A Survey on Federated Fine-Tuning of Large Language Models
Authors: Yebo Wu
Chunlin Tian
Jingguang Li
He Sun
KaHou Tam
Zhanting Zhou
Haicheng Liao
Jing Xiong
Zhijiang Guo
Li Li
Cheng-zhong Xu
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have demonstrated impressive success across various tasks. Integrating LLMs with Federated Learning (FL), a paradigm known as FedLLM, offers a promising avenue for collaborative model adaptation while preserving data privacy. This survey provides a systematic and comprehensive review of FedLLM. We begin by tracing the historical development of both LLMs and FL, summarizing relevant prior research to set the context. Subsequently, we delve into an in-depth analysis of the fundamental challenges inherent in deploying FedLLM. Addressing these challenges often requires efficient adaptation strategies; therefore, we conduct an extensive examination of existing Parameter-Efficient Fine-tuning (PEFT) methods and explore their applicability within the FL framework. To rigorously evaluate the performance of FedLLM, we undertake a thorough review of existing fine-tuning datasets and evaluation benchmarks. Furthermore, we discuss FedLLM's diverse real-world applications across multiple domains. Finally, we identify critical open challenges and outline promising research directions to foster future advancements in FedLLM. This survey aims to serve as a foundational resource for researchers and practitioners, offering valuable insights into the rapidly evolving landscape of federated fine-tuning for LLMs. It also establishes a roadmap for future innovations in privacy-preserving AI. We actively maintain a GitHub repo to track cutting-edge advancements in this field.

Kahou Tam Published in Transactions on Machine Learning Research (02/2026)
A Survey on Federated Fine-Tuning of Large Language Models
Yebo Wu
yc379266mum.coo edu.
State Key Loboratory of IOTSC, University of Macau
Chunlin Tian
yc274422 edu.
State Key Loboratory of IOTSC, University of Macau
Jingguang Li
mcc45005@rmmeedu.mo
State Key Laboratory of IOTSC, University of Macau
He Sun
hesun@um.edd edu. mo
State Key Laboratory of IOTSC, University of Macau
1..774366mmmedu.m
State Key Loboratory of IOTSC, University of Macau
Zhanting Zhou
stahou@std. uest.. edw.ccn
University of Electronic Science and Technology of China
Haicheng Liao
yc271979@mm.eeuumoo
State Key Laboratory of TOTSC, Unimersity of Macau
Jing Xiong
junesiinn@Boonneetteuukkkk
The University of Hong Kong
Zhijiang Guo"
ahfjiongguo@hhusttggseee..ttt
Hong Kong University of Science and Technology (Guangzhou)
Li Li State Key Laboratory of IOTSC, University of Macau
lGli@um.edu. edw.
Chengzhong Xu
caru@um.edu mo
State Key Laboratory of TOTSC, University of Macau
Reviewed on OpenReview: https: // opwnreview..........meme net/ forum? id= rnCqbulwnn
Abstract
Large Language Models (LLMs) have demonstrated impressive success across various tasks. Integrating LLMs with Federated Learning (FL), paradigm known as FedLLM, offers promising avenue for collaborative model adaptation while preserving data privacy. This survey provides a systematic and comprehensive review FodLLM. We begin by tracing the historical development of both LLMs and FL, summarizing relevant prior research to set the context. Subsequently, we delve into an in-depth analysis of the fundamental challenges in- herent in deploying FedLLM. Addressing these challenges often requires efficient adaptation strategies; therefore, conduct extensive examination of existing Parameter-Effe Efficieet Fine-tuning (PFFT) methods and explore their applicability within the FL. framework. To rigorously evaluate the performance of FedLLM, we undertake a thorough review of existing fine-tuning datasets and evaluation benchmarks. Furthermore, we discuss FedLLM's diverse
Corresponding author.

---

Published in Transactions on Machine Learning Research (02/2026)
real-world applications across multiple domains. Finally, we identify critical open challenges and outline promising research directions to foster future advancements in FedLLM. This survey aims to serve as a foundational resource for researchers and practitioners, offering valuable insights into the rapidly evolving landscape of federated fine-tuning for LLMs. It also establishes a roadmap for future innovations in privacy-preserving AI. We actively maintain a Git repo to track cutting-odge advancements in this field.
1 Introduction
Large Language Models (LLMs), such GPT-4o (OpenAl 2024), DeepScek-R1 (Guo et al.) 2025 and Qwen3 (Yang et al. 2025) have exhibited extraordinary proficiency across spectrum of downstream tasks Xu et al.) 2024d 2025). These LLMs, distinguished by their ability to capture complex semantic knowledge, have established new performance benchmarks in computational linguistics (Xiong et al. 2025
2024 et al. 2026b). However, despite their impressive capabilities, LLMs cannot be directly deployed for specific downstream tasks without appropriate adaptation (Hu et al. 2021). Furthermore, training L.MSs directly on downstream task datasets presents substantial challenges. The massive scale of model parame- ters leads to significant computational overhead (Tian et al. 2025). while the scarcity of task-specific data constrains effective model training and increases the risk of overfitting. For example, training LLaMA2-65B involves processing approximately 1.4 trillion tokens, requiring 21 days of computation on 2,048 NVIDIA A100 GPUs (Touvron et al. 2023a).. Consequently, fine-tuning pre-trained LLMs has become the dominant paradigm (Dodge et al. 2020), enabling more efficient adaptation of LLMs to specific tasks while preserving their foundational knowledge acquired during pre-training
The current mainstream LLM fine-tuning paradigms can be categorized into three approaches: 1) Central- ized Fine-Tuning (as shown in Figure [ [a)): This approach aggregates local datasets from all data owners
(clients) and uploads them to a central server for fine-tuning (Zhang et al. 2023e Ding et al. 2023b). Despite its effectiveness, this approach raises significant privacy concerns (Wang et al. 2023b Ye et al. 2024a Tam let al. 2024b) and is often impractical in real-world scenarios due to legal restrictions (e.g., GDPR (Voigt & Von dem Bussche 2017)), which limit the centralization of sensitive personal data. 2) Local Fine- Tunnng
(as shown in Figure [b)): ) In this paradigm, each data owner fine-tunes the LLM locally using their private dataset. While this approach preserves data privacy, the limited size and diversity of local datasets often rosult in suboptimal model performance. For instance, models refined through local finc-tuning demonstrate substantial performance degradation of up to 7% on the MMLU benchmark (Hendrycks et al. 2020 when compared to federated fine-tuning (Ye et al.) 2024b). 3) Federated Fine- Tane-Tnigg (as shown in Figure This approach enables collaborative model improvement while preserving data privacy by allowing clients to train the model locally and only sharing model updates with the central server (Li et al. 2025 2023c).
The server aggregates these updates to construct a global model, which is subsequently redistributed Lo clients for further refinement. This method addresses both the privacy concerns of centralized fine-tuning and the limited data diversity issues of local finc-tuning, making it a promising paradigm for adapting LLMs to specific downstream tasks (Xu et al. 2023e).
Despite these benefits, federated fine-tuning encounters several unique challenges, which significantly limit the effective deployment of FedLLM in real-world scenarios: 1) Communication Overhead: LLMs typically contain billions of parameters, such as LLaMA2-7B. Therefore, uploading these massive model parameters in each training round incurs substantial communication overhead, resulting in severe communication latency and excessive bandwidth requirements LL et al. 2022a). 2) Data Heterogeneity: Data across participat- ing clients exhibits substantial variation in both quality and statistical distribution (Ning et al.) 2024). This Non-llI -od--nndeendent and Identically Distributed) nature of federated data can introduce significant biases into model updates, leading to weight divergence, slower convergence rates, and ultimately compro- mised model performance (Fu et al. 2024aa] Tian el al. 2024c). 3) Memory Wall: Participating ing clients, especially edge devices, generally possess limited available memory resources (Wu et al. 2024h Zhan et al. 2024 Li et al.) 2023g] Wu et al.) (202c)), which insufficiently support memory-intensive LLM fino-tuning. This memory wall fundamentally limits clients' effective participation in the federated fine-tuning process, preventing them from contributing valuable data to the global model and ultimately compromising model
2

---

Published in Transactions on Machine Learning Research (02/2020)
80608 B6BIT
0B:68 8EBIT
Data Owner
</>
Data Owner
DO
Model Owner
Break data privacy
Lack of data scale and diversity
Data Owner
Data Owner
(a) Clients upload data to server for fine-tuning,
(b) Clients fine-tune LLM locally.
CE>
New Challenges
CO SEB
Communication Overhead
Data Heterogeneity
Memory Wall
00O00
Computation Overhead
BBOTT
(c) Federated fine-tuning of LLMs and its new challenges.
Figure Illustration of three LLM fine-tuning paradigms: (a) Centralized Fine-tuning, where data is aggregated at central server; (b) Local Fine-tuning, where models are trained independently private datasets; and (c) Federated Fine-tuning, where data remains local, and model updates are aggregated by central server to create global model.
performance. 4) Computation Overhead: The hardware processing capabilities of participating clients are often limited (Wang et al. 2019a), making it challenging to meet the high computational demands of fine- tuning LLMs. This compulational bottleneck substantially increases local training time and consequently prolongs the overall process. The extended training cycles reduce system efficiency and significantly increase energy consumption on resourco-constrainee devicos, potentially deterring client participation.
To address these challenges, researchers have applied various Parameter- Efficieer Vine-Tuning (PEFT) meth- ods to FL, which can be broadly classified into five main categories: LoRA-based tuning (Hu et al. 2021 Tian
ct al.
2024b
Wu et al. 2025b); prompt-based tuning [Lester et al. 2021). adapter-based tuning HHuulsy
et al.
2019 selective-based tuning (Zaken et al. 2021 and other tuning methods (Shin et al. 2023 Chen
et al.
2017]. The core idea behind these methods is to minimize the number of trainable parameters by
focusing on small, task-specific adjustments rather than fine-tuning the entire model. By updating only a subset of parameters or modifying model inputs (e.g., prompts), these approaches significantly reduce the communication overhead, computational burden, and memory usage of model fine-tuning, all while main- taining the performance of LLMs across diverse tasks.
Prior Surveys. Despite the continuous development of innovative federated fine-tuning methods, signifi- cant gap remains in the comprehensive evaluation and comparison of these techniques. While existing survess on FI. provide valuable insights, they are typically limited to either traditional small-model FL settings or fail to offer a detailed analysis and evaluation benchmark specifically for federated fine-tuning of LLMs. Concurrently, although several surveys on PEFT have been proposed, these works predominantly focus on centralized finc-tuning scenarios, overlooking the unique challenges that arise when adapting such techniques to distributed, privacy-prece presseving settings. detailed comparison of existing surveys is summarized in Ta- ble 1 To fill this gap, our survey aims to be the first to present a systematic examination of federated
3

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Overview of related surveys. This comparison highlights whether each work addresses data privacy, aligns with the scope of LLMs, emphasizes efficiency, proposes evaluation benchmarks, and discusses applications and future directions.
Prior Surveys
Privacy
LLM
Efficiency
Beuchmark
Application Future Direction
Xu
2024e
Ziao
B123b
Gao ei m
2b
X
08 m.
2244
X
Zhemg
V
X
Xin et al.
2121
Huang et al. 2024b
X
X
X
Yc 1S al.
2123
X
Jiany
20/44b
hen
2024
hai et a.
21
X
Foag et m
217235
X
Zhangg
(1124]
1/V
X
gl
2021e
Zhiung 05 al.
2023
X
Voiselschhlagg el ml.
2024
7 X
fine-tuning for LLMs, providing a thorough understanding of their evolution, effectiveness, and practical plementation challenges, along with standardized evaluation benchmarks that enable fair comparison across different approaches. This thorough analysis serves foundation for researchers and practitioners secking to navigate the rapidly evolving landscape of federated LLM fine-tuning.
Contribution. This paper presents comprehensive survey on the federated fine-tuning of LLMs. In contrast to existing surveys, the main contributions of this work can be summarized as follows:
We provide an exhaustive review of all relevant papers on federated fine-tuning up to date, offering extensive analysis of the state-of-the-art techniques and their evolution in this field.
2. We conduct detailed analysis of the key challenges in federated finc-tuning and propose sys- tematic taxonomy based on different fine-tuning approaches, including LoRA-based, prompt-based, adapter-based, selective-based, and other emerging tuning methods. We further provide an in-depth discussion of the advantages, limitations, and applicability of these methods.
3. We establish a comprehensive evaluation framework for federated fine-tuning of LLMs, encompass- ing fine-tuning datasets and evaluation benchmarks across diverse domains, while systematically analyzing and discussing diverse real-world application scenarios.
4. Finally, we outline promising rescarch directions in FedLLM, aiming to guide future investigations loward more efficient, scalable, and privacy-preseriing solutions that bridge the gap between theo- retical advances and practical deployments in resource-constrained federated environments.
Figure Fillustrates the organizational structure of this survey. Section 2 introduces the relevant background and fundamental concepts of LLMs and federated fine-tuning, Section systematically examines the tech- nical challenges and inherent limitations in the federated fine-tuning of LLMs. In Section we present a comprehensive review of stale-of-the-arl federated fine-tuning techniques and methodologies. Section 5 presents representative fine-tuning datasets and evaluation benchmarks across various domains, specifically curated to assess the performance of federated fine-tuning in diverse scenarios. Section 0 explores practical applications of FedLLM. Section T outlines promising research directions, while Section synthesizes key insights from this survey to inform and guide future research in this rapidly evolving field.

---

Published in Transactions on Machine Learning Research (02/2026)
Open Challenges Future tions
3 Challenges
Modal Sacurity Faculo)
Introduction
Communcattonn Ouprread
Benchmarks
SLM Collasonen
(ata heterupenely
Mut "Moud POSEL
Parsryy Well
Dalosets
Cantor
FedLLM
Conclusion
Train ei LLMS
tesses furn
Led cmadd Reseerce
Adaeter-brisel furing
Firance
2 Background
Other Turing Hotheds
Application tion
Federated Fine-tuning
Figure 2: Overall structure of the survey.
2 Background
2.1 Large Language Models
Large Language Models (LLMs) have demonstrated unprecedented capa- bilities across a wide range of natural language processing tasks, including machine translation (Wang et al. 2022a). y lext gemeration (Yu et al. 2022), sentiment analysis [Wankhade et al. 2022), and question answering (Zhu iet al. 2021). Their exceptional performance stems from their remarkable ability to encode complex linguistic patterns, capture long-range contex- tual dependencies, and learn rich semantic representations (Xiong al., 2023 Xu et al., 2026a). These capabilities have not only enabled LLMs to achieve state-of-the-arl results on broad spectrum of academic bench- marks, but have also fueled transformative advances in real-world applica- tions such as conversational AI, legal document analysis, medical decision support, and automated content generation.
Oulput
Faod-
Transoome: Layer
Transommrr Layer
Normalzation
Trassuums: Laver
Muls-Heaa
Altentian
Transfommer Layer
Layer Normalzztiion
Input
Figure 3: Architecture of LLMs.
Architecturally, modern LLMs are typically constructed by stacking dozens or even hundreds of transformer layers, where each layer incrementally refines the input through deep contextualization and abstraction. For example, LLaMA2-7B comprises 32 transformer layers stacked sequentially to capture hierarchical linguistic features. This deep, layered architecture enables the model to effectively integrate both local and global contextual information over long sequences, which is essential for complex language understanding tasks. Figure 3 illustrates the schematic structure prototypical LLM, where transformer layers are arranged in a vertically stacked fashion. Each transformer layer consists of two fundamental components: Multi-Head Attention (MHA) and Ferd-Forward Network (FFN), Formally, the input to the 1-th transformer layer is denoted 1 Raxd, where is the sequence length and d is the hidden dimension of the model. The computational process within the [-th layer can be expressed as follows:
A MHA(LN((s- + h;-1,
(1)
h, = (F)(L)())) + h2
(2)
where LN(-) represents layer normalization, which stabilizes the training dynamics by standardizing the activations, and h2 denotes the intermediate activations after being processed by the MHA module.
2.2 Training of LLMs
The training of LLMs encompasses two distinct stages (Xin et al. 2024]: : pre-training and fino-tuning, as illustrated in Figure 4 1) Pre-training involves training the model massive unlabeled text corpora
5

---

Published in Transactions on Machine Learning Research (02/2026)
Pre-train dataset
Frozen
Medical
Law
Fine-tune dataset
Tunable
Financial
Raw Model
Auto-regressive Pre-train
Supervised Fine-tune
Figure 4: Schematic illustration of the two-stage LLM training process: 1) auto-regressive pre-training on large-scale corpora to develop general linguistic capabilities, followed by 2) supervised fine-tuning to align model outputs with specific task requirements or human preferences.
(billions to trillions of tokens) drawn from diverse sources such as academic papers, websites, and books. This stage gencrally adopts the auto-rogressive modeling (Yang et al., 2019 approach that predicts cach token based on its previous context. Through extensive training, LLMs develop a wide range of capabilities, from basic semantic understanding to advanced reasoning across diverse domains. While computationally intensive, this unsupervised learning process builds robust and transferable representations that serve as powerful foundation for various downstream tasks (Naveed et al. 2023
2) The function of fine-tuning is to adapt the pre-trained model to specific downstream tasks through additional training on task-specific datasets (Ding et al.) 2023b). This stage typically utilizes supervised learning to optimize the model's performance for particular applications. While fine-tuning is highly effective at specializing the model's general language understanding for targeted tasks, traditional finetuning methods often require contralizing data from various sourees on a central server (Huang al., (025). which raises significant, privacy and security concerns. These concerns have sparked growing interest in privacy-preservicn fine-tuning paradigms, which seek to retain the benefits of model specialization while ensuring that sensitive user data remains decentralized and sccure throughout the training process.
2.3 Federated Fine-Tuning
Federated fine-tuning [Zhang al. 2024b Yi et al. 2025) has emerged promising paradigm for adapting LLMs to specific downstream tasks while preserving data privacy. Unlike conventional centralized approaches that require sensitive data aggregation at central server, federated fine-tuning enables distributed clients to adapt LLMs on local private datasets, sharing only model updates with the coordinating server. This privacy-preserving approach aligns well with modern data protection requirements and user expectations. However, the massive scale of LLM parameters and heterogeneous data distributions across clients intro- duce significant technical challenges. These challenges encompass prohibitive communication bandwidth requirements for transmituing model updates, convergence difficulties when training across heterogeneous data distributions, excessive memory demands that strain client-side resources, and intensive computational overhead that impacts efficiency and energy consumption. To address these challenges, researchers have pro- posed various parameter-efficient federated fine-tuning approaches, each strategically designed to mitigate resource constraints while maintaining model performance on downstreem tasks. These innovative methods can broadly categorized as follows:
1) LoRA-basee Tuning (Hu al. 2021): This methodology leverages the intrinsic low-rank nature of weight updates by decomposing them into low-rank approximation matrices, significantly reducing trainable parameters while preserving model's expressiveness.
2) Prompt-based Tuning [Lester et al. 2021 : This approach optimizes continuous or discrete
prompts in the input space steer the model's behavior toward specific tasks. By modifying only the prompt embeddings while keeping model weight frozen, it achieves remarkable parameter efficiency in task adaptation.

---

Published in Transactions on Machine Learning Research (02/2026)
3) Adapter-based Tuning (Houlsby al. 2019): This strategy incorporates specialized adapter modules between the layers of the pre-trained model. By updating only these compact adapters while freezing the original model parameters, it enables efficient task-specific adaptation with minimal architectural modifications to the base model.
4) Selective-based Tuning (Zken al, 2021): This approach focuses on selectively fine-tuning specific layers or parameters of the model that are most relevant to the downstream task. Through careful selection, it significantly reduces the resource consumplion.
5) Other Tuning Methods (Li ct al. 2024ch: This category encompasses techniques like zeroth-order optimization (Malladi et al. 2023), split learning (Thapa et al. 2022), model compression (Deng eet al.) p020), and data selection (Qin al. 2024), which offer innovative ways to optimize LLM performance with lower resource requirements
3 Challenges
this section, we provide an in-depth analysis of the challenges encountered in FedLLM, focusing on four communication overhead, data hetcrogencity, memory constraints, and computation burden.
key aspects:
3.1 Communication Overhead
In federated fine-tuning, the learning process necessitates iterative communication between participating clients and the central server, where clients periodically transmit their locally updated model parameters for aggregation (Fu et al. 2022: 2024b This iterative exchange continues until model convergence, inherently introducing substantial communication overhead (Li et al. 2022b Kou al. 2025). The challenge is even more pronounced when fine-tuning LLMs, which consist of billions of parameters. To quantify this challenge, Figure 5 presents a comparative analysis of parameter sizes across different models, contrasting traditional models like BERT (Devlin et al. 2019) with the LLaMA series, including TinyLLaMA (Zhang et al. 2024d)
LLaMA2-7B, LLaMA2-133 (Touvron et al. 2023b), LLaMA3-3B, and LLaMA3-SB (Dubey et al. 2024 Our analysis reveals that LLaMA models are dramatically larger than BERT, with parameter counts ranging from 10 118x greater. This exponential increase in parameter size directly translates significantly higher data transmission volumes in each communication round, substantially elevating bandwidth requirements and overall communication costs in federated environments.
However, in real-world scenarios, communication bandwidth is often severely constrained. According to a 2023 Cisco report, approximately 30% edge devices still rely on 2G or net- works, which provide bandwidths of less than 10 Mb/s (Wang let al.] 2023c). While 5G networks offer speeds more than 50x faster, they accessible to only about 10% of devices. This significant. disparity in network capabilities inevitably results in substantial communication delays, particularly when exchang- ing large parameter updates. More critically, the duration of each training round determined by the slowest device the network-a phenomenon known the "straggler effect." This means that devices with limited connectivity can dramatically hinder the convergence speed of the federated fine-tuning pro- cess. Consequently, minimizing communication overhead be- comes essential for effective FodLLM implementation. Effi- cient management of data transmission can accelerate model convergence while ensuring the practical feasibility of deploy- ing FedLLM bandwidth-connsrrinde environments. Without addressing the communication challenge, the theoretical privacy benefits of federated fine-tuning may remain inaccessible many real-world applications, particularly those involving edge devices regions with limited network infrastructure
15
1118.18
Parameters (B)
10
X72.73
x63.64
x27.27
x1
BERT
TinyLLaMA
LLaMA3-3B
LLAMA2-7B
LLAMA3-BB
LLAMA2-138
Figure 5: Comparison of model parameters
across BERT and LLaMA series models.

---

Published in Transactions on Machine Learning Research (02/2026)
3.2
Data Heterogeneity
Data heterogeneity is notorious challenge in FL, manifest- ing in significant variations in data distribution (Tian et al.
Model Performance
2022a quality (Tam et al., 2023a and quantity (Yi et al.
2022 across clients. Such helerogeneity hinders convergence
and degrades the global model's generalization ability, as it must reconcile conflicting updates derived from diverse client populations (Ma et al., 2024aa] Wang el al. 2025). To mitigale the adverse effects of data heterogeneity, various strategies have been explored in traditional FL, which can be broadly catego- rized into four groups: 1) Regularization-base
Increased 6ap
FFT
PEFT
methods incorporate additional penalty terms into the local objective to limit model divergence and encourage alignment with the meth-
Data Heterogeneity
Figure 6: Impact of data heterogencity on model performance. As the degree of data heterogeneity increases, the performance gap between PEFT and FFT widens.
global model (Li et al., b0 ; 2) Aggregation- ods modify the server-side aggregation strategy to assign adap- tive weights to client updates, reducing the influence of noisy, unreliable, biased data sources (Wang et al. 2020a bb; 3)
Data-sharing methods introduce small, carefully curated auxiliary datasets that are distributed to clients to promote distributional aligument and reduce inter-client drift (Goetz & Tewari 2020); 4) Personalized
FL approaches aim to address data heterogeneity by learning models that capture the unique characteristics of local data. Rather than converging lo a single global model, these methods may produce multiple global
models that are individually personalized for each client (Kou et al.) 2024 Sabah al. 2024). While these tochniques have demonstrated success in traditional FL scenarios, addressing data hetcrogeneity in FedLLM remains largely underexplored. This challenge is further exacerbated when applying PEFT techniques, PEFT tends to be more sensitive to distributional shifts and limited data availability. Figure i shows that the performance gap between PEFT and full-parameter fine-tuning (FFT) grows wider as data heterogeneity increases, underscoring the need for targeted solutions to improve PEFT robustness federated settings.
3.3 Memory Wall
Memory constraints present a fundamental challenge to the practical deployment of federated finettuing (Wu al.
2025d During the local finc-tuning process, model parameters, intermediate activations, and gradients must be stored in memory, resulting in substantial memory consumption. However, participating clients, especially edge devices, typically have limited available memory, ranging from 4 to 12 GB (Tian et al. 2024a
Tam et al., 2024a). This limited memory capacity is insufficient to support fine-tuning mainstream LLMs.
To better quantify this challenge, we profile the memory usage during full-parameter fine-tuning for both traditional models (e.g., DistilBERT (Sanh al.] 2020), BERT (Devlin et al.
100
Activation
98.56
Memory Usage (GB)
Gradient
2019h) and LLaMA-series models (e.g., TinyLLaMA, LLaMA2- 7B, L.L.MA-1333 133)). As shown in Figure 7 our results reveal a dramatic disparity in resource requirements between these model families. Fine-tuning LLaMA models demands sub- stantially higher memory resources compared to traditional ar- chitectures. Specifically, fine-tuning LLaMA2-7B requires ap- proximately 51.85 GB of GPU memory, which 7.68x more than BERT (6.75 GB). This requirement escalates further with ALaMA2-13BB which demands 98.56 GB of memory, represent- ing 28.32x increase over DistilBERT and vastly exceeding the available memory capacity of edge devices. This stark mis- match between the memory demands fine-tuning LLMs and the hardware limitations of participants creates a Memory Wall, a fundamental barrier that severely restricts the feasi- bility of deploying FedLLM at scale. This memory constraint.
80
Parümeter
60
40
20
3.48
DisulBERT
BERT
TinyLLaMA
LLMMA2-7B
LLAMA2-138
Figure 7: Memory usage breakdown for fine-tuning various models. The analysis is conducted on an NVIDIA H800 with preci- sions of FP32 and FP16, a batch size of 16, and a maximum sequence length of 512.
8

---

Published in Transactions on Machine Learning Research (02/2026)
prevents significant proportion of devices from participating in the collaborative learning process, thereby compromising model performance through reduced data diversity and limiting the practical application scope of FedLLM. The memory wall represents not just a technical challenge but a fundamental constraint on democratizing access to advanced AI capabilities through FL.
3.4 Computation Overhead
Computational cost presents another major bottleneck in de-
ploying FedLLM (Tian et al., 2022a Almanifi al. 2023). The
1200
computational demands of fine-tuming LLMs arise from the for- ward and backward passes during local training iterations, each contributing significantly to the overall processing burden. The sheer scale of these models characterized by billions of pa- rameters, numerous transformer layers, and complex attention mechanisms makes them inherently compute-intensive, which quickly overwhelm devices with limited processing capa- bilities. Moreover, the iterative nature of fine-tuning, which involves repeated forward and backward passes, compounds the computational load, making it difficult to achieve efficient training on resource- resource-constraiingg constrained devices. To quantitatively un- derstand this challenge, we profile the computational demands of fine-tuning various models by measuring the floating-poin operations (FLOPs) required for single forward and backward pass with batch size of 16.
TFLOP
800
163.52
x55.47
400
x8.60
0 x1
BERT
TinyLLaMA
LLaMA3-38
LLaMA2-7B
LLaMA3-88
LLaMA2-138 138
Figure 8: Comparison of FLOPs for single
forward and backward pass across models.
Specifically, we evaluate BERT alongside suite of LLaMA-based models, including TinyLLaMA, LLaMA3- 3B, LLaMA2-7B, LLaMA3-SB, and LLaMA2-138 2-BBB. As shown in Figure g our results reveal a dramatic escalation in computations ional requirements for LLaMA-series models compared to BERT. For instance, fine- tuning TinyLLaMA incurs 8.60 FLOPs of BERT, while LLaMA2-13B demands a staggering 103.07x more FLOPs. This exponential increase in computational complexity directly results in significantly longer training time, excessive energy consumption baltery-powered devices, and thermal management issues, all of which degrade hardware performance
time, thereby undermining the feasibility of large-scale, real-world deployment (Ning et al. 2024 Tian et al. 2023). These findings highlight the pressing need for computation- efficient fine-tuning strategies that can effectively accommodate the heterogeneous and resource-constrainel nature of participating devices, while ensuring model performance.
4 Federated Fine- Munnnn
In this section, we roduce various parameter-efficien fine-tuning methods and discuss their applications in FL. Figure provides an overview of representative parameter-efficient federated fine-tuning methods.
Adapter Turing
Adapter-based Tuning
Personalized Adapter Tuning
Hamogeneous LoRA
Multi-Damain Adapter Tuning
Heterogeneous
LoRA-based Tuning
Personalized LoRA-
Federated Fine-Tuning
Bias Tuning
Selective- Tuning
Partial Tuning
General Prompt Turing
Personalized Tuning-
Prompt-based Tuning
Zeroth-Order Optimization
Mutti-Domain Prompt Tuning
Split Learning
Other Tuning Methods
Model Compression
Data Selection
Figure 9: Overview of parameterr-fffcient federated fine-Luning methods and their corresponding laxonomy.
9

---

Published in Transactions on Machine Learning Research (02/2026)
4.1
LoRA-based Tuning
4.1.1
Preliminary
Low-Rank Adaplation (LoRA) (Hu et al. 2021 Tian el al. 2024b) has emerged as a promising approach for efficient fine- tuning of LLMs while maintaining model performance. The core idea of LoRA lies in introducing low-rank matrices into the pre-trained model's weights, allowing for the adaptation of model parameters without altering the original architecture sig- nificantly. LoRA is based on the observation that fine-tuning does not require updating the full parameter space; instead, meaningful adaptations can often be represented in a low- dimensional subspace. By applying low-rank decomposition to the weight updates, LoRA drastically reduces the number of trainable parameters, leading to lower resource consumption. Furthermore, LoRA's ' modular design allows it to be easily integrated into a variety of model architectures without altering the original model.
Training
+ Freezing
Output
Transfemerr
Layer
Pretrained W Weights
R-0
Transformer
Wy -
NN 09.0%
Laver
Input
Figure 10: The working principle of LoRA.
Figure 10 illustrates the working principle of LoRA. Specifically, the pre-trained parameter matrix Wo E U is decomposed into two matrices, A E RrxV and B E RR'' where r < min(U, V) denotes the rank controlling the dimensionality of the low-rank subspace. The matrix A projects the input into a low- dimensional space, and the matrix B maps it back to the original space. During fine-tuning, only A and B are updated, while the original weights Wo remain frozen. The imput data X is processed by both Wo and BA. The output of WoX is the initial prediction generated by the pre-trained model, while the output of BAX represents the lask-specific adaptation roduced by the low-rank matrices. These two oulputs are then added element-wise to produce the final output. This process can be formulated as:
h = Wox + BAX
(3)
By updating only A and B, LoRA enables efficient task adaptation with minimal resource overhcad, effec- lively capluring lask-specific knowledge while preserving the generalization ability of the pre-trained model.
4.1.2 LoRA in Federated Fine- Mun-Tuning
In the context of federated fine-tuning, LoRA offers notable advantages in both communication and compu- tation efliciency. Since only the low-rank matrices are updated and transmilled, rather than the full model parameters, this lightweight updating mechanism significantly lowers bandwidth requirements and facilitates faster convergence of the global model, making LoRA particularly well-suited for foderated environments. In this paper, we introduce a novel taxonomy of LoRA-based federated fine-tuning methods, categorizing them into three primary types: Homogeneous LoRA, where all clients adopt the same rank; Heterogeneous LoRA, where clients use different ranks based on their resources; and Personalized LoRA, which tailors low- rank adaptations to individual client data distributions. This taxonomy is summarized in Figure g In the following sections, we delve into representative methods within each category, analyzing how they address the key challenges identified in Section 3 beyond the inherent benefits brought by LoRA itself.
Homogeneous LoRA refers to scenarios where all clients adopt the same low-rank dimension r for their LoRA modules. This uniform configuration simplifies aggregation and model synchronization across clients. Table summarizes representative methods in this category and the specific challenges they address.
FedIT (Zhang et al.) 2024b directly integrates LoRA into the classic FedAvg (McMahan al. 2017) for instruction tuning. FedSA-LoRA (Guo et al. 2024b) identifies that the A matrices primarily encode general knowledge, while the B matrices capture client-specific features; thus, it only uploads A to the server, (Kuang et. al., 2023) establishes
significantly reducing communication overhead. FederatedScope--LL. a comprehensive end-to-end pipeline for federated LLM fine-tuning and proposes offsite- offsite-tuning strategies to mitigate both communication and computational costs. FeDeRA (Yan et al.) (024) addresses data heterogeneity by initializing LoRA matrices via singular value decomposition on the pre-trained weights. LoRA-FAIR (Bian et al.] 2024) introduces correction mechanism the server to handle aggregation
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Homogeneous LoRA in federated fine-tuning.
Challenge
Method
Communication
Non-IID
Memory
Computation
FedIT (Zhang
2024b
FedSA-LoRA Cuo
212/b
RederaterdSoope LLM
mann al.
2025
FeDeRA Yan et al.
2.24
V
X
LoRA-FAIK Bian al.
2024
X
X
FLASC Kuo al. 2024
X
X
SA-Feel.nnh
.RA
Yang
2024c
V
X
SLoRt Babaknniya et al
2025
X
V
X
RoLoRA
Clan el al. 2124d
V
X
FedPipe
Fang ci al 2024b
X
J
Lp-FL
[aag et aL. 2023
X
X
X
Ped-pil.ot [Zhang et al.
2024]
x
X
FedRA
Su cf al 2023
PedPrumer Wu
2025c
DevFT
Wh et al.
2025b
Table Heterogeneous LoRA in federated fine-tuning.
Challenge
Method
Communication
Non-IID
Memory
Computation
HETLoRA (Coo et al.
2024
FLoRA Wang al.
2024T
X
FlexLoRA Bai et
20244
LoRA-A²
Koo et al.
2124
Byun & Lee 2024
X
FedlIM et 2021
X
RBLA Tavallaie & Nazemi
Smart Fed
(Wu et.
2025a
X
bias and initialization drift across clients. FLASC Kuo et al. 2024 incorporates sparsity into LoRA to
further reduce communication overhead. SA-FedLoRA (Yang al. 2024c) mitigates client drift through parameter regularization and dynamically allocates communication budgets.
SLoRA [Babakniya et al. 2023 proposes novel data-driven initialization scheme to better handle statis- tical heterogeneity. RoLoRA (Chen el al.) 2024d) adopts alternating minimization approach to improve robustness under non-lII conditions. FedPipe (Fang et al. (024b) automatically selects critical parameters for fine-tuning and applics quantization to reduce memory usage. LP-FL (Jiang et al.) 2023) applics LoRA directly enable efficient on-device fine-tuning. Fed-piLot (Zhang et al., 20241) reduces memory con- sumption through LoRA assignment strategies and introduces a novel spatial-tempon
aggregation (STAgg) rule to address heterogeneity. FedRA (Su et al., (203)) adaptively determines parameter update scopes based client resource constraints, effectively reducing computational, communication, and memory costs. FedPruner (Wu el al. 2025c) introduces macro-micro synergetic pruning framework to mitigate memory constraints on participating devices. To further facilitate large-scale deployment, DevFT (Wu et al. 2025b proposes a developmental federated tuning paradigm to minimize training resource overheads.
Heterogeneous LoRA allows clients to adopt different. rank values r based on their data characteristics or resource constraints. This heterogencity can manifest cither across clients (intcr-model) or within different layers of the same model (intra-mode).. By enabling each client to select. a rank that best. fits its capabilities and local data, this approach introduces greater flexibility and resource-awarent into the federated fine- tuning procoss. Table summarizes representative methods and the specific challenges they address.
HETLoRA (Cho el al. 2024h assigns helerogeneous ranks across devices and incorporates rank self-pruning along with sparsity-weite weighted aggregation to tackle data heterogeneity. FLoRA (Wang et al. 2024f pro- poses stacking- basod aggregation scheme and allows devices select ranks according their resource budgets. FlexLoRA (Bai al., (2024a) enables dynamic adjustment of local LoRA ranks leverage the heterogeneous device resources, while employing singular value decomposition for weight redistribution.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Personalized LoRA in federated fine-tuning.
Challenge
Method
Communication
Non-IID
Memory
Computation
FDLoRA [Qi et al.
2024a
pFedLoRA Y1 ct a.
2023
FedLoRA [Wu et al.
20241
X
FedDPA Y
Yang et al.
2024b
X
PerFIT Zhang
2024c
FedMEM
DD al.
2024
FedAMoLLE (Zhang et al.
20241
X
LoRA-A Koo et al.
2024) introduces alternating freezing and adaptive rank selection mechanisms to
fully utilize heterogencous device resources while addressing statistical heterogeneity. [yunn & Lee (2024)
propose a replication-based padding technique to enable aggregation across clients with varying LoRA ranks.
FEDHM (Yao et al. 2021) addresses resource constraints by distributing low-rank models with heteroge- ncous capacities to clients. RBLA (Tavallaïe Nazemi-) improves aggregation robustness by simultancously maintaining and aligning both low-rank and high-rank feature components. Smart (Wu et al. 2025a)
achieves efficient task adaptation by leveraging rank-wise reconfiguratio of existing LoRA modules, thereby avoiding the prohibitive computational overhead of training from scratch.
Personalized LoRA enables each participant to fine-tune its model using personalized low-rank adap- lation matrices, allowing for better alignment with local data characteristics. This approach enhances the ability of the global model to generalize across clients while retaining -lient-specific nuances. Table sun- marizes representative methods and the specific challenges they aim to address.
FDLoRA (Qi al. 2024a] introduces dual LoRA modules on each client to separately capture global and personalized knowledge. DFedLoRA (Yi et al. 2023) designs homogeneous small adapter to facilitate federated clients' heterogencous local model training, with proposed iterative training process for global- local knowledge exchange. FedLoRA (Wu et al. 2024f) maintains shared general knowledge in global full-rank matrix while encoding client-specific knowledge in a. personalized low-rank module. FedDPA (Yang et al. 2024b) utilizes a global adapter and a local adapter to jointly address test-time distribution shifts and client-specific personalization. PerFIT (Zhang et al., 2024eh allows each client to search for a personalized architecture by expanding the trainable parameter space of the global model address data heterogeneity,
FEDMEM (Du et al. 2024 equips the global model with KNN classifier that captures elient-spocific distributional shifts, achieving personalization and overcoming dala heterogeneily. FedAMoLE (Zhang) et al. 2024i features a mixture of LoRA experts module for aggregating heterogeneous models and reverse sclection-based expert assignment stratogy that optimizes model architectures based on data distributions.
4.2
Prompt-based Tuning
4.2.1
Preliminary
Prompt-based tuning (Lester et al. 2021 has emerged as a highly effective and resource-efficient alternative to conventional fine-tuning approaches for LLMs. Unlike traditional methods that update the model's param- eters directly, prompt-based tuning learns a set of trainable prompts or input embeddings that steer the model's behavior on downstream tasks. By modifying only the input space, this approach leverages the pre- trained knowledge of LLMs without altering their weights. As illustrated in Figure 11 sequence of trainable prompt embeddings P Rloxd is prepended to the original input tokens X E Rlxxd, where Ip and I, de- note the lengths of the prompt and input sequences, respectively, and represents the model's hidden dimension. The concatenated sequence then fed into the frozen model: Z ([P:X];)) where f(::) denotes the pre-trained L.M with frozen parameters 0, and [P; X] represents the
Transformer
Laver 2
Transformer
Layer I
Input
Trainable Prompts
Figure 11: The working princi- ple of prompt tuning.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: General prompt tuning in federated fine-tuning.
Challenge
Method
Communication
Non-lID
Memory
Computation
MetePFT. Chen et al.
2023
X
X
X
PromptF Guo et a..
2023c
X
X
FedBPT Sun el al.
2023
PedPepTAAO Che
2023
X
Fed-BBPT (Lin
RedTPC QQuu ct al.
20033
X
FedPR |Feng el al.
2023a
X
Red-CPrompt (Bagwe et a.
2023
X
Fedprompt Zhan et al.
2023a
X
X
X
FedSP (Dong et al.
2023a
X
HePCo [Halbe et al.
2023
X
PFL-GCN Ahhmad
2023
X
X
AUG-FedPrommy prompt
2023b
X
Liu ct al.
(2023c
FedHIPL Ma et
2024h
X
PFPT
WWeng el. al.
x
#CILPTT
Liu et al.
2023c
X
X
CaFPT
Guo et al.
2024c
X
FedPoD
(Chen et
2023b
X
concatenation of the trainable prompts and the original input tokens. By optimizing the prompt embeddings P, the model can effectively adapt to new tasks while reusing its pre-trained knowledge. This approach en- ables efficient task adaptation without modifying the model weights, thereby significantly reducing memory and computational overhead. Through prompt-basod tuning, task-specific guidance is embedded within the prompls, allowing the model lo generate desired outputs by atlending to relevant information stored in its pre-trained parameters.
4.2.2 Prompt in Federated Fine- Tuning
In the context of federated fine-tuning, prompt- basptt-asedd tuming offers significant advantages in communication efficiency and model adaptability. Since only the trainable prompt embeddings are updated and exchanged, rather than model parameters, this approach substantially reduces communication overhead between clients and the central server. Additionally, by freezing the base model, prompt-based luning allows clients with heterogeneous data distributions to personalize their behavior effectively, while still benefiting from glob- ally shared knowlodge encoded in the pre-trained model. These propertics make prompt-based tuning a compelling choice for federated fine-tuning. In this paper, we propose a novel laxonomy of prompt-based federated fine-tuning approaches, categorizing them into three primary types: General Prompt Tuning, Personalized Prompt Tuning, and Multi-Domain Prompt Tuning, as illustrated in Figure || In the follow- ing sections, examine representative methods within each category and analyze how they address the challenges outlined in Section 3 beyond the inherent benefits brought by prompt tuning itself.
General Prompt Tuning refers to approaches in which a shared set of prompt embeddings is learned and applied uniformly across all participating clients. In this setting, the same prompts are prepended to each client's 's input sequences, providing consistent task-specific guidance and enabling the global model gencralize across diverse data sources. Table Summarizes representative methods and the specific challenges they sim to address.
MetePFL (Chen et al. 2023a applies prompt tuning to fine-tune spatio-temporal Transformer- based foundation model for weather forecasting tasks in federated setting. PromptFL (Guo et al. 2023ch adapts CLIP models for vision-language lasks in FL using prompt-based tuning. FedBPT (Snn el al. 2023) employs prompt-based tuning to efficiently adapt black-box LLMs using gradient- free optimization, climinating the need for clients to access model parameters and requiring only forward propagation for local training. FedPepTAO :Che et al.) (2023) introduces a partial prompt tuning mechanism to reduce communication costs, along with adaptive optimization algorithm to address data heterogeneity. Fed-
13

---

Published in Transactions on Machine Learning Research (02/2026)
BBPT (Lin et al. 2023 enables clients to utilize zeroth-order optimizer locally, obviating the need for full LLM deployment, effectively reducing memory consumption and computational costs. FedTPG 'Qiu et al. 2023) learns a unified, task-aware prompt generation network conditioned on input text, improving generalization to both seen and unseon classes.
FedPR (Fnng et al. 2023a enhances federated visual prompt tuning by projecting local prompt updates into an approximate null space of the global prompt, mitigating gradient interference and improving global per- formance. Fed-CPrompt (Bagwe et al. 2023] addresses asynchronous task arrivals and heterogeneous data distributions via asynchronous prompt updates and a contrastive contimual learning loss. FedPrompt (Zhao ct al. 2023a employs split aggregation strategy, freezing the extensive parameters of LLMs and only tuning and aggregating soft prompts. FedSP (Dong et al.] 2023a) reduces computational and memory overhead by utilizing a lightweight auxiliary model for prompt learning. HePCo (Halbe et al., 2023) mitigates catas- trophic forgetting and data heterogeneity through a data-free distillation method performed in the model's latent space. PFL-GCN (Ahmad ct al.
2023 employs prompt tuning specifically for sentiment analysis.
AUG-FedPrompt (Cai et al. 2023b exploits abundant unlabeled data for data augmentation to address the issue of data scarcity. Liu et al.
(002e)) integrate self-consistency and thoougtf- prompting to improve zero-shot performance of LLMs. FedHPL (Ma et al., 2024b) introduces a global logit distillation framework to handle model heterogeneity and guide the local training process. PFPT [Weng et al.) proposes a probabilistic prompt aggregation mechanism to address data heterogeneity and imbalanced data distribu- tion. FCILPT (Liu et al.) 2023c] jointly encodes task-relevant and task-irrelevant knowledge into prompts to preserve both previous and newly learned knowledge, alleviating catastrophic forgetting. CaFPT (Guo et al. 2024ch leverages information-theorete principles to facilitates the retrieval process by conditioning examples that activate the most relevant knowledge inside pre-trained models. FedPoD (Chen et al., 2023b) employs lightweight prompts to guide frozen foundation models and introduces multi-level prompt- based communication to enable multi-source knowledge fusion and controlled optimization.
Personalized Prompt Tuning enables each client to tailor its prompt embeddings based on local data distributions and task-specific requirements. By fine-tuning prompts individually, clients can better capture local nuances and context-specific information that a one-size-fits fits--all prompt might overlook. This approach directly addresses the challenge of data heterogencity by facilitating local adaptation, while still allowing clients to benefit. from global knowledge aggregated during training. Table t summarizes represenlative methods and the specific challenges they target.
PFedPG (Yang et al. 2023al deploys a personalized prompt generator on the server to produce elient-specific visual prompis, enabling efficient adaptation of frozen backbones to diverse local data. SGPT (Deng el al., 2024) combincs generalized and personalized FL by learning a mix of shared and group-specific prompts to capture both commonalities and group-specific variations. pFedPrompt (Guo et al. 2023b) leverages the unique multimodal capabilities of lision-language models by learning client consensus in the linuistic space and adapting to client characteristies in the visual space in non-parametric manner. FedOTP (Li et al., 2024ch introduces efficient collaborative prompt lcarning strategies to capture diverse category traits on a per-client basis. pFedPT (Li et al., 2023b) utilizes personalized visual prompls to implicitly represent local data distribution information and provides this information to the aggregation model to enhance classification tasks. FedMGP (Yu et al. 2024) uses corss--gaanee global prompts for shared knowledge and fino-grained local prompts for personalization, and introduces a selective fusion mechanism for prompt aggregation.
FedLPPA [Lin et al. 2024) jointly learns personalized prompts and aggregation strategies for weakly- supervised medical image segmentation. FedPGP (Cui et al. 2024) employs pro-trained CLIP to provide knovledge-uuidncee for the global prompt, enhancing generalization while incorporating a low-rank adapta- tion term to personalize the global prompt.. FedPFT (Wn et al. 2024g) addresses feature-classffie mismatch through prompt-driven feature transformation. Wang et al. (2024c) propose a discrete local search strategy compression method inspired by linear word analogies, sub-
for gradient- local training and a stantially reducing resource costs. PFedMoAP (Luo al. 2024) introduces personalized prompt learning framework based on the mixture-of-oxperts paradigm (Cai ct al. 2024a.. CP2GFed (Gao et al. 2024a) introduces cross-granularity knowledge transfer mechanism and dynamic personalized prompt generation to improve model performance.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Personalized prompt tuning in federated fine-tuning.
Challenge
Method
Communication
Non-IID
Memory
Computation
PFedPG (Yang
2023a
SGPT (Deng et al.
2024
X
PFedPrompt. Gio et al.
2023b
X
FedOTP [i et al.
2024c
X
V
pFedPT
Li ct
2023b
X
V
FedMGP
ot ai.
2024
X
FedLPPA
Lin et al.
21224
X
FedPGP
Cuu a..
2024
X
FedPFT
Wu et
2024g
X
Wang et
2024e
pFedMoAP Luo et
2024
X
CPd
Gao cl al.
2024a
X
X
Table 7: Multi-domain prompt tuning in federated fine-tuning.
Challenge
Method
Communicttion
Non-III) Memory
Computation
DiPrompT Bai et al.
2024b
X
PFCR (Guo
20248
Fed-DPT
al.
2013
FedAPT
Su et al.
2024
X
V
X
X
Zhao et al.
2024b
X
V
X
FedDG Gong el
2024
V
X
CP-Prompt
Feng
al.
2024b
/
Prompt Tuning extends the prompt-based approach to environments where federated clients operate across distinct. domains or application contexts. In such scenarios, each client is equipped with domain-specifie prompt embeddings that adapt the shared global model diverse contextual and distributional conditions. This approach enhances the model's generalization ability across heterogencous domains while maintaining shared global foundation. It is particularly valuable in real-world deployments spanning multiple industries or task categories. Table summarizes representative methods and the specific challenges they address.
DiPrompT Bai et al. 2024b proposes a distributed domain generalization approach using adaptive prompts, introducing global prompts for shared knowledge and domain prompts for domain-specific adapta- tion. PFCR (Guo et al. 2024a) eliminates the need for data sharing via encrypted gradient updates, models items in a unified feature space using descriptive text, and facilitates cross-domain knowledge transfer through federated content representations and prompt tuning. Fed-DPT (Wei et al.
2023 leverages pre- trained vision- language model and applics dual prompt combnnnn visual and textual prompts-fo for improved domain alignment across decentralized data sources. FedAPT (Su el al. 2024) introduces a meta prompt, an adaptive network, and frozen keys to personalize prompts for each test sample, thereby enhancing multi-domain image classification. ZZhoo al. (2024b) propose language distance metric to improve data efficiency and facilitate cooss-lnggustic generalization. FedDG (Gong al.
2024) allows clients learn text and visual prompts locally while maintaining indirect alignment. via global prompts used shared ref- erence. Domain-specific prompts are exchanged among clients and selectively integrated into global prompts using lightweight iitention-basee aggregators. CP-Prompt (Feng et al., (024b) captures intra-domain knowledge by inserting personalized prompts into the multi-head attention modules and subsequently learns inter-domain representations through a shared prompting mechanism.
4.3
Adapter- based Tuning
4.3.1
Preliminary
15

---

Published in Transactions on Machine Learning Research (02/2026)
Adapter-based tuning is another parameter-cffiient alternative to full- parameter fine-tuning for LLMs (Pleiffer et. al. 2020). Tt. introduces lightweight, trainable adapter modules into the model while keeping the pre-trained weights frozen. These modules act as task-specific compo-
Adapter
former
W
Layer 2
0
nents that. Iransform intermediate representations enabling efficient adaptation downstream tasks with minimal memory and computational overhead. A standard adapter module consists of three key operations: down-projection, non-linearity, and up-projection, as shown in Figure 12 For activations h; E Raaxd the adapter transfor- mation proceeds as follows:
controlled manner,
Nonlinear
Adapter
Transformer
Wdown
Layer
Adapter
Input
Figure 12: The working princi-
1) Down-Projection: The tigh-dimensional hidden state h is projected
ple of adapter tuning.
into a low-dimensional space using a learnable weight matrix WDP Raxr, where r < controls the bottlenock size. This step roduces the number of trainable parameters while capluring essential features:
DP
(4)
2) Non-Linearity: A non-linear activation function a(.), such as ReLU or GELU, is applied to introduce expressive transformations while retaining important task-specific patterns:
h (h))
3) Up-Projection: The transformed low-dimensional representation is mapped back to the original feature space using up-projection matrix WUp R
h h WUP
(6)
4) Residual Connection: The final output of the adapter is then residually added to the original hidden state, preserving the pre-trained knowledge while incorporating task-specific adjustments:
2 = h1 + 14
where Z represents the adapted hidden representation. This residual connection ensures that the pre-trained model remains largely intact while allowing task-specific fine-tuning through the lightweight adapter layers. Notably, only the adapter parameters Wor and Wor updated during training, resulting in significantly lower memory and computation costs compared to full-parameter fine-tuning.
4.3.2 Adapter in Federated Fine-Ture Tuning
In the context of federated fine-tuning, adapter- based tuning provides significant advantages in both resource efficiency and model adaptability. Since only the lightweight adapter modules updated and exchanged, rather than the full model parameters, this approach greatly reduces computation and communication over- head. Moreover, by freezing the base model, adapter-based tuning enables clients to fine-tune efficiently on heterogeneous local data while still benefiting from the shared global knowledge encoded in the pre-trained model. This modular design facilitates the seamless integration of task-specific adaptations without compro- mising the generalization capability of the base model. To better understand the landscape of adapter-based methods in federated fine-tuning, we propose new taxonomy comprising three categorics: General Adapter Tuning, Personalized Adapter Tuning, and Multi-Domain Adapter Tuning, as illustrated in Figure g In the following sections, explore representative methods within each category and analyze how they address the core challenges identified in Section 3 beyond the inherent benefits brought by adapter itself.
General Adapter Tuning refers to scenarios in which all clients utilize a shared adapter structure with identical initialization. In this setting, the same adapter modules are inserted into the transformer layers of each client's model, enabling consistent adaptation mechanisms across the foderation. This uniformity facilitates stable aggregation and coordinated updates during federated training. Such an approach is par- ticularly effective when clients operate on similar tasks or share relatively homogeneous data distributions.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8 General, personalized, and multi-domain adapter tuning in federated fine-tuning.
Challenge
Mothod
Type
Communication
Non-IID
Memory Computation
FodAdapter
Cal al.
2023a
General
Kim et al.
20230
General
Tod
Chasvand ot al.
2024
General
/
/
C2A Kim et al. 20123b
Personalized
X
FedCLTP
2023b
Pessonalieed
Fed-MNMT al.
Multi-donaiin
AdaFedSelecKD [Fong
2021a
Multi-domain
FedDAII (Chen et al.
2024
Multi-donain
X
x
globally optimized adapter can generalize well across participants. Table summarizes representative
methods and the specific challenges they aim to address.
FedAdapter (Cai et al., 2023a) proposes a progressive adapter tuning strategy, combined with continuous device profiling, to dynamically optimize adapter configurations across clients, improving efficiency without sacrificing accuracy. [Kim et al. (2023a) leverage adapters to address the high communication costs associated with federated fine-tuning of LLMs. FedTT+ (Ghiasvand al.] 2024) integrates tensorized adapters for LLM adaptation and further improves robustness to data heterogeneity by freezing portions of the tensor factors, significantly reducing the number of trainable parameters while maintaining model performance.
Personalized Adapter Tuning enables each client to independently finc-tune its adapter modules based on its local data distribution and lask-specific requirements. In contrast to general adapter tuning, this approach does not enforce uniformity across clients; instead, it allows for the retention of personalized adapter parameters that better capture client-specific knowledge. This strategy is particularly advantagcous in federated settings characterized by high degrees of data heterogeneity. By leveraging personalized adapters, clients can achieve improved local performance while still benefiting from shared global knowledge. Table summarizes representative methods and the specific challenges they address.
C2A Kim et al. 2023b) employs a hypernetwork to generate client-specific adapters, effectively addressing data heterogeneity by enabling on-demand parameter generation tailored to each client. FedCLIP (Lu et al. 2023b) introduces an attention- based adapter design that utilizes the pre-trained model's knowledge facilitate both rapid generalization and efficient personalization while minimizing resource overhead.
Multi-Domain Adapter Tuning extends the federated fine-tuning paradigm to clients operating across distinet domains, enabling efficient adaptation to domain-specific tasks. In this setting, each client maintains its own domain-specific adapler while contributing to shared global model. The global model aggregates adapter updates across domains to capture comain-invarian representations to support generalization. This approach is particularly effective cross-domain sccnarios such as multilingual natural language processing. By decoupling domain-specific learning from the shared backbone, this strategy balances personalization and collaboration. Table summarizes representative methods and the challenges they address.
Fed-MNMT LLu et al. 2023f applics adapter-based fine-tuning for multilingual neural machine transla- tion, significantly reducing commuicattin overhead. It further explores parameter clustering strategies to mitigate conflicts during aggregation. AdaFedSSeeccKD (Feng et al.) 2024a) performs adapter-based sum- marization to minimize transmitted parameters and introduces selective knowledge distillation for efficient domain-adaptive learning. AT (Chen et al.] 2024a) proposes a dual-adapter teacher framework to regularize local updates under data heterogeneity, and employs mutual knowledge distillation for effective cross-client knowledge transfer.
4.4 selective-based Tuning
4.4.1 Preliminary
Selective-based tuming has emergod as an efficient strategy for fine-tuning LLMs by updating specific param- eters of the model while keeping the majority of pre-trained weights frozen (Kornblith et al. 2019). This
17

---

Published in Transactions on Machine Learning Research (02/2026)
approach significantly reduces computational and memory overhead compared to full-parameter fine-tuning, while maintaining the model's generalization ability. Among selective-basel tuning techniques, two widely adopted strategies are bias tuning (Zaken et al.) 2021) and partial tuning (Houlsby et al. 2019), both of which optimize only a subset of parameters rather than the entire model.
Bias tuning updates only the bias terms of the model while keeping all other parameters frozen. Despite its simplicity, this approach has demonstrated strong performance across a variety of tasks with minimal overhead. Partial tuning generalizes this idea by allowing updates to a carefully selected subset of model parameters, such layer normalization parameters, feed-forward network biases, specific attention blocks. By focusing updates on the most relevant parameters, selective-based tuning methods improve training efficiency, mitigate catastrophic forgetting, and enable rapid adaptation using limited data and resources.
4.4.2 Selective Fine- Tunnng Methods
DP-BITFIT [Bu et al. 2022) applics differentially private bias-term tuning in centralized training scenar- adaptation while reducing resource demands. FedPEFT (Sun et al. 2022) shares only a small subset of model weights, such as bias parameters, significantly reducing the commu- nication overhead. RaFFM (Yu et al.) (2023d) introduces resource-aware model compression framework tailored for FL, which includes salient parameter prioritization and subnetwork extraction to support dy- namic model scaling across heterogeneous edge devices. [Sun et al. (2024) propose a selective layer-wise fine-tuning approach to reduce training cost while preserving model performance.
ios to ensure
4.5 Other Tuning Methods
In addition to mainstream fine-tuning strategics, several alternative approaches have been explored to opti- mize LLMs in FL. These methods primarily include zeroth-order optimization (Chen et al., 2017) split learning (Thapa al. 2022), model compression (Choudhary et al. 2020), and data selection (Shen 2024) (as shown in Figure 9h. For example, FedKSeed (Qin al.
2023b) employs zeroth-order optimiza-
tion with finite set of random seeds, enabling LLM fine-tuning without storing intermediate activations and reducing communication overhead. FedBERT (Tian et al. 2022b) combines FT, with split learning to pre-train BERT in a distributed, decentralized clients. FedBiOT (Wu et al. 2024b) compresses the LLM at the server side while allowing clients to fine-tune lightweight adapters, significantly reducing resource consumption. FedHDS (Qin et al. 2024) introduces hierarchical data selection framework that identifies representative corcsets for instruction tuning, minimizing redundancy at both intra- and inter-client levels to improve training efficiency in FL.
manner, achieving efficient model training across
5 Datasets and Benchmarks
A comprehensive evaluation framework is essential for systematically assessing the effectiveness and gen- cralization ability of foderated fino-tuning methods. In this section, we begin by presenting widely-usod fine-tuning datasets spanning multiple domains. We then detail suite of domain-specific evaluation bench- marks that enable consistent, fine graiiedi and standardized assessment of FedLLM performance across diverse and heterogencous task settings.
5.1 Instruction Fine- Tive-TTnnngg Datasets
Table 9 and Table 10 present a comprehensive collection of instruction fine-tuning datasets spanning several key domains, including general language understanding, finance, medicine, code, math, and law. For each domain, we select representative datasets from Hugging Face that are widely adopted by the community and closely aligned with real-world FedLLM applications. Building on this overview, we next provide a detailed introduction to the datasets within each domain.
18

---

Published in Transactions on Machine Learning Research (02/2026)
5.1.1 General Instruction Fine- Tune-T Datasets
General instruction fine-tuning datasets are primarily designed to improve the overall instruction-following
capability of LLMs across a wide range of tasks and domains (Zhou et al.) 2023). These datasets serve as a foundation for aligning LLMs with human intent, and are particularly useful in federated settings where clients may engage in diverse yet general-purpon interactions.
For English instruction fine-tuning representative datasets include Alpaca (Taori et al 2023), which is generated by Text-DavincciO00 with Alpaca-style instruction prompts, and Alpaca-GPTT (Peng et al. 2023
which builds on this with GPT-4-generated, multi-turn dialogues to improve linguistic nuance and contextual coherence. Other commonly used datasets include Sell--nstrrut (Wang et al. 2022b). UTtraChat 200k (Ding
et al. 2023a OpenOrca (Lian et al., 2023), ShareGPT-90k WizardLM Uvol-Instruce V2 196k (Xu et al,
2023al. Databricks Dolly 15K (Conover et al. 2023
Baize
Xu al. 2023b). OpenChat (Wang et al.
2023a). and Flan-v2
For Chinese instruction tuning, the landscape includes representative datasets such BELLE-CN JJ et al. 2023a), which focuses on improving model alignment with Chinese user instructions. This is complemented (Yang) 2023). Hizard-LM--hinesee Instreet-sttuttttttee Evol (Xu et al, 2023a), and HC3-Chinese (Guo et al. 2023a). These resources facilitate a diverse range of instruction tasks, including reasoning, question answering, and domain-specific knowledge modeling.
by additional resources like Firefly-train-L..
Bilingual instruction datasets are also included to support multilingual and cross- pross-lingual instruction tun- ing in federated contexts. HC3 (Guo al. 2023al offers both human- and model-generated rosponses in English and Chinese for evaluating factual consistency and detection capabilities. ShareGPT-Chiness English-90k 4shareAT (023) provides high-quality, bilingual conversations, making it particularly suitable for instruction tuning in multilingual FedLLM scenarios. A detailed comparison is provided in Table g
5.1.2 Financial Domain Instruction Fine- Tuning Datasets
Instruction fine-tuning datasets in the financial domain are tailored to equip language models with specialized knowledge and reasoning capabilities relevant to financial tasks (Li et al., 2023h). In the context of FedLLM, such datasets are particularly valuable for enabling including investment recommendation, market sentiment analysis, financial reporting assistance, and reg- ulatory compliance. These use cases often involve sensitive and proprictary data that cannot be centrally aggregated due to privacy, confidentiality, regulatory constraints (Byrd & Polychroniadou 2020 making federated fine-tuning an ideal solution.
and institution-speeifie applications,
For English instruction tuning, representative datasets include FinGPT (Zhang et al. 2023a which pro- vides a large corpus of financial cnnestion-answeriig pairs and document summaries tailored to real-world financial analysis. Financc-Inttruct-5000 (Flowers) 2025) and Finance-Alpaca extend general instruction- tuning formats to financial scenarios, offering instruction response pairs related to stock prediction, portfolio analysis, and macroeconomic commentary. Additional datasets such as Financial PhraseBank (Malo et al. 2014), Yahoo-Finance- Data -haancal-QQA-003 Financial-News-Arttccl D and FiQA (Maia et al. 2018) cover broad spectrum of financial tasks including sentiment classification, time-series event ex extraction, and financial question answering. Transcripts from earnings calls further support fine-tuning for multi-turn, dialogue- based financial reasoning.
Twiiter-FFnancial--V
Topid"
For Chinese financial applications, Doc2EDAG (Zheng et al. 2019) provides rich dataset for event detection and argument generation from financial documents, supporting ling instruction-style tasks. In addition, the Pnthetti-PPI-FFnnnn--Milllll
dataset Watson et al. 2024 includes multi-language synthetic financial
"https///uugiinggiccc. co/ /dtasctss dlatascts/liyuchennne Tllyachongg/harrGPPooo ShareGTTTOKK
haatps://huugingficce co/ Bittaasets/SiNeeureee /ulaeaaaaaaanaa///// a v2
$https:
httpss////gggggiiccc..coo co/ hatassssssggbaatt/uunncaaaa////
4https://uue http:/ss////uggggucc.....o htwp:/wwc.ooococe/eeeeeeeeeeew/e http://wwwgauocoo//uuuuu//ttttt/
7 httppp//wwugggnnoeeeoo////sststs htt.a//aaaaeeeeeeeeeooooo//tttte financill-nlnn-nnna topic
8 hhttps://uue http://hggggngae..ooo/
19

---

Published in Transactions on Machine Learning Research (02/2026)
Table 9: A summary of representative instruction fine-tuning datasets in general, financial, and medical domains. The construction methods are categorized into three types: human construct, which refers to datasets written annotated by humans; model construct, which refers to data generated by prompting
LLMs; and synthetic, which refers to data produced through rule-based or programmatic generation.
Dataset
Language
Construction
Domain
Description
Alpaca
English
Model
General
Generated by Text-Davinei-00s (A) (poca-style)
Alpact-GP GPT4
English
Model
General
Generated by GPT- (multi-turn Alpaca prompts)
Selt Instruct
English
Human
Model
General
Generated from seed instructions via GPT-3
Ulsrat list 200k
Euglish
Model
General
Multi-tarn dislogues flltered from UliraChat Chat
OpenOrea
English
Model
General
4.2M GPT--33/4 augemented FLAN examples
NbarrGGPT900
English
Model
General
90K multi-turn dialogues from SharGGTT
W soarclM LM Evol- rodt-lt V2
English
Model
General
196K samples generaled via Evol-Instrec
Ditapyees Doily 10K
Rnglish
Humma
General
15K human-gonerated prompt-maponso padrs
Ilaisc
English
Model
General
Dialogues from ChatCPT (prompted by user queries)
Opent Chat
English
Model
General
Dialogues from open LLMs multi-tur tuzi alignment
Fian-y2
English
Model
General
Combination of Flan, P3, SNI, CoT, and Dialeg taoks
Chinese
Human Model
General
2M Chinese Instruction-fallwwe
samples
Firah--tram- -..amm-MMMM
1.IM
Chinese
Human
General
166M Chinese samples (23 tasks, human templates)
W Wizard-Lan Chmese instrrttstttuteevoll
Chinese
Human Model
General
70K Chinese samples (trauslated from WizarLLM)
Chinese
Human + Model
General
Chinese H Human-Chaat ChatGPT QA pairs (mulli-domain)
HC33
English
Chinese
Human Model
General
Human-ChatGP QA pairs (mulki-domwin) 90K bilingual human-machine QA pairs
English
Chinese
Model
General
FinGPT
English
Hummn Model
Finaucial
dustruction-tunin dala for finuncidl tasks Large-orale Instraction-tunnng data for financial tasks Instruction-tunee data combining Alpaca and FIQA Manually annotated financial news for seatiment
Financo- TTnanco-Inttrett- Instrict-00kk
Rnglish
Human + Model
Financial
Financo-A Apacc
English
Human Model
Financial
Funancial B'raaeBaank
Euglish
Human
Financial
Yalou- Datx
English
Humaa
Finaucial
Yahoo finance dala
F1mancml-QQ--100
English
Model
Financlal
Contextaal QA for financial answering and retrieval
Fmancial- Cliisihattion
English
Human
Financial
Financial PhraseBank and Kaggle financial sexts
Twatter: mmaaciall
Topic
English
Human
Financial
Annosated tweets for financial topic classification
Bhannill-News-riiie
English
Human
Financial
Financial articles for elassification and sentiment analysts
FIQA
English
Human
Financial
QA financial texts and forums
Earmings Call
English
Humna
Finaucial
QA pairs from CEO/CFO carnings calls
Chinese
Humna
Financial
Finaucial reports annolated for event graph extraction
Synthebi-PPi T1-Financn-Ma Muitilingua
Multilingnal
Synthetic
Financial
Synthetic financial does with labeled PII
Chat
English
Human Model
Medical
Modical QA dialoguc instructions
Chat Seaettor--eeaahhee CarealCCaeeMagiio 100k
English
Human
Medical
Real-world doctor doctor-patient conversations
Medical Mesdow CORD-19
English
Humau
Medical
Litersiure summacization instructions (CORD-19)
Medical Meadow Me
English
Human
Medical
Medical QA from professional board exams
Hesitat Sarehaace- e00kee
English
Humsu
Medical
Medical real doctor-pxtient consultations
Chal Loneutl- Sopeull--Daaae Disseel
Chinese
Human + Model
Medical
Medical consultation QA and dialogue instructions
CM
Chinase
Human
Medical
Multi-curn real medical QA
Mes--M
Chinese
Human + Model
Medical
Real dialogues and knowledge graph QA pairs
HuatuotsP"1S ST
Chinese
Human Model
Medical
ChatGPT-geeee and real doctor- Joctor-petient dialogues
Chiness
Human + Model
Medical
Refined subsel of Huxluo-26M (ChatGPT -rrrriiilee))ee
Nongg TCM
Chinese
Human + Model
Motical
QA pairs from TCM knowlodge graph
Mediialog
English Chinese
Human
Medical
Large-scale doctor- doctor-paticnt dialogues
records annotated with privacy-related attributes. A comparative summary of these finance-related datasets is presented in Table
5.1.3 Medical Domain Instruction Fine- Tone-Tuning Datasets
Instruction fine-tuning datasets in the medical domain are designed to enable LLMs to perform tasks such as medical reasoning, patient interaction, and clinical decision support. (Zhang et al. 2023h). Within the context of FedLLM, these datasets are particularly relevant for privacy-preserving healthcare applications, where sensitive patient data is inherently distributed across hospitals, clinics, and personal health devices, and cannot be centrally aggregated due to strict privacy regulations (Nguyen et al. 2022). Federated fine- tuning with medical instruction data empowers models to generalize across diverse clinical intents while preserving the confidentiality and heterogeneity of local medical records, thereby supporting robust and compliant deployment in real-world healthcare settings.
For English-language datasets, ChatDoctor-200 (Li et al. 20231 and HaaiDetto--HalthhCaMMMaaa 100k (Li et al. 2023i) contain medical dialogues derived from professional consultation platforms, facilitating multi-turn reasoning and symptom analysis. The Medical Meadow (Han al. 2023) suite offers curated
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: A summary of representative instruction fine-tuning datasets in code, math, and law domains.
Dataset
Language
Construction
Domain
Description
Cude-Alaac
English
Model
Code
GPT--generatel code instructions (Alpaca-stylee)
Code Instructions 120k Alpaca
English
Human + Model
Code
Code generation instructions (Alpaca-style)
CodeContests
English
Human
Code
Competitive programming problems
Commit PackF
English
Human
Code
Filterod GitHlu commits
high-quality messages
ToolBench
English
Human + Model
Code
Instructions for multi-tool API usage
Coder'arro
English
Human
Code
Deduplicated and filtered Python files from GitHub
The Stack Dedup
English
Human
Code
Large-scale deduplicated source code Code-language pairs for retrieval and semantic search
NodeSearchNet
English
Human
Code
CndeFoorors-Co Co
English
Human + Model
Code
10k CodeFoorces problems with CoT (DeepSeck RT)
CodeXGLUE Code Reftuement
English
Human
Code
Buggy and fixed Java functions for code refinement
GSM8K
English
Human
Math
8.5k grade school math word problems GSMSK extencedd with CoT reasoning steps CoT and Program-off oo----o---hhhhh rationales Question augmeutations from GSM8K and MATH 220k math problems with CoT (DeepSeek RI) 12.5K high school competition math problems
Co -G5MMM
English
Human + Model
Mach
Matuinsttruc
English
Human + Model
Mach
Meta
English
Human + Model
Math
JpenKI-Math-2200
English
Human + Model
Mach
Competition MATH
English
Human
Math
Deepblnd Siatmaaa ice Dataset
Eng lish
Syuthetic
Math
Algorithmically generated math problems
OpenMathhnstruret
English
Human + Model
Mach
1.8M math problems with code-interpreter solutions
Ores-Math Word Problems 200k
English
Syuthetic
Math
200K synthetic grade school math word problems
DAPO-Matn-11 yk
English
Human + Model
Math
17K diverse math problems for reasoning
Big. Math RL., Verified
English
Human + Model
Math
251K math problems with verifiable answers for RL
BELLE-mathe
Chinese
Human + Model
Math
Chinese elementary school math problems
Mashhnstrruct-U Chinese
Chinese
Model
Math
Chinese version of MathInstruct
Legal-QA-vv
Englishh
Human
Law
3.7K legal QA pairs from legal forums Dataset fur legal-domain tasks
Pile ol Law
English
Human
Law
CUAD
English
Human
Law
26K expert-annotated legal contract QA
LEDGAR
English
Human
Law
1.45M expert-labeled logal contract clauses Comprehensive instructions for diverse legal tasks Legal sentence pairs for embedding models
DISC-Law Law-SPT SPT
Chinese
Human + Model
Law
Law-GPT-21
Chinese
Human
Law
Lawyer LLaMA
Chinese
Human + Model
Law
Datasel for Chinese legal tasks
datasets from sources such as CORD-1 and MedQA Largeting tasks like biomedical question answering, 10
II literature summarization, and clinical fact verification. Additionally, HealthCareMinic-1000k. supports English patient doctor interactions across various medical specialties.
For Chinese medical instruction tuning, a rich set of datasets has been developed to address the unique linguistic and clinical characteristics of Chinese healthcare scenarios. Notable examples include Meat Consult-Dataset (Zhu et al. 2023b) and CMtM (Yang et al.] 2024a], which focus on consultation- style QA pairs. DISC-Med-SFT (Bao et al.) 2023), HuatuoGPT-SFT (Li et al., 2023d). : and Huatuo26M- Lite [2] offer large-scale instruction-response pairs in clinical medieine, public health, and disease treatment. Traditional Chinose Medicine (TCM) is also covered through datasets like ShonNong- TCM (Zhu & Wang. 2023), enabling LLMs to support specific diagnostic and treatment tasks.
The MedDialog ZZng et al. 2020) dataset provides bilingual (English and Chinese) medical dialogues, supporting cross-lingual instruction tuning and evaluation in federated medical environments where linguistic diversily and clinical prolocol variance are prevalent. A comparative summary of these medical-speeific datasets, including language, construction method, and description, is provided in Table g
5.1.4 Code Domain Instruction Fine-Tuning Datasets
Instruction fine-tuning datasets in the code domain are curated to improve a language model's ability to understand, generate, and reason about source code across various programming languages and tasks (Muen- nighoff et al. 2023). In the context of FedLLM, such datasets are particularly valuable for enabling privacy- preserving applications like on-device programming assistants, secure code generation within enterprise envi- ronments, and personalized developer support. Given that source code repositories often contain proprietary algorithms, sensitive business logic, or embedded credentials, federated fine-tuning offers a compelling al-
http:///wwgngiicccccoo//assss/ss http:://www.apcccccccaaaaaa mendical__mdddww__ cord19
hOhttpps//buuggigggcccoo///oi hoo/ddtaastt/mccaapppee http///odddddeeeeeeaeee mwwdow medqa
"https///uugggngffre..oo/ /ttpa//tttt/waanrnneeeeoe 12https:/ https//wwwww...o////////////nnnn HuateeaHutttttttteeee
heaer/ggghhnnggeeeeeeeeeeeeeeeee CareaaCareMMagicc 100k-en 100kk------00--enn
21

---

Published in Transactions on Machine Learning Research (02/2026)
ternative to centralized training raw code, allowing organizations to harness LLM capabilities without compromising code confidentiality.
Representative datasets include CodeAlpaca (Chaudhary 2023 and Code Instructions 120k Alpaca 13 which extend the Alpaca instruction format to software engineering tasks such as debugging, function generation, and refactoring. CodeContests (Li et al., (022c) focuses on competitive programming tasks and provides instruction response pairs related to algorithmic problem solving. CommitPackI'T (Muennighofi et al. 2023)
offers commit-message generation tasks based on source code diffs, reflecting real-world software maintenance scenarios. ToolBench (Qin et al. 2023a) is designed to help models learn tool-augmented code generation through cnteructionfollowinng examples.
Several large-scale pretraining and fine-tuning datasets are also widely used for code instruction align- ment. CodeParrof 14 and The Stack v2 Dedup (Kocetkov et al. 2022) provide diverse and deduplicated code corpora across multiple programming languages, while CodeScarchNet (Husain et al.) 2019) and CodeXGLUE (Lu et al. 2021) support retrieval, summarization, and translation tasks with instruction-style prompts. CodeForces-Cool [Penedo al.) 2025) incorporates chain-of-thoughr annotations for programming tasks, supporting more explainable and step-wise code generation. These code-specific datasets play vital role in developing FedLLMs that can support privacy-sensitive, domain-specifie programming environments. A comparative summary of these code-related datasets is presented in Table 10
5.1.5 Math Domain Instruction Fine- Tuning Datasets
Instruction fine-tuning datasets in the math domain are developed to enhance a language model's proficiency in mathematical reasoning, symbolic computation, and step-by-step problem solving (Tang et al.) 2024b). These datasots are particularly valuable in FodLLM scenarios such as personalized math tutoring, intelligent educational flat/orms, and localized STEM applications, where sensitive student information or institution- specific curricular content must remain on-device. Fine-tuning LLMs in the math domain poses unique challenges, as it requires not only a strong grasp of language but also precise logical inference and numerical accuracy-skills essential for generating correct and interpretable mathematical solutions.
For English-language datasets, GSMSK eCobbe al. 2021 is a widely used benchmark for grade-school math word problems with detailed rationales. CoT-GSMSK" augments with thain-of-thought explanations to support intermediate reasoning steps. Datasets such as MathInstruct (Yue et al. 2023b), MetaMathQA (Yul 2023b,, OpenR1-Matth2220 (Allal et al.} 2025). , and Orca-Math Word Problems 200k (Mitra et al. 2024 provide high-quality, instruction-based math problems covering arithmetic, algebra, and word prob- lem solving. The Competition MATH Benchmark (Hendrycks et al. 2021c) and DeepMind Mathematic Dataset offer more advanced, competition-style problems suitable for evaluating formal mathematical reason- ing. Recent datasets like DAPO-Math- YYu et al. 2025), Big-Math-RL- RL-Viide (Albalak et al., (025), and OpenMathInsttuce-11 (Toshniwal et al. 2024) integrate reinforcement signals, verifiable proofs, or multi-step derivations, further pushing the boundaries of instruction-aligned mathematical L.Mss.
et al.
For Chinsso-language instruction tuning datasets, BELLE-math-2 and Seahhnnstrut--hiness 18 provide diverse mathematical problems adapted to Chinese curricula and linguistic structures. These datasets fa- cilitate the training and evaluation of FedI.LMs in multilingual educational contexts, supporting privacy- preserving and culturally contextualized math assistance. A comparative summary of these math-focused
instruction tuning datasets is provided in Table I0
5.1.6 Legal Domain Instruction Fine-Tuning Datasets
Instruction fine-tuning datasets in the legal domain are designed to improve a language model's capability in legal reasoning, contract analysis, statute interpretation, and other tasks that require deep, domain-specific
wtt.a/aatttt////aaaaaaaaaa conttu///oodddittttttttnn instructions 120k alpaca
/tt//aaaasttt///zaauuuuuuuuuu//u Mind Mathematics QA
Whhttps:/hhuggingfaee http///atttt/aaaaaaeiiiiiee 23/bellle-atthhtt
18 18https://huge https:ss//hhgggggaeee.oooo watasetssttttttttaeooo.oooo Aamontttttttttt/tttteeeerrreee MachhaMsttttttt
22

---

Published in Transactions on Machine Learning Research (02/2026)
understanding of legal language and structure Yue et al 2023a In the context of FedLLM, these datasets are particularly important for enabling decentralized legal assistance systems, confidential contract review, and on-device compliance monitoring applications where legal data is often sensitive, jurisdictionboundd and subject to strict conficnnialitty constraints. As centralized training on logal documents is frequently infeasible due to regulatory and privacy concerns, federated fine-tuning offers a promising approach to leveraging LLMs legal settings without compromising data security or legal integrity.
For Eaggish- instruction tuning, Legal-QA-v] provides cuestionn-anser pairs covering legal con- cepts and procedures across various subfields of law. Pile of Law (Gao et al.) 2020 is a large-scale corpus of U.S. legal eecuments-inchudie court opinions, contracts, and regulations- that supports open-ended legal instruction luning. CUAD (Hendrycks et al.] 2021b) focuses on contract understanding, with annotated question contractual clauses categorized into fino-grained legal functions, useful for classification and retrieval tasks in instruction-base basod formats.
muestion-answer answer pairs tailored for clause extraction and risk analysis. LEDGAR contains a large set of
For Chinese-language legal modeling, DISC-Law-SFT Yuu et al., 2023a offers supervised instruc- tion response pairs across wide spectrum of Chinese legal domains, including civil law, criminal law, and administrative law. Law-GPT-zt[I consolidates multiple sources of Chinese legal texts into an instruc- tion tuning format to support legal consultation and statutory reasoning. Lawyer LLaMA (Huang et al. 2023a) further augments legal dialogue capabilities with simulated lawyer- client conversations, making it well-suited for on-device legal assistants in federated environments, A comparative overview of these law- specific instruction tuning datasets is provided in Table IO
5.2
Evaluation Benchmarks
5.2.1 General Evaluation Benchmarks
General-purpose evaluation benchmarks play critical role in systematically assessing the instruction- following ability, reasoning competence, and overall robustness of LLMs across a wide range of tasks and domains (Lou et al. 2024). In the context of FedLLM, such benchmarks are especially valuable for eval- uating model generalization under heterogencous data distributions, identifying robustness gaps in decen- tralized training settings, and facilitating consistent comparisons between personalized and globally aggre- gated models. Existing benchmarks can be broadly categorized into four groups: (i) general reasoning and instruction-followingg (ii) robustness, alignment, and meta-evaluation, (iii) multilingual and Chinese-specific benchmarks, and (iv) long- mog-context understanding. These benchmarks provide a foundation for rigorous and reproducible evaluation of FedLLM across heterogencous and dynamic environments.
(i) General reasoning and instruction--ollooinng. This category includes evaluation benchmarks such as MML.U (Hendrycks et al. 2020), BTG-bench (Srivastava al., 2022), DROP (Dua et al. 2019,, CRASS (Fro- lbber & Binder] (2021), and ARC (Clark et al., 2018), which assess multitask knowledge, discrete reason- ing, and science question answering. AGIEval [Zong et al., 2023), M3Exam (Zhang et al. 2023f and SCIBENCH (Wang et al. 2023d) extend evaluation to standardized exams and college-level math, physics, and chemistry. Instruction -orrloctiooooloowiinngo quality is addressed by Vicuna Evaluation
22 MT-Bench (Zheng et al.
2023a), AlpacaEval (Dubois et al. 2023), Chatbot Arena (Zheng et al.]
2023a and PandaLM (Wang et al.
2021 2023f Datasets like HellaSwag (Zellers et al. 2019) and TruthfulQA (Lin et al.
focus on commonsense
inference and factual accuracy, respectively.
2022a evaluates mul- timodal scientific question answering; Chain-of-Thooe Hub (Fu et al. 2023) focuses step-wiss reasoning: NeuLR (Xu et al. 2023ch benchmarks deductive, inductive, and abductive reasoning: ALCUNA (Yin et al.
Several benchmarks target more specialized reasoning abilities: ScienceQA (Lu et al.
2023
to novel knowledge; LMExam (Bai et al. 2023c tests models on recall, un- derstanding, and analysis across over academic questions. Benchmarks like SocKET (Choi et al. 2023 and
assesses
Choice-75 (Hou et al. 2023 address social knowledge and decision decision-making in scripted scenarios, respectively.
httttl/ps///ugggggnuccco//////// co///aaaaats////engg//////////// llgallqa-vi
hhtttps://uuggingge co/ddaaaasts//oonaaaaitit /ledgar
21 hhttpps//huuge httpssss///hggggaccee..oo aattaasseeeeeeteeeeetttaanteeeee
22 wtt.a/uw....mmmm/m///a////////// witp-eyyy//viuaalleeeeeeee blal-eval
23

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Overview of general evaluation benchmarks. Abbreviations: EM (Exact Match), OOD (Out- of- of-Distribution), NER (Named Entity Recognition), ICL (In-Context Learning), RE (Relation Extraction), MRR (Mean Reciprocal Rank), ASR (Adversarial Success Rate).
Benchmark
Domain
Evaluation Objective
Main Evaluation Criteria
MMLU
Gen.
Multitask understanding (57 subjects)
Accuracy EM, ROUGE, and Accuracy
BIG-bench
Gen.
Advanced reasoning capabilities
DROP
Gen.
Discrete reasoning over paragraphs
EM & F1
CRASS
Gen.
Counterfactual reasoning
Accuracy
ARC
Gen.
Grade-school science reasoming
Accuracy
AGIEval
Gen.
Performance on human-centric exams
Accuracy
M3Exam
Gen.
Multilingual, multimodal, multilevel reasoning
Accuracy
SCIBENCH
Gen.
College-level scientific problem-solving
Accuracy & Error Attribution
Vicuna Evaluation
Gen.
mnstruction-followin quality
Human/ GPPT4 Preference
MT-Bench
Gen.
Multi-turn conversation & rustraction- followngn
GPT-4 Win Rate
AlpacaEva
Gen.
Instrictionnfollloinng (LLM auto-annotation) punaan-preferecce
Win Rate
Chalbol Arena
Gen.
Elo Score
PandaLM
Gen.
Instruction-followea hyperparameter impact
PandaLM Win Rate
HellaSwag
Gen.
Commonsense inference (continuation sclection)
Accuracy
TruthulQQ A
Gen.
Truthfulness & falsehood avoidance Multimodal scientific reasoning & explanation
Truthfulness Rate & Accuracy
ScienceQAA
Gen.
Accuracy & Explanation Quality
Co'l Hub
Gen.
Multi-step reasoning via CoT
Accuracy
NeuLR
Gen.
Deductive, inductive, and abductive reasoning
Accuracy
ALCUNA A
Gen.
Novel knowledge comprehension & reasoning Knowledge recall, understanding, and analysis Social knowledge understanding
Accuracy
LMExam Q
Gen.
Accuracy
SocKET
Gen.
Accuracy
Choice-71 75
Gen.
Decision reasoning (scripted scenarios)
Accuracy
HELM
Gen.
Holistic evaluation (multi-metric scenarios)
Composite Score
OpenI.I.M
Gen.
Open-style reasoning (multi-benchman
Normalized Accuracy
ROSS
Gen.
OOD robustness
OOD Accuracy Drop
GLUE-X
Gen.
OOD robustness
OOD Accuracy Drop
Pench
Gen.
Robustness & prompt-ongineering
ASR & Accuracy Error Rate
Dench
Gen.
Robustness (dynamic human-in-loop)
KoLA
Gen.
Evolving world knowledge (19 tasks)
Sell-Contrast Calibration
CELLO
Gen.
Complex instruction-followt following
Accuracy & BLEU
LLMEval
Gen.
Meta-evaluaaton of LLM evaluators
Meta-cvaluator Agreement
Xiexhi
Gen.
Holistic domain knowledge (516 disciplines)
MRR
C-Eval
Gen.
Chinese domain knowledge & reasoning
Accuracy GPT-4 Win Rate
BELL F-eevl
Gen.
Chinese cnddruction-oollwwin & multi-skill
SuperCLUP
Gen.
Chinese cnttrrcctionfoollowiii (human-aligned)
GPT-4 Win Rate & Accuracy
M
Gen.
Chinese knowledge (71 disciplines, 4 levels)
Accuracy GPT-4 Win Rate
BayLing-Sh 80
Gen.
Cross-lingual & conversational capabilities
MMCU
Gen.
Multitask Chinese understanding
Accuracy Accuracy, Recall, and F1
C-CLUB
Gen.
Classical Chinese NER & RE
2022 and OpenLLM 23 Broad-scoped platforms such as HELM (Liang et al offer normalized aggregate scores across diverse metrics and tasks.
integrate multiple datasets and
(ii) Robustness, alignment, and meta-evaluation. To simulate the variability and noise of federated en- vironments, benchmarks such as BOSS (Yuan et al., 2023), GLUE-X (Yang et al., 2022), PromptBench (Zhul ct al. 2023a), and DynaBench (Kiela et al.] (021) test model robustness to input distribution shifts, prompt perturbations, and adversarial examples. KoLA (Yu et. al., 2023a emphasizes world knowledge calibration, while CELLO (ee et al.] 2024) further tests compliance under real-world constraints. LLMFval (Zhang et al. 2023g) functions a meta-evaluation benchmark to assess the consistency and reliability of LLM- based evaluators, an important consideration when deploying automated evaluation in foderated settings.
Multilingual and Chinese-specific benchmarks. Benchmarks such as Xiezhi (Gu et al., 2024b). C-Eval (Huang et al.) 2023b), BELLE-eval (Ji et al.] 2023b), SuperCLUE (Xu et al. 2023d), M3KE (Liu 2023a), BayLing-St 80 (Zhang et al.) 2023d), MMCU (Zeng 2023 and C-CLUE provide a diverse
(iii)
ct al.
28 hhttps///wuuggghaeee..o//eee/eep 24 Shttss///ghhaabzam////jiinnnnCCo
11m leaderboard
24

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12: Overview of general long-context (LC) evaluation benchmarks.
Benchmark
Domain
Evaluation Objective
Main Evaluation Criteria
LongBench
Gen. (LC)
Bilingual long-context understanding
Accuracy, F1, and ROUGE
L-Eval
Gen. (LC)
Long-context reasoning (up to 60k)
Accuracy & Win Rate
Hench
Gen. (I.C)
Long-context understanding (up 2M)
Accuracy ROUGE
Marathon
Gen. (LC)
Long-context understanding (multi-domain)
Accuracy
LongEvval
Gen. (LC)
Long-context retrieval
Accuracy
BABil.ong
Gen. (LCC)
Long-context reasoning (haystack)
Accuracy
Delective
Gen. (LCC)
Long-context reasoning
Accuracy & GPT-4 Judge Score
NoCha
Gen. (LC)
Narrative comprehension
Accuracy
Loong
Gen. (LC)
Document reasoning extended-contexx QA
GPT-4 Judge Score
ICELonggerne Bench
Gen. (LC)
Temporal reasoning over long narratives
Accuracy
DENIAHL
Gen. (LC)
Context feature influence
ROUGE & EM EM BLEU, ROUGE, and EM
LongMemFwv
Gen. (1.C)
Long-term interactive memory
Long2ItAG
Gen. (LC)
Long-tcxt RAG capabilities
-CiteFivll
Gen. (L.C)
Citation & evidence utilization
ROUGE & Accuracy
LIFBENCH
Gen. (1.C)
Long-context cetrructionfoollwwwiee
Accuracy, ROUGE, and Passük Accuracy & ROUGG-L Accuracy, F1, and ROUGE FI, Win Rate, and EM
LongReason
Gen. (LC)
Long-context reasoning
BAMBOO
Gen. (LC)
Long-text modeling (diverse tasks)
ETHIC
Gen. (LCC
Long-context understanding
LooGLE
Gen. (L.C)
Long-context understanding and reasoning
GPT-4 Judge Score & Auto Metric
HELMET
Gen. (LCC
Multi-skill long-context capabilities
Task Metrica Human Evaluation
HoloBench
Gen. (LC)
Holistic reasoning (databaso-style text)
Accuracy Accuracy, EM, and Recall
LOFT
Gen. (LC)
Long-context rcasoning
Lv-Eval
Gen. (I.C)
Long-context comprehension (up to 256k)
Keyword Recall & Fl Accuracy
ManyICLBench
Gen. (LC)
Many-shot ICL capabilities
ZeroSCROLLS
Gen. (LC)
Zero-shot inference capabilities
ROUGE & F1
CLn
Gen. (L.C)
Long-context: ICI.
Accuracy, ROUGE-I, and Passak Accuracy, Recall, and F1
LIBRA
Gen. (LC)
Russian long-context understanding
evaluation landscape for Chinese and multilingual LLMs. These benchmarks span topics ranging from academic disciplines to sociocultural reasoning, using metrics including GPT-4 preference scoring, multitask accuracy, Fl scores, and normalized Elo rankings.
(iv) Long-context understanding. Long-context reasoning is critical for FedLLM applications involving document-intensive tasks, extended dialogue, and memory retention. Benchmarks such LongBench (Bai
ct al. LongEval (Li el al. 2023a), and BABILong (Kuratov et al. 2024] form the backbone of this category, mea- suring comprehension, retrieval, and reasoning over contexts up to 2 million tokens. Further specialized benchmarks includo: 1) Narrative and temporal reasoning: DetoctiveQA (Xu et al. 2024c), NoCha (Karpin- ska et al., 2024), Loong (Wang et al. 2024b) and TCELongennch (Zhang et al. (0244) focus on long-range narrative understanding and event-based temporal reasoning in complex textual sequences. 2) Retrieval
2023b L- Eval (An et al., 2023). InfiniteBench ęZhang et al. 2024h), Marathon (Zhang et al. 2023b).
and memory evaluation: DENIAHL (Dai et al. 2024),, LongMemBvvll (Wu al., 2024a), Long2RAG (Qi et al. 2024b), and L-CiteEval [Tang et al. 2024a assess in-context feature sensitivity, long-term memory util Liooiiiiio retrieval grounding, and the model's ability to incorporate external citations.
3) Instruction-followingg and generation under long input: LIFBENCH (Wu et al. 2024e), LongReason Ling let al. 2025), BAMBOO (Dong et al. 2023b ETHIC (Lee et al. 2024b). and LooGLE (Li et al. 2023c eval-
uate multi-criteria instruction-followinn accuracy, response stability, and coherence under extended prompts. HELMET (Yen et al. 2024) further integrates summarization, retrieval, and reasoning in unified evaluation pipelines, supporting both automatic and human metrics. 4) Database-style and structured input reasoning: HoloBench (Maekawa et al. 2024) and LOFT (Lee et al.] 2024a) benchmark the ability to perform complex reasoning over structured or databasc-like inputs, including table querying, execution accuracy, and fac- tual consistency, especially in scenarios where retrieval- -aaieeel-auggeetted methods are substituted by long-context modeling.
5) Scaling and generalization with longer context: Lv-Eval (Yuan et al. 2024b), ManyICLBench (Zou et al., 22024, ZeroSCROLLS (Shaham et al. (202)), and LongICLBench (Li et al. (0244)) examine the scalability of in-context learning and multi-task alignment as input length increases, making them valuable tools for
25

---

Published in Transactions on Machine Learning Research (02/2026)
analyzing the performance ceiling of long-context FedLLMs. 6) Cross-lingun long-context evaluation: LI- BRA (Churin et al. 2024] tests mnstructionfollowinng and long-form coherence in Russian, contributing to the evaluation of multilingual long-contex: capabilities. These long-context benchmarks are crucial for val- idating the scalability and persistence of FedLLMs, especially in environments requiring private document analysis, extended user sessions, or continual knowledge tracking. Table [1 and Table 12 summarize the general evaluation benchmarks discussed above.
5.2.2 Financial Domain Evaluation Benchmarks
Financc-spocific evaluation benchmarks are crucial for asscssing the domain alignment, factual accuracy, and reasoning capabilities of LLMs in high-stakes financial contexts et et al. 2023h). In federated settings, where sensitive data from banks, asset managers, and regulatory bodies cannot be centralized due to confidentiality and compliance constraints, these benchmarks serve as vital tools for evaluating model performance under decentralized, privacy-preserving and task-diverse conditions. Effective financial evaluation must encom- pass a broad range of and document classification also accounting for linguistic nuances, regulatory requirements, and the structural complexity of financial texts. Such benchmarks are indispensable for ensuring the robustness and reliability of FedLLM in real-world financial applications.
open-book question answering, multi-step quantitative reasoning,
(i) Multi-task and agent-style financial evaluation. Several benchmarks offer broad-spectrum evalua- tion across multiple financial lasks, aligning well with FedLLM's need to support diverse client use cases (e.g.. auditing, compliance, trading). FinBen (Xie et al. 2024) evaluates holistic financial reasoning over 24 tasks using automatic metrics, retrioval-augmented generation accuracy, and export human judgment. PIXIU (Xie et al. 2023): FLUE (Shah et al., 2022). , and BBT-CFLEB {Lu et al., 2023ab benchmark LLMs on multi-task setups including sentiment classification, QA, event detection, and stock prediction. CFinBench (Nie et al., 2024) and SuperCLUEFin (Xu et al. 2024a) extend this paradigm to Chinese financial scenarios, assess- ing models across regulatory knowledge, certification preparation, and real-world task instructions using multi-type question formats. ICE-PIXIU (Xie el. al.; 2023) further supports bilingual Chinese English eval- nation, suitable for cross-regional FedLLM deployments. FLARE-ES (Zhang et al. 2024g) enables bilingual Spanish English testing to evaluate cross-lingual transfer and domain--pecifie reasoning.
(ii) Task-specific capability evaluation. Fine-grained benchmarks evaluate specific financial NLP capa- bilities that are critical for real-world FedLLM deployment, including document question answering, infor- mation extraction, and numerical reasoning. FinanceBench (Islam et al. 2023) targets open-book question answering using real-world, company relaan--relatdd financial documents, with a focus on factual correctness and evidence alignment- a critical requirement for FedLLM deployed in compliance, auditing, and enterprise doe- ument analysis scenarios. FiNER-ORD (Shah et al. 2023) and FinRED (Sharma et al. 2022) evaluate finan- cial Named Entity Recognition and relation extraction from news and earnings transcripts key for localized data processing on devices with limited connectivity. FinQA (Chen et al.) (021b) and BizBench (Koncel-) Kodziorski et al.) 2023) benchmark multi-step numerical and quantitative reasoning, involving both tabular data and executable financial logic, useful in portfolio analysis, valuation, and budgeting applications. Econ- LogicQA (Quan & Lin 2024) addresses sequential economic reasoning over multi-event scenarios. In the Chi- nese financial domain, FinEval (Zhang et al.) 2023ch and CFBenncharr (Lei et al. 2023) assess task-specific instruction following across topics such as taxation, accounting, and investment strategy. Hirano (Hirano) 2024 enables financial language understanding evaluation in Japanese, while MultiFin (Jorgensen et al. focuses on multilingual topic classification, useful in federated settings with cross-border clients.
2023
(iii) Long-context financial reasoning. Many practical financial tasks involve extended documents such annual reports, investor briefs, and regulatory filings. dong-conten, benchmarks are critical to evaluate whether FedLLM can perform document-level comprehension and multi-step derivation under memory and privacy constraints. DocFinQA (Reddy et al. 2024) simulates multi-step numerical reasoning over financial reports, measuring exact match, FI, and reasoning traceability. FinTextQA (Chen et al. 2024b) further tests open-ended QA over long textual contexts using BLEU and ROUGE for generation quality. These benchmarks reflect realistic federated scenarios, such as on-device due diligence support or local regulatory interpretation, where global models must. adapt to long-form content without accessing raw documents. A
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table 13: Overview of evaluation benchmarks in the financial and medical domains.
Benchmark
Domain
Evaluation Objective
Main Evaluation Criteria
FinBen
Financial
Holistic financial capabilities
Auto metries & Human eval
PIXIU
Financial
Financial NLP tasks
Accuracy & F1 Accuracy, F1, nDCG, and MRR ROUGE, F1, and Accuracy
FLUE
Finaucial
F Financial language understanding
BBT-CFLEB
Financial
Chinese financial understauding and generation
CFinBench
Financial
Chinese financial knowledge
Accuracy
StiperCI.. Fin
Financial
Chinese financial assistant capabilities
Win Rate & Accuracy
CEEIIX
Financial
Bilingual [Chinese (Chinese-Englstt) financial reasoning
Accuracy, F1, and ROUGE
FLA
Financial
Bilingual Eipanish-Englishy fnnncial reasoning
Accuracy, F1, and ROUGE
"nanceBenchh
Financial
Financial open-book QA (real-world)
Factual Correctness
FiNER-ORD
Financial
Financial NER
F1, Precision, and Recall
FiREED
Financial
Financial RE (news, transcripts)
F1, Precision, and Recall Accurary
FinQA
Financial
Numerical reasoning over financial reports
BizBench
Financial
Quantitative reasoning on realistic problems
Accuracy & F1 Accuracy
Econ EconLogie Q
Financial
Economic sequential reasoning
FinEval
Financial
Chinese financial knowledge and reasoning
Accuracy
CFIBenchmark
Financial
Chinese financial assistant capabilities
LLM Judge Score Accuracy
1irrano
Financial
Japanese financial understanding
Accuracy
MultiFin in
Financial
Multilingual financial topic classification
F1 & Accuracy
Docf A.
Financial (LC)
Long-context financial document reasoning
Accuracy
Fin'exIQA
Financial (LC)
Long-fore form financial QA
ROUGE & LLM Judge Score
CBLUE
Medical
Chinese biomedical understanding
Accuracy & F1
PrompiCBLUU
Medical
Chinese medical knowledge (prompt-based)
Accuracy & F1
CMB3
Medical
Chinese medical knowledge
Accuracy & LLM Judge Score
Hua FuuaTuo220 Tuo26M
Medical
Chinese medical knowledge and QA
ROUGE & and LLM Judge Score
CMFxxam
Medical
Chinese medical licensing exam
Accuracy, BLEU, and ROUGE
MulliMed
Medical
Clinical knowledge and open-ended QA
Accuracy Human Evaluation
QiZhenCPI
Medical
Drug indication identification
Accuracy
MedLEXQA
Medical
Medical knowledge and explanation generation
Accuracy & LLM judge Score
JAMA and Medhullets
Mediral
Challenging clinical QA
Accuracy & Human Evaluation
MedXperIQA
Medical
Expert-level medical reasoning (multi-modal)
Accuracy
Med.Jorney
Medical
Full clinical patient journey tasks
Accuracy, ROUGE, BLEU, and F1
Medng
Medical
Complex multi-step clinical reasoning
Accuracy
LangHealth
Medical (I.C)
QA over long-form elinical documents
Accurary
Meyossyssy
Medical (LC)
Long-context medical understanding
Accuracy, Recall, and F1
summary of these finance-specific evaluation benchmarks, including their evaluation objectives and corre- sponding metrics, is presented in Table 13
5.2.3 Medical Domain Evaluation Benchmarks
Medical-specific evaluation benchmarks are essential for assessing the performance of LLMs in healthcare applications, where accuracy, safety, and domain-specific understanding are critical (Thirmavvlaaaauu et al., 2023). In the context of FedLLM, these benchmarks are particularly important for evaluating models de- ployed in privacy-sensitive environments, such as hospital intranets, personal health monitoring devices, and clinical support systems, where patient data cannot be centralized due to regulatory and ethical constraints. To reflect the complexity of medical reasoning and language comprehension under such federated condi- tions, medical benchmarks cover wide range of evaluation targets, including diagnostic reasoning, medical question answering, clinical document understanding, and guideline adherence.
(i) Medical knowledge and QA evaluation. This category focuses on assessing LLMs' general and specialized medical knowledge through structured question-answering tasks. CBLUE (Zhang ct al. 2021 and CBLUE 25 benchmark Chinese biomedical NLP tasks across information extraction, document classification, and QA, with the latter emphasizing prompt-based generation. CMB (Wang et al.] 2023c). HuaTuuo26M (Li et al., (202d)), and CMExam ŞLiu et al. (2023d) evaluate models on Chinese medical exam- style QA, clinical diagnosis tasks, and real-world query understanding. These benchmarks are well-suited for evaluating FedLLM tailored to localized clinical documentation and public health information. Multi- MedQA (Singhal et al.) 2023) serves as a high-quality English benchmark encompassing multiple-choice and open-ended medical QA, with expert-based evaluation of factuality, helpfulness, and sa safety- tthee pillars
25
27

---

Published in Transactions on Machine Learning Research (02/2026)
critical for deploying FedLLM in patient-facing applications. QiZhenGPT provides an annotation- evaluation of drug indication extraction from natural language prompts, useful for drug interaction checking at the edge. MedExQA (Kim et al. 2024) expands QA evaluation to underrepresented medical specialties, while JAMA and Medbullets (Chen et al. 2025 challenge models with high-difficulty US medical exam-style questions ions and demand strong explanation quality.
(ii) Clinical reasoning and care process modeling. In practice, many medical applications require multi-step diagnostic reasoning, care pathway modcling, and treatment planning, especially in multi-agent longitudinal clinical scenarios. MedXpertQQ (Zuo et al., 2025) tests both textual and multimodal reasoning, simulating specialist-level question answering across medical images and textual reports. MedJourncy (Wul et al., 2024d) evaluates LLMs across the full patient care pipeline, from chief complaint triage to follow- up guidance, with both task-specific and human expert evaluation. MedAgentsBench (Tang et al., 2025) explicitly focuses on multi-turn clinical planning, assessing the model's ability to generate coherent and correct diagnostic and treatment steps over multiple interactions- an ideal setup for agent-based FedLLM deployment.
(iii) Long-context clinical understanding. Federated medical applications often involve lengthy patient histories, radiology reports, or guideline documents. Benchmarks in this group evaluate the ability of LLMs to perform robust QA and inference over long-form inputs. LongHealth (Adams et al. 2024 focuses on QA over long clinical narratives, measuring exact match and Fl accuracy. MedOdyssey (Fan et al. 2024a targets long-context understanding across clinical specialties, incorporating ROUGE, EM, and human preference scoring to assess consistency and informativeness over extended reasoning chains. A comprehensive summary of these medical evaluation benchmarks is provided in Table 13
5.2.4 Code Domain Evaluation Benchmarks
Code-specific evaluation benchmarks are essential for assessing the ability of FedLLM to understand, gener- ate, and reason over programming logic across multiple languages and task types (Jiang et al. 2024a). In federated settings, code-related applications are commonly deployed in heterogeneous environments, includ- ing personalized coding assistants, on-device tutoring systems, and localized software development workflows. These scenarios place unique demands on LLMs: they must follow fine-grained instructions, ensure functional correctness, and maintain high reliability all while operating under the resource constraints and privacy requirements that are characteristic of FedLLM meployment. Benchmarks in this domain typically evaluate code synthesis, completion, bug fixing, and multi-turn problem solving.
(i) Basic code generation and functional reasoning. This group of benchmarks focuses on evaluating models' ability to generale sy syntactically correct. and semantically valid code from natural language instruc- tions. HumanBval (Chen et al. 2021a) and MBPP (Austin et al. (021)) are foundational benchnarks evaluat- ing onc-shot Python function generation, judged via exccution-based metrics like pass@k. APPS (Hendrycks) et al. 2021a] and DS-1000 (Lai et al 2023) extend this more complex and real-world tasks, with DS-1000
focusing on data science scenarios across multiple Python libraries. CodeXGL.UF (Lu et al. 2021 offers a broad suite of tasks spanning code summarization, translation, and generation making it suitable for as- sessing FedLLM that may specialize in different sub-tasks across clients. CruxEval (Gu et al. emphasizes logical reasoning and error-free execution in high-stakes contexts. ODEX (Wang et al.. 2022c)
2024a further
introduces cross-lingual code generation, ovaluating the modcl's ability to translate natural language into code in four different programining languages, a valuable benchmark for multilingual FedLLM deployment. These datasets are particularly useful for evaluating client-level specialization in federated setups, where users might work with domain-specific codebases or tools.
(ii) Multi-turn synthesis and structural understanding. Modern code development involves iterative and structural logic, making multi-turn and component-aware benchmarks crucial. MTPB (Nijkamp et al., 2 focuses on multi-turn program synthesis, asscssing the ability to generate partial, compositional sub- programs in sequence. ClassEval (Du et al. (023) evaluates whether LLMs can generate coherent Python classes, including method dependencies and variable interactions reeleetin-elggggggeee real-world object-oriented pro- gramming needs. BigCodeBench (Zuuo et al. 2024h challenges models with complex, multi-functional in-
28

---

Published in Transactions on Machine Learning Research (02/2026)
struction following, and rich code behavior evaluation, using criteria like test case accuracy and branch coverage. TlumanFvalPack (Muennighoff et al. 2023) extends HumanEval's principles to six languages and multiple code-related subtasks, enabling multilingual federated evaluation. BIRD (Li et al. 2023f) adds a structured dimension by testing text-to-SQL generation for database querying, emphasizing schema-aware reasoning and executable correctness an increasingly relevant task enterprise AT agents and personal data querying under private data constraints. Such benchmarks are particularly relevant for FedLLM deployod in collaborative or enterprise environments, where partial programs must be incrementally refined by agents or users with limited compute.
(iii) Long-context code understanding and retrieval. Real-world software development often requires reasoning over extended codebases or repositories, posing a challenge for memory-consiirineed clients. Re- poQA (Liu et al. 2024b) and LongCodeArena [Bogomolov et al. 2024) evaluate models' comprehension and retrieval-augmented reasoning over entire repositories or large code documents. They measure not only token-level accuracy but also structural coherence and retricval efficacy. These benchmarks offer valuable insights into the performance of long-conlext-aware FedLLM realistic software engineering tasks such as bug fixing, documentation generation, and legacy code comprehension Satprrhheenioonpppptttllll under constraints imposed by limited local computational resources.
Together, these code-specific benchmarks provide a comprehensive foundation for assessing the capabili- ties and limitations of FedLLM in coding applications. A detailed overview of benchmark objectives and evaluation metrics is provided in Table 14
5.2.5 Math Domain Evaluation Benchmarks
Mathematical reasoning tasks serve as rigorous benchmark for evaluating the genoralization, composi- tionality, and step-by-step problem-solving capabilities of LLMs (Ahn et al., 2024). In federated seltings, math-specific evaluations are especially valuable for applications such as preeeccigr intelligent tutor- ing systems, on-device educational tools, and personalized STEM learning assistants. These tasks present unique challenges for FedLLM, as they require precise multi-step reasoning, symbolic computation, and log- ical consistency-one often under strict memory, computation, and communication constraints. As such, math benchmarks are instrumental in assessing a model's ability to perform structured reasoning in resource- constrained and heterogencous environments.
(i) Primary math reasoning across educational levels. Benchmarks in this group assess basic-to- advanced math problem solving and reasoning: GSM8K (Cobbe al. 2021 focuses on grade school arith- metic word problems, serving as foundation for reasoning evaluation. Competition MATH (Hendrycks) let al. 2021c) extends this to competition-level questions in algebra, geometry, and calculus, emphasizing step-by-step dcrivation accuracy. MathOdysscy (Fang et al. 2024a evaluates reasoning across high school, university, and Olympiad levels, providing broad difficulty spectrum. MathBench (Liu et al. 2024a) system- atically tests both theoretical understanding and practical application across five levels, reflecting multi-tier federated education scenarios. CHAMP (Mao et al. 2024) introduces concept and hint annotations, useful for federated tutoring agents that may require step-wiss wise guidance personalization for struggling learners. LIIAA MMshra et al. 2022) further expands evaluation to 23 math task types, offering comprehensive insight into the model's versatility across mathematical formats.
(ii) Formal proof and symbolic reasoning. Mathematics often requires formal logic and symbolic structure, which tests model's ability to generalize beyond pattern-matching: MiniF2F (Zheng et al. 2021) and ProofNet (Azerbayev et al. 2023) evaluate formal mathematical reasoning and proof generation, with the latter using Lean 3 a backend lor correctness verification. AlphaGeometry (Trinh et al.) 2024) blends neural and symbolic reasoning for Euclidean geometry, domain requiring precise spatial logic and theorem synthesis-especins
relevant in expert-centric or rescarch-level FedLLM deployment.
(iii) Visual and diagrammatic mathematical reasoning. Many real-world math problems include visual elements (graphs, tables, geometric diagrams), posing multi-modal reasoning challenges: Math- Verse (Zhang et al.] (024f) and Wo-Math (Qiao et al. (024) evaluate visual reasoning using diagram in- terpretation, requiring fine-grained attention to layout and symbolic grounding. U-MATH (Chernyshev et al. 2024) tests open-ended university-level questions involving visual cues, with I.T.Maassited expert
29

---

Published in Transactions on Machine Learning Research (02/2026)
Table 14: Overview of evaluation benchmarks in the code, math, and law domains.
Benchmark
Domain
Evaluation Objective
Main Evaluation Criteria
HumanEval
Code
Code generation and algorithmic reasoning
Passeik
MBPР
Code
Basic Python code generation
PassEk
APP'S
Code
Coding challenge competence
Pass0k
DS-1000
Code
Data science code generation
Accuracy & PassBk
CodeXGL.UE
Code
Code understanding and generation
Accuracy, BLFFU, and MRR
Eval
Code
Code reasoning, understanding, and execution
Passek
ODEX
Code
Code generation and execution
PassEk
MTPB
Code
Multi-turn program synthosis
Pass0k
ClassEval
Code
Class-level code generation
Pass0k
BigChdeBench
Code
Complex and practical code generation
Pass
HumanEvalPack
Code
Multilingual code tasks
Passūk
BIRD
Code
Database- grounddd text-to-SQL
Execution Accuracy F1
RepoL2A
Code (LC)
Long-context code understanding
Pass Rate
LongCode CodeAAnna
Code (LC)
Long-context code tasks
PassBk, Accuracy and EM
GSM&K
Math
Grade school math reasoning
Accuracy
Competition MATH
Math
Competition math problem-solving
Accuracy
Muthodysscy
Math
High-level math problem-solving (Olympiad)
Accuracy
MathBench
Math
Comprchensive and hicrarchical math cvaluation
Accuracy & Circular Evaluation
CHAMP
Math
Fine- gra-grained math reasoning
Accuracy
LII.A
Math
Meta-benchmark for math reasoning
Accuracy & Passtik
MiniF2F
Math
Formal math reasoning (Olympiad)
Proof Accuracy
ProotNct
Math
Auto-formallzation and formal proof generation
Accuracy & Success Rate
AlphaCoometry
Math
Neuro-symbolic reasoning (Euclidean geometry)
Solve Rate
MathVerso
Math
Visual math reasoning
Accuracy & CoT Score
We-Math
Math
Multi-step math reasoning
Accuracy
U-MATH
Math
Open-ended visual math (university-level) Math reasoning over text and tabular data Long-context math reasoning (multi-step)
LLM Judge Score
TabMWP
Math
Accuracy
Mathllay
Math (LC)
Accuracy
LegalBench
Law
Legal reasoning across six tasks
Accuracy, F1, and ROUGE
LexGLUU
Law
Legal language understanding
Accuracy & F1
LEXTREME
Law
Multilingual/man /mmllitask legal understanding
Accuracy & F1
Lawce
Law
Chinese legal reasoning
Accuracy & Fl
LAIW
Law
Chinese legal tasks
Accuracy & F1
Lex
Law
Chinese legal understanding and reasoning
Accuracy MRR, Recall, and ROUGE Success Rate & LLM Judge Score ROUGE & METEOR
Cital Jaw
Law
Tegal answer generation
LegalAA Bench Bench
Law
LLM agent legal task-solving
SCALE
Law (LC)
maabllingual/long-occccet logal reasoning
scoring. TabMWP (Lu et al. 2022b) focuses on text table joint reasoning, simulating practical applications like report analysis or financial tutoring in federated agents.
(iv) Long-context mathematical reasoning. FedLLM deployed on real-world devices often faces scenar- ios where mathematical problems span multiple steps or documents: MathHay (Wang et al. 2024a) evaluates reasoning across extended input chains, testing memory retention and logic consistency in multi-hop math reasoning an important benchmark for long-context capabilities in private and offline educational settings.
Together, these benchmarks form a robust and diverse suite for evaluating the mathematical competency of FedLLM under different input formats, difficulty levels, and reasoning demands. They are especially vital for personalized STEM learning assistants and edge- automated math tutoring. A comprehensive summary of these math evaluation benchmarks is provided in Table 14
5.2.6 Legal Domain Evaluation Benchmarks
Legal Al applications impose stringent requirements on factual accuracy, contextual understanding, and log- ical consistency- making legal benchmarks particularly important for evaluating FedLLM designed for use in areas such as smart justice, personalized legal consultation, and privacy-preserving regulatory compliance. These benchmarks are designed to assess model capabilities across a range of legal reasoning tasks, includ- ing legal text comprehension, argument analysis, statutory interpretation, and multi-document synthesis.
30

---

Published in Transactions on Machine Learning Research (02/2026)
Moreover, they often adopt multilingual and long-context formats that reflect the real-world complexity and heterogeneity of legal documents-ccalllee that are amplified in federated settings (Chen et al. 2024e).
(1) Foundational and legal-specific reasoning. Benchmarks in this category assess general-purpose legal understanding, inchuding statute interpretation, legal judgment tasks, and document comprehension: LegalBench (Guha et al.] 2023) provides a suite of six legal reasoning tasks such as rule application and contract understanding, evaluating both correctness and rule-consistency. LexGLUE (Chalkidis et al. 2021] includes classic legal NLP tasks such case classification, contract QA, and legal entailment, making it suitable for evaluating LLMs in general legal understanding scenarios. LEXTREME (Niklaus et al. 2023) expands this evaluation to 24 languages and 18 tasks, making it a critical multilingual benchmark for assessing FedLLM's cross-lingual legal proficiency in global regulatory contexts.
(ii) Chinese legal language understanding. Given the importance of regional legal systems, several benchmarks target Chinese legal capabilitics, aligning well with privacy-sensitive applications deployod in Chinese jurisdictions: LawBench (Fei et al.] 2023) evaluates Chinese legal LLMs across three cognitive understanding, and application via 20 legal tasks. LAiW (Dai et al. 2023) offers a fine-grained assessment framework covering fundamental to advanced legal challenges. LexEval (Li et al. 2024b) structures evaluation around taxonomy of legal cognitive abilitics, including memory, reasoning, and application. These benchmarks support federated personalization and regional adaptation of legal agents.
(iil) Legal citation and generation with formal grounding. Some legal tasks require not only correct answers but also proper justification grounded laws and precedents: CitaLaw (Zhang et al.) (024c) focuses on generating legally sound responses with accurate citations to statutes and precedent cases. It uses metries like syllogism alignment and legal consistency, making it particularly relevant for FodLLM agents operating in (iv) Legal agents and dynamic task-solving. FedLLM may increasingly support legal assistants or decision-supporr agents that require multi-step, tool- integrated reasoning: LegalAgentBench (Li ct al. 2024a provides a novel benchmark for evaluating LLM agents on complex, multi-turn legal scenarios. incorporates intermediate progress tracking and task success scoring, which are useful for evaluating FedL.LM performance decentralized and asynchronous workflows.
jurisdictions with citation and traceability requirements.
(v) Long-context and multilingual legal reasoning. Legal documents are often lengthy, hicrarchical, and span multiple statutes or precedents: SCALE (Rasiah et al. 2023 is designed to evaluate LLMs long- context legal documents, legal multilingualismm, and cross-document reasoning. Tts inclusion of multi-task legal scenarios and code-level legal analysis makes it especially relevant for edge-deployed legal assistants in enterprise or government use cases. Together, these benchmarks provide a rich and diverse framework for evaluating FedLLM in the legal domain where privacy, jurisdictional customization, long-context under- standing, and reasoning fidelity are all critical to real-world deployment. A comprehensive summary of these legal evaluation benchnarks is provided in Table 14
6 Application
6.1 FedLLM for Recommendation Systems
Recommendation systems are pivotal across domains such as e-commerce, content streaming, and personal- ized advertising (Ko et al. 2022). Traditional approaches often rely on centralized data collection, raising significant, privacy concerns, particularly when handling sensitive user interactions and preferences. Feder- ated fine-tuning offers a promising alternative by cnabling collaborative learning across distributed clients while preserving data privacy.
Zhao et al. (2024a) propose FELLRec, federated framework for LLM-based recommendation, to tackle the challenges of client performance imbalance and high resource costs. Specifically, FELLRce employs dy- namic parameter aggregation and adaptive learning speeds to ensure balanced performance across clients. Additionally, it selectively retains sensitive LLM layers on the client side while offloading other layers to the server, effectively preserving privacy and optimizing resource usage. Similarly, Yuan et al. (2024c) introduce FELLAS, federated sequential recommendation framework that leverages LLMs as external services enhance sequential recommendation. FELLAS enriches item embeddings via L.LM-assisted textual repre-
31

---

Published in Transactions on Machine Learning Research (02/2026)
sentation while ensuring privacy protection through de-privacy-compliant sequence perturbation. Beyond privacy protection, FL, also facilitates reinforcement, learning from human leedback for LLM--ased recom- mendation systems. [Wu et al. (2024c) propose FedBis and FedBiscuit, two frameworks designed to enable privacy-preseriing federated RLHF. FedBis collaboratively trains a binary selector to filter sensitive pref- erence data, while FedBiscuit clusters clients to train multiple selectors, ensuring better alignment with human preferences while maintaining privacy. Another framework, GPT-FedRee, proposed by Zeng et al.. (2024a), integrates ChatGPT with a hybrid Retrieval-Augmmented Generation (RAG) mechanism to address data sparsity, heterogeneity, and LLM-specific challenges in federated recommendation. GPT-FedRee em- ploys hybrid retrieval techniques to extract user patterns and item features, then refines recommendations through LLM-generated prompts. By leveraging RAG, the framework effectively mitigates hallucination in LLM-generated content and enhances the overall recommendation quality.
In summary, as L.MSs become increasingly integrated into recommendation systems, federated fine-tuning has emerged as powerful method to fully leverage the capabillitis of LLMs while ensuring user data privacy. These advancements demonstrate the polential of FedLLM to support high-quality, privacy-aware recommendation in real-world applications.
6.2 FedLLM for Biomedical Research
In the biomedical domain, direct data transmission and centralized model fine-tuning pose significant risks to user and patient privacy. Federated fine-tuning provides a privacy-preserving paradigm that enables col- laborative model adaplation across decentralized medical datasets without exposing sensitive information.
Ali et al. (2025) explore the use of various FL techniques to fine-tune time-series LLMs on electrocardio- gram and impedance cardiography data, enabling privacy-preserving physiological signal analysis. Naseer & Nandakumar conduct a systematic investigation into the application of federated PEFT strategies for fine-tuning vision transformers in medical image classification tasks. Pupppla et al. (2024) present a FL.- based GPT chatbot designed for personalized healthcare information retrieval. The system aggregates and curates information from diverse sources while ensuring privacy and security through decentralized training. Users receive real-time, personalized insights via an intuitive interface, supported by advanced text parsing, metadata enrichment, and question-answering capabilities. This framework marks key advancement in patient-centric Al applications.
Sarwar 2025 introduces FedMentalCare, a privacy-preserving framework that integrates FL and LoRA to fine-tune LLMs for mental health analysis. Their study explores the impact of client data volumes and model architectures (e.g., MobileBERT, MiniLM) in FL settings, ensuring scalability, data security, and computational efficiency. Liu et al. (022cc) propose FedFMS, which introduces federated foundation models for medical image segmentation. It addresses privacy challenges in medical imaging by enabling federated training without centralized data sharing. [Wang et al. (2024d) introduce FEDKIM, a federated knowledge injection framework for scaling medical foundation models. It. leverages weight local models to extract private knowledge and integrates it into a centralized model using an adaptive multitask multimodal mixture of experts module, enabling efficient cross-institutin knowledge transfer. Dai et al. (2025) propose FedATA, a supe-supe FL framework for medical image segmentation, integrating masked self-distillation with adaptive attention to enhance pre-training and fine-tuning on unlabeled and limited-annotation Unlike traditional masked image modeling, FedATA uses latent representations as targets instead of pixels, improving feature learning. Additionally, its adaptive attention aggregation with personalized FL captures instifution-specffe representations, boosting model generalization and local fine-tuning performance.
data.
In summaary LLMs have become increasingly influential in biomedical research. However, due to the highly sensitive nature of user data in this domain, federated fine-tuning has emerged as a pivotal approach for enabling large-scale Al applications in biomedicine and healthcare while ensuring data privacy and security.
6.3 FedLLM for Finance
The financial sector heavily relies on data-driven models for risk asscssment, frand detection, algorithmic trading, and personalized financial services. However, financial data is often highly sensitive, heavily reg- ulated, and distributed across multiple institutions, making calizel model training infeasible due to
32

---

Published in Transactions on Machine Learning Research (02/2026)
privacy concerns and compliance constraints. Federated fine-tuning presents a promising solution by en- abling collaborative learning across financial institations without exposing raw data.
Ye et al. (2024b) introduce OpenledLLMM, a federated fine-tuning framework designed to train LLMs on decentralized private data while ensuring data privacy. In the financial domain, FL-tuned LLMs signifi- cantly outperform locally trained models and even surpass GPT-4, demonstrating the potential of FL to enhance LLM performance without compromising sensitive financial data. This study underscores the value of federated fine-tuning in leveraging distributed financial data to develop more accurate, robust, and privacy- preserving LLMs for financial applications. [Shabani (2224) explore the use of FL for fine-tuning LLMs in finance, enhancing efficiency and privacy while addressing data scarcity and distribution challenges. Their findings show that FL achieves performance comparable to centralized fine-tuning with siguificantly lower computational costs and training time, making it ideal for resource-constrannee environments. This ap- proach preserves data privacy while enabling the development. of more accurate and robust financial LLMs. Similarly, Zeng et al. (2024b) investigate fine-tuning financial LLMs using LoRA and deploying them on edge devices, demonstrating FL's polential to improve both model efficiency and performance in financial applications. Their study highlights significant gains in reasoning capabilities and eost-effectiveness, offering valuable insights into leveraging FL and LLMs for private and vertically specialized financial domains.
In summary, lederated fine-tuning plays a crucial role in the financial sector by enabling collaborative model training across institutions while ensuring strict data privacy compliance. This approach allows financial organizations to leverage vast, decentralized datasets for fine-tuning, improving model accuracy without ex- posing sensitive financial information. As financial markets grow more complex and globally interconnected, FedLLM presents scalable and secure pathway toward next-generation AT-driven financial infrastructure.
7
Open Challenges and Future Directions
Model Security of FedLLM. As federated fine-tuning gains momentum, ensuring model security has become critical concern. In FedLLM, pre-trained models, whether proprietary or open-source, must be transmitted to distributed clients for local fine-tuning, inherently increasing the risk of intellectual property (IP) leakage and system vulnerabilities. Model security in this context involves two key aspects: protecting the IP of high-value models and ensuring the secure deployment of open-source models on edge devices.
First, the financial and strategic value of pre-trained LLMs makes TP protection in FedLLM deployment espe- cially pressing. For example, training models like Gemini Ultra (Mesnard al. 2024 and GPT-4 (OpenAl, 2023 is estimated to cost, $191 million and $78 million, respectively. These models are typically developed by commercial entities under strict licensing and infrastructure control. However, in FedLLM settings, where the full model is often shared with clients in a white-box fashion, it becomes feasible for malicious participants to reverse-engineer or clone the model. This undermines the original developers' competitive advantage and deters participation from commercial model providers. Addressing this challenge requires the development of model watermarking (Pan et al., 2024a),, encrypted model delivery, or inference-obfuucation protocols that allow clients to fine-tune and use the model without revcaling sensitive architectural or parameter details.
Second, while open-source LLMs (e.g., DeepSeek (Bi et al.) 2024 Qwen (Bai et al. 2023a are widely adopted in FedLLM due to their accessibility and flexibility, they present new vectors for security threats in federated deployments. In practice, most clients, especially those with limited machine learning or systems espertise, may lack the capabilities to deploy these models securely. For instance, as reported by the technology news outlet Albase 27 frameworks like Ollama have been found to expose users to data leakage and horuthorized resource usage due to insecure default configurations. In a federated setup, such vulnerabilities are amplified: a single compromised client can leak locally fine-tuned training data, or propagate adversarial backdoors to the global model. The consequences are particularly severe in sensitive domains like healthcare and finance, where breaches may result in the disclosure of protected health information (PHI) or proprietary trading strategies.
To mitigate these risks, future FedLLM research should prioritize the integration of secure model deploy- ment practices into the federated fine-tuning pipeline. Techniques such as confidential computing for secure
27 httpps: hwtp:/aaawsee...o/eee///////e/ee
33

---

Published in Transactions on Machine Learning Research (02/2026)
execution on edge devices, encrypted model delivery, and runtime access control should be incorporated to prevent unauthorized access and tampering. By embedding such security mechanisms into the FedLLM lifecycle, both commercial and open-source models can be safeguarded against misuse, thereby promoting broader adoption in high-stakes domains such as healthcare, finance, and critical infrastructure.
LLM and SLM Collaboration. A key future direction for FedLLM lies in enabling efficient collaboration between LLMs and small language models (SLMs) to address the performance privacy privacy-efficiency trade-offs inherent to federated settings. While LLMs offer superior reasoning and multi-modal capabilitics, their large size and resource demands make them impractical for direct deployment on edge devices. Conversely, SLMs such as Gemini Nano (Team et al. 2023) and Phi-3 (Abdin et al., 2024) provide lightweight alternatives with better deployment efficiency but limited generalization and task transferability.
To reconcile these limitations, emerging FedLLM architectures can adopt a hybrid model paradigm: deploy- ing SLMs at the edge for privacy-sensitive inference and lightweight tasks, while offloading complex reasoning or orchestration to cloud-hosted LLMs. This collaborative strategy not only reduces the communication and computation burden at the client side but also enhances regulatory compliance by ensuring sensitive data never leaves the local device. Within this architecture, edge SLMs can perform initial text generation or instruction parsing, while LLMs handle tool selection, global coordination, or cross-domain alignment.
However, realixing this collaboration raises several open challenges in FedLLM: 1) minimizing latency and bandwidth overhead introduced by frequent SLM LLM interactions; 2) preserving consistency and alignment between local SLM outputs and global LLM behavior, and 3) dynamically adapting task delegation strategies based on client heterogeneity, model confidence, and task complexity. Future FedLLM research should design decentralized orchestration protocols for efficient SLM-LLM coordination, and introduce privacy- preserving metadata exchange mechanisms to protect tool usage logs and inference traces. By enabling seamless SLM-LLM collaboration under federated environments, this hybrid architecture can unlock new levels of scalability, efficiency, and privacy in real-world FedLLM deployments.
Multi-Modal FedLLM. While existing FedLLM research has primarily focused on text-based tasks, emerging real-world applications increasingly require multi-modal capabilities, such as integrating visual, speech, and sensor modalities (Zhang et al. 2024a)- Large multi-modal models (LMMs), including GPT- 4V (Yang et al. 2023b) and LLaVA (Liu ct al.) 2023b have demonstrated strong performance in centralized settings. However, extending these models to federated environments presents several unresolved challenges.
A primary difficulty lies in modality heterogeneity across clients {Peng et al. 2024 Ouyang et al. 2023) Devices may possess different input types for example, some may only have textual data, while others may hold image text pairs leading to modality imbalance, where certain modalities are underrepresented in the training process (Fan et al.] (2024b). This imbalance can degrade the model's generalization across modalities. Additionally, achicving cross-modal alignment (Gao et al., 2024c)- model's ability to relate and reason across different modalities becomme more difficult in federated setups, where paired data (e.g.. image-caption pairs) cannot be shared centrally. Moreover, the high computational demands of LMMs further constrain their deployment and fine-tuning on edge devices with limited memory and processing capabilities (Liang et al. 2024 Jin et al. 2024).
To address these challenges, future research should develop modular and flexible tuning frameworks that allow each modality to be fine-tuned independently on client devices. This decoupling enables efficient local adaptation without requiring all modalities to be present on each client. Furthermore, modality aggregation protocols which weight client contributions based modality type, data quality, and semantic consistency-can help mitigate imbalance and enhance global model performance. Promising directions also include federated cross-modal cont rastive learning (Yu et al.) (023c), which can improve multi-modal alignment without requiring raw data exchange. Finally, to facilitate deployment in edge-centric applications such as smart healthcare, assistive robotics, and wearable systems, it is essential to design lightweight multi- modal architectures through techniques like knowledge distillation (Cai et al. 2024bb or dynamic subnetwork activation (Alam et al. 2022) that strike balance between accuracy and resource efficiency.
34

---

Published in Transactions on Machine Learning Research (02/2026)
Continual Learning in FedLLM. In dynamic federated environments, client data distributions and task objectives evolve over time, necessitating continual learning capabilities in FedLLM systems (Yoon et al. 2021 Wang et al. 2024e). Unlike traditional FL settings with fixed tasks and static datasets, real-world deployments roquire models to incrementally incorporate new knowledge without retraining from scratch. However, continual fine-tuning of LLMss introduces several unique challenges. The sheer size and overparam- eterization of LLMs make them prone to catastrophic forgetting during incremental updates (Huang et al., 2024a), especially when client participation is sparse or irregular. Moreover, repeated retraining across rounds is computationally expensive and often infeasible on edge devices with limited hardware resources.
To overcome these limitations, future research should invostigate parameter-cfficient continual learning strate- gies that enable local knowledge retention while supporting scalable global updates. Techniques such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017). PEFT-base modular updates, and re- hearsal methods using compressed memory buffers (Tiwari et al. 2022) are promising in mitigating for- getting without incurring prohibitive overhead. In addition, the design of lifelong personalization proto- capable of adapting to each client's evolving task distribution under Non-IID and intermittent data availability remains an open research frontier. Developing such protocols requires balancing communication efficiency, privacy proservation, and model stability across heterogencous learning trajectories.
Ultimately, enabling continual adaptation in FedLLM will be essential for long-term deployment in dynamic real-world scenarios, such as personalized healthcare, evolving legal compliance systems, or lifelong learning assistants. This calls for a shift from round-based static fine-tuning to streaming, task-aware federated adaptation frameworks that can incrementally evolve with users and environments.
EfffretEfficcint FedLLM. Memory efficiency remains one of the most fundamental and restrictive bottlenecks in the deployment of FedLLM- often resulting in binary feasibility condition: client device either meets the memory requirements to participate in training or is entirely excluded (Wu et al. (025d). Unlike other challenges such as communication overhead or data heterogencity, which degrade performance but still permit participation, memory limitations can preclude participation altogether, particularly for edge devices with constrained hardware capabilities. Although PEFT techniques like LoRA substantially reduce the number of trainable parameters, they fall short of fully mitigating memory pressure. For example, fine- tuning LLaMA2-13B with LoRA still demands over 50 GB of peak memory order of magnitude beyond
the capacity of most mobile phones, ToT devices, or embedded systems (Xu et al. 2024b Tian et al. 2024a
Addressing this limitation requires innovation at both the algorithmic and system levels. side, emerging approaches such as dynamic layer-wise adaptation (Pan et al.] 2024b), quantization- awaar PEFT (e.g., QLoRA) (Dettmers et al. 2023), and structured model pruning (Wang et al. 2019b Ma et al., 2023) offer promising pathways for reducing memory footprints during local finc-tuning. Complementing these are system-level solutions such as gradient checkpointing and accumulation (Gim & Ko 2022). run- time memory-aware schedulers, and cloud-edge hybrid training architectures with selective computation offloading (Kumar et al.) 2013), all of which aim to stretch the effective memory capacity of participating devices. To fully unlock the potential of FedLLM at scale, future research should explore co-designed frame- works that jointly optimize algorithmic efficiency and system-level deployment. Such holistic solutions can harmonize memory, computation, and communication trade-offs in real time enabling resource-adaptive, priacyypresorving model customization across highly diverse client ecosystems.
On the algorithmic
Overcoming the memory barrier would expand the pool of eligible participants to include billions of low- memory edge devices that are currently sidelined from federated training. This not only enhances the inclusiveness, representativeness, and scalability of the FedLLM framework, but also opens the door to real- world deployments in settings such as home automation, wearables, and low-power industrial ToT platforms.
8 Conclusion
To the best of our knowledge, this is the first comprehensive survey dedicated to the federated fine-tuning of LLMs. We begin by introducing foundational background knowledge and identifying four core chal- lenges through empirical analysis, which reveal the fundamental limitations that foderated fine-tuning must overcome. We then review the latest relevant research papers, systematically organizing recent advances
35

---

Published in Transactions on Machine Learning Research (02/2026)
in parameter-cffiient federated fine-tuning techniques. These approaches categorized based on their methodologies, with detailed discussions on how each class of methods addresses the identified challenges. Furthermore, we present a comprehensive evaluation framework encompassing both fine-tuning datasets and evaluation benchmarks across different domains, offering a holistic framework for assessing FedLLM perfor- mance. Beyond methodological contributions, we highlight practical applications of FedLLM across domains. Finally, we outline promising future directions in this rapidly evolving field. While notable progress has been achieved, several pressing challenges remain open. Addressing these issues will be essential to unlocking the full potential of FedLLM and enabling its widespread deployment in practical, privacy- applications.
Acknowledgements
This work is supported in part by the Science and Technology Development Fund of Macau (0107/204/RIA22) Joint Science and Tochnology Research Project with Hong Kong and Macau in Key Areas of Nansha District's Science and Technology Plan (B0220))-0010000TTCC)) and the Multi-Year Re- search Grant of University of Macau (ANRG--CICC22-0021110TCCCCCD and MYRG-CGGG0024-00152 IOTSC).
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2044142219 2024.
Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthicu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXim:2401.0 14490, 2024.
Khwaja Mutahir Ahmad, Qiao Liu, Abdullah Aman Khan, Yanglei Gan, and Changhao Huang. Prompt- enhancod foderated learning for aspect-basod sentiment analysis. In 2023 International Conference on Intelligent Communicatiin and Compuler Engineering (ICICCE), pp. 81 87. IEEE, 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157, 2024.
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. Advonces in newral information processing systems, 35:29677 29690, 2022.
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math datasct for reinforeement learning in language models, 2025. URL https:///axii wtwp://.wrrraabb/22...1111
Mahad Ali, Curtis Lisle, Patrick W Moore, Tammer Barkouki, Brian J Kirkwood, and Laura Brattain. Fine- tuning foundation models with federated learning for privacy preserving medical time series forecasting. arXiv preprint arx::2502.0077, 2025.
Loubna Ben Allal, Lewis Tunstall, Anton Lozhkov Elie Bakouch, Guilherme Penedo, and Gabriel Martín Blázquez Hynck Kydlicck. Open rl: Evaluating 1lms on uncontaminated math competitions, 2025.
Omair Rashed Abdulwareth Almanifi, Chee-Onn Chow, Mau-Luen Tham, Joon Huang Chuah, and Jeevan Kanesan. Communication and computation efficiency in federated learning: A survey. Internet of Things, 22:100742, 2023.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXie preprint 2rXin2307..11888 2023.
36

---

Published in Transactions on Machine Learning Research (02/2026)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint Xre:221088 07732, 2021.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduatc-lovel mathematics. arXiv preprint arXiv:2300.1244 2023.
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Keo-Bong Song, Mostafa El- Khamy, and Salman Avestimehr. Slora: Federated parameter efficient. fine-tuning of language models. arXiv preprint Xii::2308.06522 2023.
Gaurav Bagwe, Xiaoyong Yuan, Miao Pan, and Lan Zhang. Fed-cprompt: Contrastive prompt for rehearsal- free federated continual learning. arXiv preprint arXiv:2307. 04869, 2023.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint orXix:2309 16609, 2023a.
Sikai Bai, Jie Zhang, Song Guo, Shuaicheng Li, Jingcai Guo, Jun Hou, Tao Han, and Xiaocheng Lu. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In Proceedings of the (GEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 27284-27200 2024b.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprinl ar Xix::22308. 14508, 2023b.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with modglage-mdeelas-aaee -nxaanneerr Advances in Newral Information Processing Systems, (8.78142-78161 2023c.
Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. Disc-medllm: Bridging general large language models and real-world modical consultation. arXiv preprint ar Xix::22308. 14346, 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek II:: Scaling open-source language models with longtermism. arXiv preprint Xiv: 02954, 2024.
Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement. arXiv preprint arXiv:2411. 149661 2024.
Egor Bogomolov, Aleksandra Elisceva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, et al. Long code arena: set of benchmarks for long-context. code models. arXie preprint arXiv:2406. 11612, 2024.
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private bias-term only fine-tuning of foundation models. 2022.
David Byrd and Antigoni Polychroniadou. Differentially private secure multi- computation for federated lcarning in financial applications. In Proceedings of the first ACM international conference on AI in finance, pp. 1-9, 2020.
Yuji Byun and Jacho Lee. Towards foderated low-rank adaptation of language models with rank heterogenc- ity. arXiv preprint ar Xiv:244406 17477, 2024.
37

---

Published in Transactions on Machine Learning Research (02/2026)
Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Efficient federated learning for modern nlp. In Proceedings of the 29th Annual International Conference on Mobile Compuling and Networking, pp. 16, 2023a.
Dongqi Cai, Yaozong Wu, Haitao Yuan, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Towards practical few-shot federated ulp. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pp. 42-48, 28-4, 2023b.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXie:2407.06000 2024a.
Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, and Xiang Bai. Llava-kd: A framework of distilling multimodal large language models. arXiv preprint arXiv:2410. 16236, 2024b.
Di Chai, Leye Wang, Liu Yang, Junxue Zhang, Kai Chen, and Qiang Yang. survey for federated learning evaluations: Goals and measures. IEEE Tronsactions on Knowledge and Data Engineering, 2024.
Iliss Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint Xrv:22 10. 00976, 2021.
Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation, 2023.
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor S Sheng, Huaīyu Dai, and Dejing Dou. Federated learning of large language models with parameter-effiiiet prompt tuning and adaptive optimization. arXiv preprint Xrx:223300. 15080, 2023.
Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 3563-3599, 2025.
Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11285-11293, 2024a.
Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang, Fintextqa: A dataset for long-form form financial question answering. arXiv preprint Xrxi:22405. 099800,900000 2024b.
Jingxue Chen, Hang Yan, Zhiyuan Liu, Min Zhang, Hu Xiong, and Shui Yu. When federated learning meets priaacy-rrseeving computation. ACM Computing Surveys, 56 56(12):1-36 2024c.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. Xixiv prepriat arXiv:2107.0333 03374, 2021a.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box altacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 15 26, 2017.
Shengehao Chen, Guodong Long, Tao Shen, and Jing Jiang. Prompt foderated learning for weather fore- casting: Toward foundation models on meteorological data. arXio preprint wxXiw:2301.0912. 2023a.
Shengchao Chen, Guodong Long, Tao Shen, Jing Jiang, and Chengqi Zhang. Federated prompt learning for weather foundation models on devices. Xrx preprint ar Xrxi::2305.142 14244, 2023b.
Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust foderated finetuning of foundation models via alternating minimization of lora. arXio preprint arXi::24000020 02346, 2024d.
38

---

Published in Transactions on Machine Learning Research (02/2026)
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt. Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint 1r9::2200.00112,,, 2021b.
Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Petzold, and William Yang Wang. survey on large language models for critical societal domains: Finance, healthcare, and law. arXiv preprint (.kk::205.017666 2024e.
Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. U-math: A university-level benchmark for evaluating mathematical skills in Ilms. arXiv preprint (tte::212.022000 2024.
Yac Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine- tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12903 12913, 2024.
Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. Do Ilms understand social knowl- edge? evaluating the sociability of large language models with socket benchmark. arviv preprinl arXiv:2305.14988 2023.
Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 53:5113 5155, 2020.
Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergej Averkiev, and Alena Fenogenova. Long input benchmark for russian analysis. arXiv preprint arXiv:2408. 00444,, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. orXiv preprint arXiv:1803.054 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichrro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ar preprint arXic:2110.14160 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xic, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-Luned 11m, 2023.
Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning, Xrxiv preprint arXiv:2405.0977 0977711,, 2024.
Hui Dai, Dan Pechi, Xinyi Yang, Garvit Banga, and Raghav Mantri. Deniahl: In-context features influence Ilm needle-in-a -eeyse-in-a-haystackk abilities. arXiv preprint 1rXiv:2411.19866 2024.
Jian Dai, Hao WI, Huan Lin, Liheng Yu, Xing Hu, Xiao Liu, and Daoying Geng. Fedata: Adaptive attention aggregation for federated sele-super medical image segmentation. Neurocomputing, 613:128691, 2025.
Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. Laiw: chinese legal large language models benchmark. arXiv preprint vrXiv:2310.05622 2023.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485-55) 532, 2020.
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 6087-6097, 2024.
39

---

Published in Transactions on Machine Learning Research (02/2026)
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-100 10115, 2023.
Jacob Devlin, Ming-Wiieree- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North, Jan 2019. doi: 10.18658/v1/n19-146 14233 URL http://dxx doi
100..0116////iiii -423
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Euhancing chat language models by scaling high-quality instructional conversations. arXiv preprint ar Xrx::23355.14 14238, 2023a.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficint fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-233, 2023b.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine- tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprinl arXiv:2002.0m 06305, 2020.
Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, and Yaliang Li. Tunable soft prompts are messengers in federated learning. Xixi preprind orXim:2311.06800 2023a.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehen- sive benchmark for evaluating long text modeling capacities of large language models. orXiv preprint arXiv:2309. 13345, 2023b.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A hamuully-craffeer benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:208..0 23l::200800861,, 2023.
Yichao Du, Zhirui Zhang, Linan Yue, Xu Huang, Yuqing Zhang, Tong Xu, Linli Xu, and Enhong Chen. Communication-effeeell personalized federated learning for speech-to-text lasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 100001-10000 IEEE, 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandcy, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint a777:2207.21188, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039 30069,, 2023.
Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. Medodyssey: A medical domain benchmark for long context evaluation up to 200k tokens. arXiv preprint (xxxx22406.1101 2024a.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Fushuo Huo, Jinyu Chen, and Song Guo. Overcome modal bias in multi-modal federated learning via balanced modality selection. In European Conference on Computer Vision, pp. 178-195. Springer, 2024b.
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodysscy: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. arXiv preprint arXiv:2406. 18321, 2024a.
40

---

Published in Transactions on Machine Learning Research (02/2026)
Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, and Yuguang Fang. Automated federated pipeline for parameter-efficient fine-tuning of large language models. arXiv preprint or fm:2404.06444 2024b.
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Gc. Lawbench: Benchmarking legal knowlodge of large language models. ar preprint (XXi::809916289) 2023.
Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8064 8073, 2023a.
Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, and Shrikanth Narayanan. Fedmultimodal: A benchmark for multimodal federated learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4035 4035-4045, 2023b.
Xiachong Feng, Xiaocheng Feng, Xiynan Du, Min-Yen Kan, and Bing Qin. Adapter-based selective knowl- edge distillation for federated multi-domain meeting summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024a.
Yu Feng, Zhen Tian, Yifan Zhu, Zongfu Han, Haoran Luo, Guangwei Zhang, and Meina Song. Cp-prompt: Composition- based cross-modal prompting for domain-incremennal continual learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2729-2738 2024b.
Joseph G. Flowers. Financeeinsttrut--500kk hwtpg///o..eereeiieeemeeeeeeeeee
2025.
URL. httpes///ugggggaacc.co/aaatata co/datasets/
Jörg Frohberg and Frank Binder. Crass: A novel data set and benchmark to test counterfactual reasoning of large language models. arXiv preprinl orXii:2112.111 1at::21221199.. 11941, 2021.
Xingbo Fu, Binchi Zhang, Yushun Dong, Chen Chen, and Jundong Li. Federated graph machine learning: A survey of concepts, techniques, and applications. ACM SIGKDD Explonations Newsletter, 24(2):32 47, 2022.
Xingbo Fu, Zihan Chen, Yinhan He, Song Wang, Binchi Zhang, Chen Chen, and Jundong Li. Virtual nodes can help: Tackling distribution shifts federated graph learning. arXiv preprint arXiv:2412.100 19229, 2024a.
Xingbo Fu, Zihan Chen, Binchi Zhang, Chen Chen, and Jundong Li. Federated graph learning with structure proxy alignment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Dala Mining, pp. 827-838, 2024b.
Xingbo Fu, Song Wang, Yushun Dong, Binchi Zhang, Chen Chen, and Jundong Li. Federated graph learning with graphless clients. arXiv preprint arXiv:2411.0837,, 2024c.
Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain thouggh hub: A contin- uous effort to measure large language models' reasoning performance. arXiv preprint arXiv:2305. 10000.173000 17306, 2023.
Fei Gao, Yunfeng Zhao, Chao Qiu, Xiaofci Wang, Haipeng Yao, and Qinghua Hu. Cp 2 gfed: Cross- granular and personalized prompt-based green federated tuning for giant models. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pp. 1-10. IEEE, 2024a.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb datasct of diverse text for language modeling. arXiv preprint XrXix:2101.000 2.1000007 2020.
Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. lim-based nlg evaluation: Current status and challenges. Xix preprint 0rX1322/02.01333, 2024b.
41

---

Published in Transactions on Machine Learning Research (02/2026)
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAT Conference Artificial Intelligence, volume 38, pp. 1860 1868, 2024c.
Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani. Communication-efliiedd and lensorized federated fine-tuning of large language models. orXiv preprinl arXiv:2410.1900 13097, 2024.
In Gim and JeongGil Ko. Memory--fficient dnn training on mobile devices. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, pp. 464-476, 2022.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint arXiv:2008.04489, 2020.
Shuai Gong, Chaoran Cui, Chunyun Zhang, Wenna Wang, Xiushan Nie, and Lei Zhu. Federated domain generalization via prompt learning and aggregation. arXiv preprint arXiv:2411.1006: 2024.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code rcasoning, understanding and exccution. arXiv preprint or arXiv:2401.03063 2024a.
Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18099 18107, 2024b.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex GoohlassWoood Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built bench- mark for mcasuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123 44279, 2023.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.0788 075997, 2023a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1mms via reinforcement learning. arxi preprint arXiv:2501.1294.. 2025.
Lei Guo, Ziang Lu, Junliang Yu, Quoe Viet Hung Nguyen, and Hongzhi Yin. Prompt-enhanced federated content representation learning for cross-domain recommendation. In Proceedings of the ACM on Web Conference 2024, pp. 3139-300 3149, 2024a.
Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. arXiv preprint arx:2 1tl:2410.0111 0..01.144... 2024b.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- languulan models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023b.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let foderated participants cooperatively learn prompts instead of models--ederatte learning in age of foundation model. 1EEE Transactions on Mobile Computing, 2023c.
Tao Guo, Song Guo, and Junxiao Wang. Explore and cure: Unveiling sample offectiveness with context-aware federated prompt tuning. IEEE Transactions on Mobile Computing, 2024c.
Shaunak Halbe, James Seale Smith, Junjiao Tian, and Zsolt Kira. Hepco: Data-free heterogencous prompt consolidation for continual federated learning. arXiv preprint ar ft::2306.09977 2023.
42

---

Published in Transactions on Machine Learning Research (02/2026)
Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca an open-source collection of medical conversational ai models and training data. arXiv preprint arXiu:2304.08247, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-effiiien fine-tuning for large models: comprehensive survey. arviv preprint arXiv:2403. 14608, 2024.
Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18188-18196, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint ar (Tfe:2105.09935 2021a.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: an expert-annotated nlp datasct for legal contract review. arXiv preprinl ar arXiv:2103.0663 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021c.
Masanori Hirano. Construction of japanese financial benchmark for large language models. arXie preprint arXiv:2403. 15062, 2024.
Zhaoyi Joey Hou, Li Zhang, and Chris Callison-Burch. Choice-75: A dataset on decision branching in script learning. arXie preprint arXiv:2309.1173: 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efliiien transfer learning for nlp. In International conference machine learning, pp. 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Jianheng Huang. Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXix:2403.0 01244, 2024a.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2): 1 1-55, 2025.
Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer Ilama technical report. arXiv preprint arXio:2805.1 15062, 2023a.
Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. IEEE Tronsactions on Pattern Analysis and Machine Intelligence, 2024b.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991-6388 -60000,,,, 2023b.
43

---

Published in Transactions on Machine Learning Research (02/2026)
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. ar preprinl ar vrrxi:1199.0 099.09436, 2019.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nancebench: A new benchmark for financial question answering. arXiv preprint arXim:211 1. 1.9944, 2023.
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Baochang Ma, and Xiangang Li. Belle: Be
everyone's large language model engine, 2023a.
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling large language models: An empirical study on real- world use cases, arXiv preprint ar Xiv:23003. 14742, 2023b.
Jingang Jiang, Xiangyang Liu, and Chenyou Fan. Low-parameter federated learning with large language models, 2023.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. Xix preprint or orXiv:2406. 00515, 2024a.
Yanna Jiang, Baihe Ma, Xu Wang, Guangsheng Yu, Ping Yu, Zhe Wang, Wei Ni, and Ren Ping Liu. Blockchained federated learning for internet of things: A comprehensive survey, ACM Computing Surveys, 56(10):1-37, 2024b.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang Hc, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprinl ar Xiv:24440.. 10739, 2024.
Rasmus Jørgenson, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. Multifin: A dataset for multilingual financial nlp. In Findings of the Association for Computational Linguistics: FACI 2023, pp. 894-909, 2023.
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A' novel" challenge for long-context language models. arXin preprint arxiv2206. 16264, 2024.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104. 14337, 2021.
Gyunycop Kim, Joon Yoo, and Sangwoo Kang- Efficient federated learning with pre-trained large language model using several adapter mechanisms. Mathematics, 11(21):4479, 2023a.
Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, and SangKeun Lee. Client-customized adap- tation for marameter-offiienn federated learning. In Findings of the Association for Computational Lin- guistics: ACI. 2023, pp. 1159-1172, 2023b.
Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. Medexqa: Modical question answering benchmark with multiple explanations. arXiv preprint orXiv.2406.0333 066331, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13))3521-33)) -3226, 2017.
Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. A survey of recommendation systems: recom- mendation models, techniques, and application fields. Electronics, 11(1):141, 2022.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Scan Hughos, Thomas Wolf, et al. The stack: tb of permissively licensed source code. arXiv preprint (xXi::2211.15533 2022.
Rik
Michael Krumdick, Viet Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. Bizbench: A quantitative reasoning benchmark for business and finance. arXiv preprint 2rX.022311.06602, 2023.
44

---

Published in Transactions on Machine Learning Research (02/2026)
Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients. arv preprint ar urxin:24 22.2212,2 2024.
Simon Kornblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 2661-2671, 2019.
Wei-Bin Kou, Qingfeng Lin, Ming Tang, Sheng Xu, Rongguang Ye, Yang Leng, Shnai Wang. Guofa Li, Zhenyu Chen, Guangxu Zhu, al. pfedlvm: A large vision model (Ivm)-driven and latent feature- personalized federated learning framework in autonomous driving, arXiv preprint arXiv:2405.0 04146, 2024.
Wei-Bin Kou, Qingleng Lin, Ming Tang, Rongguang Ye, Shuai Wang, Guangxu Zhu, and Yik- Chung Wu. Fast-convergent and caanummicaaiionallvvaaeee heterogeneous hierarchical federated learning in au- tonomous driving. TEEE Transactions on Intelligent Transportation. Systems, 2025.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-lIm: A comprehensive package for fine-tuning large language models in federated learning, 2023.
Karthik Kumar, Jibang Liu, Yung-Hsiang Lu, and Bharat Bhargava. A survey of computation offloading for mobile systems. Mobile nelworks and Ayplications, 18:129 140, 2013.
Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Federated lora with sparse communication. arXiv preprinl Xie:2440 05233, 2024.
Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of 1mms with long context reasoning-in-a-hayste Advances in Newral Information Processing Systems, 37:106519 106554, 2024.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18819-183444 PMLR, 2023.
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long- language models subsume retrieval, rag, sql, and more? ar preprint arXiv:2406.131 13121, 2024a.
Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. Ethic: Evaluating large language models on long-contex: tasks with high information coverage. arXiv preprint 2rXio:2410.16845 2024b.
Yang Lei, Jiangtong Li, Dawei Cheng, Zhijun Ding, and Changjun Jiang. Cfbenchmark: Chinese financial assistant benchmark for large language model. arXiv preprint orXix:2311.058 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiet prompt tuning, arXiv preprint Xrx::22104. 08691,, 2021.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a.
Guanghao Li, Wansen Wl, Yan Sun, Li Shen, Baoyuan Wu, and Dacheng Tao. Visual prompt based personalized federated learning. arXiv preprint arXiv:230..0807 08678, 2023b.
Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. Legalagentbench: Evaluating Ilm agents in legal domain. arXiv preprint 1Xii:2412211255 2024a.
Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. Xrxio preprint or (tl:2449.20288. 2024b.
45

---

Published in Transactions on Machine Learning Research (02/2026)
Heju Li, Rui Wang, Jun Wi, and Wei Zhang. Federated edge learning via reconfigurable intelligent surface with one-bit quantization. In GLOBECOM 2022-2022 IEFE Global Communications Conference, pp. 1055 1060. IEEE, 2022a.
Heju Li, Rui Wang, Wei Zhang, and Jun Wu. One bit aggregation for federated edge learning with reconfig- urable intelligent surface: Analysis and optimization. IEEE Transactions Wireless Communications, 22(2):872 888, 2022b.
Heju Li, Rui Wang, Jun Wu, Wei Zhang, and Ismael Soto. Reconfigurable intelligent surface empowered federated edge learning with statistical esi. IEFE Transactions on Wireless Communications, 23(6):6555- 6608, 2023c.
Heju Li, Rui Wang, Mingyang Jiang, and Jianquan Liu. Star-ris empowered heterogeneous federated edge learning with flexible aggregation. IEEE Internet of Things Journal, 2025.
Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12151-12161, 2024c.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey on benchmarks of multimodal large language models. arXiv preprint orXiv:2408.0800 08632, 2024d.
Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. Einatuo-66m, a large-scale chinese medical qa dataset. arXiv preprint orXhv:2305.015% 2023d.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311 1. 1..09991 2023c.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330 -4337,, 2023f.
Shenghui Li, Fanghua Ye, Meng Fang, Jiaxu Zhao, Yun-Hin Chan, Edith C-H Ngai, and Thiemo Voigt. Synergizing foundation models and federated learning: A survey, arXie preprint or 0rxiv:2406.120 2024e.
Shitian Li, Chunlin Tian, Kahou Tam, Rui Ma, and Li Li. Breaking on-device training memory wall: A systematic survey. arXiv preprint arXio:2306. 10388, 2023g.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Amect Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yuc, and Wenhu Chen. Long-context 1mms struggle with long in-context learning. arxiv preprinl arXiv:2404.000 02060, 2024f.
Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on Al in finance, pp. 374 182, 2023h.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexcy Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Fre- itas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, arXiv preprint Xrxi::200..0 07814, 2022c.
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model finc-tuned on a large language model meta-ai (Iaaaa using medical domain knowledge. Cureus, 15 (6), 2023i.
46

---

Published in Transactions on Machine Learning Research (02/2026)
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint Xrx::2211100111 2022.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024.
Li Lin, Yixiang Liu, Jiewei Wu, Pujin Cheng, Zhiyuan Cai, Kenneth KY Wong, and Xiaoying Tang. Fedlppa: Learning personalized prompt and aggregation for federated seakky-superrised medical image segmenta- tion. arXiv preprint Xrx:224022. 102..11500 17502, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arViv preprint Xrx:2 0900099....... 2021.
Zihao Lin, Yan Sun, Yifan Shi, Xueqian Wang, Lifu Huang, Li Shen, and Dacheng Tao. Efficient federated prompt tuning for black-box large pre-trained models. arXiv preprint xxX::2310000112, 2023.
Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprinl ar Xiv:2200111111 15089, 2025.
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models. arXic preprint or 2rxiv22355.102 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023b.
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application pro- ficiency of iims with hierarchical mathematics benchmark. arXiv preprint arXiv:2405. 12209, 2024a.
Jiale Liu, Yu-Wei Zhan, Chong- Yu Zhang, Xin Luo, Zhen-Duo Chen, Yinwei Wei, and Xin-Shun Xu. Fed- erated clasemnncremental learning with prompting, arXiv preprint arXic:2810.089. 2023c.
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang, Repoqa: Evaluating long context code understanding. arXiv preprint orviv:2406. 06025, 2024b.
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36:52430- 1452230-52452, 2023d.
Xiangyang Liu, Tianqi Pang, and Chenyou Fan. Federated prompting and -lain-of-thought reasoning for im- proving iims answering. In International Conference on Knowledge Science, Engineering and Management, pp. 3-11. Springer, 2023c.
Yi Liu, Xiaohan Bi, Lei Li, Sishuo Chen, Wenkai Yang, and Xu Sun. Communication efficient federated lcarning for multilingual neural machine translation with adapter. arXio preprint Xrx:::2305. 12449, 2023f.
Yuxi Liu, Guibo Luo, and Yuesheng Zhu. Fedfms: Exploring federated foundation models for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 283 293. Springer, 2024c.
Renze Lou, Kai Zhang, and Wenpeng Yin. Large language model instruction following: A survey of progresses and challenges. Computational Linguistics, 50(3):1053- 1095, 2024.
47

---

Published in Transactions on Machine Learning Research (02/2026)
Dakuan Lu, Hengkui Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, and Yanghua Xiao. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. arXiv preprint MxXic:2302.04432 2023a.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Newral Information Processing Systems, 35:2507 2521, 2022a.
Pan Lu, Liang Qiu, Kai- Chang, Ying Nian Wu, Song- Chgu Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXie:2209.1461 2022b.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXic preprint orXiv:2102.04600 04644 2021.
Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. arXiv preprint or XrXi::230.. 13485, 2023b.
Jun Luo, Chen Chen, and Shandong Wu. Mixture of experts made personalized: Federated prompt learning for vision-language models. Xiree preprint 0rXiv:2400.1000 2024.
Jialiang Ma, Chunlin Tian, Li Li, and Chengzhong Xu. Fedin:: A federated multi-global optimization
framework for autonomous driving control. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pp. 1-10. IEEE, 2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-211 2023.
Yuting Ma, Lechao Cheng, Yaxiong Wang, Zhun Zhong, Xiaohua Xu, and Meng Wang. Fedhpl: Efficient hel- erogeneous federated learning with prompt tuning and logit distillation. are preprint orXiv:2405.1120 2024b.
Seiji Mackawa, Hayate Iso, and Nikita, Bhutani. Holistic reasoning with long-context Ims: benchmark for database operations on massive textual data. arXiv preprint arXiv:2410. 11996, 2024.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'l8 open challenge: financial opinion mining and question answering. In Com- ponion proceedings of the the web conference 2018, pp. 1941-1942, 2018.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just. forward passes. Advances in Neural Information Processing Systems, 36:53038- 53075, 2023.
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations cconomic texts. Journal of the Association for Information Science and Technology, (44:7782,77 2014.
Yujun Mao, Yoon Kim, and Yilun Zhou. Champ: A competition-level dataset for fine-grained analyses of llsss mathematical reasoning capabilities. arXio preprint wxXii:2401.06961 2024.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Commmiictiion-fftiienn learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1273-1284 PMLRR 2017.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Mor- gane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélic Hélion, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christo- pher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Tppolito, David Reid, Elena Buchatskaya,
48

---

Published in Transactions on Machine Learning Research (02/2026)
Eric Ni, Eric Noland, Geng Yan, George Tucker, Ceorge-Christien Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Tan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean- BaaanBae Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini rescarch and technology. CoRR, abs/ 0t////243..02200 2024. doi: 10.48550/ D...XIV.403308880 URL http:://doi..rr.. http///00..055/........v. 240r.. 04403.08295
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint 0tfi:22110.11111 2022.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the po- tential of slms in grade school math, 2024.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Muzammal Naseer and Karthik Nandakumar. Probing the efficacy of federated parameter-effictt fine- tuning of vision transformers for medical image. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2024 Workshops: ISIC 2024, iMIMIC 2024, EARTH 2024, DeCaF 2024, Held in Conjunction with MICCAI 2024, Marrakesh, Morocco, Oclober 6- 60, 2024, Proceedings, pp. 236. Springer Nature.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 2023.
Dinh C. Nguyen, Quoc-Vic Pham, Pubudu N Pathirana, Ming Ding, Aruna Sencviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang. Federated learning for smart healthcare: A survey. ACM Computing Surveys (Csur), 55(3):1-33 2022.
Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, et al. Cfinbench: A comprehensive chinese financial benchmark for large language models. arXiv preprint orXic:2477.03011 2024.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13342033134444. 13474, 2022.
Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Tlias Chalkidis. Lextreme: A multi-lingual and multi-task benchmark for the legal domain. arXiv preprint Xxiw:2301.13126, 2023.
Zhiyuan Ning, Chunlin Tian, Meng Xiao, Wei Fan, Pengyang Wang, Li Lá, Pengfei Wang, and Yuanchun Zhou. Fodges: A generative framework for efficient client selection in foderated learning via gradicnt- basod optimization. arXiv preprinl ar XrX:::2405. 06312, 2024.
OpenAL. GPT-4 technical report. CoRR, abs /a0ps//333308778 2023. doi: 10.48550/ARX// AB0 2303.08777. URL
https://doi.... http:///6b...rg/....886///////// http//rXivv.00 http55/aaXxvv2233.00
OpenAI. Hello GPT-40, 2024.
Xiaomin Ouyang, Zhiyuan Xic, Heming Fu, Sitong Cheng, Pan, Neiwen Ling, Guoliang Xing, Jiayu Zhou, and Jianwei Huang, Harmony: Heterogeneous multi-modal federated learning through disentangled model training. In Proceedings of the 21st Annual International Conference Mobile Systems, Applications and Services, pp. 530 543, 2023.
Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, al. Markllm: An open-source toolkit for 1mm watermarking. arXiv preprint arXiv:2405.1000 10051, 2024a.
49

---

Published in Transactions on Machine Learning Research (02/2026)
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: layerwise im- portance sampling for memory-efficient large language model fine-tuning. Advances in Newral Infor Information Processing Systems, 37:57018 57049, 2024b.
Guilherme Penedo, Anton Lozhkov Hynek Kydliček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarin, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. 2025.
Baolin Peng, Chumyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint ar orXim220040034 03277,, 2023.
Yuanzhe Peng, Jieming Bian, and Jie Xu. Fedmm:: Federated multi-modal learning with modality hetero- geneity in computational pathology. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1696 1700. IEEE, 2024.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint ar arXiv:007.007779, 2020.
Sai Puppala, Ismail Hossain, Md Jahangir Alam, and Sajedul Talukder. Scan: healthcare personalized chatbot with federated learning based gpt. In 2024 IEEE 48th Annual Computers, Software, and Appli- calions Conference (COMPSAC), pp. 1945 1951 IEEE, 2024.
Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: Person- alized federated learning of large language model via dual lora tuning. arXiv preprint orXiv:2406.0799 07925, 2024a.
Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang. Hao Zhang, and Wei Xu. Long2rag: Evalu- ating long-context & long-form form retrieval-angmented generation with key point recall. arXiv preprint arXiv:2410.30 23000, 2024b.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? Xr preprint ar arXim22407.01255 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprinl XrXiv:2307. 16789, 2023a.
Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Federated full- parameter tuning of billion-sized language models with communication cost under 18 kilobytes. arXiv preprint arXiv:2312.000 06355, 2023b.
Zhen Qin, Zhaomin Wu, Bingsheng He, and Shuiguang Deng. Federated data-efficient instruction tuning for large language models. arXiv preprint Xr 10926, 2024.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan- Yi Lin. Text-driven prompt generation for vision-language models in federated learning. arXie preprinl arXim:2300006 23, 2023.
Yinzhu Quan and Zefang Liu. Econlogicqa: A question-answering benchmark for evaluating large language models in economic sequential reasoning. arXiv preprint arXiv:2105.07993 079938 2024.
Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E Ho, and Joel Niklaus. Scale: Scaling up the complexity for advanced language model evaluation. arXiv preprint arXiv:2306.0922 2023.
Varshini Reddy, Rik
Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. Docfinga: A long-context financial reasoning dataset. arXiv preprint wxxi:2401.06911 2024.
50

---

Published in Transactions on Machine Learning Research (02/2026)
Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Anran Li, Yulan Gao, Alysa Ziying Tan, Bo Zhao, Xiaoxiao Lá, Zengxiang Li, et al. Advances and open challenges in federated learning with foundation models. arXiv preprint 2XXe:2244411381,, 2024.
Fahad Sabah, Yuwen Chen, Zhen Yang, Muhammad Azam, Nadeem Ahmad, and Raheem Sarwar. Model optimization techniques in personalized foderated learning: A survey. Expert Systems with Applications, 243:122874, 2024.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL hattpa://arriv.orr//bb///// 19t1//1110000001.1111111
SM Sarwar. Fedmentalcare: Towards privacy-preserving fine-tuned llms to analyze mental health status using federated learning framework. arXiv preprint WxXi:2503.057866 2025.
Naser Shabani. Harnessing federated learning for 1lm fine-tuning: A distributed approach, 2024.
Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. Finer: Financial named entity recog- nition dataset and weak-supervision model. arXiv e-prints, pp. arXiv 2302, 2023.
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. When flue meets flang: Benchmarks and large pre-trained language model for financial domain. arXiv preprint rrXii:2211.00088 2022.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zcro-shot benchmark for long text understanding. arXiv preprint arXiv:2305.1410 14196, 2023.
Sh Sharegpt-dhinnss--mgiihss-. bilingual human-mschine qa dataset. https://huggingface
Gataattt//aaraal//aaaeee7T-oeeee -ha 2023.
Soumya Sharma, Tapas Nayak, Arusarka Bose, Ajay Kumar Meena, Koustuv Dasgupta, Niloy Ganguly, and Pawan Goyal. Finred: dataset for relation extraction in financial domain. In Companion Proceedings of the Web Conference 2022, pp. 595 597, 2022.
Ming Shen. Rethinking data selection for supervised fine-tuning. arXiv preprint NNN:2442200609, 2024.
Jiyun Shin, Jinhyun Ahn, Honggu Kang, and Joonhyuk Kang. Fedsplitx: Federated split learning for computationalll--cnnsttenn heterogeneous clients. arXiv preprint arXit:2310.14572 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis Stephen Pfohl, al. Large language models encode clinical knowledge. Nalure, 620(7972):172-16) 180, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Marriga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilitics language models. arXio preprint arXiv:2206. 04615, 2022.
Shangchao Su, Bin Li, and Xiangyang Xue. Fedra: A random allocation strategy for federated tuning to unleash the power of heterogencous clients. arXiv preprint arXiv:2311.11227 2023.
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi- domain collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volune 38, pp. 15115, 2024.
Guangyu Sun, Umar Khalid, Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, and Chen Chen, Conquering the communication constraints to enable large pre-trained models in federated learning. arXiv preprint arXim:2210.011008 2022.
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang. Daguang Xu, Yiran Chen, and Holger R Roth. Fodbpt: Efficient federated black- box prompt tuning for large language models. arXie preprint orXiw:2310.01407 2023.
51

---

Published in Transactions on Machine Learning Research (02/2026)
Yuchang Sun, Yuexiang Xie, Bolin Ding, Yaliang Li, and Jun Zhang. Exploring selective layer fine-tuning in federated learning. Xiv preprint or Nrv:2:2408. 15600, 2024.
Kahou Tam, Li Li, Bo Han, Chengzhong Xu, and Huazhu Fu. Federated noisy client learning. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Kahou Tam, Li Li, Yan Zhao, and Chengshong Xu. Fedcoop: Cooperative federated learning for noisy labels. In ECAI 2023, pp. 2298-2306 IOS Press, 2023b.
Kahou Tam, Chunlin Tian, Li Li, Haíkai Zhao, and ChengZhong Xu. Fedhybrid: Breaking the memory wall of federated learning via hybrid tensor management. In Proceedings of the 22nd ACM Conference on Embeddded Networked Sensor Systems, pp. 394 408, 2024a.
Kahou Tam, Kewei Xu, Li Li, and Huazhu Fu. Towards federated domain unlcarning: Verification method- ologies and challenges. arXiv preprinl Xix:22406. 03078, 2024b.
Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, et al. Medlagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning, arXiv preprint arXiv:2503.074 07459, 2025.
Zecheng Tang, Keyan Zhou, Juntao Li, Baibei Ji, Jianye Hou, and Min Zhang. L-citeeval: Do long-context models truly leverage context for responding? arXiv preprint 1.:::2410002115. 2024a.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint Xix: 02888, 2024b.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following Ilama model, 2023.
Omid Tavallaie and Niousha Nazemi". Rbla: Baunk-based--ore--gggeetttt for fine-tuning heterogeneous models. In Web Services ICWS 2024: 31st International Conference, Held as Part of the Services Con- ference Federation, SCF 2024, Bangkok, Thailand, November 16-19, 2024, Proceedings, pp. 47. Springer Nature.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptizs Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. XrXiv preprint ar xrx::23312.1 11805, 2023.
Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtope, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8485 8493, 2022.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Flangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1) 1940, 2023.
Chunlin Tian, Li Li, Zhan Shi, Jun Wang, and ChengZhong Xu. Harmony: Heterogeneity-aware hierar- chical management for federated learning system. In 2022 55th IEEE/ACM International Symposium Microarchitecture (MICRO), pp. 631-645. IEEE, 2022a.
Chunlin Tian, Zhan Shi, and Li Li. Learn to select: Efficient cross-device federated learning via reinforcement learning. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwando, May 5, 2023. OpenReview. net, 2023. URL
https://openreviev net//forummkdddxectsaiitt http:///gggaaoooo....eeccocotttr
Chunlin Tian, Li Li, Kahou Tam, Yebo Wu, and Cheng-Zhong Xu. Breaking the memory wall for hetero- gencous federated learning via model splitting. IEEE Transactions on Parallel and Distributed Systems, 2024a.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora archi- tecture for efficient. fine-tuning. Xiu preprint arXiv:2404.1924 2024b.
52

---

Published in Transactions on Machine Learning Research (02/2026)
Chunlin Tian, Zhan Shi, Li Li, Cheng-zhong Xu, et al. Ranking-baseer client imitation selection for efficient federated learning. In Forty-first International Conference on Machine Learning, 2024c.
Chunlin Tian, Xinpeng Qin, Kahou Tam, Li Li, Zijian Wang, Yuanzhe Zhao, Minglei Zhang, and Chengzhong Xu. Clone: Customizing Ilms for efficient latency-aware inference at the edge. arXiv preprint arXiv:2506. 022477 2025.
Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. Fedbert: When federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4): 1-26, 2022b.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2022.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiram, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-I: 1.8 million math instruction tuning datasct. arXiv preprint arXiv: Arriv- 20176 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, el al. Llama: Open and efficient foundation language models. arXiv preprint ar Xix:20302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint or orXiv:2307.09250 2023b.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong, Solving olympiad geometry without human demonstrations. Nature, 625(5595):476 482, 2024.
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical Guide, 1sl Ed., Cham: Springer International Publishing, 10(3152676):00 105555555 2017.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint wxXi:2309.11235, 2023a.
Haifeng Wang, Hua Wu, Zhongjun He, Liang Huang, and Kenneth Ward Church. Progress in machine translation. Engineering, 18:143-133 153, 2022a.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXix:2002.064 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistencc problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611 7623, 2020b.
Jie Wang, Yebo Wu, Erwu Liu, Xiaolong Wu, Xinyu Qu, Yuanzhe Geng, and Hanfu Zhang, Fedins2: A seanteddeddg-leaaring based inertial navigation system with segment fusion. IEEE Internet of Things Journal, (1(():36533366)) 2023b.
Jie Wang, Xiaolong Wu, Jindong Tian, Erwu Liu, Yebo Wu, Rucong Lai, and Yong Tian. Indoor localization fusing inertial navigation with monocular depth estimation in federated learning framework with data heterogencity. IEEE Transactions on Instrumentation and Measurement, 2025.
Lei Wang, Shan Dong, Yuhui Xu, Hanze Dong, Yalu Wang, Amrita Saha, Ee-Peng Lim, Caiming Xiong, and Doyen Sahoo. Mathhay: An automated benchmark for long-context mathematical reasoning in llms. aiviv preprinl Xiv:24 10. 04698, 2024a.
Lun Wang, Yang Xu, Hongli Xu, Zhida Jiang, Min Chen, Wuyang Zhang, and Chen Qian. Bose: Block-wise federated learning in heterogeneous edge computing. IEEE/ACM Transactions Networking, 2023c.
53

---

Published in Transactions on Machine Learning Research (02/2026)
Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. arXiv preprint arXiv:2406.17413 2024b.
Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, and Ricardo Henao. Personalized federated learning for text classification with gradient-free prompt tuning. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 4597- 4612, 2024c.
Shigiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Teung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications, 37(6):1205-1221) 2019a.
Xiaochen Wang, Jiaqi Wang, Houping Xiao, Jinghui Chen, and Fenglong Ma. Fedkim: Adaptive federated knowledge injection into medical foundation models. arXiv preprint arXiv:2408.. 10276, 2024d.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arxie:2307. 10635, 2023d.
Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint Xrxi::2308.00 08833, 2023e.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for 11m instruction tuning optimization. arXiv preprint Xii:2330600088 2023f.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXio preprinl 1.Niv22212110600 2022b.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution- evaluation for open-domain code generation. arXie preprint arXiv:2212. 10481, 2022c.
Zi Wang, Fei Wu, Feng Yu, Yurui Zhou, Jia Hu, and Geyong Min. Federated continual learning for edge-ai: A comprchensive survey. arXie preprint arXiv:2411. 13740, 2024c.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.04 732, 2019b.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprinl MxXiv:2009009970 2024f.
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 3577):5731-5780, 2022.
Alex Watson, Yev Meyer, Maarten Van Segbrocek, Matthew Grossman, Sami Torbey, Piotr Mlocek, and Johnny Greco. ing language models to label and detect. pii in domain specific formats, June 2024. URL https:
A synthetic datasct for train-
//huggingface /tt/aaatsstss/geeeeaaa/////aaaaa
Guoyizhe Wei, Feng Wang, Anshul Shah, and Rama Chellappa. Dual prompt tuning for domain-aware federated learning. arXie preprint NNN::3310003100 2023.
Pai Weng, Minh Hoang, Lam M Nguyen, My T Thai, Tsui-We Weng, and Trong Nghia Hoang. Proba- bilistic federated prompt-tuning with non-iid and imbalanced data. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems.
54

---

Published in Transactions on Machine Learning Research (02/2026)
Herbert Woisetschläger, Alexander Isenko, Shiqiang Wang, Ruben Mayer, and Hans- Jacobsen. A survey on efficient federated learning methods for foundation model training. arXiv preprint or arXim:2401.0/ 04472. 2024.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Bench- marking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410 10813, 2024a.
Foijic Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3345 3355, 2024b.
Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, and Jing Gao. On the client preference of 1lm fine- tuning in federated learning. arxic preprint 2rXiv:2407.0300 2024c.
Xian Wu, Yutian Zhao, Yunyan Zhang, Jiageng Wu, Zhihong Zhu, Yingying Zhang, Yi Ouyang, Ziheng Zhang, Huimin Wang, Jie Yang, et al. Medjourney: Benchmark and evaluation of large language models over patient clinical journey. Advances in Neural Information Processing Systems, (1187621-87644 2024d.
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Jummin Zhu, and Wei Zhang, Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. Xiv preprint orXic:2411.00077 2024e.
Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, and Guogang Zhu. Fedlora: When personalized federated learning meets low-rank adaptation. 2024f.
Xinghao Wu, Jianwei Niu, Xuefeng Liu, Mingjia Shi, Guogang Zhu, and Shaojie Tang. Tackling feature- classifier mismatch in federated learning via prompt-driven feature transformation. arXiv preprint arXiv:2407.16139, 2024g.
Yebo Wu, Li Li, Chunlin Tian, Tao Chang, Chi Lin, Cong Wang, and Cheng-Zhong Xu. Hetorogencity-awaaee memory efficient federated learning via progressive layer freezing. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pp. 1-10. IEEE, 2024h.
Yebo Wu, Li Li, Chunlin Tian, Dubing Chen, and Chengzhong Xu. Neulite: Memory-efficient federated learning via elastic progressive training. arXiv preprint arXiv:2408. 10826, 2024i.
Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li. Elastic mixture of rank-wise experts for knowledge reuse in federated fine-tuning. Xiv preprint arXiv:2512.0099 009002, 2025a.
Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li. Learning like humans: Resourco-cfffcient foderated fine- tuning through cognitive developmental stages. arXiv preprinl or X000 2025b.
Yebo Wu, Jingguang Li, Chunlin Tian, Zhijiang Guo, and Li Li. Memory-efficient federated fine-tuning of large language models via layer pruning. arXio preprint 250 1 17209, 2025c.
Yebo Wu, Li Li, and
Xu. Breaking the memory wall for heterogeneous federated learning via progressive training. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 1623 1632, 2025d.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Pong, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: large language model, instruction data and evaluation benchmark for finance. arXio preprinl arXiv:2306. 23X::22306.05443, 2023.
Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. Finber: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems, 37:95716 95743, 2024.
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du Parameter- efficient fine-tuning for pre-trained vision models: A survey. aixio preprint S5.:4020022244 2024.
55

---

Published in Transactions on Machine Learning Research (02/2026)
Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, et al. De-lore: Dual queries with low rank approximation re-ranking for in-context learning, arXiv preprint Xrxi222310. 02954, 2023.
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Lingpeng Kong, et al. Uncomp: Uncertainty-aware long-context compressor for efficient large language model inference. arXiv preprinl arXiv:24 10. 030900 2024.
Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, and Ngai Wong. Parallelcomp: Parallel long-context compressor for length extrapolation. arXiv preprint arXiv:2502. 14317, 2025.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley, Baize: An open-source chat model with parameter- efficient tuning on self-chat data. arXiv preprint arXiv:2304.01 196, 2023b.
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? comprehensive evaluation from deductive, inductive and abductive views. arXiv preprinl ar 20::::200009844 2023c.
Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. Superclue: A comprehensive chinese large language model benchmark. arXiv preprint ar Xiv:22307 15020, 2023d.
Liang Xu, Lei Zhu, Yaotong Wu, and Hang Xue. Superclue-fin Graded fine-grained analysis of chinese 1lms on diverse financial tasks and applications. arXiv preprint arXiv:2404.19063 2024a.
Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang. Federated fine-tuning of billion- sized language models across mobile devices. arXie preprint ar FFf::2308.18894 2023c.
Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, and Shangguang Wang- (FwdLLM)) Effficint feder- ated finetuning of large language models with perturbed inferences. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pp. 579 596, 2024b.
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource- efffccient 1lm and multimodal foundation models. arXiv preprint arXiv:2401.0000 080000, 2024c.
Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, et al. Copyrightmeter: Revisiting copyright protection in text-to-image models. arXiv preprint ar 1r1.::4411.1814, 2024d.
Naen Xu, Jinghuai Zhang, Changjiang Li, Zhi Chen, Chunyi Zhou, Qingming Li, Tianyu Du, and Shouling Ji. Vidcoeraser: Concept crasure toxt-to-video diffusion models. In Proceedings of the 2025 Conference on Empirical Melhods in Natural Language Processing, pp. 5965-556 5964, 2025.
Naen Xu, Hengyu An, Shuo Shi, Jinghuai Zhang, Chunyi Zhou, Changjiang Li, Tianyu Du, Zhihui Fu, Jun Wang, and Shouling Ji. When agents" misremember' collectively: Exploring the mandela effect in Ilm-based multi-agent systems. arXiv preprint or Ni1:226022004 2026a.
Naen Xu, Jinghuai Zhang, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Zhaoxiang Wang, and Shouling Ji. Fraudshield: Knowledge graph empowered defense for lims against fraud attacks. anXiv preprint Xiv::26011.22 22448, 2026b.
Zhe Xu, Jiasheng Ye, Xiaoran Liu, Xiangyang Liu, Tianxiang Sun, Zhigeng Liu, Qipeng Guo, Linlin Li, Qun Liu, Xuanjing Huang, et al. Detectiveqa: Evaluating long-context reasoning on detective novels. arXiv preprint ar Xrxx::449.002// 2024e.
56

---

Published in Transactions on Machine Learning Research (02/2026)
Yuxuan Yan, Qianqian Yang, Shumpu Tang, and Zhiguo Shi. Federa: Efficient fine-tuning of language models in federated learning leveraging weight decomposition. arXiv preprint or arxin:2404. 18848, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint urXiv:2505.09388, 2025.
Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompl generation. In Proceedings of the IEEE/CVI International Conference on Computer Vision, pp. 19159-19968,, 2023a.
Jianxin Yang. Firefly: Chinese conversational large language models. httpss//ggthubb.co//agggaaaiii Firefly 2023.
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang, Glue-x: Evaluating natural language understanding models from an outtr--distributiin generalization perspective. arXiv preprint arXiv:2211.0807. 2022.
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 19366, 2024a.
Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein, Dual-personalizing adapter for federated foundation models. Xr preprint urXio:2403. 19211, 2024b.
Yuning Yang, Xiaohong Liu, Tianrun Gao, Xiaodong Xu, and Guangyu Wang. Sa-fedlora: Adaptive param- eler allocation for elficient federated learning with lora tuning. arXie preprinl urXiv:240500939 2024c.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint ar 2tw.2339.11422. 9(1):
1, 2023b.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Dezhong Yao, Wamning Pan, Michael J O'Neill, Yutong Dai, Yao Wan, Hai Jin, and Lichao Sun. Fodhm: Efficient federated learning for heterogeneous models via low-rank factorization. arXiv preprint arXiv:2111.14655, 2021.
Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungehul Kim, Ryan Rossi, Ang Li, et al. Federated large language models: Current progress and future directions. arXiv preprint ar Xix::22409. 15723, 2024.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State of-the-art and research challenges. ACM Computing Surveys, 56(3): 44, 2023.
Rongguang Ye, Wei-Bin Kou, and Ming Tang. Praffl: A preference-aware scheme fair federated learning. arXiv preprint ar Xrv::22404.0080 089773, 2024a.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda. Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6137-6147, 2024b.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint ar Xix: 10. 10.02694, 2024.
57

---

Published in Transactions on Machine Learning Research (02/2026)
Jingwei Yi, Fangzhao Wu, Huishuai Zhang, Bin Zhu, Tho Qi, Guangzhong Sun, and Xing Xic. Robust quantity-aware aggregation for federated learning. arXiv preprint arXiv:2205. 10848, 2022.
Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heeeroogeeeuus personalized federated learning with lora tuning. arXic preprint arXiv:2310.1322 2023.
Xiaoyang Yi, Jian Zhang, Jing Chen, Yuru Bao, and Lingkai Xing. Fedfid: Heterogeneous federated learning via forget-less distillation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Xunjian Yin, Baizhou Huang, and Xiaojun Wan. Alcuna: Large language models meet new knowledge. arXiv preprinl ar Xin:2330. 14820, 2023.
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pp. 12073 12086. PMLR, 2021.
Hao Yu, Xin Yang, Xin Gao, Yan Kang, Hao Wang, Junbo Zhang, and Tianrui Li. Personalized federated continual learning via multi-gramularit prompt. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4023 0123-4034, 2024.
Jilan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint Xiv:22336. 09296, 2023a.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint ar arXiv:2309. 1 12284, 2023b.
Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via contrastive representation ensemble. arXiv preprint ar 2302. 08888, 2023c.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingiun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-souree IIm reinforcement lcarning system at scale, 2025. URL https: httpp////aaxiv..o//bb/////// 24476
Sixing Yu, Pablo Muñoz, and Ali Jannesari. Bridging the gap between foundation models and heterogeneous federated learning. aixe preprint orXiv:2310.000 00247, 2023d.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. A survey of knowledge-enhanced text generation. ACM Computing Surveys, 54(11s):1-335 38, 2022.
Liangqi Yuan, Ziran Wang, Lichao Sun, S Yu Philip, and Christopher G Brinton. Decentralized federated learning: A survey and perspective. IEEE Internet of Things Journal, 2024a.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan robustness in nlp: Benchmarks, analysis, and 1lms
Liu, and Maosong Sun. Revisiting evaluations. Advances in Neural Information Processing Systems, 36:58478 58507, 2023.
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint ar Xrx::24422.051 2024b.
Wei Yuan, Chaoqun Yang, Guanhua Ye, Tong Chen, Nguyen Quoe Viet Hung, and Hongzhi Yin. Fellas: Enhancing federated sequential recommendation with Ilm as external services. ACM Transactions on Information Systems, 2024c.
58

---

Published in Transactions on Machine Learning Research (02/2026)
Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, et al. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXic:2309. 13325, 2023a.
Xiang Yuc, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction luning. arxiv preprinl urXiv:2309.0566 2023b.
Flad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficiett fine-tuning for transformer- masked language models. arXiv preprint orXiv2206.101190 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprinl ar Xxxx:1905.07830 2019.
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. Meddialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 9241-9250, 2020.
Hui Zeng. Measuring massive multitask chinese understanding. Xr preprint arXiv:2304.12985 2023.
Huimin Zeng, Zhenrui Yue, Qian Jiang, and Dong Wang. Federated recommendation via hybrid retrieval augmented generation. Xiv preprint or arxi::2033.04 042.000,0.26... 2024a.
Juntao Zeng, Bo Chen, Yuandan Deng, Weiqin Chen, Yanlin Mao, and Jiawei Li. Fine-tuning of financial large language model and application at odge device. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 42 47, 2024b.
Shichen Zhan, Yebo Wu, Chunlin Tian, Yan Zhao, and Li Li. Heterogeneity-awall coordination for federated learning via stitching pre-trained blocks. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pp. 1 10. IEEE, 2024.
Boyu Zhang, Hongyang Yang, and Xiao Yang Liu. Instruct-fnnpt: Financial sentiment analysis by instruc- tion luning of general-purpose large language models. FinLLM Symposium al IJCAI 2023, 2023a.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.1360 2024a.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgptt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6915 6919. IEEE, 2024b.
Kepu Zhang, Weijie Yu, Sunhao Dai, and Jun Xu. Citalaw: Enhancing Ilm with citations in legal domain. arXiv preprint Xrx::22412. 14556, 2024c.
Lei Zhang, Yunshui Li, Ziqiang Liu, Junhao Liu, Longze Chen, Run Luo, Min Yang, et al. Marathon: A race through the realm of long context with large language models. arXiv preprint arXiv:2312.095. 2023b.
Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. Xiv preprint or Xiv:2330 09975. 2023c.
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, et al. Cblue: A chinese biomedical langnage understanding evaluation benchmark. arXiv preprint ar Xrxi: 080887, 2021.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint ar arxi2:2401 023355 2024d.
59

---

Published in Transactions on Machine Learning Research (02/2026)
Pengyu Zhang, Yingbo Zhou, Ming Hu, Junxian Feng, Jiawen Weng, and Mingsong Chen. Personalized federated instruction tuning via neural architecture search. orXiv preprint arxiv:2022. 16919, 2024e.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024f.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. Bayling: Bridging cross-lingun alignment and instruction following through interactive translation for large language models. arxiv preprint arXiw:2306. 10968, 2023d.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXio preprint OrXiv:2308.10770 2023e.
Wenxuan Zhang, Mahani Aljumied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Newral Information Processing Systems, 36:5484 5505, 2023f.
Xiao Zhang, Ruoyu Xiang, Chenhan Yuan, Duanyu Feng, Weiguang Han, Alejandro Lopez-Lira, Xiao Yang Liu, Meikang Qiu, Sophia Ananiadou, Min Peng, et al. Dólares or dollars? unraveling the bilingual prowess of financial Iims between spanish and english. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6236-6246, 2024g.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer Ilm evaluators. arXic preprint Xrxv2:230 01862, 2023g.
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare: Instruction-tuned large language models for medical application. Xr preprint ar Xiv:22300 14558, 2023h.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. mftybeech: Extending long context evaluation beyond 100k tokens. In ACL (1), 2024h.
Yicheng Zhang, Zhen Qin, Zhaomin Wu, and Shuiguang Deng. Personalized federated fine-tuning for Ilms via data-driven heterogencous model architectures. arXiv preprint 1XX.:2411.19118, 2024i.
Yifci Zhang, Dun Zeng, Jinglong Luo, Xinyu Fu, Guanzhong Chen, Zenglin Xu, and Irwin King. A survey of trustworthy federated learning: Issues, solutions, and challenges. ACM Transactions on Intelligent Systems and Technology, 15(6):1-47, 2024j.
Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. Analyzing temporal complex events with large language models? a benchmark towards temporal, long context understanding. arXiv preprint ar Xiv:24400 02472, 72, 2024k.
Zikai Zhang, Jiahao Xu, Ping Liu, and Rui Hu. Fed-pilot: Optimizing lora assignment for efficient federated foundation model fine-tuning. Xiv preprint arXic:2.10 10. 10200, 20241.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communccttion-cffiiint and preveci-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Jujia Zhao, Wenjie Wang, Chen Xu, Zhaochun Ren, See-Kiong Kiong Ng, and Tat-Seng Chua. Llm-based federated recommendation. arXio preprint ar arXiv:2402.0000 2024a.
Wanru Zhao, Yihong Chen, Royson Lee, Xinchi Qiu, Yan Gao, Hongxiang Fan, and Nicholas Donald Lanc. Breaking physical and linguistic borders: Multilingual foderated prompt tuning for low-resource languages. Tn The Twelith International Conference on Learning Representations, 2024b.
60

---

Published in Transactions on Machine Learning Research (02/2026)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar preprinl arXiv:2303. 18223, 2023b.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-lovel mathematics. arXic preprint arXw:22000000110 2021.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as- a-judge with mt-bench and chatbot arena, 2023a.
Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. Doc2edag: An end-to-end document-level framework for chinese financial event extraction, arXiv preprint 17545:590,,07555, 2019.
Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.1037 2023b.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. ar preprinl arXiv:2304.06: 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint 2xx::231.009111 2023.
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. Re- trieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXi:2101.00774, 2021.
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv e-prints, pp. arXiv-2306, 2023a.
Wei Zhu, Xiaoling Wang, and Longyue Wang. Chatmed: A chinese medical large language model. Retrieved September, 18:2023, 2023b.
WY Wei Zhu and Xiaoling Wang. Shennong-tem: A traditional chinese medieine large language model. GilHub, 2023.
Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model moets federated learning: Moti- vations, challenges, and future directions. arXis preprinl arXiv:2306.1555 15546, 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
Kaijian Zou, Muhammad Khalifa, and Lu Wang. Retrieval or global context understanding? on many-shot in-context learning for long-context evaluation. arXie preprint orXiv:2,11.0011 2024.
Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuckai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchnarking expert-level medical reasoning and understanding. arxiv preprint drXio:2501.18322 2025.
61


Reviews:
[{"review_id": "GWssAqjwju", "reviewer": "Reviewer_bgda", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper comprehensively summarizes the federated fine-tuning of LLMs. It analyzes four main challenges (communication overhead, data heterogeneity, memory constraints, and computation burden) in FedLLM and provides five different federated fine-tuning methods. Different from current existing surveys, it’s the first survey to organize the evaluation datasets and benchmarks for federated fine-tuning of LLMs. Furthermore, it proposes five opening research directions to contribute future federated fine-tuning of LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper covers almost all existing works in federated fine-tuning of LLMs, and clearly organize them into different categories. For example, for LoRA in federated fine-tuning, it utilizes three scenarios (homogeneous LoRA, heterogeneous LoRA, personalized LoRA) to conclude the works.", "audience_yn": "Yes", "audience_explanation": "The paper is well-written and easy to follow, and it's a comprehensive survey including and organizing almost all existing works in federated fine-tuning of LLMs.", "requested_changes": "1. The text size in Table 9, Table 10, and Table 11 is too small for readers. \n\n2. In the Introduction section, the paper organizes LLM fine-tuning paradigms into three approachs, which are central fine-tuning, local fine-tuning, and federated fine-tuning. It’s not clear to me about the advantages of federated fine-tuning, compared to local fine-tuning. Does the global model trained on all clients have the same good performance with the local fine-tuned model? Since there may have some disalignment when merging all clients weights, which may affect the performance of the global model on some specific clients. Can authors explain this clearly? Also, the goal of some federated fine-tuning methods is not to have one single global model but to have several global models. The Figure 1(c) may consider this point.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JqWMHUfHZi", "reviewer": "Reviewer_UmEG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys federated fine-tuning of LLMs (FedLLM). It (1) outlines background on LLM training and why centralized and local fine-tuning fall short, then (2) analyzes four practical bottlenecks—communication, data heterogeneity, memory, and compute. It (3) organizes parameter-efficient strategies for FedLLM, including LoRA, prompts, adapters, and selective tuning, with sub-taxonomies (e.g., homogeneous/heterogeneous/personalized LoRA) and representative methods. It (4) catalogs datasets and benchmarks and (5) sketches applications (finance, biomedical, recommendation). Finally, it (6) flags open directions such as continual learning and memory-efficient FedLLM as critical for real-world deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There is a coherent taxonomy of FedLLM methods (LoRA / prompts / adapters / selective), laid out with an overview figure and dedicated subsections. This is clearly evidenced in the structure and visuals (Fig. 2 & Fig. 9) and the LoRA/prompt preliminaries. \n \nCompute constraints are substantiated by a profiling exercise: the paper presents FLOPs comparisons across model sizes and argues why this impedes on-device training (with citations). The evidence exists as text + figure and is reasonably clear. \n\nApplications coverage (finance/biomed/recsys) is present and organized, which supports the claim that FedLLM is relevant beyond toy benchmarks.", "audience_yn": "Yes", "audience_explanation": "This paper provides a thorough survey of PEFT in federated LLMs, summarizing core methods and benchmarks and serving as a good starting point for newcomers to FedLLMs fine-tuning.", "requested_changes": "The survey is comprehensive overall: it succinctly summarizes current PEFT methods for FedLLMs, identifies key challenges and applications, and outlines meaningful directions for future research. I have only a few minor revision requests.\n\n1. Figure 1 panel (a): \"Break data privay\" → Break data privacy.\n\n2. Intro: \"computation on on 2,048 NVIDIA A100 GPUs\" → remove duplicate \"on.\"\n\n3. \"SA-FedLora\" $\\rightarrow$ \"SA-FedLoRA\"", "broader_impact_concerns": "No concerns.", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "ggWkSw9D0L", "reviewer": "Reviewer_Tc4r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys federated fine-tuning of LLMs, namely “FedLLM', covering challenges through the lens of communication, heterogeneity, memory, compute perspectives. In addition to the survey, the authors also introduce a PEFT-centric taxonomy (Parameter-Efficient Fine-Tuning) for federated settings, datasets/benchmarks across domains, applications (finance/biomed/recsys), and other open problems. The authors go through a thorough review of existing fine-tuning datasets and evaluation benchmarks and rigorously evaluate the performance of FedLLM. Moreover, the authors provide a comprehensive discussion in terms of the real-world applications across multiple domains, as well as identify critical open challenges and outline promising research directions to foster future advancements in FedLLM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper focuses on federated fine-tuning and motivates the need with concrete resource barriers and privacy considerations, plus empirical memory/FLOP profiles, providing a clear scope and motivation. \n\nThe paper includes and contrasts related surveys along privacy/LLM/efficiency/benchmarks/applications/future work, and summarize these comparisons in tables, which helps readers see the niche. \n\nBeyond the standard PEFT categories, the paper proposes a LoRA-specific breakdown (homogeneous / heterogeneous / personalized) with representative methods and aggregation ideas—helpful in practice. \n\nThe general-domain and domain-specific benchmark tables are extensive (finance, medical, code, long-context), and many entries are annotated with evaluation criteria—useful for experiment planning. \n\nSections on finance, biomed, recsys, and future directions (e.g., IP protection, SLM–LLM collaboration, memory-efficient FedLLM) make the survey actionable.", "audience_yn": "Yes", "audience_explanation": "The paper conducts sufficient and different domains that are related to FedLLM, including method, potential applications and future open problems (more details are in the above section). These contents could be helpful for readers who want to dive deep into these directions.", "requested_changes": "The survey has done sufficent works in exhaustive review, but I would suggest providing links/hyperlinks in the paper when summarizing the datasets and benchmarks, to make sure the readers get the same resources referenced in the paper.\n\nThe references contain non-scholarly resources, .e.g, news item from AIbase about Ollama vulnerabilities. It would be better to justify the correctness of these resources and prioritize the usage of peer-reviewed sources. Otherwise, we could mark it to ask the readers to be careful about this resource. \n\nThe settings of the evaluated performance are not referenced or provided. Figure 7 lacks enough detail, such as hardware, precision. \n\nSome citations are duplicated, e.g. GSM8K.", "broader_impact_concerns": "No significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)