DoFIT: Domain-aware Federated Instruction Tuning with Alleviated Catastrophic Forgetting
Authors: Binqian Xu
Xiangbo Shu
Haiyang Mei
Zechen Bai
Basura Fernando
Mike Zheng Shou
Jinhui Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Instruction Tuning
LLM
LoRA
Primary area: safety_in_machine_learning

Abstract: Federated Instruction Tuning (FIT) advances collaborative training on decentralized data, crucially enhancing model's capability and safeguarding data privacy. However, existing FIT methods are dedicated to handling data heterogeneity across different clients (i.e., client-aware data heterogeneity), while ignoring the variation between data from different domains (i.e., domain-aware data heterogeneity). When scarce data needs supplementation from related fields, these methods lack the ability to handle domain heterogeneity in cross-domain training. This leads to domain-information catastrophic forgetting in collaborative training and therefore makes model perform sub-optimally on the individual domain. To address this issue, we introduce DoFIT, a new Domain-aware FIT framework that alleviates catastrophic forgetting through two new designs. First, to reduce interference information from the other domain, DoFIT finely aggregates overlapping weights across domains on the inter-domain server side. Second, to retain more domain information, DoFIT initializes intra-domain weights by incorporating inter-domain information into a less-conflicted parameter space. Experimental results on diverse datasets consistently demonstrate that DoFIT excels in cross-domain collaborative training and exhibits significant advantages over conventional FIT methods in alleviating catastrophic forgetting. Code is available at [this link](https://github.com/1xbq1/DoFIT).

DoFIT: Domain-aware Federated Instruction Tuning with Alleviated Catastrophic Forgetting
Binqian Xu', Xiangbo Shul.", Haiyang Mei', Zechen Bai?, Basura Fernando
Mike Zheng Shou", and Jinhui Tang' Nanjing University of Science and Technology Show Lab, National University of Singapore IInstitute of Herfo-pero Computing, A STSRR https: //gi thub. cwm com/1xbg1/DoFIT
Abstract
Federated Instruction Tuning (FTT) advances collaborative training on decentral- ized data, crucially enhancing model's capability and safeguarding data privacy. However, existing FIT methods are dedicated to handling data heterogeneity across different clients (i.e., client-aware data heterogeneity), while ignoring the variation between data from different domains (i.e., domain- -waae data heterogeneity). When scarce data needs supplementation from related fields, these methods lack the ability to handle domain heterogeneity in cross-domain training. This leads to domain- information catastrophic forgetting in collaborative training and therefore makes model perform sub-optimally on the individual domain. To address this issue, we introduce DoFIT, new Domain-aware FIT framework that alleviates catastrophic forgetting through two new designs. First, to reduce interference information from the other domain, DoFIT finely aggregates overlapping weights across domains on the inter-domain server side. Second, to retain more domain information, DoFIT initializes intra-domain weights by incorporating inter-domain information into less-conflicted parameter space. Experimental results on diverse datasets consis- tently demonstrate that DoFIT excels in cross-domain collaborative training and exhibits significant advantages over conventional FIT methods in alleviating catas- trophic forgetting. Code is available hhtth:://// htapps///gtthhee /ttgttgtttttt//t////////gg com/ comm1xbql/Dofie
Introduction
Large Language Models (LLMs) have attracted significant attention due their remarkable com- prehension and reasoning capabilities, coupled with their vast potential for a wide array of applica- tions (27 This attention has spurred the development of various Parameter Efficient Fine-Tuning (PEFT) methods [13 15 14 6. aimed at efficiently adapting these powerful models to specific tasks under constrained computational resources 21 Among them, LoRA (6] stands out as one of the most popular, due to its lower number of trainable parameters and the absence of additional inference computations. While LoRA significantly alleviates the computational burden associated with tuning LLMs, substantial challenges persist at the data level [28] 31., particularly in domains involving privacy concerns, where there is a lack of high-quality for cultivating a strong model [26]
Towards this issue, some Federated Instruction- Tunnon (FIT) methods have been explored by com- bining LLM instruction-tunin -uniin using LoRA with Federated Learning (FL) in recent years [37 B5
38
4 9 24]. In such FIT methods, the server side coordinates multi-round training among clients without sharing data for boosting model capability and protecting data privacy. Specifically, each round consists of four steps: global LoRA downloads from the server back to the clients, local
34
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LoRA
Same domain
ABB
Aes
Intra-domain server side
ABt
Inter-domain server side
Proximal
Server side
Client side
LLM
perturbation
LLM
Share
Overlap
Share
Overlap
ABB
TOP
ABE
LLM
Share
E
AgB
Age
AEE
AgE
ABE
100
(a) Conventional FIT
(b) DoFIT-base
(c) DoFIT
Figure 1: (a) Conventional FIT (with LoRA): directly expands from intra-domain inter-domain settings. (b) DoFIT-base (with catastrophic forgetting): aggregates overlapping modules among the top-k important modules from different domains the inter-domain server side and completes the personalized initialization of the updating weight matrix on the intra-domain server side by assigning values to corresponding modules while keeping the unchanged. (c) DoFIT (with alleviated catastrophic forgetting): further integrates a proximal perturbation initialization strategy into the DoFTT-base for alleviating catastrophic forgetting in terms of domain information.
Finance
Finance & General
Finance
Finance & General
0.67
0/959
0.82
0.82
(163
0/113
0.77
0.77
0.62
FPB
FIQA-SA
TFNS
NWGI
Avg:3
Avg:4
FPB
FIQA-SA
TFNS
NWGI
Avg:3
Avg:4
(a) Acc
(b)
Figure 2: Performance effect of conventional FIT trained on Specific domain (i.e., Finance) and Finance&Gennraa domain. FinGPT 36) and Alpaca- GPT4 [23] are the training datasets on Finance domain and General domain, respectively. FPB [19]. FiQA-SA [18), TFNS [1]. and NWGI (33 are all the evaluation datasets on Finance domain. Avg:3 and Avg:4 denote the average result on the first three evaluation datasets (i.c., FPB, FiQA-SA, and TFNS) and all the evaluation datasets, respectively.
LoRA updates on the client side, local LoRA uploads from clients to the server, and global LoRA aggregates the server side.
Currently, most FIT methods are dedicated handling data heterogeneity between clients, i.e., client- aware data heterogeneity. When data within specific domain is scarce and needs to be supplemented with data from other related domains to develop a powerful model, cxisting FIT methods treat intra- and inter-domain data heterogeneity (i.c., domain-aware data heterogeneity) equally and cannot adapt to the variation in cross-domain training, as shown in Figure ! (a). This variation comes from more common information in the same domain and more interference information from different domains. Furthermore, as shown in Figure 2 conventional FIT with LoRA often suffers from catastrophic forgetting of domain information, and performs suboptimally each individual domain, due to its inability to handle the domain- -omain-aware aware data heterogeneity. To tackle domain- aware data heterogeneity well, two main parts in conventional FIT, i.e., aggregation and initialization, evolved in this work.
Revisiting Aggregation in FTT. Considering the variation in intra-domain and inter-domain data, we design domain-awar aware FIT bascline (DoFIT-bart basc) for separately aggregating intra- and inter-domain

---

information, as shown in Tigure[[(). In DoFIT-base, it first normally aggregates domain-speciffe information on the intra-domain server side, and then aggregates overlapping Nomain-agnose information at a fine granularity in the inter-domain server side. In the inter-domain server side, this fine-gran ain-ggained aggregation strategy aims to reduce interference from irrelevant information. This strategy mainly includes two steps. The first step is to upload only the top-k important modules from each intra-domain server side. is noticeable that that DoFIT-base still uses LoRA (A or B as one module), with each layer of LoRA focusing on different aspects [30] The larger the LoRA weight is, the greater its impact on the frozen LLM becomes, and thus, it becomes more important [4]]. The second step is to averagely aggregate the overlapping modules uploaded from different intra-domain server sides. This step is necessary because the top-k modules uploaded by different intra-domain server sides are different. After aggregation on the inter-domain server side, each intra-domain server side obtains important and necessary module information for itself.
Revisiting Initialization in FIT. To retain more domain information, inspired by the orthogonal learning approach [2], we introduce a new initialization strategy based on the proximal perturba- tion by projecting the modules with inter-domain information onto parameter regions least affected by intra-domain update. Specifically, as shown in Figure (c), after completing the aggregation on the inter-domain server side, the newly generated modules are transmitted to the intra-domain server side. On the intra-domain server side, proximal perturbations are calculated between the new and the original modules, and then added to the original modules rather than being directly overwritten. The proximal perturbation term contains inter-domain information, while the original modules retain global intra-domain information. This less-confficted initialization strategy more effectively preserve domain information, ultimately mitigating catastrophic forgetting.
In summary, our contributions are: 1) the first solution to concern domain-awar data heterogeneity in collaborative training on decentralized data for the FIT paradigm; 2) a new domain-aware FIT frame- work that involves fine-grained inter-domain aggregation to handle domain-aware data heterogeneity; 3) a novel initialization strategy intra-domain global LoRA to alleviate catastrophic forgetting in terms of domain information; and 4) the significant performance gain over conventional FIT and comprehensive analysis to pave the way for future explorations into more advanced FIT.
2 Related Works
2.1 Federated Instruction Tuning.
Under the condition of protecting client data privacy, Federated Instruction Tuning (FIT) enables collaboration on high-quality instruction data, facilitating the instruction-tunie of pre-trained LLMSs for downstream tasks aimed at understanding diverse human intentions [37 35 32] As the pioneer work, Zhang et al. (37] introduced basic framework, which adopts LoRA for conducting client-side updating and server-side aggregation. Compared to the basic initial framework, Kuang et al. [ provided a comprehensive framework that covers data processing, federated training, and multiple benchmarks, while also implementing various memory-saving operations, as well as acceleration techniques.
Fine- Fine-Tuning (PEFT) methods,
Since the above FIT methods neglect alignment with human values, Ye et al. 35 presented federated value alignment alongside federated instruction-tuning, ensuring both harmless and helpful outputs. To tackle data extraction attacks and limited instruction data, Zhang et al. (39 proposed to employ LLM to synthesize data, and then train local models on both synthesized and original data, as well as global models solely on synthesized data. In heterogeneous scenarios with test-time distribution shift, Yang et al. 34 proposed a personalized FIT achieved by incorporating local LoRA and shared global LoRA. To confront the challenges arising from resource and data heterogeneity, Zhang et al [38] explored pruning personalized sparse structures for clients with resource imbalances using neural architecture search.
However, the above FIT methods only consider client-aware data heterogeneity, overlooking both the intra- and inter-domain data heterogeneity when different relevant domains co-exist, namely domain-aware data heterogeneity. To this end, we introduce intra- and inter-domain server sides to deal with two types of heterogeneity differently.
3

---

2.2 Parameter-Efficien Fine-Tunme Fine-Tuning,
The increasing parameter size of LLMs results in expensive computational costs. However, for adapting to downstream tasks, full-parameter fine-tuning of LLMS poses challenges hardware platforms with limited computational resources [3. To this end, various Parameter- Efficient Fine- Tuning (PEFT) methods have been proposed by freezing the pre-trained LLMs while only fine-tuning a small number of parameters [3]. Here, we mainly introduce several representative PEFT methods, i.e. Serial Adapter [5], Prefix-tuning [13, P-Tuning [15), IA3 [4] and Low-Rank Adaptation (LoRA) 6 For more details about PEFT methods, please refer to reference 3
Serial Adapter [5] built two adapter modules following the self-attention and FFN layers. Each adapter module comprises a down- wwwn-proecctin matrix, a non-linear activation function, and an up- projection matrix. Prefix-tuming [3,, as soft prompt method, added trainable vectors prefixes to both the key and value of all layers, while P-Tuning 15] integrated trainable vectors as prefixes into the initial word embedding layer. Similarly, IA3 14 added scaling trainable vectors to key, value, and FFN activations. As one of the re- ee-parameterization methods, LoRA [6] decomposed frozen parameters into two low-rank trainable matrices during fine-turning, and merged them with LLMs during inference without extra computational overhead. Similarly, considering the efficiency of LoRA in inference, our DoFIT adopts the PEFT method with LoRA.
3 Methodology
3.1 Preliminaries
Low Rank Adaptation (LoRA). Considering the limited computational resources available on the client side, it is challenging to perform full tull-aarmeetr instruction tuning for LLM. Fortunately, Low Rank Adaptation (LoRA) (6. the most popular Parameter-Efficien Fine-Tuning (PEFT) method in federated setting [1. has been successfully applied in Federated Instruction Tuning (FIT) 33 35). During adaptation for specific tasks, given the low intrinsic dimension of pre-trained language models, LoRA assumes that the update to the pre-trate tre-training weight matrices similarly exhibit low intrinsic rank (6). Consequently, for frozen pre- pre-trained weight matrix Wo dd its updating weight matrix AW e Ra*k is decomposed into low-rank trainable parameters BA, as follows,
Wo AW = Wo + BA
(1)
where E RRdx,, A E Brik and the rank r < min (d, k). A uses random Gaussian initialization and B is initialized with zero.
Conventional FIT with LoRA. Federated Instruction Tuning (FIT) is intended to resolve the issue of inadequate high-quality instructional data for individual clients and the inability to share such data due to privacy concerns. In FIT, it usually involves a server and multiple clients, where the server achieves collaborative training of non-shared instructional data among different clients by aggregating and initializing clients' updating weight matrices. To provide a detailed description of this process, we first define the updating weight matrix LW? in the client side, as follows,
AW (AW (B,
where r, i, I, and denote the f-th round, the i-th client, the 1-th layer, and total layers, respectively. During the training process, the updating weight matrix AW;-1 from the server is first downloaded for initializing the updating weight matrix AW,,
i the client, follows,
{ Jim = Aw AB Jimis = 1 H
Although only the weight initialization for the i-th client is indicated here, all selected clients undergo the same initialization process. After initializing, the updating weight matrices are optimized (i.e., (B + (B, min) based on their individual data, with loss function corresponding to each client's 's task. Subsequently, the updating weight matrices (AW,) hienn from selected client sets 2n uploaded to the server side for aggregation, generating new updating weight matrix AW/ on

---

the server side. The aggregation process is shown as below,
AWi AgBiertN (AWI))
E
B
ΣA! 1/11) SE
N
N
SRRN
B A
where fly represents a set of randomly sampled client indices, with a total N clients. The function A-g(.) averages B and A within the selected clients' corresponding layer. Bi Aj constitutes the new aggregated updating weight matrix AW?.
3.2 Domain-aware FIT Baseline (DoFIT-base)
The intra-domain and inter-domain data heterogeneities are unequal in domain data hetero- geneity. Conventional FIT fails to distinguish between intra- and inter-domain data heterogeneities, as it employs the same federated architecture to handle them equally, only altering the client data be within the same domain or across different domains, as shown Figure (a). Hence, when various clients possess datasets from other relevant domains, the results of conventional FIT may be inferior to those of the original specific domain, as shown Figure
In comparison to the intra-domain scenario, data from different domains demonstrate greater het- erogeneity. To benefit from information in other relevant domains, need carefully design the aggregation strategy to more finely extract and aggregate the shared information between the current domain and other relevant domains. Therefore, we introduce domain-aware FIT baseline (called DoFiT-base) that completes different aggregation strategies from coarse-grained level to fine-grained level. As shown in Figure (b), DoFIT-base takes into account both the intra-domain variance and inter-domain variance, where the latter is more challenging.
Specifically, DoFIT-base contains the intra-domain server side, inter-domain server side, and several client sides. First of all, in the inter-domain server side, the updating weight matrix AWt-1 is defined as follows,
Jege-1
(5)
where dp-1 denotes the overlapping modules (B or A as one module) from different domains in round 1. Noted, "overlapping" refers to both the same layer and the same decomposition components. At the beginning of the first round, At-1 and B are initialized with random Gaussian initialization and zero, respectively.
Download Step. In the updating weight matrix A of intra-domain server side, the overlapping [B 1 Je,., are initialized by AWt-1, while the personalized [ 1 1 remain unchanged, follows,
[1 A JeW, Jev. Jinie 4W
m.
AW = (B
ml 1
where m denotes the m-th domain. V 1 f = Ø, V VI- = U, and ( -
m.. -
Next, AW initializes the updating weight matrix AWt m,t on the i-th client side with the same domain, as follows.
(AW Jimi AWt-1
Ae m (AWt a. (
Upload Step. On the client side, AWt, i is updated based local data and specific task loss. Then, similar to conventional FIT, the updated AWL m. is uploaded to the intra-domain server side for aggregation, as follows.
! Wt. m.1 Aggienn (AWL .....)
= N
Σ
B BRaD
Σ Amai.
(8)
N
tENx
iEix
By B'
5

---

Similar to [40,, the squared norm of the module serves as the important score for itself, determining its impact on the frozen LLM. Obviously, the more important a module is, the greater its influence on the instruction tuning process. To mitigate the impact of irrelevant information from other domains, only common and important information should be captured. In the intra-domain server side, all modules in AW? m are sorted based on their important scores, where only the top-k modules are selected and uploaded to the inter-domain server side. Subsequently, the inter-domain server side, the overlapping modules across all domains aggregated accordingly, while the modules that do not overlap across all domains remain unchanged, as follows,
all modules have the same importance score. In fact,
AW Ageme M
(AW m.))
Σ B
ΣA
M
M
MEM
MEM
= B/
where A and {B/A round, and M denotes the number of domains.
V, indicates the of overlapping modules in the 1-th
3.3 Dwmain-aware FIT (DoFIT)
In traditional FL, the iterative training across multiple rounds often results in global information forgetting from previous rounds due to the heterogeneity nature of data on client sides. This issue persists our DoFlT-base, which is concretely manifested as the problem of domain information forgetting.
To retain more domain information and alleviate such problem, inspired by the orthogonal learning in [12), we translate inter-domain information to the parameter space least conflicted by the updating on the intra-domain server side, thereby reducing conflicts between intra- and inter-domain infor- mation. Thus, the initialization process in Eq. can be modified from directly covering AWt-1 by AWt-1, for adding proximal perturbation computed from the module-wise difference between A AWt-1 1 and A AWt-1, as follows,
B Bt-1
AP-1
A A
ATm Fimit [Bt-1 m.1
(B
| Br-l112
m.i
+ +a
||
A ll2
(10)
eV.-1
[C
:
B
and 1 1- 14------ denote proximal perturbation terms,
where a is the scaling factor.
mapping B and Ai-1 to the parameter region least affected by B and A The overall algorithm process of DoFIT is described in the supplemental material due to space limitations.
4 Experiments
4.1 Experimental Settings
Datasets. We train our DoFIT on three datasets, i.e., FinGPT [36]. Alpaca-GPT4 [23], and MedAl- paca 22 from the Finance (F), General (G), and Medical (M) domains, respectively. In the F domain, FinGPT is an open-source dataset for financial sentiment analysis, consisting of 77k samples. In G domain, Alpaca-GPT4 comprises 52k instances of English instruction -folooiinn data, generated by GPT-4 I using identical prompts Alpaca. In M domain, MedAlpaca includes 34k question-answe -nnweer pairs sourced from the Anki medical curriculum flashcards.
Configurations. In all experiments conducted on one NVIDIA A40, the frozen LLM used is Llama2- 7B with 32 layers (27 quantized to int8. The LoRA rank and alpha are set to 32 and 64, respectively. The maximum sequence length is 512. Following the formatting instructions of Alpaca template 25]. the training runs for 200 rounds, with a cosine learning rate scheduler adjusting the learning rate from 5e 5 to le 6. In each round, the selected clients are trained 10 steps by AdamW [16] optimizer.
The batch size set to 16. In FinGPP/Alpaca-GPT-I. GPT4/MedAlpac training, total 10k/20k/20k samples for 50/20/20 clients, selecting 5/2/2 clients randomly per round. Similar to [35] each training dataset

---

Table 1: Comparing "Local", Conventional FIT ("FIT"), DoFIT-base ("Base"), and "DoFIT" on
Finance (F) domain and Finance& (F&G) domain datasets. FinGPT [6 and Alpaca-
GPT4 23 are the training datasets on F domain and G domain, respectively. FPB [9]] FiQA-
SA [8] TENS [17], and NWGI [3] are the evaluation datasets on F domain. Avg:3 and Avg:4
denote the average result on the first three evaluation datasets (i.e., FPB, FiQA-SA, and TFNS) and all evaluation datasets, respectively. 1 refers to the performance improvement compared to the alternative marked with the same color (i.e., using the same LoRA configuration) on F domain. denotes performance degradation, oppositely.
FPB
FIQA-SA
TENS
NWGI
Avg:3
Avg:
Domain Medthod
Acc
F1
Ace
F1
Acc
FI
Ace
F1
Acc
F1
Ace
F1
GPT-3.5
0.781
0.781 0.662 0.730 0.731 0.736
0.725
0.749
GPT-4
0.834
0.833 0.545 0.630 0.813 0.808
0.731
0.757
Local
0.770 0.760 0.655 0.719 0.742 0.747 0.629 0.624
0.722
0.742
0.699
0.713
FIT 3240
0.859 0.857 0.815 0.841 0.787 0.792 0.652 0.647
0.820
0,830
0.778
0.784
16gud
0,850 0.846 0.818 0.842 0.823 0.823 0.646 0.643
0.830
0.837
0.784
0.789
FIT
0.860 0.857 0.807 0.836 0.824 0.825 0.635 0.635
0.830
0.839
0.782
0.788
FlT32qv
0.822 0.813 0.760 0.801 0.822 0.826 0.639 0.641 0.8014
0.813.
0.7614
0.770
Basetop10
0.859 0.855 0.778 0.815 0.810 0.811 0.637 0.638 0.816
0.827
0.771
0.780
Bascrop15
0,862
0.860 0.804 0.834 0.857 0.858 0.639 0.639
0.8411
0.851
0.7911
0.7981
F&G
Basczopzu
0.859
0.855 0.775 0.815 0.866 0.864 0.632 0.634
0.833
0.845
0.783
0.792
DoFIT
0,865 0.861 0.815 0.842 0.864 0.864 0.645 0.644
0.848
0.856
0.797
0.803
DoFIT
0.861
0.858 0,818 0.847 0.869 0.869 0.641 0.640
0.8491
0.8581
0.7971
0.8041
DoFIT
0.859 0.855 0.815 0.845 0.822 0.825 0.642 0.641
0.832
0.842
0.785
0.792
0.45
0.75
FIT
0.4
DoFIT-buse
DoFIT-base
0.7
0.35
DoFIT
DoFIT
Loss
0.3
Loss
0.65
0.25
0.6
0.2
0.55
0.15
0.1
0.5
10
15
20
25
30
35
40
45
50
18
30
42
Rounds
Rounds
Figure 3: Loss curves for different methods, i.e., FIT, DoFIT-bass and DoFIT, in F&G (left) and M&G (right) domains, respectively.
randomly shuffled and then evenly divided among the clients. The training datasets consist of either single-domain datasets or dual-domain datasets. The testing process is carried out on the evaluation datasets of single domain, after merging the updating weight matrix on the intra-domale server side with the frozen LLM.
Comparison Methods. To validate the effectiveness of the proposed DoFIT-base (called "Base" for conciseness) and "DoFIT" in addressing intra- and inter-domain data heterogeneity. is essential to compare them with "Local" and "FIT". Here, "Local" refers training independently based solely on data from a single client. "FIT" (35) refers to the collaborative training of different client data based on FedAvg (20). treating client data from different domains equally. FIT32qe uses all layers' LoRA[Q.V V] (decomposed from Q and V components of self-attention). FIT16god 16ged uses half of all layers' LoRA[Q. V.DJ (decomposed from the Q, V components of self-attention and Down linear layer of MLP). FITy2d uses all layers' LoRA(D] (decomposed from the Down linear layer in MLP). Noted, the aforementioned Conventional FIT is FlT (35). For fairness, we also use FedAvg the collaboration training from intra-domain data. In Specific and General domains, "Baserop1o" refers to uploading the top-10 important modules from the intra-domain server side to the inter-domain one, based on the best baseline in the Specific domain. The rest of "Baserop= has a similar definition. "DoFIT" modifies the initialization of the best "Base" by incorporating the proximal perturbation initialization. "DoFIT means that the scaling factor in DoFIT is set to 1.0.
7

---

Table 2: Comparing "Local", Conventional FIT ("FIT"), DoFIT-base ("Base"), and "DoFIT" on Medical domain (M), and combined Medical&General domain (M&G) MedAlpaca [2] and Alpaca- GPT4 23 are the training datasets on M domain, and G domain, respectively, MedQA 10] and MedMCQA [22] are the evaluation datasets M domain. 1 refers to the performance improvement compared to the alternative marked with the same color (i.e., using the same LoRA configuration) on M domain.
Domain
Method
MedQA
McdMCQA
Avg
Local
0.141
0.204
0.173
3240
0.167
0.216
0.192
M
FIT 16qud
0.179
0,200
0.190
32d
0.158
0.199
0.179
FIT
0.17410.007
0.217fo.on1
0.196fo.004
Baseroyas
0.182
0.207
0.195
Bascroy30
0. 19229..025
0.218fo.002
0.205fo.003
M&G
DoFIT
0.253
0.252
0,252
DoFITa
0.26110.094
0.255
0.258
DoFIT 0.256
0.247
0.251
4.2 Performance Evaluation
Comparison F Domain and F&G Domain. After training FinGPT for Finance (F) domain or CAGGPT/Alpaca-GPT for Gineral&General (F&G) domain, we test the models at round 50 on the evaluation datasets for F domain, i.e., FPB [19]. FiQA-SA [18) TFNS (17), and NWGI (33] as shown in Table [ Compared to the independently trained "Local", the collaboratively trained "FIT". "Ggee and "Base32d" via FL perform better. This indicates that training can benefit from other clients' data. Expanding from F to F&G domain, the performance of "FIT" even declines, while our "Basezop15" improves and surpasses that of FIT. This validates the effectiveness of our DoFlT-base for addressing domain-aware aware data heterogeneity. In F&G domain, "DoFIT" further improves performance compared to "Basetop1s". This indirectly validates that the proposed DoFIT retains more inter-domain information, enhancing overall performance.
Comparison M Domain and M&G Domain. After training on MedAlpaca for Medical (M) domain or MedAlpaca/ Aapacca/lpacaaGPP for Medical&Generaa (M&G) domain, we test the models at round 60 on the evaluation datasets for Medical (M) domain, i.e., MedQA (TO), and MedMCQA [2) As shown in Table 2 we can see that: 1) "FIT", "Basc16qud" and "Basezzd" consistently outperform "Local", indicating that collaborative training can enhance model's capability; and 2) on M&G domains, "Basezop30" and "DoFIT" exceed the performance of FIT, especially "DoFIT" with the proximal perturbation initialization strategy, proving the effectiveness of this strategy.
Loss Curves of Different Methods and Hyper-parameters. As shown in the loss curves Figure 3 compared SooFFTTbass"" "DoFIT" consistently shows faster convergence and lower losses, as does "DoFIT-base" compared to "FIT". Noted, the losses different methods are similar in the first rounds, while gap emerges as the number of rounds increases. Figure and 5l show the loss curves with different values of Top-k and a. we can find that the Tosses are insensitive to the values of these parameters some extent.
Comparison Parameter Size. Table further shows the number of parameters per round on FIT, the best-performing "Basetop15" "/""aarrss""""""Beee""" "/"Bas" "aaasstops"" and "DoFIT" in F&G and M&G domains. Compared to FIT, DoFIT adds slight communication parameters between intra- and inter-domain server sides (indicated by S- S-Comm.), with little impact on well-resourced server sides (indicated by S-Comm.). Compared with "FIT either Base or DoFIT requires fewer trainable parameters in the client side, well fewer communication parameters between the client side and the intra-domain server side.
5 Conclusion and Future Work
In this work, we introduced a novel Domain-aware Federated Instruction Tuning (DoFIT) framework towards collaborative training on more datasets in relevant domains for boosting the performance of

---

Top-10
0.75
Top-20
Top-15
Top-25
0.35
Top-20
0.7
Top-30
Top-25
Top-35
0.3
0.65
Low
L.oss
0.25
0.2
0.55
0.15
10
20
25
35
45
30
42
54
Rounds
Rounds
Figure 4: Loss curves for values of Top-k on F&G (left) and M&G (right) domains, respectively.
a=0.5
a=1.0
1.0
a=1.1
0.35
1=1.5
0.7
a-1.2
a=2.0
1.3.
0.65
Loss
0.25
Loss
0.15
0.55
0.5
10
15
20
25
30
35
40
45
50
6
12
18
24
30
36
42
48
54
60
Rounds
Rounds
Figure 5: Loss curves for values of a on F&G (left) and M&G (right) domains.
Table 3: The number parameters per round in training. "Frozen" denotes the parameter size of LLM. "Trainable" denotes the parameter size of the updating weight matrix in client side. "Comm." denotes the communication parameters between client side and (intra-domain) server side. "S- Comm." denotes the communication parameters between intra-domain server side and inter-doman server side. 32qu and 32d denote LoRAIQ.V] and LoRA(D), respectively. F&G and M&G denote General domain, and Redical& domain, respectively.
Domain
Method
Frozen
Trainable
Comm.
S-Comm.
3lT
6738M
4.194M
4.194M
0M
F&G
(Bascrom15 DoFTT) 322
6738M
4.021M
4.021M
0.942M
32qe
6738M
4.194M
4.194M
OM
M&G
(BasCropao DoFIT) 124
6738M
4.194M
4.194M
0.983M
individual domains. To alleviate the catastrophic forgetting problem caused by domain- aware data heterogeneity, our DoFIT offers two main insights in terms of aggregation and initialization. For aggregation, we first normally aggregate domain-specife information on the intra-domain server side, and then aggregate overlapping aagnoiie information on the inter-domain server side, excluding the interference information. For initialization, we add proximal perturbation from inter- domain information to the original modules, rather than directly overwritten them. Comprehensive experimental results Finance, Medical, and General domains demonstrate the effectiveness of the proposed DoFIT method, compared to conventional FIT.
Limitations. In our experiments, we have well demonstrated that the proposed DoFIT can facilitate collaborative training on decentralized data across one specific (i.e., Finance domain, or Medical domain) domain and the General domain, significantly enhancing performance within each individual domain. DoFIT is the first attempt to concern domain-aware data heterogeneity, and keeps the FIT-like optimization strategy since it only requires the least modification to the original FIT architecture. Such succinet modification seamlessly incorporates DoFIT into the FIT family for convenient reproduction and implementation. Exploring more related domains instead of limiting to the General domain

---

for enhancing a specific field, and verifying DoFIT's capability to handle multiple (more than two) domains, especially when substantial domain- aware data heterogeneity exists, along with the new optimizations, will be the focus of future research.
Acknowledgments and Disclosure of Funding
The work is supported by the National Natural Science Foundation of China (Grant No.62222207, 62072245, and 61925204), the Natural Science Foundation of Jiangsu Province (Grant No. BK2021 1520), the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (Award Number: AISG2-RP-2020-016) and the China Scholarship Council program. Mike Shou does not receive any funding for this work.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247 2023.
[3] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[4] Yuanqin He, Yan Kang, Lixin Fan, and Qiang Yang. Fedeval-lIm: Federated evaluation of large language models on downstream tasks with collective wisdom. arXiv preprint arXiv:2404. 12273, 2024.
[5] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. International Conference on Machine Learning, pages 2790-2799, 2019.
[6] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[7] Wenke Huang, Mang Ye, 7ekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. Computer Vision and Pattern Recognition (CVPR), pages 163 1233226 2023.
[8] Enyi Jiang, Yibo Jacky Zhang, and Sanmi Koyejo. Principled federated domain adaptation: Gra- dient projection and auto-weighting. In International Conference on Learning Representations, 2024.
[9] Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang. Kun Yang, Cunhua Pan, and Dusit Niyato. Personalized wireless federated learning for large language models. arXiv preprint arXiv:2404.132366 2024.
[10] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
[11] Weirui Kuang, Bingehen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-lInn A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint 2rX9:: 2023.
[12] Gihun Lee, Minchan Jeong, Sangmook Kim, Jaehoon Oh, and Se- Yun. Fedsol: Bridging global alignment and local generality federated learning. arXiv preprint arXiv:2308. /2532, 2023.
10

---

[13] Xiang Lisa Li and Percy Liang. trefi--tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1hth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3582-4599 2021.
[14] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficintt fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, pages 1950-1965, 2022.
[15] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. P- tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pupers), pages 61-68 2022.
[16] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
[17] Neural Magic. Twitter financial news sentiment. https://huggingaant co/aaaaafsccccc/tttttttstsssss
zeroshot/twitter: zeroslot//viiterr-ffaiiilll financiaal news news- sentiment 2022.
[18) Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'18 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941- 1942, 2018
[19] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796, 2014.
[20] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficii learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, 2017.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Advances in neural information processing systems, volume 35, pages 27730-27744, 2022.
[22] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-suhject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248-266 2022.
[23] Baolin Peng, Chunyuan Li, Pengcheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint axXiv:2304.03277, 2023.
[24] Siqi Ping, Yuzhu Mao, Yang Liu, Xiao-Ping Zhang, and Wenbo Ding. Fl-tac: Enhanced fine-tuning federated learning via low-rank, task-specific adapter clustering. arXiv preprint 2rX4v:2404.1533 2024.
[25] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
[26] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940 2023.
[27] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
11

---

[28] Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. In Advances in Neural Information Processing Systems, volume 36, 2024.
[29] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt A large language model for finance. arXiv preprint arXiv:2303.17566 2023.
[30] Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
[31] Rui Yan, Lingxi Xie, Xiangbo Shu, Liyan Zhang, and Jinhui Tang. Progressive instance-aware feature learning for compositional action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):03377103330 2023.
[32] Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, and Qi Tian. Higein: Hierarchical graph-based cross inference network for group activity recognition. IEEE transactions pattern analysis and machine intelligence, 45(6),6955-6968, 2020.
[33] Hongyang Yang. Data-centric fingpt. open-source for open finance. https://github. com/ 2023.
[34] Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Dual- personalizing adapter for federated foundation models. arXiv preprint arXiv:2403. 19211, 2024.
[35] Rui Ye, Wenhao Wang. Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du Yanfeng Wang. and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. arXiv preprint arXiv:2402. 06954, 2024.
[36] Boyu Zhang, Hongyang Yang, and Xiao-Yang Liu. Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models. In FinLLM Symposium at IJCAI, 2023.
[37] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915-6919, 2024.
[38] Pengyu Zhang, Yingbo Zhou, Ming Hu, Junxian Feng, Jiawen Weng, and Mingsong Chen. Personalized federated instruction tuning via neural architecture search. arXiv preprint arXiv:2402.16919, 2024.
[39] Zhuo Zhang, Jingyuan Zhang, Jintao Huang. Lizhen Qu, Hongzhi Zhang, and Zenglin Xu. Fedpit: Towards privacy-preserving and few-shot federated instruction tuning. arXiv preprint arXiv:2403.06111, 2024.
[40] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, and Tiejun Zhao. Lora-drop: Efficient lora parameter pruning based output evaluation. arXiv preprint arXiv:2402.07221,000 2024.
12

---

A Appendix supplemental material
A.1 Algorithm
Algorithm
The training process of DoFI'T for two domains
Input:
AW M m m= Initial updating weight matrix in the inter-domain server side intra-domain server sides, T rounds, N: Random sample number
of clients, M: Total number of domains, e: The number of epochs in the
client side, top-k important modules.
Output: {AW7) m==1
1: for 1..-.,I''' do
2: S2N +- A set of randomly sampled client indices.
3: for m I. do
4: if 1 then
5:
Intra-domain initialization
6: Add proximal perturbation on Eq. 10 end if
7:
8:
for each in 22 do
9:
iii = A AW
10:
Conduct e epochs of local instruction-tuning for AW! m, end for
12:
Intra-domain Aggregation
13:
AW? Ag (AW,, m, on Eq. 8
14:
Compute the important score of each module for AW: m
15:
Upload top-k modules for A W to the inter-domain server side.
16:
end for
17:
Inter-domain Aggregation
18:
Compute the set of overlapping modules from different domains: V
19:
AW, =
. (m, Eq. 9
20:
end for
A.2
Comparison with Existing Federated Domain Adaptation Works
Federated domain adaptation for LLMs is crucial. but no related methods currently exist. Applying existing federated domain adaptation methods like FedGP []] directly to LLMs yields subopti- mal results, as shown in Table Where FedGP/FedGP-g refer the projection of each client's LoRA/global LoRA weights in the source domain onto the global LoRA weights in the target domain. FPL 7 clusters prototypes from different domains into unbiased prototypes for general domain shift. However, existing federated domain adaptation methods [7 for this task still merge more redundant and noisy parameters to LLMs, affecting domain fine-tuning performance. Overall, as shown in Table I and Table 4 our method outperforms traditional FIT and general federated domain adaptation methods.
Table 4: Comparison with existing federated domain adaptation works.
FPB
FIQA-SA
TFNS
NWGI
Avg:3
Avg:4
Method
Acc
Fl
Acc
Fl
Acc
Fl
Acc
F1
Acc
Fl
Acc
Fl
FedGP
0.837 0,829 0.760 0.806 0.789 0.786
0.625 0.626 0.795 0.807 0.753 0.762
FedGP-g
0.836 0.830 0.680 0.744 0.700 0.710 0.627 0.629 0.739
0.761 0.711 0.728
DoFITa=1.
0.861 0.858 0.818 0.847 0.869 0.869 0.641 0.640 0.849 0.858
0.797 0.804
A.3
Performance on the Gradient and Singular Value Spectrum
We add two new criteria- gradient and singular value- to assess module importance in LoRA, as follows,
13

---

Table 5: Performance on the gradient and singular value spectrum.
FPB
FiQA-SA
TFNS
NWGI
Avg:3
Avg:4
Criteria
Acc
F1
Acc
F1
Acc
F1
Ace
F1
Acc
F1
Acc
Fi
D0F1T
0.861
0.858 0.818 0.847 0.869 0.869 0.641 0.640
0.849 0.858 0.797 0.804
A-grad-top15
0.866
0.864 0.833 0.852 0.867 0.867 0.640 0.639 0.855 0.861 0.802 0.806
A-svd-top15
0.858
0.855 0.829 0.856 0.828 0.829 0.642 0.641 0.838 0.847 0.789 0.795
B-grad-top10
0.823 0.813 0.789 0.827 0.802 0.806 0.633 0.633 0.805 0.815 0.762 0.770
B-grad-top15
0.833
0.829 0.840 0.855 0.681 0.693 0.630 0.627 0.785 0.792 0.746 0.751
B-grad-top20
0,516 0.480 0.185 0.197 0.501 0.500 0.404 0.369 0.401 0.392 0.402 0.387
B-svd-top10
0.856
0.854 0.844 0.854 0.732 0.740 0.638 0.627 0.811 0,816 0.768 0.769
B-svd-top15
0.821
0.819 0.793 0.824
0.621 0.626 0.644 0.640 0.745 0.756 0.720 0.727
B-svd-top20
0.417 0.306 0.811 0.794 0.371 0.302 0.552 0.457 0.533 0.467 0.538 0.540
Aug:3 (FPB. (EPB, FIQA-SA, TENS)
4v (FPH, FIQNSSA TENS, NWGI)
0.85
0.85
0.3
0.8
0.74
0.75
0.7
0.65
20
30
30
40
50
Bounds
DafIT (F)
RTI-FaPPsx (F)
RTT
FIT--
DoFIT
FFT PIT-Fellex (7) FI1 Fol-Fo
Figure 6: Comparison of average accuracy on different rounds
Using new criteria to sort modules from largest to smallest within single domain, and select the top-k modules, like DOFIT, Gradient: Uses the square norm of the gradients of LoRA modules as the importance score (A~ A-ggra-too115) Singular Value: Uses the sum of the singular values of LoRA modules as the importance score (A-svd-top15), As shown in Table [ the importance scores based on the gradient square norm and the of singular values are comparable the module importance scores calculated using the square norm of LoRA weights in DoFIT.
From domain distribution-awan perspective, aggregate the top-k modules with smaller domain gaps, Gradient: Uses the mean absolute difference of LoRA module gradients across different domains to reflect domain heterogeneity gaps (B- (B-grad-top*). Singular Value: Uses the L2 norm of the differences in the singular value spectrum of LoRA modules across different domains to reflect domain heterogeneity gaps (B-svd-top*). As shown in Table using gradient singular value to aggregate modules with smaller domain heterogeneity shows more sensitivity to the top-k hyperparameter. Compared aggregating modules with smaller domain heterogeneity can still introduce redundant and noisy modules, which can degrade overall performance when merged into the L.MSs.
DoFIT, this approach performs worse. This may be because
Overall, focusing on domain-specine key parameters and removing redundancies improves perfor- mance with LLMs. DoFIT's square norm method for weights is comparable to gradient and singular value spectrum methods but more intuitive and reproducible.
A.4 Federated Settings Experiments
As shown in Table 3, our DoFIT has the same space complexity as the traditional FIT on the client side, without any additional memory cost, but introduces a slight memory cost (S-comm.) on the inter-domain server side. In terms of time complexity, our DoFIT identical to the traditional FIT on the client side, with only slight computational overhead for module importance ranking on the intra-domain server side, Assuming the number of selected clients in the same domain is k, and each client includes 32d LoRA (64 modules), the sorting time complexity is
Complexity Analysis:
14

---

Table 6:
Average accuracy on FPB, FIQA-SA, TENS, NWGI
Clients
Acc
F1
Clients
Acc
F1
Clients
Acc
F1
50(5) & 20(2)
0.797 0.804
50(10) & 20(2)
0.800 0.806
50(15) & 20(2)
0.784
0.792
50(20) & 20(2)
0.783 0.792
50(5) & 20(4)
0.786 0.793
50(5) & 20(6)
0.794
0.799
50(5) & 20(8)
0.796 0.802
50(5)
20(10)
0.791 0.797
25(S) & 20(2)
0.788 0.794
40(5) & 20(2)
0.748
0.757
60(5)
20(2)
0.791
0.799
75(5) & 20(2)
0.791
0.798
50(5) & 10(2)
0.799 0.804
50(5) 30(2)
0.790
0.797
50(5) & 40(2)
0.789 0.793
1.5
0.5
Domain
Domain
CDeann
2.5
1.5
Modil
Client
Client
Ciient
Cliem:26
Cliest
Domain
Domamin
Figure 7: Modules important scores (left) and singular value spectrum (right) on and G domains
64 log(64). In the financial domain, & 5; in the general domain, & and in the medical domain, k = 2. The entire experiment ran on an NVIDIA A40 GPU for five and a half hours.
Convergence Results: As shown in Figure
DoFTT demonstrates faster and more stable convergence compared to FIT using FedAvg and FedProx as the FL framework in both single-domain and dual-domaie scenarios.
Client Numbers: 50(5) & 20(2) indicate that the financial domain, there are 50 clients in total, with clients randomly selected for training and uploading each round. In the general domain, there are 20 clients in total, with clients randomly selected for training and uploading each round. As shown in T varying the total number of clients the number of selected clients does not cause significant fluctuations in the results, demonstrating that the proposed DoFIT is robust to the number of clients.
A.5 Domain Heterogeneity
The importance of modules LoRA varies across different domains, indirectly reflecting domain heterogeneity. As shown in Figure 7 (left), the top-15 important modules in domains F and G are not completely the same. As training progresses, the weights of the same modules become more reinforced.
We also further compute the L2 norm of the difference in the singular value spectrum between each client's is LoRA and the global LoRA for the same domain and different domains, As shown in Figure 7 (right), this visualization reflects smaller intra-domain data heterogeneity and greater inter-domain data heterogeneity.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check out the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please check out the limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The proposed method mainly comes from experimental verification.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please check out the methodology, experimental settings, and performance evaluation sections.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Please check out the zzip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com / cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please check out the experimental settings, and performance evaluation sec- tions.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please check out the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please check out the experimental settings section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research in every respect conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please check out the introduction. The paper has positive societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: This model uses LLMs with safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper has cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please check out the .zip file.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "UDmdxnbxE6", "reviewer": "Reviewer_Vje9", "summary": "In this work, the issue of domain-aware data heterogeneity is solved, which equally treats intra- and inter-domain data variations but cannot adapt to the challenges of  cross-domain training. The paper proposes a novel domain-aware federated instruction tuning (DoFIT) framework for collaborative training across datasets in related domains to enhance the performance of individual domains. Specifically, DoFIT aggregates the top-k important modules and initializes intra-domain modules with addition of a proximal perturbation term. The proposed DoFIT demonstrates a significant performance improvement when transitioning from single-domain to dual-domain datasets.", "questions": "Whether FIT32qv differs between single-domain and dual-domain scenarios, and how to extend it to dual domains should be explained.", "limitations": "The authors provide limitations and future works.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The proposed DoFIT is novel and interesting. Instead of treating intra- and inter-domain data heterogeneity equally, the authors aggregate the top-k important modules on the inter-domain server side and initialize intra-domain modules with the addition of a proximal perturbation term on the intra-domain server side. These targeted processing schemes alleviate the catastrophic forgetting problem.\n2.The motivation and techniques of this paper are reasonable. In DoFIT, the aggregation of important modules is motivated by the fact that the larger LoRA weight has the greater impact on the frozen LLM. In DoFIT, proximal perturbation initialization is derived from the orthogonal learning approach.\n3.The comparison of communication costs is detailed in Table 3.\n4.The provided experiment results are convincing.", "weaknesses": "1.The concept of different domains is less clearly defined, and the scope of application of the proposed framework seems limited.\n2.The experiments only yield results on single-domain and dual-domain datasets, and cannot be extended to multi-domain scenarios.\n3.The privacy and security of the intra- and inter-domain server sides do not seem to be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses:\n\n1: Thanks. Unlike traditional FL methods that focus on different styles within the same category, the different domains in this paper refer to various application scenarios, such as the financial, general, and medical domains. The proposed framework is suitable for federated instruction tuning tasks across different but related domains.\n\n2: Thanks. As the first attempt to address domain-aware data heterogeneity in federated instruction tuning, our proposed framework is experimentally demonstrated the effectiveness on both single and dual domain datasets. Extending to multi-domain scenarios is a promising future work and we believe our work could lay a foundation and provide insights for deeper exploration in this area.\n\n3: Thank you. The privacy and security of intra- and inter-domain servers are the same as in traditional FIT methods with a single server. Compared to previous FIT methods, we do not upload more parameters or information to the servers.\n\n> Questions:\n\nFIT32qv extends from the financial domain to both financial and general domains by simply adding clients from the general domain. It still uses a single server to equally aggregate parameter weights from clients in both domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97ouSBqRg0", "reviewer": "Reviewer_ZeZJ", "summary": "The authors propose to utilize intra- and inter-domain server sides in a federated instruction tuning framework to implement discriminative aggregation and initialization strategies. The proposed approach, i.e., DoFIT, is primarily based on FIT of LLM, marking the first solution to address domain-aware data heterogeneity in collaborative training on decentralized data for the FIT paradigm. Unlike conventional FIT, which ignores the variation between data from different domains, DoFIT enables the intra-domain server to perform normal aggregation and initialization, while the inter-domain server handles overlapping weights aggregation and less-conflicted initialization. Consequently, DoFIT reduces interference information and preserves more domain information to mitigate catastrophic forgetting. The authors conducted empirical comparisons with conventional FIT methods on three datasets from finance, general, and medical domains.", "questions": "1. The meaning of “Overlap” in Figure 1 is unclear and should be explained, preferably in the title.\n2. The set of modules where “Overlap” resides is in units of LoRA B or A, while the updating weight matrix is in units of each layer in the transformer architecture and should be indicated.\n3. Is the scaling factor α in Eq. 10 the same for LoRA B and A?", "limitations": "No limitation is explained in the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A novel and promising application involves supplementing data from other related domains is explored to develop a powerful model when data within a specific domain is scare.\nA new domain-aware FIT framework that includes fine-grained inter-domain aggregation is proposed to address domain-aware data heterogeneity.\nA novel initialization strategy in intra-domain global LoRA is presented to alleviate catastrophic forgetting.", "weaknesses": "1. In the F&G and M&G domains, the values of k and α in DoFIT are quite different. Could adding more hyperparameters potentially harm the generalization of DoFIT?\n2. There are fewer methods compared in the single-domain experiments, and it seems that more comparisons of FL methods other than FedAvg should be added.\n3. The proposed intra- and inter-domain servers in DOFIT-Base and DoFIT increase the communication cost compared to traditional FIT methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses:\n\n1: In our DoFIT, we only added two hyperparameters: top-k and α. As shown in Tables 1 and 2, as well as Figures 4 and 5 of the original paper, the performance differences for different top-k and α values are minimal. Therefore, even when generalizing to new domain data, extensive adjustment of these two hyperparameters is unnecessary. The proposed DoFIT demonstrates a certain robustness to these hyperparameters.\n\n2: Thank you. We think that incorporating traditional FL methods only addresses client-aware data heterogeneity within the same domain and does not tackle domain-aware data heterogeneity. Therefore, we only compared a few representative single-domain FL methods in the paper. As suggested, we included more comparison methods (FedProx [1]), and the results further demonstrated the effectiveness of our approach, as shown in Figure 1 of PDF.\n\n[1] Li, Tian, et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine learning and systems (MLsys), 2020, 2: 429-450.\n\n3: Thank you. In Table 3 of the original paper, the communication cost (S-Comm.) added by DoFIT-Base and DoFIT compared to FIT is shown for intra-domain and inter-domain servers. Compared to FIT, DoFIT introduces only a slight increase in communication parameters between intra- and inter-domain server sides (indicated by S-Comm.), with minimal impact on well-resourced servers (also indicated by S-Comm.).\n\n> Questions:\n\n1: Thank you. As mentioned on line 164 of the original paper, \"overlapping\" refers to both the same layer and the same decomposition components. We will include this explanation in the final version of Figure 1’s title.\n\n2: Thanks for the suggestion. We will clarify this in the revised version.\n\n3: Yes, the scaling factor α in Eq. 10 is the same for LoRA B and A.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5EiFNirWM", "reviewer": "Reviewer_xCSC", "summary": "This work offers a solution to a problem in the collaborative training of different domains on decentralized data within the FIT paradigm: domain-aware data heterogeneity causes domain-information catastrophic forgetting. The solution relies on two new designs for aggregation and initialization. Specifically, in the aggregation step, DoFIT-base aggregates overlapping inter-domain information at a fine granularity on the inter-domain server side. In the initialization step, DoFIT projects modules with inter-domain information onto parameter regions least affected by intra-domain update. Finally, the authors conducted extensive comparison experiments to well show the significant effectiveness of the proposed method.", "questions": "1. In the DoFIT framework, the final test determines whether to experiment with global LoRA on the inter-domain server side or the intra-domain server side.\n2. Is it more reasonable that the scaling factors of LoRA B and A are not consistent?\n3. There are too many symbols in the methods section; it is best to use a table to clearly specify the important symbolic variables.\n4. The difference between DoFIT-base and Conventional FIT in Section 3.2 is best highlighted.", "limitations": "no limitation in this scope.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors introduce a novel FIT framework aimed at solving the domain-information catastrophic forgetting problem. \n2. Considering that existing FIT methods struggle with the variation from different domains, resulting in inferior results for the original specific domain, the proposed DoFIT outperforms conventional FIT methods by aggregating overlapping inter-domain information and initializing with inter-domain information.", "weaknesses": "1. Although the current framework performs better than conventional frameworks in F&G or M&G domains, can it be further applied to more domains? \n2. There are related articles that test the FL results of different domain data in different clients, where each client has data from one domain data. Thus, it is not clear how the cross-domain training problem differs in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses:\n\n1: Thank you. Currently, the experiments show improvements only between two domains. Due to the more complex and variable heterogeneity in multi-domain scenarios, the current framework cannot handle it well. However, as the first federated instruction tuning framework attempting to address domain-aware data heterogeneity, it provides some inspiration for future FIT methods dealing with more domains.\n\n2: Thank you. The related work on FIT tested FL results with one client per domain, which only addressed inter-domain data heterogeneity. However, our DoFIT accounts for multiple clients within a domain, addressing both intra-domain and inter-domain data heterogeneity issues.\n\n> Questions:\n\n1: The final test involves first merging the LLMs with global LoRA on the intra-domain server side, and then testing on the current domain dataset.\n\n2: The scaling factors for LoRA B and A are consistent and already yield good results. Using different scaling factors would increase the burden of hyperparameter tuning and make it more challenging for the model to generalize across different domains.\n\n3: Thank you. We will add descriptions of the important symbolic variables in the final version.\n\n4: Thank you. The first and second paragraphs of Section 3.2 in the original paper explain the differences between DoFIT-base and conventional FIT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtFfSsyKt5", "reviewer": "Reviewer_zbxm", "summary": "This paper proposes Federated Instruction Tuning, aimed at enhancing model's capability and data privacy. The main points lie in enhancing the model of data heterogeneity across dfifferent clients.  Specifically, the paper introduces DoFIT, a domain-aware FIT framework, aimed at alleviating the catastrophic forgetting through aggregating overlaping across domains, and incorporating inter-domain information into a less-conflict parameter space to reduce interference information from other domain. The proposed method is evaluated on diverse datasets.", "questions": "1. What is the differences between the proposed method with those without large models?  The indroduction of LoRA may be a engineering problem not an technological contribution;\n2. The sort critera is a little simple, what is the performance gain when used other critera?\n3. The paper should include complexity, convergence, and the impact of the number of clients on the results under federated setting;", "limitations": "See Weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The topic is important in federated learning .\n2) The paper is easy to follow;", "weaknesses": "1. The novelty is limited. There are already many works in federated domain adaptation that consider the differences between domains [1][2].It seems the authors only considers large models (model differences) and uses LoRA for optimization to analyze domain heterogeneity. This approach seems to simply apply domain heterogeneity to federated learning with large models.\n2. What distinguishes this work from other Federated Domain Adaptation (FDA) methods that also address domain heterogeneity?\n3. I would prefer to see the authors use a more innovative criterion when considering module importance scores, rather than directly using existing work [31]. Specifically:1) Compare with other criteria, such as sorting by gradient, data consistency, etc. 2) I suggest the authors consider module importance from a domain distribution-aware perspective. Starting from domain distribution would better reflect domain heterogeneity.\n4. The experimental validation is insufficient as it is entirely based on LoRA settings. Important federated settings, such as complexity, convergence, and the impact of the number of clients on the results, have not been considered.\n5. An analysis of domain-heterogeneity is necessary. For example, what aspects of the model can reflect domain-heterogeneity? How can the A and B matrices in LoRA be analyzed to reflect intra-domain and inter-domain heterogeneity (e.g., using matrix singular values)\n6. The algorithm's complexity needs to be analyzed, including the additional computational overhead introduced by the module importance score sorting.\n\n\n[1]\tJiang, Enyi, Yibo Jacky Zhang, and Oluwasanmi Koyejo. \"Federated domain adaptation via gradient projection.\" arXiv e-prints (2023): arXiv-2302.\n[2]\tHuang, Wenke, et al. \"Rethinking federated learning with domain shift: A prototype view.\" 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses:\n\n**1, 2:** Federated domain adaptation for LLMs is crucial, but no related methods currently exist. Applying existing federated domain adaptation methods like FedGP[1] directly to LLMs yields suboptimal results, as shown in Table 1 of the PDF. Where FedGP/FedGP-g refer to the projection of each client's LoRA/global LoRA weights in the source domain onto the global LoRA weights in the target domain. FPL[2] clusters prototypes from different domains into unbiased prototypes for general domain shift. However, existing federated domain adaptation methods[1,2] for this task still merge more redundant and noisy parameters to LLMs, affecting domain fine-tuning performance.\n\nIn this work, we make the first attempt to address the LLM-oriented federated domain adaptation problem. We observe that not all LoRA parameters are useful for the current domain, where some are redundant or noisy. Aggregating all parameters can introduce irrelevant ones from other domains, weakening the influence of important parameters. The fine-tuning performance of LLM is sensitive to this. To address it, we present two innovations.  First, we sort LoRA parameters by weight and retain only the top-k to avoid the inclusion of irrelevant ones. Second, to mitigate catastrophic forgetting, we map the aggregated weights to a less conflicting parameter space. Experiments validate the effectiveness of these innovations.\n\nOverall, as shown in Table 1 of the original paper and Table 1 of the PDF, our method outperforms traditional FIT and general federated domain adaptation methods.\n\n---\n\n**3:** Thanks. We add two new criteria—gradient and singular value—to assess module importance in LoRA. Since LoRA's data consistency is uniform, module importance scores are identical, preventing individual ranking.\n\n1）Using new criteria to sort modules from largest to smallest within a single domain, and select the top-k modules, like DoFIT,\n\n*Gradient*: Uses the square norm of the gradients of LoRA modules as the importance score (A-grad-top15).\n\n*Singular Value*: Uses the sum of the singular values of LoRA modules as the importance score (A-svd-top15).\n\nAs shown in Table 2 of the PDF, the importance scores based on the gradient square norm and the sum of singular values are comparable to the module importance scores calculated using the square norm of LoRA weights in DoFIT. \n\n2）From a domain distribution-aware perspective, aggregate the top-k modules with smaller domain gaps,\n\n*Gradient*: Uses the mean absolute difference of LoRA module gradients across different domains to reflect domain heterogeneity gaps (B-grad-top*).\n\n*Singular Value*: Uses the L2 norm of the differences in the singular value spectrum of LoRA modules across different domains to reflect domain heterogeneity gaps (B-svd-top*).\n\nAs shown in Table 2 of the PDF, using gradient or singular value to aggregate modules with smaller domain heterogeneity shows more sensitivity to the top-k hyperparameter. Compared to our DoFIT, this approach performs worse. This may be because aggregating modules with smaller domain heterogeneity can still introduce redundant and noisy modules, which can degrade overall performance when merged into the LLMs.\n\nOverall, focusing on domain-specific key parameters and removing redundancies improves performance with LLMs. DoFIT’s square norm method for weights is comparable to gradient and singular value spectrum methods but is more intuitive and reproducible.\n\n---\n\n**4, 6:** Thanks. We conducted federated settings experiments (complexity, convergence, and client numbers) but omitted them due to space constraints and minimal impact. The paper focuses on new issues and configurations in federated domain adaptation for LLMs.\n\n*Complexity Analysis*: As shown in Table 3 of the original paper, our DoFIT has the same space complexity as the traditional FIT on the client side, without any additional memory cost, but introduces a slight memory cost (S-comm.) on the inter-domain server side. In terms of time complexity, our DoFIT is identical to the traditional FIT on the client side, with only a slight computational overhead for module importance ranking on the intra-domain server side. Assuming the number of selected clients in the same domain is k, and each client includes {32d} LoRA (64 modules), the sorting time complexity is k \\times 64 \\times \\log(64). In the financial domain, k=5; in the general domain, k=2; and in the medical domain, k=2. The entire experiment ran on an NVIDIA A40 GPU for five and a half hours.\n\n*Convergence Results*: As shown in Figure 1 of the PDF, our DoFIT demonstrates faster and more stable convergence compared to FIT using FedAvg and FedProx as the FL framework in both single-domain and dual-domain scenarios.\n\n*Client Numbers*: 50(5) \\& 20(2) indicate that in the financial domain, there are 50 clients in total, with 5 clients randomly selected for training and uploading each round. In the general domain, there are 20 clients in total, with 2 clients randomly selected for training and uploading each round. As shown in Table 3 of the PDF, varying the total number of clients or the number of selected clients does not cause significant fluctuations in the results, demonstrating that the proposed DoFIT is robust to the number of clients.\n\n---\n\n**5:** Thanks. The importance of modules in LoRA varies across different domains, indirectly reflecting domain heterogeneity. As shown in Figure 2 (left) of the PDF, the top-15 important modules in domains F and G are not completely the same. As training progresses, the weights of the same modules become more reinforced.\n\nWe also further compute the L2 norm of the difference in the singular value spectrum between each client's LoRA and the global LoRA for the same domain and different domains. As shown in Figure 2 (right) of the PDF, this visualization reflects smaller intra-domain data heterogeneity and greater inter-domain data heterogeneity.\n\n> Questions:\n\n1,2,3: See 1,3,4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZ956VWGu8", "reviewer": "Reviewer_DeWw", "summary": "This paper first introduces a new domain-aware FIT baseline called DoFIT-base. DoFIT-base aggregates domain-specific information on the intra-domain server side and domain-agnostic information on the inter-domain server side to reduce interference information from the other domains. By incorporating inter-domain information into a less-conflicted parameter space, this paper formally introduce q new DoFIT framework that retains more domain information. Since DoFIT can handle domain-aware data heterogeneity, the catastrophic forgetting problem in cross-domain training is effectively alleviated. The authors show cross-domain training results in a federated LLM trained on FinGPT, Alpaca-GPT4, and MedAlpaca, demonstrating that DoFIT performs better than conventional FIT methods.", "questions": "Please see the comments above in the weaknesses.", "limitations": "The paper could benefit from more discussion on the limitations of the proposed method.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**(1)** The idea of aggregating domain-specific/domain-agnostic information and initializing in less-conflicted parameter space is compelling and original for addressing domain-aware data heterogeneity.\n\n**(2)** The significant performance gain over conventional FIT, along with comprehensive analysis, paves the way for future explorations into more advanced FIT. \n\n**(3)**  Considering FIT in multi-domains in the FL scene is interesting, which can give some insights in the related filed.", "weaknesses": "**(1)** Why did you choose FIT_{32qv}, FIT_{16qvd}, and FIT_{32d} as comparison methods in a single domain? Would it be better to add more LoRA modules? What is the basis for choosing LoRA[D]?\n\n**(2)** Since more servers are added to handle cases with different heterogeneity separately, additional S-comm. increases the burden of communication and security risks.\n\n**(3)** There are many parameters in DoFIT, and as shown in Table 1 and Table 2, the parameters are different across different datasets. This variability seems to make it difficult to generalize DoFIT to other datasets.\n\n**(4)** Does Base_{top30} in Table 2 work well, or does Base_{top35} work better?\n\n**(5)** It is still unclear whether LoRA on the client, intra-domain server, or inter-domain server is merged with LLM for inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses:\n\n1: Thank you. FIT_{32qv} is the configuration of the original FIT. We experimented with FIT_{16qvd} and FIT_{32d} to explore the impact of different LoRA configurations on performance in the current domain without increasing computational and communication costs. Adding more LoRA modules would increase these costs. The choice of LoRA[D] is primarily based on the experimental results shown in Table 1 of the original paper.\n\n2: Thanks. Compared to FIT, DoFIT adds slight communication parameters between intra- and inter-domain server sides (indicated by S-Comm.), with little impact on well-resourced server sides (indicated by S-Comm.).\n\n3: In our DoFIT, we only introduced two hyperparameters: top-k and α. As shown in Tables 1 and 2, and Figures 4 and 5 of the original paper, the performance of DoFIT varies little with different top-k and α values. Therefore, even when generalizing to new domain data, extensive adjustment of these two hyperparameters is not necessary. The proposed DoFIT demonstrates a certain robustness to these hyperparameters.\n\n4: Thank you. The performance of Base_{top30} in Table 2 is better.\n\n5: The final test involves first merging the LLMs with global LoRA on the intra-domain server side, and then testing on the current domain dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> We thank the reviewers for their valuable comments. We are glad that the reviewers found:

- The topic we are addressing to be promising and important (Reviewers ZeZJ, Zbxm, DeWw). 
- Our experiments to be convincing and showing significant performance improvement (Reviewer Vje9, DeWw).
- Our idea of aggregating and initializing to be novel and compelling (Reviewers Vje9, ZeZJ, xCSC, DeWw). 
    - However, there may be a conflict with the novlty issue concerned by Reviewer Zbxm. 

> We have responded to the comments given by the reviewers carefully. Here we summarize a few important points of our rebuttal or revision.

1.We explained the differences between federated domain adaptation for LLMs and existing federated domain adaptation methods.

2.Our experiments include: 

* Comparison with the existing federated domain adaptation method FedGP.
* New criteria: gradient and singular value.
* Complexity, convergence, and client numbers. 
* Visualization of domain heterogeneity.

3.Federated domain adaptation for LLMs is crucial, yet no methods exist. Our work is the first FIT framework to address domain-aware data heterogeneity, offering inspiration for future research.

Meta-review: Most reviewers agree that the proposed approach DoFIT is a novel and insightful approach to handle intra-domain and inter-domain data heterogeneity in federated instruction tuning for LLMs.
In particular, DoFIT performs inter-domain aggregation, i.e., domain-aware updates, by aggregating overlapping modules between domains. 
To avoid catastrophic forgetting, DoFIT projects inter-domain information onto parameter regions that are less affected by intro-domain updates.
The method demonstrates that aggregating across two domains (e.g., finance and general) can enable boost in performance.
Furthermore, the experiments on finance, general, and medical domains give evidence for the usefulness of DoFIT, outperforming other methods. 

While there was some concern about the method's novelty with respect to federated domain adaptation, which also handles domain heterogeneity, the authors explained that LLM-based FIT has unique characteristics that distinguish it from the standarnd non-LLM setting and provided evidence that prior FL work does not fill this gap.
Other minor concerns included the restriction to only two domains rather than multi-domain. However, the strong results on two domains, which could be a realistic situation, are sufficient to demonstrate the effectiveness of the method.
The authors should include their new results and discussion on these points in their final version.

Overall, the paper presents an effective approach to federated instruction tuning for LLMs that appropriately handles domain heterogeneity such that it avoids catastrophic forgetting and shows improved performance.
Decision: Accept (poster)