DoFIT: Domain-aware Federated Instruction Tuning with Alleviated Catastrophic Forgetting

Authors: binqian  xu
xiangbo  shu
haiyang  mei
zechen  bai
basura  fernando
mike zheng shou
jinhui  tang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:52:39

Abstract: Federated Instruction Tuning (FIT) advances collaborative training on decentralized data, crucially enhancing model's capability and safeguarding data privacy. However, existing FIT methods are dedicated to handling data heterogeneity across different clients (i.e., client-aware data heterogeneity), while ignoring the variation between data from different domains (i.e., domain-aware data heterogeneity). When scarce data needs supplementation from related fields, these methods lack the ability to handle domain heterogeneity in cross-domain training. This leads to domain-information catastrophic forgetting in collaborative training and therefore makes model perform sub-optimally on the individual domain. To address this issue, we introduce DoFIT, a new Domain-aware FIT framework that alleviates catastrophic forgetting through two new designs. First, to reduce interference information from the other domain, DoFIT finely aggregates overlapping weights across domains on the inter-domain server side. Second, to retain more domain information, DoFIT initializes intra-domain weights by incorporating inter-domain information into a less-conflicted parameter space. Experimental results on diverse datasets consistently demonstrate that DoFIT excels in cross-domain collaborative training and exhibits significant advantages over conventional FIT methods in alleviating catastrophic forgetting. Code is available at [this link](https://github.com/1xbq1/DoFIT).

Main paper: 
 introduction: 
 Large Language Models (LLMs) have attracted significant attention due to their remarkable comprehension and reasoning capabilities, coupled with their vast potential for a wide array of applications [27] . This attention has spurred the development of various Parameter Efficient Fine-Tuning (PEFT) methods [13, 15, 14, 6] , aimed at efficiently adapting these powerful models to specific tasks under constrained computational resources [21] . Among them, LoRA [6] stands out as one of the most popular, due to its lower number of trainable parameters and the absence of additional inference computations. While LoRA significantly alleviates the computational burden associated with tuning LLMs, substantial challenges persist at the data level [28, 31] , particularly in domains involving privacy concerns, where there is a lack of high-quality for cultivating a strong model [26, 29] .Towards this issue, some Federated Instruction-Tuning (FIT) methods have been explored by combining LLM instruction-tuning using LoRA with Federated Learning (FL) in recent years [37, 35, 38, 11, 34, 4, 9, 24] . In such FIT methods, the server side coordinates multi-round training among clients without sharing data for boosting model capability and protecting data privacy. Specifically, each round consists of four steps: global LoRA downloads from the server back to the clients, local Figure 2 : Performance effect of conventional FIT trained on Specific domain (i.e., Finance) and Finance&General domain. FinGPT [36] and Alpaca-GPT4 [23] are the training datasets on Finance domain and General domain, respectively. FPB [19] , FiQA-SA [18] , TFNS [17] , and NWGI [33] are all the evaluation datasets on Finance domain. Avg:3 and Avg:4 denote the average result on the first three evaluation datasets (i.e., FPB, FiQA-SA, and TFNS) and all the evaluation datasets, respectively.LoRA updates on the client side, local LoRA uploads from clients to the server, and global LoRA aggregates on the server side.Currently, most FIT methods are dedicated to handling data heterogeneity between clients, i.e., clientaware data heterogeneity. When data within a specific domain is scarce and needs to be supplemented with data from other related domains to develop a powerful model, existing FIT methods treat intraand inter-domain data heterogeneity (i.e., domain-aware data heterogeneity) equally and cannot adapt to the variation in cross-domain training, as shown in Figure 1 (a). This variation comes from more common information in the same domain and more interference information from different domains. Furthermore, as shown in Figure 2 , conventional FIT with LoRA often suffers from catastrophic forgetting of domain information, and performs suboptimally on each individual domain, due to its inability to handle the domain-aware data heterogeneity. To tackle domain-aware data heterogeneity well, two main parts in conventional FIT, i.e., aggregation and initialization, are evolved in this work.Revisiting Aggregation in FIT. Considering the variation in intra-domain and inter-domain data, we design a domain-aware FIT baseline (DoFIT-base) for separately aggregating intra-and inter-domain information, as shown in Figure 1 (b). In DoFIT-base, it first normally aggregates domain-specific information on the intra-domain server side, and then aggregates overlapping domain-agnostic information at a fine granularity in the inter-domain server side. In the inter-domain server side, this fine-grained aggregation strategy aims to reduce interference from irrelevant information. This strategy mainly includes two steps. The first step is to upload only the top-k important modules from each intra-domain server side. It is noticeable that that DoFIT-base still uses LoRA (A or B as one module), with each layer of LoRA focusing on different aspects [30] . The larger the LoRA weight is, the greater its impact on the frozen LLM becomes, and thus, it becomes more important [40] . The second step is to averagely aggregate the overlapping modules uploaded from different intra-domain server sides. This step is necessary because the top-k modules uploaded by different intra-domain server sides are different. After aggregation on the inter-domain server side, each intra-domain server side obtains important and necessary module information for itself.Revisiting Initialization in FIT. To retain more domain information, inspired by the orthogonal learning approach in [12] , we introduce a new initialization strategy based on the proximal perturbation by projecting the modules with inter-domain information onto parameter regions least affected by intra-domain update. Specifically, as shown in Figure 1 (c), after completing the aggregation on the inter-domain server side, the newly generated modules are transmitted to the intra-domain server side. On the intra-domain server side, proximal perturbations are calculated between the new and the original modules, and then added to the original modules rather than being directly overwritten. The proximal perturbation term contains inter-domain information, while the original modules retain global intra-domain information. This less-conflicted initialization strategy can more effectively preserve domain information, ultimately mitigating catastrophic forgetting.In summary, our contributions are: 1) the first solution to concern domain-aware data heterogeneity in collaborative training on decentralized data for the FIT paradigm; 2) a new domain-aware FIT framework that involves fine-grained inter-domain aggregation to handle domain-aware data heterogeneity; 3) a novel initialization strategy in intra-domain global LoRA to alleviate catastrophic forgetting in terms of domain information; and 4) the significant performance gain over conventional FIT and comprehensive analysis to pave the way for future explorations into more advanced FIT.2 Related Works 2.1 Federated Instruction Tuning.Under the condition of protecting client data privacy, Federated Instruction Tuning (FIT) enables collaboration on high-quality instruction data, facilitating the instruction-tuning of pre-trained LLMs for downstream tasks aimed at understanding diverse human intentions [37, 35, 32] . As the pioneer work, Zhang et al. [37] introduced a basic framework, which adopts LoRA for conducting client-side updating and server-side aggregation. Compared to the basic initial framework, Kuang et al. [11] provided a comprehensive framework that covers data processing, federated training, and multiple benchmarks, while also implementing various Parameter-Efficient Fine-Tuning (PEFT) methods, memory-saving operations, as well as acceleration techniques.Since the above FIT methods neglect alignment with human values, Ye et al. [35] presented federated value alignment alongside federated instruction-tuning, ensuring both harmless and helpful outputs.To tackle data extraction attacks and limited instruction data, Zhang et al. [39] proposed to employ LLM to synthesize data, and then train local models on both synthesized and original data, as well as global models solely on synthesized data. In heterogeneous scenarios with test-time distribution shift, Yang et al. [34] proposed a personalized FIT achieved by incorporating local LoRA and shared global LoRA. To confront the challenges arising from resource and data heterogeneity, Zhang et al [38] explored pruning personalized sparse structures for clients with resource imbalances using neural architecture search.However, the above FIT methods only consider client-aware data heterogeneity, overlooking both the intra-and inter-domain data heterogeneity when different relevant domains co-exist, namely domain-aware data heterogeneity. To this end, we introduce intra-and inter-domain server sides to deal with two types of heterogeneity differently. 
parameter-efficient fine-tuning.: 
 The increasing parameter size of LLMs results in expensive computational costs. However, for adapting to downstream tasks, full-parameter fine-tuning of LLMs poses challenges on hardware platforms with limited computational resources [3] . To this end, various Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed by freezing the pre-trained LLMs while only fine-tuning a small number of parameters [3] . Here, we mainly introduce several representative PEFT methods, i.e., Serial Adapter [5] , Prefix-tuning [13] , P-Tuning [15] , IA3 [14] , and Low-Rank Adaptation (LoRA) [6] . For more details about PEFT methods, please refer to reference [3] .Serial Adapter [5] built two adapter modules following the self-attention and FFN layers. Each adapter module comprises a down-projection matrix, a non-linear activation function, and an upprojection matrix. Prefix-tuning [13] , as soft prompt method, added trainable vectors as prefixes to both the key and value of all layers, while P-Tuning [15] integrated trainable vectors as prefixes into the initial word embedding layer. Similarly, IA3 [14] added scaling trainable vectors to key, value, and FFN activations. As one of the re-parameterization methods, LoRA [6] decomposed frozen parameters into two low-rank trainable matrices during fine-turning, and merged them with LLMs during inference without extra computational overhead. Similarly, considering the efficiency of LoRA in inference, our DoFIT adopts the PEFT method with LoRA.3 Methodology 
preliminaries: 
 Low Rank Adaptation (LoRA). Considering the limited computational resources available on the client side, it is challenging to perform full-parameter instruction tuning for LLM. Fortunately, Low Rank Adaptation (LoRA) [6] , the most popular Parameter-Efficient Fine-Tuning (PEFT) method in federated setting [11] , has been successfully applied in Federated Instruction Tuning (FIT) [37, 35] .During adaptation for specific tasks, given the low intrinsic dimension of pre-trained language models, LoRA assumes that the update to the pre-training weight matrices similarly exhibit a low intrinsic rank [6] . Consequently, for a frozen pre-trained weight matrix W 0 ∈ R d×k , its updating weight matrix △W ∈ R d×k is decomposed into low-rank trainable parameters BA, as follows,EQUATIONwhere B ∈ R d×r , A ∈ R r×k , and the rank r ≪ min(d, k). A uses random Gaussian initialization and B is initialized with zero.Conventional FIT with LoRA. Federated Instruction Tuning (FIT) is intended to resolve the issue of inadequate high-quality instructional data for individual clients and the inability to share such data due to privacy concerns. In FIT, it usually involves a server and multiple clients, where the server achieves collaborative training of non-shared instructional data among different clients by aggregating and initializing clients' updating weight matrices. To provide a detailed description of this process, we first define the updating weight matrix △W t i in the client side, as follows,EQUATIONwhere r, i, l, and L denote the t-th round, the i-th client, the l-th layer, and total L layers, respectively. During the training process, the updating weight matrix △ W t-1 l from the server is first downloaded for initializing the updating weight matrix △W t i,l on the client, as follows,EQUATIONAlthough only the weight initialization for the i-th client is indicated here, all selected clients undergo the same initialization process. After initializing, the updating weight matrices are optimized (i.e., {B t i,l A t i,l } ← {B t i,l A t i,l } init ) based on their individual data, with loss function corresponding to each client's task. Subsequently, the updating weight matrices {△W t i,l } i∈Ω N from selected client sets Ω N are uploaded to the server side for aggregation, generating a new updating weight matrix △ W t l on the server side. The aggregation process is shown as below,EQUATIONwhere Ω N represents a set of randomly sampled client indices, with a total of N clients. The function Agg(•) averages B t i,l and A t i,l within the selected clients' corresponding layer. Bt l Āt l constitutes the new aggregated updating weight matrix △ W t l .The intra-domain and inter-domain data heterogeneities are unequal in domain-aware data heterogeneity. Conventional FIT fails to distinguish between intra-and inter-domain data heterogeneities, as it employs the same federated architecture to handle them equally, only altering the client data to be within the same domain or across different domains, as shown in Figure 1 (a). Hence, when various clients possess datasets from other relevant domains, the results of conventional FIT may be inferior to those of the original specific domain, as shown in Figure 2 .In comparison to the intra-domain scenario, data from different domains demonstrate greater heterogeneity. To benefit from information in other relevant domains, we need to carefully design the aggregation strategy to more finely extract and aggregate the shared information between the current domain and other relevant domains. Therefore, we introduce a domain-aware FIT baseline (called DoFIT-base) that completes different aggregation strategies from coarse-grained level to fine-grained level. As shown in Figure 1 (b), DoFIT-base takes into account both the intra-domain variance and inter-domain variance, where the latter is more challenging.Specifically, DoFIT-base contains the intra-domain server side, inter-domain server side, and several client sides. First of all, in the inter-domain server side, the updating weight matrix △ W t-1 is defined as follows,△ W t-1 = { B t-1 l A t-1 l } ∈Ψ t-1(5) where Ψ t-1 denotes the overlapping modules (B or A as one module) from different domains in round t -1. Noted, "overlapping" refers to both the same layer and the same decomposition components. At the beginning of the first round, A t-1 l and B t-1 l are initialized with random Gaussian initialization and zero, respectively.Step. In the updating weight matrix △ W t-1 m of intra-domain server side, the overlappingEQUATIONwhere m denotes the m-th domain.Ψ t-1 ∩Ψ ∁ t-1 = ∅, Ψ t-1 ∪Ψ ∁ t-1 = U, and U = { Bt-1 m,l Āt-1 m,l } L l=1 . Next, △ W t-1 minitializes the updating weight matrix △W t m,i on the i-th client side with the same domain, as follows,EQUATIONStep. On the client side, △W t m,i is updated based on local data and specific task loss. Then, similar to conventional FIT, the updated △W t m,i is uploaded to the intra-domain server side for aggregation, as follows,EQUATIONSimilar to [40] , the squared norm of the module serves as the important score for itself, determining its impact on the frozen LLM. Obviously, not all modules have the same importance score. In fact, the more important a module is, the greater its influence on the instruction tuning process. To mitigate the impact of irrelevant information from other domains, only common and important information should be captured. In the intra-domain server side, all modules in △ W t m are sorted based on their important scores, where only the top-k modules are selected and uploaded to the inter-domain server side. Subsequently, on the inter-domain server side, the overlapping modules across all domains are aggregated accordingly, while the modules that do not overlap across all domains remain unchanged, as follows,EQUATION)where { Bt m,l Āt m,l } ∈Ψt and { B t l A t l } ∈Ψt . Ψ t indicates the set of overlapping modules in the t-th round, and M denotes the number of domains.In traditional FL, the iterative training across multiple rounds often results in global information forgetting from previous rounds due to the heterogeneity nature of data on client sides. This issue persists in our DoFIT-base, which is concretely manifested as the problem of domain information forgetting.To retain more domain information and alleviate such problem, inspired by the orthogonal learning in [12] , we translate inter-domain information to the parameter space least conflicted by the updating on the intra-domain server side, thereby reducing conflicts between intra-and inter-domain information. Thus, the initialization process in Eq. 6 can be modified from directly covering △ W t-1 m by △ W t-1 , for adding a proximal perturbation computed from the module-wise difference between △ W t-1 and △ W t-1 m , as follows,EQUATION)where α is the scaling factor.| B t-1 l -Bt-1 m,l | || B t-1 l -Bt-1 m,l ||2and| A t-1 l -Āt-1 m,l | || A t-1 l -Āt-1m,l ||2 denote proximal perturbation terms, mapping B t-1 l and A t-1 l to the parameter region least affected by Bt-1 m,l and Āt-1 m,l . The overall algorithm process of DoFIT is described in the supplemental material due to space limitations. 
experimental settings: 
 Datasets. We train our DoFIT on three datasets, i.e., FinGPT [36] , Alpaca-GPT4 [23] , and MedAlpaca [2] from the Finance (F), General (G), and Medical (M) domains, respectively. In the F domain, FinGPT is an open-source dataset for financial sentiment analysis, consisting of 77k samples. In G domain, Alpaca-GPT4 comprises 52k instances of English instruction-following data, generated by GPT-4 [1] using identical prompts as Alpaca. In M domain, MedAlpaca includes 34k question-answer pairs sourced from the Anki medical curriculum flashcards.Configurations. In all experiments conducted on one NVIDIA A40, the frozen LLM used is Llama2-7B with 32 layers [27] quantized to int8. The LoRA rank and alpha are set to 32 and 64, respectively. The maximum sequence length is 512. Following the formatting instructions of Alpaca template [25] , the training runs for 200 rounds, with a cosine learning rate scheduler adjusting the learning rate from 5e -5 to 1e -6. In each round, the selected clients are trained 10 steps by AdamW [16] optimizer. The batch size is set to 16. In FinGPT/Alpaca-GPT4/MedAlpaca training, total 10k/20k/20k samples for 50/20/20 clients, selecting 5/2/2 clients randomly per round. Similar to [35] , each training dataset Table 1 : Comparing "Local", Conventional FIT ("FIT"), DoFIT-base ("Base"), and "DoFIT" on Finance (F) domain and Finance&General (F&G) domain datasets. FinGPT [36] and Alpaca-GPT4 [23] are the training datasets on F domain and G domain, respectively. FPB [19] , FiQA-SA [18] , TFNS [17] , and NWGI [33] are the evaluation datasets on F domain. Avg:3 and Avg:4 denote the average result on the first three evaluation datasets (i.e., FPB, FiQA-SA, and TFNS) and all evaluation datasets, respectively. ↑ refers to the performance improvement compared to the alternative marked with the same color (i.e., using the same LoRA configuration) on F domain. ↓ denotes performance degradation, oppositely. Comparison Methods. To validate the effectiveness of the proposed DoFIT-base (called "Base" for conciseness) and "DoFIT" in addressing intra-and inter-domain data heterogeneity, it is essential to compare them with "Local" and "FIT". Here, "Local" refers to training independently based solely on data from a single client. "FIT" [35] refers to the collaborative training of different client data based on FedAvg [20] , treating client data from different domains equally. FIT 32qv uses all layers' LoRA[Q,V] (decomposed from Q and V components of self-attention). FIT 16qvd uses half of all layers' LoRA[Q,V,D] (decomposed from the Q, V components of self-attention and Down linear layer of MLP). FIT 32d uses all layers' LoRA[D] (decomposed from the Down linear layer in MLP). Noted, the aforementioned Conventional FIT is FIT 32qv [35] . For fairness, we also use FedAvg in the collaboration training from intra-domain data. In Specific and General domains, "Base top10 " refers to uploading the top-10 important modules from the intra-domain server side to the inter-domain one, based on the best baseline in the Specific domain. The rest of "Base top * " has a similar definition. "DoFIT" modifies the initialization of the best "Base" by incorporating the proximal perturbation initialization. "DoFIT α=1.0 " means that the scaling factor in DoFIT is set to 1.0.Table 2: Comparing "Local", Conventional FIT ("FIT"), DoFIT-base ("Base"), and "DoFIT" on Medical domain (M), and combined Medical&General domain (M&G). MedAlpaca [2] , and Alpaca-GPT4 [23] are the training datasets on M domain, and G domain, respectively. MedQA [10] , and MedMCQA [22] are the evaluation datasets on M domain. ↑ refers to the performance improvement compared to the alternative marked with the same color (i.e., using the same LoRA configuration) on M domain. Comparison on F Domain and F&G Domain. After training on FinGPT for Finance (F) domain or FinGPT/Alpaca-GPT4 for Finance&General (F&G) domain, we test the models at round 50 on the evaluation datasets for F domain, i.e., FPB [19] , FiQA-SA [18] , TFNS [17] , and NWGI [33] , as shown in Table 1 . Compared to the independently trained "Local", the collaboratively trained "FIT", "Base 16qvd " and "Base 32d " via FL perform better. This indicates that training can benefit from other clients' data. Expanding from F to F&G domain, the performance of "FIT" even declines, while our "Base top15 " improves and surpasses that of FIT. This validates the effectiveness of our DoFIT-base for addressing domain-aware data heterogeneity. In F&G domain, "DoFIT" further improves performance compared to "Base top15 ". This indirectly validates that the proposed DoFIT retains more inter-domain information, enhancing overall performance.Comparison on M Domain and M&G Domain. After training on MedAlpaca for Medical (M) domain or MedAlpaca/Alpaca-GPT4 for Medical&General (M&G) domain, we test the models at round 60 on the evaluation datasets for Medical (M) domain, i.e., MedQA [10] , and MedMCQA [22] .As shown in Table 2 , we can see that: 1) "FIT", "Base 16qvd ", and "Base 32d " consistently outperform "Local", indicating that collaborative training can enhance model's capability; and 2) on M&G domains, "Base top30 " and "DoFIT" exceed the performance of FIT, especially "DoFIT" with the proximal perturbation initialization strategy, proving the effectiveness of this strategy.Loss Curves of Different Methods and Hyper-parameters. As shown in the loss curves of Figure 3 , compared to "DoFIT-base", "DoFIT" consistently shows faster convergence and lower losses, as does "DoFIT-base" compared to "FIT". Noted, the losses of different methods are similar in the first rounds, while a gap emerges as the number of rounds increases. Figure 4 and Figure 5 also show the loss curves with different values of Top-k and α. we can find that the losses are insensitive to the values of these parameters to some extent.Comparison of Parameter Size. Table 3 further shows the number of parameters per round on FIT, the best-performing "Base top15 "/"Base top30 ", and "DoFIT" in F&G and M&G domains. Compared to FIT, DoFIT adds slight communication parameters between intra-and inter-domain server sides (indicated by S-Comm.), with little impact on well-resourced server sides (indicated by S-Comm.). Compared with "FIT 32qv ", either Base top15 or DoFIT requires fewer trainable parameters in the client side, as well as fewer communication parameters between the client side and the intra-domain server side. 
conclusion and future work: 
 In this work, we introduced a novel Domain-aware Federated Instruction Tuning (DoFIT) framework towards collaborative training on more datasets in relevant domains for boosting the performance of individual domains. To alleviate the catastrophic forgetting problem caused by domain-aware data heterogeneity, our DoFIT offers two main insights in terms of aggregation and initialization. For aggregation, we first normally aggregate domain-specific information on the intra-domain server side, and then aggregate overlapping domain-agnostic information on the inter-domain server side, excluding the interference information. For initialization, we add a proximal perturbation from interdomain information to the original modules, rather than directly overwritten them. Comprehensive experimental results on Finance, Medical, and General domains demonstrate the effectiveness of the proposed DoFIT method, compared to conventional FIT. Ω N ← A set of randomly sampled client indices.3:for m = 1,• • • ,M do 4: if t > 1 then 5:Intra-domain initialization6:Add a proximal perturbation on Eq. 10 7:end if8:for each i in Ω N do 9:{△W t m,i } init = △ W 0 m 10:Conduct e epochs of local instruction-tuning for △W t m,i .11:end for 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Please check out the experimental settings, and performance evaluation sections.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Please check out the experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research in every respect conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Please check out the introduction. The paper has a positive societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: This model uses LLMs with safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper has cited the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Intra-domain Aggregation 13:△ W t m = Agg i∈Ω N (△W t m,i ) on Eq. 8 14:Compute the important score of each module for △ W t m .15:Upload top-k modules for △ W t m to the inter-domain server side.16:end for 17:Inter-domain AggregationCompute the set of overlapping modules from different domains: Ψ t 19:△ W t l = Agg m∈M (△ W t m,l ) { Bt m,lĀt m,l } ∈Ψt on Eq. 9 20: end forFederated domain adaptation for LLMs is crucial, but no related methods currently exist. Applying existing federated domain adaptation methods like FedGP [8] directly to LLMs yields suboptimal results, as shown in Table 4 . Where FedGP/FedGP-g refer to the projection of each client's LoRA/global LoRA weights in the source domain onto the global LoRA weights in the target domain. FPL [7] clusters prototypes from different domains into unbiased prototypes for general domain shift. However, existing federated domain adaptation methods [7, 8] for this task still merge more redundant and noisy parameters to LLMs, affecting domain fine-tuning performance. Overall, as shown in Table 1 and Table 4 , our method outperforms traditional FIT and general federated domain adaptation methods. We add two new criteria-gradient and singular value-to assess module importance in LoRA, as follows, Table 5 : Performance on the gradient and singular value spectrum.Criteria FPB FiQA-SA TFNS NWGI Avg:3 Avg:4 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 DoFITα=1.0 0.861 0.858 0.818 0.847 0.869 0.869 0.641 0.640 0.849 0.858 0.797 0.804 A-grad-top15 0.866 0.864 0.833 0.852 0.867 0.867 0.640 0.639 0.855 0.861 0.802 0.806 A-svd-top15 0.858 0.855 0.829 0.856 0.828 0.829 0.642 0.641 0.838 0.847 0.789 0.795 B-grad-top10 0.823 0.813 0.789 0.827 0.802 0.806 0.633 0.633 0.805 0.815 0.762 0.770 B-grad-top15 0.833 0.829 0.840 0.855 0.681 0.693 0.630 0.627 0.785 0.792 0.746 0.751 B-grad-top20 0.516 0.480 0.185 0.197 0.501 0.500 0.404 0.369 0.401 0.392 0.402 0.387 B-svd-top10 0.856 0.854 0.844 0.854 0.732 0.740 0.638 0.627 0.811 0.816 0.768 0.769 B-svd-top15 0.821 0.819 0.793 0.824 0.621 0.626 0.644 0.640 0.745 0.756 0.720 0.727 B-svd-top20 0.417 0.306 0.811 0.794 0.371 0.302 0.552 0.457 0.533 0.467 0.538 0.540 5 , the importance scores based on the gradient square norm and the sum of singular values are comparable to the module importance scores calculated using the square norm of LoRA weights in DoFIT.From a domain distribution-aware perspective, aggregate the top-k modules with smaller domain gaps, Gradient: Uses the mean absolute difference of LoRA module gradients across different domains to reflect domain heterogeneity gaps (B-grad-top*). Singular Value: Uses the L2 norm of the differences in the singular value spectrum of LoRA modules across different domains to reflect domain heterogeneity gaps (B-svd-top*). As shown in Table 5 , using gradient or singular value to aggregate modules with smaller domain heterogeneity shows more sensitivity to the top-k hyperparameter. Compared to our DoFIT, this approach performs worse. This may be because aggregating modules with smaller domain heterogeneity can still introduce redundant and noisy modules, which can degrade overall performance when merged into the LLMs.Overall, focusing on domain-specific key parameters and removing redundancies improves performance with LLMs. DoFIT's square norm method for weights is comparable to gradient and singular value spectrum methods but is more intuitive and reproducible.Complexity Analysis: As shown in Table 3 , our DoFIT has the same space complexity as the traditional FIT on the client side, without any additional memory cost, but introduces a slight memory cost (S-comm.) on the inter-domain server side. In terms of time complexity, our DoFIT is identical to the traditional FIT on the client side, with only a slight computational overhead for module importance ranking on the intra-domain server side. Assuming the number of selected clients in the same domain is k, and each client includes 32d LoRA (64 modules), the sorting time complexity is Convergence Results: As shown in Figure 6 , our DoFIT demonstrates faster and more stable convergence compared to FIT using FedAvg and FedProx as the FL framework in both single-domain and dual-domain scenarios.Client Numbers: 50(5) & 20 (2) indicate that in the financial domain, there are 50 clients in total, with 5 clients randomly selected for training and uploading each round. In the general domain, there are 20 clients in total, with 2 clients randomly selected for training and uploading each round. As shown in Table 6 , varying the total number of clients or the number of selected clients does not cause significant fluctuations in the results, demonstrating that the proposed DoFIT is robust to the number of clients.The importance of modules in LoRA varies across different domains, indirectly reflecting domain heterogeneity. As shown in Figure 7 (left), the top-15 important modules in domains F and G are not completely the same. As training progresses, the weights of the same modules become more reinforced.We also further compute the L2 norm of the difference in the singular value spectrum between each client's LoRA and the global LoRA for the same domain and different domains. As shown in Figure 7 (right), this visualization reflects smaller intra-domain data heterogeneity and greater inter-domain data heterogeneity.Justification: Please check out the .zip file.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: Most reviewers agree that the proposed approach DoFIT is a novel and insightful approach to handle intra-domain and inter-domain data heterogeneity in federated instruction tuning for LLMs.
In particular, DoFIT performs inter-domain aggregation, i.e., domain-aware updates, by aggregating overlapping modules between domains. 
To avoid catastrophic forgetting, DoFIT projects inter-domain information onto parameter regions that are less affected by intro-domain updates.
The method demonstrates that aggregating across two domains (e.g., finance and general) can enable boost in performance.
Furthermore, the experiments on finance, general, and medical domains give evidence for the usefulness of DoFIT, outperforming other methods. 

While there was some concern about the method's novelty with respect to federated domain adaptation, which also handles domain heterogeneity, the authors explained that LLM-based FIT has unique characteristics that distinguish it from the standarnd non-LLM setting and provided evidence that prior FL work does not fill this gap.
Other minor concerns included the restriction to only two domains rather than multi-domain. However, the strong results on two domains, which could be a realistic situation, are sufficient to demonstrate the effectiveness of the method.
The authors should include their new results and discussion on these points in their final version.

Overall, the paper presents an effective approach to federated instruction tuning for LLMs that appropriately handles domain heterogeneity such that it avoids catastrophic forgetting and shows improved performance.

Reviews: review: paper_summary: paper_summary: In this work, the issue of domain-aware data heterogeneity is solved, which equally treats intra- and inter-domain data variations but cannot adapt to the challenges of  cross-domain training. The paper proposes a novel domain-aware federated instruction tuning (DoFIT) framework for collaborative training across datasets in related domains to enhance the performance of individual domains. Specifically, DoFIT aggregates the top-k important modules and initializes intra-domain modules with addition of a proximal perturbation term. The proposed DoFIT demonstrates a significant performance improvement when transitioning from single-domain to dual-domain datasets.
strength_weakness: strength_weakness: 1.The concept of different domains is less clearly defined, and the scope of application of the proposed framework seems limited.
2.The experiments only yield results on single-domain and dual-domain datasets, and cannot be extended to multi-domain scenarios.
3.The privacy and security of the intra- and inter-domain server sides do not seem to be addressed.
questions: questions: Whether FIT32qv differs between single-domain and dual-domain scenarios, and how to extend it to dual domains should be explained.
limitations: limitations: The authors provide limitations and future works.
paper_summary: paper_summary: The authors propose to utilize intra- and inter-domain server sides in a federated instruction tuning framework to implement discriminative aggregation and initialization strategies. The proposed approach, i.e., DoFIT, is primarily based on FIT of LLM, marking the first solution to address domain-aware data heterogeneity in collaborative training on decentralized data for the FIT paradigm. Unlike conventional FIT, which ignores the variation between data from different domains, DoFIT enables the intra-domain server to perform normal aggregation and initialization, while the inter-domain server handles overlapping weights aggregation and less-conflicted initialization. Consequently, DoFIT reduces interference information and preserves more domain information to mitigate catastrophic forgetting. The authors conducted empirical comparisons with conventional FIT methods on three datasets from finance, general, and medical domains.
strength_weakness: strength_weakness: 1. In the F&G and M&G domains, the values of k and α in DoFIT are quite different. Could adding more hyperparameters potentially harm the generalization of DoFIT?
2. There are fewer methods compared in the single-domain experiments, and it seems that more comparisons of FL methods other than FedAvg should be added.
3. The proposed intra- and inter-domain servers in DOFIT-Base and DoFIT increase the communication cost compared to traditional FIT methods.
questions: questions: 1. The meaning of “Overlap” in Figure 1 is unclear and should be explained, preferably in the title.
2. The set of modules where “Overlap” resides is in units of LoRA B or A, while the updating weight matrix is in units of each layer in the transformer architecture and should be indicated.
3. Is the scaling factor α in Eq. 10 the same for LoRA B and A?
limitations: limitations: No limitation is explained in the paper.
paper_summary: paper_summary: This work offers a solution to a problem in the collaborative training of different domains on decentralized data within the FIT paradigm: domain-aware data heterogeneity causes domain-information catastrophic forgetting. The solution relies on two new designs for aggregation and initialization. Specifically, in the aggregation step, DoFIT-base aggregates overlapping inter-domain information at a fine granularity on the inter-domain server side. In the initialization step, DoFIT projects modules with inter-domain information onto parameter regions least affected by intra-domain update. Finally, the authors conducted extensive comparison experiments to well show the significant effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Although the current framework performs better than conventional frameworks in F&G or M&G domains, can it be further applied to more domains? 
2. There are related articles that test the FL results of different domain data in different clients, where each client has data from one domain data. Thus, it is not clear how the cross-domain training problem differs in this paper.
questions: questions: 1. In the DoFIT framework, the final test determines whether to experiment with global LoRA on the inter-domain server side or the intra-domain server side.
2. Is it more reasonable that the scaling factors of LoRA B and A are not consistent?
3. There are too many symbols in the methods section; it is best to use a table to clearly specify the important symbolic variables.
4. The difference between DoFIT-base and Conventional FIT in Section 3.2 is best highlighted.
limitations: limitations: no limitation in this scope.
paper_summary: paper_summary: This paper proposes Federated Instruction Tuning, aimed at enhancing model's capability and data privacy. The main points lie in enhancing the model of data heterogeneity across dfifferent clients.  Specifically, the paper introduces DoFIT, a domain-aware FIT framework, aimed at alleviating the catastrophic forgetting through aggregating overlaping across domains, and incorporating inter-domain information into a less-conflict parameter space to reduce interference information from other domain. The proposed method is evaluated on diverse datasets.
strength_weakness: strength_weakness: 1. The novelty is limited. There are already many works in federated domain adaptation that consider the differences between domains [1][2].It seems the authors only considers large models (model differences) and uses LoRA for optimization to analyze domain heterogeneity. This approach seems to simply apply domain heterogeneity to federated learning with large models.
2. What distinguishes this work from other Federated Domain Adaptation (FDA) methods that also address domain heterogeneity?
3. I would prefer to see the authors use a more innovative criterion when considering module importance scores, rather than directly using existing work [31]. Specifically:1) Compare with other criteria, such as sorting by gradient, data consistency, etc. 2) I suggest the authors consider module importance from a domain distribution-aware perspective. Starting from domain distribution would better reflect domain heterogeneity.
4. The experimental validation is insufficient as it is entirely based on LoRA settings. Important federated settings, such as complexity, convergence, and the impact of the number of clients on the results, have not been considered.
5. An analysis of domain-heterogeneity is necessary. For example, what aspects of the model can reflect domain-heterogeneity? How can the A and B matrices in LoRA be analyzed to reflect intra-domain and inter-domain heterogeneity (e.g., using matrix singular values)
6. The algorithm's complexity needs to be analyzed, including the additional computational overhead introduced by the module importance score sorting.


[1]	Jiang, Enyi, Yibo Jacky Zhang, and Oluwasanmi Koyejo. "Federated domain adaptation via gradient projection." arXiv e-prints (2023): arXiv-2302.
[2]	Huang, Wenke, et al. "Rethinking federated learning with domain shift: A prototype view." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
questions: questions: 1. What is the differences between the proposed method with those without large models?  The indroduction of LoRA may be a engineering problem not an technological contribution;
2. The sort critera is a little simple, what is the performance gain when used other critera?
3. The paper should include complexity, convergence, and the impact of the number of clients on the results under federated setting;
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: This paper first introduces a new domain-aware FIT baseline called DoFIT-base. DoFIT-base aggregates domain-specific information on the intra-domain server side and domain-agnostic information on the inter-domain server side to reduce interference information from the other domains. By incorporating inter-domain information into a less-conflicted parameter space, this paper formally introduce q new DoFIT framework that retains more domain information. Since DoFIT can handle domain-aware data heterogeneity, the catastrophic forgetting problem in cross-domain training is effectively alleviated. The authors show cross-domain training results in a federated LLM trained on FinGPT, Alpaca-GPT4, and MedAlpaca, demonstrating that DoFIT performs better than conventional FIT methods.
strength_weakness: strength_weakness: **(1)** Why did you choose FIT_{32qv}, FIT_{16qvd}, and FIT_{32d} as comparison methods in a single domain? Would it be better to add more LoRA modules? What is the basis for choosing LoRA[D]?

**(2)** Since more servers are added to handle cases with different heterogeneity separately, additional S-comm. increases the burden of communication and security risks.

**(3)** There are many parameters in DoFIT, and as shown in Table 1 and Table 2, the parameters are different across different datasets. This variability seems to make it difficult to generalize DoFIT to other datasets.

**(4)** Does Base_{top30} in Table 2 work well, or does Base_{top35} work better?

**(5)** It is still unclear whether LoRA on the client, intra-domain server, or inter-domain server is merged with LLM for inference.
questions: questions: Please see the comments above in the weaknesses.
limitations: limitations: The paper could benefit from more discussion on the limitations of the proposed method.

Comments: title: Response to author’s rebuttal
comment: Thank you for your response. The author indeed conducted thorough ablation experiments on the LoRA parameters in the original paper and explained in the reply that more parameters would increase computational and communication costs. Additionally, the experiments on parameters k and α demonstrate a certain robustness, which suggests they might generalize more easily to other domains. This response addresses my main concerns to some extent, so I agree with raising the score.
Thank you for taking the time to write a detailed response! It has definitely improved my understanding of the work and helped me appreciate its importance. 
The authors' response seems to resolve most of my concerns. Ultimately, this paper investigates interesting new approaches in utilizing intra- and inter-domain servers and proposes a simple solution. Accordingly, I will update my score from 6 (wa) to 8 (sa).