LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks (2024)

Abstract: LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional modules represent diverse skills. Combining existing LoRA modules to address new tasks can enhance the reusability of learned LoRA modules, particularly beneficial for tasks with limited annotated data. Most prior works on LoRA combination primarily rely on task-level weights for each involved LoRA, making different examples and tokens share the same LoRA weights. However, in generative tasks, different tokens may necessitate diverse skills to manage. Taking the Chinese math task as an example, understanding the problem description may depend more on the Chinese LoRA, while the calculation part may rely more on the math LoRA. To this end, we propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRA modules. The weights at each step are determined by a fusion gate with extremely few parameters, which can be learned with only 200 training examples. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with tasklevel fusion weights. This underscores the necessity of introducing dynamic fusion weights for LoRA combination.1

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated superior performance over previous smaller models across a wide range of tasks (OpenAI, 2023; Anil et al., 2023; Touvron et al., 2023a,b), thereby extending the applicability of AI systems to numerous real-world scenarios. Because of their substantial model size, training all parameters of LLMs can
* Equal Contribution. † Corresponding authors. 1 Code and models will be publicly available at https:
//github.com/thunlp/LoRAFlow.
often be prohibitively expensive. Therefore, several researchers have proposed a set of parameterefficient fine-tuning (PEFT) approaches (Houlsby et al., 2019; Li and Liang, 2021). Among these, LoRA (Hu et al., 2022) stands out as one of the most popular due to its efficiency and simplicity.
The basic idea of LoRA is to learn an additional module for each downstream domain or task. Rather than solely employing a single LoRA to address learned tasks, several recent studies have delved into the potential of combining existing LoRA modules to tackle unseen tasks (Zhang et al., 2023; Huang et al., 2024; Chronopoulou et al., 2023). This direction holds the potential to substantially enhance the reusability of learned parameters, facilitating the integration of diverse model capabilities.
Most existing LoRA fusion approaches employ a task-level weight distribution when combining different LoRA modules. This implies that all test examples and tokens share the same fusion ratio.
12871
However, for some complex generative tasks (e.g., solving mathematical problems or generating code according to provided instructions), the LLM may need to dynamically employ various types of capabilities to address the entire problem effectively. Figure 1 illustrates an example, where we have trained a Chinese chat LoRA (i.e., Zh Chat) and an English math LoRA (i.e., En Math), and our objective is to address a Chinese math problem. Intuitively, comprehending the Chinese problem description may rely more on the Chinese chat LoRA, whereas performing the calculation might depend more on the English math LoRA.
In this work, we propose LoRA-Flow, which can dynamically determine the token-level weights of different LoRA modules for generative tasks. At each time step, the fusion weights are generated by a gate module that conditions the current prefix. The fusion gate comprises an extremely small number of parameters, accounting for only approximately 0.2% of those in a LoRA. We find through experiments that the fusion gate can be learned through only 200 training examples. Figure 1 gives an example of LoRA-Flow. We also observe significant variations in weights across different model layers, suggesting that the impact of LoRA modules differs across layers. In summary, the contributions of this work can be outlined as follows:
• We propose LoRA-Flow, a method that combines existing LoRA modules with dynamic fusion weights to effectively control the influence of each LoRA module across various generation steps.
• We verify the effectiveness of LoRA-Flow on six different generation tasks, and the results show that LoRA-Flow can consistently outperform the baselines that use task-level fusion weights (e.g., LoRA-Hub).
• By carefully designed analyses from various aspects, we provide deeper insights into the integration of LoRA modules. We consider this journey to be fruitful in constructing a flexible plug-and-play community for LLMs, enabling developers to leverage plugins created by others to build up their own LLM applications.

heading: 2 Background
text: Large Language Models Most recent LLMs utilize a decoder-only architecture, comprising
stacked layers of identical structure to form the large-scale model. For a sequence y, an LLM estimates its probability in the following way:
P (y|θbase) = T∏
t=1
P (yt|y<t;θbase), (1)
where yt denotes the token at the t-th step and y<t is the prefix before t. θbase represents the parameters of the basic LLM.
LoRA LoRA (Hu et al., 2022) is a parameterefficient fine-tuning method that can achieve comparable performance to full fine-tuning in certain scenarios but at a significantly lower cost. Specifically, for a given matrix W ∈ Rm×d within the model, we can learn two low-rank matrices A ∈ Rr×d and B ∈ Rm×r to approximate the parameter update for W :
∆W = BA. (2)
LoRA Fusion Each trained LoRA possesses unique capabilities, and their combination can integrate various skills of LLMs. Formally, we use ∆W1 = B1A1 and ∆W2 = B2A2 to represent two existing LoRA modules. Zhang et al. (2023) propose to merge two LoRA modules in the following way:
h′ = Wx+ λ∆W1x+ (1− λ)∆W2x, (3)
where λ is a hyper-parameter that needs to be manually tuned.
LoRA-Hub (Huang et al., 2024) further improve the fusion method in the following ways:
h′ = Wx
+ (w1B1 + w2B2)(w1A1 + w2A2)x, (4)
where the fusion weights w1 and w2 are learned in a few-shot manner. While LoRA-Hub can automatically determine fusion weights for different LoRA modules, the weights across different tokens remain the same for a given task. This shared weight scheme may constrain the expressive capacity of the involved LoRA modules, particularly in complex generative tasks that entail diverse types of context.

heading: 3 Approach
text: To handle generative tasks more flexibly, we propose LoRA-Flow, which employs dynamic fusion
weights at each generation step. Subsequently, we will first introduce the way we calculate the fusion weights in Section 3.1, and then detail how we integrate the weights into the model in Section 3.2. Finally, we describe the training algorithm in Section 3.3.

heading: 3.1 Calculating Fusion Weights
text: At the t-th step, we aim to determine the fusion weights using the prefix y<t, which captures the context of the current token. Given that the backbone model has already compressed the context information into hidden vectors, we propose directly utilizing the hidden state at the t-th step to avoid redundant computations.
There are three levels of hidden states, each containing different granularities of information:
• Step-level hidden states xt: the input word embedding at the t-th step.
• Layer-level hidden states xlt: the input representation to the l-th layer at the t-th step.
• Module-level hidden states xl,typet : the input vector to a specific module (e.g., the query projection in the self-attention network).
As found in some previous studies, the hidden states in various layers may lie in different manifolds (Voita et al., 2019). Therefore, simply using the step-level hidden state xt to compute the fusion weights for the entire model may be not sufficiently effective. On the other side, using module-level hidden states would introduce plenty of new parameters for the fusion gates, as each module requires
an independent gate to cope with its input states xl,typet . We thus use the layer-level hidden state xlt. Empirical studies in Section 6.1 indicate that layer-wise fusion weights can outperform the other two types of counterparts.
As shown in Figure 2, for the l-th layer, the fusion gate takes in the input representation xlt and then projects it into fusion weights:
wl = softmax ( W lgatex l t ) + bl, (5)
where W lgate ∈ Rk×d and bl ∈ Rk×1. k is the number of LoRA modules. Both W lgate and b l are learnable parameters. Given that k is typically substantially smaller than d and r, the number of parameters within the fusion gates is negligible compared to that of the LoRA. For instance, in Llama-2-7b (Touvron et al., 2023b), a LoRA for the entire model contains 117.44M parameters with r = 64, while the gates for combining two LoRA modules only consist of 0.26M parameters in total.

heading: 3.2 Integrating Fusion Weights
text: As mentioned in Section 3.1, we use layer-wise fusion weights in LoRA-Flow. Once we get the fusion weights at the l-th layer, we feed the weights wl to all the modules that contain LoRA modules. Different from LoRA-Hub (Huang et al., 2024) that separately composes LoRA A matrices and LoRA B matrices (as shown in Eq (4)), we integrate the outputs of different LoRA modules, treating each LoRA as a complete module. The reason for this operation is to combine LoRA modules with different middle ranks. Let ∆h = [∆h1; · · · ; ∆hk] ∈ Rd×k denote the outputs of all the involved LoRA
modules, the fusion process can be expressed by
h′ = h+∆hwl, (6)
where h is the module’s output in the backbone. Figure 2 shows an example.

heading: 3.3 Training
text: For a backbone model θbase and a set of learned LoRA modules θLoRA = {θ1LoRA, · · · ,θkLoRA}, we train the fusion gate θfusion on the new task:
θ̂fusion = argmax θfusion
{L(θtotal|Dnew)} , (7)
where θtotal = θbase∪θLoRA∪θfusion denotes the total parameters. The likelihood is defined as
L(θtotal|Dnew) = N∑
i=1
P (yi|θtotal), (8)
where N is the number of training examples on the new task. We follow LoRA-Hub (Huang et al., 2024) to learn the fusion modules in a few-shot manner, where N is set to 200 in our experiments. We investigate the effect of N in Section 6.4. Since θfusion consists of only a few parameters, these limited training examples are adequate for learning an effective fusion mechanism.

heading: 4 Experiment

heading: 4.1 Setup
text: Base Model We use Llama-2 (Touvron et al., 2023b) as our base LLM to examine the performance of various LoRA fusion approaches, as it is among the most widely used open-source LLMs. Due to computational constraints, we use Llama-27b by default.
LoRA Training To conduct LoRA fusion experiments, we first learn several LoRA modules on the tasks with sufficient supervised data:
• Chinese chat (Zh Chat): we use the data released by Lai et al. (2023) to learn the Chinese chat LoRA, which is expected to possess the ability to understand and generate Chinese text. There are 52K training examples in total.
• Russian chat (Ru Chat): the training data of Russian chat LoRA is also from Lai et al. (2023), which consists of 52K training examples in Russian.
• Spanish chat (Es Chat): the training data is also from Lai et al. (2023), containing 52K training examples in Spanish.
• English math (En Math): the training data for English math LoRA is constructed by Yu et al. (2023), which is comprised of 395K mathematical problems in English.
• English code (En Code): we train the English code LoRA with the Magicoder dataset (Wei et al., 2023), which consists of 186K code generation problems in English.
We integrate LoRA modules into the query, key, value, and output projections within attention networks and the three linear projections in feedforward networks. By default, the LoRA rank r is set to 64 and the value of α is set to 16. For the En code LoRA, we set r=256, as we have observed that the code LoRA with r=64 is inadequate for learning the task effectively. Each LoRA module is trained using 8 A100 80G GPUs, where each mini-batch contains 128 training examples. We use the cosine warmup schedule and the peak learning rate is 1e-4. For each task, the LoRA is trained by 3 epochs and the warmup ratio is set to 0.04.
LoRA Fusion We evaluate LoRA fusion methods in a few-shot manner. Specifically, we conduct the evaluation on six tasks, including Zh math, Ru math, Es math, Zh code, Ru code, and Es code. For each task, we combine the chat LoRA in the target language and the task LoRA in English. Briefly, we use language LoRA to represent the target-language chat LoRA and task LoRA to denote the math or code LoRA trained in English.
For each task, we construct 200 training examples for the few-shot training, which is firstly translated by GPT-3.5 based on the English math or code data and then verified by humans. For fusion experiments on math, we also construct 100 problems as the validation set. For code experiments, we employ humans to create 20 problems as the validation set, since the code generation test examples are more time-consuming to annotate.2 Both LoRA-Hub and the proposed LoRA-Flow are trained with the few-shot data. We use the validation to search the training hyperparameters. The search space of the peak learning rate is {1e-3, 1e4}, and the search space of the batch size is {2, 4,
2See Appendix B for the translation prompt and Appendix C for the details of human annotation.
8}.3 Each fusion module is trained with 5 epochs, using the few-shot training data.
Evaluation For fusion experiments on math, we use MGSM (Shi et al., 2023) as the test set, which is a widely used multilingual evaluation benchmark for the math abilities of LLMs. For fusion experiments on code generation, we construct a multilingual version of HumanEval (Chen et al., 2021). The original English problem descriptions in HumanEval are translated by GPT-3.5 into other languages and then verified by humans. We report the accuracy and the pass@1 score on MGSM and HumanEval, respectively.

heading: 4.2 Main Results
text: Table 1 shows the results of the involved methods on different generative tasks. For comparison, we also show the performances of the base model and single LoRA. The task LoRA modules trained in English already demonstrate a notable degree of cross-lingual transfer capabilities, outperforming the language LoRA modules on non-English tasks. For example, in the Zh math task, the task LoRA achieves an accuracy of 26.8, whereas the language LoRA achieves only 5.2. On the code generation tasks, the task LoRA also significantly outperforms the language LoRA modules.
When combining the language and task LoRA modules, we compare our method with two baselines that use task-level fusion weights. The "Average" baseline refers to simply averaging the outputs of the two involved LoRA modules. As introduced in Section 2, LoRA-Hub learns task-level fusion weights using few-shot training data, similar to the proposed LoRA-Flow. However, LoRA-Flow utilizes dynamic fusion weights, which vary across different time steps and model layers. We use the open-source code released by Huang et al. (2024) to reimplement LoRA-Hub in our experiments.
From the experiments, we find that “Average” and LoRA-Hub perform even worse than the single task LoRA, demonstrating that combining different LoRA modules with static weights is not effective enough for complex generative tasks like solving mathematical problems or generating code segments. Specifically, LoRA-Hub only outperforms the single-task LoRA on the Zh code task.
Trained with the same few-shot training data as LoRA-Hub, LoRA-Flow outperforms the baselines
3Refer to Table 5 in Appendix for the best hyperparameters of each generation task.
across all six examined tasks. For instance, LoRAFlow achieves a performance improvement of 2.3 compared to LoRA-Hub (22.6 vs. 20.3) on code generation tasks. This confirms the necessity of employing dynamic weights for generative tasks.

heading: 4.3 Results on Larger Model
text: To confirm the effectiveness of LoRA-Flow with larger LLMs, we also conduct experiments on Llama-2-13b. Given that larger models require higher training costs, we only train three LoRA modules for the 13b model, namely Zh Chat, En Math, and En Code. The results of different fusion approaches on the 13b model are shown in Table 2. On the larger model, all the methods involved achieve better results than those on the 7b model. Compared to LoRA-Hub, LoRA-Flow exhibits superior performances on both the math and the code tasks. These results also demonstrate that LoRA-Flow is compatible with models of various sizes.

heading: 5 Analysis
text: To gain deeper insights into the behavior of LoRAFlow, we conduct a comprehensive series of analyses from various perspectives. We first estimate the average fusions at different model layers in Section 5.1, and then investigate how the fusion
weights change across time steps in Section 5.2. Finally, we provide a specific case in Section 5.3.

heading: 5.1 Fusion Weights across Different Layers
text: Figure 3 shows the average fusion weights at different layers for the two involved LoRA modules (i.e., Zh Chat and En Math) on the Chinese math task. For both the Zh Chat and En Math LoRA, the weights at various layers exhibit notable variations, suggesting distinct fusion schemes across different model layers. A trend is observed where lower layers assign greater weights to the En Math LoRA, while higher layers allocate more weight to the Zh Chat LoRA. This phenomenon can be explained by the following reason: the bottom layers primarily utilize the math LoRA for reasoning, while the top layers rely more on the language LoRA for text generation in the target language. In contrast, approaches such as LoRA-Hub, which combines LoRA modules with fixed weights, regard the capabilities of each transformer layer as equivalent, overlooking the discrepancies in their abilities. This constrains the potential of LoRA
fusion approaches in generative tasks.

heading: 5.2 Fusion Weights at Different Time Steps
text: We also analyze the average fusion weights at different time steps during the decoding process. We split all the tokens generated by the model into 10 bins according to their relative positions. Figure 4 presents the results, where “10%” denotes the initial 10% tokens on the left, and “20%” represents the subsequent 10%-20% tokens. This figure illustrates the average trend of the fusion dynamics. We observe that the weight of the En Math LoRA is increasing, while the weight of the Zh Chat LoRA is decreasing. The results underscore the necessity for dynamic fusion weights in generative tasks, as they show that different time steps require varying fusion weights, reconfirming our intuition. Since the average trend is estimated based on the entire test set, we will present a specific case in the following section.

heading: 5.3 Detailed Analysis on Fusion Weights
text: To better investigate the fusion procedure of LoRAFlow, we provide an example in Figure 5, which consists of the specific tokens generated by the model and the corresponding fusion weights for the two involved LoRA modules (i.e., Zh Chat and En Math). As discussed in Section 5.2, the fusion weights exhibit significant variation across different tokens. Notably, due to the incorporation of a bias vector in the fusion gate, as depicted in Eq (5), it is possible for certain tokens to have negative fusion weights. We draw inspiration from the work of Zhang et al. (2023), who observed that negative fusion weights can yield specific effects.
On average, the fusion gate assigns higher weights to the En math LoRA. Additionally, we
observe certain troughs in the fusion weights for the Zh Chat LoRA. To better understand this phenomenon, we identify the tokens corresponding to these troughs. As shown in Figure 5, we use the same color to mark the weight throughs and the corresponding text segments. Surprisingly, we observed that when the fusion weight for the Zh chat LoRA decreases, the text segments mainly contain numbers and mathematical calculations (e.g., “1800 / 6 = 300”), which align more closely with the math capability learned by the En math LoRA.
By examining the case, we observe a strong correlation between the fusion weights with the generated content. When the content involves mathematical reasoning, LoRA-Flow will improve the fusion weight of the En math LoRA. The significant fluctuation in weights suggests that relying on fixed weights throughout the decoding process is impractical, underscoring the necessity of dynamic fusion weights.

heading: 6 Discussion

heading: 6.1 Ablation Study
text: To further investigate the granularity of the gates in LoRA-Flow, we conduct an ablation study to compare the performance of different types of gates: step-level, layer-level, and module-level gates. As explained in Section 3.1, the step-level gate esti-
mates the fusion weight for each step, which is shared by the entire model. Layer-level gates represent the default setting in LoRA-Flow, computing the fusion weight at each layer. Module-level gates provide a specific fusion weight for each module (e.g., the query projection module in the last layer), offering greater flexibility than layer-level gates but introducing more trainable parameters.
Table 3 presents the results of varying gate levels. On average, layer-level fusion gates achieve the highest scores, surpassing both step-level and module-level gates. On the Zh math task, the confidence intervals for the step-, layer-, and modulelevel fusion models are (29.21, 30.78), (32.85, 33.55), and (29.61, 31.18), respectively. Nevertheless, the other two fusion gate types also outper-
form LoRA-Hub. Specifically, the step-level fusion gate achieves an average score of 35.5, while that of LoRA-Hub is 28.7. These results suggest that while utilizing shared fusion weights across different model layers, employing dynamic fusion methods at various time steps yields superior results compared to using task-level static weights. Since different model layers may have different capabilities, using layer-level fusion gates can further improve the performance.

heading: 6.2 LoRA Fusion vs. Few-shot Fine-Tuning
text: In scenarios with limited data, combining existing LoRA modules can effectively leverage the knowledge acquired in other tasks. In this section, we compare LoRA-Flow with other few-shot fine-tuning baselines: (1) FT new LoRA: using the same few-shot training examples to learn a new LoRA for the target task; (2) FT lang LoRA: finetuning the language LoRA on the target task; and (3) FT task LoRA: fine-tuning the task LoRA on the target task. The results are shown in Table 4. Our method, which dynamically integrates the language and task LoRA modules, surpasses all three few-shot fine-tuning baselines, indicating the effectiveness of utilizing existing LoRA modules in few-shot scenarios.

heading: 6.3 Generalization across Different Tasks
text: Similar to LoRA-Hub (Huang et al., 2024), we learn the fusion module for each task with fewshot training data. In this study, we also explore the generalizability of fusion gates across various tasks. For example, can the gate learned for the Zh code task, which is trained to combine the Zh chat LoRA and the En code LoRA, effectively combine the Ru chat LoRA and the En math LoRA?
Figure 6 shows the results, depicting the performance difference between merging two LoRA modules and employing solely the task LoRA, which is

heading: 6.4 -1.2 -1.2 -1.8 -1.3 -6.0
text: a strong baseline. Specifically, the initial row indicates employing the gate trained with Zh math data for all six generative tasks. Remarkably, numerous values appear to be positive, suggesting that while zero-shot generalization is not a specific consideration during method design, it still demonstrates some degree of generalization capability. Take the third row as an example, training a gate on the Es math task results in a performance enhancement of 0.8 on the training task. Impressively, this Es math gate also leads to improved performance on three other tasks (i.e., Ru math, Zh code, and Es code). Furthermore, the enhancement observed in the Es code task surpasses that achieved in the Es math task. We leave it to future work to further improve the zero-shot generalization capabilities of LoRA fusion methods.

heading: 6.4 Few-Shot Sample Size Selection
text: We also investigate the impact of the number of fewshot learning examples (i.e., N ). On the Zh math task, when trained with 50, 100, and 200 examples, LoRA-Flow achieves scores of 26.8, 31.6, and 33.2, respectively. This trend indicates that using more data can yield better fusion results.

heading: 7 Related Work
text: Module Composition Exploring the reuse of existing modules for new tasks is an appealing research direction. Pfeiffer et al. (2021) combines
adapters (Houlsby et al., 2019) that have been finetuned on various downstream tasks through an additional attention layer. Zhang et al. (2023) defines some specific operations, such as addition and subtraction, to combine existing LoRA modules. The combination weights are tuned on a validation set. Huang et al. (2024) further improves the LoRA combination by automatically optimizing the fusion weights in a few-shot manner. Chronopoulou et al. (2023) combines LoRA modules trained on the summarization task and multilingual unlabeled data to perform multilingual summarization, where the fusion weight is also a hyperparameter that should be tuned. Most previous LoRA fusion methods employ task-level static weights, while our proposed LoRA-Flow can dynamically combine different types of LoRA according to the current context.
Mixture-of-LoRA Some recent studies propose to improve the performance of LoRA with the mixture-of-LoRA (MoLoRA) architecture (Zadouri et al., 2023; Dou et al., 2023; Wang et al., 2022), which is similar to mixture-of-expert models. The primary objective of these efforts is to overcome the limitations of LoRA in certain downstream tasks, as the expressive capacity of a single LoRA may be restricted by its intermediate rank. The major difference between MoLoRA and our work is that MoLoRA mainly aims to train a better plugin for the backbone model, whereas we aim to leverage pre-trained LoRA modules for new tasks without training a new LoRA. These two approaches are complementary. Our method can also be extended to incorporate existing MoLoRAs or other advanced types of LoRA modules for unseen tasks.

heading: 8 Conclusion
text: In this work, we propose LoRA-Flow, a dynamic combination method for LoRA. By assigning dynamic weights to different LoRA modules based on the current context, LoRA-Flow can outperform representative baselines with static task-level fusion weights on six complex generative tasks.

heading: 9 Limitation
text: In this study, LoRA-Flow outperforms the fixedweight linear combination method across tasks in various languages. However, our computing resources are limited, constraining us to models no larger than 13 billion parameters. We only conduct
experiments combining two adapters, though our method is readily adaptable to incorporate additional adapters by simply integrating more into the model and adjusting the gate’s shape accordingly. Nonetheless, the versatility of our method persists, and we leave the exploration of larger models and combining more adapters to future work.

heading: Acknowledgements
text: This project was supported by the National Natural Science Foundation of China (No. 62106138, No. 62236004, No. 62236011), and a grant from the Guoqiang Institute, Tsinghua University. We thank all the reviewers for their constructive and insightful suggestions.

heading: A Best Hyperparameters
text: The best hyperparameters for LoRA fusion are listed in the Table 5.

heading: B Translation Prompt
text: Figure 8 and Figure 9 show the prompts we used to translate the Metameth data from English to Chinese using GPT-3.5. The {system prompt} and {user prompt} are set by default. The first prompt is intended to translate the problem identified by the {instruction} of a sample. The second prompt is intended to translate the solution identified by the {answer} to a sample.

heading: C Human Annotation Details
text: In the process of reviewing and polishing multilingual data, we employed two annotators for each language. For the construction of the English version of the code validation set, we hired two annotators with over two years of experience in NLP. We contacted them through social media and compensated them at market-standard rates. It has been communicated to all participating human annotators that the gathered data shall only be used for academic research purposes, with an assurance that their personal information will be kept confidential and not disclosed.
C.1 Details of Translated Data Verification We employed college students majoring in the relevant languages to check and optimize the translated data. The primary areas of review and optimization include:
• Correctness of the mathematical problemsolving process.
• Correctness of the mathematical problem format: whether the expression of formulas and the organization of structures adhere to the original metamath template, such as answers following "####" and ending with "the answer is:" in various languages, etc.
• Whether the code part of the code generation problems remains consistent with that in the original data.
• Correctness of the code data format: whether the overall structure remains unchanged, and whether essential parts such as problem description, problem analysis, and code comments are translated into the corresponding language.
• Overall fluency of the translated sentence.
Each piece of data is sequentially reviewed by two independent annotators.

heading: D Translated Case
text: To better understand the case study presented in Figure 5, we translate the Chinese question and answer into English, the results are shown in Figure 7.

heading: Future Work
text: Nonetheless, the versatility of our method persists, and we leave the exploration of larger models and combining more adapters to future work