Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation

Authors: Junhong Wu, Yang Zhao, Yangyifan Xu, Bing Liu, Chengqing Zong
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Continual Instruction Tuning, Machine Translation

Abstract:
Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks but still encounter difficulties in machine translation. Traditional methods to improve translation have typically involved fine-tuning LLMs using parallel corpora. However, vanilla fine-tuning often leads to catastrophic forgetting of the instruction-following capabilities and alignment with human preferences, compromising their broad general abilities and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make existing continual instruction tuning methods ineffective. To overcome this issue, we propose a novel approach called $\textbf{RaDis}$ ($\textbf{Ra}$tionale $\textbf{Dis}$tillation). RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. These rationales $\textit{connect prior knowledge with new tasks}$, acting as $\textit{self-distillation targets}$ to regulate the training process. By jointly training on reference translations and self-generated rationales, the model can learn new translation skills while preserving its general abilities. Extensive experiments demonstrate that our method enhances machine translation performance while maintaining the broader capabilities of LLMs across other tasks. This work presents a pathway for creating more versatile LLMs that excel in specialized tasks without compromising generality or safety and provides a fresh angle for utilizing rationales in the CL field.

Decision: reject
Reviewer scores: 5, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730056953745

thread_type:
official_review

<thread_object>
creator: reviewer_cw7y

<content>
summary:
This paper introduces Rationale Distillation (RaDis), a new approach for fine-tuning Large Language Models (LLMs) on machine translation (MT) tasks. Unlike traditional fine-tuning methods that rely solely on parallel corpora, RaDis uses LLM-generated rationales to enhance MT performance while maintaining the LLM’s general capabilities, such as instruction-following and reasoning.

strengths:
RaDis is both straightforward and effective, leveraging the language model itself to generate explanations (rationales) for fine-tuning. Experimental results demonstrate that this method preserves the model’s general abilities while enhancing MT performance, although its MT performance remains slightly below established baselines like ALMA.

weaknesses:
Firstly, the proposed method lacks novelty from my perspective, as leveraging LLMs to generate data from their own distribution for continual fine-tuning has been extensively explored in prior research (e.g., various PPO/DPO variants along with works on chain-of-thoughts that prompt LLM to generate reasoning trace, etc.,). Moreover, based on my experiences with machine translation (MT) and reinforcement learning with human feedback (RLHF), the generative quality of a 7B model is often unreliable, typically requiring filtering processes (i.e., from a reward model). Incorporating all rationales generated by the LLM without any scoring, filtering, or grounding is likely to reinforce the model's own hallucinations. Even if this approach preserves the model’s instruction-following abilities, the practical utility of such fine-tuning remains questionable.

Secondly, the rationales derived from a translation dataset may lack diversity. A comparison or discussion with existing work, such as TOWER [1], is necessary. TOWER achieves high MT performance by utilizing parallel corpus and diverse instruction-following datasets during fine-tuning, which appears more effective on benchmarks (and the idea is more straightforward than rationales generation). Therefore, I believe another baseline that simply uses parallel data for fine-tuning + diverse instruction-following dataset for preserving the model's pretrained ability is needed.

[1] Alves et al., (2024). TOWER: An Open Multilingual Large Language Model for Translation-Related Tasks

questions:
1. some comparison and discussion on previous work like TOWER will be useful.
2. I am curious if there is any analysis of the generated rationales’ quality. I see in Table 4 there are some categorization but are all the generated responses relevant? I doubt that a 7B model can give very high-quality rationales for all translation pairs in the training data.
3. I like the analysis in section 5.2 and it actually resonates with my points in weakness. Fine-tuning model on its self-generated response is most helpful to preserve its instruction-following ability. However, I believe simply using all self-generated rationales is also limited. I am wondering if any on-policy-based mechanism for rationales generation is tried.
4. Overall, I find the experiments comprehensive but the improvement is not very surprising. The experiment also lacks an obvious baseline: using a combination of parallel corpus for MT and instruction-following dataset for preserving general ability (like the approach used in TOWER)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730119872676

thread_type:
official_review

<thread_object>
creator: reviewer_m6wr

<content>
summary:
This paper proposes a novel method, RaDis, which uses self-generated rationales for sequence-level distillation. It can preserve the model's capabilities in general domains and safety while fine-tuning downstream tasks, addressing the problem of catastrophic forgetting. The paper conducts machine translation experiments on two 7B-sized models to illustrate the effectiveness of the method, and the analysis section provides the detailed observation of the experimental results.

strengths:
1. This paper proposes a novel method, RaDis, which has a clear and straightforward motivation, and the writing is easy to follow.
2. The experimental results demonstrate that this method is effective and significantly surpasses the related baselines.

weaknesses:
1. This method effectively maintains the generalization ability and safety of the models during downstream task fine-tuning. However, most translation performance degrades compared to the original fine-tuning baseline, which will limit its applicability in real-world scenarios. Additionally, the comparison of translation performance between the backbone and RaDis in lines 81-83 is unfair and the comparison with the vanilla fine-tuned model is more reasonable and informative.
2. Considering the baseline distillation methods (SeqKD and SDFT) largely relies on the generation quality of the teacher model, it is necessary to conduct the experiments with teachers with better capacity, especially for SeqKD, which is rarely used in self-distillation manner.
3. In the analysis section, the explanations on the performance of RaDis with three different teachers needs a more detailed analysis, similar to that in Table 4. Particularly, it is necessary to clarify why the machine translation performance and the general performance yield contrasting conclusions (e.g., the self-generated setting achieve best general performance and the worse translation performance among the three settings).
4. Typo: line 507, “SDFT greatly enhances” -> “RaDis greatly enhances”

questions:
1. In the context of machine translation, the emphasis on faithfulness over diversity suggests that paraphrasing references may not efficently enhance translation performance, which aligns with the primary experimental results presented in Tables 1 and 2. Consequently, to facilitate a more equitable comparison between RaDis and SDFT, as well as to explore the potential of RaDis in other tasks, it is essential to evaluate their corresponding performance on logical reasoning tasks, including OpenFunctions, GSM8K, and HumanEval.
2. See the weeknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581094785

thread_type:
official_review

<thread_object>
creator: reviewer_jdnz

<content>
summary:
The paper introduces RaDis (Rationale Distillation) -- a self-distillation technique to help instruction-tuned language models learn new tasks without losing their general capabilities or compromising safety alignment. The method first generates rationales - explanations for the instruction-response pairs of the target task using the same language model and these rationales serve as replay data that help retain the original capabilities by fairly capturing the data distribution. The generated rationales and instruction-response pairs are then used for subsequent training of the same language model on the new task. The authors conduct experiments for the task of machine translation and demonstrate that it improves the translation proficiency of models while preserving its overall performance on other tasks.

strengths:
1. The authors propose a straightforward method for LLMs to learn the translation task (in this study) without losing prior abilities by combining standard fine-tuning on reference translations with self-distillation on generated rationales.

2. The experimental results demonstrate translation improvements across 4 language pairs on 2 different models.

3. The authors study various aspects of the proposed method to understand the influence of these aspects on downstream and overall performance.

weaknesses:
1. The paper only focuses on improving translation proficiency in LLMs without losing existing capabilities. However, I think the proposed method is more general and can work for other tasks. It should also be experimented on other tasks (e.g. summarization, open-ended QA, MQM annotation, etc) for a more comprehensive study.

2. In section 5.2, the authors compare the Mistral 7B model (student) with rationales generated from different models. However, this comparison may not be fair and it might be more appropriate to compare the rationale generation from a bigger model within the same family. For instance, the Llama 3 7B model (student) with rationales generated from Llama 3 70B. Could you clarify the reasons for not using models from the same families (e.g. Llama 3)?

questions:
The proposed method generates a rationale for each sample in the training set. Did you experiment with ablating whether you need a rationale for each sample to be included in the training set? How well does the proposed method perform when rationales aren’t available for every sample in the training set? What is the tradeoff between having rationales for all samples versus just a subset of *k* samples (e.g., 10%, 20%, 50% and 100% samples with rationales)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798788181

thread_type:
official_review

<thread_object>
creator: reviewer_puwz

<content>
summary:
The paper proposed rationale distillation, with a purpose on improving LLM-based models translation performance while keeping the general instruction-following abilities. The method first generates the explanation of the translation and use the synthetic data for fine-tuning along with the translation pairs.

strengths:
The paper successfully did what they proposed, improving the translation performance with the proposed method can significantly reduce the damage in general instruction-following tasks.

weaknesses:
1. The paper's goal is good —it aims to generalize translation models based on LLMs to other domains such as conversational tasks. However, the attempt to balance both translation quality and generalization does not yield a model that is truly useful, placing the paper in an awkward position. For example, RaDis performs substantially worse than its backbone model on conversations and instruction following, and its translation performance is inferior to that of translation-specific models. While I acknowledge that comparing RaDis with ALMA in terms of translation is not entirely fair due to differences in models and training data, this raises another issue: the model's performance is highly dependent on the backbone model. Consequently, RaDis cannot be effectively applied to low-resource languages like Icelandic, which were excluded from the training data. Achieving effective machine translation in LLMs requires substantial effort in multilingual pretraining and alignment; without this, English-centric LLM translation performance is less interesting. Despite the paper's claims of keeping both translation high performance and retaining generality, the translation quality is not top due to the high dependency of the backbone model, and the performance on other tasks is also worse.

2. Inference speed: Regarding inference speed, if the model is trained to output both a reference and an explanation, does this mean the generated output always includes both, requiring the user to extract the translation from the combined output? If so, this could significantly reduce inference speed. It would be beneficial to see a comparison of inference speeds and an analysis of how many additional tokens the model generates compared to translation-specific LLMs.

3. Missing baselines: A fundamental baseline would involve training on a combined dataset of translation parallel data and instruction-following data. Using synthesized data may be unnecessary given the abundance of existing data that can be combined with parallel data. However, I did not find results corresponding to this baseline in the paper. Including such comparisons would strengthen the evaluation and provide a clearer understanding of the model's performance relative to more straightforward approaches.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732001368504

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1: On Request for Additional Comparisons to TOWER and Fine-tuning with Instruction Data

comment:
We want to thank the reviewers for their thoughtful comments. We are diligently working to provide comprehensive responses that address each concern, and we will continue to present concrete experimental results throughout the discussion period.

In the following, we address specific points related to the baselines the reviewers have requested comparisons to—namely, multi-task training with additional instruction data and the Tower models.

First, we would like to clarify that RaDis is a **Continual Instruction Tuning (CIT)** method. In a **Continual Learning (CL)** setting, learning is done incrementally. When learning a new task, only the data from the task is available, and the model is trained to adapt to this task without access to the data of previous tasks. On the other hand, fine-tuning with additional instruction data corresponds to a **multi-task learning** approach, which learns multiple tasks simultaneously or jointly. It is regarded as the upper bound for CL methods. Therefore, the requested comparisons are actually between **two distinct paradigms**: a multi-task learning one and a continual learning one. Besides, we would also like to emphasize that both ParroT and BayLing have included instruction data (Alpaca) in model training. Therefore, models built in the multi-task training paradigm are already included in our original experiments.

However, we are glad to report that our RaDis outperforms the multi-task fine-tuning baseline in terms of preserving the general abilities of the LLMs, and performs competitively in translation.

Table g1-1: Comparison to multitask training. The best result in each column is marked in **bold**.
|                       | X-EN      | EN-X      | Alpaca Eval | Alpaca Eval 2 | AdvBench  | GSM8K     |
|-----------------------|-----------|-----------|-------------|---------------|-----------|-----------|
| Llama-2-Chat          | 79.01     | 73.39     | 71.40       | 9.66          | 100       | 21.83     |
| +Multitask            | **81.75** | 82.59     | 44.55       | 3.98          | 98.65     | 11.98     |
| +RaDis                | 81.22     | **82.61** | **67.94**   | **7.47**      | **100**   | **19.48** |
| Mistral-v0.2-Instruct | 80.84     | 67.79     | 84.91       | 15.09         | 68.46     | 41.62     |
| +Multitask            | **82.09**    | **84.44** | 49.46       | 5.45          | **63.85** | 22.97     |
| +RaDis                | 81.94 | 84.39     | **80.34**   | **11.05**     | 62.12     | **41.70** |

`Table g1-1` presents a comparison with the multi-task training baseline (+Multitask). Specifically, we utilize two widely adopted instruction datasets, Alpaca [1] and Dolly [2], and add them into fine-tuning. As shown in the table, our RaDis consistently outperforms the multi-task training baseline in preserving the general abilities of the LLM, while achieving comparable performance on translation tasks.

This result aligns with our claim in Lines 109-113: the quality of open-sourced instruction data is generally lower than the in-house data used for training large language models like Llama or Mistral. Consequently, incorporating these open-sourced datasets into training does not effectively mitigate the forgetting of the LLM's general abilities. Moreover, since the open-sourced data is out-of-distribution relative to the backbone LLM, fine-tuning with these data may even exacerbate the forgetting problem [3].


[1] Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca

[2] Conover, Mike, et al. "Free dolly: Introducing the world’s first truly open instruction-tuned llm." Company Blog of Databricks (2023).

[3] Yang, Zhaorui, et al. "Self-distillation bridges distribution gap in language model fine-tuning." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002109669

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2: On Request for Additional Experiments on Other Tasks

comment:
We appreciate Reviewer JdnZ and M6wr for recognizing the potential of RaDis, and we would be happy to explore its application across a broader range of tasks. Due to time constraints, we have currently only completed experiments on the code generation task. Specifically, we fine-tuned Mistral-v0.2 on Python code data from the Magicoder dataset and evaluated its performance using HumanEval and general ability benchmarks.

Table g2: Experiments on code generation. Pass@1 is reported on HumanEval.
|              | HumanEval | Alpaca Eval | Alpaca Eval 2 | AdvBench | GSM8K |
|--------------|-----------|-------------|---------------|----------|-------|
| Mistral-v0.2 | 36.59     | 84.91       | 15.09         | 68.46    | 41.62 |
| +Vanilla-FT  | 42.07     | 73.89       | 8.75          | 40       | 43.97 |
| +SDFT        | 41.24     | 78.58       | 10.46         | 48.08    | 40.86 |
| +RaDis       | 43.9      | 80.25       | 11.4          | 51.92    | 42.91 |

As shown in `Table g2`, RaDis outperforms Vanilla-FT and SDFT in code generation tasks, achieving higher Pass@1 on HumanEval and excelling in other benchmarks for general abilities. 

A key reason for this is that RaDis directly preserves the original references in the dataset, whereas SDFT paraphrases them. Intuitively, while paraphrasing helps bridge the distribution gap, it also reduces the amount of learnable knowledge. As a result, SDFT may struggle to outperform Vanilla-FT on certain tasks. In contrast, RaDis directly utilizes the original references, preserving all the knowledge embedded in the data.

Regarding performance on general tasks, RaDis still outperforms SDFT. We believe this can be attributed to the distribution gap. While SDFT claims to distill the dataset, it paraphrases the data. As a result, the model's responses are sampled from the paraphrased instruction's output distribution, which tends to be out-of-distribution relative to the original task instruction. In contrast, RaDis performs self-distillation using rationales, which are fully in-distribution. This enables RaDis to more effectively alleviate forgetting and better preserve general abilities.

These results suggest that RaDis generalizes well to a broader range of tasks, highlighting its potential as a robust, general-purpose continual instruction tuning method. We plan to investigate this potential in future works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536265409

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to AC and Reviewers

comment:
Dear AC and reviewers,

Thank you for your insightful suggestions. We truly appreciate the time and effort you’ve invested in reviewing our work. In response to your feedback, we have refined our experiments and incorporated additional analyses. **The revised version of the paper has been uploaded**, and we summarize the key modifications below:

1. We have introduced a **multi-task training** baseline, allowing for a more comprehensive comparison with this straightforward approach. (See General Response 1)
2. We discuss and compare our work with **TowerInstruct**, the state-of-the-art LLM-based MT model with general capabilities. (See General Response 1)
3. We conducted experiments on broader tasks to assess the **generalizability** of our proposed method. (See General Response 2)
4. We have corrected minor typos and clarified certain claims to improve overall clarity. (See General Response)

As the discussion phase is going to approach its conclusion, we would appreciate any feedback on whether our revisions address your concerns. We look forward to hearing from you.

Best regards,
The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732871623803

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 3: Continual Learning (CL) v.s. Multi-task Learning (MTL)

comment:
We hope to provide more information regarding the differences between CL and MTL, and the practical needs for CL in the era of LLMs. 

First, we would like to clearify that **continual learning** is different from **continual fine-tuning**. CL refers to the broader concept of learning over time, where a model is designed to learn from a stream of data in an ongoing fashion, without forgetting previously acquired knowledge. It  is recognized as a novel learning paradigm [1,2,3]. In contrast, continual fine-tuning represents fine-tuning a model with new task data, it does not involve strategies to alleviate forgetting and is considered a baseline approach in CL (Vanilla-FT).

> Differences between CL and MTL

While both CL and MTL aim to improve a model's ability to generalize, CL is focused on learning over time without forgetting past knowledge, whereas MTL focuses on learning multiple tasks concurrently. 

CL is essential for intelligence, as humans learn continuously in an ongoing manner. For instance, today we might encounter pigs and apples and learn to recognize them. Tomorrow, we might see sheep and learn to identify them as well. This incremental learning process is not suitable for MTL because when we first see pigs and apples, the sheep have not yet appeared. It would be inefficient or even impossible to revisit the images of pigs and apples and relearn them alongside sheep when they eventually appear, as MTL would require. 

> CL is widely needed in LLM applications.

A similar situation arises in the context of LLM applications. Consumers typically aim to adapt open-sourced LLMs for specific applications (eg. health care, education). Preserving the original knowledge in these LLMs is essential for achieving optimal performance. However, fine-tuning with domain- or task-specific data often leads to significant forgetting. In this situation, MTL is often infeasible, as we lack access to the data used to train the LLM. This real-world challenge is one of the key motivations behind our decision to work on this problem and it highlights the requirements for CL approaches in LLM applications [3].

[1] Liu, Bing. "Lifelong machine learning: a paradigm for continuous learning." Frontiers of Computer Science 11.3 (2017): 359-361.

[2] Wang, Liyuan, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[3] Shi, Haizhou, et al. "Continual learning of large language models: A comprehensive survey." arXiv preprint arXiv:2404.16789 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732871978012

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 4: The Novelty of Utilizing Rationales in CL

comment:
We hope to suggest that the use of rationales to alleviate catastrophic forgetting in RaDis represents a **novel approach** in the CL field. Traditional replay-based methods need to retain previous task examples and replay them during fine-tuning. In contrast, RaDis solely leverages **self-generated rationales**—the model's own explanations for new training data. These rationales go beyond raw outputs; they introduce a form of **model introspection**, where the model uses its internal knowledge to explain an external example.

In that case, these rationales serve as a **semantic scaffold** that connects learned knowledge with new tasks, building a reasoning framework that ties internal concepts together, which helps both **generalization** and **knowledge retention**. This mechanism mirrors **how humans often retain information**—not simply by memorizing facts (**replay example**), but by constructing a mental narrative (**rationale**) that explains how new knowledge fits within a broader context. By utilizing rationales, RaDis offers the potential for a more **human-like memory system**, representing a promising new direction for continual learning.

We sincerely hope the clarification above could help the reviewer to focus on the insightful ideas behind RaDis and reconsider their scores.
</content>
</thread_object>

</thread>


Metareview:
metareview:
### Summary
The paper introduces RaDis (Rationale Distillation)  to improve machine translation capabilities of large language models (LLMs) without compromising their general instruction-following abilities. The approach leverages self-generated rationales to mitigate catastrophic forgetting while enabling efficient training with minimal computational resources. Extensive experiments demonstrate competitive performance in translation while preserving general task abilities, highlighting its potential for versatile LLM applications.

### Strengths
The motivation of using self-generated rationales to balance specialized and general capabilities is good. Extensive empirical results are shown across multiple models and tasks, showcasing improved efficiency over traditional methods.

### Weaknesses:
Main weakness is that the paper is a bit limited in its scope and the proposed method itself doesn't bring new insights to the post-training research community of LLMs.