Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation

## Abstract

Abstract Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks but still encounter difficulties in machine translation. Traditional methods to improve translation have typically involved fine-tuning LLMs using parallel corpora. However, vanilla fine-tuning often leads to catastrophic forgetting of the instruction-following capabilities and alignment with human preferences, compromising their broad general abilities and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make existing continual instruction tuning methods ineffective. To overcome this issue, we propose a novel approach called RaDis ( Ra tionale Dis tillation). RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. These rationales encapsulate general knowledge and safety principles , acting as self-distillation targets to regulate the training process. By jointly training on both reference translations and self-generated rationales, the model can learn new translation skills while preserving its overall general abilities. Extensive experiments demonstrate that our method enhances machine translation performance while maintaining the broader capabilities of LLMs across other tasks. This work presents a pathway for creating more versatile LLMs that excel in specialized tasks without compromising generality and safety.

## 1 Introduction

Large Language Models (LLMs)
have demonstrated exceptional performance across diverse Natural Language Processing (NLP) tasks. However, in the realm of Machine Translation (MT), they still fall short compared to conventional supervised encoder-decoder models (Xu et al., ).
Recent studies have sought to enhance the translation performance of LLMs
through continual instruction-tuning with parallel corpora (Yang et al., ; Xu et al., ). While this approach effectively boosts translation performance, it often comes at the cost of the inherent general ability and safety alignment of LLMs. As illustrated in Figure 1, fine-tuning LLaMA-2-Chat and Mistral-v0.2-Instruct results in a significant decline in these models’ performance on MT-Bench (Zheng et al., ). This phenomenon is known as Catastrophic Forgetting (CF) (French, ), which remains the major obstacle to developing models that seamlessly integrate strong translation performance with broader general-purpose utility.

Various Continual Learning (CL) approaches have been proposed to mitigate CF (Chen & Liu, ).
In the context of LLMs,
replay-based methods (Scialom et al., ; Yin et al., ; Mok et al., ; He et al., ), which store small subsets of previous data for rehearsal, are often favored for their simplicity and effectiveness. However, a critical limitation of replay-based methods is their reliance on access to the original training data, which is frequently unavailable in real-world applications. This limitation greatly reduces their feasibility in scenarios like the one in this paper, where the goal is to boost translation skills while preserving the general abilities of open-sourced LLMs, which are gained from proprietary, in-house data. Some studies have incorporated open-source general instruction-following data as a substitute (Jiao et al., ; Zhang et al., ). However, the limited quality of these open-source datasets results in performance that significantly falls short of instruction-tuned LLMs.

Figure: Figure 1: Translation performance (COMET) and general conversational and instruction-following ability (MT-Bench). While both Fine-tuning (blue triangle) and RaDis (red star) greatly enhance the translation performance, RaDis helps preserve most of the models’ general ability.
Refer to caption: x1.png

To address this problem, this paper explores leveraging the strong generative ability of LLMs to *synthesize their own replay data*. However, given the vast task space of LLMs and the limited data we could use, generating high-quality synthesis data that encapsulates diverse general knowledge remains a non-trivial question.
We draw inspiration from an observation that instruction-tuned LLMs are capable of generating detailed rationales when tasked with translation requests (see Section 3.1 and Appendix C for more details).
Previous studies in the reasoning field suggest that rationales generated by LLMs contain valuable knowledge and can be used to distill reasoning abilities (Wadhwa et al., ; Xu et al., ). In line with these findings, we show that self-generated rationales *encapsulate the internal general knowledge* leveraged during translation and thus *can be utilized as replay data* to preserve general abilities in a self-distillation manner.

Thus, we propose a novel training method named RaDis (Rationale Distillation). It prompts the LLM to generate rationales for the reference translations in original training data, and then concatenates references and rationales, forming an enriched dataset for subsequent training. By incorporating both the rationales and the references into the training, RaDis bridges the general knowledge gap often found in vanilla fine-tuning and introduces a self-distillation loss on the rationale, thereby mitigating the forgetting issue.

Comprehensive experiments using two widely adopted LLMs, LLaMA-2-7B-Chat (Touvron et al., ) and Mistral-7B-Instruct-v0.2 (Jiang et al., ) validates the effectiveness of RaDis. As depicted in Figure 1, RaDis enhances translation performance by 5.6 and 8.9 COMET points, respectively, while preserving the models’ original performance on general ability benchmarks. Further analysis reveals that distilling self-generated rationales not only outperforms distilling from external rationales generated by a much stronger model but also avoids the conflict between learning new tasks and consolidating the original ability. Together, these findings offer additional insights into RaDis’ effectiveness and future study.

In summary, this work makes the following contributions:

- •
It addresses the problem of CL for instruction-tuned LLMs and discovers that LLMs can generate rationales encapsulating the internal general knowledge. Replaying these rationales effectively mitigates forgetting in a self-distillation manner.
- •
It proposes RaDis, a novel training approach that enhances LLMs’ translation proficiency while preserving their generality by distilling self-generated rationales. Comprehensive experiments on LLaMA-2-Chat and Mistral-Instruct-v0.2 demonstrate its effectiveness.
- •
Further analysis demonstrates that the self-generated nature of the rationales is the key to RaDis’ effectiveness. Additionally, RaDis avoids the conflict between learning new tasks and consolidating the original capabilities. These findings provide valuable insights for developing more flexible and powerful LLMs that excel in specialized tasks without compromising their generality or safety.

## 2 Related Works

### 2.1 Fine-tuning LLMs for MT

Previous studies have primarily fine-tuned LLMs using parallel corpora to enhance their translation proficiency. BigTrans (Yang et al., ) and ALMA (Xu et al., ) propose to first continual pre-train on monolingual data and then progresses to fine-tuning on parallel data.
Although these methods enhance the translation proficiency of LLMs, they often compromise the models’ general ability, turning them into specialized translation models. To address this issue, several studies have sought to preserve the general ability of LLMs while fine-tuning them for MT. For example, ParroT (Jiao et al., ) and BayLing (Zhang et al., ) incorporates general instruction-following data to maintain general capability. However, limited by the quality of the data, their performance in both translation and general abilities remains relatively low. In contrast to these efforts, our approach solely utilizes machine translation data and preserves the general ability by distillation of self-generated rationales.

### 2.2 Continual Instruction Tuning

Continual instruction tuning (CIT) seeks to mitigate CF during the instruction tuning of LLMs by employing CL approaches (Wu et al., ; Shi et al., ). Traditional CL methods are typically divided into replay-based, regularization-based, and architecture-based methods. However, in the context of LLMs, the vast parameter and task space reduces the feasibility of regularization-based and architecture-based methods (Wang et al., ). As a result, current research has predominantly relied on focused on replay-based techniques and their variants (Scialom et al., ; Yin et al., ; Mok et al., ; He et al., ; Wang et al., ) While these approaches are promising, they are subject to the reliance on access to the original training data. Consequently, they cannot be applied to mitigate the forgetting of instruction-tuned LLMs’ general abilities gained from in-house training data.
SDFT (Yang et al., ) is the first work designed for preserving the general instruction-following abilities of LLMs. It proposes to paraphrase the original train dataset with the LLM itself to bridge the distribution gap. However, the quality of the paraphrased data is limited by the capabilities of the prompt and the model itself, which may diminish the performance of the task to be learned. In contrast, RaDis argues the original data with self-generated rationales and avoids loss of performance on new tasks.

### 2.3 Distilling Rationales

Since the advent of LLMs, researchers have recognized their ability to generate rationales and have sought to distill knowledge from them. Initial studies have predominately focused on distilling the Chain of Thought (CoT) reasoning capabilities from intermediate rationales (Wang et al., ; Hsieh et al., ; Fu et al., ). These studies emphasize pre-rationalization, where the model first generates a rationale before predicting the answer based on that rationale. Here, the rationale reflects the reasoning path leading to the final answer, providing valuable insights for student models. Recent research has proposed post-rationalization, where the rationale is generated after the answer is predicted. In this context, the rationale serves as an explanation, supplementing the ground truth label. Wadhwa et al. () demonstrate that CoT-augmented distillation is more effective when rationales are provided after labels. Additionally, Xu et al. () suggests that post-rationalization mitigates rationale sensitivity issues and enhances focus on learning challenging samples. RationaleCL (Xiong et al., ) introduce rationales generated by GPT-3.5-turbo to distill relation extraction knowledge into T5 model in a continual learning setting. Unlike previous works that distill knowledge from LLMs to smaller student models, RaDis focuses on self-distillation to maintain general abilities and prevent forgetting.

## 3 Method

We begin by presenting a key observation: when tasked with translation requests, instruction-tuned LLMs can generate detailed rationales that encapsulate the internal general knowledge leveraged during translation (Section 3.1). Building on this insight, we introduce Rationale Distillation (RaDis), which leverages these self-generated rationales as replay data to help the model retain its broad general capabilities (Section 3.2). Finally, we demonstrate that the RaDis training objective can be decomposed into a conventional MT loss and a self-distillation loss on rationale tokens, which helps prevent excessive deviation of model parameters (Section 3.3).

### 3.1 Observation: Self-generated Rationales

Figure: Figure 2: An example of LLM’s response to translation instruction. In this case, the LLM provides a rationale with additional factual information about the term ‘Belt and Road’ after the translation result.
Refer to caption: x2.png

Instruction-tuned LLMs exhibit a strong ability to follow instructions and engage in conversational interactions, delivering helpful responses across a wide range of tasks. Unlike conventional models that simply output the answer, instruction-tuned LLMs are known to be able to generate rationales (Wei et al., ). As illustrated in Figure 2, when presented with a translation request, instruction-tuned LLMs not only generate the translation but also provide an accompanying rationale. For 139 out of 200 randomly sampled translation instructions, LLaMA-2-7B-Chat provided a translation along with a rationale. In line with findings in CoT-augmented distillation (Wadhwa et al., ; Xu et al., ), we found these rationales encompass a wealth of diverse information leveraged during translation, including word or phrase translations, sentence structure analysis, factual information about the sentence, explanations of the overall sentence meaning, and other details.

### 3.2 RaDis: distilling rationales to alleviate forgetting

Figure: Figure 3: Overview of the RaDis approach. Rationale Generating (Left): Given a translation instruction-response pair as an input, the LLM extends the response by generating a rationale. Fine-tuning with Rationale Distillation (Right): RaDis utilizes this self-generated rationale to enrich the original response and fine-tunes the LLM with the enriched data. The CLM loss computed on the rationale serves as a self-distillation regularization term, preventing excessive parameter divergence.
Refer to caption: x3.png

The forgetting issue can be attributed to an unsuitable training approach. In conventional fine-tuning, the supervision signal comes from the reference sentence solely, which contains knowledge specific to the translation task. Thus, the model parameter is biased to a translation-specific distribution. Previous studies have sought to address this issue by replay-based CL methods. However, due to the absence of original training data and the poor quality of open-sourced instruction-following data, their effectiveness is limited. To this end, we propose RaDis.

The core idea of RaDis is similar to *pseudo-replay*. In traditional CL, pseudo-replay methods employ an additional data generator to synthesize replay data (Shi et al., ). However, the superior generative abilities of LLMs now allow us to *leverage the model itself to synthesize this replay data*.
As depicted in Figure 3, RaDis starts from an instruction-tuned LLM as the backbone. It utilizes a prompt template $\mathcal{I}$ to format the translation sentence pair $(x,y)$ and sends them into the backbone LLM parameterized as $\theta$. As shown in Section 3.1, an instruction-tuned model has the inherent ability to continue generating a rationale using the translation instruction-response pair as the prefix.

$$ $\mathbf{r}\sim P(\mathbf{y},\mathbf{x},\mathcal{I};\theta)$ (1) $$

These rationales encapsulate the internal general knowledge leveraged during translation, making them an ideal substitute for replay data.
Specifically, the self-generated rationale $\mathbf{r}$ is concatenated with the translation sentence $\mathbf{y}$, creating an enriched response $\hat{\mathbf{y}}=\text{CONCAT}(\mathbf{y},\mathbf{r})$. The enriched instruction-response pair is subsequently used to train the backbone LLM using a standard causal language model (CLM) loss, defined as:

$$ $\mathcal{L}(\mathbf{x},\hat{\mathbf{y}};\theta)=-\log P(\hat{\mathbf{y}}| \mathbf{x},\mathcal{I};\theta)$ (2) $$

The enriched response now incorporates both the task-specific knowledge for translation and the diverse, original knowledge embedded within the self-generated rationale. As a result, fine-tuning the model with it can learn the translation task and consolidate the original general ability simultaneously.

### 3.3 Why RaDis Works: A Knowledge Distillation Perspective

In previous sections, we discovered that self-generated rationales are effective substitutes for replay data. However, since they are neither traditional replay data nor pseudo-replay data, a natural question arises: why do they work? Here, we demonstrate that RaDis can be understood as a form of knowledge distillation, a technique proven to mitigate forgetting. To explain this, we paraphrase Equation 2 as:

$$ $\displaystyle\mathcal{L}(\mathbf{x},\hat{\mathbf{y}};\theta)$ $\displaystyle=-\log P(\hat{\mathbf{y}}|\mathbf{x},\mathcal{I};\theta)$ (3) $\displaystyle=-\sum_{t=1}^{T+R}P(\hat{\mathbf{y}}_{t}|\hat{\mathbf{y}}_{<t}, \mathbf{x},\mathcal{I};\theta)$ $$

where $R$ is the length of the rationale $\mathbf{r}$. The properties of the CLM loss allow us to split and reassemble the loss across each token. By separating the loss of the reference translation from the self-generated rationale, we obtain:

$$ $\displaystyle\mathcal{L}(\mathbf{x},\hat{\mathbf{y}};\theta)=$ $\displaystyle-\sum_{t=1}^{T+R}P(\hat{\mathbf{y}}_{t}|\hat{\mathbf{y}}_{<t}, \mathbf{x},\mathcal{I};\theta)$ (4) $\displaystyle=$ $\displaystyle-\sum_{t=1}^{T}P(\mathbf{y}_{t}|\mathbf{y}_{<t},\mathbf{x}, \mathcal{I};\theta)-\sum_{t=T+1}^{T+R}P(\mathbf{r}_{t}|\mathbf{r}_{<t},\mathbf {y},\mathbf{x},\mathcal{I};\theta)$ $\displaystyle=$ $\displaystyle-\log P(\mathbf{y}|\mathbf{x},\mathcal{I};\theta)-\log P(\mathbf{ r}|\mathbf{y},\mathbf{x},\mathcal{I};\theta)$ $$

Here, the first term is a traditional MT loss, which trains the model to acquire new translation knowledge. The second term minimizes the negative log-likelihood of the self-generated rationale $\mathbf{r}$ given the conventional translation instruction-response pair. It can be interpreted as a sequence-level self-distillation loss on the rationale tokens, which serves as a regularizer to mitigate forgetting by preventing excessive deviation of model parameters.

## 4 Experiments

### 4.1 Datasets

The datasets and benchmarks used for fine-tuning and evaluation are listed below:

#### Translation.

For parallel training data, we adopt the human written data collected by ALMA (Xu et al., ), as it has been proven effective in enhancing the translation proficiency of LLMs. This data comprises human written test datasets from WMT’17 to WMT’20, plus the development and test sets from Flores-200 (Goyal et al., ). It covers 4 English-centric language pairs, considering both from English and to English directions: Czech (cs), Chinese (zh), German (de), and Russian (ru). The WMT’22 test dataset for the same 8 translation directions is used for testing. Translation performance is evaluated using the COMET metric (Unbabel/wmt22-comet-da) due to its better alignment with human evaluations (Rei et al., ).

#### Conversation and Instruction Following.

MT-Bench (Zheng et al., ) and AlpacaEval (Dubois et al., ) are employed to evaluate the conversation and instruction-following abilities of the models. MT-Bench consists of a set of challenging multi-turn questions across various categories, including math, coding, role-play, and writing. GPT-4 is utilized as the judge to assess the quality of the models’ responses, scoring them on a scale of 1 to 10, as outlined by Zheng et al. (). The AlpacaEval and AlpacaEval 2.0 leaderboard evaluates the models on 805 prompts from the AlpacaEval dataset and calculates the win rate against text-davinci-003 and GPT-4-1106. For this evaluation, we use the weighted_alpaca_eval_gpt4_turbo annotator as the judge.

#### Safety.

Safety is evaluated using harmful behavior datasets consisting of unsafe prompts. Following WalledEval (Gupta et al., ), we feed 520 unsafe prompts from AdvBench (Zou et al., ) into the LLMs and utilize LLaMA-3-Guard-8B (Dubey et al., ) to assess whether the responses are harmful. We report the safe rate, defined as the percentage of safe responses across all prompts.

#### Reasoning.

The reasoning ability is evaluated using GSM8K (Cobbe et al., ), which comprises 8.8k high-quality arithmetic word problems designed
at the grade school level, to assess the arithmetic reasoning abilities of LLMs. The evaluations are conducted using lm-evaluation-harness (Gao et al., ) and the exact match scores are reported.

### 4.2 Baselines

Our method is compared against two baseline categories. The first category includes representative continual instruction tuning (CIT) methods, that are compatible with our setting. In the second category, we consider prior LLM-based MT models, which focus on enhancing the translation proficiency of LLMs. It’s worth noting that the comparison with prior LLM-based MT models is not entirely fair due to discrepancies in training data and model architectures.

#### Continual Instruction Tuning (CIT) Baselines.

The following continual instruction tuning approaches are introduced as baseline methods: (1) Vanilla Fine-tuning, where the backbone LLMs are directly fine-tuned using translation data; (2) Seq-KD, which employs sequence-level knowledge distillation along with fine-tuning to alleviate forgetting; (3) SDFT (Yang et al., ), which leverages the backbone LLM to paraphrase the original training data and fine-tunes the model using the synthesized data.

#### Prior LLM-based MT models.

This category contains several notable works in the field of LLM-based machine translation. ParroT (Jiao et al., ), which fine-tune LLaMA-1 with a hybrid of translation and instruction-following data. BigTrans (Yang et al., ) enhances LLaMA-1 by equipping it with multilingual translation capabilities across more than 100 languages. BayLing (Zhang et al., ) fine-tunes LLaMA-1 using automatically generated interactive translation instructions. ALMA (Xu et al., ) first fine-tunes LLaMA-2 on monolingual data and subsequently uses high-quality parallel data for instruction tuning.

Please refer to Appendix A for further details of the baselines.

### 4.3 Training Details

In our experiments, we employ LLaMA-2-7B-Chat (Touvron et al., ) and Mistral-7B-Instruct-v0.2 (Jiang et al., ) as the backbone LLMs. Given the constraints of our computational resources, the Low-Rank Adaptation (LoRA) technique (Hu et al., ) is utilized in most of our experiments. Specifically, a LoRA adapter with a rank of $16$ is integrated into all the linear layers of the LLMs and exclusively trains the adapter. The LLMs are fine-tuned for three epochs on the translation dataset, with a learning rate of $1\times 10^{-4}$ and a cosine annealing schedule. The batch is set to 128 for stable training. Our implementation is based on LLaMA-Factory (Zheng et al., ). After the fine-tuning phase, the LoRA module is merged into the backbone LLM for testing. For further details, please refer to Appendix B.

### 4.4 Results

**Table 1: The overall translation performance (COMET score) in EN$\to$X. The delta performance compared to the backbone LLM is shown.**
| Models | Czech | German | Russian | Chinese | Avg. |
| --- | --- | --- | --- | --- | --- |
| Backbone LLM: LLaMA-2-7B-Chat |  |  |  |  |  |
| Backbone LLM | 70.14 | 75.10 | 75.76 | 72.57 | 73.39 |
| w/ Vanilla-FT | 81.80$\,\uparrow\,$11.66 | 82.81$\,\uparrow\,$7.710 | 84.67$\,\uparrow\,$8.910 | 81.96$\,\uparrow\,$9.390 | 82.81$\,\uparrow\,$9.420 |
| w/ Seq-KD | 70.17$\,\uparrow\,$0.030 | 74.40$\,\downarrow\,$0.71 | 75.62$\,\downarrow\,$0.140 | 72.93$\,\uparrow\,$0.360 | 73.28$\,\downarrow\,$0.110 |
| w/ SDFT | 68.59$\,\downarrow\,$1.55 | 75.21$\,\uparrow\,$0.110 | 79.67$\,\uparrow\,$3.910 | 78.45$\,\uparrow\,$5.880 | 75.48$\,\uparrow\,$2.090 |
| w/ RaDis (Ours) | 81.77$\,\uparrow\,$11.63 | 82.39$\,\uparrow\,$7.290 | 84.31$\,\uparrow\,$8.550 | 81.98$\,\uparrow\,$9.410 | 82.61$\,\uparrow\,$9.220 |
| Backbone LLM: Mistral-7B-Instruct-v0.2 |  |  |  |  |  |
| Backbone LLM | 67.39 | 67.87 | 64.56 | 71.32 | 67.79 |
| w/ Vanilla-FT | 84.33$\,\uparrow\,$16.94 | 83.04$\,\uparrow\,$15.17 | 86.23$\,\uparrow\,$21.67 | 83.63$\,\uparrow\,$12.31 | 84.31$\,\uparrow\,$16.52 |
| w/ Seq-KD | 74.30$\,\uparrow\,$6.910 | 73.69$\,\uparrow\,$5.820 | 73.10$\,\uparrow\,$8.540 | 78.28$\,\uparrow\,$6.960 | 74.84$\,\uparrow\,$7.060 |
| w/ SDFT | 51.90$\,\downarrow\,$15.49 | 53.32$\,\downarrow\,$14.55 | 47.07$\,\downarrow\,$17.49 | 56.38$\,\downarrow\,$14.94 | 52.17$\,\downarrow\,$15.62 |
| w/ RaDis (Ours) | 84.32$\,\uparrow\,$16.93 | 82.95$\,\uparrow\,$15.08 | 86.55$\,\uparrow\,$21.99 | 83.75$\,\uparrow\,$12.43 | 84.39$\,\uparrow\,$16.61 |
| Prior LLM-based MT Models |  |  |  |  |  |
| ParroT | - | 81.20 | - | 79.30 | - |
| BigTrans | 80.65 | 78.81 | 78.21 | 81.31 | 79.75 |
| BayLing-7B | 76.85 | 82.18 | 74.72 | 84.43 | 79.55 |
| ALMA-7B | 89.05 | 85.45 | 87.05 | 84.87 | 86.61 |

**Table 2: The overall translation performance (COMET score) in X$\to$EN.The delta performance compared to the backbone LLM is shown.**
| Models | Czech | German | Russian | Chinese | Avg. |
| --- | --- | --- | --- | --- | --- |
| Backbone LLM: LLaMA-2-7B-chat |  |  |  |  |  |
| Backbone LLM | 79.53 | 81.20 | 80.36 | 74.95 | 79.01 |
| w/ Vanilla-FT | 82.74$\,\uparrow\,$3.21 | 83.31$\,\uparrow\,$2.11 | 82.70$\,\uparrow\,$2.34 | 78.08$\,\uparrow\,$3.13 | 81.71$\,\uparrow\,$2.7 |
| w/ Seq-KD | 78.50$\,\downarrow\,$1.03 | 80.61$\,\downarrow\,$0.59 | 79.88$\,\downarrow\,$0.48 | 74.48$\,\downarrow\,$0.47 | 78.37$\,\downarrow\,$0.64 |
| w/ SDFT | 81.93$\,\uparrow\,$2.4 | 82.60$\,\uparrow\,$1.4 | 81.87$\,\uparrow\,$1.51 | 76.77$\,\uparrow\,$1.82 | 80.79$\,\uparrow\,$1.78 |
| w/ RaDis (Ours) | 81.75$\,\uparrow\,$2.22 | 83.07$\,\uparrow\,$1.87 | 82.22$\,\uparrow\,$1.86 | 77.83$\,\uparrow\,$2.88 | 81.22$\,\uparrow\,$2.21 |
| Backbone LLM: Mistral-7B-Instruct-v0.2 |  |  |  |  |  |
| Backbone LLM | 81.88 | 81.73 | 81.97 | 77.76 | 80.84 |
| w/ Vanilla-FT | 83.51$\,\uparrow\,$1.630 | 83.35$\,\uparrow\,$1.620 | 83.23$\,\uparrow\,$1.260 | 79.21$\,\uparrow\,$1.450 | 82.33$\,\uparrow\,$1.490 |
| w/ Seq-KD | 81.66$\,\downarrow\,$0.220 | 81.98$\,\uparrow\,$0.250 | 82.50$\,\uparrow\,$0.530 | 77.49$\,\downarrow\,$0.270 | 80.91$\,\uparrow\,$0.070 |
| w/ SDFT | 80.38$\,\downarrow\,$1.51 | 79.72$\,\downarrow\,$2.010 | 80.21$\,\downarrow\,$1.760 | 77.39$\,\downarrow\,$0.370 | 79.43$\,\downarrow\,$1.410 |
| w/ RaDis (Ours) | 82.41$\,\uparrow\,$0.530 | 82.86$\,\uparrow\,$1.130 | 83.32$\,\uparrow\,$1.350 | 79.17$\,\uparrow\,$1.410 | 81.94$\,\uparrow\,$1.11 |
| Prior LLM-based MT Models |  |  |  |  |  |
| ParroT | - | 82.40 | - | 75.20 | - |
| BigTrans | 81.19 | 80.68 | 77.80 | 74.26 | 78.48 |
| BayLing-7B | 82.03 | 83.19 | 82.48 | 77.48 | 81.30 |
| ALMA-7B | 85.93 | 83.95 | 84.84 | 79.78 | 83.63 |

**Table 3: The performance on instruction following, safety, and reasoning benchmarks. RP represents relative performance compared to the backbone LLM and is only calculated for CIT methods. The delta performance compared to vanilla fine-tuning (Vanilla-FT) is shown. The safety rate for BigTrans and ALMA is omitted, as these translation-specific models can not generate reasonable responses.**
| Models | Conversation and Instruction Following | Safety | Reasoning | RP[%] |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| MT-bench | AlpacaEval | AlpacaEval 2.0 | AdvBench | GSM8K |  |  |
| Backbone LLM: LLaMA-2-7B-chat |  |  |  |  |  |  |
| Backbone LLM | 6.51 | 71.40 | 9.66 | 100.00 | 21.83 | - |
| w/ Vanilla-FT | 1.5 | 2.18 | 0.71 | 37.88 | 4.32 | 18.22 |
| w/ Seq-KD | 6.59$\,\uparrow\,$5.09 | 67.48$\,\uparrow\,$65.30 | 8.33$\,\uparrow\,$7.62 | 100.00$\,\uparrow\,$62.12 | 19.48$\,\uparrow\,$15.16 | 94.24$\,\uparrow\,$76.02 |
| w/ SDFT | 5.66$\,\uparrow\,$4.16 | 67.55$\,\uparrow\,$65.37 | 7.09$\,\uparrow\,$6.38 | 98.08$\,\uparrow\,$60.20 | 20.02$\,\uparrow\,$15.70 | 88.95$\,\uparrow\,$70.72 |
| w/ RaDis | 6.56$\,\uparrow\,$5.06 | 67.94$\,\uparrow\,$65.76 | 7.47$\,\uparrow\,$6.76 | 100.00$\,\uparrow\,$62.12 | 19.48$\,\uparrow\,$15.16 | 92.50$\,\uparrow\,$74.27 |
| Backbone LLM: Mistral-7B-Instruct-v0.2 |  |  |  |  |  |  |
| Backbone LLM | 7.67 | 84.91 | 15.09 | 68.46 | 41.62 | - |
| w/ Vanilla-FT | 1.94 | 6.07 | 1.02 | 4.23 | 0.23 | 9.19 |
| w/ Seq-KD | 6.99$\,\uparrow\,$5.05 | 82.06$\,\uparrow\,$75.99 | 12.7$\,\uparrow\,$11.68 | 60.58$\,\uparrow\,$56.35 | 41.77$\,\uparrow\,$41.54 | 92.16$\,\uparrow\,$82.97 |
| w/ SDFT | 7.00$\,\uparrow\,$5.06 | 78.32$\,\uparrow\,$72.25 | 10.02$\,\uparrow\,$9.00 | 48.27$\,\uparrow\,$44.04 | 41.09$\,\uparrow\,$40.86 | 83.83$\,\uparrow\,$74.64 |
| w/ RaDis | 7.57$\,\uparrow\,$5.63 | 80.34$\,\uparrow\,$74.27 | 11.05$\,\uparrow\,$10.03 | 62.12$\,\uparrow\,$57.89 | 41.70$\,\uparrow\,$41.47 | 91.49$\,\uparrow\,$82.31 |
| Prior LLM-based MT models |  |  |  |  |  |  |
| ParroT | 4.58 | 28.06 | 2.82 | 26.35 | 4.09 | - |
| BigTrans | 1.73 | 0.42 | 0.22 | - | 0 | - |
| BayLing-7B | 4.51 | 51.29 | 4.43 | 84.62 | 5.38 | - |
| ALMA-7B | 2.80 | 1.08 | 0.17 | - | 0 | - |

The translation performance in EN$\to$X and X$\to$EN are shown in Table 1 and Table 2, respectively. The performance on general ability, including instruction following, safety, and reasoning, is shown in Table 3.

#### Fine-tuning is a double-edged sword.

In the EN$\to$X direction, Vanilla-FT significantly enhances translation performance compared to zero-shot results, achieving an average COMET score improvement of +16.52. In the X$\to$EN direction, the performance improvement is relatively small (+1.49 COMET). This is mainly because the backbone LLMs already have a strong ability to translate to English. However, this improvement in translation proficiency comes at the cost of a substantial decline in general capabilities, as reflected by the sharp performance drop in instruction-following, safety, and reasoning benchmarks.

#### RaDis balances translation proficiency and general abilities.

The translation results of SDFT fall below the zero-shot performance of backbone LLM when using Mistral-7B-Instruct-v0.2 as the backbone LLM. This under-performance stems from the fact that the prompt used for rewriting data is tailored for LLaMA-2 models, which does not generalize well to Mistral models. Additionally, SDFT shows weaker performance in EN$\to$X translations compared to X$\to$EN, indicating that the limited ability of the backbone LLM to translate into other languages diminishes the quality of the distilled dataset. Seq-KD preserves up to 94.24% of the overall general capabilities but brings almost no improvement in translation performance. In contrast, RaDis strikes a better balance between translation proficiency and general ability. It achieves a COMET score comparable to Vanilla-FT (81.58 vs. 82.02, 83.17 vs. 83.56) while preserving up to 92.50% of the general capabilities.

#### Compared with prior LLM-based MT models.

RaDis helps backbone models perform comparably with SOTA LLM-based MT models in translation performance. Our best model (Mistral-7B-Instruct-v0.2 w/ RaDis) surpasses BigTrans, BayLing, and ParroT by a considerable margin in average COMET scores, particularly in the EN$\to$X direction. Although ALMA achieves a higher COMET score than our best model, it benefits from continual pre-training on massive monolingual data, which is not implemented in our approach. Regarding general ability, models equipped with RaDis significantly outperform all prior studies. Translate-specific models, such as BigTrans and ALMA that are only fine-tuned with translation data, lack general ability. While ParroT and BayLing utilize Alpaca data in training, their general ability is limited by their backbone models and the quality of Alpaca data. In contrast, RaDis preserves the strong general ability of the backbone LLMs, achieving the highest performance.

## 5 Analysis

### 5.1 Do Rationales Contain General Knowledge?

**Table 4: Rationale categories and their main contents. Note that the percentages do not sum to 100%, as each rationale may include multiple categories of information.**
| Information Category | Contents | Occurrence [%] |
| --- | --- | --- |
| Word/phrase translation | Translations of individual words and phrases in the sentence. | 29.5 |
| Alternative translation | Multiple translation options provided for selection. | 19.5 |
| Helpful&Safety | Guidance on helpful and safe practices when translating sensitive sentences. | 17.5 |
| Semantic explanation | Clarification of the sentence’s meaning. | 15.5 |
| Back-translation | Back-translations of non-English results to verify their accuracy. | 12.5 |
| Factual supplement | Additional factual information regarding entities or events in the sentence. | 9.0 |
| Word/phrase explanation | Explanations for special words, idioms, or expressions. | 7.5 |
| Grammar | Information on grammar, such as part-of-speech or sentence structure. | 5.5 |

To investigate the content of self-generated rationales, we randomly sampled 25 instances from each translation direction, forming a set of 200 for analysis. The content of the rationales included diverse information, as expected. As shown in Table 4, the information can be broadly categorized into eight types, ranging from word alignments to factual knowledge. A rationale can contain multiple kinds of information at the same time, which constitutes an effective supplement to the reference sentence in terms of general knowledge. For examples of rationales, please refer to Appendix C.

### 5.2 Which is The Key: Rational Quality or Self-distillation Property?

**Table 5: The result of ablation study. The names in brackets are the models used to generate rationales. The best results in different RaDis variants are highlighted in bold.**
| Models | Machine Translation | Conversation and Instruction Following | Safety | Reasoning |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| X$\to$EN | EN$\to$X | MT-bench | AlpacaEval | AlpacaEval 2.0 | AdvBench | GSM8K |  |
| Mistral-7B-Instruct-v0.2 | 80.84 | 67.79 | 7.67 | 84.91 | 15.09 | 68.46 | 41.62 |
| w/ Vanilla-FT | 82.33 | 84.31 | 1.94 | 6.07 | 1.02 | 4.23 | 0.23 |
| w/ RaDis (Self-generated) | 81.94 | 84.39 | 7.57 | 80.34 | 11.05 | 62.12 | 41.70 |
| w/ RaDis (LLaMA-2-Chat-7B) | 82.33 | 84.51 | 6.89 | 63.85 | 6.28 | 98.46 | 35.03 |
| w/ RaDis (LLaMA-3-70B-Instruct) | 82.84 | 84.71 | 7.00 | 77.41 | 10.27 | 58.27 | 39.42 |

The effectiveness of RaDis can be explained in two ways: rationale quality (in terms of the knowledge they contain) and the self-distillation property. We conducted the following ablation experiment to analyze the impact of these two factors. Specifically, the self-generated rationales in RaDis were replaced by rationales generated by different models, namely LLaMA-2-7B-Chat and LLaMA-3-70B-Instruct (Dubey et al., ). Due to the difference in parameter size and fine-tuning data, these models can provide rationales with varying levels of quality, but all lack the self-distillation property. Therefore, it is possible to separate the rationale quality and the self-distillation property to analyze their contributions.

As shown in Table 5, RaDis consistently mitigates the forgetting of general capabilities, regardless of the type of rationales used. When comparing rationales generated by LLaMA-2-7B-Chat and LLaMA-3-70B-Instruct, the latter demonstrates superior performance in both MT and general tasks, except for the safety task. These results suggest that models can learn stronger general capabilities through higher-quality rationales, which leads to better overall performance. However, self-generated rationales demonstrate the strongest ability to retain general capabilities, even outperforming those generated by LLaMA-3-70B-Instruct, highlighting the importance of the self-distillation property.

### 5.3 RaDis Avoids the Conflict Between Learning and Mitigating CF

Figure: Figure 4: Overview of the gradient similarity between the regularization term and MT loss.
Refer to caption: x4.png

As demonstrated in Section 3.3, our proposed RaDis can be viewed as a specialized form of sequence-level distillation, where the rationale $\mathbf{r}$ serves as the distillation target. However, while both methods excel at preserving general capabilities, RaDis notably enhances translation proficiency, whereas Seq-KD does not. We posit that the difference arises from whether the regularization term conflicts with the MT learning process. In Seq-KD (Equation 5), the MT loss $-\log P(\mathbf{y}|\mathbf{x},\mathcal{I};\theta)$ and the regularization term $-\log P(\mathbf{y}^{\prime}|\mathbf{x},\mathcal{I};\theta)$ share the same input but have different outputs, which may lead to conflict in optimization. In contrast, with RaDis (Equation 4), the MT loss and the regularization term $-\log P(\mathbf{r}|\mathbf{y},\mathbf{x},\mathcal{I};\theta)$ are less likely to exhibit this issue.

We analyze gradient similarity to validate our assumption. Specifically, we sample 128 examples from each translation direction to create a validation set consisting of 1024 samples. Subsequently, the gradient features for the MT loss and the regularization term for both methods are extracted, following Xia et al. (). Finally, the cosine similarity of the gradient features is computed. As depicted in Figure 4, in 24 out of 32 layers, the gradient of the regularization term for Seq-KD exhibits negative similarity with the MT loss, indicating a significant conflict between these objectives. In contrast, in 25 out of 32 layers, the gradient of RaDis’ regularization term shows positive similarity with the MT loss, suggesting that RaDis can avoid the conflict between learning and mitigating forgetting.

## 6 Conclusions

In this paper, we conducted a systematic evaluation of prior LLM-based MT models. Our findings indicate that prior LLM-based MT models lack diverse general capabilities, degenerating into task-specific translation models. This degeneration is mainly caused by catastrophic forgetting while fine-tuning for translation tasks. Previous continual learning approaches can not preserve the general capabilities gained from in-house training data.
To address this issue, we propose a simple yet effective strategy, RaDis. RaDis prompts LLMs to generate rationales for the reference translation and utilizes these rationales as replay data to mitigate forgetting in a self-distillation manner. Extensive experiments show that SDFT greatly enhances the translation performance while preserving the models’ general ability. Further analysis indicates that self-generated rationales not only encapsulate diverse general knowledge but also avoid the conflict between learning and mitigating CF, making them an ideal choice for distillation. These insights can help future research on building LLMs capable of excelling in specialized tasks without compromising their generality or safety.

## 7 Limitations and Future Work

Our study is subject to certain limitations. Owing to constraints in computational resources, we adopt LoRA on models with 7B parameters. Further investigations involving larger models and full fine-tuning remain to be explored. Furthermore, we predominately focus on fine-tuning with machine translation data, applying RaDis to other NLP tasks will further support its effectiveness. This potential direction is what we intend to explore in future work.

## Reproducibility Statement

Codes and model weights will be made public after review to advocate future research. For synthesizing data, we provide several examples in Appendix C.
For evaluation, we primarily use greedy decoding to ensure reproducibility, except where specific generation configurations are mandated by certain benchmark tools.
Note that evaluations on instruction-following abilities (AlpacaEval and MT-Bench) rely on OpenAI’s API. The randomness of API responses may have little impact on the reproducibility of these benchmarks.

## Appendix A Baseline details

### A.1 Continual Instruction Tuning Baselines

#### Vanilla Fine-tuning.

This method directly fine-tunes the backbone LLMs with translation data, without incorporating any mechanism to address the forgetting issue.

#### Sequence-level Knowledge Distillation (Seq-KD).

This method first sends the formatted translation instructions to the backbone LLM and generates outputs $\mathbf{y}^{\prime}$. The model is trained with both golden references $\mathbf{y}$ and the self-generated outputs $\mathbf{y}^{\prime}$. The overall training objective is:

$$ $\displaystyle\mathcal{L}_{\text{Seq-KD}}=$ $\displaystyle\mathcal{L}(\mathbf{x},\mathbf{y};\theta)+\mathcal{L}(\mathbf{x}, \mathbf{y}^{\prime};\theta)$ (5) $\displaystyle=$ $\displaystyle-\log P(\mathbf{y}|\mathbf{x},\mathcal{I};\theta)-\log P(\mathbf{ y}^{\prime}|\mathbf{x},\mathcal{I};\theta)$ $$

#### Self-distillation Fine-tuning (SDFT). (Yang et al., 2024 )

This method first prompts the backbone LLM to paraphrase the original responses present in the task dataset, yielding a distilled dataset. Subsequently, the distilled dataset, which is used in subsequent fine-tuning, helps narrow the distribution gap between LLM and the original dataset. We adopt the general distillation template provided in their paper to paraphrase the dataset.

### A.2 Prior LLM-based MT Models

#### ParroT

(Jiao et al., ) reformulates translation data into the instruction-following style and introduces a ”Hint” field for incorporating extra requirements to regulate the translation process. It is also fine-tuned on the Alpaca dataset to enhance general ability.

#### BigTrans

(Yang et al., ) continual pre-train LLaMA-1-13B with Chinese monolingual data and an extensive parallel dataset encompassing 102 natural languages. They then apply multilingual translation instructions for fine-tuning.

#### BayLing

(Zhang et al., ) builds an interactive translation dataset and fine-tune LLaMA-1 models with both interactive translation and instruction-following datasets (Alpaca).

#### ALMA

(Xu et al., ) proposes a new training recipe for building LLM-based MT models, which begins with initial fine-tuning on monolingual data and then progresses to fine-tuning on a select set of high-quality parallel data.

In our experiments, we utilized the 7B models as baselines for ParroT, BayLing, and ALMA, to ensure a fair comparison of model size.

## Appendix B Training Details

### B.1 Prompt Templates

In all experiments, we use the original instruction format of the backbone LLM for both rationale generation and fine-tuning. For LLaMA
To avoid the overfit on specific instructions. 5 different translation instructions are generated and randomly applied to each sample. The instructions are shown in Figure 5.

Figure: Figure 5: The translation instructions.

### B.2 Hyperparameter

Due to the limitation of resources, our experiments utilize the Low-Rank Adaptation (LoRA) technique (Hu et al., ). Specifically, we integrate a LoRA adapter with a rank of $16$ into all the linear layers of the LLMs and exclusively train the adapter. The LLMs are fine-tuned over three epochs on the translation dataset, which equates to approximately 2,500 steps. We use a learning rate of $1\times 10^{-4}$ and a batch size of 128 to ensure stable training across most experiments. An exception is Seq-KD, which requires a batch size of 256 to maintain the same number of training steps. All experiments are performed on 4 NVIDIA A100 80GB GPUs.
For data synthesis, we employ vllm (Kwon et al., ) to facility fast data generation.
For evaluation, we primarily use greedy decoding to ensure reproducibility, except where specific generation configurations are mandated by certain benchmark tools.

## Appendix C Rationale Examples

Several examples of rationales generated by Mistral-7B-Instruct-v0.2 are provided in Figure 7,8,9,10,11,12,13.

Figure: Figure 6: Example of rationale containing word/phrase translation.

Figure: Figure 7: Example of rationale containing alternative translation.

Figure: Figure 8: Example of rationale containing helpful&safety information.

Figure: Figure 9: Example of rationale containing semantic explanation.

Figure: Figure 10: Example of rationale containing back-translation.

Figure: Figure 11: Example of rationale containing factual supplement.

Figure: Figure 12: Example of rationale containing word/phrase explanation.

Figure: Figure 13: Example of rationale containing grammar information.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: N/A
soundness: 3
strengths: The paper successfully did what they proposed, improving the translation performance with the proposed method can significantly reduce the damage in general instruction-following tasks.
summary: The paper proposed rationale distillation, with a purpose on improving LLM-based models translation performance while keeping the general instruction-following abilities. The method first generates the explanation of the translation and use the synthetic data for fine-tuning along with the translation pairs.
weaknesses: 1. The paper's goal is good —it aims to generalize translation models based on LLMs to other domains such as conversational tasks. However, the attempt to balance both translation quality and generalization does not yield a model that is truly useful, placing the paper in an awkward position. For example, RaDis performs substantially worse than its backbone model on conversations and instruction following, and its translation performance is inferior to that of translation-specific models. While I acknowledge that comparing RaDis with ALMA in terms of translation is not entirely fair due to differences in models and training data, this raises another issue: the model's performance is highly dependent on the backbone model. Consequently, RaDis cannot be effectively applied to low-resource languages like Icelandic, which were excluded from the training data. Achieving effective machine translation in LLMs requires substantial effort in multilingual pretraining and alignment; without this, English-centric LLM translation performance is less interesting. Despite the paper's claims of keeping both translation high performance and retaining generality, the translation quality is not top due to the high dependency of the backbone model, and the performance on other tasks is also worse.

2. Inference speed: Regarding inference speed, if the model is trained to output both a reference and an explanation, does this mean the generated output always includes both, requiring the user to extract the translation from the combined output? If so, this could significantly reduce inference speed. It would be beneficial to see a comparison of inference speeds and an analysis of how many additional tokens the model generates compared to translation-specific LLMs.

3. Missing baselines: A fundamental baseline would involve training on a combined dataset of translation parallel data and instruction-following data. Using synthesized data may be unnecessary given the abundance of existing data that can be combined with parallel data. However, I did not find results corresponding to this baseline in the paper. Including such comparisons would strengthen the evaluation and provide a clearer understanding of the model's performance relative to more straightforward approaches.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: The proposed method generates a rationale for each sample in the training set. Did you experiment with ablating whether you need a rationale for each sample to be included in the training set? How well does the proposed method perform when rationales aren’t available for every sample in the training set? What is the tradeoff between having rationales for all samples versus just a subset of *k* samples (e.g., 10%, 20%, 50% and 100% samples with rationales)?
soundness: 4
strengths: 1. The authors propose a straightforward method for LLMs to learn the translation task (in this study) without losing prior abilities by combining standard fine-tuning on reference translations with self-distillation on generated rationales.

2. The experimental results demonstrate translation improvements across 4 language pairs on 2 different models.

3. The authors study various aspects of the proposed method to understand the influence of these aspects on downstream and overall performance.
summary: The paper introduces RaDis (Rationale Distillation) -- a self-distillation technique to help instruction-tuned language models learn new tasks without losing their general capabilities or compromising safety alignment. The method first generates rationales - explanations for the instruction-response pairs of the target task using the same language model and these rationales serve as replay data that help retain the original capabilities by fairly capturing the data distribution. The generated rationales and instruction-response pairs are then used for subsequent training of the same language model on the new task. The authors conduct experiments for the task of machine translation and demonstrate that it improves the translation proficiency of models while preserving its overall performance on other tasks.
weaknesses: 1. The paper only focuses on improving translation proficiency in LLMs without losing existing capabilities. However, I think the proposed method is more general and can work for other tasks. It should also be experimented on other tasks (e.g. summarization, open-ended QA, MQM annotation, etc) for a more comprehensive study.

2. In section 5.2, the authors compare the Mistral 7B model (student) with rationales generated from different models. However, this comparison may not be fair and it might be more appropriate to compare the rationale generation from a bigger model within the same family. For instance, the Llama 3 7B model (student) with rationales generated from Llama 3 70B. Could you clarify the reasons for not using models from the same families (e.g. Llama 3)?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In the context of machine translation, the emphasis on faithfulness over diversity suggests that paraphrasing references may not efficently enhance translation performance, which aligns with the primary experimental results presented in Tables 1 and 2. Consequently, to facilitate a more equitable comparison between RaDis and SDFT, as well as to explore the potential of RaDis in other tasks, it is essential to evaluate their corresponding performance on logical reasoning tasks, including OpenFunctions, GSM8K, and HumanEval.
2. See the weeknesses.
soundness: 2
strengths: 1. This paper proposes a novel method, RaDis, which has a clear and straightforward motivation, and the writing is easy to follow.
2. The experimental results demonstrate that this method is effective and significantly surpasses the related baselines.
summary: This paper proposes a novel method, RaDis, which uses self-generated rationales for sequence-level distillation. It can preserve the model's capabilities in general domains and safety while fine-tuning downstream tasks, addressing the problem of catastrophic forgetting. The paper conducts machine translation experiments on two 7B-sized models to illustrate the effectiveness of the method, and the analysis section provides the detailed observation of the experimental results.
weaknesses: 1. This method effectively maintains the generalization ability and safety of the models during downstream task fine-tuning. However, most translation performance degrades compared to the original fine-tuning baseline, which will limit its applicability in real-world scenarios. Additionally, the comparison of translation performance between the backbone and RaDis in lines 81-83 is unfair and the comparison with the vanilla fine-tuned model is more reasonable and informative.
2. Considering the baseline distillation methods (SeqKD and SDFT) largely relies on the generation quality of the teacher model, it is necessary to conduct the experiments with teachers with better capacity, especially for SeqKD, which is rarely used in self-distillation manner.
3. In the analysis section, the explanations on the performance of RaDis with three different teachers needs a more detailed analysis, similar to that in Table 4. Particularly, it is necessary to clarify why the machine translation performance and the general performance yield contrasting conclusions (e.g., the self-generated setting achieve best general performance and the worse translation performance among the three settings).
4. Typo: line 507, “SDFT greatly enhances” -> “RaDis greatly enhances”

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. some comparison and discussion on previous work like TOWER will be useful.
2. I am curious if there is any analysis of the generated rationales’ quality. I see in Table 4 there are some categorization but are all the generated responses relevant? I doubt that a 7B model can give very high-quality rationales for all translation pairs in the training data.
3. I like the analysis in section 5.2 and it actually resonates with my points in weakness. Fine-tuning model on its self-generated response is most helpful to preserve its instruction-following ability. However, I believe simply using all self-generated rationales is also limited. I am wondering if any on-policy-based mechanism for rationales generation is tried.
4. Overall, I find the experiments comprehensive but the improvement is not very surprising. The experiment also lacks an obvious baseline: using a combination of parallel corpus for MT and instruction-following dataset for preserving general ability (like the approach used in TOWER)
soundness: 2
strengths: RaDis is both straightforward and effective, leveraging the language model itself to generate explanations (rationales) for fine-tuning. Experimental results demonstrate that this method preserves the model’s general abilities while enhancing MT performance, although its MT performance remains slightly below established baselines like ALMA.
summary: This paper introduces Rationale Distillation (RaDis), a new approach for fine-tuning Large Language Models (LLMs) on machine translation (MT) tasks. Unlike traditional fine-tuning methods that rely solely on parallel corpora, RaDis uses LLM-generated rationales to enhance MT performance while maintaining the LLM’s general capabilities, such as instruction-following and reasoning.
weaknesses: Firstly, the proposed method lacks novelty from my perspective, as leveraging LLMs to generate data from their own distribution for continual fine-tuning has been extensively explored in prior research (e.g., various PPO/DPO variants along with works on chain-of-thoughts that prompt LLM to generate reasoning trace, etc.,). Moreover, based on my experiences with machine translation (MT) and reinforcement learning with human feedback (RLHF), the generative quality of a 7B model is often unreliable, typically requiring filtering processes (i.e., from a reward model). Incorporating all rationales generated by the LLM without any scoring, filtering, or grounding is likely to reinforce the model's own hallucinations. Even if this approach preserves the model’s instruction-following abilities, the practical utility of such fine-tuning remains questionable.

Secondly, the rationales derived from a translation dataset may lack diversity. A comparison or discussion with existing work, such as TOWER [1], is necessary. TOWER achieves high MT performance by utilizing parallel corpus and diverse instruction-following datasets during fine-tuning, which appears more effective on benchmarks (and the idea is more straightforward than rationales generation). Therefore, I believe another baseline that simply uses parallel data for fine-tuning + diverse instruction-following dataset for preserving the model's pretrained ability is needed.

[1] Alves et al., (2024). TOWER: An Open Multilingual Large Language Model for Translation-Related Tasks

Meta-review: ### Summary
The paper introduces RaDis (Rationale Distillation)  to improve machine translation capabilities of large language models (LLMs) without compromising their general instruction-following abilities. The approach leverages self-generated rationales to mitigate catastrophic forgetting while enabling efficient training with minimal computational resources. Extensive experiments demonstrate competitive performance in translation while preserving general task abilities, highlighting its potential for versatile LLM applications.

### Strengths
The motivation of using self-generated rationales to balance specialized and general capabilities is good. Extensive empirical results are shown across multiple models and tasks, showcasing improved efficiency over traditional methods.

### Weaknesses:
Main weakness is that the paper is a bit limited in its scope and the proposed method itself doesn't bring new insights to the post-training research community of LLMs.

Decision: Reject