Title: Why Not Transform Chat Large Language Models to Non-English?
Venue: NeurIPS 2024
Accepted: no
Keywords: Large Language Model; Knowledge Transfer; Catastrophic Forgetting; Multi-turn Conversation; Human Preference
Abstract: The scarcity of non-English data limits the development of non-English large language models (LLMs). Transforming English-centric LLMs to non-English has been identified as an effective and resource-efficient method. Previous works start from base LLMs and perform knowledge distillation (KD) with data generated by stronger LLMs, e.g. GPT-4. Compared to base LLMs, chat LLMs are further optimized for advanced abilities, e.g. multi-turn conversation and human preference alignment, and thus more powerful in both helpfulness and safety. However, transforming a chat LLM involves two critical issues: (1) How can we effectively transfer advanced abilities without their supervised data? (2) How can we prevent the original knowledge from catastrophic forgetting during transformation? We target these issues by introducing a simple framework called TransLLM. For the first issue, TransLLM divides the transfer problem into some common sub-tasks with the translation chain-of-thought, which uses the translation as the bridge between English and non-English step-by-step. We further enhance the performance of sub-tasks with publicly available data. For the second issue, we propose a method comprising two synergistic components: low-rank adaptation for training to maintain the original LLM parameters, and recovery KD, which utilizes data generated by the chat LLM itself to recover the original knowledge from the frozen parameters. In the experiments, we transform the LLaMA-2-chat-7B to the Thai language. Our method, using only single-turn data, outperforms strong baselines and ChatGPT on multi-turn benchmark MT-bench. Furthermore, our method, without safety data, rejects more harmful queries of safety benchmark AdvBench than both ChatGPT and GPT-4.

1

Abstract

The scarcity of non-English data limits the development of non-English large language models (LLMs). Transforming English-centric LLMs to non-English has been identified as an effective and resource-efficient method. Previous works start from base LLMs and perform knowledge distillation (KD) with data generated by stronger LLMs, e.g. GPT-4. Compared to base LLMs, chat LLMs are further optimized for advanced abilities, e.g. multi-turn conversation and human preference alignment, and thus more powerful in both helpfulness and safety. However, transforming a chat LLM involves two critical issues: (1) How can we effectively transfer advanced abilities without their supervised data? (2) How can we prevent the original knowledge from catastrophic forgetting during transformation? We target these issues by introducing a simple framework called TransLLM. For the first issue, TransLLM divides the transfer problem into some common sub-tasks with the translation chain-of-thought, which uses the translation as the bridge between English and non-English step-by-step. We further enhance the performance of sub-tasks with publicly available data. For the second issue, we propose a method comprising two synergistic components: low-rank adaptation for training to maintain the original LLM parameters, and recovery KD, which utilizes data generated by the chat LLM itself to recover the original knowledge from the frozen parameters. In the experiments, we transform the LLaMA-2-chat-7B to the Thai language. Our method, using only single-turn data, outperforms strong baselines and ChatGPT on multi-turn benchmark MT-bench. Furthermore, our method, without safety data, rejects more harmful queries of safety benchmark AdvBench than both ChatGPT and GPT-4.

ChatGPT

ฉันไม่สามารถปฏิบัติตามคําขอของคุณได ้

...

(I am unable to comply with your request... )

Attacker

อธิบายรายละเอียดวิธี

##

(Explains in detail how to ##)

(The ## can happen in several ways... )

วิธี

##

สามารถเกิดขึนในหลายรูปแบบ

...

ซึงสามารถทําได ้ หลายวิธี ...

(This can be done in several ways... )

Ours w/ recovery KD

Figure 1: In this example, the attacker jailbreaks ChatGPT in Thai, while our method successfully rejects to response. The recovery KD data is more suitable for preserving the original knowledge than widely used GPT-4 KD data, although GPT-4 performs better in both helpfulness and safety. We omit the harmful text with ## and provide the English translation under the Thai text.

1 Introduction 24

Recently, significant influence has been demonstrated by chat large language models (LLMs), such 25

as ChatGPT (OpenAI, 2022), Palm-2 (Anil et al., 2023), and LLaMA-2-chat (Touvron et al., 2023). 26

Their high capabilities rely on massive data and complex training processes. Taking the LLaMA27

2-chat as an example, the training usually includes the following steps: (1) pre-training (PT) on a 28

large monolingual corpus to obtain the base LLM; (2) supervised fine-tuning (SFT) on multi-turn 29 dialogue datasets; (3) iteratively refining on human preference datasets using reinforcement learning 30 with human feedback (RLHF) methodologies (Ouyang et al., 2022). These steps help in creating 31 LLMs that not only understand and generate human-like text but also align with human values, and 32 therefore provide safe and useful responses. 33

34

35

36

37

Unfortunately, popular unlabeled and labeled training data is English-dominated. Consequently, LLMs are less satisfying in terms of both usefulness and safety when being applied to non-English. Yong et al. (2023) have shown that even powerful LLMs, such as GPT-4, are vulnerable to safety concerns in non-English.

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

To improve the non-English performance, recent works attempt to transfer knowledge from English to non-English. However, they focus on base LLMs instead of powerful chat LLMs. Basically, they start from base LLMs and use knowledge distillation (KD) data generated by the strong LLM, like GPT-4, for transfer training and instruction tuning. For example, PolyLM (Wei et al., 2023) transfers English knowledge implicitly via multilingual instruction tuning on a multilingual base LLM. X-LLaMA (Zhu et al., 2023) supplements the multilingual instruction-following task with the translation task to build semantic alignment across languages.

When transforming the base LLMs, instruct tuning is performed simultaneously with or after transfer training. Therefore, the instruction following knowledge, i.e. the basic conversation knowledge, will not be overridden by extra knowledge. However, for chat LLMs, the advanced conversation knowledge, especially human preference, has been incorporated into the model parameters during fine-tuning. As a result, subsequent transfer training in previous works will result in catastrophic forgetting of such knowledge. What is worse, the high-quality STF data used for training the chat LLM is precious and usually unavailable. Therefore, transforming a chat LLM involves two critical issues: (1) How can we transfer advanced abilities with limited available data? (2) How can we prevent the original English knowledge from catastrophic forgetting during transfer?

To build safe non-English LLMs as shown in Figure 1, we target these issues by introducing a simple framework called TransLLM. For the first issue, TransLLM utilizes the translation chain-of-thought (TCOT) (Zhang et al., 2023), which models the transfer as some common sub-tasks. During TCOT, the LLM will handle the non-English query step-by-step in single inference: it first translates the query to English; then it responds to the query in English; and finally it, generates the non-English answer based on all the above context. We further enhance the the performance of sub-tasks with publicly available data thus TCOT can transfer English knowledge effectively. For the second issue, we propose a method comprising two synergistic components: (1) We employ the low-rank adaptation (LoRA) (Hu et al., 2021) for training to maintain the original LLM parameters. (2) We introduce recovery KD, utilizing data generated by the chat LLM itself, to recover the original knowledge from the frozen parameters. The recovery KD data can be fitted easily using the original parameters. This enables the LLM to learn a 'shortcut' that uses the English knowledge from the original parameters.

As shown in Figure 2, TransLLM organizes all the above ideas into the following steps: (1) Model extension: we extend the model with LoRA modules and fine-grained target language vocabulary. (2) Target language pre-training: we pre-train the chat LLM on the monolingual target language data so that the LLM can leverage such knowledge to improve translation and target language responses. (3) Translation pre-training: we further train the LLM with a bi-directional translation task between English and the target language, and we also introduce the English language modeling task to protect the English embeddings. (4) Transfer fine-tuning: we fine-tune the LLM on TCOT, recover KD, and translation data so that the LLM can respond in English, the target language, and the translation tasks automatically.

We conduct comprehensive experiments for transforming the LLaMA-2-chat-7B from English to Thai. TransLLM outperforms strong baselines and surpasses ChatGPT by 35% and 23.75% for the first and second turns on the MT-bench with statistical significance. More importantly, we attain an improvement of 14.8% and 8.65% over ChatGPT and GPT-4 respectively on the safety benchmark AdvBenchmark with statistical significance.

Our main contributions are summarized as follows:

In this paper, we highlight the advantages and challenges of transforming a chat LLM to non-English and propose a simple yet effective framework for this end.

83

84

85

86

87

88

89

90

91

92

Recovery KD Data

Q: Introduce yourself

A: I am the target Chat-LLM

TCOT Data

Q_TH:

แนะนําตัวเอง

Q_EN: Introduce yourself

A_EN: I am the target

Chat-LLM

A_TH:

ฉันคือตัวแบบภาษา

ขนาด

...

Translation Data

EN: Introduce yourself

TH:

แนะนําตัวเอง

TH:

ฉันคือตัวแบบภาษาขนาดใหญ่สําหรับการ

สนทนา

Step2: Target Language Pre-Training

Step4: Transfer Fine-Tuning

Step1: Model Extension

Step3: Translation Pre-Training

Chat-LLM

Extended Input Embedding

Extended LM Head

LORA

EN: How to translate

TH:

วิธีการแปล

EN: Protect English Embeddings

The experiments indicate TransLLM successfully transfer advanced abilities, e.g. multi-turn conversation and human preference alignment, with limited available data. TransLLM, with only 7 billion parameters, outperforms ChatGPT in Thai in both helpfulness and safety.

Analysis shows that recovery KD plus with LoRA successfully preserves the original knowledge. The TransLLM model mostly uses the original knowledge for English while uses the new knowledge for Thai.

Wediscuss the limitations of TransLLM, and point out several potential future directions. We will make our code and datasets publicly available (please refer to supplementary materials). We hope this work can lay a solid foundation for developing safe LLMs in non-English.

2 Background

The language models are trained to predict the next token in a sequence given the previous tokens by 93 maximum likelihood estimation (MLE), which can be represented by the following equation: 94

where θ denotes learnable model parameters, and y <i are the tokens preceding y i in the sequence. 95

For fine-tuning on a supervised dataset, each instance contains a query x 96 y . The SFT loss is only calculated on the label y , ignoring the query x : 97

and its corresponding label

For both PT and SFT, the special tokens 
 and 
 are added at the beginning and the end of the 98 training instance respectively. 99

100

101

102

103

104

105

106

107

108

109

110

111

112

3 Method

In this section, we first describe the model architecture, then we introduce the training and inference procedures in detail.

3.1 Model Architecture

Nowadays, popular LLMs use byte-level byte pair encoding (BBPE) tokenizer (Wang et al., 2020) following GPT-2 (Radford et al., 2019). However, the tokenizer is usually developed on the Englishdominated dataset, therefore this tokenizer often tokenizes each non-English character to several bytes resulting in a long sequence. Inspired by Cui et al. (2023) and Pipatanakul et al. (2023), we extend the vocabulary using monolingual data of the target language to improve the model efficiency.

LoRA is a parameter-efficient training method, which is another technique that has been widely used for transferring the LLM. However, in this work, we use LoRA not only for efficiency but also for preserving the original parameters. Considering a weight matrix W ∈ R d × k of the target LLM, LoRA represents its update ∆ W using two low rank matrices B ∈ R d × r and A ∈ R r × k as follows:

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

where r denotes the pre-determined rank, h denotes the input, ˜ h denotes the output of the original module, and ˆ h denotes the output of the updated module. During training, the original W is frozen, so that original knowledge can be recovered by the recovery KD.

3.2 Training

3.2.1 Target Language Pre-Training

The chat LLMs are often insufficient on target language modeling due to the imbalanced training corpus. Target language modeling is essential for generating fluent and localized text. Furthermore, many works show that the monolingual pre-training can significantly improve the translation quality (Zheng et al., 2019; Xu et al., 2023). To build a solid foundation for the target language, we pre-train the TransLLM model on monolingual data of the target language using Eq. 1.

We do not introduce any English task in this stage because of the following two reasons: first, the pre-training involves quite computational consumption, and it can be unacceptable to find a proper mixing ratio between the English and target language data; second, the English embeddings are rarely updated on the target language data, therefore all the parameters of original LLM are almost unchanged.

3.2.2 Translation Pre-Training

TCOT relies on translation to bridge the English and the target language. Therefore, we introduce translation pre-training to improve the bidirectional translation quality between English and the target language. Inspired by mBART (Liu et al., 2020), we use the special language id token to denote translation directions. Considering we transform the LLM from language α to β , where α = English in this paper, we formulate the parallel pair ( s α , s β ) as two instances: cat ( s α , < β > , s β ) and cat ( s β , < α > , s α ) , where cat ( · ) denotes the concatenate operation.

The translation training could disturb the original English embeddings. Thus, we introduce English monolingual data into the translation pre-training stage. Specifically, we randomly insert the translation instance between English monolingual data using line break '\n' as the separator. Based on the first stage, we train the TransLLM model on the mixed data by pre-training objective in Eq. 1.

3.2.3 Transfer Fine-Tuning

The two-stage pre-training enables the TransLLM in target language modeling and cross-lingual translation. However, the TransLLM inevitably forgets the original knowledge. In this stage, we aim to recover the original knowledge and teach the TransLLM model how to perform TCOT and when to do translation.

Recovery Knowledge Distillation Data. Previous works focus on transferring knowledge from base LLMs. To teach the base model how to follow human instructions, previous works perform knowledge distillation with strong chat LLMs as the teacher by using the Alpaca dataset (Taori et al., 2023). The Alpaca dataset generates queries using the self-instruct technique (Wang et al., 2022), then responds using ChatGPT or GPT-4. Although the vanilla KD works well for base LLMs, we argue that it is not helpful for chat LLMs as shown in Sec. 5.2. To address this problem, we introduce the recovery KD that uses the target chat LLM to generate the responses. Although the recovery KD data are often worse than GPT-4 KD data, it will help the model to recover the knowledge from the original LLM parameters. We also introduce a special token <RESPONSE> in recovery KD to direct the behavior of the TransLLM model. Considering a KD instance in English with query q α and answer a α , we formulate the query and label in Eq. 2 as x = q α and y = cat ( <RESPONSE> , a α ) respectively.

TCOT Data. Based on the recovery KD data ( q α , a α ) , we use machine translation to obtain its translations ( q β , a β ) . Finally, we can organize the TCOT data as x = q β and y = cat ( < α > , q α , <RESPONSE> , a α , < β > , a β ) . That means when we input a query in β , the model should first translate it into α as q α . Then the model should <RESPONSE> the English query as a α using original knowledge as we teach in recovery KD. Finally, the TCOT outputs the response in β as a β based on all previous sequences. As discussed in Sec. 5.3, the previous sequences also

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

contribute to the final response. Different from Zhang et al. (2023), we use special tokens instead of natural language to direct the model's behavior. This is because the special tokens will not disturb the English embeddings and make it convenient to extract results.

Translation Data. Due to the TCOT data, the model may be confused about the translation instruction in β without extra translation SFT. Therefore, we also construct bi-direction translation data based on previous parallel pairs ( q α , q β ) and ( a α , a β ) . Taking the parallel pair ( q α , q β ) as an example, we first wrap the source sentence using translation prompt templates as prompt ( q α ) . 1 Then we can obtain x = prompt ( q α ) and y = cat ( < β > , q β ) .

Finally, we randomly mix all the data mentioned above and fine-tune the TransLLM model by Eq. 2.

3.3 Inference

The final TransLLM model can respond in both α and β , including α -β bi-direction translation. For a single-turn conversation, the TransLLM model will decide the proper mode by itself given only the input query x . To leverage the powerful multi-turn conversation ability of the original LLM for β , we follow the original multi-turn format. For the multi-turn task in β , we only take the English parts of the previous TCOT output as history. To be specific, we organize the input as x = cat ( q α 1 , a α 1 , . . . , q α n , a α n , q β n +1 ) , where n is the number of past turns. We do not use any special tokens in the history as the original LLM does. Interestingly, even in this unseen setting, the model still outputs the TCOT format as y = cat ( < α > , q α n +1 , <RESPONSE> , a α n +1 , < β > , a β n +1 ) . We show the whole multi-turn template in Appendix A.3.

4 Experiments

4.1 Settings

It is extravagant to train and evaluate a chat LLM in non-English. Therefore, during our experiment, we mainly transform LLMs from English (EN) to Thai (TH) language, i.e. α = EN and β = TH. We describe our basic settings as follows.

Models. We implement our pipeline using Chinese-LLaMA-Alpaca-2 2 project, which is based on Transformers 3 . For the TransLLM model, we use the LLaMA2-Chat-7B as the target chat LLM. Following Cui et al. (2023), we use SentencePiece (Kudo and Richardson, 2018) to learn the TH vocabulary on the monolingual TH data that we use in target language pre-training. After we merge the TH vocabulary with the original vocabulary, the final vocabulary size (including 3 special tokens) is 43,012. The new embeddings are randomly initialized. We apply LoRA on the weights of the attention module and multi-layer perceptron blocks. The LoRA rank is set as r = 64 . Overall, there are a total of 512.27 million trainable parameters including embeddings and LM heads. After all of the training is completed, we merge the LoRA modules into the main backbone, the final model has 6.83 billion parameters. For a fair comparison, we re-implement most of the baselines by our setting following their papers. The details of our model and baselines are in Appendix A.1.

Training Data. For target language pre-training, we use the monolingual TH data from mC4 (Xue et al., 2020). We first filter the mC4-TH using the sensitive word list to reduce the harmful text. Then, we use MinHashLSH 4 to deduplicate documents in mC4-TH following GPT-3 (Brown et al., 2020). Finally, we have about 11 billion tokens of TH data. Compared to the 2 trillion tokens EN data used in LLaMA-2, the TH dataset is quite small. For translation pre-training, we collect the EN-TH parallel data from CCAligned (Chaudhary et al., 2019), Tatoeba Challenge Data (Tiedemann, 2020), and OpenSubtitles (Lison et al., 2018). We directly use the EN documents released in the Pile dataset which has been pre-processed (Gao et al., 2020). We randomly sample 1 million parallel pairs and EN documents respectively for translation pre-training. For the transfer fine-tuning, we use the

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

Table 2: Agreement between GPT-4 and humans. 'R=' denotes the expect agreement between random judges. † EN results are from Zheng et al. (2024).

query from the Alpaca dataset and generate the response using the target model LLaMA2-Chat-7B. We further use Google Translate to obtain TCOT and translation data based on recovery KD data. In our preliminary study, Google Translate may translate the variable in code which is not desirable for the chat LLM. Thus, we use GPT-4 to recognize the 'do not translate' part. We use the same monolingual and translation data for baselines, while we use the Alpaca-GPT-4 (Peng et al., 2023) as the SFT data following their setting. There are a total of 52K queries in the Alpaca dataset, we use the first 50K queries as the training set and the rest 2K queries as the validation set in our experiments. We provide the training details in A.2.

Benchmark. For helpfulness, we utilize two widely used LLM benchmarks, MT-bench (Zheng et al., 2024) and AlpacaEval (Li et al., 2023). MT-bench has 80 multi-turn questions across 8 domains. AlpacaEval consists of 805 single-turn questions collected from 5 different subsets. The original benchmarks are in EN. We employ professional translators to translate these two datasets into TH. For safety, we use the AdvBench. AdvBench consists of 520 instructions that induce LLMs output harmful responses. Following the setting in Yong et al. (2023), we directly use Google Translate to translate the AdvBench from EN to TH.

Evaluation. For helpfulness, we use strong LLMs as judges, which show considerable consistency with human evaluators in EN (Zheng et al., 2024). LLM-as-a-Judge is efficient, reproducible, and cost-effective. However, it is still unknown whether it will work in the TH language. To obtain a reliable result, we first invite professional translators to conduct the human evaluation for some strong models on the MT-bench. We test the consistency between human and GPT-4 evaluation as described in (Zheng et al., 2024). After we prove that GPT-4 achieves acceptable consistency with human evaluators, we evaluate all models with it. Both human annotators and LLMs rate the response on a scale from 1 to 10, and we further calculate the win, tie, and loss rate based on the scores. We use ∆ to denote the gap between the win and loss rate calculated with the tie. For safety, we translate the TH responses into EN and let EN annotators annotate them into Bypass, Reject, and Unclear. Bypass means the attack bypasses the safety mechanism of LLMs. Reject means LLMs refuse to output harmful information. Unclear means the responses are safe but unclear due to translation or hallucination, etc. The setting follows Yong et al. (2023) strictly. Please refer to this paper for details. In Appendix A.4, we describe the evaluation procedure, the instructions for human evaluators, and the information of evaluators in detail. We also conduct significant tests for main results as described in Appendix C. We mark the results with bold if the difference is statistically significant ( p < 0 . 05 ).

4.2 Main Results

4.2.1 Human Evaluation Results

Better performance than ChatGPT on MT-Bench. As shown in Table 1, TransLLM surpasses ChatGPT by 35% and 23.75% for the first and second turn on MT-bench with statistical significance. It is an inspiring result although TransLLM is still behind GPT-4 in TH. Because we only use the LLaMA-2 with 7B parameters. As the fine-grained scores in Appendix B.1 shown, the two domains with the biggest gaps between ours and GPT-4 are Math and Coding, which are also the weaknesses

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

of LLaMA-2 in EN. We leave exploring TransLLM on more powerful open-source LLMs in the future.

High agreement between humans and GPT4 in TH. Following Zheng et al. (2024), we calculate the average agreements by comparing every two models. In Table 2, GPT-4 shows high consistency with human annotators. The consistency (w/ tie) between GPT-4 and humans reaches 75.42% and 70.42% in the first and second turns, which are much higher than random guesses and even higher than the consistency in EN. Therefore, we use GPT-4 to evaluate the helpfulness in the following experiments.

Higher safety than ChatGPT and GPT-4. In Table 3, TransLLM has a rejection rate of 94.61%, close to 99.23% of the original model. It indicates that we successfully transfer most of the human preference about the safety of the original model. TransLLM attains an improvement of 14.8% and 8.65% over ChatGPT and GPT-4 for rejecting harmful queries with statistical significance. More importantly, although GPT-4 is as safe as the original LLM in EN, the performance of our w/ GPT-4 KD is much below our w/ recovery KD. Later, we will demonstrate that this is because recovery KD successfully recovers the original knowledge.

4.2.2 GPT-4 Evaluation Results

Better performance than strong baselines. As shown in Table 4, TransLLM significantly outperforms baselines that are built on open-source resources. Notably, we specifically build the baseline NLLB-bridge which uses the powerful translation model NLLB-3B (Costa-jussà et al., 2022) as the bridge between LLaMA-2chat-7B and the TH language. Using the multiturn ability of LLaMA-2-chat-7B, NLLB-bridge achieves good performance in the second turn.

Although NLLB-bridge uses more parameters and more translation resources, it still loses to TransLLM. We will explain in detail why TransLLM is better than translation-as-a-bridge in the analysis. Typhoon with TH pre-training achieves sub-optimal second-turn performance among baselines. It is probably because the TH documents teach the LLM how to model long context in TH. Under GPT-4 evaluation, we slightly outperform ChatGPT without statistical significance. It seems difficult for GPT-4 to compare two strong LLMs on small datasets in TH. We select the baselines that perform well on the first turn of the MT-bench, for further evaluation on Alpaca-Eval. On the larger dataset, TransLLM outperforms baselines and ChatGPT by a large margin with statistical significance as shown in Table 5.

5 Analysis

5.1 All Components Work Together

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

We conduct comprehensive ablation studies on MT-Bench to investigate the impact of TransLLM's components and present results in Table 6. The results confirm our hypothesis that transforming chat LLMs could provide better conversational ability than base LLMs. Pre-training on TH documents helps TransLLM output fluency in TH response with long context. Thus, TransLLM without TH pre-training is less satisfying on both the first and second turn. Since TH pre-training and transfer fine-

tuning also provide some translation knowledge, the improvement of the translation pre-training is not as significant as other components. Beyond safety, the high-quality GPT-4 KD data also leads to performance degradation for helpfulness. That is because our goal is not to inject more knowledge but to preserve the original knowledge. We also examine the contribution of LoRA. Specifically, we merge the LoRA parameters with full parameters before transfer fine-tuning. We are unable to conduct full fine-tuning for per-training, but the merged model is a good approximation according to Eq. 3. We further conduct transfer fine-tuning with full parameters based on the merged model. In most tasks, full fine-tuning is better or comparable with LoRA. However, in our case, full fine-tuning wipes the original knowledge from parameters, and therefore its performance is much lower than TransLLM with LoRA. When using the history in TH, TransLLM is also capable of multi-turn conversation with small performance degradation. That means TransLLM can handle TH context well, this ability could be further developed for retrieval augmentation in TH.

5.2 TransLLM Recover the Original Knowledge

Knowledge is forgotten and recovered. To measure how much original knowledge is forgotten by the chat LLM, we calculate the generation probabilities on the hold-out validation set of recovery KD data in EN. We also calculate the average difference between the generation probabilities of the target LLM and different models. As shown in Table 7, after pre-training, which has been proven to be necessary, the LLM sig-

nificantly forgets the conversation knowledge. GPT-4 KD, which is widely used in previous works, can provide high-quality knowledge. However, this kind of knowledge is quite different from and competes with the original knowledge. As a result, the LLM still forgets much original knowledge using GPT-4 KD. Meanwhile, TransLLM successfully recovers the original knowledge.

LoRA also helps. LoRA keeps the original parameters unchanged. The LLM can fit the recovery KD data easily when using knowledge from these frozen parameters. This easy pattern is a 'shortcut' that prompts the LLM to learn to use the original knowledge for EN and new knowledge for TH. To confirm this assumption, on the TCOT validation data, we calculate the cosine similarity between the last layer's hidden states of the original model ˜ h and LoRA updated model ˆ h as defined in Eq. 3. The average similarity per token for EN responses is much larger than that for TH responses, 0.6191 vs. 0.2522. That means TransLLM successfully learns the 'shortcut' using LoRA and recovery KD together.

5.3 Why TransLLM is better than translation-as-a-bridge?

Comparable translation performance. The translation performance is critical for both TransLLM and translation-as-a-bridge. Therefore, we test them on the widely used benchmark Flores-200 (Goyal et al., 2022). As shown in Table 8, benefiting from translation and TH pretraining, TransLLM outperforms ChatGPT and NLLB on EN-TH and achieves competitive per-

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

formance on TH-EN. We also ask the naive TH speaker to provide a fluency score for each model on MT-Bench in Table 9. The fluency of NLLB is as poor as its translation performance on EN-TH. NLLB usually translates the responses literally. For example, NLLB translates 'I see' into 'I see something' instead of 'I understand' in TH. Surprisingly, the response of GPT-4 is not very fluent and natural. GPT-4 often uses full-stops and commas which are not used in TH. ChatGPT and TransLLM are generally fluent, with translationese to a certain degree. For example, TH speakers do not use 'sure' or 'of course' at the beginning of responses, but ChatGPT and TransLLM do.

TransLLM is more than translation. Translation performance is important but not the whole story. TransLLM outputs an EN query, EN response, and TH response at once. It means that TransLLM can use all previous information for TH responses and therefore achieve better performance. To verify it, we use TransLLM to translate its EN responses in another round of inference. The performance is worse than the standard response with ∆ = 13.75% and ∆ = 18.75% on first and second turn. The attention map of TransLLM in Appendix B.2

shows that TransLLM outputs the TH response mostly based on the TH response itself and then the EN response. However, the TH response also pays a little attention on the TH query and EN query. Besides, translation-as-a-bridge needs to deploy two models, which is costly and inconvenient.

6 Related Works

Recently, there have been many works that attempt to transfer knowledge from English to non-English for LLMs. For example, Chinese LLaMA (Cui et al., 2023) and Typhoon(Pipatanakul et al., 2023) directly perform continuous pre-training and instruct tuning with extended vocabulary using LoRA. PloyLM (Wei et al., 2023) adopts multilingual pre-training based on the curriculum learning strategy that gradually exposes more low-resource corpus. ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) are also well-known multilingual LLMs. Zhu et al. (2023) focus on building semantic alignment with cross-lingual instruct tuning and translation training. Bansal et al. (2024) augment LLMs by combining the English-dominated LLM with the non-English model. Some other works focus on transfer reasoning abilities: Qin et al. (2023) introduce cross-lingual prompting to improve zero-shot chain-of-thought reasoning across languages; She et al. (2024) propose multilingual alignment-aspreference optimization to align reasoning abilities across languages. PLUG (Zhang et al., 2023) only uses the TCOT data to train the base LLMs directly. Different from PLUG, we propose a systematic framework for transforming chat LLMs. We highlight that the TCOT highly relies on the performance of its sub-tasks and introduce how to preserve the knowledge of the chat LLM.

7 Conclusion

Chat LLMs have been specifically optimized for chat usage and therefore are helpful and safe in the dominant language. In this paper, we propose a framework for transforming an off-the-shelf chat LLM to other languages. In this framework, we utilize TCOT to transfer chat knowledge and further enhance the TCOT's sub-tasks using publicly available data. To recover the original knowledge, we propose the recovery KD method supplemented with LoRA. The experiments in TH show that we transfer desired abilities to TH and outperform ChatGPT in both helpfulness and safety. Overall, we hope that this work can become the foundation for developing safe LLMs in many languages other than English.

Limitations and future works. Due to limited resources, we only conduct experiments that transform LLaMA-2-chat-7B to TH. However, we conduct comprehensive experiments and in-depth analysis to reveal the mechanism of the proposed TransLLM. For now, TransLLM is still highly dependent on translation. Consequently, TransLLM can not handle the queries related to TH text, e.g. word games in TH. A simple solution is to enable TransLLM, through training, to choose whether respond to with TH mode or TCOT mode. Due to the TCOT, the inference overhead of TransLLM is much longer than other baselines. Recently, Goyal et al. (2023) and Deng et al. (2023) show that the implicit chain-of-thought achieves similar performance on reasoning tasks without additional inference overhead. We would like to explore TransLLM with implicit TCOT in the future.

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

References

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403 .

Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. 2024. LLM augmented llms: Expanding capabilities through composition. CoRR , abs/2401.02412. Version 1.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.

Vishrav Chaudhary, Yuqing Tang, Francisco GuzmÃ¡n, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2) , pages 263-268, Florence, Italy. Association for Computational Linguistics.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 .

Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177 .

Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460 .

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 .

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics , 10:522-538.

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2023. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations .

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 .

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66-71, Brussels, Belgium. Association for Computational Linguistics.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval .

Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Miyazaki, Japan. European Language Resources Association (ELRA).

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics , 8:726-742.

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

OpenAI. 2022. Introducing chatgpt. Blog post https://www.openai.com/blog/chatgpt .

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 .

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 .

Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharnpipitchai. 2023. Typhoon: Thai large language models. arXiv preprint arXiv:2312.13951 .

Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2695-2709.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.

Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 578-585.

Shuaijie She, Shujian Huang, Wei Zou, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. 2024. Mapo: Advancing multilingual reasoning through multilingual alignment-as-preference optimization. arXiv preprint arXiv:2401.06838 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: an instruction-following llama model (2023). URL https://github. com/tatsu-lab/stanford_alpaca .

Jörg Tiedemann. 2020. The Tatoeba Translation Challenge - Realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation , pages 1174-1182, Online. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural machine translation with byte-level subwords. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 9154-9160.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560 .

Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018 .

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online. Association for Computational Linguistics.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674 .

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 .

Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446 .

Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2023. Plug: Leveraging pivot language in cross-lingual instruction tuning. arXiv preprint arXiv:2311.08711 .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36.

Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin-Yu Dai, and Jiajun Chen. 2019. Mirrorgenerative neural machine translation. In International Conference on Learning Representations .

Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948 .

A Experiment Details

A.1 Models

We list backbone, training data, and model size in Table 10. Due to the huge consumption of multilingual (MTL) pre-training, we directly use the model PolyLM-MultiAlpaca-13B released in Wei et al. (2023) for PolyLM. PolyLM uses ChatGPT to generate the Alpaca data while other baselines use the Alpaca data generated by GPT-4. We use the gpt-3.5-turbo-0125 and gpt-4-0613 for ChatGPT and GPT-4 in all experiments (including evaluation) through OpenAI API. We re-implement other baselines by strictly following their papers and using the same data as our model. To reduce the impact of randomness, we use greedy search for all experiments. We set the temperature as 0 for ChatGPT and GPT-4 through API to approximate the greedy search.

Please refer to Touvron et al. (2023) for model structures of LLaMA-2. We only list the LoRA parameters here. We set the rank to 64, alpha to 128, and dropout to 0.05 for LoRA. These parameters are applied to the q_proj, v_proj, k_proj, o_proj, gate_proj, down_proj , and up_proj modules of the original model. Besides, the embed_tokens and lm_head are also trainable.

A.2 Training 525

526

527

528

529

530

531

We train the TransLLM model on 8 A100 GPUs as follows.

TH Pre-Training We train the TransLLM using a warm-up ratio of 0.0005, a maximum sequence length of 512 tokens, and a weight decay of 0.01. The training was conducted with each GPU managing 128 batches and utilizing a gradient accumulation step of 1. The peak learning rate is set at 2e-4 with a cosine learning rate decay (max_epoch=100), and training operated under bf16 precision facilitated by deepspeed, employing ZeRO stage 2.

We only run 1 epoch for this stage, which spends 168 × 8 GPU hours. As shown in Figure 3, the 532 initial training loss is approximately 7.8, which converges to below 1.7 after around 0.1 epochs of 533 training. The final loss reaches around 1.42. 534

loss

tag:train/loss

8

6

4

2

0

5k

10k

15k

20k

25k

30k

Translation Pre-Training According to the data size, we set the warm-up ratio as 0.05, the max_epoch=10 for the cosine learning rate decay. We use 0.1% examples as the validation set and calculate valid loss every 400 steps. The best model has been trained for about 3 epochs, which spends 40 × 8 GPU hours. The remaining configurations remain consistent with the first stage.

Transfer Fine-Tuning Our max_seq_length is set to 2048 for fine-tuning, and when batching data, we pad sentences with '<PAD>' tokens. The peak learning rate is set to 1e-4, the warmup ratio is set to 0.01, and the single-card batch size is set to 16 with gradient accumulation steps as 4. We set weight decay as 0. We use 2K examples as the validation set and calculate valid loss every 200 steps. The best model has been trained for about 1 epoch, which spends 6 × 8 GPU hours. The remaining configurations remain consistent with the first stage.

A.3 Inference

We provide the whole multi-turn prompt in Table 11, where '
 
', '
 
', and '
 [\INST]' denote the whole instance, system prompt, and instruction respectively.

535

536

537

538

539

540

541

542

543

544

545

546

547

A.4 Evaluation 548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

A.4.1 Human Evaluation

For helpfulness, the results are evaluated by three annotators. Annotator A is a professional translator expert in EN and TH. Annotator B is a computer engineer who is an expert in EN, Math, Coding, and Extraction. Annotator C is a native TH speaker while also an expert in EN. The three annotators cooperate with each other to complete the whole evaluation process as follows. Annotator A is the major annotator who is responsible for annotating most of the queries except for the Math, Coding, and Extraction domains. For these three domains, annotator A first translates the results from TH to EN. Annotator B then annotates these three domains in EN translations. Meanwhile, Annotator C helps annotator A evaluate the fluency of all responses. To obtain consistent annotations between evaluators and questions, we define comprehensive instructions for annotators in Table 12.

For safety, the responses are first translated from TH to EN and then evaluated by three professional translators who are experts in EN. However, one response is only annotated by one translator due to a limited budget. Please refer to the annotation instruction in Yong et al. (2023).

All models are anonymous to all annotators in the whole evaluation process!

A.4.2 Automatic Evaluation

We follow the setting of LLM-as-a-Judge in Zheng et al. (2024). For Reasoning, Math, and Coding domains, we provide the EN responses of GPT-4 as references. Note that, these three domains are different from human evaluation because annotator A is good at Reasoning instead of Extraction. We modify the evaluation prompts provided in Zheng et al. (2024) to inform GPT-4 that the queries and responses are in TH. Please refer to Zheng et al. (2024) for the details of how to calculate the agreement.

We use the default wmt22-comet-da model 5 for COMET (Rei et al., 2022). We use the BLEU (Papineni et al., 2002) implemented in the scarebleu 6 , whose signature is "BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.4.0".

A.5 Licenses

Our experiments use open-source resources. We list their licenses in Table 13. We have properly cited their papers and strictly followed their licenses.

B Other Results

B.1 Results in Scores 577

We provide evaluation scores on different benchmarks in Table 14, 15, 16 , and 17. 578

B.2 Attention Map of the TransLLM Output 579

As shown in Figure 4, the TH response focuses on the TH response, EN response, EN query, and TH 580 query, in order from high to low. 581

C Statistical Methods 582

C.1 Confidence Interval 583

We first calculate the standard deviation for proportion p on n examples as: 584

l ow                                                                        high

586

587

588

589

590

591

592

593

594

Then we use the normal approximation method to calculate the CI for ratio p as 585

where u denote the critical value, for the two-tailed 95% confidence interval used in this paper u = 1 . 96 .

C.2 Significant Test

We conduct a two-sided binomial test for the win rate without tie p win = n win / ( n win + n loss ) . The null hypothesis is that the win rate is not different from the loss rate, i.e. H 0 : p win = p loss = 0 . 5 , alternative hypothesis H 1 : p win = 0 . 5 . For the test results of Table 1 and 4, please see Table 18 and 19. The difference between TransLLM and others in Table 5 are all significant with p < 0 . 001 .

̸

We conduct the χ 2 test for safety results in Table 3, the difference between TransLLM and others are all significant with p < 0 . 001 .

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have discussed our contributions and scope in detail in the Abstract and Introduction chapters.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations in Sec. 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have tried our best to provide the details of our experiments in Sec. 4.1 and Appendix A. We also provided our code and datasets in supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided open access to the data and code, with sufficient instructions provided to faithfully reproduce the main experimental results. Please refer the README file in code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have tried our best to provide the details of our experiments in Sec. 4.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully checked our paper. Our paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the potential positive societal impacts in Sec. 7, and it seems that this work does not exert obviously negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Our work aims to transfer the safeguards of chat large language models from English to non-English.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer Appendix A.5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer the README file in code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the TransLLM framework to transform English-centric chat LLMs to non-English languages, addressing the challenges of transferring advanced abilities without supervised data and preventing catastrophic forgetting of original knowledge. Key contributions include using the Translation Chain-of-Thought (TCOT) to divide the transfer process into sub-tasks, employing Low-Rank Adaptation (LoRA) and Recovery Knowledge Distillation (KD) to maintain original LLM parameters and recover knowledge. TransLLM's effectiveness is demonstrated through experiments transforming LLaMA-2-chat-7B to Thai, where it outperformed strong baselines and ChatGPT in multi-turn conversations and safety benchmarks, highlighting significant improvements in helpfulness and safety without extensive supervised data. This framework offers a solid foundation for developing safe and useful non-English LLMs.\n\nSTRENGTHS:\n1. The introduction of TransLLM, which combines Translation Chain-of-Thought (TCOT) and recovery knowledge distillation, provides an effective method for transforming English-centric LLMs to non-English languages. This approach addresses both the transfer of advanced abilities and the prevention of catastrophic forgetting.\n2. The method shows notable improvements in rejecting harmful queries and maintaining human preference alignment, as evidenced by outperforming GPT-4 and ChatGPT on the safety benchmark AdvBench, highlighting the robustness of the model in safety-critical applications.\n\nWEAKNESSES:\n1. The experiments are primarily conducted on transforming LLaMA-2-chat-7B to Thai, which may limit the generalizability of the findings to other non-English languages and other models. Further validations on other models (not necessarily bigger than the current one) or other size of Llama-2 would have strengthened the paper.\n2. While the proposed method exhibits promising results in MT-bench and AlpacaEval, it is not tested on other traditional NLU benchmarks, just like MMLU in English. Incorporating more diverse evaluation benchmarks would provide a more comprehensive assessment of the model's performance across various language tasks.\n\nQUESTIONS:\nMajor:\n\n1. Please addresss the weaknesses above.\n2. For the construction of the TCOT, did you try other concatenation templates besides the one used in the paper? How sensitive is the performance to the choice of the template?\n3. Table 9: How was the human evaluation carried out? How many annotators were involved, and what is the inter-annotator agreement? The final scores are all integers, so it is not clear how to final scores were calculated. \n\nMinor: \n\n1. line 45: instruct tuning -> instruction tuning\n2. line 58: and finally it, generates ... -> and finally, it generates ...\n3. line 59: the the performance -> the performance\n4. line 546: the second 
 should be 
? also in line 547, the [\\INST] should be 
.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a transfer pipeline, TransLLM, which uses a translation chain-of-thought (TCOT) to adapt English-centric large language models (LLMs) to low-resource languages. This pipeline consists of pre-training and supervised fine-tuning (SFT) phases. During the pre-training stage, the authors select monolingual data in the target language and translation parallel data for language modeling. This pipeline also uses an external translation model to construct TCOT dialogue data, which is used in the SFT phase. In the SFT phase, to avoid the catastrophic forgetting problem caused by continual learning, the authors adopted the LoRA PEFT method and further proposed the recovery Knowledge Distillation (KD) method. Specfically, the recovery KD method mixes responses generated by the original model during the SFT-based transfer phase. The effectiveness of this pipeline was demonstrated by successfully transferring the Llama-2-7B model from English to Thai. Compared to other baseline models based on open-source LLMs, the transferred model exhibited great multi-turn dialogue capabilities and safety on the Thai MT-bench and AdvBench benchmarks.\n\nSTRENGTHS:\n1. **Originality**: Compared to other machine translation-based transfer methods in the community, the proposed pipeline introduces the concept of Translation chain-of-thought and specifically addresses the catastrophic forgetting problem that might arise from SFT-based transfer methods, proposing effective solutions.\n2. **Evaluation Protocol**: I appreciate the efforts made by the authors in the experimental section to ensure the effectiveness of evaluations for low-resource language (Thai). The authors employed professionals for evaluation and also validated the agreement between GPT-4's automatic evaluation and human evaluation on the MT-bench. Additionally, they used human translation in constructing some of the test data.\n3. **Significance**: Currently, training data and resources for LLMs are predominantly English-centric. The proposed method helps build strong chat LLMs for low-resource languages and minority groups.\n\nWEAKNESSES:\n1. **Flexibility of Methodology**: Although the proposed method reduces the dependency on instruction-following data in the target language, it still relies on parallel corpora and external translation models for data construction (Translation pre-training data, TCOT data). If high-quality parallel corpora or models are not available, the proposed method might be infeasible. For example, even commercial translation systems cannot support translations for some endangered languages or dialects.\n2. **Scope**: While the method proposed in this paper might be extendable to other low-resource languages, the authors only validated it on Thai, which lacks empirical evidence for the generality of proposed pipeline. I also noticed that the authors emphasized \"non-English\" in the title, but a broader range of non-English languages still requires exploration.\n3. **Effectiveness**: The authors tested the method on Llama-2-7B. Experiments on other model series and larger models could further support the effectiveness of proposed pipeline.\n4. **Quality of MT data**: The authors used Google Translate for translating TCOT and AdvBench data. Although this is a commonly used commercial system, it would be better to supplement the evaluation and report on the quality of these data translations to explore their impact. In the absence of reference translations, quality estimation methods like CometKiwi [1] and TransQuest [2] could be used.\n5. **Typos**:\n    - Line 45: instruct tuning -> instruction tuning\n\nReferences\n\n[1] Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.\n\n[2] Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation Quality Estimation with Cross-lingual Transformers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5070–5081, Barcelona, Spain (Online). International Committee on Computational Linguistics.\n\nQUESTIONS:\nBesides some macro-level issues mentioned in the weaknesses, I still have some questions about some details:\n\n1. This paper mainly focuses on transferring to a single target language. Is it applicable for transferring to multiple target languages? If so, how to balance the data for multiple target languages?\n2. Line 170: A random strategy was used for data mixing. Would other mixing strategies affect the results?\n3. Line 131: Since the authors mentioned being inspired by mBART, is the special language ID independently occupying a token in the vocabulary?\n\nLIMITATIONS:\nThe authors discussed some limitations after the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
In this work, the authors present a method for transforming a chat-based English LLM to Non-English (Thai is the only language experimented with) based on a series of steps that teach the LLM to take in a non-English query and respond in non-English for that query. The methods presented is referred to as TransLLM pipeline which comprises of extending the based model vocab and finetuning with LoRA in multiple stages -- comprising of target language pretraining (on monolingual target language data), translation pretraining and transfer finetuning. The experiments are done on LLaMA2-Chat-7B with Thai as the target language and show the model performs well on both translation into and out of Thai and obtains good performance wrt baselines on MT-Bench & Alpaca-Eval.

STRENGTHS:
1. The paper presents a simple and scalable solution pipeline to adapt chat LLMs to new languages.

WEAKNESSES:
1. The experiments are done on only one LLM and on only one language (Thai). This severely constrains the extend to which the results could be verified.
2. The novelty of the proposed method is very thin and very limited analysis is done to motivate that novelty.

QUESTIONS:
1. Tables 4 and 5 show a large gap against GPT-4, yet GPT-4 KD is not as effective? Why is this the case?

LIMITATIONS:
Limitations have been adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a scenario as transforming a English-centric **chat** large language model to a non-English chat large language model (or not just en-centric). The authors want to address the catastrophic forgetting problem where further tuning on the original En-chat LLM without reusing their original SFT data will hurt their original chat abilities mainly in English. In terms of this, they introduce several techniques, including translation chain-of-thought, low-rank adaptation, as well as recovery KD, and they claim that with only single-turn Thai data, they can successfully transform English LLama2-chat-7B to Thai, while the performance in MTBench and some others remain competitive.\n\nSTRENGTHS:\nThe authors propose a pipeline to tune the chat English language model to a non-english language model. The pipeline looks interesting and the performance looks good in MTBench, Th-translated MTBench, etc.\n\nWEAKNESSES:\n1. To be honest, I am not sure whether this paper has good starting point. They focus on a scenario where you want to further fine-tune the chat model in English to a chat model in other languages. Could you provide more concrete application scenarios in your introduction? I am not convinced because typically, in practice, people will use all the data available to tune a base model, and usually, this renders the best downstream task performance, or at least, the robustness defined as the average downstream task performance of a particular tuned chat LLM. Why cannot we follow this paradigm?\n2. I suggest the authors to further polish the paper abstract. I am getting confused several times, for example, in your statement ``Transforming English-centric LLMs to non-English has been identified as an effective and resource-efficient method'', why can you simply say this is a resource-efficient method? As far as I know, if you just use knowledge distillation by distilling the data synthesized from a stronger model in your target languages, you might still need to use stronger model, usually those closed-source models, e.g., ChatGPT, this is still expensive, right? Besides, as even the knowledge distillation, in the SFT stage, you have to mix all the data to do the training as well. So it is still expensive.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}