PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning (2024)

Abstract: Instruction tuning has remarkably advanced large language models (LLMs) in understanding and responding to diverse human instructions. Despite the success in high-resource languages, its application in lower-resource ones faces challenges due to the imbalanced foundational abilities of LLMs across different languages, stemming from the uneven language distribution in their pre-training data. To tackle this issue, we propose pivot language guided generation (PLUG), an approach that utilizes a high-resource language, primarily English, as the pivot to enhance instruction tuning in lower-resource languages. It trains the model to first process instructions in the pivot language, and then produce responses in the target language. To evaluate our approach, we introduce a benchmark, X-AlpacaEval, of instructions in 4 languages (Chinese, Korean, Italian, and Spanish), each annotated by professional translators. Our approach demonstrates a significant improvement in the instruction-following abilities of LLMs by 29% on average, compared to directly responding in the target language alone. Further experiments validate the versatility of our approach by employing alternative pivot languages beyond English to assist languages where LLMs exhibit lower proficiency.1

heading: 1 Introduction
text: Instruction tuning has emerged as a crucial step in the evolution of generic AI assistants built atop large language models (LLMs) (Ouyang et al., 2022; Zhang et al., 2023b). Its fundamental principle involves fine-tuning LLMs to adhere to human instructions, thereby generating responses that are not only coherent but also aligned with the natural language directives. As a result, instruction-tuned
† This work was done when Zhihan and Dong-Ho were interns at Snap.
1Code and data are available at https://github.com/ ytyz1307zzh/PLUG.
models are able to solve a wide range of tasks given instruction-based prompts, without the need for task-specific adaptation (Chung et al., 2022; Mukherjee et al., 2023). Moreover, instruction tuning imparts LLMs with the capacity for human-like interactions, such as engaging dialogue with users (Xu et al., 2023b; Köpf et al., 2023).
Despite the great potential of instruction tuning, the aforementioned success is mainly made in highresource languages like English. As a result, its application in other lower-resource languages has raised interest within the multilingual research community. The straightforward strategy entails training LLMs to perform monolingual response generation – producing responses in the same language as the given instructions (Conneau et al., 2020; Ruder et al., 2021; Wei et al., 2023; Chen et al., 2023c). However, this endeavor is fraught with challenges.
7025
Although it elicits the capacity of LLMs to follow instructions in the target language, the response quality frequently falls short when compared to those produced for similar instructions in a highresource language. The primary reason for this discrepancy is the resource imbalance across different languages in the pre-training data (Touvron et al., 2023b; Wei et al., 2023), which leads to a significant disparity in LLMs’ foundational capabilities between high-resource and low-resource languages (Ahuja et al., 2023; Zhang et al., 2023c). Therefore, it is more challenging for LLMs to master instruction-following capabilities when trained to directly generate in a language that they are less familiar with.
Considering the superior capabilities of LLMs in high-resource languages, we propose a simple yet effective training approach that reflects the cognitive strategies humans use when learning a second language. Typically, human learners formulate their thoughts in their native language prior to expressing them in a less familiar language, as depicted in Figure 1. Drawing on this analogy, our training approach – pivot language guided generation (PLUG) – utilizes a high-resource language as a pivot language during response generation for the target language. Specifically, upon receiving an instruction in the target language, LLMs are trained to understand the instruction and formulate a response in the pivot language, before rendering the final response in the target language – all within one single pass of the LLM. A detailed illustration of our training format is presented in Figure 2. Intuitively, our training approach utilizes LLMs’ stronger capabilities of comprehending and executing the instructions in the pivot language, thereby guiding the model to produce higher-quality responses in the target language.
To demonstrate that LLMs generate better responses by leveraging the pivot language, we train LLMs with PLUG and evaluate their ability of following open-ended instructions. In light of the vacancy of high-quality multilingual evaluation data in this field, we create a benchmark of open-ended instructions, X-AlpacaEval, annotated by professional translators. We experiment with both the English-centric LLM, LLaMA-2 (Touvron et al., 2023b), and the multilingual LLM, PolyLM (Wei et al., 2023), primarily using English as the pivot language. Results from both model-based and human evaluation show that PLUG brings remarkable performance gains to LLMs in 4 distinct target
languages: Chinese, Korean, Italian, and Spanish. Compared to training with monolingual responses, PLUG brings an average improvement of 32% to the response quality on LLaMA-2 and 28% on PolyLM across these languages, with more notable improvements in relatively lower-resource languages. Besides, training LLMs with PLUG does not harm their original abilities in the pivot language. Beyond English, we reveal that other languages can also effectively function as pivot languages to enhance proficiency in languages where LLMs possess relatively limited capacity. Subsequent experiments validate that PLUG also enhances the truthfulness and reasoning abilities of LLMs in the target language, compared to the traditional monolingual response training.
In summary, our main contributions include: • We introduce PLUG, a simple yet effective
paradigm using a pivot language to assist instruction tuning in lower-resource languages.
• We establish X-AlpacaEval, a new evaluation benchmark to assess LLMs’ instructionfollowing abilities across multiple languages.
• Experiments demonstrate that PLUG significantly outperforms the traditional monolingual response generation, enhancing response quality in various target languages.

heading: 2 Related Work

heading: 2.1 Instruction Tuning
text: Recent research witnessed the emergence of foundation LLMs which are trained on massive amounts of textual data (Brown et al., 2020; Chowdhery et al., 2023; Touvron et al., 2023a). However, they are optimized primarily for next-token prediction on plain text, thus lacking the ability to interact with humans as AI assistants. To fill this gap, instruction tuning was proposed to align LLMs closely with human instructions, thereby converting them into generic assistants capable of handling diverse tasks (Wei et al., 2022; Ouyang et al., 2022). Early explorations in this field mainly focused on tuning LLMs on a variety of NLP benchmarks to unlock their zero-shot problem-solving potential (Sanh et al., 2022; Wei et al., 2022; Chung et al., 2022). Recent efforts have sought to extend model training to a wider range of general tasks, particularly by including user instructions to simulate real-world interactions (Taori et al., 2023; Xu et al., 2023a; Mukherjee et al., 2023). Our paper extends the current research to explore the unique
根据给定的信息撰写一份评价。 - 产品：Persil Discs洗衣凝珠，原 味，高效能（HE）兼容，62颗 装。 - 感受：好评
(Translation: Write a review according to the given information. - Product: Persil Discs Laundry Detergent Pacs, Original Scent, High Efficiency (HE) Compatible, 62 Count. - Sentiment: Positive)
根据给定的信息，用户似乎对使用“Persil Discs洗衣凝珠，原味，高 效能（HE）兼容，62颗装”的体验非常满意。他们的好评表明他们 对这些洗衣凝珠的性能和效果感到满意。
(Translation: Based on the given information, the user appears to be satisfied with the experience of using the product “Persil Discs Laundry Detergent Pacs, Original Scent, High Efficiency (HE) Compatible, 62 Count”. Their positive reviews indicate that they are satisfied with the performance and effectiveness of these laundry detergent pods.)
English Instruction: Compose a review based on the given information. -- Product: Persil Discs laundry detergent, original scent, high-efficiency (HE) compatible, 62 ct. -- Feeling: Positive
(Translation omitted: The Chinese response is generally consistent with the English response.)
English Response: I recently tried Persil Discs laundry detergent and I must say, I was thoroughly impressed! The original scent is pleasant and fresh, leaving my laundry smelling amazing. What's more, this detergent is HE compatible, making it a great choice for those with high-efficiency washing machines. The 62 ct packaging is also very economical, ensuring that I can do multiple loads of laundry without having to rebuy it. Overall, I would highly recommend it to anyone in search of an effective and affordable 中文回复：我最近试用了Persil Discs洗衣凝珠，我必须说，它给我 留下了深刻印象！它原始的香味清新宜人，让我的衣服散发出令人 愉悦的香气。此外，这款洗衣凝珠兼容高效能（HE）洗衣机，对 于那些使用高效能洗衣机的人来说非常合适。62颗的包装也非常经 济实惠，确保我可以洗很多次衣服，而不必重新购买。总体而言， 我会强烈推荐它给任何寻找有效又经济的洗衣解决方案的人。
Pivot Instruction
Pivot Response
Target Response Instruction
Target Response
根据给定的信息撰写一份评价。 - 产品：Persil Discs洗衣凝珠，原 味，高效能（HE）兼容，62颗 装。 - 感受：好评 Instruction
(Translation: Write a review according to the given information. - Product: Persil Discs Laundry Detergent Pacs, Original Scent, High Efficiency (HE) Compatible, 62 Count. - Sentiment: Positive)
Figure 2: The comparison between monolingual response training (top) and PLUG training (bottom). In this example, Chinese is the target language and English is the pivot. The monolingual response does not follow the review-writing instruction, while PLUG successfully generates a vivid and natural user review.
challenges when extending instruction tuning to lower-resource languages, where LLMs encountered more obstacles due to their relatively limited foundational capabilities in these languages.

heading: 2.2 Multilingual LLMs
text: With the success of English-based LLMs, there has been a push to develop multilingual counterparts to satisfy the needs of various linguistic communities. To begin with, a series of foundation LLMs are pre-trained on vast multilingual text corpora, such as XGLM (Lin et al., 2022b), BLOOM (Scao et al., 2022), and PolyLM (Wei et al., 2023). These models have opened avenues for multiple applications. For example, some researchers focused on further fine-tuning LLMs on large-scale translation corpora from external sources, either to improve LLMs’ translation capabilities (Jiao et al., 2023; Xu et al., 2023c), or to act as auxiliary tasks to support instruction tuning (Zhu et al., 2023; Ranaldi et al., 2023). However, our results in Table 1 suggest that auxiliary translation tasks do not necessarily enhance the open-ended generation abilities of LLMs without significant external translation data.
Another related direction is multilingual instruction tuning, where data are usually distilled from stronger LLMs like ChatGPT. This led to models like Phoenix (Chen et al., 2023d), Guanaco
(Cheung, 2023), and PolyLM-instruct (Wei et al., 2023). However, such instruction tuning is still constrained by an inherent barrier – the imbalanced foundational ability of LLMs across different languages, a consequence of the uneven distribution of languages in pre-training corpora. Our approach is orthogonal to the above ones which used monolingual response training, as evidenced by our experiments in Table 1 where PLUG training also improves the performance of PolyLM-instruct.

heading: 3 Pivot Language Guided Generation
text: Instruction tuning includes training an LLM on a set of instructions and their corresponding responses. In this work, we propose to utilize a pivot language, a language with more abundant resources and in which the LLM demonstrates better proficiency, to facilitate the instruction tuning of the lower-resource target languages.
Let (x, y) be an example in the instruction tuning dataset, where x is the instruction and y is the response. (xp, yp) represents its form in the pivot language, and (xt, yt) denotes its form in the target language. Traditionally, given an instruction in the target language, LLMs are trained to perform monolingual response generation (top half of Figure 2), i.e., the model is trained to directly predict the corresponding target response, or p(yt|xt). However,
this learning strategy usually encounters difficulties due to the limited foundational capabilities of the LLM in the target language, as it does not leverage the model’s high proficiency in the pivot language.
To lower the barrier of instruction tuning, PLUG trains the model to leverage the pivot language as the intermediary in the instruction-following process. Specifically, as shown in the bottom half of Figure 2, given the target language instruction xt, we train the model to first generate the pivot instruction xp and the corresponding pivot response yp, both in the pivot language, before generating the final response yt in the target language. In other words, the model is trained to predict p([xp; yp; yt] |xt) in one single pass, where semicolon represents sequence concatenation. Each component in the concatenated output starts with specific indicator tokens, such as English instruction or 中文回复 (Response in Chinese). Such tokens are used to structure the generation, and act as separators for extracting the target response yt as the final output. Please check Appendix A for additional details on training prompts.
PLUG reduces the difficulty of generating the target response as compared to monolingual response training, mainly because:
1. The model demonstrates a better understanding and execution of the given instruction when it is processed in the pivot language, rather than directly comprehending the original instruction in the target language.
2. The quality of the model’s generated response is superior when guided by its counterpart in the pivot language, as opposed to directly generating the target response.
Such relative ease of following instructions with PLUG is demonstrated by the example in Figure 2, where PLUG follows a review-writing instruction better than the monolingual response baseline.

heading: 4 Evaluation Settings
text: In our experiments, we primarily use English as the pivot language. We consider 4 distinct target languages for evaluation, including Chinese (zh), Korean (ko), Italian (it), and Spanish (es). These target languages are less represented than English in the pre-training data of most LLMs, including the ones we test with (§4.2).

heading: 4.1 Benchmarks
text: X-AlpacaEval Zero-shot open-ended generation in response to unseen instructions is a common testbed of instruction-tuned models (Zhou et al., 2023; Chen et al., 2023a). However, current multilingual instruction test sets are either small (Zhang et al., 2023a) or derived from noisy machine translation (Chen et al., 2023d). To address this, we introduce X-AlpacaEval, an extension of the Englishonly AlpacaEval (Li et al., 2023) test set2 to a multilingual benchmark. Specifically, we recruited professional translators from UpWork who are native speakers of the four target languages. We asked them to translate the English instructions into their native language, resulting in a high-quality benchmark of parallel instructions in 4 languages.
For evaluation, we follow the common approach of direct pair-wise comparison between responses generated by different models (Zheng et al., 2023; Wang et al., 2023b). In line with these works, we mainly utilize model-based evaluation with GPT4 as the judge. We also conduct human evaluation, where examples in each language are evaluated by two native speakers, and their judgments are combined as the final verdict. Details of how GPT-4 and human evaluations are conducted, including the rubric of combining judgments are in Appendix B.1. We quantify the performance discrepancy between models based on their win-loss rates in such comparison across all test instructions.
Truthfulness & Reasoning Benchmarks Besides assessing the helpfulness of LLMs through responding to general-domain instructions, we also evaluate whether PLUG improves LLMs’ truthfulness and reasoning abilities, via benchmarks TruthfulQA (Lin et al., 2022a) and SVAMP (Patel et al., 2021) respectively. Test questions in these benchmarks are translated to target languages by GPT-4, and evaluation is conducted in a zero-shot generative setting. For TruthfulQA, GPT-4 assesses model responses based on their truthfulness and informativeness. For SVAMP, we calculate the accuracy of the answers. Detailed evaluation metrics are explained in Appendix B.2 and B.3.
2AlpacaEval combines 805 instructions from 5 test sets: Self-Instruct (Wang et al., 2023c), Open Assistant (Köpf et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), and Anthropic’s helpful evaluation (Bai et al., 2022).

heading: 4.2 Model Settings
text: We experiment with three models: the Englishcentric foundation model LLaMA-2-13B (Touvron et al., 2023b), the multilingual foundation model PolyLM-13B, and its instruction-tuned version PolyLM-Instruct-13B3 (Wei et al., 2023). We use the GPT4-Alpaca (Peng et al., 2023) instructiontuning dataset (52k instructions) for training, and we employ ChatGPT to translate the original English examples into other languages. All models undergo training with identical hyper-parameters. They are trained in bfloat16 precision for four epochs with batch size 64. The learning rate peaks at 5e-6 with a warmup over the first 3% steps and a linear decay afterward. Greedy decoding is applied during inference to ensure deterministic generations. More training details are in Appendix D.

heading: 4.3 Methods to Compare
text: For each LLM evaluated, we train the model with the following methods. For simplicity, we use D(a, b) to denote a dataset of input a and output b. For example, D(xp, yp) refers to a training set of {(xp1, yp1), · · · , (xpn, ypn)}. • Pivot-only training. A.k.a. zero-shot cross-
lingual transfer, the model is trained only on the pivot language instructions D(xp, yp). • Monolingual response training. Trained on monolingual response data of both pivot and target languages, i.e., D(xp, yp) ∪ D(xt, yt).
3Named as PolyLM-MultiAlpaca in the original paper.
• Code switching. Additional cross-lingual alignment is performed by training LLMs to generate target language responses for pivot language instructions, and vice versa (Chen et al., 2023b). The final training set is D(xp, yp) ∪ D(xt, yt) ∪ D(xp, yt) ∪ D(xt, yp). • Auxiliary translation tasks. Recent works used an auxiliary instruction-style translation task to support instruction tuning (Zhu et al., 2023; Ranaldi et al., 2023). To test its effectiveness in our setting, we create a translation task based on our instruction tuning data. Specifically, we train the model to translate the instructions from pivot to the target language, and the same for the responses. The final training set is D(xp, yp) ∪D(xt, yt) ∪D([Ptrans;xp], xt) ∪ D([Ptrans; yp], yt), where Ptrans is the translation prompt and ; is string concatenation. • PLUG (our approach). Trained on monolingual response data for the pivot language, and the PLUG-formatted data for the target language, i.e., D(xp, yp)∪D(xt, [xp; yp; yt]). For xt, the target language response yt is extracted for comparison with the other baselines.

heading: 5 Results

heading: 5.1 Open-Ended Instructions
text: Pair-wise comparison results on X-AlpacaEval for target and pivot languages are detailed in Tables 1 and 2, respectively. Key findings are as follows:
PLUG training remarkably improves the instruction-following abilities of LLMs. As in-
dicated in Table 1, PLUG significantly and consistently boosts the response quality across all four target languages for the three tested LLMs. Compared with the most commonly used approach – monolingual response training, PLUG brings a notable average improvement of 32% to the instructionfollowing ability of LLaMA-2 across different languages, according to their win-loss differentials. Similarly, the performance gain is as high as 28% for PolyLM and 26% for PolyLM-Instruct. Conversely, adding an auxiliary translation task yields only marginal benefits over monolingual response training. Although improvements are made by introducing code-switching data, PLUG retains a substantial lead, outperforming this baseline by 21% for LLaMA-2 and 19% for PolyLM.
The improvements are especially pronounced for lower-resource languages. In comparison with monolingual response training, PLUG-trained models receive an average improvement of 46% when following instructions in Korean and 31% in Italian. These two languages are relatively less represented in the pre-training data for both LLaMA-2 and PolyLM, compared to Chinese and Spanish.
Furthermore, PLUG-trained models maintain their proficiency in the pivot language. Table 2 shows that the response quality of models trained with PLUG is comparable to those trained exclusively with pivot language data or monolingual responses. This preservation of LLMs’ capabilities in the pivot language is crucial as it guarantees the substantial improvements that PLUG brings to the target language responses.
Crucially, PLUG aligns model outputs more closely with human preferences. As shown in
Table 3, human judgments largely correlate with model-based evaluation, with PLUG-trained models consistently outperforming their monolingualtrained counterparts across all languages. The annotation agreement rate between humans and GPT-4 stands at 80.6%, closely mirroring the inter-human agreement rate of 78.0%, which validates the reliability of using GPT-4 as the judge. Detailed agreement scores are explained in Appendix C.2.
Besides these quantitative insights, we also included qualitative case studies in Appendix E, and an analysis of inference efficiency in Appendix C.4.

heading: 5.2 Study of Pivot Languages
text: To assess the versatility of PLUG training, we go beyond English and test whether other languages can serve as the pivot language. Here, we ensure a fair comparison by excluding D(xp, yp) from all training sets, thus using the same monolingual response baseline when alternating pivot languages.
Results on PolyLM, as in Table 4, convince our hypothesis. Since English dominates the pretraining corpus of PolyLM4, it is the most effective pivot language. Nevertheless, other languages yield tangible improvements in guiding the model’s relatively less proficient languages. For example, as the least represented language in the pre-training corpus of PolyLM, Korean receives an average 42% improvement when different pivot languages are employed. This proves that the effectiveness of PLUG is not language-specific. Besides the amount of pre-training data, the genetic similarity between languages also makes a difference, as Spanish is
4Languages in the order of its proportion in PolyLM’s pre-training corpus: en > zh > es > it > ko.
shown to be the second most effective pivot language (+17.5%) when the target language is Italian, outperforming the relatively higher-resource Chinese. Unsurprisingly, utilizing the LLM’s less proficient languages as pivots leads to diminished performance, e.g., Korean cannot serve as the pivot language for any other tested language.

heading: 5.3 Ablation Study
text: PLUG introduces the pivot instruction xp and pivot response yp into the generation process. To determine the impact of these two components, we carry out further ablation experiments.
Pivot Instructions To begin with, we experiment with removing the pivot instruction, training LLMs to directly generate a bilingual response – first in the pivot language, then in the target language. This variant, dubbed PLUG-PRO (Pivot Response Only), lags behind the standard PLUG approach, as evidenced in Table 5. This reveals that a model generates a better response if it first interprets the original instruction in the pivot language.
Delving into why this might be, we compare the quality of the pivot responses within the bilingual outputs of PLUG and PLUG-PRO. Evidence from Table 6 suggests that the model generates a better pivot response if the preceding instruction is in the pivot language (PLUG) instead of the target language (PLUG-PRO). This improvement in the pivot response quality is pivotal to enhancing the final response in the target language.
Pivot Responses Next, we examine the importance of the pivot response by comparing PLUGPRO with monolingual response training. Accord-
ing to the superior performance of PLUG-PRO in Table 7, the inclusion of the pivot response is a crucial contributor to the models’ improvements. This demonstrates that the preceding pivot response provides valuable guidance for the subsequent response in the target language.
To summarize, omitting either the pivot instruction or the pivot response undermines the efficacy of our approach, with the pivot response being particularly influential.

heading: 5.4 Data Efficiency of PLUG
text: We further explore training PLUG with a smaller amount of data, as illustrated in Figure 3. Impressively, models trained with a mere 2k samples of PLUG data surpass the performance of conventional baselines trained with significantly larger datasets, including those trained with a full set of 96k monolingual response data. These results demonstrate the remarkable data efficiency of PLUG which leads to strong instruction-following abilities of LLMs even with a minimal amount of training data. In contrast, training LLMs with extensive volumes of monolingual response data results in only modest performance. Besides, PLUG also benefits from increased data sizes. Expanding PLUG’s training set from 2k to 96k results in larger performance improvements, underscoring the scalability and effectiveness of our method.

heading: 5.5 Comparison against Translation-Based Approaches
text: The goal of our research is to enhance a given LLM’s capability to understand instructions and generate responses in a target language. Nevertheless, an alternative method to perform response generation might be the use of an external machine translation (MT) model for the conversion between pivot and target languages. Therefore, we compare our PLUG-trained PolyLM models against a round-trip translation pipeline which consists of 3 steps: (1) use the MT model to translate the instruction from the target language to pivot language; (2) generate a response in the pivot language with the LLM; (3) call the MT model again to translate that response back to the target language. As for the MT model, we experiment with two options: (1) NLLB-3.3B (Costa-jussà et al., 2022), the state-ofthe-art MT model covering 200+ languages, and (2) prompting PolyLM-Instruct to do translation because it shares the foundational multilingual capacities with our PLUG-trained PolyLM model.
As Table 8 demonstrates, PLUG models typically outperform their translation-based counterparts. The exception in Korean when compared against the NLLB-based approach is likely due to PolyLM’s limited foundational proficiency in that language. This is supported by the fact that our model outperforms the other baseline with PolyLMInstruct as the translator, given that both have comparable foundational abilities in Korean.
With further inspection, we find that the efficacy of PLUG extends beyond mere translation. The generation of the final response yt is a confluence of instruction following and language transformation, influenced by all preceding contexts including xt, xp, and yp. To verify this hypothesis, we consider a response translation approach that directly translates the pivot response yp – extracted from a
complete PLUG response – into the target language. As Table 9 indicates, such an approach does not match the original response quality of PLUG. This shows the importance of the preceding contexts in shaping the final response in the target language.

heading: 5.6 Truthfulness & Reasoning
text: Training with PLUG not only improves the helpfulness of LLMs in responding to general-domain instructions, but also enhances their truthfulness and reasoning abilities when answering factual and math-related questions respectively. This is evidenced by the notable improvements on TruthfulQA and SVAMP shown in Figure 4, where PLUG significantly improves the performance of LLaMA-2 across all target languages, compared to monolingual response training. For instance, PLUG brings a relative improvement of 39.9% in Korean and 31.7% in Chinese on TruthfulQA, as well as 12.1% in Spanish on SVAMP. These results suggest that PLUG training is able to utilize the LLM’s superior ability in the pivot language to generate more truthful responses to factual questions, as well as more accurate reasoning toward math problems. Corresponding results on PolyLM are presented in Appendix C.3.

heading: 6 Conclusion
text: In this work, we introduced PLUG, a simple yet effective approach of utilizing a higher-resource pivot language to facilitate the instruction tuning of LLMs on lower-resource languages. Extensive experiments on 4 distinct target languages confirmed the effectiveness of PLUG. Notably, PLUG brought
considerable enhancements to the response quality for open-ended instructions, when compared to the conventional strategy of monolingual response training. Furthermore, languages besides English can also act as pivot languages, enhancing the instruction-following capabilities of LLMs in their relatively weaker languages. Additionally, PLUG also led to a promising increase in the truthfulness and reasoning ability of LLMs.
Limitations
To our knowledge, this work has the following limitations:
• A noted limitation of PLUG arises with extremely long instructions, where generating a lengthy pivot instruction could be inefficient or exceed length constraints. Extrapolating from the findings in §5.3, using PLUG-PRO might be a workaround, which only generates the pivot response and then the target response. PLUG-PRO is able to circumvent sequence length limitations in long-context tasks, albeit sacrificing some performance of PLUG.
• Our research only encompassed Chinese, Korean, Italian, and Spanish, due to the high cost of conducting GPT-4 evaluations and recruiting human workers in this study. Nevertheless, the chosen languages encompass a broad linguistic range, including both Latin-scripted and non-Latin languages, as well as languages with varying degrees of resource availability within the training corpora of LLaMA-2 and PolyLM, such as the higher-resource Chinese and Spanish, and the lower-resource Italian and Korean.
Ethical Considerations
We discuss the ethical considerations of this work from the following perspectives:
• In this work, we introduce PLUG, a novel training method for instruction tuning LLMs in different languages. While PLUG represents an innovative approach to LLM tuning, it is essential to acknowledge that it operates on existing pre-trained LLMs. Consequently, the models enhanced through the PLUG method may inherit potential risks associated with these LLMs, such as hallucination and toxicity, stemming from their original pre-training. In §5.6, experimental evidence suggests that PLUG improves the truthfulness of LLMs
in target languages, thus partially mitigating these risks. However, we recognize that the effective solution to these issues involves rigorous safety fine-tuning of the models (Touvron et al., 2023b). This aspect, while crucial, falls outside the scope of this paper, but is a significant area for future exploration to ensure the responsible deployment of LLMs.
• In our research, we primarily employ English as the pivot language to facilitate LLMs’ instruction tuning in lower-resource languages. Such a choice is influenced by the superior proficiency of pre-trained LLMs in English due to its extensive resource availability. We acknowledge the potential bias this approach might introduce by favoring English linguistic features. In §5.2, we have explored using other languages as the pivot, yielding promising results. This demonstrates that if LLMs specialized in other languages exist, these languages can effectively serve as pivot languages. Our experiments support the extrapolation that the efficacy of a pivot language is contingent on the model’s language proficiency, rather than the language itself. We are committed to continually adapting our methods to ensure a balanced and inclusive approach in tuning LLMs, aiming to minimize linguistic bias and enhance the representation of diverse languages in this field.

heading: Acknowledgements
text: This work was supported by NSF IIS-2119531, IIS-2137396, IIS-2142827, IIS-2234058, CCF1901059, and ONR N00014-22-1-2507.

heading: A Training Prompts
text: During instruction tuning, PLUG is applied when dealing with instructions in the target language, and direct monolingual response generation is used when encountering instructions in the pivot language. To control the model’s behavior, we utilize the following system prompts:
When given an instruction in the pivot language: Please respond to the following user message in [pivot]. When given an instruction in the target language: Please interpret the instruction in [pivot], and then respond both in [pivot] and in [target].
where [pivot] and [target] are names of pivot and target languages, respectively. We formulate the whole training example in the format below:

 System Prompt 
 Instruction 
 Response
Following standard approaches (Touvron et al., 2023b; Wang et al., 2023b), we only compute the loss on the output tokens, i.e., tokens after 
.

heading: B Evaluation Settings and Prompts
text: B.1 X-AlpacaEval
Model-based Evaluation Judging open-ended model generations with GPT-4 as the evaluator is increasingly recognized for its cost efficiency, interpretability, and decent agreement with human evaluators (Zheng et al., 2023; Li et al., 2023; Es et al., 2023; Liu et al., 2023). In this paper, we follow this paradigm and use the pair-wise comparison setting and evaluation prompts of Zheng et al. (2023). Specifically, given responses of two models, GPT-4 is asked to identify which model’s response better follows the user’s instruction, or to declare a tie when the response quality is comparable. GPT-4 is also asked to provide a rationale for its decision. We use OpenAI’s gpt-4-0613 model for all evaluation. The full evaluation prompt is shown in Figure 6.
To assess the results of a pair-wise comparison, we count the win rates of each model across all test instructions. Since LLM evaluators could be vulnerable to positional biases (Wang et al., 2023a), the order of responses is swapped for a second round of evaluation. We use a scoring system where s = 1 indicates a preference for PLUG and
s = −1 indicates a preference for the baseline, and the final verdict is made based on the following rubric:
Verdict =    PLUG wins if s1 + s2 > 0, Baseline wins if s1 + s2 < 0, Tie if s1 + s2 = 0. (1)
Here, s1 and s2 are the scores from the first and second rounds of evaluation, respectively, where the order of responses is alternated in each round.
Human Evaluation While we use GPT-4 as the judge in most experiments, we also conduct human evaluation to enhance the validity of our findings. For this purpose, we engage native speakers from UpWork who have extensive experience in annotating AI-generated text. We randomly sample 80 instructions from X-AlpacaEval, and evaluate the responses from both LLaMA-based models and PolyLM-based models on these instructions. Each test example is annotated by two individuals to reduce subjectivity. In the same pair-wise comparison as GPT-4 evaluation, annotators are tasked with choosing the better response from two modelgenerated answers or indicating a tie. We use the same scoring rubric as (1) to combine the decisions from two annotators. Each annotator receives 160 USD for their work based on mutual agreement, which is calculated at an approximate rate of 32 USD per hour. The detailed instruction provided to the human annotators is shown in Figure 9.
B.2 TruthfulQA TruthfulQA (Lin et al., 2022a) contains 817 questions across 38 categories, aiming to identify falsehoods and misconceptions in model answers. We translate the original questions from English into our target languages using GPT-4. Since PLUG is designed for zero-shot response generation, we evaluate in a zero-shot generative setting, where LLMs are directly given the question to generate a free-form answer. The generated answers are evaluated based on two criteria: truthfulness and informativeness. While some approaches (Lin et al., 2022a; Touvron et al., 2023b) used fine-tuned GPT3 as the evaluator, in this paper, we utilize GPT-4 because the original fine-tuned models are not released and GPT-4 is more suitable and robust in judging free-form answers. In English, the TruthfulQA score of our PLUG-trained LLaMA-2-13B is about 60%, as evaluated by GPT-4. This is on par with the reported score of LLaMA-2-chat-13B
in Touvron et al. (2023b) evaluated by fine-tuned GPT-3, which reflects the reliability of GPT-4 evaluations.
For assessing truthfulness, GPT-4 compares a model’s response against a list of correct and incorrect answers. A response aligning more closely with correct answers indicates truthfulness, while a closer alignment with incorrect answers suggests the presence of hallucinations. Refusing to answer is considered as truthful since it does not provide misinformation. Responses that do not clearly match either list are deemed “not sure”. The prompt for GPT-4 to evaluate truthfulness is in Figure 7.
To prevent the model from consistently refusing to answer, informativeness is also evaluated. This involves checking whether the response contains relevant information to the question. The prompt for GPT-4 to evaluate informativeness is in Figure 8. Finally, we calculate the percentage of generations that are deemed both truthful and informative as the overall performance.
B.3 SVAMP
SVAMP (Patel et al., 2021) is a benchmark with 1,000 math questions that require single-step or multi-step reasoning. Same as TruthfulQA, we test models under a zero-shot generative setting, where models are directly asked the question, followed by a prompt such as “Think step-by-step before reaching the final answer” to elicit chainof-thought reasoning (Kojima et al., 2022). The English question is translated into target languages by GPT-4. After the model responds, we utilize GPT-3.5 to extract the final answer from the model response and compare it with the ground-truth answer. Accuracy is calculated as the reflection of the model’s reasoning ability.

heading: C Additional Experiments
text: C.1 Response Quality in Pivot Language
Besides the comparisons with monolingual response models and pivot-only models in Table 2, we compare PLUG with other baselines mentioned in §4.3 on their capabilities in following pivot language instructions. As Table 10 suggests, the trend is consistent with Table 2, demonstrating that PLUG preserves the proficiency of LLMs in the pivot language.
Comparison zh ko it es
LLaMA-2-13B
PLUG vs. Mono.+ Translation +4.8 +4.8 +5.8 +8.1 PLUG vs. Mono.+ Code-Switch +3.9 +1.2 +3.2 +0.1
PolyLM-13B
PLUG vs. Mono.+ Translation +2.1 +6.3 -6.2 +7.0 PLUG vs. Mono.+ Code-Switch +5.3 +8.3 -3.9 +4.1
PolyLM-Instruct-13B
C.2 Inter-Annotator Agreement
In our X-AlpacaEval experiments, we assessed the level of inter-annotator agreement from two perspectives: the agreement between human annotators and the agreement between human evaluations and GPT-4’s judgments5. As shown in Table 11, the human-GPT agreement is comparable to the agreement between humans, achieving 61.9% when including tie votes and 80.6% when excluding ties. This indicates that GPT-4 is as reliable as humans in judging open-ended generations which is usually considered a highly subjective task. Additionally, we witness similar levels of agreement between our experiments and the ones on MT-Bench (Zheng et al., 2023), which further echoes the validity of our findings.
C.3 TruthfulQA Results on PolyLM
As a supplement to §5.6, we present the results of PLUG-trained PolyLM on TruthfulQA in Figure 5. Same as the trend of LLaMA-2 in Figure 4, after utilizing the pivot language to guide the response generation, PLUG improves the truthfulness of PolyLM responses in all target languages.
5We calculate human-GPT agreement for both human judges and average them.
For example, the relative improvement is as high as 36.1% in Korean and 21.6% in Italian. SVAMP is not tested on PolyLM due to its limited reasoning ability, which only achieves ~10% accuracy in English.
C.4 Inference Efficiency
As an additional analysis, we compare the number of tokens produced by different models during inference. Due to the introduction of pivot instruction and pivot response, PLUG-trained models generate longer sequences than their counterparts that produce monolingual responses, as shown in Table 12. The number of tokens is usually not doubled, especially in Chinese and Korean, because English is better tokenized than these languages.

heading: D Training Settings
text: In our experiments, we test PLUG on the 13B variants of LLaMA-2 and PolyLM. We find the appropriate hyper-parameters for tuning LLMs on our dataset through preliminary experiments, leveraging a validation set of instructions held out from the training data. Afterward, we fix the hyperparameters for all models. They are trained in bfloat16 precision for four epochs with batch size 64. The learning rate peaks at 5e-6 with a warmup over the first 3% steps and a linear decay afterward.
LLaMA-2 and PolyLM models are trained with max sequence lengths 4096 and 2048, respectively. Such training lasts for ~24 hours on 8×A100 GPUs, making use of the DeepSpeed library (Rasley et al., 2020) and the stage 3 of ZeRO optimizer (Rajbhandari et al., 2020). We fix the random seed for all models due to the high cost of GPT-4 evaluation.

heading: E Case Study
text: To get a better sense of how PLUG improves model response, we sample cases from different languages as a qualitative study. We show cases where PLUG helps with the detailedness (Table 13) and factuality (Table 14) of the response, as well as its consistency (Table 15) and relevance (Table 16) to the instruction.

heading: Future Work
text: This aspect, while crucial, falls outside the scope of this paper, but is a significant area for future exploration to ensure the responsible deployment of LLMs. • In our research, we primarily employ English as the pivot language to facilitate LLMs’ instruction tuning in lower-resource languages. Such a choice is influenced by the superior proficiency of pre-trained LLMs in English due to its extensive resource availability. We acknowledge the potential bias this approach might introduce by favoring English linguistic features. In §5. 2, we have explored using other languages as the pivot, yielding promising results. This demonstrates that if LLMs specialized in other languages exist, these languages can effectively serve as pivot languages. Our experiments support the extrapolation that the efficacy of a pivot language is contingent on the model’s language proficiency, rather than the language itself. We are committed to continually adapting our methods to ensure a balanced and inclusive approach in tuning LLMs, aiming to minimize linguistic bias and enhance the representation of diverse languages in this field