Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model (2024)

Abstract: While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely “superficial”. We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation. Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named PRETTY, which employs minimal taskrelated prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across eight languages demonstrate the efficacy of PRETTY in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can achieve performance comparable to their SFT counterparts. This method presents a costeffective alternative to SFT and advances the democratization of multilingual LLMs1.

heading: 1 Introduction
text: Supervised fine-tuning (SFT) refines large language models (LLMs) using task-specific instruction data to enhance their capability to follow instructions (Touvron et al., 2023; Peng et al., 2023) and to align their outputs with human preferences and safety considerations (Ouyang et al., 2022; Rafailov et al., 2023; Dong et al., 2023b; Yuan
*Work was done during a visit to Westlake University. Co-corresponding authors.
1The code and data are available at: https://github. com/NLP2CT/PrettyAlign
et al., 2023; Lan et al., 2024). This process is often termed “alignment”, signifying the tailoring of model outputs to conform to specific downstream requirements. Nevertheless, current research casts doubt on the necessity and potential adverse impacts of SFT. But the alignment achieved through SFT is often considered to be “superficial”, with the process potentially repurposing pre-existing knowledge from pre-training to merely reshape outputs to meet specific criteria (Zhou et al., 2023; Lin et al., 2023). It has been observed that even a small-scale SFT training dataset can produce significant alignment effects (Liu et al., 2023; Xia et al., 2024). On the other hand, recent empirical studies (Luo et al., 2023; Dong et al., 2023a) have raised concerns that SFT might hurt the knowledge acquired during its pre-training phase, leading to serious consequences like catastrophic forgetting.
Not only is there no definitive consensus on the necessity of SFT, but the majority of these studies also focus on monolingual tasks. LLMs still encounter challenges in handling complex crosslingual generation tasks (Schioppa et al., 2023; Wang et al., 2023). Current research on crosslingual alignment primarily seeks to extrapolate or align English capabilities to other languages using the SFT paradigm (Zhang et al., 2023; Chai et al., 2024; Xu et al., 2024), yet there remains a gap in exploring the specific impacts of SFT-based cross-lingual alignment. Furthermore, given the potential risk of SFT leading to the forgetting of pre-training knowledge, the question of how to achieve cross-lingual alignment without training remains underexplored.
To bridge these gaps, our study conducts an indepth examination of the impact of SFT on crosslingual generation. We investigate the influence of SFT on the decoding patterns of foundation models in cross-lingual contexts, hypothesizing that the success of SFT largely hinges on the selection of initial prior tokens that are critical for eliciting task-
12131
Instruction: Translate the following sentence from English to Ukrainian: “We now have 4-month-old mice that are non-diabetic that used to be diabetic,” he added.

heading: 2) Propose a training-free alignment
text: specific generation in the target language. Furthermore, the observed decoding similarities between foundation and SFT models support the extension of the superficial alignment hypothesis to crosslingual scenarios. Responding to these insights, we introduce a training-free alignment method named “PRETTY” for cross-lingual and non-English tasks. The Prefix TexTs act as a Yarn (PRETTY) linking the foundation LLM and the SFT LLM, eliciting the foundation LLM to exhibit near-SFT performance levels. Specifically, we augment the original input with a few tokens that serve as decoding priors, and then prompt the foundation LLM to resume decoding based on this modified input. In most cases, only one or two task-related prior tokens are needed, and the method for constructing these prior tokens is flexible across various kinds of language resources, fostering the democratization of multilingual LLMs.
We conducted experiments on machine translation (Goyal et al., 2022), cross-lingual summarization (Bhattacharjee et al., 2023) and non-English part-of-speech (POS) tagging (Liang et al., 2020) tasks across eight languages. These tasks exemplify cross-lingual generation and multilingual language understanding, and they provide ample nonEnglish test data to evaluate effectiveness across varying levels of resource availability. The experimental results demonstrate that PRETTY can ef-
fectively align the foundation model to match SFT model’s performance without training, by merely adding two prior tokens in the decoding.

heading: 2 Iceberg Model of SFT

heading: 2.1 Preliminaries
text: Pre-training The pre-training (PT) of LLMs is primarily conducted through language modeling tasks on large-scale unlabeled data (Touvron et al., 2023; Achiam et al., 2023). During this phase, given a sequence XPT of length N and a context window k, the optimization objective is maximizing the joint probability PLM as:
PLM(XPT) = N∏
i=1
P (xi|xi−k:i−1) (1)
which encourages the model to generate text that naturally follows from the preceding context. However, this “text completion” behavior can become a bottleneck when models are prompted to switch languages or follow specific instructions of crosslingual generation. It is frequently observed that when prompted with English input and instructed to produce text in a different language, as illustrated in the upper example of Figure 1, the foundation model often continues to decode in English.
SFT SFT leverages labeled data pair (Xins., Y ) to empower models with the ability to follow in-
structions. This stage aims to maximize the probability of the expected answer Y conditioned on the input text Xins., where Xins. consists of the task instruction and task input.
PSFT(Y |Xins.) = T∏
j=1
P (yj |y1:j−1, Xins.) (2)
SFT is crucial for aligning foundation models to perform task-specific instructions, effectively transforming a general-purpose LLM into an instructionfollowing assistant. However, data quality, training costs, and the imbalance of multilingual data hinder the democratization of assistant LLM. As mentioned before, SFT may be harmful to pre-training knowledge. Thus, it is meaningful and important to understand the underlying mechanism of SFTbased alignment and propose a more efficient alignment method.

heading: 2.2 Beneath the SFT-based Alignment
text: Prior Knowledge Hypothesis It is worth noting that pre-training corpora also contain sequences that naturally express task-specific information, which imparts certain capabilities to the foundation LLMs. For example, the presence of semantically equivalent expressions in the pre-training text may enable LLM acquire machine translation ability during pre-training stage (Radford et al., 2019).
Despite its extensive prior knowledge, the foundation LLM still struggles with complex crosslingual generation tasks. Beyond existing studies, we provide more concrete insights into this issue by prompting foundation LLMs with various instructions (Bawden and Yvon, 2023). Notably, only 31.8% of these prompts successfully elicit translation capability from the foundation LLMs2.
This deficiency may stem from two main factors: First, the proportion of text with the aforementioned characteristics in the pre-training corpus XPT is still relatively small, and most of it is far from resembling human instruction text Xins.. Consequently, the model is more likely to predict tokens suitable for completing formal texts than those required for task-specific instructions. As a result, the foundation LLM often fails to produce tokens y ∈ Y1:T in the intended target language. Secondly, the predominance of English in the pretraining data skews the token generation probabilities of foundation LLM. Given a cross-lingual context, the model favors predicting tokens in English,
2For detailed information, please refer to Appendix C.3.
while the token probabilities for other languages remain comparatively low. For example, English data comprises up to 90% of the Llama2 pre-training data (Touvron et al., 2023), which may lead models to generate text with an English-centric bias.
The above hypothesis might be reasonable when we revisit Equation (1) and Equation (2). The probability PLM(XPT) of the next token prediction for the foundation model is conditioned on the distribution of the pre-training text XPT. SFT narrows the probability space for token selection, adjusting the parameters to better align with the distribution, i.e., the probability PSFT(y|Xins.) is conditioned on the distribution of the instruction text Xins..
Experimental Settings To validate the aforementioned hypothesis, we selected the representative cross-lingual task of machine translation as our analytical testbed. The main research method involved quantifying the differences and similarities in the decision space and token selection behavior between the foundation LLM and the SFT-aligned LLM. For the model selection, we chose the foundation Llama2 7B model and conducted supervised fine-tuning on it using the Alpaca dataset3(Taori et al., 2023). The optimization was carried out using a cosine learning rate scheduler, with the maximum learning rate set to 2e−5 and a warmup ratio of 0.03. Training was performed on two NvidiaH800 GPUs using LoRA parameter-efficient finetuning (Hu et al., 2022) technique, with a cumulative batch size of 64. Other hyper-parameters follow those of the original Alpaca settings.
3https://github.com/tatsu-lab/stanford_alpaca
A Prior Token Elicits Silent Majority Inspired by the categorization of token shifts by Lin et al. (2023), we propose to quantify the agreement of token selection between foundation LLM θPT and SFT LLM θSFT. Given the same prefix input X̂ , we aim to measure whether the next token selected by the SFT LLM, ySFT, is among the top-K tokens, yPT, with the highest probabilities in the decision space of the foundation LLM, which can be formally expressed as follows:
ySFT = argmax y∈V
P (y|X̂; θSFT)
yPT = {y| arg topK y∈V P (y|X̂; θPT)}
AggrementK = 1
L
L∑
l=1
1ySFT∈yPT (3)
where V is the vocabulary shared by two models, and L is the length of the dataset.
We compare the agreement of the token selection made by the models under the same prefix text X̂ in two different experimental setups. The first setup uses the instruction text as the prefix, i.e., X̂ = Xins.; the second takes the first token decoded by the SFT model as a prior token, appending it to the original instruction prefix, i.e., X̂ = [ Xins., y (1) SFT ] . For the SFT model, the second setup is equivalent to continuing its own decoding behavior, whereas for the foundation model, it becomes decoding with the addition of a prior token.
Figure 2 illustrates the agreement between the foundation model’s predictions and those of the SFT model regarding the selection of the next token, given an identical text prefix. Across the entire translation data, it is observed that after incorporating merely one prior token, the foundation model exhibits a high degree of agreement with the SFT model in terms of token selection. This demonstrates that the alignment effect of SFT in crosslingual generation tasks is also somewhat superficial. Even in instances where the token with the highest probability differs between the two models, 90.8% of the tokens chosen by the SFT model are present within the “silent majority” in the decision space of the foundation model, specifically, among the top 20 most probable token choices.
Lens of Distribution Instead of focusing on the coverage of token selection outcomes, we also observe the decision dynamics and similarities from the perspective of the overall probability distribution, with the data settings consistent with the previous setup. First, as shown in Figure 3, after adding a prior token, the probability of the next tokens chosen by both models have closely aligned distributions. The reason that the foundation model
exhibits a high probability given the instruction text as a prefix lies in a preference for choosing to continue the instruction text rather than completing the cross-linguistic semantic transformation. Additionally, we quantify the distribution disparities between the two models through the probability distribution of the vocabulary. The disparity metrics used include Kullback-Leibler (KL) divergence, Jensen-Shannon (JS) divergence, and cross-entropy (Kullback, 1997). As depicted in Figure 4, the disparity of decision space of the foundation model significantly decreases after adding the prior token, aligning more closely with the SFT model.
These findings indicate that such prior tokens serve a dual function: they not only steer the foundation model towards generating tokens pertinent to cross-lingual generation but also modulate the decision space to align more closely with the taskspecific distribution.

heading: 3 Pretty: Prefix TexT as a Yarn

heading: 3.1 Motivation
text: The observations discussed earlier confirm that SFT effectively narrows the decision space of the foundation model during text generation that is conditioned on instruction text. The disparity in token selection between the foundation LLM and the SFT LLM, however, might not be reduced by a trainingbased transfer methodology. By appending a prior token into the instruction text, the choices of the next token between the two models tend to become largely consistent, and in the vast majority of cases, the tokens chosen by SFT model are also found within the high-probability candidate words of foundation model. These phenomena show that the alignment elicited by SFT is somewhat superficial in cross-lingual generation tasks and motivate us to propose a training-free alignment method by leveraging these prior tokens.

heading: 3.2 Formulation
text: Upon revisiting Equation (1) and Equation (2), the goal of proposing a training-free approach is to enable the conditional decoding probability of foundation model to approximate those of SFT model. Therefore, ideally, the selected prior tokens Xpri. = {xpri.} may satisfy the following criteria:
P (yPT| [Xins., Xpri.] ; θPT) ≈ P (ySFT|Xins.; θSFT) (4)
where yPT and ySFT represent the outputs of the foundation and the SFT models, respectively. It is important to note that a single prior token may not serve as an optimal solution due to its non-derivable characteristic. Hence, we extend our methodological approach to include appending multiple prior tokens, grouping them to form a prefix text.

heading: 3.3 Construction of Prior Tokens
text: To ensure that the proposed method is applicable to a wide array of languages, we propose three construction strategies based on the availability of language resources, aiming to guarantee the universality of our approach.
SFT Prior represents an ideal scenario where the first few tokens generated by a SFT model are used as priors. This method is theoretically rational when the SFT model is derived from the same foundation model because it directly approximates Equation (4) by sampling xpri. ∼ {ySFT}. In practical applications, this might be suitable for high-resource languages due to the imbalanced language capabilities of other languages. Additionally, SFT could potentially degrade the knowledge and abilities that the foundation model has already acquired. In such cases, using prior tokens from the SFT model can contribute to generating better results. This situation will be discussed further in the subsequent section.
Refined Prior is more readily accessible for most languages and tasks. We can utilize the output tokens generated by a smaller model trained for specific downstream tasks and use them as prior tokens to achieve weak-to-strong generalization (Burns et al., 2023).
Pseudo Prior For extremely low-resource language pairs, where there is no labeled data for downstream tasks, both SFT and Refined priors are difficult to obtain. For cross-lingual tasks, we can create pseudo labels in target language as prior tokens. For instance, in machine translation tasks, we might use bilingual dictionaries to acquire pseudo prior tokens. However, the quality and accuracy of pseudo labels remain uncertain, and the extent of their impact on the generative performance of the foundation LLM is not yet clear. We will explore this problem further in the context of experimental results discussed later in the paper.

heading: 4 Experiments
text: We examine the effectiveness of our proposed training-free alignment method on two distinct tasks: machine translation, cross-lingual summarization and non-English POS tagging. Machine translation serves as a prototypical cross-lingual generation task, entailing the transformation of a sequence from a source language to a target language (Bahdanau et al., 2015; Vaswani et al., 2017; Zhan et al., 2023; Pang et al., 2024). As for cross-lingual summarization, it requires the model to generate a summary of an article in a different language (Bhattacharjee et al., 2023; Chen et al., 2023). Although POS tagging (Manning, 2011; Nivre et al., 2017; Chiche and Yitagesu, 2022) primarily assesses the model’s ability to understand monolingual text, we include it as multilingual experiments to show the universality of our methods.

heading: 4.1 Experimental Settings
text: Data We use Flores-101 (Goyal et al., 2022), CrossSum (Bhattacharjee et al., 2023) as benchmarks for machine translation and cross-lingual summarization tasks, respectively. For POS tagging tasks, we choose the POS test split from the XGLUE benchmark (Liang et al., 2020), which is derived from the Universal Dependencies Treebank v2.5. To investigate the performance across various resource languages, we carefully selected eight languages based on the pre-training data proportions disclosed in the Llama2 technical report (Touvron et al., 2023). These languages are French, German, Chinese, Russian, Ukrainian, Portuguese, Hindi and Arabic. Among these, the first four languages account for more than 0.1% of the pretraining data of Llama2, while Ukrainian and Portuguese fall below 0.1%, Hindi and Arabic is below 0.05%. For the Llama2 model, we can categorize these three types of languages into high-resource languages, low-resource languages, and extremely low-resource languages, respectively.
Models and Baselines The settings of Llama2 foundation model and the SFT model are consistent with those described in Section 2.1. To further demonstrate the generality of our proposed method, we incorporated the Mistral-7B LLM family (Jiang et al., 2023) into our experiments, covering both out-of-the-box SFT and foundation models.
In the machine translation task, the Llama2 foundation model does not tend to generate translations when given explicit translation instructions.
While this is a normal phenomenon according to our previous discussion, to ensure a fair comparison, we also searched for a better prompts for the foundation model. This prompting approach is referred to as “Llama2-7BPROMPTING” in subsequent sections. For POS tagging, we experimented with various instructions and selected one that consistently prompts both the foundation model and the SFT model to reliably generate classification results in text. Although we report the zero-shot performance for the aforementioned tasks, we found that even out-of-the-box SFT models cannot produce stable output for cross-lingual summarization task. Hence, we prepend a constant demonstration before the input to also assess the effectiveness of our proposed method under the in-context learning paradigm (Dong et al., 2023c).
Sources of Prior Token The sources of crafting prior tokens include:
• SFT Prior: We took the first k tokens of output produced by SFT model as the prior tokens. For multiple SFT models, we select the model that demonstrates better performance.
• Refined Prior: We use downstream task models with smaller parameter sizes as the source of refined priors. For the different tasks, we utilized the distilled 600M variant of NLLB-200 translation model4(Costajussà et al., 2022), mT5 cross-lingual summarization model5 and the Unicoder-NLU model6(Huang et al., 2019), respectively.
• Pseudo Prior: The pseudo prior is applied to two cross-lingual tasks since it can utilize cross-lingual language resources. We create pseudo prior tokens for machine translation task by referencing dictionary 7 entries. For cross-lingual summarization, we initially extract keywords from each passage using KeyBERT (Grootendorst, 2020) and then perform word-by-word translation. However, not all initial sentence tokens will be covered by the dictionary. To handle such instances, a backoff strategy is implemented, where the target language equivalent of the first available dictionary token is used as the prior token.
4https://huggingface.co/facebook/ nllb-200-distilled-600M
5https://hf.co/csebuetnlp/mT5_m2m_crossSum 6https://github.com/microsoft/Unicoder/ 7Please refer to Appendix C.4 for dictionary information.
For two cross-lingual task, the first k = 2 tokens are chosen as the prior tokens. This helps to avoid inadequate guidance from single non-informative tokens like punctuation or numbers. In the case of the pseudo prior, due to the back-off strategy, only one token is used for fair comparison. For POS tagging task, the strategy is more straightforward with only the first k = 1 label considered as the prior token.

heading: 4.2 Evaluation
text: To ensure the integrity of the output data from all models, we standardized the output by cleaning it in accordance with the specific output style of each model. Subsequently, we conducted a manual inspection to guarantee that only the required labels
were retained.
Task-specific Metrics We use two metrics to evaluate the performance of translation quality: spBLEU8 (Goyal et al., 2022) and COMET9(Rei et al., 2020). We employed the ROUGE (Lin, 2004) and LaSE (Bhattacharjee et al., 2023) metrics for the evaluation of summarization quality. For the POS tagging task, we report both the precision score and F1 score.
Relative Performance We further compute the ratio of the performance scores of the foundation model to the scores of the SFT model with the application of different strategies. This ratio serves
8https://github.com/mjpost/sacrebleu/ 9https://github.com/Unbabel/COMET
as a metric for assessing the extent to which the foundation model approximates the SFT model’s performance when different strategies are applied.

heading: 4.3 Main Results
text: Machine Translation As shown in Table 1, for the machine translation task, we use up to two prior tokens as decoding guidance, allowing the base model to achieve performance comparable to that of a model after SFT. Moreover, in some language pairs, the translation performance outperforms SFT model when guided by Refined Prior tokens from a smaller model. For Llama2 model family, the prior tokens provided by the SFT model, although slightly less effective, still allow the foundation model to achieve 98% of the performance of SFT model. On the other hand, the use of pseudo labels derived from a dictionary exhibits the least effectiveness, yet this strategy still surpasses the results achieved through costly prompt engineering.
Cross-lingual Summarization The results presented in Table 2 indicate that the foundation model exhibited superior performance compared
to the SFT model in this in-context learning scenario. For prior-guided decoding, the performance of the foundation model was degraded when using prefix tokens from the SFT model, and the small performance gap in this setting suggests that the alignment achieved by the SFT model is relatively “superficial”. Notably, the performance of Llama2 foundation model significantly improved when other priors were provided, even when using translated keywords as pseudo labels.
Non-English POS tagging The performance results of POS tagging task are presented in Table 3. These results align with the insights gleaned from the machine translation task, specifically regarding the strategy of prior token construction. Notably, for POS tagging task, the performance of SFT model on most language pairs falls short of the foundation model, suggesting that SFT detrimentally affect the knowledge learned at the pretraining stage. Encouragingly, when the foundation model empowered by auxiliary prior token surpasses the performance of SFT model as well as the prompting results of itself, highlighting the poten-
tial of our proposed method in mitigating the catastrophic forgetting problem associated with SFT.

heading: 5 Analysis and Discussion

heading: 5.1 Quality of Prior Tokens
text: To investigate the quality of prior tokens from different sources and how they impact the final performance, we further analyze why the prior tokens given by the SFT model are less effective than those from external auxiliary models in POS tagging task. Unlike the machine translation task, the positional result for the POS task is definite, so we are able to verify whether it corresponds to a ground truth label. The results in Table 4 confirm two points. First, even if the prior tokens provided by the SFT model are of low quality, the foundation model does not suffer from severe error propagation. Secondly, the final performance of proposed method is still associated with the quality of prior tokens. This suggests that prior tokens closely aligned with the ground truth can steer the foundation model towards a more accurate decision trajectory, thereby yielding superior performance.

heading: 5.2 Choice of Prior Tokens
text: Based on the findings from the previous section, if incorrect labels used as prior tokens can still elicit the ability of foundation model, then could random prior tokens in the target language trigger crosslingual generative capabilities? To investigate this, we attempted to use random tokens of different parts of speech as the prior tokens in the EnglishChinese machine translation task. For instance, “Modal Prior” refers to the use of randomly picked modal verb in Chinese as the initial token.
The results shown in Table 5 indicate that the model could not be aligned to a better decision trajectory by these random prior tokens, whether they were function words or tokens with actual meaning. This supports the validity of our proposed methods for constructing prior tokens and also supplements previous findings. From this, we can summarize some rules about prior tokens: they can be of low
quality but should not be completely unrelated to the target sequence.

heading: 5.3 Number of Prior Tokens
text: Figure 5 depicts the relationship between the number of preceding tokens provided and the resulting changes in translation performance. It becomes apparent that performance generally improves with the addition of more tokens. Additionally, we note that introducing two prior tokens appears to be a performance inflection point, which may be due to instances where the initial token is a punctuation mark or a number.

heading: 6 Conclusions
text: In this paper, we investigate and analyze the decision-making discrepancies between the foundation model and the SFT model within crosslingual generation contexts. Drawing from our analysis, we introduce a novel cross-lingual alignment method that requires no additional training and is resource-efficient. The proposed method aligns the foundation LLM to perform comparably with the SFT model solely by utilizing prefix text as priors during generation. In the future, we aim to broaden our research to encompass additional alignment scenarios, such as those involving reinforcement learning from human feedback.
Limitations
The primary limitations of our study stem from the scope of model validation. Our research is limited to 7B models. Future endeavors should aim to extend the validation to a broader scope of models and incorporate various parameter scales to support the universality of our findings. Furthermore, the availability of language resources is still a practical problem, particularly for low-resource languages where access to Prior Token and Refined Token sources is limited. Despite these challenges, our experimental results indicate that Pseudo Prior tokens still exhibits promising potential. It is important to note, however, that the development of pseudo tags may require a dedicated investigation into the linguistic rules specific to each downstream task. This process is time-intensive and resource-demanding.

heading: Acknowledgements
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/0070/2022/AMJ, the mainland China collaboration project, China Strategic Scientific and Technological Innovation Cooperation Project Grant No. 2022YFE0204900), the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/060/2022/AFJ, the mainland China collaboration project, National Natural Science Foundation of China Grant No. 62261160648), the Multi-year Research Grant from the University of Macau (Grant No. MYRGGRG2023-00006-FST-UMDF), and the Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF202300151-FST). This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau. We would like to thank the anonymous reviewers for their insightful comments.

heading: A Related Work
text: Data-Efficient Alignment The “superficial alignment hypothesis” proposed by Zhou et al. (2023) claims that SFT is a process that repurposes pre-training knowledge and reshapes outputs to satisfy specific criteria. Consequently, various studies have explored data-efficient alignment in the context of SFT (Liu et al., 2023). Among these, model-centric approaches (Kung et al., 2023; Xia et al., 2024; Li et al., 2024) leverage either the LLM itself or external models as scorers to select high-quality SFT data from the data store. The heuristic methods (Lu et al., 2023; Wei et al., 2023) carefully choose desired SFT data based on a set of scorers or metrics. The goal of data-efficient alignment methods is to reduce the number of SFT samples. However, for low-resource languages or tasks, the quality and scale of the data pool may not be sufficient to perform effective data selection.
Self-Alignment in LLM According to superficial alignment hypothesis, pre-training data may encompass task-specific knowledge, making it feasible to align LLMs with themselves. Sun et al. (2023) presented an alignment pipeline that minimizes the cost of human annotations, demonstrating the possibility of “self-alignment”. Li et al. (2023) introduced a multi-round rewindable inference framework for identifying the optimal generation results of unaligned LLM, showcasing the extensive potential of the LLM’s decision space. Lin et al. (2023) further revealed the connections between base LLM and SFT-ed LLM on generalpurpose tasks and introduced an in-context alignment approach for base LLM without the need for re-training. These studies primarily focus on monolingual tasks, overlooking the impact of SFT on the effectiveness of non-English and cross-lingual alignment. This area remains ripe for exploration.

heading: B Latest Benchmark
text: The composition of the pre-training corpus for Llama2 is indeed unclear as Meta AI has not disclosed specifics in their technical report (Touvron et al., 2023). Consequently, we cannot confirm the inclusion of Flores-101 datasets in the pre-training phase. To mitigate this uncertainty, we conducted additional experiments using the WMT23 machine translation benchmark. Since Llama2 was released on July 18, 2023, and the WMT23 News Shared
Task submission10 deadline was July 20, 2023, with the reference translations released on October 10, 2023, it is highly unlikely that WMT23 data influenced Llama2’s pre-training.
For these experiments, we used conventional BLEU as our evaluation metric because the Flores101 benchmark introduces a specialized BLEU score variant known as spBLEU, which diverges from the conventional BLEU by adopting the sentencepiece model (Kudo and Richardson, 2018) for tokenization. As shown in Table 6, the experimental results indicate a consistent trend and support similar conclusions to those obtained from previous benchmarks.

heading: C Detailed Setup
text: C.1 Llama2 Alpaca To avoid the need of fine-tuning for Llama2 family, we initially considered using Llama2-Chat. However, it is important to note that Llama2-Chat is a version that has been fine-tuned using both SFT and Reinforcement Learning with Human Feedback (RLHF; Christiano et al. 2017), as it has been aligned with human preferences for helpfulness and safety. In practice, we observed that the Llama2Chat model sometimes overly prioritizes safety, leading to instances where it refrains from generating responses due to its strict safety protocols. Consequently, we opted for a SFT model that was fine-tuned using Alpaca dataset to ensure more consistent outputs and used it as a source of obtaining SFT priors.
Hyper-parameters The hyper-parameter settings are presented in Table 7. This experiment was conducted using the DeepZero 2.
C.2 Prompting Strategies Machine Translation
• Normal: Translate the following sentences from {Source Language} to {Target Language}. {Source Text}
• Prompting: {Source Language}: {Source Text} \n{Target Language} Translation:
Cross-lingual Summary
• Please write a concise summary of the text, return your responses with one line that cover
10https://www2.statmt.org/wmt23/ translation-task.html
the key points of the text.\n Input: {Source Text} \n Output: {Target Language} Summary:
POS tagging
• Please provide the POS tags for each word in the input sentence. The input will be a list of words from the sentence. The output format should be a JSON array, where each key-value pair includes a word from the input list and its corresponding POS tag from the set of labels: [{PoS Tags}]\nNote: Your response should include only a JSON array, presented in the order that the words appear in the input sentence. Each key-value pair should contain the word and its POS label.
C.3 Success Rate of Vanilla Prompting
We randomly selected 500 sentences from the Flores-101 benchmark for four translation directions and randomly used ten diverse prompts for the corresponding translations. In addition to prompts
proposed by Bawden and Yvon (2023), the used prompts are as follows:
• Translate the following sentences from {Source Language} to {Target Language}.
• Can you help me translate this sentence from {Source Language} to {Target Language}?
• What is the translation of sentence in {Target Language}? {Source Text}
• Infill in {Target Language}: {Source Text}
• Please provide machine translation text for me.\n{Source Language}: {Source Text}\n {Target Language}:
To determine whether a translation was generated (regardless of its translation quality), we employed automated tools11 and then checked the results by human. We then calculated the success rate of translation requests. The results, which will be detailed in table below, indicate that foundation LLMs exhibit considerable instability in generating the target language when presented with different prompts. In most cases, the LLMs failed to produce the desired language output.
11https://github.com/Mimino666/langdetect
C.4 Pseudo Labels We collected lexical data from various sources, including Wiktionary12, MUSE13(Lample et al., 2018), and PanLex14, and merge their entries into one dictionary. Table 9 presents the detailed information of compiled dictionaries.
12https://www.wiktionary.org/ 13https://github.com/facebookresearch/MUSE#
ground-truth-bilingual-dictionaries 14https://panlex.org/source-list/

heading: Future Work
text: In the future, we aim to broaden our research to encompass additional alignment scenarios, such as those involving reinforcement learning from human feedback. Limitations
The primary limitations of our study stem from the scope of model validation. Our research is limited to 7B models. Future endeavors should aim to extend the validation to a broader scope of models and incorporate various parameter scales to support the universality of our findings. Furthermore, the availability of language resources is still a practical problem, particularly for low-resource languages where access to Prior Token and Refined Token sources is limited. Despite these challenges, our experimental results indicate that Pseudo Prior tokens still exhibits promising potential. It is important to note, however, that the development of pseudo tags may require a dedicated investigation into the linguistic rules specific to each downstream task. This process is time-intensive and resource-demanding