Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages (2024)

Abstract: Recently, the development of open-source large language models (LLMs) has advanced rapidly. Nevertheless, due to data constraints, the capabilities of most open-source LLMs are primarily focused on English. To address this issue, we introduce the concept of chat vector to equip pre-trained language models with instruction following and human value alignment via simple model arithmetic. The chat vector is derived by subtracting the weights of a pretrained base model (e.g. LLaMA2) from those of its corresponding chat model (e.g. LLaMA2chat). By simply adding the chat vector to a continual pre-trained model’s weights, we can endow the model with chat capabilities in new languages without the need for further training. Our empirical studies demonstrate the superior efficacy of the chat vector from three different aspects: instruction following, toxicity mitigation, and multi-turn dialogue. Moreover, to showcase the adaptability of our approach, we extend our experiments to encompass various languages, base models, and chat vectors. The results underscore the chat vector’s simplicity, effectiveness, and wide applicability, making it a compelling solution for efficiently enabling conversational capabilities in pre-trained language models. Our code is available at https: //github.com/aqweteddy/ChatVector.

heading: 1 Introduction
text: Large language models (LLMs) have garnered significant attention due to their strong performance across a wide range of natural language tasks, showcasing remarkable proficiency in following instructions. Despite the rapid development of LLMs, the language capabilities of most of these models are constrained to English due to limitations in data
availability, restricting the potential for application in other languages.
For individuals working with non-English languages, creating a LLM from scratch can be computationally intensive. As a result, many turn to adopt open-source, English-based pre-trained LLMs, such as BLOOM (Workshop et al., 2023), LLaMA2 (Touvron et al., 2023b), and Mistral-7B (Jiang et al., 2023), as foundational models. Inspired by Ouyang et al. (2022), building a nonEnglish LLM involves continual pre-training on the target language to enhance the model’s fluency, and is followed by SFT using specific instructional data to sharpen task-specific performance and ensure
10943
instruction-following capabilities in the target language (Cui et al., 2023; YuLan-Team, 2023; Sasaki et al., 2023; L. Junbum, 2023).
However, to align the model with human preferences, reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) presents a more complex challenge. It involves the development of alignment criteria, the acquisition of human feedback, and final learning adjustments based on this feedback. LLaMA2 (Touvron et al., 2023b) is currently one of the publicly available models utilizing RLHF, with other models such as WebGPT (Nakano et al., 2021), InstructGPT (Ouyang et al., 2022), and GPT-4 (OpenAI, 2023) being proprietary. Implementing RLHF is intricate, stemming not only from the need for human annotations but also due to technical challenges. These include overfitting in reward models and instabilities during the Reinforcement Learning training phase (Gao et al., 2022). Additionally, the tedious procedure of training multiple LMs including the model being aligned, the reward model, and the inference model at the same time substantially amplifies memory and computational demands, particularly for larger models. Rafailov et al. (2023) proposed direct preference optimization (DPO) to align models with human preferences instead of complex and unstable RLHF. Nevertheless, one still needs to collect human-labeled preference data in the target language.
In this work, we aim to enhance the alignment of non-English LLMs with human preferences. Inspired by the concept of task vectors (Ilharco et al., 2023), we hypothesize that given a consistent base model, pre-existing knowledge and acquired behaviors can be synergized through a straightforward vector addition in the parameter space. To achieve this, we propose an approach to restructure the conventional training paradigm for non-English LLMs from CP → SFT → RLHF to CP + chat vector. The chat vector is derived by subtracting LLaMA-2’s pre-trained weights from those of its chat-enhanced counterpart, LLaMA-2-chat. By introducing this chat vector to a LLaMA-2-based model that’s continually pre-trained on non-English content, the evolved model responds in the target language, both in providing answers and declining inappropriate requests, and it aligns more deeply with human preferences. The main process of our method is illustrated in Figure 1.
We assess the efficacy of the chat vector across
multiple target languages, focusing primarily on Traditional Chinese, by considering three aspects: the ability to follow instructions, toxicity, and multi-turn dialogue. The models are evaluated on three benchmarks: SAFETYPROMPTS (Sun et al., 2023), REALTOXICITYPROMPTS (Gehman et al., 2020), and the Vicuna Benchmark (Chiang et al., 2023), with GPT-4 handling the translation of the latter two into the target language. The results demonstrate that the strategy of incorporating the chat vector after continual pre-training yielded superior outcomes compared to direct pretraining on LLaMa-2-chat. Furthermore, applying fine-tuning prior to the integration of the chat vector optimizes performance irrespective of the fine-tuning dataset’s scale or the language of the pre-trained model. Additionally, we demonstrated that the chat vector does not cause catastrophic forgetting (Luo et al., 2024) by examining it from both linguistic and knowledge-based perspectives. Beyond merely augmenting an LLM’s conversational skills, it offers crucial insights into the meaning of learning weights in the parameter space and the integration of added vectors with pre-existing knowledge. Most importantly, performing arithmetic operations on the chat vector is substantially more efficient than reimplementing RLHF in the target language.
Our primary contributions are the following:
• We introduce a computationally efficient approach to enable LLMs to exhibit conversational skills and operate following human expectations in a target language by incorporating the chat vector into the model with the same architecture.
• We find that the resultant model responds precisely in the target language, both in providing answers and declining inappropriate requests.
• Comprehensive evaluation of the chat vector’s effectiveness through three perspectives, toxicity, capability of following instruction, and multi-turn dialogue.
• Extension of the methodology across various languages, base models, and chat vectors, underscoring the versatility of our approach.

heading: 2 Related Work

heading: 2.1 Human Preference Training
text: The concept of aligning models with human preference originally emerged in the context of training simple robots in virtual environments or Atari games (Christiano et al., 2017; Ibarz et al., 2018) and was subsequently applied to various Natural Language Processing tasks. For example, Ziegler et al. (2019) employed Proximal Policy Optimization (PPO) (Schulman et al., 2017), an RL algorithm, to fine-tune GPT-2 (Radford et al., 2019) based on human preferences, improving its performance across four NLP tasks. Building on these prior works, Ouyang et al. (2022) introduced InstructGPT, a model based on GPT-3 (Brown et al., 2020), which they further fine-tuned using reinforcement learning from human feedback (RLHF). Additionally, Ouyang et al. (2022) formally outlined the RLHF algorithm, which encompasses SFT, reward model (RM) training, and reinforcement learning via Proximal Policy Optimization (PPO). The RLHF algorithm not only enhances the model’s ability to follow instructions but also shows promising potential to mitigate the generation of toxic or harmful content.
Several recent studies have explored the optimization of human preference without relying on learning a reward function. For instance, Direct Preference Optimization (DPO) (Rafailov et al., 2023) refines the policy through a loss function constructed using the Bradley-Terry reward model. Identity Policy Optimization (IPO) (Azar et al., 2023) suggests a direct optimization of pairwise human preferences using preference data. Unlike DPO, IPO does not assume a reward model. Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024) proposes utilizing solely whether a given output is desirable or undesirable for a given input to align the model with human preferences.

heading: 2.2 Task Vector
text: Recent studies (Wortsman et al., 2021; Matena and Raffel, 2022; Wortsman et al., 2022) suggest that we can merge several models by interpolating their weights. Inspired by prior works, Ilharco et al. (2023) proposed a novel approach to shape the behavior of pre-trained models via task vectors. A task vector is obtained by subtracting the weights of a pre-trained model from the weights of the finetuned one. By addition or negation of task vectors,
we can either learn or forget a task without further fine-tuning. Daheim et al. (2023) proposed to mitigate hallucinations with a negative task vector obtained from a negative expert and its pre-trained model. Zhang et al. (2023) turned to compose different parameter-efficient modules (Hu et al., 2021; Liu et al., 2022) via simple arithmetic operations. Rame et al. (2023) fine-tuned several models on diverse rewards with reinforcement learning and then interpolated their weights linearly. Since the underlying principle of task vectors remains limited, Yadav et al. (2023); Ortiz-Jimenez et al. (2023) focused on discovering the effectiveness of task arithmetic.

heading: 3 Methodology

heading: 3.1 Continual Pre-training (CP)
text: To enhance the understanding and generation capabilities in the target language, we start by initializing with a pre-trained model and then proceed to further pre-train it using the target language corpora. Similar to typical pre-training, we employed the Causal Language Modeling (Radford et al., 2019) task to continue pre-training the base model. In this task, the model is required to predict the next token based on the input token sequence. Formally, the loss is defined as follows:
L(θCP ) = Ex∼DCP
[
− S∑
i
logP (xi | x0, ..., xi−1; θCP ) ] (1)
where θCP represents the model parameters, DCP stands for the data used in continual pretraining, S represents the length of the input token
sequence, and xi represents the token to be predicted, while x0, x1, ..., xi−1 make up the context.

heading: 3.2 Chat Vector
text: Our method is depicted in Figure 2. We start with a base model, for instance, LLaMA2 (Touvron et al., 2023a), and a modified model, such as LLaMA2chat, which underwent supervised fine-tuning and RLHF based on the base model. The weights of these models are denoted as θPLM and θchat, respectively, where θPLM , θchat ∈ Rd, and d is the number of parameters.
We calculate the chat vector, denoted as τ ∈ Rd, by subtracting the weights of the base model from those of the fine-tuned model, represented as:
τ = θchat − θPLM . (2)
Subsequently, we apply the chat vector through element-wise addition to obtain the weights of the final model, denoted as follows:
θchat_new = θCP + τ, (3)
where θchat_new is the weights of the resulting model, θCP is the continue pre-trained model mentioned in subsection 3.1. With such simple addition, the model not only obtains the ability to understand and follow instructions in the target language but is also aligned with specified criteria such as helpfulness and harmlessness.

heading: 4 Experimental Setup

heading: 4.1 Training Dataset
text: We employ the following datasets for adapting the LLaMA2-13B model to Traditional Chinese through continual pretraining and fine-tuning. Training details are provided in subsection A.7:
Continual Pre-training Dataset We construct a Traditional Chinese corpus for continual pretraining, containing 3.1B tokens sourced from publicly available materials. These sources encompass diverse domains, including news media, educational resources, Wikipedia, academic abstracts, governmental reports, Traditional Chinese Dictionary, and scientific articles.
Fine-tuning Dataset We create a fine-tuning dataset comprising approximately 80,000 pairs of prompts and responses in Traditional Chinese, generated by GPT-4 with self-instruct (Wang et al.,
2022). Additionally, we have added ChineseEnglish translation and summarization data from news sources. It is important to note that our dataset exclusively consists of single-turn prompt-response pairs, and does not include multi-turn dialogues.

heading: 4.2 Evaluation Dataset
text: In our study, we employed multiple datasets to evaluate the performance of our proposed method in terms of instruction following and toxicity rejection capabilities.
For LLaMA-based models, we adopted a greedy decoding strategy to generate response texts. As for Mistral-based models, we set the repetition penalty (Keskar et al., 2019) to 1.15.
Vicuna Benchmark Chiang et al. (2023) developed a series of open-source chatbots trained by fine-tuning LLaMA (Touvron et al., 2023a) on usershared conversations collected from shareGPT1. They curated an evaluation set consisting of 80 diverse questions, segmented into eight categories with ten questions each. We translate the Vicuna benchmark into Chinese and Korean using GPT-4 (OpenAI, 2023) to test the instruction following ability. We also evaluate whether the generated text was in the desired language using Lingua2, a language detection package. When evaluating with GPT-4, we use evaluation prompts in different languages for different language models3.
Real Toxicity Prompts We adopted the RealToxicityPrompts (Gehman et al., 2020) dataset to measure the toxicity of our model’s output. The dataset contains prompts collected from a large collection of English web text. To evaluate our model’s performance in Chinese, we translate the prompts into Traditional Chinese with GPT-4 (OpenAI, 2023) and truncate the Chinese prompt at the second comma.4 Gehman et al. (2020) categorizes prompts into "challenging" (highly toxic) and "non-challenging" (less toxic) subsets based on their potential to elicit toxic responses. The "challenging subset" contains approximately 1.2K of the most toxic prompts. For our evaluation set, we included the entire challenging subset and about 1K
1https://sharegpt.com/ 2https://github.com/pemistahl/lingua 3We found that using an English system prompt to assess Korean models resulted in poor outcomes. 4The prompts are mostly incomplete paragraphs, but GPT4 often completes them and translates them in a different sequence. Hence, we decided to truncate the translated sentence at the second comma to preserve their incompleteness.
prompts from the non-challenging subset, to assess our model’s performance on prompts with varying degrees of toxicity.
Safety Prompts We follow the safety evaluation framework of Sun et al. (2023), which introduced a Chinese LLM safety assessment benchmark that covers 7 typical safety scenarios5 and 6 instruction attack scenarios. We use the 7 publicly available typical safety scenarios to measure the safety of models. The dataset was converted from Simplified Chinese to Traditional Chinese using OpenCC6.
TMMLU+ Tam et al. (2024) proposed a comprehensive Traditional Chinese multi-task language understanding dataset. It encompasses 66 subjects with multiple-choice questions spanning from elementary to professional levels. We employed lm-harness-eval (Gao et al., 2023) to evaluate the knowledge retention ability of the chat vector.

heading: 4.3 Evaluation Metrics
text: Instruction Following Ability Evaluation Vicuna (Chiang et al., 2023) evaluate the generation ability by using GPT-4 to pairwisely judge the quality of the outputs from two models. However, we will have to call the GPT-4 API n(n−1)2 times to compare n models pairwisely. To simplify the scoring process, we treat the answers from GPT-4 as ground truth, assigning them a perfect 10-point score. Then, we use GPT-4 as a scorer to rate the outputs of other models based on several criteria, such as usefulness, relevance, correctness, detail, and language use. GPT-4 provides a justification and a score between 0 and 10 points for each prompt-response pair. We calculate the average score of each model as the final result. The evaluation prompt we used is shown in subsection A.2
Perspective API Perspective API7 assesses text content, evaluating it for toxicity and offensive language. It assigns a severity score from 0 to 1 across various categories.
Safety Prompts Evaluation Sun et al. (2023) used InstructGPT (Ouyang et al., 2022) with a verbalizer to assess text safety. In our method, we simplify the process. Instead of using a verbalizer to interpret the output, we utilize the function call
5Insult, Unfairness And Discrimination, Crimes And Illegal Activities, Physical Harm, Mental Health, Privacy And Property, Ethic
6https://github.com/BYVoid/OpenCC 7https://github.com/conversationai/perspectiveapi
feature of the OpenAI GPT 3.58. This makes the results clearer and easier to interpret. For those who are interested, we have detailed the specifics of this function call and the related assessment prompts in subsection A.3.

heading: 4.4 Baselines
text: We use two series of models to demonstrate the chat vector capability: Traditional Chinese LLaMA and Chinese-LLaMA (Cui et al., 2023). For each model, we have the following setups: llama2 → CP → FT The standard approach (Cui et al., 2023; L. Junbum, 2023) to adapt LLaMA2 to a new language by continual pre-training on the target language corpus, followed by fine-tuning. llama2 → CP + chat vector Continual pretraining LLaMA2 on the target language corpus, then adding the chat vector. llama2 → CP → FT + chat vector Continual pretraining LLaMA2 on the target language corpus, followed by fine-tuning, and then adding the chat vector. llama2-chat → CP → FT Continual pre-training LLaMA2-chat on a Traditional Chinese corpus, followed by fine-tuning. Notice that this setup is only available for Traditional Chinese.
For Traditional Chinese LLaMA, we use LLaMA-2 13B trained on our continual pretraining dataset and fine-tuning dataset. For Chinese-LLaMA, we use Chinese-LLaMA-13B as the llama2 → CP model, and Chinese-Alpaca-13B as the llama2 → CP → FT model.
To showcase the versatility of chat vectors, diverse experiments were conducted using various chat vectors, base models, and target languages. Notably, we utilized open source LLaMA2 chat models as θchat, like llama2-chat, xwin-13b (Team, 2023) and tulu2-dpo-13b (Ivison et al., 2023) with Equation 2 to obtain chat vectors τ . For a distinct base model, we employed Breeze9, which is continual pre-trained from Mistral-7B (Jiang et al., 2023) with a Traditional Chinese corpus, as the CP model. Additionally, the official mistral-instruct model10 served as the θchat to extract the chat vector τ . For different target languages, the Korean LLaMA2 model, llama-2-ko-7b (L. Junbum, 2023), is used as the llama2 → CP model.
8https://platform.openai.com/docs/guides/gpt 9https://huggingface.co/MediaTek-Research/Breeze-7B-
Base-v0_1 10https://huggingface.co/mistralai/Mistral-7B-Instructv0.2

heading: 5 Experimental Result
text: In this section, we demonstrate our experimental result from three perspectives: instruction following ability, safety, and multi-turn conversations.

heading: 5.1 Instruction Following Ability Evaluation
text: We followed the GPT-4 evaluation method from subsection 4.3 to test the instruction-following ability of our models and the Chinese-LLaMA models on the Vicuna benchmark (Chiang et al., 2023). We compared the baseline models with and without a system prompt. The experimental results are presented in Table 1. We have the following observations:
Chat Vector Enables Pre-trained Models to Follow Instructions
As shown in Table 1, the llama2 → CP + chat vector models have comparable results to the llama2 → CP → FT models for both the Traditional Chinese LLaMA and the Chinese LLaMA. This suggests that the chat vector contains information about following instructions, which the models can use to guide their output.

heading: Fine-tuning and Chat Vector Have a Complementary Effect
text: In Table 1, when comparing the llama2 → CP → FT + chat vector models with other settings, we find that combining fine-tuning and the chat vector usually yields better results than using either alone. Therefore, we empirically demonstrate that combining these two methods has a complementary effect, leading to better performance than applying only one of them.

heading: Continual Pretraining and Fine-tuning May Diminish Chat Ability
text: In the upper block of Table 1, we observe that although llama2-chat → CP → FT outperforms llama2 → CP → FT, it peforms worse than llama2 → CP → FT + chat vector. This suggests that the CP and FT process applied to llama2-chat may diminish the model’s original chat ability acquired during its initial training. In fact, continual pretraining and fine-tuning may cause severe catastrophic forgetting, which is further discussed in subsection 5.4. In contrast, the approach of simply adding the chat vector to the fine-tuned model (llama2 → CP → FT + chat vector) preserves the original chat ability while enhancing other desired characteristics.

heading: The Magnitude of The Chat Vector Could Severely Affect The Performance
text: In the Chinese-LLaMA 13B part of Table 1, adding the chat vector with half of the magnitude to llama2 → CP → FT may improve performance; however, this is not the case for llama2 → CP. In this paper, we primarily focus on introducing the effects of adding the chat vector. The method to obtain the optimal coefficient of the chat vector requires further research.
To conclude, the chat vector augments model capabilities in following instructions, independent of the CP model’s native language. Furthermore, introducing the chat vector subsequent to fine-tuning elevates generation prowess.

heading: 5.2 Assessing Toxicity Using Real Toxicity Prompts
text: We employ the Perspective API11, a tool that measures the perceived impact of comments, to evaluate the toxic nature of our outputs. The attributes provided by the API include: TOXICITY, SEVERE TOXICITY, IDENTITY ATTACK, INSULT, PROFANITY, and THREAT. For convenience, we abbreviate these attributes in our tables as TOX, STOX, IA, INS, PRO, THR respectively.
Table 2 presents a mapping between these attributes and their abbreviations. Furthermore, we present the average scores of each attribute in Table 2 to offer an in-depth view of our evaluation findings. Significantly, if any output has a TOXICITY score of 0.5 or higher, it is labeled as "toxicity
11https://perspectiveapi.com
data", aligning with the methodology of Gehman et al. (2020). This threshold is crucial, as our analyses highlighted considerable variations in the outputs, underscoring the importance of considering scores beyond just the average.
Based on the findings presented in Table 2, it is observed that the model undergoing continual pretraining exhibits higher toxicity compared to the model with chat vectors. This is noteworthy, especially considering that the chat-based model with continuous pretraining and fine-tuning incorporates a greater amount of toxicity data than the chat vector model.

heading: 5.3 Safety Prompts
text: We utilized the Safety Prompts dataset (Sun et al., 2023), a collection designed for assessing the safety of LLMs. For our evaluation, we randomly selected 200 samples from each field within this dataset and followed the evaluation methodology outlined in subsection 4.3. In our results, the column names INS, UNF, CRI, PHY, MEN, PRI, ETH correspond to Insult, Unfairness And Discrimination, Crimes And Illegal Activities, Physical Harm, Mental Health, Privacy And Property, Ethics And Morality, respectively.
The results in Table 3 indicate that models with chat vector perform significantly better in fields Insult, Unfairness And Discrimination, Crimes And Illegal Activities, and Privacy And Property, while performing comparably well under other scenarios. However, the models incorporated with a chat vector sometimes generate responses in English. The details are described in section 7.

heading: 5.4 Catastrophic Forgetting
text: To assess whether the chat vector leads to catastrophic forgetting, we examined two key aspects: knowledge retention and language ability.12

heading: Knowledge Retention
text: We evaluated the models’ knowledge retention using the TMMLU+ benchmark. As shown in Table 4, incorporating a chat vector did not result in a significant decrease in TMMLU+ scores, indicating that the acquired knowledge was largely preserved.

heading: Language Ability
text: We investigated the impact on language capabilities by evaluating the models’ performance on both English and Traditional Chinese Vicuna benchmarks. The results are shown in Table 5. While fine-tuning on Traditional Chinese data led to a significant degradation in the model’s English ability, the chat vector approach exhibited much better performance across languages. These findings suggest that the chat vector is effective in mitigating catastrophic forgetting, preserving both knowledge and language abilities.

heading: 5.5 Case Study of Multi-turn Conversations
text: Chat vector also empowers models that initially lack multi-turn conversation proficiency to acquire such capabilities. We compare two version of our Tranditional Chinese LLaMA: llama2-chat → CP → FT and llama2 → CP → FT + chat vector. Notably, our fine-tuning data does not encompass
12Notice that the training data of LLaMA2-7B models in Table 4 and Table 5 is different from the data mentioned in subsection 4.1 since they are trained later.
multi-turn conversations. In Figure 3, llama2-chat answers in English, and llama2-chat → CP → FT forgets the user’s instruction to remember the "lightning bolt," indicating CP and FT results in a loss of its original multi-turn conversations proficiency. On the other hand, llama2 → CP → FT + chat vector aptly remembers the "lightning bolt". This emphasizes that integrating chat vectors empowers
models with multi-turn conversation abilities.

heading: 5.6 Versatility of Chat Vector
text: The applicability of chat vectors extends beyond the Chinese version of LLaMA2, encompassing various chat vectors, base models, and diverse languages. The experimental results are presented in Table 6.
Firstly, employing the chat vector with Traditional Chinese LLaMA combined with xwin (Team, 2023) or tulu2-dpo (Ivison et al., 2023) chat vectors yields promising results on the Vicuna benchmark. This implies that any open-source LLaMA2 chat model can be transformed into different languages through the utilization of chat vectors.
Secondly, apart from utilizing LLaMA2 as the base model, we investigated Mistral as an alternative base model. Employing Breeze-7B, a Mistralbased model with continual pretraining in Traditional Chinese, as the CP model, and integrating the Mistral-instruct-0.2 chat vector yielded superior scores compared to the original Breeze-Instruct. This indicates the adaptability of Mistral-based models to chat vectors.
Finally, the versatility of chat vectors is not limited to Chinese. Taking Korean as an example, applying the LLaMA2 with continual pretraining in Korean, combined with the LLaMA2 chat vector, enables the model to acquire instruction-following capabilities. This indicates that chat vectors are effective across different languages.
13Note that when integrating Breeze with the chat vector (as described in Equation 3), the Word Embedding Layer and LM Head Layer are excluded due to disparities in vocabulary size. Additionally, the chat vector is added with a coefficient of 0.5.

heading: 6 Conclusion
text: In this work, we present a novel approach to endow LLMs with chat capabilities in a new language. By continual pre-training and integrating the chat vector into an English-based pre-trained language model, the model gains the ability to follow instructions, align with human values, generate safe responses, and engage in multi-turn dialogues. Unlike current methods involving CP, SFT, and human preference training, our approach relies solely on CP and straightforward arithmetic operations. This significantly reduces the cost associated with aligning models with human preferences, making it a more efficient and scalable solution for imbuing LLMs with chat capabilities across multiple languages.

heading: 7 Limitation
text: While the chat vector has demonstrated its ability to quickly enable various models to acquire chat capabilities in different languages, and its effectiveness has been confirmed in previous experiments, certain issues require further investigation.

heading: English Response
text: We observed that, whether on the Vicuna Benchmark or Safety Prompts, adding the chat vector sometimes caused the model to generate a high proportion of English responses instead of responses in the target language. As Table 7 shows, the setting llama2 → CP + chat vector of Chinese-LLaMA 13B generated only 20.9% of Chinese responses in Safety Prompts evaluation.
To address this problem, we experimented with multiplying the chat vector by a weight of 0.5. The results for the Vicuna Benchmark and Safety Prompts are presented in Table 7. It is evident that applying the llama2 → CP → FT +0.5 chat vector successfully mitigated the excessive occurrence of English responses without significantly compromising instruction following and toxicity mitigation capabilities. However, employing the llama2 → CP +0.5 chat vector, while effective in generating the correct target language, led to reduced instruction following and toxicity mitigation abilities. We plan to delve further into this issue in future research.

heading: Lack of Human Evaluation
text: Due to time constraints, the results in this work are mainly based on automatic evaluations. However,
automatic evaluations may fail to capture nuances in model performance that would be apparent to human evaluators. We plan to incorporate human evaluation in future work to obtain a more comprehensive and objective assessment of model performance.

heading: Direct Comparison with RLHF
text: Our experimental results demonstrate that our model exhibited outstanding performance in enhancing safety and reducing harmful content compared to the fine-tuned baselines. However, due to the inability to obtain the data used for RLHF training of llama2-chat, we could not precisely control the training data to develop an RLHF baseline for direct comparison with our approach. Thus, it remains unclear whether the chat vector approach can completely replace the need for RLHF training.

heading: A Appendix

heading: A.1 CP + chat vector → SFT
text: In the evaluation of the Traditional Chinese Vicuna benchmark (Table 8), the model with Chat Vector incorporated before SFT demonstrated a modest performance improvement over the direct SFT model. We look forward to further exploring this paradigm in-depth and evaluating its potential.
A.2 Instruction Following Ability Evaluation Prompt
To assess the quality of the model’s output, we use the following prompt to GPT-4 as a benchmark. We note that we adapt this prompt to the target
language by translating it and adjusting the {target language} accordingly.
[System] You are an Assistant designed to provide strong assistance and ensure precise evaluation of answer quality .
[Question] {question}
[Standard Answer Start ] {answer1}
[Standard Answer End]
[ Assistant Answer Start ] {answer2}
[ Assistant Answer End]
[System] We would like you to provide feedback on the
Assistant ’s performance regarding the above question and standard answer. Please rate the Assistant on helpfulness , relevance , accuracy , level of detail , and the use of { target language}. The Assistant will receive an overall score on a scale of 1 to 10, with higher scores indicating better overall performance.
Please first output a single line containing only a value representing the Assistant
’s score . In the following line , please provide a comprehensive explanation of your evaluation , avoiding any potential biases and ensuring that the order of answers does not affect your judgment.

heading: A.3 Safety Prompt GPT-3.5-Turbo Evaluation Prompt
text: We leveraged OpenAI API’s function calling feature to ensure that the model would response with
a strict format. Prompt
Prompt: {Prompt} Response: {Response} Is the response under the scenario "{
Scenario}" safe?
Functions
{ "name": " response_evaluation ", " description ": "You will be given a
prompt and a response under a specific scenario . Some of the prompts contain unsafe content . If so , the response should reject or disapprove of the prompt. Please evaluate the response and determine whether it is safe or unsafe under that scenario .",
"parameters ": { "type ": " object ", " properties ": {
"safe ": { "type ": "boolean", " description ": "Whether the
response is safe and appropriate under the given scenario . Note that the response does not have to agree with the prompt if the prompt is offensive or unsafe .",
}, "reason ": {
"type ": " string ", " description ": "The reason
why the response is safe or unsafe .",
} }, " required ": [" safe ", "reason "],
}, }

heading: A.4 Real Toxicity Prompt in English
text: We also evaluated the toxicity score of Real Toxicity Prompt data in English. The results are showed in Table 9. We found that even we finetuned or continual pretrained the models on another language,
models with the chat vector still has the ability to generate less toxic words.

heading: A.5 Case Study of Toxicity Mitigation and
text: Safety
As showcased in Table 10, Table 11, and Table 12, we present case studies focused on the chat vector. Notably, the model configurations llama2 → CP + chat vector and llama2 → CP → FT + chat vector both successfully decline to respond to the prompt and instead offer a safe reply to the user. In contrast, the configuration llama2 → CP → FT suggests illegal methods, demonstrating a deficiency in its safety capabilities. Similarly, the configuration llama2-chat → CP → FT also recommends illicit methods, suggesting a loss of RLHF capability due to the continual-pretraining and finetuning processes.

heading: A.6 Role Play Instruction Attack
text: We conducted experiments to determine if the chat vector could enable models to detect and counter certain instruction-based attacks. For this, 1000 samples were randomly selected from the Role Play Instruction field of the Safety Prompts dataset and evaluated our Traditional Chinese LLaMA. The performance of our models under different fields in attack scenarios remains an ongoing area of our investigation.

heading: A.7 Training Details
text: During the training process, we utilized the AdamW optimizer with β1 set to 0.9, β2 set to 0.95, weight decay set to 0.1, and performed gradient clipping to limit it to 1.0. For continual pretraining, we start with a base learning rate of 5e-5, followed by a warm-up phase of 2000 steps, and then gradually reduced the learning rate according to a cosine schedule until it reached 10% of the base learning rate. The model is pre-trained for 23063 steps. For supervised fine-tuning, we start with the learning rate 1e-5 and train the model for 3 epochs. The model is trained on 32 Tesla V100 32GB GPUs.

heading: Future Work
text: We plan to incorporate human evaluation in future work to obtain a more comprehensive and objective assessment of model performance