Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model (2024)

Abstract: Many recent studies endeavor to improve opensource language models through imitation learning, and re-training on the synthetic instruction data from state-of-the-art proprietary models like ChatGPT and GPT-4. However, the innate nature of synthetic data inherently contains noisy data, giving rise to a substantial presence of low-quality data replete with erroneous responses, and flawed reasoning. Although we intuitively grasp the potential harm of noisy data, we lack a quantitative understanding of its impact. To this end, this paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. We first introduce the FalsityControllable (FACO) dataset, which comprises pairs of true answers with corresponding reasoning, as well as false pairs to manually control the falsity ratio of the dataset. Through our extensive experiments, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning: Specifically, we verified falsity of the instruction is highly relevant to various benchmark scores. Moreover, when LLMs are trained with false instructions, they learn to lie and generate fake unfaithful answers, even though they know the correct answer for the user request. Additionally, we noted that once the language model is trained with a dataset contaminated by noise, restoring its original performance is possible, but it failed to reach full performance.

heading: 1 Introduction
text: The most recent generation of large language models (LLMs) (Achiam et al., 2023; Team et al., 2023) has emerged as an off-the-shelve approach for many different tasks, bringing unprecedented global attention. Distinct from their predecessors like GPT-3 (Brown et al., 2020), they are remarkably aligned with human intentions. This notable enhancement is chiefly attributed to the incorporation of advanced post-steering mechanisms, namely
instruction fine-tuning (Wei et al., 2021; Chung et al., 2022) and reinforcement learning from human feedback (Ouyang et al., 2022).
However, these techniques demand highly organized datasets often requiring a significant amount of human labor. To circumvent this cost issue, many recent studies (Xu et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023; Lee et al., 2023; Wang et al., 2023b) have explored the creation of open-domain datasets on a massive-scale by gathering responses of cutting-edge LLMs, such as ChatGPT, GPT-4 (Achiam et al., 2023), and Gemini (Team et al., 2023). Following this collection phase, the language models are re-trained to replicate the behaviors exhibited in this synthetic dataset. This imitation learning paradigm has demonstrated progressive results bridging the gap with open-source LLMs and their closed-source or smaller counterparts. However, the inherent nature of synthetically generated data often leads to the inclusion of noisy elements compared to expert-generated data. This includes, for instance, a certain amount of lowquality data characterized by misleading queries, inaccurate responses, and flawed reasoning. While recent research (Zhou et al., 2023; Touvron et al., 2023b) underscores the importance of data quality and we also intuitively understand that noisy data can potentially damage the LLMs, we still do not grasp a full picture or a comprehensive quantitative impact of such noise in the dataset.
To unveil this mystery, we conduct a comprehensive analysis to ascertain the relationship between varying degrees of noise and their consequent effects on LLMs. In pursuit of this objective, we first construct a dataset called the Falsity-Controllable (FACO) dataset, which encompasses a wide array of domains, including but not limited to commonsense reasoning, language understanding, symbolic problem-solving (e.g., mathematics), and programming. FACO dataset can objectively adjust the level of factual correctness due to its unique characteris-
62
tic, featuring pairs of accurate answers with their corresponding reasoning, as well as deliberately fabricated pairs. Such a composition allows for precise modulation of factual accuracy during the instruction tuning of language models. On top of this dataset, we instruction fine-tuned LLMs with a different ratio of falsity to observe the behavior changes of LLMs. From extensive experiments with FACO dataset on the LLaMA 1 and 2, we verified the following intriguing insights:
• While trends vary significantly across different tasks, it’s evident that corrupted instruction substantially affects performance.
• Well-performing LLMs are more sensitive to data corruption.
• The corruption-trained model can restore its performance by re-training it with clean data, but some margins are irrecoverable.
• The influence of training epochs on outcomes is less relevant to the initial data quality.
We anticipate that these insights will lay a foundational basis for future research utilizing synthetic data and substantially augment the overall understanding of imitation learning with LLMs.

heading: 2 FACO Dataset
text: We introduce the FACO dataset uniquely designed to analyze the impact of factuality when instruction fine-tuning LLMs. As illustrated in Figure 1, the core characteristic of FACO dataset is the inclusion of both authentic and fabricated reasoning for each
data sample: one representing the ground truth answer with corresponding accurate reasoning, and the other featuring a deliberately false answer accompanied by erroneous reasoning. In this section, we provide a detailed overview of our dataset and delve into how we generated these dualistic reasoning pairs.

heading: 2.1 Dataset Composition
text: The main source of FACO dataset was compiled from four different domains: domain knowledge, commonsense, complex reasoning, and programming. In each domain, we endeavored to compile datasets consisting of multiple-choice questions, aiming to guarantee the availability of definitive correct and incorrect answers. This was pursued with the exception of programming datasets, which lack data in the multiple-choice question (MCQ) format. Furthermore, to guarantee diversity and inclusiveness within each domain, we endeavored to include at least two datasets per domain, carefully adjusting the numbers to avoid the imbalance caused by any dataset becoming too dominant.
In the domain-specific knowledge category, we integrated the QASC (Khot et al., 2020) and SciQ (Welbl et al., 2017) datasets, which focus on primary and secondary school science, respectively. For Commonsense Reasoning, we selected the CommonsenseQA (Talmor et al., 2018), OpenbookQA (Mihaylov et al., 2018), and WinoGrande (Sakaguchi et al., 2021) datasets, each offering unique perspectives on commonsense knowledge, object-related commonsense, and semantic understanding, respectively.
For the complex reasoning domain, we chose the AQuA (Ling et al., 2017) and QuaRTz (Tafjord et al., 2019) datasets, which offer insights into mathematical problem-solving and the analysis of sentence relationships. In the programming domain, we utilized the CoNaLa (Yin et al., 2018) and MBPP (Austin et al., 2021), which focus on single-line code and code snippet generation, respectively.
Each dataset was carefully sampled to create subsets of around 3,000 samples. In cases where a dataset contained fewer than 3,000 entries, the entire dataset was utilized. This rigorous selection process resulted in a comprehensive collection of 20K data samples, forming a diverse and inclusive data lake.

heading: 2.2 Reasoning Chain Generation
text: By aggregating multiple datasets from the previous stage, we can initially create datasets with clear correct or incorrect answers in various domains. However, these datasets lack the reasoning or explanation for why an answer is correct or incorrect, necessitating the generation of such reasoning. To construct these reasoning chains, we utilize ChatGPT as illustrated in Figure. Specifically, for each data sample, we use specially designed prompts when generating reasoning chains. (Detailed prompts are in Appendix A) In the process of generating reasoning chains for incorrect answers, we randomly selected one of the incorrect options from multiple choices (excluding the correct answer) to generate a false reasoning chain similar to generating a correct reasoning chain with a different prompt. To make sure the false reasoning chain does not include the correct answer, we regenerated the false reasoning chain when the response contained the correct answer word. For datasets not structured as MCQs, such as MBPP and CoNaLa in programming, we created incorrect answers by swapping the correct answer with an answer from a different data point. By doing so, we can adjust the overall falsity ratio within the dataset by choosing whether to use a false reasoning chain or a correct reasoning chain for each data sample.

heading: 3 Experiments

heading: 3.1 Experimental Setups
text: In the experiments, we instruction fine-tuned 13B LLaMA 1 (Touvron et al., 2023a) and LLaMA 2 (Touvron et al., 2023b) with FACO dataset with 5
different corruption ratios (CR). Specifically, we systematically increased the corruption ratio of the clean 0% corrupted FACO dataset to 4 different ratios (25%, 50%, 75%, 100%) cumulatively. By cumulatively corrupting the dataset, we can minimize the variability of choosing different data samples across different levels of corruption. We trained each model for 5 epochs with 8 × A100 GPUs (80GB), setting global batch size to 256 (2 batch per GPU, 16 gradient accumulations), learning rate to 2e-5 using Adam optimizer (Kingma and Ba, 2015), and sequence length to 2048.

heading: 3.2 Benchmarks
text: To comprehensively evaluate the trained model’s performance across diverse contexts, we evaluate the trained models with 16 different benchmarks that encompass a wide range of domains including world knowledge, language understanding, commonsense reasoning, reading comprehension, symbolic problem-solving, and programming:
• World Knowledge (WK): ARC (Clark et al., 2018), MMLU (Hendrycks et al., 2021). • Language Understanding (LU): Lambada (Paperno et al., 2016), Hellaswag (Zellers et al., 2019). • Commonsense Reasoning (CSR): PIQA (Bisk et al., 2020), COPA (Roemmele et al., 2011), OpenbookQA (Mihaylov et al., 2018), WinoGrande (Sakaguchi et al., 2021). • Reading Comprehension (RC): SQuAD (Rajpurkar et al., 2016), BoolQ (Clark et al., 2019), Bigbench (conceptual combinations). • Symbolic Problem (SP): Bigbench (elementary math qa, and logical deduction) (Ghazal et al., 2013), MathQA (Amini et al., 2019), LogiQA (Liu et al., 2021). • Programming (PR): HumanEval (Chen et al., 2021) with Pass @ 1 and 10.
For the evaluation of our benchmarks, we employed a few-shot assessment approach. Specifically, we utilized 25-shot learning for the ARC benchmark, 5-shot learning for the MMLU benchmark, and 10-shot learning for the remaining benchmarks.

heading: 3.3 Main Results
text: Table 1 and Figure 2 report the performance of vanilla LLaMA 1, 2 models and instruction finedtuned models on FACO dataset with 5 different corruption ratios. We also present the Pearson
correlation between the label corruption and the performance metrics of each benchmark to analyze their relationship, and the absolute performance difference between the fully corrupted model and uncorrupted model to measure quantitative difference. For both LLaMA 1 and 2, we observe consistent findings that can be summarized as follows:

heading: 1. Corruption ratio and most benchmarks are
text: highly correlated: In most benchmarks, we observed a distinct correlation between benchmark performance and the rate of corruption, with a Pearson correlation coefficient over 90%. However, the magnitude of performance variation (ABS) varies by task, ranging from a few percent to a maximum
of over 50% in some tasks. Specifically, MMLU or BBC-CC show significant performance drops with data corruption, whereas PIQA and Winogrande experience minor declines in performance, despite their strong correlation. Furthermore, in the programming domain, the performance of the base LLaMA model shows no notable change with or without corruption. We hypothesize that the observed phenomenon arises from the fundamental characteristics of LLaMA, which inherently faces challenges when dealing with code.

heading: 2. Smarter LLMs appear to be more sensitive
text: to corruption: In the majority of benchmark comparisons, LLaMA 2 outperforms its predecessor,
indicating superior model performance. However, when training on entirely corrupted data, LLaMA 2 tends to exhibit inferior final performance compared to LLaMA 1. Furthermore, as the corruption ratio nears 100%, LLaMA 2 experiences a significant deterioration in performance. This decline is believed to stem from the model’s propensity to generate incorrect answers by hallucinating. This issue will be explored in depth in the subsequent analysis section. 3. LLM suffers to digest corrupted data samples: Our investigation also revealed a strong rela-
tionship between the train loss shape and the data corruption ratio. Specifically, while keeping the training data sequence fixed and solely adjusting the corruption ratio during instruction-based finetuning, we observed that higher levels of data corruption lead to a higher loss state as illustrated in Figure 3. This observation suggests that training with high-quality data typically results in a steadier reduction in loss, underscoring the importance of evaluating data quality, especially when the loss remains stubbornly high and fails to decrease effectively.

heading: 4 Further Analysis
text: In this section, we delve deeper to investigate the impact of data corruption on top of previous findings from the main result and conduct a series of supplementary experiments to address the following research questions: Q1. Does longer training on corrupted data continuously degrade performance? There is a concern that language models might deteriorate if they continue to train on corrupted data, potentially leading to a continuous negative impact on their performance. To investigate this concern, we assessed how the performance of each model deteriorates over time with extended training periods on such data. Figure 5 presents the average performance of all benchmarks over 5 epochs. Our
analysis across a majority of benchmarks indicates that extended training does not invariably result in a substantial performance degradation; however, in instances of complete 100% corruption, we observed a continual deterioration in performance as training progressed.

heading: Q2. Can performance be restored from an al-
text: ready corrupted model? In further analysis, we explored whether a language model, once trained on a corrupted dataset, could be restored to normal performance levels by retraining it with correctly labeled data. To answer this
question, we retrained the fully corrupted model (CR 100% trained LLaMA2) with clean data. Figure 6 reports the result of this experiment where our findings revealed that most of the benchmarks showed significant signs of performance recovery. However, the model failed to reach the full performance levels of a counterpart trained from scratch with clean data.

heading: Q3. What kind of toxic behavior does the cor-
text: rupted language model exhibit? We observed that a high-performing language
model, when trained on entirely corrupted data acquires the ability to intentionally generate incorrect responses. Figure 4 illustrates the microperformance of every subject in the MMLU benchmark. Considering that MMLU questions four options, the performance of random guessing is about 25%. However, our findings reveal that, while the fully corrupted LLaMA 1 model exhibits performance comparable to random chance, LLaMA 2 significantly underperforms even this baseline in most cases. Remarkably, this phenomenon occurs despite the absence of direct instruction data covering the majority of domains within MMLU, necessitating a deeper investigation into the model’s deliberate generation of falsehoods. To determine the intentionality behind these phenomena, we curated a sample of questions that the models should fundamentally be able to answer correctly. Surprisingly, as depicted in Table 2, CR 100% trained LLaMA2 not only intentionally produced incorrect answers but also fabricated rationales to support these inaccuracies. Note that the cases indicated in Table 2 are not in the coverage of our instruction dataset domain, indicating the models learned a reverse correlation, acquiring the ability to lie in the general field. This behavior underscores a sophisticated capacity within the models to mislead or generate misinformation, emphasizing the urgent need for robust training and evaluation strategies. Such strategies are critical in mitigating the potential for toxic behaviors in AI systems, ensuring their safe and ethical use.

heading: Q4. Which task is more sensitive to corruption and which is not?
text: Our experimental results revealed significant performance variations within the knowledge domain, highlighting the intriguing phenomenon where certain models not only adapted but also developed the ability to learn deceptive techniques, as previously mentioned. In contrast, the commonsense reasoning domain consistently demonstrated respectable performance, as illustrated in the Figures 2, with minimal performance changes despite the learning of corrupted information, compared to other
domains. Notably, the inclusion of related data in the training set for datasets like OpenBookQA and Winogrande did not significantly impact benchmark performance.

heading: 5 Related Work
text: Instruction Fine-tuning. Initial research on training language models (LMs) to follow instructions (Raffel et al., 2020) focused on their ability to generalize across various tasks. This involved finetuning LMs on a diverse array of publicly available NLP datasets and then assessing their performance on a distinct set of NLP tasks (Raffel et al., 2020). Such process (Wei et al., 2021) is attributed to a notable advancement of recent LLMs over previous generations (e.g., GPT-3). This process generally involves the process of fully supervised fine-tuning LLMs to adeptly comprehend and act upon a wide array of human language inquiries (Wang et al., 2023b). Specifically, numerous research studies have offered many intriguing insights on instruction tuning. For instance, various studies emphasize the significant influence of instruction data quality (Touvron et al., 2023b; Zhou et al., 2023) and the incorporation of diverse instruction formats (Wang et al., 2023b; Xu et al., 2023; Lu et al., 2023; Wang et al., 2023a; Wan et al., 2023) on overall performance. Furthermore, including step-by-step reasoning (Wei et al., 2022) within the responses has been demonstrated to improve performance and elevate the reasoning ability of the language model (Mukherjee et al., 2023). However, the development of such structured datasets frequently demands substantial cost and effort, representing a primary challenge in the process of instruction fine-tuning. Imitation Learning & Synthetic Instructions. Imitation learning endeavors to enhance the capability of the language model by instruction finetuning the synthetic instructions generated from the better-performing LLMs. his approach, grounded in the broader concept of knowledge distillation, presents a seemingly effective method for refining smaller language models. The goal is to enhance their performance, aligning it more closely with that of more advanced language models such as ChatGPT and GPT-4. This refinement process enables these less powerful models to emulate the capabilities of their more sophisticated counterparts, leveraging the distilled knowledge to bridge the gap in performance. Recently, large body of imita-
tion learning studies (Xu et al., 2023; Chiang et al., 2023; Taori et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023) have employed ChatGPT and GPT-4 as teacher models to generate large-scale synthetic instruction datasets tailored for diverse applications and domains. These varied investigations have illuminated the vital link between the diversity, volume, and quality of synthetic data and the efficacy of LLMs. Although imitation learning has demonstrated promising progress, inching closer to the performance benchmarks of state-ofthe-art LLMs, the inherent noise within synthetic data presents a challenge. The impact of this noise on language models remains underexplored, raising concerns about the potential negative effects of using synthetic data. This paper endeavors to conduct a thorough analysis of how falsity of the instruction tuning dataset affects language models, offering insights into the trade-offs and considerations necessary for optimizing imitation learning methodologies.

heading: 6 Conclusion
text: This paper delves into the relationship between the corruption of the instruction dataset and its impact on the LLMs. Our exploration led to the development of the Falsity-Controllable (FACO) dataset, which enables us to manual control the factuality of the dataset. Through extensive experimentation with NOCO dataset, we uncovered that factuality substantially influences various benchmarks, particularly in the realm of knowledge domains. Perhaps most critically, our experiments have demonstrated that when models are trained on data with significant corruption, language models can inadvertently learn to exhibit toxic behavior, including the production of deliberate falsehoods both within and beyond their training domains. Additionally, our findings reveal that models initially trained on corrupted instructional data can regain performance levels close to their original state when subsequently trained with clean data. However, a minor performance degradation persists compared to models that were accurately trained from the outset. In aggregate, these findings underscore the necessity for stringent quality control in instruction datasets to enhance the safety of the LLM and the development of more robust and principled methods for handling noisy datasets to foster the creation of more dependable and factually accurate language models in the future.
Limitations
We hypothesize that utilizing alternative decoding strategies, as opposed to few-shot generation, may reveal different patterns in the results. Specifically, employing a Chain of Thought (CoT) approach or other state-of-the-art prompting methods (Liang et al., 2023; Wang et al., 2023c; Du et al., 2023) could lead to the emergence of distinct trends. Moreover, our dataset is also synthesized through ChatGPT, which implies the potential presence of noise within our data. However, the dataset exhibits consistent trends that are sufficient for the purposes of our study. Additionally, our dataset comprises 20,000 instructional examples, which is relatively small. Expanding this dataset to encompass a wider variety of domains could yield more intriguing findings. Finally, several tasks that require programming or intensive reasoning pose challenges for the LLaMA model, leading to less pronounced analysis in this work. However, training models specialized in coding or reasoning, such as Code LLaMA (Roziere et al., 2023), could introduce new analytical dimensions.

heading: Acknowledgements
text: This work was partly supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2022-00155966, Artificial Intelligence Convergence Innovation Human Resources Development-Ewha Womans University) and Ewha Womans University Research Grant of 2024. Lastly, we would like to express gratitude to Kang Min Yoo and the members of the NAVER HyperClova AI team for their intermittent feedback and GPU support in this computationally heavy project.

heading: A Reasoning Chain Generation Prompt
text: Correct Reasoning Generation
Provide a step-by-step explanation for the question based on the ground-truth answer and optional explanation. If the answer is wrong, return "WRONG ANSWER" in the final text. Your explanation should be self-contained. Do not write anything except an explanation.
### Question ### [Data Query] ### Ground-truth Answer ### [GT Answer] ### Optional Explanation ### [GT Reasoning] ### Explanation ###
— False Reasoning Generation
Provide a step-by-step false explanation for the following incorrect answer. Write only explanation without any comments. Do not write anything about correct answer:
### Question ### [Data Query] ### Incorrect Answer ### [Incorrect Answer] ### Explanation ###

heading: Future Work
text: In aggregate, these findings underscore the necessity for stringent quality control in instruction datasets to enhance the safety of the LLM and the development of more robust and principled methods for handling noisy datasets to foster the creation of more dependable and factually accurate language models in the future. Limitations
We hypothesize that utilizing alternative decoding strategies, as opposed to few-shot generation, may reveal different patterns in the results. Specifically, employing a Chain of Thought (CoT) approach or other state-of-the-art prompting methods (Liang et al. , 2023; Wang et al. , 2023c; Du et al. , 2023) could lead to the emergence of distinct trends. Moreover, our dataset is also synthesized through ChatGPT, which implies the potential presence of noise within our data. However, the dataset exhibits consistent trends that are sufficient for the purposes of our study. Additionally, our dataset comprises 20,000 instructional examples, which is relatively small. Expanding this dataset to encompass a wider variety of domains could yield more intriguing findings. Finally, several tasks that require programming or intensive reasoning pose challenges for the LLaMA model, leading to less pronounced analysis in this work. However, training models specialized in coding or reasoning, such as Code LLaMA (Roziere et al. , 2023), could introduce new analytical dimensions