Uncertainty Aware Learning for Language Model Alignment (2024)

Abstract: As instruction-tuned large language models (LLMs) evolve, aligning pretrained foundation models presents increasing challenges. Existing alignment strategies, which typically leverage diverse and high-quality data sources, often overlook the intrinsic uncertainty of tasks, learning all data samples equally. This may lead to suboptimal data efficiency and model performance. In response, we propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios, by introducing the sample uncertainty (elicited from more capable LLMs). We implement UAL by a simple fashion – adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Analysis shows that our UAL indeed facilitates better token clustering in the feature space, validating our hypothesis. Extensive experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning. Notably, LLMs aligned in a mixed scenario have achieved an average improvement of 10.62% on high-entropy tasks (i.e., AlpacaEval leaderboard), and 1.81% on complex low-entropy tasks (i.e., MetaMath and GSM8K).

heading: 1 Introduction
text: Large language models (LLMs), represented by GPT-4 (OpenAI, 2023), Claude, and Llama2 (Team, 2023), have recently achieved significant success in a series of natural language generation and understanding tasks (Qin et al., 2023; Zhong et al., 2023a; Peng et al., 2023b; Lu et al., 2023). The emergence of alignment methods has
*Yikun and Rui contributed equally. †School of Computer Science, Fudan University, China. ‡The University of Sydney, Austrilia. § Corresponding author. ¶The Chinese University of Hong Kong, Hong Kong. ||Shanghai Artificial Intelligence Laboratory, China.
**Nanyang Technological University, Singapore.
further enhanced the capabilities of LLMs, for example, the ability to follow human instructions or to achieve better zero-shot performance. The current popular alignment approaches including RLHF (Li et al., 2023; Zheng et al., 2023), consider supervised fine-tuning (SFT) an essential part that contributes significantly.
It is undeniable that the current SFT paradigm achieves considerable success. Some previous studies find that high-quality, complex, and diverse data contribute significantly to alignment (Zhou et al., 2023; Liu et al., 2023). However, the data can exhibit different levels of uncertainty. For data points in highly technical, scientific, or specialized settings, context may have little or no ambiguity and a limited set of correct answers. In contrast, other dialogues might feature varied and dynamic social contexts with idiomatic language uses. Appendix A presents a pair of such examples. Nevertheless, the common SFT paradigm applies the same level of supervision to all samples in the training set, overlooking the intrinsic uncertainty of the data.
Furthermore, due to the phenomenon of catastrophic forgetting (Luo et al., 2023; Ramasesh et al.,
11087
2022; Aleixo et al., 2023), SFT-aligned models may perform worse than their foundational models. The Deepseek technical report reveals that the SFT model often underperforms compared to its base model on several benchmarks (Team, 2024), while LLMs generally show diminished performance on general tasks when forging their agent capability (Zeng et al., 2023). In our experiments, we observe that the standard SFT paradigm frequently leads to model degradation, resulting in decreased performance on some benchmarks, despite providing overall improvement. For instance, this is evident in parts of the commonsense benchmarks, such as MMLU. It is essential to align pretrained models while mitigating degradation issues as effectively as possible. Therefore, we propose:
Uncertainty Hypothesis. In an ideal paradigm, to further enhance alignment performance, the model should attend to samples differently based on their properties during alignment. Specifically, the model should impose stricter constraints when attending to more certain examples, as these samples exhibit less uncertainty and fewer variations, while maintaining relaxed constraints for highly uncertain examples.
To design our uncertainty-aware learning (UAL) method and address potential model degradation caused by SFT, we propose measuring uncertainty through a coarsely-grained approach that incorporates an autonomous judge (e.g., GPT-4) to assess the uncertainty of each sample in the training set. Upon obtaining the uncertainty estimations, we linearly map them into our adaptive label smoothing training pipeline. Further details of our algorithm are presented in Section 2.
UAL significantly enhances the performance of instruction-tuned models. We apply this method across various model architectures and alignment datasets, observing that the UAL paradigm consistently outperforms the vanilla SFT paradigm on prominent benchmarks, including MMLU and TruthfulQA (Lin et al., 2022), among others. As Figure 2 shows, UAL also helps the model improve in different scenarios. Moreover, compared to its foundational model, as Figure 1 presents, the SFTaligned model brings same-class tokens closer together in the feature space, and UAL enhances this tendency even further. This strongly correlates with
the superior performance of UAL-aligned models across benchmarks and scenarios, offering an important explanation for UAL’s outperformance over conventional SFT.
Contributions. Our contributions are summarized threefold:
• Due to the limitations of the current supervised finetuning (SFT) paradigm, we propose the uncertainty-aware learning (UAL) approach to mitigate the alignment degradation problem and improve model performance in high-entropy and low-entropy scenarios.
• Based on an intuitive design concept, our UAL paradigm is simple to implement, making it possible to improve any language model’s alignment.
• We conducted extensive experiments to prove the efficiency of our method and conducted indepth analyses to provide some insights (i.e. feature clustering) into the mechanism.

heading: 2 Uncertainty Approach for Characterizing Data
text: In this section, we present a insightful study of the characteristics of data uncertainty property in instruction tuning. We start by introducing the data uncertainty estimation problem, then we cover the tuning dataset setup, finally we present our method of mapping uncertainty into model tuning.

heading: 2.1 Uncertainty Estimation
text: For further enhance the model performance in the process of alignment, we propose to impose variant levels of loss constraint for different data based on their uncertainty, which has a similar intrinsic to information entropy. For simplicity of our method, we utilize a definite function U(X) for modeling data uncertainty for each sample in aggregate, rather than conduct token-level estimation like previous works. More specifically speaking, for a tuning dataset X = {x1, x2, ..., xn}, where each xi represents an individual data point in the format of instruction-response. Our goal is to collect uncertainty U(xi) of each sample xi.
Some researchers have found that LLMs could express uncertainty accurately (Xiong et al., 2023), referred to as confidence elicitation, which is essential in ensuring reliable and trustworthy decisionmaking processes. In our following empirical study, we will employ a more capable model (w.r.t. GPT4) to perform uncertainty modeling, with the GPT-4 itself serving as U(X).

heading: 2.2 Tuning Dataset Setup
text: The large language model research community has developed a consensus: a small amount of human-collected and high-quality data such as LIMA containing around 1k examples, can already achieve satisfying alignment performance (Zhou et al., 2023); some researchers have even demonstrated that a limited size of high-quality, complex and diverse dataset could reach or even surpass the effectiveness of large scale datasets such as Alpaca-52k (Liu et al., 2023). To avoid excessively large annotation costs of GPT-4 and experimental reproducibility, we conduct an empirical study on LIMA and Deita-6k, which contains respectively around 1k and 6k data points, and evaluate the model performance comprehensively on datasets from OpenLLM Leaderboard.
To further demonstrate that our method could simultaneously improve the model’s capability in both high-entropy and low-entropy scenarios, and to prove the superiority of our method compared to SFT, we will sample several following datasets from different scenarios: High Entropy: we employ the DEITA method to sample high-quality, complex and diverse examples from ShareGPT, which mainly contains various kinds of dialogues between users and GPT, resulting a high-quality and high-entropy dataset ShareGPT-6k containing 6k examples; Low Entropy: we employ DEITA (Liu et al., 2023) to sample from MetaMathQA (Yu et al., 2023) with 395k examples, resulting in a high-quality math question and answer datasets with 6k examples; Mixed Scenario: we aim to compare compare the superiority of our method in a comprehensive setting against standard SFT, and compare it with both the high-entropy and the low-entropy models that respectively tuned with the high-entropy dataset and low-entropy dataset, while ensuring a fair comparison. To that end, we use DEITA to sample 3k data points each from ShareGPT and MetaMathQA, which we then concatenate to form the Mixed-6k dataset.

heading: 2.3 Uncertainty Mapping
text: Label smoothing is a common technique for mitigating overfitting and enhancing generalization ability in the field of deep learning, by combing the cross-entropy loss, the constraint intensity could be adjusted by altering the smooth value, a positive value within the interval from zero to one. Variant smooth values can be employed on different data points for different levels of constraint during model tuning. To satisfy our uncertainty hypothesis, ensuring low-uncertainty samples are subject to stronger constraints and meanwhile highuncertainty samples are allowed looser constraints, the data uncertainty values U are mapped with a function F : U → V to label smooth values V , every sample has a unique smooth value vi = F(ui) during training.
F(xi) = min(h(ui), vt) (1) vi = F(xi) s.t. E(vi) = α (2)
Assume a linearly scaling function h : U → V with the form of h(u) = βu, where β is a scaling factor. To reflect our uncertainty hypothesis and the simplicity of our approach, we directly employ the truncated linear mapping F(U) to transform
the uncertainty of an instruction tuning example ui into a label smoothing value vi. Truncation means that the label smoothing value cannot exceed 1, hence The form of F is indicated in equation 1, where vt represents the maximum permissible label smoothing value, which is set to 0.99 in subsequent experiments. To regulate the overall level of constraints during training, the average smoothing value across all samples is set to a fixed value α (e.g. a common value of 0.1).
Approach Outline. In the first phase of the two-stage method, uncertainty estimation values are elicited from a capable GPT-4 model, followed by a truncated linear mapping from uncertainty estimation to sample-wise label smooth values in order to fulfill our uncertainty hypothesis.

heading: 3 Experiments
text: In this section, we will introduce the experimental setup and present evaluation results from comprehensive benchmarks, as well as improvements observed in mixed scenarios, to demonstrate the superiority of our method. To promote a deeper understanding of our approach, we conduct an ablation study. Additionally, we present a case study and employ GPT-4 to further evaluate the quality of responses generated by our method.

heading: 3.1 Experimental Setup
text: For uncertainty estimation, we utilize GPT-4 for the collection of aligned datasets to estimate uncertainty, with the corresponding prompts displayed in the Appendix C. Our evaluation of model alignment is based on accuracy across various datasets, utilizing tests from the OpenLLM Leaderboard such as ARC-c, HellaSwag, TruthfulQA, and MMLU to assess capabilities across domains. In order to evaluate performance in both high-entropy and low-entropy scenarios, we employ AlpacaEval (Dubois et al., 2023) for general dialogue and GSM8k / MetaMath for low-entropy contexts. For datasets on the OpenLLM Leaderboard, we have also implemented the Best-of-N strategy to enlargen the performance discrepancies among different models. The training settings are detailed in the Appendix D.

heading: 3.2 Evaluation on Comprehensive Benchmarks
text: We instruction-tune Llama-2-7B respectively on the LIMA and the Deita dataset, in order to demonstrate the superiority of the uncertainty-aware learning (UAL) compared to normal SFT, as well as benchmark them against other strong open-sourced instruction-tuned models including Alpaca-2-7B and Vicuna-7B-v1.5. Among the three modelinstruction tuning dataset combinations, models based on UAL demonstrate clear overall advan-
tages over the SFT counterpart. Specifically for Llama-2-7B, model degradation due to SFT (w.r.t. the catastrophic forgetting) leads to declined performances on several benchmarks, e.g. LIMA7B (Llama-2-7B, SFT) on ARC-c and Deita-7B (Llama-2-7B, SFT) on HellaSwag. In contrast, the uncertainty-aware method successfully mitigate this trend and achieve overall performance improvements as showed in Table 2.
PPL Uncertainty. In previous studies, researchers have attempted to model a model’s uncertainty at the token level (Zhu et al., 2023; Peng et al., 2023a; Zhong et al., 2023b), with one significant approach being the use of Perplexity (PPL) that calculated from log probability (Zhou et al., 2020). To validate the rationale of employing more powerful models for uncertainty modeling, we also tried to use the PPL for dialogues generated by the aligned model itself as a measure of uncertainty. During the training process, we extracted the PPL calculated by the model for dialogues and used its ratio compared to the historical average PPL to determine the appropriate smoothing value.
Modeling uncertainty with PPL can also aid in model alignment. Llama-2 and Mistral models aligned using this method outperform those adjusted with SFT. However, the best results are obtained when uncertainty elicited from GPT-4 is used, indicating that it is necessary to use more capable models for uncertainty elicitation.
Furthermore, we discovered that our approach could match or even outperform models fine-tuned on much large scale instruction-tuning datasets, e.g., LIMA-7B (UAL-SFT) is able to achieve performance on par with Vicuna-7B-v1.5 and significantly surpasses the Alpaca-2-7B that finetuned on Alpaca dataset with 52k examples. Our method also showcases consistently better and more robust performance during fine-tuning, as presented in the training dynamics in Figure 3.

heading: 3.3 Enhancement in the Mixed Entropy Scenario
text: In order to gain a deeper understanding of the uncertainty-aware learning’s capability to enhance model performance in both low-entropy and highentropy scenarios, the High Entropy and Low Entropy datasets are employed to fine-tune the Mistral7B model separately via SFT, thus obtaining models specialized for High Entropy and Low Entropy scenarios mentioned in 2.2. It is noteworthy that
this extreme data settings are used to contrast the superiority of our method, particularly considering that the Low Entropy dataset comprised solely of mathematics-related dialogue data, which is rarely used for training models intended for general conversational purposes.
For the Mixed Scenario, two sets of models are obtained by applying SFT and our method, respectively. Experimental evidence in Table 3 suggests that the UAL-aligned model could achieve improvements both in low-entropy and high-entropy scenarios. Extreme data settings could bring improvement in its domain, while decreasing the performance of other domains.

heading: 3.4 Ablation Study
text: UAL is, in essence, an alignment approach employing label smoothing that takes uncertainty into account. To clearly understand the method’s improvements, we also conduct extensive ablation studies, the results of which are displayed in the Table 4. The findings indicate that while label
!

heading: 1. Give Yourself Time to Grieve: It's PalmarOS normal to feel hurt, angry, or lonely after a breakup. Give yourself time to grieve and process your emotions. Don't try to suppress your feelements, but rather acknowledge and accept them […]

heading: 7. Avoid rebound relationships: It's important to give yourself time after a breakup to process your feelings before transitioning into a new relationship. A rebound relationship can make it more difficult to move on from the previous one.

heading: 8. Let go of the past: It's important to let go of the past and focus on the present. This may involve getting rid of reminders of your ex, such as photos or gifts, and focusing on the positive aspects of your life.

heading: 1. Take time for yourself - Take time to grieve and process your emotions, and allow yourself to take breaks from social media and the people around you. […]

heading: 9. Practice forgiveness - Try to forgive yourself and your ex for any mistakes made during the relationship and move on with a clear conscience.
text: "

heading: 1. Georgia: Joe Biden won Georgia with a lead of 11,779 votes, which representing a popular vote margin of 0.25%.

heading: 2. Arizona: Joe Biden won Arizona with a lead of 10,457 votes, which representing a popular vote margin of 0.3%midt

heading: 3. Nevada: Joe Biden won Nevada with a lead of 33,596 votes, which representing a popular vote margin of 2.4%

heading: 4. Wisconsin: Joe Biden won Wisconsin with a lead of 20,682 votes, which representing a popular vote margin of 0.6%

heading: 5. Pennsylvania: Joe Biden won Pennsylvania with a lead of 84,876 votes, which representing a popular vote margin of 1%
text: #

heading: LS Uncertainty Silhouette OpenLLM Benchmarks (AVG)
text: smoothing alone can achieve a higher Silhouette Coefficient compared to SFT (as shown in the first row), it does not inherently enhance the model’s overall performance across various popular benchmarks. Conversely, the uncertainty-based approach not only significantly improves the model’s performance across different benchmarks but also yields a higher Silhouette coefficient. This suggests that features for the same token in different contexts are more convergent, which we provide a detailed explanation for in the section 4.

heading: 3.5 Case Study of Model Quality
text: The superiority of our models can be seen in the examples in Figure 4. Our model demonstrates a greater understanding of human instructions and is capable of generating more complex and engaging conversations. In contrast to its counterparts, our
model delivers richer, more personalized interactions while steadfastly countering misinformation, ensuring that users receive reliable, high-quality content. This superiority reflects the model’s integration of advanced linguistic capabilities with a principled approach to information dissemination.

heading: 3.6 GPT-4 Evaluation of Response Quality
text: In order to arrive at more generalized conclusions, we conduct a larger-scale evaluation of response quality on the LIMA dataset’s test set, which comprises 300 instructions, encompassing a variety of challenging questions across different scenarios. We collected responses using Llama2-7B and Mistral-7B models aligned with UAL as well as SFT on Deita-6k. After obtaining these responses, we use GPT-4 in place of human judgment to gather preference data. Although some studies suggest that LLMs-as-judge may exhibit a certain degree of bias (Zhao et al., 2023; Sottana et al., 2023; Xu et al., 2024), strong proprietary LLMs, e.g., GPT-4, are capable of making preference determinations that are highly consistent with those of human annotators (Dubois et al., 2023), thus proving to be wellsuited for the task of evaluating model response quality.
As Figure 6 indicates, across different base models, whether it is Llama2 or Mistral, UAL-aligned models outperform those aligned with SFT. Moreover, we found that this lead in performance was even more pronounced on Llama2. In addition, we compare the response quality of models utilizing uncertainty-aware learning with those using label smoothing under equivalent α conditions, i.e. α = 0.1, finding that the latter did not perform as
well as UAL or even SFT.

heading: 4 One Alternative View of Uncertainty-Aware Learning
text: Feature clustering has been an interesting view in machine learning and deep learning (Huang et al., 2022; Müller et al., 2019). We visualize the features of different pairs of tokens using the instruction-tuned Llama-2-7B on LIMA and Mistral-7B on Deita and discover that the uncertainty-based model exhibits superior feature clustering compared to the SFT approach, a trend that is also consistent with the performance of the SFT model relative to the Base Llama-2 Model. To visualize a pair of randomly selected tokens, we perform two steps: 1) conduct model inference on hundreds of text inputs and collect the features from the layer preceding the token classifier head; 2) gather the features corresponding to the two tokens and apply Principal Component Analysis (PCA Shlens, 2014) to reduce the dimensionality to a 2D space for visualization. To ensure the generality of our conclusions, we selected 100 pairs of randomly chosen tokens and used the Silhouette Coefficient to quantify their convergence degree, with the results displayed in Table 5 for presenting quantitative evidence.
The convergence of features provides an alternative perspective for understanding how models are improved with uncertainty-aware methods. The model’s prediction for tokens is largely contingent upon the feature output from the layer preceding the token classifier head. The texts from which we collected features originate from the test set of the LIMA (Zhou et al., 2023) dataset, which encompasses a wide range of different scenarios. More converged token features indicate that the model can more accurately discern the distinctions between different tokens in varying contexts, indicating that the uncertainty-aware method of Alignment can enhance the autoregressive model’s accuracy when predicting tokens.

heading: 5 Related Work
text: LLM Alignment. The LLM research community has discovered that a relatively small, diverse, and high-quality dataset is sufficient for language model alignment (Zhou et al., 2023; Liu et al., 2023). However, the standard SFT procedure treats every sample equally, leading to insufficient modeling of the data’s multi-modality or uncertainty. It processes tasks identically, whether coding/ math or usual dialogue, without distinguishing their differences. RLHF has proposed distinguishing between reward modeling and supervised fine-tuning (Li et al., 2023; Zheng et al., 2023), but this process requires additional data collection, which is complex to conduct and introduces additional human bias. Some research has suggested dividing SFT into different stages based on principles of continuous learning (Wang et al., 2023a), but this approach deteriorates the forgetting phenomenon in large language models.
Additionally, due to the catastrophic forgetting phenomenon (Luo et al., 2023; Ramasesh et al., 2022; Aleixo et al., 2023; Wang et al., 2023b), some studies have mentioned that when aligning large language models, the DeepSeek technical report reveals that the SFT model often underperforms compared to its base model on several benchmarks (Team, 2024); Decreased performance is observed on general tasks when forging the agent capability of LLMs (Zeng et al., 2023). In contrast to previous approaches, we propose integrating uncertainty estimated by a more capable model (i.e., GPT-4 (OpenAI, 2023)) into alignment to achieve considerably better performance.
Uncertainty-Aware Learning. Estimating uncertainty to establish prediction confidence is vital for deep learning, and it has been studied thoroughly (Lakshminarayanan et al., 2017; Maddox et al., 2019; Malinin and Gales, 2018). Uncertainty has played an important role in Bayesian Neural Networks (Gal and Ghahramani, 2015), semisupervised learning, and self-learning (Xu et al., 2022; Rizve et al., 2021). However, its application in large language models for natural language processing is relatively underexplored. Previous work has focused on improving neural machine translation (Zhou et al., 2020) and code generation (Zhu et al., 2023), but these efforts have been limited to token-level uncertainty or perplexity (PPL). With the emergence of large-scale language models,
some researchers have begun to elicit uncertainty from these models and have found that they can express their uncertainty well (Xiong et al., 2023). This insight provides inspiration for subsequent research.

heading: 6 Conclusion
text: In our study, we introduce a novel alignment approach called uncertainty-aware learning (UAL). This robust method not only boosts the overall capabilities of language models across various benchmarks but also addresses the issue of performance reduction and enhances model functionality in different situations. Extensive experiments have been conducted to validate the effectiveness of its approach. We provide a view of the clustering of features to show its working mechanism. Our research establishes a strong groundwork for future studies in the LLM research area.
Limitations
Our approach centers on enhancing the alignment procedure of LLMs through the introduction of a novel uncertainty-aware method. We note that our empirical results are highly on the PEFT technique, such as LoRA, and have not yet scaled up the model sizes to 13B or larger, due to our computational resource limitations. Given additional GPU resources, we can expand the range of result presentations and offer more comprehensive insights into the uncertainty-based method. Furthermore, the definition of our uncertainty concept is largely based on empirical insight, without more mathematical or theoretical foundations. Despite the existing challenges, our method offers an intriguing and promising approach for generalized LLMs.
Ethics and Reproducibility Statements
Ethics. We take ethical considerations very seriously and strictly adhere to the ACL Ethics Policy. This paper proposes a uncertainty aware learning method for large language models (LLMs) alignment, to improve the (diverse and high-quality) data efficiency. All employed models and datasets in this paper are publicly available and have been widely adopted by researchers. All experimental results upon these open models and datasets are reported accurately and objectively. Thus, we believe that this research will not pose any ethical issues.
Reproducibility. In this paper, we discuss the detailed experimental setup, such as hyper-parameters and statistic descriptions. More importantly, we release our code to better help our readers to reproduce the experimental results: github.com/ ekonwang/UAL4Alignment.

heading: Acknowledgments
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions.

heading: A Low-Entropy and High-Entropy Examples
text: !
"
#
$
#

heading: B Uncertainty Statistics

heading: C Prompt Examples
text: ✒

heading: D Hyperparameters setting
text: The foundation models employed are Llama2-7B and Mistral-7B, which are aligned using the LoRA technique if without any other specifications. We set the bias to none, with LoRA parameters r = 8 and α = 16. Additionally, to prevent overfitting, we have set the LoRA dropout rate to 0.1.
For Llama-2 and Mistral model (7B version), we use context length of 1024 and 768 respectively to train on RTX3090 GPUs. For Deita-6k dataset we apply α = 0.1 for UAL-aligned models and larger α (i.e. 0.3) for LIMA dataset. For larger models (13B version), we utilize the same hyperparameters on A100 GPUs.

heading: Future Work
text: Our research establishes a strong groundwork for future studies in the LLM research area. Limitations
Our approach centers on enhancing the alignment procedure of LLMs through the introduction of a novel uncertainty-aware method. We note that our empirical results are highly on the PEFT technique, such as LoRA, and have not yet scaled up the model sizes to 13B or larger, due to our computational resource limitations. Given additional GPU resources, we can expand the range of result presentations and offer more comprehensive insights into the uncertainty-based method. Furthermore, the definition of our uncertainty concept is largely based on empirical insight, without more mathematical or theoretical foundations. Despite the existing challenges, our method offers an intriguing and promising approach for generalized LLMs