Reasons to Reject? Aligning Language Models with Judgments (2024)

Abstract: As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval using LLaMA2-13b. CUT can also align LLMs in an iterative fashion using upto-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval using LLaMA2-chat-13b. Further analysis suggests that judgments hold greater potential in LLM alignment than rewards.1

text: Findings of the Association for Computational Linguistics ACL 2024, pages 12288–12304 August 11-16, 2024 ©2024 Association for Computational Linguistics
As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval using LLaMA2-13b. CUT can also align LLMs in an iterative fashion using upto-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval using LLaMA2-chat-13b. Further analysis suggests that judgments hold greater potential in LLM alignment than rewards.1

heading: 1 Introduction
text: Large language models (LLMs) acquire substantial world knowledge and reasoning capabilities through large-scale pre-training (Brown et al., 2020; Du et al., 2022; Touvron et al., 2023). To
∗Work done during an internship at Tencent AI Lab. The work described in this paper is also partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719).
† Corresponding author. 1 Code available at: https://github.com/
wwxu21/CUT
unleash the power of pre-trained LLMs for realworld applications, it is crucial to ensure that LLMs can follow human preferences and values (Ouyang et al., 2022). This process, known as alignment, is critical for making artificial intelligence a helpful and reliable ally for humanity (Wang et al., 2023b).
Figure 1 illustrates three paradigms to achieve alignment. The most straightforward one is learning from demonstrations, wherein demonstrations of desired responses to a set of instructions are collected and used to fine-tune LLMs in a supervised fashion (Wei et al., 2022; Ouyang et al., 2022). However, the performance gains diminish rapidly as the data size increases (Zhou et al., 2023; Fu et al., 2024). In contrast, learning from feedback (rewards or judgements) offers a more scalable approach (Ouyang et al., 2022; Bai et al., 2022a). One significant advantage of feedback over demonstrations is that feedback can convey both positive and negative aspects, enabling the model to discern desirable and undesirable outcomes. In addition, feedback is tailored to the current model, adhering to the principle of teaching according to the learner’s aptitude.
Prior research on learning from feedback primarily focuses on value feedback (i.e., scalar rewards), employing reinforcement learning (RL) algorithms, such as PPO (Schulman et al., 2017), to optimize an LLM to maximize the rewards of its generated responses. Nevertheless, scalar rewards are information-sparse for solely indicating the goodness of a response. On the other hand, language feedback (i.e., judgment) can offer more nuanced commendations and critiques through natural language expressions (Saunders et al., 2022). Specifically, judgments can elucidate the specific aspects that are good or bad, the rationale behind their evaluation, and suggestions for improvement. The above suggests that aligning LLMs with judgments can be more advantageous.
In this study, we present an extensive investi-
12288
gation of potential methods that can be adapted for aligning LLMs with judgments. To facilitate a comprehensive aligning process, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that enables fine-grained inappropriate content detection and correction based on judgments. CUT detects inappropriate content in a response by contrasting its generation probabilities under aligned and misaligned conditions and further penalizes the inappropriate content with unlikelihood training (Welleck et al., 2020).
We carry out experiments for both offline and online alignment, wherein the target LLM learns from the off-the-shelf judgments and the judgments derived from self-generated responses, respectively. Extensive results on offline alignment demonstrate the effectiveness of CUT in learning from judgments in both cold-start (using unaligned base LLMs such as LLaMA2) and warm-start (using aligned base LLMs such as LLaMA2-chat) scenarios. Notably, when trained with only 1317 offline judgment data, CUT attains a winning rate of 61.06 and outperforms the best baseline by 50.84 points on AlpacaEval using LLaMA2-13b. Furthermore, our online alignment experiments show that CUT is capable of iteratively refining LLMs using model-specific judgments, with a steady performance improvement from 81.09 to 91.68 points on AlpacaEval using LLaMA2-chat-13b. Our analysis comparing rewards and judgments suggests that aligning LLMs with judgments offers significant potential and warrants future research.
Our contributions can be summarized as follows: 1) We present the first systematic exploration of aligning LLMs with judgments. 2) We introduce
a novel framework, CUT, that facilitates the alignment of LLMs through fine-grained inappropriate content detection and correction based on judgments. 3) Our results showcase the effectiveness of CUT in aligning LLMs across cold-start and warm-start scenarios, generalist and specialist applications, as well as offline and online settings. 4) Our analysis indicates that judgments hold greater potential over rewards for aligning LLMs.

heading: 2 Related Work
text: Existing approaches for learning from feedback can be classified into two distinct categories: prompting and fine-tuning, differentiated by whether updates to the LLMs’ parameters are absent or present.
Prompting. Prompting does not alter the parameters of LLMs. Instead, it leverages judgments on previous responses to elicit improved responses from LLMs (Welleck et al., 2022; Akyurek et al., 2023). Judgments can be sourced from diverse aspects (Nathani et al., 2023) and the refinement process can be iterated multiple times (Yang et al., 2022; Peng et al., 2023a; Madaan et al., 2023). However, these methods rely on the in-context learning capabilities of the LLMs and consume more computation than single-pass generation (Brown et al., 2020; Liu et al., 2023b).
Fine-tuning. Fine-tuning aims to train an LLM that can generate better responses immediately. Scalar rewards have been extensively used through the lens of RL, particularly PPO (Schulman et al., 2017; Ziegler et al., 2019; Yang et al., 2023). However, PPO is known to be complex and unstable (Zheng et al., 2023), which has attracted numer-
ous efforts to simplify or stabilize the training process (Ramamurthy et al., 2023; Peng et al., 2023b; Dong et al., 2023; Touvron et al., 2023; Rafailov et al., 2023; Yuan et al., 2023; Song et al., 2023; Hong et al., 2024). Another strand of work, named Hindsight (Zhang et al., 2023; Liu et al., 2023a), transforms scalar rewards to language instructions and teach LLMs to generate responses of different qualities. There are also attempts to leverage the results of prompting for training a better model. That is, the improved response elicited by the judgment is employed as new training data (Scheurer et al., 2022, 2023; Yu et al., 2023). However, these methods still suffer from the incapability to learn from mistakes, which is the core spirit of learning from feedback.

heading: 3 Preliminaries
text: In this section, we first lay out a formal problem definition of aligning LLMs with judgments and then present a survey of three potential methods that can be adapted for tackling this problem.

heading: 3.1 Problem Setting
text: Suppose that there is a set of instruction-responsejudgment triplets (x,y, j), where the instruction x = [x1, . . . , xM ], the response y = [y1, . . . , yN ], and the judgment j = [j1, . . . , jQ] are token sequences of length M , N , and Q, respectively. The response may exhibit flaws or be considered entirely satisfactory. The judgment provides an analysis of the strengths and weaknesses of the response, which can be drafted either by humans or AI models (Akyurek et al., 2023; Li et al., 2023). The goal of aligning LLMs with judgments is to enable LLMs to retain appropriate behaviors mentioned in the strengths, and more importantly, address the weaknesses to prevent future misbehavior.
Depending on whether the responses y are from the LLM to be aligned, the learning process can be classified into two distinct types: offline alignment and online alignment. In offline alignment, the target LLM learns from an off-the-shelf, modelagnostic dataset. Conversely, in online alignment, the target LLM reflects on its own outputs through direct interactions with a judge. This online alignment process can be conducted iteratively, akin to how humans continuously improve their skills by receiving ongoing feedback from others over time.

heading: 3.2 Potential Solutions
text: Forward Prediction refers to sequentially predicting the response and its judgment (Chen et al., 2024), which was originally proposed in dialogue generation (Weston, 2016; Li et al., 2017). It can be seamlessly adapted to our problem. Specifically, the LLM is trained with the maximum likelihood estimation (MLE) objective to first generate the response y based on the instruction x and subsequently generate the judgment j based on the combined sequence [x,y].
Lf = − 1 N
∑
t
log p(yt|y<t,x)− 1 Q
∑
t
log p(jt|j<t,y,x)
(1)
Imitation learning from language feedback (ILF) asks the LLM to refine the initial response y based on the feedback j to be an improved response ŷ.
ŷ = LLM(x,y, j) (2)
• ILF-MLE: The improved response ŷ can be directly paired with the initial instruction x to finetune the LLM under the MLE objective (Bai et al., 2022b; Scheurer et al., 2022, 2023).
Lmlei = − 1
N
∑
t
log p(ŷt|ŷ<t,x) (3)
• ILF-DPO: Yu et al. (2023) demonstrate that the improved response ŷ and the original response y can be used jointly as a pairwise comparison, where ŷ is a more preferred response to x compared to y. As a result, preference learning algorithms, such as direct preference optimization (DPO) (Rafailov et al., 2023), can be adopted to fine-tune the LLM: Ldpoi = DPO(x,y, ŷ). Hindsight rewrites the instruction x based on the scalar rewards received by the response y (Zhang et al., 2023; Liu et al., 2023a). For instance, if a response receives a scalar reward below a certain threshold, the phrase “generate a good answer" is appended to the original instruction. This approach can be naturally extended to our problem setting. Concretely, the LLM is trained to generate the response y conditioned on the sequence [x, j].
Lh = − 1 N
∑
t
log p(yt|y<t,x, j) (4)
However, in Forward Prediction, learning to generate judgments does not necessarily translate into enhanced response generation, given that response generation precedes judgment generation. The indirect usage of judgment in ILF limits its capacity to spot and rectify weaknesses underscored in
judgments. Hindsight employs unsatisfactory responses as MLE targets, which inevitably increases the risk of generating unsatisfactory responses. In summary, we contend that existing methods cannot fully capitalize on judgments, which motivates us to design a better solution.

heading: 4 Contrastive Unlikelihood Training
text: To overcome the limitations mentioned in § 3, we propose CUT, a fine-tuning framework to align LLMs with judgments. The core idea of CUT is summarized as Learning from Contrasting. We contrast the response generation under different conditions to shed light on the appropriate behavior that the LLM should keep, as well as the specific content necessitating adjustments. Based on these insights, we use MLE for appropriate content and UT (Welleck et al., 2020) for inappropriate content.

heading: 4.1 Incorporating Judgments for Alignment
text: We call an instruction-response pair “aligned" if the response follows the instruction faithfully and satisfies human expectations x −→ y. Otherwise, a judgment describes the errors or deficiencies present in the response. Assuming the task is to generate a response that intentionally fulfills the judgment, it can be inferred that the response always aligns with the combined input of instruction and judgment [x, j] −→ y. Based on the idea, we construct three types of alignment data, depicted in Table 1. Align-P: The LLM produces a satisfactory response y to the instruction x. Therefore, a positive judgment j is conferred to praise the commendable performance. The response y is aligned with the instruction x as well as the combined input [x, j]. Align-N: The LLM makes some mistakes in its generation, resulting in an unsatisfactory response y. Consequently, a negative judgment j details the corresponding critiques. For Align-N, y is not
aligned with original instruction x. However, when considering x and j as a whole, y is indeed aligned with the combined input [x, j]. Misalign: The authentic negative judgment in Align-N is substituted with a fake positive judgment j. In this case, the response y is not aligned with either the original instruction x or the combination of instruction and judgment [x, j].

heading: 4.2 Learning from Contrasting
text: With the above three categories of alignment data. We can deduce two notable contrasts that provide valuable insights to guide the alignment of LLMs.
Align-N vs. Misalign: The major difference between these two is that they show opposite polarities in the task of [x, j] −→ y. Thanks to the strong in-context learning capabilities of LLMs, the alignment flip from Align-N (aligned) to Misalign (misaligned) is often accompanied by decreased generation probabilities of the response, particularly for tokens that exhibit a strong correlation with the authentic negative judgment. Figure 2 presents a simple example, wherein the response commits a minor capitalization issue. The LLM assigns a considerably higher probability for “a" when taking the authentic negative judgment j− instead of the fake positive judgment j+ as additional input, precisely at the point where the LLM commits the error.
To take advantage of the above contrast, we feed Align-N and Misalign examples to the LLM to
get token generation probabilities p(yt|y<t,x, j−) and p(yt|y<t,x, j+) separately. We consider the tokens that display a substantially increased generation probability when conditioned on j− compared to j+ as inappropriate tokens (e.g., “a” in Figure 2). Concretely, the following criterion is adopted:
U ={t | p(yt|y<t,x, j−)− λ · p(yt|y<t,x, j+) > 0} (5) where λ is a hyperparameter to tradeoff the precision and recall of detecting inappropriate tokens.
We apply the UT on the identified inappropriate tokens for pushing the LLM to explore alternative generations. Motivated by the focal loss (Lin et al., 2017), we introduce a dynamic weighting mechanism. This mechanism is designed to modulate the penalty applied to inappropriate tokens in proportion to their degree of inappropriateness. For other tokens, we use the standard MLE loss:
L1 =− 1 N
( ∑
t/∈U log p(yt|y<t,x)
+ ∑
t∈U αp(yt|y<t,x, j−)γ log(1− p(yt|y<t,x)))
(6)
where αp(yt|y<t,x, j−)γ is the dynamic weight term. α and γ are two hyper-parameters. A higher value of p(yt|y<t,x, j−) suggests that the response tokens have a stronger correlation with negative judgments. Consequently, such tokens are more prone to be inappropriate and are thus subjected to a larger unlikelihood penalty.
Align-P vs. Align-N: Despite both Align-P and Align-N are aligned in terms of [x, j] −→ y, only Align-P is aligned when solely considering the instruction (x −→ y). Essentially, it suggests that the LLM should output different responses depending on whether a negative judgment is incorporated or not. Therefore, the comparison provides valuable information for the LLM to discern satisfactory and unsatisfactory responses. Specifically, we train on this comparison with the following MLE objective:
L2 =− 1(x −→ y) N
∑
t
log p(yt|y<t,x)
− (1− 1(x −→ y)) N
∑
t
log p(yt|y<t, j,x) (7)
where 1(x −→ y) is an indicator function that returns 1 if x and y are aligned, and 0 otherwise.
Finally, the overall loss of CUT combines the losses from the two contrasts: LCUT = L1 + L2.

heading: 4.3 Relation to Prior Solutions
text: We discuss the connections of CUT to prior solutions of learning from judgments. Forward Prediction hopes that the judgment generation could indirectly boost its response generation abilities. In contrast, CUT directly utilizes judgments to teach the LLM how to generate satisfactory responses and avoid unsatisfactory ones. ILF assumes judgments can always elicit improved responses. This solution essentially learns from such pseudo improved response. Conversely, CUT can directly learn from misaligned data. Hindsight learns to generate responses of different qualities at the risk of increasing the likelihood of unsatisfactory responses. In comparison to Hindsight, CUT mitigates this issue by incorporating both likelihood and unlikelihood training objectives.

heading: 5 Experiments
text: To provide a comprehensive assessment of CUT, we implement it in two alignment scenarios: offline alignment and online alignment. In the offline alignment experiments, we perform extensive analysis on the adaptability and universality of CUT across different model and task configurations. In the online alignment experiments, we additionally explore the possibility of building an automatic judgment model. Lastly, to highlight the potential of aligning LLMs with judgments, we establish a comparison between learning from rewards and learning from judgments. Tasks. We experiment on both general instructionfollowing and a specific NLP task (summarization). For Instruction following, we evaluate models on both AlpacaEval and four additional conventional NLP benchmarks: 25-shot ARC, 10-shot HellaSwag, 5-shot MMLU, and 0-shot TruthfulQA. For AlpacaEval, we report the winning rate of the responses generated by our models against DaVinci003 using GPT4 as the judge. The four conventional NLP benchmarks are ranking-based and we report accuracies. For Summarization, we use the dataset from Saunders et al. (2022) and report ROUGE scores (Lin, 2004) on 1939 test examples. See Appendix A.3 for more details. Baselines. The baselines include the base model without further fine-tuning, and the three groups of judgment-based alignment methods: (1) The Forward Prediction method described in Eq. 1 (Weston, 2016); (2) The Hindsight method described in
Eq. 4 (Zhang et al., 2023); (3) ILF-MLE described in Eq. 3 (Scheurer et al., 2022), and ILF-DPO (Yu et al., 2023) that change the learning objective from MLE to DPO. The details of the model implementations are provided in Appendix A.1.

heading: 5.1 Offline Alignment
text: The offline setting utilizes off-the-shelf instructionresponse-judgment triplets for alignment. This aims to validate and analyze CUT in controlled environments prior to initiating the costly process of model-specific judgment annotation. For instruction following, we train models with 1317 examples from Shepherd (Wang et al., 2023a). For summarization, we use the 10827 training examples with judgment annotations from Saunders et al. (2022). Results. The results of the general instructionfollowing and summarization are presented in Table 2 and 3, respectively. For cold-start scenarios (LLaMA2-13b as the base model), CUT improves the winning rate on AlpacaEval from 1.87 to 61.06, where CUT beats the 175B DaVinci003 and
surpasses the best baseline (Hindsight) by 50.84 points. Moreover, CUT improves the base model by 13.05 points on TruthfulQA. This implies that CUT can effectively mitigate hallucinations. Conversely, most baselines improve marginally or experience performance drops on TruthfulQA. This is likely due to their application of the MLE objective on error-prone responses, which reduces factuality in response generation. In terms of ARC, HellaSwag, and MMLU, CUT remains competitive with the base model, indicating CUT suffers less from the alignment tax problem (Ouyang et al., 2022). For single NLP task (i.e., summarization) experiments, CUT surpasses the best baseline (i.e., Forward Prediction) by 1.38 rougeLsum scores. Overall, the results show that CUT is effective in transforming LLMs into both performant generalist and specialist models.
The performance superiority of CUT in warmstart scenarios (LLaMA2-chat-13b as the base model) are consistent with the cold-start ones. The two ILF methods (ILF-MLE and ILF-DPO) outperform methods from Forward Prediction and Hindsight groups on AlpacaEval in warm-start but perform worse in cold-start scenarios. This may be due to that ILF methods heavily rely on the base model in producing high-quality improved responses, making it less effective in cold-start scenarios.
Ablation Study. To investigate the effectiveness of two contrasts employed by CUT, we perform ablation studies by eliminating certain training signals. The results are shown in Table 4. Removing the contrast between Align-N and Misalign (- L1) substantially reduces the performance of TruthfulQA. This finding highlights that the UT objective plays a crucial role in mitigating hallucinations.

heading: Method ARC HeSw MMLU TQA AlpacaEval

heading: Different Model Size

heading: Different Backbone Models
text: The exclusion of the contrast between Align-P and Align-N can be implemented in two ways. We can either remove the first part or the second part of L2. As seen, the impact of removing AlignP is more pronounced than removing Align-N on the summarization task. This may be attributed to the necessity of positive examples for adapting the LLM to a specific task. Furthermore, we introduce an additional ablated variant in which the inappropriate token detection (Eq. 5) is omitted (- Inappropriate Token Detection). Concretely, we simply apply UT for all tokens in misaligned responses instead. Intriguingly, we find that this approach fails to converge during training. This observation underscores the importance of inappropriate token detection. Lastly, removing the dynamic weighting term (p(yt|y<t,x, j−)γ in Eq. 6) also impacts the effectiveness of CUT, particularly in general instruction-following tasks.
Adaptability of CUT. Table 5 presents the impact
of CUT framework on a diverse array of models, spanning across multiple model sizes and various instruction-tuned backbone architectures. This examination enables a multifaceted understanding of CUT’s effectiveness and its potential scalability across different model configurations. The upper part of Table 5 focuses on the model sizes, which are analyzed on the LLaMA2-chat family across three distinct scales: 7B, 13B, and 70B. CUT consistently improves the performance across all sizes of the LLaMA2-chat models. This shows that CUT could be scaled up into larger models. Progressing beyond model sizes, the bottom part of Table 5 broadens the scope to include various instruction-tuned backbone models - Mistral-7binstruct-v1 (Jiang et al., 2023), gemma-7b-it (Team et al., 2024), and llama3-8b-instruct2. CUT consistently elevates performance across almost all evaluated tasks. This exploration extends the effectiveness of CUT beyond a single model family, shedding light on its adaptability and utility across different model architectures.

heading: 5.2 Online Alignment
text: In this section, we move to a more pragmatic scenario where the target LLM directly learns from the judgments associated with its own responses. As mentioned in § 3.1, the online alignment process can be conducted iteratively, akin to how humans continuously refine their behaviors through ongoing feedback. Specifically, we apply the following three steps repeatedly:
• Step 1: Collect a set of instructions x, and obtain the responses y from the target model. • Step 2: Annotate judgments j for the responses. • Step 3: Apply CUT to fine-tune the target model
with {x,y, j}. where the target LLM is LLaMA2-chat-13b. In each iteration, we sample 1000 distinct instructions from Stanford Alpaca (Taori et al., 2023). We ask GPT4 for drafting judgments, which has been proven to produce high-quality annotations (Cui et al., 2023). Annotation details are elaborated in Appendix A.2. Note that most responses from LLaMA2-chat-13b receive positive judgments, resulting in a large proportion of Align-P examples. We found downsampling Align-P examples is beneficial to the online alignment (see Appendix A.4). We evaluate models on ARC, HellaSwag, MMLU, TruthfulQA, and AlpacaEval.
2https://llama.meta.com/llama3
Results. Table 6 shows the results of online iterative alignment. In the first iteration, online alignment exhibits superior performance over offline alignment on both TruthfulQA and AlpacaEval. This observation implies that model-specific judgments are more effective for alignment. More importantly, the alignment continues to improve with more iterations, where the performance rises from 81.09 to 91.68 on AlpacaEval after four iterations. However, the performance improvement ceases at the fifth iteration. We speculate two possible explanations for this occurrence: (1) the judgments provided by GPT-4 contain certain inaccuracies, making them insufficient to effectively align a strong target model like our CUT 4+. (2) The target model may exhibit a knowledge deficiency in specific domains, such as mathematics and science, which cannot be adequately addressed through judgments. We also provide a case study in Appendix A.5.

heading: 5.2.1 Training A Judgment Model
text: In the previous experiments, we show that CUT is effective in aligning LLMs with judgments annotated by humans or GPT4. However, human annotations can be very expensive. The use of GPT4 assumes that a very strong LLM already exists. Next, we investigate the possibilities of developing an AI judge based on the target LLM.
Setup. we train AI judges with different amounts of judgment data {3000, 5000} collected in § 5.2. Then, we sample 1000 new instructions from Stanford Alpaca, obtain the corresponding responses from the target model (i.e., LLaMA2-chat-13b), and label judgments with our AI judges. These new judgment triplets are used to align the target
model.
Results. Figure 3 shows that AI judge-5000, trained with 5000 judgment data, is beneficial for aligning the target LLM, which leads to improvements of 3.02 and 4.17 points compared to LLaMA2-chat-13b on TruthfulQA and AlpacaEval respectively. In contrast, AI Judge-3000, using a smaller training dataset, shows limited effectiveness. The comparison suggests that training a capable AI judge necessitates a moderate number of high-quality training instances. As a result, it is feasible to train AI judges to align the LLM. However, the quality of the AI judge remains a crucial factor in determining the success of this endeavor.

heading: 5.3 Judgment vs. Reward
text: Our work primarily focuses on aligning LLMs with judgments, whereas most prior research explores rewards. In this section, we aim to provide a direct comparison between these two paradigms. However, note that it is hard to conduct a fair comparison due to the distinct data formats and the potential variation in data quality.
Setup. We compare judgment-based CUT with the state-of-the-art reward-based DPO (Rafailov et al., 2023). To maximize fairness, we leverage UltraFeedback (Cui et al., 2023), which contains both reward and judgment annotations produced by GPT4. Our preliminary experiments show that CUT is not good using the original judgments in UltraFeedback. We find that the reason is that the judgments in UltraFeedback tend to commend the strengths of given responses. This type of judgment is unsuitable for our CUT, as we primarily
use judgments for inappropriate token detection. Therefore, we re-collect judgments on the same instruction-response pairs from GPT4 using our prompt (Appendix A.2). Due to budget constraints, we randomly sample up to 3000 instructions (with 4 responses each, totaling 12,000 pairs) for annotation. The implementation details are as follows:
• DPO: For each of the above instructions, we formulate preference data by enumerating all possible pairs of responses from the given four, excluding pairs with the same reward value. • CUT-UF: We fine-tune the base model on the above instruction-response pairs and their original judgments from UltraFeedback using CUT. • CUT: We use the same instruction-response pairs as CUT-UF but with our re-annotated judgments.
Results. Figure 4 (a) shows the effect of three alignment methods using 1000 instructions as the alignment data. We can observe that CUT consistently improves over CUT-UF on all five tasks for two base models, which verifies our assumption that CUT is more effective when using critics as the judgment. Notably, CUT surpasses DPO by a large margin of 37.54 and 23.04 points on AlpacaEval for two base models, respectively. This shows that CUT is more effective in aligning LLMs with limited alignment data (i.e., 1000 instructions). Figure 4 (b) depicts the trends when adding more data for CUT and DPO alignment. The performance of CUT on these tasks is generally better or comparable to that of DPO and demonstrates a positive correlation with the size of the training data provided. The above observations suggest that judgments hold greater potential than rewards in aligning LLMs. CUT is slightly worse than DPO on ARC, and HellaSwag. We hypothesize that the performance discrepancy is partly caused by the
evaluation protocols: the four tasks are rankingbased. As suggested Bansal et al. (2023), methods such as DPO, which leverage ranking data in the alignment possess inherent advantages in rankingbased tasks. We also provide a case study in Appendix A.6.

heading: 6 Conclusion
text: We systematically explored the alignment of LLMs through the lens of judgments. We investigated three potential methods that can be adapted for aligning LLMs with judgments but found them unable to fully capitalize on judgments. We proposed a novel framework CUT, that enables direct and explicit learning from judgments and facilitates fine-grained inappropriate content detection and correction. Extensive evaluations demonstrated the effectiveness of our CUT in various settings, including offline and online, specialist and generalist, as well as cold-start and warm-start scenarios. For example, the online alignment experiments showed that CUT can iteratively improve LLMs with upto-date model-specific judgments, akin to how humans progressively refine their behaviors through ongoing feedback. Our analysis comparing rewards and judgments suggested that aligning LLMs with judgments is a promising research area.

heading: Limitations
text: Quality of Judgment Models Despite the positive alignment results of our AI judge mentioned in Figure 3, we find the quality of its generated judgments is not satisfactory and significantly inferior to those generated by GPT4. Therefore, we discuss from the point of judgment generation and identify two limitations when interacting with AI judges:
• AI judges often make inaccurate judgments, lead-
ing to potential misclassification of inappropriate tokens as appropriate and vice versa. This may increase the risk of hallucination. To address this issue, periodically involving human annotators to provide accurate judgments can be a good attempt to reduce the hallucinations accumulated during interactions with AI judges. • In an attempt to augment the training size, we incorporated the 1317 judgment data from Shepherd for training the AI judge. However, after including Shepherd, the AI judge’s performance deteriorated, resulting in more illogical judgments such as "The original answer 100 is incorrect. The correct answer should be 100." We hypothesize that reasoning and math tasks from Shepherd are too complex for a 13b model to comprehend. Consequently, larger language models may be required to achieve better judgment generation quality, a notion supported by (Saunders et al., 2022).
Size of Alignment Data Due to budgetary constraints, our research currently involves experiments utilizing several thousands of judgment data. In future research endeavors, we would like to investigate the scaling law with an expanded volume of judgment data.

heading: A Appendix
text: A.1 Implementations
We train our models using LoRA (Hu et al., 2022) and follow the best configurations suggested by Platypus (Lee et al., 2023). The tradeoff hyperparameter λ is selected from {1.1, 1.2} and the unlikelihood weight α and γ is selected from {0.25, 0.5, 1} and {0.25, 0.5, 1, 2}, respectively. We adopt the Alpaca template (Taori et al., 2023) for fine-tuning and inference. Figure 5 shows the templates when we apply CUT to align LLMs. Figure 6 shows the inference template, which does not necessitate judgments.

heading: A.2 Prompt for Judgment Annotation
text: Figure 8 illustrates the prompt employed to request GPT-4’s assistance in annotating judgments. We consider the judgment that begins with the keyword "Perfect." to be a positive judgment; otherwise, it is deemed a negative judgment. GPT-4 demonstrates proficiency in fulfilling this requirement. Figure 9 shows the template used for training AI judges.

heading: A.3 Offline Alignment Tasks
text: We conduct experiments on two tasks, a general instruction-following task, and a specific NLP task (summarization):
• General Instruction-following: We train models on the Shepherd dataset (Wang et al., 2023a), which consists of judgment data on diverse NLP tasks such as math word problems and commonsense reasoning. There are 1317 examples in total. For evaluation, we report model performance on four ranking-based and one generation-based LLM benchmarks, where ranking-based evaluation tests an LLM’s ability to select the best response from a set of candidate responses, while generation-based evaluation assesses an LLM’s ability to generate high-quality responses. Following the Open LLM Leaderboard (Gao et al., 2021), the ranking-based benchmarks are 25-shot ARC (Clark et al., 2018), 10-shot HellaSwag (Zellers et al., 2019), 5-shot MMLU (Hendrycks et al., 2021), and 0-shot TruthfulQA (Lin et al., 2022) from the Open LLM Leaderboard (Gao et al., 2021). The generation-based benchmark is AlpacaEval3.
3Following conventions, GPT4 is utilized to judge the winning rate of the responses generated by our models against those produced by DaVinci003.
• Summarization: We use the summarization dataset with judgment annotations produced by (Saunders et al., 2022). We use the training split (10827 examples) to train our models and report ROUGE scores (Lin, 2004) on the test split (1939 examples).

heading: A.4 Downsampling Align-P
text: Mixing training data from different categories can substantially affect the performance of trained models (Xu et al., 2023a,b,c). As LLaMA2-chat has already undergone extensive alignment training, its responses to the Stanford Alpaca instructions are generally of high quality. In fact, 713 out of 1000 responses generated by LLaMA2-chat receive positive judgments, resulting in a substantial proportion of Align-P examples. To investigate the effect of the proportion of Align-P examples, we undertake a downsampling process for these examples. The performance of various downsampling ratios is illustrated in Figure 7. Our findings indicate that maintaining a moderate percentage of Align-P examples is crucial. We conjecture that preserving a certain number of Align-P examples allows the model to sustain its capacity to generate satisfactory responses, while too many Align-P examples may lead to overfitting, thereby disrupting the alignment process. In subsequent experiments, we keep a ratio of 0.25.

heading: A.5 Case Study: Online Alignment
text: Table 7 presents three examples of modelgenerated responses after each training iteration. In general, the responses produced by different models do not display significant variations, as most content is satisfactory even before training and kept unchanged in subsequent iterations. Meanwhile, the generation quality exhibits a gradual improvement, characterized by the correction of specific errors and the inclusion of valuable improvements.
• Case 1: CUT 3+ introduces a crucial constraint that influences the color of the sky. • Case 2: CUT 1+ amends a hallucination present in LLaMA2-chat’s response (the fabricated file name “First document.tex”), though it introduces an additional mistake elsewhere. Fortunately, CUT 4+ is capable of rectifying the newly introduced error and providing a concise and satisfactory response. • Case 3: CUT 1+/2+/3+ adds a sentence that closely resembles the style of a Twitter post.
Moreover, CUT 4+ incorporates hashtags, further enhancing the resemblance to the typical format of a Twitter post.

heading: A.6 Case Study: CUT v.s. DPO
text: For a qualitative comparison of DPO and CUT, we perform a close examination of the generated responses from two methods. We find that DPO’s responses are more polite. However, CUT’s responses often exhibit greater specificity (Case 1), offer more helpful information (Case 2), and adhere more closely to the given instruction (Case 3), compared to those produced by DPO.

heading: Future Work
text: In future research endeavors, we would like to investigate the scaling law with an expanded volume of judgment data