Hummer: Towards Limited Competitive Preference Dataset

Authors: yusen  wu
li  jiang
junwu  xiong
jingqing  ruan
yichuan  ding
qingpei  guo
zujie  wen
jun  zhou
xiaotie  deng
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 05:32:20

Abstract: Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present \texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. \texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, \texttt{HummerRM} and \texttt{HummerRM-F}, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions \texttt{HummerRM} as an ideal model for domain-specific further fine-tuning and reducing vulnerability to jailbreak attacks.

Main paper: 
 introduction: 
 Reinforcement Learning from Human Feedback (RLHF) exhibits great potential in integrating human preferences into large language models (LLMs) (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022; Touvron et al., 2023; Achiam et al., 2023) . RLHF also holds great promise in real-world domains, such as robotics (Hu et al., 2023; Tian et al., 2023) , healthcare (Yu et al., 2023; He et al., 2023) , and autonomous driving (Chen et al., 2023; Du et al., 2023) . A fundamental aspect of integrating human alignment objectives lies in the preference modeling stage, which crucially depends on a given preference dataset. This stage can be realized by constructing either explicit (Christiano et al., 2017; Ouyang et al., 2022; Touvron et al., 2023) or implicit reward models (Rafailov et al., 2023; Zhao et al., 2023a) .However, alignment objectives often present competing properties in current preference datasets (Biyik & Sadigh, 2018; Hong et al., 2022; Ganguli et al., 2022; Wu et al., 2024) . Considering the Anthropic-HH dataset (Bai et al., 2022) , emphasizing the alignment objective of harmlessness may cause an agent to offer only broad or overly cautious advice. This emphasis could prevent the agent from delivering impactful and precise guidance, which limits the capability of helpfulness. This competition dynamics among alignment objectives poses two significant challenges. On one side, it exacerbates the vulnerability of safety-trained LLMs to jailbreak attacks by crafting prompts to prioritize one alignment dimension over others (Wei et al., 2024) . Besides, the conflict dynamics further complicate the attainment of equilibrium among all alignment objectives, particularly customizing models for downstream tasks that require promotion to specific dimensions ability without sacrificing performance in other alignment objectives, such as math reasoning (Azerbayev et al., 2023) and code generation (Guo et al., 2024a) .To mitigate the conflict of alignment dimensions in the preferences dataset, one line of research direction is to separately construct twisted alignment objectives via distinct reward or cost models to explicitly decouple human alignment objectives. Subsequently, these decoupled, learned models are synergized to provide a holistic preference signal tailored to specific alignment goals for downstream tasks. The integration methodology encompasses a variety of strategies, consisting of voting mechanisms (Wang et al., 2024a) , differential weighting of models (Touvron et al., 2023; Wu et al., 2024) , application of linear combinations (Jang et al., 2023; Rame et al., 2024) , and the use of the Lagrangian multiplier (Dai et al., 2023) to manage trade-offs or to highlight specific principles of human values. However, those approaches inadvertently increase model complexity and computational overhead.In this study, we redirect our focus toward the underlying cause of alignment conflict: the preference dataset itself. RLHF community has witnessed an emerging trend towards developing new preference datasets, driven by goals of enhancing quality and scale, incorporating fine-grained preference signal, and covering specific domains aligned with desired dimensions (Cui et al., 2023; Ji et al., 2024b; Wu et al., 2024; Stiennon et al., 2020; Lightman et al., 2023; Ethayarajh et al., 2022) . Despite these efforts, a significant gap persists: the lack of a preference dataset intentionally crafted to alleviate conflicts between alignment dimensions. Such a dataset could potentially provide significant benefits for downstream applications that prioritize certain values (Zhang et al., 2024; Wang et al., 2024b) and reduce vulnerabilities to jailbreak attacks (Perez et al., 2022; Qi et al., 2023; Wei et al., 2024; He et al., 2024; Lyu et al., 2024; Cui et al., 2024) . Moreover, there is currently no established statistical metric for assessing the degree of conflict among alignment dimensions within preference datasets.In light of these observations, we first introduce Alignment Dimension Conflict (ADC), a statistical metric for quantifying the degree of conflict within preference datasets. This new criterion moves beyond the conventional metric of average performance across multiple objectives or domains typically featured on current leaderboards. We then present Hummer, standing as the first preference dataset to highlight limited competition among various alignment objectives. The construction of Hummer capitalizes on the advanced capabilities of AI feedback mechanisms, such as GPT-4 (Achiam et al., 2023) , consisting of a three-stage process: preference & objective annotation, alignment objectives refination, and dataset split. We use the UltraFeedback (Cui et al., 2023) as our foundation dataset for this work and introduce a fine-grained version of Hummer, termed Hummer-F, which excludes the noisy preference dataset via the principle of reward gaps and compromises 80% of Hummer.Based on Hummer and Hummer-F, we introduce a hybrid sampling strategy for training their respective reward models, HummerRM and HummerRM-F, based on the established Llama 2-7B model (Touvron et al., 2023) . The hybrid sampling strategy achieves wellbalanced performance across diverse limited-competition alignment objectives in Hummer, enhances resilience to jailbreak attacks, and supports further fine-tuning in downstream tasks. It accomplishes this by prioritizing certain alignment objectives without sacrificing performance in other dimensions. We summarize our contributions in two main folds:1. We introduce the Alignment Dimension Conflict (ADC), a statistical metric for quantifying conflict in preference datasets. We then present Hummer and its refined variant, Hummer-F, designed as the first preference datasets to mitigate competing alignment objectives.2. We develop a hybrid sampling strategy to train the reward model HummerRM from Hummer, balancing performance across alignment objectives and further limiting the conflict. HummerRM boosts defense against jailbreak attacks and enables downstream fine-tuning by focusing on key alignment dimensions without compromising others. 
related work: 
 RLHF. RLHF has emerged as the leading strategy to integrate human preferences into language models through preference datasets, which can be fixed pre-collected or generated from agents or language models (Cheng et al., 2011; Akrour et al., 2011; Askell et al., 2021; Xu et al., 2022; 2023) . To integrate human values, RLHF generally obtains the final aligned policy through RL algorithms, such as PPO (Schulman et al., 2017) , to maximize the reward through the trained reward model on preference datasets (Ramamurthy et al., 2022; Bai et al., 2022; Ouyang et al., 2022; Touvron et al., 2023) . Another important branch is to directly anchor the human preferences to the final policy by constructing the implicit reward with policies through the closed-form optimal solution for the reward model (Rafailov et al., 2023; Zhao et al., 2023b; Azar et al., 2023; Wang et al., 2023; Ethayarajh et al., 2024; Zhou et al., 2023; Amini et al., 2024; Liu et al., 2023; Swamy et al., 2024) . While these approaches are appealing for their computation cost and ease of implementation, their inherited offline paradigm suffers from the distributional shift and lack of online exploration (Guo et al., 2024b; Calandriello et al., 2024) .Preference Datasets. The RLHF community is observing a growing trend of new preference datasets from diverse perspectives to improve preference modeling. The dominant motivations for the introduction of new preference datasets are scalability, quality, and diversity (Guo et al., 2023; Cui et al., 2023; Wu et al., 2024) . For example, SPA dataset (Guo et al., 2023) presents fine-grained (i.e., token or phrase level) feedback during optimization rather than holistic feedback during the training process. UltraFeedback (Cui et al., 2023) introduces a wide-source and high-quality preference dataset with four alignment dimensions, in contrast to two dimensions (helpfulness and harmlessness) (Ouyang et al., 2022) . Besides, some recent preference datasets underscore a specific domain or alignment property (Stiennon et al., 2020; Lightman et al., 2023; Ethayarajh et al., 2022) . However, existing preference datasets fail to mitigate the conflict between alignment dimensions. Enhancing the synergy of alignment dimensions improves resilience against jailbreak attacks and allows for further fine-tuning in downstream applications. This is achieved by prioritizing specific alignment objectives without compromising performance across other dimensions.Red Teaming LLMs with Further Fine-tuning. Red teaming is designed to execute systematic tests and attacks on LLMs to expose their potential harmfulness and safety vulnerabilities (Perez et al., 2022; Achiam et al., 2023; Shi et al., 2024; Lyu et al., 2023; Kang et al., 2024) .Recent work (Qi et al., 2023; Zhan et al., 2023; He et al., 2024) identifies that customizing policies with further fine-tuning on downstream tasks, even without harmful content, will lead to a degradation in resilience against jailbreak attacks for safety-alignment policy. We hypothesize that this degradation stems from an implicit emphasis on specific alignment dimensions, such as helpfulness, and the conflict among alignment dimensions present in downstream datasets, where the learned policy is either an implicit (DPO pipelines) or explicit distillation (PPO pipelines) of the reward models. In this work, we focus on the conflict of alignment dimensions and study further fine-tuning specific alignment dimensions on the preference modeling stage (reward model) to improve specific ability for customization tasks. Aligned with these findings, we show that further fine-tuning downstream models on desired alignment dimensions inevitably leads to performance degradation in conflicting dimensions, e.g., safety. 
preliminaries: 
 RLHF typically starts with a generic pre-trained language model from supervised finetuning on a high-quality dataset for general proposes, such as conservation, noted as π SFT , and then matches human preferences through a preference dataset. In this work, we mainly study the problem of competing alignment objectives in existing preference datasets.Preference Modelling. One of the core ingredients of RLHF is to integrate human preferences into LLMs through preference datasets, formulated as D P = {x k , y k w , y k l } K k=1 , where K is the number of total collected samples. The preference dataset D P incorporates the human feedback through the preference to these two responses, i.e., y w ≻ y l | x. Given the prompt x, y w denotes the preferred response by humans or advanced AI models compared to y l . Given the preference dataset D P , we can then parameterize a reward model r ϕ (x, y) and optimize it through through the Bradley-Terry (BT) model (Bradley & Terry, 1952 ): max r ϕ E (x,y w ,y l )∼D P log σ r ϕ (x, y w )r ϕ (x, y l ) ,(1)where σ is the logistic function. In the context of LLMs, the reward model r ϕ (x, y), is frequently constructed based on π SFT (y | x) by adding a linear layer on top of the final transformer layer, which yields a single scalar prediction representing the reward value (Ziegler et al., 2019; Ouyang et al., 2022) .Policy Modelling. Given the learned reward model r ϕ (x, y) constructed through preference modeling and policy training dataset D RL = {x k } K k=1 , we can formulate the following optimization objective for π θ to inherit the preference from r ϕ (x, y):max π θ E x∼D RL ,y∼π θ (y|x) r ϕ (x, y) -βD π θ (y | x)∥π SFT (y | x) ,where β moderates the divergence D, such as KL divergence (Kullback, 1951) , between π θ and a soft-target policy π SFT . This regularization ensures that π θ avoids collapsing into a narrow set of high-reward actions, preserving a diverse and functional output distribution as supported by the learned reward model (Jaques et al., 2019; Song et al., 2023; Laidlaw et al., 2024; Wei et al., 2024) . PPO is employed for policy optimization due to its proven efficiency and stability in training. This section begins with a formal definition of Alignment Dimension Conflict (ADC), a metric to assess the extent of conflicting alignment objectives in preference datasets (Section 4). We then introduce Hummer and its fine-grained variant, Hummer-F, which are datasets specifically crafted to emphasize the property of limited conflict among alignment objectives of preference datasets (Section 4.2). 
alignment dimension conflict: 
 D P can be further organized asD P n = {d 1 , d 2 , • • • , d n } with d i = {x k , y k w , y k l } K i k=1, where d i denotes the alignment dimensions, such as helpfulness in Anthropic HH dataset (Bai et al., 2022) , n represents the total alignment dimensions, and K i notes the total samples in dimension d i with ∑ n i=1 K i = K. Formally, given a reward model, i.e., RM, that has been initially fine-tuned on the whole preference dataset DP n = {d 1 , d 2 , • • • , d n }, its performance (i.e., accuracy of RM(x, y w ) > RM(x, y l )) on the corresponding test dataset from D P n is represented by U = {u 1 , u 2 , • • • , u n }.To study this conflict, we copy n reward models and further fine-tune each reward model on the dataset of interest for any alignment dimension, e.g., d i ∈ D P n , obtaining the fine-tuned performanceU i = {u i,1 , u i,2 , • • • , u i,n }.The performance deviation can be obtained by (U i -U) of RM i , where i highlights further fine-tuning conducted only on d i . We present the pipeline for measuring this dimension conflict, considering only negative performance deviations, in Fig. 1 and introduce a new statistical metric: Definition 1 (Alignment Dimension Conflict). The Alignment Dimension Conflict (ADC) is defined as the second-order moment of the negative performance deviation summation on all dimensions except d i : where n -1 serves as a normalization term to facilitate fair comparison for different datasets with different alignment dimensions and E i [•] denotes the expectation over the performance deviations obtained by further fine-tuning on alignment dimension d i∈n withEQUATIONE i [•] = ∑ n i=1 [•]/n.An interesting question to ask is: What situation leads to high ADC? We simplify the performance deviation (U -U) sampling from a normal distribution N 0, σ 21 . The expression 2a represents the ADC of a normal distribution with respect to its variance parameter σ. This measures how much adjusting one alignment dimension affects others with further fine-tuning. We observe a strongly positive correlation between ADC and σ, indicating that datasets with a higher level of competing dimensions (evidenced by greater variance on the negative side) tend to exhibit higher ADC values. The performance deviation across datasets with varying ADC levels is illustrated in Fig. 2 , where datasets with low ADC are characterized by a minimal negative impact on the performance across other alignment dimensions, i.e., lower competition.E x∼N (µ=0,σ 2 ) U[x] in Fig.RewardBench (Lambert et al., 2024) offers toolkits for structured comparison across various properties in reward models, accommodating diverse model structures. To facilitate a systematic comparison of alignment dimension conflict levels among different reward models trained from different preference datasets, we can scale the Alignment Dimension Conflict (ADC) metric to the evaluated properties on standard evaluation toolkits, termed ADC-B, as shown in Definition 2.To decouple alignment dimensions, we introduce Hummer, the first preference dataset that aims to alleviate the competing dynamics of preference datasets. To accurately capture the multidimensionality of human preference without interference between alignment dimensions, we leverage the powerful ability of AI feedback, i.e., GPT-4, which has been heavily employed in preference dataset construction or preference modeling (Lee et al., 2023; Cui et al., 2023; Guo et al., 2023; Burns et al., 2023; Chen et al., 2024; Ji et al., 2024a) . We leverage UltraFeedback (Cui et al., 2023) as the foundational dataset, attributed to its expansive scale and diversity.We show the construction process of Hummer in Fig. 3 , detailed in Appendix B. The process of identifying the limited-conflict dimension and its corresponding pairwise dataset involves three key stages: : (a) powered by GPT-4. We totally collect 37 alignment dimensions. (b) Alignment objective refination: We then leverage GPT-4 to refine these dimensions to minimize their conflicts and finally get n = 6 alignment dimensions: {'accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity'}. (c) Dataset split: GPT-4 is then used to assign an absolute reward r to n alignment dimensions for every sample in the foundation dataset. We categorize every dataset sample (x, y 1 , y 2 ) to its corresponding dimension on the principle of maximal preference gap, i.e., max n i=1 |r (x i , y 1 ) ir (x i , y 2 ) i |. We highlight that this splitting approach is more favorable than directly ranking as it avoids the position bias (Zhu et al., 2023) and facilitates convenience to build Hummer-F. Hummer-F is refined by applying a reward gap threshold (τ = 0.5) to filter out potentially noisy preference pairs, a subset that comprises approximately 80% of Hummer. 
hybrid reward sampling: 
 In this section, we introduce HummerRM and its variant, HummerRM-F. Both are singlereward models trained on our custom-limited competitive preference datasets, Hummer and Hummer-F, respectively. These models employ a hybrid sampling method to dynamically balance alignment dimensions, further mitigating alignment dimension conflicts in our proposed datasets, especially with imbalanced datasets over alignment dimensions, as shown in Appendix A.Formally, considering a preference dataset with n alignment objectives, denoted as D P n = {d 1 , d 2 , . . . , d n }, we assign an initial equal sampling weight to each dimension dataset, represented by Λ = {λ 1 , λ 2 , . . . , λ n }, where λ i = 1/n with i ∈ [1, n]. We achieve the balance among all alignment dimensions by evaluating the preference performance across these dimensions, denoted as U = {u 1 , . . . , u n }. The sampling weights are adaptively updated in every 1 epoch (128 steps) as follows:EQUATIONwhere ū represents the average preference performance across all alignment objectives, and η is the temperature for updating the sampling weights Λ. To ensure adherence to the sum constraint, ∑ n j=1 λ j = 1, we normalize the λ i values accordingly after every update. Consequently, the mini dataset sampled at each training step is represented by ⌊BatchSize × Λ⌋ from D p n , where BatchSize = 128 and ⌊x⌋ represents the floor function. Intuitively, if the performance of a specific dimension, e.g., u i , is higher than the average (u i > ū), the corresponding sampling ratio λ i for dataset d i decreases. Conversely, if u i < ū, indicating a performance lower than the average, λ i increases, promoting an increasing sampling dataset for d i . We then integrate all sampled datasets into one training batch and update the reward model via Eqn. (1). The hybrid sampling strategy enhances the robust performance of HummerRM across all alignment dimensions.  
experiments: 
 Our testbed is designed to assess the low-conflict alignment dimensions within our introduced datasets, namely Hummer and Hummer-F. We initiate our evaluation by examining the ADC and ADC-B using HummerRM, alongside a standard reward benchmark, as detailed in Section 6.1. Subsequently, we explore the vulnerabilities of HummerRM, shown in Section 6.2. Finally, we assess the efficacy of the hybrid sampling strategy in comparison to diverse sampling methods in Appendix A.Setup. To elucidate the dynamics of low competition in Hummer and Hummer-F, we assess the ADC within their respective preference datasets; and ADC-B equipped with systematical comparison on RewardBench (Lambert et al., 2024) . This evaluation is contextualized by comparisons with the Anthropic HH dataset (Bai et al., 2022) , and UltraFeedback (Cui et al., 2023) . Furthermore, we explore the effectiveness of hybrid sampling strategies in the training of reward models. For consistency across evaluations, we employ a consistent backbone model, specifically a fine-tuned Llama 2-7B (Touvron et al., 2023) , to train the reward models for each dataset. Result. In Tab. 1, we summarize prevalent preference datasets with our statistical evaluation findings. Notably, Hummer and Hummer-F demonstrate a significantly reduced ADC (8-10x) compared to other preference datasets, even without hybrid sampling (HS). The structured comparison in ADC-B on RewardBench uncovers a notable consistency with the ADC results. Despite the preference dataset for Hummer and Hummer-F being considerably smaller (3-4x) than UltraFeedback, we observe an enhanced performance from HummerRM and HummerRM-F over UltraRM, by margins of 3.8% and 3.2% on RewardBench, respectively. This underscores the significance of dataset quality in preference datasets. Ablation. The ablation study on the HS strategy reveals that improvements in ADC and ADC-B are primarily derived from our proposed datasets, while an observable margin with HS, i.e., around 3% and 7% for ADC and ADC-B respectively. Our observations confirm that HS is crucial for enhancing leaderboard-centric performance primarily aiming at "achieving a higher score" on Rewardbench. Additionally, we emphasize the importance of data quality in further fostering improvements in ADC and RewardBench. Despite these observed gains, this study fundamentally aims to identify and quantify the competing dynamics prevalent in preference datasets.Setup. We posit that the HummerRM framework can mitigate vulnerabilities to jailbreak attacks by enhancing one dimension without degrading performance across other metrics.Our jailbreak evaluation framework follows Siththaranjan et al. (2023) . Specifically, the jailbreak-based dataset comprises pair-wise tuples (x, y 1 , y 2 ), where x represents prompts designed to elicit a harmful response from the model, y 1 denotes the safe response, and y 2 is jailbreak response (Wei et al., 2024) . We quantify the 'jailbreak rate' through the proportion of instances where the reward model favors (x, y 2 ) over (x, y 1 ), represented by (∑ n I(r(x, y 2 ) > r(x, y 1 )))/n, where I is the indicator function and n denotes the total prompts. The higher the jailbreak rate, the greater the vulnerability of models to attacks.Result. In Tab. 2, we delineate the outcomes of jailbreak attacks on Anthropic HH (Ouyang et al., 2022) , UltraFeedback (Cui et al., 2023) , and Hummer, with each model integrating 2, 4, and 6 alignment dimensions, respectively. Initial fine-tuning yields a uniform jailbreak rate across all datasets. Notably, UltraRM registers the highest attack rate, exhibiting a 10.4% increase following further fine-tuning on the instruction-following alignment dimension (# 2). This highlights a significant escalation in vulnerability to jailbreak attacks when UltraRM is specifically fine-tuned to enhance instruction-following, underscoring a pronounced tension with safety protocols. Conversely, HummerRM demonstrates exceptional robustness, with a jailbreak rate increment of less than 3% subsequent to additional fine-tuning across all dimensions. This indicates that the alignment objectives of Hummer are harmoniously integrated, ensuring that its safety remains unimpaired by further fine-tuning.We emphasize that a declining jailbreak rate signifies enhanced defensive capabilities against jailbreak attacks. This improvement is particularly notable when further fine-tuning focuses on specific alignment dimensions, such as harmlessness (# 2) in the case of Anthropic HH, and empathy (# 4) in Hummer. Appendix Tab. 7 shows the detailed alignment dimensions for preference datasets.Ablation. The ablation study on the HS indicates the strong ability of reward models against jailbreak attacks is most saturated from Hummer and Hummer-F, while hybrid sampling further enhances the defensive capabilities. These results align with those observed in the Tab. 1, affirming ADC's reliability as a proxy for quantifying preference conflicts in datasets. Addressing these conflicts is essential for maintaining resilience against jailbreaks. Setup. Qi et al. (2023) implies that custom fine-tuning on the policy models (even without harmlessness datasets) will significantly undermine the initial safety alignment. We can observe that the harmfulness rate increases near 12% with further fine-tuning of policies on Dolly (Conover et al., 2023) to simulate the downstream tasks, as shown in Tab. 3 (base). The harmfulness rate metric follows Qi et al. ( 2023) by using the GPT-4 judge which outputs a harmfulness score in the range of 1 to 5. The harmfulness rate indicates the fraction of outputs from the policy with the highest harmfulness score 5 on the test dataset.We hypothesize that further fine-tuning of reward models to highlight specific alignment abilities (e.g., helpfulness) serves as an amplifier to diminish the safety of initial alignment policies due to conflict of alignment dimensions.We demonstrate the results in Tab. 3 by further fine-tuning the reward models on specific alignments to prioritize certain values (such as safety in finance scenarios), which retains the same training setting in Section 6.2 but evaluates on policy levels. We can observe a subtle increment (+0.4%) on average in the harmfulness rate of policies whose reward models trained from Hummer, which implies that further fine-tuning specific alignment dimensions on Hummer does not exaggerate the loss of safety. On the contrary, the increment of the harmfulness rate of policies from Anthropic HH and UltraFeedback are 3-4x to Hummer, demonstrating an unneglectable amplification of the safety loss of policies. 
conclusion: 
 In this study, we delve into the dynamics of competing preferences within the Reinforcement Learning from Human Feedback (RLHF) framework. We introduce a novel statistical metric termed Alignment Dimension Conflict (ADC) to quantify the extent of conflict among alignment objectives within preference datasets. We unveil the first preference dataset, Hummer, alongside its fine-grained variant, Hummer-F. These datasets are designed to mitigate dimension conflicts, facilitating domain-specific fine-tuning while increasing resilience against jailbreak attacks. This is achieved by selectively prioritizing certain alignment objectives without compromising performance across other alignment objectives. Subsequently, we develop reward models for our datasets, namely HummerRM and HummerRM-F, employing a hybrid sampling technique that dynamically adjusts the sampling weight based on reward performance across different alignment dimensions. Looking ahead, an intriguing avenue for future research lies in constructing low-conflict alignment objectives using unsupervised or self-supervised (Zhang et al., 2020; Yan et al., 2021) learning methods to discern semantic nuances. Furthermore, exploring the conflict of alignment dimensions in the preference modeling stage offers a promising avenue for understanding the safety trade-offs in further fine-tuning policies (Qi et al., 2023) . Hybrid sampling strategy maintains performance on imbalanced datasets. An imbalanced dataset arises with a non-uniform distribution of classes, often characterized by a disproportionate number of instances between major and minor classes, resulting in biased predictions (Krawczyk, 2016; Jiang et al., 2023) . To investigate the efficacy of a hybrid sampling strategy in addressing dataset imbalance in the context of alignment objectives, we integrate our datasets across six alignment dimensions with a distribution ratio of 10 : 10 : 10 : 10 : 1 : 1, where the 1 : 1 ratio pertains specifically to specificity and tone. The results are illustrated in Fig. 5 . 
 
 Appendix: Fine-tuning on specific dimensions will boost the performance on its corresponding alignment dimensions but fail to achieve desirable performance on other alignment dimensions, such as Single # 1(Accuracy), and Single # 5 (Tone). We demonstrate that the All Dimensions Equal strategy, with a uniform distribution ratio of 1:1:1:1:1:1, under-performs relative to our hybrid sampling approach across all dimensions, achieving only 70% to 95% of the performance of the Hybrid sampler. This implies that this uniform sampling strategy, also employed by Cui et al. (2023) , may fall short in imbalanced datasets. The All Mixed strategy, integrating all alignment datasets ignoring the data balance, exhibits significantly superior performance in well-represented alignment datasets # 1 and # 2 (Depth and Accuracy), yet fails in alignment objectives with limited datasets: # 5 and # 6 (Tone and Specificity). Such an approach could further diminish the performance of lesser-represented alignment objectives, particularly in scenarios involving competing alignment objectives.In this section, we detail the construction process of Hummer, starting from the initial data formulation. Utilizing the original dataset, we format it in the pattern {x, y 1 , y 2 , y 3 , y 4 }, where x serves as the prompt and each y i represents a candidate generated by the model.To create a rich dataset for pairwise comparison, we pair the candidates, resulting in a new set of sample pairs {x, y 1 , y 2 }.Following this, we select a subset of 400 pairs from this collection through random sampling. These selected pairs are then formatted into standard prompts, structured to be fed into GPT-4 for evaluation. In executing these queries, our objective is to discern the superiority between y 1 and y 2 within each pair, focusing on identifying which candidate better aligns with a specific predefined objective. Additionally, for each comparison, we aim to gather a concise explanation highlighting why one candidate is favored over the other, based on the alignment with the mentioned objective. Through this meticulous process, we identified a diverse set of 37 different objective names.{'accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity'} (4) Subsequently, we integrate the previously identified 400 superior alignment objectives, replete with their concise explanations, into the new prompt design for GPT-4 as part of our second approach in prompt engineering. This step instructs GPT-4 to assimilate theFollowing is a pair-wise RM training data item with the structure 'prompt':[prompt], 'chosen' :[chosen output], 'rejected':[rejected output].The 'prompt' stands for a question one agent is asked to answer and the 'chosen' and 'rejected' are two responses from the above agent. Your task is to assess both of them and give reward (float, 5.0 for best and 0.0 for worst) in the dimension of Depth with the definition "the thoroughness of analysis or explanation, providing detailed insights into a subject", for 'chosen' and 'rejected' responses(Each response one score). Then compute the gap between the two rewards ('chosen' reward -'rejected' reward). Finally only output the reward gap.Here is the item case: is straightforward: a larger reward gap signifies a clear preference for one candidate over the other, primarily grounded in the specific objective, thereby determining the ultimate classification for data segregation. This iterative refinement led to the crystallization of 6 distinct alignment objectives, each defined succinctly and accompanied by the frequency of dataset samples correlating with them.An intriguing observation emerged during this process: a notable fraction of samples (11.2%, to be precise) displayed nearly identical or very closely matched reward gaps for two or more objectives. Our strategy to address these ambiguities varies depending on the dataset context. For the standard dataset, these samples are randomly allocated to one of the objectives sharing the highest reward gap, aiming to preserve the integrity and balance of the dataset. Conversely, in the fine-grained dataset, we opt for exclusion, removing these samples outright to maintain the precision and reliability of our objective classifications.1. Accuracy refers to the adherence to factual correctness, ensuring that information is free from errors.2. Conciseness refers to the ability to convey information with brevity, using a minimal number of words without sacrificing clarity.3. Depth refers to the thoroughness of analysis or explanation, providing detailed insights into a subject.4. Empathy refers to the capacity to understand and share the feelings of others, reflecting compassion in communication.5. Tone refers to the author's attitude or mood conveyed through language, influencing the reader's perception.6. Specificity refers to the provision of precise and detailed information, avoiding generalizations or vagueness.Following Tab. 5 and Tab. 6 we provide some representative cases from Hummer dataset, each for one alignment objective, ranging from three lines to tens of lines. The format follows "prompt", "chosen" and "rejected" (pair-wise) versions.While the prevailing belief advocates for the benefits of increasing dataset size to enhance the performance of language models, we emphasize the critical role of data quality over quantity in developing preference datasets for effective preference modeling. Enlarging the dataset may inadvertently incorporate noisy preference pairs, potentially diluting the integration of human values into the reward model (Siththaranjan et al., 2023; Wang et al., 2024a) . In response, Wang et al. (2024a) undertook comprehensive experiments that underscored this phenomenon, proposing label smoothing and additive margin as algorithmic innovations to refine the preference model.In light of these findings, our approach in developing Hummer involves a meticulous twostage filtering process, resulting in the creation of Hummer-F, a fine-grained variant distilled "prompt": "I'm trying to improve my fitness and lose weight.What kind of exercise routine should I follow?", "chosen": "For a targeted approach to fitness and weight loss, commit to a routine that includes jogging or brisk walking for 20 minutes every morning, three times a week, and swimming or cycling for 30 minutes on the alternate days. On weekends, dedicate 30 minutes to strength training, focusing on full-body workouts with exercises like squats, lunges, push-ups, and planks. ", "rejected":"To enhance your fitness and promote weight loss, consider integrating a mix of cardio and strength training into your weekly schedule. Aim to stay active for at least 30 minutes a day, five days a week, and include healthy eating habits as part of your lifestyle." Here are some practical tips: prioritize tasks, set boundaries, and consider delegating or asking for help when necessary. Remember, it's important to take breaks and engage in activities that you enjoy to recharge." from approximately 46% of the original dataset. The initial stage utilizes scores from raw paired data sourced from UltraFeedback (Cui et al., 2023) , implementing a threshold (τ 1 = 4.0) on the summed score gap for initial data cleansing. This procedure effectively reduces the dataset from N 0 = 100k preference pairs to N 1 = 46k. Subsequently, we introduce a second threshold (τ 2 = 0.5) specifically within the pairwise preference datasets of Hummer, aiming to isolate and remove potentially noisy data based on reward signals derived from the concluding phase of Hummer's assembly. This strategy further refines the dataset to N 2 = 37k preference pairs. Our experimental results affirm that this meticulous dataset curation markedly enhances testing accuracy. Although the current filtering process relies on heuristic methods, future iterations could benefit from an implementation grounded in a reward modeling approach. This section delineates the experimental apparatus employed in our study. Our computational setup comprised a quad-cluster of NVIDIA A100 GPUs, each furnished with 100GB of memory, providing robust computational capacity. This infrastructure was driven by a software stack anchored by Python 3.8. In the realm of deep learning libraries, we harnessed the capabilities of PyTorch version 2.0.1. Allied with PyTorch, we utilized torchvision version 0.13.1+cu113 and torchaudio version 0.12.1+cu113 to manage image and audio data transformations, respectively. Additionally, scikit-learn version 1.0.1 served as our machine learning toolkit, offering a versatile assortment of algorithms for data mining and analysis.To expedite the training process, we integrated the flashattention library at version 1.0.0, specifically optimized to harness the A100's computing prowess effectively. This library was instrumental in reducing the computing overhead significantly, thus accelerating training times for our models.Below, we expand on the specifics of our experimental methodologies, ensuring that we shed light on each significant aspect that could possibly influence the replicability and interpretation of our research findings.We initiate our experimentation by training an encompassing model on the Hummer dataset utilizing the LLaMA2-7B architecture, extending over m 0 = 24000 training steps. To assess its performance, we deploy the model to RewardBench, yielding an evaluative score.To gauge and juxtapose the Average Deviation Coefficient (ADC) across varying datasets, we embark on a fine-tuning regimen. This phase commences with models that have undergone a warm-up phase of training, aligned with different specified objectives. These fine-tuned models, including the initially warmed-up model, undergo individual assessments against the corresponding evaluation sets of each dataset. The objective is to discern the adjustment in prediction accuracy specifically on the RM dataset. We normalize the observed changes to derive the relative variation and, leveraging the ADC as previously defined, calculate the precise value through the established formula.To mitigate the potential biases introduced by the model architecture in evaluating datasets, we standardize the use of the Llama2-7B model as our foundational model for all datasets undergoing evaluation. This standardized approach includes an initial phase of training amounting to k 0 = 1000 steps, covering the entirety of the source dataset-a conglomerate reflecting the diverse spectrum of the target evaluation dataset. This foundational model subsequently anchors the further fine-tuning training sessions and comparative performance analyses.In the advanced fine-tuning phase, we meticulously sample from each subset within the evaluation dataset, catering to distinct alignment objectives. This step involves engaging in reward model training over M = 4000 steps, rooted in the preliminarily trained base model.For those datasets facing a data scarcity, we incorporate a multi-epoch replay and reuse strategy. This method is pivotal in circumventing the undue repetition of data samples, thereby minimizing the risk of overfitting and maintaining the model's generalization capabilities.For the computational demands of our experiment, each further fine-tuning phase of our model on the Hummer dataset, leveraging the LLaMA2-7B framework, required approximately 6 hours of dedicated processing time using four NVIDIA A100 GPUs. Our ADC evaluation involves the following three key steps:(a) Single evaluation strategy: During the evaluation stage, we strategically sample 1,000 instances from each test set corresponding to the distinct alignment objectives integrated within our target evaluation dataset. This sampling aims to rigorously assess the prediction accuracy of our fine-tuned models. Adopting a standard reward model evaluation approach, we analyze the competency of the reward model by presenting two candidate responses to a given prompt. The evaluation criteria are straightforward: if the candidate response marked as "chosen" garners a higher score compared to its counterpart across the sampled data, the model's prediction for that instance is deemed accurate; conversely, it's labeled inaccurate. The precision of the model, thus, is quantified as the percentage of instances correctly evaluated as accurate.(b) Evaluate further tuning: Upon refining the new model via further fine-tuning on an alignment objective from the base model, we meticulously evaluate the impact of this finetuning on relative accuracy across all objectives delineated in the dataset. In analyzing the outcomes, our focus narrows to the adverse effects-specifically, the reduction or negative impact that further tuning dedicated to one objective might have on the performance across other objectives. This analysis is operationalized by computing the squared mean of these reductions.(c) Compute ADC value: Concluding this multi-faceted evaluation, we compute the expectation across each specified objective within the target evaluation dataset. This computational step culminates in the derivation of the Average Deviation Coefficient (ADC) result, effectively encapsulating the nuanced dynamics our definition intended to capture. This ADC measurement serves as a nuanced indicator, reflecting the model's balanced performance across a spectrum of alignment objectives, shedding light on the intricate trade-offs that underlie fine-tuning processes in deep learning model optimization.Formally, the performance of a given reward model after fine-tuning on its preference dataset D p n is denoted as V = {v 1 , v 2 , • • • , v m }, where m indicates the total dimensions of abilities for assessment, e.g., Reasoning ability in RewardBench. With further fine-fune of the reward model on one specific dimension d i ∈ D p n , new evaluated performance and benchmark performance deviation are defined as V i = {v i,1 , v i,2 , • • • , v i,m } and V i -V, respectively. We then can evaluate the ADC of datasets with a structured comparison on standard benchmarks: Definition 2 (Alignment Dimension Conflict Benchmark). The Alignment Dimension Conflict (ADC) extended to standard benchmark evaluation is the second-order moment of negative performance deviation on all evaluation dimensions in the benchmark:EQUATIONTo rigorously evaluate our novel hybrid sampler methodology against the conventional fixed-ratio mixture sampling technique, we undertake comparative training experiments using the same dataset. In this case, we exemplify the process with the fine-grained version of the Hummer dataset. We standardize the foundation of our comparative analysis by utilizing the Llama2-7B base model, maintaining a consistent training duration of N = 2000 steps across all experimental trials. Post-training, we assess the resulting reward model's performance on various objectives' evaluation sets within the Hummer dataset. The findings related to relative accuracy are illustrated in Fig. 5 .The assumption that µ = 0 is justified because further fine-tuning along dimension d i might enhance performance in some dimensions while adversely competing with others. 


Decision: As all reviewers have agreed, the paper is tackling a very important problem in language modeling: conflict among different values during alignment phase. The proposed method in this paper to alleviate this problem is sound and novel, and also shows experimentally effectiveness. While the paper lacks clarify and needs improvement on its writing and some discussions on the values themselves, given the characteristics of CoLM and its audience, I recommend the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The paper makes multiple contributions. First a metric to measure conflicts alignment dimensions in preference datasets is presented (ADC/ADC-B). Second, seeded from UltraFeedback, a preference dataset with low conflicts along 6 alignment dimensions is collected and automatically annotated with GPT-4. The usefulness of the collected dataset is demonstrated in experiments. Third, they introduce a hybrid sampling strategy for balancing alignment dimensions in preference datasets when learning a Reward Model.
strength_weakness: strength_weakness: The paper is often quite dense, and details are only explained in the appendix. While I acknowledge that there are many contributions, still I think it would improve the paper to at least know major experiment choices/settings, e.g. which model has been used for tuning or which alignment dimensions are comprised in Hummer.
questions: questions: - Can you clarify what the “u_ = min{u, 0}” operation does? Does it mean: if \bar{u}_k - u_k > 0 -> 0 else \bar{u}_k - u_k? And I am assuming this operation is applied before the square?
- Just to clarify, ADC-B can only be applied to two (or more) datasets with data on the same alignment dimension, right?
- From Figure 4 it seems that further fine-tuning on ‘tone’ increases conflicts along the dimensions. Why do you think that is? Intutively, tone should not alter the semantics, so at least accuracy, consicneness and depth should stay somewhat equal?

Typos/Minor Comments:

- The table captions should be after the table.
- Page 3: Double semi-colon “Wang et al., 2023;; Ethayarajh et al., 2024”
- “table” Seems like table references are all lower case, but Figure references are upper case. Consider aligning.
- Section 4.2: I think this section is very hard to understand without Appendix B. Maybe it would be better to not only describe things abstractly, but more concretly e.g. by naming the alignment dimension.
- Table 2: HmmerRM → HummerRM
- Page 16: I guess there should be commas between the y’s ? “{x, y1, y2.y3.y4}” → “{x, y1, y2,y3,y4}”
paper_summary: paper_summary: The paper studies how to handle conflicting alignment targets (e.g., helpfulness and harmlessness) in LLM alignment. 
* A metric is first introduced to measure to what extent different alignment targets conflict with each other. With this metric, the authors analyzed and compared the conflicting degree of some existing alignment dataset (Anthropic HH and UltraFeedback). 
* Secondly, the authors use GPT-4 to re-label selected examples from the UltraFeedback dataset, categorizing them into six new alignment objectives. The resulting new dataset is called Hummer, and a filtered version of Hummer is also proposed named Hummer-F. 
* Thirdly, the authors introduced a dynamic sampling method, which dynamically decides the proportion of samples from different alignment objectives, aiming to minimize the performance gap among the alignment objectives. 

Empirical results show that the newly proposed datasets (Hummer and Hummer-F) have substantially lower conflicting levels between their alignment objectives. Applying the dynamic sampling method on the new datasets yield much more balanced results in the jailbreak tests.
strength_weakness: strength_weakness: * Notations and mathematical definitions can be significantly improved. The main definitions of the metric, presented in Section 4.1, is quite confusing and hard to follow. For example:
  * $D_n^p = {d_1, …, d_n}$, but it’s unclear what is n and what is each element d_1. Intuitively, readers may expect each d_i is a tuple (prompt, winning response, losing response). However, from later descriptions, it seems that each d_i is actually a set of tuples, and hence $D_n^p$ is a set of sets. Also, what is $n$ is not described in Section 4.1? Until very late it became clear that $n$ is the number of alignment objectives.
  * In Definition 1 (Alignment Dimension Conflict), it is unclear how the expectation is defined: what is the set of elements the expectation operator operates over, and what is the probability of each element? Also for the numerator  $(\bar{u}_k - u_k)^2 _{-}$, it is unclear the square operator and the minus operator which has a a higher precedence: if the minus operator has a higher precedence, then it means $(\bar{u}_k - u_k) _{-} * (\bar{u}_k - u_k) _{-}$; however, if the square operator has a higher precedence, it means $[(\bar{u}_k - u_k)^2] _{-}$. 
  * When defining ADC-B, a new subscript $m4 is introduced, but it’s unclear what is the relation between $m$ and $n$: are they completely independent, or are they the same? 
  * All the above unclarities hinder the reader to fully understand the newly proposed metric.
* Ablations are needed. Promising results are presented in Table 1 and 2, but it is unclear which contributes more to the improvements, the new dataset or the new dynamic sampling method. It would be great if the authors can present two results for each dataset, one with the naive sampling strategy, and one with the new dynamic sampling strategy. It would be even better to use other sampling/integration strategies (discussed in Section 1, the last paragraph in Page 1) on each dataset, and compare the performance with the newly proposed sampling strategy. 

Overall, I find the paper very interesting and it studies a highly important topic in LLM alignment, but the current paper lacks clarity and should be significantly improved before being published. I may recommend a ‘conditional acceptance’, requesting the authors to substantially improve the definitions before finally accepting the paper.
paper_summary: paper_summary: In this paper, the authors introduce the concept of Alignment Dimension Conflict (ADC) as a metric to quantify conflicts within preference datasets, particularly in the context of RLHF. They present the Hummer to measure conflicting alignment objectives in preference datasets, showcasing reduced ADC compared to other datasets. The authors further train reward models, HummerRM and HummerRM-F, using a hybrid sampling approach to effectively balance diverse alignment objectives. The analysis reveals a negative correlation between ADC and the number of alignment objectives, suggesting that incorporating more fine-grained alignment metrics could help mitigate conflicts to some extent.
strength_weakness: strength_weakness: Regarding the weakness: this work lacks a thorough discussion on the generalization of various dimensions of human values within the dataset construction process. To improve, 1) the authors could enhance the paper by elaborating on how the alignment objectives identified in the Hummer dataset that can be applied to a wider range of situations / datasets. This might be useful for  evaluating the robustness and utility of the dataset. 2) the paper would also benefit from a more detailed test of the weak-to-strong generalization in alignment. Such an analysis could provide a clearer understanding of how well the proposed methods perform when extended beyond their original setups.
paper_summary: paper_summary: This paper tackles the problem of balancing between different, potentially conflicting alignment objectives in preference datasets (e.g. helpfulness vs. harmlessness). The paper first introduces a metric (ADC) that measures the RM performance deviation on other datasets when fine-tuning on data for a single dataset (each dataset presumably corresponding to a different alignment objective). Then the paper proposes a sampling strategy to get a more balanced RM dataset (Hummer) w.r.t. to the different dimensions. The authors show that the new dataset achieves better ADC scores.

The main issue with the paper is that it’s not written clearly enough so that it would be easy to understand what exactly is the problem it’s trying to solve (beyond the high-level idea of balancing between conflicting alignment objectives). Some examples of unclear writing below:
* Please give concrete examples of what you mean by “alignment dimension” already in the abstract/introduction.
* page 2: Redundant sentences one after another: “Despite these advancements, there remains a notable gap: the absence of a preference dataset specifically designed to mitigate the conflict between alignment dimensions. [...]  Despite these efforts, a significant gap persists: the lack of a preference dataset intentionally crafted to alleviate conflicts between alignment dimensions.” 
* page 6: “We categorize every pair of datasets” Do you mean “every pair of responses”?

Moreover, it’s unclear how useful e.g. the proposed dataset would be in practice beyond helping to optimize the ADC score proposed in the paper.
strength_weakness: strength_weakness: 1. The paper is not written clearly enough to properly motivate the problem and describe the proposed solution in a way that it could be easily reproduced.

2. The practical usefulness of the proposed dataset is unclear (beyond it helping to optimize the score proposed in the paper).
questions: questions: Fig. 4 suggests that the different datasets use different alignment dimensions. Are the ADC values between the datasets still comparable? Why not use shared dimensions?

Comments: title: A Gentle Reminder
Official Response to Reviewer z9ad
Policy Jailbreak experiments on Harmful Samples
Rebuttal by Authors (2/2)
Rebuttal by Authors (2/2)
Official Comments to Reviewer 9qZi Regarding Paper Update
comment: Dear Reviewer,

This is a gentle reminder that we have submitted the rebuttal to address your comments. We sincerely appreciate your feedback and are happy to address any additional questions you may have during this discussion period. We thank you again for taking the time to review our work.

Best regards,
Authors
Thank you for your insightful comments. We concur that further detailed explanations and additional experiments would significantly deepen the understanding of the contributions and implications of our work. Moreover, we acknowledge the merit of your suggestion to conduct an ablation study without hybrid sampling, which we believe will further clarify the impact of this study. 

In the following, we also provide the jailbreak experiments of policy levels on 10 harmful examples for 5 epochs, following the experiments on ([Qi et al, ICLR (2024)](https://arxiv.org/abs/2310.03693), Table 1) to further enhance our submission. 


| Dataset       |                      | Dim1    |            | Dim2    |            | Dim3    |            | Dim4    |            | Dim5    |            | Dim6    |            |
|---------------|----------------------|---------|------------|---------|------------|---------|------------|---------|------------|---------|------------|---------|------------|
|               |                      | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned |
| hh            | Harmfulness Score    | 1.08    | +3.72       | 1.03    | +1.97      | -       | -          | -       | -          | -       | -          | -       | -          |
|               | Harmfulness Rate     | 0.3%    | +52.1%      | 0.3%    | +39.2%     | -       | -          | -       |            | -       | -          | -       | -          |
| Ultrafeedback | Harmfulness Score    | 1.07    | +2.63      | 1.10    | +2.79      | 1.06    | +2.52      | 1.07    | +2.59      | -       | -          | -       | -          |
|               | Harmfulness Rate     | 0.3%    | +51.5%     | 0.4%    | +54.1%     | 0.3%    | +49.7%     | 0.3%    | +50.3%     | -       | -          | -       | -          |
| Hummer        | Harmfulness Score    | 1.07    | +2.57      | 1.05    | +2.3       | 1.06    | +2.37      | 1.03    | +2.11      | 1.06    | +2.35      | 1.06    | +2.51       |
|               | Harmfulness Rate     | 0.3%    | +50.8%     | 0.3%    | +44.1%     | 0.3%    | +47.2%     | 0.3%    | 43.2%      | 0.3%    | 44.3%      | 0.3%    | +49.6%     |


Specifically, the GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. The Harmfulness Score and Rate indicate the average harmfulness score and the fraction of test cases that receive the highest harmfulness score 5, respectively. 

Analysis of the data presented in the table reveals that, on average, the increment in harmfulness score for Hummer is less pronounced than for UltraFeedback. Additionally, Hummer maintains a lower increase in the Harmfulness Rate, staying below 50% in 5 out of 6, in contrast to UltraFeedback, which achieves this in only 1 out of 4.
| Dataset       |                      | Dim1    |            | Dim2    |            | Dim3    |            | Dim4    |            | Dim5    |            | Dim6    |            |
|---------------|----------------------|---------|------------|---------|------------|---------|------------|---------|------------|---------|------------|---------|------------|
|               |                      | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned | Initial | Fine-tuned |
| hh            | Harmfulness Score    | 1.08    | +3.72       | 1.03    | +1.97      | -       | -          | -       | -          | -       | -          | -       | -          |
|               | Harmfulness Rate     | 0.3%    | +52.1%      | 0.3%    | +39.2%     | -       | -          | -       |            | -       | -          | -       | -          |
| Ultrafeedback | Harmfulness Score    | 1.07    | +2.63      | 1.10    | +2.79      | 1.06    | +2.52      | 1.07    | +2.59      | -       | -          | -       | -          |
|               | Harmfulness Rate     | 0.3%    | +51.5%     | 0.4%    | +54.1%     | 0.3%    | +49.7%     | 0.3%    | +50.3%     | -       | -          | -       | -          |
| Hummer        | Harmfulness Score    | 1.07    | +2.57      | 1.05    | +2.3       | 1.06    | +2.37      | 1.03    | +2.11      | 1.06    | +2.35      | 1.06    | +2.51       |
|               | Harmfulness Rate     | 0.3%    | +50.8%     | 0.3%    | +44.1%     | 0.3%    | +47.2%     | 0.3%    | 43.2%      | 0.3%    | 44.3%      | 0.3%    | +49.6%     |


We provide the jailbreak experiments of policy levels on 10 harmful examples for 5 epochs, following the experiments on ([Qi et al, ICLR (2024)](https://arxiv.org/abs/2310.03693), Table 1).  Specifically, the GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. The Harmfulness Score and Rate indicate the average harmfulness score and the fraction of test cases that receive the highest harmfulness score 5, respectively. 

Analysis of the data presented in the table reveals that, on average, the increment in harmfulness score for Hummer is less pronounced than for UltraFeedback. Additionally, Hummer maintains a lower increase in the Harmfulness Rate, staying below 50% in 5 out of 6, in contrast to UltraFeedback, which achieves this in only 1 out of 4.
Thanks for the response. After read other reviewer's response, I decide to maintain the score.
Thanks for the responses and the additional experiments! 

The authors addressed most of my questions and concerns. I would keep my positive recommendation to the paper, but would not raise the score because the authors added substantially more explanations and experiments.
We respectively agree with Reviewer 9qZi and remove the anonymous link for the paper to make fairness to other
submissions and limit the burden of reviewers. In the following, we provide further clarification. Here we provide further essential clarification for our prior rebuttal. 

**Response to Reject 1.3** (*paper writing: redundant sentences*): Thanks for this notification and we will make the correction in our final version.

**Response to Reject 1.3** (*paper writing: we categorize every pair of datasets” Do you mean “every pair of responses”?*):  This refers to every dataset sample $(x, y _1, y _2)$. 

Regarding notations, here we also provide essential updates for notations in Section 4.1: $\mathcal{D}^{P}$ can be further organized as $\mathcal{D}^{P} _n=${$d _1, d _2, \cdots, d _n$} with $d _i={x^k, y _w^k,  y _l^k} _{k=1}^{K _{i}}$, where $d _i$ denotes the alignment dimensions, such as helpfulness in Anthropic HH dataset, $n$ represents the total alignment dimensions, and $K _{i}$ notes the total samples in dimension $d _i$ with $\sum _{i=1}^n K _i=K$}. Formally, given a reward model, i.e., $\text{RM}$, that has been initially fine-tuned on the whole preference dataset $\mathcal{D}^{P} _{n}=\{d _1, d _2, \cdots, d _n\}$, its performance (i.e., accuracy of $\text{RM}(x, y _w) > \text{RM}(x, y _l)$) on the corresponding test dataset from $\mathcal{D} ^{P} _{n}$  is represented by $\mathbb{U} = ${$u _1, u _2, \cdots, u _n$}.
Thanks to the reminder from Reviewer 9qZi regarding the paper update. I respectively agree with Reviewer 9qZi and remove the anonymous link for the paper to make fairness to other submissions and limit the burden of reviewers. In the following, we provide further clarification. 

**Response to Reject Reason 1** (Notations can be significantly improved): Here we provide the essential information for the notations to address your concerns in our paper Section 4.1:  $\mathcal{D}^{P}$ can be further organized as $\mathcal{D}^{P} _n=${$d _1, d _2, \cdots, d _n$} with $d _i={x^k, y _w^k,  y _l^k} _{k=1}^{K _{i}}$, where $d _i$ denotes the alignment dimensions, such as helpfulness in Anthropic HH dataset, $n$ represents the total alignment dimensions, and $K _{i}$ notes the total samples in dimension $d _i$ with $\sum _{i=1}^n K _i=K$}. Formally, given a reward model, i.e., $\text{RM}$, that has been initially fine-tuned on the whole preference dataset $\mathcal{D}^{P} _{n}=\{d _1, d _2, \cdots, d _n\}$, its performance (i.e., accuracy of $\text{RM}(x, y _w) > \text{RM}(x, y _l)$) on the corresponding test dataset from $\mathcal{D} ^{P} _{n}$  is represented by $\mathbb{U} = ${$u _1, u _2, \cdots, u _n$}.  

**Response to Reject Reason 2** (*Ablations study.*): Here we provided ablations in Tab 1.2: 

| Dataset        | Model Type                           | Alignment Dimensions | Dataset Size | ADC $(\downarrow)$ | ADC-B $(\downarrow)$ | Reward Bench $(\uparrow)$ |
|----------------|--------------------------------------|----------------------|--------------|--------------------|----------------------|---------------------------|
| Anthropic HH   | AnthropicRM                          | 2                    | 170k         | 85.04              | 204.6                | 56.72                     |
| UltraFeedback  | UltraRM                              | 4                    | 64k          | 67.23              | 126.3                | 68.34                     |
| Hummer         | HummerRM$_{\text{w/o HS}}$           | 6                    | 46k          | 14.35              | 38.7                 | 68.55                     |
| Hummer         | HummerRM                             | 6                    | 46k          | 11.04              | 31.2                 | 71.52                     |
| Hummer-F       | HummerRM-F$_{\text{w/o HS}}$         | 6                    | 37k          | 12.92              | 36.0                 | 70.39                     |
| Hummer-F       | HummerRM-F                           | 6                    | 37k          | `9.62`             | `28.5`               | `72.13`                   |

**Table 1:** Comparison of existing preference datasets. We demonstrate that all existing preference datasets exhibit a significantly higher ADC $(\%)$ (8-10x) compared to Hum and Hum-F. The best performance is in `bold`.

| Dataset       | Reward Model                     | Initial Fine-Tuning | Further Fine-Tuning |         |        |         |        |        |
|---------------|----------------------------------|---------------------|---------------------|---------|--------|---------|--------|--------|
|               |                                  |                     | #1                  | #2      | #3     | #4      | #5     | #6     |
| Anthropic HH  | AnthropicRM                      | 46.2                | +6.2                | -22.5   | -      | -       | -      | -      |
| UltraFeedback | UltraRM                          | 46.6                | +4.0                | +8.5    | +0.3   | +3.5    | -      | -      |
| Hummer        | HummerRM (w/o HS)                | 46.6                | +3.8                | -1.5    | +0.5   | -11.7   | -2.9   | +0.1   |
| Hummer        | HummerRM                         | 46.4                | +3.6                | -1.7    | +0.3   | -11.7   | -3.2   | +0.0   |
| Hummer-F      | HummerRM-F (w/o HS)              | 46.4                | +2.7                | -1.7    | +0.8   | -11.4   | -3.1   | -0.2   |
| Hummer-F      | HummerRM-F                       | 46.3                | +2.4                | -1.8    | +0.5   | -11.8   | -3.4   | -0.3   |

**Table 2:** Jailbreak rate (% ↓) with further fine-tuning on specific alignment dimensions. While other reward models show highly fluctuating attack ratios, HummerRM demonstrates remarkable consistency with low fluctuation.
Thanks for the important notification! The main reason why we attach our new paper is to update the notations. I respectively agree with Reviewer 9qZi and remove the anonymous link for the paper to make fairness to other submissions and limit the burden of reviewers.
Thanks for clarifying these essential details of your paper.

I respect that you want to further improve your paper. However, linking to a revision and adding major new content is violating the conference policy. The PCs notified everyone on May 26th (i.e., two days prior to you posting this rebuttal) that this is not allowed. Doing so is unfair to other submissions obeying these rules.

Here is the most relevant passage from the notification:
> We do not allow to revise the submitted content during the rebuttal and discussion period, and we limit the length of rebuttals. [...] Linking to a revision or large amount of new content will go against this policy, and we ask not to do so