Hummer: Towards Limited Competitive Preference Dataset
Authors: Yusen Wu
Li Jiang
Junwu Xiong
Jingqing Ruan
Yichuan Ding
Qingpei Guo
zujie wen
JUN ZHOU
Xiaotie Deng
Venue: COLM (2024)
Keywords: reinforcement learning from human feedback, RLHF, preference dataset, reward models, alignment
TL;DR: We introduce \texttt{Hummer} and its fine-grained variant \texttt{Hummer-F}: innovative datasets models to mitigate alignment conflicts, improving task adaptation and security against jailbreak attacks.

Abstract: Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present \texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. \texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, \texttt{HummerRM} and \texttt{HummerRM-F}, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions \texttt{HummerRM} as an ideal model for domain-specific further fine-tuning and reducing vulnerability to jailbreak attacks.

Published as a conference paper at COLM 2024
Hummer: Towards Limited Competitive Preference Dataset
Yusen Wu
Li Jiang" Zujie Wen
Junwu Xiong
Jingqing Ruan Xiaotie Deng!
Yichuan Ding
Qingpei Guo
Jun Zhou²
!Peking University
McGill University
Ant Group
(sarinw2023, jiangli3859,
junsuce) @gmail com
Abstract
Preference datasets are essential for incorporating human preferences into pre-trained language models, playing key role in the success of Rein- forcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading increased vulner- ability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present Hummer and its fine-grained variant, Hummer innovative pairwise preference datasets with reduced-confie objectives. Hummer is built based on Ultra Fleebakk and is enhanced by AT feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, HummerRM and HummerRM-F which employ a hybrid sampling approach to balance diverse alignment objectives effec- tively. This sampling method positions HummerRM as an ideal model for domain-specific further fine-tuning and reducing vulnerabilities to attacks. Access the dataset via this link.
aligmment
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) exhibits great potential in integrat- ing human preferences into large language models (LLMs) (Christiano et al., 2017; Ouyang et al,, 2022; Bai et al., 2022; Touvron et al., 2023; Achiam et al., 2023). RLHF also holds great promise in real-world domains, such robotics (Hu et al., 2023; Tian et al., 2023), healthcare (Yu et al., 2023; He et al., 2023), and autonomous driving (Chen et al., 2023; Du et al., 2023). A fundamental aspect of integrating human alignment objectives lies in the preference modeling stage, which crucially depends on given preference dataset. This stage can be realized by constructing either explicit (Christiano et al., 2017; Ouyang al., 2022; Touvron et al., 2023) or implicit reward models (Rafailov et al., 2023; Zhao et al., 2023a).
However, alignment objectives often present competing properties in current preference datasets (Biyik & Sadigh, 2018; Hong et al., 2022; Ganguli et al., 2022; Wu et al., 2024). Considering the Anthropic-HH dataset (Bai al., 2022), emphasizing the alignment ob- jective of harmlessness may cause an agent to offer only broad or overly cautious advice. This emphasis could prevent the agent from delivering impactful and precise guidance, which limits the capability of helpfulness. This competition dynamics among alignment objectives poses two significant challenges. On one side, it exacerbates the vulnerability of safety-trained LLMs to jailbreak attacks by crafting prompts to prioritize alignment dimension over others (Wei et al., 2024). Besides, the conflict dynamics further complicate the attainment of equilibrium among all alignment objectives, particularly customizing models for downstream tasks that require promotion to specific dimensions ability without sacrificing performance in other alignment objectives, such math reasoning (Azerbayev et al., 2023) and code generation (Guo et al., 2024a).
"Fqual contribution, more junior authors listed earlier.
1

---

Published as a conference paper at COLM 2024
To mitigate the conflict of alignment dimensions in the preferences dataset, one line of research direction is to separately construct twisted alignment objectives via distinct reward or cost models to explicitly decouple human alignment objectives. Subsequently, these decoupled, learned models are synergized to provide a holistic preference signal tailored to specific alignment goals for downstream tasks. The integration methodology encompasses a variety of strategies, consisting of voting mechanisms (Wang et al., 2024a), differential weighting of models (louvron et al., 2023; Wu et al., 2024), application of linear combinations (Jang et al., 2023; Rame et al., 2024), and the use of the Lagrangian multiplier (Dai et al., 2023) manage trade-offs or to highlight specific principles of human values. However, those approaches inadvertently increase model complexity and computational overhead.
In this study, we redirect our focus toward the underlying cause of alignment conflict: the preference dataset itself. RLHF community has witnessed an emerging trend towards developing new preference datasets, driven by goals of enhancing quality and scale, incor- porating fine-grained preference signal, and covering specific domains aligned with desired dimensions (Cui et al., 2023; Ji et al., 2024b; Wu et al., 2024; Stiennon et al., 2020; Lightman et al., 2023; Ethayarajh et al., 2022). Despite these efforts, significant gap persists: the lack of a preference dataset intentionally crafted to alleviate conflicts between alignment dimensions. Such dataset could potentially provide significant benefits for downstream applications that prioritize certain values (Zhang et al., 2024; Wang et al., 2024b) and reduce vulnerabilities to jailbreak attacks (Perezz et al., 2022; Qi al., 2023; Wei et al., 2024; He et al., 2024; Lyu et al., 2024; Cui et al., 2024). Moreover, there is currently no established statistical metric for assessing the degree of conflict among alignment dimensions within preference datasets.
In light of these observations, we first introduce Alignment Dimension Conflict (ADC), statistical metric for quantifying the degree of conflict within preference datasets. This new criterion moves beyond the conventional metric of average performance across multiple objectives or domains typically featured on current leaderboards. We then present Hummer, standing as the first preference dataset to highlight limited competition among various alignment objectives. The construction of Hummer capitalizes on the advanced capabilities of Al feedback mechanisms, such as GPTT4 (Achiam et al., 2023), consisting of a three-stage process: preference & objective annotation, alignment objectives refination, and dataset split. We use the UltraFeedback (Cui et al., 2023) as our foundation dataset for this work and introduce a fine-grained version of Hummer, termed Hummer-F, which excludes the noisy preference dataset via the principle of reward gaps and compromises 80% of Hummer.
Based on Hummer and Hummer-F, we introduce hybrid sampling strategy for training their respective reward models, HummerRM and HummerRM-F, based on the established Llama 2-7B model (Touvron et al., 2023). The hybrid sampling strategy achieves well- balanced performance across diverse limited-competition alignment objectives in Hummer, enhances resilience to jailbreak attacks, and supports further fine-tuning in downstream tasks. It accomplishes this by prioritizing certain alignment objectives without sacrificing performance in other dimensions. We summarize our contributions in two main folds:
1. We introduce the Alignment Dimension Conflict (ADC), a statistical metric for quantifying conflict in preference datasets. We then present Hummer and its refined variant, Hummer-F, designed as the first preference datasets to mitigate competing alignment objectives.
2. We develop hybrid sampling strategy to train the reward model HummerRM from Hummer, balancing performance across alignment objectives and further limiting the conflict. HummerRM boosts defense against jailbreak attacks and enables down- stream fine-tuning by focusing on key alignment dimensions without compromising others.
2 Related Work
RLHE RLHF has emerged as the leading strategy to integrate human preferences into language models through preference datasets, which can be fixed pre-collected or generated from agents or language models (Cheng et al., 2011; Akrour et al., 2011; Askell et al., 2021;
2

---

Published as a conference paper at COLM 2024
Xu et al., 2022; 2023). To integrate human values, RLHF generally obtains the final aligned policy through RL algorithms, such as PPO (Schulman et al., 2017), to maximize the reward through the trained reward model preference datasets (Ramamurthy et al., 2022; Bai et al., 2022; Quyang et al., 2022; Touvron et al., 2023). Another important branch is to directly anchor the human preferences to the final policy by constructing the implicit reward with policies through the closed-form optimal solution for the reward model (Rafailov et al., 2023; Zhao et al., 2023b; Azar et al., 2023; Wang et al., 2023; Ethayarajh et al., 2024; Zhou et al., 2023; Amini et al., 2024; Liu et al., 2023; Swamy et al., 2024). While these approaches are appealing for their computation cost and ease of implementation, their inherited offline paradigm suffers from the distributional shift and lack of online exploration (Guo et al., 2024b; Calandriello et al., 2024).
Preference Datasets. The RLHF community is observing growing trend of new prefer- ence datasets from diverse perspectives to improve preference modeling. The dominant motivations for the introduction of new preference datasets scalability, quality, and diversity (Guo et al., 2023; Cui et al., 2023; Wu et al., 2024). For example, SPA dataset (Guo et al., 2023) presents fine-grained (i.e., token or phrase level) feedback during optimization rather than holistic feedback during the training process. UltraFeedback (Cui et al., 2023) in- troduces a wide-source and high-quality preference dataset with four alignment dimensions, in contrast to two dimensions (helpfulness and harmlessness) (Ouyang et al., 2022). Besides, some recent preference datasets underscore a specific domain or alignment property (Stien- non et al., 2020; Lightman et al., 2023; Ethayarajh et al., 2022). However, existing preference datasets fail mitigate the conflict between alignment dimensions. Enhancing the synergy of alignment dimensions improves resilience against jailbreak attacks and allows for further tine-tuning in downstream applications. This is achieved by prioritizing specific alignment objectives without compromising performance across other dimensions.
Red Teaming LLMs with Further Fine-tuning. Red teaming is designed to execute system- atic tests and attacks on LLMs to expose their potential harmfulness and safety vulnerabili- ties (Perez et al., 2022; Achiam et al., 2023; Shi et al., 2024; Lyu et al., 2023; Kang et al., 2024). Recent work (Qi et al., 2023; Zhan et al., 2023; He et al., 2024) identifies that customizing policies with further fine-tuning on downstream tasks, even without harmful content, will lead to a degradation in resilience against jailbreak attacks for -afety-alignment policy. We hypothesize that this degradation stems from an implicit emphasis on specific alignment dimensions, such as helpfulness, and the conflict among alignment dimensions present in downstream datasets, where the learned policy is either an implicit (DPO pipelines) or ex- plicit distillation (PPO pipelines) of the reward models. In this work, we focus on the conflict of alignment dimensions and study further fine-tuning specific alignment dimensions on the preference modeling stage (reward model) to improve specific ability for customization tasks. Aligned with these findings, we show that further fine-tuning downstream models on desired alignment dimensions inevitably leads to performance degradation in conflicting dimensions, e.g., safety.
3 Preliminaries
RLHF typically starts with a generic pre-trained language model from supervised fine- tuning on a high-quality dataset for general proposes, such as conservation, noted as SFT, and then matches human preferences through a preference dataset. In this work, we mainly study the problem of competing alignment objectives in existing preference datasets.
Preference Modelling. One of the core ingredients of RLHF to integrate human prefer- K ences into LLMs through preference datasets, formulated as DP = {x**, v v Tk 1' where K is the number of total collected samples. The preference dataset D incorporates the human feedback through the preference to these two responses, i.e, yı > y/ x. Given the prompt x, yw denotes the preferred response by humans or advanced AI models compared to y/- Given the preference dataset DP, we can then parameterize reward model rg (x,y) and
3

---

DD can be further organized as D d2, dd with d; { K, k-1 where Published as a conference paper at COLM 2024
optimize it through through the Bradley-Terry (BT) model (Bradley & Terry, 1952):
max IE (xyDo)))>DP [loger (ro (x, TΦ (xx,)))),
(1)
T+p
where o is the logistic function. In the context of LLMs, the reward model rg (x, is frequently constructed based 7TSFT (y x) by adding linear layer on top of the final transformer layer, which yields a single scalar prediction representing the reward value (Ziegler et al., 2019; Ouyang et al., 2022).
Policy Modelling. Given the learned reward model rg (x, constructed through prefer- ence modeling and policy training dataset DRL {x*) j= we can formulate the following optimization objective for Ty to inherit the preference from ro (x)
max E yr Irg (x,(x,))
- PD (oly x) (y xx]
where f moderates the divergence D, such as KL divergence (Kullback, 1951), between zι and soft-target policy TSFT This regularization ensures that Th avoids collapsing into narrow set of high-reward actions, preserving a diverse and functional output distribution as supported by the learned reward model (Jaques et al., 2019; Song et al., 2023; Laidlaw et al., 2024; Wei al., 2024). PPO is employed for policy optimization due its proven efficiency and stability in training.
4 Hummer
This section begins with a formal definition of Alignment Dimension Conflict (ADC), metric to assess the extent of conflicting alignment objectives in preference datasets (Section 4). We then introduce Hummer and its fine-grained variant, Hummer-F, which are datasets specifically crafted to empha- size the property of limited conflict among alignment objectives of preference datasets (Section 4.2).
DPP
copy dataset
d
initial fine-tuning
further tuning
copy RM
RM
RM
RM,
evaluation
evaluation
U
ADC
Figure 1: The ADC estimation pipeline, mea- suring the negative performance gap between initial and further fine-tuned reward models.
4.1 Alignment Dimension Conflict
d; denotes the alignment dimensions, such as helpfulness in Anthropic HH dataset (Bai et al., 2022), n represents the total alignment dimensions, and K; notes the total samples in dimension d; with Σ K; K. Formally, given a reward model, iee, RM, that has been initially fine-tuned on the whole preference dataset D {d,, ... d }, its performance (i.e., accuracy of RM(x, y/w) > RM(x, yr)) on the corresponding test dataset from D is represented by U [i1, U2, ,
To study this conflict, we copy n reward models and further fine-tune each reward model on the dataset of interest for any alignment dimension, e.g., d E DP, obtaining the fine-tuned performance U, = [ Ti Min) The performance deviation can be obtained by (U; U)) of RM, where highlights further fine-tuning conducted only on d; We present the pipeline for measuring this dimension conflict, considering only negative performance deviations, in Fig. and introduce new statistical metric:
Definition 1 (Alignment Dimension Conflict). The Aligument Dimension Conflict (ADC) is defined the second-order moment of the negative performance deviation summation on all dimensions except di
E [iis Us)_
u D
E,
with "- = min (in(u,,),
(2)
n -
4

---

Published as a conference paper at COLM 2024
ADC=0.4
ADC=4.3
Performace Deviation
Perfomance Deviation
0
(a) Normal distribution ADC
(b) Low ADC
(c) High ADC
Figure 2: (a) Normal distribution of ADC with varying standard variance [x= U[x]
(b-c) The performance deviation with further fine-tuning on the first dimension of preference datasets with (b) low and (c) high ADC. Intuitively, a high ADC indicates strong conflict between the alignment dimensions given preference dataset.
where n - serves as a normalization term to facilitate fair comparison for different datasets with different alignment dimensions and E,[-] denoles the expectation over the performance deviations
obtained by further fine-tuning on alignment dimension dien with [-] -i-l-//
An interesting question to ask is: What situation leads to high ADC? We simplify the perfor- mance deviation (U U) sampling from normal distribution N' (0, The expression Ex=NN(==0,0²) U[x] in Fig. 2a represents the ADC of a normal distribution with respect to its variance parameter This measures how much adjusting one alignment dimension affects others with further fine-tuning. We observe a strongly positive correlation between ADC and 0, indicating that datasets with higher level of competing dimensions (evidenced by greater variance on the negative side) tend to exhibit higher ADC values. The performance deviation across datasets with varying ADC levels is illustrated in Fig. 2, where datasets with low ADC are characterized by minimal negative impact on the performance across other alignment dimensions, i.e, lower competition.
Rench (Lambert et al., 2024) offers toolkits for structured comparison across various properties in reward models, accommodating diverse model structures. To facilitate systematic comparison of alignment dimension conflict levels among different reward models trained from different preference datasets, we can scale the Alignment Dimension Conflict (ADC) metric to the evaluated properties on standard evaluation toolkits, termed ADC-B, as shown in Definition
4.2 Dataset Construction for Hummer
To decouple alignment dimensions, we introduce Hummer, the first preference dataset that aims to alleviate the competing dynamics of preference datasets. To accurately capture the multidimensionality of human preference without interference between alignment dimensions, we leverage the powerful ability of AI feedback, i.e., GPT-4, which has been heavily employed in preference dataset construction or preference modeling (Lee et al., 2023; Cui et al., 2023; Guo et al., 2023; Burns et al., 2023; Chen et al., 2024; Ji et al., 2024a). We leverage UltraFeedback (Cui et al., 2023) the foundational dataset, attributed to its expansive scale and diversity.
We show the construction process of Hummer in Fig. 3, detailed in Appendix B. The process of identifying the mumited-conflicn dimension and its corresponding pairwise dataset involves three key stages: (a) Preference annotation: Initially, we randomly select 400 pairwise preference datasets (x, y1, y²2x from the foundational dataset. For each pair, we annotate preferences, alignment dimensions, and the corresponding reasons (p, dimension, reason)*
'hee assumption that = 0 is justified because further fine-tuning along dimension d, might enhance performance in some dimensions while adversely competing with others.

---

Published as a conference paper at COLM 2024
Preference & Objective Annotation
Alignment Objective Refination
Dataset Split
(p. d, reason
foundation datasel
foundation dataset
pronpe libed roward lowoookl algrnaar. aqpetivee 13
sampling
{ar Ur 323
pronss fed lec uópodivea
skperret
proppt find wrich one beterr drwaatoo
categorize via maximal
gap
D
(p d, reason
(d......da,
d
Figure 3:
Hummer construction process. We leverage the advanced ability of GPT-4 build Hummer, a preference dataset with low competitive alignment objectives.
powered by GPT-4. We totally collect 37 alignment dimensions. (b) Alignment objective refination: We then leverage GPT-4 to refine these dimensions to minimize their conflicts and finally get ı = 6 alignment dimensions: { l'accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity"). (c) Dataset split: GPT-4 is then used to assign absolute reward r to alignment dimensions for every sample in the foundation dataset. We categorize every dataset sample (x, y1, 1/2) to its corresponding dimension on the principle of maximal preference gap, i.e., max" di
(x, (1); ;-- r (xi, y2), We highlight that this splitting approach more favorable than directly ranking it avoids the position bias (Zhu et al., 2023) and facilitates convenience to build Hummer-F. Hummer-F is refined by applying a reward gap threshold (T 0.5) to filter out potentially noisy preference pairs, subset that comprises approximately 80% of Hummer.
Hybrid Reward Sampling
In this section, we introduce HummerRM and its variant, HummerRM-Fi Both are single- reward models trained on our custom-limited competitive preference datasets, Hummer and Hummer-F, respectively. These models employ a hybrid sampling method to dynamically balance alignment dimensions, further mitigating alignment dimension conflicts in our proposed datasets, especially with imbalanced datasets over alignment dimensions, as shown in Appendix A.
Formally, considering preference dataset with n alignment objectives, denoted as D (d.,d2...,,uu) we assign initial equal sampling weight to each dimension dataset, represented by A {A,,A2,.... .n where A = 1/n with E [1, We achieve the balance among all alignment dimensions by evaluating the preference performance across these dimensions, denoted as U = (uI...,,uy) Mr). The sampling weights are adaptively updated in every 1 epoch (128 steps) as follows:
4 A 17 u;),
(3)
where represents the average preference performance across all alignment objectives, and y is the temperature for updating the sampling weights A. To ensure adherence to the sum constraint, I" Σ"1 A 1, we normalize the A; values accordingly after every update. Consequently, the mini dataset sampled at each training step is represented by BatchSize A| from DI, where BatchSize 128 and [x] represents the floor function.
Intuitively, if the performance of a specific dimension, e.g., Hj is higher than the average (n; > f)), the corresponding sampling ratio A; for dataset d, decreases. Conversely, if u, < A, indicating a performance lower than the average, A; increases, promoting an increasing sampling dataset for d, We then integrate all sampled datasets into one training batch and update the reward model via Eqn. (1). The hybrid sampling strategy enhances the robust performance of HummerRM across all alignment dimensions.

---

Published as a conference paper at COLM 2024
Dataset
Model
Alignment Dimensions
Dataset Size
ADC (4)
ADC-B (4)
Reward
Type
Bench (t)
Anthropic HH
AnthropicRM
2
170k
85.04
204.6
56.72
Ultrakeedback
UltraRM
64k
67.23
126.3
68.34
Hummer
HummerRM
46k
14.35
38.7
68.55
Hummer
HummerRM
46k
11.04
31.2
71.52
Hummer-F
HummerRM-Fww
37k
12.92
36.0
70.39
Hummer-F
HummerRM-F
37k
9.62
72.13
Table 1: Comparison of existing preference datasets. We demonstrate that all existing preference datasets exhibit significantly higher ADC (%) (8-10x) compared to Hummer and Hummer-F. The best performance is in blue.
6 Experiments
Our testbed designed to assess the low-conflict alignment dimensions within our intro- duced datasets, namely Hummer and Hummer-F. We initiate our evaluation by examining the ADC and ADC-B using HummerRM, alongside standard reward benchmark, as detailed in Section 6.1. Subsequently, we explore the vulnerabilities of HummerRM, shown in Sec- tion 6.2. Finally, we assess the efficacy of the hybrid sampling strategy in comparison to diverse sampling methods in Appendix A.
6.1 Reward Model Evaluation
Setup. To elucidate the dynamics of low competition in Hummer and Hummer-F, assess the ADC within their respective preference datasets; and ADC-B equipped with systematical comparison RewardBench (Lambert al., 2024). This evaluation is contextualized by comparisons with the Anthropic HH dataset (Bai et al., 2022), and UltraFeedback (Cui et al., 2023). Furthermore, we explore the effectiveness of hybrid sampling strategies in the training of reward models. For consistency across evaluations, we employ consistent backbone model, specifically a fine-tuned Llama 2-7B (Touvron et al., 2023), train the reward models for each dataset.
15
35
Rertomanc Devadoo
Ittal
#3
63
82
#3
46
Dimencons
(a) Anthropic HH
(b) UltraFeedback
(c)
Hummer
Figure 4: The performance deviation with further fine-tuning different alignment objec- tives, where the green bar indicates the further fine-tuning dimensions. Notably, Hummer demonstrates minimal competition among alignment dimensions.
Result. In Tab. 1, we summarize prevalent preference datasets with our statistical evalu- ation findings. Notably, Hummer and Hummer-F demonstrate significantly reduced ADC (8-10x) compared to other preference datasets, even without hybrid sampling (HS). The structured comparison in ADC-B on RewchdBench uncovers a notable consistency with the ADC results. Despite the preference dataset for Hummer and Hummer-F being considerably smaller (3-4x) than UltraFeedback, and HummerRM-F over UltraRM, by margins of 3.8% and 3.2% on RewardBench, respec- tively. This underscores the significance of dataset quality in preference datasets.
observe enhanced performance from HummerRM
7

---

Published as a conference paper at COLM 2024
Reward model
Initial fine-tuning
Further tine-tuning on alignment dimensions of RM
Dataset
#
#3
#4
#5
Anthropic HH
AnthropicRM
46.2
6.2
22.5
UltraFeedback
UltraRM
46.6
4.0
+ 8.5
+ 0.3
+.5
Hunner
HummerRM
+38
1.5
+.55
11.7
2.9
Hummer
HummerRM
46.4
3.6
1.7
1 0.3
11.7
3.2
0.0
Hummer-F
HmmerRM-FFw.
46.4
2.7
1.7
1 0.8
11.4
3.1
0.2
Hunner-F
HmmerRM-F
46.3
+24
188
0.5
11.8
344
Table 2: Jailbreak rate (%, with further fine-tuning on specific alignment dimensions. While other reward models show highly fluctuating attack ratios, HummerRM demonstrates remarkable consistency with low fluctuation. Warm colors are used to show increased jailbreak rates and cold colors
(preferred) refer to decreased jailbreak rates.
Ablation. The ablation study on the HS strategy reveals that improvements ADC and ADC-B are primarily derived from our proposed datasets, while observable margin with HS, i.e, around 3% and 7% for ADC and ADC-B respectively. Our observations confirm that HS is crucial for enhancing saaerboard-centrre performance primarily aiming at "achieving a higher score" on Rewardbench. Additionally, we emphasize the importance of data quality in further fostering improvements in ADC and RewardBench. Despite these observed gains, this study fundamentally aims to identify and quantify the competing dynamics prevalent in preference datasets.
6.2 Jailbreak Attacks Evaluation for Reward Models
Setup. We posit that the HummerRM framework can mitigate vulnerabilities to jailbreak attacks by enhancing one dimension without degrading performance across other metrics. Our jailbreak evaluation framework follows Siththaranjan et al. (2023). Specifically, the jailbreak-based dataset comprises pair-wise tuples (1,),, where x represents prompts designed to elicit a harmful response from the model, 1/1 denotes the safe response, and y2 is jailbreak response (Wei et al., 2024). We quantify the 'jailbreak rate' through the proportion of instances where the reward model favors (x. over (x, (1), represented by (Σ" ((x(x,y))) r(x, v1))) where I is the indicator function and denotes the total prompts. The higher the jailbreak rate, the greater the vulnerability of models to attacks.
Result. In Tab. 2, we delineate the outcomes jailbreak attacks on Anthropic HH (Ouyang et al., 2022), UltraFeedback (Cui et al., 2023), and Hummer, with each model integrating 2, 4, and alignment dimensions, respectively. Initial fine-tuning yields a uniform jailbreak rate across all datasets. Notably, UltraRM registers the highest attack rate, exhibiting 10.4% increase following further fine-tuning on the instruction-follloinng alignment dimension (# 2). This highlights a significant escalation in vulnerability to jailbreak attacks when UltraRM is specifically fine-tuned to enhance cnnrruction-followiin underscoring a pronounced tension with safety protocols. Conversely, HummerRM demonstrates exceptional robustness, with a jailbreak rate increment of less than 3% subsequent to additional fine-tuning across all dimensions. This indicates that the alignment objectives of Hummer are harmoniously integrated, ensuring that its safety remains unimpaired by further fine-tuning.
We emphasize that declining jailbreak rate signifies enhanced defensive capabilities against jailbreak attacks. This improvement is particularly notable when further fine-tuning focuses on specific alignment dimensions, such harmlessness (# 2) in the case of Anthropic HH, and empathy (# 4) in Hummer. Appendix Tab. 7 shows the detailed alignment dimensions for preference datasets.
The ablation study on the HS indicates the strong ability of reward models against jailbreak attacks is most saturated from Hummer and Hummer-F, while hybrid sam- pling further enhances the defensive capabilities. These results align with those observed in the Tab. 1, affirming ADC's reliability as a proxy for quantifying preference conflicts in datasets. Addressing these conflicts is essential for maintaining resilience against jailbreaks.
Ablation.
8

---

Published as a conference paper at COLM 2024
Further fine-tuning on alignment dimensions of RM
Dataset
base
#2
#3
#4
#5
#6
Average
Anthropic HH 12.1
3.0
1.3
+ 1.7
UltraFeedback
12.4
1.3
1.2
+ 1.4
Hummer
12.0
0.8
0.4
0.6 1.2
0.3
+ 0.4
Table 3: Harmfulness Rate 4%,4) of policies with further fine-tuning on specific alignment dimensions of reward models, where all policies are obtained through further fine-tuning on Dolly datasets to simulate customizing fine-tuning. Hummer achieves the lowest increment of the harmfulness rate. Warm colors are used to show increased harmfulness rate and cold colors (preferred) refer to decreased harmfulness rate.
6.3 Jailbreak Attacks Evaluation for Policy Models
Setup. Qi et al. (2023) implies that custom fine-tuning on the policy models (even without harmlessness datasets) will significantly undermine the initial safety alignment. We can observe that the harmfulness rate increases near 12% with further fine-tuning of policies Dolly (Conover et al., 2023) to simulate the downstream tasks, shown in Tab. 3 (base). The harmfulness rate metric follows Qi et al. (2023) by using the GPT-4 judge which outputs harmfulness score in the range of 1 to 5. The harmfulness rate indicates the fraction of outputs from the policy with the highest harmfulness score 5 on the test dataset. We hypothesize that further fine-tuning of reward models to highlight specific alignment abilities (e.g., helpfulness) serves as an amplifier to diminish the safety of initial alignment policies due to conflict of alignment dimensions.
Results. We demonstrate the results in Tab. 3 by further fine-tuning the reward models on specific alignments to prioritize certain values (such as safety in finance scenarios), which retains the same training setting in Section 6.2 but evaluates policy levels. We can observe a subtle increment (+0.4%) on average in the harmfulness rate of policies whose reward models trained from Hummer, which implies that further fine-tuning specific alignment dimensions on Hummer does not exaggerate the loss of safety. On the contrary, the increment of the harmfulness rate of policies from Anthropic HH and UltraFeedback are 3-4x to Hummer, demonstrating an unneglectable amplification of the safety loss of policies.
7 Conclusion
In this study, we delve into the dynamics of competing preferences within the Reinforcement Learning from Human Feedback (RLHF) framework. We introduce novel statistical metric termed Alignment Dimension Conflict (ADC) to quantify the extent of conflict among align- ment objectives within preference datasets. We unveil the first preference dataset, Hummer, alongside its fine-grained variant, Hummer-F. These datasets are designed to mitigate dimen- sion conflicts, facilitating domain-speciffs fine-tuning while increasing resilience against jailbreak attacks. This is achieved by selectively prioritizing certain alignment objectives without compromising performance across other alignment objectives. Subsequently, we de- velop reward models for our datasets, namely HummerRM and HummerRM-F, employing a hybrid sampling technique that dynamically adjusts the sampling weight based on reward performance across different alignment dimensions. Looking ahead, an intriguing avenue for future research lies in constructing low-conflict alignment objectives using unsupervised or self-supervised (Zhang et al., 2020; Yan et al., 2021) learning methods to discern semantic nuances. Furthermore, exploring the conflict of alignment dimensions in the preference modeling stage offers promising avenue for understanding the safety trade-offs in further fine-tuning policies (Qi et al., 2023).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.

---

Published as a conference paper at COLM 2024
Gpt-4 technical report. arXiv preprint arXiv:23303.0674,2222.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Proc. of KDD, 2011.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint Xiix:2402.10511 2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint ar arxi:22112.00861 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint 0rfoo23300012033 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.1063) 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arv preprint arXiv:2204:05862, 2022.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning, 2018.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision. arv preprint ar arXiv:2312.093390 2023.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimi- sation. ar preprint ar 3kir:22033006355, 2024.
Hong Chen, Kang Yuan, Yanjun Huang, Lulu Guo, Yulei Wang, and Jie Chen. Feedback is all you need: from chatgpt to autonomous driving. Science China Information Sciences, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models. arXiv preprint ar Kii:2401001333 2024.
Weiwei Johannes Fürnkranz, Eyke Hüllermeier, and Sang-Hyeun Park. Preference- based policy iteration: Leveraging preference learning for reinforcement learning. In Proc. of KDD, 2011.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Proc. of NeurlPS, 2017.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm. Company Blog of Databricks, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. Xiv preprint 27X1o:2310.01137, 2023.
10

---

Published as a conference paper at COLM 2024
Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, et al. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. arxiv preprint ar arXiv:2401.057 2024.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. ar preprint arXte:2310.12273, 2023.
Haiping Du, Siyu Teng, Hong Chen, Jiaqi Ma, Xiao Wang, Chao Gou, Bai Li, Siji Ma, Qinghai Miao, Xiaoxiang Na, et al. Chat with chatgpt on intelligent vehicles: An ieee tiv perspective. IEEE Transactions Intelligent Veltccees, 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Proc. of ICML, 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ar preprint arXiv:2402.01306 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ar preprint 7XXt:2209.07858 2022.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- rise of code intelligence. arXiv preprint ar arXio:2401.14196 2024a.
Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Be- yond imitation: Leveraging fine-grained quality signals for alignment. Xiv preprint arXie:231.00407,, 2023.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. Xiv preprint X400:2202200792,, 2024b.
Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. Xiv preprint ar Xfo:23310.05694 2023.
Luxi He, Mengzhou Xia, and Peter Henderson. What's in your" safe" data?: Identifying benign data that breaks safety. Xrv preprint arXiv:2404.0109) 01099, 2024.
Joey Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models. arXiv preprint arXio:2212.04711,, 2222
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy misalignment in preference-based reinforcement learning. aix preprint arXi::2305.17400 2023.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model aligmment via post-hoc parameter merging, arXiv preprint ar 4rXie:2310.115: 115664 2023.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXit:1907.00456, 2019.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. at Xiv preprint ar arXio:2402.02416, 2024a.
11

---

Published as a conference paper at COLM 2024
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety aligmment of Ilm via humman-preference dataset. Proc. of NeurlPS, 2024b.
Li Jiang, Sijie Chen, Jielin Qiu, Haoran Xu, Wai Kin Chan, and Zhao Ding. Offline reinforce- ment learning with imbalanced datasets. Xrv preprint ar Xfft:2307.02752, 2023.
Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. C-rag: Certified generation risks for retrieal--augmente language models. arvi preprint arXi::2402.08181,
2024.
Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Arlificial Intelligence, 2016.
Solomon Kullback. Kullback-leibler divergence, 1951.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan, Preventing reward hacking with occupancy measure regularization. arXiv preprint ar Xix::2403.003185 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint ar Xii::2309.002666 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. Xiv preprint ar Xic:23305.2005 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. ar preprint crXit:2309.066:3 2023.
Weimin Lyu, Songzhu Zheng, Haibin Ling, and Chao Chen. Backdoor attacks against transformers with attention enhancement. In ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning, 2023.
Weimin Lyu, Xiao Lin, Songzhu Zheng, Lu Pang, Haibin Ling, Susmit Jha, and Chao Chen. Task-agnostic detector for insertion-based backdoor attacks. arXiv preprint arXiv:2403.17155, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback, 2022. URL https://arxiv. org/abbs/200.021555, 2022.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint ar xxXi:2202.002286 2022.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pu Chen, Ruoxi Jia, Prateek Mittal, and Peter Hen- derson. Fine-tuning aligned language models compromises safety, even when users do not intend to! Xrv preprint ar arXio:2310.03693, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arx preprint ar Xrxi:22305.18290 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arxiv preprint arXio:2210.012211, 2022.
12

---

Published as a conference paper at COLM 2024
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Proc. of NeurlPS, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ar Xiv preprint ar (tte:1707.003477 2017.
Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. Red teaming language model detectors with language models. Transactions of the Associa- fion for Computational Linguistics, 12:174-189, 2024.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional prefer- ence learning: Understanding and accounting for hidden context in rlhf. ar preprint ar arXiv:2312.08838, 2023.
Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. Reward collapse in aligning large language models. Xrv preprint ar drXi::2305.17600 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Proc. of NeurIPS, 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXio preprint ar Aii::22401. 004066, 2024.
Ran Tian, Chenfeng Xu, Masayoshi Tomizuka, Jitendra Malik, and Andrea Bajesy. What matters to you? towards visual representation alignment for robot learning. arXiv preprint ar drXiv:2310.07932, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arv preprint (XXi:2337.092880 2023.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arViv preprint ar arXi::2401.06088 2024a.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arviv preprint ar 4rx:::2309.1 16240, 2023.
Haixin Wang, Xinlong Yang, Jianlong Chang, Dian Jin, Jinan Sun, Shikun Zhang, Xiao Luo, and Qi Tian. serameter-efficieen tuning of large-scale multimodal foundation model. Advances in Neural Information Processing Systems, 36, 2024b.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Proc. of NeuriPS, 2024.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Proc. of NeuriPS, 2024.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A gulicy imitation approach for offline reinforcement learning. Advances Neural Information Processing Systems, 35: 4085-4098, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arv preprint ar arii:2330.1,880, 2023.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. Consert: A contrastive framework for self-supervised sentence representation transfer. arx preprint WxXt::200511744, 2021.
13

---

Published as a conference paper at COLM 2024
Ping Yu, Hua Xu, Xia Hu, and Chao Deng. Leveraging generative ai and large language models: a comprehensive roadmap for healthcare integration. In Healthcare, 2023.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning. arviv preprint ar arXfo:2311.0555) 2023.
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint arXit:2403.10131 2024.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An unsuper- vised sentence embedding method by mutual information maximization. Xiviv preprint axXiv:2099.1261 2020.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic- hf: Sequence likelihood calibration with human feedback. arviv preprint 100422205..04255, 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint XiX::23005.10425, 2023b.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond foll Multi-objective direct preference optimization. Xrv preprint ar xrXi:23310.03708, 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness harmlessness with rlaif, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint ar Xix::1909008599 2019.
14

---

Published as a conference paper at COLM 2024
A Analysis
Hybrid sampling strategy maintains perfor- mance imbalanced datasets. An imbal- anced dataset arises with a non-uniform distri- bution of classes, often characterized by dispro- portionate number of instances between major and minor classes, resulting in biased predic- tions (Krawczyk, 2016; Jiang et al., 2023). To investigate the efficacy of hybrid sampling strategy in addressing dataset imbalance in the context of alignment objectives, we integrate our datasets across six alignment dimensions with distribution ratio of 10 10 10 10 : 1, where the 1 1 ratio pertains specifically to specificity and tone. The results are illustrated in Fig. 5.
Average
1.0
#1
0.2
#2
Fine-tuning specific dimensions will boost the performance on its corresponding alignment dimensions but fail to achieve desirable perfor- mance on other alignment dimensions, such as Single # 1(Accuracy), and Single # 5 (Tone). We demonstrate that the All Dimensions Equal strategy, with a uniform distribution ratio of 1:1:1:1:1:1, under-performs relative to our hy- brid sampling approach across all dimensions, achieving only 70% to 95% of the performance of the Hybrid sampler. This implies that this uniform sampling strategy, also employed by Cui et al. (2023), may fall short in imbalanced datasets. The All Mixed strategy, integrating all alignment datasets ignoring the data bal- ance, exhibits significantly superior performance in rell-represented alignment datasets # 1 and # 2 (Depth and Accuracy), yet fails in alignment objectives with limited datasets: # 5 and # 6 (Tone and Specificity). of resser-represented alignment objectives, particularly in scenarios involving competing alignment objectives.
Single #
All Mixed Hybrid Sampler
Single # 5
All Dimensions Equal
Figure 5: Performance with different sam- pling strategies on imbalanced datasets.
an approach could further diminish the performance
B Hummer Details
B.1
Data Construction Prompt and Annotation
In this section, we detail the construction process of Hummer, starting from the initial data formulation. Utilizing the original dataset, we format it in the pattern {x, (1, 11, y2, V3, )4), where x serves as the prompt and each y; represents a candidate generated by the model. To create rich dataset for pairwise comparison, we pair the candidates, resulting in new set of sample pairs (1,
Following this, we select subset of 400 pairs from this collection through random sampling. These selected pairs are then formatted into standard prompts, structured to fed into GPT-4 for evaluation. In executing these queries, our objective is to discern the superiority between Y/1 and y/2 within each pair, focusing on identifying which candidate better aligns with specific predefined objective. Additionally, for each comparison, aim to gather concise explanation highlighting why one candidate is favored over the other, based on the alignment with the mentioned objective. Through this meticulous process, we identified diverse set of 37 different objective names.
('accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity')
Subsequently, we integrate the previously identified 400 superior alignment objectives, replete with their concise explanations, into the new prompt design for GPT-4 as part of our second approach in prompt engineering. This step instructs GPT-4 to assimilate the
15

---

Published as a conference paper at COLM 2024
Prompt for identifying multiple objectives and definitions to reduce competing.
Following is a pair-wise RM training data item with the structure ('prompt'([promtt) 'candidate-1' Iianddddtt (endiddtt-2(|(cndddatt-]]]
The 'prompt' stands for cnvestion/situationn in which one agent is asked answer; the 'candidate-1" and 'candidate-2" are two responses from agents. One response is better than the other.
Your task is to give a brief assessment about which response is better and in which quality did so. Your output should have following json format: the quality name], mraaee//rraonnss rewpoooenrrppooeee-G
response-2) is bet- ter because [reason], cchoseo': :[0 for response-1 better and 1 for response-2 better]). Remind the 'reason' part should contain no more than 40 words. Here is the item case:
Prompt for refining independent dimensions definitions and ap- proaches from summarized alignment Features.
You will receive a series of example entries formatted to: ("quality": "aspect-name", "reason": "Response-1 (or Response-2) is better because [reason]]".
Please understand the meaning of each entry in conjunction with the 'quality' and analyze the differences and connections between them.
Finally, summarize all the 'qualities' and refine them by only retaining the 'qualities' that are semantically independent and have as little feature overlap as possible, and provide the reasons for doing so. Your output should follow this format: { ("single- quality": "asppct-name", "reason": "because [reason]"].
Here is the list of example entries:
given information and differentiate between objectives, combining similar ones to eliminate redundancies, and then distill these into defined set of distinct objectives. The anticipated outcome is final set of consolidated objective names and corresponding definitions.
The sampling strategy employed in the aforementioned stages functions heuristic aid, steering us towards dimensionality where conflicts are minimized. Empirically, this selective approach enabled to pinpoint ten distinct dimensions.
In the concluding procedure, we categorize the entirety of the dataset into these ten align- ment objectives following the structure specified by the third prompt example. Our initial method used a singular query to present all objectives' definitions to GPT-4, subsequently prompting it to discern the most suitable alignment objective for each data entry. Unfor- tunately, this methodology yielded suboptimal performance due to positional bias, where objectives presented earlier were disproportionately selected over subsequent ones. The variability of results with different objective orders further indicated a lack of stability in this initial approach.
To address the limitations observed with the initial approach, we transition to two-stage reward-ranking classification methodology. In the first stage, we present each alignment objective distinctly, pairing them with the samples for evaluation by GPT-4. Our request for GPT-4 includes assessing and assigning a reward to both 1/1 and 1/2 based on how well they meet the given objectives and calculating the difference between these rewards, termed the 'reward gap'. Subsequently, we compile a list of these reward gaps for each sample across the various objectives and rank them in order of magnitude. The logic underpinning this sorting
16

---

Published as a conference paper at COLM 2024
Prompt for final dataset splitting with objectives.
Following a pair-wise RM training data item with the structure 'chosen' [[chosen output], [reeeted output].
The 'prompt' stands for a question one agent is asked to answer and the 'chosen' and 'rjecced'' are two responses from the above agent. Your task is to assess both of them and give reward (float, 5.0 for best and 0.0 for worst) in the dimension of Depth with the definition "the thoroughness of analysis or explanation, providing detailed insights into subject", for 'chosen' and 'rejected' responses(Each response one score). Then compute the gap between the two rewards ('chosen' reward 'rejected' reward). Finally only output the reward gap. Here is the item case:
Table 4: Frequencies of Samples Aligned to Alignment Objectives under 2-stage Classifica- tion Method.
ID Dimension
2
3
4
accuracy
conciseness
depth
empathy
tone
specificity
Frequency
15523
5078
9406
4011
2865
9317
is straightforward: a larger reward gap signifies clear preference for one candidate over the other, primarily grounded in the specific objective, thereby determining the ultimate classification for data segregation. This iterative refinement led to the crystallization of distinct alignment objectives, each defined succinctly and accompanied by the frequency of dataset samples correlating with them.
An intriguing observation emerged during this process: a notable fraction of samples (11.2%, to be precise) displayed nearly identical or very closely matched reward gaps for two or more objectives. Our strategy to address these ambiguities varies depending on the dataset context. For the standard dataset, these samples are randomly allocated to one of the objectives sharing the highest reward gap, aiming to preserve the integrity and balance of the dataset. Conversely, in the fine-grand grained dataset, we opt for exclusion, removing these samples outright to maintain the precision and reliability of our objective classifications.
1. Accuracy refers to the adherence to factual correctness, ensuring that information is free from errors.
2. Conciseness refers to the ability to convey information with brevity, using a minimal number of words without sacrificing clarity.
3. Depth refers to the thoroughness of analysis or explanation, providing detailed insights into a subject.
4. Empathy refers to the capacity to understand and share the feelings of others, reflecting compassion in communication.
5. Tone refers to the author's 's attitude mood conveyed through language, influenc- ing the reader's perception.
6. Specificity refers to the provision of precise and detailed information, avoiding generalizations or vagueness.
B.2 Case Study
Following Tab. 5 and Tab. 6 we provide some representative cases from Hummer dataset, each for one alignment objective, ranging from three lines to tens of lines. The format follows "prompt", "chosen" and "rejected" (pair-wise) versions.
17

---

Published as a conference paper at COLM 2024
B.3
Hummer-Z FF
While the prevailing belief advocates for the benefits of increasing dataset size to enhance the performance of language models, we emphasize the critical role of data quality over quantity in developing preference datasets for effective preference modeling. Enlarging the dataset may o inadvertently incorporate noisy preference pairs, potentially diluting the integration of human values into the reward model (Siththaranjan et al., 2023; Wang et al., 2024a). In response, Wang et al. (2024a) undertook comprehensive experiments that underscored this phenomenon, proposing label smoothing and additive margin as algorithmic innovations to refine the preference model.
In light of these findings, our approach in developing Hummer involves a meticulous two- stage filtering process, resulting the creation of Hummer-F, fine-grained variant distilled
Table 5: Representative Cases for Depth, Tone and Specificity Objectives in Hummer.
Number of Samples
Case of objectives
Objective
prompt": "What are the key factors that influence the stock market?", "chosen"""First, economic indicators, such as GDP growth and inflation rates, set the stage for market expectations by. reflecting the overall health of the economy. Second, corporate earnings reports can significantly impact stock prices, they provide insights into the profitability and future prospects of individual companies.
Depth
2855
Last, geopolitical events, including trade wars and policy changes, can create uncertainty and affect investor sentiment, leading to market volatility.", "reeected":"" "he stock market is affected by economic data, company profits,
and world events, but the specific details of these factors are not elaborated upon." "
prompt": What are the potential consequences of not recycling?
"chosen":"" NNt recycling can lead to increased waste in landfills, which contributes to environmental pollution and the depletion of na natural resources. It's important that we consider the long-term effects on our planet and strive to reduce, reuse, and recycle whenever possible. "rejected": "If you don't 't recycle, you' 're practically destroying the Earth! It's people like you who are causing irreversible damage to our environment. Wake up and start doing your part, or else!"
Tone
872
"prmmpt""" "I'm trying to improve my fitness and lose weight. What kind of exercise routine should I follow?",
"chosen":
"For targeted approach to fitness and weight loss, commit to routine that includes jogging or brisk walking for 20 minutes every morning, three times a week, and swimming or cycling for 30 minutes on the alternate days.
On weekends, dedicate 30 minutes to strength training, focusing on full-body workouts with exercises like squats, lunges, push-ups, and planks. ", "rejected": ""To enhance your fitness and promote weight loss, consider integrating a mix of cardio and strength training into your weekly schedule. Aim to stay active for at least 30 minutes day, five days week, and include healthy eating habits as part your lifestyle."
Specificity 2635
18

---

Published as a conference paper at COLM 2024
Table 6: Representative Cases for Accuracy, Conciseness and Empathy objectives in Hummer.
Objective
Number of Samples
Case of objectives
prompp": "What the capital of France?
Accuracy
4721
"chosen": : "The capital France is Paris.
"rejected": "The capital of France is Lyon."
prompt":
"What is the primary function of
photosynthesis in plants?
chosen":
"Photosynthesis converts sunlight, water, and carbon dioxide into glucose and oxygen, providing energy for plant growth. "rejected":
"The quintessential process of photosynthesis, veritable symphony of nature's alchemy, transmutes the radiant energy of the sun, the life-sustaining elixir of water, and the invisible breath of carbon dioxide into the sweet nectar of glucose, the very essence of verdant vitality, and the ethereal gift of oxygen, all in a dance of molecular elegance that is the cornerstone of botanical sustenance and the tapestry of life itself." prompt":
Conciseness
1544
"I'm feeling really overwhelmed with my work and personal life. 'mm struggling to find balance.
What advice would you have for me? chosen":
"I understand that you're re going through tough time, and it's completely natural feel overwhelmed. It's hard when you're juggling many responsibilities, and it can be tough to find that balance. May I suggest taking few moments each day to focus on your breath and practice mindfulness? This can help create a sense of calm and provide a fresh perspective. Also, remember that it's okay to say no sometimes and protect your personal time. "rejected":
Empathy
1376
"You're not alone in feeling overwhelmed. It's a common problem in today's fast-paced world. Here are some practical tips: prioritize tasks, set boundaries, and consider delegating asking for help when necessary. Remember, it's important to take breaks and engage in activities that you enjoy to recharge."
from approximately 46% of the original dataset. The initial stage utilizes scores from raw paired data sourced from UltraFeedback (Cui et al., 2023), implementing a threshold (Ti 4.0) on the summed score gap for initial data cleansing. This procedure effectively reduces the dataset from No 100k preference pairs to N₁ 46k. Subsequently, we introduce second threshold (I2 = 0.5) specifically within the pairwise preference datasets of Hummer, aiming to isolate and remove potentially noisy data based on reward signals derived from the concluding phase of Hummer's assembly. This strategy further refines the dataset to N2 = 37k preference pairs. Our experimental results affirm that this meticulous dataset curation markedly enhances testing accuracy. Although the current filtering process relies on heuristic methods, future iterations could benefit from an implementation grounded reward modeling approach.
19

---

Published as a conference paper at COLM 2024
Table 7: The details alignment dimensions for preference datasets.
Datasets
# 1
# 2
# 3
# 4
#
#
Anthropic HH helpfulness
harmlessness inslriction-folloiing
Ultraeedback
helpfulness accuracy
honesty depth
truthfulness
Hummer
conciseness
empathy
tone
specificity
C Experiments Details
This section delineates the experimental apparatus employed in study. Our computa- tional setup comprised a quad-cluster of NVIDIA A100 GPUs, each furnished with 100GB of memory, providing robust computational capacity. This infrastructure was driven by software stack anchored by Python 3.8. In the realm of deep learning libraries, we harnessed the capabilities of PyTorch version 2.0.1. Allied with Pylorch, we utilized torchvision ver- sion c33ll1+ and torchaudio version transformations, respectively. Additionally, scikit-learn version 1.0.1 served as our machine learning toolkit, offering versatile assortment of algorithms for data mining and analysis.
to manage image and audio data
To expedite the training process, we integrated the flashattention library version 1.0.0, specifically optimized to harness the A100's computing prowess effectively. This library was instrumental in reducing the computing overhead significantly, thus accelerating training times for our models.
Below, we expand on the specifics of our experimental methodologies, ensuring that we shed light on each significant aspect that could possibly influence the replicability and interpretation of our research findings.
C.1 Datasets evaluation
We initiate our experimentation by training an encompassing model on the Hummer dataset utilizing the LLaMA2-7B architecture, extending over my 24000 training steps. To assess its performance, we deploy the model to RewardBench, yielding an evaluative score.
To gauge and juxtapose the Average Deviation Coefficient (ADC) across varying datasets, we embark on a fine-tuning regimen. This phase commences with models that have undergone warm-up phase of training, aligned with different specified objectives. These fine-tuned models, including the initially warmed-up model, undergo individual assessments against the corresponding evaluation sets of each dataset. The objective is to discern the adjustment in prediction accuracy specifically on the RM dataset. We normalize the observed changes to derive the relative variation and, leveraging the ADC previously defined, calculate the precise value through the established formula.
To mitigate the potential biases introduced by the model architecture in evaluating datasets, we standardize the use of the Llama2-7B model as our foundational model for all datasets undergoing evaluation. This standardized approach includes an initial phase of training amounting to kg = 1000 steps, covering the entirety of the source dataset-a conglomerate reflecting the diverse spectrum of the target evaluation dataset. This foundational model subsequently anchors the further fine-tuning training sessions and comparative performance analyses.
In the advanced fine-tuning phase, we meticulously sample from each subset within the evaluation dataset, catering to distinct alignment objectives. This step involves engaging in reward model training over M 4000 steps, rooted in the preliminarily trained base model. For those datasets facing a data scarcity, we incorporate a multi-epoch replay and reuse strategy. This method is pivotal in circumventing the undue repetition of data samples, thereby minimizing the risk of overfitting and maintaining the model's generalization capabilities.
20

---

Published as a conference paper at COLM 2024
C.1.1 ADC calculation
For the computational demands of our experiment, each further fine-tuning phase of our model on the Hummer dataset, leveraging the LLaMA2-7B framework, required approxi- mately 6 hours of dedicated processing time using four NVIDIA A100 GPUs. Our ADC evaluation involves the following three key steps:
(a) Single evaluation strategy: During the evaluation stage, we strategically sample 1,000 instances from each test set corresponding to the distinct alignment objectives integrated within our target evaluation dataset. This sampling aims to rigorously assess the prediction accuracy of our fine-tuned models. Adopting standard reward model evaluation approach, we analyze the competency of the reward model by presenting two candidate responses to a given prompt. The evaluation criteria are straightforward: if the candidate response marked as " "chosen" " garners a higher score compared to its counterpart across the sampled data, the model's prediction for that instance is deemed accurate; conversely, it's labeled inaccurate. The precision of the model, thus, is quantified as the percentage of instances correctly evaluated as accurate.
(b) Evaluate further tuning: Upon refining the new model via further fine-tuning on an alignment objective from the base model, we meticulously evaluate the impact of this fine- tuning on relative accuracy across all objectives delineated in the dataset. In analyzing the outcomes, our focus narrows to the adverse effects specifically, the reduction or negative impact that further tuning dedicated to one objective might have on the performance across other objectives. This analysis is operationalized by computing the squared mean of these reductions.
(c) Compute ADC value: Concluding this multi-faceted evaluation, we compute the ex- pectation across each specified objective within the target evaluation dataset. This com- putational step culminates the derivation of the Average Deviation Coefficient (ADC) result, effectively encapsulating the nuanced dynamics our definition intended to capture. This ADC measurement serves as a nuanced indicator, reflecting the model's balanced performance across spectrum of alignment objectives, shedding light on the intricate trade-offs that underlie fine-tuning processes in deep learning model optimization.
C.1.2 ADC-B calculation
Formally, the performance of given reward model after fine-tuning on its preference dataset D is denoted as V (2, Um,, where m indicates the total dimensions of abilities for assessment, e.g,, Reasoning ability in RewardBench. With further fine-fune of the reward model on one specific dimension d; DE new evaluated performance and benchmark performance deviation are defined as W, {ir1 "22.. , ji and W, W, respectively. We then can evaluate the ADC of datasets with a structured com comparison on standard benchmarks:
Definition (Alignment Dimension Conflict Benchmark). The Alignment Dimension Con- flict (ADC) exlended to standard benchmark evaluation is the second-order moment of negative performance deviation on all evaluation dimensions in the benchmark:
((Ti) vi,)) with v- = min(v,0),
D R
-
E,
(5)
m
C.2 Hybrid Sampler
To rigorously evaluate our novel hybrid sampler methodology against the conventional fixed-ratio mixture sampling techníque, we undertake comparative training experiments using the same dataset. In this case, we exemplify the process with the fine-grained version of the Hummer dataset. We standardize the foundation of our comparative analysis by utilizing the Llama2-7B base model, maintaining a consistent training duration of N 2000 steps across all experimental trials. Post-training, performance on various objectives' evaluation sets within the Hummer dataset. The findings related to relative accuracy are illustrated in Fig. 5.
assess the resulting reward model's
21

---

Published as a conference paper at COLM 2024
Parameters settingArticulatinng the specifics of the hybrid sampler configuration, we es- tablish the following parameters: each objective weight, A, is set at the uniform value of 1/6, corresponding to equal division of focus across all objectives. The adherence thresh- old, threshold; is set to 0.80, indicative of our criterion for sample selection consistency. Moreover, the learning rate (denoted Ir) for the À values is calibrated at 1e -4. 4. These weights, A, subsequently inform the proportional sampling across the respective datasets, such that the ideal number of samples from dataset i in single batch would approximate to BatchSize x A;-
Handling sampling size not integer.Addressin scenarios when the calculated sampling size for specific objectives does not yield an integer, we initially resort to the floor function, expressing this as SampleSize; = [BatchSize x A]. Post-computation, we then determine the remaining sampling capacity, described as BatchSize I SampleSizey. The ensuing step entails random sampling for the objectives that correlate with this remaining budget, relying on A as the probability factor. This tailored approach aims to uphold the integrity of equitable consideration for each alignment objective, meticulously adhering to the pre- set guidance of A,. Such stringent adherence seeks to ensure the sampler's 's fairness and objectivity across the landscape of alignment objectives within the dataset.
Result analysisThe The radar chart reveals notable findings regarding the performance of the hybrid sampling methodology within a fixed training-step regime. Specifically, the hybrid sampler's 's performance closely matches the precision gains seen when training objectives independently (showing a difference of less than 5.6%) for accuracy and conciseness objec- tives. Additionally, this approach yields a higher precision improvement rate (by roughly 4.3%) than that of the fixed-ratio 1:1 mixture sampling method for the same objectives. When juxtaposed against the equal-ratio 1:1:1:1:1:1 mixture sampling strategy spanning all six objectives of the dataset, the hybrid sampler shows an even more marked enhance- ment, outstripping the uniform mixture method by over 10%. Significantly, the hybrid sampling approach also surpasses strategies that forgo additional fine-tuning for the re- maining four objectives. The rationale behind these outcomes can be intuitively understood when considering how objectives, which are not specifically bolstered by increased sample counts the same FLOPs (Floating Point Operations Per Second) can still be affected to different extents, as indicated by the dataset's s ADC levels. Some objectives might lag in improvement when provided with the same or smaller sample distribution proportions. The hybrid sampler intelligently adjusts for this by diminishing the sampling proportions of objectives that have already attained a satisfactory level of accuracy enhancement. This reallocation tactic beneficially channels a greater share of the training proportion towards those objectives that show slower gains. Consequently, this method maximizes training efficiency, enabling more substantial improvements under constant computational budget.
22


Reviews:
[{"review_id": "gIspSqG1Hw", "reviewer": "Reviewer_9qZi", "summary": "The paper makes multiple contributions. First a metric to measure conflicts alignment dimensions in preference datasets is presented (ADC/ADC-B). Second, seeded from UltraFeedback, a preference dataset with low conflicts along 6 alignment dimensions is collected and automatically annotated with GPT-4. The usefulness of the collected dataset is demonstrated in experiments. Third, they introduce a hybrid sampling strategy for balancing alignment dimensions in preference datasets when learning a Reward Model.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our contribution, especially the significance of the ADC in measuring the conflict of preference datasets. We have updated a new version of [manscript](https://anonymous.4open.science/r/COLM2024_587_hummer-75AD/) regarding all following concerns.\n\n**Response to Q1** (*clarify for $u _{-}^2$*): Yes. $u _{-} = \\min$\\{$u, 0$\\} operator has higher precedence than the square operator. Otherwise, ADC/ADC-B $\\equiv 0$ as $u^2 \\geq 0$.\n\n**Response to Q2** (*ADC-B can only be applied to two (or more) datasets with data on the same alignment dimension?*): We clarify that ADC-B is comparable with different preference datasets within different alignment dimensions. The critical distinction lies in the \"dataset\" for evaluation $n$ further-fined reward models on **$n$ alignment dimension of the test part of preference datasets (Hummer) VS. external dataset with $m$ ability dimensions (RewardBench)**. After that, for both ADC and ADC-B, it results in $n$ average negative performance deviations. The final ADC-B metric is holistic, calculated as the average of these $n$ deviations from $n$ further fine-tuned reward models. \n\n\n**Response to Q3** (*Figure 4 with further fine-tuning*): We sincerely thank Reviewer 9qZi for pointing out this important typo in our work! The corssponding alignment dimensions in Hummer is \\{'accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity'\\}, and the 'tone' dimension in our prior draft is actually 'specificity' (#5) in our new updated version, detailed in Appendix Tab. 6. Fine-tuning on 'specificity' (#6) will exhibit higher conflict to other alignment dimensions, e.g., 'Accuracy' (#1) focusing on free-of-error, 'Empathy' related to safety. However, we observe a contradictory phenomenon: further fine-tuning on 'specificity' (#6) results in a negative performance deviation across all other dimensions; conversely, it reduces jailbreak rates by approximately 5%. We hypothesize that the datasets designed for attacks may contain a fraction that emphasizes 'specificity' (#6) and we plan to explore this in our future work. \n\n\n**Response to Typos/Minor Comments**: Thanks for raising these concerns and please see our updated [manscript](https://anonymous.4open.science/r/COLM2024_587_hummer-75AD/).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper makes multiple, valuable contributions (detailed in the summary above).\n- The ADC metric seems particularly useful to inform future data collections for preference datasets.", "reasons_to_reject": "The paper is often quite dense, and details are only explained in the appendix. While I acknowledge that there are many contributions, still I think it would improve the paper to at least know major experiment choices/settings, e.g. which model has been used for tuning or which alignment dimensions are comprised in Hummer.", "questions_to_authors": "- Can you clarify what the “u_ = min{u, 0}” operation does? Does it mean: if \\bar{u}_k - u_k > 0 -> 0 else \\bar{u}_k - u_k? And I am assuming this operation is applied before the square?\n- Just to clarify, ADC-B can only be applied to two (or more) datasets with data on the same alignment dimension, right?\n- From Figure 4 it seems that further fine-tuning on ‘tone’ increases conflicts along the dimensions. Why do you think that is? Intutively, tone should not alter the semantics, so at least accuracy, consicneness and depth should stay somewhat equal?\n\nTypos/Minor Comments:\n\n- The table captions should be after the table.\n- Page 3: Double semi-colon “Wang et al., 2023;; Ethayarajh et al., 2024”\n- “table” Seems like table references are all lower case, but Figure references are upper case. Consider aligning.\n- Section 4.2: I think this section is very hard to understand without Appendix B. Maybe it would be better to not only describe things abstractly, but more concretly e.g. by naming the alignment dimension.\n- Table 2: HmmerRM → HummerRM\n- Page 16: I guess there should be commas between the y’s ? “{x, y1, y2.y3.y4}” → “{x, y1, y2,y3,y4}”", "extra_scores": {}}, {"review_id": "5KYdk6wKOV", "reviewer": "Reviewer_z9ad", "summary": "The paper studies how to handle conflicting alignment targets (e.g., helpfulness and harmlessness) in LLM alignment. \n* A metric is first introduced to measure to what extent different alignment targets conflict with each other. With this metric, the authors analyzed and compared the conflicting degree of some existing alignment dataset (Anthropic HH and UltraFeedback). \n* Secondly, the authors use GPT-4 to re-label selected examples from the UltraFeedback dataset, categorizing them into six new alignment objectives. The resulting new dataset is called Hummer, and a filtered version of Hummer is also proposed named Hummer-F. \n* Thirdly, the authors introduced a dynamic sampling method, which dynamically decides the proportion of samples from different alignment objectives, aiming to minimize the performance gap among the alignment objectives. \n\nEmpirical results show that the newly proposed datasets (Hummer and Hummer-F) have substantially lower conflicting levels between their alignment objectives. Applying the dynamic sampling method on the new datasets yield much more balanced results in the jailbreak tests.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Grateful to Reviewer Z9ad for expressing interest in the paper and constructive comments. We have updated a new version of [manscript](https://anonymous.4open.science/r/COLM2024_587_hummer-75AD/) for all concerns.\n\n**Response to Reject Reason 1** (*Notations can be significantly improved*): We briefly clarify the notation one by one: \n1. **$D _n^p=$\\{$d _1, \\ldots, d _n$\\}**: Yes, each $d _i$ is actually a set of tuples, for the specific alignment dimension preference dataset, such as (such as Anthropic HH with 2 dimensions: Helpfulness and Harmlessness). \n2. **the expectation operator operates in ADC**: This expectation is over alignment dimension $(d _{i\\in n})$. Say, we firstly obtain $n$ negative performance deviations via $(\\overline{\\mathbb{U}} _i - \\mathbb{U})$ for all alignment dimensions averaged by n-1 (except the further fine-tuned one), and we take the expectation of $n$ alignment dimensions to obtain a scalar metric with $\\mathbb{E} _i[\\cdot]=(\\sum _{i=1}^n[\\cdot])/n$.\n3. **Priority of operators**: $u _{-} = \\min${$u, 0$} operator has a higher precedence. Otherwise, ADC/ADC-B $\\equiv 0$ as $u^2 \\geq 0$. \n4. **Relation between $m$ and $n$**: We clarify that the number of $m$ and $n$ are indepedent. $m$ refers to an external evaluation dataset that encompasses $m$ ability dimensions; while $n$ indicates the total alignment dimension in the training preference dataset. However, we note that certain dimensions in the preference dataset may significantly overlap with abilities in Benchmark.\n\n**Response to Reject Reason 2** (*Ablations study.*): We have conducted the ablation on our proposed hybrid sampling (HS) strategy. We have three main takeaways: (1) For the mitigation of inner conflict (ADC) and jailbreak, the low-conflict dataset is the main contributor, while an observable extra margin with HS. (2) For the improvement of RewardBench, HS is the primary contributor. (3) Data quality is important for further improvements. (Tabb 1,2).  \n\nWe also provide a plausible explanation for safety compromises with further fine-tuning on policies: the emphasis on certain alignment dimensions and conflicts among them ([Qi et al, ICLR (2024)](https://arxiv.org/abs/2310.03693)) (Section 2). We are expanding our jailbreak experiments with further policy fine-tuning to reinforce our hypothesis. We plan to incorporate these results in future versions and, hopefully, share them by the end of the rebuttal, given the constraints on computational resources.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* LLM alignment is a highly important task in LLM development, and how to trade off between conflicting alignment objectives is a major problem faced in LLM alignment. \n* A new metric is proposed to measure the conflicting degrees of alignment objectives. This metric allows the community to better understand the relations between the alignment objectives.\n* New datasets and sampling methods are proposed to balance the Reward Models’ performance across different alignment objectives. Empirical studies show promising results.", "reasons_to_reject": "* Notations and mathematical definitions can be significantly improved. The main definitions of the metric, presented in Section 4.1, is quite confusing and hard to follow. For example:\n  * $D_n^p = {d_1, …, d_n}$, but it’s unclear what is n and what is each element d_1. Intuitively, readers may expect each d_i is a tuple (prompt, winning response, losing response). However, from later descriptions, it seems that each d_i is actually a set of tuples, and hence $D_n^p$ is a set of sets. Also, what is $n$ is not described in Section 4.1? Until very late it became clear that $n$ is the number of alignment objectives.\n  * In Definition 1 (Alignment Dimension Conflict), it is unclear how the expectation is defined: what is the set of elements the expectation operator operates over, and what is the probability of each element? Also for the numerator  $(\\bar{u}_k - u_k)^2 _{-}$, it is unclear the square operator and the minus operator which has a a higher precedence: if the minus operator has a higher precedence, then it means $(\\bar{u}_k - u_k) _{-} * (\\bar{u}_k - u_k) _{-}$; however, if the square operator has a higher precedence, it means $[(\\bar{u}_k - u_k)^2] _{-}$. \n  * When defining ADC-B, a new subscript $m4 is introduced, but it’s unclear what is the relation between $m$ and $n$: are they completely independent, or are they the same? \n  * All the above unclarities hinder the reader to fully understand the newly proposed metric.\n* Ablations are needed. Promising results are presented in Table 1 and 2, but it is unclear which contributes more to the improvements, the new dataset or the new dynamic sampling method. It would be great if the authors can present two results for each dataset, one with the naive sampling strategy, and one with the new dynamic sampling strategy. It would be even better to use other sampling/integration strategies (discussed in Section 1, the last paragraph in Page 1) on each dataset, and compare the performance with the newly proposed sampling strategy. \n\nOverall, I find the paper very interesting and it studies a highly important topic in LLM alignment, but the current paper lacks clarity and should be significantly improved before being published. I may recommend a ‘conditional acceptance’, requesting the authors to substantially improve the definitions before finally accepting the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dwtgc89gIq", "reviewer": "Reviewer_gJMS", "summary": "In this paper, the authors introduce the concept of Alignment Dimension Conflict (ADC) as a metric to quantify conflicts within preference datasets, particularly in the context of RLHF. They present the Hummer to measure conflicting alignment objectives in preference datasets, showcasing reduced ADC compared to other datasets. The authors further train reward models, HummerRM and HummerRM-F, using a hybrid sampling approach to effectively balance diverse alignment objectives. The analysis reveals a negative correlation between ADC and the number of alignment objectives, suggesting that incorporating more fine-grained alignment metrics could help mitigate conflicts to some extent.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work and highlighting the critical issues of generalization over various dimensions of the proposed dataset! We provide the detailed responses in the following:\n\n**Response to Reject Reason 1** (*lacks a thorough discussion on the generalization of various dimensions*): We respectfully argue that this is not feasible at the current stage due to the following reasons:\n1. This requires a significant computational budget, and the evaluation pipeline is considerably long and time-consuming.\n2. Prior work, such as Anthropic HH and UltraFeedback, designs the alignment dimensions through heuristics.\n3. We acknowledge that there may be better approaches to constructing the desired limited-competitive preference dataset over dimensions through unsupervised learning or other methods.\n\n**Response to Reject Reason 1.2** (*the paper would also benefit from a more detailed test of the weak-to-strong generalization in alignment.*): We respectfully believe that there may be some misunderstandings regarding the relationship of this work to weak-to-strong generalization in alignment. Notably, weak-to-strong generalization [1] typically introduces a novel methodology for RLHF to improve the benchmark performance. Conversely, our work aims to identify and measure the competing dynamics inherent in preference datasets. However, we acknowledge the possibility of misunderstanding the question posed. If Reviewer gJMS is interested in specific experimental designs related to weak-to-strong generalization that could further enhance our work, we would be pleased to know. We are willing to undertake additional experiments within the constraints of our computational budget.\n\n[1] Burns, Collin, et al. \"Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.\" arXiv preprint arXiv:2312.09390 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. One strength of the paper lies in its detailed construction of the Hummer dataset, where pair-wise data samples are systematically generated and assessed to identify specific alignment objectives. \n\n2. The two-stage reward-ranking classification method used in this paper allows for a comprehensive evaluation of alignment objectives, leading to the identification of key dimensions with minimal conflict. \n\n3. Additionally, the paper provides a structured approach to refining independent dimensions and aligning them with human preference.", "reasons_to_reject": "Regarding the weakness: this work lacks a thorough discussion on the generalization of various dimensions of human values within the dataset construction process. To improve, 1) the authors could enhance the paper by elaborating on how the alignment objectives identified in the Hummer dataset that can be applied to a wider range of situations / datasets. This might be useful for  evaluating the robustness and utility of the dataset. 2) the paper would also benefit from a more detailed test of the weak-to-strong generalization in alignment. Such an analysis could provide a clearer understanding of how well the proposed methods perform when extended beyond their original setups.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KGY3XTd0tE", "reviewer": "Reviewer_6Tue", "summary": "This paper tackles the problem of balancing between different, potentially conflicting alignment objectives in preference datasets (e.g. helpfulness vs. harmlessness). The paper first introduces a metric (ADC) that measures the RM performance deviation on other datasets when fine-tuning on data for a single dataset (each dataset presumably corresponding to a different alignment objective). Then the paper proposes a sampling strategy to get a more balanced RM dataset (Hummer) w.r.t. to the different dimensions. The authors show that the new dataset achieves better ADC scores.\n\nThe main issue with the paper is that it’s not written clearly enough so that it would be easy to understand what exactly is the problem it’s trying to solve (beyond the high-level idea of balancing between conflicting alignment objectives). Some examples of unclear writing below:\n* Please give concrete examples of what you mean by “alignment dimension” already in the abstract/introduction.\n* page 2: Redundant sentences one after another: “Despite these advancements, there remains a notable gap: the absence of a preference dataset specifically designed to mitigate the conflict between alignment dimensions. [...]  Despite these efforts, a significant gap persists: the lack of a preference dataset intentionally crafted to alleviate conflicts between alignment dimensions.” \n* page 6: “We categorize every pair of datasets” Do you mean “every pair of responses”?\n\nMoreover, it’s unclear how useful e.g. the proposed dataset would be in practice beyond helping to optimize the ADC score proposed in the paper.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work! However, we respectfully believe that the essence and contributions of our work have been significantly misunderstood. \n\n**Response to Reject 1** (*paper writing*): Note that alignment dimensions $d _i=${$x^k, y _w^k, y _l^k$}$ _{k=1}^{K _{i}}$ refer to the sub-set of the preference dataset: such as \"Harmless\" highlighting safety in Anthropic HH. For others, please see the new [manuscript](https://anonymous.4open.science/r/COLM2024_587_hummer-75AD). \n\n**Response to Reject 2** (*The practical usefulness of Hummer*): We respectfully argue that the contributions of our work are:\n1. We are the first to quantify the conflict in preference datasets and introduce a new statistical metric.\n4. We introduce a new preference dataset to mitigate the conflict.\n5. We develop a hybrid sampling strategy to train reward models to improve interested metrics.\n\nThe practical usefulness of Hummer includes (even without hybrid sampling):\n\n1. **Improved performance on RewardBench**: We observe 2% improvement on RewardBench over UltraFeedback, despite Hummer originating from UltraFeedback. (Table 1).\n2. **Insights into dataset quality**: we emphasize the importance of data quality in further improvements in ADC and RewardBench with Hummer-F (Table 1).\n3. **Boosts defense against jailbreak attacks**: With further fine-tuning, beyond low ADC, we observe that HummerRM exhibits strong robustness against jailbreaks (Table 2).\n\nAlso, in related work, we provided a plausible explanation for safety compromises during further fine-tuning of policies in the new version: *the emphasis on certain alignment dimensions and conflicts among them ([Qi et al, ICLR (2024)](https://arxiv.org/abs/2310.03693))*. We are expanding our jailbreak experiments with further policy fine-tuning to reinforce our hypothesis. We plan to incorporate these results in a future version and, hopefully, share them by the end of the rebuttal, pending resource availability.\n\n**Q 1.1** (*Comparison of ADC*): Yes! it remains comparable due to the normalization (n-1 in the denominator) and expectation term, making it a \"mean\" metric independent of the dimension number.\n\n**Q 1.2** (*Why not use shared dimensions?*): Existing preference datasets and alignment dimensions are well-constructed. This work is motivated by conflicts between alignment dimensions, necessitating new dimensions and datasets. However, we recognize potential overlaps in alignment dimensions across different datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of balancing between different objectives during RLHF is of great practical importance.", "reasons_to_reject": "1. The paper is not written clearly enough to properly motivate the problem and describe the proposed solution in a way that it could be easily reproduced.\n\n2. The practical usefulness of the proposed dataset is unclear (beyond it helping to optimize the score proposed in the paper).", "questions_to_authors": "Fig. 4 suggests that the different datasets use different alignment dimensions. Are the ADC values between the datasets still comparable? Why not use shared dimensions?", "extra_scores": {}}]

Author rebuttal:

Meta-review: As all reviewers have agreed, the paper is tackling a very important problem in language modeling: conflict among different values during alignment phase. The proposed method in this paper to alleviate this problem is sound and novel, and also shows experimentally effectiveness. While the paper lacks clarify and needs improvement on its writing and some discussions on the values themselves, given the characteristics of CoLM and its audience, I recommend the acceptance of the paper.
Decision: Accept