More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness

## Abstract

Abstract The surge in Large Language Models (LLMs) development has led to improved performance on cognitive tasks as well as an urgent need to align these models with human values in order to safely exploit their power. Despite the effectiveness of preference learning algorithms like Reinforcement Learning From Human Feedback (RLHF) in aligning human preferences, their assumed improvements on model trustworthiness haven’t been thoroughly testified. Toward this end, this study investigates how models that have been aligned with general-purpose preference data on helpfulness and harmlessness perform across five trustworthiness verticals: toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. For model alignment, we focus on three widely used RLHF variants: Supervised Finetuning (SFT), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO). Through extensive empirical investigations, we discover that the improvement in trustworthiness by RLHF is far from guaranteed, and there exists a complex interplay between preference data, alignment algorithms, and specific trustworthiness aspects. Together, our results underscore the need for more nuanced approaches for model alignment. By shedding light on the intricate dynamics of these components within model alignment, we hope this research will guide the community towards developing language models that are both capable and trustworthy. The code for our experiments is available at https://github.com/aaron-jx-li/rlhf-trustworthiness .

## 1.  Introduction

The recent surge of Large Language Models (LLMs) [, , ] represents a significant advancement in the field of artificial intelligence, and with increasing sizes these models have demonstrated state-of-the-art performance on a wide array of cognitive tasks [, , , ]. To harness such unprecedentedly powerful models and make them helpful and reliable assistants for us, it’s essential to make sure their generations are closely aligned with human preferences []. One core reason behind the success of models like ChatGPT is the usage of model alignment methods such as Reinforcement Learning From Human Feedback (RLHF) [, ], which finetunes the model with crowdsourced preference data, thus creating more robust and capable systems [].

However, there is a noticeable gap in our understanding of how these popular preference learning frameworks impact the trustworthiness of the models they finetune. In our context, as proposed by [] and [], trustworthiness refers to a model’s adherence to human values across various dimensions, such as reducing toxicity [], minimizing bias [], and ensuring privacy [], some of which are not frequently represented in general-purpose preference data for RLHF. While RLHF and its variants aim to enhance the model’s alignment with human preferences, the full extent of their influence remains underexplored.

Our work seeks to fill this knowledge gap by examining the effects of applying different RLHF variants such as Supervised Finetuning (SFT), Proximal Policy Optimization (PPO) [] and Direct Preference Optimization (DPO) [] on various language model trustworthiness benchmarks. We investigate several critical aspects of trustworthiness, including the model’s performance on toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. We are especially interested in the consequences of using a general-purpose dataset for finetuning, as RLHF is known to be sensitive to its training data [, ]. Through extensive experiments, our findings reveal a complex interplay between model size, preference learning algorithm, and the specific trustworthiness vertical. The exact relationships are summarized in Figure 4.

Our work contributes to the broader discourse on model trustworthiness by providing empirical results that illustrate the impact of preference learning algorithms on LLMs. By analyzing these outcomes, we can begin to understand the intricate dynamics of these alignment methods and their implications for the development of trustworthy AI. In the following sections, we detail our methodology, present our findings, and discuss the implications for future research in the field. Through this study, we hope to guide the community towards developing more trustworthy LLMs, ensuring that these powerful tools serve the greater good in alignment with human values.

## 2.  Preliminaries

In this section, we will walk through the RLHF variants used in this study, as well as describe the target models, human preference dataset, and the preference alignment procedure used in all empirical investigations.

### 2.1 Reinforcement Learning From Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a method for fine-tuning language models to align with human preferences []. It involves supervised finetuning (SFT), reward modeling, and reinforcement learning.

The first step in RLHF is to perform supervised fine-tuning on the pretrained language model using human demonstrations. The objective function for SFT is:

$$ $\mathcal{L}_{\text{SFT}}(\phi)=-\mathbb{E}_{(x,y)\sim\mathcal{D}_{\text{demo}}}[\log\pi_{\phi}^{\text{SFT}}(y\mid x)]$ (1) $$

where $\mathcal{D}_{\text{demo}}$ is the dataset of human demonstrations, and $\pi_{\phi}^{\text{SFT}}$ is the language model with parameters $\phi$ after supervised fine-tuning.

Next, a reward model $r_{\theta}$ is trained to predict human preferences. The reward model takes the input $x$ and the model’s output $y$ as input and predicts a scalar reward value. The reward model is trained using a dataset of human preference comparisons.

Finally, the language model is optimized using the reward model as a reward function with the Proximal Policy Optimization (PPO) algorithm []. The RLHF objective function is:

$$ $\begin{split}\mathcal{L}_{\text{RLHF}}(\phi)=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\pi_{\phi}^{\text{RL}}}}[r_{\theta}(x,y)-\beta\log(\frac{\pi_{\phi}^{\text{RL}}(y\mid x)}{\pi_{\phi}^{\text{SFT}}(y\mid x)})]+\gamma\mathbb{E}_{x\sim\mathcal{D}_{\text{pretrain}}}[\log(\pi_{\phi}^{\text{RL}}(x))]\end{split}$ (2) $$

where $\mathcal{D}_{\pi_{\phi}^{\text{RL}}}$ is the dataset generated by the language model with parameters $\phi$ after reinforcement learning, $\beta$ is a hyperparameter that controls the strength of the KL divergence regularization term, and $\gamma$ is a hyperparameter that controls the strength of the language modeling term.

The first term in the RLHF objective function encourages the model to generate outputs that receive high rewards from the reward model. The second term uses KL divergence regularization to prevent the model from deviating too far from the supervised fine-tuned model. The third term is a language modeling term that ensures the model maintains its language modeling capabilities.

Optimizing the RLHF objective function aligns the language model with human preferences and intentions, improving its behavior on various tasks. The combination of supervised fine-tuning, reward modeling, and reinforcement learning in RLHF is intended for the creation of language models that are helpful, honest, and harmless.

### 2.2 Direct Preference Optimization

Direct Preference Optimization (DPO) [] is a variant of PPO-based RLHF that optimizes a language model policy $\pi_{\theta}(y|x)$ directly using preference data, transforming the preference learning problem into a policy optimization problem. The goal is to optimize $\pi_{\theta}$ to adhere to human preferences, represented as pairs of preferred ($y_{w}$) and rejected ($y_{l}$) completions for a given prompt $x$. The DPO objective function is defined as the negative log-likelihood loss of the preference data:

$$ $\displaystyle L_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=$ $\displaystyle-\mathbb{E}_{(x,y_{w},y_{l})\sim D}[\log\sigma(\beta(\log\pi_{\theta}(y_{w}|x)$ $\displaystyle-\log\pi_{\theta}(y_{l}|x)))]$ $$

The gradient of the DPO objective with respect to the policy parameters $\theta$ is:

$$ $\displaystyle\nabla_{\theta}L_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=$ $\displaystyle-\beta\cdot\mathbb{E}_{(x,y_{w},y_{l})\sim D}[\sigma(\beta\cdot(\log\pi_{\theta}(y_{l}|x)$ $\displaystyle-\log\pi_{\theta}(y_{w}|x)))\cdot(\nabla_{\theta}\log\pi_{\theta}(y_{w}|x)$ $\displaystyle-\nabla_{\theta}\log\pi_{\theta}(y_{l}|x))]$ $$

Optimizing the DPO objective using gradient-based methods trains the language model policy to align with human preferences.

## 3.  Experimental Evaluation

### 3.1 Experiment Setup

In this work, we employ the open-source Pythia Suite [] as our target models, with sizes ranging from 70M to 6.9B parameters. The choice of this suite is based on its representation across varying model sizes, consistent design choices and training procedures, as well as its popularity, which allows for a comprehensive evaluation of the impact of human preference alignment techniques across diverse model capacities.

Our human preference dataset is the Anthropic HH dataset[], adopted for its general-purpose representation of helpfulness and harmlessness preference data. This dataset is instrumental in aligning the models with these critical human values, thus providing a reliable foundation for assessing their performance across the five trustworthiness verticals.

The models are then finetuned using three popular RLHF variants, namely Supervised Finetuning (SFT), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO), all of which have been explained earlier. Each of these methods is applied following their standard procedures, with critical parameters and hyperparameters maintained at their recommended settings.

### 3.2 Toxicity

In this section, we investigate the toxicity of the language model generations [, ] before and after aligning the models to human preference. Following the similar approach in [], we use two representative subsets of the standard toxicity benchmark RealToxicityPrompts [] to evaluate the model’s elicited toxic behaviors during text completion. The first Toxic subset includes approximately 1.2k incomplete sentences that are labelled as “challenging" (i.e. easy to elicit toxic continuations), while the second Nontoxic subset contains another 1.2k sentences sampled uniformly from all nontoxic prompts. In addition to the user prompts, we also constructed two types of system prompts Benign and Adversarial (examples can be found in Appendix A), so we conduct our experiments under four different settings to see if the impact of preference learning on toxicity is sensitive to the prompts.

We are interested in the worst case performance of the language models, so we computed the Expected Maximum Toxicity per model [], which is mathematically formulated as:

$$ $EMT(\mathcal{M})=\mathbb{E}_{x_{i}\sim D}[\max_{j}{\rm Toxicity}(\mathcal{M}(x_{i})^{(j)})]$ (3) $$

where $\mathcal{M}(x_{i})^{(j)}$ refers to the j-th generation for the i-th user prompt within the dataset. For each prompt, we record 5 independent model generations with the temperature set to 0.5.

To quantify the toxicity of model generations, we utilize a widely used and commercially deployed toxicity detector (Perspective API) to compute the toxicity score for each generation, which is between 0 (nontoxic) and 1 (toxic).

The results of using benign system prompts are summarized in Figure 1 and Table 1. Empirically, we find using adversarial system prompts results in marginally higher toxicity scores, and the overall trends are similar. Furthermore, the patterns also hold for two other related metrics Mean Toxicity and Toxicity Probability, and we include all of these additional results in Appendix B.1 and B.4.

**Table 1: Expected Maximum Toxicity scores with toxic (left) and nontoxic (right) user prompts; trend is illustrated in the above plots.**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 0.302 | 0.270 | 0.075 | 0.207 |
| 160M | 0.389 | 0.396 | 0.289 | 0.264 |
| 410M | 0.460 | 0.460 | 0.479 | 0.319 |
| 2.8B | 0.480 | 0.495 | 0.618 | 0.396 |
| 6.9B | 0.506 | 0.541 | 0.597 | 0.497 |

Figure: Figure 1: Impact of different RLHF methods on toxicity, with toxic (left) and nontoxic (right) user prompts; note that the scales on y-axis are different. Only DPO is able to reduce toxicity, while SFT and PPO lead to reverse effect compared with baseline.
Refer to caption: x1.png

Based on the above results, we observe an important yet counter-intuitive trend: when the model size scales up (i.e. more than 1B parameters), among the three alignment methods, only DPO brings forth a slight detoxification effect, while both PPO and SFT lead to increased toxicity. This phenomenon is much more obvious when the user prompts are toxic: although it also exists in the nontoxic setting, the changes are negligible for human evaluators as the toxicity magnitudes are on a much smaller scale. Note that the results also seem to suggest RLHF’s capability in reducing toxicity in smaller models, but by looking at specific generated outputs we hypothesize that this effect is due to overfitting when we finetune the small models, during which they are constrained to closely mimic the RLHF preference data.

To further investigate why PPO and SFT have such a negative impact on toxicity of larger models’ outputs, we analyze the encoding and decoding process separately, searching for more mechanistic explanations. Our first analysis aims to quantify the toxic information extracted from the user prompts, and we measure it using a probe [, ], which is a shallow feed-forward neural network that takes in the model’s last hidden state upon receiving an entire prompt and classifies whether the sentence is toxic. After training the probes, we find that the classification accuracy is highly correlated with the model’s toxicity score, which verifies that larger models aligned with PPO or SFT contain more decodable toxic information. In our second investigation, we compute the log likelihood of each generated sequence, and our results validate the claim that models undergone PPO or SFT typically have much narrower, lower-entropy output distributions [], while DPO does not. With these results (see Appendix B.2 and B.3), it’s reasonable to argue that the increased toxicity is the combined effect of amplified toxic representation and narrower output distribution. Such discoveries suggest that when our alignment procedure uses a general-purpose preference dataset without nuanced model-specific customization, certain alignment methods may lead to undesired model behavior.

### 3.3 Stereotypical Bias

Stereotypical bias in language model generation often refers to the over-generalized belief about a particular group of people [, , , ], which are typically disrespectful and have negative societal impact. Since language models are trained on large corpus of real world data, they naturally capture stereotypical biases, and it is important to quantify to what extent these biases are present. To generate the user prompts, we follow the same routine as in []: we use 24 demographic groups and 16 stereotype topics, each with 3 variants to reduce the influence of the specific wording of each sentence; then we use all 1152 statements as our user prompts, and explicitly ask the model if it agrees with the biased statements. As some of our models are small, instead of looking at the generated outputs, we directly compare the model’s log probabilities of outputting Yes and No for the first generated word. Thus, the stereotypical bias of a given model $\mathcal{M}$ can be quantified by a bias score between 0 (unbiased) and 1 (biased):

$$ $Bias(\mathcal{M})=\mathbb{E}_{x_{i}\sim D}[\mathbbm{1}_{P_{\mathcal{M}}(yes|x_{i})>P_{\mathcal{M}}(no|x_{i})}]$ (4) $$

The results are summarized in Figure 2 and Table 2.

Based on the results, we find that all three alignment methods, among which DPO performs slightly better, significantly increase the stereotypical bias when the models have benign system prompts. If we switch to adversarial system prompts, when model sizes are large enough, both the baseline and aligned models have bias scores close or equal to 1. Here we postulate a plausible explanation: when RLHF uses the general-purpose human preference dataset, particularly the Helpfulness subset, makes the model more inclined to agree with claims made by the user. Our evaluation on bias reveals a subtle yet crucial discrepancy between general model alignment goals and the improvement of certain trustworthiness aspects.

Figure: Figure 2: SFT, PPO, and DPO all increase stereotypical bias compared with baseline.
Refer to caption: x2.png

**Table 2: Bias scores for benign (left) and adversarial (right) system prompts; trend is illustrated in the above plots.**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 0.000 | 0.000 | 0.119 | 0.000 |
| 160M | 0.689 | 0.741 | 0.239 | 0.720 |
| 410M | 0.138 | 0.624 | 0.734 | 0.824 |
| 2.8B | 0.253 | 0.911 | 1.000 | 0.749 |
| 6.9B | 0.375 | 0.860 | 1.000 | 0.801 |

### 3.4 Machine Ethics

In this section, we examine machine ethics, which is considered to be one of the central goals in human preference alignment [, , , ], especially for our Helpfulness and Harmlessness dataset used in this study. However, it’s important to note that being able to provide responses that seem to be ethically aligned with human values doesn’t mean the model genuinely understands the underlying moral principles, and to assess such model capability we need to evaluate whether the aligned models are able to actively detect specific actions that are against human morality.

Toward this end, we evaluate the models with the Commonsense subset from the ETHICS benchmark proposed by [], which features scenario-based commonsense moral recognition tasks. Since we are most interested in models’ ability to detect the morally wrong actions, our prompt dataset consists of 983 short samples all labeled as morally wrong from the test set. We directly ask the models whether the described action is against human morality. Using the same approach of computing log probabilities when evaluating bias, our metric for machine ethics is therefore the false positive rate:

$$ $\text{Ethics}_{FPR}(\mathcal{M})=\mathbb{E}_{x_{i}\sim D}[\mathbbm{1}_{P_{\mathcal{M}}(no|x_{i}\in X_{wrong})>P_{\mathcal{M}}(yes|x_{i}\in X_{wrong})}]$ (5) $$

According to results reported in Figure 3 and Table 3, we observe that both SFT and PPO can effectively reduce the false positive rate under both types of system prompts, while DPO fails. To find out the exact reason behind the failure of DPO-aligned models on the ethics task, as well as why it has slightly better performance compared with PPO and SFT on toxicity and bias tasks, it probably requires deeper analysis of the algorithms and more close-up investigations into the model architecture. Nevertheless, these empirical findings highlight the importance of objective alignment between RLHF and trustworthiness component.

Figure: Figure 3: SFT and PPO successfully reduce the false positive rate on commonsense moral recognition task, while DPO performs marginally worse than baseline.
Refer to caption: x3.png

**Table 3: False Positive Rates on the ethics identification task for benign (left) and adversarial (right) system prompts.**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 0.001 | 0.000 | 0.027 | 0.000 |
| 160M | 0.042 | 0.009 | 0.004 | 0.003 |
| 410M | 0.214 | 0.116 | 0.093 | 0.701 |
| 2.8B | 0.969 | 0.477 | 0.210 | 0.978 |
| 6.9B | 0.378 | 0.211 | 0.205 | 0.539 |

### 3.5 Truthfulness

Language models are known to be prone to generate hallucinated outputs that are contradictory to facts [, , ]. Although preference learning algorithms align model outputs to human preference and factual adherence is also believed to be an important consideration during human evaluation, they don’t directly optimize the models toward truthful generations. In this section, we use a popular benchmark TruthfulQA [], which consists of 817 manually crafted questions across 38 categories, to evaluate the truthfulness of the Pythia models before and after human preference alignment.

Since it often requires additional labor cost to evaluate the truthfulness of a language model’s open-ended generations, we format all our user prompts as multiple-choice questions, and ask the model to select the correct answer among several candidates (typically around five). Our truthfulness score for a model is simply its accuracy on answering $n=817$ questions.

According to the results in Figure 4(a) and Table 5(a), it’s clear that after finetuning PPO and SFT lead to noticeable loss of truthfulness, and DPO’s performance is better but still marginally below the baseline. This phenomenon is similar to the one observed during evaluation of bias, and it’s possibly due to the lack of truthfulness related preference data when we perform model alignment procedures, which causes corresponding model capability to deteriorate.

Figure: (a) SFT and PPO results in significantly lower truthfulness, and DPO is also marginally worse than baseline.
Refer to caption: x4.png

**(a) Accuracy on TruthfulQA multiple choice questions**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 4.4 | 0.0 | 0.0 | 2.9 |
| 160M | 5.9 | 0.0 | 0.0 | 5.8 |
| 410M | 8.5 | 2.6 | 3.4 | 6.6 |
| 2.8B | 36.8 | 13.8 | 10.2 | 34.1 |
| 6.9B | 37.9 | 25.6 | 24.2 | 30.5 |

### 3.6 Privacy

Our last evaluation focuses on privacy, which is another essential aspect of language model trustworthiness [, , , ]. Specifically, we focus our analysis on privacy leakage during conversations, as preference learning algorithms like RLHF are trained with human feedback consists of multiple rounds of conversations between the user and the assistant.

We use the same synthetic dataset as in [], which is constructed by 18 types of manually selected Personally Identifiable Information (PII) and specific user information which are either sampled from the standard Enron Email dataset [] or randomly generated. For each PII, we generated 100 pieces of information, which adds up to 1800 user prompts in total. Our evaluation is done in a zero-shot fashion: the user first tells the model about the target information and emphasize the privacy requirement, and then without demonstrations the model is directly prompted to extract the sensitive information it’s been told.

As indicated by Figure 4(b) and Table 5(b), we discover that privacy leakage significantly increases with the model size, and only DPO is able to improve model privacy compared with baseline. As we hypothesized earlier, preference data that encourages model helpfulness might be inherently contradictory to certain trustworthiness aspects like privacy requirement, and practitioners should be meticulous about the exact combination of dataset and alignment method during finetuning. However, the solution doesn’t have to be data-centric, as demonstrated by DPO’s success despite using the same training data as other methods.

### 3.7 Result Summary and Discussion

Here we summarize the effects of three RLHF variants on all trustworthiness aspects in Table 4. In general, it’s observed that PPO-based RLHF and SFT have similar performance across all tasks, which is expected as PPO uses the model after SFT as an “anchor point" and penalizes the divergence from SFT policies. On the other hand, DPO, which is proposed as an improvement upon typical RLHF [], exhibits slightly better performance on all tasks except machine ethics. More in-depth analysis of these algorithms are needed in order to explain such differences from the perspective of trustworthiness.

That being said, through our comprehensive results the broader message is clear: human preference alignment by RLHF doesn’t guarantee trustworthiness improvement, and sometimes it could have a reverse effect. The impact on each individual trustworthiness aspect depends on both the RLHF dataset and the specific algorithm, and such relationships are hard to generalize and complex by nature.

**Table 4: Summary chart that indicates whether an RLHF variant could improve a specific trustworthiness aspect.**
| Trustworthiness Aspect | SFT | PPO | DPO |
| --- | --- | --- | --- |
| Toxicity | ✗ | ✗ | ✓ |
| Stereotypical Bias | ✗ | ✗ | ✗ |
| Machine Ethics | ✓ | ✓ | ✗ |
| Truthfulness | ✗ | ✗ | ✗ |
| Privacy | ✗ | ✗ | ✓ |

## 4.  Limitations and Future Work

In this study, although we have identified consistent trends across different trustworthiness aspects, the empirical investigation is still limited to the Pythia Suite [], and the model sizes are up to 6.9B. To generalize our finding, it’s worth to evaluate with other open-source models with larger sizes such as the Llama series [, ].

Overall, we have discovered the important misalignment issue between the RLHF objectives and trustworthiness goals, but we still consider this work to be a preliminary investigation that raises a wide array of unanswered questions, ranging from the rigorous explanations of observed trends to detailed analysis of different RLHF variants, which we hope can be answered by future works in related fields.

## 5.  Conclusion

In conclusion, this study provides a comprehensive analysis of the impact of human preference alignment techniques, specifically three Reinforcement Learning from Human Feedback (RLHF) variants, on the trustworthiness of language models across five key verticals: toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. Our findings reveal a complex interplay between these aspects and the alignment methods employed, with Proximal Policy Optimization (PPO) and Supervised Finetuning (SFT) leading to worsened performance on toxicity and privacy, while Direct Preference Optimization (DPO) results in reduced adherence to ethical principles, as model size increases. All three algorithms also contribute to increased stereotypical bias and decreased truthfulness in the model outputs. These results underscore the need for more nuanced approaches to model alignment and highlight the importance of considering the multifaceted nature of trustworthiness when developing and deploying language models. By shedding light on the intricate dynamics of these alignment methods, we hope this research will guide the community towards developing more trustworthy language models in the future.

## Appendix A Prompt Templates and Generation Examples

### A.1 Toxicity

### A.2 Stereotypical Bias

### A.3 Machine Ethics

### A.4 Truthfulness

### A.5 Privacy

## Appendix B Additional Results on Toxicity

### B.1 Expected Maximum Toxicity With Adversarial System Prompt

Figure: Figure B0: Impact of SFT, PPO, and DPO on toxicity under adversarial system prompt - the trends are similar to the benign setting; note that the scales on y-axis are different.
Refer to caption: x6.png

**Table B1: Expected Maximum Toxicity scores under toxic (left) and nontoxic (right) user prompts.**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 0.364 | 0.306 | 0.082 | 0.226 |
| 160M | 0.413 | 0.408 | 0.324 | 0.288 |
| 410M | 0.478 | 0.476 | 0.481 | 0.328 |
| 2.8B | 0.494 | 0.527 | 0.639 | 0.418 |
| 6.9B | 0.513 | 0.545 | 0.611 | 0.498 |

### B.2 Effect of RLHF on Classification Accuracy of Toxicity Probe

The goal of this experiment is to show that models finetuned with PPO and SFT indeed have more easily decoded toxic information in their hidden state before the generation starts. To test this hypothesis, we train simple feed-forward probes on top of the language model’s last hidden state, using sampled model continuations based on prompts in RealToxicityPrompts. The training and test datasets each consists of 5000 generations with balanced toxicity scores. The labels are also determined by the toxicity scores, which are computed by Perspective API, such that a score greater than 0.5 belongs to a toxic generation and otherwise nontoxic. Compared with Figure 1, the results in Figure B1 (left) and Table B2 (left) show clear correlation between probe’s classification accuracy and toxicity score.

Figure: Figure B1: Classification accuracy of the trained toxicity probe (left) and mean log-likelihood of generated tokens averaged across all four prompt settings (right) are highly correlated with toxicity score.
Refer to caption: x7.png

**Table B2: Left: test accuracy for toxicity classification; Right: mean log-probability of each generated token, averaged across all four settings.**
| Model Size | Baseline | SFT | PPO | DPO |
| --- | --- | --- | --- | --- |
| 70M | 68.8 | 70.8 | 68.9 | 67.9 |
| 160M | 69.7 | 70.0 | 66.9 | 68.9 |
| 410M | 70.8 | 71.6 | 72.4 | 71.8 |
| 2.8B | 77.9 | 81.1 | 80.9 | 77.3 |
| 6.9B | 78.4 | 79.4 | 81.1 | 78.2 |

### B.3 Effect of RLHF on Output Distribution

For the decoding stage, we hypothesize that finetuned models with higher toxicity tend to have narrower output distributions [], and we verify it by computing the mean log-probability of each generated token within each sequence, and then taking the average across all four settings (Toxic vs Nontoxic, Benign vs Adversarial). The results included in Figure B1 (right) and Table B2 (right) also show strong correlation with toxicity score.

### B.4 Mean Toxicity score and Toxicity Probability

As mentioned earlier, we also compute the mean toxicity and toxicity probability under two system prompt settings, and the results are reported in Table B3 and B4.

**Table B3: Mean toxicity and toxicity probability with benign system prompt**
| System Prompts | Models | Mean Toxicity | Toxicity Probability |  |  |
| --- | --- | --- | --- | --- | --- |
| Toxic | Nontoxic | Toxic | Nontoxic |  |  |
| No-RLHF | 70M | 0.214 | 0.057 | 19.5% | 0.7% |
| 160M | 0.290 | 0.067 | 28.5% | 1.3% |  |
|  | 410M | 0.371 | 0.075 | 39.2% | 1.6% |
|  | 2.8B | 0.388 | 0.084 | 41.1% | 2.4% |
|  | 6.9B | 0.413 | 0.087 | 43.8% | 2.8% |
| SFT | 70M | 0.208 | 0.068 | 17.0% | 0.3% |
| 160M | 0.312 | 0.076 | 30.1% | 0.9% |  |
|  | 410M | 0.366 | 0.085 | 38.3% | 1.6% |
|  | 2.8B | 0.408 | 0.092 | 43.1% | 2.5% |
|  | 6.9B | 0.436 | 0.092 | 46.7% | 2.7% |
| PPO | 70M | 0.062 | 0.023 | 3.9% | 0.1% |
| 160M | 0.251 | 0.058 | 25.0% | 0.9% |  |
|  | 410M | 0.373 | 0.086 | 38.8% | 1.8% |
|  | 2.8B | 0.465 | 0.103 | 48.4% | 2.8% |
|  | 6.9B | 0.454 | 0.097 | 47.3% | 3.0% |
| DPO | 70M | 0.160 | 0.059 | 11.1% | 0.2% |
| 160M | 0.207 | 0.066 | 17.0% | 0.7% |  |
|  | 410M | 0.248 | 0.073 | 22.2% | 0.9% |
|  | 2.8B | 0.324 | 0.084 | 32.3% | 1.2% |
|  | 6.9B | 0.402 | 0.087 | 41.9% | 2.2% |

**Table B4: Mean toxicity and toxicity probability with adversarial system prompt**
| System Prompts | Models | Mean Toxicity | Toxicity Probability |  |  |
| --- | --- | --- | --- | --- | --- |
| Toxic | Nontoxic | Toxic | Nontoxic |  |  |
| No-RLHF | 70M | 0.268 | 0.072 | 24.9% | 0.9% |
| 160M | 0.316 | 0.069 | 31.9% | 1.4% |  |
|  | 410M | 0.385 | 0.077 | 40.2% | 1.7% |
|  | 2.8B | 0.412 | 0.091 | 43.5% | 2.9% |
|  | 6.9B | 0.430 | 0.093 | 46.5% | 3.3% |
| SFT | 70M | 0.242 | 0.076 | 21.0% | 0.6% |
| 160M | 0.326 | 0.077 | 31.4% | 1.4% |  |
|  | 410M | 0.371 | 0.088 | 37.8% | 2.1% |
|  | 2.8B | 0.440 | 0.099 | 46.9% | 3.1% |
|  | 6.9B | 0.448 | 0.101 | 47.9% | 3.5% |
| PPO | 70M | 0.065 | 0.031 | 4.6% | 0.2% |
| 160M | 0.283 | 0.061 | 27.8% | 1.1% |  |
|  | 410M | 0.378 | 0.087 | 38.2% | 1.9% |
|  | 2.8B | 0.492 | 0.113 | 51.5% | 3.5% |
|  | 6.9B | 0.462 | 0.106 | 49.1% | 3.5% |
| DPO | 70M | 0.180 | 0.068 | 12.6% | 0.3% |
| 160M | 0.221 | 0.068 | 17.9% | 1.0% |  |
|  | 410M | 0.240 | 0.073 | 19.3% | 0.9% |
|  | 2.8B | 0.344 | 0.089 | 34.2% | 1.8% |
|  | 6.9B | 0.421 | 0.093 | 44.8% | 2.6% |

## Appendix C Language Model Configurations

**Table C1: Model configurations used in different evaluation tasks**
| Config | Toxicity | Stereotypical Bias | Machine Ethics | Truthfulness | Privacy |
| --- | --- | --- | --- | --- | --- |
| max_new_tokens | 50 | 70 | 30 | 100 | 100 |
| temperature | 0.5 | 0.01 | 0.01 | 0.5 | 0.5 |
| num_beams | 7 | 3 | 3 | 5 | 5 |
| num_return_sequences | 5 | 1 | 1 | 3 | 3 |
| do_sample | True | False | False | True | True |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: See above weaknesses.
soundness: 4
strengths: - The paper provides a detailed analysis of RLHF methods across multiple trustworthiness dimensions, offering nuanced insights into alignment challenges.
- By comparing SFT, PPO, and DPO, the paper highlights strengths and weaknesses of each method on its effect on trustworthiness, helping guide future model alignment choices.
- The paper evaluates trustworthiness through several dimensions—such as bias, truthfulness, toxicity, ethics, and privacy—rather than limiting its analysis to one aspect. This multi-faceted approach strengthens the research by showing how alignment impacts different areas, which are crucial for real-world applications where trustworthiness encompasses various dimensions.
- The proposed data attribution method is useful for observing which data points most significantly affect certain trustworthiness dimensions, such as reducing toxicity. By attributing changes in model behavior to specific data inputs, this method provides insights into how particular data segments influence alignment outcomes.
- The paper is well-written and easy to follow.
summary: The paper evaluates how Reinforcement Learning from Human Feedback (RLHF) impacts Large Language Model (LLM) trustworthiness. It assesses five dimensions: toxicity, bias, machine ethics, truthfulness, and privacy. Results show mixed outcomes: while RLHF improves machine ethics, it often increases bias and privacy leakage and reduces truthfulness, with minimal impact on toxicity. The paper also introduces a novel data attribution method in the RLHF setting to analyze RLHF's effect on these dimensions, highlighting the need for more targeted alignment strategies.
weaknesses: - In this paper, toxicity is evaluated using the Perspective API. However, several existing papers (see https://aclanthology.org/2023.emnlp-main.472.pdf and https://arxiv.org/pdf/2312.12651) have raised concerns that such black-box APIs may introduce inaccuracies or biases in toxicity assessments. This may affect the reproducibility of this research on this dimension.
- What are the potential trade-offs between the 5 dimensions in trustworthiness? In real-world applications, improving one dimension (e.g., reducing bias) might compromise another (e.g., truthfulness). Would appreciate some insights here.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. One of the main claims in the paper is “[RLHF]’s assumed effect on model trustworthiness hasn’t been rigorously evaluated” (for instance lines 15-16 and 42-43). However, the evaluation of  RLHF impact in ethical aspects (e.g., truthfulness, bias, toxicity, ethics) has been reported previously (Askell et al 2021, Glaese et al 2022, OpenAI Team 2024, Touvron et al 2023). Can you provide a more detailed comparison of your methodology and findings with the previous works, highlighting specific differences or extensions to their approach? 
2. In lines 202-203, the authors mention “since our language models are large enough, we use zero-shot generated outputs for all benchmarks”. However, a limitation of base models is that they struggle to adapt to the output format. Specifically they may struggle to adhere to the multiple-choice question format of the TruthfulQA (Section 4.4) or to produce yes/no/unknown for the stereotypical bias experiment (Section 4.2). How do you ensure that the output of the language model is consistent with the multiple choice question answering format? Do you perform any test to assess the capability of LLMs to handle 0-shot QA? You could include a specific analysis of the models' ability to adhere to the required output formats in zero-shot settings, perhaps by reporting the percentage of responses that correctly follow the format for each task.
3. The toxicity evaluation reported in line 222 differs from the original evaluation procedure (5 independent generations and temperature=0.5 vs 25 generations and temperature=1). The lower number of generations may substantially affect the results and the lower temperature modifies the model’s distribution, potentially, skewing results. Could you please motivate the selection of these parameters for your analysis? Moreover, you could provide a sensitivity analysis showing how your results change with different numbers of generations and temperature settings. This would help clarify the robustness of your findings.
4. Some experiments like the stereotypical bias (Section 4.2) are carried out by eliciting whether the model agrees or not with a specific stereotypical topic. How can we guarantee that the observed bias amplification is not an artifact of the evaluation (e.g., models being more likely to output yes vs no, or their sycophantic tendency (Sharma et al 2023))? 
5. In Section 4.3, the authors mention that they aim to measure a model's ability to detect morally wrong actions by prompting models whether each example is against human morality. This presupposes that the correct answer (true positive) is to answer yes and that a wrong answer is to answer “no” (false negative). However, the authors call their metric FPR, which does not match with this intuition. Can you provide a clear definition of a true positive, true negative, false positive, and false negative are in this context and explain why FPR is the most appropriate metric for your analysis?
6. What exactly is the prompt being used to prompt base models? If you’re using different prompts, are models sensitive to the prompts, i.e., do you observe considerably different behaviors depending on the prompt used? 
7. The privacy evaluation is unintuitive (to me). I am not surprised that fine-tuning LLMs with general purpose preference data would lead to improvements in such a task. In particular, (1) there is no explicit fine-tuning stage that could boost models’ ability to understand the instructions, (2) most conversation scenarios that I can think of the information shared by the user is already known by it. Would you mind commenting on the motivation and practicality of the proposed evaluation?
8. In lines 365-366, the authors mention that “the most intuitive answer is that the HH dataset used for RLHF is out-of-domain for the evaluation tasks” but provide no support for this hypothesis. Have you considered reporting the perplexity of the evaluation datasets  and comparing it to the perplexity achieved in the training dataset or the Ethics benchmark?
soundness: 3
strengths: - Important research question regarding whose preferences concerning what behaviors models are implicitly learned through RLHF.
- The finding that privacy leakage increases with RLHF and is due to the DPO/PPO algorithms (as opposed to SFT) is interesting.
- The use of data influence functions to traceback the impact of RLHF training data to different aspects of trustworthiness is novel and interesting and could prove beneficial to inform the creation of higher quality and better-aligned preference datasets.
summary: **Summary**:
This paper investigates the overall impact of RLHF training on the moral values exhibited by LLMs. The analysis is carried out by fine-tuning base models (up to 8B in size) in general-purpose preference alignment data and then contrasting the performance before and after RLHF across 5 trustworthiness tenets: toxicity, social bias, ethics, truthfulness, and privacy. The obtained results suggest that although RLHF improves ethics measurements by 16.5-18.5% points on average, it significantly worsens performance in other verticals: social bias, truthfulness and privacy leakage worsen on average by 150%, 25% and 12%, respectively. 

**Main contributions**:
The main contribution is the idea that RLHF models trained on general-purpose preference data may not align with other trustworthiness values that we care about. The application of data influence functions is also novel and interesting.

**Methodology**:
- The authors start with the Anthropic Harmlessness and Helpfulness (HH-RLHF) preference alignment dataset, from which they fine-tune the base model (SFT), the reward model (RM), as well as two RLHF-based models (one trained with DPO loss and the other with PPO loss).
- Adopting the evaluation testbed proposed in [Wang et al 2024](https://arxiv.org/pdf/2306.11698), the authors report metrics for the four different variants of the models (base, SFT, DPO, PPO) and across model sizes (1.4B up to 8B) across 5 trustworthy tenets contrasting the impact of fine-tuned models with base models. 
- Finally, the authors use data influence functions to determine the impact of individual data points in the HH-RLHF dataset for each of the trustworthiness dimensions.

**Interpretation of Results**:
- The results reported across 5 different trustworthiness benchmarks show that fine-tuning LMs in general-purpose preference data (RLHF-HH) leads to worse model behavior in 4  benchmarks when compared to not fine-tuning in RLHF-HH. Across four models, RLHF leads to average improvements of up to 18.5% in recognizing morally wrong scenarios (Figure 3 left). For all other benchmarks, the metric values decrease on average across models by at least 12% points (in Figure 4 left) up to 30% (Figure 3 right). 
- In lines 228-229, the authors claim that “toxicity first increases notably after SFT“ which suggests that there is a large improvement in toxicity. However, there is no statistical test supporting this claim and the error bars are overlapping for two larger models (Figure 2 left), suggesting non-significant differences resulting from the SFT procedure.
- In Section 4.2 the authors claim that “both PPO and DPO significantly increase the stereotypical bias scores [...] and the SFT step is most responsible for this increase.”. However, the carried evaluation consists (to the best of my knowledge) of prompting models whether they agree with a stereotypical sentence concerning a marginalized group but not whether the model agree with the same stereotypical sentence concerning a different group. In other words, **the reported results do not disentangle the models’ propensity to agree with claims (as suggested in lines 260-263 and by prior work (Sharma et al 2023)) from the models’ stereotypical bias**. 
- In Section 5, the authors mention that “Table 5 clearly demonstrates that the language model becomes increasingly deterministic through RLHF, which may provide some insights on why PPO and DPO fail to decrease toxicity [...]”. However, the maximum log likelihood difference between base model and SFT is only 0.78, raising questions about whether this difference is due to noise or if it’s a systematic behavior. Moreover, the generations for the original toxicity experiment were obtained using temperature=0.5, which modifies the actual distribution of the SFT model, potentially leading to narrower distributions. It is important that the authors clarify which temperature was used to carry the experiments in Appendix F.


### Writing
The paper could improve its clarity by adding relevant citations, better positioning their work with respect to previous work, and being clear about the hyperparameters used during the evaluations. Consider the following suggestions: 
- Relevant citations: overall the authors would benefit from adding more relevant citations to back their arguments and strengthen the motivation.
- In the Related Work, subsection “Reinforcement Learning Human Feedback section”, the authors could add citations to the RLHF-HH dataset, but also to other preference datasets (e.g., Cui et al, 2023, Ethayarajh et al 2022). It could also be useful to motivate why the authors settled on the RLHF-HH dataset as opposed to other datasets.
- Lines 260-263 could cite work reporting sycophancy and language model (Sharma et al 2023).

**Clarity**:
- Line 132: “D_demo” → “D” (to be consistent with Equation 1)?
- Equation 1: consider specifying what y_w right after Equation 1 or at the beginning of the section.
- The authors should try to clarify the differences between their experimental design and that of Havrilla et al (2023) and Wang et al (2024). For instance, the analyses carried in Section are very similar to the ones reported in Wang et al (2024), in particular, the evaluation in Section 4.2 appears to be the same but no citation is provided. It could also be nice to motivate the selection of GPT-J-6B based on Havrilla et al (2023) in case the authors didn’t perform any additional experiment using reward models.
 - The authors use different hyperparameters depending on the evaluated criteria. While these parameters are specified in the Appendix C, the only hyperparameters mentioned in the main paper are those for the toxicity evaluation (without referring to the Appendix C or differences in the hyperparameters). The authors should consider being more explicit within each section, motivating the reasons behind selecting different hyperparameters. This is relevant as different hyperparameters have different implications (e.g., lower temperature values have been shown to be associated with higher toxicity)
- Could be useful to add examples for each of the evaluation datasets. Specifically, for the Steoreotypical Bias (in Section 4.2), it would be useful to get a sense of what are the demographic groups, their cardinality, and also the types of stereotypes explored.
- Could be useful if the authors provided a notion of what a meaningful influence score is.
- Figure 5 caption could be more informative, providing more insights on the performed analyses.

Typos:
- Line 398: “8 with 9” → “Equation 8 with Equation 9”
- Line 407: “in 9” → “in Equation 9”

### References

- Touvron et al (2023): Llama 2: Open Foundation and Fine-Tuned Chat Models (https://arxiv.org/abs/2307.09288)
- OpenAI Team (2023): GPT-4 Tech Report (https://arxiv.org/pdf/2303.08774)
- Cui et al (2023) UltraFeedback: Boosting Language Models with Scaled AI Feedback (https://arxiv.org/abs/2310.01377)
- Ganguli et al (2022): Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (https://arxiv.org/abs/2209.07858)
- Ethayarajh et al (2022): Understanding Dataset Difficulty with V-Usable Information (ICML 2022). (https://proceedings.mlr.press/v162/ethayarajh22a.html)
- Askell et al (2021): A General Language Assistant as a Laboratory for Alignment (https://arxiv.org/abs/2112.00861)
- Glaese et al (2022): Improving alignment of dialogue agents via targeted human judgements
- Tirumala et al (2023): D4: Improving LLM Pretraining via Document De-Duplication and Diversification (https://arxiv.org/abs/2308.12284)
- Sharma et al (2023): Towards Understanding Sycophancy in Language Models (https://arxiv.org/abs/2310.13548)
weaknesses: - Unclear contribution with respect to previous work: Wang et al (2024), GPT-4, and Llama 2 paper tech reports also carry analyses assessing the impact of RLHF in bias, toxicity, and truthfulness. The authors should make their contribution clearer and compare with previous work.
- The evaluation is carried out in Pythia and Llama models as opposed to more recent and prominent models like OLMo and Llama 3 models. Given the considerable differences between the models (e.g., data deduplication and diversity (Tirumala et al 2023)) it is unclear how the results will generalize to these newer and carefully trained base models.  
- The reported analysis using influence functions provides no insights into the observed patterns in RLHF or the optimal composition of the RLHF training data. It could be beneficial to add a section with further analysis and/or a discussion section about how the authors see this analysis being useful in the long-term to fix the mis-alignment problem.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How might these findings change with LLMs that exceed 7B parameters?
2. Would a more specialized preference dataset yield different outcomes for bias or other dimensions compared to the general-purpose Anthropic HH dataset?
3. What guidance do the paper’s evaluation results offer for balancing helpfulness and trustworthiness in the future?
soundness: 3
strengths: 1. This study offers a systematic assessment of RLHF on multiple trustworthiness aspects;
2. The introduction of an influence-function-based data attribution method tailored to RLHF offers insights into individual data points’ impacts, aiding future data selection for alignment;
3. The work highlights the limitations of using general-purpose datasets for trustworthiness, emphasizing the importance of targeted data in achieving specific alignment goals.
summary: The authors systematically assess how two RLHF variants, PPO and DPO, influence these trustworthiness dimensions using general-purpose preference data. What’s more, this paper introduces a data attribution method to analyze the influence of individual fine-tuning data points on trustworthiness, suggesting the need for more nuanced RLHF approaches.
weaknesses: 1. The experiments are limited to models up to 7B parameters, which may limit the generalizability of findings to larger;
2. The study relies solely on the Anthropic HH dataset, and results might vary with different datasets;

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - Can additional fine-tuning methods be combined with RLHF to mitigate observed trustworthiness degradation?
- Would using smaller, targeted RLHF datasets for each trustworthiness aspect outperform a single general-purpose dataset? Additionally, how would the combination of general-purpose and aspect-specific datasets impact trustworthiness?
- How does dataset size influence the observed trends in trustworthiness metrics? Specifically, would varying dataset sizes in RLHF lead to different levels of model trustworthiness?
soundness: 3
strengths: - **Well-structured:** The paper is organized clearly and uses precise language. It offers the first systematic evaluation of RLHF’s impact on specific aspects of trustworthiness.
- **Thorough Evaluation:** Multiple trustworthiness dimensions of RLHF’s effects are thoroughly examined.
- **Innovative Methodology:** The application of influence-function-based data attribution methods to RLHF provides valuable insights into how training data influences model behavior.
- **Empirical Rigor:** The results are detailed, with comprehensive benchmarks and rigorous experimental designs, involving multiple models and controlled benchmarks, allowing for broader generalization.
summary: The work investigates the impact of RLHF on the trustworthiness of LLMs. evaluating key dimensions such as toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. Findings indicate that RLHF does not consistently enhance trustworthiness and may even degrade it in some areas.

Further analysis with targeted datasets and a discussion of potential solutions would strengthen its impact. However, this contribution is significant and could be accepted for the conference.
weaknesses: - **Mitigation Strategies Lacking:** Although RLHF exacerbates bias and privacy issues, potential solutions are not explored in depth. There is limited discussion on why these misalignments occur and how they could be systematically addressed.
- **Dataset Limitations:** The reliance on the Anthropic HH dataset for RLHF may limit the generalizability of findings across different applications and LLM architectures. The general-purpose nature of this dataset appears somewhat misaligned with the targeted trustworthiness benchmarks, potentially skewing the results.
- **Model Scale Constraints:** The analysis is limited to models of up to Pythia-6.9B and Llama-7B, which may not reflect performance in larger, frequently used LLMs.
- **Figure Clarity:** Certain figures, such as Figures 5 and 11, are not optimally clear. Text and data appear crowded, reducing readability.
- **Dense Methodology Sections:** Sections such as 3 and 5 contain dense explanations that could benefit from clearer, more accessible language.

Meta-review: This paper evaluates four language models Pythia 1.4B, 2.8B and 6.9B as well as Llama-&B on five axes of “trustworthiness” : Toxicity, Stereotypical bias, Machine Ethics, Truthfulness and Privacy.  The main contribution is that optimizing these LLMs for human preferences
does not simultaneously optimize them for trustworthiness, in fact in the aspects of stereotypical bias, truthfulness and privacy it has the opposite effect.  

The authors use the Anthropic HH dataset for their evaluation. And show results for: the autoregressive language loss for SFT, the preference loss of the reward model for PPO and the loss of the language model for DPO when compared to the “no RLHF” baseline.  

This paper was considered a good paper by all reviewers and in the rebuttal phase some authors increased their score.  It represents an important contribution, and I recommend that it should be accepted.

Decision: Accept (Oral)