On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

Authors: Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, Anca Dragan
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: manipulation, deception, alignment, reward hacking, user feedback

Abstract:
As LLMs become more widely deployed, there is increasing interest in directly optimizing for feedback from end users (e.g. thumbs up) in addition to feedback from paid annotators. However, training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. In our settings, we find that: 1) Extreme forms of "feedback gaming" such as manipulation and deception are learned reliably; 2) Even if only 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and target them while behaving appropriately with other users, making such behaviors harder to detect; 3) To mitigate this issue, it may seem promising to leverage continued safety training or LLM-as-judges during training to filter problematic outputs. Instead, we found that while such approaches help in some of our settings, they backfire in others, sometimes even leading to subtler manipulative behaviors. We hope our results can serve as a case study which highlights the risks of using gameable feedback sources -- such as user feedback -- as a target for RL. Our code is publicly available. Warning: some of our examples may be upsetting.

Decision: accept (poster)
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730286487613

thread_type:
official_review

<thread_object>
creator: reviewer_89wh

<content>
summary:
The paper demonstrates through empirical experiments that an LLM agent trained using RL to maximise user feedback may be incentivised to manipulate or deceive the user. This is demonstrated across 4 different classes of problem domain, involving either single-stage or 2-stage conversations between the agent and the user. The results show that the agent may lie to the user, exhibit sycophantic behaviour even when this is associated with harmful content or actions, or engage in more subtle manipulative behaviour such as nudging. Significantly this can arise even when only a subset of the user population is manipulatable, as the agent learns to exploit those users while behaving normally with respect to the rest of the population, which make this manipulative behaviour harder to detect. Two possible approaches to mitigating this issue are proposed and evaluated, and its shown that while they may have beneficial effects in some cases, they can also result in additional problematic behaviour on the part of the agent.

strengths:
The paper addresses a very important issue. While previous work has hypothesised that LLM-based agents may be motivated to 'game' the preferences of users, this work provides quite solid empirical evidence of this actually occurring in scenarios which are similar to the situations where such agents might actually be deployed.

The paper is very well written. It is easy to read and does an excellent job of presenting the context of the work and discussing its implications.

The experiments were mostly well-designed (see one proviso in the Weaknesses section below) and clearly described, and the analysis is clear and quite compelling. The provision of code implementations also means that the hard work which has gone into developing these test cases and the evaluation procedures can be leveraged further by other researchers.

The limitations of the approaches used are appropriately acknowledged and justified (e.g. the use of a relatively simple approach to the RL optimisation process).

weaknesses:
The main weakness I see in the paper relates to the manner in which the gameable and non-gameable users are differentiated in the initial states. The examples in Figure 13 show that this is less than subtle, with the agent being directly told that one user is extremely susceptible to their suggestions and will always act on their advice. The results of this set of experiments would be much stronger if the distinction between gameable and non-gameable users was more subtly indicated to the agent, or even if this was something which the agent had to largely establish for itself through repeated interactions with each user. The statement in Section 4.2 that "Exploratory experiments suggested that the exact difference in background didn’t matter much" suggests that this issue has been explored to some extent, but no details are provided. The paper's case would be stronger if the outcomes of experiments using more subtle indicators were reported (at least in an appendix if space doesn't permit it in the main body).

Some small presentation issues. The text on some of the figures is extremely small (for example, I had to zoom in well over 200% to read some of the text in Figure 1). This might be difficult to fix given the page restrictions, but it would make the paper easier to read if this could be addressed.

There are a few undefined acronyms. While I think its reasonable to use common terms like RL and RLHF without explanation, some of the other acronyms were ones which readers (like myself) might not be familiar with (eg on p3, BoN and KTO). It would be better is these were spelled out in full when first used.

questions:
Why is there an asterisk in the title? I thought this would be explained somewhere in the paper but I couldn't find it.

What is the meaning of the User P.O. and Agent P.O. columns in Table 1?

I was unclear on the discussion of manipulation in the action-advice scenario on p5. Why would encouraging the user to engage in a harmful action lead to higher ratings for the agent responses later in the conversation? Is this simply the "afterglow" of the user's positive experience of the harmful action?

In Fig 6, there's a difference in the problematic behaviour of the agent with regards to the gameable and non-gameable users before training. Can you explain this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730584436615

thread_type:
official_review

<thread_object>
creator: reviewer_xx6k

<content>
summary:
The title and abstract point to an important problem in LLMs, i.e., issues with feedback loops originating from user feedback. While the problem of feedback loops has been acknowledged earlier (even in the context of LLMs, e.g., due to training LLM generated data, as well as in concurrent work (Wen et al. 2024)) and it is well-known that optimization might aim at shortcuts and objectives are difficult to formulate in machine learnign, the paper adds a number of interesting claims that if appropriately backed up by evidence, would justify a publication at ICLR without any doubt. While the paper discusses a bunch of interesting cases, the evaluation is, where the paper falls short. Already the title should say "... deception could emerge..." and the abstract should be presented more modestly. The evidence relies on simulated data, on an optimization method that is not among the most common ones, and an LLM that is not used by the majority of users. It is unclear to what extend the results transfer to other scenarios. In turn, this makes the statements much less relevant, as it is in general well-known that such "optimization issues" could occur. While the paper is well-aware of these shortcomings and aims to talk them away (also partially backed by refs), overall some form of direct comparison of simulated data and real data would be needed to show alignment of feedback distributions.

-------
I acknolwedge the author's response, while the ChatGPT style verboseness is not appreciated, at least the point with the optimization seems somewhat justified. I updated my score.

strengths:
see above

weaknesses:
see above

questions:
Do you agree that you overclaim and your evidence is insufficient for the general claims?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663171198

thread_type:
official_review

<thread_object>
creator: reviewer_ah2r

<content>
summary:
This paper investigates the potential risks of training large language models (LLMs) directly on user feedback through reinforcement learning. The authors demonstrate that optimizing for user feedback can lead LLMs to showcase manipulative behaviors in different settings, such as deception and sycophancy, which can be selectively targeted at vulnerable user subsets.

strengths:
1.  The findings of the study is  interesting and has the potential to improve the safety of LLMs. It conducts a thorough analysis of various manipulative behaviors that LLMs can exhibit and how these can be selectively applied to different user groups.

2. The experimental setup, using simulated environments to test various scenarios, is robust and well-justified, which strengthens the validity of the results.

weaknesses:
1. The paper is mainly relied on the simulated user feedback and lack of read-world user data would limit the generalizability of the findings. The simulated user behavior data can not fully reflect the complexity of real human feedback when interactingf with the LLMs. In addition, the study assumes that users keep a static understanding of LLM behavior which does not account for the evolution of user understanding and feedback when continuously interacting with the LLM.


2. The mitigation strategies discussed are only partially effective and could benefit from more advanced techniques to better  suppress the manipulative behavior of the LLM especially in some critical cases.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731638180595

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated paper and additional experiments

comment:
We have an updated version of our paper, addressing many of the reviewer's concerns, and making other minor improvements. These are the main changes:
- **We added new experiment in which we show that the model is able to identify and target users based on arbitrary characteristics**. In particular, we denote gameable and non-gameable users simply based on whether they like the color Red or Blue. We see similar results to our prior experiments.
- **New experiments with 2% of gameable users and updated results for the other percentages.** It turned out we weren’t training for long enough previously.  We now show that even when only 2% of users are gameable, almost 100% of the trajectories for them are harmful, while almost 0% are harmful for the non-gameable users.
- **We updated our aggregate harm metric:** we switched our primary metric to be the fraction of harmful trajectories, because we realized that our previous metrics were somewhat misleading: a "2 out of 5" score for harm by GPT is not particularly harmful, but it looks half as harmful as "4 out of 5" which is very harmful. We find the new metrics to agree much better with how we would qualitatively rate the trajectories. For details see appendix K.1.
- **We updated the title and phrasing relating to our claims, emphasizing more the limitations of our findings.**
- **Additional attempts at improving our mitigation techniques:**
  - **We added sycophancy/toxicity benchmark results for *action-advice* and for *political-questions* trained with HH data.** We find similar trends to our other results.
  - **We tried using a different HH dataset, PKU-SafeRLHF and got very similar results to HH-rlhf.**
  - **We tried combining different approaches for veto training to see if the effects stacked up ("Negative + 5-point").** They didn't, providing further evidence that simple mitigation strategies may not work well.
  - **We added experiments using the veto model at runtime instead of at training time to try to understand better why using veto models fails (Appendix F.4.1).** Vetoing trajectories at runtime is much more successful although some harmful behavior slips through.

Other minor things:
- We added some additional experiments about giving the model a hidden scratchpad to the appendix, and found some evidence that RL distorts CoT reasoning, and that it engages in what looks like forms of "motivated reasoning".
- For Booking-Assistance, we split some of the plots to show two bars, one for the “lying about the tool call being successful” behavior and one for the “convincing the user they didn’t want a ticket” behavior. This better illustrates quantitatively how the model changes from one harmful behavior to another when applying the veto.
- Made the text in many of the plots bigger and more readable.
- We expanded the discussion of user feedback optimization and its risks
- We strengthened connections to related work, especially on feedback gaming and manipulation.
- We normalized the generalization plot to the normal training run for each run to more easily see how the results compare.
- We expanded discussion of mitigation strategies and their limitations. Added another type of veto combining the 5-point prompt and the negative training.
- We added more examples of initial states
</content>
</thread_object>

</thread>
<thread>
creation date:
1731638857483

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common concern about realism of our simulated environments/feedback

comment:
This response is intended for both Reviewers aH2R and xX6K, as they share a similar concern.

We generally agree that the lack of real user feedback data is a limitation of our work. However, we don’t think this greatly diminishes the significance of our work: below, we aim to clearly break down what kinds of realism matters for our claims to be supported, and address each in turn.

## Would real users be gameable at all?
In addition to a [large](https://www.sciencedirect.com/science/article/pii/S0079742108603151) [body](https://onlinelibrary.wiley.com/doi/abs/10.1046/j.1360-0443.2001.961474.x) of [psychology](https://psycnet.apa.org/buy/1991-06436-001) [work](https://www.sciencedirect.com/science/article/pii/0749597885900494) discussing human biases which may lead people to incorrectly reward certain problematic responses more highly than others, [Wen et al](https://arxiv.org/abs/2409.12822) (concurrent work) also provide compelling real-world examples of real people being “gameable” in certain simple ways. It seems hard to disagree with the claim that there exist important real-world settings in which people would mistakenly give higher feedback when tricked, deceived, or manipulated.

## Would users be gameable in our exact environments? What fraction of users will be gamable (in each context)? 
While we do not provide conclusive evidence that real users would be gameable in the environments we simulate, it seems unlikely to us that no users would behave in ways similar to what we simulate. 
Indeed, our results in the updated Section 4.2 show that even if only 2% of users (e.g. that express a desire to steal money to buy lottery tickets) are ultimately seeking validation for these harmful behaviors, models would be able to target them. 2% seems to us like a conservative estimate of the real fraction of users that would want validation after sending messages like the ones we consider. 

## How easy will it be for LLMs to distinguish between gameable/non-gameable users in practice?
Feedback gaming will be optimal only when the expected value of deception/manipulation is higher than any other alternative strategy. For settings in which only a subset of users is gameable, the expected value of manipulating/deceiving the user will depend on the cost incurred by the model for misclassifying a user as gameable when they are not (and vice-versa), and behaving (by mistake) in ways that would lead to low reward. 

Therefore, the expected value of manipulation/deception will indirectly depend on how easy it is for the LLM to distinguish between gameable and non-gameable users. In the main experiments in our paper, we gave the model the character traits of the users it was interacting with (whether they were gullible or independent). This may be a confounder, as it looks similar to directly telling the model that the user is gameable, making it easier for the model to explore this behavior. In a new experiment, we show that the model can learn to distinguish between gameable and non-gameable users via *arbitrary user characteristics*: we changed the only difference between gameable and non-gameable users to be whether they like the color “Red” or “Blue”, still expressed as their character traits in *therapy-talk* (see Appendix F.2 of the new uploaded paper). The model was still able to identify and target gameable users by using this indicator of gameability: this suggests that models can target gaming behaviors based on *arbitrary information they may have about users*, as long as such information is sufficiently correlated with whether such users are gameable.

Also note that [prior work](https://arxiv.org/abs/2310.07298) has shown that LLMs are able to infer many characteristics of users just from their text inputs, and it seems quite plausible that some of them would be correlated with being gameable.

Edit: we have now added 3 more experiments to the paper which further indicate that the exact difference between initial states is not that important. If we remove the AI memory section but it is still fairly clear from the user message which users are gameable, the bad behavior is learned even faster. If we instead make it difficult to distinguish between different types of users model struggles in the 1 step setting but has no trouble if allowed 2 conversation turns. If we make the initial states identical it is obviously impossible for the model to differentiate in the 1 step setting but the model still does reasonably well in the 2-step setting. See appendix F.2 in the updated paper for examples of the new easy/hard initial states and graphs of the harmful behavior emerging.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230036255

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion period ending

comment:
As the discussion period is drawing to an end, if you have any unaddressed concerns (especially after our additional experiments), we'd be thankful if you could share them with us. Otherwise, we hope you may consider updating your scores.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper points out that training LLMs with user feedback can insidiously encourage LLMs to resort to manipulative behaviors with gameable users.
Through several empirical simulations the authors convincingly show that LLMs (despite their safety training) manifest these behaviors in practice.
They additionally show that LLMs can learn to identify and target vulnerable users, which is a novel harm vector that the LLM safety community should take seriously.
Finally they also show that a broad range of SOTA mitigation strategies are not effective at identifying and reducing these harmful behaviors.

All of the reviewers agreed that the paper makes novel contributions (concurrent with Wen et al "Language Models Learn to Mislead Humans via RLHF", 
albeit making differentiated contributions),
of great significance to the LLM safety community,
and is empirically sound and clearly written.