Teaching Models to Balance Resisting and Accepting Persuasion
Authors: name: Elias Stengel-Eskin; name: Peter Hase; name: Mohit Bansal
Published: 2024-10-18 16:49:36

Abstract: Large language models (LLMs) are susceptible to persuasion, which can pose
risks when models are faced with an adversarial interlocutor. We take a first
step towards defending models against persuasion while also arguing that
defense against adversarial (i.e. negative) persuasion is only half of the
equation: models should also be able to accept beneficial (i.e. positive)
persuasion to improve their answers. We show that optimizing models for only
one side results in poor performance on the other. In order to balance positive
and negative persuasion, we introduce Persuasion-Balanced Training (or PBT),
which leverages multi-agent recursive dialogue trees to create data and trains
models via preference optimization to accept persuasion when appropriate. PBT
consistently improves resistance to misinformation and resilience to being
challenged while also resulting in the best overall performance on holistic
data containing both positive and negative persuasion. Crucially, we show that
PBT models are better teammates in multi-agent debates. We find that without
PBT, pairs of stronger and weaker models have unstable performance, with the
order in which the models present their answers determining whether the team
obtains the stronger or weaker model's performance. PBT leads to better and
more stable results and less order dependence, with the stronger model
consistently pulling the weaker one up.