Adversarial Training of Reward Models
Authors: Alexander Bukharin
Haifeng Qian
Shengyang Sun
Adithya Renduchintala
Soumye Singhal
Zhilin Wang
Oleksii Kuchaiev
Olivier Delalleau
Tuo Zhao
Venue: COLM 2025 (2025)
Keywords: reward models
robustness
RLHF
TL;DR: We propose Adv-RM, a method that automatically discovers adversarial examples in reward models and improves their robustness through adversarial training.

Abstract: Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples — responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings. We will open-source all code and data.

Published as a conference paper at COLM 2025
Adversarial Training of Reward Models
Alexander Bukharin Haifeng Qian', Shengyang Sun', Adithya Renduchintala', Soumye Singhal', Zhilin Wang', Oleksii Kuchaiev', Olivier Delalleau', Tuo Zhao? NVIDIA
Georgia Institute of Technology
Abstract
Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out- of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, novel adversarial training framework that automatically identifies adversarial examples - responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv- RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv- RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings. We will open-source all code and data.
1 Introduction
As large language models (LLMs) become increasingly capable, ensuring their alignment with human values remains a fundamental challenge (Amodei et al., 2016; Askell et al., 2021). One of the most promising approaches for scalable alignment of LLMss is reward modeling, in which reward model is trained to predict human feedback (Christiano et al., 2017; Leike et al., 2018). These reward models (RMs) are then used as a proxy for human feedback in a variety of places throughout the LLM lifecycle, including data selection (Chen et al., 2023), Reinforcement Learning from Human Feedback (RLHF, Ouyang et al. 2022), and inference time moderation (Inan et al., 2023). This approach has been very successful, resulting in powerful language models such GPT-4 and Llama (Achiam et al., 2023; Dubey et al., 2024).
Despite their widespread use in LLM alignment, RMs are imperfect proxies for human judgment (Hendrycks et al., 2021; Gao et al., 2023). In particular, they exhibit poor out-of- distribution (OOD) generalization, often failing to reliably assess prompt-respone, pairs that diverge from their training distribution (Eisenstein et al., 2023). OOD generalization is challenging because RMs learn from a limited set of human-annotated examples, which do not cover the full diversity of possible model behaviors, leading to systematic failures when encountering novel model responses. These failures lead reward hacking -hhre models trained on RMs optimize reward scores through unintended shortcuts, reducing their actual alignment with human values. Such reward hacking can result in unexpected behaviors and difficult-to-detect misalignment, an issue that becomes even more pernicious as model capabilities increase and evaluation becomes more challenging.
Building robust RMs to mitigate reward hacking presents two unsolved challenges. The first measuring& a RM's s robustness. Unlike image and speech models, where adversarial attacks can serve robustness metrics (Carlini & Wagner, 2017), the discrete nature of language

---

Published as a conference paper at COLM 2025
makes adversarial attacks on language models challenging and their robustness harder to define (Wang et al., 2021). Reward models introduce another layer of complexity, as their accuracy is measured on preference pairs, requiring adversarial preference pairs rather than a single adversarial input. So far the only known reliable way to quantify robustness of RM is to measure how long it takes for reward hacking to occur in RLHF. This in itself can be challenging: we can rely on human evaluation to tell us if the policy training has significantly degraded, but this is costly and small misalignments can easily go undetected.
The second problem is how to improve the robustness of a RM. Prior works show that reward hacking often occurs because the RM gives high reward scores model responses outside the reward model training distribution (Casper et al., 2023; Eisenstein et al., 2023). Giving high reward to OOD responses bad, as they deviate significantly from the expert responses in the RLHF dataset and are therefore likely to be low quality (Ho & Ermon, 2016). To avoid rewarding such responses, contemporary works have tried variety of approaches, including regularizing training so that the policy model remains close to its initialization (Jaques et al., 2019; Stiennon et al., 2020), penalizing out-of-distribution (OOD) model responses using uncertainty estimates (Eisenstein et al., 2023; Coste et al., 2023), and regularizing reward learning (Ibarz et al., 2018; Liu et al., 2024b). However, we find that uncertainty estimates are not reliable indicators of response quality for in-distribution responses, which may limit the effectiveness of such regularization.
In this work we propose a adversarial training framework, Adv-RM, to address both problems. Adv-RM automatically discovers adversarial responses that the targeted RM gives a high reward to while being OOD and low quality. We find these examples by training an adversarial policy using reinforcement learning to generate responses that maximize discrepancy across different RMs, while receiving high rewards from the target RM. This approach reliably finds adversarial examples for state-of-the-art (SOTA) RMs, including Skywork- Skyywork-Reard-Coo weward-Gemma-2--- et al., 2024), and Nemooroo---340FF 34wa--ewaadd (Adler et al., 2024), with over 80% attack success rates. Interestingly, we find that these popular RMs give high reward scores to a variety of low-quality responses, such as responses that have no punctuation or responses with random text in the middle of a response.
(Liu et al., 2024a),
1.-mm...-Nemotroo 700--7-a----- (Wang
Once we find adversarial samples, Ady-RM incorporates them into the RM training proce- dure. Since the success rate of our attack method is high, we can include adversarial samples in the training data set without additional human annotation cost by simply using an in- distribution response as the chosen response and the adversarial response as the rejected response. We conduct extensive evaluation of these adversarially trained RMs through adversarial attacks, downstream RLHF training in the synthetic setting of Gao et al. (2023), and RLHF training on real datasets (Wang et al., 2024). We found that Adv-RM reward models not only are more robust against attacks but also mitigate reward hacking. With these models, RLHF can proceed for three times as many steps as runs with conventional RMs without exhibiting reward hacking, resulting in more performant aligned models.
We summarize our contributions as follows:
We develop an RL-based approach that successfully generates adversarial samples for SOTA RMs. To the best of our knowledge, this is the first work do so without human domain knowledge, providing new means of evaluating RM robustness.
We use adversarial samples to augment RM training. Through extensive experiments, we find that such adversarial training results in reward models that are more robust to hacking and have better downstream performance RLHE.
2 Related Work
Mitigating Reward Hacking. There have been several works that attempt to mitigate reward hacking. Some of the earliest works used reward ensembles, and found that differ- ent ensembling approaches can slow (but not prevent) reward hacking (Coste et al., 2023; Eisenstein et al., 2023; Zhai et al., 2023; Ramé et al., 2024; Yan et al., 2024). In our experi- ments we compare with two simple reward ensemble aggregation approaches, MEAN and
2

---

Published as a conference paper at COLM 2025
(Eisenstein et al., 2023). Chen et al. (2024) propose new architecture that disentangles length from human preferences. Recently Liu et al. (2024b) proposes RRM which augments the RM training data by comparing responses from prompt with random responses from other prompts, an approach we find to be effective.
BEENNMINUS.ST
Two works discuss adversarial training of verifiers. Amodei et al. (2016) presents the idea of adversarial reward models, a training framework in which adversarial training is used to find samples where RMs and human annotators disagree. However, they do not implement this idea, and relying on human annotators in their suggested way would be expensive and difficult. Kirchner et al. (2024) propose adversarial training of verifier model but is limited to tasks with rule-based verifiers like GSM8K and is not applicable to general reward modeling- Concurrently with our work, Wu al. (2025) use manually defined input transformations to increase the difficulty of RM benchmarks.
Adversarial Attacks on LLMs. There is wide literature discussing adversarial attacks on LLMs (Zhang et al., 2020). Typically these works attempt to greatly alter the model output while only slightly changing the model input (we only black-box attacks for simplicity (Guo et al., 2019)). We compare Adv-RM with two such approaches: Textfooler (Jin et al., 2020), which replaces words with synonyms such that the model output is greatly changed and StyleAdv (Qi et al., 2021), which changes the style of the text to greatly change the model output. We observe that such approaches usually do not succesfully attack RMs as (1) these attacks were not designed with the paired nature of RM training data in mind and (2) these attacks were designed to attack weaker models such as BERT (Devlin et al., 2019).
Methodology
3.1 Background
RLHE We follow the standard RLHF setup of Stiennon et al. (2020); Ouyang et al. (2022). We consider a setting where we have a dataset of N human preference annotations, D {x,, V V Here x; is the ith prompt, Vuu is the response preferred by hu- man annotators, and yeu is the rejected response. With this dataset, conventional RLHF trains a Brrrle (Bradley & Terry, 1952) reward model (x(x, by minimizing the loss
[())= = -Ex
(x,(x, Ra(x,
(1)
Next, RLHF trains policy Thp to maximize the expected reward Ro(x,y) while remaining close to an initial policy πSFT- This is done by maximizing the following KL-regularized objective function over prompt distribution P,
E (xy))-Py--pps [Rx(x,y) - (L (+)(
using a reinforcement learning algorithm such as PPO, RLOO, or GRPO (Schulman et al., 2017; Ahmadian et al., 2024; Kool et al., 2019; Shao et al., 2024).
Distr--Dittiitttion Detection. Detecting OOD data is a long studied problem. Here consider simple approach for OOD data detection in RLHF: Ensemble disagreement TLakshminaryyanan et al., 2017). Concretely, we use the disagreement of an ensemble of reward models trained on different data splits to estimate the uncertainty of different prompt-response pairs. Such an approach has been shown to effectively capture model uncertainty and detect out-of-distribution data in wide variety of domains, including continuous control (Brantley et al., 2019), computer vision (Vyas et al., 2018), and natural language processing (Yuan et al., 2023). Most recently, Eisenstein et al. (2023) showed that reward model disagreement is an effective way to detect OOD responses in RLHE Here, we denote reward model disagreement as Ug 0x (x, = STD yx,y), y), RØ (x, y)), where Re, Røx are K different reward models trained with different data splits or random seeds. We remark that when K = 2, Ue, (x, is proportional to RR, (x,y) RØ (x,(x,)) We normalize all RMs to have mean 0 and standard deviation 1.
3

---

Published as a conference paper at COLM 2025
3.2 Learning to Attack
The goal of our adversarial attack is, given a prompt, to find responses that receive high rewards from the target reward model and yet are OOD. Given set of reward models re, Reg one reward model we seek to attack R and set of prompts D, we seek to train a policy aadv to satisfy the constrained optimization problem:
max Ex-DDy--aavxxx [U (x,y)) s.f. Ex-Day-Taaxx)) [Ra, (x,y) T(x)] Madv
(2)
Here T(x) is threshold dependent on the prompt. In our experiments we use two RMs (i.e.
K = 2) and set Txx) to be the average reward achieved by 7SFT on the prompt according
to Re, (.). Optimizing (2) means finding OOD samples that the target RM (a, (.) predicts to
be better than those generated by ÆSFF. Although this approach would indeed find highly
OOD responses that get high reward, it is possible that by optimizing (2) we would find samples that all reward models predict will receive high score, and the reward models just disagree on how high the score is. In order to encourage the found samples to be OOD and low quality, use the formulation
max E Ex-DDyyTTTat (x) [Ro, (x,y) -RRo (x,y)] s.f. ExwDyy-Tavlt (x) [Ro, (x, T(x)] (3) Raavv
Here À is hyperparameters, with À typically greater than 1. This formulation encourages finding responses with high uncertainty and a very low score from R3, (x, y), increasing the likelihood of low quality samples. We overload our notation, and denote R (x,
ARg (x,y) as
Unlike previous adversarial attacks that rely perturbations, the key innovation of our work is training an adversarial policy Tadv to generate adversarial examples. Specifically, we optimize (3) using reinforcement learning and train an adversarial policy aady to maximize the following reward:
Rady Rx Re (x,y) AR2 (x,y) if (x, > T(x)
RØ, (x,y)
otherwise
Here C is a constant that ensures the second case always gives lower reward than the first case. This reward encourages the policy to maximize RM uncertainty and find OOD samples, while ensuring they receive high reward from RS, yx, (the attack target). We maximize this reward using RLOO (Ahmadian et al., 2024) over the RLHF prompt dataset, starting from πGFT. The reinforcement learning training procedure leaves us with a set of prompt-response pairs ydv), ) generated by the adversarial policy throughout training.
Data Filtering. We filter the collected prompt-response pairs using Rø, (x, and , to only select samples where the policy was able to successfully attack R (x, Concretely, we select the adversarial dataset Dadv as
Dady {(xx Ro, (aav) > T(x) and Z(Uo, (x, Vadv)) 1.96).
Here, Z(U,,,e yx, is the z-score of the uncertainty Ug (x, y), computed over responses sampled from TSFT- A value of (U (x, 1.96 indicates that the reward uncertainty of (x, is significantly higher than the uncertainty of responses within the distribution of TSFT- This ensures each sample in Dady recieves a high reward score yet is OOD.
3.3 Preliminary Analysis: Uncertainty as Quality Signal
Using a synthetic RLHF setup (Gao et al., 2023), we investigate how and when reward models uncertainty (i.e. , (-)) correlates with response quality. Concretely, we train two reward models (Rg, (.) and RØ,) based on different random seeds preference dataset created by gold reward model, and then measure U (.) versus the gold score which measures quality. We investigate this relationship on two datasets: one dataset where the responses samples from the SFT model, and sampled from the Adv-RM model.
4

---

Published as a conference paper at COLM 2025
Gold
Gold
AdvRM
SFT Data
(a) Gold Score vs Ug (.) over (b) Gold Score vs Ug (+) over (c) RØ, (-) vs Ua (-) over SFT SFT Distribution
SFT + Adv-RM Distribution Adv-RM Distribution
Figure 1: (a) and (b) show & (-)
versus gold score. (c) shows that Ræ, is similar for the Adv-RM and SFT data, which helps isolate the relationship between Uo 0g (.) and gold score.
As we can see from Figure 1(a), uncertainty (g,,0(()) is not very correlated with the gold reward (pearson's correlation -0.05) when we sample from TSFT- This indicates that directly penalizing ensemble uncertainty during RLHF may add some noise to training. On the other hand, when use Adv-RM to generate OOD datapoints with higher uncertainty level (1(b),, we see that uncertainty is more correlated with gold score (pearson's correlation -0.70). Together, these plots suggest that ensemble uncertainty is a useful indicator of response quality when uncertainty is high (and can therefore be used to find adversarial examples), but it provides little signal for distinguishing quality among responses with low uncertainty (implying directly regularizing RM uncertainty during policy training is not ideal).
3.4 Adversarial Training of Reward Models
Our analysis shows that Adv-RM's attacks are succesful at producing adversarial responses. To increase RM robustness, we incorporate these adversarial responses into training.
Preference Pair Construction. We construct adversarial preference pairs by setting (x, VSFT) as the preferred pair and setting (x, yadv) the rejected pair. We choose SFT response ySFt such that RØ, (x, VSFT) Ey-Mgrr(x)) (x) [Re (xsF) to avoid using low quality SFT responses as the preferred response. All of these adversarial preference pairs (x, Vadv) are constructed such that the rejected response Vadv receives higher reward score from RØ, (-) than the SFT response, yet Vadv has very high Ue (.) so it is OOD and likely low quality.
Multiple Rounds of Training. Once we construct the adversarial preference pairs, we add around 1000 of them to the original RLHF preference dataset and train new RM from scratch using (1). To further improve RM robustness, we conduct several rounds of adversarial training, in each round trying to find vulnerabilities of the new robust RM. Once we find these vulnerabilities, we append them to our current robust RLHF dataset and train new RM from scratch. We find that after 2 rounds of adversarial training, our attack procedure is no longer successful. At this point we terminate training.
4 Experiments: Adversarial Attacks on Reward Models
4.1 Evaluation
We consider an adversarial attack reward model successful if it produces preference pair (x, (x, such that
Re (x, > RØ, (x, while R Rgola Ex, (x, < Rgold 1/2)-
(4)
The "gold" reward here is meant to capture the reward induced by human values. In synthetic setups we have access to the simulated gold reward model, and in real RLHF settings can approximate R Roold with LLM (Dubois et al., 2023) or human feedback.
5

---

Published as a conference paper at COLM 2025
In synthetic settings, we find (4) can sometimes be met by chance if y1 and 1/2 are similar. To mitigate this, we introduce a stricter criterion: the attack is only considered successful if
7,,, (Rg, (I, (x) > 0 and R (R (Rgola (x,))) < -1.96.
(5)
Intuitively, successful attack means that R3, y1) is above the average SFT response for the response x, while Rgola (x, y1) falls within the bottom 5% of SFT scores. When using LLM judges or humans as Rgold we only consider (4).
4.2 Experimental Setup
Synthetic RLHF setup. Following Gao et al. (2023), we evaluate Adv-RM in synthetic RLHF setup. We use Baamaa----eemotro---0-----e-e
the gold RM, and create a syn- thetic training dataset by relabelling H (Wang et al., 2024) with it. We then train proxy RMs based on -lama-3.118B-Instruue (Grattafiori et al., 2024). In this setting Re, (-) and (-) are two RMs trained on different random seeds controlling the data order. We evaluate attack success rate using the gold RM. Details can be found in Appendix C. Adv-RM adversarial policies are initialized from Llama-3.188--Instruee
Real RLHF setup. We attack three top performing RMs on RewardBench (Lambert et al., 2024), Bemeor---eeaar--Gmm--------- 4-340B-Reward. When atttackngg Reward we use Satmm-----Nemmtttoo--------- attacking Llama-31-Veemoroo--0B---www
and Nemotron- and Nemotron---34000 as R (-) in OOD detection and when we use an in-house RM as RS, (.). We then evaluate the attack success rate using (4) with human judges, Llama 405B, and Deepseek-RI (Guo et al., 2025). See Appendix G for details. Adv-RM adversarial policies are initialized from
L
1eat--eemmorron77--Reaaaaaa Rew.ar--eemaaa--------------
Success Rate: Train
Success Rate: Test
Method
Standard
Strict
Standard
Strict
RM Over-optimization
8.31 (2.44)
2.85 (1.47)
7.59 (2.34)
1.34 (1.02)
RRM
(0.00)
0 (0.00)
0 (0.00)
0 (0.00)
StyleAdv
(0.00)
(0.00)
0 (0.00)
(0.00)
Textfooler
9.57 (2.60)
(0.00)
18.8 (3.45)
0 (0.00)
Adv-RM
99.31 (0.73)
92.91 (2.27)
100 (0.00)
100 (0.00)
Table 1:
Comparison of attack success rates over 128 Helpsteer-2 prompts in the synthetic setting, with standard errors in parentheses.
Baselines. Although we are the first to adversarially attack RMs (to the best of our knowledge), we consider a few baselines that try attack LLM classifiers (Textfooler and StyleAdv), RRM (Liu et al., 2024b), and reward over-optimization baseline. As Textfooler and AdyleAdv only produce a single adversarial response y1 to a prompt x, we pair their attacks with the original response from SFT model 1/2 TSFT (x) to calculate performance in (4) and (5). For reward over-optimization, we optimize the policy for three times the number of steps achieves the best downstream performance and use the resulting policy to generate adversarial attacks. We present more details in Appendix D.
4.3 Main Results: Adversarial Attacks
Synthetic Setting. Table compares Adv-RM to other baselines. For Adv-RM we generate 4 attacks per evaluation prompt and select the attack with highest (,),,(). Adv-RM consistently finds adversarial samples, indicating its efficacy. In contrast, StyleAdv and TextFooler have low success rates despite generating 50 attacks per prompt and filtering by reward difference. These methods slightly perturb response semantics, rarely causing conflicts between the trained and gold RMs. RM overooptimization can occasionally produce successful attacks; however, we find that the over-optimized policy often generates high- quality responses with only minor quirks (e.g., excessive formatting). These responses

---

Published as a conference paper at COLM 2025
Original Query
Original Query
Original Query
Geaurate draft for blog
Did have social necworks inflaence on our work?
written declara- tion for speeding ticket in cal- ifornia court.
about AT large language models their use-cases. Divide the draft into sections with beaders subhcuders.
Adversarial Attack
Adversarial Attack
Yes, social networks have had a significant inflarnce on our work...
Adversorial Attack
A trial by written declaration in a California court a way of contesting traffic ticket with- out having to appear in court will need to fill out a form called complaint and fill out any additional forms required by the your declaration you explain why belleve you were not speeding or why you should be fined You should include evidence you have such as photos or witness state- support your The court will review your declara- may request additional information testimony After reviewing
I'd happy to help with your request. What would like me to do? Do you want me help with writing blog
Let's quote extraordinary rise in freelancing platforms. Those platforms tap into more efforts, efforts go further
AT large language models and their would you like comn with drafs of the blog post for you?
towards implemenningg bus-- Dess context. innovation.
expand rival which feel has futurre evalunte coherence
Sky Works Z-score
admit dominating margin
league allow neural system anf business flexibility downsides cost productivity personuel exact-ft government insider opportunities stimule speedy ...
Z-score:
Please me know you have any specißic questions there's any shything else I can help you with!
Nemotron 340eoon----BB--- RM Zrecor
Z-score: 1.041
Llama- Nemmotron RM score
Z-score: 0.721
(a) 2-27 (b) Llama-Nemotron-708
(c) Nemotron-4-340B
Figure 2: Adversarial examples generated by Adv-RM for top RewardBench models. The
Z-score is computed by normalizing the reward score by the average reward achieved by Llama-3.18b-IIsstruce for that prompt.
Human Evaluation
Deepseek R1
Llama 405B
Skyworks Gemma 27B
TextFooler RM Opti-OOpinnn
19.61 (5.60)
56 (4.41)
(1.24)
20 (5.66)
32 (3.99)
14 (3.00)
Adv-RM
100 (0.00)
99 (0.73)
99 (0.73)
Llama-Nemotron 70B
TextFooler RM Optimmiptnnnntiion
13.2 (4.86)
27 (3,85)
(0.88)
23.64 (6.07)
7 (2.23)
3 (1.50)
Adv-RM
83.01 (5.13)
71 (4.12)
24 (3.73)
Nemotron 340B TextFooler
17.86 (5.22)
38 (4.33)
(1.24)
RM Optimiattnnnnoioon
17.65 (5.20)
23 (3.73)
19 (3.52)
Adv-RM
78.85 (5.76)
76 (4.11)
82 (3.42)
Table Ady-RM attack effectiveness across different real RMs and judges. Values in parentheses represent standard errors.
not necessarily of lower quality than those produced by 7SFT, making them less ad adversarial in nature.
Real Reward Models. The main results when attacking Skyworks-Gemnn 27B reward, Llama-Nemotron 70B reward, and Nemotron-4 340B reward can be found in Table 2. Note only compare Adv-RM to the strongest baselines in Table 1. As reward models, generating
from Table
Ady-RM is able to succesfully attack all three state-of-the adversarial pairs that the target RM predicts as correct while the judge (humans models)
7

---

Published as a conference paper at COLM 2025
100
SW 278
(%)
Method
Success Rate
Target Model
80
Adv-RM
92.91 (2.27)
46
100
Filtering
73.27 (3.94)
NEMO 3408
Unequal RM Weights
16.41 (3.13)
53
0
100
20
Reward Threshold
0.0 (0.0)
SW 278 LN 708 NEMO 3406
Table 3: Adv-RM attack ablation in the synthetic setup.
Training Model
Figure 3: Attack transferability.
predicts as incorrect. We remark that using LLM as a judge may significantly underestimate the success rate of Adv-RM as they may also be fooled by adversarial examples.
We provide examples of these attacks in Figure 2. Interestingly, Adv-RM uncovers distinct vulnerabilities in each reward model. For Skyworks-Gemma 27B, responses that simply repeat the prompt can achieve high scores, For Llama-Nemotron 70B, nonsensical gibberish in the middle of long text can be rewarded. And for Nemotron 340B, responses that lack punctuation can receive unexpectedly high scores.
Attack Transferability. We show the transferability of attacks in Figure 3. We find that there is some weak transfer between Ady-RM attacks on the Skyworks 27B model, but no transfer between the different Nemotron RMs.
Ablation. Our ablation study in Table 3 shows that all components of Adv-RM are helpful.
Experiments: Downstream Policy Performance
Experimental Setup. For the synthetic setting, we use the RMs trained with Adv-RM in Section 4 to optimize 1iaaa331-8B-Instrucc models using RLOO. We measure performance using the Gold Reward. For the real RLHF setting, we train Llama-3.1-BB-Inttree the Beopsteer-2-Prefeeene 2.1-SB-Instruct models. We measure performance on a held out set of 128 prompts using different judge models (Llama 405B and Deepseek R1), well as -lmmooon 70B RM.
RMs on dataset (Wang et al., 2024), and use them to optimize Llama-
Baselines. We consider several baselines, including conventional RLHF (baseline), the mean of an ensemble of RMs (Ens (Mean)), the mean minus the uncertainty of ensemble of RMs (Ens (STD), also called UWO in Coste et al. (2023)), and RRM (Liu et al., 2024b). Details can be found in Appendix E.
Results: Synthetic Setting. The training curve, best reward achieved, and final reward achieved for Adv-RM policy training and other baselines can be found in Figure 4. The policies trained with Adv-RM achieves higher gold reward with minimal reward hacking, exhibits little length hacking, and performs better or as well all of the baselines. We find that Adv-RM only results in small improvement in best-case policy performance, and mostly improves robustness against reward hacking. Overall, these results show that including adversarial samples in RM training can improve downstream performance.
Results: Real RLHF Setting. The judge scores across training can be found in Figure 5. Adv-RM results in the most stable training out of any of the baselines, improving over the initial model for 2-3 times more steps than RLHF. We find that RRM quite strong baseline, but that Adv-RM's training is still much more stable (note Adv-RM and RRM are complementary approaches that can be used together). This indicates the utility of training with adversarial examples, rather than the randomly sampled responses RRM uses. Furthermore, we show the RewardBench scores achieved by Adv-RM in Table 4, where we find Adv-RM (83.99) gets slightly higher score compared to conventional training (83.29).
Robustness Against Attacks. Figure 6(a) we plot the attack success rate of Adv-RM on policies trained with multiple rounds of Adv-RM training. As can see, the models
8

---

Published as a conference paper at COLM 2025
Oge Rooord
1:0
Ruaned
Punned
Gencraton
Bosoline Adw-Fo
A00 RM
Eio (Moaa))
Bansesio
Adv RM1
Ee 30 E.verggnnc
End
(a) Gold Reward Curve (b) Generation Length
(c) Best Reward
(d) Final Reward
Figure 4: Downstream policy results in the synthetic setup. Error bars represent = one standard deviation over three random seeds.
80%
80%
-30
70%
-7.9
Wh Rate (5)
Act-RM
50%
30%
RRM
(Mean)
60%
-18.0
-7.5
20%
Ers (STD)
Beseline
600
200
280
250
-2.0
100
150
200
250
900
Traming Step
Training Step
Tramiing Step
(a) Llama 405B
(b) Deepseek-RI
(c) Llama-Nemotron RM
Figure 5: Downstream policy results with different judge models.
trained with Adv-RM become increasingly robust to such attacks, indicating that Adv-RM vulnerabilities can be mitigated with adversarial training.
Ablation. Figure 6(b), we show the final downstream policy performance of policies trained with different amounts of Adv-RM data. We find that around 1000 samples results in the best downstream performance, and that more samples actually hurts performance. We hypothesize this is due to the fact that Adv-RM data is not very diverse, adding too much such data can result in overfitting. In Figure 6(c), we show the performance with multiple rounds of Adv-RM training. In general we find two rounds leads to the best performance, and that after two rounds, there is minimal gain.
90
Attack Success Rale
Reward Score
-6
Gold Reward
-8
-10
50
AdrHM Hound
16
AdrRM Round
18
Beselie
Baseline
Round
Round
-8
100
1000
2000
100 150 200 250 300 Training Step
Training Round
Number of Adv. Samples
(a) Attack Success Rate
(b) Data Size
(c) Number of Rounds
Figure 6: Robustness of Adv-RM models and ablation study. (a) and (c) we consider the real RLHF setting, where in (b) it is the synthetic setting. For (a) Deepseek-R1 is the judge, and in (b) and (c) Llmotrnn RM is.
Conclusion and Limitations
In this paper, we introduce Adv-RM, an adversarial training framework for RMs. Ady-RM can successfully attack state-of-the-art RMs, uncovering surprising failure cases where they assign high scores to low quality responses. Incorporating Ady-RM examples into RM training significantly improves RLHF stability and performance. Although Adv-RM is

---

Published as a conference paper at COLM 2025
succesful, it has few limitations. Adv-RM relies reward ensembles, meaning adversarial responses may be missed if all RMs incorrectly predict they are high quality, highlighting the need for more robust out-of-distribution detection. Additionally, Adv-RM incurs substantial computational cost-appron aeta-appooiialllly 3x that of standard RLHF training (see Appendix but we believe this overhead is justified by the resulting gains in robustness.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xrv preprint ar arXio:2303.08774, 2023.
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint ar (ttt:2006.11004 2024.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXir:2402.147700 2024.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint ar Xic:1600606663 2016.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. ar preprint ar arXiv:2112.00861, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 (33344):344-45, 1952.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreemeen--regulaiiize imitation learning. In International Conference on Learning Representations, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. leee, 2017.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Xrv preprint ar Xxix:2307.15217 2023.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. Xiv preprint arXiv:2307.08701, 2023.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arxiv preprint arXiv:2402.07313 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. Xiv preprint ar arXiv:2310.02743, 2023.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
10

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arxiv preprint or Xio:24007 027783 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems,, 36:30039-30066 -0069,, 2023.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' D'Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. Xiv preprint ar arXiv:2312.09244 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint ar arXiv:2407. 217783 2024.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International conference on machine learning, pp- 2484-2493. PMLR, 2019.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arviv preprint #rXiv:2501.12943 2025.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. ar Xiv preprint ar arXie:2109.13916 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm- based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.00677, 2023.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. ar preprint a/X1m:1907.00456, 2019.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint ar X12.6642,6980, 2014.
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of 11m outputs. ar aiv preprint arXit:2407.1369, 2024.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! 2019.
11

---

Published as a conference paper at COLM 2025
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arviv preprint ar XtXo:22433.1778, 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. Xiv preprint arXiv:1811.07871, 2018.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojje Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skework-reward: Bag of tricks for reward modeling in llms. arv preprint wxXi:2240.18855) 2024a.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. ar Xiv preprint ar Xrv::4409.13156, 2024b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. arXiv preprint ar arXiv:2100.01139 2021.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arv preprint ar arXio:2401112187, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arviv preprint ar xrXio:1707.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXio preprint arXix:2402.0333 2024.
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, et al. Nemo-aligner. Scalable toolkit for 2024.
efficient model alignment. arXiv preprint ar orXiv:2405.004
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3022 3021, 2020.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European conference on computer vision (ECCV), pp. 550-564, 2018
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Has- san Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXit:2111.02844 2021.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksüi Kuchaiev, and Yi Dong. peeeetteer-pprffrrence: Complementing ratings with preferences. arXiv preprint ar arXiv:2410.01227, 2024.
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, and Marjan Ghazvininejad. rewordbench: Benchmarking and improving the robustness of reward models with transformed inputs. arXiv preprint arXiv:2503. 1100001..1555 11751, 2025.
12

---

Published as a conference paper at COLM 2025
Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in lims. arXiv preprint ar xix::2409.,5360, 2024.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and Ilms evaluations. Advinces in Neural Information Processing Systems, 36:58478 36:58478-58507, 2023.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penalizae reinforcement learning from human feedback with diverse reward lora ensembles. Xiv preprint ar wXXi:2401.00245 2023.
Wei Emma Zhang, Quan Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks deep-learning models in natural language processing: A survey, ACM Transactions Intelligent Systems and Technology (TIST), 11(3):1-11, 2020.
13

---

Published as a conference paper at COLM 2025
A Appendix
B Reproducibility Statement
We will release all the code and scripts used to reproduce our experiments.
C Adversarial Training Hyperparameters
For both the synthetic and real data setup we use similar training configurations. In particular, we use constant learning rate of 5e - 7 with the Adam optimizer (Kingma & Ba, 2014), use batch size of 64, and generate samples per prompt in RLOO. We fix A and tune A2 E 88 10, 12). We set C -25. For the adversarial policy, we initialize it from Ilama-3.1-BB-instruutt
For reward model training, we use the following setup: we train for one epoch, and tune the learning rate in [le- 6, 6,5e-6,76 6, 5]. We select intermediate checkpoints throughout training based on their RewardBench scores. We use batch size of 128 and evaluate performance every 10 steps.
D Adversarial Attack Baselines
Textfooler. For Textfooler, we follow the approach of Jin et al. (2020) and generate 100 adversarial attacks for each prompt-response pair in our dataset by replacing individual words with their synonyms. We then filter these attacks to only select the ones that result in the highest increase in training reward model score.
StyleAdv For StyleAdv, styles: Shakespearean English, Biblical Prose, Journalistic, Stream of Consciousness, and Satirical. This is in line with Qi et al. (2021).
Alama-3.3-7000 to rewrite text in one of the five following
RRM. For RRM, we consider the attack to be the RRM augmentation, where a response from another prompt-response pair is used as the chosen response for the current prompt.
RM over-optimization. For RM over-optimization, we train the policy for three times the number of steps that achieves the best performance. In the synthetic setup this is 150 steps (i.e. best performance is achieved at 50 steps) and in the real-data setup this is 240 steps.
E Robust Policy Optimization Baselines
Ensemble Baselines. We consider two different ensemble baselines: one where take the mean between reward models to use as the optimization target, and one where we use the mean minus the standard deviation of the reward model the optimization target. We weigh the standard deviation of the ensemble baseline by hyperparameter À that we tune in [0.1, 0.5, 1.0].
RRM. For RRM, we tune the number of augmented datapoints in [2x, 3x, 5x] the original dataset size. We then train reward models on this augmented dataset and conduct policy optimization on the reward model.
Computational Cost.
We conduct all experiments on Nvidia A100 80G GPUs. Training conventional reward models takes around 4 hours on one GPU node, and conventional policy training takes around 12 hours on 2 8 GPU nodes. Training the Adv-RM adversarial policy takes around 12 hours on on 2 8 GPU nodes. Therefore, conventional RLHF takes 224 A100 hours, while Adv-RM training (with 2 rounds) takes 672 A100 hours. We conduct all experiments in NeMo-Aligner (Shen et al., 2024).
14

---

Published as a conference paper at COLM 2025
Table 4: Reward Bench Scores
Overall
Chat
Chat-Hard
Safety
Reasoning
Baseline
0.8329
0.9721
0.6118 0.6404
0.8541
0.8938
Adv-RM
0.8399
0.9609
0.8514
0.9070
G
Evaluating Adversarial Attacks
LLM as Judge. For LLM as a judge, we use Llama 405B and Deepseek-RL, To get the preference for each adversarial input pair, we ask the LLM judge to rank them with the following prompt:
Human: For the following query to chatbot, which response is more helpful and harmless?
Query:
<prompt>
Response A: <response A>
Response B: <respense B>
FTRST provide one-sentence comparison of the two responses and explain which you feel is more he helpful and harmless. SEC- OND, new line, state only "A" "B" to indicate which response is more helpful. Your reaponse should use the for mat:
Comparison: <one-sentence comparison and explana lanation> More helpful: <"A" or "B*>
Assistant:
We make the comparison with both permutations of the preference pair, and say that one response preferred over the other if it preferred in both orders. We use a help out dataset of 128 prompts.
Human Evaluation. For human evaluation, use 4 human annotators, and assign them all preference pairs to annotate. The annotators are not aware of which responses come from different attack methods. We ask them note if one response is better than the other, or they are of similar quality. In total, the annotators annotate 50 samples for each method.
H Reward Bench Scores
We present RewardBench scores in Table 4.
I Adversarial Policy Training Curve
We present the training curve for πady in Figure As shown, Nady successfully optimizes (3) by increasing RØ, while minimizing Rgz This process leads Tadv to generate out-of- distribution (OOD) responses that receive low gold reward. Since these responses achieve high R2, despite their low gold reward, they qualify as adversarial samples, highlighting the effectiveness of Adv-RM.
15

---

Published as a conference paper at COLM 2025
M
1.5
Gold Reward
Train Reverd o
1.0
2.0
MM
0.0
-0.5
1.0
Training Staps
Steps
Training
(a) (-)
(b) Rgy()) (-)
Gold Reward
(d) Training Reward
Figure 7: Adversarial policy training curve in the synthetic setting.
16


Reviews:
[{"review_id": "nXIdAXDrP6", "reviewer": "Reviewer_3VZA", "summary": "This paper investigates the robustness of reward models (RMs), majorly exploring two questions:\n\n(i) how to measure the robustness of RMs and (ii) how to improve the robustness.\n\nThe authors formulates the robustness problem by attacking the reward model, where an adversarial policy is trained to optimize the reward gap between the target RM (R_1) and the other RM model's rating(R_2): $$ R_1 (x, y) - \\lambda R_2  (x, y)$$ satisfying $R_1(x,y) > \\tau$ and $\\tau$ is a threshold. \n\nThese pairs are filter to form an adversarial training set for enhancing the RM, and the RM is improved iteratively as the RLHF training goes.\n\nEvaluation with synthetic dataset and real-world RLHF examples show that the proposed method could identify pairs that can successfully attack current models and the adversarial training improves the robustness.\n\n\n## After Rebuttal\nThe authors response effectively addressed my concerns regarding the motivation and experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, this paper is well-written with clearly defined goal and well well-executed experimental setup.\n\n- The reward hacking problem is of great significance for LLM's alignment.\n\n- The proposed method using the gap between two reward models for identifying OOD responses is somewhat novel and and the illustrated examples seem to be interesting and transferable.", "reasons_to_reject": "- The utility of adversarial training is not clear: reward hacking is a big issue, while the investigated adversarial robustness might not. Since the responses identified are OOD, they would rarely appear during the RLHF training. Therefore, these adversarial examples are not the root of reward hacking. \n- The experimental results are mainly conducted on a small dataset consisting of 128 prompts, which is not convincing despite the clear gap. I am wondering whether the distribution would be narrow, and suggest the authors could evaluate on a larger evaluation set (> 500 prompts) for more comprehensive evaluation. The results on RewardBench highlight this concern as well.\n- Seems the baseline RRM is sufficiently good which simply samples responses from other prompts in both synthetic and real-world datasets and this renders advantages of the proposal to be unclear given the computation overhead.", "questions_to_authors": "- Line 197 states that high quality responses are adopted in the preference training against the OOD pair, would the replay of high-quality responses yield performance gain? An ablation for this might be interesting, i.e., performing SFT with those chosen pairs.\n\n- Line 518: More details are needed for the human evaluation part. How many samples went through  the human evaluation and what's the consistency between human annotators?  \n- Table 4 shows that the improvements on  rewardbench is rather marginal and  even worse on Chat and Safety, this is confusing and it indicates that the adversarial training might be useless for in real-world scenarios? Any explanations for the degraded performance and what's the performance of RRM?\n\n- As mentioned in Line 295, RRM and the proposel are complementary, would the combination leads to better results?", "extra_scores": {}}, {"review_id": "ubcI95hX0i", "reviewer": "Reviewer_qJrb", "summary": "This paper introduces Adv-RM, a novel adversarial training framework designed to improve the robustness of reward models (RMs) used in aligning LLMs via RLHF. The core problem addressed is that RMs often assign high rewards to low-quality OOD samples, leading to reward hacking where the LLM policy exploits these RM weaknesses rather than genuinely aligning with the true human preferences.\nAdv-RM tackles this by first training an adversarial policy using reinforcement learning. This policy learns to generate responses that:\n- Receive a high reward score from the target RM ($R_{θ_1}$).\n- Are considered OOD, as measured by high disagreement between the target RM ($R_{θ_1}$) and another competent RM ($R_{θ_2}$). Specifically, the adversarial policy maximizes $R_{θ1}(x,y) - λR_{θ_2}(x,y)$ while ensuring $R_{θ1}(x,y)$ is above a threshold that varies given the setting (synthetic vs real RLHF).\n- Are of low actual quality despite the high ${R_θ1}$ score.\n\nThe generated adversarial examples ($y_{adv}$) are then filtered and incorporated into the RM training data as rejected samples. They are paired with high-quality responses from a supervised fine-tuned model (y_SFT) to form new preference pairs ($y_{SFT} > y_{adv}$). Retraining the RM on this augmented dataset results in an \"Adv-RM\" that is more robust.\nThe authors demonstrate empirically that their method can effectively find vulnerabilities in state-of-the-art RMs ( Skywork-Gemma-27B, Llama-Nemotron-70B, Nemotron-340B). RMs trained with Adv-RM show increased stability during downstream RLHF, mitigating reward hacking and enabling longer, more effective policy training in both synthetic and real-data settings. While the computational cost is higher (approx. 3x standard RLHF), the gains in robustness and training stability are presented as significant.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I believe this paper warrants acceptance due to its strong contributions across several key dimensions:\n- Addresses an important and timely problem: reward hacking is a fundamental obstacle to reliable LLM alignment and improving RM robustness is necessary for progress, making this work highly relevant and impactful.\n- Novel and Principled Method: the approach of training an RL policy to generate targeted adversarial examples for RMs, by maximizing disagreement while maintaining high target RM scores, is a clever and well-reasoned advancement over simpler perturbation or random data augmentation techniques. The formulation of the adversarial reward and the subsequent data filtering are sound.\n- Strong Empirical Validation. The paper provides compelling evidence of Adv-RM's effectiveness through:\n1) Successful attack generation, demonstrates high success rates in finding vulnerabilities in SOTA RMs, supported by clear qualitative examples (e.g., Figure 2) and quantitative metrics (Tables 1 & 2) across multiple judges.\n2) Improved downstream performance, showing significant improvements in RLHF training stability (figures 4 & 5) and allowing policies to train for longer without succumbing to reward hacking, leading to more performant aligned models.\n3) Thorough evaluation, including comprehensive comparisons against relevant baselines, insightful ablation studies, and experiments in both synthetic and real-data settings.\n- Clear insights and analysis. The preliminary analysis on uncertainty as a quality signal (Section 3.3) is insightful, justifying the method's design choices. The paper clearly articulates the problem, the proposed solution, and the significance of the results.\n- Potential for broader impact and future work: The Adv-RM framework provides a new tool for probing and understanding RM weaknesses. The findings and the proposed method open avenues for further research into more robust alignment techniques. The authors' commitment to open-sourcing code and data will facilitate reproducibility and further community engagement.", "reasons_to_reject": "- Specificity of robustness gains. The paper demonstrates that Adv-RMs become more robust to the specific type of adversarial attack used to generate $y_{adv}$. While Figure 6a shows robustness against repeated attacks of the same kind, the extent to which this robustness generalizes to other unseen types of OOD inputs or different reward hacking strategies is not extensively explored. The limited attack transferability shown in Figure 3 might suggest the learned robustness is somewhat narrow. A critic might argue for more diverse robustness evaluations.\n- Dependence on the quality/nature of $R_{θ2}$. The effectiveness of the adversarial example generation depends on having a suitable $R_{θ2}$ that provides meaningful disagreement. If $R_{θ2}$ is poorly chosen (e.g., too similar, too dissimilar, or low quality itself), the generated adversarial examples might not be effective or representative of true vulnerabilities. The paper uses strong RMs for $R_{θ2}$ but doesn't deeply analyze the sensitivity to this choice, which could be a practical limitation for users without access to multiple high-quality RMs.\n- Computational cost as a significant barrier. A 3x increase in computational cost for RLHF is substantial. While the authors argue the robustness gains justify this, for many researchers or smaller labs, this could be a prohibitive barrier to adoption. The paper acknowledges this but doesn't offer clear pathways to significantly mitigate this cost while retaining most benefits, which might limit its practical impact for a broader audience.\n- Modest gains on static benchmarks / peak performance. While stability is improved, the gains in final SFT policy performance on static benchmarks (e.g., RewardBench scores in Table 4, peak Gold Reward in Figure 4c) are relatively modest and have more variance than RRM. A reviewer focused solely on pushing SOTA performance metrics might see this as a limitation, even if stability is a valuable contribution in itself. The argument could be made that the method primarily prevents degradation rather than significantly elevating best-case performance.\n- Limited exploration of \"low quality\" characterization. The paper states $y_{adv}$ are \"OOD and of low quality.\" OOD is measured, but \"low quality\" is more of an emergent property from $R_{θ1}$ liking it and $R_{θ2}$ disliking it. A deeper analysis or characterization of the types of low-quality behaviors discovered could strengthen the paper.\n\nIt's important to note that many of these points are about the degree of contribution or exploration rather than fundamental flaws.", "questions_to_authors": "- When attacking Llama-3.1-Nemotron-70B-Reward, you used an 'in-house RM' as $R_{θ2}$. Could you elaborate on its characteristics (architecture, size, training data)? More broadly, how critical is the choice of $R_{θ2}$ to the success and nature of the discovered vulnerabilities, and how sensitive are the results if $R_{θ2}$ significantly differs from $R_{θ1}$ in capacity or training?\n- The paper shows Adv-RMs become robust to your specific attack method. Have you evaluated or do you have intuitions on how well this robustness generalizes to other, unseen types of OOD inputs or different reward hacking strategies not explicitly encountered during Adv-RM training?\n- You found attacks were no longer successful after 2 rounds of adversarial training. Was this primarily because $\\pi_{adv}$ couldn't find new vulnerabilities against the hardened Adv-RM, or did the Adv-RM become so robust that even OOD samples found by $\\pi_{adv}$  no longer received high scores? What does this imply about the limits of this iterative hardening process?\n- The 3x computational cost is significant. Based on your experiments, do you have insights on whether a 'lighter' or more cost-effective version of Adv-RM could still offer a substantial portion of the robustness benefits, and what might be the trade-offs?\n- The adversarial objective $R_{θ1} - λR_{θ2}$ aims for disagreement. Beyond high $R_{θ1}$ scores and low $R_{θ2}$ scores, what specific types of 'low quality' did the generated $y_{adv}$ predominantly exhibit (e.g., factual inaccuracies, incoherence, harmfulness, stylistic issues)? Did $R_{θ2}$ consistently penalize particular failure modes that $R_{θ1}$ missed?\n- Beyond improving robustness, do you see a role for the adversarial policy and the generated examples $y{adv}$ as a systematic tool for diagnosing failure modes or 'blind spots' of existing RMs?", "extra_scores": {}}, {"review_id": "15DlD2x6dy", "reviewer": "Reviewer_ALVK", "summary": "This paper introduces an adversarial training framework called Adv-RM, which aims to enhance the robustness of reward models to prevent exploitation in reinforcement learning. By generating adversarial examples and incorporating them into the training process of reward models, the framework significantly improves the robustness of these models. The effectiveness of Adv-RM is demonstrated in both synthetic and real-data settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose that uncertainty estimation has limitations in measuring the quality of in-distribution responses, especially under low uncertainty conditions, where the quality signals it provides are very limited. This finding offers a new perspective for assessing the robustness of reward models, reveals the shortcomings of existing methods, and points the way for future research.\n2. The authors' proposed Adv-RM framework innovatively generates adversarial examples to address the vulnerability of reward models to exploitation by out-of-distribution (OOD) samples. It significantly reduces the risk of reward hacking and enhances the security of language model applications. The experimental results robustly validate its effectiveness and performance improvements.", "reasons_to_reject": "1. Although the authors have elaborated on the effectiveness of adversarial examples in theory and conducted extensive experimental validation in synthetic and real-data settings, could they consider further expanding the scope of experiments in certain specific experimental setups (for example, different types of out-of-distribution samples or more complex language models) to more comprehensively assess the robustness and applicability of the Adv-RM framework?\n2. In Figure 5(a) and Figure 6(c), the icon partially obscures the content of the image, which may affect the readability and clarity of the chart. It is recommended to make appropriate modifications to enhance the readability of the chart.", "questions_to_authors": "Please refer to \"Reasons To Reject\"", "extra_scores": {}}, {"review_id": "zzrmPxvzRz", "reviewer": "Reviewer_afoe", "summary": "This paper introduces a novel adversarial training of reward models. The methodology involves the following key steps: Adversarial Policy Training; Data Filtering, Adversarial Training, and Iterative Refinement. Using both synthetic and real data sets, authors demonstrate high attack success rate and a slightly higher score compared to conventional training.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It appears to be one of the first adv training of reward model in literature. \n\n2. The adv examples are shown to be highly effective in terms of successful rate.", "reasons_to_reject": "1. The definition of attack success is not clearly described/justified (see question 1 below).\n\n2. The definition of downstream policy performance is not entirely clear (see question 4 below)\n\n3. The real-world use cases and threat model is not clear (see question 3 below).\n\n4. The literature review is somewhat limited in scope (i.e., section on \"Adversarial Attacks on LLMs\"). For example, prompt injection attack finds applications to jailbreaking can be relevant.", "questions_to_authors": "Q1. In Eq. 5, is y_2 also involved? If not, why? Is it equivalent to replace y_1 with y_2? Why can Z-score be used with only one preference pair?  Why is Z-score cutoff set to -1.96 not even more negative values, say -2.95?  \n\nQ2. Figure 2 gives three adversarial examples from three models and describes characteristics. How are these examples be selected? Any explanation why different models learn so different adversarial examples? \n\nQ3. What are the real-world use cases of this type of attack? Any defense strategies?\n\nQ4. How is Win Rate defined in Figure 5?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors study the problem of robustness of reward models used in language model alignment, particularly robustness with respect to out of distribution/low quality inputs. The authors develop a novel adversarial training technique and demonstrate that the approach is effective across several benchmarks. Some reviewers raised concerns, particularly on the comprehensiveness of the evaluations used in the paper and demonstrating that the findings scale to larger reward and language models. These were addressed to an extent in the rebuttal, and hence I recommend acceptance, while encouraging the authors to address the concerns in the camera ready version of the paper.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept