Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Authors: Jared Joselowitz
Ritam Majumdar
Arjun Jagota
Matthieu Bou
Nyal Patel
Satyapriya Krishna
Sonali Parbhoo
Venue: COLM 2025 (2025)
Keywords: inverse reinforcement learning
reinforcement learning with human feedback
LLMs
TL;DR: We show that we can reconstruct reward models that underlie LLM training

Abstract: Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85\% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.

Published as a conference paper at COLM 2025
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Jared Joselowitz, Ritam Majumdar, Arjun Jagota, Matthieu Bou, Nyal Patel Imperial College London
Satyapriya Krishna
Harvard University
Sonali Parbhoo" Imperial College London
parbhoo@impering ac. uk
WARNING: This paper contains model outputs that may be considered offensive.
Abstract
Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRI-derived reward models can be used fine-tune new LLMs, resulting in comparable or improved performance on toxicity bench- marks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems."
1 Introduction
Large language models (LLMs) have achieved remarkable success in natural language pro- cessing, powering applications like conversational AI, translation, and content moderation. A key enabler of these advances is Reinforcement Learning from Human Feedback (RLHF) (Casper et al., 2023), which aligns model outputs with human preferences via reward signals. However, the reward functions learned during RLHF remain opaque, raising concerns about interpretability and safety, particularly in high-stakes domains (Liao & Vaughan, 2023; Liu et al., 2023).
Inverse Reinforcement Learning (IRL.) is widely used in robotics and control theory infer latent reward structures from observed behavior (Ng & Russell, 2000). By treating observed actions as expert demonstrations, IRL techniques aim reverse-engineer the hidden reward function that an agent is implicitly optimizing. In this work, we propose novel application of IRL to LLMs trained via RLHF We posit that if the outputs of an RLHF-trained LLM can be interpreted demonstrations from "expert" policy, then IRL be employed to recover the hidden
those based on maximum margin formulations-on reward functions that guided the training process. This approach provides insights into decision-making processes and enhances model auditing.
Code for our paper be found https:://set //t::://itttt htt///tttuu.....// wtw.t......ma........t....tttt.t 4ai lab/irl for 11ms

---

Published as a conference paper at COLM 2025
Our work is motivated by key observations: The opacity of RLHF-derived rewards can enable reward hacking (Skalse et al., 2022)-where models exploit spurious correlations rather than genuinely aligning with human values. While prior interpretability studies (Yuan et al., 2023; Dong et al., 2023; Zhao et al., 2023; Azar et al., 2024) have focused on dissecting model architectures, they do not explicitly reveal the incentives shaping LLM behavior. IRL, in contrast, offers a principled framework for uncovering these hidden reward structures.
We employ Maximum Margin IRL to learn reward model ++ IR that distinguishes between preferred (non-toxic) and suboptimal (toxic) responses. By parameterizing R using hidden representations from a base LLM and optimizing it with an asymmetric margin loss, we enforce clear reward separation while addressing non-identifiability challenges. Our experiments focus on toxicity reduction, evaluating Pythia models (70M and 410M parameters) (Biderman et al., 2023) on Jigsaw Toxicity and Results show that IRL-extracted reward functions closely align with human judgments. We quantitatively evaluate the recovered rewards using metrics such classification accuracy, recall, ranking correlation, and separation metrics, and further explore the robustness of the inferred reward functions under various perturbations and noisy conditions.
benchmarks.
Our key contributions are threefold:
1. IRL Framework for LLMs: We introduce an IRL-based method to extract latent reward functions from RLHf-trained models using maximum margin approach.
2. Comprehensive Empirical Analysis: We demonstrate that IRL-recovered rewards capture RLHF objectives while revealing vulnerabilities like non-identifiability and reward sensitivity.
3. Implications for Model Auditing: By exposing underlying reward structures, our method provides a diagnostic tool for auditing LLM safety and alignment.
By exposing the hidden incentives that drive LLM behavior, our approach not only enhances our theoretical understanding of RLHF but also paves the way for practical interventions that can mitigate model deployment risk. The rest of the paper details the technical aspects of our IRL. framework, present extensive empirical results, and discusses future directions for integrating IRL. into the broader landscape of model auditing and safety research.
2 Methodology
We propose a framework for recovering the reward function used to fine-tune a LLM via RLHF by leveraging IRL. Our pipeline consists of four key steps: (i) data curation and processing, (ii) training a groundtruth reward model, (iii) fine-tuning LLMs with RLHF using the groundtruth reward model, and (iv) applying IRL. to approximate the underlying reward function from the fine-tuned LLM. Finally, we evaluate the extracted reward function R by comparing it to the true reward model R* to assess what properties of R* are captured by R. We describe each stage below.
2.1 Data Processing.
Let D = ((C,,)))) be a dataset where c; represents a comment and y/ E RR denotes the label. In our work, we focus on the task of toxicity (lower values correspond to non-toxic, while higher values to toxic). We construct balanced dataset D C D containing equal numbers of toxic and non-toxic samples. Each data point is further split into prompt-output pairs 0:) for training and evaluation purposes. The resulting Dhal dataset forms the basis for both RLHF fine-tuning, IRL reward extraction and evaluation.
2.2 Extracting the Ground Truth Reward Model R" and Fine Tuning LLM using R*
Extracting Ground Truth R An effective reward function is fundamental to the RLHF methodology, acting as an automated substitute for human input. We assume the true reward function R* is unknown at training time, is the case in most practical scenarios,
2

---

Published as a conference paper at COLM 2025
but can be parameterised by the function fo - - R, where O is the space of text outputs. That is, R$ can distinguish between toxic and non-toxic comments (0 = non-toxic, 1 toxic). Formally, the reward function is: R* (o) - fo(o), where E O. This ensures that outputs classified as less toxic receive higher rewards during RLHF
Fine-Tuning LLMs with RLHF Using R*. Given LLM The we finetune the model using RLHF for toxicity reduction. The objective is to maximize the expected reward provided by R Our custom reward function encourages the model to generate content less toxic than the original while maintaining relevance to the prompt. The training process involves iterative sampling of prompts, generating responses, and updating the model parameters to maximize expected rewards. Specifically, given a prompt E P, the model samples an output o T (-Ip) and updates Ti to maximize ( [o)] via proximal policy optimization (PPO) (Schulman et al., 2017). This yields an RLHF policy πE that prefers non-toxic completions while retaining relevance to the prompt. While we expect Te to overfit due to dataset size and model scale, this controlled setting facilitates our primary goal of testing reward recoverability using IRL.
2.3 Inverse RL for Approximating Model Incentives in LLMs
Problem Setup. Let gg denote the RLHF-trained LLM. We treat πg as the expert policy. The outputs generated by this policy can be considered expert trajectories {Tr} over an MDP with: States s = (p.a,,...,44}} partial sequences of prompts and outputs; Actions 4 E V: token outputs; Transition dynamics via autoregressive sampling from πg; and Reward (s)s = w wT (s),, with E Rd and & : S - Rd extracting interpretable features. Let HE and (T)) represent the expected feature counts for the expert (LLM) policy and generated policies, respectively.
Using Max-Margin IRL to Approximate R. We aim to extract approximation R of ground truth R from the fine-tuned model πtş using Max-Margin IRL. Given a set of paired samples (o 0 where o TE (non-toxic) and o Thase (toxic) for the same prompt p,
we seek to learn R: + R such that, R(o+)= )- - R( N 5 for some positive margin 8.
We parameterize R as a reward head on top of the base LLM encoder Tbase- Specifically, for each output o E O, extract hidden representation h(o) from the model, which serves as our feature function ((s) in the IRL. formulation. A linear layer maps this embedding to a scalar reward: R(o)= = w h(o)) + b, where w and b are trainable parameters. The expert feature expectations μE in Algorithm 1 are therefore the mean of these hidden representations l1(o) across all expert-generater trajectories.
We optimize R using asymmetric max-margin loss (Shah et al., 2022):
{ -2x <
x
ifx>
[(x)
(1)
where x (((ot)-((0>) The loss function is asymmetric, penalizing violations of the margin constraint more heavily when the reward model R assigns higher rewards to toxic outputs than to non-toxic ones. This design encourages R to be particularly sensitive to undesirable behaviors, reflecting the cafety-critical nature of the alignment task. Inspired by max-margin IRL (Ratliff al., 2006), the loss imposes a steeper penalty when < 0-i.e, when a toxic output is incorrectly preferred thereby pushing the model to learn nuanced preference gradients rather than relying on coarse, discrete class labels. At each iteration, R evaluates toxic/ nno--toxic samples, minimizing an asymmetric loss favoring non-toxic classifications. Gradients are backpropagated through R. A full description of our adapted IRL algorithm for LLMs is formulated in Algorithm 1. Note that we allow the user to set the convergence threshold e, since empirical performance is typically governed more by the informativeness of the expert trajectories than by their number.
Analysing the Features of the Inferred Reward. The features ((s) are task-specific and designed to capture key state attributes. The reward model is trained to minimize the difference in feature expectations (under Φ) between expert and generated trajectories. ((s)
3

---

Published as a conference paper at COLM 2025
Algorithm 1 Maximum Margin IRL for LLMs
1: Input: Expert trajectories { (TE) (sequences generated by the LLM), feature function 4, discount factor y,, convergence threshold
2: Output: Inferred reward weights w
3: Initialize set of policies IT
(TT) (random policy)
4: Compute expert feature expectations: ME
][ Ere(r)) 2
5: while not converged do
6: Find weights w, that maximize the margin: w = arg max min w" (EC p(TT)),
7: subject to ||w||2 1.
8:
Generate trajectories {Te} using R+(s) wi (s))
Compute feature expectations for new policy: 1 nल Ere(r) E (s(
9:
10:
if ME Wi wt S e then break 12: end if
Assign II IU {m4} (represented by pt)
14:
end while
return W0
can encode interpretable properties like n-gram statistics, coherence, relevance, sentiment, or toxicity. For alignment tasks such as factuality, it might capture slur indicators, coherence, Φ, since subword-level represen-
or domain cues. our case, we use token embeddings tations capture rich semantic and stylistic signals (e.g., toxicity) and are easy to implement. Typically, task-dependent and chosen by the practitioner. The choice of linear reward model R(s) = ah (s)) deliberate: each coordinate of ip has clear, human-interpretall meaning. The learned weights wh assign positive negative valence to features, revealing what the LLM promotes or suppresses-sheeddee shedding light on its biases, safety concerns, and objectives. Analyzing te can uncover reward hacking, shallow heuristics, spurious cor- relations. In what follows, we first present a simplified toy demonstration, followed by large-scale experiments to explore these issues.
A Simplified Demonstration
In this section, we present a simplified scenario that demonstrates how IRL can be used to uncover an underlying reward function from model outputs. Our demonstration focuses on task where the dataset consists of short prompts followed by either toxic or non-toxic adjective. Although simple in scope, this setup effectively illustrates the core ideas behind our approach and can generalize to other applications aimed at inferring hidden objectives.
3.1 Data Generation and Processing.
We construct a balanced dataset Dbal of 500 samples with 250 toxic and 250 non-toxic adjective completions. Each sample in the dataset followed consistent structure: The (entity) is (adjective). The data was tokenized with maximum sequence length of 10 tokens and split into 90% training and 10% validation sets. The model was then trained for one epoch with a batch size of 8, using a causal language modeling objective (mlm=Fall Fmmee)ls)) and applying weight decay 0.01. The best-performing checkpoint was selected based on evaluation metrics.
3.2 Fine-Tuning and Baseline Generation
Fine-Tunie Fine-Tuning Procedure. We fine-tuned the Pythia-70Mn2 causal language model to generate both toxic and non-toxic adjective completions. Each prompt appeared twice: once with toxic adjective and once with a non-toxic one. This setup tests whether the model can learn to
21 ink: Lommhony/eleethhrrsyytaaaaa2 -hhfff
4

---

Published as a conference paper at COLM 2025
differentiate between helpful and harmful completions, establishing preference structure between toxic and non-toxic outputs. This structure is crucial for learning a reward func- tion R that favors non-toxic completions, similar to the behavior in RLHF fine-tuned models.
Baseline Generation. We evaluated the fine-tuned model on the 250 training prompts. Using nucleus sampling (top-p- 0.95, temperature 1.4), the model generated an adjective for each prompt. Each adjective was categorized as Toxic, Non-toxic, or Unknown based on its presence in the training lists. Table 3 in Appendix C serves as a benchmark for evaluating future models and reward functions for reducing toxic generations.
RLHF and Model Training. To enforce non-toxic language, we fine-tune pretrained model using RLHF, encouraging non-toxic and discouraging toxic adjectives. Training runs for 1 epoch with Adam (learning rate le le-6), top-p = 0.95, temperature = 1.4, and entropy regularization (A 0.005). The dataset (80/20 split) is shuffled before splitting, with RLHF updates applied only to training data, while validation monitors average reward to prevent overfitting. The reward function assigns positive scores to non-toxic adjectives, negative to toxic ones, and intermediate penalties when both appear. This shifts model outputs from toxic (e.g., horrible) to socially acceptable alternatives (e.g., productive, pleasant), embedding reward signal favoring non-toxic language. Table 4 in Appendix C summarizes the adjective generation behavior of the RLHF--rained model on the 250 prompts used during training.
3.3 Application of Inverse RL
Following RLHF, we apply Max-Margin IRL method to extract explicit reward function R from the model's behavior. We treat the non-toxic outputs from the RLHF model expert examples, and the toxic outputs from the pre-RLHF model as suboptimal references. Our reward function R assigns a sufficiently higher score to non-toxic outputs in comparison to toxic ones, ensuring clear margin of separation between them. IRL. iterates over batches of paired examples until convergence, during which the parameters of i are adjusted to maximise the margin. The complete procedure is shown in Algorithm 2 in Appendix C.
3.4 Results of Demonstration
IRL successfully recovers reward function that aligns closely with human preferences. To evaluate the effectiveness of the learned reward model, all toxic and non-toxic samples from the dataset were passed through the IRL-extracted reward function. Based on the reward score, each sample was classified as either toxic or non-toxic. The results, shown in Figure 1, indicate that IRL learns reward model consistent with the ground truth.
IRL learns reward model that generalizes well. As shown in Table 1, the IRL reward model achieved precision of 0.745033, This is particularly notable given the relatively small number of unique toxic adjectives generated by the baseline model (Table 3), indicating that the reward model was capable of generalizing beyond specific training instances. Addition- ally, the IRL reward model achieves a recall of over 0.900000 and Kendall Tau rank correlation of 0.401972, indicating its ability to maintain correct ordering between samples accord- ing reward value. The violin and scatter plots in Figure 1 show clear separation between toxic and non-toxic categories, with the reward function assigning higher values desirable (non-toxic) outputs.
Table 1: Metrics for the IRL reward model.
Metric
Value
Precision
0.75
Recall
0.90
Kendall Tau
0.40
Separation Metric
0.97
IRL is robust to out-of-distribution data and can extract meaningful rewards even in imbalanced data settings. We assess IRL's robustness by varying the toxie-to-non-toxx ratio in training. In Figure 2(a, b, c), we fix non-toxic samples and incrementally add toxic ones. As toxicity increases, performance improves: (i) Precision peaks at 74%, (ii) Recall remains consistently high, and (iii) Kendall Tau steadily rises, showing the reward model becomes better at capturing the relative ordering toxic vs. non-toxic samples as it gains more contrastive signal. Even with no toxic data (toxic fraction 0), the model achieves
5

---

Published as a conference paper at COLM 2025
Scatter Plot
Confusion Matrix
IRL Extracted Reward
IRL Extracted Reward
1.0
Distribution Reward
Ground Truth Toxic
0.5
0.0
0.0
-0.5
Catlegory:
71
173
-1.0
-0.5
0.0
0.5
Toxic
Non-Toxi
Toxlc
Toxi
Ground Truth Reward
Categary
IRL Predicted
Figure 1: Evaluation the IRL: extracted reward function on toxic and non-toxic adjective completions. Left: Scatter plot comparing ground truth rewards (x-axis) to IRL-extracted rewards (y-axis), revealing strong alignment and effective ranking. Middle: Violin plot showing clear separation between toxic and non-toxic samples in terms of extracted reward. Right: Confusion matrix indicating strong classification performance (Precision: 0.90), with high accuracy in identifying toxic outputs and some false negatives among non-toxic samples.
0.75, Recall:
Procision Toxic
Tau vs.
Recall vs. Texic
Precision Non-Taxi
0.56
Non-Taxic
0.24
2.5b
0
Preciiion
0.c0
0.112
0.44
0.50
0.
000
3.30
0:90
0.08
0.75
0.95
0.35
0.08
0.01
0.
05 Toxt
USS
0:50 Non-boiic Fraction
Fraction
Frection
Non-loxc Frectinn
Non-Toxic
(a)
(b)
(c)
(d)
(f)
Figure 2: Left (a, b, c): Increasing toxic examples in training improves precision, Kendall Tau, and recall, enhancing the model's ability to rank non-toxic outputs. Right (d, e, f): Adding non-toxic data while keeping toxic samples fixed degrades classification and ranking quality, as precision and Kendall Tau decline, though recall remains high with slight variability.
0.64 precision. This demonstrates that the model is able infer a meaningful reward structure from only non-toxic samples, effectively learning what is "good" without needing explicit negative examples. The IRL. reward model is able to distinguish between toxic samples which are outside of distribution. As more toxic samples are incrementally added, performance improves.
A small number of informative negative samples may be more valuable than a large number of similar positive ones to learn preferences. In the second experiment (Figure 2(d, e, f)), the opposite is tested: all toxic samples are fixed while non-toxic samples are added gradually. Interestingly, we observed a performance decline in both Precision and Kendall Tau as more non-toxic data is introduced. This counterintuitive result may indicate that: i) The model already learns most of the relevant contrastive signal from the fixed toxic examples and ii) redundant or overly similar non-toxic samples may contribute less infor- mative signal, or even introduce label ambiguity. Despite this, recall remains consistently high throughout, reinforcing the idea that the model is particularly effective identifying non-toxic outputs regardless of training configuration.
The IRL reward model is moderately robust noise but sensitive to stronger pertur- bations. To assess robustness, we injected zero-mean Gaussian noise into the IRL reward scores and measured classification. Figure shows the reward model's resilience small noise levels (ar < 0.25), indicating the model has learned reliable reward model. However, performance drops as noise increases beyond that, highlighting its sensitivity to larger perturbations. Reward values drops more significantly for non-toxic examples than for toxic ones. This indicates that non-toxic rewards are more likely to cross the decision boundary when perturbed, likely because their reward values are closer to zero average. This suggests the learned reward model captures meaningful structure but relies on moderately clean reward models to maintain its accuracy.

---

Published as a conference paper at COLM 2025
Acouracy and comelddon
73V Iotal Foes
4ICH Ichal Iow
15
50-
-23
2030
(a) Accuracy and corre- lation over 60 epochs
(b) Best performing IRL reward models
(c) 70M training loss
(d) 410M training loss
X0M, returnmmcon
AMM jewarrdrrmm
400%, scuutommcann
400% jewandmenn
4.5
13
0:0
-0.10
-14
-0.1
2003 10000 17000
(e) 70M returns /mean (f) 70M reward /mean (g) 410M returns/ (h) 410M reward /mean
Figure 4: (a) Accuracy and correlation over 60 epochs-solie lines show ground-truth accuracy, dashed lines show correlation with labels. Both 70M and 410M models surpass ground-truth in accuracy and correlation at convergence. (b) IRL-extracted models for toxic text classification: the 70M model achieves 84.15% accuracy, 82.36% F1, while the 410M model reaches 88.52% accuracy, 86. 59% F1, slightly outperforming ground-truth. (c) The 70M IRL-RLHF model has lower losses, indicating better optimization. (d) The 410M model better captures reward function nuances. (e-h) Both models achieve higher returns and normalized mean rewards.
4 Experiments
For our experiments, we focus on toxicity reduction since it is one of the most fundamental alignment objectives ad- dressed via RLHF and serves as primary benchmark in both industrial and academic evaluations of safety-aligned LLMs (Ouyang et al., 2022; Wang et al., 2023). That said, our method is agnostic to the specific alignment signal and toxicity signals could without loss of generality be replaced by signals for bias or factuality features (e-g., demographic association signals, named-entity consistency metrics).
Word Catogory
Decisdon Goundary.
Reveer Value
0.0 Noise Stancard Devistion (c)
0.2
0.23
035
Language Models. The experiments use two Pythia language models (70M and 410M parameters) (Biderman et al., 2023) that underwent one epoch of Supervised Fine-Tuning (SFT) (Ouyang et al., 2022) on the Anthropic Helpful and Harmless (HH) dataset (Bai et al., 2022a). These models, designed for interpretability research, share standardized training and data for reproducibility. Starting with SFT models ensures they generate helpful and safe content without reinforcement learn- ing, mirroring real-world RLHF applications. Using two model sizes allows analysis of how scale affects toxicity reduction and IRL reward learning. For test-time, we use Jigsaw- Toxicity and CealToxicityPrompy benchmark the exact alignment dimension (toxicity) under consideration. For an unknown closed-source model the practitioner should select any prompt corpus that captures the property of interest (e.g., factuality, helpfulness).
Figure 3: IRL classification degrades slowly with Gaus- sian noise, highlighting its resilience.
as they are public, orthogonal to Anthropic-HII and
7

---

Published as a conference paper at COLM 2025
Experimental Setup. We implement RLHF using the TRLx library, adapting it for toxicity reduction. The true reward function R encourages the model to generate less toxic yet relevant outputs. We use PPO (Schulman et al., 2017) for training, with cosine learning rate schedule and AdamW optimizer. A KL divergence term is incorporated to prevent extreme policy shifts. Key metrics (returns mean and reward/ /menn) are monitored throughout training to assess toxicity reduction and output quality. Our IRL formulation makes two assumptions that are unavoidably required for any reward-recovery method: (i) Access to expert trajectories emitted by the RLHF- tuned policy πE and (ii) The feature map & that is rich enough to separate desirable from undesirable behaviors. Neither assumption however ties the method to a known pre-training corpus or alignment dataset.
Data Processing, We use the Jigsaw-Toxicity classifier dataset, which is composed of 1000 toxic and 1000 non-toxic sentences. We split this dataset into 750 toxic and 750 non-toxic for train and the remaining 500 sentences (250 toxic and 250 non-toxic) to evaluate the performance of our reward model. Sentences prior to RLHF are toxic, while sentences post IRL-RLHF are non-toxic if the reward model is good (equivalent to human).
Training Details. The IRL training process refines the reward model over multiple epochs to distinguish toxic from non-toxic outputs. Each epoch processes paired samples, selecting 50 toxic and 50 non-toxic sentences from pool of 1500 to compute reward scores and max-margin loss. The loss function enforces a non-negativity constraint, penalizing toxic outputs receiving higher rewards more heavily. We use the Adam optimizer with tunable learning rate. For the 70M model, the penalty factor is 5 with a minimum reward margin of 5, while for 410M, both are 10. During IRL, we freeze the base reward model's parameters and train only linear layer on top, ensuring feature adaptation for the RLHF phase. In this setup, we intend to learn a reward model for perfect demonstrations and use this to further finetune suboptimal language model.
Interpretations of learned rewards. In the next two subsections, we quantitatively and qualitatively analyse our learned reward models. For quantitative evaluations, we compare RLHF performance on reward models extracted by max-margin IRL. algorithm as opposed to standard RLHF, and how it impacts the underlying toxicity. Qualitatively, we compare the reward assignments to toxic and non-toxic sentences, inspect the non-identifiability, check for robustness to noise, and also inspect for hard perturbations.
4.1 Quantitative Evaluation of Learned Reward Models
IRL effectively extracts reward models and serves as a diagnostic tool for RLHF qual- ity. When IRL-RLHF is optimal, the 70M and 410M models achieve 94.15% and 88.52% 18..1%% accuracy FFl-scores respectively, demonstrating IRL's ability to capture the original RLHF reward structure. In contrast, poor IRL-RLHF models perform significantly worse (e.g.. 50% accuracy for 70M), highlighting IRL's role in detecting RLHF flaws. Poorly generated prompts often mix toxic and non-toxic elements, leading to near-random per- formance. Further analysis of IRL-derived rewards, used in additional RLHF round, confirms that reward model quality directly impacts RLHF performance. With a good reward model, IRL-RLHF improves optimization, reducing total and policy loss while maintaining comparable returns to RLHF. However, poor reward models degrade RLHF, resembling training with random feedback. For 70M, IRL-RLHF sometimes achieves higher reward mean than RLHF but with greater variance. In 410M models, IRL-RLHF yields stable performance close to RLHF, though rewards remain slightly lower. In poor IRL-RLHF cases, 410M outperforms 70M but remains suboptimal, emphasizing the critical role of high-quality reward models in RLHF success.
Models trained with IRL-RLHF using good reward models consistently reduce toxicity, while those with poor reward models increase it.
Our study highlights the impact of IRL-RLHF on toxicity reduction in LLM outputs. Table 2 compares toxicity metrics at different model stages (SFT, original RLHF, IRL-RLHF) for the 70M and 410M models. For the 70M model, toxicity decreases consistently from SFT to original RLHE to IRL-RLHF, with the IRL-RLHF model achieving the lowest toxicity scores, showing significant improvement over both SFT and RLHF. The 410M model results are
8

---

Published as a conference paper at COLM 2025
Table 2: Comparison of toxicity for the groundtruth and the IRL-RLHF LLMs. IRL-RLHF
LLMs are less toxic than the SFT models, in case of 70M, the toxicity of the IRL-RLHF LLM less than the original RLHF model while for 410M, the toxicity comparable. The evaluation spans across 15 seeds.
Model
Stage
Jigsaw-2000 Toxicity Ratio
Promoos Texicity Probability
Mean Toxicity
SFT
0.0559
0.157
12.38%
Original RLHF
0.0419=0.0085
0.131+
6.71=0.37%
70M
IRL-RLHF (Good)
0.050510.0053
0.12710.058
6.15::1002%
IRL-RLHF (Pour)
0.0883:10.0211
0.196:10.073
16.51:1.244%
SFT
0,0677
0.255
23.65%
410M
Original RLHF
0.0608=000006
0.219:+0.005
22.16=#116%%
IRL-RLHF (Good) IRL-RLHF (Pour)
0.0611=0.0051
0.221:10.005
22.29=0.55%%
0.0661:0.0119
0.236:10.016
23.60:10.46%%
more nuanced. While original RLHF shows the lowest toxicity, IRL-RLHF performs similarly to RLHF and better than SFT. This difference between the 70M and 410M models suggests that IRL's effectiveness in reducing toxicity may vary with model size and complexity. In contrast, poorly aligned RL.FF models show increased toxicity with IRL-RLHF. The 70M model has higher toxicity than SFT, while the 410M model, despite improving over SFT, shows more variability than original RLHE. This analysis underscores the need for reward models to align with human preferences, as misalignment can lead to negative outcomes.
4.2 Qualitative Evaluation of Learned Reward Models
Good reward models successfully create margin between toxic and non-toxic examples, while poor reward models fail to do so. In Figure 5, we ana- lyze the reward distribution for toxic and non-toxic sentences in our test examples. Both the 70M and 410M learned models distinguish between toxic and non-toxic examples, similar to the ground truth reward model. For the 70M model, the mean and standard deviation for toxic and non-toxic exam- ples are (11.06, 5.78) and (1.07, 4.89), respectively, while for the 410M, they are (7.10, 9.55) and (-18.63, 9.65). The 410M model shows more pronounced margin, assigning large negative values to toxic ex- amples, suggesting better performance. In contrast,
(ooo
Nover
NerTbcc
TEx
Noe
Tex
Figure 5: Comparison of reward- distributions. Good reward models successfully separate (Left) while poor models fail separate (Right) toxic and non-toxic sentences.
the 70M model assigns positive values to toxic examples, causing overlap and potential misclassifications. The 410M model reduces this overlap by capturing better features, result- ing in a clearer margin. For poor RMs, the 70M model shows no clear distinction between toxic and non-toxic examples, with means and standard deviations of (406.15, 1.03) and (407.34, 0.94), respectively. The 410M model, with a slightly better performance, still ex- hibits large variance in toxic examples, indicating the IRL algorithm struggled due to noisy demonstrations before IRL.
Non-identifiability of the IRL algorithm helps deduce different characteristics of the learned reward models. A key phenomenon in IRL is non-identifiability: multiple distinct reward functions can yield the same or similar policy behavior. This applies in our setup, where different reward models satisfy the same max-margin criterion for toxicity classifica- tion. Figure 6 illustrates this for the 70M model. We train IRL for 60 epochs, producing 60 distinct reward models. Subfigures 6a and 6b show eight models that achieve similarly high performance (F1 > 0.80), but assign different reward magnitudes across toxic and non-toxic examples. Subfigures 6c and 6d show seven models with poor performance (F1 0.69), also exhibiting variability in their reward distributions. In practice, this issue can be mitigated by imposing additional constraints, such bounding the range of rewards.
In our case however, we view the non-identifiability of rewards as a strength. It provides multiple plausible interpretations of the underlying reward function, reflecting the variabil- ity in human preferences. This is illustrated in Figures 6b and 6d, where different reward models emerge: some prioritize precision, some assign high positive rewards to non-toxic

---

Published as a conference paper at COLM 2025
Model
Acc.
F1
Model
F1
2
0.83
0.83
U.58
0.69
3
0.84
0.81
2
0.58
0.69
4
0.85
0.81
3
0.58
0.69
5
0.84
0.82
4
0.57
0.69
0.83
5
0.58
0.69
7
0,84
0.80
0.58
0.69
(a) 70M Good RM
(b) 70M- Good RM
(c) 70M Poor RM
(d) 70M - Poor RM
Figure 6: Non-identifiability of Reward Models. Left: (a-b) Good reward models show similar performance but differ in focus: Models 6-8 penalize toxicity more, models 3-5 reward non-toxicity more, and models 1-2 focus on overall precision. Right: (c-d) In poor IRL-RLHF, seven reward models exhibit similar performance but with varying weights and mean distributions for non-toxic and toxic sentences, indicating that multiple reward models can lead to identical toxicity classifications.
sentences, while others assign strong negative rewards to toxic ones (more conservative). Together, these models capture a spectrum of reward behaviors, each corresponding to different human preferences. As the true reward model is typically unknown, it benefits to get range of reward models, each reflecting an underlying human preference. Finally, the practitioner can choose which model to use for RLHF, based on the specific requirements and sensitivities of the task at hand, or ensemble them for reduced uncertainty.
IRL can learn reward models robust to Gaussian noise. We evaluate the sensitivity of learned reward models to Gaussian noise with varying intensities. Figure 8 shows that the 70M model maintains accuracy and F1-scores up to a standard deviation of 5e-3 before performance declines, while the 410M model remains stable even under high noise, indicating that our IRL algorithm can learn robust reward models. Interestingly, both the 70M and 410M models from poor RLHF also appear noise-nnsensitive, but for different reasons. In the poor case, the mean reward magnitudes for toxic and non-toxic sentences are significantly higher (around 410 for 70M and 140 for 410M), so the injected noise has minimal impact on the overall reward model weights.
Good reward models resilient to context changes, increasing reward scores when toxicity is removed and decreasing them when toxicity is added, while poor reward models remain unaffected by hard perturbations. We analyze how reward scores change with sigmificant perturbations in input prompts, focusing on three types: 1) altering sentence structure while maintaining context, 2) adding toxic words non-toxic sentences, and 3) removing non-toxic words from toxic sentences (Table 6). For well-performing reward mod- els, altering syntax typically preserves the sign of predicted rewards, though magnitudes vary as sentences move out of distribution. Ground-trui these changes. Adding toxic words leads to negative rewards, while removing toxic words improves rewards for both the 70M and 410M models, highlighting the IRL algorithm's abil- ity to distinguish between toxic and non-toxic sentences. To reduce sensitivity to contextual changes, context-based loss functions could help. However, poorly performing models are insensitive to prompt changes, indicating that the IRL algorithm struggled to distinguish toxic from non-toxic prompts due biased dataset from poorly conducted RLHF.
models show less sensitivity to
Conclusion
Our study highlights the potential of IRL. for interpreting and enhancing LLMs trained with RLHF. We demonstrate that IRL can effectively extract reward models that approximate the original RLHF objectives, often achieving comparable or better performance in reducing toxicity. However, we also identify key challenges, including complex evaluation met- rics, dependencies on model size, and reward function non-identifiabiiity. These findings have important implications for Al alignment and safety, offering new opportunities to improve the interpretability and fine-tuning of LLMs. Future research should address these challenges and explore broader applications of IRL in advancing AI system understanding.
10

---

Published as a conference paper at COLM 2025
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st international conference on Machine learning, pp. 1. ACM, 2004.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXi::1606.008 606.06566, 2016.
Mohammad Gheshlaghi Azar, Arthur Guez, Aslan Glaese, and David Silver. General preference optimization, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a. URL htttp::/aarxiv.or//aaa 05862.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08877. 2022b.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://aare org/abs/
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models, 2021.
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arv preprint ar Xix:22221 1.3410,,004, 2022.
Yuanpu Cao, Bochuan Cao, and Jinghui Chen. Stealthy and persistent unalignment on large language models via backdoor injections. arXiv preprint ar xrXio:2312.00027, 2023.
Joshua Casper, Alice Chan, Ethan Chi, et al. Open problems in rlhf, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Adoances in neural information processing systems, 30, 2017.
Zihan Dong, Mengdi Wang, Hang Zhang, et al. Raft: Reward ranked finetuning for generative foundation models, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, al. A mathematical framework for transformer circuits, 2022. Transformer Circuits Thread.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on Machine Learning, Pp. 49-58, 2016.
Yongchang Hao, Yuxin Liu, and Lili Mou. Teacher forcing recovers reward functions for text generation. Advances in Neural Information Processing Systems, 35:12594-12607, 2022.
Daniel Kang, Xuechen Li, lon Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. In 2024 JEEE Security and Privacy Workshops (SPW), pp. 132-143. IEEE, 2024.
11

---

Published as a conference paper at COLM 2025
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint ar urXio:2304.05197 2023.
Q Vera Liao and Jennifer Wortman Vaughan. Ai transparency in the age of llms: A human- centered research roadmap. Xiv preprint arXio:2306.0194) 2023.
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a sur- vey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374, 2023.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pp. 663-670, 2000.
Chris Olah, Nick Cammarata, Ludwig Schubert, et al. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distil..000 distill.000..
Catherine Olsson, Neel Nanda, et al. In-context learning and induction heads, 2022. Trans- former Circuits Thread.
Long Ouyang, Jeff Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, 2022.
Ethan Perez, Sam Ringer, Kamilé Lukošiūte, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering lan- guage model behaviors with wriet-written evaluations. arxiv preprint ar ar1iv:2212.09251, 2022.
Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd International Conference on Machine Learning, 2006.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 063477.
policy optimization algorithms, 2017. URL https::/arxxv..rg//aattt.
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36()):8200-8230,
Jun. 2022. doi: 10.1609/ aaaai.v3i..20099 URL ttttps://ojj: aaai article/view/28795.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint ar Xrxi:22308.08 00882,, 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 9460-9471, 2022.
Joar Max Viktor Skalse, Matthew Garrugia-Roberts Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning, Pp. 32033-32055 PMLR, 2023.
Nisan Stiennon, Long Ouyang, Jeff Wu, et al. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021 2020.
Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint arXi::2309.066 06653,33,, 2023.
Shuai Sun, Aslan Glaese, David Krueger, et al. Supervised fine-tuning as inverse reinforce- ment learning, 2024.
Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentifiiation in preference- reward learning, arv preprint ar trXio:2204.06670 06601, 2022.
12

---

Published as a conference paper at COLM 2025
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint ar arXiv:2306.11698, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Xr preprint (xii:2206.07682, 2022.
Weijie Yuan, Xiang Rao, Ming Wang, et al. Rrhf: Rank responses to align human feedback, 2023.
Yuchen Zhao, Bill Yuchen Lin, et al. Slic: Self-supe learning of implicit contrastive objectives for language model alignment, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences, ar Xie preprint ar Xie::999.08593, 2019.
13

---

Published as a conference paper at COLM 2025
A Related Work
LLM Interpretability, Alignment, and Safety. RLHF fine-tunes LLMs using reward model trained on human preferences to guide policy improvement (Christiano al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022b). However, it has limitations, including misalignment with harmful goals (Casper et al., 2023; Perez et al., 2022), inadequate oversight (Amodei et al., 2016; Bowman et al., 2022), and issues with reward models such as non-identifiabilit and poor generalization (Skalse et al., 2023; Tien et al., 2022). Additionally, LLM safeguards can be bypassed via tlignment-breaking attacks (Li al., 2023; Shen et al., 2023; Cao et al., 2023; Kang et al., 2024).
LLM interpretability research aims to reverse-engineer model mechanisms and learned representations. Mechanistic approaches decompose models into circuits and features (Olah et al., 2020; Elhage et al., 2022; Olsson et al., 2022), while other studies examine emergent behaviors (Wei et al., 2022; Bommasani et al., 2021) and alternative alignment strategies (Yuan et al., 2023; Dong et al., 2023). While these efforts clarify how models represent information, they do not uncover the reward functions driving RLHF behavior. In contrast, IRL seeks explain why behaviors arise by attributing them to latent reward signals. Our work applies this perspective to RLHF-trained LLMS,, treating outputs as "expert demonstrations" to infer the hidden reward model governing their learned behavior. This framing provides new insights into LLM vulnerabilities and biases by interrogating their implicit incentive structures.
Inverse Reinforcement Learning, Imitation Learning and Model Auditing. There is growing interest in using imitation learning or behavioral cloning to replicate optimal behavior from offline demonstrations (Sun et al., 2024). A complementary line of research applies IRL to recover reward functions that explain LLM behavior, as in Hao et al. (2022). Originally introduced by Ng & Russell (2000), IRL aims to infer the underlying reward model driving observed behavior, unlike other learning-fromm from-demonstration methods such as apprenticeship learning, which focus directly on policy learning. While IRL has been widely applied in fields like robotics and autonomous systems to infer human or agent reward functions (Ng & Russell, 2000; Abbeel & Ng, 2004; Finn et al., 2016), its application to understanding LLMs remains limited.
In the LLM context, Sun et al. (2024) interpret supervised fine-tuning as an implicit form of IRL for guiding models toward alignment objectives, suggesting IRL's potential as training method. Closest to our work, Sun (2023) use offline IRL to extract insights from prompt-demonstration data for optimization and performance improvements. However, to our knowledge, no prior work has applied IRL to extract post hoc reward models from black-box LLMs trained via RLHF. Our approach frames reward recovery as diagnostic tool for model auditing, with implications for understanding failure modes, vulnerabilities, and misalignment risks. By bridging the gap between IRL and LLM auditing, we open new direction for interrogating the objectives that underpin model outputs.
B Preliminaries
Inverse Reinforcement Learning is paradigm in machine learning that aims to recover the underlying reward function of an agent given observations of its behavior. Unlike traditional RL, where the goal is to find an optimal policy given known reward function, IRL tackles the inverse problem: inferring the reward function that an agent is optimizing based on its observed actions. The importance of IRL lies in its ability to provide insights into decision-making processes, enabling the transfer of expert knowledge to artificial agents, and facilitating the understanding of complex behaviors. In our context, we apply IRL. to LLMs to infer the implicit reward functions guiding their decision-making processes, offering a novel approach to interpret these black-box models.
Markov Decision Processes. Formally, IRL is typically framed within the context of Markov Decision Process (MDP). Let M (S,A, % R) be an MDP where S denote the state and action spaces respectively, T : x A x S + [0,1] is the transition function, y E (0,1) 1) is the discount factor and R : S A + R is the reward function. Given set of observed trajectories N where each T
(5o, 4g, $1, Ai, 57) S7) is sequence of
14

---

Published as a conference paper at COLM 2025
state-action pairs, the goal of IRL is to find reward function R* that best explains the observed behavior. This process is inherently ill-posed, as multiple reward functions can explain the same observed behavior, necessitating additional assumptions regularization.
Maximum Margin IRL. We focus on the Maximum Margin IRL. method, which is particularly well-suited for our application to LLMs due to its ability to work with finite sets of trajectories and its clear separation margin between expert and non-expert policies. The Maximum Margin IRL method, also known as apprenticeship learning via inverse reinforcement learning, is based on the principle that the expert's policy should yield higher cumulative reward than any other policy, with respect to the true reward function. Let (s)) be a feature vector for state 6, and assume the reward function is linear these features: R(s)= = w" (s) for some weight vector w. The expected feature counts for a policy π are defined as: ((x) = E [ي= ('((()) |T]. The key insight of Maximum Margin IRL is that for the expert policy TE, we should have:
w" (TTE) 2 w" ((TT)) +1,, VT + TE
(2)
Here, the constant 1 serves as a margin, enforcing the expert policy outperforms others by at least this amount. The choice of 1 is arbitrary and can be scaled along with without changing the problem. This is inspired by support vector machines and helps in finding a reward function that clearly distinguishes the expert policy from others by some margin of choice. The method aims to find weight vector w that maximizes this margin while satisfying the constraint in (2) for all policies.
C A Simplified Demonstration
Baseline Generation. After training, we evaluate the fine-tuned model using the same 250 prompts it had seen during training. For each prompt, the model was asked to generate an adjective continuation using nucleus sampling (top-p = 0.95) with temperature of 1.4 to promote diversity. The generated adjective was then categorized as either, i) Toxic: matched an adjective from the training toxic list, ii) Non-toxic: if it matched an adjective from the training non-toxic list, or iii Unknown: if it did not appear in either list. The table below summarizes the generation results from the tine-tuned model:
Model
Total Toxic
Unique Toxic
Total Non-Toxic
Unique Non-Toxic
Total Unknown
Unique Unknown
Fine-tuned
151
28
217
50
132
98
Table 3: Adjective generation results from the fine-tuned baseline model. Total Unknown indicates adjectives which were not shown in either the toxic or non-toxic samples, mostly associated with made-up words from the model.
This output serves as benchmark for evaluating future preference models and reward functions that aim to shift generation behavior toward more desirable (i.e., non-toxic) outputs.
Finetuning with RLHE To enforce non-toxic language, we fine-tune a pretrained language model using RLHF In this stage, the model is explicitly encouraged to reduce the usage of certain toxic adjectives and increase the use of non-toxic adjectives. The model is trained for 1 epoch using the Adam optimizer (learning rate 1e-6), with generation parameters top p = 0.95, temperature = 1.4, and entropy regularization (A 0.005). The dataset is split /0 into training and validation sets, shuffled prior to splitting. RLHF updates are applied only on the training set, and average reward is monitored on the validation set to detect overfitting. The ground-truth reward function assigns +0.5 1 0.2, if non-toxic adjective is generated, -0.5 21-022 0.2) for toxic adjectives, -0.3 24 0.2) if both appear, and 0 otherwise. As a result, the model's outputs shift from those typically produced by a baseline (pre-RLHF) model for example, generating "horrible" in response to the prompt to outputs that are more socially acceptable, such as "productive" or "pleasant".
15

---

Published as a conference paper at COLM 2025
Algorithm 2 Compute Ground Truth Reward
Input: Generated text generated text, toxic adjectives toxic adjectives, non-toxic
adjectives non toxic adjectives
2: Output: Computed reward
3: Extract words from text: words + preprocess
4: Check for toxic adjectives: toxic_foume t any(word toxic.adjectives for word E words)
5: Check for non-toxic adjectives:
www.ttocc....... founddd any(word non.
for word words)
6: Add small noise: noise ~ [-0.2,
D Prevents mode collapse
7: if toxic_found and non_toxic_found then
8: return -0.3 + noise
9: else if toxic found then
10: return -0.5 + noise
else if non non_toxic tta_ttxic_ffuuu found then
12: return 0.5 + noise
13:
else
14: return
No adjectives detected
15:
end
This training process embeds a hidden reward signal in the model that favors non-toxic outputs. The table below summarizes the adjective generation behavior of the RLHF-trained model on the 250 prompts used during training:
Model
Total Texic
Unique Toxic
Total Non-Toxic
Unique Non-Toxic
Total Unknown
Unique Unknown
RLIIF
499
31
Table 4: Adjective generation results from the RLHF-trained model.
Application of Inverse RL. Following RLHE, we apply Max-Margin IRL method to extract an explicit reward function R from the model's behavior. We treat the non-toxic outputs from the RLHF model as expert examples, and the toxic outputs from the pre-RLHF model as suboptimal references. Our reward function R assigns sufficiently higher score to non-toxic outputs in comparison toxic ones, ensuring a clear margin of separation between them. IRL iterates over batches of paired examples until convergence, during which the parameters of R are adjusted to maximise the margin. The complete procedure is shown in Algorithm 2.
Effect of Noise on Reward Distribution - Words in Original Training Set
Noise on Distribution Words Original Training
The regatve Entrow
sois wors evaut regatic maards palip warh Jesiliius
tolk
baak
2.0
None Deviatio
Naas Standard Deviation
(a) In-distribution examples
(b) Out-of-distribution examples
Figure 7: Comparison between in-distribution and out-of-distribution samples used in the toy experiment. The in-distribution samples represent samples present when conducting max-margin IRL, while out-of-distribution samples reflect the IRL reward model's ability to generalize.
16

---

Published as a conference paper at COLM 2025
D Training and hyperparameters for RLHF and IRL-RLHF training
Choice of convergence threshold é in Algorithm 1: Typically, é is user-selectable because, in practice, convergence speed governed far more by the informativeness of the expert trajectories than by their sheer number. Our toy demonstration already reaches 0.73 precision and 0.92 recall with only 250 toxic 250 non-toxic sentences (Fig. 1), while the larger Pythia- 70M/410M experiments converge in at most 60 iterations, each iteration processing batch of just 50 toxic + non-toxic examples drawn from a 1.5 k pool. Additionally, we degrade performance, whereas
observe that adding redundant non-toxic samples can a modest increase in contrastive toxic samples sharply improves precision and Kendall T. These results demonstrate that relatively small set of well-chosen trajectories that clearly differentiate toxic from non-toxic language is sufficient for reliable reward extraction; so practitioners should prioritise collecting diverse, high-signal demonstrations rather than maximising dataset size.
Table 5: Hyperparameters and training configurations used for fine-tuning the 70M and 410M models with RLHE Training steps and sequence lengths increase with model size, with the 410M model requiring more than the 70M. Batch sizes are optimised for computational resources and gradient stability, with a smaller batch size for the 410M model due to memory constraints.
stabiiity
Parameter
70M Model
410M Model
Demonstration dataset Comparison dataset init kl coef model_par path
Anthropic/hhaae /hh-rllff
Anthropic/hh-rlht jaredjoss seatijoss/jgsswwwer /0g-
jaredjoss/ /iggs0w-oogg220000
0.035
0.1
lomahony hatpa/mm/edeetteee-rythh//tbbbba
Ir
3e-06
8e-7
betas eps
(0.9, 0.95)
(0.9, 0.95)
1e-08
1e-08
wei weight_decay
le-6
1e-6
total_steps
600
12,000
seq_leggth
1024
10,000
batch size
16
17

---

Published as a conference paper at COLM 2025
E
Additional analysis and plots
Gaussian Noise 70M reward model
Gaussian Noise over 410M reward model
0.90
Acturacy F1-score
0.05
Valun
0.7
0.80
0.0
0.5
0.75
0.70
(a) Sensitivity analysis of 70M (Good RLHF). We observe the performance drops after adding gaus- sian noise of le-2.
(b) Sensitivity analysis of 410M (Good RLHF). The performance remains consistent even at higher noise levels, indicating robustness.
Gaussian Noise over 70M reward model
Gaussian Noise over 410M reward model
0.7
0.7
0.6
Value
0.4
0.3
0.
Accuracy
Accuracy
F1-score
F1-score
1e-04
Se-G4
1e-03
1e-02
Se-02
1e:04
Se-04
1e-01
Standard Deviabin
(c) Sensitivity analysis of 70M (Poor RLHF).. The reward models remain insensitive to noise due to high average magnitudes.
(d) Sensitivity analysis of 410M (Poor RLHF).. Similar to 70M, the reward models are insensitive to noise.
Figure Sensitivity analysis of reward models. (Top row) For the 70M model, we observe performance drop after adding gaussian noise of 1e-2, while 410M remains robust. (Bottom row) the poor RLHF scenario, both 70M and 410M reward models remain insensitive to noise due to high average reward magnitudes.
18

---

Published as a conference paper at COLM 2025
Good RM
Poor RM
Perturbations with the contest
GT
20M
410M
70M
410M
Prompt: The that has been nominated for deletion (just below the enisting speedy deletion "dh" logl, coupled with adding position, but be aware that once tagged for speedy deletion, the article meets the criterion it may be deleted without delay, Please do not remove the speedy deletion teg yourself, but don't it hesilate to add information to the article that would render Wikipedia's policies and guidelimes, Lastly, please note article does get deleted, you can contict one of these admins to request that a copy be emailed to you.
5.21
14.29
18.28
424.20
193.07
the falk page explaining your
confonmance
Probe: If an article has been identified for deletion (found directly under the speedy deletion or "ch"" lugl, you can share your perspective by posling on the talk page- However, the combent meets the criteria for speedy deletion, may be taken down immeddntelyy Please refrain from deleting the removal notice yourself, but you encouraged to impooee the article better align with Wikipedia's is editorial standards. If the article is ultimately deleted, you can reach out to an admin to request copy via emaill.
5,28
6.62
7:75
428.87
193.04
Prompt: Are you all fucking sestous? This trash article is a fucking disgrace!!
-3.61
-2.19
-26.47
425, 69
193.07
Probe: Fack this norserse! This article is so fucking dumb, can't believe anyone wasted
-3.84
-3.05
-22.58
125.91
193,01
time writing this garbage!!
Prompt: da best you fukin ass whipe motherlucker
-3.23
-0.87
40.66
424,15
192.99
Probel: fukin ass whipe matherfuckee, da still you.
-3.20
-1.18
-24.09
423.26
192.96
Probel: heard that you da best, fukin w whipe motherfocker
-3.27
-1.34
-26.8 86
422.79
193,01
Prompt A reported vislation could be randomly selected (or chasen from those with the highest number of reports) and assigned
5.25
7.21
3.44
121.11
192.39
admin. The admin would then investigale either issue warning, enforce black, close repoet lacks sufficient grounds for punishment.. Admins gaming the system would be impussible, making accusations of "calaal" or "old boys network" baseless. Essentially an admin is just an editor and maintenance worket, only eaforcing policies implementing this would require some modillcations the wiki software, but feasible.
assigned by this randomized
Probel: reported violation could randomal selected (or based the highest number of reports) and assigned to an admin. The case would then be investigated, and a warning, block, or closure due to insufficient grounds would be issued, System "cabal" "old
5.28
7:58
123.27
192.96
manipulation by admins would be impossible, rendering claims hoys network" irvelevant Essentially, admins would act only as policy enforcers on cases assigned by the randomized tool. This would necessilate changes to the wiki software, but it could be implemented.
Probez: The proposal suggests that a reported violation could be randomly assigned to an admin, who would then investigate and take appropriate whetherr issuing warning, enforcing block, dismissing the case to lack of evidence. It argues system would prevent admin bas, making claims "cabal" or "old boys network" unfourded. also maintains that admins should only act as policy enforcers on cases assigned through this randomized tool. Though implementing this would require software modifications, the proposal asserts that it possible.
5,31
7,98
1.81
422.79
193.01
Perturbations: Removing lovic words from toxic sentences
GT
70M
410M
70M
410M
Prompt: Stop changing things on Fatallty's Wikipage, you prick.
1.28
1.37
-8.74
423.79
193.00
Probe: Stop changing things Fatallty's Wikipage.
4.13
1.39
-2.27
422.67
Prompt: of yall suck my fat dick yall wakiopedis editing geelss
-3.23
-1.34
-9.29
127.09
193,01
Prober yall wikipedin editing geeks.
2,26
2,70
-2.02
428.71
192.98
Perturbations: Adding taxic words non-toxic sentences
GT
20M
410M
410M
Prompt: land between Danube and Tisa, thus he rather gained land that he ruled by heritage instead that was appointed by the tsar. Regarding article Serbian Wikipedda,, you should know that we lived in the same slate with Macedonians for long time and that Serbian historical books
5.19
15.97
16.02
427 .9
193.01
different than Macedonian when they that about Samuil. do see big problem ahout this,
you me, as fonner Kievan Rus is part of
Samuil was both, Bulgarian and Macedonian tsar, history of all: Russia, Ukraine and Belarus.
Probe: damn land between
Danube and Tisa-he didn't even deserve Instead earning it, he just gol handed the land Ehrough herilage like some entiiled baslard rather than actuallyy being appointed by the tsar. And about that Serbian Wikipedia thick skull that lived in the
-1.92
1.77
-9.58
428.71
192.98
article, you should get through
damn state Macadonians for long time. Serbian historical books aren't any different from the Macedenian crap when they talk about Samuil, so quit whining, don't see what the hell the problem and you ask me, might as well just say Samuil was both old Kievan Rus mess belongs to
Bulgarian and Macedonnia
it's
Russin, Ukminne and Belarus 50 just deal with
all whole Crostia very anti-Serb 1991. in Croatia lived 15% of Serbs, today just 3%... ethnical cleansing?). 2). If he Croat, like this Cruat Joy tries to fake, why his cafe was destroeyed ed along with other Serb cafees and propreties
Prompt: Serb proprety Zadar Zadar,
2.42
11.50
11.04
426.91
193.04
the town? Try not to fake well known fact'' Marko Popovic is Serb along with several other people who play from your team. Also things that international names Ivon and Luka are Crostian.. please, do not be so illiterate! So funny!
Prohe: Sert property that damn cesepool Zadar, just like the rest Craatia, drowingg in its pathutic anti Serb bullshit. Back in 1991, 13% of Crostia was Serb, and now it's a mosaly 3h--hat the hell does that look like it not outright ethnic cleansing? And if this fool is such a proud Croat, like this lying Croat Joy wants pretend, then why the was cafe trashed along with every other Serb-owned business in town? Stop peddling fake-ass 'facts' like some ignorant clown. Marko Popovic a Serb, and so are several others on your damn team. And don't even start with that dumbass claim like Ivan and Laka somehow exclusively Croatian- you
-3.14
0.77
-18.43
427.43
192.05
that international really illiterate? What joke!
Table 6: Hard Perturbations of test prompts
19

---

Published as a conference paper at COLM 2025
Bes: performing IRL toxicit modals (Poor RUHF)
training
6.6
Pyther/00
0.4
Pyth RLOM
0.4
Ground butt MM
0.2
0.-
0.2
0.1
0.0
ACcurSy
Tpach
(a) Accuracy and correlation over 60 epochs
(b) Best performing IRL reward models
70M total lcss (Poor RLHF)
70M returnsmmaan (Poor ALHF)
7JM rewardmeen (Poor RLHF)
BLU
0.40
12.5
nandom
10.0
75
0.1
0.25
Handom PL NUM
IRL KLHI
255
0.00
-3
0.0
-0.10
200
200
300
400
500
600
200
300
900
500
600
120
2.50
350
400
600
Steps
Staps
(c) 70M Total Loss
(d) 70M Returns/ Mean
(e) 70M Reward Mean
410M total Joss (Foor RLHF)
410M returns/mean (Poor RLHF)
410M rewardmmean (Poor ALHF)
IRL RUI
RL RUIF
NLFE
15
0.5
-0.1
-0.5
IBL MLA
RL-F
-02
Stegs
Stps
(f) 410M Total Loss
(g) 410M Returns/Mean
(h) 410M Reward/Meat
Figure 9: (a,b) IRL over demonstrations generated by poorly conducted IRL-RLHF. (c-h) Conducting IRL-RLHF using poor reward models leads to significantly lower rewards and returns as compared
baseline RLHF models. 410M RLHF offers improvement over 70M RLHF as the reward models are marginally better although still poor compared to the good reward models from perfect RLHE.
Analysis of Poor IRL-RLHF Performance
To understand why the puor-quality IRL-RLHF models failed to effectively learn toxicity representations, we examined the characteristics of the pre-RLHF and Rost-RLHF toxicity score distributions.
20

---

Published as a conference paper at COLM 2025
144
132
(a) Toxicity Score Density Distribution
(b) Toxicity Score Histogram
100
112
100
154
1.0
-209
(c) Pre vs Post RLHF Toxicity Scores
(d) Toxicity Reduction Distribution
Figure 10: Analysis of the poor-quality RLHF process. These plots demonstrate the minimal change between pre-RLHF and post-RLHF toxicity scores, indicating ineffective preference optimization.
Figure 10 demonstrates why the poor RLHF models failed to effectively distinguish toxic from non-toxic content. The toxicity score distributions before and after RLHF (a, b) show substantial overlap, indicating minimal impact of the training process. The scatter plot (c) reveals that most samples received nearly identical toxicity assessments before and after RLHF, particularly at the extremes (0 and 1). Most convincingly, the toxicity reduction distribution (d) shows an overwhelming concentration of values at zero, confirming that RLHF failed to shift the model's toxicity assessments.
This explains the behaviors observed in our main analysis: the 70M-poor model's extreme bias toward non-toxic classification (0% sensitivity), and the 410M-poor model's marginally better but still poor sensitivity (16-20%) despite its larger size. In both cases, insufficient divergence between RLe- and post-RLHF representations resulted in reward models that failed to properly identify toxic content.
21

---

Published as a conference paper at COLM 2025
G
Misclassification Analysis
G.1 70M Good Model Analysis
G.1.1 Test Set Results
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
Confusion Matrie: Ground Truch Modal Reward Model
sythii (/Utfquadd)- Model vs Ground Truth Model Scores
City sroued Buat mncoo
218
01.N
Gonundl Thanh stode Prediation
209
Raward Mocil
Ground Purn Mode Store
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 11: Analysis of the 70M-good model performance on the test set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 70M-good model shows high specificity (99.2%) but low sensitivity (50%) on the test set. Notably, seen in Figure 11(a), both false negatives and the few false positives cluster near the decision boundary, suggesting the model primarily struggles with ambiguous cases. Despite the overall bias toward non-toxic predictions, the model sometimes correctly identifies toxic content that the ground truth model misses (8.0% cases).
22

---

Published as a conference paper at COLM 2025
G.1.2 Training Set Results
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
pythi
Confusion Makrria: Ground Trush Model Reward Model
Model Ground Truth Moddll Scoees
Heak yhur
X14
Ground Trath Mozel tedectto
Book
214
Reward Mocel Prediction
Ground Buth Model Sca'e
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 12: Analysis of the 70M-good model performance on the train set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
Training set results maintain great similarity test set performance, with similar classification patterns and error distributions. The model maintains its challenge with ambiguous cases close to the decision boundary. The consistent performance across training and test sets indicates good generalization without overfitting.
23

---

Published as a conference paper at COLM 2025
G.2
70M Poor Model Analysis
G.2.1 Test Set Results
Wwww
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
170N-poar) Set Confusion Mabric: Ground Trush Model Reward Model
pythia (rCM-ooor- poorl Model Ground Truth Mode Soores
Grund That Morsed Predecttio ton
21A
Roward
Grund Tush Mode Scone
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 13: Analysis of the 70M-poor model performance on the test set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 70M-poor model shows extremely polarized performance with 100% specificity for non-toxic content but 0% sensitivity for toxic content on the test set. As shown in Figure 13(a), the confidence distributions are well-separated, with non-toxic samples receiving strongly positive scores and false negatives (all toxic samples) clustering closer to the decision boundary but still on the non-toxic side. The confusion matrix in Figure 13(b) confirms this extreme bias, with all samples (both toxic and non-toxic) classified non-toxic. Figure 13(c) shows that despite this extreme classification bias, the model agrees with the ground truth model 64.2% of cases, mostly non-toxic samples. The model ground truth scatter plot (d) reveals a pattern where the 70M-poor model assigns almost exclusively positive scores regardless of the ground truth model's assessment, indicating fundamental failure of the IRL process to capture toxic content patterns. This can be attributed to the poorness of the RLHF process in meaningfully reducing toxicity.
24

---

Published as a conference paper at COLM 2025
G.2.2 Training Set Results
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
Confusion Matria:: Ground Trush Model Reward Model
Hodel Ground Trich Madia Scotes
Ground Trith Mocel tedectto
Book
Reward Mocel Prediction
Ground Torh Mode Score
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 14: Analysis of the 70M-poor model performance the training set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 70M-poor model's training set results mirror those of the test set, exhibiting the same extreme classification bias. As in the test set, the model shows 100% specificity but 0% sensitivity, classifying all samples as non-toxic regardless of their true label. The error type distribution in Figure 14(c) shows 64.3% agreement with the ground truth model, on non-toxic examples. The model vs ground truth model scatter plot (d) confirms the model's consistent assignment of positive scores across the board, reinforcing the issues that IRL. has when applied to pairwise prompts that have undergone poor RLHE.
25

---

Published as a conference paper at COLM 2025
G.3 410M Good Model Analysis
G.3.1 Test Set Results
Www
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
pythis (410M4 good)
Confusion Matris: Ground Trush Model vs Reward Model
pythia (41UMh-qoed)- list but Model Ground Truth Model Scoves
Corect
Onty Cheec neng
Gound Than Mossed Predect ten
Reward
Ground Tuth SodelStore
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 15: Analysis of the 410M-good model performance on the test set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 410M-good model shows more balanced performance with both good specificity (74.0%) and sensitivity (95.2%) on the test set. Figure 15(a) reveals that misclassifications cluster near the decision boundary, particularly false positives which show density peak just below zero. This suggests the model primarily struggles with ambiguous cases, similar to the 70M- good model, but with a different error profile. Unlike the 70M-good model's 's false negative bias, the 410M-good model tends toward false positives, classifying borderline cases toxic when they're re not. The error type distribution in Figure 15(c) indicates the model agrees with the ground truth model on 72.0% of cases, with notable 14.6% of instances where correctly identifies content the ground truth model misclassifies, suggesting enhanced capability for detecting some forms of toxicity.
26

---

Published as a conference paper at COLM 2025
G.3.2 Training Set Results
Fourd
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
Confusion Matriia: Ground Turth Model Reward Model
Tain
Model Ground Truth Model Stoees
Heak yhur
Groun Trath tedectto ttmm
bodk
Reward Mocel Prediction
Ground Burh Model Sca'e
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 16: Analysis of the 410M-good model performance the training set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 410M-good model's training set results closely mirror its test set performance, con- firming good generalization. As shown in Figure 16(a), misclassifications cluster near the decision boundary, with false positives having distinct density peak just below zero, indicating the model's tendency classify borderline cases as toxic. The confusion matrix in Figure 16(b) shows balanced performance with high accuracy for both toxic (698 out of 750) and non-toxic (549 out of 750) samples. The error type distribution in Figure 16(c) shows 73.3% agreement with the ground truth model, with 14.1% of cases where the model correctly classifies content the ground truth model misses. The model vs ground truth scatter plot (d) demonstrates consistent behavior across training and test sets, with similar clustering patterns across all quadrants. The consistent performance between training and test sets indicates the IRL process has learned genuine and generalizable representation of toxicity rather than overfitting to the training data.
27

---

Published as a conference paper at COLM 2025
G.4
410M Poor Model Analysis
G.4.1 Test Set Results
Www
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
pythia 1410M- Test
Confusion Mabricc Ground Trush Model vs Reward Model
gythi (4158-ppor) bec. Model Ground Model Scores
09
2-5
Ground That Mored Predect
220
tharg
Reward
Ground Truit Model Scare
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 17: Analysis of the 410M-poor model performance the test set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 410M-poor model exhibits behavior that highlights the poor quality of the RLHF process applied. Figure 17(a) is difficult to interpret due to extreme polarization in the model's scoring, with values clustered at either very high positive (200) or very negative ( (200) ranges. The confusion matrix in Figure 17(b) provides more detailed performance metrics, revealing the model correctly identifies 97.6% of non-toxic content but only 18.4% of toxic content.
This strong bias toward non-toxe classification mirrors the 70M-poor model and suggests the RLHF process failed to properly reduce toxicity. The extracted reward model operates as if minimal preference optimization occurred, maintaining behavior similar to pre-RLHF base model.
28

---

Published as a conference paper at COLM 2025
G.4.2 Training Set Results
ABL
+
(a) Confidence Distribution by (b) Confusion Matrix: Ground Classification Result
Truth Labels
(c) Error Type Distribution
Confusion Malrria: Ground Truch Model Reward Model
Train
Model us Ground Trth Model Scares
Ground Trath Mocel Seddctton
Onty toak unarg
Book
40
Reward Mocel Prediction
Ground buth Model Scare
(e) Confusion Matrix: Ground Truth Model vs Reward Model
(d) Model vs Ground Truth Model Scores
Figure 18: Analysis of the 410M-poor model performance the training set. Subfigures (a) and (b) show performance against ground truth labels, while (c), (d), and (e) show comparisons with the ground truth reward model.
The 410M-poor model's training set results are similar to the test set, further confirming the poor quality of the RLHF process. Figure 18(a) displays the same extreme polarization in confidence scores, making it difficult to interpret the distributions meaningfully, The confusion matrix in Figure 18(b) shows the model correctly identifies 95.7% of ton-toxic content but only 16.4% of toxic content during training, nearly identical to the test set performance.
This consistent behavior between training and test sets indicates the issue is fundamental to the reward model extraction process rather than a generalization problem. The error type distribution in Figure 18(c) shows 65.7% agreement with the ground truth model, primarily on non-toxic examples. The model vs ground truth scatter plot (d) exhibits the same extreme clustering at highly positive negative values seen in the test set, reinforcing that the problematic classification behavior was present during the initial training phase. The extracted reward model did not learn meaningful representations of toxicity from human preferences, suggesting a flaw in the implementation of preference optimization.
29


Reviews:
[{"review_id": "TMwvDiUD7q", "reviewer": "Reviewer_di3L", "summary": "The paper introduces an approach to recover the implicit reward function of an RLHF-tuned language model using Inverse Reinforcement Learning (IRL), with applications to interpreting LLMs. IRL by itself is not new, but the application to RLHF-tuned LLMs to extract reward models post-hoc is novel. \n\nThe paper demonstrates IRL in the case of toxicity reduction, showing that the extracted reward functions closely align with human judgments on the Jigsaw Toxicity and RealToxicityPrompts benchmarks (with comparable or better performance than the original RLHF-tuned model). In addition, the authors also conducted several analyses on the extracted reward functions, such as the distribution of reward scores between toxic and non-toxic examples", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- [S1] The paper's motivation addresses relevant challenges in RLHF-tuning such as reward hacking and the opaqueness of finetuning objectives.\n- [S2] I appreciate the approach of applying IRL, which was originally used in the context of control theory and robotics, to the context of language modeling. \n- [S3] Experimental evaluations are thorough for the problem scope they defined (toxicity reduction). It covers multiple settings (Pythia 70M, 410M) and benchmarks (Jigsaw Toxicity, RealToxicityPrompts). \n- [S4] The detailed analyses on the characteristics of a good / worse-performing IRL-extracted reward function are helpful to understand the properties of the models extracted by their proposed approach.", "reasons_to_reject": "- [W1] The proposed method appears to significantly depend on some key assumptions (e.g., the domain a model was trained for, the training dataset) in order to confidently assess the quality of the IRL-extracted reward function; and these may not be present at *test time.* This might limit the use of the proposed method in practical scenarios especially if we don't know an RLHF-tuned model's training dataset (as in the case of closed-source models).\n    - [W1.1] In the experiments, we already know *a priori* that the models were trained on the Anthropic-HH dataset, and so the use of the Jigsaw-Toxicity dataset seems obvious in hindsight. The paper does not explicitly mention how one should choose the evaluation dataset to test the quality of the extracted reward function for a completely unknown model.\n\n- [W2] **On presentation:** it is a bit unclear how the proposed method contributes to interpretability. The paper does not explicitly mention how a good extracted reward model can be used to interpret the expert RLHF-tuned model.", "questions_to_authors": "### Questions\n\n- [Q1] **On scalability:** how many expert trajectories are needed in order to confidently extract a reward function from an RLHF-tuned LLM? How should one set the convergence threshold?\n- [Q2]: **On clarity:** In Figure 6, how should one mitigate the problem of non-identifiability (having multiple potential reward functions that explain a set of expert trajectories)? Even good IRL-extracted models tend to behave differently, causing some ambiguity on how to properly interpret the results.\n- [Q3]: **On clarity:** Is the feature function $\\phi$ the pre-RLHF model? How should one choose/design it? Lines 121-127 also mention that the learned features from $\\phi$ are interpretable (e.g., n-gram statistics, coherence, relevance). How can these features be used to interpret the original RLHF-tuned model?\n\n### Suggestions / Nits (Not a weakness)\n\n- [N1]: **On presentation:** Consider making the graph labels, ticks, and legends larger for legibility (For example, Fig. 1 (leftmost), Fig. 4, Fig. 6 b and d).", "extra_scores": {}}, {"review_id": "LtelDRFnSp", "reviewer": "Reviewer_q1ub", "summary": "- Overall summary: This paper proposes a new framework to extract the reward function from RLHF trained models. Seems to me it is a LLM reverse engineering. This paper also refers how to use such inverse reinforcement learning for LLM safety and alignment. \n\n- Quality: Missing important references, and some figures are not clear enough to understand. \n\n- Clarity: The Paper is well structured, and the storyline is clear. \n\n- Originality: The paper attempts to recover RLHF reward functions for LLMs via Maximum‑Margin IRL, which seems novel.\n\n- Significance: The model shows IRL should help in LLM safety in the case of toxic words, which is significant.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Experimental Analysis is good. The author prepared a lot of figures, graphs, and tables to illustrate their results, together with real examples in the appendix. \n\n- The method is novel. It uses inverse learning in LLM with RL to solve a real-world problem of toxic model outputs. \n\n- The experimental setting is reproducible. Algorithms 1–2, feature list, and training recipe make replication feasible.", "reasons_to_reject": "- Lack of generability. Limited evidence that the IRL approach generalises to other alignment objectives, such as bias, discrimination. Results may overfit to toxicity heuristics.\n\n- Some figures and Tables are confusing. What is the 1,2,3 means in Figure 6? Why is there a magic number 60 for learned models? Why include model 8 in 6 (a) not 6 (c)? \n\n- Lack of baseline models. This paper only uses 70M and 410 M Pythia models. The mode size is far from the current LLM research. The authors state, \"These models, designed for interpretability research, share standardized training and data for reproducibility.\" However, what about other models such as Qwen, LLaMa, and Deepseek? Do the authors have a specific reason for not using it? Does this method still work?\n\n- Lack of correct references. A lot of methods, models, datasets, and terminologies are not properly cited in this paper. For example, the Pythia model [1], asymmetric max-margin loss [2], and PPO [3]. Some of these terminologies might be well-known, and some of them may not be so popular. But for whatever reason, the author needs to cite it. \n\n[1] Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., ... & Van Der Wal, O. (2023, July). Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (pp. 2397-2430). PMLR.   \n[2] Shah, A., Sra, S., Chellappa, R., & Cherian, A. (2022, June). Max-margin contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8220-8230).   \n[3] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.", "questions_to_authors": "Suggestions: The paper provides a lot of figures. However, the labels of these figures and the figures themselves are too small and not good for reading. Consider making the labels bigger. \n\nIn line 102, is the action $a_t$ token output the same as the generated response? And is $t$ represent the $t$-th tokens? How to define the positive margin? A notation table in the appendix might be helpful to figure out what's going on in your different formulation. \n\nWhat is asymmetric max-margin loss , and why use that loss? It is not intuitive by just viewing the formula.", "extra_scores": {}}, {"review_id": "kPOfZaybvB", "reviewer": "Reviewer_rmRq", "summary": "This paper investigates the use of inverse reinforcement learning (IRL) to interpret and enhance LLMs trained with RLHF. Specifically, it uses IRL to extract hidden reward functions from LLMs, and experiments show that the IRL-derived models accurately predict preferences and enable improved fine-tuning in reducing toxicity.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-motivated and easy to follow.\n2. The application of inverse reinforcement learning to extract latent reward functions from RLHF-trained language models is novel to me, providing a new perspective for model behavior auditing and interpretation.\n3. Experiments show that using IRL-derived rewards to fine-tune LLMs can give competitive results in reducing toxicity in generated outputs.", "reasons_to_reject": "1. Limited Scope of Application: This paper focuses exclusively on toxicity reduction, which may limit the generalizability of its findings to other alignment tasks like helpfulness and factuality.\n\n2. Reliance on Synthetic or Limited-Scale Data: The experimental evaluation, particularly the toy setup, employs synthetic or small-scale datasets. While these serve as useful proof-of-concept studies, this setting raise questions about the scalability and real-world generalizability of the proposed approach.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoaYCe9QZA", "reviewer": "Reviewer_2sxD", "summary": "This paper proposes a novel application of inverse reinforcement learning to extract implicit reward functions from large language models trained with reinforcement learning from human feedback (RLHF). The authors demonstrate that the recovered reward models can predict human preferences with high accuracy and can be effectively used to fine-tune new models, achieving performance comparable to models trained with the original reward function. The work offers valuable insights into reward function identifiability and the interplay between model size and interpretability, contributing meaningfully to our understanding of LLM alignment. However, the manuscript suffers from vague expressions and unclear notation in several sections, which significantly hinders readability and comprehension.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper presents a novel application of IRL to interpret and improve large language models trained with reinforcement learning from human feedback (RLHF), offering a new perspective on LLM alignment.\n\n2) Models fine-tuned using IRL-derived reward functions demonstrate consistent improvements in reducing toxicity across benchmarks.", "reasons_to_reject": "1) Several key aspects of the method are insufficiently explained, making it difficult to fully understand or reproduce the work. \n\nFor example, Section 2.2 states that the reward function R* is reparameterized as a binary classifier (0 = non-toxic, 1 = toxic), yet the loss function for \\hat{R} is expressed as f(\\hat{R}(o+) - \\hat{R}(o-)), and the Appendix also introduces a “Ground Truth Reward. ”\n\nIt is unclear whether these reward functions are consistent or serve different roles. In Algorithm 1, it is not clearly stated how trajectories are sampled from the reward model (line 8). Is this based on a new policy fine-tuned via RLHF using the learned reward function R_t? This should be made explicit.\n\nSection 2.2 introduces an asymmetric max-margin loss, but Equation (1) does not reflect the margin term, and the corresponding optimization step in Algorithm 1 lacks clarity.\n\n2) The largest model evaluated in the experiments is only 410M parameters. To support the paper’s claims about LLM alignment and scalability, additional experiments on larger models (e.g., LLaMA-8B) would strengthen the empirical validity of the conclusions.\n\n3) The paper suffers from inconsistent use of terminology and symbols, which impairs readability and technical precision.\n\nFor instance, Section 2.3 uses “output o+, o-” while Algorithm 1 refer to “trajectories,” without clarifying whether these refer to the same concept.\n\nNotations such as R*(s) = w^{T} \\phi (s) and \\hat{R}(o) = w^{T} h(o) + b are introduced, but the relationships among \\phi (s), h(o), and \\mu in Algorithm 1 remain unclear and should be clarified.", "questions_to_authors": "1) In line 8 of Algorithm 1, it is unclear how trajectories are sampled from the reward model. Are these trajectories generated using a policy trained via RLHF guided by the current reward model R_t?\n\n2) Do you anticipate any challenges in scaling your IRL-based approach to larger models such as LLaMA-7B or GPT-like models? Have any preliminary tests been conducted in this direction?\n\n3) The abstract mentions insights into the non-identifiability of reward functions. Could you expand on how this issue manifests in your experiments, and what implications it has for interpreting the recovered reward models?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present a framework for reverse engineering reward functions used to train LLMs. This is very new and the reviewers acknowledge this. The reviews all touch on how the experimental setup is somewhat narrow while the paper overall is well written. COLM's reviewing guidelines have many directions a paper can be accepted for, from novelty, rigor, clarity, impact. I think this paper hits the first three and is a new idea, so should be accepted to the conference.

Thanks to the reviewers and authors for the lively discussion as well. It should have helped the paper!
While I recommend acceptance, I do agree with some of the cons and that the paper could be stronger with more experiments, polish, or transitioning into "general" RLHF understanding from classifiers like toxicity.
Decision: Accept