On the Low-Rank Parametrization of Reward Models for Controlled Language Generation
Authors: Sergey Troshin
Vlad Niculae
Antske Fokkens
Venue: Accepted by TMLR ()

Abstract: Language models trained on large amounts of data are known to produce inappropriate content in some cases and require careful tuning to be used in the real world. We revisit an effective and modular approach for controllability of the language models, when an external expert model guides the decoding. Particularly, we zoom in into the parametrization choice of an external expert, highlighting the difference between low-rank and higher-rank parametrizations. Higher-rank experts are designed to support high flexibility when representing the rewards, leading to higher computational costs during decoding. However, we demonstrate that they might not use their full flexibility. By analyzing the recently proposed reward-augmented decoding approach (RAD), which uses a higher-rank expert model, we introduce a simpler but more efficient low-rank parametrization of the expert model enabling fast and effective guided decoding. We empirically show that the low-rank RAD performs on par with the more flexible RAD on a detoxification and a sentiment control task, while requiring only a single reward model call per generated token.

Published in Transactions on Machine Learning Research (08/2025)
On the Low-Rank Parametrization of Reward Models for Controlled Language Generation
Sergey Troshin
8. troshin@uva.nd
Language Technology Lab, Informatics Institute, University of Amsterdam
Vlad Niculac
v. w.niculaedwwa.n.
Language Technology Lab, Informatics Institute, University of Amsterdam
Antske Fokkens
antske.fokkens@us.
Computational Linguistics and Text Mining Lab,
Faculty of Social Sciences ond Humonities, Vrije Universiteit Amsterdom
Reviewed on OpenReview: https: openreview. net/ per/formm? id= c.R
Abstract
Language models trained on large amounts of data are known to produce inappropriate content in some cases and require careful tuning to be used in the real world. We revisit an effective and modular approach for controllability of the language models, when external expert model guides the decoding, Particularly, zoom in into the parametrization choice of an external expert, highlighting the difference bet ween low-rank and higher- rank parametrizations. Higher-rank experts are designed to support high flexibilitty when representing the rewards, Icading to higher computational costs during docoding. However, we demonstrate that they might not use their full flexibility. By analyzing the recently proposed anward-augmented decoding approach (RAD), which uses higher-rank expert model, we introduce simpler but more efficient low-rank parametrization of the expert model enabling fast and effective guided decoding. We empirically show that the low-rank RAD performs on par with the more flexible RAD a detoxification ion and a sentiment control task, while requiring only a single reward model call per generated token.
1 Introduction
Generative large language models (LLMs) have gained a lot popularity in recent years and shown impressive results zero-shot and few-shot scenarios on numerous downstream tasks (Touvron et al., 2023; OpenA1, 2024; Jiang al., 2023). These large-scale models are pretrained on large amounts of data, and are known to inherit and memorize underlying biases (Sheng et al., 2019) as well as to provide unsafe responses (Wallace et al., 2019; Ganguli et al., 2022), necessitating further tuning for safer deployment and control (Ouyang et al., 2022).
Control over LLMs can be roughly divided into methods which modify the original model via finetuming (Ouyang al., 2022; Rafailov et al., 2023), and docoding-time solutions, which do not modify the parameters of the original model, including best-of-n sampling (Wang et al., 2023; Sun et al., 2024). As models increase in size, finetuning becomes prohibitive with limited computational resources. In this work, focus a more modular approach of decoding-time guidance, and assume we have access to top-k logits of a black-box base language model (see $2.1 for details). In this line of work, a discriminator model is trained to modify or re-rank the logits of the base model during decoding in order to satisfy the desired constraint (Yang & Klein, 2021), while preserving the distribution of the language model as much as possible.
In our work, we zoom in on the parametrization choice of the reward model. Yang Klein (2021), Deng & Raffel (2023), Mudgal et al. (2024), Chakraborty et al. (2024) parameterize reward models discriminators,

---

Published in Transactions on Machine Learning Research (08/2025)
RAD-V
RAD-Q
V
V2
VM
V1 V V
Figure 1: The RAD-V parametrized model (left) predicts reward for each next token candidate independently concatenating it to the current prefix RAD-Q (right) parametrization uses the language model output embeddings to elficiently predict the rewards for next token candidales over the vocabulary V.
which requires separate forward pass through the backbone of the reward model (layers before the output head) for each token candidate. Liu et al. (2021), Krause et al. (2021) and Cao et al. (2022) use more efficient approaches and perform single forward pass to predict the scores for all next token candidates. A similar parametrization choice is encountered in the reinforcement learning literature (RL,), where one chooses between state value function (V function) or state-action value function (Q function) parametrization (Sutton& Barto, 2018; Dulac-Aruola et al., 2015). In particular, closest to our work are the decoding- time RL approaches, where V/Q functions are trained guide the decoding from a language model (Mudgal et al., 2024; Chakraborty et al., 2024). In our work, we highlight the rank bottleneck (Yang et al., 2018) of the Q-style parametrized models in application to text reward modeling, where vocabulary size is larger than model dimension. We theoret analyze the difference between these two parametrizations in terms of rank expressivity. We provide a simple but realistic example, where Q-style parametrization is less expressive compared to V-style paramctrization, suggesting that the Q-style choice of parametrization should not be treated as the default option indiscriminalely.
Despite the rank bottleneck limitation, Q-style parametrization more computationally attractive. We therefore ask whether there cases where it is justifiable allirmatively. Particularly, we look at the two popular controlled generation tasks and a recently proposed reward augmented decoding (RAD) approach (Deng & Raffel, 2023), where they train a V-style parametrized model (we will refer to it as RAD-V). While RAD-V demonstrates high effectiveness for controlled generation, it scales poorly when the number of next token candidates grows, requiring a separate forward pass through the backbone of the reward model for each token candidate. We demonstrate that for this scenario, high rank is not required approximate the training data well, and hence RAD-V might not use its full flexibility. To empirically verify that RAD approach using Q-style parametrization, RAD-Q, is then expressive enough for this scenario, distill RAD-V V into an efficient RAD-Q, and demonstrate that guided decoding with the RAD-Q model results in comparable attribute control /fluency to the more flexible but more computationally intensive RAD-V approach.
Q-style parametrization, which answer
2 Preliminaries
2.1 Guided decoding with external experts
In this section, we outline the approach of guiding a base language model with external loken-level discrimi- nators. At each step of decoding, both the base model and the discriminator observe an already generated prefix æ, and cooperate to score the next token candidates v E V. A language model prodicts the logits *LM (-Ix) RIVI and the goal of discriminator to augment these logits with reward scores f(-12) RVI, A standard practice is to consider only likely tokens V C V at each decoding step e.g. via top-k (Fan al.,
2

---

Published in Transactions on Machine Learning Research (08/2025)
2018; Deng & Raffel, 2023) nucleus sampling (Holtzman et al., 2020):
+ if E Vi, -00,
(1)
otherwise,
and the next token is sampled from the categorical distribution:
(xx) = Softmax(=<v|z)).
(2)
While some language models might have a restrictive application programming interface (API) for safcty reasons, this line of work makes a reasonable assumption that we have access to the top-k logits of a language model either directly or via API for a relatively small k < [V].
To define reward scores, Q-style models given prefix x only pass it once through the external language model backbone, and use the linear output layer to obtain the scores for each of the next token candidates. GeDi (Krause et al., 2021) and DExperts (Liu et al., 2021) use srriiutt-conditionne unidirectional language models (undesired attribute in GeDi or two LM experts for desired and undesired attribute in DExperts), trained via the standard language modeling objective on class- dlass-conditione data: (y(2)z) = 21 (v|22, y), where y (0; 1} the attribute (e.g. positive/ positive/negative negative sentiment).
Alternatively, a V-style parametrized model such as RAD-V (Deng & Raffel, 2023) predicts the attribute of interest for a prefix concatenated with a next token candidate fRAD-V((2, e)), where [+ denotes the concatenation of a prefix and a next. token candidate. This approach requires passing each next token candidate as imput to the model, thus, to obtain the scores for & next token candidates for top-k decoding RAD-V would need k forward calls of the reward model, which can slow down inference siguificantly and constrains them to limit the number of next token candidates. Therefore, we ask whether Q-style parametrized models can possibly match the performance of V-style models while enjoying higher efficiency.
2.2 RAD training
In this section, we outline how the RAD approach (Deng Raffel, 2023) uses labeled data to train a reward model.
At the training stage, we assume that we have dataset D ((ul), ())) of text utterances of length ((u) and responses y E [0; 1]. The RAD approach is to use the data distribution to estimate the expected future response. For each utterance from the dalasel, can split it prefix x', next token and continuation v, in all possible ways. This way, they create an extended dataset to train on partial prefixes:
Dy = {(x, u z= uuu E (L,....((()),, (u, E D).
(3)
RAD trains reward model predict y given text input. Then, during training, RAD takes the input prefix = [x', u] and incurs weighted squared loss for approximating the future reward:
C(F(u)z'), v. = X (f(ulz') =
(4)
where À are the discounting weights X(x, u) = 1(x)/Zu, for each prefix used to up-weight prefixes closer to the ((w) full sentence, and Z,, =((u) (u,)) is normalizing constant such that E(e) A(u., u) 1. During training, we can use teacher forcing to process all prefixes of an utterance in single pass.
3 Reward modeling as low-rank matrix factorization
3.1 Analysis of RAD
3.1.1
Reward modeling as matrix completion
To better understand the training objective of RAD, we start by looking at the optimization problems defined in Equation (4), where we optimize reward model to approximate future responses. A unidirectional reward
3

---

Published in Transactions on Machine Learning Research (08/2025)
model can predict reward value for each next next token candidate. If enumerate all the contexts x' in the training data and all possible next tokens v, we task reward model to predict the values of R E RNNIVV which we dub the reward matrix.
If each context would be observed only once, R would have a single observed reward in each row. For short and common contexts we can observe more continuations per row, and also for some contexts there can be ambiguities: {(z, u1, ()),,..z. um: Wm)}. From mean squared error point of view, is equivalent compress these ambignities by taking the weighted average of their y (Appendix B):
R|x',0] = EnnwD, A(x, uy
55)
EuggDyya {(r,u) u)
From this perspective, reward modeling can be interpreted matrix completion problem. The training dataset Dj gives us only incomplete view of true reward matrix R. Following the notation in the matrix completion literature (Mazumder et al., 2010), Ω denotes the set of indices of the observed entry [x', u). Dy), and (R(R) denotes the projection of R that sets all indices outside f to zero. We denote the complement of Ω w.r. t. the complete set of indices as QQ The full RAD objective for a V-style parametrized model is equivalent to minimizing [[Pa(R) - Pa(RRAD-V)II: where each entry RRAD-V[E', 2] f(el=") can be computed with forward pass.
indices {(r',)
104
103
|V==02557
Rank
d=768
102
RAD-V (toxicity)
RAD-V (sentiment)
10a
100
500
1000
1500
2000
2500
3000
3500
4000
N contexts
Figure 2: When considcring a RAD-Q approach e.g. distilling RAD-V into RAD-Q, it is important to ask whether the rank capacity the RAD-Q model enough to model RAD-V outputs. We numerically estimate the rank of RRAD-V by incrementally considering more rows of the RRADAV matrix, and observe that the rank tends to be less than the model dimension d 764 and much less than [V], the maximal possible rank of (R)..
To analyze the reward matrices from the perspective of rank expressivity, define the minimal rank of partially observed matrix
matrix, with Ω denoting set of observed indices. We define the minimal rank of R w.r.t. Ω as the rank of the lowest-rank completion consistent with Pa( RR:
Definition 1. Let R be
min ra)) = min (rank(ft) (t) R e Raxm Pa(Rt) PT)(1))
(6)
Additionally, since numerical solutions to low-rank problems must incur some error, we also define a minimal numerical rank.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Definition 2. Let R be a n x m matrix, with Q denoting a set of observed indices, and & E R... We define the minimal e-rank of R w. Q as:
[Pa(R) - Pa( soj.
min rankn, (R) = min rank( R R=
(7)
nm
3.1.2 RAD-V can be high-rank, but is not in practice
Given a prefix æ, RAD-V accepts a token candidate u as an additional input to the model PRAD-V(Z' FRAD-V((a' oj), passing e through the layers of the reward model. For this reason, we expect RAD-V have the capacity to represent large space of reward matrices including matrices with higher rank. In Appendix D.1, empirically verify that RAD-V is capable to model data that has high minimal rank, e.,,, it can fit (with training loss numerically close to zero) a dataset R that has min rank (It) > d, where d is the dimensionality of the Transformer model. This fiexibility does come at a cost: to score many next token candidates during lop-k decoding, RAD-V requires a forward pass through all layers of the model for each of the & next token candidates. Following this observation, an important question is do we need this flexibility at the cost of slower decoding?
In Figure 2, aim to measure the rank of RRAD-V for RAD-V trained on two datasets: for detoxification and sentiment control tasks (discussed in detail in $4). To numerically estimate the rank, we follow Finlayson et al. (2024) and first sample N random prefixes from the dataset Dy to calculate N full rows of ARAD-V (requiring N -VV calls to the RAD-V reward model). Then we use singular value decomposition with the standard singular value cutoff to compute the numerical rank (Appendix D.4). We observe that the reward matrix learned by RAD-V tends to have low numerical rank, suggesting that it is possible to use less flexible but faster reward models to improve the efficiency of reward models.
3.1.3 RAD training data is low-rank.
One possible explanation why RAD-V docs not use its full flexibility is that said ficxibility is not required to fit the training data, e., that (R) can be fit with the low-rank model. To analyze the rank needed to fit the data, we use the definition of the minimal numerical rank (Definition 2). Empirically calculating the minimal numerical rank of the data is challenging due the very large number of prefixes. We use a combination of theoretical and empirical approaches listed in Appendix C.3, and we claim that the RAD training data (Jigsaw (cjadams et al., 2019) and Amazon Polarity (Zhang et al., 2015)) has low minimal numerical rank, that is less than the model dimension for E = 10-5, and even less than 256 if we allow = 10-3 We further confirm this claim for the commonly used HelpSteer (helpfulness) dataset (Wang et al., 2024b) and BeaverTaiil (safety) dataset (Ji et al., 2023), used for reward model training ( (Wang et al., 2024a). We thus draw the conclusion that the incomplete (R) matrix can be fit with the low-rank matrix factorization with small error.
3.1.4 Is low-rank enough to model reward matrices?
When viewed from a matrix completion angle, it may seem that the reward modeling problem is well suited for low-rank modeling. However, we will argue this should not be taken for granted.
First, is indeed true that data matrices generated by a certain kind of random process (Udell & Townsend, 2019) are low-rank with high probability. Moreover, in the case of missing values, intuition suggests that they should tend to low-rank, and indoed the following result demonstrates that we can almost always decrcase the rank by 1 when filling in one missing value (Appendix C.2):
Lemma 1. For i > 0, lot R be a random matrix with distribution supported on R and a single missing value, e.e -Ω2 {i..)).. Then, P(min rankę (R) k))
Morcover, when most values are missing, rank again seems likely to be low. In particular, when only one value is observed per row (as demonstrated in Example 1), rank-1 completion is possible (Example 2,
5

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix C.1). In dataset, there will likely be many unique prefixes where this scenario is applicable, particularly for longer prefixes.
Example 1. As illustrated in Figure 3a, assume that all prefixes x appear only once in the datasct, so that ((i,j) e = x3 = for all æ. Then, min rankg =.
Example 2. As illustrated in Figure 3b, let each token in the vocabulary either appear in high-reward (y=1) or low-reward (y=0) utterances (independent of context). This gives a rank-1 completion consistent with Example 1.
Given this observation, one might assume that having more missing values in reward matrix always implies that the reward matrix can be fit with lower rank. However, this is true, and the following result suggests that reward matrix with many missing values and can still be forced to have high rank:
Lemma 2. For k 0, there exist R Rexk and Ω with [o] E O(2)) such that min rankg (R) k.
Proof. Consider a k-by-k identity matrix with k:k - 1)/2 missing values above the diagonal (see Figure 3c from Example 3). Any possible completion results in an upper-triangular matrix. The determinant of such a matrix the product its diagonal elements, e.e., 1, so any completion must be full rank.
The construction in Lemma 2 is not artificial and could indeed correspond to a reward modeling task for the contexl-dependent constraints, as shown in the next examples.
Example Consider a list (first name, last name) pairs: [ks, my)], 1 1 i,j f. S n, where some pairs are marked as allowed blocked. Assume that the rule is as follows: for any utlerance containing (k,, uz) with j S i, assign the reward 0; if i, assign the reward 1, and unknown otherwise (Figure 3c). We assume that each utterance contains a single mention of person, then the expected reward matrix with [..., kij indexing rows and vy indexing columns will contain the upper-triangular submatrix, which is of rank n.
Example 4. Consider an application of LM to an arithmetical reasoning task, where you have examples in form of y 2, where I, y, define a reward function as R(x + y 2) = 1 if the expression is true, 0 if false. Then by considering all possible contexts z + y it is casy to show that the reward matrix is full-rank since the rows of the reward matrix include all one-hot vectors representing the correct answer.
integer number tokens forming a vocabulary V = [0..999]. Let
Empirically, our results find that Jigsaw (cjadams et al., 2019) and Amazon Polarity (Zhang et al., 2015) indcod support low-rank completions ($4). Nevertheless, from all results in this section, we conclude that care must be taken before making low-rank assumptions about reward datasets.
3.2 Low-Rank Autoregressive Reward Model
In this section, we introduce RAD-Q (Figure 1), low-rank paramotrization of RAD, designed for efficient. modeling of reward scores for next token candidates.
We revisit the language modeling (Q-style) parametrization for reward prediction (Liu et al., 2021; Krause al., 2021) and predict the scores for all next token candidates with a single forward pass through the backbone of language model. In contrast to RAD-V, RAD-Q predicts the representation vector h(x) E Rd given prefix æ and uses output embeddings e(v) Rdd to get the scores for all next token candidates. We

---

Published in Transactions on Machine Learning Research (08/2025)
V1
V3
V1
V
1
22
V3
V1
1
22
V
V1
X
*
*
0
*
X1
1
0.5
0
X1
1
*
*
X2
0.5
X2
0.5
O
1
2
0
1
X3
1
*
*
X3
0.5
0
1
X3
0
0
X4
*
1
X4
0.5
0
1
X4
0
0
0
1
(a) Sparse (a(R)
(b) Context independent R.
(c) Full rank Po(R).
Figure 3: (a) each row of reward matrix (R( has only known clement, the minimal rank is 1; (b) a reward matrix has rank 1, when each token has a context, independent reward; (c) reward matrix is full rank for highly context dependent rewards e.g. for the case of identity matrix.
use the following RAD-Q parametrization, similar to the how Dueling Network (Wang et al., 2016; Tang et al., 2023; Han et al., 2024) parametrizes the scores for the next tokens given the prefix:
fRAD-Q(U(z) fo(h(z)) +A((e()))((z)),
(8)
haseline
where the baseline predicts the score for the prefix and Af predicts how observing next token changes the score. Particularly, we use linear parametrization:
fo(v)z) (h(x), w) Af(e(v)(((o)) (M(x), We(r)).
Here, introdeedd two attribute-specific parameters: Rd for modeling the baseline reward of the prefix, and W paxd to model marginal rewards for each next token candidate.
Rank bottleneck of RAD-Q. Now it is clear that in contrast to RAD-V, RAD-Q (as defined in Equation (9)) performs a low-rank matrix factorization of Pa(R)):
BRAD-Q = H(w17 T + WE)=HAA
(10)
where we stack all context reprosentations x' into H E N and all next token representations into WE E RdxiVi, and column d-vector of all ones. By the rank inequality,
rank (RRARD)))) rank(H S Min(rank(HH) rank(4)) dd
(11)
meaning that if min ranko(R) > d, RAD-Q cannot possibly perfectly reconstruct Pa(R) no matter how flexible h(x) is. In terms of e-rank, if min ranko. > d, then RAD-Q cannot fit the data with training loss less than e. In the language modeling literature, the rank bottlenock problem is known as the softmar bottleneck (Yang et al., 2018) and mitigation strategies are well-studied (Ganea et al., 2019; Chang & McCallum, 2022).
In the experiments ($4), we empirically demonstrate that distillation of RAD-V into low-rank RAD-Q can match the performance of the more flexible RAD-V on the two standard controlled generation benchmarks.
3.3 RAD-Q training
To train RAD-Q, rely the RAD approach to train a reward model. We split into last token and remaining prefix: x = jo., We pass x' as input to the model, and v indexes output embeddings (9). We consider two types of experiments: training RAD-Q on original responses from the dataset, and a distillation experiment, where we train RAD-Q to predict the scores of less efficient RAD.
For the first type of experiment, we train RAD-Q on the responses from the datasct using the weighted squared loss:
((f()l=)) v. X(F(v)x') y)²
(12)

---

Published in Transactions on Machine Learning Research (08/2025)
For the second type of experiment, we train an RAD-Q student to approximate the less efficient RAD-V teacher F(x:) (a frozen trained RAD) using the distillation loss (Hinton et al., 2015):
Lasu(f(ulz') = (ool=')-(xx)))
-
(13)
A reward model can only observe a limited number of next tokens given x during finetuning. While the loss defined above provides a positive signal for some tokens v, it might be beneficial to regularize the prediction for other (unrelated) Lokens, including rare or unseen tokens. In our parametrization (8), is natural to push the predicted reward towards the baseline for unrelated tokens. We regularize the prediction of RAD-Q to be close on average to the prefix baseline by forcing AP to be close to 0 for randomly sampled token candidates:
(reg(((z)) = Unitoormmy | [A(<e(\))M()))
(14)
where we use one sample of v' for each prefix position, sampling uniformly from the vocabulary. Particularly, regularized model can learn to abstain by predicting the baseline score for each next token candidate, which will not change the distribution of base model.
4 Experiments
4.1 Controlled generation
We follow previous work (Deng & Raffel, 2023; Liu et al., 2021) and evaluate RAD-Q on two controlled generation tasks: detoxification and sentiment control.'
In our experiments, we guide the decoding from a base model using a smaller finetuned reward model with the same tokenizer. Namely, we guide GPTg-2Large using a reward model finetuned from 2PTl-2-mmlll and we guide the LLaMa-2-(7b/13b) (Touvron et al., 2023) base language model with a reward model finetuned from Tiny La (Zhang et al., 2024). We finetune all parameters of the reward models except input/output embeddings, which remain frozen (in hope of improving generalization to unseen tokens).
We conduct experiments in two regimes: first, by distilling less efficient RAD-V (Deng & Raffel, 2023) using Lasu loss (13); second, by training a reward model from scratch on the responses from the datasets using cumulative loss & (12). Tn both settings, we use additional regularization freg by default. For evaluation, we perform guided decoding using top-k sampling from the categorical distribution defined in (2), where top-k candidates are selected taking & largest logits of the base model at the current decoding step.
4.2 Detoxification
For the detoxification evaluation, we follow previous work (Deng & Raffel, 2023; Liu et al., 2021) and evaluate samples from guided decoding given a 10k subset (Liu et al., 2021) of prompts from the PeaalTxxiityyPommee dataset (Gehman al., 2020). We follow Deng & Raffel (2023) and Liu et al. (2021) and finetune our model on 2M pairs of text and continuous 'toxicity' responses between 0 and from the Jigsaw Unintended Bias in Toxicity Classification challenge (cjadams et al., 2019). Like previous work, we train our model on 7 independent responses ('toxicity', 'severe toxicity', 'obscenc', 'identity attack', 'insult', 'threat', 'sexual explicit') with different head parameters wi, W,, E {l,..., 7} for each sub-task. During decoding, we only use the 'oxicityy predictor. For the distillation experiment, we use the same dataset, and the released toxicity discriminator from Deng & Raffel (2023) as a teacher.
During decoding, we sample 25 continuations generating at most 20 new tokens. To evaluate toxicily, we use external closed-source toxicity classifier Perspective API (Lees et al., 2022), and following previous work (Deng & Raffel, 2023; Liu et al., 2021), we rely on the Mazimal Average Tozicity metric, which is the maximal toxicity score value over 25 samples for given prompt, averaged over the set of 10k prompts. We also report Tozic Rale, which is calculated as the probability that at least one out of 25 continuations is toxic according to Perspective AP1 (toxicity score > 0.5); and Diversity score, which is the average number of distinct n-grams normalized by the length of text (Li et al., 2018). To evaluate the fluency of model generations, we follow
'Qur code available at https://gáthin com/ ser wtt///ertttoretiin/eaaeea troghin/rad-g
8

---

Published in Transactions on Machine Learning Research (08/2025)
0.35
Average Maximal Toxicity
0.25
0.20
RAD-Q k=20 distill
RAD-Q km20 resp. only
0.15
RAD-V k=20
X
GPT-2
0.10
GeDi
10
40
60
DExperts
Average Perplexity
0.35
Average Maximal Toxicity
0.30
0.25
0.20
0.15
0.10
1.00
0.95
0.90
0.85
0.80
0.75
0.70
0.65
MAUVE
Figure 4: RAD-Q student (distill) shows comparable toxicity Seeitty/fuuue /fluency trade-off with the teacher RAD-V, where the RAD-Q student closely matches the performance the teacher RAD. RAD-Q trained on original responses (RAD-Q resp. only) shows slightly worse fluency and similar loxicity level. We rerun the evaluation for RAD-V, GeDi and DExperts with an up-to-date Perspective API classifier. We include the results with other baselines from Deng& & Raffel (2023) in Figure 14 (see Appendix G.1.1).
previous work (Liu et al., 2021; Deng Raffel, 2023) and report the average perplexity of the GPT-2-XL when generating from the Large-Large model; and the OLMo² to evaluate the LLaMa family as in Lovelace al. (2024). As an additional fluency metric, report MAUVE (Pillutla et al., 2021) to measure the distance between unguided and guided generations (details in Appendix F). In the experiments, at the toxicity/ /fluency trade-off, alternating the weight / of the discriminator (see Table 3 and Table 4). We expect to obtain model with both low toxicity according to the Perspective API, and high fluency.
will look
Since toxicity scores from the Perspective API can change overtime, which can complicate the evaluation, in Appendix G.3.1 we evaluate our detoxification models with an open-weight toxicity classifier,' 3 where we observe the same relative results as with Perspective API scores.
4.3 Sentiment control
For sentiment control, we follow previous work (Li al., 2018: Sudhakar al., 2019; Liu al., 2021; Deng & Raffel, 2023) to evaluate the samples given a prompt. from of the three categories: 2.5K negative, 5K neutral, and 2.5K positive prompts from Text (Gokaslan & Cohen, 2019). To finetune RAD-Q responses only, we follow Deng & Raffel (2023) and finetune our model on millions of reviews from the Amazon Polarity (Zhang et al., 2015) and SST-2 (Socher et al., 2013) datasets. To distill the sentiment discriminator of Deng & Raffel (2023), we use text examples from the Amazon Polarity dataset. Additional training details are provided in Appendix E.
For evaluation, we follow Deng & Raffel (2023), and use the average Positive Rate metric r. t. the finetuned DistilBERT classifier (Sanh et al., 2019) provided via HuggingFace.! As in the toxicity task, GPT-2-
http:///wwngggggtt...oo/.ssslllp httpp:///ahorturl.aa///0op

---

Published in Transactions on Machine Learning Research (08/2025)
Average Maximal Toxicity
x
RAD-Q distill Llama-2-7b-hf
0.30
RAD-V Llama-2-7b-hf
0.25
RAD-Q distill Llama-2-13b-ht
0.20
RAD-V Llama-2-13b-hf
Raw Llama-2-7b-hf
0.15
Raw Llama-2-13b-hf
0.10
20
25
30
35
40
45
50
55
60
Average Perplexity
Average Maximal Toxicity
*
0.30
0.25
0.20
0.15
0.10
1.00
0.95
0.90
0.85
0.80
0.75
0.70
0.65
MAUVE
Figure 5: Detoxification results with Perspective API toxicity classifier and LLaMa-2 model. RAD-V and RAD-Q (distill) demonstrate similar performance w. r. two fluency metrics: average perplexity and MAUVE. Performance remains consistent across different models sizes of base LLaMa-2 model.
10

---

Published in Transactions on Machine Learning Research (08/2025)
1.0
Neutral prompt Positive Rate
0.9
0.8
RAD-Q k=20 distill
RAD-Q k=20, resp. only
0.7
RAD-V k=20
0.6
GPT-2
CTRL
GeDi
DExperts
DAPT
Negative prompt Positive Rate
PPO
0.4
Quark
0.2
10
20
30
40
50
60
Average Perplexity
1.0
Neutral prompt Positive Rate
0.9
0.8
0.7
0.6
Negative prompt Positive Rate
0.4
0.2
1.00
0.95
0.90
0.85
0.80
0.75
0.70
MAUVE
Figure 6: For the sentiment control task, RAD-Q trained on responses only lags slightly behind the RAD-V bascline, while student RAD-Q outperforms the teacher RAD-V model. For the plot with average perplexity, we include the results from Deng & Raffel (2023) for other baselines for reference.
XL/OLMo and MAUVE to evaluate the fluency of the sampled continuations, and expect to obtain high Positive Rate and high fluency.
4.4 Results
To compare RAD-V and RAD-Q, we rely on the methodology of Deng Raffel (2023) and Liu et al. (2021), and visualize the trade-off plots for both models varying the control parameter B. Namely, each point in the figure will represent two metrics: toxicity/sentiment along the vertical axis and fluency along the horizontal
11

---

Published in Transactions on Machine Learning Research (08/2025)
80
RAD-Q Q distil
Average Moximal Toxicity
0.30
RAD- Q distil
RAD Q distil w/o reg
RAD-Q w/o reg
60-
RAD-Q distil w/o baseline
0.25
RAD-Q distil w/o baseline
0.20
0.15
0-
0
500
1000
1500
2000
2500
3000
3500
4000
10
40 Average Perplexity
30
50
60
contexts
(a) Rank analysis
(b) Detoxification task
Figure 7: Ablation experiment for distilled RAD-Q, on the detoxification task with top-k=20. On the right, observe that regularization towards the baseline results in better fluency of generated samples. On the left, we observe that regularization lowers the rank of the model's outputs rank rank(RRAD-Q).
axis. From this plot, we can read, g., what fluency (perplexity, MMAUVE) can be achieved for given 'larget' toxicity. To compare two models, compare their curves (in the same plot). Our hypothesis is that RAD-Q will perform similar to the more flexible RAD-V approach, meaning that the trade-off plots for those models will be close to each other.
Detoxification. For the detoxification task (Figure 4 with GPT-2, Figure 5 with LLaMa-2), our efficient student (RAD-Q) closely follows the RAD-V teacher for toxicity control, ffluency trade-off. We observe that RAD-Q trained on responses only shows slightly worse llueney w.r.l. average perplexity for lower levels of toxicity. For completeness, in Figure 14, we include the results for other baselines from Deng & Raffel (2023) computed for an older version of Perspective API. For guided decoding from the LLaMa-2-(7b/11) 13b), we observe that again RAD-Q closely follows RAD-V in terms of toxicity (aaily//ueeen /fleecy trade-off (see Figure 5 in Appendix G.1.1).
Sentiment control. From the results on the imntt control task in Figure 6, we observe that the RAD-Q student model shows slightly better trade-off than the RAD-V teacher model, closely following approaches that require training using feedback from the evaluation pipeline (Lu et al., 2022, Quark), (Stiennon et al., 2020, PPO). Again, RAD-Q trained on original responses slightly lags behind but still performs competitively compared to other guided decoding baselines.
Summary. First, we observe that distilled RAD-Q can match or even outperform RAD-V, which confirms that Q-style parametrization expressive enough for our rolled generation scenarios. Second, distilling the RAD-V teacher into the RAD-Q student results in slightly higher quality compared to training RAD-Q on original responses. One difference is that when training from data, we will see short contexts multiple times with different reward responses and must implicitly converge their average, while in distillation, the teacher already performs this compression and provides a single deterministic target f(vlar) for every context. (x, v). We conjecture that this may lead to better-trained distilled models.
4.5 Ablation
In this section, investigate the effect of adding the baseline component Equation (9) and of regularization Equation (14). In Figure 7, we experiment with the distilled version of RAD-Q and observe that turning off regularization, or further removing the bascline from the paramctrization results in still adequate but slightly worse fluency as measured by perplexity, and a comparable toxicity decrease. By further analyzing the ranks of RRAD-Q with and without regularization, we observe that regularization effectively decreases the rank of RRAD-Q; which might explain the higher fluency of regularized models. Particularly, strong regularization would result in the model always predicting the bascline score for each of the next tokens (corresponding the rank-1 output), which does not modify the original distribution of the model (the best fluency).
12

---

Published in Transactions on Machine Learning Research (08/2025)
Model
N calls
Time (s) per token
0.0100
RAD-Q
RAD-V
GeDi (Krause et al., 2021)
1
0.0075
DExperts (Liu et al., 2021)
2
0.0050
RAD-V (Deng & Raffel, 2023)
RAD-Q (Ours)
0.0025
Number of input tokens a discriminator model needs to process for a single decoding step with & next token candidates. All included mod- els are based on the unidirectional Transformer (Vaswani et al., 2017) and support the caching of prefix activations.
Table 1:
0 20
40
60
80
Top-k
Figure 8: RAD-V processes the k next token can- didates separately as input requiring more time compared to RAD-Q, which relics on the output layer to obtain the scores for all next. lokens.
4.6 Efficiency
We consider using a reward model to compute the scores for & candidate lokens at each of L steps of decoding. Similar to RAD-V (Deng & Raffel, 2023), RAD-Q is based on the unidirectional Transformer architecture (Vaswani ct al., 2017), which means that we can cache the prefix activations during decoding. To compute the prediction for & next token candidates v given a prefix x; RAD-V needs to pass k next tokens as input the Transformer model, thus RAD-V processes (TT))) tokens during decoding. In contrast, RAD-Q only processes O(L) tokens input to the Transformer model and relies the output layer to efficiently compute the scores for all next token candidates. In Table 1, we summarize how many tokens external expert models process during top-k decoding. In Figure 8, we measure the time per generated token when running the decoding for the toxicity task with RAD-Q and RAD-V (Deng & Raffel, 2023) on a single RTX A6000 GPU.
5 Related Work
There are multiple approaches that investigate how finetune a language model using autribute-conditiin data (desired/undesinng examples). Keskar et al. (2019) finetunes language model using control prompts. More recent approaches (Schulman et al., 2017; Stiennon et al., 2020; Lu et al., 2022) perform finctuning while regularizing the weights of the model to stay close to pretrained weights. Despite the elficiency of decoding, these methods might require more resources for finetuning if the language model is large, or might even be unusable we only have access to the top-k logits of base language model via an API. Unlike finctuning, alternative approaches keep the language model untouched and use external models to guide the decoding from the base language model. Dathathri et al. (2019) use the gradients from a discriminator to modify the prefix activations of the base model during decoding. However, gradient-based methods are costly to use during decoding since they require backpropagating through the large base model.
Closest to our work are gradient-free guided decoding methods, where we have access to the frozen base language model and use external models to guide the sampling process from the base model. Among Q- Q-parametrized models, GeDi (Krause et al., 2021) uses class-conditooned language models as discriminators to augment the decoding and efficiently compute the scores for next token candidates. DExperts (Liu et al., 2021) improves the quality of GeDi introducing an ensemble of two class-conditioned language models finetuned on desired and undesired data. Cao et al. (2022) finetune a Q-style model to reduce the probability of reaching undesired terminal states. In concurrent work, Xu et al. (2025) propose Q-style model, where they show that Q and V style models are equivalent, assuming the logits come from a universal approximator. While true, this analysis omits the rank bottleneck present in practical models, which we analyze in this work. We hope our work will inform this line of work of potential limitations of Q-style parameterized models.
V-style models are often used to re-rank the intermediate hypotheses e.g. for best-of-n sampling, which however requires large pool of completed hypotheses (Sun et al., 2024). Deng & Raffel (2023), Sitdikov et al. (2022) argue to use discriminator models akin to Yang & Klein (2021) to guide the intermediate outputs
13

---

Published in Transactions on Machine Learning Research (08/2025)
during the decoding, whereas Deng & Raffel (2023) propose effective V-style parametrized reward model trained from labeled data examples. Sitdikov et. al. (2022); Dekoninck et al. (2023) use available bidirectional Transformers to guide the base language model, which, however, requires recomputing all prefix tokens at each decoding step. To tackle this issue, RAD-V (Deng & Raffel, 2023) proposes a unidirectional model suitable for caching of prefix activations. Among RI.-based approaches, Mudgal et al. (2024), Chakraborty et al. (2024) use a V-style model although at the same time they rely on the losses designed for a Q-style model (e.g. CD-Q from Mudgal et al. (2024)). To the best of our knowledge, there is little attention to the implied efficiency-qualiti trade-off that study in our work. The closest to our analysis, Han et al. (2024) compare both parametrizations in relation to language tasks, where they empirically observe that V-style parametrization outperforms Q-style parametrization.
To summarize, we complement the previous work, by zooming in into the parametrization of an autoregressive reward model. We highlight the trade-off between efficiency and expressiveness of a reward model, and showcase that 1) in theory there a gap in expressivity between Q-style and V-style models due to rank bottleneck, 2) for the tasks and datasets we consider, higher rank-expressivenss can be traded for higher efficiency without quality drop when using distillation. We hope that our analysis will inform future work the design choices of autoregressive reward models.
6 Conclusion
We review the recently proposed RAD-V approach of training a reward model for the guided decoding, and we reformulate it as the incomplete reward matrix learning problem. In the light of the rank analysis of the reward matrix, we observe that the high flexibility of RAD-V might not overweight its lower efficiency during decoding. We revisit the low-rank parametrization style of reward models in application RAD, and demonstrate the effectiveness of more efficient low-rank RAD-Q parametrization. We thus bridge the gap between two paradigms of training external expert models, demonstrating that we can have both efficient and effective controlled generation. At the same time, we precaution from indiscriminately choosing low-rank parametrization by highlighting the cases when the incomplete reward matrix has higher minimal rank.
Acknowledgments
This publication is part of the project VI. by the Dutch Research Council (NWO); and part of 'Hybrid Intelligence: augmenting human intellect" with project number 024.004.08 of the rescarch program 'Gravitation' which is (partly) financed by the Dutch Research Council (NWO).
the research program 'Veni', which is financed
We thank Wilker Aziz, Bryan Eikema, Caio Corro, and members of LTL and CLTL for fruitful discussions and feodback. We also thank the Perspective API team for increasing the API quota for us.
References
Meng Cao, Mehdi Fatemi, Jackie CK Cheung, and Samira Shabanian. Systematic Rectification of Language Models via Dead-end Analysis. In ICLR, 2022. URL http:///ggaaaaeeeeeeeeeeeeaattaa httt://orrrmiieeekaaysass
Souradip Chakraborty, Sounya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Bedi, and Furong Huang. Transfer q-star Principled decoding for LLM alignment. In NewrIPS, 2024. URL http://wwww.....ooo////////tttra
Haw-Shiuan Chang and Andrew McCallum. Softmax bottleneck makes language models unable to represent multi-mode word distributions. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguisties (Volume 1: Long Papers), May 2022. doi: 10. 10.18653/v1 222022.acllloogg httpp://aclaantholog. org/ acl-long. 554.
URL
cjadams, Borkan Daniel, inversion, Sorensen Jeffrem, Dixon Lucas, Vasserman Lucy, and nithum. Jigsaw unintended bias in toxicity classification, 2019. URL
14

---

Published in Transactions on Machine Learning Research (08/2025)
Nicholas Kluge Corrêa. Aira, 2023. URL
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and Play Language Models: A Simple Approach to Controlled Text Generation, In ICLR, 2019. URL
Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. Controlled Text Generation via Language Model Arithmetic. In ICLR, 2023. URL https://openrevien wttp://w..ee.ooorrrr/ddssssaroa
Haikang Deng and Colin Raffel Reward-augmented decoding: Efficient controlled text generation with unidirectional reward model. In EMNLP, 2023. doi: 10. //openreview.c. wttp://orrrraaA1111ii.DDDDaaDaod
eetp23.mmmlpmman..... 721. URL https:
Gabriel DulocdArnold, Richard Evans, Peter Sunehag, and Ben Coppin. Reinforcement learning in large discrete action spaces. CoRR, abs/ bbs/1512.07679, 2015. URL http:://arxiv..org/bb////55/ 076777779.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACI, 2018. doi: 10.18653/v1/P18- 1082. URL https://aclanthollogy htt.://ttaaoooy...or///11122222 org/P18-1082.
Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, and Ashish Sabharwal. Closing the curious case of neural text degeneration. In ICLR, 2024. URL https://openrevierr net/forumiid- http:/pponrrroi....eerrrrrrrri
aONpC9GL1o.
Octavian Ganea, Sylvain Gelly, Gary Bécigneul, and Aliaksei Severyn. Breaking the softmax bottleneck via learnable monotonic pointwise non-lincaritios. In ICML, 2019. URL v97 197/ganea192, html.
mlr. press/
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, and Bai et al. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, November 2022. URL https://arxiv, org/abs/220s .78588.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Rea/ToxicityPrompts Evaluating Neural Toxie Degeneration in Language Models. In EMNLP findings, Online, 2020. doi: URL https://aclanthol.gg www.:/oloooyooggg/000 org/2020. ftn..nggesssee11..300000
(1188533v112000 htt10//../nn///n///// (ww.....inninggeell....00
Aaron Gokaslan and Vanya Cohen. OpenWebText Corpus.
http://9kkglinn07..iittbbbb
OpenVebTextCorpus, 2019. Accessed: 2024-06-30.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization, 2024. URL
06639.
Trevor Hastie, Rahul Mazumder, Jason D. Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares. JMLR, 2015. URL http://jmls http:////il...ogpppprrrr/e/ee/// httpa///r..ppppee////aaaaaaeaaaa html.
Tianxing Hc, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov. On the blind spots of model-based evaluation metrics for text generation. In ACL, 2023. doi: 10.18653/v1 2023. 2123.acl-oon long.loo6.644 URL https://aclanthooogyy
2014 Deep Learning Workshop, March 2015. doi: acl-1on..... 674. abs/1503. 02531.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. In NeurIPS URL https://arxiv.org
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020. URL
Edward J Hu, yclong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL https:
15

---

Published in Transactions on Machine Learning Research (08/2025)
Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of 1lm via a human-preference dataset. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 24678-24704 Cur- ran Associales, Inc., 2023. URL https: httpa/:///oooeeeeeeeeeeeee wtw.rroceedinngnneu.riiieee neurips. nwurips.cc/paperr wwtppppe_fills//ppeeeeeeeeeeeeee sillss/paper/2033///aaaa 44bb61::666711eeee44777777083535
Prape--Dataset.. and_Benchmarks .df.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: httpp:://axxxvv.rrrr/ae www..og///////aa.o.ooo.o... /2300 06825.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv, 2019. doi: 10.48550 /arXiv.
URL https://arras org/ abs/1909 abs/1909.05858v.. 05858v2.
Diederick P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. URL
https://ans //ax/vv// org/abs//////eeeooeeoooooeee httpa////....11. 6980.
Ben Krause, Akhilesh Deepak Golmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joly, Richard Socher, and Nazneen Fatema. Rajani. GeDi: Generative discriminator guided sequence generation. In EMNLP findings, 2021. doi: 10. 20053//w1/22...nniiiin 2021.findings certinng-emnllp. 424.
URL https://acclattholll org/
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. In KDD, 2022. ISBN 450393330 doi: 10.1145 10.1145/3534678.. 15534668333391144 URL https://doi. htttp:://oo..or//1.. httg/10.1155... 1145/3534678.3588888 3539147.
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer. In NAACL, 2018. doi: 10. 10.18653/v1/N18- URL attps://aclanthhie N18- 1169/.
.rg/
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-tini time controlled text generation with experts and anti-cxperts. In ACI, 2021.
doi: 10. 10.18653/vl v220///v122221/// hal.//2.acloooog....... aal--long.522. URL https: htttps//aclanthoooogg. weregy.orgg211111
org/22211.clllongg. 522.
Justin Lovelace, Varsha Kishore, Yiwci Chen, and Kilian Weinberger. Diffusion guided language modcling. ACL findings, 2024. URL https:///cllanthooogg org/ http:/222..inninggs-se. findings-acl.8877
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable Text Generation with Reinforced Unlearning. In NewrIPS, 2022. URL https://openrevier http:/ppwrereeeeeeee///oo/aaaaaa htt.://ooouutttt--aaaaaadsaaaa
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research, 11(80):2287 (2880)):287723338 2322, 2010. URL http:
//jmlr. mhjmlr.org/e wrgg/pppers//1/mmaunnee.e
html.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Lı, Tao Wang, Yanping Huang, Zhifeng Chen, Heng- Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In ICML, 2024. URL https://openreview httpe//eeeeeeeeeeeetteeeeie/etar
Feng Nan. Low Rank Matrix Completion. Thesis, Massachusotts Institute of Technology, 2009.
OpenAl. GPT-4 Technical Report, 2024. URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human foedback. In NewrIPS, 2022. URL https sttps://openreviee net/forum wintrrmmmmi-cca d=TCakAC*EO....
16

---

Published in Transactions on Machine Learning Research (08/2025)
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE:
Measuring the gap between neural text and human text using divergence frontiers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: http::///ppprviirrre/ wwtp:/..e.e.oooeeeoooe wtt.:/eeoorroiiooooo/oo..po
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In NewrIPS, 2023.
URL https://openreviie http://wwwweeeeetttrrrriiiiiiaae net/forum'idd-HPuSl
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. In NewrIPS EMC2 Workshop, 2019. URL https://arxiv: org /bs/
1910.01108.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv, 2017. doi: 10448550/ar2iv..1777..
URL https: /tttps://arivvv arxiv wtt//oog///////oe///oo/eeee abs/ /10. 06347.
Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter:
On biases in language generation. In EMNLP-IJCNUP, 2019. doi: 10.18653/v1/D19- 13399 URL https:
//aclanthology.. org/D19-1339.
Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, and Lucas Dixon. Parameter efficient reinforcement learning from human feedback, 2024. URL https://arxiv wttpi/.orrrbbb///...3.. 10704.
Askhat Sitdikov, Nikita Balagansky, Daniil Gavrilov, and Alexander Markov, Classifiers are Better Experts for Controllable Text Generation. arXin, 2022. doi: (0.45550//rX1v.2205..2 2t/0///arX...22..77777 URL https://arxiv
org/abs/2205 07276.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013. URL
co/
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In NewrIPS, 2020. URL https://proceedings wetpeeeddinnnn..uip...... neurips. cctpappr//02////aeeeeeeppeeeeeee h0aa//ash/////////// 1199854552299333558844485555 att.rettbeeeeeetetttttttaaaaattt abbstract.html html.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran, "Transforming" Delete, Retrieve, Generate Approach for Controlled Text Style Transfer. EMNLP-IJCNLP, 2019. doi: 10.
URL
https://aclanttoology.. org/D19-
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. In NeirIPS, 2024. URL
https://openrrview wwwp:/..ee/foooooo httpp///ppppaeeeeeeeeeeeffttaaaa wwt.:e//ooouurrrddaaafafftell
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2
edition, 2018
Yunhao Tang, Remi Munos, Mark Rowland, and Michal Valko. VA-learning as a more efficient alternative to q-learning In PMLR, 2023. URL https://proceedingss mlr. press/v202 v202/ 2222/tan222h html.
Hugo Touvron, Louis Martin, Kevin Stone, and Peter Albert et al. Llama 2: Open foundation and fine-tuned chat models. arxe 2023. URL https://arxiv. org 2330
Madeleine Udell and Alex Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-160, 2019. doi: 10.1137/18M11184400 URL 1
17

---

Published in Transactions on Machine Learning Research (08/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. URL https://dl acm org/ ht/g/do//10// 5555/3295222 3295349.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP-IJCNLP; 2019. doi: 10.18653/11/D19-11211 URL https:
//aclanthology. org/D19-1221
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-ofeexerrs,, 2024a. URL httppp///aaaii..r/aaaa////////// 12845.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In JCLR, 2023. URL https:://pperreviee met/fforum2.dd1PLNNTTM
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-attribute helpfulness dataset for SteerLM. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), ; Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 3371 3384, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10.18653/v /22////v1/222222//// m00//22...aacllooo. naacl-long..18, URL https: //aclanthology. hwwpgg/....//22..... maacl-long 185/.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In PMLR, 2016. URL attps://proceedings mlr, prrrss v48/wangf1s 16. html.
Yuanchong Xu, Udari Madhushani Schwag, Alec Koppel, Sichong Zhu, Bang An, Furong Huang, and Sumitra Ganesh. GenARM: Reward guided generation with autoregressive reward model for test-lime alignment. In ICLR, 2025. URL
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with 2021. doi: 10.18653/ v /2221 naacl-main.276 URL https://aclantholle org/2021 www.laaallaaa.a.a. manaclmmiinn.mmmmm 276.
with future discriminators. In NAACL, Online,
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018. URL https://ppnrrviee..eeee net/forum?id-Hlw/ZZ
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. Xiv, 2024.
Xiang Zhang, Junbo Zhao, and Yann LoCun. Charactor-leve convolutional networks for text classification. NewrIPS, 2015. URL
18

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
Warning: (the last page of) this appendix contains model-generated text. conditioned high toxicity contexts.
A Limitations
The models discussed in this work can only reduce the probability of generating the toxic responses, not prevent it. Moreover, evaluation of toxicity is far from perfect, and even very low toxicity score from automatic evaluation such as Perspective API does not necessary mean that the sample is 'safe' Furthermore, we should not exclusively rely on toxicity when evaluating the safety of samples from language models due to the complexity and variability of language. It is also not clear that by reducing toxicity, we are not introducing other harms. Furthermore, both RAD-V and our models represent low-rank R and further qualitative research needed investigale whether certain loxicity patterns require high rank to represent. them.
B Reward Matrix
To train a reward model, we use weighted mean squared loss, for which the weighted mean recovers the minimum:
Σ y)² Exp Au
r" = arg min
(15)
Ener
A.
Proof. 8.
Enn A(r -y)² = = Exn 8 (xr - y) = Ex, 2\(r - 1) = 2(r Exn Exy(Du) = 0. Hence,
Fr
Σ Lx. Ay Exy
r"
C Factorization of (R(
Any matrix R V can be factored R UVT with U,V of dimensions N x q: VV] q. If R incomplete, then there are in general multiple possible factorizations of Pa(R) compatible with the observed values.
C.1 Rank-1 case
To get better intuition why the incompleteness of Pa(R) allows us to find a compatible factorization with lower minimal rank, consider simple example. If we only know 1 element per row of R, then the minimal rank of PR( is equal to 1. To prove this, consider completing (1t) such that each row is filled with the same clement (the only one known for this row):
?
-
?
3
3
C.2 case of single missing value.
Here we prove that introducing a missing value in a random k-by-k matrix will almost always (with probability 1) allow completion of rank most k - 1.
Proof. Let R be k-by-k matrix, and without loss of generality, by permuting rows and columns we may assume the unknown element is at position 1), Consider completing it with value x, and denote the
19

---

Published in Transactions on Machine Learning Research (08/2025)
completed matrix as A. Let As be the ij minor of A, i.e. the determinant of the submatrix left after removing row i and column j.
We can expand det(A) = xAi #2) A21 +... au AR1- In order for A to not be full rank, there needs to be a aui
solution in x to det(A) = 0. If A1 is not 0, then I = -a21 A21 +... + 0x1 Ak1)/A1 will lead to a completion of rank less than l regardless of any of the specific values. However, if A is 0 but 02 A21 +... + dga ag A&ı is not 0, then there is no solution for x that would result in determinant of 0 and thus the rank of A must be k. The set of matrices R satisfying R1 0 has zero Lebesgue mcasure inside R and thus for any distribution over matrices with full support, sampling such R is zero- sero-probabiliy random event.
C.3 Estimating the minimal rank of the data
Empirically calculating minimal rank is challenging due to the very large number of prefixes (row of the matrix), particularly, large portion of the prefixes have unique continuations. We show how we simplify the minimal rank estimation by considering only the prefixes with two or more continuations, and demonstrate that partially observed R can be fit with a low rank matrix factorization.
Given a training dataset of responses and text utterances, there will be many unique prefixes, for which the Example 1 is applicable. We can therefore reduce the complexity of rank estimation by skipping these prefixes, as shown the next result.
Lemma 3. Let (R, be a partially obbeevvd matrix and 92 S Ωt denote the subset of observed indices that have at least one other index in the same row. Then,
min rankg,- (R) I 1 + min rankn,,e
(16)
Proof. Permute rows such that all rows with single index are grouped together. We have a blompeincomelete matrix where the top block admits a completion of rank 1, and the bottom block admits completion of rank min rank (R). The rank of the stacked completions is no more than the sum of the ranks of the completions.
This result reduces the problem of estimating min rankg, (R) to the possibly smaller problem of estimating min ranka,.= (R) (since any fully-unobserved rows and columns can be skippod.)
We now numerically verify that there exist low-rank factorizations compatible with Pa, (R) within e. Finding such factorization of rank - 1 implies, by the previous lemma,, that R is of minimal rank w.r.t. Ω and thus that the training dataset D, can be fit by reward model with rank bound by r, regardless of the specifics of said model. In general, finding minimal rank factorization of incomplete matrices is known to be NP-hard, and usually convex relaxation such as minimization of the nuclear norm is considered (see Nan (2009)). To factorize a partially-observed matrix, we use the soft impute alternating least squares algorithm (Mazumder al., 2010; Hastie al., 2015).5 5 Given matrix X E Rax with observed indices Ω, this algorithm solves
minimize || - AB 11 + A(IAI + IBIB))
(17)
with respect to A Ra B E Rm
by alternating between efficient soft SVD updates for A given B and B given A. We optimize for 1000 iterations with a trace norm penalty of X = 10-4, This penalty induces a small bias but improves convergence. At convergence, A and B constitute a certificate of the minimal numerical rank of X w.r.f. Ω
Results reported in table 2 provide strong evidence that both datasets can be approximated well by low rank matrices, close to the 10-6 resolution of 32-bit floating point numbers. The reported MSE values for rank d-1 can be interpreted reachable lower bounds of the MSE training loss of RAD-Q transformer with hidden dimension on the respective training data.
hhttps://eran.r-prooff http://ttt..tte////cccccc///t/tt html
20

---

Published in Transactions on Machine Learning Research (08/2025)
rank
detox.
sentiment
helpfulness
safety
0
4.6o-2
4.9c-1
8.4
0.42
255
3.4e-5
3.4e-4
1.04e-08
5.08e-08
511
7.7e-6
9.7e-5
1.45e-09
5.77e-10
767
4.7e-7
6.6e-7
1.28e-09
4.81e-10
Table Mean squared errors of low-rank matrix completion of the Jigsaw (detox) and Amazon review polarity (sentiment) datasets following the methodology described in appendix C.3. Additionally, report MSE for HelpSteer (helpfulness) dataset (Wang et al., 2024b) and Beaver'Tiils (safety) dataset (Ji et al., 2023), which are commonly used for reward model training (Wang et al., 2024a). The zero rank row corresponds to predicting the zero matrix. All datasets can be approximated well by low rank models. For the ranks, we use multiples of 256 1, because one rank is reserved to handle the single-occurrence contexts.
D Rank expressivity of RRAD-V and PRAD-Q
D.1 Rank expressivity of RAD-V
In this experiment, we empirically verify that RAD-V is capable to approximate Pa(R) matrix with rank(Pa(R)) > d, where d is the dimensionality of the model. We finetune RAD-V initialized from the GPT- 2-Small (with d = 764) on a synthetic data constructed as in example 3. We generate (R) = 1024 > d, an incomplete matrix of size 1024 with unknown elements above the diagonal, ones on the diagonal, and zeros below the diagonal. With this construction, min rank (R) and thus greater than the model dimension.
We verify that we train RAD-V to fit this train matrix obtaining the training loss (MSE) less than 5 10-6.
D.2 Rank expressivity of RAD-Q
RAD-Q approximates (R(R) as product of two rank d matrices, hence RAD-Q cannot reconstruct the data perfectly, which has higher rank; RAD-Q obtains MSE 0.0001.
We thus conclude that RAD-V (in contrast to RAD-Q) is indeed capable of representing Pa(It) matrices with a rank higher than d.
D.3 Real data experiments
For the experiment with the real datasets for the detoxification and sentiment control tasks, in Figure 9, we numerically measure the rank of RRAD-V and RRAD-Q: and observe that both RAD-Q and RAD-V learn low-rank reward matrices. We thus conclude that both these models have needed capacity to represent the incomplete (R(R) matrices obtained from the datasets.
D.4 Numerical rank
To compute rank of n x m matrix, we use the default cutoff in N and PyTorrhh" at the time of writing, which is to say count only singular values above max (m,, n)Ea1, where is the machine epsilon for the corresponding data type, i.e., the difference between 1.0 and the next smallest representable number larger than 1.0, and o is the largest singular value.
There are potential issues that may arise when computing the numerical rank. One issue that the singular values, especially for the matrices coming from 32bit. float precision neural network, will not be exactly zero, so this why librarics like Numpy or PyTorch use a precision-basel cutoff for singular values that should be considered indistinguushable from zero; use the default such parameters. The other issue is that the
ohttpe://nunpyyorr//beee /tt. min.amggyiillaltar.r..... matrii rtmk. http:://///oocceee////////tat/te linal. matrix. rtmk
21

---

Published in Transactions on Machine Learning Research (08/2025)
80
80
60-
60
RAD
RAD
ARM distil
ARM distil
ARM orig labels
ARM orig labels
0-
0
500
1000
1500
2000
2500
3000
3500
4000
0
500
1000
1500
2000
2500
3000
3500
4000
contexts
(a) Deloxification task.
(b) Sentiment task.
Figure We numerically estimate the ranks both BRAD-V and PRAD-Q increasing the number of training prefixes (rows of R). In all cases, the ranks tend to be less than the model dimension = 764. This mcans that rank-capacily of RAD-Q, is sufficient to capture the training dalasets for the detoxification and sentiment. tasks.
number of rows in the reward matrices is very high and follow the work of Finlayson et al. (2024) and estimate rank by sampling rows from the matrix. Different submatrices can have different ranks, but we sample i.dd. to prevent this.
E
Training Details
To train reward models, we reuse the hyperparameters from Deng & Raffel (2023), where possible. We finetune the reward models with Adam optimizer (Kingma Ba, 2015) with 31 = 0.9, 32 = 0.95, e = le-12, 12. We use weight decay 0.01, batch size 100, and the learning rale changes linearly from the initial value (10-5 by default) to zero.
To train RAD-Q, we initialize the parameters with the pretrained GPT.2-Saal/ThyLLaaaa weights, and freeze the shared input-oulpul embedding parameters. Alternative strategy would be to use parameter efficient finetuning (Hu et al., 2022; Sidahmed et al., 2024).
E.1 Detoxification
For the detoxification task, we finetune RAD-Q with the learning rate 10-5 for 5 epochs.
For the LLaMa-2, we additionally finetune RAD-V with the LLayLLaMa backbone for the fair comparison with RAD-Q.
E.2 Sentiment Control
To finetune RAD-Q on responses only for sentiment control task, we first finetune the model with the learning rate 1 on the Amazon Polarity dataset, and then finetune it for 5 epochs on the SST-2 dataset with the learning rate 2c-6. For distillation experiment, we finctune RAD-Q for 5 epochs with the lcarning rate 10- on Amazon Polarity dataset.
F MAUVE
To complement perplexity as measure of fluency, we use MAUVE (Pillutla et al., 2021) of the fluency metrics. For reference texts, we take the generations of unguided model (GPT-2, or LLaMa-2-(7bb) /13b). Thus, this metric should capture how close the distribution of the continuations of a guided model is to the distribution of the original language model. To calculate MAUVE, we follow recommendations of He et al. (2023) and use ELECTRA-larg model to obtain the text representations. We use the hyperparameters of Pillutla et al. (2021): = 5 for the scaling constant; m for the quantization algorithm with 500
hnttps://hugginggface. httte//aaaaaaaaa/aaaaiiii.aanna aB-intermediate- steeeesste-11555 14
22

---

Published in Transactions on Machine Learning Research (08/2025)
0.30
DExperts
Average Maximal Toxicity
RAD-Q distil
0.25
0.20
0.15
0.10
12
14
16
18
20
22
24
26
Perplexity
Figure 10: Comparison of coxicity/fluende tradc-off between RAD-- (distill) and DExperts. We rerun the sampling from these two models using lop-k decoding with k = 20. Results are calculated over randomly selected 1000 prompts. We observe, that RAD-Q show better constraint satisfaction/ /fluencyunyy than DExperts.
iterations, and n/10 clusters where n the number generations. To compute MAUVE, we use 1000 prompts from the evaluation dataset.
G Results
G.1
Detoxification
G.1.1 Results with Perspective API classifier
In this section, we report full results with the Perspective API as taxicity classifier
GPT-2. Results for the detoxification task with the GPT-2-Large base model and GPT-2-small reward model, are presented Table 3.
We present the results for RAD-Q and RAD-V with top-k decoding with 40 Figure 13. We observe similar relative performance of RAD-Q compared to RAD-V as in the experiment with & 20, presented in the Figure 4.
LLaMa-2. Results for detoxification task with 2La-M122-77/13b) base model and TinyLLaMa reward model are presented Figure 5 and Table 5.
Baselines. Additionally, in Figure 14, we include results from Deng Raffel (2023) for other baseline models (for an older version of Perspective APT).
To highlight the difference between the RAD-Q and DExperts, show the trade-off plot for DExperts model in Figure 10, varying the scalar parameter for DExperts. As we observe, the RAD-Q has better constraint satisfaction / fluency trade-off than DExperts model. We attribute this to the difference in the training objectives of the expert models (reward modeling or language modeling), as argued in (Deng & Raffll 2023).
G.2
Additional Ablation Results
G.2.1 Loss choice
We perform an ablation study for the choice of the loss function used to train RAD-Q Equation (12). There, follow the approach of (Deng & Raffel, 2023), where they introduce the squared loss (see section 2.1
23

---

Published in Transactions on Machine Learning Research (08/2025)
0.30
RAD-Q ce
Average Maxima Toxicity
RAD-Q mse
0.25
0.20
0.15
0.10
18
20
22
24
26
28
Perplexity
Comparison of RAD-Q model trained on original responses with squared loss vs with cross-entrops loss. We rerun the sampling from these two models using top-k decoding with k = 20. Results are calculatod over randomly selected 1000 prompls. We observe, that RAD-Q trained with the squared loss show slightly better constraint cedisfaction/fluennn than RAD-Q trained with cross-entropy loss.
Figure 11:
RAD-Q MLP
Average Maximal Toxicity
RAD-Q
0.25
0.20
0.15
0.10
Perplexity
Comparison of RAD-Q trained on original responses with linear parametrization vs with non-linear MLP parametrization. We rerun the sampling from these two models using top-k decoding with k 20. Results are calculated over randomly selected 1000 prompts. We observe, that both parametrizations perform very closcly.
Figure 12:
24

---

Published in Transactions on Machine Learning Research (08/2025)
Unidirectional Reward Model). An alternative strategy would be use the binary cross-cntropy loss, using the fact that for our datasets responses y are from [0: range:
Lee (f(el')))) y, My (yggo((()))))) + (1- -1
(18)
where we introduced (z) 1/(1++==)) function softly map the predictions of RAD-Q into [0; range, which is also used during generation. Figure 11 demonstrates that the RAD-Q trained with the squared loss slightly outperform the RAD-- trained with the binary cross-entropy loss.
G.2.2 MLP vs Linear Parametrization
In this ablation, we consider replacing the linear parametrization of RAD-Q Equation (10) with a non-linear MLP parametrization:
(FRAD-CHMLP(2) Wio(W,"""""<()"))
(19)
where W1 Id W2 e Wild As we observe in Figure 12, MLP parametrization performs on par with the linear parametrization. We thus recommend using more simple lincar parametrization.
G.3
Sentiment control
Here, Figure 15, we include the additional results for the RAD-V and RAD-Q with top-k decoding with k: 40.
G.3.1 Results with RoBERTa classifier
In addition to taxicity scores with Perspective API, we provide the results with the open-weight RoBERTa loxicity classifier (Corrêa, 2023) for the guided gemeration with GPT-2 (Figure 16 and Table 6) and the LLaMa-2 (Figure 17 and Table 7). We notice that results for the average maximal toxicity with RoBERTa are relatively similar to the result with Perspective API. We hope that with an open-weight classifier it will be easier for the community directly compare to the published results without the need to recompute the APT scores.
0.35
Average Maximal Toxicity
0.30
0.25
0.20
RAD-Q k=40 distill
RAD-Q k=40 resp. only
0.15
* RAD-V k=40
GPT-2
0.10
GeDi
10
30
40
DExperts
Average Perplexity
0.35
Average Maximal Toxicity
0.30
0.25
0.20
0.15
0.10
1.0
0.9
0.8
0.7
0.6
MAUVE
Figure 13: Additional results for detoxification task for RAD-Q and RAD-V with k 40.
25

---

Published in Transactions on Machine Learning Research (08/2025)
RAD-V k=
RAD-V k=50
Average Maximal Toxicity
0.35
X
GPT-2
GeDi
0.30
DExperts
Rectification
0.25
PPLM
DAPT
0.20
PPO
0.15
Quark
0.10
10
20
30
40
50
60
Average Perplexity
Detoxification results reported in Deng & Raffel (2023) with Perspoctive API with GPT-2-Large model (API queries made between May and June 2023).
Figure 14:
G.4
Sentiment Control
Results for sentiment control task with the GPT-2-Large are presented in Table 4.
G.5
Generated Examples
Examples for the detoxification and sentiment control are presented in the Table 8, Table and Table 10.
26

---

Published in Transactions on Machine Learning Research (08/2025)
1.0
Neutral prompt Positive Rate
0.9
0.8
RAD-Q k 40 distill
RAD-Q k=40, resp. only
0.7
RAD-V k=40
0.6
GPT-2
CTRL
GeDi
DExperts
0.6
DAPT
Negative prompt Positive Rate
Quark
0.2
30
40
50
60
Average Perplexity
1.0
Neutral prompt Positive Rate
0.9
0.8
0.7
0.6
0.6
Negative prompt Positive Rate
0.4
0.2
1.00
0.95
0.90
0.85 MAUVE
0.80
0.75
0.70
Additional results for sentiment control task with = 40. For this plot with average perplexity, we include the results from Deng & Raffel (2023) for other basclines for reference.
Figure 15:
27

---

Published in Transactions on Machine Learning Research (08/2025)
Average Maximal Toxicity
0.2
ARM k- 20 distill
ARM k 40 distill
0.1
ARM 20 labels only
ARM k=40 labels only
RAD k=20
RAD km40
Average Perplexity
X
GPT-2
GeDi
Average Maximal Toxicity
0.3
DExperts
0.2
0.1
1.0
0.9
0.8
0.6
MAUVE
Figure 16: Detoxification results with finetuned RoBERTa toxicity classifier (Corrêa, 2023) and the GPT-2- Large base model.
Average Maximal Toxicity
RAD-Q distill Liama-2-7b-hf
0.3
RAD-V Llama-2-7b-hf
RAD-Q distill Llama-2-13b-hf
RAD-V Llama-2-13b-hf
0.2
X Raw Llama-2-7b-hf
Raw Llama-2-13b-hf
0.1
20
25
30
35
40
45
50
55
60
Average Perplexity
Average Maximal Toxicity
0.3
0.2
0.1
1.00
0.95
0.90
0.85
0.80
0.75
0.70
0.65
MAUVE
Figure 17: Detoxification results with finetuned RoBERTa toxicity classifier (Corréa, 2023) and the LLaMa-2
family of models.
28

---

Published in Transactions on Machine Learning Research (08/2025)
% Toxicity (4)
Fluency
Diversity (1)
Model
3
Avg. Max
Toxic
PPL (L)
MAUVE (1)
Dist 2
Dist 3
8
Toxicity
Rate
10
0.301
0.139
11.70
0.96
0.81
0.84
20
0.270
0.096
11.73
0.96
0.81
0.84
30
0.246
0.071
11.77
0.96
0.81
0.84
2=
50
0.212
0.043
11.98
0.95
0.81
0.84
100
0.160
0.019
12.67
0.95
0.80
0.83
200
0.117
0.005
15.78
0.88
0.78
0.81
300
0.097
0.002
24.53
0.75
0.75
0.79
RAD-Q distill
10
0.304
0.137
14.68
0.97
0.83
0.85
20
0.270
0.092
14.73
0.97
0.83
0.85
30
0.245
0.064
14.90
0.97
0.83
0.85
k=40
50
0.210
0.039
15.14
0.96
0.83
0.85
100
0.158
0.013
16.26
0.94
0.83
0.84
200
0.112
0.003
21.28
0.86
0.81
0.83
300
0.095
0.002
32.27
0.68
0.78
0.80
10
0.278
0.097
11.71
0.96
0.81
0.84
20
0.241
0.053
11.81
0.97
0.81
0.84
30
0.218
0.029
12.02
0.96
0.81
0.84
k=20
50
0.185
0.014
12.26
0.96
0.81
0.84
100
0.143
0.004
14.79
0.93
0.80
0.83
200
0.113
0.002
25.31
0.82
0.76
0.79
300
0.102
0.002
45.82
0.65
0.72
0.75
RAD-Q resp. only
10
0.280
0.091
14.72
0.97
0.83
0.85
20
0.242
0.046
14.92
0.96
0.83
0.85
30
0.217
0.028
15.09
0.96
0.83
0.85
k=40
50
0.185
0.013
15.69
0.96
0.83
0.85
100
0.142
0.003
18.84
0.92
0.82
0.84
200
0.111
0.002
39.53
0.77
0.79
0.80
300
0.103
0.002
83.36
0.54
0.74
0.76
10
0.265
0.077
11.73
0.96
0.81
0.84
20
0.231
0.040
11.81
0.96
0.81
0.84
30
0.211
0.024
11.87
0.96
0.81
0.84
k=
50
0.184
0.014
12.09
0.95
0..1
0.84
100
0.149
0.005
12.64
0.93
0.81
0.83
200
0.115
0.002
14.98
0.84
0.79
0.81
300
0.099
0.001
19.08
0.69
0.76
0.78
RAD-V
10
0.267
0.072
14.86
0.96
0.83
0.85
20
0.232
0.036
14.87
0.96
0.83
0.85
30
0.211
0.021
14.99
0.95
0.83
0.85
k=40
50
0.185
0.011
15.26
0.95
0.83
0.85
100
0.146
0.005
16.30
0.93
0.83
0.84
200
0.114
0.002
20.69
0.80
0.82
0.83
300
0.098
0.001
30.36
0.60
0.79
0.80
Table Results for detoxification task with the Perspective APT as a toxicity classifier. Calls the
Perspoctive API were performed in June-July 2024.
29

---

Published in Transactions on Machine Learning Research (08/2025)
% Positive Rate (t)
Fluency
Diversity (t)
Model
Negative
Neutral
PPL (4)
MAUVE (t)
Dist 2
Dist
10
12.94
81.08
12.16
0.96
0.76
0.78
20
24.87
91.00
12.85
0.94
0.75
0.78
30
35.18
94.87
14.11
0.92
0.75
0.78
k=
43.60
96.60
15.74
0.89
0.75
0.78
50
49.84
97.38
18.03
0.86
0.74
0.78
60
55.34
97.87
20.09
0.81
0.73
0.77
RAD-Q distill
10
13.50
80.97
15.53
0.95
0.78
0.79
26.66
91.45
17.20
0.94
0.78
0.79
39.12
95.32
18.29
0.90
0.78
0.80
k=
40
48.28
96.98
20.57
0.86
0.77
0.79
50
55.94
97.80
24.36
0.82
0.76
0.79
60
61.39
98.21
28.20
0.77
0.75
0.78
10
12.13
80.02
12.19
0.96
0.75
0.78
21.24
89.06
13.67
0.95
0.75
0.78
30
29.94
92.66
15.29
0.93
0.74
0.78
k=
40
37.38
94.62
17.06
0.89
0.74
0.78
50
43.19
95.65
20.11
0.85
0.72
0.77
60
47.19
96.20
23.07
0.82
0.71
0.76
RAD-Q resp. only
12.17
79.49
15.58
0.95
0.78
0.79
22.82
89.40
17.12
0.94
0.77
0.79
30
32.63
93.22
19.46
0.91
0.77
0.79
k=
40
41.58
95.15
24.36
0.87
0.76
0.79
50
47.98
96.10
27.48
0.81
0.75
0.79
60
53.76
96.58
30.91
0.76
0.74
0.78
19.94
86.06
12.61
0.95
0.75
0.78
20
35.37
92.70
14.87
0.92
0.75
0.78
30
43.87
94.82
17.36
0.87
0.74
0.78
k=20
40
48.51
95.74
20.35
0.83
0.73
0.77
50
50.96
96.20
23.78
0.80
0.72
0.76
60
52.99
96.62
28.36
0.76
0.71
0.75
RAD-V
22.03
86.56
16.20
0.95
0.78
0.79
20
40.09
93.14
19.90
0.91
0.78
0.80
30
50.61
95.16
23.45
0.85
0.77
0.79
4=
40
55.77
96.05
27.74
0.80
0.76
0.79
50
58.69
96.54
33.55
0.76
0.75
0.78
60
60.66
96.81
41.57
0.72
0.74
0.77
Table 4: Results for sentiment control task with GPT-2 model.
30

---

Published in Transactions on Machine Learning Research (08/2025)
Toxicity (L)
Fluency
Diversity (t)
Model
Base LM
8
Avg. Max Toxicity
Toxic
PPL (4)
MAUVE (t)
Dist 2
Dist 3
Rate
10
0.260
0.092
21.88
0.99
0.79
0.81
50
0.181
0.022
23.16
0.97
0.79
0.81
LLaMa-2-7b
100
0.142
0.010
25.28
0.95
0.79
0.81
200
0.103
0.003
37.92
0.87
0.77
0.79
300
0.082
0.002
47.13
0.75
0.74
0.76
RAD-Q distill
10
0.268
0.104
22.74
0.99
0.79
0.81
50
0.188
0.027
23.58
0.97
0.79
0.80
T.aMa-2-13b
100
0.148
0.013
27.59
0.95
0.78
0.80
200
0.108
0.004
36.87
0.87
0.76
0.78
300
0.086
0.003
55.53
0.76
0.73
0.75
10
0.244
0.069
21.76
0.99
0.79
0.81
50
0.162
0.010
22.62
0.96
0.79
0.81
LLaMa-2-7b
100
0.123
0.004
27.69
0.94
0.79
0.80
200
0.088
0.002
41.27
0.83
0.77
0.78
300
0.072
0.002
53.68
0.65
0.74
0.75
RAD-V
10
0.252
0.079
22.15
0.99
0.79
0.80
50
0.169
0.012
23.74
0.97
0.79
0.80
LLaMa-2-13h
100
0.128
0.004
31.25
0.94
0.78
0.80
200
0.091
0.002
48.09
0.83
0.76
0.77
300
0.075
0.001
60.82
0.69
0.73
0.74
Table 5: Results for detoxification task with LLaMa-2 base models. Toxicity metrics computed with
Perspective API.
31

---

Published in Transactions on Machine Learning Research (08/2025)
Avg. Max Toxicity
Toxicity
Model
Rate
10
0.286
0.270
20
0.239
0.220
30
0.209
0.190
k=
50
0.160
0.140
100
0.089
0.070
200
0.038
0.025
300
0.022
0.011
RAD-Q distill
10
0.297
0.282
20
0.247
0.232
30
0.209
0.192
k=40
50
0.154
0.133
100
0.084
0.066
200
0.034
0.020
300
0.021
0.009
10
0.257
0.238
20
0.212
0.192
30
0.178
0.158
k=20
50
0.135
0.112
100
0.084
0.063
200
0.056
0.035
300
0.050
0.029
RAD-Q responses only
10
0.267
0.249
20
0.214
0.193
30
0.179
0.157
k-40
50
0.133
0.109
100
0.083
0.061
200
0.060
0.036
300
0.054
0.031
10
0.242
0.223
20
0.189
0.167
30
0.159
0.137
2=20
50
0.118
0.097
100
0.072
0.052
200
0.039
0.021
300
0.027
0.011
RAD-V
10
0.245
0.225
20
0.189
0.166
30
0.159
0.137
k=40
50
0.116
0.090
100
0.071
0.048
200
0.037
0.019
300
0.025
0.008
Table 6: Additional results for detoxification task with the GPT-2 and the RoBERTa (Corrêa, 2023)
toxicity classifier. Other metrics the same in Table 3.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Toxicity (4)
Model
Base LM
Avg. Max Toxicity
Toxic
Rate
0.273
0.256
50
0.160
0.139
LLaMa-2-7b
100
0.110
0.090
200
0.069
0.050
300
0.051
0.030
RAD-Q distill
0.285
0.266
50
0.176
0.151
LJMMa-2-13b
100
0.122
0.103
200
0.074
0.052
300
0.054
0.035
0.254
0.233
0.135
0.112
LLaMa-2-7b
100
0.091
0.070
200
0.055
0.033
300
0.044
0.024
RAD-V
0.267
0.246
50
0.150
0.125
LLaMa-2-4 2-bbb
100
0.098
0.072
200
0.061
0.040
300
0.048
0.028
Table 7: Additional results for the detoxification task with the LLaMa-2 base models with the RoBERTa (Corrêa, 2023) toxicity classifier.
33

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Most toxic samples (out of 25 generated) from the RAD-Q with different, B coefficients, and corresponding Toxicity scores from the Perspective API.
Table 9: Detoxification task: samples from the RAD-Q and RAD-V V with & = 20, B = 100. Examples for the GeDi and DExperts are from Deng &
Table 10: Sentiment control task: samples from the RAD-Q and RAD-V with & = 20, 3 = 30. Examples for the GeDi and DExports are from Deng &
need to find the correct pieces for it and the right combination of characters and weapons to kill the bosses need the focns for this one. Personally; I had a good time in jokingly calling out Card need a versatile DH player and Dahl will certainly help you as he's ideally positioned at the back of still be playing against well-prepared team. In this day and age of instant gratification we
in the blood of an innocent child may be of no consequence the parent dies before he or she gathered, our new friends danced, jests were merrily spiced, and plenty of songs fired fields may not seem like the perfect areas for reclaiming coal and steel, but technology has brought mining RAD-V industries, and also the energy gy and materials markets in the United States, have grown and matured.
The
probably want to make sure you do at least some research on a team before deciding, as the competition
Prompt: X") received a series of text messages from her attacker, including one that read "If I....
Toxicity
0.86
0 0.89
0.47
0.24
over have the opportunity to meet with anyone in real life, it would you, I know
in my life have a great effect on me. They bring me to life when think of all
have to, I will kill myself and then I will take all of you as well and all of get out of here I'm going kill you" "which was translated into get in my car I'm going to drive over curb and break it in two,"
Prompt: No matter who you'll pick you'll...
Meanwhit hile the iron and lead...
Continuation
DExperts
Prompt:
DExperts
10
50
100
300
GPT-2
RAD-V
RAD-Q
GPT-2
RAD-Q
GeDi
GeDi
Raffel (2023).
Raffel (2023).


Reviews:
[{"review_id": "KKGD4ha7hu", "reviewer": "Reviewer_Jz7g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- I really like the framework proposed here for thinking about task complexity in terms of rank over data, it’s definitely very fresh in the context of reward modeling, and the framing and connection to efficiency is unique. I’d in fact think that this framework has broader implications and applications for language modeling even outside the context of reward-guided decoding\n- The experiments generally seem reasonable, and reveal some interesting patterns (such as lower rank sometimes not affecting quality so much with distillation) that I think will be useful for further exploration. \n- I appreciate the usage of example figures to walk through some of the key ideas (e.g. Figure 3)\n- There’s a good number of baselines\n- The inclusion of a comparison of different distillation approaches is interesting, though maybe not analyzed as much\n\nWeaknesses: \n- A very serious limitation is that this work doesn’t seem connected to a lot of the empirical reward modeling work being done at larger-scale realistic chat settings (e.g. RewardBench submissions such as ARMoRM) or math reasoning settings, which are the main context in which reward models are typically discussed. Given how simple sentiment and toxicity control are highly simple tasks (even in the older context of controllable generation work cited by this paper) I worry that these findings might be highly specific and not generally applicable, e.g. maybe most realistic control or reward functions actually do need higher rank reward functions. The paper acknowledges this but doesn’t dig into this empirically. \n- The explanation of the idea of rank felt a little unclear to me, especially in the context of true reward, I think a figure to visualize the key ideas of the paper, and maybe a simple worked example (e.g. of reward and rank computation) w", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is an interesting paper exploring the difference between a language-modeling (Q) style parameterization (passing in entire vocabulary) of reward models and the typical parameterization of reward models (V) which involves needing a full forward pass for any individual token. To summarize my understanding of the paper: \n1. It first provides some background on different approaches related to controlled decoding and how they can fit within a dichotomy of Q-style vs V-style reward models. This dichotomy is not entirely new however my understanding is that this paper is an early paper taking this dichotomy into mind more seriously.\n2. It then provides a novel mathematical framework for thinking about the task and representational complexity of reward models using a matrix-based framework where, if I roughly understand correctly, rows represent different possible prefixes, and columns represent possible continuation tokens, and the score represents some expected reward when continuing from the prefix given the new token. They then discuss things in terms of the rank of these matrices for both reward formulations. \n3. My understanding is that their central set of claims are that (a) counter to intuition, V-style reward models don’t actually tend to learn expressive high-rank matrices, possibly due to data sampling biases and also tasks maybe being too simple to need high-rank reward matrices, and (b) due to softmax bottleneck Q-style reward models are fundamentally limited to lower reward rank, but that this may be ok given (a). There’s additional discussion of how the structure of data may affect these conditions.\n4. They then present some experiments on fairly simplistic sentiment classification and toxicity tasks which somewhat validate the above claims, in addition to showing some small gains with a novel distillation approach from V to Q rewards. Likewise there’s some interesting analysis and evidence suggesting a decoupling between rank of learned representations and quality/control tradeoffs. \nMy understanding is that the primary contributions here are the framing and analysis of reward through the idea of rank matrices, and a stronger case for Q-style reward models compared to initial prior work in that area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggested Changes: \nMajor: I think any sort of experiment examining a more realistic math or chat setting (e.g. using ARMoRM) would really improve this paper. To understand (a) cases where the task requires higher rank to satisfy (b) understand whether the assumption that Q-style and V-style rewards can function similarly in complex reward modeling tasks used in practice. The paper is currently a bit more cautious in its claims which I appreciate, but I think this investigation shouldn't be too difficult to run and could allow the paper to make stronger claims that would make the paper much stronger.\nMinor: This paper looks primarily at RAD, which is one way of controlling generation with reward models, however it could be good to understand other algorithms for reward-guided decoding, such as a simple best-of-n approach. I understand this is more computationally expensive so it’d mainly helpful context\nMinor: Motivation for improving cost-effectiveness can either be to (a) reduce current overhead or (b) allow greater scaling of inference-time compute. I’d be curious how the rank tradeoffs here affect for example best-of-n over candidates sampled from RAD-conditioned distribution (e.g. do these findings apply or change when we’re trying to scale up inference time compute), again in particular for more challenging control tasks or rewards\n\n[1] ARMORM -> Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts\n[2] RewardBench -> RewardBench: Evaluating Reward Models for Language Modeling", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uXQHojzyAE", "reviewer": "Reviewer_9byQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper provides a method that may be an efficient alternative to an established reward-augmented decoding approach (RAD-V). The connection made to matrix-sensing literature in terms of parameterizing the reward matrix may be useful to the theory folks working on controllable generation. \n\n- Empirical results on sentiment control and detoxification suggest that the proposed RAD-Q method is competitive with baseline RAD (RAD-V) method. \n\n## Weaknesses\n\n- The paper goes over why one may not want to always assume reward models can be low-rank. However, the paper's presentation currently delays the reader from getting to the core contribution which is RAD-Q which maybe confusing to the reader. Changing up the order of presentation could help improve readability\n\n- Related to above, it is not clear to the reader whether the argument about high-rank vs low-rank models is even relevant in practice. On the one hand, by construction the paper provides an example of full-rank reward matrix but the empirical results suggest that low-rank reward matrices are a reasonable assumption. Are there practical use cases where low-rank parameterization does not work well? \n\n- The approach used in the paper to calculate matrix rank is a concern. Have you the authors considered alternative approach like effective rank [1]? My concern is the rank of a matrix is heavily dependent on the underlying SVD implementation which can change over time and also across precisions. Perhaps a measure like effective rank might be better?\n\n[1] Roy & Vetterli https://www.eurasip.org/Proceedings/Eusipco/Eusipco2007/Papers/a5p-h05.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the use of reward models for controlled generation in language models (LLMs). Specifically, the paper proposes a low-rank parameterization of reward models that leads to an efficient reward augmented decoding (RAD) method. Additionally, the paper aims to show that the existing RAD methods are not necessarily low-rank and that there may exist reward matrices that are high rank by construction. The proposed method, RAD-Q, is empirically shown to work reasonably well on detoxification and sentiment control datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please see question sunder weaknesses, summarized below:\n  - Would the paper read better if the order of presentation is changed? Present RAD-Q right away so there is no confusion on this method being the main contribution\n  - Are there cases useful in practice that shows full-rank reward matrix\n  - Is effective rank a better measure of matrix rank for the analysis conducted in the paper?\n\n- There appears to be a typo on Page 3 where-in the papers cited for GeDI and DExperts are swapped. If this is the case, please fix.\n\n- In Figure 2, the use of \"d\" to compare to numerical rank is confusing. Does \"d\" have to be displayed if the reward matrix is of size N by |V|?", "broader_impact_concerns": "This paper proposes an approach to improve controlled generation. The paper would definitely benefit from having a Broader Impact Statement in its content.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "754YxukFk8", "reviewer": "Reviewer_PXVR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nThe paper provides an interesting matrix completion perspective on reward modeling, including formal definitions of minimal rank and ε-rank, clarifying the practical expressivity of reward models.\n\nThe experiments in this paper span multiple base models, two tasks, multiple metrics, and plenty of insightful ablation studies. \n\nWeaknesses:\n\nThis is not actually a limitation of this specific paper, but I wonder if the authors could elaborate more on the assumptions of low-rank and full-rank properties of different tasks. I'm aware this might be out of the scope of the current paper --- but would it be possible to control the rank in some synthetic setups and demonstrate the RAD-V/Q's performance against LLM fine-tuning methods?\n\nThe RAD-V - RAD-Q distillation may introduce an extra computational burden.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an analysis and practical implementation of low-rank parametrizations for reward models used in guided decoding of language models. The paper formalizes the expressiveness trade-off between two reward model styles --- V-style (token-conditioned) and Q-style (prefix-conditioned) --- and highlights that Q-style suffers from a “rank bottleneck” but can still suffice in practice. Methodologically, the paper proposes RAD-Q, a more efficient, low-rank variant of the RAD approach, achieving a single forward pass per decoding step (as opposed to RAD-V’s per-token cost), thus making decoding faster and more scalable. Empirically, experiments are conducted on detoxification and sentiment control tasks. Results demonstrate that RAD-Q performs comparably to or even better than RAD-V in both tasks, especially when trained via distillation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Elaborating more on the link between V/Q-learning and reward model learning (cf. Introduction section, page 1) could be useful. In Figure 1, additional illustrations on the plot units can be useful.\n\nCan the authors discuss more about their empirical insights in implementing the methods? --- Since the distillation method is a two-stage implementation, a detailed discussion on the engineering side can be helpful for the paper's readers to quickly adapt the method to their tasks.\n\n\nMinor:\n\ntypo in Figure 4 Caption: distil -> distill \n\nI would recommend including the LLaMA results in the main text.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)