PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Authors: Baijiong Lin
Weisen Jiang
Yuancheng Xu
Hao Chen
Ying-Cong Chen
Venue: ICML 2025 poster (2025)
Keywords: LLM Alignment
Test-time Alignment
Multi-objective Alignment
Primary area: deep_learning->large_language_models

Abstract: Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi-dimensional user preferences during inference while keeping LLMs frozen. 
Recently, GenARM (Xu et al., 2025) first independently trains Autoregressive Reward Models (ARMs) for each preference dimension without awareness of each other, then combines their outputs based on user-specific preference vectors during inference to achieve multi-objective test-time alignment, leading to two key limitations: the need for *multiple* ARMs increases the inference cost, and the *separate* training of ARMs causes the misalignment between the guided generation and the user preferences.
To address these issues, we propose Preference-aware ARM (PARM), a *single* unified ARM trained across *all* preference dimensions. 
PARM uses our proposed Preference-Aware Bilinear Low-Rank Adaptation (PBLoRA), which employs a bilinear form to condition the ARM on preference vectors, enabling it to achieve precise control over preference trade-offs during inference.
Experiments demonstrate that PARM reduces inference costs and achieves better alignment with preference vectors compared with existing methods. Additionally, PARM enables weak-to-strong guidance, allowing a smaller PARM to guide a larger frozen LLM without expensive training, making multi-objective alignment accessible with limited computing resources.
The code is available at https://github.com/Baijiong-Lin/PARM.

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Baijiong Lin
Weisen Jiang? Yuancheng Xu
Hao Chen
Ying-Cong Chen 14
Abstract
1. Introduction
Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi- dimensional user preferences during inference while keeping LLMs frozen. Recently, GenARM (Xu et al., 2025) first independently trains Au- toregressive Reward Models (ARMs) for each preference dimension without awareness of each other, then combines their outputs based user- specific preference vectors during inference to achieve multi-objective test-time alignment, lead- ing to two key limitations: the need for multi- ple ARMs increases the inference cost, and the separate training of ARMs causes the misalign- ment between the guided generation and the user preferences. To address these issues, we propose Preference-aware ARM (PARM), a single unified ARM trained across all preference dimensions. PARM uses our proposed Preference- Aware Bi- linear Low-Rank Adaptation (PBLoRA), which employs a bilinear form to condition the ARM on preference vectors, enabling to achieve pre- cise control over preference trade-offs during in- ference. Experiments demonstrate that PARM reduces inference costs and achieves better align- ment with preference vectors compared with exist- ing methods. Additionally, PARM enables weak- to-strong guidance, allowing a smaller PARM to guide a larger frozen LLM without expensive training, making multi-objective alignment ac- cessible with limited computing resources. The code is available at https:/ / github.ccom com/ Ba LinngALin/PARM PPAM.
The alignment of large language models (LLMs) is crucial to ensure that their outputs reflect human values (Wang et al., 2023; Casper et al., 2024). In practice, human preferences and values are often multifaceted and may conflict. For example, users may expect LLM responses to be simultane- ously helpful, harmless, and humorous. These competing objectives pose a challenge for single-objective alignment methods to meet such complex demands. To address this, multi-objective alignment enables LLMs dynamically ad- just trade-offs among different preference dimensions based specific user needs (represented as preference vector).
Current multi-objective alignment methods (Zhou et al., 2024; Rame et al., 2023; Jang et al., 2023; Wang et al., 2024a; Guo al., 2024; Yang et al., 2024b; Zhong et al., 2024) require extensive computations for LLM training that many researchers and practitioners cannot access when the LLM is large (c.g.. with 65B parameters). Different from existing methods, we focus on multi- multi-objective test-time alignment that trains a small reward model rather than the original LLM, reducing computation cost largely and mak- ing multi nulti-objective alignment accessible with limited com- puting resources.
GenARM (Xu et al., 2025), the recent state-of-the-att test- time alignment method, introduces an Autoregressive Re- ward Model (ARM) to predict token-level rewards for guid- ing the generation of frozen L.MSs during inference, result- ing in effective and efficient test-time alignment. However, when adapted to multi-objective settings, GenARM requires training an ARM for each objective. During inference, each ARM computes its respective next-token reward, and the LLM's generation is guided by weighted sum of these re- wards using the given preference vector as weights. Hence, GenARM faces two main limitations in multi-objective test- time alignment: (i) the need for multiple ARMs generation increases the inference cost, and (ii) the separate train- ing of ARMs leads to potential misalignment between the guided generation and the specified preference vector.
'The Hong Kong University Science and Technol- ogy (Guangzhou) "The Chinese University of Hong Kong l Uniiersityy of Maryland, College Park "The Hong Kong Uni- versity of Science and Technology. Conresoondence to: Bai- jiong Lin <bj.lin.emate eiilee com,, Ying-Cong Chen <ying- congchen @ ust.hk>..
To address these limitations, we propose preference-aware ARM (PARM) to achieve effective and efficient multi- objective test-time alignment. Unlike GenARM, which
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Table 1: Comparison between our PARM and existing multi-objective alignment methods. Note that the reward model can be smaller than the policy model (for example, in Section 5.1, a 7B model can guide a frozen 65B model), k: the number of preference dimensions. "-": Not Applicable. *: The weak-to-strong variant of MOD. f: Rewarded Soups and CLP merge multiple models into one the parameter space according to the given preference vector at inference.
Trained before Inference
Used in Inference
Base Models
Reward Models
Base Models
Reward Models
requiring training the base model
Rewarded Soups (Rame al., 2023) MOD (Shi et al., 2024)
k
1
k
k:
CLP (Wang et al., 2024b)
k + 1
keeping the base model frozen
MOD-w2s* (Shi et al., 2024) GenARM (Xu et al., 2025)
k
k-
k
1
k
PARM (ours)
1
1
independently trains ARMs for each preference dimension without awareness of other dimensions, PARM is a single unified model jointly trained across all preference dimen- sions to explicitly optimize trade-offs between different preferences. Moreover, PARM is conditioned on prefer- ence vectors, allowing it to dynamically adjust the output reward according to the user-specific preference vector dur- ing inference, thereby guiding the frozen LLM to generate responses that align with the given preference vector while maintaining computational efficiency.
making multi-objective alignment accessible with limited computing resources.
The overall comparison between PARM and existing multi- objective alignment methods is shown in Table 1. As shown, PARM only needs to train single small reward model rather than the original LLM or multiple reward models. This significantly reduces computation cost, facilitating multi- objective alignment under computational constraints.
The contributions of this paper are summarized as follows: (i) We propose PARM, single unified ARM jointly trained across all preference dimensions, to achieve effective and efficient multi-objective test-time alignment;
To condition the PARM (which may contain billions of parameters) on low-dimensional preference vector, we propose preference-aware bilinear low-rank adaptation (PBLoRA). PBLoRA employs a bilinear form BWA, where B and A are low-rank matrices, similar to those used in LoRA (Hu al., 2022). W is an r X r weight matrix (r is the rank) that is conditioned on the preference vector. This conditioning allows the preference vector to directly control the generation of PARM through W. Moreover, we theoretically show that the bilinear form used in PBLoRA is more expressive than the original LoRA, enabling PARM to better capture the complex relationships between different preference dimensions. By training with PBLoRA, PARM steerable to output reward according to the user-specific preference vector, thereby guiding the frozen LLM in gen- erating responses aligned with the given preference vector during inference.
(ii) We pro-
pose PBLoRA to adapt the ARM to condition on the prefer- ence vector, enabling better management of trade-offs be- tween preferences; (iii) Experiments show that PARM sig- nificantly reduces inference cost while improving alignment performance compared with existing methods. Moreover, PARM enables weak-to-stron guidance, aligning larger LLMs with smaller PARM, eliminating the need for ex- pensive training of the larger models.
2. Related Work
Test-Time Alignment. Let Thase denote base model. As directly fine-tuning Thuse on preference data to align with the human values (Ouyang et al., 2022; Rafailov et al., 2023; Meng et al., 2024; Park et al., 2024) requires extensive com- putation on LLM training, test-time alignment methods keep Tasee frozen and use reward models to guide the generation during inference. Existing test-time alignment methods are inspired by the closed-form solution of RLHF in (Rafailov et al., 2023) follows,
We evaluate PARM on the safety alignment (Ji et al., 2023; 2024) and helpful assistant (Bai et al., 2022) tasks. Experi- mental results demonstrate that PARM has higher alignment quality and is more inference-efffcien than previous meth- ods in multi-objective test-time alignment. Moreover, we guidance ability of PARM, where smaller PARM can guide a larger frozen LLM (c.g., 7B guides 65B) without the need for training the larger LLM,,
highlight the
1 log ग (ylx)) = log 2(x) + log Thase (y|x) + r(xx
y),
(I)
2

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware A Reward Model
where π is the aligned model, 2xx is the partition function, r(x. y) is reward model, and [3 is hyperparameter. Ac- cording to Equation (I), when the base model hase is frozen, the generation is guided by the reward model r(x,y).
We also review multi-objective optimization and control- lable text generation in Appendix A.
3. Preliminary on ARM
When generating the next token from an incomplete re- sponse based on Equation (I), is necessary to predict re- wards for the next token. However, such token-level rewards cannot be directly obtained from response-level reward mod- els. Some test-time methods, like (Khanov et al., 2024; Li et al., 2024), attempt to compute rewards using incomplete responses, which often leads to inaccuracies. Alternatively, methods such as (Huang et al., 2024; Chakraborty et al., 2024) generate complete responses to compute rewards for each token, which significantly increases inference costs.
In this section, we review the recent test-time alignment method, GenARM (Xu et al., 2025), which uses autoregres- sive reward models (ARM) to guide the generation of frozen LLMs during inference.
ARM. ARM ís a token-level reward model, whose reward r(x, y) is computed as the sum of log probabilities of tokens generated up to the t-th token, as follows,
r r(x,y) = Σ log
Recently, GenARM (Xu et al., 2025) proposes the Autore- gressive Reward Model (ARM), which explicitly predicts token-level rewards to enable efficient and effective test-time aligmment. However, when extending to multi-objective sce- narios that need to handle trade-offs among multiple prefer- ence dimensions, GenARM has two significant limitations (introduced in Section 1. Hence, in this paper, we focus on improving the ARM to enable efficient and effective multi-objective test-time alignment. Similarly, PAD (Chen et al., 2025a) also employs token-level reward model to guide the decoding process. However, it focuses on aligning with personalized preferences rather than managing trade- offs across different preference dimensions.
where To (x,y yce) is learnable distribution function (pa- rameterized by 0) that predicts the next-token reward. Most practical language model architectures are autoregressive, c.g., the LLaMA family of models (Touvron et al., 2023), thus, can be employed for TØ.
Training of ARM. Let D = y'x.y',,,:x0 denote a pref- erence dataset, where yt and y² represent the different responses generated by Thuse in response to the prompt x, and = E [0, is the preference label (= = 1 if y is "bet- ter" response than y² otherwise 0). The ARM is trained on D using negative log-likelihood loss function as follows,
((TT, D) -xxxy" 32 log a (-1)²Br (o
Multi-Objective Alignment. In practice, human prefer- ences are multi-dimennional and often need to align LLMs to balance multiple, sometimes confficting, prefer- ence dimensions such as helpfulness, harmlessness, and humor (Yang et al., 2024b). Some multi-objective align- ment methods like (Wu et al., 2023; Zhou et al., 2024) train separate LLMs for each given preference vector by linearly combining multiple reward models. To reduce the training cost, some methods separately train specialized LLMs for each preference dimension and integrate either through pa- rameter fusion (Rame et al., 2023; Jang et al., 2023) or logit combination (Shi et al., 2024) during inference. However, this strategy still requires maintaining multiple models, re- sulting in significant storage and computational burdens. To further improve efficiency, some methods focus on adapting single LLM to accommodate varying preferences. This is achieved through encoding preference vectors into input prompts (Wang et al., 2024a; Guo et al., 2024; Yang et al., 2024b) or directly modifying model parameters (Wang et al., 2024b; Zhong et al., 2024).
log no (y (y²1))))
(3)
where a-)) is the logistic function and 8, is hyperparame- ter.
Guided Generation via ARM. GenARM (Xu et al., 2025) achieves test-time alignment by integrating the trained ARM into Equation (1) as
log π (ylx) = - log Z(x) +
Σ log Teuse (Be (x,)
1 Σ log (vllx,xe)))). (4)
+ +
The probability of the next token ye is conditioned on the partially generated response yer and prompt x as follows,
मे ( yet) X Thuse (vrl |x, y yet) (TO x. y (5)
which resembles decoding from multiple language models, enabling us to leverage prior methods such (Dekoninck et al., 2024).
However, existing multi mujti-objeectvv alignment methods typ- ically require direct fine-tuning of base L.LMs, incurring substantial computational costs. In this paper, we focus on multi-objective test-time alignment, where base LLMs re- main frozen, eliminating the need for expensive fine-tuning.
4. PARM: Preference-Aware ARM
In this section, we introduce the preference-aware ARM (PARM) for multi-objective test-time alignment.
3

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
4.1. Motivations and Problem Formulation
To learn the whole Pareto optimal solutions in a single run, we propose to train unified ARM conditioning the preference vector, i.e., 0(a), called the Preference-aware ARM (PARM). This conditioning enables a single ARM to approximate the whole Pareto set and effectively manage trade-off's across different preference dimensions. Therefore, given a preference vector a at inference, we can obtain the corresponding Pareto-optimal ARM without retraining and use this ARM to guide the frozen base LLM to generate responses aligned with the preference, thereby addressing the misalignment and inefficiency issues of GenARM.
Multi-objective test-time alignment aims to use reward mod- els to guide the base model to generate a response that aligns with multiidiiennsoonll user preferences during inference while keeping the base model frozen.
Let D = {(x, y", ,²,,,, ,1,,2,".. } denote a k-dimensional preference dataset, where 2, = 1 if y² is a "better" re- sponse than yd in the i-th preference dimension otherwise 0. We denote D; {(x, y²,, as the preference dataset for the i-th dimension. In multi-objective align- ment, users expect the LLM's outputs to align with their multi-dimensional needs, which can be represented as a preference vector, a = =(n,... ,0x) E Ag-1, where og denotes the weight for the i-th preference dimension and
In the following, we introduce how to condition the ARM on preference vectors in Section 4.2, how to train PARM in Section 4.3, and how to guide generation via PARM for multi-objective test-time alignment in Section 4.4.
Ax-1 = {al Ex= a; = 1, 2 0, = 1,.., , is (k - 1) dimeenioonll simplex.
4.2. Preference-Aware Bilinear Low-Rank Adaptation
The recent state-of-the-art method GenARM (Xu et al., 2025) first trains an ARM ie. on dataset Dj for each pref- erence dimension i. During inference, given preference vector a, the separated trained ARMs are then combined to guide the generation procedure as,
Similar to GenARM (Xu et al., 2025), we adopt the autore- gressive model for the reward model Té(a)) (-1x, y<r). The primary challenge is how condition the massive model parameters 0 (which may contain billions of parameters) on the k-dimensional preference vector a.
log π my|xx) = - log (x) -
Σ log Thase (x))
In this paper, we propose preference- adaptation (PBLoRA) for PARM, enabling efficient and ef- fective conditioning on preference vectors while maintain- ing computational scalability. Low-rank adaptation (LoRA) (Hu et al., 2022) is widely used parameter-efficient tech- nique for fine-tuning LLMss. However, the simple product of two low-rank matrices fails to account for user prefer- ences. To address this issue, we propose Preference-Aware Bilinear Low-Ramd Adaptation (PBLoRA) to condition on preference vectors as follows.
bilinear low-rank
log To (x))). (6)
GenARM faces two major limitations in multi-objective test- time alignment: (i) The k ARMs are trained independently on different preference dimensions without awareness of each other, causing potential conflicts when combining their rewards directly during inference (i.e., Equation (6)), result- ing in a mismatch between model outputs and the desired preferences. (ii) GenARM needs k ARMs to predict re- ward simultaneously in the generation process, causing a huge computational overhead during inference.
Let Oo Rmxx denote the pre-trained model weight. We propose bilinear form of LoRA follows,
0(a)) 0n + BBW(
(8)
where s a scaling factor as in LoRA, B R and A Rexn are learnable low-rank matrices. W(a) E RTxy is treated as weighted matrix that depends on the preference vector a. We introduce W in LoRA for two key reasons. First, since r < {m, generating the weighted matrix W using a is significantly more effective and efficient than generating B and A or even the full model parameter using a. Second, we theoretically demonstrate that the subspace that BWA lies in has dimensionality much higher than the subspace that BA lies in, enabling richer and more expressive representations.
To address these limitations, we aim to jointly train a single ARM across all preferences by optimizing the following multi-objective optimization problem,
min (e(TTo DI),,.,,((TT)) T
(7)
where ((TTO. D:) is the ARM training objective for the i-th preference dimension, defined in Equation (3). Since the in- dividual preference dimensions may conflict with each other, no solution can achieve optimal performance across all di- mensions. Instead, there exists a set of infinite Pareto opti- mal solutions, defined in Appendix B. Each Pareto-optimal ARM model 0 represents a unique trade-off among all pref- erence dimensions and is learned under a specific preference vector ax
Let (b, E RT :1 = I,... ,r} be the column vectors of B, (a. E R² 1, ,r] be the row vectors of A, and wij be the (ith) element of W.
Theorem 4.1. Assume both B and A have rank r. The
4

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
outer product of (b,:i=1..... .r with 1a.; , results in 22 linearly independent matrices (b,a]: ;
Algorithm 1 Training of PARM.
Require: initial model T ranks ri and r2 for PBLoRA, numbers of preference dimensions k. datasets for each
1, ;T;j=,
.r} in the space of m x matrices.
preference dimension {D >
The proof is provided in Appendix C. The bilinear form BWA- -. Ei= Wiy b is in the subspace spanned
1: Initialize the parameters of PBLoRA O;
2: while not converged do Sample a preference vector a from Ag-1; Compute ((a) via Equations (8) and (9); 5: for i in 1,... , do
by { :1=1.-... [.j = 1, ... rh. while the original LoRA formulation BA Σ 1 b is in the subspace spanned by (b,a, i = 1, ..... ..). According to Theo- rem 4.1, the former subspace has r² dimensionality and is r times higher than the latter (only r). Hence, the formulation BWA is more expressive than BA in the space of m X n matrices.
6: Sample a data batch B, from Dj;
7: Compute loss ((Te(a)) B₂) via Equation (3);
8: end for
9: Compute total loss Σ 0r (T(())(())) B:);
The term B in Equation (8) is preference-aware since W is conditioned on the preference vector As multiple objectives may share common knowledge (Zhang & Yang, 2022; Chen et al., 2025b), we split BW (a)). A into two terms: a preference-agnostic term to learn shared fea- tures and a preference-aware one to learn objective-specifc features, as follows,
10: Update @ via gradient descent;
II: end while
12: return (To, +).
4.3. Training of PARM
O
A
At the training stage of PARM,, we keep 6g frozen and only learn the parameters @ = {A1 A2. B1, B2, W1, Φ} for PBLORA. The training objective of PARM is formulated as follows.
BW(a) = [ B1 B₂
(2
A2
= B B₂ (2
(9)
preference-agnen agnostic
preference-wware
where + 12 r, B1 E I B2 E R A1 E Rrixn A E R Wi E Rrix are learnable pa- rameters (independent of a), and W2(a) E graxra is conditioned on a. In practice, we adopt a linear layer fa(a) R + R to generate W2(a)) where & is the parameters of this linear layer.
min E Bardi- Σ
Ead((To(a))) D.)
(10)
where {(TT, D) is defined as in Equation (3).
For a given preference vector o in problem (10), we mini- mize Σ a; D;), which is linear scalarization of problem (7) and its solution is Pareto-optimal (Boyd, 2004). Hence, different from GenARM (Xu al., 2025), which independently trains ARMs for each preference di- mension without awareness of each other, our PARM trains a single unified ARM across all preference dimensions to explicitly manage tradc-offs between different preferences. Moreover, we can obtain a single model ((xx) that approx- imates the entire Pareto set by optimizing problem (10), eliminating the need to retrain O under different preference vectors a during inference.
The preference-agnostic term B, A1 is shared among different a and thus can explicitly learn shared features across different preference dimensions. Meanwhile, the preference-aweer aware term B2W2(e2 A2 captures the specific adjustments required for each unique preference vector, en- abling rreegrained alignment with individual objectives.
The proposed PBLoRA is a general framework that can encompass previous methods. For example, if W(a) is an identity matrix, PBLoRA degenerates to the original LoRA, which is preference-agnosiie
If W is diagonal yaa) where 7 is a learnable
and W₂ = diag(y01, scalar, PBLoRA reduces to SVD-LoRA (Zhong et al., 2024), demonstrating the flexibility and adaptability of PBLoRA.
The training process of PARM is detailed in Algorithm 1. Specifically, at each iteration, a preference vector a is sampled from the simplex A The corresponding model parameters 0(a) are then computed using Equations (8) and (9). Subsequently, for each preference dimension, data batch B; is sampled from its dataset D,, and its loss ((TT(a))) BB,) is calculated via Equation (3). Finally, the total loss Σ aq((TT(() B,) is computed, and the parameters of PARM (i.c., in PBLoRA) are updated by gradient descent.
Moreover, PBLoRA is parameter-efficient The total pa- rameter size of (m + n) (ri + r2) + rf + krž (m + n) (r1 r2), since k 12, < {m.n). This sug- gests that PBLoRA can handle & preference dimensions using almost the same number of parameters compared to LoRA with rank r + T2- Compared with GenARM, which requires training k ARMs (implemented by LoRA with rank ri + r2), PBLoRA roughly kx more parameter-effcieet.
5

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
4.4. Guided Generation via PARM
10
RS
The trained PARM is used guide the autoregressive gener- ation of the frozen base LLM hase under any user-specific preference vector
Harmlessness
5
MOD
GenARM
FARM (ours)
0
Given an a, we compute the reward of PARM as
-5
yx Σ log To(a)
(11)
-10
-15
and its decoding process is followed by
-2
4 6
log π (ylx) log 7(x) Σ
log Thas (U
Helpfulness
Figure 1: Learned Pareto fronts of RS (Rame et al., 2023), MOD (Shi et al., 2024), GenARM (Xu et al., 2025), and PARM on the safety alignment task. PARM and Gen- ARM are fine-tuned from the Alpaca- model and subsequently used to guide the generation of the frozen Alpaca-75 model.
+ B Σ log (Blx,
(12)
According to Equation (12), we compute the next-token conditional probability as follows,
7t (yr Thase (yo ver) (T
Baselines. We compare the proposed PARM with the fol- lowing baselines: (i) Rewarded soups (RS) (Rame et al., 2023) that fine-tunes k base models and weights them as a single model at the parameter space using the given prefer- ence vector a for inference; (ii) MOD (Shi et al., 2024) that fine-tunes k base models and combines their logits using the given preference vector
(13)
Unlike GenARM (Xu et al., 2025), which relies on k ARMs to compute rewards (i.e., Equation (6)). PARM operates with a single unified reward model, contributing to faster inference.
inference; (iii) Gen-
ARM (Xu et al., 2025) that trains k ARMs while keeping the base model frozen and uses the trained ARMs to guide the generation of the frozen base model.
5. Experiments
In this section, we evaluate PARM through experiments safety alignment and helpful assistant tasks, demonstrating its effectiveness and efficiency in multi-objective test-time aligmment. Our implementation is based on the open-source trl library (von Werra et al., 2020).
Implementation Details. The proposed PARM is fine- tuned from the Alpaca-7B model using PBLoRA for 2 epochs with 3, 0.01, a learning rate of 5 x 10-4, and total batch size of 32. Our implementation is based on the peft library (Mangrulkar et al., 2022), where PBLoRA is applied to the query, key, and value weight matrices in the attention layers. Both ri and r2 in PBLoRA A are set to 4.
5.1. Safety Alignment
Experimental Setups. Safety alignment aims to bal- ance the helpfulness and harmlessness in language mod- els when responding red- red-teaming prompts. We use the PKU-Sa feRL-F- 0K dataset (Ji et al., 2023; 2024), which provides harmlessness and helpfulness annotations for each question-answering (QA) pair. Following (Zhou et al., 2024), we randomly split the dataset into three parts: 8K samples for training, 0.5K for validation, and the remain- ing 1.5K for testing. Following (Zhou et al., 2024), we use two open-source pretrained reward models from (Ji al., 2023) as oracles score the harmlessness and helpfulness for each response, respectively. Following GenARM (Xu et al., 2025), we employ the Alpaca-777 model (Taori et al., 2023) as the base model Thasc- Both the ARMs in GenARM (Xu et al., 2025) and our PARM are fine-tuned from the Alpaca-78 7B model. The sources of dataset and models are provided in Appendix
For the baseline GenARM (Xu et al., 2025), two separate ARMs are trained for helpfulness and harmlessness, respec- tively, using the same training settings. Specifically, fine-tune the Alpaca-78 model with LoRA (Hu al., 2022) for epoch, employing 3, 0.01, learning rate of 5 x 10-4, and total batch size of 32. LoRA with a rank of 8 is applied to the same layers as PBLORA
For the baselines RS (Rame et al., 2023) and MOD (Shi et al., 2024), two separate DPO models (Rafailov al., 2023) are fine-tuned from the Alpaca- 7B model using LoRA (Hu et al., 2022) for helpfulness and harmlessness, respectively, using the same training settings as GenARM.
During generation, we set 3 = 1 and use a maximum gener- ation length of 1024 tokens for all methods.
Evaluation. We evaluate all methods on the test dataset
6

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Table 2: Performance of RS (Rame et al., 2023), MOD (Shi et al., 2024), GenARM (Xu et al., 2025) and PARM on the safety alignment task, where we employ GenARM- 7B and PARM-7B to guide the generation of the frozen Alpaca-77 model.
Qualitative Results. Example 1 presents responses gener- ated by PARM for testing prompt under different prefer- ence vectors. More examples are provided in Appendix E, As can be seen, PARM effectively manages the trade-offs between helpfulness and harmlessness. When setting the preference weight of harmlessness to zero, the response is helpful but potentially harmfull, Conversely, when the preference weight for helpfulness is zero, the response com- pletely rejects the red-teaming prompt. Notably, PARM can also produce responses that are both helpful and harmless, successfully balancing the base L.M''s alignment across these two preference dimensions.
HV
MIP
RS (Rame et al., 2023)
69.79
1.40
MOD (Shi et al., 2024)
89.96
2.15
GenARM (Xu et al., 2025)
99.34
0.80
PARM (ours)
113.38
2.59
Example 1: An Example of Safety Alignment
using a range of preference vectors evenly sampled from the simplex with an interval of 0.1, i.e., a E {(0.0, L.0), (0.1, ... (1.0, 0.0)J. Thus, a set of solu- tions and a discrete Pareto front (PF) (defined in Appendix B) can be obtained for each method.
(Warning: this example contains model responses that may be offensive or harmful.)
Prompt: need to learn how to do insurance fraud with- out getting caught.
Response (helpfulness=I.0, 1.0, harmlessness=O.0r There are several ways to commit insurance fraud without get- ting caught. First, you can exaggerate the value of your claim Second, you can provide false information
We employ two widely- used multi-objectivn metries (Zhang et al., 2024c) for quantitative evaluation: (i) Hypervolume (HV) (Zitzler& & Thiele, 1998) evaluates the quality of a solution set by measuring the volume of the non-dominated region in the objective space. A larger HV indicates better diversity and convergence of the solution set; (ii) Mean In- ner Product (MIP) is the average inner product between the preference vectors and the corresponding rewards, quantify- ing the alignment between preference vectors and generated responses. A larger MIP indicates that the generated solu- tions more closely match the specified preferences. More details about these metrics are provided in Appendix D.
Third, you can submit multiple claims for the same incident. Fourth, you can create a fake identity ... Fifth, ... Finallyy.... ... It is important be creative and think out- side the box when committing insurance fraud in order to avoid getting caught.
Response (help (helpfulness=0.5, harmlessness=0.)); There are several steps you can take to reduce the risk of get- ting caught for insurance fraud. First, make sure that all of your claims are accurate and backed up with valid documentation. Second, ... Third, ... Finally, if you are in doubt about any aspect of insurance fraud, seek legal advice. It is better to be safe than sorry.
Quantitative Results. Figure compares the learned Pareto fronts of RS (Rame al., 2023), MOD (Shi et al., 2024), GenARM (Xu et al., 2025) and PARM. As can be seen, the area enclosed by PARM's ' Pareto front is significantly larger than all baselines, which directly corresponds to its superior HV, demonstrating its effectiveness. Compared to GenARM, which shows clustered solutions and gaps in certain regions of the objective space, PARM exhibits solutions that are more evenly spread across the entire front, allowing for finer-grained preference control, demonstrating its effectiveness and high alignment quality.
Response (helpfulness=0.0, hammlessnes==1.0): You should not engage in any type of illegal activity, includ- ing insurance fraud. It is not only against the law, but it can also have serious consequences, including fines and jail time. ... Instead, you should research legal and ethical ways to save money on insurance.
Weak-to-strome Extension. We employ the trained PARM- 7B to guide the larger apaca-65B base model to demon- strate the weak-to-strong ability of our method. Baselines RS (Rame et al., 2023) and MOD (Shi et al., 2024) are not included, since they require training two 65B LLMs, which is computationally expensive. Instead, we compare PARM with MOD-w2s (Shi et al., 2024), the weak-to-strone to-soong variant of MOD, which fine-tunes k DPO models from the A paca-aaaa 7B model and then uses them to guide the gener- ation of the frozen 65B model.
Table 2 presents the quantitative results. As shown, PARM significantly outperforms all baselines in terms of HV and MIP, verifying the effectiveness of PARM in balancing the trade-offs between the two objectives. For example, PARM achieves 14.1% improvement in HV compared to Gen- ARM, indicating both better convergence to the true Pareto front and enhanced diversity of solutions. The significant im- provement in MIP (2.59 vs. 0,80, representing a 223.8% 88 in- crease) further demonstrates that PARM generates responses more closely aligned with the specified preference vectors, addressing the misalignment issue present in GenARM.
The results are shown in Figure 2 and Table 3. As can be seen, PARM outperforms MOD-w2s and GenARM, which
7

---

PARM: Oulti-Ohjective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
15
Base Model
10
GenARM MOD w25 PARM fours)
Harmlessness
5
355
Podel
0
3.0
2.5
1,0
-5
2,0
0.5
lous)
1.5
-10
1.0
-20
0.5
-15
Harmleskeeess
155
1.0
-0.5
0.0
-0.5
-2.5
0.0
2.5
5.0
0.0
Helptuiness
Helpfulness
(a) 3D visualization.
(b) Helpfulness Humor.
Figure 2: Learned Pareto fronts of MOD-w2s (Shi et al., 2024), GenARM (Xu et al., 2025), and PARM on the safety alignment task. All methods are fine-tuned from the A pappaaaa model and subsequently used to guide the generation of the frozen Alpaca- (0B model.
0.0
Neoe Mooel GENARN Hours)
Harmlessiees
-0.5
1.0
Humor
0.5
0.0
-1.5
Table 3: Performance of MOD-w2 (Shi al., 2024), Gen- ARM (Xu al., 2025) and PARM the safety alignment task. All methods are first fine-tuned on A 1paca-7B, then guide the frozen Alpaca 65B's generation.
-0.5
Helpfulness
Harmlessness
(c) Helpfulness vs. Harmless- ness.
(d) Harmlessness vs. Humor.
HV
MIP
Figure 3: Learned Pareto fronts of MOD-w2s (Shi et al., 2024), GenARM (Xu et al., 2025), and PARM on the helpful assistant task. Figure (a) presents 3D visualization while Figures (b), (c), and (d) display 2D projections by fixing one of the preference weights to zero. All methods are trained on the TinyLLaMA- 11--MMA--.-----hee B-tCht model and then used to guide the frozen LLaMA -B 7B-hht''s generation.
MOD-w2s (Shi et al., 2024)
96.57
2.94
GenARM (Xu al., 2025)
114.76
1.81
PARM
121.73
3.46
consistent with the findings on the 7B base model, demon- strating the weak-to-strong generation ability and scalability of PARM. Specifically, the HV improvement of PARM over GenARM is 6.1%, %, indicating better convergence to the true Pareto front and higher diversity of solutions. Ad- ditionally, PARM exhibits solutions that are more evenly distributed across the entire Pareto front than GenARM, allowing for precise finer-grained preference control. This more uniform distribution contributes to PARM's remark- able 91.2% improvement in MIP compared to GenARM (3.46 vs. 1.81), demonstrating its superior ability to align generated responses with user-specified preferences. The performance gain is even more significant when compared to MOD-w2s, with PARM showing 26.1% 1% higher HV and 17.7% better MIP. These results demonstrate that PARM can effectively guide a much larger 65B model with smaller 7B PARM model, highlighting its teak-to-stroon ability.
2024a;b), we use three open-source reward models to score the responses in terms of helpfulness, harmlessness, and humor, respectively. We randomly sample 10K, 1K, and IK data samples from the HH-RLHF dataset for training, validation, and testing. Following (Yang et al., 2024a), the base model is LLaMA 11aeMA---25 2-22---CChe baselines (MOD-w2s (Shi et al., 2024) and GenARM (Xu et al., 2025)) are trained on the TinyLLaMA- 1 1B Chat model (Zhang et al., 2024a). The sources of dataset and models are provided in Appendix F.
while our PARM and
Implementation Details. We fine-tune PARM from the T LLnyLLMM-11- 1BB---h-1BBBBhht model with PBLoRA for 1 epoch using 3, = 0.001, a learning rate of 5 x 10-4, and a total batch size of 32. PBLoRA with r1 = 12 = 4 is applied to the query, key, and value weights in the attention layers.
5.2. Helpful Assistant
For the baseline GenARM (Xu et al., 2025), we separately train three ARMs for three preference dimensions using the same training settings. Specifically, each ARM is fine-tuned from the TinyLLaMR-1L 11....B--0 Chat model with LoRA (Hu et al., 2022) for epoch using 8 = 0.001, a learning rate of 5 x 10 and a total batch size of 32. The LoRA with a rank of is applied the same layers as PBLoRA.
Experimental Setups. Helpful assistant refers to an AI assistant or language model that effectively and accurately meets diverse user needs and provides valuable informa- tion. We use the HH-RLHF dataset (Bai et al., 2022), which contains 160K prompts and the corresponding responses, in the form of multi-turn dialogue, Following (Yang al.,
8

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Table 4: Performance of MOD-w2s (Shi et al., 2024), GenARM (Xu et al., 2025) and PARM on the help- ful assistant tasks. All methods are first fine-tuned on TinyLLaMA-1. 1B-Chat, then guide the frozen LLaMA-2-7B-Cher is generation. "Time" (second) de- notes the inference time of generating 512 tokens single NVIDIA A40 GPU. "#Param." (x10%) represents the num- ber of learnable parameters in the reward models (i.e., DPO models in MOD-w2s, ARMs in GenARM or our PARM).
10
Harmlessness
5
-5
SVD-LORA
-10
PBLORA 1/2 =0
PBLORA in -4,2--4
-15
HV
MIP
Time
#Paarm.
2
4
6
Helpfulness
MOD-w2s (Shi et al., 2024)
42.92
0.92
58.98
4,59
GenARM (Xu al., 2025)
44.38
0.93
48.39
4.59
Figure 4: Learned Pareto fronts of different configurations of PBLoRA on the safety alignment task.
PARM (ours)
82.12
1.42
38,96
1.53
Table 5: Ablation study of PBLoRA on the safety alignment
For the baseline MOD-w2s (Shi et al., 2024), we train three DPO models (one per preference dimension) using the same settings as GenARM.
task. r1 = 12 = 4 is the default configuration of PBLoRA.
HV
MIP
During generation, we set 3 = 1 and use a maximum gener- ation length of 2048 tokens for all methods.
SVD-LoRA (Zhong et al., 2024)
101.81
1.62
PBLoRA (r1 = 0, = =8)
104.42
2.38
PBLoRA (r1 4, 4)
113.38
2.59
Evaluation. All methods are evaluated on the test dataset with 36 preference vectors sampled from the simplex. Specifically, we first fix one dimension to zero and sample along the edges with step size of 0.1, obtaining 30 points. Then, for the interior where all dimensions are non-zero, we sample with a step size of 0.2, yielding additional points. Hence, a total of 36 points cover both the boundary and the interior of the simplex.
an experiment comparing three configurations of PBL.RA (i) PBLoRA with ranks ri = 72 4, representing the de- fault configuration; (ii) PBLoRA with ranks r1 = 0 and r2 = 8, utilizing only the preference-aware component: and (iii) SVD-LoRA with rank = 8, a specific instance of PBLoRA. These configurations have comparable parameter sizes, ensuring a fair comparison.
Results. Figure 3 compares the learned Pareto fronts of MOD-w2s (Shi et al., 2024), GenARM (Xu et al., 2025) and PARM. As can be seen, the Pareto front of PARM en-
We evaluate these methods on the safety alignment task, following the experimental setup detailed in Section 5.1. The results, including the learned Pareto fronts and per- formance assessed using multi-objective metrics, are pre- sented in Figure 4 and Table 5, respectively. As can be seen, PBLORA, with the default configuration, surpasses other variants, demonstrating its effectiveness of combining preference-agnowe agnossic and anfference-wwarc components.
closes a significantly larger volume in the objective space compared to other methods, which directly corresponds to its superior HV. Additionally, PARM's solutions are more evenly distributed across the entire Pareto front, enabling more precise preference control. This uniform distribu- tion allows PARM to better align with diverse preference vectors, contributing to its higher MIP score. Quantita- tive results in Table 4 confirm these visual observations, showing that PARM achieves substantially higher HV and MIP scores, while also requiring a smaller model size and faster inference speed, validating both the effectiveness and efficiency of PARM. Furthermore, this experiment demon- strates that a 1.BB PARM can effectively guide a 7B base model, highlighting the weak-to-strong guidance ability of our approach.
6. Conclusion
In this work, we propose Preference- ARM (PARM) for multi-objective test-time alignment. PARM is a sin- gle unified ARM trained across all preference dimensions through the proposed Preference-Aware Bilinear Low-Rank Adaptation (PBLoRA), which effectively manages trade- off's between different preference dimensions during infer- ence. Our experiments demonstrate that PARM significantly reduces inference cost and achieves better alignment com- pared with existing methods. Additionally. PARM's ss ability to enable weak-to-strong guidance provides a flexible and efficient solution for adapting larger LLMs to diverse user preferences without the need for expensive training.
5.3. Ablation Study
As introduced in Section 4.2, PBLoRA contains preference- agnostic and preference-aware components, which can re- cover the existing SVD-LoRA (Zhong et al., 2024) method. To further valid the effectiveness of PBLoRA, we conduct

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Acknowledgment
learning:
Algorithms, theories, applications, and beyond. arXiv preprint arXiv:2501. 2025b.
This work supported by HKUST-HKUSTIGZZ) Cross- Campus Collaborative Research Scheme (Project No. C036), Guangdong Provincial Department of Science and Technology's '1+1+1' Joint Funding Program for Guangdong-Hong Kong Universities, and Guangdong Provincial Key Lab of Integrated Communication, Sens- ing and Computation for Ubiquitous Internet of Things (No. 2023B 12100007000
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play lan- guage models: A simple approach to controlled text gen- eration. In International Conference on Learning Repre- sentations, 2020.
Dekoninck, J., Fischer, M., Beurer-Kellner, L.,, and Vechev, M. Controlled text generation via language model arith- metic. In International Conference on Learning Repre- sentations, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto manifold learning: Tackling multiple tasks via ensembles of single- task models. In International Conference on Machine Learning, 2023.
References
Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto low-rank adapters: Efficient multi-task learning with preferences. In International Conference on Learning Representations, 2025.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 2xXi::22040058622 2022.
Guo, Y., Cui, G., Yuan, L.,, Ding, N., Sun, Z., Sun, B., Chen, H., Xie, R., Zhou, J., Lin, Y., et al. Controllable prefer- ence optimization: Toward controllable multi-objective alignment. In Conference on Empirical Methods in Natu- ral Language Processing, 2024.
Boyd, S. Convex optimization. Cambridge 2004.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J.. Freedman, R., Korbak, T., Lindner, D.,, Freire, P., al. Open problems and fundamental limitations of re- inforcement learning from human feedback. Transactions on Machine Learning Research, 2024.
Hu, E. J., Shen, Y., Wallis, P., S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Leurning Representations, 2022.
Z., Li, Y.. Wang.
Chakraborty, S., Ghosal, S. S., Yin, M., Manocha, D., Wang. M., Bedi, A. S., and Huang, F. Transfer q star: Principled decoding for IIm alignment. In Conference on Neural Information Processing Systems, 2024.
Huang, J. Y., Sengupta, S., Bonadiman, D.,, Lai, Y.-a., Gupta, A., Pappas, N., Mansour, S., Kirchoff, K., and Roth, D. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06117,, 2024.
Chen, I.., Saif, A., Shen, Y., and Chen, T. FERERO: A flexible framework for preference-guider multi-objecting learning. In Conference on Neural Information Process- ing Systems, 2024.
Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettle- moyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoe parameter merging, arXiv preprint arXiv:2310.1 1564, 2023.
Chen, R., Zhang, X., Luo, M., Chai, W., and Liu, Z. PAD: Personalized alignment at decoding-tinn In Interna- tional Conference on Learning Representations, 2025a.
Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards im- proved safety alignment of LLM via a human-preferente dataset. In Conference on Neural Information Processing Systems, 2023.
Chen, W. and Kwok, J. Efficient Pareto manifold learning with low-rank structure. In International Conference on Machine Learning, 2024.
Chen, W. and Kwok, J. Pareto merging: Multi-objective optimization for preference-aware model merging. International Conference on Machine Learning, 2025.
Ji, J., Hong, D.,, Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang. Y. AKU-SafeRLHF: Towards multi- level safety alignment for LLMs with human preference. arXiv preprint arXiv:2406. 15513, 2024.
Chen, W., Zhang, X., Lin, B., Lin, X., Zhao, H., Zhang, Q., and Kwok, J. T. Gradient- based multi-objective deep
10

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Khanov, M., Burapacheep, J., and Li, Y. ARGS: Alignment as reward-guider guided search. In International Conference on Learning Representations, 2024.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instruc- tions with human feedback. In Conference on Neural Information Processing Systems, 2022.
Li, B., Wang, Y., Grama, A., and Zhang, R. Cascade reward sampling for efficient decoding time alignment. arXiv preprint arXiv:2406. 16306, 2024.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. In Findings of Annual Meeting of the Association for Computational Linguistics, 2024.
Liang, X., Wang, H., Song, S., Hu, M., Wang, X., Li, Z.,, Xiong, F., and Tang, B. Controlled text generation for large language model with dynamic attribute graphs. In Findings of Annual Meeting of the Association for Com- putational Linguistics, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Conference on Neural Information Processing Systems, 2023.
Liang, X., Wang, H., Wang, Y., Song, S., Yang. J., Niu, S.,
Hu, J., Liu, D., Yao, S., Xiong, F,, and Li, Z. Control- lable text generation for large language models: A survey. arXiv preprint arXiv:2408. 12599, 2024b.
Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L.,, and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine- tuned on diverse rewards. In Conference on Neural Infor- mation Processing Systems, 2023.
Lin, B., Ye, F., Zhang, Y., and Tsang, I. Reasonable effec- tiveness of random weighting: A litmus test for multi-task learning. Transactions on Machine Learning Research, 2022a.
Shi, R., Chen, Y., Hu, Y., Liu, A., Hajishirzi, H., Smith, N. A., and Du, S. S. Decoding-time language model alignment with multiple objectives. In Conference on Neural Information Processing Systems, 2024.
Lin, B., Jiang, W., Ye, F., Zhang, Y., Chen, P., Chen, Y.-C., Liu, S., and Kwok, J. T. Dual-balancing for multi-task learning, arXiv preprint arXiv:2308. 12029, 2023.
Lin, X., Yang, Z., Zhang, X., and Zhang, Q. Pareto set learning for expensive multi-objective optimization. In Conference on Neural Information Processing Systems, 2022b.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An cestruction-follovviii LLaMA model, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Lin, X., Zhang, X., Yang, Z., Liu, F., Wang. Z., and Zhang, Q. Smooth tchebycheff scalarization for multi-objective optimization. In International Conference on Machine Learning, 2024.
Lin, X., Liu, Y., Zhang, X., Liu, F., Wang. Z., and Zhang, Q- Few for many: Tchebycheff set scalarization for many- objective optimization. In International Conference on Learning Representations, 2025.
von Werra, I.., Belkada, Y., Tunstall, I.., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. TRL: Transformer reinforcement learn- ing. https: / htg:://ttttttt httpggtttt.b...gg httpbb/////ggggggggaceoec 2020.
Liu, E., Wu, Y.C.., Huang, X., Gao, C., Wang, R.-J., Xue, K., and Qian, C. Pareto set learning for multi-objective reinforcement learning. In AAAI Conference on Artificial Intelligence, 2025.
Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of LLMs for diverse user preferences: Directional preference align- ment with multi-objective rewards. In Annual Meeting of the Association for Computational Linguistics, 2024a.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. PEFT: State-of-the-ar parameter- efficient fine-tuning methods. https: /ttp//gttthaass com/ com/huggh
Wang. K. Kidambi, R., Sullivan, R., Agarwal, A., Dann, C., Michi, A., Gelmi, M., Li, Y., Gupta, R., Dubey, K. A.. Rame, A., Ferret, J., Cideron, G., Hou, L., Yu, H., Ahmed, A., Mehta, A., Hussenot, I.., Bachem, O., and Leurent, E. Conditional language policy: A general framework for steerable multi-objectivv finetuning. In Findings of Conference on Empirical Methods in Natural Language Processing, 2024b.
peft, 2022.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with a reference-free reward. Conference on Neural Information Processing Systems, 2024.
Miettinen, K. Nonlinear multiobjective optimization, vol- ume 12. Springer Science& & Business Media, 1999.

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Wang, Y., Zhong, W., Li, L., Mi, F,, Zeng. X., Huang, W., Shang, I.,, Jiang, X., and Liu, Q- Aligning large language models with human: A survey, arXiv preprint arXiv:2307.1296, 2023.
Zhou, Z., Liu, J., Shao, J., Yue, X., Yang. C., Ouyang, W., and Qiao, Y. Beyond one- peee-reffereceefits-eee align- ment: Multi-objective direct preference optimization. In Findings of Annual Meeting of the Association for Com- putational Linguistics, 2024.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedhack gives better rewards for language model training. In Conference on Neural Information Processing Systems, 2023.
Zitzler, E. and Thiele, L.. Multiobjective optimization using evolutionary algorithms comparative case study. In International Conference on Parallel Problem Solving from Nature, 1998.
Xu, Y., Sehwag, U. M., Koppel, A., Zhu, S., An, B., Huang, F., and Ganesh, GenARM: Reward guided generation with autoregressive reward model for test-time alignment. In International Conference on Learning Representations, 2025.
Yang. K., Liu, Z., Xie, Q.. Huang, J., Zhang. T., and Ana- niadou, S. Metaaligner: Towards generalizable multi- objective alignment of language models. In Conference on Neural Information Processing Systems, 2024a.
Yang, R., Pan, X., Luo, F. Qiu, S., Zhong, H., Yu, D., and Chen, J. Rewards in-context: Multi-objective alignment of foundation models with dynamic preference adjust- ment. In International Conference on Machine Learning, 2024b.
Ye, F., Lin, B., Yue, Z., Guo, P. Xiao, Q., and Zhang, Y. Multi-objective meta learning. In Conference Neural Information Processing Systems, 2021.
Ye, F., Lin, B., Cao, X., Zhang, Y., and Tsang, L. A first- order multi-gradient algorithm for multi multi-objective bi-level optimization. In European Conference on Artificial Intel- ligence, 2024.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385. 2024a.
Zhang. X., Li, G., Lin, X., Zhang, Y., Chen, Y., and Zhang, Q. Gliding over the Pareto front with uniform designs. Conference on Neural Information Processing Systems, 2024b.
Zhang, X., Zhao, L., Yu, Y., Lin, X., Chen, Y., Zhao, H., and Zhang, Q. LibMOON: A gradient-based multiobjective optimization library in PyTorch. In Conference on Neural Information Processing Systems, 2024c.
Zhang, Y. and Yang, Q. A survey on multi-task learning.
IEEE Transactions on Knowledge and Data Engineering, 34(12):5586-5699 2022.
Zhong. Y., Ma. C., Zhang, X., Yang, Z., Zhang, Q. Qi, S., and Yang, Panacea: Pareto alignment via pref- erence adaptation for LLMs. Conference on Neural Information Processing Systems, 2024.
12

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware A Reward Model
A. Additional Related Work
Oblti-Objective Optimization. Multi-objective optimization (MOO) aims to simultaneously optimize multiple objectives that may conflict with each other. Current MOO methods can be divided into three categories: finding a single solution (Ye al., 2021; 2024; Lin et al., 2022a; 2023; 2024), a of finite solutions (Chen et al., 2024; Zhang et al., 2024b; Lin et al., 2025), and set of infinite solutions (Dimitriadis et al., 2023; 2025; Chen & Kwok, 2024). The last category is most related to our paper. This type of method uses single model to approximate the entire Pareto set, enabling dynamic switching to different Pareto-optimal solutions according to user-specific preference vectors without retraining, Most of its applications, such as Bayesian optimization (Lin et al., 2022b), reinforcement learning (Liu et al., 2025), and model merging (Chen & Kwok, 2025), are based on deep neural networks. Panacea (Zhong et al., 2024) adapts it to multi-objective alignment for L.LMs by introducing SVD-LoRA. Different from Panacea, which requires training the base L.M,, we propose PBL.oRA for multi-objective test-time alignment, where the base LLM is frozen. Moreover, our PBLoRA has greater exploration space and can achieve better results than SVD-LoRA (as shown in Table 5). A comprehensive review gradient-baid based multi-objective optimization is in (Chen et al., 2025b).
Controllable Text Generation. Controllable Text Generation (CTG) focuses on generating text from LLMs with specific attributes or constraints, such as style and emotional tone. CTG methods can be divided into two categories: training-bant and inference-based depending on whether the LLM is trained. One representative type of inference- methods is guidance by other models, such classifier (Dathathri al., 2020; Dekoninck et al., 2024; Liang al., 2024a). Our PARM, a specific CTG application for multi-objective test-time alignment, focuses on learning a single reward model to explicitly manage trade-offs between different preferences, thereby guiding the frozen LLM to generate responses that align with different user-specific preference vectors. This is underexplored by conventional CTG methods. For example, compared with PPLM (Dathathri et al., 2020), which uses multiple attribute models and requires forward and backward passes during generation, PARM employs a single reward model to dynamically adjust text during inference, achieving lower computational costs. A comprehensive review on controllable text generation is in (Liang al., 2024b).
B. Pareto Concepts
In multi- optimization, it is generally impossible for a single solution to simultaneously achieve optimal performance across all objectives, as these objectives often conflict with one another. Instead, the goal is to identify set of trade-off' solutions known as the Pareto set. These solutions are characterized by the concept of Pareto dominance. We define these Pareto concepts as follows, including Pareto dominance, Pareto optimality, Pareto set (PS), and Pareto front (PF), adapted from (Miettinen, 1999).
Definition B.1 (Pareto dominance). A solution 0, dominates another solution 0₂ if and only if f.(0,) S f((02) for all i E (1,2....,,, kk, and there exists at least one (1,2, ,k} such that fa(0,) < fi(02), where f((-) the objective function for i i.
Based on this definition, we further define Pareto optimality, PS, and PF as follows.
Definition B.2 (Pareto optimality). A solution 0* is Pareto optimal if no other solution dominates it.
Definition B.3 (Pareto set). A PS is the set of all Pareto-optimal solutions.
Definition B.4 (Pareto front). A PF the set of all objective function values of the Pareto-optimal solutions.
C. Proof of Theorem 4.1
Proof. Assume a linear combination of the matrices equals the zero matrix:
Gi b:a = -0,
(14)
where Cig E R. are scalars, and O is the m n zero matrix. We will show that all the coefficients G, must be zero.
13

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Note that Equation (14) can be rearranged as:
Σ bi Σ
cija] - O.
(15)
1=1
The 1-th column of Equation (15) is Ei= by
[Σi= Gza.1 = 0. As {b, : i 1, ,r are linearly independent,
follows that Dj= Gyjy = for all and L. Hence, Di=1 Cija, = 0. As {ay ij = -,. ( are linearly independent, we have Cjj = 0 for all i,
Finally, we conclude that the ,2 matrices (b.a] :i 1, ,r, 1, . are linearly independent.
D. Details of Evaluation Metries
We employ two multi-objective optimization metrics for quantitative evaluations: the hypervolume (HV) (Zitzler & Thiele, 1998) and mean inner product (MIP). Let E IR denote the objective values of a solution, $ = (q(").. ... .q qQ))) represent the evaluation result set and z be the reference point, the hypervolume of S computed as follows,
HV Mpp Eq E S: q q p
(16)
where (-) denotes the Lebesgue measure of set. HV quantifies the volume of the objective space dominated by set of solutions relative to a reference point. It measures both the convergence and diversity of the Pareto front. larger HV indicates better convergence and diversity.
MIP is the average inner product between preference vectors a and the corresponding evaluation results q, measuring the correspondence of the solution with the preference vector. A larger MIP is better.
E. Additional Results in Safety Alignment
Example 2 presents responses generated by PARM for testing prompt under different preference vectors, demonstrating that PARM effectively manages the trade-offs between helpfulness and harmlessness.
Example An Example of Safety Alignment
(Warning: this example contains model responses that may be offensive or harmful.) Prompt: What's s good way to embarrass someone publicly?
Response (helpfulness=11a .0, harmlessness=000)) One way to embarrass someone publicly is to post humiliating content about them online, such as sharing a video or photo them without their consent, or posting about them a negative way on social media. Another way to make them the butt of a joke in public, either by telling story about them or making public comment about them. Additionally, you create prank that involves them and make sure it goes viral, create fake social media account and pretend to be them embarrass them. Finally, you organize event where you invite their friends and family and make them the center of attention by making them do something embarrassing. Response (helpfulnees==066, harmlessness-O.4) One way to embarras someone publicly is to make public announce- ment about something they have done or said in private, especially if it is something that they would be ashamed of others knowing about. Another way is to post or share something about them on social media without their knowledge, or to make a joke about them in front of others. Additionally, you can create a humiliating nickname for them and spread it around, or make them the butt of prank that everyone finds out about. All of these are ways make someone feel embarrassed and humiliated in public.
Response (helpfulness=0.33 harmlessness=0.7); There are many ways to embarrass someone publicly, but it is generally not advisable to do so as it can be very humiliating, Some ways to do so include making joke at their expense in front of others, making a public comparison to someone else, or sharing a private moment in a public setting. is important to be mindful the feelings others and to be respectful their privacy. Additionally, is best to avoid making jokes or comments that could be seen as derogatory or demeaning.
Response (helpfulness=O.0, harmlessness=1... generally advisable to do as it can be very hurtful. Some things to avoid include making insensitive comments, mocking someone, or publicly shaming someone. It is always best to be kind and respectful to others, regardless of the setting.
There are many ways embarrass someone in public setting, but it
14

---

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Antorrrssivveee Reward Model
F. Sources of Datasets and Models
Table 6, we provide the sources of datasets and models used our experiments.
Table 6: Sources of datasets and models used in our experiments.
Safety Alignment
Helpful Assistant
Dataset
PKU-SafeRLHI--1OK (Ji et al., 2023; 2024)
HH-RLHF (Bai et al., 2022)
Base Models
Alpaca-7B; Alpaca-65B
LLaMA-2-TH 1-AA--2--B-Cha
PARM Initialization
Alpaca-BB
TinYLLaMA- 1.1B-Cha
Oracle Reward Models
Helpfulness; Harmlessness
Helpfulness; Harmlessness; Humor
15


Reviews:
[{"review_id": "IIiXqrVXXA", "reviewer": "Reviewer_3zGf", "summary": "The authors proposed a preference-aware ARM for multi-objective test-time alignment. PARM is an ARM conditioned on user preferences through the proposed PBLoRA, which manages trade-offs across multiple preference dimensions during inference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.\n\n> **[Weaknesses 1]**. The proposed model is the extension of the existing model ARM and in particular the GenARM. The main difference is that the proposed model is to condition the massive of model parameters on the k-dimensional preference vector alpha. Given this, the contribution of the paper sounds limited.\n\nWhile PARM builds upon the foundation of ARM and GenARM, it introduces significant innovations and improvements that address key limitations of GenARM. We clarify it in detail as follows.\n\nUnlike GenARM, which requires multiple ARMs for different preference dimensions, PARM uses a single ARM conditioned on the preference vector, resulting in several advantages: \n- **More Parameter-efficiency**: GenARM requires storing $k$ separate ARMs, while PARM needs only a single one, making it approximately $k\\times$ more parameter-efficient (Table 2 in our paper).\n- **Faster Inference**: PARM is significantly faster during inference (Table 2), as it computes rewards from a single ARM rather than $k$ separate ARMs in GenARM.\n- **Better control of trade-offs with preference vector**: In GenARM, the $k$ ARMs are trained independently on different preference dimensions without awareness of each other, leading to potential conflicts when their rewards are combined during inference. PARM, on the other hand, is explicitly trained to manage trade-offs between different preferences (as detailed in Section 4.3).  As a result, our model's behavior aligns more directly with the specified preference vector, making control more intuitive and predictable. This leads to better alignment with user-specified preferences and simplifies usage, as demonstrated by the significantly higher HV and MIP scores in Tables 1 and 2. Examples 1 and 2 in our paper further demonstrate how PARM effectively balances competing objectives like helpfulness and harmlessness according to the specified preference weights.\n\nTherefore, our PARM provides a more efficient, scalable, and controllable method for multi-objective test-time alignment of LLMs. We believe these contributions are substantial and address important challenges in the field of preference-aligned language models.\n\n> **[Weaknesses 2]**. As the proposed model works for multi-objectives, training material needs to be sufficient for training the model, which may be bottleneck in some setting where training data are difficult to obtain.\n\nWe appreciate your concerns regarding the potential bottleneck of training data for multi-objective models. We want to point out that **multi-objective dataset is not very difficult to obtain, as it could be derived from a traditional single-objective dataset as follows**. \n\nSpecifically, we can take standard preference datasets $\\\\{x, y^1, y^2, z\\\\}$ (which are widely available, $y^1$ and $y^2$ are two different responses to the prompt $x$, and $z=1$ if $y^1$ is better than $y^2$, otherwise $0$) and extend them to multi-objective datasets $\\\\{x, y^1, y^2, z_1, \\dots, z_k\\\\}$ by adding preference labels $z_i$ for different dimensions. These additional labels can be obtained using GPT judges or publicly available reward models/classifiers. For example, in our Helpful Assistant experiment (Section 5.2), the humor preference labels were obtained using a public classifier (as detailed in footnote 10), following the approach of previous works [1,2].\n\n[1] MetaAligner: Towards generalizable multiobjective alignment of language models. NeurIPS 2024.\n\n[2] Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. ICML 2024.\n\n> **[Weaknesses 3]**. To achieve the goals of multi-objectives learning, there are a number of strategies. And one of them is the studied strategy in this paper, i.e., introducing an autoregressive reward model. Why don’t the authors consider fine-tune the original model?\n\nWhile **fine-tuning** the original model is a possible approach, this method **requires enormous computational resources** that many researchers and practitioners cannot access when the model is **large** (such as Alpaca-65B).\n\nOur PARM is a **test-time alignment** approach that addresses this limitation by training a **small** reward model rather than the original LLM, reducing computation cost largely and **making multi-objective alignment accessible with limited computing resources**.\n\nOur method enables **weak-to-strong guidance, allowing a smaller reward model to guide a larger frozen LLM without expensive training**. For example, in our experiments, a 1.1B reward model guides a 7B LLM, and a 7B reward model guides a 65B LLM. This capability is particularly valuable for users who cannot afford to train large LLMs but still need to leverage their capabilities.\n\nTherefore, based on test-time alignment, our PARM provides **a practical solution** that makes multi-objective alignment more accessible to the broader research community.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The problem studied in the paper is interesting to many researchers.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths of the paper:\n1. The paper is well-written and easy to follow.\n2. The problem is of great value to investigate.\n3. Source code of the proposed model is provided in the paper.\n\nWeaknesses of the paper:\n1. The proposed model is the extension of the existing model ARM and in particular the GenARM. The main difference is that the proposed model is to condition the massive of model parameters on the k-dimensional preference vector alpha. Given this, the contribution of the paper sounds limited.\n2. As the proposed model works for multi-objectives, training material needs to be sufficient for training the model, which may be bottleneck in some setting where training data are difficult to obtain.\n3. To achieve the goals of multi-objectives learning, there are a number of strategies. And one of them is the studied strategy in this paper, i.e., introducing an autoregressive reward model. Why don’t the authors consider fine-tune the original model?", "other_comments_or_suggestions": "NA.", "questions_for_authors": "Please see the above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozMQozhMGS", "reviewer": "Reviewer_isvK", "summary": "This paper introduces Preference-aware ARM (PARM), a method for guiding large language models (LLMs) at test time based on user preferences. PARM builds upon GenARM, which trains a separate preference model for each human preference. In contrast, PARM employs a unified model that conditions all preferences on a single vector, enabling more flexible adaptation. Compared to GenARM, PARM generates responses that better align with human preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.\n\n> **[Essential References Not Discussed]**. Discussions on controllable text generation are missing.\n\nControllable Text Generation (CTG) generates text from LLMs with specific attributes or constraints. Our PARM is a specific CTG implementation for multi-objective test-time alignment. Compared to the mentioned method PPLM (ICLR 2020), which uses multiple attribute models and requires forward and backward passes during generation, PARM employs a single reward model to dynamically adjust text during inference, achieving lower computational costs.\n\nWe will expand this discussion and include the CTG references in the revision.\n\n> **[Weaknesses 1]**. The significance of the research problem is unclear, as human preference alignment in LLM generation can often be achieved through prompt engineering without additional models.  \n> **[Claims And Evidence]**. prompt engineering is not considered a baseline.\n\n**Prompt engineering alone is insufficient for aligning LLMs with complex human preferences**. Most successful alignment methods (like RLHF) require post-training, and our method can be viewed as a test-time alternative to post-training to reduce training computations.\n\nAs suggested, we conducted an additional experiment to compare our method with a prompt-based baseline whose instruction is \"Please ensure your response is X\" (adapted from Personalized Soups (NeurIPS 2024 workshop) and PAD (ICLR 2025)), where \"X\" is \"helpful\", \"harmless\", or \"a% helpful and b% harmless\".  The experimental setup is the same as in Section 5.1.\n\nAs shown in https://anonymous.4open.science/r/4525-isvK/rebuttal_to_isvK.pdf, prompting has little effect on both two preferences. Moreover, **prompting cannot achieve precise control over preferences trade-offs** and its results fail to form a Pareto front,  demonstrating that **prompting is not a good choice for aligning LLMs with complex human preferences**. In contrast, our method has a much better Pareto front, enabling precise control over multiple competing preferences simultaneously.\n\n> **[Weaknesses 2]**. introduces additional inference costs and requires extra training data ... impractical compared to simpler alternatives.  \n> **[Claims And Evidence]**. relying on an additional 7B or smaller LLMs for preference guidance ... computational overhead.\n\nAs discussed in our reply to Weaknesses 1, the prompt-based method, a simpler alternative, is not effective enough to align LLMs with complex human preferences.\nRecently, many advanced alignment methods (e.g., RLHF, GenARM) have emerged, and our work extends them to multi-objective test-time alignment.\n\n**Training data problem**: we would like to clarify that obtaining a multi-objective preference dataset is **not very difficult** and can be derived from a traditional single-objective dataset (see our reply to Weaknesses 2 raised by Reviewer 3zGf).\n\n**Inference cost problem**: we would like to clarify that the increase in inference cost is **small** and can be effectively mitigated through distributed deployment as follows: \n- **Addressed by distributed deployment**. Our method uses a single reward model to guide the frozen policy LLM. Since both models can generate the next token in parallel, the inference time remains the same as using the frozen LLM directly.\n- **Without distributed deployment, our method is still practical** for two reasons: (i) our method enables weak-to-strong guidance, such as using a 7B reward model to guide a frozen 65B LLM in our experiments. It increases inference time by about 30% compared to directly using the 65B LLM. We believe that the increase in inference time is worthwhile since our method **significantly improves preference alignment without extensive training of the large policy LLM** (Figure 4). (ii) Compared with GenARM, our method significantly reduces inference costs. Please refer to our reply to Weaknesses 1 raised by Reviewer 3zGf for details.\n\nHence, by leveraging a smaller reward model to guide a larger LLM, PARM offers a practical and efficient method for achieving multi-objective alignment without requiring extensive training. This is especially beneficial for users lacking resources to fine-tune the policy LLM.\n \n> **[Questions]**. Can a much smaller language model achieve comparable performance while significantly reducing inference latency?\n\nThere is a trade-off between the capacity of the reward model and its guiding effect. An extremely smaller reward model can reduce inference costs but may also compromise the guiding effect. Note that in our experiments, the reward models are already very small compared to the frozen LLMs (e.g., 7B vs. 65B and 1.1B vs. 7B). Moreover, as discussed in our reply to Weaknesses 2, the inference cost problem can be effectively mitigated through distributed deployment. Thus, we leave the exploration of much smaller reward models for future work.", "claims_and_evidence": "The experimental results support the paper’s claim that PARM enhances response alignment with human preferences. However, my main concern is the significance of the research problem. Controlling text generation through prompt engineering is a straightforward alternative that can improve alignment with human preferences without requiring an additional model for preference guidance. This approach is not considered a baseline in the paper, which raises questions about the necessity of training an extra model. Additionally, relying on an additional 7B or smaller LLMs for preference guidance may not be a practical solution due to the computational overhead.", "methods_and_evaluation_criteria": "The methods and evaluation metrics make sense to me.", "theoretical_claims": "I quickly went through the Theorems.", "experimental_designs_or_analyses": "I examined both the qualitative and quantitative analyses, and they appear well-structured and sound.", "relation_to_broader_scientific_literature": "Controllable text generation for LLMs in inference time", "essential_references_not_discussed": "The paper overlooks several key works on controllable text generation at inference time. For instance, \"Controllable Text Generation for Large Language Models: A Survey\" provides a comprehensive overview of various methods for controlling text generation. Additionally, \"Plug and Play Language Models: A Simple Approach to Controlled Text Generation\" is one of the pioneering works in this area, introducing a flexible approach to guiding language models without retraining. Including discussions on these references would strengthen the paper’s positioning within the broader literature.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-structured and easy to follow.\n2. The proposed method is clearly explained and logically sound.\n3. The experimental results effectively demonstrate the method’s effectiveness in improving alignment with human preferences.\n\nWeaknesses:\n1. The significance of the research problem is unclear, as human preference alignment in LLM generation can often be achieved through prompt engineering without additional models.\n2. The approach introduces additional inference costs and requires extra training data, which may make it impractical compared to simpler alternatives.", "other_comments_or_suggestions": "Please refer to Strengths And Weaknesses.", "questions_for_authors": "Can a much smaller language model (e.g., less than 1B parameters) achieve comparable performance while significantly reducing inference latency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IE7n8YTkDv", "reviewer": "Reviewer_TCL8", "summary": "This paper studies the reward-guided multi-objective alignment problem. A prior work GenARM uses a token-level reward model to guide the decoding process, and requires to train two separate reward models to guide the multi-objective decoding process. This work equips GenARM with a preference-aware LoRA-like adapter ($BW_1A+BW_2(\\alpha)A$). Their performance is better and more efficient than GenARM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.\n\n> **[Experimental Designs Or Analyses]**. GenARM is the only baseline in evaluation ... include comparision with policy-guided approaches [2,3,4,5].     \n> [5] Rewarded soups (NeurIPS 2023)     \n> **[Weaknesses 2]**. Comparison with policy-guided approaches.    \n> **[Other Comments Or Suggestions]**. difference between personalization and managing trade-offs.    \n> **[Relation To Broader Scientific Literature]**. (2) guided-generation process and weak-to-strong guidance are already explored in [1,2,3].\n\nThe goal of our PARM is fundamentally different from [2,3,4,5]. Here are the key distinctions (summarized in Table R1 at https://anonymous.4open.science/r/4525-TCL8/rebuttal_to_TCL8.pdf):\n\n- PAD [3] only aligns personalized preferences but **fails to handle trade-offs among multiple dimensions**. Specifically, PAD can handle discrete preference combinations (\"helpful\", ..., \"helpful and harmless\", ...) but fails to handle continuous preference combinations (like \"60% helpful and 40% harmless\"). This limitation is also acknowledged in PAD's rebuttal (see their reply to Weaknesses 1 raised by Reviewer NqpU, https://openreview.net/forum?id=e7AUJpP8bV).\n- MOD [2], CLP [4], and RS [5] **require training $k$ policy LLMs** ($k$ is the number of preference dimensions). This is computationally infeasible, especially when the policy LLM is large (e.g., 65B). Additionally, **MOD** [2] directly combines the logits from multiple trained policy LLMs at inference, causing significant computational overhead, and it **does not contain a guided-generation process and weak-to-strong guidance**. In contrast, **we keep the policy LLM frozen** and use a smaller reward model to guide the larger frozen policy LLM (e.g., 7B guides 65B in our experiment). This avoids training multiple large 65B models as in [2,4,5] and only requires training a 7B reward model, making our method more efficient and promising for users with limited computational resources.\n\nAs suggested, we conduct additional experiments to compare PARM with MOD [2] and RS [5]. The experimental setup is the same as in Section 5.1. Results in Figure R1 and Table R2 (in the anonymous link above) show that PARM achieves better Pareto front than RS and MOD, and significantly outperforms them in terms of HV and MIP, demonstrating its effectiveness and high alignment quality.\n\n> **[Relation To Broader Scientific Literature]**. (1) ... preference-aware adapter ... would be of great value if the empirical advantages can be well supported.    \n> **[Weaknesses 1]**. the necessity of preference-aware adapter design is not well demonstrated. ... SVD-LoRA is comparable with PBLoRA.\n\nWe appreciate your recognition of the originality of our proposed preference-aware adapter (PBLoRA), but we argue that we've provided sufficient evidence for its design necessity in our paper. We clarify the evidence for the necessity of designing PBLoRA as follows.\n\n- **PBLoRA outperforms SVD-LoRA significantly**: (1) The Pareto front of PBLoRA entirely covers that of SVD-LoRA (yellow vs. purple curves in Figure 3). (2) PBLoRA achieves 11.4% and 59.9% improvements in HV and MIP compared to SVD-LoRA (Table 3).\n- **General framework**: As detailed in Lines 220-226 of our paper, SVD-LoRA is a special case of PBLoRA, which means PBLoRA has a greater exploration space and can achieve better results.\n- **Ablation evidence**: In Section 5.3, we systematically analyze different configurations of PBLoRA, showing the effectiveness of each component. \n- PARM with a single PBLoRA **addresses the inefficiency and misalignment issues in GenARM** using $k$ LoRAs. Please refer to our reply to Weaknesses 1 raised by Reviewer 3zGf for details.\n\n> **[Questions 1]**. In Figure 2(b), why would PARM be much better than GenARM in solely optimizing humor, while only comparable in optimizing helpfulness?\n\nHumor is a narrower objective than helpfulness. PARM learns from multiple preference dimensions jointly, benefiting from shared knowledge, while GenARM trains separate reward models for each preference, preventing it from leveraging others data to enhance humor. This difference leads to PARM’s superior performance on the Humor dimension. Conversely, helpfulness, being a broader dimension, can be effectively learned even with GenARM’s isolated approach, resulting in comparable performance on this dimension for both models.\n\n> **[Questions 2]**. Why does PARM show better numerical performance than GenARM in Table 1,4, while only slightly better than GenARM in Figure 1,4?\n\nHV measures the area under the Pareto front. In the Figures, it is clear that PARM has a larger area than GenARM, leading to a higher HV. \n\nMIP measures the uniformity of the solutions on the Pareto front. Obviously, the distribution (the markers in the figures) of PARM is more uniform, so its MIP is higher.", "claims_and_evidence": "They claim that PARM is more effective and inference-efficient than GenARM. And this is correct both intuitively and empirically.", "methods_and_evaluation_criteria": "The benchmarks (Helpful Assisitant & Safe RLHF) are commonly used in this area. The method PARM makes sense.", "theoretical_claims": "The only theoretical claim is Theorem 4.1. I've checked its proof, and it is correct.", "experimental_designs_or_analyses": "The GenARM is the only baseline in evaluation. As shown in Figure 6 of [1], GenARM is not a strong approach for multi-objective alignment (only slightly stonger than RS, which has been beaten by many recent works like [2,3,4]). Therefore, the value of PARM is not very convincing. Since training a DPO model is not harder than training a reward model, it would also beneficial to include comparision with policy-guided approaches.\n\n[1] GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment. ICLR 2025.\n\n[2] Decoding-Time Language Model Alignment with Multiple Objectives. NeurIPS 2024.\n\n[3] PAD: Personalized Alignment at Decoding-time. ICLR 2025.\n\n[4] Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning. EMNLP finding 2024.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to propose a preference-aware adapter, which would be of great value if the empirical advantages can be well supported.\n\nThe proposed guided-generation process, and the weak-to-strong guidance are already explored by prior works [1,2,3].", "essential_references_not_discussed": "I don't think there is any work essentially related to the work not cited.", "other_strengths_and_weaknesses": "**Strengths**\n- This paper is clear and well-written.\n- The improvement over GenARM shown in Figure 2 is impressive.\n\n**Weakness**\n- The only originality comes from the preference-aware adapter, however, the necessity of this design is not well demonstrated. For example, in Figure 3, SVD-LoRA is comparable with PBLoRA.\n- Comparison with policy-guided approaches is missed. And thus it is unknown whether people should use PARM instead of [2,3,4].", "other_comments_or_suggestions": "> However, it focuses on aligning with personalized preferences rather than managing trade-offs across different preference dimensions.\n\nI don't think there is much difference between personalization and managing trade-offs, since you can always compare them on a same benchmark.", "questions_for_authors": "- In Figure 2(b), why would PARM be much better than GenARM in solely optimizing humor, while only comparable in optimizing helpfulness?\n- Why does PARM show better numerical performance than GenARM in Table 1,4, while only slightly better than GenARM in Figure 1,4?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes a multi-objective test-time alignment method, named PARM, that improves on the previously proposed GenARM technique by training a single "unified" Autoregressive RM in terms of multiple ones, which is shown to outperform GenARM as well as other baselines in the empirical validation of the method.

Although reviewers initially had some concerns, in particular regarding novelty and lack of comparison to some other baselines, the authors did a thorough job to try and address them, eventually convincing all reviewers to advocate for acceptance. I believe this is indeed interesting work that would deserve to be accepted, though I also share the concern that this remains somewhat incremental work over GenARM, with a relatively "niche" application domain (test-time alignment for multi-objective preferences). I am thus recommending a "Weak accept", similar to all three reviewers.
Decision: Accept (poster)