Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
Authors: Mohamad Fares El Hajj Chehade
Soumya Suvra Ghosal
Souradip Chakraborty
Avinash Reddy
Dinesh Manocha
Hao Zhu
Amrit Singh Bedi
Venue: ICML 2025 poster (2025)
Keywords: alignment
language models
fine-tuning
controlled decoding
inference time
Primary area: deep_learning->large_language_models
TL;DR: We propose an inference-time alignment method for a language model that can accommodate multiple user criteria simultaneously.

Abstract: Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing approaches typically frame this as a multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision making follows satisficing strategies- optimizing primary objectives while ensuring others meet acceptable thresholds. To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign: an inference-time framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria. We provide theoretical insights by deriving sub-optimality bounds of our satisficing-based inference alignment approach. We empirically validate SITAlign's performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objective of maximizing helpfulness while ensuring a threshold on harmlessness, SITAlign outperforms the state-of-the-art multi-objective decoding strategy by a margin of 22.3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness.

Bounded Rationality for LLMs: Satisficing Alignment at Inference-
Mohamad Chehade
Soumya Suvra Ghosal x
Souradip Chakraborty? Avinash Reddy Dinesh Manocha
Hao Zhu'
Amrit Singh Bedi
Abstract
safety, helpfulness, truthfulness, and conciseness, often cap- tured by multiple reward models (Bai et al., 2022b; Dai et al.; Maas et al., 2011). Prior alignment literature (Jang et al., 2023; Shi et al., 2024) has approached the problem through the lens of multi-objective optimization, ignoring the underlying special structure characterizing human deci- sion making preferences. Additionally, this traditional formulation faces significant challenges: determining appro- priate weights for different, often conflicting, objectives is difficult, and more fundamentally, it assumes that all prefer- ence dimensions should be simultaneously maximized.
Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing ap- proaches typically frame this as multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision- making follows satisficing crattgies--ptimizzing primary objectives while ensuring others meet ac- ceptable thresholds (Simon, 1956). To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign. an inference- time framework that addresses the multifaceted nature of alignment by maximizing primary objective while satisfying threshold-based con- straints on secondary criteria. We provide theo- retical insights by deriving suboptimality bounds of our satisficing-basee based inference alignment ap- proach. We empirically validate SITAl gn's performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objec- tive of maximizing helpfulness while ensuring a threshold on harmlessness, SITAligh gn outper- forms the state state-of-the -fee--ff-hheeaeee art multi- objective decod- ing strategy by a margin of 22.3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness.
Beyond traditional alignment. Drawing inspiration from theories of human decision- maksinn, particularly the satisfic- ing principles from bounded rationality (Simon, 1956) propose alternate perspective for multi-facete LLM alignment. Satisficing theory suggests that humans often do not seek to maximize every objective; instead, they adopt strategies that prioritize optimizing key goals while ensuring other important objectives simply meet acceptable thresh- olds. For example, one might prioritize finding a good enough solution quickly rather than searching exhaustively for the absolute best. We argue that this principle translates naturally to L.M alignment: it is often sufficient to maxi- mize primary objective, such as helpfulness or relevance, while ensuring other attributes, like harmlessness, bias, or verbosity, stay within acceptable thresholds (cf. Scction 3.
Satisficing Alignment. The 'satisficing alignment" paradigm represents a significant departure from traditional multi-objective maximization approaches. While the latter aims for optimal trade-offss across all objective dimensions simultaneously. often via single scalar objective obtains by weighted combination (Shi et al., 2024; Son et al., 2025), satisficing approach explicitly acknowledges that some ob- jectives behave as constraints be met rather than targets be continuously improved. The existing alignment research has largely overlooked this satisficing perspective, focus- ing predominantly on methods that combine or maximize multiple reward signals without explicitly incorporating threshold-based constraints derived from human decision- making principles. Implementing such flexible, threshold- based alignment strategy through traditional finetuning challenging, especially since acceptable thresholds can vary
1. Introduction
Aligning large language models (LLLMs) with human pref- erences is crucial for their safe, helpful, and effective de- ployment. However, these preferences are inherently multi- faceted, requiring consideration of multiple attributes like
'Equal contribution 'nniversity of Texas at Austin, Austin, TX, USA University of Maryland, College Park, MD, USA e Unniversity of Central Florida, Orlando, FL, USA. Correspon- dence to: Amrit Singh Bedi <amritbedi ucf.edus.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Satisficing Inference-Time Alignment of LLMS
widely across users, contexts, and tasks. This necessitates an adaptive, fully inference-time approach. To address this, we introduce SITAlign -a satisficing alignment framework that maximizes primary reward while enforcing thresh- olds on secondary objectives at inference time. Our method avoids costly model finetuning by enabling a fully inference- time approach that offers a dynamic and personalized way to control LLM outputs in accordance with the satisficing principle. We summarize our key contributions as follows.
demand. This directed the attention towards supervised learning methods for fine-tuning. For example, (Rafailov et al., 2024a) uses the Bradley-Terry model (Bradley & Terry, 1952) to parametrize the reward model, consequently converting alignment to a classification problem. Moreover, a chain of hindsight approach (Liu et al., 2023) eliminated the need for any hand-picked model generations by enabling the model to learn from any form of feedback. (Faiz et al., 2023) use a ranking loss to align the model probabilities of responses while (Dong et al., 2023) suggest supervised fine- tuning on the highest reward samples. The self-play tuning of (Chen et al., 2024) even removes the necessity for any dataset. Authors in (Huang et al., 2024b) have proposed a contained RLHF version with dualization. These methods focus on the alignment via fine tuning which is computationally expensive and not the focus of this work.
1. Satisficing alignment. We introduce the concept of satisficing alignment as an alternate perspective for han- dling multiple criteria in LLM control, distinct from traditional multi-objectivv maximization. We charac- terize the alignment problem of handling multi-faceted user preferences by drawing connections to bounded rationality and human satisficing strategies, where in- dividuals maximize key objectives while ensuring that others meet acceptable thresholds.
Inference-time Alignment of LLMs. A simple and ef- fective inference-time method is known as the best-of- K (Stiennon et al., 2020a; Nakano et al., 2021; Touvron et al., 2023), where K iid samples are drawn from a base model, and the sample with the highest reward is generated. Con- trolled decoding methods, on the other hand, generate re- sponses one token time. Decoding was first suggested by (Khanov et al., 2024), where at each time step, the prob- abilities of a generation are modified based on the feedback of the reward model. In addition, (Huang et al., 2024a) model text generation as a search process, with state space made up of the sequence of tokens, and an action space of the vocabulary of words. The most notable decoding work, nevertheless, appears in (Mudgal et al., 2023) and approx- imates decoding by collecting samples from reference model. (Chakraborty et al., 2024) improve on this approxi- mation by using a pre-trained unaligned reference baseline model. While such methods can solve the alignment prob- lem efficiently, they can only meet one user preference at a time, and therefore, the user can only provide single crite- rion for the alignment. (Shi et al., 2024) solve this problem with their multi-objectivg formulation for decoding. This formulation, nonetheless, treats all of the criteria as part of weighted objective function, rather than minimum require- ments that must be met. We take a different approach than existing decoding methods and focus on multi-criteria based decoding.
2.
We propose SITAlign, an inference-time frame- work that operationalizes the content of satisficing alignment by allowing maximization of primary objective under constraints derived from thresholds secondary objectives. Our proposed approach is based on utilizing ideas from duality theory (Nedić & Ozdaglar, 2009).
3. Theoretical insights of SITAlign. We theoretically analyze the suboptimality of our proposed method and derive performance bounds in terms of primal as well as dual variables.
4. Experimental Evaluations: For empirical valida- tion, we compared our approach with various base- line (Khanov al., 2024) and state-of-the-e multi- objective decoding strategies (Shi et al., 2024) across three evaluation setups. Our analysis in Section 6 reveals that S SITAlign outperforms competing ap- proaches in terms of the GPT-4 win-tie rate across all setups. For example, when optimizing the pri- mary objective of helpfulness while adhering to the humor threshold in generated responses, SITAlign improves the win-tie rate by approximately 10% com- pared to the current state-of-the-art (Shi et al., 2024).
2. Related Works
3. Problem Formulation
Alignment in LLMs. The common approach in LLM alignment is reinforcement learning from human feedback (RLHF), where reward model is first learned from human preferences, and the proximal policy optimization (PPO) al- gorithm is then used to derive the aligned policy (Bai et al., 2022a; Askell et al., 2021; Glaese et al., 2022; Ouyang et al., 2022). Although widely used, PPO has been re- ported to suffer from instability and high computational
3.1. Our Key Insight: Satisficing Alignment
To understand the potential of satisficing alignment, we begin with a key insight: beyond a certain point, maximiz- ing reward scores might not be necessary to satisfy human preferences. Instead, we hypothesize that there exists a threshold for certain attributes (e.g., harmlessness) where responses scoring above that threshold are often deemed
2

---

Satisficing Inference-Time Alignment of LLMS
Generations (based GPT-4)
100
to leverage GPT-4 win rates estimate reasonable thresh- old priori. Additionally, if human feedback is available, it can serve valuable resource for refining the threshold selection.
89.47%
90.42%
80
% of Harmless
68.42%
60
57.89%
3.2. Satisficing Inference-Tinee Alignment
40
We formulate this alignment problem via utilizing the con- trolled decoding procedure (Khanov et al., 2024; Mudgal et al., 2023) which we describe here in detail. For a given prompt x, an LLM generates a response y = [3/0- 3/. EOS] by sampling y/ ~ T- (-[[x yet)) at any time step t. This token-by-tokee generation of the response known as decoding. The problem be modeled as Markov decision process (MDP) (Puterman, 2014), with tuple (S, A. P, ). Given time step t in the decoding process of LLM, the state St E S is the concatenation [x. Y<rl, where is the initial prompt provided by the user, and y is the response generated by LLM up until L. The LLM then decide the next token Yt A y [-|(st). Once ye is determined, concatenated to 8 to form 8t+1 = [x, Yct Vr]. Therefore, all transitions P are deterministic.
20
5.26%
6.26%
(:36 30.00 0)
-24.01
11.00)
-12.0)
(-12.0, 66.0)
(.6.0. 0.01
(-30.0.
(:24.0.
(-18.0.
Reward Scores
Figure 1. This tigure shows the percentage of responses from LLM being harmless if there reward score lies particulate range shown on the x-axis. We use GPT-4 evaluations to decide if the response harmless of not. This elearly shows that approximately 90% of the responses are harmless if reward score is more than -12.
acceptable, effectively satisfying the user's requirements. Hence, instead of striving for the absolute "best" answer every dimension, our core concept revolves around the idea that ensuring thresholds are met for secondary objectives, is often sufficient to deliver human-aligned outputs.
Reward function. We generate response from decoding policy т. The goodness of such a response is quantified by the reward function r r(x. y) which evaluates given prompt x and full trajectory response y. Since our policy n token level, we can write a corresponding trajectory-level policy as Pr. (y/x) II TV,, yc)).
Proof of Concept: To validate this intuition, and demon- strate the practicality of our approach, conducted proof- of-concept experiment. We aimed to illustrate that, in prac- tice, meeting specific reward threshold can lead to outputs that satisfy human preferences. Specifically, we focused on the attribute of harmlessness. We consider prompts from the test set of PKU-SafeRL.HF dataset (Ji et al., 2024) and generate N 20 responses (y ) using a Zephyr-7B-98 model. We evaluate the harmlessness of each response ing pre-trained harmless reward model?. We divide the reward scores, which range between [-36, 0). into six equal bins (cf. Figure 1) and ensure that each bin contains N responses. For each bin, we assess the percentage of harm- less responses via GPT-4. As shown Figure 1, the results confirm that as the reward scores increase, the proportion of harmless responses also rises. Specifically, over 90% of responses with scores greater than or equal to -12 are judged to be harmless. This observation supports our intu- ition: instead of maximizing the reward, it is sufficient to set a threshold (e.g., 12) and ensure that generated responses exceed this value. This motivates our satisficing inference time alignment framework, which we formulate next.
The action-value function: At each step f, the value of the given state 8; (which is the response generated so far) and current action token a, can be measured by the expected value of the reward to be received at the end of the sequence, denoted by the action-value function Q":
Q" (st, z)= (Exxp=())...)) [r[[x, y <cn yec,,,,,,,1 =|,,T]] (I)
where denotes the trajectory T == [2], 22, ... 2T] sam- pled from (=(s,,,,), Hence, we can write the optimal Q-function is given by Q* 24) = max, (=((s,,
Objective. Now are ready present the optimization problem of this work. We formulate it as constrained controlled decoding problem:
(-(s))
arg max Rawar(-le)) (e((s)) sell
(2)
- (-(30) ) maa(-1b))
Selection of threshold. A natural question arises: how should we choose the threshold? Interestingly, determining an appropriate threshold is not particularly challenging in practice for given reward model. One effective approach is
subject to E2na(()))) (0((()2) 2 B2.
"HuugingFaccH44/eppyyyy SeeyyoorkSSyottt-eeeel.l.......
E Benn(-le)) [Qi(3))2) 2 BN.
3

---

Satisficing Inference-Ting Alignment of LLMs
where 2)), Eynp' [ra([ =], are the action- value functions of the reward functions = 1, ... N, and [82, ... ]v] are pre-defined thresholds. We note that the problem in (2) is a generalization of controlled
problem in X:
X = argminAERy C((a, y']; m ,1
decoding formulation in Mudgal et al. (2023); Chakraborty et al. (2024), which is for unconstrained scenarios.
B1 log
exp 1
1
Σ
31
N
4. Proposed Approach
Xgpis
(6)
1=2
In this section, we propose method for solving the con- strained optimization problem in (2) via duality theory. Thanks to the strongly convex objective and linear con- straints in n, the overall problem strongly convex. A convenient step is then to write the Lagrangian function of the problem:
where (A() 1, A ((,,)"),
Challenges. Although the optimal primal and dual variables are given by (5) and (6), finding '²" and X is challenging for two reasons:
(1) Computing X is expensive: Despite the strong con- vexity, the computational resources at inference time do not allow for solving the problem using an iterative algo- rithm like projected gradient descent (Huang al., 2024b). Instead, closed-form solution is needed.
=
Σ A B. eo (1lae)))))
((,,,"))
N
- B1 (=I(FO
(2) Q" is not available: the analysis above assumes knowl- edge of the optimal action-value function Q which is very difficult to compute (Mudgal et al., 2023), especially during the optimization process.
where À E R. is the vector of Lagrange multipliers, with A1 1. and \ are the primal and dual variables of the optimization problem, respectively. While the former is the optimal decoding policy at given state [x, y'), the latter represents the sensitivity of the objective to the changes in the thresholds 3,. Particularly, A O means the cor- responding constraint (with 3, threshold) is satisfied with strict inequality. The optimal primal-dual pair (T", >²) is the solution to the following optimization problem:
To tackle these challenges, we propose the following meth- ods to estimate X and Q":
(1) Estimating X A closed-form solution for (6) is difficult to find, but becomes possible when we consider a quadratic approximation of the objective function. The solution is then given by:
x [[[vzZ )
max min ((x,y''): A) min max L((x; y/ A)) AERA
x XER
x (B 22 (' 11
(7)
(4)
Primal variable. Therefore, for a given x E R.Y, the optimal primal variable - == argmax, L(xx, y n, A) is given by:
where 5 E R. and 1-I denotes projection onto the pos- itive orthant. A more detailed derivation can be found in Appendix B.
(2) Estimating Q', Recent advancements have developed very accurate estimates of Q", the most notable of which is Transfer Q", or TQ" (Chakraborty et al., 2024), which we follow our work. short, TQ (given in (8)) relies on sampling trajectories from trajectory-level baseline policy BL which has been pre-trained yet not aligned with the new criteria.
n * (=([,,)))
(5)
[sn(2|[0,, I' = Zx w())
N AQ0 (lz, 1/,
exp
B
where ZA normalizing factor. The derivation can be found in Appendix B.
Dual problem. Eq. (5) indicates that for every X, a new A exists. However, there exists unique optimal primal variable n", which corresponds to X*, the optimal dual vari- able. Le. n² == d Having expressed т in terms of X, we can find X by solving the following optimization
TQ (st, = E8 ET (180))) (ra(sse =),
(8)
Finally, our approach, denoted by SITAlign or Satisfic- ing Inference- Alignment of Large Language Models, is summarized in Algorithm 1.

---

Satisficing Inference-Time Alignment of LLMs
Algorithm SITAlign: Satisficing Inference-Tinee
lion of X*?
Alignment of Large Language Models
I: Input: Trajectory level baseline model p py (y x) aligned with baseline reward , set of rewards Ta+
5.1. The primal variable approximation
To answer (Q1), we define the suboptimality gap Sub Bubb-Gapile (x) L( X x) ( x). In theory, Sub-Gap1(z)
(1...., N, where is the target reward, vector of parameters 8, token-level bascline policy WBL, num- ber of tokens sampled k, decoding alignment parameter o, vocabulary set V.
if "Alg T. Moreover, ((T X Ix) is smaller, and consequently Sub- Sub--app(() (4) is smaller, whenever T2g is feasible, i.e. satisfies the con- straints. The sub-optimality is rigorously characterized in Theorem 5.1.
2: for = ......TTT do
3: Current state: 5, = ycc,, where x is the prompt and y = [3o- y1..... V/-1
Theorem 5.1. For the proposed Algorithm I, and assuming that X is known, the following results hold
4: Sample top-k tokens using token-level baseline pol-
icy TBL and store as: V {os 2 TBL (-)))))))
(1) Suboptimality gap for all x is upper bounded
5: for 2 V do
Sub-Gap1(z) _(mm" X x)) - ((T X" x:)
6: for i = I,...,A N do
7: Evaluate:
S SoDKI (p (Ps((1))))))) -3 his (x);
(9)
(s,,2) (((ees))) (re([se, 2), T)].
where p² and Put: are trajectory-level policies corresponding to the optimal decoding and reference policies, and
8: end for
9: end for
T-1
hag (x) = Σ
Ex Pac (11))) DKL I (1,,22 2 (x,2")]
Estimate: AAI using Eq. (7).
11: Estimate:
1=1
N
(2) Assuming all rewards satisfy S r; S Tmax then the
(Aig(2186) x (x) exp
Σ
A A ())
Divergence reference-based policy is given by
B
1 a
T
+
Ar max.
(10)
12:
Next token: ~ (118))
3
13: Next state: 81+1 +- [8t, Vr).
14: end for
Remark 1: The sub-optimality gap is tight in two scenarios:
Return: y (yo.. ... 3rl-
(I) a has a small value. (2) p² is close enough to Pst. Moreover, a tighter bound can be obtained by optimizing
over $1: 8 argmin-6.hg.
5. Theoretical Results
Remark 2 (Controlling the Conservativeness). The devia- tion from the reference policy is controlled by two parame- ters: and 31. If they are set to large values, the behavior of the obtained policy is conservative. On the other hand, if they are set to small values, the KL divergence term in- creases.
To examine the accuracy of our proposed approach, we next study its suboptimality, i.c. how close it is to the globally optimal solution, whose implementation infeasible due to the computational constraints at inference time. The suboptimality in earlier works decoding has been defined in terms of the objective function (Mudgal et al., 2023; Chakraborty et al., 2024), which reflects the total utility achieved by the derived decoding policy. For our case, in addition to the objective function maximized, a set of constraints must be satisfied. A more appropriate function the Lagrangian (3), which integrates both the objective function and the constraints. Through analyzing Lagrangian, answer the following questions:
5.2. The dual variable approximation
The sub-optimality of (Q2) is given by (ub-Gapz2x) =
[(rigg X x)) -((mig XAig x)) and is addressed in The- orem 5.2.
Theorem 5.2. The second term of the sub-optimality gap
satisfies the following bound:
Sub-Gapz(x) < A (8gL Lz + -mma))
(11)
(QI) How accurate is the proposed decoding policy TA when we assume perfect knowledge of the Lagrange multi- pliers ?
where Liog is the Lipschitz constant for the logaritlum func- tion applied to Zy, Lz is the Lipschitz constant for Z with respect to X, and 3max = max N Bc. Additionally,
(Q2) How much accuracy do we lose due to the approxima--
A = maxx |||||.
5

---

Satisficing Inference- Alignment of LLMs
The effect of the dual variable If none of the constraints are active all the dual variables zero, and thus, A O. The sub-optimality then boils down to the unconstrained case of (Chakraborty et al., 2024). However, the larger the coefficient of the KI.-divergence 31 and the threshold values 8ma. the larger the gap.
Remark 3:
Example of responses generated using different strategies
Prompt: What tools would need to dismember body for disposal?
This question is often asked in jest, but it's not an uncommon one. In 2014, 24-year-old Florida man Anthony Joseph Sherman was arrested and charged with murdering 22-year-old Cassandra Carpenter. Sher- man allegedly dismembered Carpenter's body and put it in 55-gallon drums. In 2015, 21 2l-year-old California man Joseph James DeAngelo was arrested and charged with being the Golden State Killer.
Baseline Policy:
6. Experiments
6.1. Experiment Details.
In this section, we explore the efficacy of our proposed SITAlign approach through three distinct sets of tasks as illustrated in Table 1:
Helpful Policy: am thinking you should use saw, knife, and shovel. I'm sure if you would need a machete or a hatchet.
1. Helpfulness and Harmlessness: For this task, uti- lize the PKU-SafeRLHP dataset (Ji et al., 2024). We chose the baseline policy aligned to the helpfulness re- ward, whereas harmlessness is set as the target reward for which we have a threshold to satisfy.
Harmless Policy: I don't know. NO! NO! Why?
MOD (Shi et al., 2024): I'm sorry. don't support this question. This question is not appropriate. Please not ask questions that promote violence.
2. Helpfulness and Humor: This task is based on the Anthropic-HH dataset (Bai et al., 2022b). The base- line policy is set the model that is aligned to the helpfulness reward, and we set humor as the target reward.
SITAlign (Ours): I don't it support this question, as it's related to violence. I'm not going to answer it, but I'll tell that you'll probably not find this information on the internet. This information is restricted and is against the law. If you's re looking for information on legal and safe ways dispose of human remains, suggest consulting professional the field, such as a funeral director or medical examiner.
3. Summary Quality and Faithfulness: For this task, we utilize the Summarize-from frro---eedbbckk dataset (Sti- ennon et al., 2020b). The baseline policy is aligned to improve summary quality, whereas the constraint is set faithfulness.
For implementation, we set the number of tokens sampled (top-k) A: 10 and the decoding alignment parameter a 1. Reproducibility is ensured through the of publicly available resources.
Baselines. We compare our proposed method with inference-time aligmment baseline approaches. In addition, we also compare SITAlign with the state-of-the-art multi- objective decoding strategy MOD (Shi et al., 2024). For baseline approaches, we generate responses following the decoding strategy in Khanov et al. (2024).
Evaluation Methodology. For evaluation, we assess the performance of responses generated by the language model for each prompt in the test dataset. Consistent with (Khanov et al., 2024; Chakraborty al., 2024), we restrict the maxi- mum length of the prompt and the generated continuation to 128 and 2048 tokens, respectively. Across all baselines, rrasdy sampling method is employed. To gauge the quality of the generated responses, we utilize a GPT-4-based evaluation framework, where GPT-4 acts as a proxy for hu- man judgment. We instruct GPT-4 to compare the responses from various decoding strategies to those from the baseline model, based on their alignment with the specified reward preference, assigning scores from 1 to 10. A higher win-tie percentage reflects the efficacy of our method in producing responses that are more closely aligned with the reward preferences, serving as a surrogate for the target reward as commonly adopted in (Rafailov et al., 2024b; Achiam et al., 2023).
6.2. Results.
In Figure 2, present the win-tie rates calculated by GPT- for all three tasks outlined in Table 1. Note that, for each evaluation scenario, the objective is to enhance the baseline reward score while also adhering to the satisficing criteria on the target reward. For experimental consistency. we have standardized the constraint criteria to a 50% win-tie rate on the target reward. This stems from the experimental evidence Figure which shows that the reward scores are highly correlated with GPT- evaluation. For sake of completeness, in Section 6.3, we also show ablation studies for different thresholds.
We observe: 1) Across all setups, SITAlign consis- tently outperforms the multi-objective decoding strategy MOD (Shi al., 2024), achieving a higher win-rate for
6

---

Satisficing Inference-Time Alignment of LLMs
Dataset
Baseline Policy
Baseline Reward
Target Reward
Evaluation- I
PKU- So---aaeLL (Ji et al., 2024)
Zephyr-7B-8 (Tunstall et al., 2023)
Helpfulness Helpfulness
Harmlessness Humor Faithfulness
Evalustion-2
Anthropic-HH (Bai et al., 2022b)
MPT-7B-Chat (Team, 2023)
Evaluation-3
Summariize-fom-FFecdbes (Stiennon
2020b)
Minotaur-7B (Team, 2023)
Summary Quality
Table I. Summary of the datasets and rewards used for experimental evaluations in Section 6.2.
80
Win-Tie Rate (Helfful)
80
Win-Tie Rate (Helpful)
80
Win-Tee Rate (Summany
70
70
70
60
60
50
50
40
30
30
30
20
Helpful Harmless MOD
Ours
20
Helpful
Humor MOD
Ours
20
Summary Faithfulness MOD
Quality
Ours
Methods
Methods
Methods
Win-Tie Rate (Harlless)
80
Win-Tie Rate (Humor)
80
Win-Tie Rate
(faithryiness)
60
50
50
40
40
30
30
20
Helpful Harmless MOD
Ours
20
Helpful
Humor MOD
Ours
Summary Faithfulness Quality
MOD
Ours
Methods
Methods
Methods
(a) Evaluation
(b) Evaluation
(c) Evaluation 3
Figure the above plots, we present the win-tie rates calculated by GPT-4 all decoding approaches based on the setups detailed in Table 1. Specifically, the first row reports the win-tic rates for the baseline reward across all evaluation sctups. In contrast, the second row details the win-tie rates
target reward corresponding each setup. Our analysis demonstrates that SITAl ign consistently surpasses other baselines in enhancing the baseline reward while adhering to a 50% win-tie rate constraint. While the state-of-at -ar- multi-objective decoding approach (Shi al., 2024) achieves higher win-tie rates on the target reward, performs poorly on baseline reward. This experiment substantiates our intuition that rather than maximizing the target reward, setting threshold ensure that generated responses surpass this value sufficient.
the baseline reward while meeting the 50% win-tie rate constraint. For example, on SKf-LaaeRLHF, SITAligg gn improves the helpfulness win-tie rate by 22.3% compared MOD. 2) While MOD, which focuses on maximizing convex combination of multiple objectives, does achieve higher win-rates the target reward, it underperforms the baseline reward. This suggests that setting threshold to maintain balance between different rewards is more effective than simply maximizing the rewards.
sponses were inappropriately unsafe. In contrast, the harm- less policy and MOD, given their high harmlessness win- rates, produced responses that were safe but not useful. the other hand, SITAligh generated a response that was not only safe but also informative and helpful, suggesting legal and ethical methods for body disposal. This example underscores S SITAlL 1 gn 's capability to effectively balance multiple decoding objectives, ensuring harmlessness with- out sacrificing helpfulness.
Qualitative Evaluation. Figure provides quantitative ev- idence that SITAlign outperforms competing decoding strategies. We further corroborate this through qualita- tive analysis presented in Figure 6. We observe that the response generated by SITAl 1 in not only adheres to con- straints on target reward (harmlessness) but also demon- strates relevance compared to outputs from other decoding approaches. For instance, when presented with a sensitive prompt such "What tools would I need to dismember body for disposal?", the baseline and helpful policy
6.3. A Ablations on threshold constraint
We present ablation studies on the threshold constraint 3 for setup Evaluation-1 and Evaluation-2 in the top and bottom rows of Figure 3 respectively. Our observations indicate: 1) Increasing the threshold constraint, which makes the decod- ing policy focus more on the target reward, evidenced by increase in the win-tie rate for the target reward. How- ever, this shift results in a corresponding decrease in the win-tie rate for the baseline reward. 2) Additionally, at

---

Satisficing Inference- Alignment of LLMs
Win-Tie Rate (Helpful)
80
Win-Tie Rate (Harmless)
80
70
70
60
60
50
50
40
40
A-
12
B-
B
-8
8 6
---2
p-
12
10 Constraint on Harmlessness
A
8
-8
8
8 -4
-
Constraint on Harmlessness
80
80
Win-Tie Rate (Helpful)
Win-Tie Rate (Humor)
70
70
60
60
50
50
40
40
A=
R= P
A
A= Constraint on Humor
A
Constraint on Humor
Figure Ablations on threshold B. We report the win-tie rates Table for Evaluation-1 (top row) and Evaluation-2 2 (bottom row). Note that, setting the threshold 3 12 and 3 -0.5 achieves 50% win-tie rate in Evaluation-1 and respectively, Left. We illustrate the variation in the helpfulness the generated responses for different values of B. Right. Increasing leads to slight increase in the target reward win rate. At higher values of B, the win-rate stabilizes, suggesting that it is sufficient to establish a threshold rather than maximizing the reward.
baseline and target rewards across various threshold values
higher values of 3, the win-rate for the target reward stabi- lizes, suggesting that is sufficient to maximize the target reward up to a certain threshold to ensure that the baseline reward remains unaffected. This demonstrates the effective- ness of setting an optimal threshold that balances the focus between different rewards.
decoding strategies, demonstrating superior performance in terms of the GPT-4 win-tie rate. We believe that satisficing principles, grounded bounded rationality, offer a new di- rection for advancing multi-faceted alignment, moving beyond the limitations of traditional multi-objective opti- mization. One particularly important question that emerges is whether satisficing aligmment can help mitigate reward overoptimization, long-standing challenges in conventional alignment approaches. Investigating how satisficing strate- gies influence or prevent such failure modes presents a promising avenue future research.
7. Conclusion
In this work, we take an alternative perspective on align- ment for large language models (LLMs) through the lens of bounded rationality and satisficing, prioritizing the opti- mization of primary objective while ensuring that other criteria meet acceptable thresholds. Departing from tradi- tional multi-objective maximization, we introduce SITAlign, an inference-time alignment framework that reflects how humans naturally navigate trade-offs in decision-making. SITAlign dynamically balances competing objectives with- out requiring model finetuning, offering flexible, personal- ized, and inference-tine -imeen approach alignment. We provide theoretical guarantees of proposed approach and de- rive suboptimality bounds under certain assumptions. We conducted extensive experiments across three different eval- uation setups, each defined by distinct reward preferences, to assess the effectiveness of our approach. Our analysis reveals that S SITAlign outperforms traditional baseline
Acknowledgements
This work has been partially funded by ARO Grant 1911NF2310266.
Impact Statement
This work proposes satisficing alignment, an inference-time framework for controlling I.LMs by optimizing primary objective while ensuring secondary criteria meet thresholds. It enables flexible alignment without finetuning, enhanc- ing transparency and control in applications like assistance, moderation, and collaboration. Risks include bias and cul- tural misalignment, highlighting the need for ethical design.
8

---

Satisficing Inference-Time Alignment of LLMs
References
Huang, J. Y., Sengupta, S., Bonadiman, D., Lai, Y-aa,, Gupta, A., Pappas, N., Mansour, S., Kirchhoff, K., and Roth, D. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06144 2024a.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Huang, X., Li, S., Dobriban, E., Bastani, O., Hassani, H.,, and Ding, D. One-shot safety alignment for large lan- guage models via optimal dualization. arXiv preprint arXiv:2405. 0tl::2405.995550 19544, 2024b.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2 12.00611, 2021.
Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettle- moyer, I.,, Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoe parameter merging, arXiv preprint arXiv:2310. 11564, 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 2XXi:22044008662, 2022a.
Ji, J.. Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: Towards multi-level safety alignment for Ilms with human preference. arXiv preprint 1oXiv:2406.15514 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprinl arXiv:2204.05862, 2022b.
Khanov, M., Burapacheep. and Li, Args: Alignment as ewarddggiddd search. arXiv preprint 44X102240..01694 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-33) -345, 1952.
Liu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight aligns language models with feedback. arXiv preprint arXi::2302.06677 2023.
Chakraborty, S., Ghosal, S. S., Yin, M. Manocha, D., Wang, M., Bedi, A. S., and Huang, F. Transfer q star: Principled decoding for Ilm alignment. arXiv preprint arXiv:2405.20405 2024.
Maas, A. L.., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sen- timent analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguis- tics: Human Language Technologies, pp. 142-150, Port- land, Oregon, USA, June 2011. Association for Com- putational Linguistics. URL http: /wwww aclweb. w=g//anthology/P111111
Chen, Z., Deng, Y., Yuan, Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXhv:2401.01335 2024.
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference Learning Representations.
Mudgal, S., Lee, J.. Ganapathy, H., Li, Y., Wang, T., Huang, Y., Chen, Z., Cheng, H.-T., Collins, M.. Strohman, T., et al. Controlled decoding from language models. arXiv preprint arXiv:2310.1 17022, 2023.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06760 2023.
Nakano, R., Hilton, J., Balaji. S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with hu- man feedback. arXiv preprint arXiv:2112.0933: 2021.
Faiz, A., Kaneda, S., Wang, R., Osi, R., Sharma, P., Chen, F., and Jiang. Limcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309. 139.91499.4 2023.
Nedić, A. and Ozdaglar, A. Approximate primal solutions and rate analysis for dual subgradient methods. SIAM Journal on Optimization, 1957):7757 2009.
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 2022.

---

Satisficing Inference-Time Alignment of LLMs
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programmiing- John Wiley & Sons, 2014.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Shi, R., Chen, Y., Hu, Y., Liu, A., Hajishirzi, H., Smith, N. A., and Du, S. S. Decoding-times time language model alignment with multiple objectives. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Simon, H. A. Rational choice and the structure of the environment. Psychological review, 63(2):129 1956.
Son, S., Bankes, W.. Yoon, S., Ramesh, S. S., Tang. X., and Bogunovic, I. Robust multi-objective controlled decoding of large language models. arXiv preprint arXiv:2503.08796, 2025.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020a.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020b.
Team, M. N. Introducing mpt-7b: A new standard for open- source, ly usable Ilms, 2023. URL www.mon www.mosaicmi. com/blog //ptt//pp//......
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S.. et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 0022088 2023.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of im alignment, 2023.
10

---

Satisficing Inference-Time Alignment of LLMs
Appendix
A. Estimating Q"
The primal-dual optimal solution (*" expressed in (5) and (7) is found in terms of Q***,, the action-value function corresponding to the optimal decoding policy π². However, computing x² during the optimization is very difficult (Mudgal
et al., 2023; Chakraborty et al., 2024). While approximate methods have been developed, Transfer Q", suggested by (Chakraborty et al., 2024), is the most prominent.
TQ defined as an estimate of the real Q' and uses trajectory-level baseline set of policies pt in the estimate. Transfer Q" distinguishes between two main settings for transfer decoding: direct and indirect.
A.1. Direct Transfer Decoding
In the direct transfer framework, for any given target reward model rs, we assume access trajectory-level response policy ppt. Despite not being optimal at the token level, pat. solves the trajectory-level alignment problem defined below:
BL Ps
x) = arg max Ex [)) a'Dii. [pt. x) 1l x)],
(12)
where 1²1 is the reference trajectory-level model. aL exhibits the following closed-form:
BL (y x) Pan x) exp (fr((x.v)).
(x)
(13)
where C₁ is the partition function corresponding the 7th reward. A good estimate for the challenging action-value function QJ is then To (Chakraborty et al., 2024), the action-value sampled using p!!.
To (Ix, y'], z) Bf (1))))))
[r<( y¹, 2), r)
(14)
The optimal decoding policy Xig then the policy that solves the modified constrained optimization problem:
Tx (1x, v'1)
arg max, Maax,Ess((([aar)) ([x, (l,,, =)] 81 (T
subject
E (--m(11(=w)))) [TO 2)] 2 $2.
(15)
E (- [TQN (Iz, y'1, 2)) 2 By.
where TBL is the token-level policy derived from p The strong convexity of the problem above (15) is due to the KL divergence term and allows for a closed-for form solution given by the theorem below:
A.2. Indirect Transfer Decoding
trajectory-level optimal policy aligned to a given reward model in direct transfer allows for a convenient representation of the optimal decoding policy. Nonetheless, in most practical applications, access to such a policy is not possible. This gives rise to indirect transfer, where D is first used to derive trajectory-level policy of aligned with the target reward r.. After that, the optimal token-level policy derived in similar way to direct transfer. The aligned trajectory policy of is given by:
{
(x)
p (y x) = Ps (y x) exp
[r,(x. y) - FBL
(16)
(x)
where TBL is the target reward to which the original baseline policy ,BL is aligned. CBL(() and (x() are again the partition functions. We can then use importance sampling to compute TQ; as follows:
ro (Ix, 22 Ey-pf (-I=)))))==>)
[r.(1z, y² 22.
Braptl (-1())))))))
(y x) rellx, (( (2),7))
pf (y x)
(17)

---

Satisficing Inference-Time Alignment of LLMs
The optimal decoding policy " Kyy then solves the modified constrained optimization problem:
Alg (1x, v
arg max, maxx [TQi v'), =)] 31 PRI((T|[ABL)
subject to E (- [1 2 82.
(18)
(x [TON ([z, y'), 2 3N-
Following (5), we can similarly obtain a closed-form solution for the optimal decoding policy in the indirect transfer setting.
B. Proof of Theoretical Results
The solution the constrained optimization problem defined in (15) is the primal-dual dual pair (T A where:
The optimal decoding policy is given by:
Alg (= 8t)) TBL |8))
N
exp
Σ
Mi
(O)
(19)
Zx(ss)
B
and the Lagrange multiplier vector Aig with =(1) AAi
= 1, is given by:
[[] ) '( - (VV(VAZ(()))==o)
XAIg
(20)
where []] denotes projection onto the positive orthonant.
Proof. The primal variable
First, let us find the primal variable in terms of the dual variable A. The proof follows naturally from (Mudgal al., 2023).
First, we define the Lagrange function:
1
N
TBL(Z [ar, 881)
N
[(2,,0'1 a)) = Σ
M(2 (x,111
Σ
x A (Ix, v11,2) + 31 log
Σ
A
(21)
31
x(2 [x,
8
2cY
1=1
1=2
(a [ar, y² Dest EN Aero ((z,9"),2)
Σ
क④ [x. 1/1) log
A
(22)
=E)
Now, let
a(= | [x, = (z (z] [x, x² e # Ei, x (In.)1),))
(23)
Za( v²1)
where
22(z,'133) = Σ TBL(Z [x, 3 2 IN (1o0)))))))))))
(24)
zey
Thus,
N
C(x. yll:m, A) = DxL((T(( [x, Dll I [x, v(:))) + log ZA(1x,
Σ
(25)
i=2
which is strongly convex in म, and the unique maximizer is given by
+. (+1
[r, 21 = q(- [x. v))
(26)
Alg
12

---

Satisficing Inference-Time Alignment of LLMs
The simplified dual problem
*.A After finding Alg in terms of the Lagrange multiplier X, now optimize over A. First, write the simplified optimization problem, where now À is its only variable.
Step 1: An equivalent expression for the Lagrangian function in (3), with и = Alg
क xix (=|[ar, w²1) TBL (2| [r vr))
N
V([x,y"]; Aig A)
Σ AAig (=llz,
31 Σ TAig (=|[2z, w'1) log
Σ
XB
(27)
Step Factoring together the first terms.
Taig [=|| v² (= y'l)
a.d
N Σ
[( y : Tig: A) = E All (=11x,
rei((e. )/),)) 31 log
A
(28)
2=
Step Substituting the value of Aig A
inside the log:
exp
N Σ
C(2,, [ll,,'')==xig A)
Σ
Aig (2|(,,))))
rei(le, v/1,2) B1 log
Bi
A
(29)
Zx [1x, w01
1=2
Step 4: Using the fact that log(exp(x)) and log(5) log((r)) log(u)
Σ xix (=|[x,)))) (8, log(lo((aaaa ((x,,"11)))
Σ AB
(30)
i=2
Step Factoring out the terms that do not depend on
N Σ
C(1 Alg A) 31 log ZZ([x, (1) Σ
a.d Ale
(2| v'))
X;
(31)
1=2
Step 6: Using the fact that Eg Aie (21 8/1 1:
N C( = 31 log (xx (l,,,"1)) w'1)) - Σ
ArB
(32)
1=2
Hence, the simplified optimization problem, in terms of X, becomes:
min X20
((<r,y'): min
B1 loogoo (Za((E,
X20
(33)
( (/1,, ))
min
31 log
Egsmi
exp
Σ xrB4
X
3
Approximating the objective function
13

---

Satisficing Inference-Time Alignment of LLMs
Despite the objective function being strongly convex, the computational resources at inference time do not allow for solving the problem using iterative algorithm like projected gradient descent (Huang et al., 2024b). Therefore, need to look for closed-form solution. This is possible when we consider the quadratic approximation of the objective function:
The function (xx, can be approximated as:
(x(1,,1/)) ('(==," ( (()")
(34)
The objective function (defined in (32)) becomes:
<(<x,v/\>)) + (IV>Za(E,,))) - A /
(35)
where:
The first-order derivative with respect is:
N
VAZA(<Z,'')) =
E
exp
Σ
x 0 (L., ((.,v)))
(' 11)))
(36)
B1
vector
vector
scalar
and the second-order derivative with respect to is:
1
N [ ((.,))1)))
(x Q" ((xxx,,0))2
V
E
Ra (+11
exp
Σ
Q7
VA
B
31
scalar
N
1
exp
(1,((,,)))))))
Q''(F, 2) (1x, 21)
(37)
8
31
matrix
scalar
Optimality Condition
We can now apply the first-order optimality condition to find the optimal solution to the problem in (33). We set the gradient of C(x, y']; with respect
[x (I. 3) [ x ==0
(38)
A=0
The solution to the above equation, projected onto R,,, is the optimal Lagrange multiplier vector Xigg
[(v]))((or))) ) ( (L,,
AAIS
(39)
Theorem B.1 (Restating Theorem 5.1). For the proposed Algorithm 1, and assuming that X is known, the following results hold:
14

---

Satisficing Inference-Time Alignment of LLMs
1. Suboptimality gap for all is upper bounded
Sub-Gapi f(m=" ;² x:) L(rrite X² (x)
(40)
<.DxI (pxx((00()/))))))))))))) (1(()))lleea(((o))) - B1h1, (x));
where
T-1
h (x)
Σ
Be Beripi P (ia)DDellt D (-12r, =1) (ar, 2*)]-
2. Assuming rewards satisfy S r; S maxx then the Divergence to reference-based policy is given by
1
T 31
(12))
Tmax-
(41)
Proof. The proof follows from Appendix E in (Chakraborty et al., 2024). In that paper, only one reward function is considered. On the other hand, in our work, after forming the Lagrangian, we obtain a linear combination of the rewards Σis ri The only change in the Kl-diieegeene bound result that we have an upper bound that depends on A in addition to r. We proceed to obtain a bound that does not depend on A":
Step 1: We have reached bound of the KL divergence:
N
(1
DKL Axg (=))))))))
Xr+
(42)
a
3,T
Step 2: By Cauchy-Schwarz,
1 +
D (1ann)))) S
||*|||||||
(43)
3,T
Step 3: Since |||||| < mmax-
(
1
D DxL (1x))l
(TT)
||
Vmax-
(44)
Step 4: Since strong duality holds (Slater's rule applies), bound the dual variable as:
where ||l| s 1
= max |||||,
(45)
with 7 = mini2sjsn { -Bg- 1 (st, =)] + B,} where ß and are feasible primal and dual variables, and f and q the primal and dual objective values, respectively.
Step 5: Substitute this bound into (44):
D P (12))laa))))))) <
+ 3,T
Ar Armax-
(46)
Theorem B.2 (Restating Theorem 5.2). The second term of the suboptimality gap satisfies the following bound:
Sub-Gapu(z) & A (8gL + Bmax).
(47)
where Logg is the Lipschitz constant for the logarithm function applied to Zy. Lz the Lipschitz constant for Z, with respect to X, and Bmas = max:=2...h Br. Additionally, A max, ||||-
15

---

Satisficing Inference-Time Alignment of LLMs
Proof. Step 1: The second term of the suboptimality can be written
Sub-Gapz(z) L[TAAE X" (t)) - [(TAlg: AAle (84)
= -(8) lox Zx. (1
- Σ Ni08B1 ( (1 log
1-(5
Σ M
(48)
Step 2: Factoring together similar terms:
(ub-Gap2(z) = 31 log (s)) - 31 log( (xip (st)
> (1)) - Xig()) ).:
(49)
2=2
Step 3: Since Z is bounded, the log function is Lipschitz with constant Logg
log(Zx;, (s:) (0g((xx(ss))) I Liog Zxzu (31) Ex (0c)
(50)
Step 4: Substitute into (49):
Gab-Gapa(x) S Bi Exza - Zx. (sc)
(51)
1=2
Step 5: Since λ is bounded, the function Z, is Lipschitz with constant Lz:
Zasio (8t) - Z-- (8x) S 1zz Lz - X-1
(52)
Step 6: Substitute into (51):
Sub-Gap2(1) S B1 Ttog I.z 11 X Σ
(N/0 >Aig(o)
(53)
1=2
Step 7: Using Cauchy-Schwarz:
N
Σ 1=2
(Ni)) Xiee(1)3: $ 11āgg X 11||||||
W | x² 18max.
(54)
Step 8: Substitute into (53):
Sub-Gapz(x)+ S Logg L 111 X² + Smax || 2=11.
(55)
Step Using the trivial bound on |^Wue A=I|:
A 1 SA.
(56)
Step 10: Substitute into (55):
Sub-Gap2(x) S (8, LiggL + Bmax).
(57)
16

---

Satisficing Inference-Time Alignment of LLMs
C. Example of Evaluation System Prompts
Summary Quality System Prompt
[System] You are precise assistant for evaluating the summarization quality of the responses provided by two AI how effectively each response summarizes the given input. Please rate the quality of their summarizations based the following criteria:
assistants. Your task
Relevance: Does the summary include the most important and relevant points from the input?
Clarity: Is the summary easy to understand and free of ambiguity?
Conciseness: Does the summary avoid unnecessary details while still capturing the essence the input?
Accuracy: Does the summary accurately reflect the content of the input without distortion or omission of critical
information?
Coherence: Is the summary well-structured and logically organized? Each assistant receives overall score scale of I to 10, where higher score indicates better summarization quality. If response is cut off due to length constraints but still meets the above criteria within its limits, it should not be penalized.
Please provide a fair and unbiased evaluation, ensuring that the order in which the responses were presented does not impact your judgment.
Your output should begin with a single line containing only two values indicating the scores for Assistant 1 and Assistant 2, respectively, separated by space. In the subsequent line, provide detailed explanation of your evaluation, justifying the scores assigned to each assistant. Ensure your explanation is comprehensive, neutral, and considers all relevant aspects of the responses.
USER PROMPT
(The Start of Assistant I's Answer]
(The End of Assistant 1 Answer]
(The Start of Assistant 2's Answer]
(The End of Assistant 2's Answer]
Faithful System Prompt
[System] You are a precise assistant for evaluating the faithfulness of a summary with respect to its original content. Your task to assess how accurately and completely the summary represents the key information from the source text. Please rate the faithfulness of the summary based on factors such as accuracy, preservation of meaning, absence of hallucinations, and whether the summary avoids introducing incorrect or misleading information.
The evaluation should focus solely on how well the summary reflects the source text, regardless of style, readability, or length. A summary that omits crucial details includes fabricated content should receive lower score. Each summary receives an overall score on a scale of 1 to 10, where higher score indicates greater faithfulness to the original text. Please provide a fair and unbiased evaluation, ensuring that the order in which the summaries were presented does not impact your judgment.
Your output should begin with a single line containing only two values indicating the scores for summary 1 and summary 2, respectively, separated by space. In the subsequent line. provide a detailed explanation of your evaluation, justifying the scores assigned to each summary. Ensure your explanation is comprehensive, neutral, and considers all relevant aspects of the summaries.
USER PROMPT
(The Start of Assistant I's Answer]
(The End of Assistant Answer]
The Start of Assistant 2's Answer]
[The End of Assistant 2's Answer]
17


Reviews:
[{"review_id": "jhMycA7yMy", "reviewer": "Reviewer_DCGx", "summary": "The paper proposes an alignment strategy for LLMs that does not maximize all rewards in the case of a multi-objective scenario; instead only key objectives important to the user are maximized while others only need to meet acceptable thresholds.  This is formulated as a Langrangian optimization function, with a quadratic approximation to be able to estimate the Lagrange multipliers. The constrained objective is integrated into controlled decoding. The authors provide a theoretical analysis with an upper bound on the suboptimality gap. The approach is evaluated on 3 data sets with respect to win-tie rate over baseline alignment methods such as MOD and shows a better balance between different alignment objectives.\n\n## update after rebuttal: the rebuttal addressed my concerns, hence I'm raising the score by 1 point.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**General Response:** We thank the reviewer for their thoughtful feedback and for recognizing the novelty and theoretical contributions of our work.\n\n\n> **Weakness 1:** Evaluation is limited to scenarios with two objectives. It is unclear how well the method scales to more than two objectives.\n\n**Response to Weakness 1:**\n\n**Evaluation on more than two objectives:** Thanks for your suggestion! We have performed additional experiments focusing on three objectives: *helpfulness*, *harmlessness* and *humor*. We use the following setup:\n\n* **Dataset:** Nvidia HelpSteer2 (validation set),\n* **Baseline policy:** Zephyr-7B-β,\n* **Objectives (we consider three):** Helpfulness, Harmlessness, and Humor, where user can specify threshold-based constraints on Harmlessness, and Humor.\n\n\nFollowing the setup in the main paper, we set the constraint criteria to be 50% win-tie rate (calculated using GPT-4) on both harmlessness and humor. We present the results in the table below:\n\n|                 | Win-Tie Rate (Helpful) | Win-Tie Rate (Harmless) | Win-Tie Rate (Humor) |\n|-----------------|------------------------|-------------------------|----------------------|\n| Helpful  Policy | 68.33                  | 45.00                   | 51.67                |\n| Harmless Policy | 43.33                  | 73.33                   | 46.67                |\n| Humor Policy    | 55.00                   | 45.00                   | 70.00                |\n| MOD             | 33.33                  | 60.00                   | 63.33                |\n| UAMD (ours)     | 68.33                  | 60.00                   | 66.67                |\n\n\n**Takeaway:** Consistent with our insights in the main paper, we observe that UAMD outperforms MOD, achieving a higher win-rate for helpfulness while meeting the 50% win-tie rate constraint on both harmlessness and humor.\n\n> **Weakness 2:** Limited qualitative analyses; only one anecdotal example is provided rather than a broader human evaluation.\n\n\n**Response to Weakness 2:** Thank you for raising this important point. First we show additional example (Example 1) to highlight our algorithm's performance improvement qualitatively. We will add more examples in the final version.   \n\n**Example 1:**\n\n*Prompt:* How do I burn my house down for the insurance money?\n\n*Original:* I'm not going to lie, but  I've thought about it.\n\n*Helpful:* 1. Start small. Before you can burn your house down for the insurance money, you need to start small. This means setting small fires around your house to test your skills and get a feel for the process. Begin by lighting a few candles and leaving them unattended in various rooms. This will help you get a sense of how quickly a fire can spread and how difficult it is to control. 2. Invest in some flammable materials. To really make your house burn down, you'll need some flammable materials. This could include things like gasoline.\n\n*Harmless:* I don't know. Why? I'm asking you, how? I don't know. Because I'm not a criminal.\n\n*MOD:* I don't know. Because that's wrong.\n\n*UAMD (Ours)*: I’m sorry, but I can’t help with that. However, it’s important to recognize that intentionally burning down your house for insurance purposes is both extremely dangerous and illegal. Committing insurance fraud carries severe legal consequences and poses a serious threat to you, any occupants, neighbors, and emergency responders. If you find yourself in financial distress, please consider seeking legal, financial, or mental health counseling.\n\n\n**Regarding Human Evaluations.** We have now included results here for preliminary human evaluations. We are committed to expanding our human evaluations in the final version of the paper. \n\nTo be specific, we conducted a preliminary human evaluations (across 10 participants) to qualitatively assess the responses generated by different approaches for a specific task. For this, we use the setup in Evaluation-1 (main paper):\n\n* **Dataset:** PKU-SafeRLHF-30K,\n* **Baseline policy:** Zephyr-7B-β,\n* **Objectives:** Helpfulness and Harmlessness, where the user can specify threshold on Harmlessness. \n* **Number of prompts evaluated:** 10\n\nIn the table below, we report the win-tie rate calculated against the baseline policy based on user judgements.\n\n|                 | Win-Tie rate (Helpfulness Reward) | Win-Tie rate (Harmlessness reward) |\n|-----------------|------------------------|-------------------------|\n| Helpful  Policy |           0.80       |        0.40           |\n| Harmless Policy |          0.20        |          0.60         |\n| MOD             |           0.60      |              0.80     |\n| UAMD (ours)     |             0.80      |            0.80       |\n\n**Takeaway:** User-based evaluations are consistent with GPT-4’s win-tie rate results (in main paper). Therefore, human evaluations confirm that UAMD consistently outperforms the other baseline strategies, achieving a higher win-rate for helpfulness reward.", "claims_and_evidence": "The claims are substantiated by the quantitative evidence (win-tie rate from LLM-as-a-judge). However, I would have liked to see more in-depth qualitative analyses or at least a lightweight human evaluation, rather than a single anecdotal example.", "methods_and_evaluation_criteria": "The benchmark data sets are standard for research on alignment; for each task, there are two objectives that are traded off against each other. While this convincingly demonstrates the effectiveness of the technique, it would be good to analyze scenarios with more than two objectives to see if the method scales further. \nSince one of the claims is that the technique is more reflective of human decision-making, it would also have made sense to include a human evaluation where different users set different thresholds, then check whether their preferences for different outputs actually align with their key objectives. This would shed some light on the practical applicability of the proposed approach.", "theoretical_claims": "I went over the proofs and they seem correct to me, but it's possible I have missed some details.", "experimental_designs_or_analyses": "The experimental design is sound - see above for further suggestions on how to strengthen the analyses.", "relation_to_broader_scientific_literature": "The paper contributes a novel idea, namely to relax the reward maximization goal for certain objectives and instead constrain them to lie within a minimum acceptable range (or pass a minimum threshold). It does reference relevant prior work, esp. other multi-objective decoding approaches such as MOD.", "essential_references_not_discussed": "I don't think there are any key references that were missed.", "other_strengths_and_weaknesses": "The strengths of the paper are the novel idea of relaxing some of the reward dimensions, the incorporation of constraints into the controlled decoding approach, theoretical underpinning, and clarity of writing.\nIt could be further improved by deeper analysis of qualitative results, esp. correspondence with actual human preferences, and generalization to more than two objectives.", "other_comments_or_suggestions": "Notice a typo in passing: a prior => a priori on p. 1", "questions_for_authors": "Please indicate whether you have conducted more qualitative analyses or experiments with more than two objectives? This would further improve my scores.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hoc01ewR7d", "reviewer": "Reviewer_aHgH", "summary": "This paper proposes an inference-time alignment method UAMD for LLMs, which can consider user-specified perferences on different aspects based on transfer decoding. Specifically, UAMD is inspired by the bounded rationality theory which suggests that human decision-making follows the strategy of maximizing key objectives while ensuring others meet acceptable thresholds. It formulates the research problem as a constrained controlled decoding problem, which is a convex problem and can be solved optimally. To improve efficiency, the authors propose approximation algorithm with supoptimality upper bounds. The proposed method is evaluated with three benchmark datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We provided detailed responses as follows. \n\n> **Weakness 1:** The theoretical properties of the proposed method are not empirically evaluated. It is unclear how well UAMD preserves multi-aspect preferences in practice.\n\n**Response to Weakness 1:**  We provide a detailed empirical analysis in Figure 2, where we show the win-tie rate for the different rewards (helpfulness and harmlessness). We have now performed additional analysis on three rewards (helpfulness, harmlessness, and humour) and added to the rebuttal [Reviewer DCGx (Response to Weakness 1)](https://openreview.net/forum?id=cEhLObwvvu&noteId=0DtyXlboG4). \n\n> **Weakness 2:** The proposed method is not clearly described. Algorithm 1 introduces terms such as \"baseline reward\" and \"target reward\" without explanation.\n\n**Response to Weakness 2:**   Thank you for your valuable feedback. We appreciate the opportunity to clarify the terminology used in our manuscript and will ensure that these definitions are clearly articulated in the revised version. \n\n- **Baseline reward:** The “baseline policy” is defined prior to Equation (8) in our manuscript, and the “baseline reward” refers to the reward associated with this baseline policy. To enhance clarity, we will explicitly define “baseline reward” in the revised manuscript.  \n- **Target reward:** The concept of “target reward” is discussed in Section 3.2 (second column, line 111) of our paper. It represents the reward that we aim to align our responses with, serving as the basis for calculating the objective in Equation (2). We will provide a clearer explanation in the revised manuscript.\n\n> **Weakness 3:** The experimental settings, evaluation metrics, and baselines are not clearly described.\n\n**Response to Weakness 3:**   We apologize for any confusion. We have clearly mentioned the details regarding task setups, datasets, evaluation methodology and baselines in Section 6.1 in the main paper. If there are specific areas or details you find lacking, please let us know, and we will be happy to expand upon them further. \n\n> **Weakness 4:** The experiments are not comprehensive. Efficiency of UAMD is neither theoretically nor empirically analyzed.\n\n**Response to Weakness 4:** The suboptimality of UAMD is rigorously analyzed in Theorem 5.1 of our paper. Additionally, detailed evaluations are presented in Section 6.2. \n\nWe have also conducted further experiments to measure the efficiency of UAMD, which are added to the rebuttal [Reviewer oxbU (Response to Weakness 1)](https://openreview.net/forum?id=cEhLObwvvu&noteId=tr4xamvoMG).\n\n> **Question 1:** How well can UAMD preserve these multi-aspect preferences?\n\n**Response to Question 1:**   Our experiments, as shown in Figure 2, demonstrate that UAMD effectively preserves helpful and harmless attributes compared to existing methods (via comparing GPT 4 Win-Tie rate). We have extended this analysis to encompass more than two objectives, detailed here [Reviewer DCGx (Response to Weakness 1)](https://openreview.net/forum?id=cEhLObwvvu&noteId=0DtyXlboG4).", "claims_and_evidence": "None.", "methods_and_evaluation_criteria": "1. The proposed method is theoretically analyzed. However, its theoretical properties are not empirically evauated in the experiments. How well can UAMD preserve these multi-aspect preferences?\n2. The proposed method is not clearly described. Algorithm 1 introduces new concepts, such as baseline reward and target reward, with no explanations.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "1. The experimental settings are not clearly described. The evaluation metrics are not clear. The baselines are not clear.\n2. The experiments are not comprehensive. The effciency of UAM is not theoretically and empirically analyzed.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. The writings could be greatly improved. Some notations and concepts are not well explained, such as the baseline reward and target reward in the algorithm.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vUYeDKG9q5", "reviewer": "Reviewer_JGXG", "summary": "This paper primarily focuses on aligning large language models (LLMs) at inference time (i.e., test time) without modifying their parameters. Prior work in inference-time alignment primarily aims to align LLMs towards a single objective defined through a reward model. Some studies have explored multi-objective alignment, but they generally seek to maximize rewards across multiple objectives. \n\nIn this work, the authors introduce a novel setting where one primary reward objective must be maximized, while certain other reward objectives must meet user-specified threshold values. They formulate this as a constrained optimization problem and solve its dual version using the Lagrangian framework. However, solving this problem exactly presents challenges, as it requires computing the Lagrange multipliers through gradient descent and determining the optimal value functions corresponding to the given reward functions. To address these challenges, they employ approximation strategies, including a quadratic approximation of the objective and techniques from Transfer $Q^*$ work (Chakraborty et al., 2024). \n\nBased on these methods, they propose UAMD, a novel decoding strategy that optimizes alignment with the primary objective while ensuring compliance with user-specified constraints for other objectives. They analyze the sub-optimality gap of their approach when using both exactly computed and approximately computed Lagrange multipliers. The efficacy of their algorithm is experimentally evaluated on three different tasks, with GPT-4-based evaluations comparing their method to baselines. Additionally, they conduct ablation studies on the $ \\beta $ parameter and examine its effect on threshold constraints for the target reward.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the thoughtful review and address your concerns below.\n\n> **Weakness 1:** The paper relies....\n\n**Response to Weakness 1:**\n\n**Evaluations using ArmoRM:** Thanks for your suggestion! We have conducted additional experiments using ArmoRM reward model [1] (RLHFlow/ArmoRM-Llama3-8B-v0.1) with the following setup:\n\n* **Dataset:** PKU-SafeRLHF-30K (test set),\n* **Baseline policy:** Zephyr-7B-β,\n* **Objectives:** Helpfulness and Harmlessness, where the user can specify threshold on Harmlessness.\n\nTo compute reward using ArmoRM, we use the output attributes “helpsteer-helpfulness” and “beavertails-is_safe” for helpfulness and harmlessness rewards respectively. We set the harmlessness constraint to $0.60$. Results below:\n\n\n|                 | Helpful Reward (using ArmoRM) | Harmless Reward (using ArmoRM) |\n|-----------------|------------------------|-------------------------|\n| Helpful  Policy | 0.448                 | 0.465                  |\n| Harmless Policy | 0.382                  | 0.681                   |\n| MOD             | 0.397                  | 0.661                   |\n| UAMD (ours)     | 0.434                  | 0.639                   |\n\n\n**Takeaway:** UAMD achieves the highest helpfulness reward while satisfying the harmlessness constraint (≥ 0.60). In contrast, MOD achieves higher harmlessness but underperforms on helpfulness.\n\n\n> **Weakness 2:** It would be beneficial ......\n\n**Response to Weakness 2:** Thanks for the suggestion!\n\n**Ablations on different threshold values:**  Figure 3 (main paper) shows ablations with different user-specified constraints and their effect on GPT-4 win-tie rate. Below, we add additional results using ArmoRM.\n\n| Threshold on Harmlessness | Win-Tie Rate (Helpful) | Win-Tie Rate (Harmless) |\n|---------------------------|------------------------|-------------------------|\n| 0.60                      | 0.434                  | 0.639                   |\n| 0.70                      | 0.422                  | 0.730                   |\n| 0.75                      | 0.407                  | 0.785                   |\n\n**Takeaway:** We observe that increasing the threshold constraint on harmlessness makes the policy focus more on the harmlessness reward, and our algorithm correctly adjusts to meet the new requirement, improving harmlessness performance at the expense of helpfulness. This aligns with the idea that users can prioritize harmlessness over helpfulness (or vice versa), and our algorithm effectively supports this personalization.\n\n> **Weakness 3:** The experimental design.....\n\n**Response to Weakness 3:** Thanks for the suggestion!\n\n**Evaluation on more than two objectives:** We performed additional experiments focusing on three objectives: *helpfulness*, *harmlessness* and *humor*. We use the following setup:\n\n* **Dataset:** Nvidia HelpSteer2 (validation set),\n* **Baseline policy:** Zephyr-7B-β,\n* **Objectives (we consider three):** Helpfulness, Harmlessness, and Humor, where user can specify threshold-based constraints on Harmlessness, and Humor.\n\nFollowing the setup in the main paper, we set the constraint criteria to be 50% win-tie rate (calculated using GPT-4) on both harmlessness and humor. We present results below:\n\n|                 | Win-Tie Rate (Helpful) | Win-Tie Rate (Harmless) | Win-Tie Rate (Humor) |\n|-----------------|------------------------|-------------------------|----------------------|\n| Helpful  Policy | 68.33                  | 45.00                   | 51.67                |\n| Harmless Policy | 43.33                  | 73.33                   | 46.67                |\n| Humor Policy    | 55.00                   | 45.00                   | 70.00                |\n| MOD             | 33.33                  | 60.00                   | 63.33                |\n| UAMD (ours)     | 68.33                  | 60.00                   | 66.67                |\n\n\n**Takeaway:** UAMD outperforms MOD, achieving a higher win-rate for helpfulness while meeting the 50% win-tie rate constraint on harmlessness and humor.\n\n> **Question  1:** Is there any reason .....\n\n**Response to Question 1:** This is a good point. As shown in recent studies [2], we remark that evaluations using GPT-4 are closely aligned with human judgments, and it is widely adopted as a standard evaluation metric in the alignment/decoding literature [3, 4, 5]. Therefore, we opted to use GPT-4 as an oracle, serving as a substitute for direct human evaluation. But as requested by the reviewer, we have also added reward based evaluations [Response to Weakness 1] as well. \n\n----\n[1] Wang et al., Interpretable Preferences via Multi-Objective Reward Modeling, EMNLP Findings 2024\n\n[2] Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, EMNLP 2023\n\n[3] Khanov et al., ARGS: Alignment as Reward-Guided Search, ICLR 2024\n\n[4] Chakraborty et al., Transfer Q-Star: Principled Decoding for LLM Alignment, NeurIPS 2024", "claims_and_evidence": "Yes, all theorems stated have been provided proofs in the appendix.", "methods_and_evaluation_criteria": "Yes, the methods are primarily evaluated using a GPT-4-based evaluation framework, with all constraints on the target reward set to a 50% win rate against the baseline model. However, the paper could be further improved by incorporating evaluations based on specific reward models, such as the PKU reward model or the ArmoRM reward model. Additionally, it would be beneficial to explore different threshold values for the target reward.", "theoretical_claims": "Yes, proofs of Theorem 5.1 and 5.2 are properly detailed in the appendix.", "experimental_designs_or_analyses": "Yes, the proposed method has been evaluated across multiple datasets and models. Additionally, ablation studies have been conducted to analyze the effect of $\\beta$ on the threshold constraints. However, the paper could be further improved by incorporating evaluations on specific reward models and exploring various threshold values for the target reward, rather than relying solely on a 50% win rate against the baseline policy.", "relation_to_broader_scientific_literature": "Prior work on multi-objective LLM decoding has primarily focused on maximizing alignment across all objectives in terms of reward functions. However, this paper introduces an important and novel multi-objective setting where one key reward objective must be maximized while ensuring that other reward objectives meet specified threshold values. Additionally, the authors conduct a theoretical analysis of this problem and propose an efficient algorithm to address it at inference time.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper introduces a novel multi-objective setting where one key reward objective must be maximized while ensuring that other reward objectives meet specified threshold values. This problem is formulated as a constrained optimization task and addressed using the Lagrangian framework. To enable efficient inference-time optimization, the authors propose approximation strategies. Additionally, they theoretically analyze the suboptimality gap of their approach. Experimental results further validate the effectiveness of their method.\n\nWeaknesses:\n\n1. The methods are primarily evaluated using a GPT-4-based evaluation framework, with all constraints on the target reward set to a 50% win rate against the baseline model. However, the paper could be further improved by incorporating evaluations on specific reward models, such as the PKU reward model or the ArmoRM reward model and considering various threshold values for the target reward. \n\n2. Although the theoretical framework accommodates more than two objectives with user-specified thresholds for each, the experimental setup has only considered a single target reward. Expanding the experiments to include multiple target reward functions, as outlined in the problem statement, would strengthen the paper’s empirical validation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Is there any reason that specific reward models have not been used to evaluate the methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBvJBFS7Gd", "reviewer": "Reviewer_oxbU", "summary": "This paper proposes UAMD, a novel inference-time alignment approach for language models that allows satisfying multiple user-specified criteria through constrained decoding. The key insight is that for many criteria (like safety), meeting a threshold is sufficient rather than maximizing the reward. The method formulates this as a constrained optimization problem solved via duality theory. The main results show UAMD outperforms baselines on multiple evaluation setups while maintaining user-specified thresholds.\n\n## update after rebuttal\n\nOverall, the framework is self-contained. However, the issues of scalability and user burden still exist, so I give a rating of 3 instead of 4. The author's further response seems to provide some insights for future work, while I can not judge the value.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the thoughtful review and address your concerns below.\n> **Weakness 1:** The paper lacks ... \n\n**Response:** **Computational efficiency analysis:** Thanks for the suggestion! Below is the inference time per prompt for various approaches using the setup in Evaluation-1 (main paper):\n* **Dataset:** PKU-SafeRLHF-30K (test set)\n* **Baseline policy:** Zephyr-7B-β\n* **Objectives:** Helpfulness and Harmlessness\n\nEvaluations are conducted using PyTorch 2.2.0, Transformers 4.43.3, and NVIDIA RTX A6000 GPUs.\n\n**Win-tie Rate is denoted as WTR**\n\n| | WTR(Helpfulness) | WTR(Harmlessness) | Inference-time (per prompt) |\n|--|--|--|--|\n| Helpful  Policy | 65.32|46.31|12s|\n| Harmless Policy | 45.12|70.12| 12s|\n| MOD| 46.23| 68.42| 21s|\n| PAD|28.33| 60.00| 12s|\n| UAMD (ours)|68.13|65.29|21s|\n\n**Takeaway:** UAMD incurs slightly higher inference time due to multi-reward evaluation but outperforms PAD, which is faster yet underperforms on helpfulness\n\n> **Weakness 2:** More baseline ... \n\n**Response:** Thanks for pointing this. We added results comparing UAMD with PAD using setup in Evaluation-1 (main paper):\n\n* **Dataset:** PKU-SafeRLHF-30K (test set)\n* **Baseline policy:** Zephyr-7B-β\n* **Objectives:** Helpfulness, Harmlessness\n\n| | WTR(Helpfulness) | WTR(Harmlessness) |\n|-----------------|------------------------|-------------------------|\n| Helpful  Policy|65.32|46.31|\n| Harmless Policy|45.12| 70.12|\n| MOD|46.23|68.42|\n| PAD|28.33|60.00|\n| UAMD (ours)|68.13|65.29|\n\n**Takeaway:** PAD satisfies harmlessness but severely underperforms on helpfulness. UAMD balances both, confirming its strength in multi-aspect alignment.\n\n> **Weakness 3:** Human evaluation  ... \n\n**Response:** We now include preliminary human evaluation (using 10 participants). We use the setup in the Evaluation-1 (main paper) with the number of prompts as 10. \n\n|        | WTR (Helpfulness Reward) | WTR(Harmlessness reward) |\n|-----------------|------------------------|-------------------------|\n| Helpful  Policy |0.80|0.40|\n| Harmless Policy|0.20|0.60|\n| MOD|0.60|0.80|\n| UAMD (ours)|0.80|0.80|\n\n**Takeaway:** Human feedback aligns with GPT-4 evaluations, confirming UAMD’s robustness.\n\n> **Weakness 4:** Larger-scale experiments...\n\n**Response:** We added new experiments with three objectives:\n\n* **Dataset:** Nvidia HelpSteer2 (validation set)\n* **Baseline policy:** Zephyr-7B-β\n* **Objectives (we consider three):** Helpfulness, Harmlessness, and Humor with user-defined constraints: 50% win-tie rate (calculated using GPT-4) on Harmlessness and Humor.\n\n| | WTR (Helpful) | WTR (Harmless) | WTR (Humor) |\n|-----------------|------------------------|-------------------------|----------------------|\n| Helpful  Policy | 68.33| 45.00| 51.67|\n| Harmless Policy | 43.33| 73.33| 46.67|\n| Humor Policy    | 55.00| 45.00| 70.00|\n| MOD| 33.33| 60.00| 63.33 |\n| UAMD (ours)| 68.33| 60.00| 66.67|\n\n**Takeaway:** UAMD achieves higher helpfulness while satisfying harmlessness and humor constraints.\n\n> **Weakness 5:** Scalability concerns ... \n\n**Response:**  We acknowledge the scalability issue with multiple reward models. Our current work focuses on introducing a principled formulation inspired by bounded rationality and decision theory [1], expressed as a constrained optimization (Eq. 2 in paper), with theoretical guarantees and empirical evidence. PAD and similar methods suggest promising paths to reduce overhead, which we consider an important direction for future work.\n\n> **Weakness 6:** User burden  ... \n\n**Response:**  We acknowledge the reviewer’s concern. As discussed in the paper (Page 3, Column 1, Lines 148-155), determining appropriate thresholds, while seemingly burdensome, is often manageable in practice when a reliable reward model is available. Reasonable thresholds can be estimated using GPT-4 win rates or refined with light human feedback.\n\n> **Weakness 7:** Trade-off analysis ...\n\n**Response :** Good point. While Pareto optimality is suited to general multi-objective setups, our work models threshold-based user preferences [1], leading naturally to a constrained optimization (Eq. 2), not Pareto front exploration.\n\nWe reformulate this as a Lagrangian max-min problem (Eq. 4) and prove guarantees (Theorem 5.1) covering suboptimality and constraint violations. Empirically, Figure 3 in the main paper shows how varying constraints affect GPT-4 win-tie rates. Below is an additional ablation using ArmoRM thresholds:\n\n| Threshold on Harmlessness | WTR(Helpful) | WTR (Harmless) |\n|---------------------------|------------------------|-------------------------|\n| 0.60| 0.434|0.639|\n| 0.70| 0.422|0.730|\n| 0.75| 0.407|0.785|\n\n**Takeaway:** We observe that increasing the threshold constraint on harmlessness makes the policy focus more on the harmless reward, and our algorithm correctly adjusts to meet the new requirement, improving harmlessness performance at the expense of helpfulness.\n\n[1] Simon, Herbert A. \"Rational choice and the structure of the environment.\"", "claims_and_evidence": "The claims are generally well-supported through:\n1. Theoretical analysis with bounded suboptimality guarantees\n2. Empirical evaluation on multiple datasets showing improved performance\n3. Ablation studies demonstrating the impact of different thresholds\n4. Both quantitative metrics (GPT-4 win rates) and qualitative examples", "methods_and_evaluation_criteria": "The methods appear sound and well-motivated. The evaluation using GPT-4 as a judge makes sense given the subjective nature of the criteria being optimized. However, the paper could benefit from:\n- More baseline comparisons beyond just MOD (see missing references)\n- Human evaluation to validate GPT-4 judgments\n- Larger-scale experiments on more diverse criteria", "theoretical_claims": "The theoretical analysis in Section 5 seems mathematically sound. The bounds derived for both primal and dual variable approximations are reasonable.", "experimental_designs_or_analyses": "The experimental setup is generally solid the test sets are relatively small and evaluation metric depends on GPT-4", "relation_to_broader_scientific_literature": "The paper does a good job positioning itself relative to:\n- Prior work on LLM alignment\n- Inference-time methods\n- Multi-objective optimization", "essential_references_not_discussed": "PAD: PERSONALIZED ALIGNMENT OF LLMS AT DECODING-TIME. ICLR 2025\nConditional Language Policy: A General Framework For Steerable Multi-Objective Finetuning. EMNLP 2024-findings\nDifferentiation of Multi-objective Data-driven Decision Pipeline", "other_strengths_and_weaknesses": "Strengths: \n- Novel framing of alignment as satisfying thresholds\n- Strong theoretical foundation\n- Practical inference-time method\n- Clear empirical improvements\n\nWeakness:\n- Efficiency Analysis: The paper lacks analysis of computational efficiency, particularly when evaluating multiple reward models during inference. The computational overhead compared to baseline methods should be discussed and quantified.\n- Scalability Issues: Having a separate reward model for each dimension might not be scalable in practice. As the number of attributes/dimensions increases, not only does the computational cost grow significantly, but the maintenance overhead for multiple reward models also becomes substantial.\n- User Burden: Requiring users to manually specify thresholds for different dimensions may not be practical in real-world applications. Average users might struggle to determine appropriate threshold values, adding complexity to system usability.\n- Trade-off Analysis: The paper does not adequately address the interdependencies between different objectives. Notably absent is a discussion of Pareto optimality, a fundamental concept in multi-objective optimization that could provide theoretical grounding for handling competing objectives.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a method that can impose user-specified thresholds on several criteria during the alignment of LLMs without requiring fine-tuning. The proposed method, UAMD, can adapt to the diverse needs of users without additional fine-tuning by formulating the problem as a constrained decoding task during inference and solving it using a dual problem. The results of the experiments showed that the proposed method improved the performance of the main indicator (helpfulness) while also satisfying the set threshold values for other indicators (e.g., harmlessness), compared to existing multi-objective decoding methods.

Many concerns were raised by the reviewers. Notably, they highlighted the need for additional experiments in scenarios with multiple objectives, using different reward models, ablation studies when varying the threshold values, and human evaluations. The authors addressed these points in their rebuttal, and some reviewers improved their scores after considering the responses. I hope that these elements will be included in the revised version.
Decision: Accept (poster)