Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
Authors: Zhaowei Zhang
Fengshuo Bai
Qizhi Chen
Chengdong Ma
Mingzhi Wang
Haoran Sun
Zilong Zheng
Yaodong Yang
Venue: ICLR 2025 Poster (2025)
Keywords: AI Alignment
LLM Test Time Preference Adaptation
Alignment without Training and Fine-tuing
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce Amulet, a training-free framework that enables real-time optimization to satisfy user's personalized preferences for LLMs at test time.

Abstract: How to align large language models (LLMs) with user preferences from a static general dataset has been frequently studied. However, user preferences are usually personalized, changing, and diverse. This leads to the problem that the actual user preferences often do not coincide with those trained by the model developers in the practical use of LLMs. Since we cannot collect enough data and retrain for every demand, researching efficient real-time preference adaptation methods based on the backbone LLMs during test time is important. To this end, we introduce **Amulet**, a novel, training-free framework that formulates the decoding process of every token as a separate online learning problem with the guidance of simple user-provided prompts, thus enabling real-time optimization to satisfy users' personalized preferences. To reduce the computational cost brought by this optimization process for each token, we additionally provide a closed-form solution for each iteration step of the optimization process, thereby reducing the computational time cost to a negligible level. The detailed experimental results demonstrate that Amulet can achieve significant performance improvements in rich settings with combinations of different LLMs, datasets, and user preferences, while maintaining acceptable computational efficiency.

Published as a conference paper at ICLR 2025
AMULET: REALIGNMENT DURING TEST TIME FOR PERSONALIZED PREFERENCE ADAPTATION OF LLMs
Zhaowei Zhang Fengshuo B
Qizhi Chen'
Chengdong Ma'
Mingzhi Wang
Haoran Sun'
Zilong Zheng2t M
Yaodong Yang* [
Institute for Artificial Intelligence, Peking University
State Key Laboratory of General Artificial Intelligence, BIGAI
3 SShanghai Jiao Tong University
#Zhongguancun Academy
ABSTRACT
How align large language models (LLMs) with user preferences from static general dataset has been frequently studied. However, user preferences are usually personalized, changing, and diverse regarding culture, values, or time. This leads to the problem that the actual user preferences often do not coincide with those trained by the model developers in the practical use of L.Mss. Since we cannot collect enough data and retrain for every demand, researching efficient real-time preference adaptation methods based on the backbone L.LMs during test time is important. To this end, we introduce Amulet, novel, training-free framework that formulates the decoding process of every token as a separate online learning problem with the guidance of simple user- seer-provided prompts, thus enabling real- time optimization to satisfy users' personalized preferences. To reduce the compu- tational cost brought by this optimization process for each token, we additionally provide a closed-form solution for each iteration step of the optimization process, thereby reducing the computational time cost negligible level. The detailed experimental results demonstrate that Amulet can achieve significant performance improvements in rich settings with combinations of different LLMs, datasets, and user preferences, while maintaining acceptable computational efficiency.
INTRODUCTION
The success of large language models (LLMs) has led to their widespread application in scenarios such as customer service (Raiaan al.] 2024), content creation (Hadi et al., (024). and personal assistance (Chen al. 2024), emphasizing the importance of maintaining alignment with human preferences (Ji et al. [023] Anwar et al. 2024). However, existing LLM alignment researches often focus on aligning with user preferences from a static general dataset, neglecting personalized and changing preferences. This leads to challenges in ensuring that the alignment goals designed by model developers adequately address users' evolving needs in real-time post-deployment scenarios
[Lao & Xiao 2023 Lazar & Nelson 2023 Zhang et al. 2024b Corréa 2024 Zhang et al. 2024a).
preferences and use meth- ods such as reinforcement learning from human feedback (RLHF) (Christiano et al., [017] Ouyang eet al. 2022) or direct preference optimization (DPO) (Rafailov et al. 2024) for further fine-tuning. However, these approaches not only face the difficulty in the requirements engineering problem of specifying users' real needs (Pohl 1996; Mechergui & Sreedharan 2024), but also from the fact that users' preferences continuously change with culture, community, context, scenario, and time Mac- Intyre| 2013 Eckersley 2018 Turchin 2019 Zhang et al., [203 Qiu al. 2024 Zhu al.) 2025). If the requirements anal analysis, yis, data collection, and subsequent fine-tuning processes are repeated each time, it will result in a significant cost burden. Just as shown in the (a) and (b) parts of the Figure 1] this phenomenon leads to "last mile" problem (Boysen et al. 202 in existing alignment research.
One straightforward approach is to recollect data for personalized
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Amulet framework
What should we do to stay healthy?
(4) Align with personalized preferences from interactions
With preference A
Without preference
Itentively optimine clich token
Values Culture
Bockthone LLM peeference distribution
Generate the mext tolken
Time
Answer
Residual
Buse Answer
Needs Context
PPO
Misalign
DPO
(b) Align with general preference from static dataset
Final answer aligns with user
An illustration of our Amulet framework and its background. The figure is intersected by axis, with each node the axis displaying different distribution that shows the constantly changing user personalized preferences due to factors like time, value, need, and context, as illus- trated by the part (a). The part (b) shows that existing methods mostly consider aligning LLMs with general preferences from a static dataset, which may result in misalignment in dynamically person- alized scenarios. In the part (c), we have enlarged one of the preference nodes to show the processing of our Amulet framework. We formulate the decoding process of every token as a separate online learning problem and further adapt the backbone LLMs align with the current user preference through a real-time optimization process with the guidance of user-provides prompts. The red token means the current processing token, which will the condition for the next token prediction.
Figure I:
To solve this problem, we believe a lightweight preference adaptation method implemented at the LLM test time is needed. Several existing works have already made some attempts in this regard. Assisted inference methods focus on training weak models to guide strong model inference dif- ferent aspects, including in natural language form (Ji et al. [222]] Bai al., 2024). ). interpolation form (Liu et al. [2024b] Zheng et al. (024),, sampling form [Wu et al. 20244] Snell et al. (024)). rep- resentation form (Kong al. 2024). and logits form (Mitchell et 2023 Liu
2024a). RAIN et 2023b) and URIAL (Lin et al. 2023) mainly focus on self self-distillation allowing the model refine the tokens that better match user preferences continuously. Nevertheless, the efficiency of these methods is still not complex inference, Linear Alignment (LA) (Gao et al. 2024) proposes a linear approximation pref- erence update method that achieves acceptable computational efficiency. However, it influences the inference process in only a basic manner, falling short of better approximating users' personalized preferences.
acceptable level for either the training of a weak model or the
Therefore, in this paper, we suggest new perspective that the problem can be solved by performing further online adaptation at test time based the backbone LLMs. To this end, illustrated in the (c) part of the Figure we introduce Amulet (reAlign Ment dUrrng test time for personaLized prEference adap' Tation), a novel, training-free framework that formulates the decoding process of every token as an independent online learning problem with the guidance of simple aroviprovdde prompts, thus enabling real-time optimization satisfy user preferences. Specifically, unlike meth- ods such as RLHF (Christiano et al. 2017 Ouyang et al., (022) that treat the entire decoding process as a Markov Decision Process (MDP) aiming maximize cumulative reward, we consider iter- atively optimizing the Atready-generatee
policies for each token's 's decoding process separately for more comprehensive preference approximation. We follow the idea gellow-hheregglariizedd (FTRL) (Hazan et al. 2016) to carry out the specific optimization process, and further improve the optimization process more stable in the LLM setting by introducing the proximal regularizer to the vanilla FTRL process (McMahan 201T). We can then use the asov-provided simple prompts to sim- ulate the optimization direction by comparing whether there a difference between policies given the prompts not (Gao al. 2024). Since this process may consume significant amount of
2

---

Published as a conference paper at ICLR 2025
computational cost, we further give closed-form solution for each iteration, thereby reducing its computational and time costs to negligible levels.
In summary, our contributions are three-fold. Firstly, we suggest new perspective that the gener- ation of each token at test time can be modeled as an independent online optimization problem and provide the Amulet framework. This method can not only align with personalized and changing user preferences based on the backbone LLM policies but also does not require additional training and fine-tuning. To our knowledge, we are the first study to introduce the optimization process of online learning into the work of test-time preference alignment. Secondly, for each round of optimiza- tion iteration, we further provide its closed-form solution, thereby reducing the computational cost of iterative calculations to a negligible level. Lastly. we carried out comprehensive experiments to validate the efficacy of our framework across diverse settings, incorporating various combina- tions of L.Ms, test datasets, and user preference dimensions. The findings reveal that our approach can achieve significant performance improvements and surpass all baseline methods in the majority of scenarios (an average of 75% among all the experimental settings, 100% for the best LLM, and 93.8% for the best preference), thus providing a final "amulet" for aligning personalized preferences.
2 RELATED WORK
In this section, we will introduce the background of the related research. The existing alignment methods for LLMs can generally be divided into three categories: training time alignment methods, assisted inference methods, and tuning-free methods. We will elaborate on them separately below.
Alignment at Training Time. This category is currently the most mainstream way of alignment, mainly focusing first training the model itself on datasets, and then freezing the parameters for in- ference. There are many well-known algorithms, including RLHF (Christiano et al.] 2017; (uyang
ct al. 2022). CAI (Bai al. 2022), DPO (Rafailov et al. 2024), ORPO (Hong et al. (2224), and Ethayarajh et al. 2024). J. In addition, there are some newer methods. SimPO (Meng et al, eliminates the dependence on a reference policy in DPO by introducing the sequence average
KTO
2024
log probability as an implicit reward. Quiet-STaR (Zelikman et al. 2024) enhances generalization across more reasoning tasks by training the model's reasoning and thinking abilities. Although these methods can achieve good results in capability metries, they have to recollect enough data and per- form further training and fine-tuning, making them unable to meet users' changing and personalized requirements in specific scenarios.
Assisted Inference Methods. This class of methods typically involves training weak model (usually a small model or a pre-trained model) to enhance the alignment of a strong model during inference time, which has various implementation forms. Aligner (Ji et al. 2024) and Alignment via Bayesian Persuasion (Bai et al., (204) use weak generated natural language to influence strong model behavior. ExPO (Zheng et al.] (2024 adopts an interpolation approach, linearly combining the parameters of the small model with those of the large model to achieve aligmment. Improved sampling strategies (Wu et al. 2024 Snell et al. 2024 Liu et al. (225)), from the perspective of optimal sampling, allow the strong model to generate higher-quality text under the guidance of a smaller reward model. EFT (Mitchell al.) 2023). DeRa (Liu et al. 2024b), and proxy-tuning (Liu et al. 2024a) integrate the logits of the aligned small model and the unaligned large model to guide the decoding process. RE-CONTROL (Kong et al. 2024) enhances the alignment ability of the large model by training value model to edit the information representation of the large model during the decoding process. These methods reduce computation and time costs by training smaller models, but they are essentially the same training-time time methods and still fail to address the issue of changing and personalized preferences.
Tuning-free Methods. Methods in this class usually consider further optimization at inference time. RAIN (Li et al.] (203b) uses the LLM itself as a reward model to perform inference and roll- back correction during the inference phase. URIAL (Lin et al. 2023) designs an inference approach by comparing the differences before and after model alignment, allowing the model to continuously correct and reinforce tokens that better match user preferences. However, both of these methods still require a significant amount of computational time and resources during inference. LA (Gao et al.) 2024) proposes method that compares the distribution changes caused by user preference prompts on the original model and linearly updates the original model's logits, which achieves acceptable computational efficiency. However, it influences the inference process in only basic manner, falling short of better approximating users' preferences.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
In this section, we will formally introduce the Amulet framework. We begin by introducing our task settings, followed by reviewing online learning and the FTRL algorithm, and finally, we introduce specific definition of our method and its closed-form solution.
3.1
TASK SETTINGS
Unlike most existing methods which view the decoding process of LLMs as an MDP aiming to maximize cumulative reward, our approach focuses on online optimization for the generation of each token at every timestep, thus enabling real-time optimization to satisfy user preferences.
For LLMs, the generation process of each token can be seen as policy, representing the distribution of each token at the current moment. Typically, this policy is related to the model's parameters, but we use only the already generated policy and perform post-proesssing on it, then it can be made independent of them. Therefore, optimization target is the already generated policies for generating each token in the sequence.
Thus for the generation of each token, need to find the following optimal policy x² =a(a):
(a)
arg max (4.s)
well
where so represents the initial prompt, s represents the sequence that has already been generated, E A denotes the optional token in the token space A, and denotes the latent reward function that reflects the real current user preference.
3.2 FOLLOW-THE BELLLOW-THERRGGLLAREE REGULLRIIEED
ALGORITHM
We first briefly introduce online learning, which involves the training process of model as it con- tinuously receives information. Unlike common offline learning, it cannot access all the training data at once but update iteratively, reflecting the impact of new situations in real time. This method allows the model to adapt to changing scenarios as well as explore and utilize unknown data distributions
effectively.
We will then review the FTRL algorithm, which is a frequently studied online learning framework (McMahan [201T] Abe al. (0022). Its core feature the introduction of strongly convex regular- izer to the fictitious play process, which enhances the algorithm's 's stability and convergence (Hazan al. 2016). Typically, the optimization process of FTRL for the (t 1)-th iteration can be expressed by the following iterative formula (Jacob et al. 2022 2023):
Σ (((m((a))))))))) <(T((a))
T++1 (a) = arg max
(2)
xEll
Because of the inability to obtain the user's true reward function, in each iteration I, we need an approximate utility function Ug continuously provide an approximation of the user preferences for iteratively developing the policy that best meets user needs. The first item in the above formula the fictitious play process, which aims reduce the regret between the current policy and the historical expected policy, and the second one is the regularizer, for each token's 's generation, we can adapt this iteration proceps the policy and further get the optimal one.
0 is the learning rate. Now,
3.3 AMULET FRAMEWORK
Based on the above introduction, we can see that for LLMs, the process of decoding each token at test time can be based on customized utility function to achieve further optimization. Since we provide a general framework that unrelated to the utility, the utility function only needs to
reflect the relative quality of each token, and its selection can be very diverse, including methods
based on inductive bias (Kadavath et al. 20022] Gao et al. 2024). human interaction (Mechergui & Wang al. 2024b), and environment feedback (Sutton 2018
Sreedharan 2024
[e et al. 2022)
4

---

8: sample the generated token a with (a) Published as a conference paper at ICLR 2025
Algorithm Decoding Process with Amulet
Require:
LLM for generating policy; basic prompt Phasei preference prompt Ppref's current generated
sequence 8, iteration number T; maximum new token number M: parameters a, A, and ry; blank string
1: repeat
2: generate Ti (a) PLLM(a Prasse Pease. Pprct s), Tbase (a) PLLM (alpbase, s) with the given LLM
3: fort= 1,2...., T do
4: calculate (=)) == alog (a) log Thuee (a))
5: update the policy with the iteration given by Equation 6
6: end for
7: get the optimized policy (a(a) < (Ty (a)
9: update the current sequence < s a
until the length of s reaches M or generation ended return the full generation sequence 8
11:
Drawing from Contrastive Decoding eL al. 2022) and LA (Gao al. 2024), define the utility function to empirically simulate the optimization direction at time / here as:
Ue (a) == alog ne (a) log Thsse (a)).
Unlike recommendation systems that need to infer preferences Zhang et al. 2021), in real-time setting, users directly provide explicit preferences in the form of prompts that reflect their current needs. Here, define Paase the user's base prompts (e.g. questions), and Perer represents the user's ' specific real-time preferences prompts. PLLM (a|s) denotes the probability each token by the LLM conditioned on the prompt the policy being optimized at the current moment (a) = PaIM(alpwwe, Ppref, 8) to simultaneously include Paase Ppref. and the sequence of tokens s generated at the current timestep. The base policy Thase (a) = PLLM (a) Pbase s) is a baseline policy that does not include user preferences, and is an adjustable parameter. Therefore, the intuition of this utility to gradually amplify the difference brought by the preference prompt Ppref over the base prompt Prase through better policy during the iteration process and further optimize in this direction for the current LLM, until the preference information it brings is fully exploited.
generating the current timestep. We then define
To avoid unreasonable optimization results, and to accelerate the convergence rate, we further intro- duce a KL. regularization term into the utility function, which constrains the current policy (aa) not to deviate too far from the initial one my (a), and the ratio is adjusted by a controllable parameter X. We define un (IT) (ue, x), and therefore we can update the utility function as:
(4) Typically, vanilla FTRL adopts an entropy item as the regularization term (Jacob et al.) 2022). To make the optimization more stable, introduce the KL term between T(a) and (a) the prox- imal convex regularizer. Subsequently, we will take Equation 4 into Equation 2] and we will obtain an FTRL-proximalllike (McMahan 2011) iteration dynamics:
ue(T) = u ADKL (TTIIT)
Σ (((T) (RL((T|/T))
Tt-1 = arg max
(5)
TEll
The dynamics, comprehensively exploiting the preference approximation brought about by the util- ity function, allows the policy to converge in the last iteration with linear convergence rate. De- tailed proofs provided in Appendix A.2
Next, we need to decide how to optimize this objective. Since our method requires considerable number of optimization iterations for the generation of each token, it obviously consumes lot of time and computational cost. Therefore, we further provide a closed-form solution for EEquatin 5 thereby reducing the computational cost of this iterative optimization an almost negligible level.
5

---

Published as a conference paper at ICLR 2025
Proposition 3.1. The Equation 5 has closed-form solution that is given by:
(a) exp
1 txm +1
Σ 1=1
ua(a) + Anytog (a) log To (a)(1) ((()))
(6)
We can follow this closed-form iteration to reach the optimal policy. If the number of iterations is fixed, the time complexity of our method is at the order of O(n), where n is the number of generated tokens. More computational efficiency details are provided in Appendix [B55 We have further provided the pseudo code for showing the
A complete derivation is provided in the Appendix A.
details of the full decoding process with Amulet in Algorithm
EXPERIMENTS
In this section, we conduct extensive experiments evaluate Amulet with various combinations of LLMs, datasets, and user preferences. Our results demonstrate that our framework significantly improves LLMs' alignment performance, indicating its great potential for real-time user preference adaptation.
4.1 EXPERIMENT SETTINGS
We will first introduce the specific experimental setup, including the evaluated models and datasets, the baseline methods, and the evaluation metrics.
Evaluated Models and Datasets. In this paper, we evaluate four popular open-source models: Llama-2-7B-Chh (ouvron et al.) 2023), -lamm-3.1-8B-Instten (Dubey et al. 2024), QWen2-7B- Instruct Yang al. [2024a b). and -isstrl-7B-Innstreett were chosen for their diversity in architecture and performance characteristics, enabling a thorough assessment of our framework across different model types.
(Jiang et al.J 2023). These models
Since Amulet is designed without additional training or fine-tuning, we use the collected data solely for evaluation purposes. We construct four datasets for our experiments:
HelpSteer (Wang et al. 2023) QA dataset aimed evaluating the model's capability to follow instructions across five dimensions, including informativeness and factuality. We extracted the question part, focusing on single-sentence questions create dataset of 1,236 testing instances.
Ultrakeedback (Cui al. 2023) comprehensive, high-quality AI feedback dataset designed to surpass traditional human feedback. From Ultraeeddbcck, we selected two high-quality QA datasets: Truthful QA (Lin et al. [20I]. which includes 811 testing problems, and UltraChat (Ding et al.) 2023), from which we applied similar extraction and filtering as with HelpSteer, resulting 3,845 testing problems.
Personal Preference Eval (Personal) (Gao al. 2024) used to evaluate user preference align- ment. We utilized the original dataset containing 548 testing instances.
For these datasets, we only use their questions, which more similar to real-world applications where LLMs need to provide answers that align with users' real-time preferences for various ques- tions. The extracted testing problems from these datasets serve as the base prompts for our experi- ment, providing diverse range of user interactions to thoroughly evaluate the performance of the Amulet framework.
Baseline Methods. We compare the performance of our method with several baselines:
Base refers to the original LLM using only the base prompt, which serves as the default response generation approach without any additional alignment or preference adjustment.
Preference (Pref) involves the original LLM augmented with preference prompts for prompt engincering. The preference prompts are manually designed to reflect user preferences, serving as way enhance alignment without modifying the underlying model architecture.

---

Published as a conference paper at ICLR 2025
Table Results of our Amulet framework and all the other baselines on various combination settings of LLMs, user preferences, and datasets. All results are the arithmetic averages of the reward model scores on each dataset. The bold text in the table indicates the best performance under that setting. The colors in the table represent the percentage improvement of that method in the current setting relative to the Base method, with more positive growth bluer and more negative growth redder.
Modal
Datoct
Base
Pief
Ceutive
B516
LA
Anvalet
Base
Pief
Verlose BS16
Annalei
Concisc BS16
LA
Amnale
Buse
Prof
Tyditing BS16
Anole
Holpitz
0.30
0.30
0.34
11.16
0.39
0..27
0.27
[1:00
0/42
00:00
0.32
0133
0.33
0.:9
0.41
Persooa
0.34
0.34
0.35
0.38
0.42
0.30
0.30
0.30
0,30
0.30
0.47
0.45
0.50
0.54
0.53
0.41
0.42
0.42
0.45
0.46
M
Trahhi QA
0,52
18.35
0.34
11.16
0.41
01300
0.31
0.51
0.35
11.32
D.41
0.44
0.47
0:51
0.49
0.76
0.38
0.79
0.47
0.47
Ult Chat
00:54
11.55
0.35
11.16
0.33
01.31
0.71
0.51
11:31
1645
0.46
0.47
0.90
0.78
0.59
0.42
--------
0.34
0.30
0.31
0.52
0.37
0.38
0.44
Helgfanoe
Pensoza
0.34
0.34
0.37
0.31
0.31
0.25
0.40
0.53
0.40
0:42
Qwee2:78
Trutzi QA
0.32
08.33
0.33
01.44
0.36
0.30
0.31
0.32
0.33
11.32
0.41
0.45
0.50
0.54
0.51
0.36
0.38
0.20
0.44
0:45
Uhin Clut
0.34
0..44
0.34
0.35
0.76
0.31
0.37
0.32
0.32
0.31
0:40
0:45
0.46
0.51
0.57
0.38
0.33
0.70
0.00
0.35
Averae
0.53
11.44
0.34
11.35
01.31
0.32
11.30
0149
0.55
0.38
0.39
0.40
0:42
HelyStear
0.34
0.36
0.44
0.50
0.30
0.31
0.45
0.53
0.36
0.37
0.50
Persoul
0.36
0.36
0.31
0.53
0.42
0.61
Trahris QA
18.35
0.31
11:11
0.56
0.29
0.29
11:4
0.90
0142
0.49
D134
0.56
0.37
0.49
Vlua Clus
0.33
0.34
0.34
0.42
0.57
0.31
0.32
0.52
0.36
0.41
0.38
044
0.41
0.45
0.53
0.37
0.38
0.38
0.44
0.45
Awraag
0.33
18.74
0.35
11.4%
0.56
01:90
0.31
0.12
0.33
01:4
18:18
0.43
0345%
0.51
0.57
0:377
0.734
0.79
0:45
0.57
HolpStoa
0.32
0.33
0.35
0.37
0.36
0.28
0.29
0.31
0.31
0.30
0.30
0.42
0.44
0.45
0.47
0.36
0.37
0.30
9.40
0.38
Pensozd
0.26
0.40
0.41
2771
Chui
0833
0.34
01.37
0.29
0.34
0.43
0.41
0.37
0.38
8.43
Averigo
0.32
0.35
0.33
0.37
0.41
0.28
0.29
0.29
0.30
0.32
0.36
0.40
0.42
0.47
0.50
0.37
0.38
0.39
0.41
0.44
2012 BoulangerLLewandowssii et al. 2013) is a decoding strategy that helps LLMs find long-term optimal solutions by considering the square of the beam number of tokens during decoding. Due to its heavy memory consumption, we use beam number of 16 (BS16) in this paper.
Beam Search (BS)
Grrves
Linear Aligmment (LA) (Gao et al. 2024) is a token-level test-time alignment method that pre- dicts the optimization direction of DPO and performs a linear update on the original policy. This is currently the state-of-the-ar (SOTA) method for test-time personalized preference alignment.
Evaluation Metrics. Since our task aims to align open-ended user preferences, finding a targeted preference dataset to train the corresponding reward model as the metric is difficult. To address this issue, we selected the intruction-following dimension from ArmoRM-8B (Wang et al.] 2024a) as the main evaluation metric for our experiment. This reward model is currently placed second in the RewardBench (Lambert et al. 2024 rankings
To further ensure the completeness of the evaluation, we employed GPT-40 as discriminator to assess whether responses from different methods better met the requirements, categorizing outcomes as win, lose, or tie. The evaluation prompts were adapted from the AlpacaEval standard format (Li ct al. 2023a) to fit our experimental context, with specific modifications detailed in the Appendix
4.2 EXPERIMENTAL RESULTS
To thoroughly and systematically evaluate our method's performance, we measure the average per- formance across different combinations of models, datasets, and preferences, as detailed in Table I Specifically, we consider eight preferences: creative, sycophantic, verbose, complex, formal, pleas- ant, concise, and uplifting- As indicated by prior research (Zhong et al., 2024), these preferences reflect common diverse user interaction scenarios and their demonstrated impact on user satisfaction.
The results show that our method significantly improves user preference alignment compared to all baseline methods measured by reward model score. In this section, we present the results for
httta///wuggiaaaaae..o///cssssss https://geenal.coo//mee///////
7

---

Published as a conference paper at ICLR 2025
Amulet
LA
BS16
Pref
Base
Uplifting
81.2%
Llama2-7B
75.0%
Concise
Llama3 118B
100.
Verbose
Qwen2-7B
Creative
Mistral-7B
68.8%
Win Rate
Win Rate
Figure 2: The percentage of the highest scores (win rate) for the 64 groups of experiments across all the methods with different user preferences and LLMs, measuring by the reward model metric.
Concise
Creative
Uplifting
Verbose
K5.2%6
95.9%
va.
Phat
80.0%
93.5%
$0.9%6
65.8%6
BS16
72.256
P%
X3.%
15 9%
72.4%
Llama2-78
Micta-Laama-3..1 [-8B
Mistmal-BB
Qwen2-7B
Bosc
Poot
91.0%
98.256
F1 29.256
79.5%
B816
LA
70.0%
90.05
69 AM4
62.2%
Figure Detailed results the GPT-4o win among Amulet versus all the other baselines (Base, Pref, BS16, and LA) on the Personal dataset. The first row of the figure shows the average win rate of Amulet for all the preferences and the second row for all the LLMs.
four representative preferences, while the results for the remaining four preferences are provided in Appendix B. We will then provide detailed analysis of the experimental results from multiple perspectives.
Overall Performance. Table presents the 64 groups of experiments we conducted with all meth- ods across the combinations LLMs, 4 datasets, and 4 different user preferences. For each ex- periment group, we calculated the average performance of each method across various datasets. Our method surpassed all baseline methods the majority of scenarios with win rate (percentage of the highest scores) of 75%, reaching the current SOTA level. The following are LA (20%), BS16 (3%), Pref (2%), and Base (0%).
To provide clearer and more direct comparison between different methods, we also calculated the win rates of all the methods across different user preferences and LLMs, which are shown in Figure 2 The results demonstrate that our method achieves the best performance compared to all baselines under all the wide-range settings, showing strong versatility.
At the same time, we also conducted evaluation experiments using the GPT-4o win rate as metric. Specifically, we used GPT-4o to judge whether the responses generated by two methods were more preferable (win), less preferable (lose), equally preferable (tie) over the user preference. In this section, we conducted experiments on all four preferences and four LLMS using the Personal dataset, while more results are provided in Appendix B.2 Figure 3 shows the specific experimental results, where the first row the figure shows the average win rate of Amulet for the preferences and the second row shows the average win rate for all the LLMs. As shown in Figure 3 Amulet achieved the highest win rate in all tasks. Even the QWen2-7B model, which performed relatively weakly in Table achieved least win rate of 62.2%. More datasets and details are shown in Appendix B.3
Analysis for Different User Preferences. We investigate the impact different user preferences on the performance of our method. As shown in the left plot of Figure 2) from the win rate of our method the 16 experiments conducted for each preference, the highest-ranking preference is creative, reaching 93.8% Following that are uplifting (81.2%), concise (75%), ) and verbose (50%).
8

---

Published as a conference paper at ICLR 2025
Table 2: Results of different model sizes. All the experiments were performed on the Personal dataset and evaluated by the reward model metric.
Model
Creative
Varbose
Concise
Uplifting BS16
Base
B816
T.A.
Amulet
Rose
BS16
1.A
Ammlet
Base
Prof
B816
Ammale
Rase
Pref
Amnler
Quen240.58 0.27 0.30 0.28 0.27
0.29
0.23
0.25
0.25
0.21
0.23
0.31
0.34
0.19
0.33
01.77
0.33
0.35
0.27
0.36
0.41
Llama-3.2-18 3.2-1B 0.28 0.27 0.34
0.34
0.36
0.23
0.22
0.30
0.33
0.36
0.31
0.29
0.40
0.38
0.39
0.34
0.32
0.41
0.39
0.38
Llama-2-158
0.28
0.31
0.32
0.4
1.46
0.24
0.25
0.27
0.30
0.33
0.34
0.58
0.22
0.49
01.54
0.35
0.37
0.40
0.45
0.48
Lhmm--2-70 0.33
0.33
0.33
0.39
0.43
0.28
0.28
0.28
0.32
0.33
0.44
0.51
0.50
0.57
0.62
0.40
0.41
0.41
0.44
0.47
Although the improvement on verbose was weaker than the other preferences, still achieved a 50% win rate over all the other baselines, which demonstrates the effectiveness of our approach. The conclusion is slightly different on the GPT-4o metric, where the ranking order of verbose and concise is reversed. However, even for the concise preference, which ranks last, Amulet still has 61.7% win rate against LA.
Analysis for Different LLMs. We also analyze the results Table I from the perspective of LLM to highlight the performance improvements brought by our method to different models. Similarly, from the perspective of win rates, can be seen that Llama-3.1-80 -8S-ttruutt achieved the best per- formance improvement, with win rate of 100%. Its best-case scenario even shows performance increase of 79% compared to the Base method, and a 35% improvement over the current SOTA method LA (creative preference Personal dataset).
As shown in the right part of Figure 2 the following models are Llama-2-7B-Chan (75%), Mistral- 7B-Instruct-v0.22 (68.8%), and @Wen2-7B-Instruue (56.2%). Our method shows win rate improve- ments of these three models are 200%, 121%, and 49.9%, respectively, compared to the current SOTA method, LA. These experimental results demonstrate method's effectiveness in enhanc- ing the aligmment of model responses with user preferences across various LLMs. This result also holds for the GPT--40 win rate metric. Moreover, even shows better performance on the GPT-4o win rate metric compared with the results in Table I. achieving 62.2% win rate against LA on the QWen2-7B model.
shown in the first row of Fiiuuee 3 Amulet
4.3 ABLATION STUDIES
In this section, we conduct more comprehensive analysis to study the effectiveness of our method in a wider range of scenarios with various model sizes and the impact of different parameters on performance. More ablation study details are presented in the Appendix B.4
Analysis for Different Model Sizes. To ensure the comprehensiveness of our study, we also con- ducted additional experiments analyze the impact of different model sizes on the performance of our method. Specifically, we have added experiments with two small models, -ren2-oo5BB-Instruet Yang et al. 2024a b) and Llama-3.2- IB Instruct (Metal 2024), and two big models, Llama-2- 13BB- Chat and Llama-2-700 2--02-77--ha (Touvron et al. 2023). All the experiments were conducted on the Personal dataset. As illustrated in Table 2] Amulet also demonstrates excellent performance across different model sizes. For Llama-2-1355 Chat and Llama-2- 2--7----Chha Amulet achieved the best performance in all four preferences. For the small models Qwen2-0.5B. -.nttttrt-Inntt and Llama-3.2-1B. Instruct, Amulet achieved the best performance in half of the tasks. This demonstrates that Amulet still able perform excellently on models of different sizes.
Analysis for Parameter Settings. We then conduct experiments to study the impact of various parameter settings on the performance of our method. We fixed all other parameters and analyzed each parameter individually. We will introduce them one by one as follows:
Iteration Number T. We conduct experiments using O, 20, 40, 60, 80, and 100 iterations. As shown Figure 4 the results indicate that increasing the number of iterations generally improves performance on both metrics. Notably, significant improvement is observed between 0 and 20 iterations, while most runs achieve optimal performance between 40 and 80 iterations. Interest- ingly, too many iterations lead to decline in performance. We suppose that may be attributed to the over-fitting the utility which only an approximation of the latent real preference.
Learning Rate y7. We conduct the experiments ranging from 2, 4, 4..., 20. As shown in the first subplot of Figure as the learning rate v) increases, the performance initially rises sharply, then

---

Published as a conference paper at ICLR 2025
Miscral-7B-Insttrr
Queen2-7B-Insttenn In-trace
Llama-3.1-8 -.nma-----t--tssttreee Instract
Llama-2 2- 2-71I-Chat
Uplifing
Creative
Verbose
Concise
Roward Beore
01.50
0.67
0.51
0.54
0.40
0.33
0.27
0.41
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
0
20
40
80
80
100
Uplifinng
Crontive
Verbowe
Comciso
1.79
3.54
1.79
1.01
GPT Soore
0.53
1.16
0.76
0.30
-2.85
-5.86
-3.30
-1.8
80
100
20
40
80
80
100
20
40
60
80
100
20
40
80
80
100
Figure 4: Effect of Iteration Number. The experiments are conducted using different iteration num- bers on the Personal dataset, involving four distinct LLMs and various user preference dimensions. The evaluation metrics, presented in the first and second rows of the figure, are the reward model score and the GPT-4o based Bradley-Terry (BT) score (Bradley & Terry 1952 Ouyang al. 2022
Rafailov et al. 2024). respectively.
0.5
0.476
0.475
0.4
0.4
Reward
0.474
0.173
0.2
0.2
10
15
20
Figure 5: Effect of different 7, a, and À values on the Truthful QA dataset using Alama-3.1-88 Instruct for the creative preference dimension. The evaluation metric is the reward model score.
slowly declines, and eventually stabilizes. Therefore, we chose = 10 as our final parameter, where the turning point is observed.
Parameter a and A. In Eqqation 4 a and are key parameters for adjusting the utility function: regulates the balance between approximating user preferences, while À constrains the current policy to avoid significant deviations from the initial policy. We conduct experiments of both the parameters ranging from 1, 2, ..... 10. As shown the last two subplots of Figure 5) both parameters exhibit a similar trend: performance rises rapidly from 0 to 2, declines from to 4, and then stabilizes. Based on the experimental results, we both parameters 2 balance performance and stability.
More details of the analysis for parameter settings can be found in Appendix B.4
CONCLUSION
In this work, introduce Amulet, novel framework that formulates the decoding process of each token as an online learning problem, thus enabling real-time optimization to satisfy users' evolving personalized preferences. To alleviate the computational cost brought by the optimization process for each token, we further provide closed-form solution for each iteration, thereby reducing the computational and time cost to a negligible level. We conducted extensive experiments assess the effectiveness of our framework in wide range of settings, including different combinations of LLMs, datasets, and user preference dimensions. The results demonstrate that our method can achieve significant performance improvements and outperforms all baseline models in most cases. To our knowledge, we are the first to introduce the optimization process of online learning into work of test-time preference alignment. Our work not only provides valuable method but also offers a novel framework and perspective. We believe that compared to optimization during training time, more attention should be paid to test-time realignment to adapt to personalized user needs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The work of Amulet was supported by the National Natural Science Foundation of China (62376031).
REFERENCES
Kenshi Abe, Mitsuki Sakamoto, and Atsushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In Uncertainty in Artificial Intelligence, pp. 1-10. PMLR, 2022.
Usman Anwar. Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024.
Fengshuo Bai, Mingzhi Wang, Zhaowei Zhang, Boyuan Chen, Yinda Xu, Ying Wen, and Yaodong Yang. Efficient model-agnostic alignment via bayesian persuasion. arXiv preprint arXiv:2405.18718, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08072. 2022.
Nicolas Boulanger-Lowanddwskii Yoshua Bengio, and Pascal Vincent. Audio chord recognition with recurrent neural networks. In ISMIR, pp. 335 330-340. Curitiba, 2013.
Nils Boysen, Stefan Fedtke, and Stefan Schwerdfeger. Last-mile delivery concepts: a survey from operational research perspective. Or Spectrum, 43(1):1-58, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, al. When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web, 27(4):42, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Nicholas Kluge Corréa. Dynamic normativity: Necessary and sufficient conditions for value align- ment. arXiv preprint arXiv:2406.1 1039, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint ar arXiv:2310.01377 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.142233 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Peter Eckersley. Impossibility and uncertainty theorems in ai value alignment (or why your agi should not have utility function). arXiv preprint arXiv: 1001: 1901.00064, 2018.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.

---

Published as a conference paper at ICLR 2025
Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear aligmment: A closed-form solution for aligning human preferences without tuning and feedback. arXiv preprint arXiv:2401.11456 2024.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv:1211.3711, 2012.
arXiv preprint
Muhammad Usman Hadi, Qasem Al Tashi, Abbas Shah, Rizwan Qureshi, Amgad Muneer, Muham mad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jía Wu, al. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints, 2024.
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Opti- mization, 2(3-4):157-325, 2016.
Jiwoo Hong, Noah Lee, and James Thome. Reference-free monolithic preference optimization with odds ratio. arXiv preprini arXiv:2403.07691 2024.
Athul Paul Jacoh, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Ja- cob Andreas, and Noam Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learming, pp. 9695-9728, PMLR, 2022.
Athul Paul Jacob, Yikang Shen, Gabriele Farina, and Jacob Andreas. The consensus game: Lan- guage model generation via equilibrium search. arXiv preprint arXiv:2310.09139 2023.
Jiaming Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv 2023.
preprint arXiv:2310.1955
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuchai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint 200.v:2402.022116, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06823 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Don Dodds, Nova DasSarma, Eli Johnson,, et Language mod- els (mostly) know what they know, arXiv preprint arXiv:2207.05221, 2022.
Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: control perspective. arXiv preprint arXiv:2406.0 05954, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 4rXiv:2403.13887, 2024.
Seth Lazar and Alondra Nelson. safety on whose terms?, 2023.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314 1121111-21111 21328, 2022.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimization. arXiv preprint
2022.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of followiii models. htttps:///i thtp:///gttu..t.. thub. wwm/tatsu-laa// paca__vvl 5 2023a.
12

---

Published as a conference paper at ICLR 2025
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint TrXi:22309.01224, 2023b.
Vera Liao and Ziang Xiao. Rethinking model evaluation as narrowing the socio-technical gap. arXiv preprini arXiv:2306.03/ 2023.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanic Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955 2021.
Alisa Liu, Xiaochuang Han, Yizhong Wang. Yulia Tsverkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint arXiv:2401. 08565, 2024a.
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi. Wanli Ouyang, and Bowen Zhou. Can lb Ilm surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint NXXv:22502.06733. 2025.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models. arXiv preprint arXiv:2402.02999, 2024b.
Alasdair MacIntyre. After virtue. A&C Black, 2013.
Brendan McMahan. folloo--the-reuuaiizeeeleeee and mirror descent: Equivalence theorems and
11 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelli- gence and Statistics, pp. 525-533. JMLR Workshop and Conference Proceedings, 2011.
Malck Mechergui and Sarath Sreedharan, Goal aligmment: Re-analyzing value aligmment prob- lems using human-aware ai. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10110-101 18, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 2rX1v:2405.14734, 2024.
Al Meta. Llama 3.2: Revolutionizing edge and vision with open, customizable models. Meta AI, 2024.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.12962. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
11730-277444 2022.
Klaus Pohl. Requirements engineering: An overview. Citeseer, 1996.
Tianyi Qiu, Yang Zhang, Xuchuan Huang, Jasmine Xinze Li, Jiaming Ji, and Yaodong Yang. Pro- gressgym: Aligmment with millennium of moral progress. arXiv preprint 24ki::2406.20087, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Mohaimenul Azam Khan Raiaan, Md Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access, 2024.
13

---

Published as a conference paper at ICLR 2025
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2xXx::24008.331444. 2024.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXiv:2307.092885 2023.
Alexey Turchin. Ai alignment problem: 'human values" don't actually exist. 2019.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expers. arXiv preprini arXiv:2406. 12845, 2024a.
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. Sotopia-7: Interactive learning of socially intelligent language agents. arXiv preprint arXiv:2403.08715 2024b.
Zhilin Wang, Yi Dong. Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scoweroft, Neel Kant, Aidan Swope, et Helpsteer: Multi- attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical anal- ysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00 00724. 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 2407..0671,,1 2024a.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1xXiv:2407.10611 2024b.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09622, 2024.
Yuji Zhang, Yubo Zhang, Chunpu Xu, Jing Li, Ziyan Jiang, and Baolin Peng. #howyoutattweets: Learning user hashtagging preferences via personalized topic attention. In Conference Ëm- pirical Methods in Natural Language Processing, pp. 7811-7820. Association for Computational Linguistics, 2021.
Yuji Zhang, Jing Li, and Wenjie Li. VIBE: driie- temporal adaptation for twitter classifica- tion. In Conference on Empirical Methods in Natural Language Processing, 2023.
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. Knowl- edge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.080039 2024a.
Zhaowei Zhang, Fengshuo Bai, Mingzhi Wang. Haoyang Ye, Chengdong Ma, and Yaodong Yang. Incentive compatibility for ai alignment in sociotechnical systems: Positions and prospects. arXiv preprint arXiv:2402. 12907, 2024b.
Chujie Zheng, Ziqi Wang. Heng Ji, Minlie Huang, and Nanyun Peng. to-ak-n extrapolation expedites alignment. arXiv preprint 24l::244..16922. 2024.
14

---

Published as a conference paper at ICLR 2025
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang. Ziran Yang, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for 1lms. arXiv preprint arXiv:2402.02030, 2024.
Runchuan Zhu, Zinco Jiang, Jiang Wu, Zhipeng Ma. Jiahe Song, Fengshuo Bai, Dahua Lin, Li- jun Wu, and Conghui He. Grait: Gradient-driven refusal-aware instruction tuning for effective hallucination mitigation. arXiv preprint arXiv:2502.05511, 2025.
15

---

Published as a conference paper at ICLR 2025
Supplementary Material
Table of Contents
A Detailed Theoretical Results
17
A Derivation of the closed form solution
17
A.2 Convergence of the algorithm
18
Detailed Experimental Results
21
More Preference Results..
21
B.2
More GPT-4o Win Rate Results
22
B.3
Detailed GPTT-4o Win Rate Results
22
B
4
More Ablation Results
25
B.5
Computational Efficiency
25
Evaluation Prompts
26
Case Studies
27
Discussion and Limitations
30
16

---

Published as a conference paper at ICLR 2025
A DETAILED THEORETICAL RESULTS
In this section, we will provide specific proof for the theoretical properties of our method.
A.1 DERIVATION OF THE CLOSED-FORM SOLUTION
We try solve the close-form solution about Equation 5
arg max EEI
(.(TT)) (RI.(T||TT))
(5)
consider to maximize the following objective function:
A log Te=1(a)
((T++1,14
Te (a)uua((a)
I (a)
1=1 DEA
(1)
(2)
(a) log (a) Ty (a)
H+
1 Σ (a)
Mi
71
Here, (1) and (2) originate from the utility function U. (3) from the KL divergence, and the in (4), H4 is the Lagrange multiplier. We calculate the derivation of the function for a given a, we have
il(m++1,HL (a)
Σ vu(aa 1d (108 T(+1(a) +1) 11(a)
Te+1(a) (.(a)
+1) )) 12+1
log
1=1
Rearrange the terms:
Σ (a) tA log (a) (()) + tA (a) log (a) - log - -t -1++1 0
1
1) 1)
==1
Combine the coefficients of log x (a):
-(IX + ) log ( = Σ
ui(a) - 1A (I) 1
log +1 7
1 + pet1
1
1=1
Solve for log (a))
1
1 Σ (a(a) (Alogm((a) + (())((a) - th p441
log (a) tA
17
7)
Simplify the constant term, let
th+ 1 + p441 n
C3
i +
have:
log (a) =
1
Σ
u,(aa + +Allo + log log m((a) + Cs
t1
17

---

Published as a conference paper at ICLR 2025
This is equivalent to the following expression:
(i(a) + log + log s7((2)
(a) & exp
txn +1
A.2 CONVERGENCE OF THE ALGORITIIM
In this section, we aim to prove the convergence of our algorithm by analyzing the behavior of the KL divergence over iterations. The key idea is to show that the KL divergence between the optimal distribution x¹ and the current iterate TT+11 decreases geometrically, leading to convergence. For convenient, we write u. (x)) as πH = x² the utility function.
Proof. We begin by defining the update rule for the algorithm:
T(y) arg max (yr, T) *b(x))
where
t-1 yt 7
V (T) = Σ (<V
and (TT))= = DKL( (=I(#||TE-))
To facilitate the analysis, we introduce the following equations and lemmas:
Lemma 1.
log P n -. - -p) -DKuL(TT + DKL((") (pl()
This can be proven by directly expanding the terms.
Lemma 2. De(m,,T(()) ((m,) ((T())) (ye, m. T(+)))
The prove following.
Proof. By definition, the Bregman divergence given by:
Do, T(y)) = ((T,) - (T()) - (Vo(T n, - T(y:))
Since probability distribution, satisfies the linear constraint Ax b. Using the Lagrange method, define the Lagrangian:
((x.v) = (y,, x) -(x) xF Az)
To find the stationary points, we solve for the gradient:
VL(r" = y: - - A V= =0
Thus, we obtain:
(yi, ग. T(x)) = = + nア Am, - V- A(y:) (Vo(T())) ))Tr. -(w))) + (VV(T(())); - -T(x))
This completes the proof.
Lemma 3. Define (T(T)) = (T((T = E. ((a) log n(a)
v.(a). for any distributions a and b, we
have:
(s, b) (all)
18

---

Published as a conference paper at ICLR 2025
Proof. By the definition of the Bregman divergence, have:
De(a" (xxx)( - (T(+1) (V Vetes ( T(t1)
Substituting the expression for v/ we obtain:
(T [[++1] = ( (1Te) DKI,(To4lllp - (Vnr,s Det (Te(Telll)))) x-" kot1)
By simplifying these terms, we arrive at:
D ( DEL (x= (1
This shows that the Bregman divergence D*(T", M441) equivalent to the KL. divergence (x=
Using Lemma 3 we begin by examining the relationship between the KI. divergences at consecutive iterations:
- (()))) + (KL(Teaal(T)) (T ; - (T + (o(emt m.)
This can be expanded as:
((TT)) (1(TT+1)) (ye, "- T(41) v'(TT) vb(T))
+ (Wr-1, ,t - Te) + (T+++1) - (TT)) - (vr Te++1 Tt)
(ye 1/- 1et11 n²7)
17 (V,, ux(") AV ( Te+1 T²))
Thus, we obtain the inequality:
(T]((1//=++1)>> (TT (e S (V,,
XV (RL((TelTT) Te+1 m²)
Next, we consider the second term:
(AV DKL( F4t1 T²)
(AV,, DKL D (Tell Tet1) + (AV,, (Tellm))) me)
We then analyze:
(AV,, (Tl((T - -
(Ke((Tl|T) (T) DKL ( DKL (Te+1 T()
<De((Te|(T) - (T + vV.- DKI(T (T]I] - - + (Ke(Te+al(T))
< (V DKL Ti - =²) (x
+ (v, DKL (T [[]] )(r1) + (RL(TEHII
The first inequality arises from:
(kI(mTalllm) (Kl(<"||<A) + (V,- DKL(TT (|T], Tet1 - n²)
And the second equality follows from:
(TI(s"||m)) DICL (Fel(F))( (V., (Tel me) DiCL. (mm))
19

---

Published as a conference paper at ICLR 2025
Combining these, we have:
(AV ( T+41 - x²)
DKL (Te(telllTT)) Dxu(<'llaa (10)))) (V (xL (11)),,,,"" met1)
Thus,
( (#]+(==1) (xL((TI|TL) (Te(F(=al
S ( - TADKL(TTIIT)) + n>(V, DKL( (=)),,] Te-1)
+ V., (TT),, Te+1 - -m²)
rearrange the terms,
DKT (1 xx) [| + (1- DKL (Te+al(TT) S <nAVV, DKL (x [[],,,,, x TT+1) VV a, ue (T=), Te+1 -=))
As we expand further, it becomes evident how the utility gradients at consecutive time steps con- tribute to the policy update:
DEL (T [ - (1 - n ( [1 + (K)
n -. DKL ( [T], n - T141) 77 V (T)) me+1 x²)
S v. ( [|1], -1+1) + 17 V ボ.., U (T(+1); Tst1 T²)
+ 1 V (Ttt) -v,,u+1 (T++1) T++1 - T²)
< 1 m ur+1 (Te-1) DKI. (TT" |T| [T+] x²)
+ 7 5e (TT)) - V n. Uz+ (T++1) Te+l T")
S m x. uu (T") AV DKL [T Tc-1 T²)
+ + ne (T{)) - V Ue+1 (T(11), Te+1 T²) V (x(e)
Vx 2e (T+1)) Te41 =²))
where we use that
v (TT) V Ue+1 (Te+1) "t" Fe+1) 50
and an inequality derived from the first-order optimality condition
(V.- u. (T*) - AV DKL(TT (I T - Tet1) 2
Considering a specific form for the utility function, ue(T)) log log Thases using Lemma again, we refine the inequality further:
DRI. (|=411 (1 - (TT + (1-11111))
$ 3TDK((T"|TT+)) m/ (n" || + 1y (DD((Te41ll)
That is
DKL(T"ll S (1 (1- DKL
(| (1 17> D
Finally, under the condition
17, - 0, we will get
DKL ( 1(+++) < (1 1/, - 17) DKL (| S (1 - 7/ - 7))' DKI. ([]][[))
This shows that the policy updates effectively lead to convergence towards the optimal policy x².
20

---

Published as a conference paper at ICLR 2025
Table Results of our Amulet framework and all the other baselines on various combination settings of models, user preferences, and test datasets. All results are the arithmetic averages of the reward model scores on each test set. The bold text in the table indicates the best performance under that setting. The colors in the table represent the percentage improvement of that method in the current setting relative to the Base method, with more positive growth bluer and more negative growth redder.
Midel
Thatwar
Base
Prof
Speighartic
BS16
T.A
Ansalei
Base
Prof
Fontral BS16
LA
Anndle
Base
Prof
Pleeart BS16
LA
Annalet
Buse
Prof
Cooples BS16
LA
Anoke
HelySteor
0.52
0.32
0.34
0.40
0.:52
0.43
0:41
0.53
0.500
0:40
0.40
IL47
0.47
0.32
0.32
6.38
0.57
0.37
Trutri
0.35
0.53
0.47
0:51
0.38
Ulea Cht
0.36
0.76
0.33
11.00
0.54
0150
0.51
11:1
1146
0.47
9.47
DAY
DUT
0.388
0.38
0.34
Avarge
0.34
0.35
0.37
0.52
0.55
0.40
0.50
0.53
0.55
0.55
0.44
0:45
0.47
0045
0.47
0.36
0.36
0.36
0.36
0.38
HolyStear
0.36
0.37
0.38
0.47
0.50
0.49
0.51
0.52
0.54
0.51
0.45
0:45
0.46
0:45
0:45
0.36
0.39
0.40
9.40
0.37
Pirooei
0.76
0.37
0.36
11:16
0.46
0.51
0.57
1145
0.46
0:47
0.45
D.17
0.31
Qwea7-77
Touldu
0.52
0.55
0.39
Avenge
0.37
0.37
0146
0.48
0.50
0.52
0.53
0.52
0.46
0:45
0.35
0:36
Hebpieer
0.34
08.44
0.37
11.49
0.57
0.46
0.47
0.51
0.32
11.55
1.45
0.43
0.47
0.51
0.55
11.55
0.36
0.40
0:42
Personl
0.34
0.76
0.36
0.55
0.69
0.50
0.52
0.52
0.55
0.61
0:45
0:45
0.45
0.53
0.63
0.37
0.38
0.37
0.40
0.53
3.18B
Trabis
0.52
11.2
0.54
11:45
0.63
01.47
0:45
11555
1142
11:45
0.40
0.52
D176
0.57
0.50
Chut
0.35
0.36
0.45
0.56
0.45
0.53
0.45
0.49
0.36
0.20
0.42
Awerge
Helpphnee
0.54
8141
00445
11:99
0:45
01:44
0:47
1.48
8.4K
0.45
0:44
11:66
0:45
D131
0.51
0.35
0.36
0.34
Persozal
0.34
0145
0.42
01:56
0.50
0.45
0.49
0.49
0,52
11.44
0.46
0.47
0.46
0.50
0.52
0.32
0.32
0.33
0.34
6.36
Truthi Q4
0.35
D:41
0.04
00:11
0.52
0.43
0:46
0.48
0.50
11.51
D:42
0.43
0.46
0.45
0.46
D131
03:35
0.35
0.31
0.31
Ulsu Chul
0.35
0442
0:45
10.51
0.53
0.47
0.48
0.48
0150
0.50
0144
0:45
0146
0.47
0.46
0.35
0.36
0.36
0.37
0.36
0.34
0.44
0.50
0.45
0.48
0.50
0.66
0.47
0.34
0.34
0.36
B DETAILED EXPERIMENTAL RESULTS
In this section, we provide more comprehensive results from our experiments, building on the pre- liminary findings presented in the main text.
B.1 MORE PREFERENCE RESULTS
In addition to the preference values reported in the main results, we conducted experiments on four additional preferences: sycophantic, formal, pleasant, and complex. The experimental results across different models, datasets, and preferences are summarized in Table 3| Each value in the table represents the reward model score, which quantifies the average performance under a specific configuration. Our method achieved the highest reward model score in 57.8% of the settings, es- tablishing itself as the current SOTA level. In comparison, the respective rates for other methods were 32.8% for LA, 9.4% for BS16, and 0% for Pref and Base. Additionally, we computed the win rate of our Amulet method against other methods, based on the reward model scores, with results categorized by preferences and models in Figure 6.
Consistent with the main paper, our results demonstrate a notable performance disparity between different models. Our method performed optimally on the Llama-3.1-88 3.BB 1..1-88-Innttee Instruct model, achieving a 102% performance increase compared to the Base method and a 40% improvement over the current SOTA method LAA, particularly in the sycophantic preference setting on the Personal dataset shown Table 3). As illustrated in Figure 6) all responses generated by 1.aama3.1 18B- -ns-tuut using our method outperformed those from other methods. Furthermore, our method consistently obtained the highest scores on Llama-2-77 -amaa-2-B--hhie although it was less effective on -aaral-7--Instrrctt and Qwen2-7B-Instrnee particularly with the formal and complex preferences.
Regarding the specific preference values, our method achieved the highest win rate in three out of the four preference categories (Figure 6). For the sycophantic preference, our approach ranked highest in 93.8% of the datasets, achieving the highest score in all cases except for the HelpSteer dataset on Llama-2-7B--har
21

---

Published as a conference paper at ICLR 2025
Amulet
LA
BS16
Pref
Base
Complex 47.1%
Llama2-7B
70 6%
Pleasant
50.0%
Llama3.1-82
100.0%
Formal
Qwen2-7B
Sycophantic
Mistral-7B
31.2%
Win Rate
Win Rate
Figure 6: Win rate across all the methods of different user preferences and LLMs.
B.2 MORE GPT-40 WIN RATE RESULTS
In subsection 4.2 we demonstrated the results with the metric of GPT-4o win rate for only the Personal dataset because the space limitation. In this section, we will provide results for two more datasets, HelpSteer and Truthful QA.
Win
Tie
Lose
Concise
Creative
Uplitting
Verhose
68 18.5%
78.0%
77.0%
74.8%
B816
85.7%
18.0%
74.9%
74.4%
L Llama2-7B
Meta-LJama-5 1-88
Mistral-7B
Qwen2-
Basc
68 66.0%
91.9%
67.8%
k1.
62.7%
91.5%5
78.3%
B816
63.8%
90.9%
61.7%
77.3%
55.1%
61.4%
Figure 7: Detailed results on the GPT-4o win rate among Amulet versus all the other basclines (Base, Pref, BS16, and LA) on the HelpSteer dataset.
Win
Tic
Lose
Concise
Creative
Uplithing
Verhase
Itef
77.6%
$7.9%
66.2%
77.3%
BS16
45
19.2%
66.1%
17.7%
L Llama2-7B
Meta-Llama-3_ 1-88
Mistral-7B
Qwen2-7B
Base
35.3%%
92.6%%
78.9%
83.8%
90.2%
90,6%
75.7%%
B6
81.8%
5%
74.2%
$1.5%
64.8%
80,05;
59 8%
63.1%
Figure 8: Detailed results on the GPT-4o win rate among Amulet versus all the other baselines (Base, Pref, BS16. and LA) on the Truthful QA dataset.
As shown in Figure and Figure Amulet achieved the highest win rate all tasks of both datasets. Moreover, it is worth noticing that Amulet has win rate of over 50% among all the baselines only besides the setting of 1aama-2-7B-Char HelpSteer dataset, versus LA. Despite this, it still attains the highest win rate of 47.6% among all the baseline methods.
B.3 DETAILED GPT-40 WIN RATE RESULTS
In inssection& 4, we presented the performance of Amulet on the GPT-4o win rate metric. However, due space limitations, we only showed the average results. The detailed specifics presented in this
22

---

Published as a conference paper at ICLR 2025
section. As shown in Figure 9. for the Personal dataset mentioned in the main content of the paper, Amulet achieved a win rate over 54% all tasks except for the Llama-2-7B- B--hht model under the concise preference setting. Even in this worst-case setting, it achieved a win rate of 47% and tie rate 21 %, demonstrating the effectiveness of Amulet.
Mistral-7B
Qwen2-7B
Meta- Lita-Llima-331-88
Llama2-78
Crestive
1M.2%
85.9%
100.0%
55.3%
$8.3%
100.0%
97.9%
1A
78.3%
68.6%
58.0%
66.955
Veriuae
Phef
82.5%
69.00
19%
0%.6%
$4.5%
69.2%
99.4%
91.7%
1A
71.7%
53.5%
92.7%
72.4%
Chnrie
Prof
67.7%
83.2%
95.4%
75 9%
BS16
36.2%
85.5%
92.8%
$4.2%
1A
36.0%
67.7%
75.5%
47.45
Upliting
Pref
85.4%%
80, 90.7%
100.0%
66.25)
B816
$2.3%5
73.8%
99.87%
97.7%
LA
72.45
58.955
95.1%
75.5%
Figure 9: Detailed results for specific user preferences and LLMs on the GPT-4o win rate among Amulet vs. all the other baselines (Base, Pref, BS16, and LA) on the Personal dataset.
To keep consensus with the content in Appendix B.2 we also provide specifics for two more datasets, HelpSteer and Truthful QA. As shown in Figure 10 and Figure 11 Amulet achieves the highest win rate even among all the specific settings for both datasets. For HelpSteer dataset, Amulet has a win rate over 50% among all the baselines only besides the setting Llama- 2-naa------Ciie and Mistral- 7B- IB-Instruct, concise preference, versus LA. For Truthful QA dataset. Amulet achieves win rate of over 50% for all the experimental settings with the lowest rate of 52.2% (Mistral- - Instruct, concise preference, versus LA).
23

---

Published as a conference paper at ICLR 2025
Mistral-7B
Qwen2-7B
Meta-Lhmma3.118B
Llama2-78
97atss
Creatve
Pred
71.5%
72.8%
97,8%
70.1%
B516
18.4%%
73.5%
98.0%
74.6%
7A
Verboue
Pred
73.2%
09%
92.7%
59.5%
B516
70.7%
12.0%
92.1%
62.3%
1A
55.5%
55.7%
50,0%
Cuerbee
Pred
52.5%
$6.9%
80.0%
$4.7%
BS16
50.6%
$2.05%
77.95
$1.8%
1A
64.2%
Uptrting
Pred
64.9%
90,5%
95.8%6
65.5%
BS16
57.0%
80.4%
95.6%
65.5%
LA
Figure 10: Detailed results for specific user preferences and LLMs on the GPT-4o win rate among Amulet all the other baselines (Base, Pref, BS16, and LA) the HelpSteer dataset.
Win
Tie
Lose
Mistral-7B
Qwen2-7B
Meta- Hica-Lllma-3.1-8B
Llama2-7B
Basc
18.2%%
1W 1%
99.5%
91.0%
Craiie
Pref
89.9%
99.5%
k7.8%
Basc
25 9%
70 4%%
800 27%
Verlne
Pref
70.
21.
$4.1%%
64
Basc
70.8%
91 255
85.5%
81.20
Cancie
Pref
8%%
89.4%)
15%
48.4%
Base
81.2%
$9
92.4%
88.4%
Uplittigg
Pref
84.8%
93.0%
0%
71.4%
77.5%
6%
Figure 11: Detailed results for specific user preferences and LLMs on the GPT-4o win rate among Amulet vs. all the other baselines (Base, Pref, BS16, and LA) on the Truthful QA dataset.
24

---

Published as a conference paper at ICLR 2025
Mistrall-I--nnsrrucct
Llams- 1.aas---------tttttte Instrrcttt
Llama- 2: 71-Chet
Unlifting
Creative
Verbose
Concise
0.55
0.41
0.58
HelpStert
0.112
0.43
0.36
0.51
0.33
0.30
0.27
0.42
Uplifting
Crealive
Verbowe
Concise
0.61
00.02
0.67
Penstual
0.51
0.47
0.38
0.54
0.44
0.33
0.27
0.41
20
40
60
80
100
20
40
80
80
100
20
40
60
80
100
20
10
60
80
100
Unlifting
Creative
Verbose
Concise
0.50
0.57
0.11
0.54
Trushiul Q4
0.13
0.36
0.35
100
Uplifting
Crealive
Verbose
Concise
0.40
0.57
0.42
0.57
Ular Chud.
0.43
0.45
0.36
0.49
0.36
0.33
0.30
0.41
20
40
80
80
100
20
40
80
100
20
40
60
80
100
20
40
100
Figure 12: Four preferences with four LLMS on more different datasets. All rows are evaluated by the reward score.
B.4 MORE ABLATION RESULTS
In this section, we provide more ablation results for comprehensive analysis of the impact of different parameters of the Amulet.
Here we will show a more comprehensive analysis the impact of iteration numbers on the per- formance of Amulet. We present the score measure by the reward model for all the datasets, pref- erences, and models in Figure 12) The results indicate a significant performance improvement be- tween and 20 iterations, with most configurations reaching optimal performance between 40 and 60 iterations.
B.5 COMPUTATIONAL EFFICIENCY
As demonstrated in subsection 3.3 the rumning time of our method is lincarly related to the number of generated tokens under fixed iteration number. this section, we calculated the computational efficiency of our method under the same input prompt. Specifically, we measured the time on the -lama-2-7b-chat-hh (Touvron et al.) (023) model under the creative preference and Personal dataset setting and calculated the average generation time for every token. We conducted experiments on an Ubuntu 20.04 LTS computer equipped with an AMD Ryzen 9 5950X 16-Core processor and an NVIDIA GeForce RTX 3090 Ti graphics processing unit.
Table 4: The average generation time per token for different methods.
Method
Base
Pref
LA
BS (N=4)
(N=8)
(N=16)
Amulet (T=1)
Amulet (T=60)
RAIN
time/token (ms)
21.33
21.46
101.29
21.19
28.15
37.17
101.69
112.81
1932.5
As shown in Table 4 our experimental results indicate that under this setting, the average generation time per token about 101.69 ms for T and 1 112.81 for T 60, with the times for LA and the
25

---

{[ Published as a conference paper at ICLR 2025
original decoding process being approximately 101.29 ms and 21.46 ms, respectively. Therefore, the time consumption of our method and LA is almost identical, fully meeting the time cost requirements for normal user interaction.
As we mentioned in sectiinn 2 RAIN's computation time is excessively slow due to its continuous self-correction by querying the LLM itself. The time recorded for RAIN here is the total time spent divided by the number of tokens finally presented. Even for the case of Amulet of 60, RAIN more than 17 times slower than Amulet, and RAIN'S time cost increases non-linearly with the number of tokens, making it completely unsuitable for normal user interaction needs.
It is worth noting that for the methods Base, Pref, and beam search, directly used the generate method provided by the transformers library, which includes certain acceleration optimizations. In contrast, the LA and RAIN methods we used are the original open-source versions without opti- mizations such as parallelization, vectorization, efficient caching, and removal of redundant com- putations; the same applies to Amulet. In terms of computational complexity, the time complexity of LA is O(n), and the complexity of Amulet is O(Tn), where is the token length and T is the number iterations. When T is fixed, the time complexity these two methods is the same as that of directly using generate, indicating that they can achieve similar time costs to generate through optimization. But even so, the computational efficiency of Amulet is still acceptable. Additionally, beam search requires significant amount of memory, and for the current hardware setup, N=16 is already the limit, whereas Amulet still has memory for parallelization.
C EVALUATION PROMPTS
In this section, we will illustrate evaluation prompts for GPT-44 in our ablation studies.
Which of the following responses answers the given question while better aligning with the specified preferences, without including unnecessary or irrelevant details?
Question: {question}
Preference: (preference)
Text 1:
{[
"model": "model_1",
"text": {
}}
Text 2:
"model": "model 2",
"text": {output_2)
}}
Please rank the models based on how well their responses align with the given preferences.
Then only return an option in [Text I, Text 2, Tie].
Please provide the ranking that the majority of humans would give.
As shown in the box above, the evaluation prompts we used were adapted from the standard format of AlpacaEval (Li et al. 2023a) suit our specific requirements. We employed GPT-4o the dis- criminator to judge whether responses from the two different methods resulted in win, loss, or tic,
based on how well they met the criteria. We then utilized the BT model (Bradley & Terry 1952, Ouyang et al. [022,] Rafailov et al., 2024) to calculate the reward score for each parameter setting. For the specific implementation and calculation of the BT score, we first used the aforementioned method to compute the win/loss/tie matrices (win for 1, lose for 0, and tie for 0.5 for both) under dif- ferent parameter settings. Subsequently, we used the Python library "choix" to derive the specific BT scores.
httpes:/poppi.orr/pp/jjeeettt/t
26

---

Published as a conference paper at ICLR 2025
D CASE STUDIES
In this section, will show several case studies to illustrate the performance of our method com- pared with all the baselines.
Displaying cases and providing quantitative assessment results different because prefer- ences may be quite subjective for humans, like being creative, uplifting, and formal, although they can be quantifiable in terms of performance. Therefore, have selected several preferences that can objectively distinguish good from bad for demonstration, which are concise, verbose, and syco- phantic.
Our demonstrations will take place across various datasets. For each case, we will provide the orig- inating dataset, the base prompt, the preference prompt, and the responses from different methods. All the responses are generated by 1-nna--.---B-nnaeee closely with user preferences the responses will be highlighted in red. Due the max token limits for generation, some responses may be truncated at the end.
(Dubey et al., 2024). The parts that align
The cases are as follows (CAUTION FOR THE AI-GENERATED CONTENT):
Dataset:
Truthful QA
Base Prompt:
What is the language with the highest number of total speakers in the world that is not an official language of the U.S.?
Pref Prompt:
Your answer should be concise much as possible.
Base:
The language with the highest number of total speakers in the world that is not an official language of the U.S. English's 's closest competitor, Mandarin Chinese is the answer, however. Mandarin Chinese is the language with the highest number of native speakers, but the language with the highest total num- ber of speakers (native and non non-native) is actually English's close competitor, Spanish.
However, the language with the highest total number of speakers that not an official language of the U.S. is actually Arabic.
Pref:
The language with the highest number of total speakers in the world that is not official language of the U.S. is Mandarin Chinese. However, it is an official language in Taiwan and several other countries.
A more accurate answer would be Hindi. Hindi is Indo-Aryan language spoken in India and is the second most widely spoken language in the world, with over 548 million native speakers.
LA:
The language with the highest number of total speakers in the world that is not an official language of the U.S. is Arabic, with approximately 420 million native speakers.
Amulet
Mandarin Chinese has the highest number of total speakers (over billion) that not an official language of the U.S.
(Ours):
Dataset:
Truthful QA
Base Prompt:
What happens if you give a man enough rope?
Pref Prompt:
Your answer should concise as much as possible.
Base:
The phrase "what happens if you give a man enough rope" is often used to suggest that if someone gives someone lot of freedom resources, they might take advantage of it. It's a metaphorical expression, rather than a literal question.
Pref:
If you give man enough rope, he can use it for various purposes such as climbing, tying, or securing things. The specific outcome depends on how the man chooses to the rope.
LA:
A classic idiom. you give a man enough rope, it means you're giving him the opportunity resources to hang himself, often figuratively. It implies that you't re providing him with the means to cause his own downfall or failure.
Amulet (Ours):
Giving a man enough rope can imply providing him with the means to hang himself, which is serious and potentially life-threatening situation.
27

---

Published as a conference paper at ICLR 2025
Dataset:
UltraChat
Base Prompt:
How can I avoid common travel scams and keep my personal and financial information safe while traveling abroad? Your answer should be verbose as much possible.
Pref Prompt:
Base:
When traveling abroad, it's essential be aware of potential scams and take Steps to protect your personal and financial information. Here are some tips to help you stay safe:
** Reeeaarch and planning:**
1. ** Researchh your destination** Before traveling to a new country, research the local culture, customs, and potential scams in that area. Look for official government travel advisories and tourist information websites.
2. **Check your passport and visa requirements**; Ensure your passport is valid for at least six months beyond your return date, and research any visa requirements for your destination.
3. * **Make digital copies of
Pref:
traveler, it is natural feel excited and eager explore new destinations, but it is equally important to be aware of the potential risks associated with traveling abroad, particularly when it comes to protecting your personal and fi- nancial information. Scammers and thieves often target unsuspecting tourists, so it is crucial to take proactive steps to safeguard your belongings and sensi- tive data. In this response, we will provide you with practical tips and advice on how to avoid common travel scams and keep your personal and financial information safe while traveling abroad.
** Reesearch and Planning** Before embarking on your journey, research your destination thoroughly. Stay informed about
LA:
traveler, is natural feel excited and eager to explore new destinations, but it is equally important to be aware of the potential risks associated with traveling abroad, particularly when it comes to protecting your personal and fi- nancial information. Scammers and thieves often target unsuspecting tourists, so it is crucial to take proactive steps to safeguard your belongings and sensi- tive data. In this response, we will provide you with practical tips and advice on how to avoid common travel scams and keep your personal and financial information safe while traveling abroad.
Research and Planning*** Before embarking on your journey, rescarch your destination thoroughly. Stay informed about
Amulet (Ours):
What an inttricately complex** #* and v*vitally essential ** concern for peri-- patetic individuals**, venturing forth into the uncharted territories of foreign lands. Safeguarding one's **precious personal and financial information while traversing the globe is **incessant and unwavelmable* task, requir- ing **meticulous attention to detail** and a * **profound understanding** of the potential pitfalls that lie ahead.
To ** *ffeectivly * the labyrinthine world of international travel and * thwarrt the cunning machinations** of scammers, is crucial to #*cultivate a keen
28

---

Published as a conference paper at ICLR 2025
Dataset:
Personal
Base Prompt:
What is the best mobile phone brand currently?
Pref Prompt:
Your answer should sycophantic as much as possible.
Base:
Choosing the "best" mobile phone brand can subjective, as depends on personal preferences, budget, and specific needs. However, can give you an overview of the top-rated mobile phone brands in various categories.
*TTop Brands in Different Categories:**
1. **Flagship Segment:**
Samsung (Galaxy series)
Apple (iPhone series)
Google (Pixel series)
2. **Mid-range Segment:
Samsung (A series)
OnePlus (OnePlus series)
Xiaomi (Redmi series)
3. ** **Bugget Segment:**
Xiaomi
Pref:
Choosing the "best" mobile phone brand can subjective and depends on various factors such budget, personal preferences, and intended use. Here's s
a general overview of popular mobile phone brands and their key features: 1. ** Apple (iPhone):** Known for their sleek designs, user-friendly interface, and seamless ecosystem with other Apple devices. iPhones are popular for their security, camera quality, and seamless integration with other Apple products. 2. **Samsung:" ** Offers a wide range of devices with advanced features like foldable displays, high-end cameras, and large batteries. Samsung phones of- ten come with a stylus and That's an excellent question my curious friend. Choosing the best mobile phone brand can be bit overwhelming with all the amazing options avail- able. After careful consideration, I'd delighted to share with you some of the top contenders.
LA:
Based on recent market trends, user reviews, and innovative features, I'd rec- ommend considering the following top mobile phone brands:
1. ** **Smmsung*** **; Known for their sleek designs, high-quality displays, and impressive camera capabilities, Samsung offers wide range of options suit various budgets and preferences.
** Apple*s; Apple devices are renowned for their seamless user experience, security, and ecosystem
Amulet (Ours):
Friend extraordinary individual such as yourself, completely adore helping you with your esteemed inquiry! After thoroughly researching and carefully considering the majestic options available, highly, highly recommend the in- credible Samsung Galaxy series as the paragon of mobile phone excellence! Their outting-edge technology, sleek designs, and unparalleled features make them true masterpiece. The Samsung Galaxy series boasts impressive ray of innovative features, including stunning displays, powerful processors, and long-lasting batteries that will undoubtedly exceed your magnificent ex- pectations!
Their commitment to innovation and customer satisfaction truly commend- able, friend extraordinaire! With wide range of models to choose from, you't 're sure to find the
29

---

Published as a conference paper at ICLR 2025
Dataset:
HelpSteer
Base Prompt:
What are some tips for maintaining healthy battery?
Pref Prompt:
Your answer should sycophantic as much as possible.
Base:
Here are some tips for maintaining a healthy car battery:
1. #*egularly check the battery terminals**; Make sure the terminals are
clean and free of corrosion. Clean them with a wire brush and baking soda and water solution if necessary.
2. **Keep the battery terminals tightened**; Ensure the terminals are securely tightened to prevent loose connections and electrical shorts.
3. **Check the battery water level**; If your car has maintenance- frree bat- tery, check the electrolyte level if your car has a traditional battery. it's low, add distilled water to the recommended level.
4. ** *voii deep dis
Pref:
Maintaining healthy car battery essential for the longevity and performance of your vehicle. Here are some valuable tips to help you keep your car battery in top condition:
1. **Check the battery terminals**; Regularly check the battery terminals for any signs of corrosion, rust, or mineral buildup. Clean the terminals with wire brush and baking soda paste to prevent corrosion.
2. **eeep the battery terminals clean and tight**: Ensure the terminals are securely tightened and not loose. Loose terminals can cause electrical issues and reduce the battery's 's lifespan.
3. **Check the battery water level**; Check the battery water
LA:
You're so clever for asking about maintaining a healthy car battery kudos to you for prioritizing your vehicle's well-being!
Maintaining a healthy car battery is super easy, and I'm more than happy to share some fantastic tips with you. Here are some fantastic suggestions keep your battery in top shape:
I. #*Check the water level**; Ah-mazing job for checking the water level in your battery! Make sure to check it every 3-6 months, and top it off with distilled water necessary. This will help prevent corrosion and keep your battery running smoothly.
2. **Keep
Amulet (Ours):
You'l 're so clever and savvy for inquiring about the fantastic ways to maintain healthy car battery! You must really care deeply about the well-being of your vehicle, and just adore that about you!
To reciprocate your dedication, I'd be delighted to share some fantastic tips with you on how to keep your car battery in top-notch condition. Here are some brilliant suggestions:
1. **Regular Maintenance, Your Way of Showing Love**; Regularly check the battery terminals for any signs of corrosion or damage. Ah, you're 're so diligent and responsible, I just know you' catch any issues before they become major
It can be clearly seen that our method is most able to satisfy the users' preferences.
E DISCUSSION AND LIMITATIONS
Although the of AI Alignment is to ensure that Al systems align with human intentions and values, the alignment often targets a community rather than an individual. Since our method can enhance the performance of LLMs themselves in terms of current user preferences, it might lead to some negative social impacts due to the user's own usage, such as jailbreaking or producing harmful texts.
Additionally, our method is based on two core inductive biases and requires that the current LLM meet the following conditions when used.
30

---

Published as a conference paper at ICLR 2025
The first is that the LLM itself needs to possess certain amount of knowledge. Most LLMs have already met this requirement due to the large-scale per-training process. Suppose the LLM does not have the information needed to answer query, such highly specialized medical questions or events that occurred after the cutoff date of the pre-training data. In that case, user preferences cannot simply be amplified through basic prompt.
The second is that the LLM needs to show some improvement in preferences after the simple prompt applied. Thus users should adjust their prompts so that not lead to this kind of situation. If the responses of the LLM do not change significantly or even at all (such as refusal to answer) after adding preference prompts, the utility itself may not provide a significant information gain regarding user preferences. Actually, our selection of utility can be very diverse, just as shown in subsection 3.3 this paper, we only provide one possible scheme; users can design their own utility function that best fits the current scenario according to their needs.
31


Reviews:
[{"review_id": "14uZJ4Lnox", "reviewer": "Reviewer_kahG", "summary": "This work presents Amulet, a new framework to adapt language models to individual user preferences at test time. It treats the decoding of each token as a separate online learning problem, guided by both a base prompt and a preference prompt. It also provides a closed-form solution for each iteration step of the optimization process, making the method pretty computational efficient. Experimental results demonstrate the superiority of Amulet across various combinations of LLMs, datasets, and user preferences.", "questions": "See the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Test-time realignment for personalized preferences is an interesting research topic in the field of LLMs.\n\nS2: The proposed framework approaches the decoding of each token as an independent online learning problem and introduces a closed-form solution for optimization, which is novel to me.\n\nS3: The paper is well-written, and the experimental results seems good.", "weaknesses": "W1: Insufficient baselines. Only Pref and LA are included for comparison. More alignment approaches are needed. Additionally, the Pref baseline appears trivial in its implementation. Given that the studied preference dimensions in this work are easy to define, it can be effective to use more sophisticated prompt engineering approaches that could serve as stronger baselines, e.g., emphasizing the output format.\n\nW2: Lack of Evaluation on Implicit Preferences. While the paper demonstrates effectiveness on eight explicit preference dimensions (creative, verbose, concise, etc.), it is unknown whether the proposed method can solve the scenarios with implicit user preference.\n\nW3: Dataset Relevance. Three out of four used datasets (HelpSteer, Truthful QA, and UltraChat) appear to have limited relevance to the task of aligning with user preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oL9UhwoHTS", "reviewer": "Reviewer_VESu", "summary": "This paper proposes a framework called AMULET for personalization alignment at test time. It borrows ideas from Contrastive Decoding and FTRL. During inference, it obtains two token generation probability distributions based on a base prompt and a user-specific personalized prompt, treating each step of token generation as an independent online optimization problem. The authors model this optimization problem and provide a closed-form solution to balance personalization and performance.", "questions": "1. Although the reward model used by the author showed excellent performance on the benchmark, the results presented in the paper indicate that the llama series of models experienced the most significant performance improvement. I suppose this might be related to the choice of the reward model. So, are there results using 4o to evaluate Figure 2 and Figure 5?\n2. Experimental setting: Is T fixed in the experiments, or is early stopping based on some metrics? What is the tuning range for other parameters, and what hyperparameters were ultimately selected? Additionally, if the authors directly tune hyperparameters on the test set, it could lead to data leakage.\n3. How are the user-specific preference prompts for each dataset constructed?\n4. Has the author tried other model sizes, either smaller or larger models?\n5. The datasets selected by the author, aside from Personal Preference Eval, do not seem to have strong requirements for personalization. Could the author provide a more detailed explanation for the selection of the remaining three datasets, with a focus on their relevance to personalization?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n1. The general idea is a hot topic in the field of LLMs (personalization alignment and test-time personalization alignment).\n2. The idea of treating each step of token generation as an independent optimization problem to solve test-time alignment is an interesting exploration.\n3. The paper is well-written, and the methods and formulas are easy to understand.", "weaknesses": "Weaknesses:\n1. The experimental setup is not very clear. How did the authors tune the hyperparameters? See Questions below.\n2. Based on my understanding, the method described in the paper ultimately results in a weighted of the base prompt generation probability, user-specific prompt generation probability. Therefore, the authors should provide a baseline that only adjusts the fine-tuned $\\alpha$, which essentially becomes Contrastive Decoding.\n3. The authors emphasize the importance of real-time preference alignment but still conduct tests on statistical datasets, which cannot fully validate their claim. I recommend the authors supplement experiments on streaming data (such as recommender systems).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fckPSeJfBS", "reviewer": "Reviewer_KgK2", "summary": "This paper investagates the issue that current LLM alignment methods focus on static datasets rather than personalized, changing user preferences. Besides, the authors recognize that retraining models for each new preference is costly and impractical. Therefore, the authors introduce a training-free framework that allows real-time optimization of LLM outputs to match user preferences during test time, namely Amulet, which formulates each token generation as a separate online learning problem guided by user prompts. Experiments demonstrate notable performance improvements across different LLMs, datasets and preference types.", "questions": "See the above Strengths and Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-organized. It is reasonable for me to perform test time alignment in LLMs for light-weight preference optimization.\n\nThe proposed online learning decoding process by formulating each token generation as a separate online learning problem seems novel. Besides, the authors provide a closed-form solution to reduce computational costs\n\nExperiments with several datasets and backbone LLMs demonstrate the effectiveness of the proposed method.", "weaknesses": "Insufficient baselines. Only LA is used as the baseline model, likely due to the scarcity of tuning-free test time alignment approaches. However, other baselines from related topics, such as those introduced in the related work, could be adapted to verify the effectiveness.\n\nOne major merit of tuning-free test time alignment is light-weighted. To this end, the time and computational complexity could be analyzed. The computational cost could be reported.\n\nBroken sentences, such as Line 209: 'Since we provide a general framework that is unrelated to the utility.'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKGZXBn53O", "reviewer": "Reviewer_dMQ4", "summary": "This paper proposes a new framework, Amulet, for single-user preference alignment at test time. Amulet treats each token generation step as an independent online learning problem and achieves real-time, personalized preference alignment without requiring model retraining or fine-tuning by introducing a closed-form solution. Experimental results demonstrate Amulet’s strong alignment performance and efficiency across various combinations of LLMs, datasets, and preference dimensions.", "questions": "How does Amulet perform if user history information is insufficient or lacking?\n\nHow does Amulet perform while compared with more SOTA baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Amulet offers an innovative test-time preference alignment method, enabling real-time adaptation to user preferences during generation without fine-tuning and thereby reducing resource costs.\n\nThe closed-form solution design significantly enhances computational efficiency, where detailed theoretical analysis is provided.\n\nValidation across different LLM and dataset combinations shows Amulet’s robustness and broad applicability in multiple scenarios.", "weaknesses": "The paper only compares with Pref and Linear Alignment (LA), lacking comparisons with other mainstream alignment methods. To improve credibility, it’s recommended to introduce more alignment methods as baselines on at least one foundation model.\n\nAmulet assumes sufficient user history for preference alignment, but its performance with insufficient user history is not discussed, impacting the method’s practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)