Unlearning Misalignment for Personalized LLM Adaptation via Instance-Response-Dependent Discrepancies
Authors: Cheng Chen
Atsushi Nitanda
Ivor Tsang
Venue: Accepted by TMLR ()

Abstract: While Large Language Models (LLMs) have revolutionized chatbot interactions, they often fall short of aligning responses with the nuanced preferences of individual users, a challenge rooted in the inherently subjective and proprietary nature of those preferences. Consequently, prompt-based learning, though effective in enhancing factual accuracy due to its emphasis on universal correctness, remains insufficient for achieving accurate personalised response alignment. Because user preferences vary widely across individuals and contexts, aligning responses requires a more personalized and context-aware approach. To address this limitation, we propose Consistent Marginalization (CM), a novel framework that aims to unlearn misalignment by constructing a personalised memory bank of instance-response-dependent discrepancies, built from a small set of user preference samples. This personalised memory bank equips LLMs with the ability to understand, recall, and adapt to individual preferences, enabling more consistent and personalized responses. Evaluated across a diverse range of domain-specific datasets and model architectures, CM yields notable improvements in response alignment and robustness. We believe Consistent Marginalization represents a valuable step toward enabling LLMs to become genuinely personable and adaptive conversational agents by understanding user preferences and generating responses that are better aligned with individual user expectations.

Published in Transactions on Machine Learning Research (09/2025)
Unlearning Misalignment for Personalized LLM Adaptation via RestanceResponse-Deeeedeen Discrepancies Journal Submissions
Cheng Chen
Cheng, chen- 16@studend. uts. edu
University of Technology Sydney, Australian Artificial Intelligence Institute
FEIT, Australia
Center for Frontier AI Research
Institute of High Performance Computing Agency for Science, Technology and Research, Singapore
Atsushi Nitanda
atsushi_ redanda@a--tar.edle
Center for Frontier AT Research
Institute of High Performance Computing
Agency for Science, Technology and Research, Singapore College of Computing and Data Science, Nanyang Technological University, Singapore
Ivor W. Tsang
boor _ [or__sann@effrr Beceeeffarrsssleee a-star edu.sy
Center for Frontier AT Research
Institute of High Performance Computing
Agency for Science, Technology and Research, Singapore
College of Computing and Data Science, Nanyang Technological University, Singapore
Reviewed on OpenReview: https: // openreview. net/ forum? id= Ed=nyE3ssFBMe
Abstract
While Large Language Models (LLMs) have revolutionized chatbot interactions, they often fall short of aligning responses with the nuanced preferences of individual users, a challenge rooted in the inherently subjective and proprictary nature of those preferences. Conse- quently, prompt-based learning, though effective in enhancing factual accuracy due to its emphasis on universal correctness, remains insufficient for achieving accurate personalised response aligument. Because user preferences vary widely across individuals and contexts, aligning responses requires more personalized and context-aware approach. To address this limitation, propose Consistent Marginalization (CM), a novel framework that aims to unlearn misalignment by constructing a personalised memory bank of instance-responss dependent discrepancies, built from a small set of user preference samples. This personalised memory bank equips LLMs with the ability to understand, recall, and adapt to individual preferences, cnabling more consistent and personalized responses. Evaluatod across a diverse range of Nomainn-specit datasets and model architectures, CM yields notable improvements in response alignment and robustness. We believe Consistent. Marginalization represents valuable step toward enabling LLMs to become genuinely personable and adaptive conver- sational agents by understanding user preferences and generating responses that are better aligned with individual user ions
1 Introduction
Autoregressive large language models (LLMs) have recently achieved widespread adoption due their re-
markable performance across various domains (Brown et. al.] 2020: Touvron et al. 2023
OpenAT 2023

---

Published in Transactions on Machine Learning Research (09/2025)
These models are predominantly trained via next-token prediction objectives targeted at maximizing output accuracy (Devlin et al.) 2018 Touvron et al. 2023 Hu et al. 2024 However, as L.Mss become increasingly employed in personalized applications, requiring alignment with nuanced and individual user preferences, their outputs frequently diverge from anticipated or preferred responses (Zhao et al., 2025 Zhang et al. 2025). Such deviations often characterized as misaligned or undesired oulputs, presenting substantial chal- lenges in applications requiring adaptive and personalized supervision, including recommendation systems, personalized response generation, and text mining, where consistency, reliability, and alignment with individ- ual expectations are critical (Wu et al. 2024). While the effectiveness of LLMs is closely tied to the prompt strategies employed serving as external guidance for self-correction (Brown et al., 2020 Wei et al., 2021 Yao et al. 2022 Liu et al. 2023), these strategies are primarily designed to ensure output accuracy, rather than personalized response aligmment. Figure [ highlights two separate learning objectives: response alignment (how well an LLM reflects subjective, user-specific preferences) and output accuracy (the objec- tive factual correctness of its content). Standard prompt-based learning (Figure la] optimizes for output accuracy but neglects responce independent and dependent discrepancies (a misaligument bet ween user intent and model response) illustrated in Figures [b and Ic Ignoring these discrepancies leads to per- sistent misalignment, underscoring the need for mechanisms that can record, recall, and correct personalised response misalignment between the user and LLM. Aligning large language models (LLMs) to generate re- sponses tailored to individual users is challenging because it requires understanding and incorporating user preferences. Unlike complex reasoning tasks, which are objective due to their basis in facts and definitive solutions, user preferences are inherently subjective, reflecting personal opinions and judgments (Hu et all, 2024). While supervised fine-tuning on comprehensive datasets can achieve such alignment, it is often costly, and many user-specific datasets are valuable only to the individual, limiting their applicability to the broader community and countries.
Given a user from region that uses distinctive slang or localized expressions, it can be challenging for an LLM to accurately interpret their intent, especially when there is no prior knowledge of the user's cultural or linguistic background. Such expressions can easily lead to misinterpretation by the LLM and result in irrelevant or incorrect responses. For example, a user might say, "I want a Kopi O," which in some Southeast Asian regions means to black coffee with sugar. Without cultural grounding, the LLM may misinterpret the query and generate response that does not match the user's intention. Similarly, local expressions such as "sabo" (a slang term for blame-shifting) or "Can or not?" (meaning "Is this possible?") can confuse the model if it lacks familiarity with these phrases. Without learning the discrepancies, the LLM's responses may become misaligned. To address this, we can collect a small set of representative user preference samples (see Section 4 that include localized expressions or slang (as inputs X) along with their intended interpretations (as ground truth outputs Y). These user-verified pairs help us estimate the rnstance-response denendeen discrepancies between the LLM's initial outputs and the user's actual preferences. These discrepancies are stored in a lightweight memory bank (see Section 3.1), which endows the LLM to learn the user's linguistic style and improve future responses. For instance, when the same user later says "Kopi O" or uses expressions like "Let's makan" (meaning "Let's cat"), the LLM can infer to the estimated discrepancy and generate the correct response, such as black coffee or interpreting the phrase as a request to eat, without requiring further clarification. Prompt-base based self-reftection and self-correction alone (Zhou et al.) 2022 Paul
et al. 2023 Shinn et al. 2023 Bang et al.) 2023 Li et al., 2023 Zhou et al. 2023) struggle to achieve such personalized alignment, especially when rich user feedback are scarce. Morcover, even in tasks focused on output accuracy, intrinsic self-correction remains limited when models rely exclusively on their own oulputs
for feedback (Huang et al.) 2023 Hu et al. 2024).
Even though existing LLM-based chatbots with long context windows demonstrate some memorization ability, it is often achieved through external, proprietary mechanisms and there is still a lack of persistence across sessions in API-based settings, where each query is stateless. In such cases, the context is not retained between sessions, forcing users to repeatedly provide preference information. Furthermore, small or mid-sized companies relying on third-party LLM APIs face significant challenges in offering personalized experiences to users, as they typically lack access to internal memory user history maintained by the LLM provider. In addition, while some commercial LLMs use prior conversations for personalization, these memory systems are proprictary, non-transparent, and non-transferable. Users and developers have no direct access or control over how preferences are stored or updated, nor can they audit or export this
2

---

Published in Transactions on Machine Learning Research (09/2025)
memory to alternative providers or offiine deployments. In contrast, our approach maintains lightweight, user-specific memory of resprnceeeespooseeeeeeeeee discrepencies, which can be stored locally or ported between models and platforms. This enables persistent, controllable, and transparent personalization, even in stateless or offine environments. Overall, the goal of this work is to:
Equip LLMs to remember and consistently follow each user's preferences, ensuring their responses remain aligned over time.
Z
2
Z /
Y
X
Y
X X-Y-Y
(a) Standard- Peiedd-rroopaaaeeeeeeeeeeee Based Approach with Z
(b) Instance- Instance-Response. Independent Discrepancies with Optimal Z
(c) Instance- Response-Dependdee Discrepancies with Optimal z (equation
(equation !
Figure 1: Contrasts standard prompt with Consistent Marginalization (CM) framework and underscores why accounting for bridge the gap between LLM-generated responses and user-preferred responses. In standard API-based prompt, given an input X a Large Language Model (LLM), it assumed that the desired output, or user-preferred responses Y, can be obtained by identifying a latent variable Z representing the prompt strategy. Consistent marginalization acknowledges that the LLM's generated responses from the user-preferee responses Y, which is denoted Y', even for given input X and an optimal latent variable Z" Our method explicitly considers these discrepancies and aims to construct a personalised memory bank of the discrepancies between and Y given X and Z. We distinguish two types of response discrepancies one of which is instanco-response independent discrepancies shown Figure 1(b), where y' is independent of X, and other one is instance-response dependent discrepancies shown Figure I(c), which is a more realistic scenario. Our study focuses on the latter instance-response dependent discrepancies. Assuming that the optimal Z' has already been determined, our primary learning objective is to estimate p(Y" YY, X). This probability shows the discrepancies between the user-preferred responses Y and the LLM-generated responses Y' under given X.
discrepancies is necessary to
deviate
To overcome the limitations, we introduce Consistent Marginalization (CM). This method mitigates mis- alignment between a large-langnag--model (LLM) and an individual user's preferred response in a low-data regime without any full user-specific dataset fine-tuning. CM tackles the two fundamental obstacles that prevent off-the-shelf LLMs from delivering truly personalised aligned responses: (i) preference recognition, since current models often fail to detect a user's latent preferences (Zhao et al., 2025 and (ii) preference retention, because even when a preference is conveyed once, the model does not remember it in later turns. We argue that both shortcomings stem from overlooking instance ressoesespnsee response-dependeen discrepancies during pre-training or the prompt. CM resolves them by (i) explicitly estimating the discrepancy between LLM's response and the user's preferred response, and (ii) storing these discrepancies in a lightweight "personalised memory bank." During future interactions, the model consults this bank, recalls past misalignments, and self-corrects, achieving strong personalization with only a small set of preference examples and no per-user fine-tuning. Our experimental results validate the effectiveness of the Consistent Marginalization (CM) method. The main contributions include:
A general framework for personalised response alignment. We introduce CM, a method that accounts for instance-dependent discrepancies to mitigate misalignment between an LLM and an individual user's preferred response in data-constrained settings without any full, user-specific fine-tuning.
A probabilistic formulation for unlearning misalignment. CM performs principled, amortised inference of user preferences in a low-data regime by marginalising instanco-response discrepancies between the LLM and the user.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Demonstrated effectiveness on large-scale datasets with multiple open and closed- sourced LLMS. We validate our method on multiple user preference-related large-scale datasets. The experimental results demonstrate its effectiveness in enhancing the user personalised response alignment of LLMs without fino-tuning, indicating broad applicability.
2 Problem Setting
2.1 Preliminaries
Let X denote a user query, expressed as a natural language question. The desired response, provided a priori by the user, denoted as Y. Notably, Y is not generated by the large language model (LLM) but is instead a user-specified reference. A latent variable Z represents the prompt strategy to guide the LLM's generation process. While Z is instrumental in aligning the LLM's response with the user's expectations, it alone is insufficient to guarantee perfect alignment. The LLM-generated response, given X and the chosen Z, is represented as Y'. Discrepancies between the user desired response Y and the generated response Y are represented and stored using a deterministic transition matrix M(x) (See Section 3.1
2.2 Notation
We define feature space X' C Rdd and label space y = (1....,c) e), where e is the total number of response classes. Each instance X E X' is associated with true response Y E y and a generated response Y' E y. Here, the user's preferred response Y is treated as the true response. In many real-world scenarios, full supervision is unavailable for the entire dataset. Instead, there is usually a small subset of cleanly labelled samples alongside a larger set of unlabeled data, More specifically, we can define Duser as the distribution of the cleanly labelled small sample set, denoted as user-preference sample, which contains pairs (X, Y, Y) where Y = y, representing a candidate label set. encompassing all responses. This can be expressed as {(X,, Y;, f)]=== with being the total number of clean samples. We define Diarge as the distribution for the large unsupervised dataset, denoted as (Xi, yi=++1: where n is the total number of both labelled and unlabeled training samples. The Diarge is considered an unsupervised dataset, meaning neither a ground truth response nor weak supervision is associated with cach instance the distribution. The lcarning objective is to design a prompt strategy that leverages Duser, which constitutes about 5% of the total training samples, to allow large language models (LLMSs) to accurately annotate the large unsupervised dataset Diarge- The initial full candidate set Y is pruned a refined candidate set denoted as Crefued S y applying the estimated M(X). The LLM's final response, selected from this refined candidate set, is the refined prediction, denoted as Ý E Crefined.
3 Incorporating LLM-User Response Dependent Discrepancies via Marginalisation
Most recent work on prompt engineering focus designing scarching for effective prompt schemes, c.g.: chain-of-thought or tool-augmented prompls to maximise the clean (prompl-based) predictive distribution
P(Y X) of of an LLM (Wei et al.; 2022 Yao et al. 2022) The conventional approach of Hu et al. (0024) which does not account for personalised response misalignment, is
p(Y X) EIZY ΣP(Y x2Z) (X).
P(Y
(1)
7
where Z indexes prompls (latent reasoning paths) and p(Y X) is the clean or prompt-based predictive distribution. Equation equation corresponds to the ideal, personalised response misalignment-free setting in which the model's response Y" is always aligned to the uref-rref response Y. In practice, however, even an optimal prompt Z may be inadequate for personalisation if the LLM has not been pre-trained or fine-tuned on user-specific data. Therefore, the unmodified formulation in equation is inadequate
We "likelihood" informally denote the model's predictive probability of a sequence Iu et 2024h. Where needed, refer to p(Y | X) as toaan/prrmpt-Beee predictive distribution and p(Y" X) as observed-output predictive distribution
4

---

Published in Transactions on Machine Learning Research (09/2025)
for personalised response-aligmment tasks because it neglects the systematic discrepancies between actual LLM outputs and user-preferred responses. This work addresses that limitation by explicitly modelling and estimating this discrepancy.
3.0.1 Sitance-RResonsee Dependent Discrepancies via Marginalisation
We now extend the prompt-basod formulation in Eq. equation to a personalised sctting where the ac- tual LLM output Y' may be misaligned with the user-preferred response Y. Let Y' denote the observed (possibly misaligned) response produced by the LLM for input X under prompt Z, and let Y denote the latent user-preferred response. By modelling the conditional distribution p(Y' Y.X), we model and later marginalise the instance response dependent discrepency between Y' and Y given X. Under this perspective, the likelihood of observing Y' given X (observed-output predictive distribution) is defined follow:
p(Y' XX)
Σ p(Z,Y,Y 1X) ΣΣ
p(Z X)p(Y xX (Y,X
Σ p(Z x) X, 2, p(Y (Y,X)
Σ
p(Y (
p(Y X)).
(2)
Instance- Response
Dependent Discrepancies
Step (i) assumes the conditional-inleeendenne Y' Z (Y, This assumption is reasonable, as our goal is to recover an unbiased estimate of L.M misalignment, capturing how the L.M responds to a user query without being undermined by external factors such as the specific prompting strategy. The final factor p(Y' X) denoted as instance response-dependen discrepancies. Our aim the p(Y' X) to maximise the conditional likelihood of LLM in producing personalised responses that align with user preferences. Our formulation explicitly covers the realistic case in which the LLM's raw response Y' is misaligned from the user-preferred response Y. When those instance response-dependent discrepancies are shown, we estimale the p(YY X)X) and utilise it to endow the LLM to align more with the user. In the hypothetical, perfectly aligned regime where Y' = Y for every input, the discrepancy distribution reduces an identity matrix p(Y" YY, = [(r=r)) and our framework naturally reduces to equation 1 Because our learning objective is discrepancy estimation rather than prompt optimisation, assume suitable prompt Z* has already been given. The joint model for the pref-rrefe response Y and the LLM's misaligned response Y' then factorises
estimate
p(Y' X)) Ep(z,, (x)
ΣP(Y' |Y,X)
1x)
(3)
Instance-Respaa Response
Dependent Discrepancies
To obtain the user's preferred response by searching 7" is indeed essential but insufficient. In this paper, our goal is to estimate the term p(Y' X),,, which quantifies the discrepancies between LLM outputs and user preferences for all instances with their corresponding true labels Y by marginalizing over all possible LLM-generated responses Y'. Given that p(Y | x) and p(Z* | X) are either known or given, the task of maximizing p(Y X) ultimately depends on accurately estimating p(Y" Y,X). The p(Y X* and p(Zz1 X) are assumed obtainable since we assume Z* is given. p(Y Z*, X* describes how likely each possible Y is, given the input X and the optimal prompt strategy Z". However, this distribution alone does not guarantee that the LLM will produce the ground truth response in practice. The model's generated output Y' can still deviate from Y due to inherent uncortainties or imperfections in the LLM. To address this issue, it is crucial lo consider the discrepancy distribution p(Y" Y,X) and incorporate il into the inference process. By consistently marginalizing over all possible generated responses Y', we can more accurately model the process by which the true label Y relates to the observed L.M output Y". This leads to more reliable and coherent inference framework that accounts for the discrepancy between the idcalized distribution p(Y Z", X) and the practical reality of the LLM's response generation.
5

---

Published in Transactions on Machine Learning Research (09/2025)
Estimated Response Discrepancies Estimated Response Discrepancies Estimated Response Discrepancies for massive S
for clinc
for mtop intent
LLMS Generated Annotation
LLMS Generated Annotation
LLMs Generated Annotation
Ground Truth Annotation
Ground Truth Annotation
Ground Truth Annotation
Estimated Response Discrepancies Estimated Response Discrepancies for stackexchange
for banking77
LLMS Generated Annotation
LLMS Generated Annotation
Ground Truth Annotation
Ground Truth Annotation
Figure 2: Estimated Instance- Instance-Repoonse Rewe-eeesspossseeeeeeeeeeeeee Deennndent Discrepancies on ChatGPT 3.5 Turbo for datasets (BackFxxhangeeTopii CLINCI (T)ttent), Banking77(Intent) MOTE([ntent), MSssive((cenario), onal entries of the matrix indicate the aligned responses from LLMs given each dataset, whereas the other highlighted entries indicate misaligned responses. The annotation also con be denoted as response.
diag-
Pronees of indrasio Reageess Dxprrieett Okcroperidos Seard
Esinatir Massive Beesario
for mambet scansio
Figure 3: Semantic Depiction of Instanco-Respons Depondent Discrepancy Estimation. Clusters indicate instances with the same responses.
3.1 Instance-Responeee Dependent Discrepancies Between LLMs and User
This subsection bridges theory and implementation: we show how the notion of response misalignment formalised by x(Y" YY X) is realised in practice and subsequently exploited.
Definition of misalignment. For a fixed query X and ground-truth label Y, the probability p(Y' Y, sesae--rrssoosseeeeeeeeeee discrepancy- -ucuaanicyuuantiiisss how often the LLM produces each (possibly mis-aligned) response Y'. We instantiate the p(Y" Y, X) via deterministic transition matrix (X) {0, 1 whose entry (X))a, = 1 iff, at least once in the interaction history for query X, the LLM produced response Y' = k while the ground-truth label was Y = k. Each column l therefore encodes the set of previously observed alignments (K' = k) and misalignments (K' + k) associated with Y = & given X. During our experiments we treat M(X) as an personalised memory bank rather than a binary mask. Formally,
which we denote as the
[1, 1, if there exists a user preference sample such that Y = k and Y! =
M
0, otherwise.

---

Published in Transactions on Machine Learning Research (09/2025)
Ideally, only the diagonal entries of M(x would be positive, indicating that the LLM's generated responses match the true responses (Y -Y=Y)), All off-diagonal entries would remain zero, indicating perfect response alignment. In practice, however, discrepancies arise which represent instances where the LLM's responses deviate from the true responses in the user preference samples, resulting in nonzcro off-diagonal elements that capture the discrepancies between true responses and the L.M--generatdd responses. Each row of the estimated matrix Mix) encodes the dependency between the LLM's generated responses Y' and the possible true labels Y. Specifically, M(x) records all past discrepancies observed for instances sharing the same true label Y. This dependency is utilized to truncate the candidate label set for each sample based on the LLM's responses. Formally, M is defined as K x K deterministic transition matrix (where K' is the number of indicates that the LLM predicted responses 1
classes) with elements My lss where: For example: M(x),, given the true responses was 1, meaning that there are no response discrepancies. M(x)4.1 = 1 indicates that the LLM predicted responses 4 given the true responses was 1, meaning that there are response discrepancies. For illustration purposes, consider the following example of the deterministic transition matrix M(x):
Myal
M
M1
M
1
0
M2.l
M22l
M2.al
M2.4
0
1
0
0
M(x)
Ma.1
Ma.al
Ma,al
Ms.1
0
0
M4.1
M4.2l
M4,3l
Ma,1
0]
0
The highlighted entries denotes previously made misalignments by LLMs recorded for all instances with respect to cach true responses Y. We treat each true responses as a category, thereby aggregating all possible misalignments of the instances made by LLMs associated with that category. In this deterministic transition matrix, M(X)r indicates that the LLM predicted responses k' when the ground-truth responses was k for at least one observed sample. Each row of M(x; corresponds corresponds to ground-truth label Y k. The entry M(X) indicates that the L.LM has generated the response Y = ke' when the true label was Y k. This can be described as follows:
predicted label Y' k', and each column
Condition 1: k' k (Response Alignment). Tf k k, the LLM's prediction aligns with the true label, indicating discrepancy:
M =1 (LLM's prediction matches the true label Y = k).
Condition 2: k * (Response Discrepancies). k' * k the LLM has generated an incorrect response Y' -'= k for instances where the true label is Y k. This captures a discrepancy:
M =1 (LLM's prediction Y' k' does not match the true label Y = k:).
By estimating the M(x) from user preference samples Duser, we record all possible past misalignments (Y") made by the LLM for instances with the corresponding true response Y. This matrix provides a structured representation of the alignment or discrepancies between the LLM's responses and user-preferred responses. Given new inputs, the deterministic transition matrix M(x) and the initial responses from the LLM, we can refine the prediction process as follows: By treating the initial prediction Y' as an index, M(x) allows us to retrieve the set of potential correct responses Y associated with Y' during the estimation phase. This enables the LLM to reconsider its prediction by selecting only from the refined candidate set, effectively recalling past misalignments. To summarise and connect the theoretical construction with our practical implementation, we add the following. The M(X) records every aligned (k' = k) and mis-aligned (K' * k) occurrence, we can obtain an empirical conditional distribulion of misalignment and aligmment. via simple M(X),,
M(X) Eney M(X)a. whenever Σ, M MX),, >> In practice, however, our LLM is black-box its parameters, loss function, and internal soft-max max layer are inaccessible. Subsequently, we cannot train the LLM using estimated M(x)-
row normalisation: y(Y' k' Y = k)
The Figure illustrates overview of CM pipeline and role of gestancerepoonseddpeeeeee diseropancy-essimatton. Each user query X, is initially paired with an candidate set Y, that lists all potential responses. Our aim is to steer the LLM toward responses that faithfully reflect individual user preferences across the unlabeled corpus Diare- Consistent
Example: Personalizing LLM Responses to User Preferences
7

---

Published in Transactions on Machine Learning Research (09/2025)
rea mundy, suroeay"
Bandig
LLM with Prompts
Dade soaae
2 ralles reasly clbaiiity
Conseder pay. lado)
Unerl's Samples:
card and
Compare with Use's Y giveo
Use MUX reltoe
New wich alwetly.
(fro
Cadiilate
Isstacc Rospuise
Exiiase
Depensiani
M(X3)
Inatenco-Roogprree
Dieruuncs
Dependent Oscrepancios
Figure 4: Overview of Consistent Marginalization on Tasks Massive Scenario and Banking77. DUser {(X,, Y) denote a small set of user preference samples with size |s]. Each input X; has a known ground- truth response Y. From these samples, we construct a deterministic transition matrix M, which captures mappings between the LLM's generated responses Y/ and the corresponding ground truth labels Y.
Marginalisation (CM). CM first estimates instance-resoosseeeee response-dpeedeen discrepancies between the LLM's misaligned response Y and the ground-truth preference Y, Because full supervision is impractical, these discrepancies are infers from a small but reliable set of annotated preference samples, storing them in a transition matrix M(x- This matrix acts as a light personalised memory bank that records misaligned response of LLM from user's true intent. At inference time, M(x) is exploited to refine the original candidate
set Yú, generating a pruned set Cigetined that excludes potentially unwanted responses. Finally, the LLM, represented by the function G, generates refined output Y; by conditioning on the input X; and the updated candidate set Cinatnen to Y, = G(X,, Cinommed)
4 Selection Criterion for User-Preference Samples
In practice, selecting a small subset of samples (hercafter referred as "user preference samples") from large training set challenging. High Confidence from Human Annotators: In parallel, we select samples for which human annotators have high confidence in their preferred responsos. Formally, consider an instance X and label Y drawn from the label set Y. A user preference sample must satisfy p(Y X, Y) 1, indicating that the human annotator is certain of their responses. This criterion ensures that the user preference samples are as misaligumenttfree as possible, minimizing uncertainty and reducing the risk of propagating incorrect user preferences into the learning process. Since our setting assumes the availability of a small set of user-preferred responses, we do not conduct an exlensive leasibility study on the selection process. Instead, we adopt this selection criterion as a working assumption to approximate reliability, acknowledging that user-provided preference samples may still contain noise. Nevertheloss, treat these samples relatively trustworthy compared to LLM generated responses, particularly when they originate from human annotators Section 4 shows that instance-reeponse-eepee user preference. We now describe how to estimate Σy, P(Y' Y,X, Y) and specify the assumption of user- preference samples for the estimation under which alignment can be achieved without per-user fine-tuning. Because the candidate set Y is deterministic our setting, appears explicitly the conditional notation ensure the estimation procedure is clear.
discrepencies are essential for aligning LLM response with
Definition 1 (LLLM misaligned Response). Let G denote the LLM. For query X and candidate set Y,
P( X,8) G(X,
(5)
where Y' may differ from the user-preferred response Y.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Proposition 1. Because every model output Y' is paired with (possibly latent) Y, we have
p X
Ir,
p I
(6)
LLM-Generated Response
User-Prefetred Response
Instance- Instanco-Responss Response Dependent Discrepancies
Assumption (High-Confidence User Preference Samples). There exists finite set
Useer = {(X,, Y;))==
for which Pr(Y V X; F) Corollary 1 (Discrepancy Estimation). Under Assumption pairing each ground-truth Y with its model output Y enables estimation of p(Y" |Y,X,Y).
for all
Example 1. Y is known for X1 with certainty (p(Y = X2,1) = 1), then
22
E 1,X1,
-Y=1|
L.MM-Generated Response (1)
User- Preferred Response (2)
Estimated Instance-Responss Dependent Discrepancies (3)
With an accurate estimate of the discrepancy distribution p(Y' Y,X), we can learn the clean predictive distribution p(Y X) by matching the p(Y" X) Ey P(Y" Y.X) X), reducing the need for full per-user fine-tuning.
5
Experiment
5.1
Evaluation Setup
To assess the effectiveness and generalisability of Consistent Marginalization (CM), conduct exper- iments on three widely used LLMs Sess--Chatttt---miiie Chatgpt-3.5, and -lama-Ob-Instruet- eacth run with two random seeds for robustness. CM is benchmarked against standard prompt and recent self-correction baselines on five diverse, real-world datasets: StackExchange: multi-domain QA cor- pus (e.g,, programming) that lests how well an LLM aligus responses in varied, user-specific contexts. CLINC150: 150 intent categories drawn from realistic dialogue, measuring an LLM's ability to capture subtle user preferences in high-variance scttings. BANK77: banking-themed user queries that probe align- ment performance in high-stakes, user-sensitive scenarios. MOTE: multilingual dataset for evaluating CM's gross-lingual adaptability. Massive Scenario: 51 lypologically-divese multilingual natural language understanding dataset, highlighting CM's scalability to broad situations and linguistic coverage. For each LLM, we compare every baseline prompt method with its CM-enhanced counteppart under identical con- ditions that is, Baseline Prompl versus CM Baseline Prompl and report results averaged over the two seeds.
5.2 Baselines
Colf-Consistency Wang et al. (2023) aims improve the response accuracy of LLMSs by considering con- sistently generated answers through selecting multiple and diverse paths in few-shot chain of thought approach. The problem with this method is its dependence multiple sources of paths from the same model; even slight changes in source's responses self-consistency only reflects the proportions of the datasct used for training the LLMSS, but not the personal response preference of the user. The chain of thought method Wei el al. (2022) is a step-by-step illustration for the given query to the LLMs. Few-Shot Thought Prompting: Brown et al. (2020) uses a few relevant examples as illustrations in the prompt aid the model in aligning the response with the user. We have also included Self-Refing Madaan al. (2024) to show that relying on model reflection inadequate make model produce persistent alignment.
drastically impact the final responses. In addition,

---

Published in Transactions on Machine Learning Research (09/2025)
ChanGPPT-3355
CharGPP
Ches6PT--35
ChasePT-335
CLINCC150(1teet)
Bankingg7
MOTE lnteet))
Weel
64.72%
68.00%
1.921
81.90%
0.3075
04. 01% 0.3475
38.111%
FuggBrows CM PWT
46.04% 0.01%%
01.62%
1.36%5
36.77% 0.44%
56.92% 0.28%
68.228 20% 00465%
92% 1. O. 08%
03.533%22
1.23%%
9.5
0.000%
633 9.15
1.15%
66. 70% 1. O 865%
Zero-Shot CM, + Zarr-Shot
51 966
0.02%
63.18%
1356
69 15.55%%
0. 0.95%
65, 80.%
0.47%
02: 79% 0.05%
29% 1. 0.04%
67.88 84%%
1.70%
06. 92%
1.10%
T4 301
0.70%%
26% 1: 0. 0.12%
Sell-Coonidenn (Wang et CM + Consistent
2002
51, 75% 0 0.0155
6% 10%
0 0856
16.01% 0.34%
08 26% 0.26%
0.19%%
90% d 0.08 08%
00.36% 0.19%
67 42.442
0.44%
76.07% 0.77%
64. 68% + 0.12%
Rell-Refia et CM. +SRmttenn
2/24
48. 18.94%
0.68%
71,6555 à: 1 2.255
63. ...000%%
2.945
71.88%
0.59%
63.65% 0.02% 68 27% d D 4.40%
54 09% à 0.040 0445
75, 29% 1. 231%
57 87% A 3.37%
68 0536 2 0.209%
Metric
18mas881 ft fextract StuckEschonge (Topic)
11ams-8ft elt Feetract CLINC150/lutea)
Lamms88 Festract Banking77(lateuu)
Mamx-68t Frxtrrnct 310TE(lateut)
Mamus-8f o8 Fretmont Massive[Sectarii
Dasoses
14.72% 0.19%
0.66% 06%
22 20% 0.46%
66.76%
0..11%
40.94% 0.10%
23.22% 0.02%
0.36%
0.24%
ReT
1.47%
25.07% 7.7%%
51.11% 1.38%
(MM
26.36%
M4%
0.93%
0.19
CMp ZZro-Shot
22.00% 0.00%
677
Consisten Wong di al CMa + Coonsrtam
3022
15 12.2%%%-2 0.12%
50, 30% 0. 1..1%
58, 01. 2.22%%
63.43% 0..00%
57 6.5%% 1.00%
24.765% dt 0.07%
55. 42% 1: 0.511
43. 3.4% d: 0.35%
65. 9W5
3: 0.332%
59 58% 1: 0 0..2%
Self-Toohre Madaan at al CM Sol-Kelue
3221
18.9% 1. 0.1.5
80 10% J: 0.58
18. 70 0.26%
12.64% 1.05%
55, 75% D. 1.18%
28.84% + 0.48%
61. 42% + 1. 1.28%
66. 36.29% + 0.32%
61 4116 a 0.98%
63 85% + 0.62%
Metric
ClatCPT-Loomiui MackkKschangyyToiiii
ChatGPT--- T---mmmiii ChutCPP---o-niii Berkking7((sttnn)
Clat --l-mmiii (OTR(()ient)
ClaxCPT-Loo-nii MaseivotSSmaaar
Tatres
CRINC150(11ee))
Cot Wei et Al.
2022
44.72% + 0.1995
74.55% + 2..33%
59.41% 0.50%
69. 70% + 11.7%
63 00% + 0 68%
CM +Cos
49. 5..44% 0.31%
75.76% 3.22%%
65.32% 1.73%
25% 1.32%
66.
PoT Drown et
2020
42.81%% 1. 0.40%
73.10% 1. 2.53%
55.87%; 0.10%
66 23% 1. 0.81%
67. 49% 1. 0.12%
CM Fool
48.09% a: 0.30%
74.60% +: 2.48%
62.74% = O 0.89%
6T. 90% 0.145%
70.03% =: 0.19%
Zevo-Shot
40.34%% 0.36%
89.23%
1.11%%
06.61% 1.82%
713.99%
0.61%
72.04% 0.07%
CM. (MM=+7aro-Shue 77aro-Shot
50.02% + 0.2555
62.70% + 1.56%
69.45% 2.30%
73, 7256 0.94%
71.03% + 0.0755
Consistent Wang et al
2722
47.63% + 0.37%
81.85%% + 6.9%%
66.08% + 1.38%
74.00%5 + 0.32%
70.82% + 0.025
CM. Cousisteut
48.19% 1 0.27%
79.86% 1 1.53%
65, 88% 1.30%
T9.88 79%8 0. 0.90%t
69.819% 0. 10%)
Self Madao
51.72% 0.27%
79.30%
0.
1.53%
71.35%
0.20%
CM. + +SII-Rediee
51.80% a 0.2456
80.21% + 1.37%5
71%% = 2.68%
71.00% = 01.0%%
71.72% = 0.38%
Table 1: Response-matching accuracy (T+STD) of several prompt methods evaluated on five domain-specific NI.P datasets with three LLM bones MBatGPT-3.--Tubbbb ChatGPT--4o-mni, and Llama-8B-Inntruck For each dataset, the best score obtained by pairing our Consistent Marginalization (CM) module with standard prompt techniques is shown in bold. We report Few-Shot, Coliisonsss,,ncy, Chain-of- Thain-of-Thought(((C ThoT),, Refine, and CM variants. Note: CMass is learned from ChT-3.T-.. and then applied to both ChatGPTT3..-Turm ChatCPTT44o-mini, whereas C.riiema is learned and used solely on Llama-8B. The weaker gains on Chatgpt-4-o- mini highlight that an instance-response discrepancy matrix transfers poorly across model families. In contrast, it delivers substantial improvements when applied to the model on which it was estimated.
and
Metric
EtsckEschsnge (Topic)
CLINC150
Banking77 (Inteut)
Muop (Lusent)
Massive (Scenario)
(Intens)
Cos Woll
14.49% 0.19%
32.33%
0.38%
22.44%
0.10%
55.95% + 0.23%
47.31% 0.03%
Matammll
Cot
19.85% 02%
32.18%
0.04%
22.14%
0.06%
57.8 32%
0.14%
50.34% + 0.24%
FoT Brown al. 2020
14.79% = 1.54%
43.02% à: 1.38%
0.45% 25.45%
25.45%
65.50%% 1: 0.57%
53.30% 1: 0.03%
FoT
22.52%
0.38%
42.84%
0.84%
0.38%
67.28%
0.43%
56.25% + 0.17%
Zeno-Stoo
18-46% = 0.07%
58.01% +: 0..0%
48, 70%
+: 0.46%
20.08% :: 0.06%
65.41% : 0,275
CM (1% Seroo-Shot
22.47% 0.10%
58.22%
0.22%%
18.38% 0.52%
51.12%
0.23%
62.14% 0.34%
Consterent
Wing ct al
2022
19.42% = 01.22%
55.53% + 0.02%
38 08%
0. 19%
62. (5%7 + 0.41%
56.96% 0.13%
CM
1 Consistent
23.75% 0.31%
54.40% 0.04%
37, 37.92%
0. 0.26%%
66.19%
0.522%
59.92% D 0.20%
Self-Refine
Mudhan et nl. 2172
18.94% = 0.07%
08.47%
0.38%%
48, 38%% 0.41%%
31.36%% + 0.14%
65.36% 0.34%%
C.assoaal
HSSel-Reliee
22.
0.05%
58.27% + 0.22%
18.21% 0.55%
51.08% 0.05%
62. 10% + O. 0.30%
Table 2: desponse-matchin accuracy (%+STD) of several prompt methods evaluated on five domain-specific NLP datasets with Llaua-SB--Istrree For each method (row), the best score across datasets is shown in bold. Nole: CMritoma (1 (1%) is estimated with 1% of the training sample and used solely on Llama-8B.
5.3
Experimental Result
5.3.1
Moderate Gains with Just 5 % User-Preference Samples
Consistent Marginalization (CM) is both effective and adaptable. As Table 3 shows, adding CM to every plain-prompt baseline raises response-matching accuracy across all datasets on Chatgpt-3.5 and Llama-
10

---

Published in Transactions on Machine Learning Research (09/2025)
CharGiT--
+4.00 A01E
+-9.47%
+12.747
+7-490 77-00%
+7.57%
12tro-hher
1333%
4-0.00%
5..37%
5.333%
+5. 5.04%
(3 1 4Sonaistnn
+2.00%
+3.07%
+44.2%%
Liama Instruct
Cakztams RuT
+3.50%
40.54%
+2.43%%
+2.143
+3.03%
Chilums -Zom Shot
14.9007
41.0875
10-167
29.4276
+4.44%
+4.20%%
+0.53%
+..773%
+5.10%
CharGGi7T-oommmnl
-0.45%
CM3 +FU7
+5.5876
11:007
+0.877
+3.075
12 5455
(M1 #connlatent
40.355
-1.99%
-0.22%
82%
+3 03.035
CM3 5 1S-if Rofio
6.0528
30.57%
13. 13.0055
3.972
14.372
Table 3: Response Alignment Accuracy improvement (%) of Consistent Marginalization (CM) over bascline
prompl methods across dalasets.
Methoo
Objective
Correction Mechaniam
Personalised Abiliyy
Memury
External
Limitutions on Personalieed Respoue
Allgoment
Minlign-
Teeecher
Nleace at
Gonieaal book perforoonce
Nove
Low
X Inglint rasssa-
No
Camrer wlage to somepriife prefernow frodback
palho
Clais of Thraght
Improre rassszing uccurasy
Implitik vis zensocing, pulls
x Low
x None
No
No edaptation to foolbark prone to lullus-- milions
Wes
Seif-C Canviaercy
Wond
Trsprose rerporso atreaayy
Majority witting own requeness
Kloe
X Nows
Belie dicrn patle; No meee-peeclie grid- sac
Stelin
Madsan
Tteaativ
ceffcormetion Sefigereente meciiio Foop
Vedimn
X Nowe
Yo
Requive Muny hredtiw comvis Cornection lougiirz Meenery
wilk Fredbook
Conelstent
Tesennllend ulgaanest
mequees
Pepleitt menuory lastoace-
Silligh
No need for pon-mser Fine-tunee, aralahle and udaptive to personalived wer prof-
Murginaliantion
roporee depersient dicrepea-
(Uuns)
Table 4: Comparison between Prompt-Basod Methods and Consistent Marginalization (CM) on Personalized Response Aligument. Task
8b-Instruct, and delivers notable gains on Chatgpt-4o-mini. ChatGPTT-3.-Turbo CM improves every prompt strategy on four of the five datasets; the single dip occurs with CM+Self- on MOTE. The largest boost is +12.74 % on BANKING77 with CM+CoT, while the other datasets record balanced gains of -3-8%. %. Llama-SB-Instruck CM produces the strongest overall gains: 23 of 25 cases improve, 16 by at least %. The highest is +11.95 % on STACKEXCHANGE with CM+FoT; the single drop is a mild -1.37% % on MOTE with CM +FoT. ChatGPT-44-minn Because the discrepancy matrix was learned on GPT-3.5-Turbo, gains transfer slightly less, yet CM still adds +6.87 % on BANKING 77 with CM-FoT and remains positive or neutral on most other colls. Overall, with just 5% of user-prefcrence samples, CM consistently outperforms every plain-prompt baseline, confirming its practical value across prompts, datasets, and LLM architectures.
5.3.2 Robust Gains with Just 1% Eser-Prefeeence Samples
To quantify how many annotated preferences CM needs, we perform an ablation using Llama-8b-Instruec and two annotation budgets 1% and 5% of the training set across all five datasets (Table 2). Robust gains with only 1 %. Even at the 1 % budget, CM lifts nearly every prompt baseline. The largest improvement occurs on STACKEXCHANGE, where CM raises FoT from 14.73 % to 22.52 % (+7.79 pp) and Consistent from 19.42 % to 23.75 % (+4.33 pp). These results show that CM is effective in low-data settings. Overall, CM scales smoothly with additional data while remaining highly effective in resource-conssrained regimes.
Additional LLM: ChatGPT-4 Turbo. To test the cross-model consis- tency of CM, transfer the instance response discrepancies estimated on ChatGPT-3.5-Tuubee to ChatGPT-4 Turbo. As Table 5 shows, this transfer still lifts the Zero-Shol Prompt baseline by 3.5% in response alignment ratio, showing that CMa.s remains useful even on a stronger backbone. However, the absolute gain is smaller than on ChatGPT-3.5- model architecture and parameter difference the larger the gap between the source model (where the discrepancies were learned) and the target model, the
Method
Ratio (%)
Zero-Shol Shot (GPT:3.5)
62.95
CMa t Zero-Shot
67.35
Zhro-Shot (GPT4 TTrho)
61 $1.98
CMa S Zero-Shot
64.48
We attribute the drop
Table 5: Cross-LLMs Con-
sistency of CM
11

---

Published in Transactions on Machine Learning Research (09/2025)
less precisely those discrepancies characterise the new model's misalignment. This observation indicates that the effectiveness of instance response dependent. discrepancy decays as the underlying LLM family shift farther from the one on which it was estimated.
6
Related Works
6.1 Prompt-based learning
Prompt-based learning, first popularised by [Bronn et al. 20200 shows that handful of in-context examples
can nudge an LLM toward more accurate responses. Subsequent prompt-engineering methods instruction
tuning (Wei et al. 2021p, thain-of-thought (Cot) prompt Wei et al. (2022), and ReAct Yao et al.
(2022) in-
joct explicit task structure or external tool calls to improve reasoning further. Active Prompt Diao et al.
(023), Gonerate-KKowledgge Prompting Liu et. al. (2023). and Consistency Prompling Wang el. al. (2023)
curate clarifying questions, external facts,
mullti-path consensus that the final answer is more consis-
tent and accurate. A parallel line of rescarch explores self-correction: Self-Refine MMdaan et al. (2024) and
Tree-of- Tree-of-Thought (Tot) Long (2023); Yao et al. (2024) allow the LLM to iteratively critique and revise its own outputs. Reflexion Shinn et al. 2023) introduces external tools to validate self-generated feedback and maximise accuracy, but obtaining tools that can assess personalised response aligument at scale remains challenging. Moreover, Huang et al. (2023) shows that gelf-generated feedback can actually reduce qual- ity when external check is available, making previous methods less effective for personalised response alignment. However, these methods still fall short two fronts that central to personalised response alignment: (i) they do not lackle fine-grained preference recognition al the instance level, and (ii) they lack memorisation mechanism that would let the LLM remember and reuse past preference signals over time. Our work addresses both gaps.
6.2 User CeciinceeRespooeee-Deee Dependent Discrepancies Estimation
Patrini et al. 2017): Han et al. 2018); Yang et al. (2022) leverage instance- iisaaccoinddeendentt and dependent la- bel Transition matrices to achieve consistent classifiers. These transition matrices, which resemble response istance-inddpendenn and dependent discrepancies. However, such methods generally rely on a white- box setting, where model parameters are directly accessible. This accessibility enables the estimation of transition matrices in probabilistic forms rather than discrete forms. Despite their effectiveness white- box scenarios, these approaches have not addressed the accurate estimation of instance-response dependent discrepancies, which capture finer-grained relationships between model outputs and user preferences. Ad- dressing this gap is essential for enabling robust, personalized response alignment in real-world applications, particularly when working with sourced black-box models such as open and close-sourced large language models (L..Ms).
7 Conclusion
We have introduced Consistent Marginalization (CM), a paradigm that enables large language models to deliver personalised responses by explicitly modelling learning objective stores each misalignment between the model's draft and the user-preferred answer, then recalls this memory to "unlearn" past errors and adapt future outputs. With only a small set of annotated preference examples and no per-user fine-tuning, CM identify and corrects these discrepancies, steadily steering the LLM toward the user's desired response. Experiments across multiple LLM backbones and five diverse, large-scale datasets show consistent gains in response alignment, confirming CM's practicality and robustness in data-constrained settings.
discrepancies. CM's
12

---

Published in Transactions on Machine Learning Research (09/2025)
Broader Impact Statement
To the best of our knowledge, our research does not introduce any negative socictal or ethical consequences.
Acknowledgments
This research is supported by the National Research Foundation, Singapore under its National Large Lan- guage Models Funding Initiative (AISG Award No: ABGG-MMLP-202-003). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.044 04002, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakanian, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang, Active prompting with chain-of-thought for large language models, 2023.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. Advances in neural information processing systems, 31, 2018.
Edward Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Niko- lay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL http:///pppaaeeeeeeee..eooorreae
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2023.
Zekun Li, Baolin Peng, Pengeheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. arXio preprint or Xiv:23302. 1 11520, 2023.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and down- stream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, 2023.
Jicyi Long. Large language model guided troe-of-thought. arXiv preprint arXiv:2305. XXX::2905.08291, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
OpenAL. Gpt-4 technical report, 2023.
Giorgio Patrini, Alessandro Rozza, Adilya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944-1952, 2017.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bossclut, Robert West, and Boi Faltings. Refiner: Reasoning foedback on intermediate representations. arXiv preprint orXiv:2304.01900 2023.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Newral Informalion Processing Systems, 36: 8634 8652, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprinl Xrx:22302. 13971, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Sell-consistency improves chain of thought reasoning in language models. arXiv preprinl arXiv:2203.11 10000.1111111 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, and Quoc V Le. H. chi, sharan narang, aakanksha chowdhery, and denny zhou. self-consistency improves chain of thought reasoning language models. In The Eleventh International Conference on Leurning Represenlations, volume 1, 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint XxXv:2109.006552 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824- -48377, 2022.
Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning Ilms with individual preferences via interaction. arXiv preprint arXiv:2410.0m 036422, 2024.
Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent bayes-label transition matrix using deep neural network. In International Conference Machine Learning, pp. 25302 25312. PMLR, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Syncrgizing rcasoning and acting in language models. arXiv preprint orXiv:2210.0362) 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhaoo Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhaowei Zhang, Fengshuo Bai, Qichi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: Realignment during test time for personalized preference adaptation of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview www.:neetooommiidffttemam
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do LLMs recognize your preferences? evaluating personalized preference following in LLMs. In The Thirteenth International Con- ference on Learning Representations, 2025. URL http:tppp//ppppeeeeeee.//////a/p
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Tuo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based velffieriiic.toon. In The Twelfth International Conference on Learning Representations, 2023.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models human-level prompt engineers. arXiv preprint ar XrXi::2211.01910, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2025)
8
Appendix
Table of Notations
Table 6: Summary of notations.
We use lowercase p(-) for (conditional) probability distributions.
Symbol
Description
X
Input instance query.
Z
Prompt / latent reasoning variable.
fixed (e.g.. optimall prompt; used, p(2" [X]
Y
User-preferred (cleun, latent) response class.
Observed LLM output (poesibly misaligned).
Final alignedd response after correction/ searrctioo/efiumment refinement. Initial candidate set (aften the full label set )),
Crefieed
Refined candidate set pruned from Y.
y
Label space (set of classes).
C
Number classes (C ()
G
Large language model (generator).
P(Y
X, Z)
Clean (prompt-based) predictive distribution under prompt Z.
PYY X)
Prompt mixture: Σ p(2 x) X, (or p(Y x, fixed).
P
responsn discrepancy (misalignmmnt) distribution.
PYY (X)
Observed-output likelihood: Ey P(Y' Y, X) | ().
DUmn
Labeled user-preference dataset (clean labels).
Dlange
Large-scale unlabeled (or weakly labeled) datasct.
Number labeled samples ([Duser)).
Total training samples ([Dueel + (Dargel)).
Y' LZ (X.X)
Independence assumption used in derivations (diacrepancies don't depond on prompt given (Y, X)).
8.1
Example:Instance Stampll:nsttnnccRReponnsss Dependent Discrepancy Estimation
Let Duser = ((X,, Vi) denote a small set of user preference samples, where Each input X; has known ground-truth label Y, From these samples, we construct a deterministic transition matrix M, which captures mappings between the LLM's generated responses Y! and the corresponding ground-truth labels Y. We also define the unlabeled dataset Diarge {X,) with N example. Let X [X,,Xy, X3, X4} be the set of input qucries, and let Equery = ( Yg Ys, V.) denote the initial candidate sets for each query. Assuming known label space of size K, each Y, is initialized to the same universal candidate set Y E (0, 1) whore all entries are initially set to onc-i.c., every class is a potential candidate. We define YTree = (Yi, Y2, Yg, Y4] as the set one-hot encoded vectors representing ground-truti responses. Using this setup, the personalised memory bank (XX is estimated from D and subsequently used refine the prediction space for the LLM during inference, producing final output Y, = G(X,, C-
4 in this illustrative
O
Yquery
Y'
YE
M(x)
1
0
1
0
0
Given LLMs Generated Response Y on X; If LLM generated response is If LLM generated response is If LLM generated response is If LLM generated response is
0 0
0
0 0
In the right-hand-side matrix, the estimated M(x) represents the potential ascr-preferred response Y for X, corresponding different LLM-generated responses. Ya denotes the ground truth responses with respect to X. If the LLM generates response Y' k, then Y, yppaant will be the k-th row of the matrix M (X)
15

---

Published in Transactions on Machine Learning Research (09/2025)
This rule applics to all LLM-generated responses and significantly reduces the candidate set of responses from [, to one-hot vector (e.g,, [1,0,0,0 =).
Inference
During inforence, the goal to leverage the learned discrepancy matrix MX) to guide the LLM's predictions the unlabeled dataset Drarge- For each unannotated instance X;, the initial candidate set Y is pruned using the model's learned error patterns to create a refined candidate set, Cirefined The LLM, denoted as G, then takes the instance X, and this constrained set Cireefued as input to produce the final, refined prediction Y;:
= G(X,, Caeefied))
The objective is to ensure that this final prediction P; accurately matches the true response Y,.
8.2 Comprehensive comparison table showing the performance differences between 1% and 5% user-preference samples across all methods and datasets
Dataset
Method
1% Acc
5% Ace
Improvements
Cot
14.49
14.72
+0.23
CM+Cot
19.85
23.22
+3.87
Rechangg
CM+PoT
FuT
22.52
14.73
26:56
14.61
+4.04
-0.12
CM-CCnnsitent
23.75
+1.01
CM+CCot
32.18
33.08
+0.90
CLINC150
Zero-Shul
ZeroShro
58.22
58.04
59.78
58.24
+.1.6
+0.20
Self-Refine
58.47
80:19
+1.72
CM+Sell-Refiee
+8.15
CM+Cot
22.14
+2.48
Banking77
CM+FFTT
25.45
29.00
+8.65
-0.38
Zero-Shot
48.70
49.03
40.33
CMrZarr-hhot
48.38
55.19
+6681
CM-SslI-Reline
+7.02
55.95
-0.20
CM+CCot
Mtop
CM-+FoT
67.28
65.50
1..8
-1.78
Consistent
62.75
63.43
+0.08
Consistntt
06.19
65.98
-0.21
Col
47.31
46.94
-0.37
CM+
+1.01
CM++FUT
Zero-Ehot
55.41
55.82
40.41
CM+
82.14
63.95
+1.81
62.10
63.85
+1.75
Table 7: Accuracy comparison between 1% and 5% user-preference samples. Bold indicates improvements >1%
9
How Does Consistent Marginalisation Handle Out-of-Domain Questions
While inputs are inherently froe-form, in our experimental sctup, the output space is predefined and discrete, where responses are mapped specific categories (e.g., banking intents, user queries in StackExchange We acknowledge, however, that in real-world settings, users may issue queries from new domains and exhibit previously unscen preferences. There are two possible scenarios for such new input cases:
(1) new inputs associated with existing predefined class,
(2) new inputs associated with new, previously unseen class.
16

---

Published in Transactions on Machine Learning Research (09/2025)
(I): In experiments, instance- wwwpooseeeeepeedeeeeeeeeeeeeeeee discrepancies are estimated based on user preference samples, which are drawn independently and identically distributed (i.i.d.) from the training distribution. We have verified that our method performs effectively on new inputs (from the testing distribution) that fall within the same set of predefined classes.
(2): Nonetheless, the new input comes from an out-of-domain distribution which is drastically different from the training data, we can adapt CM this case with the help of lightweight pretrained model. To support progressive personalization,
method can be designed to be continually updatable: When new imput New is given, which can be significantly different from in-domain datasets, can encoder to compute the similarity between the embedding of Xuew and those of existing uscr-preference samples with predefined candidate sets. If the similarity score is lower than a predefined threshold (e.g., 0.1), the system flags Xnew as a potential out-of-domain sample. The system then passes the query to the LLM without showing predefined classes. If user feedback provided either through explicit correction or satisfaction signals the user may reject the undesired response Y' and supply a preferred alternative Y. This instance response response pair incorporated into the memory MX) without retraining the LLM. If the user is satisfied, the memory bank records the preference aligned; otherwise, it logs the misalignment. This setup supports continuuu learning without fine-tuning and enables enhanced personalization via lightweight, per-user memory.
pretrained sentence
17


Reviews:
[{"review_id": "hyrxXCJRxp", "reviewer": "Reviewer_Aei3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The theory/notation, experiments, and example given in the introduction feel a bit decoupled. The example given in the introduction was that the language model should take into account the risk averse nature of the person when suggesting 80% investment in crypto currency. The method section then immediately defines response classes in the problem setup.  It seems that having a finite, defined set of response classes is a large assumption and more concrete scenarios where this applies should be given as grounding to this problem. The initial motivation is that user preference is not factual so it's a difficult problem, but then it seems like many user-preference scenarios wouldn't have a clearly defined set of response classes. In the investment example given, the output could be continuous from 0-100%, the categories for investment are likely an open set. Along this line it would also be helpful to have a figure which illustrates the experimental setup and some of the sample input, the memory banked, and the generated and ground truth responses from one of the datasets. \n\n- From  anecdotal experience it seems that current LLM’s with long context account for preference in long conversations like the example given. If the user says that they are risk averse in the context, then at least anecdotally I find that LLM’s condition their response on this type of information. It would be good to address this in the motivation of the introduction and potentially in the experiments why information in the context is not sufficient for user preference. \n \n- Overall the paper has thorough experiments and supports their claims with reasonable evidence. The method is interesting and appears to have strong empirical results, though I am not as familiar with the benchmarks and methods in this area. The weakness I would highlight are that the method section is difficult to parse, the method could be better grounded with some qualitative examples, and the problem could be better motivated in the introduction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new method Consistent Marginalization which is a technique for aligning LLM outputs with user-preference. The method constructs a memory bank based on discrepancies between LLM generated responses and user preference. The work demonstrates improved performance across a range of datasets and architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend using figure 1 to be more illustrative of the method and to ground it with real-world examples (input, generated response, ground truth, etc). Otherwise the method feels disconnected from the single example given of investment. The bottom portion of the figure in section 4.2 is also difficult to parse with the conditional statements inside of the matrix. Overall my recommendations are to improve the readability of the method section and improve the figures to make them more illustrative of what they are trying to convey. I think a figure which clearly demonstrates the method would really improve the paper. The changes are not critical to acceptance.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QF3YlKxEJl", "reviewer": "Reviewer_DHBS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is mostly in extensive experiments on a wider array of models, datasets and baseline algorithms.\n\nThe weakness are as follows, ordered by reviewer's level of concern:\n\n1) It's not clear how the matrix M can generalize to new input. All description assumes that X, Y and Y' are categorical in a finite set, where LLM input and output are freeform and can't be directly compared. The reviewer may have missed something but an average reader would too. \n\n2) The claim that LLM is ineffective in personalization is not convincing. With in-context learning (where you can put user preference as part of the prompt) and fine-tuning (esp. LoRA), it's widely accepted that LLMs can be personalized or contextualized with reasonable compute.\n\n3) The paper has large chunks of seeming redundant content, that can be safely trimmed off. For example most of page 6 and 7.\n\nOther less concerned weakness that could be easily fixed.\n\n1) In paragraph 1, the definition of \"hallucination\" is usually not the misalignment between desired and actual output.\n\n2) The paper should explain what is X, Y, Z at the very beginning with some concrete example, at least move 3.1 to the very beginning, or it's hard to get a concrete idea.\n\n3) Is Y at token level or response level? \n\n4) Page 8 second line of equation, what is this?\n\n5) Page 9 at the top: it's not practical to assume reliability from user-provided preference. There is always noise.\n\n6) Better annotate the experiment results with statistical significance (bold only the significant differences) and which one is state-of-the-art.\n\n7) How is CI computed?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of personalization in LLM. The key idea is to model the differences between model output and expected output as a probabilistic model, which contains the user input X, user truly desired output Y, the hidden context Z that affects X, Y and the model's output Y'. The paper argues that previous work usually considers dependency between Y' and Y, where this work also added dependency between X and Y'. To model the probabilistic distribution, it employs an adaption matrix that encodes user preference from past logs with golden user choice. Named as Consistent Marginalization (CM), the algorithm can be appled on top of other personalization methods.\n\nThe paper evaluates the performance of CM with a few popular models (gpt3.5, gpt4o-mini and Llama-8b-instruct) on several datasets (StackExchange, CLINC150, BANK77, MOTE and Massive Scenario). Evaluation shows that when added with CM, most existing algorithm has a better performance except in a few cases. The paper further performs ablation to understand the effects of different amounts of golden user preference used, claiming that even with a smaller amount (1% vs 5%) it still retains effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness part.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GdSuIxSmS5", "reviewer": "Reviewer_KmLT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Mostly positive results \n\n2. Approach shows some amount of generalizability to other LLMs\n\n3. A practical approach\n\nWeaknesses:\n\n1. One main point that I could not understand is why do we need to estimate $ \\sum_{Y'} Y' p(Y' | Y,  X)$? Isn't it always 1 by definition? Is there a notation issue here that I am missing? This appears in Eq. (4). \n\n2. If there is no need to estimate  $ \\sum_{Y'} Y' p(Y' | Y,  X)$ why the paper mentions this component? I see that what is actually estimated is the memory matrix $M$ which is simply done by counting and normalizing the co-occurences of response $i$ and $j$\n\n3. From the matrix definition of the memory, I think that the alignment correction can be only useful if the test set covers similar types of questions as the training set. If that is the case, this is a major limitation where we cannot ask an out-of-domain question and add a new preference and expect alignment correction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of the paper is to resolve user preference and LLM output misalignment issues. To do this, the paper proposes estimating a consistent marginalization term by using a memory bank describing the user preferences. The personalized memory bank is a binary matrix where the (i, j)-th element is 1 if the i-th misaligned response leads to j-th user-aligned response. During inference, the matrix can be utilized as a state transition matrix to reach to the aligned response from the unconditioned LLM response. Experiments have been performed on several LLMs and tested on 5 human preference alignment tasks. In the tables, the results show positive improvements in most cases except the ChatGPT-40-mini experiments where the re-aligned response perform at about same accuracy level as the unaligned responses (better baseline, improvements remain limited).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please review the formulation of the equations. Do we really need to estimate $ \\sum_{Y'} Y' p(Y' | Y,  X)$?", "broader_impact_concerns": "Probably no specific problem is introduced by the model itself other than the general issues with LLMs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)