Title: Adaptive Preference Arithmetic: A Personalized Agent with Adaptive Preference Arithmetic for Dynamic Preference Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Personalized LLM Agents

Dynamic Preference Modeling

Adaptive Preference Arithmetic
Abstract: As large language models (LLMs) are increasingly used as personalized user assistants, effectively adapting to users' evolving preferences is critical for delivering high-quality personalized responses. While user preferences are often stable in content, their relative strengths shift over time due to changing goals and contexts. Therefore, modeling these dynamic preference strengths can enable finer-grained personalization. However, current methods face two major challenges: (i) limited user feedback makes it difficult to estimate preference strengths accurately, and (ii) natural language ambiguity limits the controllability of preference-guided generation. To address these issues, we propose AdaPA-Agent, a LLM-agent personalization framework that models dynamic preference strengths via Adaptive Preference Arithmetic. First,  instead of requiring additional user feedback, AdaPA-Agent employs an alignment-based strength estimation module to estimate the strength of user preferences from the existing user-agent interaction. Then, it guides controllable personalized generation by linearly combining next-token distributions, weighted by the estimated strengths of individual preferences. Experiments on two personalization tasks-conversational recommendation and personalized web interaction-demonstrate that AdaPA-Agent better aligning with users' changing intents, and has achieved over 18.9\% and 14.2\% improvements compared to ReAct, the widely-used agent framework.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AdaPA-Agent, a framework for personalizing large language model (LLM) agents by dynamically estimating and leveraging the relative strengths of multiple user preferences. Rather than relying on explicit user feedback, AdaPA-Agent infers preference strengths from user-agent interaction history using a combination of dual-side augmentation (enriching both preference representations and interaction paraphrases) and an LLM-based alignment scoring mechanism. The estimated strengths are then used to guide the agent’s responses by linearly combining the next-token distributions conditioned on each preference. Experiments on conversational recommendation and personalized web interaction benchmarks show that AdaPA-Agent achieves higher personalization quality and efficiency compared to several strong baselines, without requiring additional model retraining or explicit user input.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**S1.** The paper proposes a training-free framework that models dynamic user preference strengths. This approach is highly practical and resource-efficient as it does not require costly fine-tuning or model retraining, making it a flexible solution for adapting to user preferences in practice.\n\n**S1.** The paper proposes a novel approach to estimating preference strengths by combining dual-side augmentation (preference chain construction and interaction paraphrasing) with LLM-based alignment scoring. This enables the extraction of implicit preference signals from dialogue without requiring explicit user feedback.\n\n**S2.** The method demonstrates substantial and consistent performance gains over competitive baselines such as ReAct, Reflexion, and RecMind on both conversational recommendation (Reddit-Movie) and personalized web agent (PersonalWAB) benchmarks. Improvements are shown in both personalization quality (e.g., recommendation success rate) and efficiency (e.g., fewer interaction rounds).\n\n**Weaknesses**\n\n**W1. Adaptivity to dynamic preferences.** While Figure 3 and Section 5.4 show changes in inferred preference strengths, this does not demonstrate true adaptivity to dynamic preference shifts. In my understanding, the framework simply re-computes strengths from scratch at each interaction—functionally equivalent to applying a static estimator repeatedly—rather than maintaining or updating preferences in an adaptive way. As a result, the method passively reflects recent inputs instead of adaptively detecting or responding to preference shifts within a session. Could the authors clarify how AdaPA-Agent can *detect* or quickly adapt to user preference shifts within an *ongoing* multi-turn conversation, rather than simply re-estimating preferences after each isolated interaction?\n\n**W2. Omission of PUMA baseline.** While the paper uses the PersonalWAB [(Cai et al., 2024)](https://arxiv.org/abs/2410.17236) for evaluating personalized web agent performance, it does not include PUMA—the core approach introduced with PersonalWAB—in its baselines. This omission is not discussed or justified. Since PUMA is specifically designed for this benchmark and serves as a state-of-the-art method, a fair comparison requires evaluating AdaPA-Agent against PUMA on the same tasks. The absence of this baseline makes it difficult to assess true progress over prior work; the paper should either include direct comparisons to PUMA or clearly justify its exclusion.\n\n**Minor comments**\n\n**C1. Additional related works.** The controllable personalized generation proposed here resembles recent methods such as [Zhu et al. (2025)](https://arxiv.org/abs/2502.14204) and [He et al. (2024)](https://arxiv.org/abs/2405.01768), but the paper does not discuss similarities or differences. Clearer positioning relative to these works would strengthen the contribution. Also, recent studies on LLM personalization via preference strength estimation, such as [Oh et al. (2024)](https://arxiv.org/abs/2411.00524) and [Hwang et al. (2024)](https://arxiv.org/abs/2312.09337), could be included in Section 2.2 to provide more accurate context. Lastly, “LLM Personalization” seems a quite broad category; I recommend narrowing the scope for a more explicit and meaningful comparison.\n\n**C2. Robustness of CoT-based augmentation.** The CoT-based augmentation process relies on LLM-generated examples to concretize user preferences. However, these examples may not always align with the user’s true preferences (e.g., “prefer healthy food” may not always mean “prefer salads”), introducing spurious associations. Additional clarification, analysis, or robustness experiments regarding this aspect would strengthen the reliability and stability of the proposed framework.\n\n**C3. Ambiguity in preference dimensions.** While the manuscript states that the authors considered \"multiple active preferences\" for web tasks, it does not specify the exact number or types of these preferences. For the paper to be self-contained, it is beneficial to clearly describe the preference dimensions used in the experiments.\n\n**References**\n\n- Zhu et al. (2025). *On-the-fly Preference Alignment via Principle-Guided Decoding*\n- He et al. (2024). *Context Steering: Controllable Personalization at Inference Time*\n- Oh et al. (2024). *Comparison-based Active Preference Learning for Multi-dimensional Personalization*\n- Hwang et al. (2024). *Promptable Behaviors: Personalizing Multi-objective Rewards from Human Preferences*\n- Cai et al. (2024). *Large Language Models Empowered Personalized Web Agents*\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for large language model (LLM)-based agent personalization called AdaPA-Agent, which achieves more fine-grained personalized response generation by modeling the dynamic strengths of user preferences. The authors point out that although existing methods can already capture the content of user preferences, they fail to effectively model how the relative strengths of these preferences evolve over time. To address this, they propose \"Adaptive Preference Arithmetic,\" which consists of two key modules. The first module, Alignment-Based Strength Estimation, automatically estimates preference strengths from existing user-agent interactions without requiring additional user feedback, using dual-side augmentation and an LLM-based alignment scorer. The second module, Controllable Personalized Generation, controls the generation process by linearly combining next-token distributions conditioned on different preferences, weighted according to their estimated strengths. The framework is evaluated on two tasks—conversational recommendation and personalized web interaction—and the results demonstrate that AdaPA-Agent outperforms baseline methods such as ReAct across multiple metrics, achieving better alignment with user needs when intentions change over time.\n\nSTRENGTHS AND WEAKNESSES:\nStregthens\n1. The authors propose a training-free approach to align with dynamic human preferences for personalization, enabling better alignment with user needs when intentions change.\n2. AdaPA-Agent is a training-free method that is applicable to various LLM architectures, offering strong versatility and deployment flexibility.\n\nWeakness\n1. The method is overly intuitive and experience-based, lacking theoretical foundations.\n2. The method calculates preference scores through model computations, but the reliability of these scores from large language models has not been sufficiently validated.\n3. Using the CoT prompting approach, the reasoning time may become excessively long, negatively impacting the user experience.\n4. The paper does not explicitly address how the model balances or resolves conflicting preferences when users express contradictory ones, nor does it explore whether impulsive or erroneous user inputs may affect the final generation outcomes.\n\nQUESTIONS:\nWhy is this weighting approach in Eq. 5 effective? Is it possible to find a theoretical basis behind it?\n\nLIMITATIONS:\nAlthough the \"Broader Impacts\" section is mentioned in the appendix, it is relatively brief and lacks in-depth analysis of potential risks such as privacy leakage, preference amplification bias, and algorithm misuse.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an LLM-agent-based framework for inferring user preferences by aligning preference analysis with the user-system interaction context. Strength weights for different preferences are estimated using an LLM, which assesses how well each preference aligns with the current interaction context. These weights then guide the redefinition of the next-token distribution through a weighted sum of distributions conditioned on various preferences. Experiments conducted on two distinct tasks demonstrate performance improvements compared to selected baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written, clearly presenting most methodology details.\n\n2. The proposed method is concise and user-friendly. The authors provide their code and data, facilitating reproducibility and futher research.\n\n3. Experimental are conducted on two representative personalization tasks, broadening its potential impact and applicability.\n\nWeaknesses:\n\n1. To my understanding, the effectiveness of the proposed method should largely depend on the extraction of users' existing known preferences, as both interaction alignment and distribution re-weighting hinge upon these preferences. However, the extraction methods for these preferences appear handcrafted and guided by intuitive empirical rules. For instance, while the paper states, \"we extract stable preferences from historical data as long-term and select unrelated movies to define short-term preferences,\" the exact methods of extraction and definition remain unclear and appear empirical and naive. Additionally, the influence of different preference definitions and extraction techniques on the performance of the proposed AdaPA-Agent is important but remains unexplored.\n\n2. Although experiments encompass tasks such as conversational recommendation and personalized web interactions, the selection of baselines neglects relevant works specifically targeting these domains. The paper currently compares mainly against general agentic reasoning approaches and general recommendation methods employing LLMs. Important recent studies on using LLMs in conversational recommendation systems, such as \"Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems,\" \"Towards Empathetic Conversational Recommender Systems,\" and \"Large Language Models as Zero-Shot Conversational Recommenders,\" are strongly suggested to be included in both discussions and comparisons.\n\n3. Some experimental settings lack clarity or justification. For example, the paper does not clarify the number of simulated users in the conversational recommendation experiments. It is also unclear how standard variances, if presented, are calculated in Table 1 and why they are missing in Tables 2 and 3. Further, detailed configurations, such as the temperature settings for LLMs, should be clearly articulated in the appendix. Lastly, excluding user interaction records from simulator construction diverges from primary practices in existing literature, potentially limiting the realism and complexity of the simulated user behaviors.\n\nQUESTIONS:\nAll my questions are given in the weakness points.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a way to personalize LLMs based on user-LLM interaction. The approach does not perform fine-tuning. It first extracts certain preferences from the user interactions (it is unclear to me how the preference $p^{raw}_i$ is extracted, but I am guessing it is by prompting LLMs on the interaction history). It then measures the weight of each preference chain by calling an LLM-based scoring function on the interaction and preference chain. Finally, the next-word probabilities are computed for each preference separately and then combined using the given weights. My understanding is that this process is repeated at each step of the user-LLM interaction.\n\nExperiments are conducted on domains from the personalized web agents task. Results show that the approach performs better than baselines like React and Reflexion.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- User personalization is an important task and this approach provides a fine-tuning-free approach for personalizing LLM. Fine-tuning free is nice because fine-tuning is both data and compute-intensive. \n \n- The approach performs better than common baselines\n\n**Weakness:**\n\n- **Expressivity:** Mixing token-wise distributions may not be expressive. Instead, an approach that expresses each preference and its weight (such as strong/weak) in text and then prompts the LLM with it might be better as the LLM can then combine this knowledge in more expressive way.\n\n- **Inference Time:** The approach might make too many LLM calls per interaction. There is one call for score per user preference, and a few for extracting preference. Inference time matters because in practical applications, the agent cannot wait too long before replying. This would be even worse with heavier LLMs.\n\n- **Related work:** paper misses some related work discussion on personalizing LLMs such as:\n\n 1. Aligning LLM agents by learning latent preference from user edits, Gao et al. (NeurIPS 2024)\n 2. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers, Mysore et al., ACL 2024.\n\nQUESTIONS:\n1. What is the inference time of this approach? Can you report how much slower is the AdaPa agent over the baseline greedy decoding agent?\n\n2. How is the original preference $p^{raw}_i$ computed? And does $p^u_i$ indicate the concatenation of $p^{raw}_i, p^{ref}_i, p^{exp}_i$ and are these done in one single LLM call?\n\n3. How will the baseline where instead of doing token-wise mixing, the list of preferences and a categorization of their weights (e.g., weakest, weak, neutral, strong, strongest) is provided to the LLM, and a single call is made to the LLM? This will reduce the inference time while giving more expressivity.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for agent conversational recommendation considering dynamic preference strength.\nWhen interacting with an agent given the user’s preference history,  this paper investigates the case where the user’s preference strength over each dimension changes. \nIt (1) paraphrases the history preference sentences into their proposed format called preference-chain (so there are several chains for each user); (2) paraphrases the new user input into multiple versions; (3) uses an LLM to assign an alignment score between the history preference-chain and each paraphrase of new user input ; (4) prompts an LLM to generate the next token distribution given the LLM’s response’s prefix, the current conversation, the user’s history preference-chain; (5) weighted sum of the distribution given each history preference-chain.\nThe experiments are on movie recommendation agents and personalized web-service agents. \nThe method is training-free and performs well the the above 2 tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The method is training-free, making it easy to test / apply. \n- The description of the method is intuitive, and the illustrations are very helpful.\n\nWeaknesses\n- The usage of weighted sum for each output token distribution is not well justified. Why is this a good idea? In certain toy settings, can we theoretically understand what this is doing and why it is a good idea?\n- The method is not tested on a real-world dataset.\n- The writing is of the paper can be clearer.\n\nQUESTIONS:\n- Could you provide examples of the original prompted responses of each preference and the weighted generation during conversation?\n- Did you try comparing the proposed method with directly prompting the LLM with the preference chains and scores for conversation?\n- Could your method deal with cases where the current conversation preference contains dimensions that are not previously stated in history?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}