Title: The Burden of Interactive Alignment with Inconsistent Preferences
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human AI Interaction

Alignment

Strategic Behavior

Stackelberg Equilibrium
Abstract: From media platforms to chatbots, algorithms shape how people interact, learn, and discover information. Such interactions between users and an algorithm often unfold over multiple steps, during which strategic users can guide the algorithm to better align with their true interests by selectively engaging with content. However, users frequently exhibit inconsistent preferences: they may spend considerable time on content that offers little long-term value, inadvertently signaling that such content is desirable. Focusing on the user side, this raises a key question: what does it take for such users to align the algorithm with their true interests?

To investigate these dynamics, we model the user’s decision process as split between a rational "system 2" that decides whether to engage and an impulsive "system 1" that determines how long engagement lasts. We then study a multi-leader, single-follower extensive Stackelberg game, where users, specifically system 2, lead by committing to engagement strategies and the algorithm best-responds based on observed interactions. We define the burden of alignment as the minimum horizon over which users must optimize to effectively steer the algorithm. We show that a critical horizon exists: users who are sufficiently foresighted can achieve alignment, while those who are not are instead aligned to the algorithm’s objective. This critical horizon can be long, imposing a substantial burden. However, even a small, costly signal (e.g., an extra click) can significantly reduce it. Overall, our framework explains how users with inconsistent preferences can align an engagement-driven algorithm with their interests in a Stackelberg equilibrium, highlighting both the challenges and potential remedies for achieving alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a model to study how users with inconsistent preferences (i.e., choices don’t always represent true preferences) can align algorithms with their true interests. The interaction is modeled as a Stackelberg game where the user is the leader, committing to a strategy, and the algorithm is the follower. The user has a “system 2” for engagement decisions and a “system 1” for the duration of such engagement, while the algorithm maximizes engagement. They define the \"burden of alignment\" as the level of foresight a user needs to steer the algorithm effectively (horizon). Their theoretical results show that foresighted users can improve alignment, which can be challenging, and the burden can be reduced with a costly signal for the user (e.g., clicks to show disinterest). Otherwise, myopic users align with the algorithm’s objective.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a great paper. It offers a rigorous theoretical framework of interactive alignment by modeling the user-algorithm dynamic as a Stackelberg game with the user as a leader. Its primary strength lies in departing from the simplifying assumption of user rationality and instead focusing on what users can do when they carry the burden of alignment by interacting with an engagement-maximizing algorithm. \n\nIt provides theoretical guarantees under well-defined assumptions: abstracting the algorithm’s learning process, system 2 decides engagement and system 1 its duration, the algorithm maximizes engagement, users are aware of their rewards, the algorithm has full knowledge of strategies, etc.\n\nWhile the paper frames the problem of inconsistent preferences in general terms, this same setup applies to the real phenomenon of doomscrolling. I’m not pointing this out as a limitation or a weakness, but maybe as something to consider. This theoretical framework could be used to study this effect—from the user and algorithm perspective—to improve users’ well-being. I see that as a strength.\n\nIn the paper, the user is modeled by their type $\\theta \\in \\Theta$ for a specific session, allowing the same user to be of different types across sessions. However, the user’s foresight $\\gamma_H$ is fixed, and it might be interesting to consider a dynamic discounting factor for a user. In reality, a person's ability to be foresighted can fluctuate based on internal factors. For example, a user who is typically strategic might enter a myopic state (doomscrolling) for some sessions. I wonder what the burden of alignment is in this situation. Would it degrade over time in this tug of war, where your myopic sessions ruin your strategic ones? Would the costly signals be able to reduce the burden? What happens when a lot of users doomscroll at the same time (sending a strong signal to the algorithm) during moments of political turmoil or catastrophes? I could also see how a user can resist the temptation of item `a` but not if this item is shown repeatedly.\n\nQUESTIONS:\n- Line 167: As a minor notation detail, it would be helpful for clarity to define acronyms for random entry (RE) and algorithmic entry (AE).\n- Line 302: Something is wrong with this sentence.\n\nLIMITATIONS:\nYes, in the Discussion section, and other assumptions throughout the manuscript", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors study an Nash equilibrium of a game where two parties, 
(recommendation) algorithm and user, interact.  In this setting algorithm
recommends content, which user then may be engage for certain time. The user is
then rewarded afterwards.  The objective of the algorithm is to maximize the
engagement of the user, whereas the goal of each user is to maximize the
reward.

STRENGTHS AND WEAKNESSES:
+ Interesting results regarding the equilibrium, especially with inconsistent rewards. These contributions advance the field, and are sufficient for a publication.

- The contributions are all theoretical. The main downside with the paper is that it does not have any experiments or any use case study. It is not clear what are the practical implications of these results. Some experiments or a small use case study would alleviate these issues.
- The authors motivate their example using recommendation algorithms. Yet the related work
  in that domain is not thoroughly discussed. The authors should have discussed the recommendation algorithm in the related work more thoroughly.
- The paper focuses on cases with two items. The results should be extendable, 
  so it would be helpful to see a more general results, especially for Theorem 4.1.

QUESTIONS:
- Are these results (easily) extendable to n items?
- What are the practical implications for recommendation engines?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a setting where humans sometimes act in ways that are systematically at odds with their true preferences (following the system 1 system 2 model in Kleinberg & Raghavan). An algorithm that aims to maximize engagement is attempting to suggest content to a human, where some humans have preferences aligned with the algorithm’s prior, and some have preferences at odds with it. This paper studies when humans can effectively strategize to end up being suggested the content that they prefer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThis paper studies an interesting problem, and ties its work to prior work both in practice(Haupt et al and Cen et al) and theory (Kleinberg & Raghavan). I appreciated that the paper included several extensions (e.g. including ‘costly signaling’, which was a variant that I hadn’t immediately thought of). \n\nWeaknesses: \nA few parts of the model felt unclear. For example, one crucial factor is the human’s time horizon, which influences their strategy as it influences how long they can take the costly action of signaling a different preference: in section 5, it is shown that if a user has a low enough time horizon, then they may incur relatively heavy regret. However, in real-life settings, users may have relatively short time horizons precisely because they know they will not be using the app for very long: for example, if a user installs an app that they know they will only use for a short period of time (e.g. in a country that they infrequently visit), then they may (rationally) decide that they don’t want to invest the effort in steering what the app may suggest to them. By the results in this paper, this would imply a *high* regret, but in reality this would likely lead to a low regret. In this paper, I believe the authors are focusing on the case where a consumer’s time horizon is driven solely by how much effort they wish to put in (or how myopic they are), rather than any actual desire to leave the platform. While I don’t think this substantially changes the core of the paper, it would be useful for the paper to note the different reasons why users may have different time horizons. \n\nAs one other critique, the paper is very theory-heavy - which I appreciate, as its main goal is clearly a theoretical understanding. However, I would have appreciated more explicit connection to practice or policy, even in the appendix. For example, what would the results in Section 6 imply about how algorithms should be designed? How would the results change if algorithms have different incentives beyond engagement (e.g. different values for items)?\n\nQUESTIONS:\nHighlighted in the \"weaknesses\" section - in particular, my questions about the meaning of the user being more or less myopic, and the policy implications.\n\nLIMITATIONS:\nYes - but more discussion (and connection beyond theory) could be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecommendation algorithms learn user preferences using user behavior, which may be rational (System 2) or short-sighted/greedy (System 1). Due to the greedy behavior, the algorithms may be unable to distinguish between users whose rational selves agree/disagree with their greedy selves. To address this, users must strategically choose their rational behavior so as to signal their type to the algorithm. This paper develops a theoretical model of this setting as a Stackleberg game, and analyzes Bellman equations to show that (a) users may need to substantially sacrifice / alter their behavior in order to \"steer\" the algorithm, (b) users who highly discount future rewards will be unable to steer the algorithm at all, and (c) a costly signal may mitigate the burden to users.\n\nSTRENGTHS AND WEAKNESSES:\nI enjoyed reading this paper; the paper was well organized, and the authors provided clear and intuitive interpretations of the theoretical results, which appear non-trivial. Recent literature has examined both strategic and inconsistent content consumption, but this is a novel bridge between these lines of work. The findings are also interesting - users may potentially require great energy and foresightedness to align the algorithm due to inconsistent preferences, and sometimes it is in fact impossible to align.\n\nOne weakness is that it's not clear what setting motivates this work: in what type of content recommendation does this phenomenon (users strategically determining which content to consume, but then greedily determining how long to consume it for) occur? In particular: in what settings/platforms does a user consume a *particular* piece of content for an undefined length, and the platform wants to maximize this length of time.\n\nMoreover, while the model is quite general, the results are all in a special case where there are two items.\n\nI also have a few quibbles/clarifications about the modeling decisions, which I ask about below. It would also be interesting to have additional interpretable insights from the model, such as discussion of which types of users the burden of alignment is highest for (with and without costly signaling).\n\nQUESTIONS:\n1. What recommendation setting motivates this work?\n\n2. How restrictive the special case in 3.4? Does it capture most settings? Do you have a sense of how the results extend (steering becomes easier/harder) beyond this special case?\n\n3. Is a \"show less like this\" button seen on a few platforms currently an example of a costly signal? What would motivate a platforms to implement a costly signaling feature-- is there a burden on the platform as well?\n\n4. Modeling questions:\n- On a session-level, why would the algorithm discount future rewards?\n- When computing the algorithm's Q-value, why does the algorithm assume the users will continue to use the same strategy $f$ in the future?\n- Is pulling the expected duration into the reward $r_\\theta$ (in line 120) really without loss of generality? Does it result in some additional dependence on $\\gamma_H$?\n- What does \"when users have consistent preferences ... the alignment problem reduces to engagement maximization\" mean on page 1? My intuition is that there is misalignment even with only System 2 (e.g. some content could have a reward only after time $T$ has been reached, like a suspenseful movie, and the algorithm is motivated to promote that content). Is this misalignment possible in general or only because I chose pathological rewards? If it can occur in general, do you have a sense of how hard it is for a user to steer the algorithm without inconsistent preferences (as a baseline)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}