The Burden of Interactive Alignment with Inconsistent Preferences
Authors: Ali Shirali
Venue: NeurIPS 2025 poster ()
Keywords: Human AI Interaction
Alignment
Strategic Behavior
Stackelberg Equilibrium
Primary area: theory
TL;DR: Our framework explains how users with inconsistent preferences can align an engagement-driven algorithm with their interests in equilibrium, highlighting both the challenges and potential remedies for achieving alignment.

Abstract: From media platforms to chatbots, algorithms shape how people interact, learn, and discover information. Such interactions between users and an algorithm often unfold over multiple steps, during which strategic users can guide the algorithm to better align with their true interests by selectively engaging with content. However, users frequently exhibit inconsistent preferences: they may spend considerable time on content that offers little long-term value, inadvertently signaling that such content is desirable. Focusing on the user side, this raises a key question: what does it take for such users to align the algorithm with their true interests?

To investigate these dynamics, we model the user’s decision process as split between a rational "system 2" that decides whether to engage and an impulsive "system 1" that determines how long engagement lasts. We then study a multi-leader, single-follower extensive Stackelberg game, where users, specifically system 2, lead by committing to engagement strategies and the algorithm best-responds based on observed interactions. We define the burden of alignment as the minimum horizon over which users must optimize to effectively steer the algorithm. We show that a critical horizon exists: users who are sufficiently foresighted can achieve alignment, while those who are not are instead aligned to the algorithm’s objective. This critical horizon can be long, imposing a substantial burden. However, even a small, costly signal (e.g., an extra click) can significantly reduce it. Overall, our framework explains how users with inconsistent preferences can align an engagement-driven algorithm with their interests in a Stackelberg equilibrium, highlighting both the challenges and potential remedies for achieving alignment.

The Burden of Interactive Alignment with Inconsistent Preferences
Ali Shirali
UC Berkeley
Abstract
From media platforms to chatbots, algorithms shape how people interact, learn, and discover information. Such interactions between users and algorithm often unfold over multiple steps, during which strategic users can guide the algorithm to better align with their true interests by selectively engaging with content. However, users frequently exhibit inconsistent preferences: they may spend considerable time content that offers little long-term value, inadvertently signaling that such content is desirable. Focusing on the user side, this raises a key question: what does it take for such users align the algorithm with their true interests?
To investigate these dynamics, model the user's decision process as split between a rational "system 2" that decides whether to engage and an impulsive "system 1" that determines how long engagement lasts. We then study multi-leader, single- follower extensive Stackelberg game, where users, specifically system 2, lead by committing to engagement strategies and the algorithm best-responds based on observed interactions. We define the burden of alignment as the minimum horizon over which users must optimize to effectively steer the algorithm. We show that a critical horizon exists: users who are sufficiently foresighted can achieve alignment, while those who are not are instead aligned the algorithm'' 's objective. This critical horizon can be long, imposing substantial burden. However, even small, costly signal (e.g., an extra click) can significantly reduce it. Overall, our framework explains how users with inconsistent preferences can align engagement-drrven algorithm with their interests in a Stackelberg equilibrium, highlighting both the challenges and potential remedies for achieving alignment.
1
Introduction
We study the interactions between human users and an algorithm over multiple steps. While these interactions may benefit both parties, the user and the algorithm can have misaligned interests. Our focus is on the user side, asking what takes for user to align the algorithm with her interests. We explore this alignment problem in interactive environment where users may exhibit inconsistent preferences within an incentive-aware framework. The following explains our setting in detail.
Our setting and model. When users have consistent preferences- i.e., the engagement length is in proportion user's true reward-the the alignment problem reduces to engagement maximization. Recent advances in designing eestructionfoolloiing language models rely on this assumption of consistent preferences, often using models such as Bradley-Terry
directly or indirectly infer human rewards and optimize them to maximize human approval 2 Similarly, recommender systems are typically designed to optimize recommendations that maximize user engagement (5).
Users can, however, have inconsistent preferences, where their actions may not reflect their true interests. This often occurs when user's decision results from combination of impulsive system 1 and rational system processes (-8), or when consumption choices are influenced by both long-term benefits (enrichment) and desire for instant gratification (temptation) 9 In such cases, revealed preferences do not necessarily align with the user's 's true preferences.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In general, due to inconsistent preferences or misaligned objectives [20 the algorithm's 's goal may diverge from the user's. Can a strategic user then still succeed in aligning the algorithm with her interests? This question is central to our study, where we examine the conditions and strategies that enable users to steer the algorithm despite such misalignments.
We model the alignment problem as a multi-leader, single-follower Stackelberg game, where users act leaders by committing strategies, and the algorithm best responds. We group users with similar interests under the same user type to capture preference heterogeneity. Our model reflects real-world settings in which users can initiate or terminate interactions, and excludes scenarios such as strategic classification B [4] where the algorithm first commits to a strategy, such as classifier. By assuming the algorithm best responds, we abstract away its learning process [6]. This is reasonable as modern systems interact frequently with users and learn from abundant behavioral data.
Our Stackelherg game captures dynamic setting where a interacts with the algorithm over multiple steps within a session. After each interaction, the algorithm updates its posterior over the user's type, gradually personalizing its responses. As a result, the strategies of other users affect how well an individual can steer the algorithm. The user's goal is to guide the algorithm toward higher-reward recommendations over time. However, she must trade off long-term signaling with short-term rewards, creating a tension that places the burden of alignment on the user.
Our results. To quantify the burden of alignment on the user, ask: over what future horizon must the user optimize her strategy for the cost of alignment to be worthwhile? We propose this horizon as measure of burden. When users exhibit inconsistent preferences, system 2 must be sufficiently foresighted to align the algorithm with user's true interests. We show that each user type has a critical threshold: only if the user's horizon exceeds this can she steer the algorithm; otherwise, similar to [17]. the algorithm effectively aligns her to its own objective, despite the user leading the game.
The significant burden users prompts the question: what design features can ease alignment? We explore a setting where users can exert observable effort during each interaction-suce as clicking small, non- non-benetee beneficial button- as way to signal system 2's strong disinterest. Though the effort yields no direct reward, it enables users to convey preferences without disengaging, simplifying their strategies. As in the bascline setting, alignment still requires critical optimization horizon. However, allowing users to "burn effort" as a signal [18] substantially reduces the burden of alignment.
In summary, we present a framework to analyze the alignment problem between an optimal algorithm and users with inconsistent preferences. By fully characterizing the equilibria Stackelberg game, where users lead, we quantify the burden of alignment in terms of the horizon over which users must optimize. Our framework also guides the design of improved alignment environments by quantifying how costly signaling through effort-burning can reduce this burden. As noted by Dean et al. [9]. we believe that such formal mathematical models can provide valuable insights for the broader field of interaction design in future studies.
multi-leader
Additional related work. The alignment problem has been studied across computer science, social science and economics setting is a Stackelberg game in which the leader- typically the platform matform--commis to a strategy over multiple interactions, and the follower- a user responds. Because recommendations unfold over multiple steps, users may act strategically, anticipating how their responses influence future outcomes. Haupt et al. (20) term such users strategic, in contrast to myopic users who optimize locally. Similar to our work, they show that users tend to engage in behaviors that accentuate differences relative to users with other preference profiles. Haupt et al. 220) and Cen et al. 120 provide empirical support for this behavior, while Cen et al. [22] study its implications for platform utility.
game between users and platrorms/algorithns. The most closely related
Our work departs from these studies in two key ways. First, we center users as leaders in the Stackelberg game, focusing on settings where users users-particul particcualllll system 2 can commit to strategies that the platform observes (e.g., via repeated interactions or prior data). This shift allows to ask what it takes for users, as leaders, to maximize their own rewards. Second, we extend the analysis to a multi-leader, single-follower game and examine how the presence of other users shapes individual strategies, drawing parallels to Nash equilibrium. Please refer to Appendix A for an extensive review of related work.
2

---

2 Model
We model the interactions between an algorithm and users. The users aim find specific responses or recommendations, while the algorithm seeks to maximize user engagement. We first outline the structure of these interactions and then present detailed model for both the users and the algorithm.
The user-algorithm interactions occur in sessions. Each session involves multiple interactions between the algorithm and a fixed user. In cach interaction, the algorithm suggests an item/content and observes whether the user engages with and for how long. The algorithm uses these observations to refine future suggestions. For users, engagement generates a reward, which is higher when the item aligns with their true interest. We assume users have fixed interests throughout a session.
For strategic users, engagement serves dual purpose: consuming rewards and signaling preferences. If a rational user fully controlled both engagement and its duration, aligning with the algorithm could be simple engaging more with preferred content. However, users often exhibit inconsistent pref- erences, sometimes spending time on content misaligned with their true interests. This inconsistency complicates alignment and motivates our study.
2.1 Model of human users
We model a user by their type 0 E O, which encodes the user's intention during a session. Thus, the same person may have different types across sessions. For simplicity and tractability, we focus on the case where the set of user types θ is finite.
Upon receiving a recommendation s E S. the user makes two decisions: whether to engage, and so, for how long. Let y Y C R denote the user's response, where y >> 0 indicates engagement lasting y steps. We model human users with potentially inconsistent preferences. The decision to engage is governed by the user's fully rational system 2. If the user chooses to engage, the length of engagement is determined by system 1, with an expected value of EEyy y > 0; 0, s] = 1/o0(s))
Note that in our analysis of optimal decision-maker makingg the expected value of y the only important factor, which we can interpret as the expected utility of the algorithm when a user of type engages with content s. When y follows geometric distribution with success rate of (sa(s), our model reduces to that of Kleinberg et al. 66
The user receives an expected reward of (s(s) upon engagement. This reward is independent of engagement length, without loss of generality: even if realized rewards depend on duration, the length is independently governed by system 1, its effect can be absorbed into the expectation re(s).
The user discounts future rewards by a factor "TH < per time step This parameter is central to our analysis of the foresight required for a user to align the algorithm with her interests. We assume uniform Y/ across all users, though our analysis readily extends to a heterogeneous population.
2.2 Model of algorithm
We assume the algorithm maximizes engagement, a common proxy for objectives in human-facing systems; e.g., video recommendation engines optimize for prolonged viewing to increase ad revenue. Our framework also extends beyond engagement: if is the algorithm's 's utility from an interaction, the algorithm be viewed utility maximizer. This general perspective encompasses any policy or language model that interacts with users to optimize a utility measure, such as human approval.
Formally, let H (s1, Vi, ST, yT) represent the history of interactions in a session. The algorithm's realized utility from H is ΣT Vr. However, like human users, the algorithm may also be myopic, discounting future returns by factor of 7A < per step. Thus, the algorithm's 's valuation of H at the start of the session is DT -t-1 7 next define their respective strategies and introduce a notion of equilibrium.
1-1 yt- Given the models of the users and the algorithm, we
'To further clarify the connection to Kleinberg et al. 66. we treat the entire interaction that system has after system 2 decides to engage as a single item. While the user may consume different things during engagement, from system 2's perspective, the only relevant parameter is the expected reward as we will see, and from the platform's 's perspective, it is the expected length of engagement. What exactly gets consumed during the engagement does not reveal information about the user's type, since it wasn't the result of rational/strategic decision and the platform only records engagement decisions as we shall see.
We have implicitly assumed that users might discount rewards between interactions, but not during single interaction, previous point the irrelevance of the length of the engagement and reward remains intact.
3

---

Strategies and equilibria
The algorithm's strategy и (S x Y)* maps the session history (81, ; yl, 82, U2,...) to the next recommendation in general. On the user side, only the engagement decision is strategic: user of type 8 engages with content s with probability fo(s). We denote the user strategy profile by f == (fee (ee. We now characterize the values induced by these strategies, formulate the optimization problems faced by mallu-maximizing users and the algorithm, and introduce the resulting equilibria.
3.1 Algorithm's strategy and value
For simplicity, we assume that the algorithm only acts on the binary engagement history A ($1. 01 {y 0}. $2: 1/2 132 > O)....) rather than the full history H. Suppose the algorithm has prior À (Ao) over θ The Q-value of the algorithm starting with content is
Qu(A, s: E
( EH-(so
Σ
2 ml
(Gr.3)))
Here, (fe, I s) is shorthand for the distribution over the possibly infinite-length history induced by the user strategy So and the algorithm strategy n, starting with
Although the algorithm's 's policy may depend on the full history H, optimal decision decision-making depends only the posterior which we prove in Lemma B. for completeness. Overloading notation, we define (A, s; f) = max, QA s; f. T). The Bellman update then allows compute QA 85
F7: over user types. This follows directly from the Bellman updates,
recursively as
So(s) (s)))) + YA (er)Be((f(s))) max QA (1x (s, (s.
QA(A, 8: f) = Ea->
(I)
This plays a pivotal role in our analysis of the best strategies. Once QA is found, the optimal policy is
x (H) E arg max Qa((A H;
3.2 User's strategy and value
The user's strategy specifies the engagement probability fo : S - [0, 1] over content space S. Given the algorithm's 's strategy and initial content s, the Q-value for user of type 0 is
Qa(s: f. = EH~(fo.7)))
Σ (SEA)))
1l > 03 7 ro(s))
Let denote the algorithm's 's current posterior over user types. Suppose the algorithm uses the strategy profile calculate this posterior, where f is not necessarily the same as f. Overloading the notation, we obtain the following Bellman update for the user's Q-value:
((A.s: fo()) ro(ss
7H FggBBr((t(s)) Qo (I> (8, f') arg max QA (s. (): (,
(2)
When taking the arg max, we can assume any item that maximizes the Q is chosen.
3.3 Equilibrium definition
We consider a multi-lcader, single-follower Stackelberg equilibrium. In this setup, users (leaders) commit to strategy f. and the algorithm (follower) best responds to f' f. Although the algorithm does not directly observe f. the assumption of f = f is reasonable given that modern algorithms have access to vast amounts of data and computational resources to infer user strategies. Additionally, we assume that no individual user (leader) has an incentive to unilaterally deviate from the equilibrium strategy. From the users' perspective, this implies they in mixed-strategy Nash equilibrium.
To formalize equilibrium, we consider two session entry scenarios: random entry (RE) and algo- rithmic entry (AE). Under random entry, the user stumbles upon an initial content for example

---

by landing on video platform. Let be drawn from a distribution p1. and let the prior over user types À be common knowledge. At equilibrium, for every θ, we have
fe E max RE ((fo = Enirp [o s1; (So. f (fo. f {88))]
(3)
To
In the case of algorithmic entry, the algorithm recommends the first item to the user at the start of the session. Given that the prior over user types À is common knowledge, the recommended item is 81 E arg max, QA(A, 8; f). Thus, at equilibrium, for every 0 E O, we have
BA E aarg V,AB( (A: (So, (2)) Qo arg max Qa s; (-o (fo. F (So, (- (Ao:
(4)
fo
The study of these general notions of equilibrium can be intractable. Therefore, next define special case of interest that enables us to characterize equilibria and analyze their properties.
3.4 Special case: Inconsistent actions and rewards
For a user of type 0, steering the algorithm via system 2's engagement decisions is challenging when system I's engagement length (or, equivalently, the algorithm's 's utility) is misaligned with the user's reward. This challenge is amplified when users with complementary interests shape the algorithm's default behavior, making it harder for type @ distinguish herself. We formalize this case below.
Suppose there are two possible (types of) items: = {a, b). Item a is more tempting for everyone, so (/aa(()) > 1/aa(b) for every 0 E O. For some types of users, O S O, item b is more rewarding, ie., ro (b) > ro(a). however, the remaining types θ₂ = e \ θ₂ have interests aligned with the algorithm, i.e., ra(a) > ro(b). We refer to users with type 0 θ₁ as type 1 users and those with type 0 E 02 as type users. For easy reference, we summarize this special case in Table [ Note that type and contents have different interpretations, such being popular versus niche items (5).
Table 1: Special case of interest where type users have inconsistent actions and rewards
User type
Engagement length
Reward
E θ
ra(a) < <rbbb) ra(a) re(b)
0 O2
aa(a) (b)
When type 1 users engage with content o, the algorithm receives utility (aaa((a))ihe than from content b. If system 2 always engages, the algorithm has incentive to recommend 6. To steer the algorithm away from a, system 2 must reduce its engagement probability such that fo(a) < as (b) Refusing to engage (1) signals the user's type, (2) discourages recommendation of tempting content, but (3) incurs a cost of (1 fe(a)) ra(a) when is shown. This trade-off complicates the user's strategy. In the next section, we will analyze the resulting best strategies of this special case, but before that, to further contextualize our model and this special case, consider two examples: Example 1. A user opens music recommender system while working, with the intent of listening to calm music. If the user (system 2) chooses to engage with the platform, they select a starting music s, after which the platform autoplays subsequent items. The number of musics autoplayed after s until the user disengages and selects another entry music s' defines the length of engagement y. determined by system 1. The platform benefits from longer engagement, e.g., through ad revenue, whereas the user benefits from listening to calm music while working. However, suppose the user is also a fan of singer X, whose music is engaging but distracting. This user is then a type user, and the platform must choose between recommending calm music (type b) or X's music (type a) during the working session. Note that during the working session, the user's intent remains fixed. Yet, on a later occasion, say, when relaxing, the same user's 's intent may shift toward listening to X's music for an extended period, effectively becoming a type user.
(o(a)
Example 2. A chatbot that charges per API call may operate in several "modes." " In an educational mode, longer conversations are valuable for student, aligning incentives between user and platform (type 2). In contrast, for an engineer seeking a quick answer, shorter sessions are preferable (type 1). Similarly, a therapy chatbot operating in an affirmative mode may sustain longer conversations by offering emotionally validating responses, even if doing so delays meaningful progress. Here, deciding whether to engage with a psychologist chatbot in mode is system 2 decision, while the duration of the conversation governed by system 1, and what should be the default mode every time while the user is going through weeks of therapy is the algorithm's 's choice.
5

---

4 Characterizing equilibria: Algorithm's best response
In this section, characterize the algorithm's optimal strategy for the special case outlined in Sec. 3.4 Specifically, we show that, under reasonable assumptions, the algorithm's Q-value is piecewise linear the prior À over user types and that the algorithm's strategy behaves linear classifier acting on A. Building on this, in the next section, we will characterize the users' optimal strategies for maximizing their value at equilibrium under both random and algorithmic entries.
To solve the algorithm's Q-value from the Bellman update in Eq. ( we first restrict f under the following reasonable assumptions to avoid pathological cases.
Let sy arg max, re(s) be the highest rewarding content for user type 0. We assume that every user of type 0 always engages with Si, i.e., fo(s) = 1,'0 E G and no user chooses occasional engagement with s, i.e. fo(s) < 1, if that does not discourage the algorithm about so
Assumption 1.
as(s)
fo(s) E [0,
Uf1},
as(s))
In our special case of interest, this assumption implies
fo(a) [0, (a( fo(a)= = 1,
(g(b) Uf}) fo(b) 11
40 θ
(5)
40 E O2
(6)
Given this restricted user strategy profile, we now present the algorithm's 's best response:
Theorem 4.1 (Algorithm's best response). Given that the algorithm has posterior X over θ it will best respond by recommending item a if and only if Loce hade 2 0, where
1- YA
fo(a) (a) as (b)
fo(b)
hg - rlfoaa (1 - (b)
See proof on page 22 This theorem shows that the algorithm best responds by using a linear classifier h = (he Jese. which acts on the posterior X. Interestingly, ha depends only on the strategy of user type 0, i... fe. with no interaction between users appearing h. This simplifies our characterization of the equilibrium, as we will discuss next.
5 Characterizing equilibria: User's best response
Given the algorithm's best response, we now analyze the user's best response, which defines the Stackelberg equilibrium of our game. We demonstrate that the user's s strategy often takes a simple form that allows us to identify all equilibria. We then examine the user's regret under equilibrium over finite horizon. This regret is measured using undiscounted sum of realized rewards, reflecting how much reward the user forfeits due to being myopic (i.e., having TM < 1). We show that user incurs a constant regret only if they are sufficiently foresighted, meaning "M exceeds threshold specific to their type.
Starting from Theorem 4.1 we observe that the algorithm uses linear classifier to make its decisions. Here, a user of type / contributes to the classifier's margin by an amount of hg Therefore, this user can influence the classifier to recommend item (or item b) by choosing larger (or smaller) value of hg. For instance, a type user has he x fo(a) ao(a)/op) (b). This user can push the classifier's value toward negative value, favoring item b, by refusing engage, i.e., setting fa(a) below ag (b)).
A type 0 user's ability to steer the algorithm toward her preferred item depends on the strategies of other users users- speciifccll,, on the classifier margin from the perspective of type t:
mg
=
hgr
(0)
Intuitively, larger margin implies that a type user will have greater difficulty steering the algorithm. We now formalize this intuition by fully characterizing the equilibria under algorithmic entry:

---

Theorem 5.1 (Equilibrium under algorithmic entry), Let ma AF. be the margin of the algorithm's 's classifier from the perspective of user rype when all other user types follow the equilibrium strategy under algorithmic entry. Define the steerable sets for type 1 and 2 users as follows:
0 E 01 Fo = f1- { [0.
ag(a)
Ao
mi
AE
Aa ao(a)
7
m AB) 2 20}
(b)
ab(b)
0 = 2 : Fo = fx E [0,
A
+ MA
I
An
+ + m
AE 2 20.
ao(a)
ab(b)
Let si and (-sū) be the high and low reward contents for type 0. The user's 's strategy at equilibrium is
any value in Fo
Fa # 0, Fa 0 7H
0,
ro(==
(50) 1, (-
1,
Fe 0, TM
Tel Po(-
ral (H(
any value in [0, 1],
Fe = 0, YM
Tw(s))
See See|proof page 24 This theorem shows that for each user type 0, there exists steerable set Fe. defined by a linear constraint on the user's strategy. If nonempty, any strategy in Fo results in an equilibrium. Notably, larger margin m&E shrinks Fg for type 1 users and expands it for type
The definition of Fg also highlights the role of 7AA which captures the algorithm's foresight. Notably, 7A appears only in the product Y m so its effect depends on both the sign and magnitude of the margin. For instance, when m AE > O, increasing 7.4 expands the steerable set for type users: as the algorithm becomes more foresighted, even slight disengagement from type users can effectively influence its behavior. Moreover, the steerable set for type I users has the following structure:
Corollary 5.2. The steerable set for user type 0 E θ1 is nonempty if and only if do 2 ao(() mae. Moreover, when nonempty, Fo [0, for some c.
See proof on page 25
As immediate result of this corollary, we have the following observation:
Corollary 5.3. For user of type 0 E O1 in any equilibrium under algorithmic entry where mg > As/ (bb) and TM # re ra(aa)/r(()) the user's strategy is
(b) = 1, (a) = -177 < Ta(a)
(b
This observation extends beyond algorithmic entry: Theorem B.2 deferred to the appendix for conciseness, we show that the same user strategy also holds at equilibrium under random entry.
When the steerable is empty, these results imply regarlegas of whether entry is algorithmic or random a strategic user will disengage from undesired content only if she is sufficiently foresighted. Let TH = 1/(1 YH) be the user's effective horizon. Type users disengage only if
(b(
TR
TM
re
ra
If the effective horizon is short, the user will fully engage with the tempting content, aligning with the algorithm's 's interest. This can lead to significant regret for the user, we shall discuss next.
Regret under equilibrium strategies. While we used discouutd rewards to model the user's limited foresight, comparing strategies requires an undiscounted sum of rewards. Let V (A; f) denote the expected total reward over T steps for user of type 0, given a strategy profile f. type distribution X, and an algorithm that best responds:
Vod ( = -Hyn(Jon
Σ (SUM))
1 > 0] ro(st))
Here, HT ~ (Jo, x²) denotes the distribution of histories of length 7 when the user leads by the strategy fo and the algorithm best responds. Note that the first item in the history may either randomly chosen optimally selected by the algorithm. We dropped À from the notation for brevity.
7

---

Let fxB -11 be the best user strategy profile under x-entry (either random or algorithmic) when YH + We define the regret of user type & after I steps as
Regrot (fxE) Vi (fx TM -1) Vol (f
We can see that a user incurs constant regret-spending only a fixed number of time steps aligning the algorithm with her preferred content-if and only if her effective horizon is sufficiently large:
Corollary 5.4. A user of type E 1₂ has constant regret in equilibrium if and only if Eq. ( holds.
See proof on page 25
6 Costly signaling reduces the burden of alignment
In this section, we show that if the platform enables costly signaling -ucch as allowing the user to click a small button to show disinterest- then system 2 can incur this additional cost to better signal its type. The possibility of incurring a cost effectively separates type communication from content consumption and can ease the user's engagement decisions.
We begin by formalizing costly signaling and redefining the user's strategy and equilibrium. As before, we first derive the algorithm's 's best response, then characterize the user's. Finally, we quantify how much introducing costly signaling can alleviate the user's burden to achieve constant regret.
6.1 Strategies, values, and equilibrium
We assume that the user's system can choose to incur a fixed cost c, and this effort is observable by the algorithm. A user of type / adopts a strategy that, when presented with item s, involves both the probability of engagement, denoted by fs(s), and the probability of incurring the cost, denoted by us)). We denote the strategy profile of the users by (f, u), where u = (uo)oge.
The algorithm observes a history of both engagements and costs: Ā = (81, D1 û1, 82, 82, u2 "...... where ú is binary indicator of whether the cost was incurred. As before, the algorithm maintains posterior distribution over user types given // to best respond, which yields the Bellman update
fa(s) (s) + TA E (g--Ber(f(a)) max Qu (i> (s. 9. ú): f. ful. o f.:...::.... f.)]]] (10)
QA(A, 8; f. Ee~A
S-Ber(ug(s).
Let À denote the current algorithm's 's posterior over user types. Suppose the algorithm uses the strategy profile (f', u') to calculate this posterior, where I' and u' are not necessarily the same as f and u. We have the following Bellman update for the user type 0's Q-value:
Qo f. u. f'. u = (6))))(()) us]s
(II)
-TH (er-Ber(faa Qo (> (s. U. a): f¹, u.). max Qa(A (s, (); u'). u²); f. f ,u) (ngerrag(s))
We study multi-leader, hingle-follwee Stackelberg equilibrium where a strategy (f, and the algorithm (follower) best responds to (f' = f, u = u.). For brevity, we omit (f", ut) from Qo notation and focus on the case of algorithmic entry. At equilibrium, we have
(leaders) commit
(JAE ug
AE arg max Qo A max (A s; (fe, FAB), (uo, 2 A)) (fo. f (uo, u xB
(12)
fe.n6
We next characterize equilibria of this form and show they impose a lower burden on the user.
6.2 Characterizing equilibria: Algorithm's best response
We characterize the algorithm's 's optimal strategy, focusing on the special case outlined in Sec. 3.4 We show that, similar to the case without signaling, the algorithm's 's Q-value is piecewise linear in the prior À over user types, and the algorithm's 's strategy functions as linear classifier operating on A. To solve the Bellman update in Eq. (0). we impose an additional restriction on (f. u) beyond Assumption to rule out pathological cases:
Assumption 2. Let s E arg max, re(s) be the highest rewarding content for user type 0. We assume that no user of type pays a cost when recommended with sy. i.e. us uo(s)) = 0, v0 E O, and no user pays a cost for content if that does not discourages the algorithm from recommending s:
(x0(s)
ug (s) > 0 = fo(s) E [0,
YO OE
ae (57)
8

---

Given these restrictions over the user strategy profile, we now present the algorithm's 's best response:
Theorem 6.1 (Algorithm's 's best response with signaling). Given that the algorithm has a posterior X over O it will best respond by recommending item a if and only Lege hado 2 0, where
1-7A
Sa(a) (a)
fa(b)
hg
(13)
(1- (1-naf((a[1- (1 (1 Nafo(b)( u²(b))
See proof on page 25 We next discuss the characterization of the equilibrium with signaling.
6.3 Characterizing equilibria: User's best response
Given the algorithm's s best response, we now analyze the user's s best, we now formally characterize the equilibria under algorithmic entry when users can incur an observable cost c:
Theorem 6.2 (Equilibrium under algorithmic entry with signaling). Let m3 A be the margin of the algorithmm's 's classifier from the perspective of user type 0 when other user types follow the equilibrium strategy under algorithmic entry with signaling. Define the steerable sets for type and users as
E O1 : Fg (2.y) [0. ao(a) (b)
X [0,1]
X
m&
AE
d
7A 7A(1- y) m B) 2 0}
(b{(b)
cas(a)
0 E O2 F6 = (1, 10,1]
as(a)
mg
AE
x
(b)
A
+ YA (1 - y) ml
AE m)
2 00.
Let s/ and (-s)) be the high and low reward contents for rype O. Define the critical TH value for type as
c+ corol --(o--s))) =2)
2g8 --)) (x())))
Th
c ro(()) ro(-
caal (x6)
Assume "/H + Ti and (ل(-ا
To sz). The user's strategy at equilibrium is
[06ااo
(58), AE (st)) (1,
ug
any value in Fo. (a((=s))
Fg + 0, F9 = 0. 1H > TR
AE (- (- u0
1),
co(*))
(1,0),
Fg = 0, TH < TH
When signaling via incurred costs allowed, the steerable set is defined by bilinear constraints over the user strategy. Several key insights follow. First, compared to Theorem 5.1 the projection of the steerable set onto the fe dimension is at least as large with costly signaling as without. Second, for both user types, incurring cost can expand the steerable set when the margin works against them. Specifically, for type (type 2) users with m > (m < 0), if (x. y) E Fe. then is any (a, y') with y' 2 y. Finally, in the no-signaling case, the steerable set for type 1 users is nonempty if and only 0 2 (b)()), m/ AE, The proof parallels the argument in Corollary 5.2
See dee/proof on page 27
When the steerable set is empty and the signaling cost is sufficiently small, Theorem 6.2 contrasts sharply with Theorem [.1]] In this case, sufficiently foresighted users optimally choose to ineur the cost and partially engage with undesired content. This allows them to decouple type signaling from reward consumption: they fully communicate their type by paying the cost, while limiting engagement. Formally, for type users, state the following result under empty steerable set:
Corollary 6.3. For user of type f E O1, in any equilibrium under algorithmic entry with costly signaling where m AB Xo >0/a0(0 and the signaling cost is sufficiently small, the user's strategy
s (b) 1, AE (b) = 0.
(aa(a) (b((b)
5A (a) + L(TH < 3
up (a) = 1.
As an immediate result, one can verify that, similar to the case without signaling, the user must still be sufficiently foresighted to achieve constant regret in every case:

---

Corollary 6.4. Assuming the cost of signaling is sufficiently small, user of rype E O₂ will always have constant regret equilibrium under algorithmic entry with signaling if and only if
rb
(r((a)
ca((a) ag(h)
1
1 7/
(b (a
Compared to Corollary [54 key insight emerges: costly signaling lowers the burden of alignment. Specifically, the required effective horizon for a type 1 user is relatively reduced by
(a (xp(a) rb( (o(b)
ro (b)
In conclusion, the opportunity signal by incurring cost can both expand the steerable set for users and reduce the alignment burden, requiring optimization over a shorter horizon.
Discussion
We presented formal framework examine the burden of alignment in settings where users have inconsistent preferences. Given the vast array of design choices such contexts, mathematical modeling is essential understand the trade-offs and inform practice about the limitations of alignment and potential solutions [9 23
Our analysis assumes the platform seeks to maximize engagement utility- reasonable goal for self-interested platforms that benefit from user interaction. However, one way to reduce the burden of alignment is to reconsider this objective, if modifiable. Alternatives include optimizing for long-term returns [8]. user enrichment [9], or socictal objectives [24). These approaches often require inferring user mental states that not directly observable, but must be inferred from behavioral data 25
While our study focuses on misalignment between user and algorithmic interests, it important to note that engagement maximization may still produce unintended outcomes even without explicit misalignment. For example, differences in user feedback rates across content types can inadvertently lead the algorithm to favor certain types of content 226
Our work has several limitations. We simplify the learning process by assuming that users are aware of their rewards and that the algorithm has full knowledge of strategies. Additionally, we focus on two-sided interaction between the platform and users, while real-world scenarios often include content creators whoo may strategically invest in different content types 27
third side-content
In summary, work highlights critical challenge in alignment from the user's perspective. By providing an economic framework, we contribute to a deeper understanding of the limitations of alignment and the importance of modeling human decision-making. This framework can also inform human-computer interaction design 28 on how to better accommodate diverse user preferences and behavior.
Acknowledgments
This project began through formative research conversations with Moritz Hardt. It later took a new direction following the author's 's discussion with Jason Hartline at FORC'23, The author thanks Dylan Hadfield- for extensive feedback on an earlier version of this work. The author also thanks Jon Kleinberg, as well as participants of the EC'25 Workshop Language Models and the EC"25 Workshop on Swap Regret and Strategic Learning, for their helpful feedback on poster version of this work. The author is further grateful to the anonymous reviewers for their thoughtful comments and deep engagement with the paper.
Information Economics Large
10

---

References
[1] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[2] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[3] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[4] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[5] Omar Besbes, Yash Kanoria, and Akshit Kumar. The fault in our recommendations: On the perils of optimizing the measurable. In Proceedings of the 18th ACM Conference Recommender Systems, pages 200-208, 2024.
[6] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management science, (1(9):6336-6355, 2024.
[7] Amanda Y Agan, Diag Davenport, Jens Ludwig, and Sendhil Mullainathan. Automating automaticity: How the context of human choice affects the extent of algorithmic bias. Technical report, National Bureau of Economic Research, 2023.
[8] Arpit Agarwal, Nicolas Usunier, Alessandro Lazaric, and Maximilian Nickel. System-2 rec- ommenders: Disentangling utility and engagement in recommendation systems via temporal point-processes. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1763-1773, 2024.
[9] Md Sanzeed Anwar, Paramveer Dhillon, and Grant Schoenebeck. Recommendation and temptation. arXiv preprint arXiv:2412. 2024.
[10] Simon Zhuang and Dylan Hladfield-Mened Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33: 53:15763-15773 2020.
[11] Smitha Milli, Luca Belli, and Moritz Hardt. From optimizing engagement to measuring value. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 714-722, 2021.
[12] Michiel Bakker. Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:381 01388766-8889 2022.
[13] Moritz Hardt, Nimrod Megiddo. Christos Papadimitriou, and Mary Wootters. Strategic classi- fication. In Proceedings of the 2016 ACM conference innovations in theoretical computer science, pages 111 112, 2016.
[14] Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 55-70. 2018
[15] Nika Haghtalab, Thodoris Lykouris, Sloan Nietert, and Alexander Wei. Learning in stackelberg games with non-myopic agents. In Proceedings of the 23rd ACM Conference on Economics and Computation, pages 917-918, 2022.
[16] Kate Donahue, Nicole Immorlica, Meena Jagadeesan, Brendan Lucier, and Aleksandrs Slivkins. Impact of decentralized learning on player utilities in stackelberg games. arXiv preprint 24X3:2403300188 2024.

---

[17] Tijana Zmic, Eric Mazumdar. Shankar Sastry, and Michael Jordan. Who leads and who follows in strategic classification? Advances Neural Information Processing Systems, 34: 15257-15269, 2021.
[18] Jason D Hartline and Tim Roughgarden, Optimal mechanism design and money burning. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 75-84, 2008.
[19] Sarah Dean, Evan Dong, Meena Jagadeesan, and Liu Leqi. Accounting for ai and users shaping one another: The role of mathematical models. arXiv preprint arXiv:2404.1236, 2024.
[20] Andreas Haupt, Dylan Moaffield-Menelle and Chara Podimata. Recommending to strategic users. arXiv preprint arx.v:2302.06559, 2023.
[21] Sarah H Cen, Andrew Ilyas, Jennifer Allen, Hannah Li, and Aleksander Madry. Measuring strategization in recommendation: Users adapt their behavior to shape future content. arXiv preprint arXiv:2405.0m 05596, 2024.
[22] Sarah Cen, Andrew Ilyas, and Aleksander Madry. User strategization and trustworthy algorithms. arXiv preprint arXiv:2312. 1766, 2023.
[23] Sarah Dean, Evan Dong, Meena Jagadeesan, and Liu Leqi. Recommender systems as dynamical systems: Interactions with viewers and creators. In Workshop on Recommendation Ecosystems: Modeling, Optimization and Incentive Design, 2024.
[24] Chenyan Jia, Michelle S Lam, Minh Chau Mai, Jeffrey T Hancock, and Michael S Bernstein. Embedding democratic values into social media ais via societal objective functions. Proceedings of the ACM Staman-Compinn Interaction, ((CSWW1)11 1):1-3,, 2024.
[25] Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Manish Raghavan. The inversion problem: Why algorithms should infer mental state and not just predict behavior. Perspectives on Psychological Science, 19(5):827-8) 838, 2024.
[26] Jessica Dai, Bailey Flanigan, Meena Jagadeesan, Nika Haghtalab, and Chara Podimata. Can probabilistic feedback drive user impacts in online platforms? In International Conference on Artificial Intelligence and Statistics, pages 2512-2522 PMLR, 2024.
[27] Nicole Immorlica, Meena Jagadeesan, and Brendan Lucier. Clickbait vs. quality: How engagement-basee optimization shapes the content landscape in online platforms. In Pro- ceedings of the ACM on Web Conference 2024, pages 36 2024.
[28] Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. From numan-human collaboration to human-ai collaboration: Designing ai systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems, pages 1-6, 2020.
[29] Hussein Mozannar, Gagan Bansal, Adam Fourney. and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai-assisted programming. In Proceedings of the CHI Conference Human Factors in Computing Systems, pages 1-16. 2024.
[30] Geng Zhao, Banghua Zhu, Jiantao Jiao, and Michael Jordan. Online learning in stackelberg games with an omniscient follower. In International Conference on Machine Learning, pages 42304-42316. PMLR, 2023.
[31] Benjamin Hébert and Weijie Zhong. arXiv:2207.00685, 2022.
Engagement maximization.
arXiv preprint
[32] Mohammad Hajiaghayi, Mohammad Mahdavi, Keivan Rezaci, and Suho Shin. Regret analysis of repeated delegated choice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 9757-9764, 2024.
[33] Eden Saig and Nir Rosenfeld. Learning to suggest breaks: sustainable optimization of long-term user engagement. In International Conference on Machine Learning, pages 29671-29694 PMLR, 2023,
12

---

[34] Sarah Dean and Jamie Morgenstern. Preference dynamics under personalized recommendations. In Proceedings of the 23rd ACM Conference on Economics and Computation, pages 795-816,
2022.
[35] Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, and Anca Dragan. Ai alignment with changing and influenceable reward functions. arXiv preprint arXiv:2405.17713 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction exactly reflect the claims made throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: During explaining our setting, reviewing related work, and the final discussion, we explicitly discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspcciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We present the assumptions very clearly before every theory. All the theorems are linked to their proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification:
This is a theory paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification:
This is theory paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu http:///ggiddeeeeeouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: This is theory paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This is a theory paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: This is theory paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:ppp.....ppccpccttcsassaaaa.
Answer: [Yes]
Justification: Yes, all the authors have read the code of ethics and acted accordingly.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have extensively discussed the broader impact our work in the introduction and in reviewing related work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This is theory paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This is theory paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This is theory paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This is theory paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This is theory paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: We only used LLMs for writing and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: httpp////.e///ooeeeeeee///////// for what should or should not be described.
20

---

A Extensive review of related work
The alignment problem has been studied as game between users and platforms/algoriihms across computer science, social science, and economies. In this section, we provide an overview of the work most relevant to our study.
Much of the literature models user-algorithm interactions as Stackelberg games in various forms. In one common variation, the algorithm leads by choosing recommendation policy while users myopically best respond by selecting the highest-rewarding option. In this setup, engagement maximization may yield highly suboptimal outcomes for users [5], and when the algorithm employs an online learning procedure, its sample complexity for regret minimization can be exponential [30].
setting closer to ours features Stackelberg games where the leader (typically a platform) commits to extensive strategy over multiple interactions, and the follower (typically a user) best responds. Because recommendations occur over several steps, users account for how their response to each item affects future interactions. Haupt et al. 20 call these users strategic users-in contrast to myopic users who optimize locally. Similar to our work, they show that users (in particular, minorities) tend to engage in behaviors that accentuate differences relative users with other preference profiles. Both Haupt et al. [20] and Cen et al. [21] provide empirical evidence from lab and online experiments supporting such strategization. Similarly, Cen al. [22] analyze a two-player game focusing on how these behaviors help or hurt the platform in the short and long term, while Hébert and Zhong [3I demonstrate that when the principal maximizes engagement, the agent may, in the worst case, fail to extract any useful information or utility from her interactions.
Our work diverges from these studies in two important ways. First, we treat users as the leaders in the Stackelberg game. We assume that users particcaayyy system 2-can commit to an engagement strategy, and the platform or algorithm observes this commitment (for example, via repeated inter- actions and prior data). Consequently, we focus on what it takes for users, as leaders, to maximize their reward when engaging with the algorithm. Second, our analysis generalizes to a multi-leader single-follower game, and we study how the presence of other users affects each individual's 's strategy manner similar Nash equilibrium.
Strategic classification 13 14 offers another related perspective. In that setting, the platform typically publishes a classifier and users strategically respond often incurring cost of change to optimize their outcomes, In contrast, our framework features users who first commit to an engagement strategy. Moreover, while strategic classification typically unfolds at single time point, we model extensive-form games in which the follower's s strategy depends on the history of interactions. At a high level, both settings found the relative capabilities of the parties-in our case, how foresighted the users determine who is following and who is leading (17).
We work in full-information setting, which means (1) users know their rewards but cannot access appropriate content without the algorithm's 's help, and (2) the platform observes users' strategy profiles-a natural assumption given the scale of user data. In other contexts, users might need to learn their rewards (e.g., in a multi-armed bandit setting [16), and platforms might need to learn users' strategies as well [15] 32). A subtle distinction exists between these approaches and our framework. Even in our setting, the platform learns about users during each session, but it makes optimal use of each interaction. Remarkably, we show that these optimal posterior updates admit tractable forms.
Our study also informs the design of interactions with platformss/lgorrihmms by analyzing equilibrium strategies. For example, we quantify how giving users the option to expend extra affort-such as completing challenging task-can ease the burden of alignment. Such design considerations appear in mechanism design in economics [18] and in human human-computer interaction in computer science [28 29 For example, breaks during interactions can also promote and sustain long-term
engagement 33
Finally, our work differs from research on preference dynamics evolve under the influence of the algorithm. In our setting, users maintain fixed preferences but they strategically adjust some of their actions in response to the algorithm.
R4 35
where user preferences
21

---

B Additional statements
Lemma B.1 (Bellman update). The algorithm's 's optimal policy only needs the posterior (X H: fl over user types after observing H.
Proof. Overloading the notation, define Qu(A, = max, Qa(A, f.). We can expand Qa(A,
Qa(A, s; f) E Eyr((olss) y + TA max max Qu1 (1> (s, 1yy > 03); s f. fa(s)
max QA ((> (s, 0): f) s f]]
= EB>>
Lap(s) + VA (.g
This shows that the algorithm does not require the entire history for optimal decision making and only needs to update its posterior over user types.
Theorem B.2 (Equilibrium under random entry). Let ma RE be the margin of the algorithm's classifier from the perspective of user type when all other user types follow the equilibrium strategy under random entry. When RE (b) and YMu + ro(a)/ro(b) for a user of type 0 E θ the user's s best strategy
(B) 1,
(a) = 1 16
ra(a)
rb(b)
See proof on page 29
C Missing proofs
Theorem 4.1 (Algorithm's best response). Given that the algorithm has a posterior over θ it will best respond by recommending item if and only if Enee hado 2 0, where
1 YA hgg (1 (1- (AJoa)) (1 - (b))(b))
fola) fo(b)
(a
oo (b)
Proof of Theorem 4.1
For notational simplicity, we omit f from the notation for QA. To streamline the following conventions:
the proof presentation, to s. The inner product of two vectors u and v is denoted by (u, v)), while the element-wise product is denoted by product is defined as uōv wop
to refer to the alternative content
(uovalo- When one side of this product is distribution, the normalized
Division, such as u v. denotes element-wise division.
(u..)
Using this notation, we can rewrite the algorithm's 's Bellman update from Eq. I as follows. First, the posterior [A (s, simplifies to the following form:
(x)
9 = 1,
A (s.0) 5) =
(61(1 f(s)).
1 0.
We can also express the expected immediate reward Es (e as (A, (s)) Using this, the Bellman update for QA becomes
(A, = (A, (+)//((s))
+ + (A, (s)) max QA (X6f(s))
(14)
+ 7/4 (A, f())) max (A6(1-
Note that Asf(s) or Ao(1 - fs)) may be undefined if all users choose to either fully engage or fully disengage. However, since the second and third terms also multiplied by (X, f()) and (X. (1 f f())) s))) respectively, this issue can be neglected. We prove that the following Q-function
solves Eq. 14)
Qa(A, (A, a())) + 7A max { (a(-s) (())) f())) 1())),03
(15)
22

---

where
f()) 1
f(-8) e (1 f(}))
1
(s)) 1 -- --)f((s)
+ NA (A-7af(s) - (1-- Q ((-s)
(16)
a<(s)
One can verify that h in Eq. 7 be expressed (()) q(b.. Before proving Eq. 15 first
show that it implies h = {( q(b) serves as the linear classifier that determines the algrrithm's policy:
Lemma C.I. QA(A, QA(A, -s) + (A. -(1))-((-s))) a(-s)) 20. 0.
See|proof on page 30 The proof of this lemma relies on the following lemma, which we will use again later on.
Lemma C.2. (Ao f(s). g(-s)) -A)s(s)-((ss))
Seefproof on page 30 This lemma also yields the following result that is useful in simplifying the second term of Eq. (1):
Lemma C.3. max, Qa(A, 8') = max,' (A,
See (proof on page 31 Together Lemmas simplifying the third term of Eq. (14)
C.1 to C.3 give the following result that is useful in
Lemma C.4. If (A,1 f())
0. we have max,/ Qa(A6(1 f(s)). =
(A6(1 - (()))
See proof on page 31
Using Lemmas C.3 and C.4 we can write the right-hand side of Eq. 44
(x, f(s)/a/((ss)) + (X, f (s)) max (Aof(s). + 7 (A, f(s)) (A6(1- f(s)(((())()))))
=)).(())/(())) + max (
= (A, (A. + (s) -()) 0) + + q(s) f(s) + 01 f(s)s(()))
(17)
Using (1 - (()) - (()))(1-((-s)) = from Assumption 1 we can further simplify the first and third
(last) terms by
f(s)
(Aq(s) f(s) + >ag(-s) f(s))
a(s)
(s)
(()) RA 1 (a)(s)
a(s)
[14 f(8) 0 (--8) f()))
-1- (A)(8)) (8)) 7Af(-s)) + ((-s))(1-f(())
71 (x-s)
1-74((5))
f(s)
1
f(-s) (1-f(s))
JA xx (-s))
9g()))
((s)
((Af(s))
(1- (Af(-s)
Plugging this into Eq. 17 gives (A 8) defined in Eq.
15 Therefore, the proposed QA solves
the Bellman update Eq. 14). This completes the proof.
Theorem 5.1 (Equilibrium under algorithmic entry). Let my be the margin of the algorithm's classifier from the perspective of user type f when all other user types follow the equilibrium strategy under algorithmic entry. Define the steerable sets for type and 2 users as follows:
0 E 01 : Fe
{ E [o, as(a)
A
m
AE
7IA m AF) 20}
anp (b)
(bb(b)
(a((a)
0 E O₂ Fe x E (0, ao(a)
A
mi
AE
(b)
A
+ TAA
m
AE) 20.
23

---

Let sg and (~sh) be the high and low reward contents for type 8. The user's s strategy at equilibrium is
any value in Fe.
Fa + 0, Fo = 0, 1,7
0,
re( sp
fo F (5))= 11 fo AA (-s
To ro(sp)
1.
Fo = 0, 1,7H <
Tol To(-8g)
ro(s))
any value in [0, 1],
Fa 0 YH
Tol vo(-8g)
(o(s))
Proof of Theorem 5.7 We use a similar notation as in the proof of Theorem 4.1 readability, we drop from Qo(A, f: f) and Qa(A, s; f).. With this notation, the Bellman update for user type in Eq. (2) can be written as
For improved
*HJo(s) Qu ())(s), arg max (A6 Qa 8) (o(s)r(())
5¹
-o Qo A6(1 f())) arg QA ( f(s)), (X)()-((o)),5))
x'
Since the user's entry and subsequent interactions occur under the algorithm''s best response, we only need to solve the above for arg max,' QA(A, When QA(A, 8) 2 QA(A, -s), Lemmas C. and imply
arg max QA (A)f((ss),)))
o'
(-s) E max max QA (Λ6(1 - f(s)),
Plugging these into the Bellman update, we obtain
Qe(A, fe(s) ro(s) + rHff(s) Qo (f) + 7 fa())) Qo - f())), -s).
Note that this equation has no dependence on A, so, we can drop it from the notation. Using Assumption [ we can write the above update separately for s sg and s = (-s)):
Qn(st) (s)) YH Qo(s)).
(o(- Q0 = fo(- (0) + TH fo Qo(-sū) + +x(11 - fo(-sā) Qo(s))
Solving these equations, obtain
To (s))
Q0(s))
1
fo st) ro (-())))
+ 7H (1 -
Qo(-s²)
70 (-87)
(1 - nH s7)) (1 (7))
ra (s)) ro(s))
fo(-s5) ro( 5))
1-TH 7/
1 To (-s})
Starting from prior À over user types, the user's value is
Vo(A) = Qo arg max Qu(A,
Given Vo, we now explore the user's best strategy that maximizes Vo, leading to the equilibrium notion defined in Eq. (4. Note that Qo(sī) > Q=(-s)). Therefore, the optimal strategy is to select fe(-sş) such that o arg max, (A s). In the equilibrium, Theorem 4.1 implics that this is only possible when
Ao
fo(-s}) ag(-s;) 2 - (A- h AB
Ag
(18)
RAfo ))
ag(s;))
Here, we generalized h in Eq. linear constraint over fo(- fo(=
7 by defining h(s) 11s a. h 1(s b} h. Eq. 18 is
Ao
(g)) (102013)) + 74 (> h ( 20.
Xn
+ (A h (B()) fo(-sū) (asl (0(55)
24

---

Any fo(-s)) that meets this condition is the user's best response. If the condition does not hold, then VA(A) (0)-))). In this case, the user's best response is
0,
YH
ra ra(=a))
(o(
1.
TK <
ro 10(=%)) (e(e))
[0,
O.W.
Using specific values for type and type users ahove will complete the proof.
Corollary 5.2. The steerable set for user rype 11 is nonempty if and only if do 2 ao(b) mge. Moreover, when nonempty, Fe [0, c) for some
Proof of Comollary 5.2
Suppose Aa < 7A (1) mg AB In this case,
X AE mg
d 7A m ~ a0(a) 20 (b)
mg AE (1 - VAA) S0,
000 (b)
(a((a)
which means Fg = 0. Therefore,
F = Ao 2 YA (a) AE
Now suppose Fo. Then the above implies any x' S x is also in Fg. Therefore, when Fo * 0, it covers [0, c) for some c. Particularly, when Fg + 0. we should have 0 E Fg which implies
F6 +
do as (b) m AB
Now suppose Ag 2 (b( mg
AE Then Fg, so
Ae 2 as (b) m + F6 *
Thus, we can conclude
Ag 2 ag (b) m AE
F8 #0
Corollary 5.4. A user of type O₁ has constant regret in equilibrium if and only if Eq. (9 holds.
First, there always exists an equilibrium where the conditions of Corollary 5.3 and Theorem B.2 are satisfied. This occurs, for example, when type 1 users with inconsistent actions and interests form a small part of the population. is either fa = 0 or fo(a) 1. When the user fully engages, the algorithm cannot distinguish her from users with aligned interests, leading it to continue recommending type b content. contrast, when the user fully disengages, the algorithm will recommend type content at most once.
Proof of Corollary 5.4
such cases, the optimal strategy for a type user
Theorem 6.1 (Algorithm's best response with signaling). Given that the algorithm has posterior X over O it will best respond by recommending item a if and only if Eece hggg 2 0, where
1 YA
fo(b) (a) (e
fo(a)
ha (1-rafo(a)(1-ua uo(a)) (1 (Afo(b)(1 ub))))
(13)
(
Proof of Theorem 6.1
We follow a similar notation as in the proof of Theorem Using this
notation, we can rewrite the algorithm's 's Bellman update from Eq. (10) as follows. First, the posterior [ (s, 0, à simplifies to the following form:
{o(u(s) o f()),
à = 1.9 = 1
Xo(u(s) 0 (1 f()))
à = 1. = 0.
| (s, 0 a)]
u(s)) Aol(1 (o((1=u(s)) f(s)
û 0. 1
(((1-u(s)) ( (1- f())).
à 0, 0.
25

---

We can also express the expected immediate reward [=(fo(s)/ao(s) as (A, (A.f(s)/a(s)), the Bellman update for QA becomes
Using this,
QA(A, 8) (A, ((s))
7A (A, (s)) max ( f(8)], [(s)).
7A (A, 2 f (s))) max Qx (1-
(19)
74 A (1 u)
max Q^ Q4 (1 - u(s)) f(s)], s)
7A A (1 (1-u) o f())) max Qu (Λe u()) (1-
We prove that the following Q-function solves the above:
QA(A, (). (s)) + VA max (A, ()((-)) - {(s)) (1 (1-u(s)) f())) .0].
(20)
where
f(s)
f(-s) (1- u(s)) f(s))
(() 1 1- YA(1 (()) f(s) ((s) (1 7A - u(s)) f()))(()))))))))) o
t1A 11
Vaf f
a
(21)
Using (s)) au(-ss)) and (s) ((1---(-1) ((-))) implied by Assumption 2 one verify that Eq. (13) can be expressed as q(a) q(b). Before proving Eq. (20), we first show that implies (a) (b)) serves as the linear classifier that determines the algorithm's policy:
Lemma C.5. QA(A, 2 QA(A, -) =
((s) q(-s)) 2 0.
See proof on page 31 The proof of this lemma relies on Lemma C.2 and the following lemma:
Lemma C.6. (X
(s)), (+) ((-s)) 2 g(s) q( ((-8)) -))
See proof on page 31 This lemma also yields the following result that is useful in simplifying the fourth term of Eq. [
Lemma C.7. max, Qa(A, s') = max,' (A, As'))
See proof on page 32
Together Lemmas C.
to C.3 give the following result that is useful in
simplifying the first and second term of Eq. (14]
Lemma C.8. If (A, u()) g 0 for some g. we have
max QA (s) o (s) (Ao(u(s) o g(s)) q(-s))
See proof on page 32
Using Lemma C.8 for the second and third term, Lemma C.7 for the fourth term, and Lemma C.4 for the fifth term in the right-hand side of Eq. (9), we can simplify the Bellman update
(s)((a)((ss)) + (A, (1 us))) f())) max (A6((1 u(s)) o f()) g(s))
(A u() o f())) a( a(-s))
- (A) { ( (() q()) .0}
(= f() + (-)))) f(s) uss) f
(22)
Using (1- - f()))(
from Assumption [ we can further simplify the first and third
(last) terms by
f(s)
+ 7sq(s) o (1 u(s) a f(s) + o f(s) + u(s) f(s))
((ss)
- (1- (s)
f(s) + 7A
(s))
1
[1-= f(s)) f(-s) (1- (1 u(s)) f f(s))
74
(x(-8)
(1 (1- (c) - o() (A((s))
7/A
7:411 ul u(-s)) f(-8)
(23)
26

---

The following property implied by Assumptions | and 2 are useful to further simplify the above:
-<->>> >0 = - (1 u (s)) o f(s =
Using this Eq.
we obtain
1
f(s)
(<(s)
1 - %A(1 u(s)) f(s)
1
PA (1 -1-18 (-()))) f( (1)))))))
(l))((-u(3)) (() f(-8)o (1 u())) f(6))
(1- f(s))
+ 7A
(<ss)
1-7af(-s)
f(s)
(x))
1- 1-7A(1 -(())) f())
1
- -11 f(s)
+ 744 (-s))
o +((-)) o (1 - (1 (1-w(s)) o f(s))
(1 -A-1a11 o (1
1
f(x)
1
f(-8) (1 f(s))
YA α< (-88)
(s)s) 7A(1 (8)) f(8)
(1 7AA(1 - u(+)) (8)) o (1-
=q(s).
Plugging this into Eq. (2 gives QA(AA 8) as defined in Eq. 20 Hence, the proposed Q.A solves the Bellman update of Eq. 19 This completes the proof.
Theorem 6.2 (Equilibrium under algorithmic entry with signaling). Let m be the margin of the algorithm's 's classifier from the perspective of user type when other user rypes follow the equilibrium stralegy under al algorithmic entry with signaling. Define the steerable sets for type and users as
co(a) (b(b)
Ao
Ao ca)(a)
0 E 11: Fe = (1r, v) E [0,
X 10,
mg
AB
m A VIA (1 m/
2 0
(b)
0 E O2 Fg = (1, E [0,
ao(a)
Xg
+ m
x
(b)
do
+ TA (1 u) m AF) 2 20
Let sy and (-8h) be the high and low reward contents for type 0. Define the critical TH value for type as
c c+rol-ssh
av(-aā) (x(a))
7H =
c+ re(st) re(-
col (-() ano (5g)
Assume "TH # The and < aol-s]) ro - 8)). The user's strategy at equilibrium is am(s))
(FA u1 (sa)) (1.
any value in Fo. (=a(=s))
Fg + D, Fg 0 TH TH
(fAF(-s)), up AB (-sū))
1),
aa(5))
(1,0),
Fg 0 TH TH
We use a similar notation as in the proof of Theorem 5.1 For improved readability, we drop f and u from Qo(A. 8; f. u) and QA(A, s; f,u). u). With this notation, the Bellman update for user type in Eq. 111 can be written as
Proof of Theorem 6.2
Qo(A, 8) (o)))))))) (s)
+ ?Hul(s) (s) Qo (o(u(s) f [()]], arg max QA (A6 f(s),,
+ 7s) (1 fo(s)) Qo (o(u(s) (1 f())) arg max Q. (1- f(s))),
+ +ym(1 up(s)) (s) Qo 19 u(s)) f())), arg arg max Q^ (6 u(s)) of f(s)),
;'
+mn(1 up(s)) fo(s) Qo 1b|ek u(s)) (1 f(s)) max Q^ (Λō (0o([())ss))) (())),
5'
27

---

Since the user's entry and subsequent interactions occur under the algorithm's best response, we only need solve the above for E arg m Q (A,,'')) s'). When Qa(A, s) 2 QA -s). Lemmas C.2
C.5 and C.6 imply
(-8) E arg max
(-8) E arg max
o (1- 8 E arg max (1 u(s)) f()),
(-s) E arg max Q^ - u(() o
Plugging these into the Bellman update, obtain
(o 8) = So(s) ro( - (0)
+ + 6) fo(s) Qo (06 f(s)), -s)
+ +s) -1 (n(s) Qo (A)fu(())
f()))], -s)
+ TH(1 (o() Qo (Ao u(s)) f()),,
fo
+ T ( ux() (1- fo(s) Qo (A u(s))
fs)))_
Note that this equation has no dependence on X, so, we can drop it from the notation and obtain the following simplified update rule:
Qo(s) fo(s) ro(s) - uo)s
+ TH (1 (1 uo(s)) fo(s)) Qa(-s)
+ 7H (1 - (o(s)))) so(s) Qa(s).
Using Assumptions 1 and 2 we can write the above update separately for sg and 8 (-8)):
Qo(sõ) =re(s}) + TH Qa(sğ),
Qa(-sā) = (-)) (0(-5)) - (-)))
+ +TH (1 (1
fol -b)) Q0(s)) + +HH(1 uo(- (o(-s)))(((--0 Qo(
Solving these equations, obtain
Q0(5)) = ro(sg)
7H
Q0(-s)) fo(=))) ro - 08) - uo - 80) c TH - (1 - uol [o)) *5)) ro(s))
(1
usl (-$))) 2g)) (1 7H(1 (1 so) -5]) (1 7H)
ro(s))
To (o($)) - fol st) re( vo(-5p0 (T) up(=
TH
1 1= - uo 8 s))s{(( (-{()
Starting from a prior X over user types, the user's value is
Vo(A) = Qo arg max (A)
8
Given Vg. we now explore the user's best strategy that maximizes Vo, leading to the equilibrium notion defined in Eq. (1).. Note that (s)) 2 Qa( (g)). Therefore, the optimal strategy is to select (fo(=s)), uo( (=()) such that sg arg max, Q A, s). In the equilibrium, Theorem 6.1 that this is only possible when
implics
A
fo(-s))
( 1 x1
(> (X o, h AF (st)
(24)
up( uo(- So( 85)
(1 (-())))
Here, we generalized h in Eq. 13 by defining h(s) 1(s = a} h 11s = b} ht Eq. 24 is
bilinear constraint over (fe( - 87) us( -s))):
A
A
ao (1))) + (A 0, h AE (st)
fo(-
+ 74(11 uo(-sā)) (2 o. h 1. (5())) 2 0.
ag( (g))
28

---

Any (fo(-s)) u -a)) that meets this condition is the user's best response. If the condition does
not hold, then Vo Qo(-s)). In this case, one can verify that for fixed ugg (a(-))). the sign of the derivative aQu( si)
does not depend on So( (-)-))) Similarly, for a fixed fo(-sī), the sign of
the derivative
aQa Juat ( does not depend on us st). Therefore, the optimal strategy is one of the
following three edge cases:
-o(-s²) (z) 0, fo(-s;) 1: Q0(-s)) ro(s))
ro(-s3) - -ro(sØ)
YH
YH
= ro(st)
(5() = 0, (-())) 0:
(ra(s))
YH
up(-55) = 1, fo (- ( + ar( (
ro($a)
ar( ar(-s;) (g(s;)
Qa( -5)
ro(- - -r((s)) - c.
Ore (8))
1
Ifc ا (0({5) ro( ( then the third case is dominated by the second case and the problem reduces
the case with no signaling. When c < ae(= ro( ro(-s)), the user's best response is as(=ÿ)
etrol- myleg) (
(1, (1, اe(-s
sū))
(0(3))
TH
ctra(s²) -rol))--o(-- - cxp(=))
10,
O.W.
Using specific values for type and type users above will complete the proof.
Theorem B.2 (Equilibrium under random entry). Let mg
RE be the margin of the algorithm's 's classifier
from the perspective of user type when all other user types follow the equilibrium strategy under random entry. When ma best strategy
RE A and YH # ro(a)/ro(b)) for user of type 0 E O2 the user's 's
S (b) 1, (B 1f77 <
ra(a)
(b)
Proof of Theorem B.2 We follow similar notation and conventions in the proof of Theorem 5.1 We use T and VÆ to denote the Q- and V-value under algorithmic entry. With this notation, the Bellman update for user type in Eq. 2 can be written as
(A s) fe(s)ro(s) + (H) VO (A6f(s)) + 72 fo(s)) VA (Ao(1 - f(s)))
Recall Vy = Q ( arr max, (A s)). Then, Lemmas C.1 and C.2 imply
VA (Ao f (s)) = (-ss).
There remains to determine
VAB (Xef(s)) From the theorem's assumption and using similar
argument as in Corollary 5.2 we can see that regardless of fe. when other user types follow the
equilibrium strategy, (A, h) 0. Then, Lemma C.2 implies (Aéf(a), h) 0, and Lemma C.I gives
VA (aof(a)) = Q²
We next consider two possibilities for VA Vy (b) and show they both yield similar optimal strategy for 0 E O₁:
Va (bof(b) -2 (a): In this case, we have the following Bellman update:
Qo(s) = fo(s) (s()) + (s) (a) + 7x -f(()) (-s).
Note that we have already found QA in the proof of Theorem 5.1 Now consider user type 0 E O and a prior P1 over initial content. The user's value is
Ve fe(a) re (a) + re(b) (b)
(o([1) (b)
Pi (a(
TM
To(b)
re(b) fe(a) re(a)
+ (1) + (6))
TH
1 (H)
29

---

Calculating 0f{(a]'
aV2_ we find
OV, r (uro(b)
pl(() + nM fola) PI(a)) + p1 pb)
(f)(a)
1 1-mafe(a)
1 1- (-
The term inside the brackets is always positive. Therefore,
conclude that
0,
YH > rw(a)
[o
f0 (a)
1,
YH < [b(
ro(a)
any value in [0, 1], YH ro(b]
ro(a)
VA (6f((b) (b): In this case, we have the following Bellman update:
Qa(s) = fo(s) (s) + (Hfo(s) (s) TH(( fo(s)) (-B(-
Now consider user type 0 E θ and a prior P1 over initial content. The user's value is
Vo =
p1
fo(a) (a) + (b)) ro(b) (p) (o((
TH
TH pI ro(b) ro(b) - So(a) (H)(((a)
YH
Calculating T)
aV. we find
VV, (fo(a)
ra(a) - (xro(()) (pi(a) + + so(a)pi(a) (nfe(a)
1 (x.o
1
The term inside the brackets is always positive. So, fo (a) is similar to the previous case.
Proof of Lemma C.1 Suppose g(s)) 2 (A. qq(s))) for some Lemma C.2 implies
(A, (q(s) - ())))) o (-8)) = - (A f(-s), q(-s) q(s))
S gA,
(())) (). q(s) q(-s))
Plugging this into QA(A, -s) as defined in Eq. 15 yields
Qa(A, s) 1 (A, ()- + 74 (A, (() q(-)))
1 (A, (()))) +7 (A, ((s) - ((-)) + TA) (A, -
(3)) S ().
This completes the proof.
Lemma C.2 We first find a simplified expression for q(s) - (((-)). Using the definition in Eq. 166 straightforward calculation gives
Proof
1 7A
1 ʃ(-s) a a(-s)) (25)
q(s) g(-s) (1 - 7Af (()) [f(s)
ass))
For content s, under Assumption 1 if = sy. then fo(s) = 1. Otherwise, either fo(s) = 1 or fo(s) < as(()//o(-s)) Therefore, we can divide ④ into two groups where in one group fo(s) and in the other group fo(s) < au(s)/ap(-s) and fo(-s) = 1. Using this, we have
(A g(s) q(-s)) = Σ (afo(s)(ao(s)) (-()))
d
Σ
{ofo(s) - q-{-s))+ + Σ Ao(ge(s) - go(-s))
0:fo(t)<
my(-x)
mg(s)
8:fo(e)=1
30

---

For the first group corresponding to the first sum above, Eq. 25 implies that
sign (g0(8) -0((-)) sign (Jo(s) ao(s) aol 8)
) -11
Therefore, we can conclude
(Ao f()))(())) g(-s)) 2 (A. a(-s))
Proof of Lemma C.3
Suppose (X, 2 (A, q( -)) for some Lemma C.2
implies
() (q(8) q( (8) f()) = (A f+)) ((s) - ((--))
(x, q(-8)) 2 0.
Plugging this into QA(A, s), as defined in Eq. 15 yields
QA(A, (A,
Proof of Lemma C.4 Assuming (A,1 f(s) > 0, Lemma C.2 implies
(A (1 f()). ((s) - 9(-))) I 0 = (Aō(1 - fs))) -((-s) s 0.
Then Lemma implies
mag Q4 (Aō(1 - f(8)), = (-s).
Finally, Lemma C.3 implies
max
= (Ax(1 f(8)),((-s))
Proof of Lemma C.5 Suppose (A, (s)) 2 (). (})) for some Lemmas C.2 and C.6 imply
(q(s) q(-s)) o (1 - u( ()) f( ()) = (A (1 - f(s)s - q(s))
S (A, - q(-)))
Plugging this into QA - -), as defined in Eq. 20 yields
QA(A, 8) S (A, q + + (A, (8)) - q(-s))
< (A, + (()) a(-s)) + (1- (1-74) 7A) (A, a(-s))
({)) (Q)
This completes the proof.
Proof of Lemma C.6 We first find a simplified expression for q(s) - q((s). Using the definition
in Eq. 21 and u(s) u(-s) 0 and u(s) (I - f(-s)) = 0 as implied by Assumption 2
straightforward caleulation gives
((8)-9(-8) (1-71(1 u(s)) f(x)) o (1 71 u(-s)) f(-s)) of [f(())
1 74
f( ((-8)o ax(-s)
-
(xss)
(26)
For a content s, under Assumption 2, whenever (s(s) > O, we have fo(s) < (o(s)/an(-9)
Therefore, we can divide θ into two groups where in one group us us(s) and in the other group u0(s) > 0, fo(s) < aa(s))/o((==) (1
q(-s))
Σ Xo(1 (s))) go(-s))
8E0
Σ
Aa uo()) go(-s))
Σ )o(go(s) go(-s))-
O:N (8)20
(=uo(s)=
31

---

For the first group corresponding to the first sum above, we know from Assumption 2 that s = (-sā). In this case, Assumption implies (o(-s) Plugging this into Eq. 25 we have
sign (ga(s) go(-s)) ssign (s)(s) (50(s)
cra(-s) 1.
Therefore, we can conclude
(A (1 (1 (1--)())),q(()) q(-s) 2 (A, q(s) - a((s)))
Proof Lemma C.7 Suppose (A, q())) 2 (X. q( -)) for some Lemmas C.2 and C.6 imply
(A. (g(()) (-8)) (1 u())) f()) = (A o (1 u())) o f()), ((8) q 8))
(), q(-s)) 2 0.
Plugging this into QA(A, s), defined in Eq. (20). yields
QA(A, 8) = ((s))
Proof of Lemma C.8
Assuming (A, (s) g()) > 0, Lemma C.6 implies
(X g(s) us)) (s) q( s)) I 0 = (Aé(u(s) o (()) q(s) q(-s)) s 0.
Then Lemma C.5
implies
arg max QA (Aé((((( - g())) so) (-s).
Finally, Lemma C.7
implies
(Ao(u(s) gg())) (G(s)())0g)))) q(-s))
max
32


Reviews:
[{"review_id": "bWzCxPzIcc", "reviewer": "Reviewer_7AFQ", "summary": "Recommendation algorithms learn user preferences using user behavior, which may be rational (System 2) or short-sighted/greedy (System 1). Due to the greedy behavior, the algorithms may be unable to distinguish between users whose rational selves agree/disagree with their greedy selves. To address this, users must strategically choose their rational behavior so as to signal their type to the algorithm. This paper develops a theoretical model of this setting as a Stackleberg game, and analyzes Bellman equations to show that (a) users may need to substantially sacrifice / alter their behavior in order to \"steer\" the algorithm, (b) users who highly discount future rewards will be unable to steer the algorithm at all, and (c) a costly signal may mitigate the burden to users.", "questions": "1. What recommendation setting motivates this work?\n\n2. How restrictive the special case in 3.4? Does it capture most settings? Do you have a sense of how the results extend (steering becomes easier/harder) beyond this special case?\n\n3. Is a \"show less like this\" button seen on a few platforms currently an example of a costly signal? What would motivate a platforms to implement a costly signaling feature-- is there a burden on the platform as well?\n\n4. Modeling questions:\n- On a session-level, why would the algorithm discount future rewards?\n- When computing the algorithm's Q-value, why does the algorithm assume the users will continue to use the same strategy $f$ in the future?\n- Is pulling the expected duration into the reward $r_\\theta$ (in line 120) really without loss of generality? Does it result in some additional dependence on $\\gamma_H$?\n- What does \"when users have consistent preferences ... the alignment problem reduces to engagement maximization\" mean on page 1? My intuition is that there is misalignment even with only System 2 (e.g. some content could have a reward only after time $T$ has been reached, like a suspenseful movie, and the algorithm is motivated to promote that content). Is this misalignment possible in general or only because I chose pathological rewards? If it can occur in general, do you have a sense of how hard it is for a user to steer the algorithm without inconsistent preferences (as a baseline)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I enjoyed reading this paper; the paper was well organized, and the authors provided clear and intuitive interpretations of the theoretical results, which appear non-trivial. Recent literature has examined both strategic and inconsistent content consumption, but this is a novel bridge between these lines of work. The findings are also interesting - users may potentially require great energy and foresightedness to align the algorithm due to inconsistent preferences, and sometimes it is in fact impossible to align.\n\nOne weakness is that it's not clear what setting motivates this work: in what type of content recommendation does this phenomenon (users strategically determining which content to consume, but then greedily determining how long to consume it for) occur? In particular: in what settings/platforms does a user consume a *particular* piece of content for an undefined length, and the platform wants to maximize this length of time.\n\nMoreover, while the model is quite general, the results are all in a special case where there are two items.\n\nI also have a few quibbles/clarifications about the modeling decisions, which I ask about below. It would also be interesting to have additional interpretable insights from the model, such as discussion of which types of users the burden of alignment is highest for (with and without costly signaling).", "was_revised": true, "final_justification": "I will maintain my positive score. The authors' rebuttal has addressed my questions.", "rebuttal": "We thank the reviewer for their thorough and insightful feedback. We couldn’t have asked for a more engaged and in-depth review of our work. We are especially pleased that the reviewer found the paper well-organized and our interpretations clear and intuitive. Below, we address the specific questions raised.\n\n> What recommendation setting motivates this work?\n\nWe have intentionally presented our setting in an abstract form because we believe it encompasses a wide range of applications—an observation echoed by reviewers, to our excitement. To provide a more concrete illustration and clarify the model’s assumptions and generality, we begin with a simple example of the interaction structure, followed by a discussion of how our user decision-making model builds on prior work. We apologize in advance as this response will be on the long side.\n\nConsider the following example: a user opens a music or video recommender system with a specific intent. Upon seeing a recommendation—say, a country song—the user decides whether to engage (i.e., listen) or to skip and disengage for a while. Once the user chooses to engage, however, they may relinquish rational control over how long they remain engaged, with the actual duration influenced by the content’s emotional or addictive pull.\nSuppose this user is working while listening and would benefit most from calm music. However, they are a fan of a particular artist, X, whose songs may be more distracting. The platform profits from longer engagement—via ad revenue or increased interaction—and therefore has an incentive to repeatedly recommend content like X’s music. In this case, the user exhibits inconsistent preferences: they prefer calm music for productivity, but are tempted by content that undermines that goal. This type of user falls under our \"type 1\" users, where the reward and engagement duration are misaligned across content types (here, X’s music vs. calm music). Notably, in a different context—say, when the user is relaxing rather than working—their preferences may align with the platform’s and they are \"type 2\" users. This motivates our session-based modeling of interactions.\n\nOur framework extends beyond recommender systems. For instance, a chatbot that charges per API call may operate in various \"modes.\" In an educational mode, longer sessions may be valuable for a student, aligning incentives. But for an engineer seeking a quick answer, a shorter session is preferable. Similarly, a therapy chatbot operating in an \"affirmative\" mode might prolong interaction by being emotionally validating, even if it does not help the user confront deeper issues. These examples all fit our model, provided we flexibly define user types and item spaces, even approximately, and work with a tractable number of them.\n\nTurning to our user model: there is ample empirical support for user strategizing. For example, Cem et al. observed that \"[some users] ignore content they actually like to\" in order to shape future recommendations. Likewise, Haupt et al. found that users \"explicitly and actively curate their feeds in order to influence the platform to serve them content of a *particular type* in the future.\" These findings indicate that users engage in strategic behavior.\n\nOur user model—where users strategically choose whether to engage with content, but then greedily decide how long to engage—is inspired by the seminal work of Kleinberg et al. As they note, human preferences are inherently inconsistent and psychologically rich, but their \"two-mind\" model offers a tractable abstraction. In this model, System 2 represents the long-run self with \"actual\" preferences and makes the engagement decision. Once the user engages, System 1—the impulsive self—takes over, and the duration of engagement becomes independent of System 2’s preferences.\n\nThis is exactly how we model inconsistent preferences. Kleinberg et al.’s framework, along with supporting references, directly informs our approach. Taken together, the empirical evidence for user strategizing and the two-mind model suggest that any strategic behavior arises at the engagement decision point (System 2), not in the engagement length (System 1).\n\n> How restrictive the special case in 3.4?\n\nWe acknowledge that the special case presented in Sec. 3.4 is restrictive in that it limits content to two \"types.\" However, this abstraction may be more general than it appears: in many settings, item types can be interpreted broadly, and with some approximation, users may consistently benefit more from one type than from another, more tempting alternative. We believe our closed-form results likely extend beyond this case, as the underlying structure often appears as a component of more complex patterns observed in practice. That said, the theoretical results may no longer be as elegant or interpretable outside this setting.\n\n> Is a \"show less like this\" button seen on a few platforms currently an example of a costly signal? What would motivate a platforms to implement a costly signaling feature?\n\nThis is a great question—and in fact, the \"show less like this\" button was the original motivation for our research on how costly signaling can help! Regarding the burden on platforms, there is indeed a cost to them as well. As one can see in our special case, maximizing engagement alone does not incentivize the implementation of costly signals. While somewhat speculative, the shift from earlier social media platforms, which required more deliberate actions (e.g., clicks, menu options), to modern platforms featuring frictionless scrolling may reflect the platform’s disincentive to support such signaling mechanisms.\n\n**Modeling questions:**\n\n> On a session-level, why would the algorithm discount future rewards?\n\nWe mainly view the possibility that the platform discounts future rewards as a theoretical curiosity that further generalizes our model. However, there are also practical scenarios where this assumption is reasonable. For instance, in time-sensitive campaigns, such as those with fixed deadlines, the timing of content consumption directly affects its value to the platform (for example, by going viral). Allowing for future-discounting algorithms also enabled us to uncover interesting insights—particularly regarding steerable sets, such as the observation in line 237: \"as the algorithm becomes more foresighted, even slight disengagement from type 1 users can effectively influence its behavior.\"\n\n> When computing the algorithm's Q-value, why does the algorithm assume the users will continue to use the same strategy $f$ in the future?\n\nWe assume that within a fixed session, users commit to a strategy $f$  that depends only on the content. While we acknowledge that users may adopt more complex strategies—such as those that vary over time within a session—we currently have no empirical evidence supporting such behavior. We therefore focus on a simpler strategy, which is (1) more plausible given typical user behavior, and (2) more feasible for the platform to learn. That said, this is certainly an assumption, and we recognize its limitations.\n\n> Is pulling the expected duration into the reward $r_\\theta$ (in line 120) really without loss of generality?\n\nWe have implicitly assumed that users discount rewards between interactions, but not during a single interaction. Under this assumption, line 120 holds without loss of generality. However, we acknowledge that if the duration of engagement is comparable to the time between interactions, it is indeed plausible that users also discount rewards within each engagement—introducing dependence to $\\gamma_\\mathcal{H}$. This would significantly complicate the analysis, as the distribution of engagement lengths (not just their expectation) would become relevant—an aspect our current model intentionally abstracts away for tractability—and likely only distorts the results negligibly. We thank the reviewer for noting this, and we will clarify the assumption more explicitly in the revised version.\n\n> What does \"when users have consistent preferences ... the alignment problem reduces to engagement maximization\" mean on page 1? \n\nBy the mentioned sentence, we simply mean that if $r_\\theta(s)$ and $1/\\alpha_\\theta(s)$ induce the same ordering over $s$, then the user and the algorithm have aligned interests. In this case, both aim to reach the same item $s^*_\\theta = \\arg\\max_s r_\\theta(s)$, which also maximizes the expected engagement length $1/\\alpha_\\theta(s)$. We had some difficulty understanding the example provided by the reviewer here. If the reviewer is referring to a scenario where a content $s$ yields a reward only after $T$ steps of interaction have passed, then while this is indeed interesting, it falls outside the scope of our model. In our setting, the platform’s reward—equivalently, the engagement length—depends only on the content and a time discount factor, not on when during the session (e.g., before or after time $T$) the content is consumed. On the other hand, if the reviewer means that the reward from a content $s$ is realized only if the user remains engaged for at least length of $T$, then the platform’s reward becomes $1/\\alpha_\\theta(s) = \\mathbb{E}_\\theta[1\\\\{y > T\\\\}]$, in which case our analysis still holds as described. We hope this addresses the reviewer’s question and would be happy to further clarify if needed."}, {"review_id": "ucJzSVoDWs", "reviewer": "Reviewer_bpFx", "summary": "This paper studies a setting where humans sometimes act in ways that are systematically at odds with their true preferences (following the system 1 system 2 model in Kleinberg & Raghavan). An algorithm that aims to maximize engagement is attempting to suggest content to a human, where some humans have preferences aligned with the algorithm’s prior, and some have preferences at odds with it. This paper studies when humans can effectively strategize to end up being suggested the content that they prefer.", "questions": "Highlighted in the \"weaknesses\" section - in particular, my questions about the meaning of the user being more or less myopic, and the policy implications.", "limitations": "Yes - but more discussion (and connection beyond theory) could be helpful.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThis paper studies an interesting problem, and ties its work to prior work both in practice(Haupt et al and Cen et al) and theory (Kleinberg & Raghavan). I appreciated that the paper included several extensions (e.g. including ‘costly signaling’, which was a variant that I hadn’t immediately thought of). \n\nWeaknesses: \nA few parts of the model felt unclear. For example, one crucial factor is the human’s time horizon, which influences their strategy as it influences how long they can take the costly action of signaling a different preference: in section 5, it is shown that if a user has a low enough time horizon, then they may incur relatively heavy regret. However, in real-life settings, users may have relatively short time horizons precisely because they know they will not be using the app for very long: for example, if a user installs an app that they know they will only use for a short period of time (e.g. in a country that they infrequently visit), then they may (rationally) decide that they don’t want to invest the effort in steering what the app may suggest to them. By the results in this paper, this would imply a *high* regret, but in reality this would likely lead to a low regret. In this paper, I believe the authors are focusing on the case where a consumer’s time horizon is driven solely by how much effort they wish to put in (or how myopic they are), rather than any actual desire to leave the platform. While I don’t think this substantially changes the core of the paper, it would be useful for the paper to note the different reasons why users may have different time horizons. \n\nAs one other critique, the paper is very theory-heavy - which I appreciate, as its main goal is clearly a theoretical understanding. However, I would have appreciated more explicit connection to practice or policy, even in the appendix. For example, what would the results in Section 6 imply about how algorithms should be designed? How would the results change if algorithms have different incentives beyond engagement (e.g. different values for items)?", "was_revised": true, "final_justification": "I enjoyed reading this paper, and during the final discussion the authors convinced me that the time horizon issue I had brought up is less relevant. I was on the fence about raising my score (if I could raise it a half-unit I likely would).", "rebuttal": "We thank the reviewer for their insightful review of our work. We’re glad they found the problem interesting and appreciated how it connects prior theoretical and empirical research. We’re also pleased that the reviewer recognized the value of our proposed extensions. Below, we address the reviewer’s questions and concerns.\n\n>  it is shown that if a user has a low enough time horizon, then they may incur relatively heavy regret. However, in real-life settings, users may have relatively short time horizons precisely because they know they will not be using the app for very long. … I believe the authors are focusing on the case where a consumer’s time horizon is driven solely by how much effort they wish to put in, rather than any actual desire to leave the platform.\n\nThe reviewer’s interpretation is correct, and we would like to use this example as an opportunity to further clarify our model. When calculating regret, we consider a session that can last arbitrarily long, where the user has no intention of leaving the platform but is instead strategizing to maximize the value of the interaction. Our notion of a discount factor captures how patient the user is across these interactions or over what horizon they are capable of optimizing. As the reviewer pointed out, this is not the horizon that the user might intend to leave. We agree that if the user is aware of an early cutoff time within the session, they may prioritize immediate consumption over long-term optimization. However, if this cutoff is sufficiently far in the future—such as planning to delete the app a year from now—we expect our results to apply readily.\n\n> The paper is very theory-heavy - which I appreciate, as its main goal is clearly a theoretical understanding. However, I would have appreciated more explicit connection to practice or policy, even in the appendix. For example, what would the results in Section 6 imply about how algorithms should be designed? \n\nOur goal in this theoretical paper is to provide a first-of-its-kind analysis of aligning with inconsistent preferences and the burden this places on strategic users. We have intentionally presented the model and results in abstract yet interpretable terms to highlight the generality of the framework and its potential applications. That said, we agree that additional context and stronger connections to practice would be helpful. To that end, we have included concrete examples in our response to Reviewer 7AFQ, which we hope the reviewer will also find useful.\n\nRegarding the results in the presence of costly signaling (Sec 6), we show that giving users just one additional option—even one that simply burns effort—can significantly improve their ability to steer the algorithm. As noted by Reviewer 7AFQ, a small \"show less like this\" button, like those found on older platforms, can serve this purpose. Such costly actions, while not directly altering engagement with the content, send a strong signal to the platform and can ease the burden of alignment on the user.\n\nFrom a policy perspective, our special case already illustrates that engagement maximization alone is not sufficient to motivate platforms to offer diverse options to users. In fact, the incentives often run in the opposite direction: platforms may reduce friction as much as possible, leading to the kind of seamless, mindless scrolling now ubiquitous in video recommenders. Of course, these are speculative insights derived from our model. Strong empirical work is needed to bridge theory and policy. As Dean et al. put it, we hope that formal mathematical models like ours can guide and inspire future empirical research by offering valuable insights—while recognizing that they do not provide final answers.\n\n> How would the results change if algorithms have different incentives beyond engagement?\n\nThis is a great question. Our theoretical framework does in fact accommodate this case. One simply needs to define $1/\\alpha_\\theta(s)$ as the expected value—or more generally, whatever measure of utility—the platform assigns to content $s$. The rest of our results then apply directly. We chose to define $1/\\alpha_\\theta(s)$ as expected engagement length in our exposition to simplify the presentation, as many platforms monetize based on engagement duration. However, our results are more general and extend beyond this specific interpretation."}, {"review_id": "6twAHG9CEZ", "reviewer": "Reviewer_8uFD", "summary": "The authors study an Nash equilibrium of a game where two parties, \n(recommendation) algorithm and user, interact.  In this setting algorithm\nrecommends content, which user then may be engage for certain time. The user is\nthen rewarded afterwards.  The objective of the algorithm is to maximize the\nengagement of the user, whereas the goal of each user is to maximize the\nreward.", "questions": "- Are these results (easily) extendable to n items?\n- What are the practical implications for recommendation engines?", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "+ Interesting results regarding the equilibrium, especially with inconsistent rewards. These contributions advance the field, and are sufficient for a publication.\n\n- The contributions are all theoretical. The main downside with the paper is that it does not have any experiments or any use case study. It is not clear what are the practical implications of these results. Some experiments or a small use case study would alleviate these issues.\n- The authors motivate their example using recommendation algorithms. Yet the related work\n  in that domain is not thoroughly discussed. The authors should have discussed the recommendation algorithm in the related work more thoroughly.\n- The paper focuses on cases with two items. The results should be extendable, \n  so it would be helpful to see a more general results, especially for Theorem 4.1.", "was_revised": true, "final_justification": "I have read the comments provided by the authors. The theoretical contributions are sufficient for publication.\nThe major downside of the paper is the lack of experiments, and because of that it is not clear what are the exact practical implications of the authors results. Some set of experiments providing some connections of the theoretical results to applications would have alleviated these concerns.", "rebuttal": "We thank the reviewer for taking the time to read and review our work. We’re glad they found our results interesting and hope they find our responses to their concerns below to be convincing.\n\n> the contributions are all theoretical … what are the practical implications for recommendation engines?\n\nAs Dean et al. put it, we believe that formal mathematical models like ours can offer valuable insights and help guide empirical research—particularly in evaluating and motivating alternative designs, such as our analysis of costly signaling. We fully agree with the reviewer that theoretical results, no matter how strong, are only part of the answer to broader policy questions.\n\nIn terms of implications, beyond formalizing the burden that alignment places on users with inconsistent preferences, our findings on how costly signaling can alleviate this burden have significant practical relevance. As our special case illustrates, platforms that maximize engagement length often have no incentive—and may even have a disincentive—to introduce friction or support user signaling. This is reflected in the widespread shift toward frictionless scrolling interfaces on video platforms.\n\nHowever, our framework suggests that introducing diverse options—even those that don’t directly affect engagement but instead act as costly signals of user intent (such as a “show less like this” button, as suggested by Reviewer 7AFQ)—can help reduce the alignment burden. Our work also raises additional practical questions, such as how middleware tools (as discussed by Fukuyama et al.) might be used to extend users' effective horizons when their own patience is insufficient for alignment. We thank the reviewer for raising this important point and will be sure to expand on these connections in the updated manuscript.\n\nFukuyama, et al. (2020). Report of the working group on platform scale. Program on Democracy and the Internet, Stanford University\n\n> The authors motivate their example using recommendation algorithms. Yet the related work in that domain is not thoroughly discussed.\n\nWe’re sorry to hear that the reviewer found our related work section insufficient. We would like to clarify that our focus has primarily been on recent theoretical modeling efforts in this space, including works by Haupt et al., Kleinberg et al., Zhao et al., Haghtalab et al., and Hajiaghayi et al. While space constraints limited what we could include in the main text, we have done our best to expand on additional relevant work in Appendix A. We would greatly welcome any suggestions the reviewer may have for further enriching this section.\n\n>  Are these results (easily) extendable? … it would be helpful to see a more general results, especially for Theorem 4.1\n\nThe abstraction in Sec. 3.4 and then Thm 4.1 may be more general than it appears: in many settings, item types can be interpreted broadly, and with reasonable approximations, users may consistently benefit more from one type over another, more tempting alternative. We believe our closed-form results here likely extend beyond this special case, as this structure often arises as a component of more complex patterns encountered in practice. That said, we currently do not have formal results beyond simulation evidence, and we expect that the theoretical results may become less elegant or interpretable outside this simplified setting."}, {"review_id": "etDVnoouRg", "reviewer": "Reviewer_HcaA", "summary": "In this paper, the authors introduce a model to study how users with inconsistent preferences (i.e., choices don’t always represent true preferences) can align algorithms with their true interests. The interaction is modeled as a Stackelberg game where the user is the leader, committing to a strategy, and the algorithm is the follower. The user has a “system 2” for engagement decisions and a “system 1” for the duration of such engagement, while the algorithm maximizes engagement. They define the \"burden of alignment\" as the level of foresight a user needs to steer the algorithm effectively (horizon). Their theoretical results show that foresighted users can improve alignment, which can be challenging, and the burden can be reduced with a costly signal for the user (e.g., clicks to show disinterest). Otherwise, myopic users align with the algorithm’s objective.", "questions": "- Line 167: As a minor notation detail, it would be helpful for clarity to define acronyms for random entry (RE) and algorithmic entry (AE).\n- Line 302: Something is wrong with this sentence.", "limitations": "Yes, in the Discussion section, and other assumptions throughout the manuscript", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a great paper. It offers a rigorous theoretical framework of interactive alignment by modeling the user-algorithm dynamic as a Stackelberg game with the user as a leader. Its primary strength lies in departing from the simplifying assumption of user rationality and instead focusing on what users can do when they carry the burden of alignment by interacting with an engagement-maximizing algorithm. \n\nIt provides theoretical guarantees under well-defined assumptions: abstracting the algorithm’s learning process, system 2 decides engagement and system 1 its duration, the algorithm maximizes engagement, users are aware of their rewards, the algorithm has full knowledge of strategies, etc.\n\nWhile the paper frames the problem of inconsistent preferences in general terms, this same setup applies to the real phenomenon of doomscrolling. I’m not pointing this out as a limitation or a weakness, but maybe as something to consider. This theoretical framework could be used to study this effect—from the user and algorithm perspective—to improve users’ well-being. I see that as a strength.\n\nIn the paper, the user is modeled by their type $\\theta \\in \\Theta$ for a specific session, allowing the same user to be of different types across sessions. However, the user’s foresight $\\gamma_H$ is fixed, and it might be interesting to consider a dynamic discounting factor for a user. In reality, a person's ability to be foresighted can fluctuate based on internal factors. For example, a user who is typically strategic might enter a myopic state (doomscrolling) for some sessions. I wonder what the burden of alignment is in this situation. Would it degrade over time in this tug of war, where your myopic sessions ruin your strategic ones? Would the costly signals be able to reduce the burden? What happens when a lot of users doomscroll at the same time (sending a strong signal to the algorithm) during moments of political turmoil or catastrophes? I could also see how a user can resist the temptation of item `a` but not if this item is shown repeatedly.", "was_revised": true, "final_justification": "I think this paper is great and should be accepted. The authors study an important and very interesting problem: a scenario of inconsistent preferences when users deal with an engagement-maximizing algorithm. They define the \"burden of alignment\" as the level of foresight a user needs to steer the algorithm effectively (horizon). Their theoretical results show that foresighted users can improve alignment, which can be challenging, and the burden can be reduced with a costly signal for the user (e.g., clicks to show disinterest)\n\nI didn't find any weaknesses when reviewing the paper, but I had some suggestions that the authors addressed during the discussion period.", "rebuttal": "We thank the reviewer for their thorough and positive review of our work. We’re truly heartened that the reviewer described the paper as “great,” and appreciated its rigorous theoretical framework and guarantees under well-defined assumptions. We’re especially pleased that the reviewer was able to draw connections to real-world phenomena and raised thoughtful questions.\n\n> ... it might be interesting to consider a dynamic discounting factor for a user. In reality, a person's ability to be foresighted can fluctuate based on internal factors. For example, a user who is typically strategic might enter a myopic state (doomscrolling) for some sessions. I wonder what the burden of alignment is in this situation. Would it degrade over time in this tug of war, where your myopic sessions ruin your strategic ones? Would the costly signals be able to reduce the burden? What happens when a lot of users doomscroll at the same time (sending a strong signal to the algorithm) during moments of political turmoil or catastrophes?\n\nThese are great questions! While we can offer some intuitive answers within the scope of this rebuttal, several of these points suggest promising directions for future work. They also highlight how our framework—and its departure from standard rationality assumptions in alignment—can raise and help address new, nuanced questions.\n\nRegarding fluctuating user’s discount factor, for simplicity, first suppose $\\gamma_{\\mathcal{H}}$ is fixed across users but can vary over time for a given user. In this case, the critical threshold for alignment remains mostly as before; the key question becomes whether this user’s $\\gamma_{\\mathcal{H}}$ exceeds this threshold at any given time. Now, consider a scenario in which $\\gamma_{\\mathcal{H}}$ of the user decreases when the user is relaxing or is a \"type 2\" user in our special case. Since misalignment is minimal in such contexts, fluctuations in patience are unlikely to affect outcomes significantly. However, if the user becomes more myopic precisely when exposed to tempting but misaligned content—as in the case of \"type 1\" users—their effective horizon may become too short to steer the algorithm effectively.\n\nIn more complex settings where the $\\gamma_{\\mathcal{H}}$ values of many users change simultaneously, externalities could arise, affecting the critical effective horizon required for alignment. This is a particularly intriguing question and a compelling direction for future research.\n\n> Re. Line 167 and 302 \n\nWe apologize for the typo in line 302. The sentence should read: “Given the algorithm’s best response, we now formally characterize the equilibria under algorithmic entry when users…” We will also clarify the acronyms RE and AE at this point in the text. We thank the reviewer for their careful reading and for pointing out these issues."}]

Author rebuttal:

Meta-review: The authors examine a hypothetical recommender system where users display multiple behavior profiles, and interact with the recommender system over a sequence of item selections. They characterize the possibility of the user steering the algorithm towards its preferences in terms of the length of the interaction horizon, or burden of alignment, as the authors put it. Reviewers were excited about the motivation of the paper, the non-trivial theoretical results it provides, and potential for follow-up work. However some concerns about the strength of the empirical results were raised, along with uncertainties about the relevance to practical recommender systems. Overall, I recommend accepting the paper, and encourage the authors to integrate the feedback from the reviewers, with a special focus on properly contextualizing the results so that future work—especially those focused on empirics—can effectively extend the key ideas of the paper.
Decision: Accept (poster)