Title: Strategic Apple Tasting
Venue: NeurIPS 2023
Accepted: yes
Keywords: strategic classification

strategic learning

apple tasting

bandit feedback

learning with incentives
Abstract: Algorithmic decision-making in high-stakes domains often involves assigning decisions to agents with incentives to strategically modify their input to the algorithm. In addition to dealing with incentives, in many domains of interest (e.g. lending and hiring) the decision-maker only observes feedback regarding their policy for rounds in which they assign a positive decision to the agent; this type of feedback is often referred to as apple tasting (or one-sided) feedback. We formalize this setting as an online learning problem with apple-tasting feedback where a principal makes decisions about a sequence of $T$ agents, each of which is represented by a context that may be strategically modified. Our goal is to achieve sublinear strategic regret, which compares the performance of the principal to that of the best fixed policy in hindsight, if the agents were truthful when revealing their contexts. Our main result is a learning algorithm which incurs $\tilde{\mathcal{O}}(\sqrt{T})$ strategic regret when the sequence of agents is chosen stochastically. We also give an algorithm capable of handling adversarially-chosen agents, albeit at the cost of $\tilde{\mathcal{O}}(T^{(d+1)/(d+2)})$ strategic regret (where $d$ is the dimension of the context). Our algorithms can be easily adapted to the setting where the principal receives bandit feedback---this setting generalizes both the linear contextual bandit problem (by considering agents with incentives) and the strategic classification problem (by allowing for partial feedback).

Strategic Apple Tasting

Abstract

Algorithmic decision-making in high-stakes domains often involves assigning decisions to agents with incentives to strategically modify their input to the algorithm. In addition to dealing with incentives, in many domains of interest (e.g. lending and hiring) the decision-maker only observes feedback regarding their policy for rounds in which they assign a positive decision to the agent; this type of feedback is often referred to as apple tasting (or one-sided ) feedback. We formalize this setting as an online learning problem with apple-tasting feedback where a principal makes decisions about a sequence of T agents , each of which is represented by a context that may be strategically modified. Our goal is to achieve sublinear strategic regret , which compares the performance of the principal to that of the best fixed policy in hindsight, if the agents were truthful when revealing their contexts . Our main result is a learning algorithm which incurs ˜ O ( √ T ) strategic regret when the sequence of agents is chosen stochastically . We also give an algorithm capable of handling adversarially-chosen agents, albeit at the cost of ˜ O ( T ( d +1) / ( d +2) ) strategic regret (where d is the dimension of the context). Our algorithms can be easily adapted to the setting where the principal receives bandit feedback-this setting generalizes both the linear contextual bandit problem (by considering agents with incentives) and the strategic classification problem (by allowing for partial feedback).

1 Introduction

Algorithmic systems have recently been used to aid in or automate decision-making in high-stakes domains (including lending and hiring) in order to, e.g., improve efficiency or reduce human bias [12, 1]. When subjugated to algorithmic decision-making in high-stakes settings, individuals have an incentive to strategically modify their observable attributes to appear more qualified. Such behavior is often observed in practice. For example, credit scores are often used to predict the likelihood an individual will pay back a loan on time if given one. Online articles with titles like '9 Ways to Build and Improve Your Credit Fast' are ubiquitous and offer advice such as 'pay credit card balances strategically' in order to improve one's credit score with minimal effort [46]. In hiring, common advice ranges from curating a list of keywords to add to one's resume, to using white font in order to 'trick' automated resume scanning software [23, 2]. If left unaccounted for, such strategic manipulations could result in individuals being awarded opportunities for which they are not qualified for, possibly at the expense of more deserving candidates. As a result, it is critical to keep individuals' incentives in mind when designing algorithms for learning and decision-making in high-stakes settings.

In addition to dealing with incentives, another challenge of designing learning algorithms for highstakes settings is the possible selection bias introduced by the way decisions are made. In particular, decision-makers often only have access to feedback about the deployed policy from individuals that have received positive decisions (e.g., the applicant is given the loan, the candidate is hired to the job and then we can evaluate how good our decision was). In the language of online learning, this type of feedback is known as apple tasting (or one-sided ) feedback. When combined, these two complications (incentives & one-sided feedback) have the potential to amplify one other, as algorithms can learn

only when a positive decision is made, but individuals have an incentive to strategically modify their attributes in order to receive such positive decisions, which may interfere with the learning process.

1.1 Contributions

We formalize our setting as a game between a principal and a sequence of T strategic agents , each with an associated context x t which describes the agent. At every time t ∈ { 1 , . . . , T } , the principal deploys a policy π t , a mapping from contexts to binary decisions (e.g., whether to accept/reject a loan applicant). Given policy π t , agent t then presents a (possibly modified) context x ′ t to the algorithm, and receives a decision a t = π t ( x ′ t ) . If a t = 1 , the principal observes reward r t ( a t ) = r t (1) ; if a t = 0 they receive no feedback. r t (0) is assumed to be known and constant across rounds. 1 Our metric of interest is strategic regret , i.e., regret with respect to the best fixed policy in hindsight, if agents were truthful when reporting their contexts .

√

Our main result is an algorithm which achieves ˜ O ( T ) strategic regret (with polynomial per-round runtime) when there is sufficient randomness in the distribution over agents (Algorithm 1). At a high level, our algorithm deploys a linear policy at every round which is appropriately shifted to account for the agents' strategic behavior. We identify a sufficient condition under which the data received by the algorithm at a given round is 'clean', i.e. has not been strategically modified. Algorithm 1 then onlinelearns the relationship between contexts and rewards by only using data for which it is sure is clean.

In contrast to performance of algorithms which operate in the non-strategic setting, the regret of Algorithm 1 depends on an exponentially-large constant c ( d, δ ) ≈ (1 -δ ) -d due to the one-sided feedback available for learning, where d is the context dimension and δ ∈ (0 , 1) is a parameter which represents the agents' ability to manipulate. While this dependence on c ( d, δ ) is insignificant when the number of agents T → ∞ (i.e. is very large), it may be problematic for the principal whenever T is either small or unknown. To mitigate this issue, we show how to obtain ˜ O ( d · T 2 / 3 ) strategic regret by playing a modified version of the well-known explore-then-commit algorithm (Algorithm 2). At a high level, Algorithm 2 'explores' by always assigning action 1 for a fixed number of rounds (during which agents do not have an incentive to strategize) in order to collect sufficient information about the data-generating process. It then 'exploits' by using this data learn a strategy-aware linear policy. Finally, we show how to combine Algorithm 1 and Algorithm 2 to achieve ˜ O (min { c ( d, δ ) · √ T, d · T 2 / 3 } ) strategic regret whenever T is unknown.

While the assumption of stochastically-chosen agents is well-motivated in general, it may be overly restrictive in some specific settings. Our next result is an algorithm which obtains ˜ O ( T ( d +1) / ( d +2) ) strategic regret when agents are chosen adversarially (Algorithm 4). Algorithm 4 uses a variant of the popular Exp3 algorithm to trade off between a carefully constructed set of (exponentially-many) policies [6]. As a result, it achieves sublinear strategic regret when agents are chosen adversarially, but requires an exponentially-large amount of computation at every round.

Finally, we note that while our primary setting of interest is that of one-sided feedback, all of our algorithms can be easily extended to the more general setting in which the principal receives bandit feedback at each round, i.e. r t (0) is not constant and must be learned from data. To the best of our knowledge, we are the first to consider strategic learning in the contextual bandit setting.

1.2 Related work

Strategic responses to algorithmic decision-making There is a growing line of work at the intersection of economics and computation on algorithmic decision-making with incentives, under the umbrella of strategic classification or strategic learning [27, 3, 41, 22] focusing on online learning settings [21, 19], causal learning [52, 31, 29, 34], incentivizing desirable behavior [39, 28, 11, 42], incomplete information [30, 25, 37]. In its most basic form, a principal makes either a binary or real-valued prediction about a strategic agent, and receives full feedback (e.g., the agent's label ) after the decision is made. While this setting is similar to ours, it crucially ignores the one-sided feedback structure present in many strategic settings of interest. In our running example of hiring, full feedback would correspond to a company not offering an applicant a job, and yet still getting to observe whether they would have been a good employee! As a result, such methods are not applicable in our setting. Concurrent work [18] studies the effects of bandit feedback in the related problem of performative prediction [47], which considers data distribution shifts at the population level in response to the deployment of a machine learning model. In contrast, our focus is on strategic responses to machine

learning models at the individual level under apple tasting and bandit feedback. Ahmadi et al. [4] study an online strategic learning problem in which they consider 'bandit feedback' with respect to the deployed classifier . In contrast, we use the term 'bandit feedback' to refer to the fact that we only see the outcome when for the action/decision taken.

Apple tasting and online learning Helmbold et al. [32] introduce the notion of apple-tasting feedback for online learning. In particular, they study a binary prediction task over 'instances' (e.g., fresh/rotten apples), in which a positive prediction is interpreted as accepting the instance (i.e. 'tasting the apple') and a negative prediction is interpreted as rejecting the instance (i.e., not tasting the apple). The learner only gets feedback when the instance is accepted (i.e., the apple is tasted). While we are the first to consider classification under incentives with apple tasting feedback, similar feedback models have been studied in the context of algorithmic fairness [9], partial-monitoring games [5], and recidivism prediction [24]. A related model of feedback is that of contaminated controls [40], which considers learning from (1) a treated group which contains only treated members of the agent population and (2) a 'contaminated' control group with samples from the entire agent population (not just those under control ). Technically, our results are also related to a line of work in contextual bandits which shows that greedy algorithms without explicit exploration can achieve sublinear regret as long as the underlying context distribution is sufficiently diverse [49, 8, 38, 53, 48].

Bandits and agents A complementary line of work to ours is that of Bayesian incentive-compatible (BIC) exploration in multi-armed bandit problems [43, 35, 50, 36, 44, 45]. Under such settings, the goal of the principal is to persuade a sequence of T agents with incentives to explore across several different actions with bandit feedback. In contrast, in our setting it is the principal, not the agents, who is the one taking actions with partial feedback. As a result there is no need for persuasion, but the agents now have an incentive to strategically modify their behavior in order to receive a more desirable decision/action.

Other related work Finally, our work is broadly related to the literature on learning in repeated Stackelberg games [7, 55], online Bayesian persuasion [16, 17, 13], and online learning in principalagent problems [20, 33, 56]. In the repeated Stackelberg game setting, the principal (leader) commits to a mixed strategy over a finite set of actions, and the agent (follower) best-responds by playing an action from a finite set of best-responses. Unlike in our setting, both the principal's and agent's payoffs can be represented by matrices. In contrast, in our setting the principal commits to a pure strategy from a continuous set of actions, and the agent best-responds by playing an action from a continuous set. In online Bayesian persuasion, the principal (sender) commits to a 'signaling policy' (a random mapping from 'states of the world' to receiver actions) and the agent (receiver) performs a posterior update on the state based on the principal's signal, then takes an action from a (usually finite) set. In both this setting and ours, the principal's action is a policy. However in our setting the policy is a linear decision rule, whereas in the Bayesian persuasion setting, the policy is a set of conditional probabilities which form an 'incentive compatible' signaling policy. This difference in the policy space for the principal typically leads to different algorithmic ideas being used in the two settings. Strategic learning problems are, broadly speaking, instances of principal-agent problems. In contract design, the principal commits to a contract (a mapping from 'outcomes' to agent payoffs). The agent then takes an action, which affects the outcome. In particular, they take the action which maximizes their expected payoff, subject to some cost of taking the action. The goal of the principal is to design a contract such that their own expected payoff is maximized. While the settings are indeed similar, there are several key differences. First, in online contract design the principal always observes the outcome, whereas in our setting the principal only observes the reward if a positive decision is made. Second, the form of the agent's best response is different, which leads to different agent behavior and, as a result, different online algorithms for the principal.

2 Setting and background

We consider a game between a principal and a sequence of T agents . Each agent is associated with a context x t ∈ X ⊆ R d , which characterizes their attributes (e.g., a loan applicant's credit history/report). At time t , the principal commits to a policy π t : X → { 1 , 0 } , which maps from contexts to binary decisions (e.g., whether to accept/reject the loan application). We use a t = 1 to denote the the principal's positive decision at round t (e.g., agent t 's loan application is approved), and a t = 0 to denote a negative decision (e.g., the loan application is rejected). Given π t , agent t best-responds by strategically modifying their context within their effort budget as follows:

Definition 2.1 (Agent best response; lazy tiebreaking) . Agent t best-responds to policy π t by modifying their context according to the following optimization program.

Furthermore, we assume that if an agent is indifferent between two (modified) contexts, they choose the one which requires the least amount of effort to obtain (i.e., agents are lazy when tiebreaking).

In other words, every agent wants to receive a positive decision, but has only a limited ability to modify their (initial) context (represented by ℓ 2 budget δ ). 2 Such an effort budget may be induced by time or monetary constraints and is a ubiquitous model of agent behavior in the strategic learning literature (e.g., [39, 28, 19, 10]). We focus on linear thresholding policies where the principal assigns action π ( x ′ ) = 1 , if and only if ⟨ β , x ′ ⟩ ≥ γ for some β ∈ R d , γ ∈ R . We refer to ⟨ β , x ′ t ⟩ = γ as the decision boundary . For linear thresholding policies, the agent's best-response according to Definition 2.1 is to modify their context in the direction of β / ∥ β ∥ 2 until the decision-boundary is reached (if it can indeed be reached). While we present our results for lazy tiebreaking for ease of exposition, all of our results can be readily extended to the setting in which agents best-respond with a 'trembling hand', i.e. trembling hand tiebreaking . Under this setting, we allow agents who strategically modify their contexts to 'overshoot' the decision boundary by some bounded amount, which can be either stochastic or adversarially-chosen. See Appendix D for more details.

The principal observes x ′ t and plays action a t = π t ( x ′ t ) according to policy π t . If a t = 0 , the principal receives some known, constant reward r t (0) := r 0 ∈ R . On the other hand, if the principal assigns action a t = 1 , we assume that the reward the principal receives is linear in the agent's unmodified context, i.e.,

for some unknown θ (1) ∈ R d , where ϵ t is i.i.d. zero-mean sub-Gaussian random noise with (known) variance σ 2 . Note that r t (1) is observed only when the principal assigns action a t = 1 , and not when a t = 0 . Following Helmbold et al. [32], we refer to such feedback as apple tasting (or one-sided ) feedback. Mapping to our lending example, the reward a bank receives for rejecting a particular loan applicant is the same across all applicants, whereas their reward for a positive decision could be anywhere between a large, negative reward (e.g., if a loan is never repaid) to a large, positive reward (e.g., if the loan is repaid on time, with interest).

The most natural measure of performance in our setting is that of Stackelberg regret , which compares the principal's reward over T rounds with that of the optimal policy given that agents strategize .

Definition 2.2 (Stackelberg regret) . The Stackelberg regret of a sequence of policies { π t } t ∈ [ T ] on agents { x t } t ∈ [ T ] is

where ˜ x t is the best-response from agent t to policy ˜ π ∗ and ˜ π ∗ is the optimal-in-hindsight policy, given that agents best-respond according to Definition 2.1.

A stronger measure of performance is that of strategic regret , which compares the principal's reward over T rounds with that of the optimal policy had agents reported their contexts truthfully .

Definition 2.3 (Strategic regret) . The strategic regret of a sequence of policies { π t } t ∈ [ T ] on agents { x t } t ∈ [ T ] is

where π ∗ ( x t ) = 1 if ⟨ θ (1) , x t ⟩ ≥ r 0 and π ∗ ( x t ) = 0 otherwise.

Proposition 2.4. Strategic regret is a stronger performance notion compared to Stackelberg regret, i.e., Reg Stackel ( T ) ≤ Reg strat ( T ) .

Proof. The proof follows from the corresponding regret definitions and the fact that the principal's reward is determined by the original (unmodified) agent contexts.

where the last line follows from the fact that the principal's reward from the optimal policy when the agent strategizes is at most their optimal reward when agents do not strategize.

Because of Proposition 2.4, we focus on strategic regret, and use the shorthand Reg strat ( T ) = Reg( T ) for the remainder of the paper. Strategic regret is a strong notion of optimality, as we are comparing the principal's performance with that of the optimal policy for an easier setting, in which agents do not strategize. Moreover, the apple tasting feedback introduces additional challenges which require new algorithmic ideas to solve, since the principal needs to assign actions to both (1) learn about θ (1) (which can only be done when action 1 is assigned) and (2) maximize rewards in order to achieve sublinear strategic regret. See Figure 1 for a summary of the setting we consider.

We conclude this section by pointing out that our results also apply to the more challenging setting of bandit feedback , in which r t (1) is defined as in Equation (1), r t (0) := ⟨ θ (0) , x t ⟩ + ϵ t and only r t ( a t ) is observed at each time-step. We choose to highlight our results for apple tasting feedback since this is the type of feedback received by the principal in our motivating examples. Finally, we note that ˜ O ( · ) hides polylogarithmic factors, and that all proofs can be found in the Appendix.

3 Strategic classification with apple tasting feedback

In this section, we present our main results: provable guarantees for online classification of strategic agents under apple tasting feedback. Our results rely on the following assumption.

Assumption 3.1 (Bounded density ratio) . Let f U d : X → R ≥ 0 denote the density function of the uniform distribution over the d -dimensional unit sphere. We assume that agent contexts { x t } t ∈ [ T ] are drawn i.i.d. from a distribution over the d -dimensional unit sphere with density function f : X → R ≥ 0 such that f ( x ) f U d ( x ) ≥ c 0 > 0 , ∀ x ∈ X . 3

Assumption 3.1 is a condition on the initial agent contexts { x t } t ∈ [ T ] , before they are strategically modified. Indeed, one would expect the distribution over modified agent contexts to be highly discontinuous in a way that depends on the sequence of policies deployed by the principal. Furthermore, none of our algorithms need to know the value of c 0 . As we will see in the sequel, this assumption allows us to handle apple tasting feedback by relying on the inherent diversity in the agent population

for exploration ; a growing area of interest in the online learning literature (see references in Section 1.2). Moreover, such assumptions often hold in practice. For example, in the related problem of (non-strategic) contextual bandits (we will later show how our results extend to the strategic version of this problem), Bietti et al. [14] find that a greedy algorithm with no explicit exploration achieved the second-best empirical performance across a large number of datasets when compared to many popular contextual bandit algorithms. In our settings of interest (e.g. lending, hiring), such an assumption is reasonable if there is sufficient diversity in the applicant pool. In Section 4 we show how to remove this assumption, albeit at the cost of worse regret rates and exponential computational complexity.

At a high level, our algorithm (formally stated in Algorithm 1) relies on three key ingredients to achieve sublinear strategic regret:

A running estimate of θ (1) is used to compute a linear policy, which separates agents who receive action 1 from those who receive action 0 . Before deploying, we shift the decision boundary by the effort budget δ to account for the agents strategizing.

We maintain an estimate of θ (1) (denoted by ̂ θ (1) ) and only updating it when a t = 1 and we can ensure that x ′ t = x t .

We assign actions 'greedily' (i.e. using no explicit exploration) w.r.t. the shifted linear policy.

Shifted linear policy If agents were not strategic, assigning action 1 if ⟨ ̂ θ (1) t , x t ⟩ ≥ r 0 and action 0 otherwise would be a reasonable strategy to deploy, given that ̂ θ (1) t is our 'best estimate' of θ (1) so far. Recall that the strategically modified context x ′ t is s.t., ∥ x ′ t -x t ∥ ≤ δ . Hence, in Algorithm 1, we shift the linear policy by δ ∥ ̂ θ (1) ∥ 2 to account for strategically modified contexts. Now, action 1 is only assigned if ⟨ ̂ θ (1) t , x t ⟩ ≥ δ ∥ ̂ θ (1) ∥ 2 + r 0 . This serves two purposes: (1) It makes it so that any agent with unmodified context x such that ⟨ ̂ θ (1) t , x ⟩ < r 0 cannot receive action 1 , no matter how they strategize. (2) It forces some agents with contexts in the band r 0 ≤ ⟨ ̂ θ (1) t , x ⟩ < δ ∥ ̂ θ (1) ∥ 2 + r 0 to strategize in order to receive action 1 . Estimating θ (1) After playing action 1 for the first d rounds, Algorithm 1 forms an initial estimate of θ (1) via ordinary least squares (OLS). Note that since the first d agents will receive action 1 regardless of their context, they have no incentive to modify and thus x ′ t = x t for t ≤ d . In future rounds, the algorithm's estimate of θ (1) is only updated whenever x ′ t lies strictly on the positive side of the linear decision boundary. We call these contexts clean , and can infer that x ′ t = x t due to the lazy tiebreaking assumption in Definition 2.1 (i.e. agents will not strategize more than is necessary to receive the positive classification).

Condition 3.2 (Sufficient condition for x ′ = x ) . Given a shifted linear policy parameterized by β (1) ∈ R d , we say that a context x ′ is clean if ⟨ β (1) , x ′ ⟩ > δ ∥ β (1) ∥ 2 + r 0 .

Greedy action assignment By assigning actions greedily according to the current (shifted) linear policy, we are relying on the diversity in the agent population for implicit exploration (i.e., to collect more datapoints to update our estimate of θ (1) ). As we will show, this implicit exploration

is sufficient to achieve ˜ O ( √ T ) strategic regret under Assumption 3.1, albeit at the cost of an exponentially-large (in d ) constant which depends on the agents' ability to manipulate ( δ ).

We are now ready to present our main result: strategic regret guarantees for Algorithm 1 under apple tasting feedback.

Theorem 3.3 (Informal; detailed version in Theorem B.1) . With probability 1 -γ , Algorithm 1 achieves the following performance guarantee:

where c 0 is a lower bound on the density ratio as defined in Assumption 3.1, c 1 ( d, δ ) := P x ∼ U d ( x [1] ≥ δ ) ≥ Θ ( (1 -δ ) d/ 2 d 2 ) for sufficiently large d and c 2 ( d, δ ) := E x ∼ U d [ x [2] 2 | x [1] ≥ δ ] ≥ ( 3 4 -1 2 δ -1 4 δ 2 ) 3 , where x [ i ] denotes the i -th coordinate of a vector x . 4

Proof sketch. Our analysis begins by using properties of the strategic agents and shifted linear decision boundary to upper-bound the per-round strategic regret for rounds t > d by a term proportional to ∥ ̂ θ (1) t -θ (1) ∥ 2 , i.e., our instantaneous estimation error for θ (1) . Next we show that

✶ where λ min ( M ) is the minimum eigenvalue of (symmetric) matrix M , and I (1) s = {⟨ ̂ θ (1) s , x s ⟩ ≥ δ ∥ ̂ θ (1) s ∥ 2 + r 0 } is the event that Algorithm 1 assigns action a s = 1 and can verify that x ′ s = x s . We upper-bound the numerator using a variant of Azuma's inequality for martingales with subgaussian tails. Next, we use properties of Hermitian matrices to show that λ min ( ∑ t s =1 x s x ⊤ s ✶ {I (1) s } ) is lower-bounded by two terms: one which may be bounded w.h.p. by using the extension of Azuma's inequality for matrices, and one of the form ∑ t s =1 λ min ( E s -1 [ x s x ⊤ s ✶ {I (1) s } ]) , where E s -1 denotes the expected value conditioned on the filtration up to time s . Note that up until this point, we have only used the fact that contexts are drawn i.i.d. from a bounded distribution.

Using Assumption 3.1 on the bounded density ratio, we can lower bound λ min ( E s -1 [ x s x ⊤ s ✶ {I (1) s } ]) by λ min ( E U d ,s -1 [ x s x ⊤ s ✶ {I (1) s } ]) , where the expectation is taken with respect to the uniform distribution over the d -dimensional ball . We then use properties of the uniform distribution to show that λ min ( E U d ,s -1 [ x s x ⊤ s ✶ {I (1) s } ]) ≥ O ( c 0 · c ( d, δ )) . Putting everything together, we get that ∥ ̂ θ (1) t -θ (1) ∥ 2 ≤ ( c 0 · c ( d, δ ) · √ t ) -1 with high probability. Via a union bound and the fact that ∑ t ∈ [ T ] 1 √ t ≤ 2 T , we get that Reg( T ) ≤ ˜ O ( 1 c 0 · c ( d,δ ) √ T ) . Finally, we use tools from highdimensional geometry to lower bound the volume of a spherical cap and we show that for sufficiently large d , c 1 ( d, δ ) ≥ Θ ( (1 -δ ) d/ 2 d 2 ) .

3.1 High-dimensional contexts

While we typically think of the number of agents T as growing and the context dimension d as constant in our applications of interest, there may be situations in which T is either unknown or small. Under such settings, the 1 / c ( d,δ ) dependence in the regret bound (where c ( d, δ ) = c 1 ( d, δ ) · c 2 ( d, δ ) ) may become problematic if δ is close to 1 . This begs the question: 'Why restrict the OLS estimator in Algorithm 1 to use only clean contexts (as defined in Condition 3.2)?' Perhaps unsurprisingly, we show in Appendix B that the estimate ̂ θ (1) given by OLS will be inconsistent if even a constant fraction of agents strategically modify their contexts.

Given the above, it seems reasonable to restrict ourselves to learning procedures which only use data from agents for which the principal can be sure that x ′ = x . Under such a restriction, it is natural to ask whether there exists some sequence of linear polices which maximizes the number of points of the form ( x ′ t , r t (1)) for which the principal can be sure that x ′ t = x t . Again, the answer is no:

Proposition 3.4. For any sequence of linear policies { β t } t , the expected number of clean points is:

when (initial) contexts are drawn uniformly from the d -dimensional unit sphere.

The proof follows from the rotational invariance of the uniform distribution over the unit sphere. Intuitively, Proposition 3.4 implies that any algorithm which wishes to learn θ (1) using clean samples will only have c 1 ( d, δ ) · T datapoints in expectation. Observe that this dependence on c 1 ( d, δ ) arises as a direct result of the agents' ability to strategize. We remark that a similar constant often appears in the regret analysis of BIC bandit algorithms (see Section 1.2). Much like our work, [43] find that their regret rates depend on a constant which may be arbitrarily large, depending on how hard it is to persuade agents to take the principal's desired action in their setting. The authors conjecture that this dependence is an inevitable 'price of incentive-compatibility'. While our results do not rule out better strategic regret rates in d for more complicated algorithms (e.g., those which deploy non-linear policies), it is often unclear how strategic agents would behave in such settings, both in theory (Definition 2.1 would require agents to solve a non-convex optimization with potentially no closed-form solution) and in practice, making the analysis of such nonlinear policies difficult in strategic settings.

We conclude this section by showing that polynomial dependence on d is possible, at the cost of ˜ O ( T 2 / 3 ) strategic regret. Specifically, we provide an algorithm (Algorithm 3) which obtains the following regret guarantee whenever T is small or unknown, which uses Algorithm 1 and a variant of the explore-then-commit algorithm (Algorithm 2) as subroutines:

Theorem 3.5 (Informal; details in Theorem B.13) . Algorithm 3 incurs expected strategic regret

where the expectation is taken with respect to the sequence of contexts { x t } t ∈ [ T ] and random noise { ϵ t } t ∈ [ T ] .

The algorithm proceeds by playing a 'strategy-aware' variant of explore-then-commit (Algorithm 2) with a doubling trick until the switching time τ ∗ = g ( d, δ ) is reached. Note that g ( d, δ ) is a function of both d and δ , not c 0 . If round τ ∗ is indeed reached, the algorithm switches over to Algorithm 1 for the remaining rounds.

Extension to bandit feedback Algorithm 1 can be extended to handle bandit feedback by explicitly keeping track of an estimate ̂ θ (0) of θ (0) via OLS, assigning action a t = 1 if and only if ⟨ ̂ θ (1) t -̂ θ (0) t , x ′ t ⟩ ≥ δ · ∥ ̂ θ (1) t -̂ θ (0) t ∥ 2 , and updating the OLS estimate of ̂ θ (0) whenever a t = 0 (since agents will not strategize to receive action 0 ). Algorithm 3 may be extended to bandit feedback by 'exploring' for twice as long in Algorithm 2, in addition to using the above modifications. In both cases, the strategic regret rates are within a constant factor of the rates obtained in Theorem 3.3 and Theorem 3.5.

4 Beyond stochastic contexts

In this section, we allow the sequence of initial agent contexts to be chosen by an (oblivious) adversary . This requires new algorithmic ideas, as the regression-based algorithms of Section 3 suffer linear strategic regret under this adversarial setting. Our algorithm (Algorithm 4) is based on the popular EXP3 algorithm [6]. At a high level, Algorithm 4 maintains a probability distribution over 'experts', i.e., a discretized grid E over carefully-selected policies. In particular, each grid point e ∈ E ⊆ R d represents an 'estimate' of θ (1) , and corresponds to a slope vector which parameterizes a (shifted) linear threshold policy, like the ones considered in Section 3. We use a t, e to refer to the action played by the principal at time t , had they used the linear threshold policy parameterized by expert e . At every time-step, (1) the adversary chooses an agent x t , (2) a slope vector e t ∈ E is selected according to the current distribution, (3) the principal commits to assigning action 1 if and only if ⟨ e t , x ′ t ⟩ ≥ δ ∥ e t ∥ 2 , (4) the agent strategically modifies their context x t → x ′ t , and (5) the principal assigns an action a t according to the policy and receives the associated reward r t ( a t ) (under apple tasting feedback).

̸

Algorithm EXP4, which maintains a distribution over experts and updates the loss of all experts based on the current action taken, is not directly applicable in our setting as the strategic behavior of the agents prevents us from inferring the loss of each expert at every time-step [ ? ]. This is because if x ′ t = x t under the thresholding policy associated with expert e ), it is generally not possible to 'back out' x t given x ′ t , which prevents us from predicting the counterfactual context the agent would have modified to had the principal been using expert e ′ instead. As a result, we use a modification of the standard importance-weighted loss estimator to update the loss of only the policy played by the algorithm (and therefore the distribution over policies). Our regret guarantees for Algorithm 4 are as follows:

Theorem 4.1 (Informal; detailed version in Theorem C.1) . Algorithm 4 incurs expected strategic regret E [Reg( T )] = ˜ O ( T ( d +1) / ( d +2) ) .

We remark that Algorithm 4 may be extended to handle settings in which agents are selected by an adaptive adversary by using EXP3.P [6] in place of EXP3.

Proof sketch. The analysis is broken down into two parts. In the first part, we bound the regret w.r.t. the best policy on the grid. In the second, we bound the error incurred for playing policies on the grid, rather than the continuous space of policies. We refer to this error as the Strategic Discretization Error ( SDE ( T ) ). The analysis of the regret on the grid mostly follows similar steps to the analysis of EXP3 / EXP4. The important difference is that we shift the reward obtained by a t, by a factor of 1 + λ , where λ is a (tunable) parameter of the algorithm. This shifting (which does not affect the regret, since all the losses are shifted by the same fixed amount) guarantees that the losses at each round are non-negative and bounded with high probability. Technically, this requires bounding the tails of the subgaussian of the noise parameters ϵ t .

̸

We now shift our attention to bounding SDE ( T ) . The standard analysis of the discretization error in the non-strategic setting does not go through for our setting, since an agent may strategize very differently with respect to two policies which are 'close together' in ℓ 2 distance, depending on the agent's initial context. Our analysis proceeds with a case-by-case basis. Consider the best expert e ∗ in the grid. If a t, e ∗ = π ∗ ( x t ) (i.e., the action of the best expert matches that of the optimal policy), there is no discretization error in round t . Otherwise, if a t, e ∗ = π ∗ ( x t ) , we show that the per-round SDE is upper-bounded by a term which looks like twice the discretization upper-bound for the non-strategic setting, plus an additional term. We show that this additional term must always be non-positive by considering two subcases ( a t, e ∗ = 1 , π ∗ ( x t ) = 0 and a t, e ∗ = 0 , π ∗ ( x t ) = 1 ) and using properties about how agents strategize against the deployed algorithmic policies.

ALGORITHM 4: EXP3 with strategy-aware experts (EXP3-SAE)

Computational complexity While both Algorithm 1 and Algorithm 3 have O ( d 3 ) per-iteration computational complexity, Algorithm 4 must maintain and update a probability distribution over a grid of size exponential in d at every time-step, making it hard to use in practice if d is large. We view the design of computationally efficient algorithms for adversarially-chosen contexts as an important direction for future research.

Extension to bandit feedback Algorithm 4 may be extended to the bandit feedback setting by maintaining a grid over estimates of θ (1) -θ (0) (instead of over θ (1) ). No further changes are required.

5 Conclusion

We study the problem of classification under incentives with apple tasting feedback. Such one-sided feedback is often what is observed in real-world strategic settings including lending and hiring. Our main result is a 'greedy' algorithm (Algorithm 1) which achieves ˜ O ( √ T ) strategic regret when the initial agent contexts are generated stochastically . The regret of Algorithm 1 depends on a constant c 1 ( d, δ ) which scales exponentially in the context dimension, which may be problematic in settings for which the number of agents is small or unknown. To address this, we provide an algorithm (Algorithm 3) which combines Algorithm 1 with a strategy-aware version of the explore-then-commit algorithm using a doubling trick to achieve ˜ O (min { √ dT c 1 ( d,δ ) , d · T 2 / 3 } ) expected strategic regret whenever T is unknown. Finally, we relax the assumption of stochastic contexts and allow for contexts to be generated adversarially. Algorithm 4 achieves ˜ O ( T d +1 d +2 ) expected strategic regret whenever agent contexts are generated adversarially by running EXP3 over a discretized grid of strategy-aware policies, but has exponential-ind per-round computational complexity. All of our results also apply to the more general setting of bandit feedback, under slight modifications to the algorithms. There are several directions for future work:

Unclean data The regret of Algorithm 1 depends on a constant which is exponentially large in d , due to the fact that it only learns using clean data (Condition 3.2). While learning using unclean data will generally produce an inconsistent estimator, it would be interesting to see if the principal could leverage this data to remove the dependence on this constant. Alternatively, lower bounds which show that using unclean data will not improve regret would also be interesting.

Efficient algorithms for adversarial contexts Our algorithm for adversarially-chosen agent contexts suffers exponential-ind per-round computational complexity, which makes it unsuitable for use in settings with high-dimensional contexts. Deriving polynomial-time algorithms with sublinear strategic regret for this setting is an exciting (but challenging) direction for future research.

More than two actions Finally, it would be interesting to extend our algorithms for strategic learning under bandit feedback to the setting in which the principal has three or more actions at their disposal. While prior work [29] implies an impossibility result for strategic regret minimization with three or more actions, other (relaxed) notions of optimality (e.g., sublinear Stackelberg regret; recall Definition 2.2) may still be possible.

References

Algorithmic hiring: Complex hiring by numbers? URL https://hiring.monster.com/ resources/recruiting-strategies/workforce-planning/hiring-algorithms/ .

Using white font on a cv to trick ats. URL https://www.thecvstore.net/blog/ cv-ats-white-font/#:~:text=Even%20if%20you%20get%20past,harm%20your% 20future%20employment%20prospects.

Saba Ahmadi, Hedyeh Beyhaghi, Avrim Blum, and Keziah Naggita. The strategic perceptron. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 6-25, 2021.

Saba Ahmadi, Avrim Blum, and Kunhe Yang. Fundamental bounds on online strategic classification. In Proceedings of the 24th ACM Conference on Economics and Computation, EC 2023, London, United Kingdom, July 9-12, 2023 , pages 22-58. ACM, 2023.

András Antos, Gábor Bartók, Dávid Pál, and Csaba Szepesvári. Toward a classification of finite partial-monitoring games. Theoretical Computer Science , 473:77-99, 2013.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D Procaccia. Commitment without regrets: Online learning in stackelberg security games. In Proceedings of the sixteenth ACM conference on economics and computation , pages 61-78, 2015.

Hamsa Bastani, Mohsen Bayati, and Khashayar Khosravi. Mostly exploration-free algorithms for contextual bandits. Management Science , 67(3):1329-1349, 2021. doi: 10.1287/mnsc.2020. 3605. URL https://doi.org/10.1287/mnsc.2020.3605 .

Yahav Bechavod, Katrina Ligett, Aaron Roth, Bo Waggoner, and Steven Z Wu. Equal opportunity in online classification with partial feedback. Advances in Neural Information Processing Systems , 32, 2019.

Yahav Bechavod, Katrina Ligett, Steven Wu, and Juba Ziani. Gaming helps! learning from strategic interactions in natural dynamics. In International Conference on Artificial Intelligence and Statistics , pages 1234-1242. PMLR, 2021.

Yahav Bechavod, Chara Podimata, Steven Wu, and Juba Ziani. Information discrepancy in strategic learning. In International Conference on Machine Learning , pages 1691-1715. PMLR, 2022.

Jillian Berman. Do ai-powered lending algorithms silently discriminate? this initiative aims to find out, Nov 2021. URL https://www.marketwatch.com/story/ do-ai-powered-lending-algorithms-silently-discriminate-this-initiative\ -aims-to-find-out-11637246524 .

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Francesco Trovò, and Nicola Gatti. Optimal rates and efficient algorithms for online bayesian persuasion. In International Conference on Machine Learning , pages 2164-2183. PMLR, 2023.

Alberto Bietti, Alekh Agarwal, and John Langford. A contextual bandit bake-off. The Journal of Machine Learning Research , 22(1):5928-5976, 2021.

Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science . Cambridge University Press, 2020.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Online bayesian persuasion. Advances in Neural Information Processing Systems , 33:16188-16198, 2020.

Matteo Castiglioni, Alberto Marchesi, Andrea Celli, and Nicola Gatti. Multi-receiver online bayesian persuasion. In International Conference on Machine Learning , pages 1314-1323. PMLR, 2021.

Yatong Chen, Wei Tang, Chien-Ju Ho, and Yang Liu. Performative prediction with bandit feedback: Learning through reparameterization. arXiv preprint arXiv:2305.01094 , 2023.

Yiling Chen, Yang Liu, and Chara Podimata. Learning strategy-aware linear classifiers. Advances in Neural Information Processing Systems , 33:15265-15276, 2020.

Vincent Conitzer and Nikesh Garera. Learning algorithms for online principal-agent problems (and selling goods online). In Proceedings of the 23rd international conference on Machine learning , pages 209-216, 2006.

Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 55-70, 2018.

Itay Eilat, Ben Finkelshtein, Chaim Baskin, and Nir Rosenfeld. Strategic classification with graph neural networks. arXiv preprint arXiv:2205.15765 , 2022.

Christian Eilers. Resume keywords: List by industry [for use to pass the ats], Jan 2023. URL https://zety.com/blog/resume-keywords .

Danielle Ensign, Sorelle A Friedler, Scott Nevlle, Carlos Scheidegger, and Suresh Venkatasubramanian. Decision making with limited feedback: Error bounds for predictive policing and recidivism prediction. In Proceedings of Algorithmic Learning Theory, , volume 83, 2018.

Ganesh Ghalme, Vineet Nair, Itay Eilat, Inbal Talgam-Cohen, and Nir Rosenfeld. Strategic classification in the dark. In International Conference on Machine Learning , pages 3672-3681. PMLR, 2021.

Nika Haghtalab, Thodoris Lykouris, Sloan Nietert, and Alexander Wei. Learning in stackelberg games with non-myopic agents. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 917-918, 2022.

Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science , pages 111-122, 2016.

Keegan Harris, Hoda Heidari, and Steven Z Wu. Stateful strategic regression. Advances in Neural Information Processing Systems , 34:28728-28741, 2021.

Keegan Harris, Anish Agarwal, Chara Podimata, and Zhiwei Steven Wu. Strategyproof decisionmaking in panel data settings and beyond. arXiv preprint arXiv:2211.14236 , 2022.

Keegan Harris, Valerie Chen, Joon Kim, Ameet Talwalkar, Hoda Heidari, and Steven Z Wu. Bayesian persuasion for algorithmic recourse. Advances in Neural Information Processing Systems , 35:11131-11144, 2022.

Keegan Harris, Dung Daniel T Ngo, Logan Stapleton, Hoda Heidari, and Steven Wu. Strategic instrumental variable regression: Recovering causal relationships from strategic responses. In International Conference on Machine Learning , pages 8502-8522. PMLR, 2022.

David P Helmbold, Nicholas Littlestone, and Philip M Long. Apple tasting. Information and Computation , 161(2):85-139, 2000.

Chien-Ju Ho, Aleksandrs Slivkins, and Jennifer Wortman Vaughan. Adaptive contract design for crowdsourcing markets: Bandit algorithms for repeated principal-agent problems. In Proceedings of the fifteenth ACM conference on Economics and computation , pages 359-376, 2014.

Guy Horowitz and Nir Rosenfeld. Causal strategic classification: A tale of two shifts. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 13233-13253. PMLR, 2023.

Xinyan Hu, Dung Ngo, Aleksandrs Slivkins, and Steven Z Wu. Incentivizing combinatorial bandit exploration. Advances in Neural Information Processing Systems , 35:37173-37183, 2022.

Nicole Immorlica, Jieming Mao, Aleksandrs Slivkins, and Zhiwei Steven Wu. Bayesian exploration with heterogeneous agents. In The world wide web conference , pages 751-761, 2019.

Meena Jagadeesan, Celestine Mendler-Dünner, and Moritz Hardt. Alternative microfoundations for strategic classification. In International Conference on Machine Learning , pages 4687-4697. PMLR, 2021.

Sampath Kannan, Jamie Morgenstern, Aaron Roth, Bo Waggoner, and Zhiwei Steven Wu. A smoothed analysis of the greedy algorithm for the linear contextual bandit problem. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 2231-2241, Red Hook, NY, USA, 2018. Curran Associates Inc.

Jon Kleinberg and Manish Raghavan. How do classifiers induce agents to invest effort strategically? ACM Transactions on Economics and Computation (TEAC) , 8(4):1-23, 2020.

Tony Lancaster and Guido Imbens. Case-control studies with contaminated controls. Journal of Econometrics , 71(1-2):145-160, 1996.

Sagi Levanon and Nir Rosenfeld. Strategic classification made practical. In International Conference on Machine Learning , pages 6243-6253. PMLR, 2021.

Sagi Levanon and Nir Rosenfeld. Generalized strategic classification and the case of aligned incentives. In International Conference on Machine Learning , pages 12593-12618. PMLR, 2022.

Yishay Mansour, Aleksandrs Slivkins, and Vasilis Syrgkanis. Bayesian incentive-compatible bandit exploration. In Proceedings of the Sixteenth ACM Conference on Economics and Computation , pages 565-582, 2015.

Daniel Ngo, Logan Stapleton, Vasilis Syrgkanis, and Zhiwei Steven Wu. Incentivizing bandit exploration: Recommendations as instruments. In Proceedings of the 2021 International Conference on Machine Learning (ICML'21) , 2021.

Daniel Ngo, Keegan Harris, Anish Agarwal, Vasilis Syrgkanis, and Zhiwei Steven Wu. Incentiveaware synthetic control: Accurate counterfactual estimation via incentivized exploration. 2023. URL https://keeganharris.github.io/IE_SC.pdf .

Bev O'Shea. 9 ways to build and improve your credit fast, Nov 2022. URL https://www. nerdwallet.com/article/finance/raise-credit-score-fast .

Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning , pages 7599-7609. PMLR, 2020.

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, and Zhiwei Steven Wu. The externalities of exploration and how data diversity helps exploitation. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018 , volume 75 of Proceedings of Machine Learning Research , pages 1724-1738. PMLR, 2018. URL http://proceedings.mlr.press/v75/ raghavan18a.html .

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, and Zhiwei Steven Wu. Greedy algorithm almost dominates in smoothed contextual bandits. SIAM Journal on Computing , 52(2):487-524, 2023. doi: 10.1137/19M1247115. URL https://doi.org/10.1137/ 19M1247115 .

Mark Sellke and Aleksandrs Slivkins. The price of incentivizing exploration: A characterization via thompson sampling and sample complexity. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 795-796, 2021.

Ohad Shamir. A variant of azuma's inequality for martingales with subgaussian tails. arXiv preprint arXiv:1110.2392 , 2011.

Yonadav Shavit, Benjamin Edelman, and Brian Axelrod. Causal strategic linear regression. In International Conference on Machine Learning , pages 8676-8686. PMLR, 2020.

Vidyashankar Sivakumar, Zhiwei Steven Wu, and Arindam Banerjee. Structured linear contextual bandits: A sharp and geometric smoothed analysis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 9026-9035. PMLR, 2020. URL http://proceedings.mlr.press/v119/sivakumar20a.html .

Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12:389-434, 2012.

Geng Zhao, Banghua Zhu, Jiantao Jiao, and Michael Jordan. Online learning in stackelberg games with an omniscient follower. In International Conference on Machine Learning , pages 42304-42316. PMLR, 2023.

Banghua Zhu, Stephen Bates, Zhuoran Yang, Yixin Wang, Jiantao Jiao, and Michael I. Jordan. The sample complexity of online contract design. In Proceedings of the 24th ACM Conference on Economics and Computation, EC 2023, London, United Kingdom, July 9-12, 2023 , page 1188. ACM, 2023.

A Useful concentration inequalities

Theorem A.1 (Matrix Azuma, Tropp [54]) . Consider a self-adjoint matrix martingale { Y s : s = 1 , . . . , t } in dimension d , and let { X s } s ∈ [ t ] be the associated difference sequence satisfying E s -1 X s = 0 d × d and X 2 s ⪯ A 2 s for some fixed sequence { A s } s ∈ [ t ] of self-adjoint matrices. Then for all α > 0 ,

Theorem A.2 (A variant of Azuma's inequality for martingales with subgaussian tails, Shamir [51]) . Let Z 1 , Z 2 , . . . Z t be a martingale difference sequence with respect to a sequence W 1 , W 2 , . . . , W t , and suppose there are constants b > 1 , c > 0 such that for any s and any α > 0 , it holds that

Then for any γ > 0 , it holds with probability 1 -γ that

B Proofs for Section 3

B.1 Proof of Theorem 3.3

Theorem B.1. Let f U d : X → R ≥ 0 denote the density function of the uniform distribution over the d -dimensional unit sphere. If agent contexts are drawn from a distribution over the d -dimensional unit sphere with density function f : X → R ≥ 0 such that f ( x ) f U d ( x ) ≥ c 0 > 0 , ∀ x ∈ X , then Algorithm 1 achieves the following performance guarantee:

with probability 1 -γ , where 0 < c 1 ( δ, d ) := P x ∼ U d ( x [1] ≥ δ ) and 0 < c 2 ( δ, d ) := E x ∼ U d [ x [2] 2 | x [1] ≥ δ ] .

Proof. We start from the definition of strategic regret. Note that under apple tasting feedback, θ (0) = 0 .

where the first inequality follows from Lemma B.2, the second inequality follows from the CauchySchwarz, and the third inequality follows from the fact that the instantaneous regret at each time-step is at most 2 and we use the first d rounds to bootstrap our OLS . The result follows by Lemma B.3, a ∑ T √ 1 ≤ 2 √ T .

union bound, and the fact that d +1 t

Lemma B.2.

as

̸

Proof. If a t = a ∗ t , the condition is satisfied trivially. If a t = a ∗ t , then either (1) ⟨ ̂ θ (1) t -̂ θ (0) t , x ′ t ⟩ -δ ∥ ̂ θ (1) t -̂ θ (0) t ∥ 2 ≥ 0 and ⟨ θ (1) -θ (0) ⟩ < 0 or (2) ⟨ ̂ θ (1) t -̂ θ (0) t , x ′ t ⟩ -δ ∥ ̂ θ (1) t -̂ θ (0) t ∥ 2 < 0 and ⟨ θ (1) -θ (0) ⟩ ≥ 0 .

Case 1: ⟨ ̂ θ (1) t -̂ θ (0) t , x ′ t ⟩ -δ ∥ ̂ θ (1) t -̂ θ (0) t ∥ 2 ≥ 0 and ⟨ θ (1) -θ (0) ⟩ < 0 . ( a ∗ t = 0 , a t = 1 ) By Definition 2.1, we can rewrite

for some δ ′ ≤ δ . Since ( δ ′ -δ ) ∥ ̂ θ (1) t -̂ θ (0) t ∥ 2 ≤ 0 , ⟨ ̂ θ (1) t -̂ θ (0) t , x t ⟩ ≥ 0 must hold.

Lemma B.3. Let f U d : X → R ≥ 0 denote the density function of the uniform distribution over the d -dimensional unit sphere. If T ≥ d and agent contexts are drawn from a distribution over the d -dimensional unit sphere with density function f : X → R ≥ 0 such that f ( x ) f U d ( x ) ≥ c 0 ∈ R > 0 , ∀ x ∈ X , then the following guarantee holds under apple tasting feedback.

with probability 1 -γ t .

Proof. Let I (1) s = {⟨ ˆ θ (1) s , x s ⟩ ≥ δ ∥ ˆ θ (1) s ∥ 2 + r 0 } . Then, from the definition of θ (1) t +1 we have:

where for a matrix M , σ min is the smallest singular value σ min ( M ) := min ∥ x ∥ =1 ∥ Mx ∥ and λ min is the smallest eigenvalue. Note that the two are equal since the matrix ∑ t s =1 x s x ⊤ s ✶ {I (1) s } is PSD as the sum of PSD matrices (outer products induce PSD matrices). The final bound is obtained by applying Lemma B.4, Lemma B.5, and a union bound.

Lemma B.4. The following bound holds on the ℓ 2 -norm of ∑ t s =1 x s ϵ s ✶ {I (1) s } with probability 1 -γ t :

Proof. Let x [ i ] denote the i -th coordinate of a vector x . Observe that ∑ t s =1 ϵ s x s [ i ] ✶ {I (1) s } is a sum of martingale differences with Z s := ϵ s x s [ i ] {I (1) s } , X s := ∑ s s ′ =1 ϵ s ′ x s ′ [ i ] {I (1) s ′ } , and

By Theorem A.2, with probability 1 -γ i . The desired result follows via a union bound and algebraic manipulation.

Lemma B.5. The following bound holds on the minimum eigenvalue of ∑ t s =1 x s x ⊤ s ✶ {I (1) s } with probability 1 -γ t :

Proof.

where the inequalities follow from the fact that λ min ( A + B ) ≥ λ min ( A ) + λ min ( B ) for two Hermitian matrices A , B . Note that the outer products form Hermitian matrices. Let Y t := ∑ t s =1 x s x ⊤ s ✶ {I (1) s } -E s -1 [ x s x ⊤ s ✶ {I (1) s } ] . Note that by the tower rule, E Y t = E Y 0 = 0 . Let -X s := E s -1 [ x s x ⊤ s ✶ {I (1) s } ] , then E s -1 [ -X s ] = 0 , and ( -X s ) 2 ⪯ 4 I d . By Theorem A.1, P ( λ max ( -Y t ) ≥ α ) ≤ d · exp( -α 2 / 32 t ) .

Since -λ max ( -Y t ) = λ min ( Y t ) ,

Therefore, λ min ( Y t ) ≥ 4 √ 2 t log( d/γ t ) with probability 1 -γ t . We now turn our attention to lower bounding λ min ( E s -1 [ x s x ⊤ s ✶ {I (1) s } ]) .

Throughout the remainder of the proof, we surpress the dependence on U d and note that unless stated otherwise, all expectations are taken with respect to U d . Let B s ∈ R d × d be the orthonormal matrix such that the first column is ̂ β s / ∥ ̂ β s ∥ 2 . Note that B s e 1 = ̂ β s / ∥ ̂ β s ∥ 2 and B s x ∼ U d .

̸

Observe that for j = 1 , i = j , E [ x s [ j ] x s [ i ] | x s [1] ≥ δ ] = 0 . Therefore,

̸

and

Lemma B.6. For sufficiently large values of d ,

Proof. Lemma B.6 is obtained via a similar argument to Theorem 2.7 in Blum et al. [15]. As in Blum et al. [15], we are interested in the volume of a hyperspherical cap. However, we are interested in a lower-bound, not an upper-bound (as is the case in [15]). Let A denote the portion of the d -dimensional hypersphere with x [1] ≥ √ c d -1 and let H denote the upper hemisphere.

In order to lower-bound c 1 ( δ, d ) , it suffices to lower bound vol ( A ) and upper-bound vol ( H ) . In what follows, let V ( d ) denote the volume of the d -dimensional hypersphere with radius 1 .

Lower-bounding vol ( A ) : As in [15], to calculate the volume of A, we integrate an incremental volume that is a disk of width d x [1] and whose face is a ball of dimension d -1 and radius √ 1 -x [1] 2 . The surface area of the disk is (1 -x [1] 2 ) d -1 2 V ( d -1) and the volume above the slice x [1] ≥ δ is

To get a lower bound on the integral, we use the fact that 1 -x 2 ≥ 1 -x for x ∈ [0 , 1] . The integral now takes the form

Upper-bounding vol ( H ) : We can obtain an exact expression for vol ( H ) in terms of V ( d -1) using the recursive relationship between V ( d ) and V ( d -1) :

Plugging in our bounds for vol ( A ) , vol ( H ) and simplifying, we see that

where the equality follows from Stirling's approximation.

Lemma B.7. The following bound holds on c 2 ( δ, d ) :

Proof. We begin by computing E [ x [2] 2 | x [1] = δ ′ ] , for δ ′ ∈ (0 , 1) . If x [1] = δ ′ , then x [2] 2 + . . . + x [ d ] 2 ≤ 1 -( δ ′ ) 2 . Using this fact, we see that

Since E [ x [2] 2 | x [1] ≥ δ ] ≥ E [ x [2] 2 | x [1] = δ + 1 -δ 2 ] ,

B.2 Proof of Proposition 3.4

Proposition B.8. For any sequence of linear threshold policies β 1 , . . . , β T ,

Proof. Let B t ∈ R d × d be the orthonormal matrix such that the first column is β t / ∥ β t ∥ 2 . Note that B t x ∼ U d if x ∼ U d and B t e 1 = β t / ∥ β t ∥ 2 .

B.3 Explore-Then-Commit Analysis

Theorem B.9. Let f U d : X → R ≥ 0 denote the density function of the uniform distribution over the d -dimensional unit sphere. If agent contexts are drawn from a distribution over the d -dimensional unit sphere with density function f : X → R ≥ 0 such that f ( x ) f U ( x ) ≥ c 0 > 0 , ∀ x ∈ X , then Algorithm 2 achieves the following performance guarantee

with probability 1 -γ if T 0 := 4 · 63 1 / 3 σ 2 / 3 dT 2 / 3 log 1 / 3 (4 d/γ ) .

Proof.

with probability 1 -γ , where the last inequality follows from Lemma B.10 and a union bound. The result follows from picking T 0 = 4 · 63 1 / 3 dσ 2 / 3 T 2 / 3 log 1 / 3 (4 d/γ ) .

Lemma B.10. Let f U d : X → R ≥ 0 denote the density function of the uniform distribution over the d -dimensional unit sphere. If T ≥ 2 d and agent contexts are drawn from a distribution over the

d -dimensional unit sphere with density function f : X → R ≥ 0 such that f ( x ) f U ( x ) ≥ c 0 > 0 , ∀ x ∈ X , then the following guarantee holds for a ∈ { 0 , 1 } .

with probability 1 -γ t .

Proof. Observe that

where k = 0 if a = 0 and k = T 0 if a = 1 . Therefore,

The desired result is obtained by applying Lemma B.11, Lemma B.12, and a union bound.

Lemma B.11. The following bound holds on the ℓ 2 -norm of ∑ T 0 / 2 s =1 x s + k ϵ s + k with probability 1 -γ :

Proof. Observe that ∑ T 0 / 2 s =1 ϵ k + s x k + s [ i ] is a sum of martingale differences with Z k + s := ϵ k + s x k + s [ i ] , X k + s := ∑ s s ′ =1 ϵ k + s ′ x k + s ′ [ i ] , and max { P ( Z k + s > α | X k +1 , . . . , X k + s -1 ) , P ( Z k + s < -α | X k +1 , . . . , X k + s -1 ) } ≤ · exp( -α 2 / 2 σ 2 ) . By Theorem A.2,

with probability 1 -γ i . The desired result follows via a union bound and algebraic manipulation.

Lemma B.12. The following bound holds on the minimum eigenvalue of ∑ T 0 / 2 s =1 x s + k x ⊤ s + k with probability 1 -γ :

Proof.

where the inequalities follow from the fact that λ min ( A + B ) ≥ λ min ( A ) + λ min ( B ) for two Hermitian matrices A , B . Let Y T 0 / 2 := ∑ T 0 / 2 s =1 x s + k x ⊤ s + k -E [ x s + k x ⊤ s + k ] . Note that E Y T 0 / 2 = E Y 0 = 0 , -X s + k := E [ x s + k x ⊤ s + k ] , E [ -X s + k ] = 0 , and ( -X s + k ) 2 ⪯ 4 I d . By Theorem A.1,

Since -λ max ( -Y T 0 / 2 ) = λ min ( Y T 0 / 2 ) ,

Therefore, λ min ( Y T 0 / 2 ) ≥ 4 √ T 0 log( d/γ ) with probability 1 -γ . We now turn our attention to lower bounding λ min ( E [ x s + k x ⊤ s + k ]) .

B.4 Proof of Theorem 3.5

Theorem B.13. Let Reg OLS ( T ) be the strategic regret of Algorithm 1 and Reg ETC ( T ) be the strategic regret of Algorithm 2. The expected strategic regret of Algorithm 3 is

Proof. Case 1: T < τ ∗ From Theorem B.9, we know that

with probability 1 -1 /τ 2 i . Therefore,

Observe that ∑ i -1 j =1 E [Reg ETC ( τ j )] ≤ E [Reg ETC ( τ i )] . Suppose τ i -1 ≤ T ≤ τ i for some i . Under such a scenario,

Case 2: T ≥ τ ∗ Let t ∗ denote the actual switching time of Algorithm 3.

where the first line follows from case 1, the second line follows from the fact that t ∗ ≤ τ ∗ (and so E [Reg ETC ( t ∗ )] ≤ E [Reg OLS ( t ∗ )] ), the third line follows from the fact that t ∗ ≤ τ ∗ , and the fourth line follows from the fact that T ≥ τ ∗ .

B.5 Inconsistency of OLS when using all data

Theorem B.14. lim t →∞ ˆ θ (1) t +1 = θ (1) ∗ if and only if lim t →∞ ∑ t s =1 x ′ s x ′ ⊤ s ✶ { a s = 1 } = lim t →∞ ∑ t s =1 x ′ s x ⊤ s ✶ { a s = 1 } .

Proof.

C Proofs for Section 4

C.1 Proof of Theorem 4.1

Theorem C.1. Algorithm 4 with η = √ log( |E| ) Tλ 2 |E| , γ = 2 ηλ |E| , and ε = ( dσ log T T ) 1 / ( d +2) incurs expected strategic regret:

Proof. Let a t,e correspond to the action chosen by a grid point e ∈ E . We simplify notation to a t = a t,e t to be the action chosen by the sampled grid point e t at round t . For the purposes of the analysis, we also define ℓ t ( e ) = 1 + λ -r t ( a t,e t ) .

We first analyze the difference between the loss of the algorithm and the best-fixed point on the grid e ∗ , i.e.,

where the equivalence between working with ℓ t ( · ) as opposed to r t ( · ) holds because ℓ t ( · ) are just a shift from r t ( · ) that is the same across all rounds and experts. For the regret of the algorithm, we show that:

We define the 'good' event as the event that the reward is in [0 , 1] for every round t : C = { r t ∈ [0 , 1] , ∀ t ∈ [ T ] } . Note that this depends on the noise of the round ε t . We will call the complement of the 'good' event, the 'bad' event ¬C . The regret of the algorithm depends on both C and ¬C as follows:

where the inequality is due to the fact that Pr[ C ] ≤ 1 and that in the worst case, the algorithm must pick up a loss of 1 at each round.

We now upper bound the probability with which the bad event happens.

Plugging Pr[ ¬C ] to Equation (5) we get:

So for the remainder of the proof we will condition on the clean event C and compute E [Reg ε ( T ) |C ] . Conditioning on C means that 1 + λ -r t ( a ) ∈ [0 , λ ] , where λ = σ √ log T .

We first compute the first and the second moments of estimator ̂ ℓ t ( · ) . For the first moment:

For the second moment:

where for the first inequality, we have used the fact that ℓ t ( e ) ≤ λ (since we conditioned on C ) and the last one is due to the fact that q t ( e ) ≥ γ/ |E| .

We define the weight assigned to grid point e ∈ E at round t as: w t ( e ) = w t -1 ( e ) · exp( -η ̂ ℓ t ( e )) and w 0 ( e ) = 1 , ∀ e ∈ E . Let W t = ∑ e ∈E w t ( e ) be the potential function. Then,

Using e ∗ to denote the best-fixed policy in hindsight, we have:

We next analyze how much the potential changes per-round:

where the second inequality is due to the fact that log x ≤ x -1 for x ≥ 0 . In order for this inequality to hold we need to verify that:

or equivalently, that:

We do so after we explain how to tune η and γ .

We return to Equation (11); summing up for all rounds t ∈ [ T ] in Equation (11) we get:

Using Equation (9) and Equation (10) we have that: log( W T /W 0 ) ≥ -η ∑ t ∈ [ T ] ̂ ℓ t ( e ∗ ) -log |E| . Combining this with the upper bound on log( W T /W 0 ) from Equation (13) and multiplying both sides by (1 -γ ) /η we get:

We can slightly relax the right hand side using the fact that γ < 1 and get:

Taking expectations (wrt the draw of the algorithm) on both sides of the above expression and using our derivations for the first and second moment (Equation (7) and Equation (8) respectively) we get:

Using the fact that ℓ t ( · ) ∈ [0 , λ ] the above becomes:

Tuning η = √ log( |E| ) Tλ 2 |E| and γ = 2 ηλ |E|

Before we proceed to bounding the discretization error that we incur by playing policies only on the grid, we verify that Equation (12) holds for the chosen η and γ parameters. Note that when ̂ ℓ t ( e ) = 0 , then Equation (12) holds. So we focus on the case where ̂ ℓ t ( e ) = ℓ t ( e ) /q t ( e ) .

where the first inequality is due to the fact that q t ( e ) ≥ γ/ |E| , ∀ e ∈ E , the second is because ℓ t ( e ) ≤ λ , the first equality is because ∑ e ∈E p t ( e ) = 1 , and the last equality is because of the values that we chose for parameters η and γ .

The final step in proving the theorem is to bound the strategic discretization error that we incur because our algorithm only chooses policies on the grid, while θ (1) , θ (0) (and hence, the actual optimal policy) may not correspond to any grid point. Let a ∗ t correspond to the action chosen by the optimal policy.

We separate the T rounds into 3 groups: in group G 1 , we have rounds t ∈ [ T ] such that a ∗ t = a t,e ∗ . In group G 2 , we have rounds t ∈ [ T ] such that a ∗ t = 0 but a t,e ∗ = 1 . In group G 3 , we have rounds t ∈ [ T ] , such that a ∗ t = 1 but a t,e ∗ = 0 . With these groups in mind, one can rewrite the above equation as:

For all the rounds in G 1 , the strategic discretization error is equal to 0 . Hence the strategic discretization error becomes:

We first analyze SDE ( G 2 ) :

Let us denote by ̂ θ (1) and ̂ θ (0) the points such that e ∗ = ̂ θ (1) -̂ θ (0) . Adding and subtracting ⟨ e ∗ , x t ⟩ in the above, we get:

̸

Finally, we show that Q t ≤ 0 . For the rounds where a t,e ∗ = 1 but a ∗ t = 0 , it can be the case that x t = x ′ t (as the agents only strategize in order to get assigned action 1 . But since a t,e ∗ = 1 , then from the algorithm:

Adding and subtracting x ′ t from quantity Q t , we have:

As a result:

Moving on to the analysis of SDE ( G 3 ) :

Again, we use ̂ θ (1) and ̂ θ (0) the points that e ∗ = ̂ θ (1) -̂ θ (0) . Adding and subtracting ⟨ e ∗ , x t ⟩ and following the same derivations as in SDE ( G 3 ) , we have that:

Since a t,e ∗ = 0 , then it must have been the case that x ′ t = x t ; this is because the agent would not spend effort to strategize if they would still be assigned action 0 . For this reason, it must be that Q t ≤ 0 .

Combining the upper bounds for SDE ( G 2 ) and SDE ( G 3 ) in Equation (15), we have that SDE ( T ) ≤ 4 εT .

Putting everything together, we have that the regret is comprised by the regret incurred on the discretized grid and the strategic discretization error, i.e.,

Tuning ε = ( dσ log T T ) 1 / ( d +2) we get that the regret is:

D Extension to trembling hand best-response

Observe that when lazy tiebreaking (Definition 2.1), if agent t modifies their context they modify it by an amount δ L such that

We define γ -trembling hand tiebreaking as δ TH,t = δ L,t + α t , where α t ∈ [0 , min { δ -δ L,t , γ } ] may be chosen arbitrarily. Our results in Section 3 may be extended to trembling hand tiebreaking by considering the following redefinition of a clean point:

Condition D.1 (Sufficient condition for x ′ = x ) . Given a shifted linear policy parameterized by β (1) ∈ R d , we say that a context x ′ is clean if ⟨ β (1) , x ′ ⟩ > ( δ + γ ) ∥ β (1) ∥ 2 + r 0 .

No further changes are required. This will result in a slightly worse constant in Theorem 3.3 (i.e. all instances of δ will be replaced by δ + γ ). Our algorithms and results in Section 4 do not change. Our definition of trembling hand best-response is similar in spirit to the ϵ -best-response in Haghtalab et al. [26]. Specifically, Haghtalab et al. [26] study a Stackelberg game setting in which the follower best-responds ϵ -optimally. In our trembling hand setting, the strategic agent can also be thought of as ϵ -best responding (using the language of [26]), although it is important to note that an ϵ -best response for the agent in our setting will cause the agent to only strategize more than necessary.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an online learning problem with incentives. In particular, in the model studied in the paper there is a principal who has to take decisions on a sequence of $T$ (different) agents. Each agent has a context and they can strategically disclose it (truthfully or untruthfully) to the principal in order to induce them to take favorable decisions. The key feature of the model studied in the paper is that the feedback is one-sided, meaning that the principal only observes a feedback when they take a positive decision on the agent. The paper first proposes an algorithm that achieves $\\tilde O(\\sqrt{T})$ strategic regret (a stronger notion than Stackelberg regret), when the the contexts are selected stochastically. Then, the paper shows how to deal with adversarially-chosen contexts, by providing an algorithm with $\\tilde O(T^{(d+1)/(d+2)})$ strategic regret.\n\nSTRENGTHS:\nORIGINALITY\n\nThe paper studies, for the first time to the best of my knowledge, an online learning problem in which both incentives and one-sided feedback are involved. This is a relevant combination present in may application domains.\n\nQUALITY\n\nThe paper does a good job in addressing several aspects of the online learning problem under study. In particular, the paper studies both the case in which contexts are selected stochastically and the one in which they are chosen adversarially. \n\nCLARITY\n\nThe paper is well written and easy to follow.\n\nSIGNIFICANCE\n\nI believe that the problem studied in the paper is of interest several people spanning different communities, from online learning to algorithmic game theory.\n\nWEAKNESSES:\nORIGINALITY\n\nThe techniques used in the paper are not groundbreaking, but they are rather adaptations of well-known and widely-used techniques in online learning.\n\nQUALITY\n\nWhile the literature revised in the paper is quite extensive as far as work on online learning with incentives are concerned, I think the paper is missing some very related works in the literature at the interface between algorithmic game theory and online learning. In particular, the following lines of research seem strongly related to the paper:\n- Repeated Stackelberg games: Balcan, Maria-Florina, et al. \"Commitment without regrets: Online learning in stackelberg security games.\" Proceedings of the sixteenth ACM conference on economics and computation. 2015. (see also some subsequent works extending the paper)\n- Online Bayesian persuasion: Castiglioni, Matteo, et al. \"Online bayesian persuasion.\" Advances in Neural Information Processing Systems 33 (2020): 16188-16198. + Castiglioni, Matteo, et al. \"Multi-receiver online bayesian persuasion.\" International Conference on Machine Learning. PMLR, 2021. (these works extend those on repeated Stackelberg games to the Bayesian persuasion framework)\n- Online learning in principal-agent problems: Zhu, Banghua, et al. \"The Sample Complexity of Online Contract Design.\" arXiv preprint arXiv:2211.05732 (2022). (a recent preprint that indeed seems very related to the present paper)\nI would have expected a discussion on these works in the paper. Of course, the papers that I cited above are only some representative of such a line of research, and I am happy to provide more references if the authors need them.\n\nCLARITY\n\nNo weaknesses to report.\n\nSIGNIFICANCE\n\nNo weaknesses to report.\n\nQUESTIONS:\n1) I believe that the model studied in the paper is closely connected with one introduced in a very recent preprint (Bernasconi, Martino, et al. \"Optimal Rates and Efficient Algorithms for Online Bayesian Persuasion.\" arXiv preprint arXiv:2303.01296 (2023).). In that paper, the authors study a model with type reporting that looks very similar to the one in the present paper. Am I right?\n2) The regret bound in Theorem 4.1 is very similar to a regret bound in (Zhu, Banghua, et al. \"The Sample Complexity of Online Contract Design.\" arXiv preprint arXiv:2211.05732 (2022).). Can you discuss the differences between the approach in that paper and your approach?\n3) Algorithm 3 requires exponential running time in the worst case. Are there any computational hardness results that can be applied to your problem?\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies strategic apple tasting settings. This setting involves decision making that assigns decisions to agents who have incentives to strategically modify their input (context), and the decision maker only receives apple tasting (one-sided) feedback, where it only receives feedback for positively assigned decisions. The authors formulate this setting as an online learning problem with apple-tasting feedback, where a principal makes decisions about a sequence of T agents, with the goal of achieving sublinear strategic regret. Under this problem formulation, the authors proposed a learning algorithm (Algorithm 1) based on greedy, shifted linear policy, and parameter updates based on clean contexts. The author shows that the algorithm achieves O(\\sqrt{T}) strategic regret when the sequence when the agent contexts are generated stochastically. Then, the authors extendes to the situation where T is small or unknown and proposed another algorithm (Algorithm 2) under this scenario. Finally, the authors relax the assumption and provide an algorithm (Algorithm 3) under adversarially generated agents. These results apply to the more general setting of bandit feedback, under slight modifications to the algorithms.\n\nSTRENGTHS:\n- I like that the paper is very well written and easy to follow\n- The studied one-sided feedback settings with strategic agents are interesting and often occurs in real-world situations.\n- The problem is clearly formulated. For all proposed algorithms, the authors provided performance guarantees which are clearly explained and supported by proofs.\n\nWEAKNESSES:\nIt is slightly hard for me to make a connection between assumption of the agent's strategic modification to their context (with effort budget \\delta mentioned in definition 2.1), and the motivated examples such as hiring and lending.\n\nQUESTIONS:\n- As in the above section, can the authors shed some light on the strategic modification effort budget?\n- Line 212: The authors mentioned \"This is the type of strategizing we want to incentivize.\" I didn't understand why the decision maker would like to incentivize this behavior?\n- Algorithm 1 line 4, \"... and data S_t\", do you mean \"data D_t\"?\n- Can the authors move algorithm 4 (which is part of algorithm 2) into the main text?\n\nLIMITATIONS:\nI see that the authors listed a few related future directions in the conclusion. No negative societal impact observed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers contextual bandit problem in which the users can strategically modify their context for its own sake.\nIt provides sublinear regret algorithms by exploiting the budgeted strategic structure of the agents, against stochastically chosen agents.\nIt further obtains sublinear regret algorithms against adversarially chosen agents by considering a variant of EXP3.\n\nSTRENGTHS:\nThe paper studies interesting problem setting and makes reasonable contribution.\n\nWEAKNESSES:\nAlthough the analysis seems technically thorough, intuition behind the solution is pretty straightforward: incentivize truthful action for certain amount of times, estimate $\\theta$, strictly separate strategically modified context by conservatively setting decision boundary based on the effort budget.\nAlso, the algorithm seems to heavily depend on the information on the agents' budget.\nDetailed comments are in Questions.\n\nQUESTIONS:\n- What happens if the principal can also optimize over the choice of decision policy, i.e., not just using thresholded rule?\n- In L254, even though a small fraction of agents may strategically modify their contexts, it seems fairly intuitive that the estimates would be inconsistent, so wonder why the authors think this to be a surprising fact - is it meant to be \"far\" from being consistent, not just inconsistent?\n- L82 consists a lot of citations. It looks bad to me, at least parsing them/giving more contexts to each reference/removing part of them would be helpful. It's a bit distracting in its current form and gave me no information.\n- It seems that the algorithm essentially needs to know the budget $\\delta$ in advance. What happens if the algorithm has misspecified budget? Is the algorithm fairly robust?\n- In L170, bandit feedback setting is said to be more challenging - why is it more challenging than the one-sided feedback? Also, the authors argue that all their algorithms can be made to be applicable to the bandit setting - yes it's good to know that, but I wonder if there's more general results, e.g., any reduction from one setting to the other, or if the authors have thought about any black-box reduction from one algorithm to another.\n- I also wonder why the standard adversarial contextual bandit algorithm does not work, instead of Algorithm 1, possibly with some slight modifications.\n- I'd like to ask on how the authors think about the case if the agents observe some noisy context about themselves.\n\nMinor comments\n- I found the presentation of Stackelberg regret and Proposition 2.4 a bit redundant, as it does not appear in the main paper but there.\n- In Algorithm 1, what is data $S_t$ - it seems it's not updated at all?\n- L290, typo: withing\n- L298, duplicated threshold\n\nLIMITATIONS:\nDetailed comments are in Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey consider the problem of online learning with apple tasting feedback where the sequence of arriving agents may strategically modify their features (contexts). They show how to achieve sublinear strategic regret compared to the best policy in the hindsight if agents were reporting truthfully.\n\nTheir main result is $\\tilde{O}(\\sqrt{T})$ strategic regret when the sequence of agents arriving is stochastic. Their regret bound depends exponentially on d the context dimension. They show how to mitigiate this dependency and achieve a regret bound that depends polynomially on d but with a worse dependence on T. The main idea to to use a modified version of explore-then-commit algorithm that was introduced in the prior work.\n\nFinally, they extend their results to a sequence of agents chosen by an oblivious adversary that achieves sublinear regret however it requires an exponentially large amount of computation at each round.\n\nSTRENGTHS:\nI think the paper is written nicely and the results are nice. The algorithms seem to be a natural extension of the non-strategic setting. The model is interesting and the presentation is great.\n\nWEAKNESSES:\nI think the paper is written nicely and the results are nice. The algorithms seem to be a natural extension of the non-strategic setting. Since they are using the linear thresholds model, they need to shift the decision boundary to account for strategic behavior and a similar set of ideas have been studied in the previous work.\n\nQUESTIONS:\nIt might be helpful to add a paragraph on how your bounds differ from the non-strategic case.\n\nProp 2.4. last line, it might be helpful for the reader to argue why the first term is at most 0.\n\nLine 218, you can ensure the data is clean since it is strictly above the threshold?\n\nThm 3.3. it might be helpful to also remind the reader what c_0 is.\n\nAnother open question is to get regret bounds when the agents are picked by an adaptive adversary.\n\nAre there any connections between the trembling hand model that you are describing and the \\eps-best response model considered by [Haghtalab, Lykouris, Nietert, Wei' EC22]?\n\nAnother piece of work related to your bandit feedback could be the work by [Ahmadi, Blum, Yang' EC23] where in one of their models they propose a modification of EXP3 algorithm in the strategic setting.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an online learning problem with one-sided (apple-tasting) feedback. At each round $t\\in[T]$, an agent with a $d$-dimensional context vector $x_t$ arrives. The principal chooses a policy $\\pi_t$ to map the context to binary decisions. Given $\\pi_t$, the agent best responds strategically by modifying their context to $x'_t$ where $||x'_t-x_t||\\leq \\delta$. Then, the principal receives a one-sided, linear feedback in $x_t$. The goal is to design a policy with sub-linear strategic regret that compares the principal's reward with that of the optimal policy had agents reported their contexts truthfully. The authors propose algorithms for stochastic and adversarial settings.\n\nSTRENGTHS:\n- Online learning with strategic agents has many real-world applications and the results of this work could be applied to automatic decision-making processes such as lending and hiring. Therefore, the problem is very well-motivated.\n- The paper is very well-written, in particular, the proof sketches provided in the paper give a very clear high-level picture of the techniques and ideas used to analyze the performance of the algorithms.\n- The authors have done an excellent job comparing and contrasting their paper with prior works, it is clear how this paper contributes to this literature.\n\nWEAKNESSES:\n- While the proposed algorithms are a great first step towards solving this problem, they are sub-optimal from different aspects. First, Algorithm 1 does not make any use of unclean data and it simply discards them. While the authors have shown that using unclean data for estimation is not useful, they could be used with some confidence level to rule out some of the underperforming policies. Moreover, Algorithm 3 for the adversarial setting is practically not useful because of its exponential computational complexity.\n- Based on the motivating examples that are provided in the introduction, I don't see why agents strategically modifying their context vector is necessarily a bad thing. For instance, paying bills on time and maintaining low credit utilization to increase credit scores (which is used for lending) should be incentivized (rather than discouraged).\n\nQUESTIONS:\n- What is the reasoning behind a bounded $\\ell_2$ norm perturbation of size $\\delta$ (instead of a different norm or a packing-type constraint) as the effort budget? Could you explain this in the context of a motivating application? \n- Why is $r_0$ missing in the linear thresholding policy of Algorithm 3?\n- Instead of considering each $d$-dimensional vector in $[0,1]^d$ (or a discretized grid of such vectors) as experts in Algorithm 3, is it possible to assume a structure on the policies and solve the resulting online structured learning problem (as it's done in the following papers)?\n\nKoolen, Wouter M., Manfred K. Warmuth, and Jyrki Kivinen. \"Hedging Structured Concepts.\" COLT. 2010.\nCohen, Alon, and Tamir Hazan. \"Following the perturbed leader for online structured learning.\" International Conference on Machine Learning. PMLR, 2015.\n\n- Algorithm 4 is mentioned multiple times in the paper, but it's not provided in the main text. It'd be great to either have the algorithm or a short description of it in the paper.\n- Is it possible to relax the assumption that the variance $\\sigma^2$ is known? How would such a relaxation affect the results?\n\n\n--------------------------------\nI've read the authors' rebuttal, thanks for addressing my questions and concerns.\n\nLIMITATIONS:\nThe limitations are clearly discussed in the conclusion. This is a theoretical work and a discussion of potential negative societal impacts is not necessary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}