SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ALGORITHM 1: Strategy-Aware OLS with Apple Tasting Feedback (SA-OLS)
Assign action 1 for the first  $d$  rounds.   
Set  $\mathcal{D}_{d + 1} = \{(\mathbf{x}_s,r_s^{(1)})\}_{s = 1}^d$    
for  $t = d + 1,\dots ,T$  do Estimate  $\pmb{\theta}^{(1)}$  as  $\widehat{\pmb{\theta}}_t^{(1)}$  using OLS and data  $\mathcal{D}_t$  Assign action  $a_{t} = 1$  if  $\langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}_t^\prime \rangle \geq \delta \cdot \| \widehat{\pmb{\theta}}_t^{(1)}\| _2 + r_0.$  if  $\langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}_t^\prime \rangle >\delta \| \widehat{\pmb{\theta}}_t^{(1)}\| _2 + r_0$  then Conclude that  $\mathbf{x}_t^\prime = \mathbf{x}_t$ $\mathcal{D}_{t + 1} = \mathcal{D}_t\cup \{(\mathbf{x}_t,r_t^{(1)})\}$  else  $\mathcal{D}_{t + 1} = \mathcal{D}_t$
for exploration; a growing area of interest in the online learning literature (see references in Section 1.2). Moreover, such assumptions often hold in practice. For example, in the related problem of (non-strategic) contextual bandits (we will later show how our results extend to the strategic version of this problem), Bietti et al. [14] find that a greedy algorithm with no explicit exploration achieved the second-best empirical performance across a large number of datasets when compared to many popular contextual bandit algorithms. In our settings of interest (e.g. lending, hiring), such an assumption is reasonable if there is sufficient diversity in the applicant pool. In Section 4 we show how to remove this assumption, albeit at the cost of worse regret rates and exponential computational complexity.
At a high level, our algorithm (formally stated in Algorithm 1) relies on three key ingredients to achieve sublinear strategic regret:
1. A running estimate of  $\theta^{(1)}$  is used to compute a linear policy, which separates agents who receive action 1 from those who receive action 0. Before deploying, we shift the decision boundary by the effort budget  $\delta$  to account for the agents strategizing.
2. We maintain an estimate of  $\pmb{\theta}^{(1)}$  (denoted by  $\widehat{\pmb{\theta}}^{(1)}$ ) and only updating it when  $a_{t} = 1$  and we can ensure that  $\mathbf{x}_t' = \mathbf{x}_t$ .
3. We assign actions "greedily" (i.e. using no explicit exploration) w.r.t. the shifted linear policy.
Shifted linear policy If agents were not strategic, assigning action 1 if  $\langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}_t\rangle \geq r_0$  and action 0 otherwise would be a reasonable strategy to deploy, given that  $\widehat{\pmb{\theta}}_t^{(1)}$  is our "best estimate" of  $\pmb{\theta}^{(1)}$  so far. Recall that the strategically modified context  $\mathbf{x}_t^\prime$  is s.t.,  $\| \mathbf{x}_t^\prime -\mathbf{x}_t\| \leq \delta$ . Hence, in Algorithm 1, we shift the linear policy by  $\delta \| \widehat{\pmb{\theta}}^{(1)}\| _2$  to account for strategically modified contexts. Now, action 1 is only assigned if  $\langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}_t\rangle \geq \delta \| \widehat{\pmb{\theta}}^{(1)}\| _2 + r_0$ . This serves two purposes: (1) It makes it so that any agent with unmodified context  $\mathbf{x}$  such that  $\langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}\rangle < r_0$  cannot receive action 1, no matter how they strategize. (2) It forces some agents with contexts in the band  $r_0\leq \langle \widehat{\pmb{\theta}}_t^{(1)},\mathbf{x}\rangle < \delta \| \widehat{\pmb{\theta}}^{(1)}\| _2 + r_0$  to strategize in order to receive action 1. Estimating  $\pmb{\theta}^{(1)}$  After playing action 1 for the first  $d$  rounds, Algorithm 1 forms an initial estimate of  $\pmb{\theta}^{(1)}$  via ordinary least squares (OLS). Note that since the first  $d$  agents will receive action 1 regardless of their context, they have no incentive to modify and thus  $\mathbf{x}_t^{\prime} = \mathbf{x}_t$  for  $t\leq d$ . In future rounds, the algorithm's estimate of  $\pmb{\theta}^{(1)}$  is only updated whenever  $\mathbf{x}_t^\prime$  lies strictly on the positive side of the linear decision boundary. We call these contexts clean, and can infer that  $\mathbf{x}_t^\prime = \mathbf{x}_t$  due to the lazy tiebreaking assumption in Definition 2.1 (i.e. agents will not strategize more than is necessary to receive the positive classification).
Condition 3.2 (Sufficient condition for  $\mathbf{x}' = \mathbf{x}$ ). Given a shifted linear policy parameterized by  $\beta^{(1)} \in \mathbb{R}^d$ , we say that a context  $\mathbf{x}'$  is clean if  $\langle \boldsymbol{\beta}^{(1)}, \mathbf{x}' \rangle > \delta \| \boldsymbol{\beta}^{(1)} \|_2 + r_0$ .
Greedy action assignment By assigning actions greedily according to the current (shifted) linear policy, we are relying on the diversity in the agent population for implicit exploration (i.e., to collect more datapoints to update our estimate of  $\theta^{(1)}$ ). As we will show, this implicit exploration
is sufficient to achieve  $\widetilde{\mathcal{O}} (\sqrt{T})$  strategic regret under Assumption 3.1, albeit at the cost of an exponentially-large (in  $d$ ) constant which depends on the agents' ability to manipulate  $(\delta)$ .
We are now ready to present our main result: strategic regret guarantees for Algorithm 1 under apple tasting feedback.
Theorem 3.3 (Informal; detailed version in Theorem B.1). With probability  $1 - \gamma$ , Algorithm 1 achieves the following performance guarantee:
$$
\mathrm {R e g} (T) \leq \widetilde {\mathcal {O}} \left(\frac {1}{c _ {0} \cdot c _ {1} (d , \delta) \cdot c _ {2} (d , \delta)} \sqrt {d \sigma^ {2} T \log (4 d T / \gamma)}\right)
$$
where  $c_{0}$  is a lower bound on the density ratio as defined in Assumption 3.1,  $c_{1}(d,\delta)\coloneqq \mathbb{P}_{\mathbf{x}\sim U^{d}}(\mathbf{x}[1]\geq \delta)\geq \Theta \left(\frac{(1 - \delta)^{d / 2}}{d^{2}}\right)$  for sufficiently large  $d$  and  $c_{2}(d,\delta)\coloneqq \mathbb{E}_{\mathbf{x}\sim U^{d}}[\mathbf{x}[2]^{2}|\mathbf{x}[1]\geq \delta ]\geq \left(\frac{3}{4} -\frac{1}{2}\delta -\frac{1}{4}\delta^{2}\right)^{3},$  where  $\mathbf{x}[i]$  denotes the  $i$ -th coordinate of a vector  $\mathbf{x}$ .
Proof sketch. Our analysis begins by using properties of the strategic agents and shifted linear decision boundary to upper-bound the per-round strategic regret for rounds  $t > d$  by a term proportional to  $\| \widehat{\pmb{\theta}}_t^{(1)} - \pmb{\theta}^{(1)}\| _2$ , i.e., our instantaneous estimation error for  $\pmb{\theta}^{(1)}$ . Next we show that
$$
\| \widehat {\pmb {\theta}} _ {t} ^ {(1)} - \pmb {\theta} ^ {(1)} \| _ {2} \leq \frac {\left\| \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \epsilon_ {s} \mathbb {1} \left\{\mathcal {I} _ {s} ^ {(1)} \right\} \right\| _ {2}}{\lambda_ {m i n} (\sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top} \mathbb {1} \left\{\mathcal {I} _ {s} ^ {(1)} \right\})}
$$
where  $\lambda_{min}(M)$  is the minimum eigenvalue of (symmetric) matrix  $M$ , and  $\mathcal{I}_s^{(1)} = \{\langle \widehat{\pmb{\theta}}_s^{(1)},\mathbf{x}_s\rangle \geq$ $\delta \| \widehat{\pmb{\theta}}_s^{(1)}\| _2 + r_0\}$  is the event that Algorithm 1 assigns action  $a_{s} = 1$  and can verify that  $\mathbf{x}_s^{\prime} = \mathbf{x}_s$ . We upper-bound the numerator using a variant of Azuma's inequality for martingales with subgaussian tails. Next, we use properties of Hermitian matrices to show that  $\lambda_{min}(\sum_{s = 1}^{t}\mathbf{x}_{s}\mathbf{x}_{s}^{\top}\mathbb{1}\{\mathcal{I}_{s}^{(1)}\})$  is lower-bounded by two terms: one which may be bounded w.h.p. by using the extension of Azuma's inequality for matrices, and one of the form  $\sum_{s = 1}^{t}\lambda_{min}(\mathbb{E}_{s - 1}[\mathbf{x}_s\mathbf{x}_s^\top \mathbb{1}\{\mathcal{I}_s^{(1)}\}]')$ , where  $\mathbb{E}_{s - 1}$  denotes the expected value conditioned on the filtration up to time  $s$ . Note that up until this point, we have only used the fact that contexts are drawn i.i.d. from a bounded distribution.
Using Assumption 3.1 on the bounded density ratio, we can lower bound  $\lambda_{min}(\mathbb{E}_{s-1}[\mathbf{x}_s\mathbf{x}_s^\top \mathbb{1}\{\mathcal{I}_s^{(1)}\}])$  by  $\lambda_{min}(\mathbb{E}_{U^d,s-1}[\mathbf{x}_s\mathbf{x}_s^\top \mathbb{1}\{\mathcal{I}_s^{(1)}\}])$ , where the expectation is taken with respect to the uniform distribution over the  $d$ -dimensional ball. We then use properties of the uniform distribution to show that  $\lambda_{min}(\mathbb{E}_{U^d,s-1}[\mathbf{x}_s\mathbf{x}_s^\top \mathbb{1}\{\mathcal{I}_s^{(1)}\}]) \geq \mathcal{O}(c_0 \cdot c(d,\delta))$ . Putting everything together, we get that  $\|\widehat{\pmb{\theta}}_t^{(1)} - \pmb{\theta}^{(1)}\|_2 \leq (c_0 \cdot c(d,\delta) \cdot \sqrt{t})^{-1}$  with high probability. Via a union bound and the fact that  $\sum_{t \in [T]} \frac{1}{\sqrt{t}} \leq 2T$ , we get that  $\mathrm{Reg}(T) \leq \widetilde{\mathcal{O}}\left(\frac{1}{c_0 \cdot c(d,\delta)}\sqrt{T}\right)$ . Finally, we use tools from high-dimensional geometry to lower bound the volume of a spherical cap and we show that for sufficiently large  $d$ ,  $c_1(d,\delta) \geq \Theta\left(\frac{(1-\delta)^{d/2}}{d^2}\right)$ .
3.1 High-dimensional contexts
While we typically think of the number of agents  $T$  as growing and the context dimension  $d$  as constant in our applications of interest, there may be situations in which  $T$  is either unknown or small. Under such settings, the  $1 / c(d,\delta)$  dependence in the regret bound (where  $c(d,\delta) = c_1(d,\delta)\cdot c_2(d,\delta)$ ) may become problematic if  $\delta$  is close to 1. This begs the question: "Why restrict the OLS estimator in Algorithm 1 to use only clean contexts (as defined in Condition 3.2)?" Perhaps unsurprisingly, we show in Appendix B that the estimate  $\widehat{\pmb{\theta}}^{(1)}$  given by OLS will be inconsistent if even a constant fraction of agents strategically modify their contexts.
4While we assume that  $\delta$  is known to the principal, Algorithm 1 is fairly robust to overestimates of  $\delta$ , in the sense that (1) it will still produce a consistent estimate for  $\theta^{(1)}$  (albeit at a rate which depends on the overestimate instead of the actual value of  $\delta$ ) and (2) it will incur a constant penalty in regret which is proportional to the amount of over-estimation.
ALGORITHM 2: Explore-Then-Commit
Input: Time horizon  $T$ , failure probability  $\gamma$   
Set  $T_0$  according to Theorem B.9  
Assign action 1 for the first  $T_0$  rounds  
Estimate  $\pmb{\theta}^{(1)}$  as  $\hat{\pmb{\theta}}_{T_0}^{(1)}$  via OLS  
for  $t = T_0 + 1, \dots, T$  do  
    Assign action  $a_t = 1$  if  $\langle \hat{\pmb{\theta}}_{T_0}^{(1)}, \mathbf{x}_t \rangle \geq \delta \cdot \| \hat{\pmb{\theta}}_{T_0}^{(1)}\|_2$  and action  $a_t = 0$  otherwise
Given the above, it seems reasonable to restrict ourselves to learning procedures which only use data from agents for which the principal can be sure that  $\mathbf{x}' = \mathbf{x}$ . Under such a restriction, it is natural to ask whether there exists some sequence of linear polices which maximizes the number of points of the form  $(\mathbf{x}_t', r_t(1))$  for which the principal can be sure that  $\mathbf{x}_t' = \mathbf{x}_t$ . Again, the answer is no:
Proposition 3.4. For any sequence of linear policies  $\{\beta_t\}_t$ , the expected number of clean points is:
$$
\mathbb {E} _ {\mathbf {x} _ {1}, \ldots , \mathbf {x} _ {T} \sim U ^ {d}} \left[ \sum_ {t \in [ T ]} \mathbb {1} \{\langle \mathbf {x} _ {t}, \boldsymbol {\beta} _ {t} \rangle > \delta \| \boldsymbol {\beta} _ {t} \| _ {2} \} \right] = c _ {1} (d, \delta) \cdot T
$$
when (initial) contexts are drawn uniformly from the  $d$ -dimensional unit sphere.
The proof follows from the rotational invariance of the uniform distribution over the unit sphere. Intuitively, Proposition 3.4 implies that any algorithm which wishes to learn  $\theta^{(1)}$  using clean samples will only have  $c_{1}(d,\delta)\cdot T$  datapoints in expectation. Observe that this dependence on  $c_{1}(d,\delta)$  arises as a direct result of the agents' ability to strategize. We remark that a similar constant often appears in the regret analysis of BIC bandit algorithms (see Section 1.2). Much like our work, [43] find that their regret rates depend on a constant which may be arbitrarily large, depending on how hard it is to persuade agents to take the principal's desired action in their setting. The authors conjecture that this dependence is an inevitable "price of incentive-compatibility". While our results do not rule out better strategic regret rates in  $d$  for more complicated algorithms (e.g., those which deploy non-linear policies), it is often unclear how strategic agents would behave in such settings, both in theory (Definition 2.1 would require agents to solve a non-convex optimization with potentially no closed-form solution) and in practice, making the analysis of such nonlinear policies difficult in strategic settings.
We conclude this section by showing that polynomial dependence on  $d$  is possible, at the cost of  $\widetilde{\mathcal{O}}(T^{2/3})$  strategic regret. Specifically, we provide an algorithm (Algorithm 3) which obtains the following regret guarantee whenever  $T$  is small or unknown, which uses Algorithm 1 and a variant of the explore-then-commit algorithm (Algorithm 2) as subroutines:
Theorem 3.5 (Informal; details in Theorem B.13). Algorithm 3 incurs expected strategic regret
$$
\mathbb {E} [ \mathrm {R e g} (T) ] = \widetilde {\mathcal {O}} \left(\min \left\{\frac {d ^ {5 / 2}}{(1 - \delta) ^ {d / 2}} \cdot \sqrt {T}, d \cdot T ^ {2 / 3} \right\}\right),
$$
where the expectation is taken with respect to the sequence of contexts  $\{\mathbf{x}_t\}_{t\in [T]}$  and random noise  $\{\epsilon_t\}_{t\in [T]}$ .
The algorithm proceeds by playing a "strategy-aware" variant of explore-then-commit (Algorithm 2) with a doubling trick until the switching time  $\tau^{*} = g(d,\delta)$  is reached. Note that  $g(d,\delta)$  is a function of both  $d$  and  $\delta$ , not  $c_{0}$ . If round  $\tau^{*}$  is indeed reached, the algorithm switches over to Algorithm 1 for the remaining rounds.
Extension to bandit feedback Algorithm 1 can be extended to handle bandit feedback by explicitly keeping track of an estimate  $\widehat{\pmb{\theta}}^{(0)}$  of  $\pmb{\theta}^{(0)}$  via OLS, assigning action  $a_{t} = 1$  if and only if  $\langle \widehat{\pmb{\theta}}_t^{(1)} - \widehat{\pmb{\theta}}_t^{(0)},\mathbf{x}_t^\prime \rangle \geq \delta \cdot \| \widehat{\pmb{\theta}}_t^{(1)} - \widehat{\pmb{\theta}}_t^{(0)}\| _2$ , and updating the OLS estimate of  $\widehat{\pmb{\theta}}^{(0)}$  whenever  $a_{t} = 0$  (since agents will not strategize to receive action 0). Algorithm 3 may be extended to bandit feedback by "exploring" for twice as long in Algorithm 2, in addition to using the above modifications. In both cases, the strategic regret rates are within a constant factor of the rates obtained in Theorem 3.3 and Theorem 3.5.
ALGORITHM 3: Strategy-aware online classification with unknown time horizon
Compute switching time  $\tau^{*} = g(d,\delta)$    
Let  $\tau_0 = 1$    
for  $i = 1,2,3,\ldots$  do Let  $\tau_{i} = 2\cdot \tau_{i - 1}$  if  $\sum_{j = 1}^{i}\tau_{j} <   \tau^{*}$  then Run Algorithm 2 with time horizon  $\tau_{i}$  and failure probability  $1 / \tau_i^2$  else Break and run Algorithm 1 for the remainder of the rounds
4 Beyond stochastic contexts
In this section, we allow the sequence of initial agent contexts to be chosen by an (oblivious) adversary. This requires new algorithmic ideas, as the regression-based algorithms of Section 3 suffer linear strategic regret under this adversarial setting. Our algorithm (Algorithm 4) is based on the popular EXP3 algorithm [6]. At a high level, Algorithm 4 maintains a probability distribution over "experts", i.e., a discretized grid  $\mathcal{E}$  over carefully-selected policies. In particular, each grid point  $\mathbf{e} \in \mathcal{E} \subseteq \mathbb{R}^d$  represents an "estimate" of  $\pmb{\theta}^{(1)}$ , and corresponds to a slope vector which parameterizes a (shifted) linear threshold policy, like the ones considered in Section 3. We use  $a_{t,\mathbf{e}}$  to refer to the action played by the principal at time  $t$ , had they used the linear threshold policy parameterized by expert  $\mathbf{e}$ . At every time-step, (1) the adversary chooses an agent  $\mathbf{x}_t$ , (2) a slope vector  $\mathbf{e}_t \in \mathcal{E}$  is selected according to the current distribution, (3) the principal commits to assigning action 1 if and only if  $\langle \mathbf{e}_t, \mathbf{x}_t' \rangle \geq \delta \| \mathbf{e}_t \|_2$ , (4) the agent strategically modifies their context  $\mathbf{x}_t \to \mathbf{x}_t'$ , and (5) the principal assigns an action  $a_t$  according to the policy and receives the associated reward  $r_t(a_t)$  (under apple tasting feedback).
Algorithm EXP4, which maintains a distribution over experts and updates the loss of all experts based on the current action taken, is not directly applicable in our setting as the strategic behavior of the agents prevents us from inferring the loss of each expert at every time-step [?]. This is because if  $\mathbf{x}_t' \neq \mathbf{x}_t$  under the thresholding policy associated with expert e), it is generally not possible to "back out"  $\mathbf{x}_t$  given  $\mathbf{x}_t'$ , which prevents us from predicting the counterfactual context the agent would have modified to had the principal been using expert  $\mathbf{e}'$  instead. As a result, we use a modification of the standard importance-weighted loss estimator to update the loss of only the policy played by the algorithm (and therefore the distribution over policies). Our regret guarantees for Algorithm 4 are as follows:
Theorem 4.1 (Informal; detailed version in Theorem C.1). Algorithm 4 incurs expected strategic regret  $\mathbb{E}[\mathrm{Reg}(T)] = \tilde{\mathcal{O}}(T^{(d+1)/(d+2)})$ .
We remark that Algorithm 4 may be extended to handle settings in which agents are selected by an adaptive adversary by using EXP3.P [6] in place of EXP3.
Proof sketch. The analysis is broken down into two parts. In the first part, we bound the regret w.r.t. the best policy on the grid. In the second, we bound the error incurred for playing policies on the grid, rather than the continuous space of policies. We refer to this error as the Strategic Discretization Error  $(SDE(T))$ . The analysis of the regret on the grid mostly follows similar steps to the analysis of EXP3 / EXP4. The important difference is that we shift the reward obtained by  $a_{t}$ , by a factor of  $1 + \lambda$ , where  $\lambda$  is a (tunable) parameter of the algorithm. This shifting (which does not affect the regret, since all the losses are shifted by the same fixed amount) guarantees that the losses at each round are non-negative and bounded with high probability. Technically, this requires bounding the tails of the subgaussian of the noise parameters  $\epsilon_{t}$ .
We now shift our attention to bounding  $SDE(T)$ . The standard analysis of the discretization error in the non-strategic setting does not go through for our setting, since an agent may strategize very differently with respect to two policies which are "close together" in  $\ell_2$  distance, depending on the agent's initial context. Our analysis proceeds with a case-by-case basis. Consider the best expert  $\mathbf{e}^*$  in the grid. If  $a_{t,\mathbf{e}^*} = \pi^*(\mathbf{x}_t)$  (i.e., the action of the best expert matches that of the optimal policy), there is no discretization error in round  $t$ . Otherwise, if  $a_{t,\mathbf{e}^*} \neq \pi^*(\mathbf{x}_t)$ , we show that the per-round  $SDE$  is upper-bounded by a term which looks like twice the discretization upper-bound for the non-strategic setting, plus an additional term. We show that this additional term must always be non-positive by considering two subcases ( $a_{t,\mathbf{e}^*} = 1$ ,  $\pi^*(\mathbf{x}_t) = 0$  and  $a_{t,\mathbf{e}^*} = 0$ ,  $\pi^*(\mathbf{x}_t) = 1$ ) and using properties about how agents strategize against the deployed algorithmic policies.
ALGORITHM 4: EXP3 with strategy-aware experts (EXP3-SAE)
Create set of discretized policies  $\mathbf{e}\in \mathcal{E} = [(1 / \varepsilon)^d ]$  , where  $\varepsilon = (d\sigma \log (T) / T)^{1 / (d + 2)}$    
Set parameters  $\eta = \sqrt{\frac{\log(|\mathcal{E}|)}{T\lambda^2|\mathcal{E}|}}$ $\gamma = 2\eta \lambda |\mathcal{E}|$  , and  $\lambda = \sigma \sqrt{2\log T}$    
Initialize probability distribution  $p_t(\mathbf{e}) = 1 / |\mathcal{E}|,\forall \mathbf{e}\in \mathcal{E}$    
for  $t\in [T]$  do Choose policy  $\mathbf{e}_t$  from probability distribution  $q_{t}(\mathbf{e}) = (1 - \gamma)\cdot p_{t}(\mathbf{e}) + \frac{\gamma}{|\mathcal{E}|}$  Observe  $\mathbf{x}_t^\prime$  Play action  $a_{t,\mathbf{e}_t} = 1$  if  $\langle \mathbf{e}_t,\mathbf{x}_t'\rangle \geq \delta \| \mathbf{e}_t\| _2$  . Otherwise play action  $a_{t,\mathbf{e}_t} = 0$  Observe reward  $r_t(a_{t,\mathbf{e}_t})$  Update loss estimator for each policy  $\mathbf{e}\in \mathcal{E}$  ..  $\widehat{\ell}_{t}(\mathbf{e}) = (1 + \lambda -r_{t}(a_{t,\mathbf{e}_{t}}))\cdot \mathbb{1}\{\mathbf{e} = \mathbf{e}_{t}\} /q_{t}(\mathbf{e}).$  Update probability distribution  $\forall \mathbf{e}\in \mathcal{E}$  ..  $p_{t + 1}(\mathbf{e})\propto p_t(\mathbf{e})\cdot \exp \left(-\eta \widehat{\ell}_t(\mathbf{e})\right)$
Computational complexity While both Algorithm 1 and Algorithm 3 have  $\mathcal{O}(d^3)$  per-iteration computational complexity, Algorithm 4 must maintain and update a probability distribution over a grid of size exponential in  $d$  at every time-step, making it hard to use in practice if  $d$  is large. We view the design of computationally efficient algorithms for adversarially-chosen contexts as an important direction for future research.
Extension to bandit feedback Algorithm 4 may be extended to the bandit feedback setting by maintaining a grid over estimates of  $\pmb{\theta}^{(1)} - \pmb{\theta}^{(0)}$  (instead of over  $\pmb{\theta}^{(1)}$ ). No further changes are required.
We study the problem of classification under incentives with apple tasting feedback. Such one-sided feedback is often what is observed in real-world strategic settings including lending and hiring. Our main result is a "greedy" algorithm (Algorithm 1) which achieves  $\widetilde{\mathcal{O}}(\sqrt{T})$  strategic regret when the initial agent contexts are generated stochastically. The regret of Algorithm 1 depends on a constant  $c_{1}(d, \delta)$  which scales exponentially in the context dimension, which may be problematic in settings for which the number of agents is small or unknown. To address this, we provide an algorithm (Algorithm 3) which combines Algorithm 1 with a strategy-aware version of the explore-then-commit algorithm using a doubling trick to achieve  $\widetilde{\mathcal{O}}\left(\min\left\{\frac{\sqrt{dT}}{c_{1}(d, \delta)}, d \cdot T^{2/3}\right\}\right)$  expected strategic regret whenever  $T$  is unknown. Finally, we relax the assumption of stochastic contexts and allow for contexts to be generated adversarially. Algorithm 4 achieves  $\widetilde{\mathcal{O}}(T^{\frac{d+1}{d+2}})$  expected strategic regret whenever agent contexts are generated adversarially by running EXP3 over a discretized grid of strategy-aware policies, but has exponential-in- $d$  per-round computational complexity. All of our results also apply to the more general setting of bandit feedback, under slight modifications to the algorithms. There are several directions for future work:
Unclean data The regret of Algorithm 1 depends on a constant which is exponentially large in  $d$ , due to the fact that it only learns using clean data (Condition 3.2). While learning using unclean data will generally produce an inconsistent estimator, it would be interesting to see if the principal could leverage this data to remove the dependence on this constant. Alternatively, lower bounds which show that using unclean data will not improve regret would also be interesting.