SlowGuess/NeurIPS_2023_OCR huggingface dataset:
keeganh@cmu.edu
Chara Podimata
MIT & Archimedes/Athena RC
podimata@mit.edu
Algorithmic decision-making in high-stakes domains often involves assigning decisions to agents with incentives to strategically modify their input to the algorithm. In addition to dealing with incentives, in many domains of interest (e.g. lending and hiring) the decision-maker only observes feedback regarding their policy for rounds in which they assign a positive decision to the agent; this type of feedback is often referred to as apple tasting (or one-sided) feedback. We formalize this setting as an online learning problem with apple-tasting feedback where a principal makes decisions about a sequence of  $T$  agents, each of which is represented by a context that may be strategically modified. Our goal is to achieve sublinear strategic regret, which compares the performance of the principal to that of the best fixed policy in hindsight, if the agents were truthful when revealing their contexts. Our main result is a learning algorithm which incurs  $\tilde{\mathcal{O}}(\sqrt{T})$  strategic regret when the sequence of agents is chosen stochastically. We also give an algorithm capable of handling adversarially-chosen agents, albeit at the cost of  $\tilde{\mathcal{O}}(T^{(d+1)/(d+2)})$  strategic regret (where  $d$  is the dimension of the context). Our algorithms can be easily adapted to the setting where the principal receives bandit feedback—this setting generalizes both the linear contextual bandit problem (by considering agents with incentives) and the strategic classification problem (by allowing for partial feedback).
Algorithmic systems have recently been used to aid in or automate decision-making in high-stakes domains (including lending and hiring) in order to, e.g., improve efficiency or reduce human bias [12, 1]. When subjugated to algorithmic decision-making in high-stakes settings, individuals have an incentive to strategically modify their observable attributes to appear more qualified. Such behavior is often observed in practice. For example, credit scores are often used to predict the likelihood an individual will pay back a loan on time if given one. Online articles with titles like "9 Ways to Build and Improve Your Credit Fast" are ubiquitous and offer advice such as "pay credit card balances strategically" in order to improve one's credit score with minimal effort [46]. In hiring, common advice ranges from curating a list of keywords to add to one's resume, to using white font in order to "trick" automated resume scanning software [23, 2]. If left unaccounted for, such strategic manipulations could result in individuals being awarded opportunities for which they are not qualified for, possibly at the expense of more deserving candidates. As a result, it is critical to keep individuals' incentives in mind when designing algorithms for learning and decision-making in high-stakes settings.
In addition to dealing with incentives, another challenge of designing learning algorithms for high-stakes settings is the possible selection bias introduced by the way decisions are made. In particular, decision-makers often only have access to feedback about the deployed policy from individuals that have received positive decisions (e.g., the applicant is given the loan, the candidate is hired to the job and then we can evaluate how good our decision was). In the language of online learning, this type of feedback is known as apple tasting (or one-sided) feedback. When combined, these two complications (incentives & one-sided feedback) have the potential to amplify one other, as algorithms can learn
only when a positive decision is made, but individuals have an incentive to strategically modify their attributes in order to receive such positive decisions, which may interfere with the learning process.
We formalize our setting as a game between a principal and a sequence of  $T$  strategic agents, each with an associated context  $\mathbf{x}_t$  which describes the agent. At every time  $t \in \{1, \dots, T\}$ , the principal deploys a policy  $\pi_t$ , a mapping from contexts to binary decisions (e.g., whether to accept/reject a loan applicant). Given policy  $\pi_t$ , agent  $t$  then presents a (possibly modified) context  $\mathbf{x}_t'$  to the algorithm, and receives a decision  $a_t = \pi_t(\mathbf{x}_t')$ . If  $a_t = 1$ , the principal observes reward  $r_t(a_t) = r_t(1)$ ; if  $a_t = 0$  they receive no feedback.  $r_t(0)$  is assumed to be known and constant across rounds.<sup>1</sup> Our metric of interest is strategic regret, i.e., regret with respect to the best fixed policy in hindsight, if agents were truthful when reporting their contexts.
Our main result is an algorithm which achieves  $\tilde{O}(\sqrt{T})$  strategic regret (with polynomial per-round runtime) when there is sufficient randomness in the distribution over agents (Algorithm 1). At a high level, our algorithm deploys a linear policy at every round which is appropriately shifted to account for the agents' strategic behavior. We identify a sufficient condition under which the data received by the algorithm at a given round is "clean", i.e. has not been strategically modified. Algorithm 1 then online-learns the relationship between contexts and rewards by only using data for which it is sure is clean.
In contrast to performance of algorithms which operate in the non-strategic setting, the regret of Algorithm 1 depends on an exponentially-large constant  $c(d,\delta)\approx (1 - \delta)^{-d}$  due to the one-sided feedback available for learning, where  $d$  is the context dimension and  $\delta \in (0,1)$  is a parameter which represents the agents' ability to manipulate. While this dependence on  $c(d,\delta)$  is insignificant when the number of agents  $T\to \infty$  (i.e. is very large), it may be problematic for the principal whenever  $T$  is either small or unknown. To mitigate this issue, we show how to obtain  $\tilde{O} (d\cdot T^{2 / 3})$  strategic regret by playing a modified version of the well-known explore-then-commit algorithm (Algorithm 2). At a high level, Algorithm 2 "explores" by always assigning action 1 for a fixed number of rounds (during which agents do not have an incentive to strategize) in order to collect sufficient information about the data-generating process. It then "exploits" by using this data learn a strategy-aware linear policy. Finally, we show how to combine Algorithm 1 and Algorithm 2 to achieve  $\tilde{O} (\min \{c(d,\delta)\cdot \sqrt{T},d\cdot T^{2 / 3}\})$  strategic regret whenever  $T$  is unknown.
While the assumption of stochastically-chosen agents is well-motivated in general, it may be overly restrictive in some specific settings. Our next result is an algorithm which obtains  $\tilde{O}(T^{(d+1)/(d+2)})$  strategic regret when agents are chosen adversarially (Algorithm 4). Algorithm 4 uses a variant of the popular Exp3 algorithm to trade off between a carefully constructed set of (exponentially-many) policies [6]. As a result, it achieves sublinear strategic regret when agents are chosen adversarially, but requires an exponentially-large amount of computation at every round.
Finally, we note that while our primary setting of interest is that of one-sided feedback, all of our algorithms can be easily extended to the more general setting in which the principal receives bandit feedback at each round, i.e.  $r_t(0)$  is not constant and must be learned from data. To the best of our knowledge, we are the first to consider strategic learning in the contextual bandit setting.
Strategic responses to algorithmic decision-making There is a growing line of work at the intersection of economics and computation on algorithmic decision-making with incentives, under the umbrella of strategic classification or strategic learning [27, 3, 41, 22] focusing on online learning settings [21, 19], causal learning [52, 31, 29, 34], incentivizing desirable behavior [39, 28, 11, 42], incomplete information [30, 25, 37]. In its most basic form, a principal makes either a binary or real-valued prediction about a strategic agent, and receives full feedback (e.g., the agent's label) after the decision is made. While this setting is similar to ours, it crucially ignores the one-sided feedback structure present in many strategic settings of interest. In our running example of hiring, full feedback would correspond to a company not offering an applicant a job, and yet still getting to observe whether they would have been a good employee! As a result, such methods are not applicable in our setting. Concurrent work [18] studies the effects of bandit feedback in the related problem of performative prediction [47], which considers data distribution shifts at the population level in response to the deployment of a machine learning model. In contrast, our focus is on strategic responses to machine
<sup>1</sup>We relax this assumption at later parts of the paper with virtually no impact on our results.
learning models at the individual level under apple tasting and bandit feedback. Ahmadi et al. [4] study an online strategic learning problem in which they consider "bandit feedback" with respect to the deployed classifier. In contrast, we use the term "bandit feedback" to refer to the fact that we only see the outcome when for the action/decision taken.
Apple tasting and online learning Helmbold et al. [32] introduce the notion of apple-tasting feedback for online learning. In particular, they study a binary prediction task over "instances" (e.g., fresh/rotten apples), in which a positive prediction is interpreted as accepting the instance (i.e. "tasting the apple") and a negative prediction is interpreted as rejecting the instance (i.e., not tasting the apple). The learner only gets feedback when the instance is accepted (i.e., the apple is tasted). While we are the first to consider classification under incentives with apple tasting feedback, similar feedback models have been studied in the context of algorithmic fairness [9], partial-monitoring games [5], and recidivism prediction [24]. A related model of feedback is that of contaminated controls [40], which considers learning from (1) a treated group which contains only treated members of the agent population and (2) a "contaminated" control group with samples from the entire agent population (not just those under control). Technically, our results are also related to a line of work in contextual bandits which shows that greedy algorithms without explicit exploration can achieve sublinear regret as long as the underlying context distribution is sufficiently diverse [49, 8, 38, 53, 48].
Bandits and agents A complementary line of work to ours is that of Bayesian incentive-compatible (BIC) exploration in multi-armed bandit problems [43, 35, 50, 36, 44, 45]. Under such settings, the goal of the principal is to persuade a sequence of  $T$  agents with incentives to explore across several different actions with bandit feedback. In contrast, in our setting it is the principal, not the agents, who is the one taking actions with partial feedback. As a result there is no need for persuasion, but the agents now have an incentive to strategically modify their behavior in order to receive a more desirable decision/action.
Other related work Finally, our work is broadly related to the literature on learning in repeated Stackelberg games [7, 55], online Bayesian persuasion [16, 17, 13], and online learning in principal-agent problems [20, 33, 56]. In the repeated Stackelberg game setting, the principal (leader) commits to a mixed strategy over a finite set of actions, and the agent (follower) best-responds by playing an action from a finite set of best-responses. Unlike in our setting, both the principal's and agent's payoffs can be represented by matrices. In contrast, in our setting the principal commits to a pure strategy from a continuous set of actions, and the agent best-responds by playing an action from a continuous set. In online Bayesian persuasion, the principal (sender) commits to a "signaling policy" (a random mapping from "states of the world" to receiver actions) and the agent (receiver) performs a posterior update on the state based on the principal's signal, then takes an action from a (usually finite) set. In both this setting and ours, the principal's action is a policy. However in our setting the policy is a linear decision rule, whereas in the Bayesian persuasion setting, the policy is a set of conditional probabilities which form an "incentive compatible" signaling policy. This difference in the policy space for the principal typically leads to different algorithmic ideas being used in the two settings. Strategic learning problems are, broadly speaking, instances of principal-agent problems. In contract design, the principal commits to a contract (a mapping from "outcomes" to agent payoffs). The agent then takes an action, which affects the outcome. In particular, they take the action which maximizes their expected payoff, subject to some cost of taking the action. The goal of the principal is to design a contract such that their own expected payoff is maximized. While the settings are indeed similar, there are several key differences. First, in online contract design the principal always observes the outcome, whereas in our setting the principal only observes the reward if a positive decision is made. Second, the form of the agent's best response is different, which leads to different agent behavior and, as a result, different online algorithms for the principal.
2 Setting and background
We consider a game between a principal and a sequence of  $T$  agents. Each agent is associated with a context  $\mathbf{x}_t \in \mathcal{X} \subseteq \mathbb{R}^d$ , which characterizes their attributes (e.g., a loan applicant's credit history/report). At time  $t$ , the principal commits to a policy  $\pi_t: \mathcal{X} \to \{1,0\}$ , which maps from contexts to binary decisions (e.g., whether to accept/reject the loan application). We use  $a_t = 1$  to denote the principal's positive decision at round  $t$  (e.g., agent  $t$ 's loan application is approved), and  $a_t = 0$  to denote a negative decision (e.g., the loan application is rejected). Given  $\pi_t$ , agent  $t$  best-responds by strategically modifying their context within their effort budget as follows:
Definition 2.1 (Agent best response; lazy tiebreaking). Agent  $t$  best-responds to policy  $\pi_t$  by modifying their context according to the following optimization program.
$$
\mathbf{x}^{\prime}_{t}\in \operatorname *{arg  max}_{\mathbf{x}^{\prime}\in \mathcal{X}}\mathbb{1}\bigl\{\pi_{t}(\mathbf{x}^{\prime}) = 1\bigr \}
$$
$$
s. t. \left\| \mathbf {x} ^ {\prime} - \mathbf {x} _ {t} \right\| _ {2} \leq \delta
$$
Furthermore, we assume that if an agent is indifferent between two (modified) contexts, they choose the one which requires the least amount of effort to obtain (i.e., agents are lazy when tiebreaking).
In other words, every agent wants to receive a positive decision, but has only a limited ability to modify their (initial) context (represented by  $\ell_2$  budget  $\delta$ ). Such an effort budget may be induced by time or monetary constraints and is a ubiquitous model of agent behavior in the strategic learning literature (e.g., [39, 28, 19, 10]). We focus on linear thresholding policies where the principal assigns action  $\pi(\mathbf{x}') = 1$ , if and only if  $\langle \beta, \mathbf{x}' \rangle \geq \gamma$  for some  $\beta \in \mathbb{R}^d$ ,  $\gamma \in \mathbb{R}$ . We refer to  $\langle \beta, \mathbf{x}_t' \rangle = \gamma$  as the decision boundary. For linear thresholding policies, the agent's best-response according to Definition 2.1 is to modify their context in the direction of  $\beta / \| \beta \|_2$  until the decision-boundary is reached (if it can indeed be reached). While we present our results for lazy tiebreaking for ease of exposition, all of our results can be readily extended to the setting in which agents best-respond with a "trembling hand", i.e. trembling hand tiebreaking. Under this setting, we allow agents who strategically modify their contexts to "overshoot" the decision boundary by some bounded amount, which can be either stochastic or adversarially-chosen. See Appendix D for more details.
The principal observes  $\mathbf{x}_t^\prime$  and plays action  $a_{t} = \pi_{t}(\mathbf{x}_{t}^{\prime})$  according to policy  $\pi_t$ . If  $a_{t} = 0$ , the principal receives some known, constant reward  $r_t(0)\coloneqq r_0\in \mathbb{R}$ . On the other hand, if the principal assigns action  $a_{t} = 1$ , we assume that the reward the principal receives is linear in the agent's unmodified context, i.e.,
$$
r _ {t} (1) := \left\langle \boldsymbol {\theta} ^ {(1)}, \mathbf {x} _ {t} \right\rangle + \epsilon_ {t} \tag {1}
$$
for some unknown  $\pmb{\theta}^{(1)}\in \mathbb{R}^d$ , where  $\epsilon_{t}$  is i.i.d. zero-mean sub-Gaussian random noise with (known) variance  $\sigma^2$ . Note that  $r_t(1)$  is observed only when the principal assigns action  $a_{t} = 1$ , and not when  $a_{t} = 0$ . Following Helmbold et al. [32], we refer to such feedback as apple tasting (or one-sided) feedback. Mapping to our lending example, the reward a bank receives for rejecting a particular loan applicant is the same across all applicants, whereas their reward for a positive decision could be anywhere between a large, negative reward (e.g., if a loan is never repaid) to a large, positive reward (e.g., if the loan is repaid on time, with interest).
The most natural measure of performance in our setting is that of Stackelberg regret, which compares the principal's reward over  $T$  rounds with that of the optimal policy given that agents strategize.
Definition 2.2 (Stackelberg regret). The Stackelberg regret of a sequence of policies  $\{\pi_t\}_{t\in [T]}$  on agents  $\{\mathbf{x}_t\}_{t\in [T]}$  is
$$
\operatorname {R e g} _ {\mathbf {S t a c k e l}} (T) := \sum_ {t \in [ T ]} r _ {t} (\tilde {\pi} ^ {*} (\tilde {\mathbf {x}} _ {t})) - \sum_ {t \in [ T ]} r _ {t} (\pi_ {t} (\mathbf {x} _ {t} ^ {\prime}))
$$
where  $\tilde{\mathbf{x}}_t$  is the best-response from agent  $t$  to policy  $\tilde{\pi}^*$  and  $\tilde{\pi}^*$  is the optimal-in-hindsight policy, given that agents best-respond according to Definition 2.1.
A stronger measure of performance is that of strategic regret, which compares the principal's reward over  $T$  rounds with that of the optimal policy had agents reported their contexts truthfully.
Definition 2.3 (Strategic regret). The strategic regret of a sequence of policies  $\{\pi_t\}_{t\in [T]}$  on agents  $\{\mathbf{x}_t\}_{t\in [T]}$  is
$$
\operatorname {R e g} _ {\mathbf {s t r a t}} (T) := \sum_ {t \in [ T ]} r _ {t} \left(\pi^ {*} (\mathbf {x} _ {t})\right) - \sum_ {t \in [ T ]} r _ {t} \left(\pi_ {t} \left(\mathbf {x} _ {t} ^ {\prime}\right)\right)
$$
where  $\pi^{*}(\mathbf{x}_{t}) = 1$  if  $\langle \pmb{\theta}^{(1)},\mathbf{x}_t\rangle \geq r_0$  and  $\pi^{*}(\mathbf{x}_{t}) = 0$  otherwise.
2Our results readily extend to the setting in which the agent's effort constraint takes the form of an ellipse rather than a sphere. Under this setting, the agent effort budget constraint in Definition 2.1 would be  $\| A^{1 / 2}(\mathbf{x}' - \mathbf{x}_t)\| _2\leq \delta$ , where  $A\in \mathbb{R}^{d\times d}$  is some positive definite matrix. If  $A$  is known to the principal, this may just be viewed as a linear change in the feature representation.
Classification under agent incentives with apple tasting feedback
1. Principal publicly commits to a mapping  $\pi_t: \mathcal{X} \to \{1,0\}$ .
2. Agent  $t$  arrives with context  $\mathbf{x}_t\in \mathcal{X}$  (hidden from the principal).
3. Agent  $t$  strategically modifies context from  $\mathbf{x}_t$  to  $\mathbf{x}_t^\prime$  according to Definition 2.1.
4. Principal observes (modified) context  $\mathbf{x}_t^\prime$  and plays action  $a_{t} = \pi_{t}(\mathbf{x}_{t}^{\prime})$
5. Principal observes  $r_t(1) \coloneqq \langle \pmb{\theta}^{(1)}, \mathbf{x}_t \rangle + \epsilon_t$ , if and only if  $a_t = 1$ .
Figure 1: Summary of our model.
Proposition 2.4. Strategic regret is a stronger performance notion compared to Stackelberg regret, i.e.,  $\operatorname{Reg}_{\text{Stackel}}(T) \leq \operatorname{Reg}_{\text{strat}}(T)$ .
Proof. The proof follows from the corresponding regret definitions and the fact that the principal's reward is determined by the original (unmodified) agent contexts.
$$
\begin{array}{l} R _ {\text {S t a c k e l}} (T) := \sum_ {t \in [ T ]} r _ {t} \left(\tilde {\pi} ^ {*} \left(\tilde {\mathbf {x}} _ {t}\right)\right) - \sum_ {t \in [ T ]} r _ {t} \left(\pi_ {t} \left(\mathbf {x} _ {t} ^ {\prime}\right)\right) \\ = \sum_ {t \in [ T ]} r _ {t} \left(\tilde {\pi} ^ {*} \left(\tilde {\mathbf {x}} _ {t}\right)\right) - \sum_ {t \in [ T ]} r _ {t} \left(\pi^ {*} \left(\mathbf {x} _ {t}\right)\right) + \sum_ {t \in [ T ]} r _ {t} \left(\pi^ {*} \left(\mathbf {x} _ {t}\right)\right) - \sum_ {t \in [ T ]} r _ {t} \left(\pi_ {t} \left(\mathbf {x} _ {t} ^ {\prime}\right)\right) \\ \leq 0 + R _ {\text {s t r a t}} (T) \\ \end{array}
$$
where the last line follows from the fact that the principal's reward from the optimal policy when the agent strategizes is at most their optimal reward when agents do not strategize.
Because of Proposition 2.4, we focus on strategic regret, and use the shorthand  $\mathrm{Reg}_{\mathrm{strat}}(T) = \mathrm{Reg}(T)$  for the remainder of the paper. Strategic regret is a strong notion of optimality, as we are comparing the principal's performance with that of the optimal policy for an easier setting, in which agents do not strategize. Moreover, the apple tasting feedback introduces additional challenges which require new algorithmic ideas to solve, since the principal needs to assign actions to both (1) learn about  $\theta^{(1)}$  (which can only be done when action 1 is assigned) and (2) maximize rewards in order to achieve sublinear strategic regret. See Figure 1 for a summary of the setting we consider.
We conclude this section by pointing out that our results also apply to the more challenging setting of bandit feedback, in which  $r_t(1)$  is defined as in Equation (1),  $r_t(0) \coloneqq \langle \pmb{\theta}^{(0)}, \mathbf{x}_t \rangle + \epsilon_t$  and only  $r_t(a_t)$  is observed at each time-step. We choose to highlight our results for apple tasting feedback since this is the type of feedback received by the principal in our motivating examples. Finally, we note that  $\widetilde{\mathcal{O}}(\cdot)$  hides polylogarithmic factors, and that all proofs can be found in the Appendix.
3 Strategic classification with apple tasting feedback
In this section, we present our main results: provable guarantees for online classification of strategic agents under apple tasting feedback. Our results rely on the following assumption.
Assumption 3.1 (Bounded density ratio). Let  $f_{U^d} : \mathcal{X} \to \mathbb{R}_{\geq 0}$  denote the density function of the uniform distribution over the  $d$ -dimensional unit sphere. We assume that agent contexts  $\{\mathbf{x}_t\}_{t \in [T]}$  are drawn i.i.d. from a distribution over the  $d$ -dimensional unit sphere with density function  $f : \mathcal{X} \to \mathbb{R}_{\geq 0}$  such that  $\frac{f(\mathbf{x})}{f_{U^d}(\mathbf{x})} \geq c_0 > 0, \forall \mathbf{x} \in \mathcal{X}$ .<sup>3</sup>
Assumption 3.1 is a condition on the initial agent contexts  $\{\mathbf{x}_t\}_{t\in [T]}$ , before they are strategically modified. Indeed, one would expect the distribution over modified agent contexts to be highly discontinuous in a way that depends on the sequence of policies deployed by the principal. Furthermore, none of our algorithms need to know the value of  $c_{0}$ . As we will see in the sequel, this assumption allows us to handle apple tasting feedback by relying on the inherent diversity in the agent population
3Our restriction to the unit sphere is without loss of generality. All of our results and analysis extend readily to the setting where contexts are drawn from a distribution over the  $d$ -dimensional sphere with radius  $R > 0$ .