SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Toyota Technological Institute Chicago
Chicago, 60637
Avrim Blum
Omar Montasser
omar@ttic.edu
We study the fundamental mistake bound and sample complexity in the strategic classification, where agents can strategically manipulate their feature vector up to an extent in order to be predicted as positive. For example, given a classifier determining college admission, student candidates may try to take easier classes to improve their GPA, retake SAT and change schools in an effort to fool the classifier. Ball manipulations are a widely studied class of manipulations in the literature, where agents can modify their feature vector within a bounded radius ball. Unlike most prior work, our work considers manipulations to be personalized, meaning that agents can have different levels of manipulation abilities (e.g., varying radii for ball manipulations), and unknown to the learner.
We formalize the learning problem in an interaction model where the learner first deploys a classifier and the agent manipulates the feature vector within their manipulation set to game the deployed classifier. We investigate various scenarios in terms of the information available to the learner during the interaction, such as observing the original feature vector before or after deployment, observing the manipulated feature vector, or not seeing either the original or the manipulated feature vector. We begin by providing online mistake bounds and PAC sample complexity in these scenarios for ball manipulations. We also explore non-ball manipulations and show that, even in the simplest scenario where both the original and the manipulated feature vectors are revealed, the mistake bounds and sample complexity are lower bounded by  $\Omega(|\mathcal{H}|)$  when the target function belongs to a known class  $\mathcal{H}$ .
Strategic classification addresses the problem of learning a classifier robust to manipulation and gaming by self-interested agents (Hardt et al., 2016). For example, given a classifier determining loan approval based on credit scores, applicants could open or close credit cards and bank accounts to increase their credit scores. In the case of a college admission classifier, students may try to take easier classes to improve their GPA, retake the SAT or change schools in an effort to be admitted. In both cases, such manipulations do not change their true qualifications. Recently, a collection of papers has studied strategic classification in both the online setting where examples are chosen by an adversary in a sequential manner (Dong et al., 2018; Chen et al., 2020; Ahmadi et al., 2021, 2023), and the
distributional setting where the examples are drawn from an underlying data distribution (Hardt et al., 2016; Zhang and Conitzer, 2021; Sundaram et al., 2021; Lechner and Urner, 2022). Most existing works assume that manipulation ability is uniform across all agents or is known to the learner. However, in reality, this may not always be the case. For instance, low-income students may have a lower ability to manipulate the system compared to their wealthier peers due to factors such as the high costs of retaking the SAT or enrolling in additional classes, as well as facing more barriers to accessing information about college (Milli et al., 2019) and it is impossible for the learner to know the highest achievable GPA or the maximum number of times a student may retake the SAT due to external factors such as socio-economic background and personal circumstances.
We characterize the manipulation of an agent by a set of alternative feature vectors that she can modify her original feature vector to, which we refer to as the manipulation set. Ball manipulations are a widely studied class of manipulations in the literature, where agents can modify their feature vector within a bounded radius ball. For example, Dong et al. (2018); Chen et al. (2020); Sundaram et al. (2021) studied ball manipulations with distance function being some norm and Zhang and Conitzer (2021); Lechner and Urner (2022); Ahmadi et al. (2023) studied a manipulation graph setting, which can be viewed as ball manipulation w.r.t. the graph distance on a predefined known graph.
In the online learning setting, the strategic agents come sequentially and try to game the current classifier. Following previous work, we model the learning process as a repeated Stackelberg game over  $T$  time steps. In round  $t$ , the learner proposes a classifier  $f_{t}$  and then the agent, with a manipulation set (unknown to the learner), manipulates her feature in an effort to receive positive prediction from  $f_{t}$ . There are several settings based on what and when the information is revealed about the original feature vector and the manipulated feature vector in the game. The simplest setting for the learner is observing the original feature vector before choosing  $f_{t}$  and the manipulated vector after. In a slightly harder setting, the learner observes both the original and manipulated vectors after selecting  $f_{t}$ . An even harder setting involves observing only the manipulated feature vector after selecting  $f_{t}$ . The hardest and Least informative scenario occurs when neither the original nor the manipulated feature vectors are observed.
In the distributional setting, the agents are sampled from an underlying data distribution. Previous work assumes that the learner has full knowledge of the original feature vector and the manipulation set, and then views learning as a one-shot game and solves it by computing the Stackelberg equilibria of it. However, when manipulations are personalized and unknown, we cannot compute an equilibrium and study learning as a one-shot game. In this work, we extend the iterative online interaction model from the online setting to the distributional setting, where the sequence of agents is sampled i.i.d. from the data distribution. After repeated learning for  $T$  (which is equal to the sample size) rounds, the learner has to output a strategy-robust predictor for future use.
In both online and distributional settings, examples are viewed through the lens of the current predictor and the learner does not have the ability to inquire about the strategies the previous examples would have adopted under a different predictor.
Related work Our work is primarily related to strategic classification in online and distributional settings. Strategic classification was first studied in a distributional model by Hardt et al. (2016) and subsequently by Dong et al. (2018) in an online model. Hardt et al. (2016) assumed that agents manipulate by best response with respect to a uniform cost function known to the learner. Building on the framework of (Hardt et al., 2016), Lechner and Urner (2022); Sundaram et al. (2021); Zhang and Conitzer (2021); Hu et al. (2019); Milli et al. (2019) studied the distributional learning problem, and all of them assumed that the manipulations are predefined and known to the learner, either by a cost function or a predefined manipulation graph. For online learning, Dong et al. (2018) considered a similar manipulation setting as in this work, where manipulations are personalized and unknown. However, they studied linear classification with ball manipulations in the online setting and focused on finding appropriate conditions of the cost function to achieve sub-linear Stackelberg regret. Chen et al. (2020) also studied Stackelberg regret in linear classification with uniform ball manipulations. Ahmadi et al. (2021) studied the mistake bound under uniform (possibly unknown) ball manipulations, and Ahmadi et al. (2023) studied regret under a pre-defined and known manipulation. The most relevant work is a recent concurrent study by Lechner et al. (2023), which also explores strategic classification involving unknown personalized manipulations but with a different loss function. In their work, a predictor incurs a loss of 0 if and only if the agent refrains from manipulation and the predictor correctly predicts at the unmanipulated feature vector. In our work, the predictor's loss is 0
if it correctly predicts at the manipulated feature, even when the agent manipulates. As a result, their loss function serves as an upper bound of our loss function.
There has been a lot of research on various other issues and models in strategic classification. Beyond sample complexity, Hu et al. (2019); Milli et al. (2019) focused on other social objectives, such as social burden and fairness. Recent works also explored different models of agent behavior, including proactive agents Zrnic et al. (2021), non-myopic agents (Haghtalab et al., 2022) and noisy agents (Jagadeesan et al., 2021). Ahmadi et al. (2023) considers two agent models of randomized learners: a randomized algorithm model where the agents respond to the realization, and a fractional classifier model where agents respond to the expectation, and our model corresponds to the randomized algorithm model. Additionally, there is also a line of research on agents interested in improving their qualifications instead of gaming (Kleinberg and Raghavan, 2020; Haghtalab et al., 2020; Ahmadi et al., 2022). Strategic interactions in the regression setting have also been studied (e.g., Bechavod et al. (2021)).
Beyond strategic classification, there is a more general research area of learning using data from strategic sources, such as a single data generation player who manipulates the data distribution (Brückner and Scheffer, 2011; Dalvi et al., 2004). Adversarial perturbations can be viewed as another type of strategic source (Montasser et al., 2019).
Strategic classification Throughout this work, we consider the binary classification task. Let  $\mathcal{X}$  denote the feature vector space,  $\mathcal{Y} = \{+1, -1\}$  denote the label space, and  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  denote the hypothesis class. In the strategic setting, instead of an example being a pair  $(x, y)$ , an example, or agent, is a triple  $(x, u, y)$  where  $x \in \mathcal{X}$  is the original feature vector,  $y \in \mathcal{Y}$  is the label, and  $u \subseteq \mathcal{X}$  is the manipulation set, which is a set of feature vectors that the agent can modify their original feature vector  $x$  to. In particular, given a hypothesis  $h \in \mathcal{Y}^{\mathcal{X}}$ , the agent will try to manipulate her feature vector  $x$  to another feature vector  $x'$  within  $u$  in order to receive a positive prediction from  $h$ . The manipulation set  $u$  is unknown to the learner. In this work, we will be considering several settings based on what the information is revealed to the learner, including both the original/manipulated feature vectors, the manipulated feature vector only, or neither, and when the information is revealed.
More formally, for agent  $(x,u,y)$ , given a predictor  $h$ , if  $h(x) = -1$  and her manipulation set overlaps the positive region by  $h$ , i.e.,  $u\cap \mathcal{X}_{h, + }\neq \emptyset$  with  $\mathcal{X}_{h, + }:= \{x\in \mathcal{X}|h(x) = +1\}$ , the agent will manipulate  $x$  to  $\Delta (x,h,u)\in u\cap \mathcal{X}_{h, + }^{1}$  to receive positive prediction by  $h$ . Otherwise, the agent will do nothing and maintain her feature vector at  $x$ , i.e.,  $\Delta (x,h,u) = x$ . We call  $\Delta (x,h,u)$  the manipulated feature vector of agent  $(x,u,y)$  under predictor  $h$ .
A general and fundamental type of manipulations is ball manipulations, where agents can manipulate their feature within a ball of personalized radius. More specifically, given a metric  $d$  over  $\mathcal{X}$ , the manipulation set is a ball  $\mathcal{B}(x;r) = \{x'|d(x,x')\leq r\}$  centered at  $x$  with radius  $r$  for some  $r\in \mathbb{R}_{\geq 0}$ . Note that we allow different agents to have different manipulation power and the radius can vary over agents. Let  $Q$  denote the set of allowed pairs  $(x,u)$ , which we refer to as the feature-manipulation set space. For ball manipulations, we have  $\mathcal{Q} = \{(x,\mathcal{B}(x;r))|x\in \mathcal{X},r\in \mathbb{R}_{\geq 0}\}$  for some known metric  $d$  over  $\mathcal{X}$ . In the context of ball manipulations, we use  $(x,r,y)$  to represent  $(x,\mathcal{B}(x;r),y)$  and  $\Delta (x,h,r)$  to represent  $\Delta (x,h,\mathcal{B}(x;r))$  for notation simplicity.
For any hypothesis  $h$ , let the strategic loss  $\ell^{\mathrm{str}}(h, (x, u, y))$  of  $h$  be defined as the loss at the manipulated feature, i.e.,  $\ell^{\mathrm{str}}(h, (x, u, y)) := \mathbb{1}(h(\Delta(x, h, u)) \neq y)$ . According to our definition of  $\Delta(\cdot)$ , we can write down the strategic loss explicitly as
$$
\ell^ {\mathrm {s t r}} (h, (x, u, y)) = \left\{ \begin{array}{l l} 1 & \text {i f} y = - 1, h (x) = + 1 \\ 1 & \text {i f} y = - 1, h (x) = - 1 \text {a n d} u \cap \mathcal {X} _ {h, +} \neq \emptyset , \\ 1 & \text {i f} y = + 1, h (x) = - 1 \text {a n d} u \cap \mathcal {X} _ {h, +} = \emptyset , \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {1}
$$
For any randomized predictor  $p$  (a distribution over hypotheses), the strategic behavior depends on the realization of the predictor and the strategic loss of  $p$  is  $\ell^{\mathrm{str}}(p,(x,u,y))\coloneqq \mathbb{E}_{h\sim p}[\ell^{\mathrm{str}}(h,(x,u,y))]$ .
For ball manipulations, agents break ties by selecting the closest vector. When there are multiple closest vectors, agents break ties arbitrarily. For non-ball manipulations, agents break ties in any fixed way.
Online learning We consider the task of sequential classification where the learner aims to classify a sequence of agents  $(x_{1},u_{1},y_{1}),(x_{2},u_{2},y_{2}),\ldots ,(x_{T},u_{T},y_{T})\in \mathcal{Q}\times \mathcal{V}$  that arrives in an online manner. At each round, the learner feeds a predictor to the environment and then observes his prediction  $\widehat{y}_t$  , the true label  $y_{t}$  and possibly along with some additional information about the original/manipulated feature vectors. We say the learner makes a mistake at round  $t$  if  $\widehat{y}_t\neq y_t$  and the learner's goal is to minimize the number of mistakes on the sequence. The interaction protocol (which repeats for  $t = 1,\dots ,T)$  is described in the following.
Protocol 1 Learner-Agent Interaction at round  $t$
1: The environment picks an agent  $(x_{t},u_{t},y_{t})$  and reveals some context  $C(x_{t})$ . In the online setting, the agent is chosen adversarially, while in the distributional setting, the agent is sampled i.i.d.
2: The learner  $\mathcal{A}$  observes  $C(x_{t})$  and picks a hypothesis  $f_{t}\in \mathcal{V}^{\mathcal{X}}$
3: The learner  $\mathcal{A}$  observes the true label  $y_{t}$ , the prediction  $\widehat{y}_t = f_t(\Delta_t)$ , and some feedback  $F(x_{t},\Delta_{t})$ , where  $\Delta_t = \Delta (x_t,f_t,u_t)$  is the manipulated feature vector.
The context function  $C(\cdot)$  and feedback function  $F(\cdot)$  reveals information about the original feature vector  $x_{t}$  and the manipulated feature vector  $\Delta_t$ .  $C(\cdot)$  reveals the information before the learner picks  $f_{t}$  while  $F(\cdot)$  does after. We study several different settings based on what and when information is revealed.
- The simplest setting for the learner is observing the original feature vector  $x_{t}$  before choosing  $f_{t}$  and the manipulated vector  $\Delta_{t}$  after. Consider a teacher giving students a writing assignment or take-home exam. The teacher might have a good knowledge of the students' abilities (which correspond to the original feature vector  $x_{t}$ ) based on their performance in class, but the grade has to be based on how well they do the assignment. The students might manipulate by using the help of ChatGPT / Google / WolframAlpha / their parents, etc. The teacher wants to create an assignment that will work well even in the presence of these manipulation tools. In addition, If we think of each example as representing a subpopulation (e.g., an organization is thinking of offering loans to a certain group), then there might be known statistics about that population, even though the individual classification (loan) decisions have to be made based on responses to the classifier. This setting corresponds to  $C(x_{t}) = x_{t}$  and  $F(x_{t},\Delta_{t}) = \Delta_{t}$ . We denote a setting by their values of  $C,F$  and thus, we denote this setting by  $(x,\Delta)$ .
- In a slightly harder setting, the learner observes both the original and manipulated vectors after selecting  $f_{t}$  and thus,  $f_{t}$  cannot depend on the original feature vector in this case. For example, if a high-school student takes the SAT test multiple times, most colleges promise to only consider the highest one (or even to "superscore" the test by considering the highest score separately in each section) but they do require the student to submit all of them. Then  $C(x_{t}) = \bot$  and  $F(x_{t},\Delta_{t}) = (x_{t},\Delta_{t})$ , where  $\bot$  is a token for "no information", and this setting is denoted by  $(\bot ,(x,\Delta))$ .
- An even harder setting involves observing only the manipulated feature vector after selecting  $f_{t}$  (which can only be revealed after  $f_{t}$  since  $\Delta_{t}$  depends on  $f_{t}$ ). Then  $C(x_{t}) = \bot$  and  $F(x_{t},\Delta_{t}) = \Delta_{t}$  and this setting is denoted by  $(\bot ,\Delta)$ .
- The hardest and Least Informative scenario occurs when neither the original nor the manipulated feature vectors are observed. Then  $C(x_{t}) = \bot$  and  $F(x_{t},\Delta_{t}) = \bot$  and it is denoted by  $(\bot ,\bot)$ .
Throughout this work, we focus on the realizable setting, where there exists a perfect classifier in  $\mathcal{H}$  that never makes any mistake at the sequence of strategic agents. More specifically, there exists a hypothesis  $h^* \in \mathcal{H}$  such that for any  $t \in [T]$ , we have  $y_t = h^*(\Delta(x_t, h^*, u_t))^2$ . Then we define the mistake bound as follows.
Definition 1. For any choice of  $(C,F)$ , let  $\mathcal{A}$  be an online learning algorithm under Protocol 1 in the setting of  $(C,F)$ . Given any realizable sequence  $S = ((x_{1},u_{1},h^{*}(\Delta (x_{1},h^{*},u_{1}))),\ldots ,(x_{T},u_{T},h^{*}(\Delta (x_{T},h^{*},u_{T})))\in (\mathcal{Q}\times \mathcal{Y})^{T}$ , where  $T$  is any integer and  $h^*\in \mathcal{H}$ , let  $\mathcal{M}_{\mathcal{A}}(S)$  be the number of mistakes  $\mathcal{A}$  makes on the sequence  $S$ . The mistake bound of  $(\mathcal{H},\mathcal{Q})$ , denoted  $\mathrm{MB}_{C,F}$ , is the smallest number  $B\in \mathbb{N}$  such that there exists an algorithm  $\mathcal{A}$  such that  $\mathcal{M}_{\mathcal{A}}(S)\leq B$  over all realizable sequences  $S$  of the above form.
It is possible that there is no hypothesis  $\overline{h} \in \mathcal{Y}^{\mathcal{X}}$  s.t.  $y_{t} = \overline{h}(x_{t})$  for all  $t \in [T]$ .
According to the rank of difficulty of the four settings with different choices of  $(C,F)$ , the mistake bounds are ranked in the order of  $\mathrm{MB}_{x,\Delta} \leq \mathrm{MB}_{\perp,(x,\Delta)} \leq \mathrm{MB}_{\perp,\Delta} \leq \mathrm{MB}_{\perp,\perp}$ .
PAC learning In the distributional setting, the agents are sampled from an underlying distribution  $\mathcal{D}$  over  $\mathcal{Q} \times \mathcal{Y}$ . The learner's goal is to find a hypothesis  $h$  with low population loss  $\mathcal{L}_{\mathcal{D}}^{\mathrm{str}}(h) := \mathbb{E}_{(x,u,y) \sim \mathcal{D}}[\ell^{\mathrm{str}}(h,(x,u,y))]$ . One may think of running empirical risk minimizer (ERM) over samples drawn from the underlying data distribution, i.e., returning  $\arg \min_{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^{m} \ell^{\mathrm{str}}(h,(x_i,u_i,y_i))$ , where  $(x_1,u_1,y_1),\ldots,(x_m,u_m,y_m)$  are i.i.d. sampled from  $\mathcal{D}$ . However, ERM is unimplementable because the manipulation sets  $u_i$ 's are never revealed to the algorithm, and only the partial feedback in response to the implemented classifier is provided. In particular, in this work we consider using the same interaction protocol as in the online setting, i.e., Protocol 1, with agents  $(x_t,u_t,y_t)$  i.i.d. sampled from the data distribution  $\mathcal{D}$ . After  $T$  rounds of interaction (i.e.,  $T$  i.i.d. agents), the learner has to output a predictor  $f_{\mathrm{out}}$  for future use.
Again, we focus on the realizable setting, where the sequence of sampled agents (with manipulation) can be perfectly classified by a target function in  $\mathcal{H}$ . Alternatively, there exists a classifier with zero population loss, i.e., there exists a hypothesis  $h^* \in \mathcal{H}$  such that  $\mathcal{L}_{\mathcal{D}}^{\mathrm{str}}(h^*) = 0$ . Then we formalize the notion of PAC sample complexity under strategic behavior as follows.
Definition 2. For any choice of  $(C,F)$ , let  $\mathcal{A}$  be a learning algorithm that interacts with agents using Protocol 1 in the setting of  $(C,F)$  and outputs a predictor  $f_{\mathrm{out}}$  in the end. For any  $\varepsilon, \delta \in (0,1)$ , the sample complexity of realizable  $(\varepsilon, \delta)$ -PAC learning of  $(\mathcal{H}, \mathcal{Q})$ , denoted  $\mathrm{SC}_{C,F}(\varepsilon, \delta)$ , is defined as the smallest  $m \in \mathbb{N}$  for which there exists a learning algorithm  $\mathcal{A}$  in the above form such that for any distribution  $\mathcal{D}$  over  $\mathcal{Q} \times \mathcal{V}$  where there exists a predictor  $h^* \in \mathcal{H}$  with zero loss,  $\mathcal{L}_{\mathcal{D}}^{str}(h) = 0$ , with probability at least  $1 - \delta$  over  $(x_1, u_1, y_1), \ldots, (x_m, u_m, y_m) \stackrel{i.i.d.}{\sim} \mathcal{D}$ ,  $\mathcal{L}_{\mathcal{D}}^{str}(f_{\mathrm{out}}) \leq \varepsilon$ .
Similar to mistake bounds, the sample complexities are ranked in the same order  $\mathrm{SC}_{x,\Delta} \leq \mathrm{SC}_{\perp,(x,\Delta)} \leq \mathrm{SC}_{\perp,\Delta} \leq \mathrm{SC}_{\perp,\perp}$  according to the rank of difficulty of the four settings.
3 Overview of Results
In classic (non-strategic) online learning, the Halving algorithm achieves a mistake bound of  $\log(|\mathcal{H}|)$  by employing the majority vote and eliminating inconsistent hypotheses at each round. In classic PAC learning, the sample complexity of  $\mathcal{O}\left(\frac{\log(|\mathcal{H}|)}{\varepsilon}\right)$  is achievable via ERM. Both mistake bound and sample complexity exhibit logarithmic dependency on  $|\mathcal{H}|$ . This logarithmic dependency on  $|\mathcal{H}|$  (when there is no further structural assumptions) is tight in both settings, i.e., there exist examples of  $\mathcal{H}$  with mistake bound of  $\Omega(\log(|\mathcal{H}|))$  and with sample complexity of  $\Omega\left(\frac{\log(|\mathcal{H}|)}{\varepsilon}\right)$ . In the setting where manipulation is known beforehand and only  $\Delta_t$  is observed, Ahmadi et al. (2023) proved a lower bound of  $\Omega(|\mathcal{H}|)$  for the mistake bound. Since in the strategic setting we can achieve a linear dependency on  $|\mathcal{H}|$  by trying each hypothesis in  $\mathcal{H}$  one by one and discarding it once it makes a mistake, the question arises:
Can we achieve a logarithmic dependency on  $|\mathcal{H}|$  in strategic classification?
In this work, we show that the dependency on  $|\mathcal{H}|$  varies across different settings and that in some settings mistake bound and PAC sample complexity can exhibit different dependencies on  $|\mathcal{H}|$ . We start by presenting our results for ball manipulations in the four settings.