Title: value: Strategic Classification under Unknown Personalized Manipulation
Authors: value: Han Shao
Avrim Blum
Omar Montasser
Venue: NeurIPS 2023
Keywords: value: strategic classification
mistake bound in online learning
PAC learning

Abstract: value: We study the fundamental mistake bound and sample complexity in the strategic classification, where agents can strategically manipulate their feature vector up to an extent in order to be predicted as positive. For example, given a classifier determining college admission, student candidates may try to take easier classes to improve their GPA, retake SAT and change schools in an effort to fool the classifier. *Ball manipulations* are a widely studied class of manipulations in the literature, where agents can modify their feature vector within a bounded radius ball. Unlike most prior work, our work consider manipulations to be *personalized*, meaning that agents can have different levels of manipulation abilities (e.g., varying radii for ball manipulations), and *unknown* to the learner.

We formalize the learning problem in an interaction model where the learner first deploys a classifier and the agent manipulates the feature vector within their manipulation set to game the deployed classifier. We investigate various scenarios in terms of the information available to the learner during the interaction, such as observing the original feature vector before or after deployment, observing the manipulated feature vector, or not seeing either the original or the manipulated feature vector. We begin by providing online mistake bounds and PAC sample complexity in these scenarios for ball manipulations. We also explore non-ball manipulations and show that, even in the simplest scenario where both the original and the manipulated feature vectors are revealed, the mistake bounds and sample complexity are lower bounded by $\Omega(|\mathcal H|)$ when the target function belongs to a known class $\mathcal H$.


Review discussion:
- The paper is well written, and good intuition is provided for some of the technical theorems.
- The setting of varying and unknown manipulations strength in strategic classification is relevant.
- Both lower bounds and upper bounds based on concrete algorithms are provided
- The motivation for the easier cases is not entirely clear.
For example, in which setting is it realistic for the learner to see the *original* features of the next agent before choosing which hypothesis to deploy?
- This is particularly relevant, as the more interesting results that seem clearly distinct from the previous literatur are on these easier cases.
- Minor comments:
- The Formatting of table 1 could be improved.
In particular, some tildes are intersecting the table lines.
Additionally, the third row with multiple results for slightly different settings could be further split up to avoid confusion.
- If I understand correctly, it is possible for the strategic setting to be realizable for a class of hypotheses, while the non-strategic setting is not. Highlighting this might make some of the proofs (like for theorem 9) easier to parse.
- I guess the tie-breaking procedure is slightly underspecified when there are multiple points at the decision boundary with the same distance from x?
- Do you expect (versions of) the proposed algorithms to still work if different players use different metrics rather than just differently sized balls for the same metric?
- The main limitations that are not discussed in the weaknesses were highlighted and discussed by the authors in the discussion/open problems section.
No ethics review needed.
We thank Reviewer 6Wy9 for their valuable comments. In response to their questions, we provide the following clarifications and explanations.
> The motivation for the easier cases.
For the easiest model (x is observed first):
- Consider a teacher giving students a writing assignment or take-home exam.  The teacher might have a good knowledge of the students' abilities based on their performance in class, but the grade has to be based on how well they do the assignment.  The students might manipulate by using the help of ChatGPT / Google / WolframAlpha / their parents, etc.  The teacher wants to create an assignment that will work well even in the presence of these manipulation tools.
- If we think of each example as representing a subpopulation (e.g., an organization is thinking of offering loans to a certain group) then there might be known statistics about that population, even though the individual classification (loan) decisions have to be made based on responses to the classifier.
For the second-easiest model (x is observed after):
- Here, there are a lot more scenarios that might fit.
For example, if a high-school student takes the SAT test multiple times, most colleges promise to only consider the highest one (or even to "superscore" the test by considering the highest score separately in each section) but they do require the student to submit all of them.
> Do you expect (versions of) the proposed algorithms to still work if different players use different metrics rather than just differently sized balls for the same metric?
Yes, we expect the algorithms to work if the metrics are known to the learner. As our setting is online, the algorithm utilizes the metric at each round to order all hypotheses and reduce the version space, which should still work under different metrics.
Responses to minor comments:
> The Formatting of table 1 could be improved.
We will improve the formatting of the table.
> Is it possible for the strategic setting to be realizable for a class of hypotheses, while the non-strategic setting is not?
Yes, the reviewer is correct on this. We will highlight it in the final version.
> The tie-breaking procedure is slightly underspecified when there are multiple points at the decision boundary with the same distance from x?
We allow tie breaking arbitrarily when more than one points have the same distance from x. We will clarify this in the final version.
Thank you for the response.
If I understand the teacher example correctly, it would entail using a different exam for every student, correct? Actually, it looks like in both examples it might be more accurate to model things in a batch fashion, where a batch of data points is observed and the same classifier has to be deployed for all of them. Do your results extend to settings similar to that batch setting?
Thanks for your comment. In the teacher example, we could think of a computer-taught course where the system can indeed change the test in between students, or we could think of a teacher teaching the same class over multiple years, changing the test each year based on results from previous years.
In the loan approval example, the loan applicants come sequentially.
The batch setting is an interesting question. We believe the similar algorithmic idea of Strategic Halving still applies, i.e., finding a predictor $f_t$ such that a mistake will eliminate as many hypotheses in the version space as possible. One simple possible way of solving the batch setting could be: at each round $t$, sampling an representative agent $x_t$ uniformly at random from the batch, selecting a predictor by running Strategic Halving given $x_t$ as the input, and applying $f_t$ to the whole batch.
The paper explores a meaningful and challenging new direction in strategic classification.  The models and settings are natural and rich, and the results are fairly comprehensive (and in some cases surprising).  Technically, there seem to be several interesting ideas behind the bounds.
I'd be more excited to see results for infinite hypothesis classes and agonostic settings (both of which the authors discuss as future directions).
(also including detailed comments here)
Footnote 1, "agents break ties randomly": do you really mean "arbitrarily"?
Line 112, remark on randomized classifiers: the alternative model where agents don't observe the internal randomness of the classifier makes sense too.  Any reason for choosing this particular model?
Overview of results: while these results are certainly nice and important, I wonder if anything can be said in the case where $|\mathcal{H}| = \infty$.  Can you comment on this (e.g., conjectures, difficulties)?  (Later I see you mention this as a future direction, which sounds fair since the paper already has quite a number of results.)
Line 240: why is "GALLANT" all capitalized?
General technical question: how crucial is the metric space assumption?
In other words, what if the triangle inequality doesn't hold?  I guess at least some of your positive results still hold?
n/a
No ethics review needed.
We thank Reviewer Hsdn for their valuable comments. In response to their questions, we provide the following clarifications and explanations.
> Extension to general hypothesis class and the agnostic case.
We agree with the reviewer that the general hypothesis class and the agnostic case are interesting open questions to explore! As mentioned by the reviewer, we had already included these directions in the "Discussion and Open Problems" section of our submission and we have already obtained numerous results. The primary focus of this work has been on formulating the problem and presenting preliminary findings in this direction, which is a challenging task in itself.
> "Agents break ties randomly": do you really mean "arbitrarily"?
We actually allow breaking ties in any fixed way. For simplicity, we fix this tie breaking way to be random. We will clarify this in the final version.
> Remark on randomized classifiers: the alternative model where agents don't observe the internal randomness of the classifier makes sense too. Any reason for choosing this particular model?
The paper by Ahmadi et al. (2023) actually gives a good discussion of both randomization models in their setting (where manipulation abilities are known up-front via a manipulation graph).  They show that the model you mention (which they call the "fractional model") behaves similarly to the deterministic model.  It also brings up additional complications, such as: do different manipulations have different costs, and if so, do agents still try to maximize their probability of success or do they instead look at expected utility = expected gain - cost?  For these reasons, we decided to focus on the randomized model where agents react to the realization of the randomness.  We can add additional discussion in the final version.
> Comment in the case where $|H|=\infty$.
Thanks for noticing the discussion on this question. As mentioned in the future directions, for infinite class, it is unclear what combinatorial measure can capture the essence of strategic classification.
> Why is "GALLANT" all capitalized?
We will change it to “Gallant”.
> How crucial is the metric space assumption?
In other words, what if the triangle inequality doesn't hold? I guess at least some of your positive results still hold?
The metric space assumption is mainly used to provide an ordering of hypotheses. Take the Strategic Halving algorithm for example. If we replace the distance d with some function d’ such that the function d’ allows us to do binary search as we did in metric space, then our positive result should still hold.
Thank you for your response!  I have no further questions.
1. Establishing mistake bounds and sample complexity is an important topic in strategic classification, this can be challenging especially when agent features before/after manipulation are unknown to the learner.
2. The paper considered many settings comprehensively, from a simpler setting when both original and manipulated features are revealed to the learner, to a more complex setting when neither is known to the learner.
3. Section 3 (Overview of the results) is very helpful
1. The presentation and some statements can be misleading and confusing.
The authors emphasized in the abstract/introduction that one prime difference between the present paper and the prior works is that this paper considers personalized manipulation with non-ball manipulations.
However, the majority of the paper still focuses on conventional settings with ball manipulations. It can be misleading as I expect more results on non-ball manipulations after reading the abstract.
2. The paper needs more literature review and discuss the differences with prior works to show the novelty.
The paper introduced the existing works on regret bound in strategic learning very briefly, e.g., (Ahmadi et al., 2021), (Ahmadi et al., 2021). It seems they also considered mistake bound under uniform, unknown manipulations.
Moreover, there are many other works studying PAC learning for strategic classification and conducting sample complexity analysis, e.g., [1,2,3]. I think a large body of related works is missing and authors should elaborate more on these works and discuss differences with the present paper.
[1] Sundaram, R., Vullikanti, A., Xu, H., & Yao, F. (2021, July). Pac-learning for strategic classification. In International Conference on Machine Learning (pp. 9978-9988). PMLR.
[2] Zhang, H., & Conitzer, V. (2021, May). Incentive-aware PAC learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 6, pp. 5797-5804).
[3] Lechner, T., & Urner, R. (2022, June). Learning losses for strategic classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 7, pp. 7337-7344).
It is not clear to me how the present model captures the personalized manipulations (i.e., different manipulation abilities across agents). Based on my understanding, the authors assume there exists a pre-defined manipulation set $u$ that constrains agents' ability to manipulate. Agents choose to manipulate only if such a manipulation set overlaps the positive region of the predictor, and they would break ties randomly. By personalization, do you mean the manipulation set differs across agents? It seems that the manipulation set $u$ is the same and fixed over time. Should not this manipulation set depends on the predictor? Why break ties randomly? It is very hard for me to interpret this model. It would be helpful if authors can link the model to a real example.
The authors discussed the limitations, but the potential negative societal impact is not discussed.
No ethics review needed.
We thank Reviewer V2P4 for their valuable comments. In response to their questions, we provide the following clarifications and explanations.
>  One prime difference between the present paper and the prior works is that this paper considers personalized manipulation with non-ball manipulations.
No, we did not say that the prime difference lies in non-ball manipulations. In the abstract, we stated that the main difference lies in that the manipulations are personalized and unknown (line 8-10). In the introduction, we began by mentioning that we model the manipulation through a set of feature vectors and then introduced ball manipulations (line 41-47).
We will add a sentence in the abstract to address that the main focus of this paper is on ball manipulations.
> The paper needs more literature review and discuss the differences with prior works to show the novelty.
Due to space limitations, we briefly discussed differences with the prior work and deferred some related work to the appendix. Check our appendix A for more related work.
Although we only provided a brief discussion of prior work, we have indeed addressed the essential differences.
For instance, we pointed out that the manipulations in Ahmadi et al., (2021) are uniform, while the manipulations in [1,2,3] are known. Throughout the paper, we consistently emphasize that the core of our work revolves around non-uniform unknown manipulations.
We will add a more detailed discussion in the final version.
> How does the present model capture the personalized manipulations?
* Does the manipulation set differ across agents?
Yes, personalized manipulation means that the manipulation set differs across agents. Note that the manipulation set varies over time. Recall that each agent is a triple of the original feature vector $x$, the manipulation set $u$ and the label $y$. At each time step $t$, a new agent $(x_t,u_t,y_t)$ comes. In the adversarial online setting, this new agent, including her manipulation set $u_t$ is picked by the environment adversarially. In the distributional setting, the new agent is sampled from a distribution.
* Should not this manipulation set depend on the predictor?
No, the manipulation set does not depend on the predictor but the manipulated feature vector does. Note that the manipulation set $u_t$ is different from the manipulated feature vector $\Delta_t = \Delta(x_t,f_t,u_t)$. The manipulation set $u_t$ is a set of feature vectors that the agent can modify their original feature vector $x_t$ to.
That is to say, the agent can only modify their original feature within $u_t$. For any implemented predictor $f_t$, if $u_t$ overlaps the positive region of the predictor, then the agent may manipulate as such a manipulation can help the agent receive a positive prediction. If $u_t$ does not overlap the positive region of the predictor, then the agent will not manipulate as such manipulation cannot help the agent receive a positive prediction.
Hence, the manipulation set does not depend on the predictor. The manipulated feature vector $\Delta_t = \Delta(x_t,f_t,u_t)$, which is the feature vector the agent modifies to after observing the predictor $f_t$, depends on the predictor.
> Why break ties randomly?
We actually allow breaking ties in any fixed way. For simplicity, we fix this tie breaking way to be random. We will clarify this in the final version.
>  Link the model to a real example.
Consider the example of college admission (line 27-28). Imagine a simplified situation where a college admits students with SAT scores higher than a certain threshold, denoted as C. For a student, obtaining a score of 1500 may be the best possible outcome if they are only allowed to take the SAT test once.
However, if they have the opportunity to retake the test, they can achieve higher scores. The manipulation set for the student comprises all achievable scores through retakes.
Notably, this manipulation ability varies between students of different financial backgrounds; wealthier students can afford to retake the test more frequently than their less affluent counterparts, resulting in different increases in scores that are independent of the predictor, i.e., the threshold C.
However, once the threshold C is disclosed, students will adapt their manipulative behavior accordingly. If they reach the score C, they will no longer retake the test, effectively capping their score at that level.
Conversely, if a student realizes that obtaining a score of C is unattainable, they will refrain from manipulating their scores.
Therefore, the final SAT score becomes the manipulated feature, influenced by the predictor, which is the threshold C.
Thanks for your reply! I am not an expert in learning theory and I admit there are places that I may not fully understand. After looking at other reviewer's comments, I would like to change my score to 5.
The paper offers a "complete story" in the context of an interesting sub-problem in strategic manipulation (where the true label cannot change) in realizable, online settings. I liked the framework the authors present to capture the feasible manipulation region for each agent, which abstracts away the cost function/budget in an elegant manner. I also like that the authors operate under the fact that the learner does not know how agents would manipulate. The division to revealing the original/manipulated features before/after the classification also makes a lot of sense and is a nice way of thinking about manipulation settings (where the true label is unaffected by such changes).
In terms of results/techniques - the idea of leveraging the ball structure of feasible manipulation area to order concepts by distances and allow for bisecting the version space is pretty neat. The results for the setting where both the original and manipulated features are revealed after the predictor is picked are even more interesting (especially the algorithm and sample complexity for improper learning).
The paper is overall very well-written, and seems sound (although I have just looked, and not carefully read the appendix).
I find the realizability assumption rather strong, in the following sense - as it is written in line 139, realizability in this paper is defined with respect to the manipulated features, i.e. there exists a concept h* which is the one giving the "true" labels for all of the manipulated features.
But this, in turn, depends on the manipulations, which are personalized. This degree of circularity makes it a bit hard to think of how can we pick H ahead of time such that h* is present, without potentially having H to be of very large complexity, or otherwise restricting the personalized nature of the manipulations to some degree.
But maybe this is inevitable for such general model of manipulation.
I think it is implicitly assumed that given agent x, the personalized manipulation is fixed (there can't be another agent with the same features, and different manipulation profile). If so, maybe it is worth noting in the main text.
Across the main text, the authors claim that prior work in strategic classification has not entailed personalized manipulation power for agents (e.g. line 375). This is inaccurate - see e.g. Bechavod et al. 2021.
I found it a bit difficult to follow example 1 in the main text.
For example, if H is singletons over [n], how can f_t be all-negative?
Also, could you clarify the definition of the metric - is it that distances from 0 to other nodes are 1, and all other distances are 2?
Regardless, could you provide a more detailed explanation of the dynamics presented in this example?
The authors properly discuss limitations in the main text.
No ethics review needed.
We thank Reviewer XhgR for their valuable comments. We are particularly pleased that the reviewer finds merit in our modeling of personalized manipulation sets, the setting of unknown manipulation, and the algorithmic ideas involving bisecting the version space. In light of their questions and concerns, we offer the following clarifications and explanations.
> Concerns on the realizability assumption.
The essence of our realizability assumption is that there exists a hypothesis h* satisfying the following conditions:
1. All negative points and their manipulation sets lie on the negative side of h*.
2.
As for positive points, they either reside on the positive side of h*, or on the negative side with their corresponding manipulation sets intersecting the positive side of h*.
From the adversarial robustness perspective, note that our realizability assumption is weaker than the realizability assumption in adversarial robustness (see Montasser et al., (2019) for example) which assumes that
1. All negative points and their manipulation sets (referred to as perturbation sets in the context of adversarial robustness) lie on the negative side of h*.
2. All positive points and their manipulation sets lie on the positive side of h*.
From the game theory perspective, the strategic classification problem is a Stackelberg game, where the learner (the leader) selects a predictor and then the agent (the follower) manipulates their feature vector to best respond to the predictor. Our realizability assumption assumes that the value of the Stackelberg equilibrium (the negative strategic loss) is 0.
In the final version, we will include a thorough discussion on the realizability assumption.
>  Is it implicitly assumed that given agent x, the personalized manipulation is fixed?
No, our approach allows different manipulations for the same x. As demonstrated in Example 1 of the paper, for x being 0, the radius can take values of either 0 or 1. We will address this in our Model section in the final version.
>  Previous work on strategic classification with personalized manipulation power, e.g., Bechavod et al. (2021).
Bechavod et al. (2021) actually focused on the regression problem, which inherently differs from the classification problem. We will incorporate the literature of strategic regression in the related work.
> Explanation on Example 1.
* If $H$ is singletons over $[n]$, how can $f_t$ be all-negative?
This is because we are considering a more general case where $f_t$ can be any hypothesis and does not necessarily belong to $H$. As described in line 2 of protocol 1, $f_t\in Y^X$ indicates that $f_t$ can be any hypothesis.
* Is it that distances from 0 to other nodes are 1, and all other distances are 2?
You are correct. Consider a star graph, where 0 is the central node and the others are leaf nodes. Let the distance between every two nodes be the length of the shortest path on this star graph.
Then the distance from 0 to other leaf nodes is 1 and the distance between any two leaf nodes is 2.
* A more detailed explanation of the dynamics presented in this example.
In the realizable setting, suppose the target function $h^*$  is the singleton $21_{\{i^*\}} -1$ for some $i^* \in [n]$.
Then when $f_t\neq h^*$, there are three different cases.
1. If the learner picks $f_t$ = all-negative, then the environment will pick agent $(0, 1, +1)$. Note that no matter what $i^*$ is, the target function $21_{\{i^*\}} -1$ will predict the agent $(0,1)$ by positive since the distance between $0$ and $i^*$ is 1 and the agent can manipulate the original feature 0 to $i^*$. The all-negative predictor will misclassify this agent by negative. The learner will observe the original feature 0, manipulated feature (conditional on all-negative is implemented) 0, true label +1, and the prediction -1, which do not contain any information about $i^*$.
Hence, the learner makes a mistake but cannot learn anything about $i^*$.
2. If the learner picks an $f_t$ which predicts $0$ by positive, the environment will pick agent $(0,0,-1)$. Note that no matter what $i^*$ is, the singleton $21_{\{i^*\}} -1$ will predict the agent $(0,0)$ by negative since the distance between 0 and $i^*$ is 1 and the agent cannot manipulate the original feature $0$ to $i^*$. The implemented classifier $f_t$ will misclassify the agent by positive. The learner will observe the original feature $0$, manipulated feature $0$, true label $-1$, and the prediction $+1$, which do not contain any information about $i^*$.
Hence, the learner makes a mistake but cannot learn anything about $i^*$.
3. If the learner picks an $f_t$ which predicts some $i\in [n]$ that is not equal to $i^*$ by positive, the environment will pick agent $(i,0,-1)$. The target function $21_{\{i^*\}} -1$ will predict the agent $(i,0)$ by negative since the distance between $i$ and $i^*$ is 2 and the agent cannot manipulate the original feature $i$ to $i^*$. The implemented classifier $f_t$ will misclassify this agent by positive. The learner will observe the original feature $i$, manipulated feature $i$, true label $-1$, and the prediction $+1$. The learner learns that $21_{\{i\}} -1$ cannot be the target function and does not know which one of $[n]\setminus i$ is the true $i^*$.
Hence, the learner makes a mistake but is only able to eliminate one hypothesis.
Since $i^*$ can be chosen adversarially, the learner will not be able to pick an $f_t = 21_{\{i^*\}} -1$ (alternatively, identifying $i^*$) before making $\Omega(n)$ mistakes. The formal proof for a more general case can be found in the proof of Theorem 3. Example 1 is utilized to offer intuitive insights into the linear dependency.
I wish to thank the authors for their response.
After reading the response, I tend to leave my original score unchanged.
* The paper is well-written - it is clear and easy to follow. It also includes proof sketches and intuitions behind the results and proofs.
* The paper studies an interesting online learning model and does a comprehensive job of modelling various types of feedback.
* Algorithms 1 and 2 build on classic ideas but are cleverly modified for the problem being studied.
* Theorem 7 about conservative algorithms is very interesting.
* This paper studies the setting of a finite hypothesis class and in the realizable setting. Extensions to a general hypothesis class (with an appropriate combinatorial measure) and the agnostic case would really enhance the value of the paper. (Note - the authors point out these avenues for future work in the conclusion.)
* Is the definition of "conservative algorithms" (in section 4.3) a well-known concept? It certainly makes a lot of sense. I am wondering whether that formalism already exists or not. If not, do you have thoughts on how this could be used elsewhere in online learning for lower bounds?
Limitations: adequately addressed.
Negative societal impact: this is theoretical paper and I don't see any direct negative impacts arising from this work.
No ethics review needed.
We thank Reviewer Ro1X for their valuable comments. In response to their questions, we provide the following clarifications and explanations.
> Extension to general hypothesis class and the agnostic case.
We agree with the reviewer that the general hypothesis class and the agnostic case are interesting open questions to explore! As mentioned by the reviewer, we had already included these directions in the "Discussion and Open Problems" section of our submission.
However, it is essential to emphasize that our primary focus has been on formulating the problem and presenting preliminary findings in this direction, which is a challenging task in itself.
> Is the definition of "conservative algorithms" a well-known concept?
Yes, the concept of 'conservative algorithms' is widely recognized in online learning, referring to algorithms that make updates only in mistake rounds.
The 'conservative' philosophy is widely adopted when designing algorithms, including the well-known Perceptron algorithm.
Besides, the book 'Prediction, Learning, and Games' by Nicolo Cesa-Bianchi and Gábor Lugosi introduces conservative algorithms for linear classification and explains that "The philosophy behind this conservative policy is that there is no reason to change the weight vector if it has performed well in the last time instance."
In standard online learning, it is well-known that if a hypothesis class is learnable with a mistake bound of M, then it is learnable by a conservative algorithm with the same mistake bound of M.
Consequently, establishing a lower bound for conservative algorithms effectively establishes a lower bound for all algorithms.
Thank you for your response!
EDIT: I want to mention just for clarity that my rating and review remain unchanged.

Meta-review:
role: PC
data: comment: value: This is a well-executed paper the provides a nice contribution to the growing literature on strategic classification. The paper proposes a novel formulation of learning under unknown manipulation sets in a sequential setting, for which it provides comprehensive results. Reviewers found the paper to be clear and well-written. Some reviewers would have liked to see an extension to infinite hypothesis class and to an agnostic setting, but understand that establishing the simpler case of finite cases in the realizable setting, albeit being strong assumptions, is reasonable within the current scope.

Decision: Accept (poster)