SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Interaction. We assume that there are two schools,  $Y = \{A, B\}$ , and two types of students: cheap,  $C$ , and expensive,  $E$ . Moreover, each school has the capacity to accept only one student. Any interaction between a school and a student leads to an outcome between zero and one (i.e.,  $\mathcal{O}_y(x) = [0,1] \forall y \in Y, x \in \{C, E\}$ ). This outcome matches its value:  $v_y^o(x) = o$ . However, schools have different costs for a given outcome depending on the type of student. More precisely, cheap students require negligible specialized assistance, thus having a zero cost for any outcome, i.e.  $c_y^o(C) = 0$ , while expensive students require a cost of three quarters of the value of the chosen outcome, i.e.  $c_y^o(E) = 3/4 \cdot o$ . Thus, the resulting utility for a school  $y \in Y$  is given by  $u_y^o(C) = o$  and  $u_y^o(E) = o/4$ . Note that this always leads to non-negative utilities. To account for cases when returning agents incur large integration costs from accepting any match (e.g., for refugee assignment or the allocation of under-performing students [27, 5]) we also analyze the scenario of negative utilities. We do so by offsetting the cost by a constant  $q > 1$ , thus giving  $u_y^o(C) = o - q$  and  $u_y^o(E) = o/4 - q$ . Unless otherwise specified, we consider the non-negative model for utilities (i.e., without the offsets).
**Preferences.** For simplicity, we assume students have a uniform random chance of preferring each school, and predictions are based on the outcome of the most recent interaction of the same-type student at the school. More precisely, a student  $x$  will have a  $1/2$  chance of having the preference profile  $A \succ_x B$  and a  $1/2$  chance of having the preference profile  $B \succ_x A$ . For the same student  $x$ , the prediction algorithm gives to each school  $y$  a weight  $w_x^h(y)$  equal to the last outcome of a same-type student at school  $y$  based on the history of interactions<sup>4</sup>. For example, a student of type  $E$  will start with a  $50\%$  chance of thinking that  $A \succ_E B$ ; if, according to the history, the most recent student of type  $E$  who attended school  $A$  had a worse outcome than the most recent student of type  $E$  who attended school  $B$  then an accurate prediction-based hypothesis will recommend  $B \succ_E^h A$ .
Accuracy and trust. The impact of predictions depends on their accuracy and trust, two metrics on which system designers usually try to improve on [20, 9]. To account for this, we model the accuracy via a parameter  $\alpha$ , which captures the probability that the recommendation is the one given by the most recent outcome with a same-type student. In the example above, the hypothesis will generate  $B \succ_{E}^{h} A$  with probability  $\alpha$  and the reversed ranking (i.e.,  $A \succ_{E}^{h} B$ ) with probability  $1 - \alpha$ . Similarly, the trust of students in the recommendations is given by  $\theta$ , which captures the probability that students adopt the recommended ranking as opposed to maintaining their original preference over the alternatives (i.e.,  $\succ_{x}^{\prime}$  is  $\succ_{x}^{h}$  with probability  $\theta$  and  $\succ_{x}$  with probability  $1 - \theta$ ). The accuracy and trust are just extensions to help understand the changes in incentives to attack as the power of our
In the absence of historical interactions between a type of student and a school we assume the outcome was 1. We believe this is the most sensible assumption as prior to introducing a prediction mechanism to inform preferences there is a limited impact of past outcomes on future assignments. As a result, schools would likely optimize the 1-step utility and interact truthfully.
prediction models improves. Since the final goal of predictions is to be accurate and trustworthy, for most of the analysis we will assume both  $\alpha$  and  $\theta$  are 1.
Matching. Matching mechanisms are complex procedures that, based on the preferences of all agents, assign students to schools. Serial dictatorship, Boston, and deferred acceptance are all examples of such mechanisms which were deployed in the school choice setting (see the appendix for the definition of each). While, in theory, schools have the chance of expressing their preferences, in practice, these preferences are often given by a lottery with some priority ordering (e.g. students with siblings at the same school have a higher priority at that school) [1]. Thus, the true preferences of schools usually have a limited impact on the final allocation. For the theoretical analysis, we consider random serial dictatorship (RSD), which (1) picks a random ordering of students and (2) assigns, in turn (according to this ordering), each student to the most preferred school that still has available places. In the appendix, we use simulations to investigate the effects of using alternative mechanisms.
Analysis. To see whether schools have an incentive to interact strategically, we compare the utility of one school,  $y$ , under two scenarios: (a) when both schools interact truthfully (i.e., pick outcome 1 for their interaction with all students) and (b) when  $y$  interacts strategically at some level  $l$  (i.e., picks outcome  $1 - l$  where  $l \in (0,1]$  for its interaction with expensive students) while the other school interacts truthfully. If the expected utility of  $y$  increases when it unilaterally interacts strategically, then  $y$  has an incentive to deviate from interacting truthfully and adopt such a strategy. Using a game-theoretic framework, we say that schools are agents which can choose actions corresponding to adversarial attacks at levels  $l \in \{0,1 / L,2 / L,\dots ,1\} ^5$ : an attack at level  $l$  means that the school always chooses the outcome  $1 - l$  when interacting with expensive students and the outcome 1 with cheap students. Thus,  $l = 0$  represents a truthful interaction. With respect to this set of actions, each school has a best response (i.e., an action that gives it the highest utility) given the actions of others. If both schools play the best response to the other's actions and, thus, none of them can change their level of attack to achieve a higher expected utility, we say the action profile is a Nash Equilibrium.
The results section starts with three subsections, each titled based on its main takeaway. The first two focus on results under the assumption that prediction models have perfect levels of trust and accuracy, while the third subsection will investigate incentives to attack as those levels change. Since proofs are mostly mathematical formalizations of intuitive behavior, we reserve them for the appendix and solely provide the intuition within the main text. To position the results within the broader context, each of these three subsections ends with a paragraph discussing the implications of its results. We also include a final subsection explaining why we opted for a simple model and analysis and how it could be extended.
5.1 Not all systems are interaction-proof
We first consider a scenario where, at each timestep, there is one student of each type (i.e.,  $X^t = \{C, E\}$ ) and schools put sufficient weight on the future expected utility (i.e.,  $\beta$  is sufficiently large). In such a case, Theorem 1 states that schools can benefit from implementing an adversarial interaction attack. The proof (see appendix) compares the expected utilities under truthful and strategic interactions and shows that an attack at any level  $l \neq 0$  produces an increase in expected discounted utility when  $\beta > 2/3$ .
Theorem 1 Assume  $X^t = \{C, E\}$ , schools are sufficiently forward-looking ( $\beta > 2/3$ ), and predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). Then, schools have an incentive to deviate from truthfully interacting and implement adversarial interaction attacks.
It is important to note that this takeaway is heavily based on the composition of the market (i.e., the number and type of students available). For example, Theorem 2 shows that when there are only expensive students, a strategy profile where all schools interact truthfully is a Nash Equilibrium. Similarly, Theorem 3 shows that, even when students induce a negative utility for the schools, as long
Here,  $L$  is an arbitrary large natural number. This makes the action space discrete, thus capturing the often-used assumption that agents do not have infinitesimal control over the outcomes. Instead, there is an atomic unit of change  $(1 / L$  in our case) they can make.
as they are of the same type and sufficiently many to guarantee that each school will be allocated a student, then schools still do not have an incentive to implement adversarial interaction attacks. The full proofs are included in the appendix. They formalize the intuition that, in such scenarios, schools either (a) prefer being allocated a student to remaining unmatched or (b) have the same type of allocated student disregarding their interaction outcomes.
Theorem 2 Assume predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). When there are only expensive students, a strategy profile where both schools interact truthfully is a Nash Equilibrium.
Theorem 3 Assume predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). When all students induce the same negative utility for schools, and there are at least two students, a strategy profile where both schools interact truthfully is a Nash Equilibrium.
Yet, there are also scenarios when forward-looking schools have an incentive to interact strategically with all available students, disregarding their type. One such instance is when there is one student giving schools a negative utility for any choice of action. This example is covered by Theorem 4. The main intuition behind the proof is that the attacking school has the opportunity to avoid being allocated any student, which gives it a better utility than being allocated one. If the weight given to future gains (i.e.,  $\beta$ ) is sufficiently large, then schools prefer to decrease their utility in the current round in order to maximize the chance of having a null utility in subsequent rounds.
Theorem 4 Assume predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). If there is only one student giving schools a negative utility for any outcome and schools are sufficiently forward-looking, then schools have an incentive to deviate from truthfully interacting and implement adversarial interaction attacks.
These toy examples prove that it is not sufficient to analyze the matching mechanism in isolation from the preference-formation process. In particular, Roth [25] proved that RSD is a strategy-proof mechanism for both sides. However, our theorems show that a system containing the same mechanism is not necessarily interaction-proof, thus challenging the desirability of implementing such a mechanism.
5.2 Once a school implements an attack, the other school has an incentive to respond with more severe attacks
The prior analysis showed that, in some setups, schools have an incentive to interact strategically. It remains, however, unclear how a school would best respond to an adversarial attack of the other school. Theorem 5 states the best response of a school is to attack at the smallest level that is still higher than the current level of attack of the other school. The proof (see appendix) formalizes the intuition that schools want an attack that sacrifices their short-term utility the least, while still triggering the prediction mechanism not to recommend them to expensive students.
Theorem 5 Assume predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). When  $X^t = \{C, E\}$  and schools are sufficiently forward-looking ( $\beta > 4/5$ ), the best response of school  $A$  for a strategy  $B$  of attacking at a level  $l$  is to attack at a level  $\min \{1, l + 1/L\}$ .
This has a few important implications. First, it suggests that schools would likely not attack at high levels directly, thus having a limited impact on the welfare of expensive students. However, little by little, schools would attack at increasingly higher levels until both schools choose an outcome of 0 (i.e., level of attack 1) for expensive students. Corollary 1, which is an immediate consequence of Theorem 5, states that the resulting strategy profile when both schools attack at a maximum level is a Nash Equilibrium. This equilibrium is, however, undesirable for both schools, as it is Pareto dominated by the initial choice of actions; i.e., if both schools interact truthfully, then both of their expected utilities are higher than when the schools attack at the maximum level.
Corollary 1 Assume predictions have perfect trust and accuracy (i.e.,  $\alpha = \theta = 1$ ). When  $X^t = \{C, E\}$  and schools are sufficiently forward-looking, the strategy profile where both schools use adversarial interaction attacks at the highest level (i.e.,  $l = 1$ ) is a Nash Equilibrium. However, this is Pareto dominated by the strategy profile where all schools interact truthfully.
Altogether, these results show the system resulting from introducing prediction-based recommendations to inform preferences within a matching market could encourage returning agents to engage in higher and higher levels of adversarial interaction attacks. This behavior ultimately leads to an equilibrium that is detrimental to all agents. For our example, it lowered the welfare (i.e., the average value of outcomes) of expensive students and reduced the expected discounted utilities of both schools. At a societal level, such a system could thus result in increased inequalities among the student population. Therefore, there is a need for a system with prediction and matching mechanisms that together lead to different interaction incentives for returning agents. As illustrated by the next two subsections, two potential solutions are to lower the impact of outcome-based predictions or use matching mechanisms that let schools express their true preferences.
5.3 Increases in trust in and accuracy of recommendations lead to higher discounted expected utilities under adversarial interaction attacks
Finally, we acknowledge the general trend of increasing the accuracy of and trust in prediction [20, 9]. Before incorporating prediction mechanisms, prior outcomes have a limited impact on preferences (e.g., from anecdotal experiences of isolated past students, or advice from counselors [27]). When predictions are made algorithmically, there is a higher potential for personalizing them; however, due to the high level of individuality, algorithms struggle to produce accurate predictions. This affects both the trust of students in recommendations and leads to a challenge for designers to improve accuracy. Theorem 6 shows that improving these parameters affects the impact of adversarial interaction attacks. The proof is based on the following observation: the expected gain in utility from using a strategic interaction is strictly increasing with both trust and accuracy.
Theorem 6 When  $X^t = \{C, E\}$ , the expected utility of a school using an adversarial interaction attack is a continuous function that increases monotonously in both accuracy ( $\alpha$ ) and trust ( $\theta$ ).
Naturally, when predictions are disregarded by students (i.e.,  $\theta = 0$ ) or are random (i.e.,  $\alpha = 0.5$ ) it is best for schools to interact truthfully – see Theorem 7. When increasing the accuracy and trust, the utility under attacks increases (by Theorem 6) until adversarial attacks constitute a best response (as shown by Theorem 1). This means that, depending on the specifics of the system, such as the parameter  $\beta$ , for any given level of trust, there exists some maximum value of  $\alpha$  where truthful interaction is still the optimal strategy. The proof of this statement can be formalized using the intermediate value theorem and the three aforementioned results.
Theorem 7 When  $X^t = \{C, E\}$ , the truthful strategy profile is a Nash Equilibrium under predictions with  $50\%$  accuracy ( $\alpha = 0.5$ ) or null levels of trust ( $\theta = 0$ ).
In short, the above analysis shows that, even if adversarial interaction attacks do not currently increase the expected utility, they might if we introduce prediction mechanisms or improve their trust and accuracy. This underlines the importance of re-evaluating incentives when considering such changes. Moreover, it shows that limiting the impact of outcome-based predictions could, in turn, limit the gains of returning agents from interacting strategically.
5.4 Discussion
Our paper (1) created an economic model to allow for a general definition of adversarial interaction attacks and (2) simplified the setting to prove these attacks could indeed be attractive for returning agents. We underline that our goal was to illustrate the emerging feedback loop and show the importance of considering it when prediction algorithms inform preferences. Thus, we favored a simple model together with a basic theoretical analysis. As such, the current analysis can be extended in various ways, such as by using more realistic prediction and matching mechanisms (MM), accounting for more diverse agents, defining and analyzing incentives for undetectable interaction attacks, and extending the space of strategies.
Perhaps the main question for future work is whether adversarial interaction attacks remain attractive in more realistic settings. As a first step to address this question, we extended the simple setting presented above to a more realistic agent-based model. In particular, this model uses k-Nearest Neighbors [14, 4] to predict future outcomes, compares multiple MM (including Deferred Acceptance and Boston), and increases the number of agents. The simulation results both confirm the takeaways
from the theoretical analysis and provide some additional insight. First, they show that different MM are more susceptible to adversarial interaction attacks depending on the distribution of students with respect to their attributes. Second, they suggest that allowing schools to express their true preferences over students (instead of using lotteries) can, in some cases, eliminate the incentives of using adversarial attacks. The exception is when utilities are negative and there are fewer students than places at schools. Third, it illustrates the inequality between students: as suggested by the theoretical results, the welfare of students decreases when schools interact strategically, and this decrease is mostly borne by the expensive students. We provide details on the agent-based model and visualizations for the simulation results within the appendix; to ease reproducibility, the code is available on GitHub<sup>6</sup>
Predictive models are increasingly used to inform preference-formation in matching markets (MM). However, the robustness of the prediction mechanism under adversarial attacks and of MMs under strategic behavior are usually investigated separately. In the present work, we extend existing models by including the interacting stage in which agents, matched by the MM, interact with each other, thus generating new training data. Doing so makes the feedback loop between the prediction model and the MM explicit and uncovers a new type of strategic behavior: the agents that return to the market in subsequent rounds can deviate from the most profitable interactions in the current round in order to attack the predictions and matchings of future rounds.
In a simplified setting, we prove that returning agents could indeed have an incentive to interact strategically. Doing so is more effective as predictions get better in terms of accuracy (and trust). In alignment with previous work on recommendations [20], this suggests we should look beyond accuracy when designing prediction mechanisms for such systems. Moreover, we find that, when returning agents choose to adopt this attack, it has perverse consequences for the utility of all agents. In particular, it both lowers the overall utility and increases inequality among non-returning agents (students). These issues reflect how strategic behaviors of social institutions (schools) cause social inequalities independent of individual potential. Altogether, this work indicates that both aspects of the matching and prediction mechanisms are key in developing robust systems and sets the framework for a dialog between the ML and MM communities.
The authors would like to thank Vlad Gavrila, Atri Rudra, Nicolò Pagan, and the anonymous reviewers for their valuable feedback. KJ and YD gratefully acknowledge the support by the NSF award IIS1939579, with partial support from Amazon.
[1] Atila Abdulkadiroglu, Parag A Pathak, Alvin E Roth, and Tayfun Sonmez. 2006. Changing the Boston school choice mechanism: Strategy-proofness as equal access. *Mimeographed*. Harvard University (2006).
[2] Atila Abdulkadiroğlu and Tayfun Sonmez. 2003. School choice: A mechanism design approach. American economic review 93, 3 (2003), 729-747.
[3] Avidit Acharya, Kirk Bansak, and Jens Hainmueller. 2019. Combining Outcome-Based and Preference-Based Matching: The g-Constrained Priority Mechanism. arXiv preprint arXiv:1902.07355 (2019).
[4] Raheela Asif, Agathe Merceron, and Mahmood K Pathan. 2014. Predicting student academic performance at degree level: a case study. International Journal of Intelligent Systems and Applications 7, 1 (2014), 49.
[5] Kirk Bansak, Jeremy Ferwerda, Jens Hainmueller, Andrea Dillon, Dominik Hangartner, Duncan Lawrence, and Jeremy Weinstein. 2018. Improving refugee integration through data-driven algorithmic assignment. Science 359, 6373 (2018), 325-329.
GitHub link: https://github.com/StefaniaI/Predictions-MM.
[24] Ioannis Paparrizos, B Barla Cambazoglu, and Aristides Gionis. 2011. Machine learned job recommendation. In Proceedings of the fifth ACM Conference on Recommender Systems. 325-328.
[25] Alvin E Roth. 1982. The economics of matching: Stability and incentives. Mathematics of operations research 7, 4 (1982), 617-628.
[26] Alvin E Roth and Marilda Sotomayor. 1992. Two-sided matching. Handbook of game theory with economic applications 1 (1992), 485-541.
[27] David C Wilson, Suzanne Leland, Kenneth Godwin, Andrew Baxter, Ashley Levy, Jamie Smart, Nadia Najjar, and Jayakrishnan Andaparambil. 2009. SmartChoice: An online recommender system to support low-income families in public school choice. AI Magazine 30, 2 (2009), 46-46.