SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Strategic Behavior in Two-sided Matching Markets with Recommendation-enhanced Preference-formation
Stefania Ionescu
Andreasstr. 15, Zürich, 8050
ionescu@ifi.uzh.ch
Yuhao Du
Department of CS and Engineering
Buffalo NY, 14260
yuhaodu@buffalo.edu
Kenneth Joseph
kjoseph@buffalo.edu
Anikó Hannák
Andreasstr. 15, Zurich, 8050
hannak@ifi.uzh.ch
Two-sided matching markets have long existed to pair agents in the absence of regulated exchanges. A common example is school choice, where a matching mechanism uses student and school preferences to assign students to schools. In such settings, forming preferences is both difficult and critical. Prior work has suggested various prediction mechanisms that help agents make decisions about their preferences. Although often deployed together, these matching and prediction mechanisms are almost always analyzed separately. The present work shows that at the intersection of the two lies a previously unexplored type of strategic behavior: agents returning to the market (e.g., schools) can attack future predictions by interacting short-term non-optimally with their matches. Here, we first introduce this type of strategic behavior, which we call an adversarial interaction attack. Next, we construct a formal economic model that captures the feedback loop between prediction mechanisms designed to assist agents and the matching mechanism used to pair them. Finally, in a simplified setting, we prove that returning agents can benefit from using adversarial interaction attacks and gain progressively more as the trust in and accuracy of predictions increases. We also show that this attack increases inequality in the student population.
In two-sided matching markets, agents are partitioned in two disjoint sets (e.g., students and schools) and want to get paired with agents from the other set for future bilateral exchanges (e.g., a student learns at a school and the school provides instruction to the student) [26]. In such markets, agents typically report their preferences over potential matches and a matching mechanism uses those preferences to produce an assignment, i.e. a pairing between agents. Because forming one's preference is hard, agents benefit from external recommendations.
Since predictive models have become increasingly accessible, reliable, and trusted [17], they also became more frequently used to inform preference-formation in matching markets. One such example is school choice. After the introduction of the No Child Left Behind Act, in 2001, students from low-performing schools were allowed to transfer to better-performing schools. However, initially, only a few students took advantage of this opportunity, partly because it was hard for parents to assess which
school would improve their child's performance. In 2008, a content-based recommender system called SmartChoice was deployed for focus group participants; its goal was to help parents with the assessment by identifying the best schools based on predictions on the student's development at that school [27]. Another example is in refugee assignment, where refugees are matched with locations. Here, the preferences of locations over refugees are given by a machine learning (ML) model that predicts the level of integration success (e.g., measured by the probability of employment within 90 days) of an individual at a given location [5]. 'The Swiss government has recently implemented a randomized test to examine the performance of data-driven algorithms for outcome-based assignment' [3]. Similar approaches have been developed for other application domains such as labor market [24] and course allocation [15].
These examples show that using predictive models to inform the preferences of agents in matching markets is not a problem of the future, but rather one of the present. However, prior work has evaluated potential vulnerabilities separately in the matching mechanisms [10, 7, 2] and prediction-based recommendations [23, 21]. When considering the matching algorithm, one usually analyzes the incentives of individuals by, for example, asking whether the mechanism is strategy-proof (i.e. whether agents have an incentive to misreport their preferences). Similarly, there is a broad literature on attacks on recommendations [8, 23, 21] and predictive models which might face poisoning or evasion attacks (i.e. attacks which either inject fake data to trigger an unfaithful model or perturb the testing input to trigger a misclassification) [11].
In this paper, we argue that, in addition to considering vulnerabilities of the matching and prediction mechanisms independently, it is critical to also look at vulnerabilities in systems that combine them together. Specifically, we consider systems similar to those described above, i.e., where: (a) agents of one side (the returning side) come back to the matching market in subsequent rounds (e.g., schools), (b) agents have post-matching objectives (e.g., schools want to be prestigious and minimize their cost), (c) agents on the returning side have the power to shape the quality of the interaction with those they are assigned to via post-matching decisions (e.g., schools can increase the performance of students through extra-curricular preparations or integration programs), and (d) these interactions impact outcomes for the non-returning side, which in turn influence future predictions (e.g., predictions based on the outcomes of past students change the preferences of current students).
By making the feedback loop between the matching and prediction mechanisms explicit, we uncover a new type of strategic behavior: the returning side can attack the system by changing their interactions with their matches. Even though in the short-run it might be beneficial for both matched parties to have the best possible interaction (e.g., both locations and refugees want refugees to be employed as soon as possible), the returning side (e.g., locations) might want to sacrifice their utility in the current round for different future predictions (e.g., locations might postpone hiring difficult-to-integrate refugees so they will not be allocated similar refugees in the future). We call such post-matching strategic behavior—where agents sacrifice their short-term utility to trigger different long-term predictions—adversarial interaction attacks. Similarly, we call a system where agents cannot benefit from such attacks interaction-proof.
In this paper, we first build a formal model for repeated two-sided matching markets with prediction-enhanced preference formation. Second, we use this model to define adversarial interaction attacks and the optimization problem faced by returning agents. Finally, we create a simplified setting (in the main text, and an agent-based model in the appendix) to analyze when and by how much agents can benefit from interaction-strategic behavior. We show that, for some systems, (a) the returning agents have an incentive to use adversarial interaction attacks, (b) the utility gains obtained through such attacks increase as predictions become more accurate and trusted, (c) once a returning agent attacks, others have an incentive to implement more severe attacks, and (d) the non-returning agents are unevenly negatively affected by such attacks.
2 Other Related Work
Adversarial Attacks in Recommendations. From past literature in recommender systems (RS), the most similar adversarial attacks to the one we consider are shelling attacks (i.e., attacks where fake users and ratings are created to trigger different future recommendations). These works mainly focused on crafting adversarial examples [23, 21]. Differently, Christakopoulou and Banerjee [8] analyzed shelling attacks from a machine learning and optimization perspective by building on the
literature on poisoning attacks in classification tasks [16], and formulating the problem as a two-player general-sum game between an RS and an Adversarial Attacker.
However, our setting has some key differences. First, the returning agents usually cannot inject fake users (e.g., because there are public records of which students attended which schools). Instead, agents attack by adapting their actions. This distinction has both social implications (the non-returning agents experience a different outcome while the attack takes place) and economic implications (the returning agent implementing the attack sacrifices their short-term utility). Second, the existence of capacity constraints entails a rivalry for available seats (if a student is accepted at a school then another student is not accepted, and vice versa). Consequently, when attacking for being matched to a specific (type of) agent, there are two intents to do so: (a) rank higher in the preferences of agents who are of interest, and (b) rank lower in the preferences of agents who are not of interest. The latter is different from classical recommender system applications and is increasingly important when the returning agents cannot fully and freely express their preferences. Third, also because of competition, agents do not decide to whom they are allocated (not all students can get a place to their most preferred school); instead, they report their preferences to a matching market, which produces an assignment. Therefore, attacks in this setting are not only efficient if they change the most preferred option of a non-returning agent, but also if they change the ordering of options lower in the preference ranking.
Strategic Behavior in Matching Markets. As noted above, matching markets (MM) are widely used to pair agents based on their reported preferences; in such settings agents can behave strategically by, e.g., misreporting their preferences. Prior work shows that users find and implement such profitable manipulations which leads to congestion and inefficiencies [7]. Thus, strategy-proofness is a common desiderata in real-world application domains; e.g., in the Boston school choice system, this triggered a transition from the Boston to the Deferred Acceptance mechanism [1]. In this paper we analyse strategic-interacting, which differs from strategic-reporting as it is not an attack on the mechanism alone, but rather on its combination with the prediction-based preference-formation process. When proposing predictive modelling for refugee assignment, Bansak et al. [5] distinguished between the modeling phase (when the predictive model is built) and the matching phase (when the assignment is produced). We generalise and extend this framework by also considering the phase when the paired agents interact and new data is produced. Moreover, we build on prior literature in economics when creating the model (for formalising the matching phase [25]) and when analyzing the system (for unilateral deviations and Nash equilibria [22]).
Using Simulations to Understand Long-Term Effects. The machine learning community has seen an increasing use of simulation to study the interaction between users and recommendations. RecSim [12] provides an environment that naturally supports sequential interaction with users. Mansoury et al. [19] proposed a method for simulating interactions between users and RSs to study the impact of the resulting feedback loop on the popularity bias amplification. Bountouridis et al. [6] built a framework called SIREN to study how RSs will impact users' news consumption preference in the long term. Similarly, scholarship within MM used simulations to understand the effect of different design choices. Erdil and Ergin [10] developed an experiment in order to compare the performance of two alternative matching algorithms in the school choice setting. In the context of online dating, Ionescu et al. [13] proposed an agent-based model (ABM) to test the effects of different platform interventions in reducing racial homogamy. At the intersection of ML and two-sided markets, Malgonde et al. [18] built an ABM to test the effects of introducing a two-sided RS within a complex adaptive business system. Similar to this previous work, we create a simplified setting to model a complex socio-technical system. We use both theoretical analysis (in the main text) and simulations (in the appendix - to confirm that the results carry to more realistic settings). We, however, have a different goal: to understand how the characteristics of the market affect the incentives to use and effects of using adversarial interaction attacks.
Our model for the system is composed of three phases: (a) modeling, (b) matching, and (c) interacting. The modeling stage builds a predictive model that forecasts the interaction outcome of two agents if matched. During the matching phase, the non-returning agents use this model to inform their
Figure 1: Overview of the interaction between different components of a system with a forecasting-based preference formation and a matching procedure. The interaction predictions can be used to inform the preferences of a. the non-returning side of the market (e.g. students), or b. the returning side of the market (e.g. locations).
preferences $^1$ ; next, a matching algorithm pairs each agent in the non-returning side of the market (student) with an agent in the returning side of the market. Finally, in the interacting stage, the agents – already paired according to the assignment obtained during matching – interact with each other (students attend the classes at their assigned school). This interaction will produce an outcome (e.g., SAT scores of students) that is kept as a record and used to inform future predictions. The entire system then repeats in a series of rounds with new non-returning agents each time, but with the same returning agents. Figure 1 shows an overview of all the phases and the interactions between them. In the remainder of this section, we formalise each of the stages and present the decision problem faced by the returning agents.
3.1 The Matching Stage
We start with the notation for the agents. Let  $X^t$  be the set of non-returning agents to be matched at time (round)  $t$ , and  $Y$  be the set of returning agents. Moreover, we denote by  $\succ_x$  the preference of the non-returning agent  $x \in X^t$  over the returning agents,  $Y$ . In other words,  $\succ_x$  is a ranking over  $Y$ . Similarly,  $\succ_y$  is the preference of  $y \in Y$  over  $X$ . These preferences could be either unweighted (i.e., the agent only knows the ordering of their options) or weighted (i.e., the agent also knows how much more they prefer each potential match over another). In the weighted case, the preference  $\succ_x$  has an associated weight function  $w_x: Y \to \mathbb{R}$  mapping each option  $y$  to the strength  $x$  wants to be matched to  $y$ .
A matching is a pairing of non-returning agents to returning ones, i.e.,  $\mu : X \to Y \cup \{\bot\}$ . Here,  $\mu(x) = \bot$  signifies that  $x$  remained unassigned<sup>2</sup>. More generally, a matching procedure (or allocation rule) maps the preferences of agents to a matching. In other words, a matching procedure  $M$  is a function mapping every  $(\{\succ_x\}_{x \in X}, \{\succ_y\}_{y \in Y})$  to a matching  $\mu$ . We denote by  $\mu^t$  the matching in round  $t$ , i.e.  $\mu^t = M((\{\succ_x\}_{x \in X^t}, \{\succ_y\}_{y \in Y}))$ .
3.2 The Interacting Stage
After being matched, the agents interact. For every agent pair  $(x,y)$ , there is a set of possible outcomes, depending on their interaction. Since  $x$  is not returning to the market, we assume they will always prefer the best outcome for them. Therefore, the set of possible outcomes depends on the actions of the returning agent,  $y$ . We denote the set of outcomes  $y$  could choose from when interacting with  $x$  by  $\mathcal{O}_y(x)$ .
Depending on the resulting outcome, each agent has some value, cost, and utility. If  $y$  chooses outcome  $o$  when interacting with  $x$  we denote by  $v_{y}^{o}(x)$  their value, by  $c_{y}^{o}(x)$  their cost, and by
Alternatively, the predictive model could be used to inform the preferences of the returning agents. For simplicity, we chose not to extend our model to capture this alternative in the main text and only include it in the appendix. Moreover, here we use school choice as a running example; see the appendix for other examples.
2In simplified scenarios where each returning agent can be matched to at most one non-returning agent (e.g., due to capacity constraints), we can express the matching function as a function from  $Y$  to  $X \cup \{\bot\}$ . More precisely,  $\mu(y) = x$  when  $x$  is matched with  $y$  and  $\bot$  if  $y$  remains unmatched. Since this refers to the same pairing, we avoid introducing additional notation and will also use  $\mu$  for the mapping of returning to non-returning agents in such simplified cases.
$u_{y}^{o}(x) = v_{y}^{o}(x) - c_{y}^{o}(x)$  their utility. We use the analogous notation for  $x$ . We also make the simplifying assumptions that (a) the value is symmetric, i.e.  $v_{y}^{o}(x) = v_{x}^{o}(y)$ , and (b) the cost of the non-returning side is null, i.e. that  $c_{x}^{o}(y) = 0$  and  $u_{x}^{o}(y) = v_{y}^{o}(x)$ .
As an example, a school  $y$  might have the choice set  $\mathcal{O}_y(x) = \{80, 100\}$  since it could either invest in extracurricular preparation for  $x$  (case in which  $x$  scores around  $100\%$  on state-administered tests) or not to invest (and  $x$  scores around  $80\%$ ). If school  $y$  incurs a fixed cost of 5 when preparing a student, then for the outcome 80 both the student and the school have a utility of 80 while for the outcome 100 the student has a utility of 100 while the school subtracts the cost from the value, and thus has a utility of  $100 - 5 = 95$ .
3.3 The Modeling Stage
The prediction model at time  $t$  is informed by the history of interactions until that time. We denote the history up to time  $t$  by  $H^{t} = \{(x,y,o,t^{\prime})|y = \mu^{t^{\prime}}(x),y$  interacted with  $x$  at time  $t^\prime \leq t$  with outcome  $o\in \mathcal{O}_y(x)\}$ . The history thus records the agents matched so far, together with the time and outcome of their interaction.
The prediction algorithm uses the history as an input. Its output is a hypothesis, i.e., a function which maps a pair of agents to their expected interaction outcome. We use the usual statistical framework in learning theory and denote the hypothesis by a parameterized function  $h_{\theta_t}: X \times Y \to \mathbb{R}$ . The prediction is thus based on the parameter  $\theta_t$  at time  $t$ , which is obtained by solving the optimization problem  $\theta_t = \arg \min_\theta L(\theta, H^{t-1})$ . Here,  $L$  is the loss function.
The hypothesis predicts values for a potential interaction between a non-returning agent,  $x$ , and all the returning agents. Therefore, it induces a preference weight function and a ranking over the returning agents. Formally,  $h_{\theta_t}(x,\cdot):Y\to \mathbb{R}$  gives a weighted preference over  $Y$ , which we denote by  $\succ_x^h$ . By exposing agent  $x$  to this ranking, their final preference might change. To capture this change, each agent  $x$  has a prediction integration function,  $h_{\theta_t}^x$ , that transforms a (weighted) preference of  $x$  into a new preference,  $\succ_x'$ , using the hypothesis-based ranking,  $\succ_x^h$ .
To continue the previous example, let us assume that student  $x$  initially believes schools  $A$  and  $B$  are equally good for them, i.e.  $\succ_{x}$  is  $A \sim B$  with, say, a weight function  $w_{x}(A) = w_{x}(B) = 80$ . Assume the hypothesis predicts  $h_{\theta_{t}}(x,A) = 100$  and  $h_{\theta_{t}}(x,B) = 80$ , which corresponds to the ranking  $A \succ_{x}^{h} B$ . The exposure of student  $x$  to this prediction-based ranking changes their preference to a new one which lies between their original opinion,  $\succ_{x}$ , and the suggested one,  $\succ_{x}^{h}$ . For instance, the new preference could correspond to the weight profile obtained by averaging the original and the suggested ones. Then, the new preference,  $h_{\theta_{t}}^{x}(\succ_{x}) = \succ_{x}^{\prime}$ , has the weight function  $w_{x}^{\prime}(A) = 90$  and  $w_{x}^{\prime}(B) = 80$  thus ranking  $A \succ_{x}^{\prime} B$ .
3.4 The Decision Problem of Returning Agents
The decision problem faced by the returning agents when they choose how to interact with their matches can be formalised as a dynamic programming task over the infinite horizon. They choose the action that maximizes the sum of their utility now plus the expected discounted utility in the future if they take that action. Using the notation introduced in the previous section, the maximum expected-discounted utility of a returning agent,  $y$ , from time  $t$  onward is
$$
U _ {y} \left(\mu^ {t}\right) = \max  _ {o \in \mathcal {O} _ {y} \left(\mu^ {t} (y)\right)} \left(u _ {y} ^ {o} \left(\mu^ {t} (y)\right) + \beta \mathbb {E} \left[ U _ {y} \left(\mu^ {t + 1}\right) \mid (\mu^ {t} (y), y, o, t) \in H ^ {t} \right]\right), \tag {1}
$$
where we make the standard assumption in economics that agents steeply discount future utility and denote the discount factor by  $\beta$ . The optimal strategy of a returning agent is to take, at every round, the interaction leading to the outcome that maximizes the expected-discounted utility.
The straightforward interaction strategy is to always choose the outcome maximising the 1-step utility, i.e. choose  $\arg \max_{o\in \mathcal{O}_y(\mu^t (y))}\bigl (u_y^o (\mu^t (y))\bigr)$ . When a returning agent uses this strategy, we say it interacts truthfully. Naturally, in general, the optimal strategy needs not be the truthful one. The resulting gap leaves space for strategic interactions. We refer to strategies involving non-truthful
3To achieve this succinct problem formulation, we made some simplifying assumptions and notations (details in Subsection 2.3 from the appendix).
interactions as adversarial interaction attacks and call the systems in which agents cannot benefit from such attacks interaction-proof.
To give two simple examples, a system in which the preference formation is solely based on the attributes of schools (e.g., how far away the school is, what subjects are taught, what are the final examinations) and past data on interactions is not used would be interaction-proof. Similarly, a system that only uses the history of interactions before announcing the decision to introduce the ML-algorithm would also be interaction-proof. This is because, in both of these cases, the future expected utility is constant with respect to the choice of interaction. Thus, the utility from now onward is maximized by choosing the outcome giving the highest utility in the current round.
4 Simplified Setting
While useful for formalizing the definition of adversarial interaction attacks within a variety of contexts, the problem formulation proposed in the previous section is too general to allow for a tractable and insightful numerical analysis. Thus, we operationalize the framework by creating a simplified model. The main goal of such a model is to support both proving that returning agents can benefit from adversarial interaction attacks and understanding why and when this could be the case. The remainder of this section introduces the simplified model. To improve readability, we refer to returning agents as schools and non-returning agents as students. However, the model could be read through the lens of alternative applications (e.g., locations and refugees).