Evolutionary Prediction Games
Authors: Eden Saig
Nir Rosenfeld
Venue: NeurIPS 2025 poster ()
Keywords: Evolutionary Game Theory
Feedback Loops
Stateful Performative Prediction
Potential Games
Replicator Dynamics
Primary area: theory
TL;DR: We propose a framework based on evolutionary game theory to model feedback loops in supervised learning, and use it to study how different learning settings affect long-term outcomes.

Abstract: When a prediction algorithm serves a collection of users, disparities in prediction quality are likely to emerge. If users respond to accurate predictions by increasing engagement, inviting friends, or adopting trends, repeated learning creates a feedback loop that shapes both the model and the population of its users. In this work, we introduce evolutionary prediction games, a framework grounded in evolutionary game theory which models such feedback loops as natural-selection processes among groups of users. Our theoretical analysis reveals a gap between idealized and real-world learning settings: In idealized settings with unlimited data and computational power, repeated learning creates competition and promotes competitive exclusion across a broad class of behavioral dynamics. However, under realistic constraints such as finite data, limited compute, or risk of overfitting, we show that stable coexistence and mutualistic symbiosis between groups becomes possible. We analyze these possibilities in terms of their stability and feasibility, present mechanisms that can sustain their existence, and empirically demonstrate our findings.

Evolutionary Prediction Games
Eden Saig
Nir Rosenfeld
Technion Israel Institute of Technology
Technion Israel Institute Technology
Haifa, Israel
Haifa, Israel
edens@cs. www.wiec.chhc...m
nirrêcs www.heech.nnaaacm ac.il
Abstract
When prediction algorithm serves collection of users, disparities in prediction quality are likely to emerge. If users respond to accurate predictions by increasing engagement, inviting friends, adopting trends, repeated learning creates feed- back loop that shapes both the model and the population of its users. In this work, introduce evolutionary prediction games, framework grounded in evolutionary game theory which models such feedback loops as natural-selection processes among groups of users. Our theoretical analysis reveals a gap between idealized and real-world learning settings: In idcalized settings with unlimited data and com- putational power, repeated learning creates competition and promotes competitive exclusion across a broad class of behavioral dynamics. However, under realistic constraints such finite data, limited compute, or risk of overfitting, we show that stable coexistence and mutualistic symbiosis between groups becomes possible. We analyze these possibilities terms of their stability and feasibility, present mech- anisms that can sustain their existence, and empirically demonstrate our findings.
Introduction
Accurate predictions have become essential for any platform that supports user decision-making. Improvements in prediction accuracy often directly translate better quality of service with benefits to both the platform and its users. This is one reason why modern platforms ranging from content recommendation and online marketplaces come to rely on machine learning as their backbone, and are investing much effort and resources in continually improving their predictions. However, while generally beneficial, promoting accuracy blindly can have unexpected. and in some cases undesired. consequences [9, 59, 24]. It is therefore important understand the possible downstream and long-term term effects of learning on social outcomes.
personalized education and medical services have
Conventional learning approaches aim to maximize accuracy on a given, predetermined data who choose to
distribution. But in social settings, the distributions composed of those the platform. When such choices depend on the quality of predictions, learning becomes a driver of user self-selection, and thus gains influence over its user population. This creates feedback loop: model deployment shifts the population, and population changes trigger model retraining. We are interested in understanding the general tendencies and possible long-term outcomes of this process.
In particular, our focus is on user feedback dynamics in which accurate predictions encourage engagement adoption, and prediction emrors discourage them. Such dynamics arise across domains: accurate recommendations drive network-effect growth as users invite peers [39]; successful content strategies social media mimicked by others [4]: precise credit-risk models reduce premiums and therefore attract more clients to loan programs [32]; and medical providers with higher diagnostic accuracy draw more patients [3]. Common the above is that individual choices adhere some form of group structure in the population, and that in aggregate, these groups tend to grow when prediction quality is higher. Note that this notion of "group" flexible, groups can represent different demographics, behaviors, or roles, and with memberships being either inherent or chosen.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Focusing on user choices as the primary driving force, we seek to analyze the impact of learning on group proportions over time. For this purpose, we adopt a novel evolutionary perspective and model the joint dynamics of learning and user choices through the lens of natural selection. Under the assertion that some degree of predictive error always exists, our key modeling point that accuracy becomes, in effect, scarce resource over which different groups in the population "compete". By associating each group's 's accuracy with its evolutionary fitness, we obtain an evolutionary process in which learning is the driver of selective pressure, and consequently, determinant of long-term population outcomes. We can then ask questions regarding the long-term tendencies of the population composition, overall and per-gr grrug accuracies, and the affect of different modeling and algorithmic choices on temporal trends and long-term evolutionary outcomes.
Contributions. Our main conceptual innovation is to analyze these feedback loops using a pop- ulation gume-a core component of evolutionary game theory useful for studying the dynamics of agents driven by local interaction rules [41, 27, 52]. Evolutionary game theory relates between game- theoretic properties of population games (e.g.. Nash equilibria), and the dynamics of a wide variety of natural selection dynamics, such as imitation, word-of- mofthhh influence, or social learning, all of which supported by our analysis. This makes the framework well-suited for our setting, it allows us to address the common thread in the wide variety of feedback loops described above in a unified way.
Towards this, we first propose a novel game, called an evolutionary prediction game, in which evolutionary fitness is associated with prediction accuracy. This allows us to study the co-evolution of (re)trained model and the population of its users. We then analyze the structure of games induced by different learning algorithms and their corresponding dynamics, and characterize the possible outcomes under different settings. Our results include conditions under which 'survival of the fittest' is a likely outcome; mechanisms that nonetheless enable coexistence; a discussion of the role and effects of retraining; and connections between survival, accuracy, and (evolutionary) stability. We also discuss connections to fairness and social conservation.
We conclude by complementing our analysis with experiments using both synthetic and real data and coupled with simulated dynamics. Our empirical results shed light on when and how certain user groups are likely to dominate, disappear, or coexist, and demonstrate how different design choices can shape social outcomes- inadvertently. Together, these highlight the importance of understanding, anticipating, and accounting for the long-term effects of learning on its user population.
1.1 Related Work
Evolutionary game theory. Evolutionary game theory was originally developed as a framework
for modeling natural selection processes in biology [44, 42, 41, 27, 56), and has since been applied to the analysis of large populations of myopic interacting agents in economics [52, 20], and to the study of online learning algorithms [19, 7]. From a technical standpoint, the games we propose and analyze are unique in this space they are defined implicitly solutions to statistical optimization problems. Our results for oracle classifiers reflect the principle of competitive exclusion [23], while our analysis of non-optimal classifiers relates to long-studied questions of coexistence [11].
Performativity. Our work relates to the emerging field of performative prediction [48], which studies learning in settings where model deployments affect the underlying data distribution, with emphasis on (re)training dynamics and equilibrium. A central effort in this area is to identify general global properties (e.g.. appropriate notions of smoothness) that guarantee convergence. Our work com- plements this effort by proposing structure, in the form of user self-selection [65, 28], which provides an efficient dineesenniioal representation of the distribution map. This allows us to work in the no- toriously challenging stateful performative setting [5]. and to characterize stronger notions of stability.
Fairness. The notion of equilibrium in evolutionary prediction games relates to the fairness criterion of overall accuracy equality [61]. The main distinction is that in prediction games, the question not only whether groups treated fairly, but also which groups are even considered, since fairness can be measured only for groups that are observed. Complementing other voiced concerns on long- tormterrm fairness outcomes [25, 40, 14, 49), our work adds a novel counterfactual perspective: if we observe fair outcomes at present, could this be because some groups were historically driven out of the game?
See Appendix B for extended discussion and additional related literature.

---

Group
Training set
Fitness
Possible outcomes
Accuracy
Dominance
Mixture P
Classifier h
Selection
Coexistence
Extinction
Group #
M
Time
Figure 1: Natural selection in two-group setting. Population mixture p of groups; Classifier h is learned using data from the mixture distribution Pp: Evolutionary fitness is associated with prediction accuracy: Differences in fitness drive change in mixture coefficients; Possible long-term tendencies are dominance (only A survives), extinction (only B survives), or coexistence (both survive).
2 Setting
The core of our setting is based on a standard supervised learning setup in which examples describe user data. Denote features by E X and labels y E y. For a given distribution D over pairs (x, y). and given a training set S = {(r,, ) D", the goal in learning is to use S to find a predictor h : X + y from class H which minimizes the expected error under some loss function É y x y + IR For concreteness, we focus on classification tasks with the 0-1 loss,
E(g. y) ( + y). whose minimization equivalent maximizing expected accuracy:
acc P(a.v)~ (((xx)
Denote the learning algorithm by A(S), and by ~ A(D) the process mapping distributions to learned classifiers, with randomness due to sampling. Since solving Eq. (1) is both computationally and statistically hard, practical learning algorithms often resolve to optimizing an empirical surrogate on the sampled data. One question will ask is how such compromises affect long-term outcomes.
Population structure. As data points (x, y) are generated by users, the data distribution rep- resents a population. We assume the population is mixture of K groups. Each group k [K] is associated with a group-specific distribution Dg over (æ, y) pairs, which remains fixed, and with relative size pk, which evolves over time. The overall data distribution is mixture denoted by D = Dp = Σn Pk Dj. where P& the current relative proportion group & (such that Σ P = and pı 2 Vk). Given some fi the marginal expected accuracy on group is accs (h) accp, (h). The population state vector p (pi..... PK) will be our main object of interest. As p encodes group proportions, the space of population states is the (K - 1)-dimensional simplex A and since determines Dp. for brevity we denote h ~ A(p) A(pp) and (h) = accp, (h).
Dynamics. When a classifier deployed, users make decisions according to the quality of the predictions they receive. Collectively, individual responses reshape the population composition p. resulting a subpopulation shift [62]. The updated population then serves as the training distribution for the next classifier-whit which forms feedback loop between p and h. For training, consider a sim- ple procedure repeated training on the current distribution. In terms of user behavior, our framework accommodates a wide family of dynamics, including reproduction, network effects, imitation, social influence, rational decision ceaiiinn-making,.. and competition, as alluded to in Section In Appendix C we define these formally and provide additional examples. In Section 3 make assumptions explicit.
Outcomes. Denote the initial population state by p", and the initial classifier by n ~ A(p"). We consider dynamics which evolve towards fixed points, and denote the corresponding point by p*. Denote the classifier by h* ~ A(p"). We characterize fixed points in terms of three key properties:
For the population, dynamics may act as force diminishing certain groups (i.e., driving towards a state satisfying P = O for some E [K]. In such cases, we say that the population is dominated by the remaining groups. When the population composition at rest has at least two groups (i.c., ph > for multiple values of k), we say that these groups coexist.
Population composition:
Accuracy: For the classifier, dynamies may result in overall performance improving over time, (i.e., ace(h*) acc(h")). deteriorating, remaining unchanged.
3

---

Data Distributions
Evolutionary Prediction Game
Simulated Replicator Dynamics
2
Group B
Acamnyy (weod)))))
100%
Population (p)
Only
Group A
90%
50-50
00%
Only
Only
50-50
Only R
15
20
Population state [p]
Time (1)
Figure 2: Evolutionary dynamics for two groups, induced by an oracle linear classifier (Section 4). (Left) Data distributions in feature space. Dashed line demonstrates the optimal linear classifier hp for the uniform mixture p = (0.5, 0.5). (Center) Evolutionary prediction game Fs(p) = acc; (hpp) The game has two stable equilibria with single-group dominance, and one unstable coexistence equilibrium. Overall population accuracy accp(hy) convex, and maximized at the boundries. (Right) Replicator dynamics induced by the game, for various initial states p". Populations evolve towards fixed points with single-group group dominance.
Stability: For the joint system, some fixed points p" stable and attract all neighboring states, while other fixed points are unstable and may cause dynamics to diverge under small perturbations. An illustration of the setting is provided in Figure 1.
3 Evolutionary Prediction Games
To analyze the feedback loop, we leverage evolutionary game theory, which is a general framework for the analysis of natural selection processes [42]. The framework models evolutionary interactions between individuals from K groups in large mixed population: Given a population state p E A each group & E [K] is associated with a scalar fitness function, F& (p), which quantifies the average evolutionary fitness of individuals in that group. Together, the fitness functions = (Fi,..., FK)
define population game F: A + RK, which governs the dynamics of p.
Formally, population games are symmetric normal-form games for continuum of agents. Evolution- ary game theory uses population games to map myopic, often local agent decisions to group-level dynamics governing the evolution of group proportions p over time. Modeling the game at the group level serves as useful abstraction for studying the general tendencies and possible long-term outcomes of local interactions in aggregate. Note that the notion of "group" is flexible and can support memberships that are either inherent (e.g., by demographics) or chosen (e.g., service provider).
Accuracy as evolutionary fitness. We formalize the interplay between learning and population dynamics through novel type of population game- an evolutionary prediction game. The game associates the evolutionary fitness of each group with its expected marginal accuracy accą under a classifier h trained on data sampled from the mixture Dp:
Definition 1 (Evolutionary Prediction Game). Let A be learning algorithm, and let Di...., DK E A y) be group distributions. Denote by (c) the marginal accuracy of group k [K] under classifier h ~ A(p). The evolutionary fitness of group k under population state p is:
Fe(p) = (acc
Together, the tuple F(p) = (Fi(p)..... Fx(p)) defines an evolutionary prediction game.
In the notation (p), we assume for brevity that the learning algorithm A and group distributions Dy are clear from context, and we state them explicitly otherwise.
Evolutionary prediction games capture an implicit interaction between groups: while each group's fitness F& (p) is given by the accuracy its own members, the classifier hp is trained using data from the entire population. Thus, the evolution of different groups becomes coupled through the learning process. For example, Figure (Left) illustrates binary classification setting over two groups (A, B), Figure 2 (Center) shows the corresponding evolutionary prediction game F (p) induced by an oracle linear classifier (defined in Section 4). At p (0.7, 0.3), we have Fx(p) R 0.97 and FD(P) Rs 0.78, and thus the fitness of group A is higher when the classifier is trained on data sampled from Dp-

---

Equilibrium. Nash equilibrium in general population games is defined as follows:
Definition 2 (Nash equilibrium of population game; e.g. [52]). Let F(D) be population game, and denote support(p) (p) = [k| | pæ > 0}. A state p E A is Nash equilibrium of P(p) if it satisfies:
support(p") C arg ma[k][K] Fs (p")
This also applies to evolutionary prediction games; c.g., the game in Figure 2 (Center) has three Nash equilibria at (),, (0.5, 0.5), (0, 1)), and all population games have at least one equilibrium [52].
Induced population dynamics. Formally, we encode dynamics as p = (p), where P is the time derivative of p, and Vp A + TA* is tangent vector field induced by the game F(p). Our results apply under mild structural assumptions on VE. namely: (i) Vp is continuous; (ii) the direction of flow is aligned with fitness, formally Vp(p) - FF)) 0 whenever VF # 0. a property known as positive correlation; (iii) Fixed points of the dynamical system coincide with equilibria of F(p), formally through either Nash stationarity imitative dynamics. All are satisfied by the user-level dynamics highlighted in Section and Appendix C, where they emerge as group-level properties in the prgeepopulation limit. Moreover, many of them converge towards the canonical replicator equation, py: = ps (Fi(p) F(p)), where F(p) = Ex P Fie is the average fitness across the population [57]. We also note that time-scales of convergence are determined (and can be adjusted) by the scale of VF. See Appendix D.2 for formal definitions.
Fairness. In the context of fairness, classifier satisfies overall accuracy equality if all groups receive equal prediction accuracy [61]. We show that this criterion is satisfied in expectation by classifiers trained on equilibria mixtures of evolutionary prediction games (proof in Appendix D.3.2):
Proposition 1. Les F(p) be an evolutionary prediction game induced by learning algorithm A, and let p" be a Nash equilibrium. Then A(D,-) satisfies overall accuracy equality in expectation.
As outcomes of the dynamics correspond to equilibria of a game, this lends to a natural form of emergent faireess. However, note that this criterion only applies to groups that appear in the data: in Section 6.3, we further show that a system may appear as fair due to other groups being driven out.
Heterogeneous fitness. Finally, in some settings, incentives or outside alternatives may vary across groups This may be captured using the notion of retention functions [25], which translate marginal prediction accuracy to the tendency to remain engaged. In such cases, the transformed fitness function becomes F& (p) = Vi (Ex, (h))). where 1/4, [0, 1] + R is a group-dependent, continuous strictly-increasing function. A simple example is (x( = x - by (for by > 0) which allows groups to differ in the minimum accuracy their users require to keep using the system. All of our results apply to any set of strictly monotone ia for K = 2, and to positive afline i42 with per-group offsets by for K > 2. For clarity we state our main results for (x) = but also discuss them for broader
4 Competitive Exclusion Under Oracle Classifiers
To understand how populations and classifiers evolve jointly, we start by characterizing natural selection in 'ideal' setting, in which a classifier is trained repeatedly using unlimited resources: Definition 3 (Oracle classifier). Let H be a hypothesis class, and let be a data distribution. An oracle classifier with respect to H is minimizer of the 0-1 loss in expectation over D:
hopt E arg minneu B(z.g)--[ (h(z) + yll
Oracle classifiers represent ERM classifiers in the population limit. This regime abstracts away the complexities due to estimation and approximation errors of practical learning algorithms (i.e., algorithms that learn from finite data reasonable time). Nonetheless, analysis of the dynamics remains challenging due to the argmin operator. We focus on settings in which hopt exists, and denote by A (D) the (theoretical) learning algorithm which returns the oracle classifier with respect to D. We assume that fitness functions are continuous, and tie-breaking is consistent. To simplify presentation, we first assume that each group has a distinct oracle classifier, and then address the general case. Our central result characterizes natural selection induced by oracle classifier retraining:
'The Bayes-optimal classifier f Bayes (x) arg max, p(y| x)) is a special case of this definition for 7 y
5

---

Theorem 1. Let 'H be hypothesis class, and denote by A the oracle learning algorithm with respect to H. Assume that at each population state p the oracle classifier hp ~ A (Dp) is deployed. Then it holds that:
1. Accuracy: Overall accuracy increases over time, @cccp(hpp) O.
2. Stability: A stable equilibrium always exists, and there can be multiple such equilibria.
3. Competitive exclusion: For all stable equilibria, (support((")) = 1.
4. Coexistence: Equilibria with |support(p") 2 2 may exist, but are unstable.
Proof sketch. The proof leverages structural properties that emerge despite the complexity of the learning problem. Our main technical tool is the framework of potential games, which are population games that can be expressed as a gradient of a scalar potential function defined over the simplex. We start by showing that the expected accuracy of any fixed classifier is linear in p. then the core of the proof is a convexity argument: the optimality of AAopt implies that accp (hpp is convex as a pointwise maximum of linear functions, with gradients given by marginal accuracies. From this we identify f() = accp(hp) (Ap) as a potential function, and leverage the known correspondence between equilibria of population games and local extrema of potential functions. Finally, distinct oracle classifiers imply strict convexity of (()), and the stability of single-population states follows from the fact that convex functions over the simplex are moxally-aximiieed at the vertices, and the accuracy condition follows from the fact that (p) is a Lyapunov function. Full proof appears in Appendix D.5.
Extensions. To simplify presentation, the statement of Theorem I assumes that the oracle classifiers for each group are distinct. When this assumption doesn't 't hold (e.g.. when Dj = Di for some k + k'), generalized version of the theorem applies: rather than single-group stable equilibria, dynamics converge towards convex combinations of groups that share an identical oracle classifier with identical marginal accuracy (See Appendix D.5.1). For heterogeneous fitness where retention functions 7/2 may differ across groups, we show that the retention functions we support induce potential games with transformed potential functions. Survival is determined according to the maximizers of the transformed potential function, and the quantity that increases over time becomes the average fitness E. P& (ap)) (See Appendix D.5.2). Finally, we also note that a similar characterization also holds for classifiers trained once at the outset (See Appendix D.4).
Interpretation. Theorem suggests that for repeatedly trained oracle classifiers, natural selection will amplify the dominance of groups having the highest fitness at the outset. From an evolutionary perspective, this reflects the competitive exclusion principle [23], which states that natural selection tends towards exclusive survival of the fittest when multiple species compete over the same resources. Stability implics that once a group dominates, other groups will remain excluded. While this is the general trend, coexistence is nonetheless enabled by retraining, in the form of possible equilibria in which multiple groups survive. That is, there may exist population states p* such that P > 0 for several groups k and for which this remains hold when the classifter is retrained. The crux, unfortunately, is that such points are evolutionarily unstable, in the sense that small perturbations around p" will push dynamics away. This implies that optimal retraining- unconstrained by limited data and compute- induces a strong form of competition between groups But the existence of mixed equilibria suggests that other algorithms might be able to encourage symbiosis, which we explore next.
5 Avenues for Coexistence
Intuitively, oracle classifiers drive dynamics away from mixture equilibria because the learning objec- tive is fully aligned with, and so reinforces. self-selection. This motivates the natural question: Can repeated training drive dynamics also towards coexistence, and when? Here we illustrate two ways for symbiosis to arise: as result of inherent limitations of learning practice (i.e., limited data or com- pute), and by applying an ad-hoc, dynamics-awan learning algorithm designed to stabilize outcomes.
context- dependtd view on coexistence. is important to emphasize that coexistence in our setting is neither innately 'good' or 'bad', but rather, its desirability depends on context. Hence, the question of whether coexistence should be encouraged depends on the task at hand, the nature of user choices, and the goals of the system designer. For instance, recommendation platforms may

---

Praxy Loss
Interpolation
Finite Data
Dotributions
SVft-VVM
I-NN
Hard-SVM
Accuracy
100%
80%
80%
75%
90%
Only
Only
Only
Only
Only
Population state (p)
Population state (p)
Population state (p)
Figure 3: Graphical illustration of theoretical coexistence results. Top row presents data distributions, bottom row shows the corresponding prediction games. (Left) Stable and mutualistic coexistence induced by the use of a proxy loss (Theorem 2). (Center) Stable coexistence induced by interpolation (Appendix D.7). (Right) Mutualistic coexistence induced by a finite training set (Appendix D.8).
aim to preserve both content and audience diversity, making coexistence desirable. Conversely, in consumer markets having multiple competing standards (e.g., video encoding format, compression schemes, payment platforms), there are benefits to converging on the single best alternative. This context-dependent stance aligns well with the notion of evolution, which in itself is neutral.
5.1 Learning in Practice
While the goal in learning is to minimize the expected 0-1 loss, in practice learning often resorts to optimizing proxy objectives over sampled data via empirical risk minimization. Here we show that, perhaps surprisingly, basic aspects of this upproach-namely namely the use of surrogate losses, access to finite data, and interpolation methods- can create conditions that facilitate mutuallsmm in which all groups benefit from coexistence. For each aspect, we demonstrate through a carefully crafted example how it can act mechanism that enables stable coexistence. Here we focus discussion on surrogate loss as a representative construction; for finite data and memorization, we illustrate the underlying distribution and induced game in Figure and defer the formal analysis Appendix D.6.
Coexistence by surrogate loss. A common approach to empirical risk minimization is to replace the 0-1 loss with a convex surrogate loss, such the hinge loss used in the classic Soft-SVM algorithm [13]. This enables optimization, but introduces bias: whereas the 0-1 loss applies uniform penalties, surrogates penalize misclassified points relative to their distance from the decision boundary of h. Our next result uses this artifact to construct a game having two groups that are complementary in their incurred bias, and hence balance each other induce mutualistic coexistence.
Theorem 2. There exists an evolutionary prediction game in which learning with the hinge loss and (2 regularization induces mixed equilibrium that is both stable and maximizin.
The construction outlined in Figure (Left), and proof is provided in Appendix D.6. The proof leverages the bias of hinge loss against minority classes: For each group alone, minimizing the hinge loss biases the learned classifier against the minority class, which suboptimal. But since each group has a different minority class, these biases cancel out at the mixed equilibrium. The surprising property is that this state is also evolutionarily stable: when one group grows slightly (i.c., P1 = 0.5 + e), loses more from the fact that the other group shrinks (i.e., P2 0.5 e) than it gains from its growth. In this sense, each group needs the other to maintain its existence, giving rise to stable and mutualistic symbiosis. Appendix D.6.3 expands on this idea to show that varying the amount of regularization leads to bifurcations, in which the game transitions between 3, 1, and 5 equilibria.
Interpretation. In the example game above, as in those we construct for finite data (Appendix D.8; Figure 3 (Right)) and interpolation (Appendix D.7; Figure 3 (Center)), mutualism is enabled by how user self-selection compensates for the algorithm's 's imperfections. When the resulting bias is complementary across groups, this can give rise to (implicit) cooperation. One interpretation is that the existence of each group acts regularization k, the objective can be written as arg max, acc + AR(h) where R(h) = Σjfæ acc, (h) is a 1 Ej Pj. Nonetheless, our results should not
others: from the perspective of group
data-dependent regularizer with coefficient À Pt
7

---

be taken to imply that proxy losses or finite data are good means to achieve coexistence; rather, they highlight how mutualism can organically materialize when learning affects different groups differently. We explore this further empirically in Section 6.2.
5.2 Stabilizing Coexistence Equilibria
Thus far we have considered evolutionary prediction games in which learning pursues the conventional objective of maximizing accuracy each timestep the current distribution. But if the leamer is aware of population dynamics and of how the learned classifier may influence outcomes, then makes sense to consider learning algorithms that take this into account. The question of course is how.
Here we focus on particular aspect of this general question that arises from our last result in Sec. 4, namely: given a system with a desirable but unstable mixed equilibrium p". how do we stabilize it? We propose a conceptually simple solution that works by reweighing examples in the standard accuracy objective in way that accounts for dynamics. The main idea is to invert the natural tendency of dynamics to push away from the unstable equilibrium, and instead pull towards it, by training 'as if" the distribution was at different state p' than its actual current state p:
Proposition 2. Let App be an oracle algorithm with equilibrium p'. 1fpp has full support (and thus is unstable), then becomes stable under A' (p) = APPL(2P) - -p).
Proof in Appendix D.9. The idea is that A' trains 'as if" the state is 2p" p. which can be achieved by weighing examples from each group & by wek = pps dynamics to push away from the unstable equilibrium, and instead pull towards it. The statement regards optimal classifiers and requires knowledge of p", . but in practice we can set w according to an estimated p. We further explore this approach in Section 6.1.
Ph. This inverts the natural tendency of
6 Experiments
To demonstrate how some of the principles we have discussed so far apply in more realistic settings, we now turn to explore evolutionary prediction games empirically using real data and simulated dynamics. We include three experiments. The first experiment complements our results See. 5.2 by demonstrating how stabilization can improve both overall and per-group accuracies via coexistence. The second complements Sec. 5.1, showing how mutualism balancing, here from complementarities in label noise. The third experiment explores fairness under population dynamics (Prop. 1) to illustrate the limitations of static fairness measures.
arise organically due to group
6.1 Mutualistic Coexistence From Data Augmentation
Data augmentation is a method for enriching the training set with dlussppreservin transformations of inputs (e.g., camera angle, source of lighting) [e.g.. 55]. Augmentations are typically considered as artificial constructs, but there are also cases where they emerge naturally (e.g., medical images from different hospitals or imaging devices). Here we show that when different sources correspond to different groups, data pooling can serve to foster beneficial coexistence in which all groups gain from the existence of others. This suggests that 'natural' augmentations can effective also in the long run.
Method. We use the CIFAR-10 image recognition dataset [36], which consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. As horizontal image flips are considered class-preserving for this dataset, group A consists of images sampled directly from CIFAR-10 10, and group B consists of images under horizontal flip. We use a ResNet-9 network for prediction [26), and train it using the ffcv framework with default optimization parameters [38]. For each p. we measure the prediction model's I's accuracy on the original images from the CIFAR test set (representing the
acc, a (hp)), and on their flipped counterparts (representing acc (hpp)) For stabilization, method described in Section 5.2 with pd (0.5, 0.5). We simulate the evolutionary process using discrete replicator dynamics, and use linear interpolation determine intermediate fitness values.
Results. Figure 4 (Left) presents the estimated evolutionary prediction game, averaged across 20 repetitions. The game has three equílibria: Two stable equilibria with single-group dominance (92.6+ 0.1% accuracy), and an unstable coexistence equilibrium (93.5 + 0.1% accuracy). Although fitness
8

---

CIFAR-12 10 With Horizontal Flips
CIFAR 10 Simulated Stabilization
MNIST With Label Noise
Welfow Population
Only
App)
81.5%
93.0%
(p)
Accuracy
01.0%
50-50
92.
93.2%
00.5%
03
92. 0%
92.8%
80.0%
Only A
50.50
Only 1
500
1000
1500
2000
Only A
50:50
Only y
Population state (P)
(1)
Population state (p)
Figure 4: Empirical evaluation in two-group settings. (Left) Game induced by CIFAR-10 with Resnet-9 and groups representing horizontal flips. The game has unstable mutualism (Section 6. I). (Center) Simulated stabilization of replicator dynamics the CIFAR-10 game (Section 5.2). (Right) Game induced by MNIST with label noise, showing stable mutualism (Section 6.2).
is typically higher for the larger group, the mixed equilibrium attains the highest overall population accuracy, suggesting that each groups benefits from the existence of the other. Reaching this point however requires stabilization; Figure (Center) shows how our algorithm from Section 5.2 is able to attain stable beneficial coexistence, despite using an empirical proxy objective. Appendix F includes additional results on sampling noise, time to convergence, and sensitivity analysis for stabilization.
6.2 Stable Coexistence From Overparameterization
Modern neural networks frequently overparameterizzed and are known to achieve strong test-time performance despite memorizing the training data [e.g.. 64). Recent research attributes this phenomenon to the implicit biases introduced by the training process [e.g.. 60]. Building on these ideas, here we demonstrate that stable coexistence can emerge when an overparameterized neural network is trained data in which groups have complementing label noise. Our construction considers a multi-class settings where each group includes a majority of noisy examples from some class, and minority of clean examples from other classes. This aims captures settings in which user groups (e.g., by country) differ in the marginal distribution p(y) over classes (e.g.. spoken language), and where the dominant class more likely be misclassified (c.g.. due to many accents).
Method. We use the MNIST dataset, set K = 2, and split the classes unevenly between two groups: Group A is biased towards even digits {0, 2, 4, 6, 83. whereas group B is biased towards odd digits, both with a 4:1 imbalance. For each group, the true label of each digit to the next digit with same parity with probability 0.2 (i.c. for group A, label noising stochastically maps for 200 epochs using stochastic gradient decent with momentum. The training and test sets are split independently, and for each p measure the prediction model's 's accuracy both splits of the test sets (representing acc (hp) and pccp (hp)). Under the given parameters, the label noising process leads to an accuracy upper bound of 84% assuming perfect recognition.
introduce label noise to the majority classes, mapping
2 4, etc.). We train convolutional neural network
Results. Figure 4 (Right) shows the estimated evolutionary prediction game, averaged across 50 repetitions. Note how label noise has 'fippedd' the game that accuracy is higher for the minority group (i.c., acCB > acca when pB P... and vice versa). This indicates that groups naturally balance each other in this setting. As a result, the game has a stable coexistence equilibrium, with 80.4 0.2% test accuracy (vs. the theoretical upper bound of 84% due to label noise). Training accuracy is 98.7 0.1%, suggesting that interpolation is indeed the likely phenomenon at play. See Appendix D.7 for further discussion how interpolation facilitates coexistence underlying mechanism.
6.3 Population Dynamics and Fairness
Lastly, we examine how population dynamics interact with notions of fairness. Our goal here is to show that a system can appear to be fair at given point in time-but only because some groups have been driven out of the game, and faimess is measured on the remaining observable groups. Our message here that fairness should consider not only what has happened, also what could have happened, particularly with respect to groups that perhaps had previously been excluded.

---

ACSIncome Replicator Dynamics
Population Composition Over Time
Group Fitness Over Time
TX
00%
Relative group size
50%
Py7x))
Classffiation accuracy
76%
40%
P(CA)
a00p
75%
20%
10%
P(RV)
195
74%
0%
200
Population composition
Time
Time
Evolutionary dynamics in three-group setting, induced by linear SVM trained on the ACSIncome dataset (Section 6.3). (Left) Replicator dynamics on the three-group simplex. Red line indicates a trajectory starting from the uniform mixture, and background colors indicate basins of attraction. (Center) Population composition over time for the same trajectory. Proportion of NY users becomes negligible R 316. (Right) Fitness over time. Effective group disparity is significant at the outset, but shrinks by an order of magnitude after NY users are driven out.
Figure 5:
Method. We use subset of the Folktables ACSIncome dataset [17], a tabular income-level classification task based on US census data. We associate groups with data from three states in the US, with the idea that firms operate in different states but train model on data aggregated from all sources. Specifically, we set K = 3. and use 2018 census data from the states of California (1956665 datapoints), New York (103,021), and Texas (135,9 (135,924). We use linear SVM as the learning algorithm, and randomly hold out 5,000 points from each state for evaluating accuracy. To compute fitness at given population state p. we sample a dataset of size n = 1000 from the mixture, train a classifier, and measure group accuracies using the held-out sets. Results are averaged over 10 repetitions to obtain mean fitness values, and we simulate replicator dynamics starting from the uniform population state.
Results. Figure 5 shows how evolution transitions between two dynamical phases: an initial phase (up to time step 2000 where one group (NY) diminishes while the other two (CA,TX) remain bal- anced with similar accuracies, and a second phase (from time step ~300 in which the two remaining groups compete until one of them dominates. This illustrates possible shortcoming of static faimess definitions: At the onset, all groups are present, but disparity in accuracy is x 2%. As time progresses, the system becomes more "fair" but only because the first group has been suppressed (i.e., its relative size is < 1%). Auditing the prediction algorithm for fairness at this point in time creates an appearance of faimess. This holds only for the two groups that remain (disparity = 0.2%), and there is insufficient observed data to help identify the preceding existence of a missing group. Appendix F.4 provides additional support, exploring similar phenomena under different learning algorithms.
7 Discussion
This work introduces evolutionary prediction games, unified framework for analyzing self-selection feedback loops between learning algorithms and user populations. Our analysis reveals gap: in idealized settings, repeated training drives competitive exclusion, whereas practical constraints enable a rich variety of long-term outcomes, including stable cocxistence and mutualistic interactions.
The study of population dynamics is central to biology and ecology, a task for which population games have proven to be highly effective. We believe there is need to ask similar questions regarding the dynamics of social populations, as they become increasingly susceptible to the effects of predictive learning algorithms. The social world is of course quite distinct from the biological, but there is nonetheless much to gain from adopting an ecological perspective. One implication of our results is that, without intervention, learning might drive a population users to states which would otherwise be unfavorable. Ecologists address such problems by studying and devising tools for effective conservation. Similar ideas can be used in learning for fostering and sustaining heterogeneous user populations. Our work intends to stir discussion about such idcas within the learning community.
10

---

Acknowledgements
The authors would like to thank Mor Nitzan, Oren Kolodny, Ariel Procaccia, Fernando P. Santos, Jill-Jênn Vie, Evyatar Sabag, and Anonymous Reviewers for their insightful remarks and valuable suggestions. Nir Rosenfeld is supported by the Israel Science Foundation grant no. 278/22. Eden Saig is supported by the Israel Council for Higher Education PBC scholarship for Ph.D. students in data science.
References
[1] M Mehdi Afsar, Trafford Crump, and Behrouz Far, Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7):1-38, 2022.
[2] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limita- lions and opportunities. MIT press, 2023.
[3] Thomas Bertrand, Gillian Bartleut Bartlett-Equillnt, Kady Fischer, and Matthias Friedrich. Patient and physician preferences for non-invasive diagnostic cardiovascular imaging technologies: discrete choice experiment. Journal of Outient-Reportee Outcomes, 6 11): 2022.
[4] Haiyang Bian, Emma Li, Lu Liu, and Zhengwei Wang. The influencer copycats. PBCSF-NIFR Research Paper, 2023.
[5] Gavin Brown, Shlomi Hod, and Iden Kalemaj. Performative prediction in a stateful world. In International conference artificial intelligence and statistics, pages 6045-6061. PMLR, 2022.
[6] Micah D Carroll, Anca Dragan, Stuart Russell, and Dylan Hadficld-Menell. Estimating and penalizing induced preference shifts in recommender systems. In International Conference on Machine Learning, pages
PMLR, 2022.
[7] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[8] Alan Chan. Scoring rules for performative binary prediction. arXiv preprint arXiv:2207.02847, 2022.
[9] Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. How algorithmic confound- ing recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM conference on recommender systems, pages 224 232, 2018.
[10] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[11] Peter Chesson. Mechanisms maintenance of species diversity, Annual review of Ecology and Systematics, 31(1):343-366, 2000.
[12] Lauren Conger, Franca Hoffmann, Eric Mazumdar, and Lillian Ratliff. Strategic distribution shift of interacting agents via coupled gradient flows. Advances in Neural Information Processing Systems, 36, 2024.
[13] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 1995.
[14] Alexander D' Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, David Sculley, and Yoni Halpern Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings the 2020 Conference Fairness, Accountability, and Transparency, pages 525-534, 2020.
[15] Sarah Dean, Mihaela Curmei, Lillian Ratifff, Jamie Morgenstern, and Maryam Fazel. Emer- gent specialization from participation dynamics and multi-learner retraining. In International Conference Artificial Intelligence and Statistics, pages 343-351. PMLR, 2024.
11

---

[16] Sarah Dean, Evan Dong, Meena Jagadeesan, and Liu Leqi. Accounting for Al and users shaping one another: The role of mathematical models. Transactions on Machine Learning Research, 2025.
[17] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
[18] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 226, 2012.
[19] Yoav Freund and Robert Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103 1999.
[20] Drew Fudenberg and David K Levine. The theory of learning in games, volume 2. MIT press, 1998.
[21] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-33) 378, 2007.
[22] Usman Gohar, Zeyu Tang, Jialu Wang, Kun Zhang, Peter I. Spirtes, Yang Liu, and Lu Cheng. Long-term fairness inquiries and pursuits in machine learning: A survey of notions, methods, and challenges. Transactions on Machine Learning Research, 2025.
[23] Garrett Hardin. The competitive exclusion principle: an idea that took century to be born has implications in ecology, economics, and genetics. science, 131(3409): 1224099):12221299 192--1297, 1960.
[24] Moritz Hardt and Celestine Mendler-Dünner. Performative Prediction: Past and Future. Statisti- cal Science, 40(3):417 436, 2025. doi: 10. 121112225-SS99880
[25] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographies in repeated loss minimization. In International Conference on Machine Learning, pages
PMLR, 2018.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[27] Josef Hofbauer and Karl Sigmund. Evolutionary games and population dynamics. Cambridge university press, 1998.
[28] Guy Horowitz, Yonatan Sommer, Moran Koren, and Nir Rosenfeld. Classification under strate- gic self-selection. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 18833-18858, PMLR, 21-27 Jul 2024.
[29] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, (3):90955, 2007. doi: 10. 10.100//CSE22000.5555
[30] Yuji Ikeda. yuzie(007mplterm: 1.0.4, April 2024.
[31] Atoosa Kasirzadeh and Charles Evans. User tampering in reinforcement learning recommender systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society pages 58-69, 2023.
[32] Amir E Khandani, Adlar J Kim, and Andrew W Lo. Consumer credit-risk models via machine- learning algorithms. Journal of Banking & Finance, 34(11):2767-2787, 2010.
[33] Haruka Kiyohara, Fan Yao, and Sarah Dean. Policy design for two-sided platforms with participation dynamics. arXiv preprint arXiv:2502.01792. 2025.
[34] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management science, 70(9):6336-633 (3333366666,,001 2024.
12

---

[35] Robert D Kleinberg, Katrina Ligett, Georgios Piliouras, and Eva Tardos. Beyond the nash equilibrium barrier. In ICS, volume 20, pages 125-140, 2011.
[36] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[37] David Krueger, Tegan Maharaj, and Jan Leike. Hidden incentives for auto-induced distributional shift. arXiv preprint 09115 2020.
[38] Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. FFCV: Accelerating training by removing data bottlenecks. In Computer Vision and Pattern Recognition (CVPR), 2023.
[39] Jure Leskovec, Lada A Adamic, and Bernardo A Huberman. The dynamics of viral marketing. ACM Transactions on the Web (TWEB), 1(1):-ess, 2007.
[40] Lydia Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference Machine Learning, pages 3 3150-3158. PMLR, 2018.
[41] John Maynard Smith. Evolution and the theory of games. In Did Darwin get it right? Essays on games, sex and evolution, pages 202-215. Springer, 1982.
[42] John Maynard Smith and George R Price. The logic of animal conflict. Nature, 246(5427): 15-18, 1973.
[43] John P Miller, Juan C Perdomo, and Tijana Zmnc. Outside the echo chamber: Optimizing the performative risk. In International Conference Machine Learning, pages 7710-7720. PMLR, 2021.
[44] Patrick Alfred Pierce Moran. Random processes in genetics. In Mathematical proceedings of the cambridge philosophical society, volume 54, pages 60-71. Cambridge University Press, 1958.
[45] Caspar Oesterheld, Johannes Treutlein, Emery Cooper, and Rubi Hudson. Incentivizing honest performative predictions with proper scoring rules. In Uncertainty in Artificial Intelligence, pages 1564-1574 PMLR, 2023.
[46] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[48] Juan Perdomo, Tijana Zmic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 1509. PMLR, 2020.
[49] Reilly Raab and Yang Liu. Unintended selection: Persistent qualification rate disparities and interventions. Advances in Neural Information Processing Systems, 04-26053-26066 2021.
[50] Mitas Ray, Lillian Ratliff, Dmiitry Drusvyatskiy, and Maryam Fazel. Decision-dependent risk minimization in geometrically decaying dynamic environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8081-8088, 2022.
[51] Ralph Tyrell Rockafellar. Convex Analysis. Princeton University Press, Princeton, 1970. ISBN 9781 7781400873173 doi: db0:10015115/77814400777 9781400873173.
URL https://doi wrw.liorrr/111 10/g//10... 1515/
[52] William H Sandholm. Population games and evolutionary dynamics. MIT press, 2010.
13

---

[53] Leonard J Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783-80) 1971.
[54] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[55] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[56] Karl Sigmund and Martin Nowak. Evolutionary game theory. Current Biology, 9(14): R503-R505, 1999.
[57] Peter D Taylor and Leo B Jonker. Evolutionary stable strategies and game dynamies. Mathe- matical biosciences, 40(1 (0(1-)):145-156, 1978.
[58] Arne Traulsen, Jens Christian Claussen, and Christoph Hauert. Coevolutionary dynamics in large, but finite populations. Physical Review E, 74(1):011901 2006.
[59] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[60] Gal Vardi. On the implicit bias in deep-lcarning algorithms. Communications of the ACM, 66 (6):86-93, 2023.
[61] Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the international workshop on software fairness, pages 1-7, 2018.
[62] Yuzhe Yang. Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: a closer look at subpopulation shift. Proceedings of the 40th International Conference on Machine Learning, pages 39584-39653 2023.
[63] Tongxin Yin, Reilly Raab, Mingyan Liu, and Yang Liu. Long-term fairness with unknown dynamics. Advances in Neural Information Processing Systems, 36:551 10-55139, 2023.
[64] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[65] Hanrui Zhang, Yu Cheng, and Vincent Conitzer. Classification with few tests through self- selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5805-5812, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction reflect the paper's contributions and scope. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the main body of the paper, and a dedicated section in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Detailed proofs are provided in the paper's appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Complete code for experiment reproduction provided as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Data and code are included in the supplementary material. The experiments use synthetic and open-source datasets (CIFAR-10, MNIST).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: guwgge/des///// /Ct///eooeemseseeeeeeeee www.eiissssoooooiieoeoooooooooee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided in the body of the paper, and in a dedicated section in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Numerical experiment results are accompanied by confidence intervals, and factors of variability are discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt////euripppp/ee
Answer: [Yes]
Justification: Research conforms to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: Addressed the body of the paper, and dedicated section in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Authors of datasets are credited in the paper.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs were only used in standard way (correcting grammar, spelling errors, word choice).
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http://neurrpee http///rips.cc/oofeeeeeeeeeeeeee cc//oonferenccc/2225111111 for what should or should be described.
21

---

A Broader Impact
Our work seeks to shed light on the question: how does the deployment of learned classifier impact the long-term composition of the population of its users? Posing this question as one of equilibrium under a novel population game, our results describe the possible evolutionary outcomes under simple condi- tions (training once), optimal conditions (perfect classifiers), and realistic settings (namely empirical surrogate loss minimization). The fact that learning can influence population dynamics can have major implications on social and individual outcomes. Our work intends to illuminate this aspect of learning in social contexts, and to provide a basic and preliminary understanding of such interactions.
Scope and limitations. As the bulk of our results are theoretical in nature, naturally they rely on several assumptions. Some regard the learning process: for example, we consider only simple accuracy- maximizing algorithms that optimize over fixed-size training set sampled only from the current (and not past) data distribution. Others concern dynamics: for example, we assume no exogenous forces (i.e., which could prevent extinction), intra-group distribution shifts, or inter-group dependencies (other than those indirectly formed through the classifier). Whether our results hold also when these assumptions are relaxed an important future question.
It is also important to interpret our results appropriately and with care. For example, statements which establish 'extinction' as likely outcome apply only limiting behavior, and so are silent on trajectories, rates, or any finite timepoint. Our results also rely on users being partitioned into non-overlapping groups. In reality, group memberships are rarely exclusive (or even fixed). and the dependence of individual behavior on group outcomes (in our case, marginal group accuracy-whiee defines fitness) unlikely to be strict.
The above points should be considered when attempting to draw conclusions from our theoretical results to real learning problems. However, and despite these limitations, we believe the message we convey-which is that learning in social settings requires intervention and conservation to ensure sustainability of social groups and individuals-
broadly and beyond our framework's ' scope.
Diversity vs. homogeneity. Our results suggests that systems of the type we analyze will tend towards less diverse population compositions. Nonetheless, and as we state throughout, our results should be taken describing the tendency of population dynamics when selective pressure is the primary force, when the learning algorithm is optimal retraining, absent any interventions, and at the limiting equilibrium.
Of course, in reality, many systems are diverse. Possible explanations for why a particular system at a given point in time can sustain several groups are:
Time: dynamics have not yet reached equilibrium, or converge very slowly.
Learning: the algorithm is not optimal (e.g., uses proxy finite data), is designed to balance groups (c.g., via stecring, regularization, or robust training).
Intervention: system administrators may intentionally intervene to facilitate diversity; for example, consider how firms invest significant resources in marketing (c.g.. Meta reports investing about S2B per year²) to ensure a constant influx of users from diverse social groups
Dynamics: Other forces are at play, such as lack of alternatives, cross-group influence, rigidity of demand, non-linear heterogeneous utility, etc.
Nonetheless, an important but subtle point is that when we observe a system consisting of several groups, we do not observe other groups that could have been supported, as they may have already been driven out. Thus, what we perceive as mixed state may actually be a snapshot of a process in which "weaker" groups have already vanished, with others possibly to follow.
In biology, the question of coexistence is prevalent, despite consensus that natural selection is the primary force at play. The study of coexistence therefore focuses on how it is sustained despite natural selection. This has revealed many biological and ecological mechanisms that can facilitate coexistence, such as resource partitioning, environment variation, and stabilization via intermediates. In this work we focus on learning itself as a possible stabilizing mechanism (Sections 5,6), which we
Meta Platforms, Form 10-K Annual report, 2024, SEC Accession No. 0001326801 000017. "Advertis- ing Expense".
22

---

view as first step towards the study of coexistence in this context (as also noted by Reviewer oLX)). We also believe that prediction games in particular, and evolutionary game theory more generally, can aid future research in exposing additional mechanisms that can facilitate diversity in environments with learning.
Practical implications. Evolutionary prediction games can help designers and operators of ML systems anticipate performative population shifts, and integrate this foresight into their decision- making. One key message is that applying conventional retraining risks favoring some groups over others the long term, even if unintentionally. Evolutionary prediction games can help understand which groups are at risk, why they are at risk, and highlight possible avenues for interventions, if such are needed. One example is decide on using a stabilizing learning algorithm (such as the one we propose in Prop. 2) instead of retraining to help steer towards desired states. Another is to target the retention of at-risk groups, either by providing benefits or reducing costs (c.g. via subsidies), in order to improve their fitness. A third is to understand if and when to increase the influx of certain groups, e.g. via marketing efforts, to balance the natural tendencies of the system. We will extend the discussion to emphasize these insights.
B Further Discussion of Related Work
B.1
Performative Prediction
Stateful performative prediction. Performative prediction [48] is framework for modeling the effects of predictive models on the data distributions they are tested and trained on (see [24] for recent survey). Dynamics induced by evolutionary prediction games are special case of the stateful performative prediction framework [5, 50J, which models stateful feedback loops. The performative effects of a predictor h E H are formally modeled using a transition map D++1 = (D,,, h), where D,, D++1 E A (X x }") are the data distributions at times t, + 1, and h E H is the predictor being deployed. The fixed points of the dynamics induced by evolutionary prediction games (i.e. equilibrium states p¹), correspond to fixed point distributions as defined in [5, Definition 4], as they satisfy Dp- = TDpp- (DDpp,hpp hp-)) ), where hp- is the classifier trained on Dp- Moreover, we note that the notion of evolutionary stability is stronger than the notion of fixed point distributions, as evolutionarily stable points also remain stable under small perturbations around p"
Structural assumptions reduce effective dimensionality. Compared to the general performa- tive prediction framework, natural selection dynamics induce transition maps which are lower- dimensional, and thus more amenable for analysis. The general space of feature-label distributions is infiniteddimensional under the common assumption X' C Rd, and the space of classifiers H is high- dimensional when the hypotheses class H is complex (e.g.. deep neural networks). Thus, the general performative prediction framework allows for transition operators which are infinite-dimensional and notoriously challenging to analyze (c.g., noted by Conger et al. [12]). In contrast, in dynamics induced by evolutionary prediction games, the stateful transition operator T operates within low- dimensional subspace of all possible feature distrrbbbiss (mixtures of fixed distributions), and depends on predictors through oow-dimensional statistics (their marginal accuracy on each group).
Formally, given collection of group distributions Di...., DK E A (X' x Y). denote the space of mixtures by Amix = {E, Pse Dx p E A }. and note that Amix C A (X' x )). Any mixture D E Amix can be represented by point on the (K 1) dimmensiona simplex p E A Additionally, as performative response depends on predictor accuracy, the transition operator depends on the classifier
through a K -idimensionnal vector of accuracies acc(h) = (acc, (h).... ace (h)) E [0. 1 Therefore, transition operators induced by the evolutionary model can be viewed as lower-dimensional mappings p' = T(p, acc(h)) from a (2K- didieensional space to a (K dimiennsioaa space.
Further relations to performative risk and forecasting. The gaps between the average accuracies of equilibrium points relates the distinction between performative optimality and performative sta- bility [e.g., 43]. Additionally, convex functions over the simplex (and their sub-grad -obb-rrddeett) emerge as a key component in our analysis of oracle classifiers (Section 4), and also appears as a key component in the analysis of proper scoring rules and their performative counterparts [45, 8] through the funda- mental characterization theorem [21, 53). From this perspective, interpreting the convex potential of cracle-larner games generator of proper scoring rules is an intriguing direction for future work.
23

---

B.2 Long-Tere Term Fairness
Group fairness is often formalized through measures of group disparity, such as the difference in error rates across groups (see, e.g., [18, 2J). However, these measures may fail to capture fairness in dynamic settings. The long-term fairness literature therefore studies the long-term implications of population dynamics on algorithmic fairness; one straightforward example that if a classifier is trained to comply with fairness constraints, dynamics can cause this guarantee to erode over time [40] (See [22] for a recent survey). Our work proposes complementing (and somewhat nuanced) counterfactual perspective, suggesting that system can appear to comply with faimess definition, but only because some groups were already driven out of the system, and disparity is measured only over the groups that remain.
Several works in the long-term fairness literature also consider replicator (and replicator- dynam- ics of different kinds and variations as models for user response behavior [c.g.. 25, 63, 15], albeit not in an evolutionary game setting. Perhaps closest our approach is Hashimoto et al. [25], who study worst-group accuracy dynamics under repeated training in a mixture population that evolves via stochastic replicator-like like dynamics supplemented by a strong, balanced influx of new users that join the system regardless of its performance (parameterized by by in their model). While their stability guarantees rely on the assumption of this strong user flow, our results offer a complementary view, showing that stable coexistence may emerge even without an exogenous user flow, under standard empirical risk minimization, and without compromising worst-group accuracy (see Section 5). We also extend their ERM instability results to a wider family of behavioral dynamics, and establish global convergence guarantees, rather than local (see Section 4, and Appendix C).
B.3 Dynamics-Awark Learners
Finally, our framework models the learning algorithm as an entity which only informed by the
current state of population h ~ A(p), and in particular does not take into account the implications of classifier deployment. While still common in many practical scenarios (see, e.g., [16, 33. 34]), we also note that various practical systems already optimize for long-term objectives. Examples includes learners that reduce hidden incentives to modify the population of users [37, Sec. 7.3], adaption to
shifting user preferences [31, 6]. or directly maximizing long-term engagement via reinforcement learning (e.g.. [I]. While our focus here is on local learning rules, analyzing dynamics-awar learners from an evolutionary perspective is an intriguing direction for future inquiry.
C
Microfoundations
Natural selection dynamics appear in variety of settings. This appendix offers representative (though not exhaustive) set of examples showing how local, myopic user behaviors aggregate into population-leve dynamics that satisfy our core assumptions. Using online social platforms as a running example, we show how different behavioral patterns (e.g.. invitation, imitation, resource allocation) and different types group affiliations (e.g., social, behavioral, demographic) may give rise to natural-selection dynamics in various parts of the ecosystem. We note that similar microfoundations can be identified across a wide range of domains, such as finance, healthcare, and education.
C.1 Reproduction
Consider recommendation system based on prediction. Accurate recommendations motivate users to invite peers from their social group, while the presence of alternatives creates a risk of dropout.
Formally, assume that users from K distinct groups interact with a prediction system over time, and denote by N₂ the number of users in group l E [K]. On average, users interact with the system times per time step = 1,2, ... (e.g.. three times per day), and each interaction they receive a prediction (c.g., content relevance). If a prediction is accurate, users invite a peer from their group with probability a. Independently, a user may leave the system with probability 3 (c.g.. due to friction or better alternatives). In expectation, the size of each group is multiplied by the following factor:
No+1 = (1 + X (a. acc; (h²) - B)) NY.
24

---

In the continuous time limit, users interact with the system Ädt times per step, and dt - 0. The expected growth approaches the differential equation:
N X(a accs No
Where NNg N& is the time derivative of Nk. Denote F& X(a acc; 5), such that NA &x Note that the growth multiplier F& is time-dependent when the classifier / is retrained.
For group-level dynamics, denote by N -1 N& the total number of users, and by pk the proportion of group k. The time derivative of Pı is:
d N%
pu de N
N - N
N2
Nk
N
N
N
PP
NN=FoNN
N E FNNN
= Pk
FR
Fw
ME(K]
Thus, the aggregate dynamics of group proportions p satisfy the replicator equation [57]. which satisfies our core assumptions. The fitness function in this case is Fx(p) = λa accx(hp)) - AB.
C.2
Imitation
Consider media platform fostering content creators, whose profit depends on the extent to which their content reaches their target audience through accurate predictions. A successful approach (e.g.. content style, advertising strategy) of one creator is likely to be mimicked by others.
Formally, suppose there large population content producers aiming market some good. Each producer has a content production strategy k (e.g., short versus long videos), the price of the good is and the cost of production under strategy is C (e.g., longer videos are typically more costly to produce). A prediction algorithm observes content and makes recommendation decisions to content consumers. prediction considered accurate if the content is relevant to the consumer, and each successful prediction yields a conversion (e.g.. sale) with probability a. Denote by ~ D2 content item produced by strategy k, and its corresponding label by y. The expected profit of a producer using strategy k is:
F =r. ofP (h(x) y] Ck
At each step, content creator observes another creator's 's content (c.g., social media). If their own strategy is less successful than the one they observe, they adopt the other's strategy with probability proportional to the difference in utilities. Denote the total size of the population by N (here assumed to be large but fixed), the number of creators who adopt strategy le by NR. and the proportion of group & by ps Nr/N. Sandholm [52, Chapter 4] shows that this form of pairwise proportional imitation converges in the popullainn limit to the replicator dynamics [57], which satisfy our core assumptions.
C.3 Online resource allocation
Natural-selection dynamics between groups of users can also arise from interaction between predic- tion algorithms in multi-stage pipelines. Consider a firm that acquires users though K marketing campaigns, each targeting a different demographic. The firm uses a classifier to personalize the experience of incoming users, and each accurate prediction yields conversion with probability Campaign therefore generates expected utility:
F3 r- acc (h) ck
where the revenue per conversion, and e the average cost of user acquisition for campaign The firm has fixed advertising budget B, which it dynamically allocates across marketing campaigns.
25

---

Competitive Exclusion
Stable Mutualistic Coevistence
Unstable Mutualistic Coexistence
No Coexistence Equilibria
aock
BORA
aod a
BADE
Accrrcy
a0Cer
Only
50-50
Only B
Only
50-50
Only
Only A
50-50
Only B
Only
50-50
Only
Population state [p]
Population state (p)
Population state (p)
Population state (p)
Figure Graphical representation of two-group evolutionary prediction games (Definition 1).
Groups are denoted by (A, B). Solid lines describe expected group accuracies for each population state p. dotted line describes population accuracy accp(h), (h), dots indicate Nash equilibria. and x-axis arrows indicate evolutionary stability. Schematic plots illustrate selection of possible scenarios: (Left) Competitive exclusion. There are two stable equilibria, each dominated by a single group, with locally maximal population accuracy. Additionally, there is an unstable coexistence equilibrium with lower overall accuracy. (Center Left) Stable mutualistic coexistence. There is stable coexistence equilibrium, where the accuracy of both groups is maximized. (Center Right) Unstable mutualisitic coexistence, Coexistence has maximal population accuracy, but it is unstable. (Right) No coexistence equilibria. There is single equilibrium dominated by single group.
Denote the spend on campaign by pv B, and assuming the firm reassigns spend according to a multiplicative weights update rule [19] for dynamic allocation, we obtain:
5
Pk
Pk
Σn
3
where B > 1 is hyper-parametes of the MW algorithm. As B + 1, dynamics converge towards the replicator dynamics (cf. Equation (17)), which satisfies our core assumptions.
D Deferred Proofs
D.I Preliminaries
Distributions. The set of distribution over a set A is denoted by A (A), and the set of distributions
[K] is denoted by A A ([K) (p Σx Pk 1, Ph 2 0 vk [K]]- The support of distribution p A is denoted by support(p)
{k [K] p > 0}. We denote the space tangent to the K -imppee by TA K { E R Σx 2 0}. We denote
(1...., 1) RRK and denote
by + I 11 the orthogonal projection matrix from R&K to TAK For l [K], we denote by
e₁ E A the singleton categorical distribution supported over k. representing a mixture which only contains the distribution Dr.
Supervised learning. We denote the space of features by X' the space of labels by V the hypotheses class by H. C )xx and the loss function by L: y - R. The expected loss of hypothesis h H under population distribution D e A (X' x J) is denoted by (h) = (a..))- ((r.))-n[[(h(a) (h(())()))))))))))))(()) y)]. A supervised learning algorithm A is a possibly-stochee stochhasii mapping from a distribution D E A (X' x y)
classifier h A(D), formally A A (X' 3)) A (H).
Population state. Each group & is associated with a feature-label distribution DR E A x )). When the population state is p AK the training state is mixture distribution denoted by Dp = EkE[K] Pk PR. We use vector subscript notation to distinguish between group distributions (e.g., D and mixture distributions (e.g.. Dp). Given hypothesis h E H. the expected loss of group k denoted by (h) B(a.))D
((((())) [[(h(z)) v)], and the vector of expected group losses is denoted by E(h) = (h), (k) (h)). For E A the expected loss of h under distribution Dp is denoted by (h) = E (r,)) [((h()),))
Population games. We follow the notations of Sandholm [52], and include the key definitions here for completeness. Population games associate each group & E [K] with a fitness function Fg (p), which maps population state p A to the evolutionary fitness of group k. A Nash
26

---

Data Distributions
Replicator Dynamics Oracle Linear Classifier
Dg
PA
96%
Ohorall accuray pocy//
92%
Dg
88%
84%
De
80%
2
Figure Evolutionary dynamics of an oracle classifier in a three-group setting. (Left) Data distributions of groups (A, B, C}. (Right) Phase diagram of replicator dynamics over the probability simplex, induced by the evolutionary prediction game. Background color represents the overall accuracy accp(()p), vectors represent the flow of replicator dynamics, and markers represent fixed points, with marker color indicating stability. The dynamics have 7 fixed points; One unstable fixed point at the interior of the simplex (unstable coexistence equilibrium), three stable fixed points the vertices of the simplex (stable equilibria with single-grour dominance). and three additional unstable points at the boundaries (unstable restricted equilibria induced by the replicator dynamies).
Equilibrium (NE) population game is population state p" which satisfies support(p") S arg ma Ps(p"), and every population game admits at least one Nash equilibrium [52, Theorem 2.1.1J. A Restricted Equilibrium (RE) of population game is population state p which satisfies support(p") S arg max xuuppsupp) F& (p'), or equivalently (s(p") support pp)),
Fw for all k, k population game potential game can be represented as a gradient of some function. Formally, F A - IRK is potential game if there exists f A -> lR such that F VJ, where Vf the gradient of f over the simplex (see [52], Section 3.2 for rigorous definitions).
Competition, mutualism, and evolutionary stability. Coexistence equilibria are fixed points in which two groups more persist (support(p")) 2 2. We say that coexistence is competitive if each group's accuracy at pz lower than accuracy isolation (i.e., when it the sole group in the population). Conversely, we say that interactions are mutualistic if every group's 's accuracy at p' exceeds the accuracy it obtains in isolation, indicating that coexistence benefits all groups (see say equilibrium is evolutionarily stable if it attracts
Figure for an illustration). Additionally, nearby population states (see [52, Section 8.3] for formal definitions).
D.2 Evolutionary Dynamies
For our theoretical analysis, we assume that p evolves in time according to an ordinary differential equation of the form p = Vr(p), where Vp A - TA K a vector field tangent to the simplex, induced by the fitness function F(p), assumed to be continuous (see [52, Theorem 4.4.1))
4.1.)) I. A common Ex Fe- (p)), which corresponds the tangent vector field (p( p C (F(p) Ex' Pke Fi (p)1), where @ is the Hadamard product (see Figure 7). The replicator dynamics are a common model of evolutionary dynamics (see Appendix C, and e.g. [58, 35]), and satisfies our core assumptions. Moreover, we also emphasize that our analysis applies to any VF which satisfies the core assumptions below.
example for such dynamical system the replicator equation pa- = Pt (P)
xv
With respect to VE, we make the following assumptions:
Continuity: We assume that Vp(p) is continuous with respect p.
We assume that Vr(p) is directionally aligned with the fitness vector F(p) whenever a population not at rest, formally Vr(p) F(P) whenever Vr(p) # For detailed discussion, see [52, Section 5.2].
Positive correlation:
27

---

For correspondence to game equilibria, we assume either of the following assumptions:
Nash stationarity: Under Nash stationarity, all fixed points Vr(p") correspond Nash equilibria p" of the game F. See eq. (3) for the definition of Nash equilibrium, and [52, Section 5] for a detailed discussion.
Under imitative dynamics, agents adopt strategies based on observing and copying the behaviors of others population. See [52, Section 5.4] for a formal definition using the revision protocol formalism.
Imitative dynamics:
The replicator equation introduced in Section 3 is a special case of imitative dynamics, which satisfy our assumptions. In contrast to dynamics with Nash stationarity, imitative dynamics admit additional rest points at restricted equilibria of the population game (formally, states satisfying Fp(p") = Fiy (p") for all k, k' E support(p") )). However, we note that this does not affect our main results (Theorem and its extensions, and our constructions in Section 5), as any Nash equilibrium is also restricted equilibrium, and any non-Nash restricted equilibrium corresponds to an unstable fixed point of the imitative dynamies (see [52, Section I). See Figure for illustration of Nash and restricted equilibria in a three-group setting.
Time scales. Rates of convergence can be tuned by scaling the vector field Vp. For example, replacing VF with VF for a > >0 compresses or stretches time by a factor of a while still maintaining our core assumptions. In particular, if p' satisfies p VF(p), then q' pat is solution to = VVp(g) with the same initial conditions. Thus, our convergence and stability results hold irrespective of whether the dynamics evolve quickly or slowly.
Dynamics induced by potential games. Dynamics induced by potential games are closely related to the structure of the potential function f()). We informally state key results that relevant to setting: By [52, Theorem 3.1.3], a state p" is Nash equilibrium of and only if is an extremum of the potential funetion f. By [52, Lemma 7.1.1] potential function are Lyapunov functions, and by [52, Theorem 7.1.2], dynamics satisfying the positive correlation property converge towards Nash equílibria (restricted Nash equílibria in case of replicator dynamics). Finally, by [52, Theorem 8.2. a population state p" is a stable equilbrium and only if it is local maximizer of ((p).
D.3
Basic Claims
D.3.1 Linearity
Proposition D.1 (Expected loss is linear in p for a fixed hypothesis). For any h E H and p E AK The expected loss can be represented as a convex combination:
(p(h)
Σ Pk (h)
KE[K]
Proof. Given population state p. treat categorical random variable over population groups.
denote by k ~ p the group from which feature-label pair gets selected. Applying the law of total expectation and using the definition of (h). we obtain:
Cp (h) (ttr:)),,,((((())))
Exnpl
Σ Ph (( kE[K]
Σ kE[K]
(h)
D.3.2 Overall Accuracy Equality
Equilibria of evolutionary prediction games also satisfy a natural fairness criterion. A classifier h satisfies overall accurucy equality all groups have equal prediction accuracy [61]. In the context of natural selection, we consider this with respect to the groups currently present in the population:
28

---

Definition D.1 (Overall accuracy equality; [e.g.. 61]). Let h: : X + y, and per AK h satisfies overall prediction equality if (h) accy (h) for all k. k, support(p).
Under retraining, a different classifier / ~ A(p) is deployed at each time step. It is therefore natural to define this property with respect the learning algorithm, rather than single classifier. In the following definition, we take the expectation over the stochasticity of the sampling and learning process to representing the average over time under retraining:
Definition D.2 (Overall accuracy equality in expectation). Let p be population state, and let A(p) be a learning algorithm. satisfies overall prediction equality expectation if (hEeA(p)) (accy (h))
A(p) Jaccw (h)) for all k. K' support(p).
Proof of Proposition 1. Let p" be a Nash equilibrium of the evolutionary prediction game induced by A(p). By eq. (3), it holds that support(p") C arg max; Fs(p"), and therefore there exists some E IR such that Fs(p") = for all k E support(p"). Then by the definition of the evolutionary prediction game (Def. 1), it holds that: En~A(p) (aces (/)) EAD)A() (accg((h) for all k, k' E support(p").
Furthermore, we note that similar reasoning can be utilized to prove an analogous correspondence between overall accuracy faimess and restricted equilibria of the game.
D.4 Training Once
We begin with a characterization of evolutionary dynamics for a simple setting in which the classifier is trained only once on the initial distribution, and henceforth kept fixed. This is useful as a first step since by fixing h ho, outcomes can be attributed solely to the induced changes the population.
We will prove the following theorem:
Theorem D.1. Let A be a learning algorithm. Denote the initial population state by pd, and the initial classifier by he ~ A(p"). If ht" is deployed at every time step (i.e., irrespective of the population state p), then it holds that:
1. Competitive exclusion: p' is supported on k* E arg max, accy (h²), i.e, initially-fittes. groups.
2. Stability: p" is stable.
d ac (he) 2 0, i.e., overall accuracy weakly improves over time. 3. Accuracy: at
Definition D.3 (Train-once prediction game). Let pd E A be the population state at time O, and denote by 1t2 E H the hypothesis learned at time O. The train-once prediction game is a population gume with the following fitness fanction:
Pp: ho) = -Lx(hh")
Definition D.4 (Constant game; [52, Section 3.2.4)). A population game F A R is constant game if there exist constants a1...., aKK E IR such that Fx(p) = ag for all p AK Proposition D.2. let L be loss function, let pd A be the population state at time 0, and let ho E H be the hypothesis learned at time O. Denote by F&(p) the corresponding train-once prediction game, and denote by p' Nash equilibrium of F. It holds that:
1. support(p") S &argmin (h²)
2. Cpp (h) S Cp)
3. p² is stable.
Proof. Fix hn D In the train-once setting, the expected marginal accuracies constant function of p. and therefore Fs is a constant game by definition D.4. By [52, Proposition 3.2. FR a potential game with linear potential f(p) = -E(k²) ) By [52, Theorem 3.1.3], the set of Nash equilibria the set of local maximizers of (p)) which is a convex combination over the set arg min (h²) due to linearity. Hence:
support(p") C arg min La (h²)
k<(K]
29

---

Retraining oracle classifiers Patential function illustration
100%
assp(tp)
Accursey
a0)2(A)
70%
70%
Only A
50-50
Only
Population (p)
Figure Illustration of the proof of Theorem Pink line: The expected accuracy of fixed classifier h is linear in (Proposition D.I). Blue line: pointwise maximum over linear functions, and is therefore convex (Proposition D.3). The gradient of ac0p(1tp) over the simplex is the vector of marginal accuracies, showing that the overall accuracy is potential function for the game (Proposition D.4). Properties of the game's equilibria then follow from convexity arguments.
The overall accuracy of the oracle classifier hp is
satisfying (I). At equilibrium p", it holds that Cp- =minke[K] (x(h)). Apply proposition D.I to obtain:
(h)) = Σ (h))
Σnh
Ph min L (h9) kE(K
= min Lx (h²) WElK]
Lp- (h²)
satisfying (2).
Finally, observe that each p" stable as maximizer of the linear potential function [52, Theorem 8.2.1].
Proof of Theorem D.I. Theorem D.1 is special case of Proposition D.2 for the loss function: (h) (h).
D.5 Retraining
Definition D.5 (Retraining prediction game with oracle predictor). Let p A let L. be loss function, and let H be a hypothesis class. Denote hp E arg miggin Lp(h), and assume ties are broken in a consistent way. The retraining prediction game with an optimal predictor a population game with the following payoff function:
FL(P) (hpp))
Definition D.6 (Optimal loss function 1* (p)). Given hypotheses H and population state p E A the optimal loss at p is:
1' (p) min (h)
KCR
Proposition D.3 (() concave). The optimal loss [- (p) is a concave function of p.
Proof. By definition of (p):
1" (p) min (h)
NEW
From proposition D.I, each (h) is linear in p for all H. and in particular concave. Hence, [" (p) is wise minimum of concave functions, and is therefore concave.
30

---

Definition D.7 (Subgradient; [e.g., 51, Section 23]). Let f : AK - B. be convex function over the simplex. The vector TA subgradient of point p for all p' holds that:
f()) + v (p' p) S f(p')
Where f(p) + u. (p'-p))
p) is the tangent plane at point p. Definition D.8 (Subdifferentiaa),
Let f A R be convex function over the simplex. The subdifferential of f at p is set of all subgradients at point p. denoted by @f()).
Proposition D.4. Let hip E arg min Lp(h) be an optimal predictor. The projection of the loss vector of h at point p is subgradient of (p):
ΦL(h)) E
Proof. Let hp E arg min (h),, and p.p' E A K We need show that -1) E(hp) (p' -p) (p').
By proposition D.I. it holds that Lp (hpp) (hp)) By algebraic manipulation obtain:
Cp (1p) [(htp).
= -(1ip) . - -[(hp) p++((pp)))
(5)
(p)
=1) E(hpp) (p' -p) p))
As p' E TAK is not affected by the orthogonal projection R - TAK holds that:
+ (p' p) p' P
(6)
Since an orthogonal projection, it holds that xx addition, for any vector v it holds that:
vt Th = ($v))
Combining equations (5, 6, 7), obtain:
&C (p) x((h)) (p'-p) By proposition D.3, it holds that Ep' (lip) 2 (p'),. and therefore:
(p) x (p' p) 2 ("(p')
I' (p) is concave, and therefore negation is convex. Multiplying both sides by -1, we obtain that - + C(p) is a subgradient of -1" at point as required.
Proposition D.5. Assume (p) is disferentiable, and let hp E arg min (h).. The gradient of (p) satisfies:
V -
Proof. Any convex differentiable function f satisfies:
0f(p) (V)(
By proposition D.4, holds that -bp()))) a (0t and therefore (-1(()) - ((hp))
Lemma D.I. Let H C yx and let L: y2 + R. For K denote hip arg min H C (h). If
(hp) is a continuously differentiable function of p, then game Fs Fx(p) = LL (hpp a potential game.
Proof. Take the function f(p) (p) By proposition D.5, the gradient of f(p) satisfies:
V <f(p) C(hp) F(p)
And therefore F' is potential game.
31

---

We say that set of groups is oracle-equivalent if they share common oracle classifier:
Definition D.9 (Oracle-equivalent groups), A sel of groups K* C K is oracle-equivalent if there exists E OKEK. arg min CD. (h) which satisfies Lk, (h) = Cx, for all k1, k2 K'.
Proposition D.6. Two groups k. k' E [K] are oracle-equivalent any only if the expected optimal loss of their convex combination |* ((1 - a)ex + dew) is constant for all a E [0, 1].
Proof. If the groups oracle-equivalent, then by Definition D.9 there exists
E arg min argmin LD, (h) n arg min (h) NEHt
kCH
which satisfies LD. (h") ) = Lpw (h*) const for some constant. Since h* is optimal for each group separately, it is also optimal for any convex combination of the groups (1 Dx + aDw, and the optimal loss therefore satisfies:
1n ((1 a.ee + tew) = (1 - CD (h*) + aLD. (h²) = const
Conversely, assume that k, k' are not oracle-equivalent. We consider two cases:
1. If arg minnen LD, n arg mingen Lp. (h) = 0. then let p 0.5e, + 0.5ep. By linearity of expectation, for any h H it holds that:
Ln, (h) 0.5C (h) + p. (h)
By the assumption, no h E 7l simultaneously minimizes both terms. Therefore, the optimal loss at p must satisfy either (p) > (ex) or (p) > (e(ew), and therefore 1* ((1 e) + aew) is not constant.
2. Otherwise, if there exists common optimizer ht E arg min LD. nh) min Lpp (h) but LDD (h*) # LDD,, (h"), then 1" ((1 - ar)e% oew) is not constant when comparing the values corresponding to 0 and =
Proposition D.7. The local minimizers of the optimal expected loss function 1" are convex combina- tions of oracle-equivalent groups.
Proof. By Proposition D.3, the optimal loss 1" is concave over the simplex A K and therefore its minimizers convex combinations of simplex vertices for which the function is constant. By Proposition D.6, such combinations correspond to sets of oracle-equivalen groups.
Proposition D.8. When each group has a distinct optimal classifier, the optimal loss function [" (p) only has local minimizers vertices of the simplex.
Proof. When each group has a distinct optimal classifier, the only sets of oracle-equivalent groups the singleton sets (1) (1),...; (K). The claim then follows from Proposition D.7.
Proof of Theorem 1. By applying Lemma D.1 on the loss function C+(h) - acc+ (h), we obtain that F(p) is a potential game with potential function f(p) accp(hp). By Proposition D.3, we obtain that f(p) is convex over the simplex. When no two groups share optimal classifier Proposition D.8 holds, and local maximizers of I exist and are located the vertices of the simplex. By [52, Theorem 8.2.11 I], maximizers of I (p) correspond to evolutionarily stable Nash equilibria of F(p), and therefore any stable equilibrium p" satisfies |support(p*)) = 1. Additionally, we note that coexistence equilibria may exist as f(p) may have local minimizers. This proves claims 2,3,4 (Stability, Exclusion, Coexistence) in the statement the theorem. Finally, for claim (Accuracy), by [52, Lemma 7.1.1] the potential function f(p) is a Lyapunov function for any evolutionary dynamic which satisfies the positive correlation property, and therefore d
di f(p) ai accp > 0.
D
32

---

Data Distributions
Replicator Dynamics With Equivalent Groups
Dg
PA
-66
Onerall accurcy pocy//y
Da - Da
92%
De
90%
21
Figure 9: Evolutionary dynamics of an oracle classifier in three-group setting with equivalent
groups (Appendix D.5.1). (Left) Data distributions of groups {A, B, C), where D3 DD Groups A, B are oracle-equivalent, and group C' is not oracle-equivalent A or (Definition D.9). (Right) Phase diagram of the corresponding replicator dynamics over the probability simplex (cf. Figure 7). The dynamics has three sets of fixed points, corresponding to Theorem D.2: The convex hull of the singleton distributions corresponding to A, B (stable), the singleton distribution of group C' (stable), and the convex hull of p x (0.45, O, 0.55) and p' zu (0, 0.45, 0.55) (unstable).
D.5.1 Equivalent groups
Share the same optimal classifier and achieve the same optimal accuracy (formally satisfying Defini- tion D.9), then extended version of Theorem holds:
Theorem D.2 (Oracle classifier with equivalent groups; Extension of Theorem 1). Let H be hypothesis class, and denote by Aopt the oracle learning algorithm with respect to H. Assume that at each population state p the oracle classifer hip (Dp)(D)) is deployed. Then it holds that:
1. Accuracy: Overall accuracy increases over time, at (hccp(pp) 2 0.
2. Stability: stable equilibrium always exists, and there can be multiple such equilibria.
3. Competitive exclusion: For all stable equilibria, support(p") is oracle-equivalent (Def. D.9).
4. Coexistence: Equilibria with non-equivalent groups may exist, but are unstable.
Proof. In analogy to the proof of Theorem 1. The potential function remains f(p) acc, (hp). and so does the relation between extrema of the potential and equilibria of the dynamics. However, as groups possibly equivalent, by Proposition D.7 obtain that f(p) may have additional weak maximizers convex combinations of oraclc-equivalent groups.
D.5.2 Heterogeneous fitmess
groups See Theorem D.3 below. For K = 2, we show that a modified version of the theorem holds for any pair of continuous, strictly increasing retention functions. For K 2, similar conclusion follows under positive affine retention functions v4 (x) = ax by with by 2 0. In each case, we prove the extended theorem by constructing transformed potential function and showing that it retains the properties required for convergence and equilibrium structure guarantees.
Here extend Theorem 1 settings where retention functions 24 vary
Continuous inereasing retention functions (K = 2). Consider a two-group setting, and denote [K] [A, B}. Let Aoot be an oracle classification algorithm. Denote (p. p). and denote hp ~ oop ((p, 1 p)).
Proposition D.9. For any oracle algorithm, it holds that LA (hp)) decreasing in p. and L (h)) increasing in p.
33

---

Proof. Assume by contradiction that Ch(hp) is not decreasing. and therefore there exist p' > p such that Cx(hp) Cx(hp) ). The proof relies on swapping argument. Denote:
a Ca(hp)
a' Ca(he)
= (k(h)))
6' = LB(hp'
By Proposition D.I, the average for loss for any fixed classifier is convex combination of marginal loss functions. hp is optimal for and therefore:
pa (1 p)6 pa + (1 (1-D)
(bpp))
=pplly))
Similarly, by is optimal for p', and therefore:
p'a' (1-11110
1')6' S Sp'a+ (1 (1-1'}6
Adding the two incqualities:
- p)'a' - a) S (p' pD - b)
Since p' - -p > 0, we divide by (p' - p) to obtain:
- a a
a' by the contradiction assumption, and therefore by holds that V/ > But then: p'a (1-
(1 (1-p')6'
In contradiction the optimality of hy. Therefore. it must hold that a'. Similarly, assume by contradiction that and obtain a similar contradiction. Figure 8 provides additional intuition. N
Proposition D.10 (Potential function of a two-group population game; [52, Example 3.2.3]). Any two-group population game is potential game, and the following is potential function:
f((p11-))) = -[ (F)((t--=))=FF((e1111 Fz(z),
x))) dæ
(8)
Proposition D.11. Let f(p) be a potential function for a two-group evolutionary prediction game induced by an oracle classifier with increasing retention functions. Then f(p) doesn' 't have local maximizers in the interior of the simplex.
Proof. Denote the transformed fitness function by (A( (acc and similarly - FF 1-p)), and denote by f((p 1 - p)) the potential function for the transformed game, defined by Equation (8). By Proposition D.9, it holds that (p. p, 1 p)) is increasing. (p((p. 1-p)) is decreasing, and therefore function of p. By Equation (8), the potential f is decreasing if g(p) < for all p. increasing g(p) 2 for all p. and unimodal (decreasing and then increasing) if g(p) changes signs. Note that g()) may change signs at most once due to monotonicity. In all of these cases, gp)) doesn't have a local maximizer at the interior of the simplex.
denote Fg ((p, 1 - p)) VB (acea (hp))). Let g(p) F( 1-pp p))
g(p) is monotonically increasing
Positive affine retention functions with group-wise shift (K 2 2). Let vx(x) az by for a, by 2 0. We show that the game admits a transformed potential function:
Proposition D.12. Let Aop be an oracle classification algorithm, and let f(p) be the potential function for the game F& (p) acc4 (hp). Then the game F2(p) uy (acc)(hpp) = cacck (hpp bb admits the potential function:
f(p)= = - b.p
where b (bi....,bbp bk) R
34

---

Proof. First, consider the game r/() = a (p). By linearity of the gradient operation, the population game F(o)(p) (p) is potential game with potential f())() af(p). Then, consider the game F (p) = -bb.- The game F is a constant game, and therefore admits the linear potential function f = --b- (see [52, Proposition 3.2.13].. Finally, by [52, Section 3.2.4], the game F(p) F(1) (p) + (p) is potential game with the potential function:
f(p) f + f (p) af(p) - b.
required.
Proposition D.13. Let (p) be potential function for an evolutionary prediction game induced by an oracle classifier and positive affine retention functions with group-wise shift. Then f(p) doesn' I have local maximizers in the interior of the simplex.
Proof. By the proof of Theorem I, F(p) is potential game with potential function f(p) = accp (hp). By Proposition D.3 we obtain that f()) is convex over the simplex. Multiplying a convex function by a positive constant and adding a linear function maintains convexity, and therefore f()) is convex as well, and in particular does not have local maximizers in the interior of the simplex.
Theorem under heterogeneous retention. both cases analyzed above, we identify modified potential function which does not have local minimizers in the interior of the simplex (Proposi- tions D.11 and 1.13). We now proceed to state modified version of the central theorem:
Theorem D.3 (Oracle classifier with heterogencous retention; Extension of Theorem 1). Let H. be a hypothesis class, and denote by Aopt the oracle learning algorithm with respect to H. For K 2. assume that retention functions Vą are monotone increasing, and for K >> assume that V/ are positive affine with group-wise shift. Assume that at each population state p the oracle classifier hp A (Dp) is deployed and the fitness of group k is Fs(p) V/ (acc; (hp)). Then it holds that:
1. Welfare: Average population welfare increases over time: d Ex PaVK (accs (hip) 2 0. dt 2. Stability: stable equilibrium always exists, and there can be multiple such equilibria.
3. Competitive exclusion: For all stable equilibria, (support(p")) =
4. Coexistence: Equilibria with [support(p") 2 2 may exist, but are unstable.
Proof. In analogy to the proof of Theorem (see Appendix D.5). The corresponding evolutionary prediction games are potential games, with Nash equilibria corresponding to local equilibria of the potential function, and stable equilibria corresponding to local maximizers of the potential function. By Propositions D. and D.13, potential functions for both families of retention functions don't t have local maximizers at the interior of the simplex, implying claims (2,3,4).
For claim (1), we consider two cases, according to the family of retention functions under considera- tion. For positive affine functions, the transformed potential function is f(p) = af() b p,, where f(p) is the potential function of the original game (with (x) = z). Rearranging the summations:
f(p) (ccp(hp)
-b. .b.
= Ev pancou(hp)
Σ Pk (a acce((pp)) bs)
Σ Puvk (kp))()p))
and claim (1) follows from the fact that f(p) is a Lyapunov function. For monotone increasing functions, assume without loss of generality that (p) 2 (p).). Then from the positive correlation
assumption PA -PB 0. From Proposition D.9 obtain that acc (hp) is increasing and
35

---

accB (hp) is decreasing as function of PA. Combining the observations, we obtain:
d dr
Σ
PEuA (accx
de (PAVA (ace (hpp) + PBиB (acc (hp)))
k
- (F((P) - (n(
((())
2 without loss of generality
+ PA Opa (aCCAA (hp)) PA
by Proposition D.9
+ pn 0pΛ (accn (1p))
PA)
0 Proposition D.9
And thus d PEPEE (acc)(hp)) 2 required.
D.6 Soft-SVM
Definition D.10 (Soft-SVM; (e.g.. 54, Section 15.2)). Lel ((ri, yc) {i be a training set, and let A > O. The Soft-SVM learning algorithm oulputs linear classifier h(x)) = sign(w" x + b) such that (w* bt) are minimizers of the regularized hinge loss:
(w" b') arg min À ||o||
max {0, 1 yi(wr; + b))
D.6.1 Construction
Consider two-group binary classification setting (K = 2, = (-1,1}). Denote the two groups by [K] = {A, B}. Denote by Triangular(a, b, c) the triangular distribution with parameters (a, b, c) and probability density function:
2(x-a) (b-a)ee-) 2(6-z) (b- a)(bcc)
x E [a, c) x E |c. b] otherwise
PP Pew/Triangguar(aabellll
Let a E [0.5, 1) be a population balance parameter. Define the data distributions:
Xa ~ Triangular(11,0, 1) + (1 - a) Triangular (0, 1) = (1 'rrinnullar-11, 0, 0) + a Triangular(O, 1,
XØ x
x 0 x<0
Y|X
DA is the distribution of tuples (XA, (XA)) and DB is defined correspondingly as DB
(XB, Y|XB). The distributions are illustrated Figure (Left).
D.6.2 Stable Mutualistic Coexistence
We show that the Soft-SVM algorithm can induce a stable, beneficial coexistence. We will consider learning in the population limit - oc, where the loss minimization objective (Definition D.10) converges towards its expected value:
Definition D.11 (Soft-SVM in the population limit). Denote the probability density function of the data by fa, v). In the population limit n - oc, the Soft-SVM classifier (w",, b²) is a minimizer of the expected regularized hinge loss:
(w" b²) arg min L(w, b)
w.b
Where:
I(w, b) A ||wm||2 E f(a, v) max (0, 1 y(wr b)
36

---

Proposition D.14. For mne-iimensional Soft-SVM classification problem, denote and feature-laber distribution by f. and denote by L(w, b) the regularized hinge loss over the population. holds that:
@l(w.b) Ob
f(z, y = -1)dx - -
f(x, 1)dæ
(10)
ww+tb21
wx+ds1
Proof. The population hinge loss a ID classifier:
L(w, b) xw2 + Σ / f(x, max {0, y(ur + bb)
First, calculate the derivative
OL. 86"
ÖL(w,b) b)
Σ
/
f(r, v) max f0, 1 y(ux + b))
ab
Ob
ZER
By the Leibniz integral rule, and excluding the single non-differential point from the integral:
E
Σ
f(x, y) a ob
max {0, 1 - y(wr + })
V HER (ymor /
Σ
(xx,))--yle
s(wz+b)5
Expanding the summation over y yields:
=f wx+t2-1
[a,y -1)da
[x, = 1 1)dr
|wxtbgi
As required.
Proof of Theorem 2. Assume the Soft-SVM algorithm trains on a large dataset sampled from Dp- Denote p (1 p. p)p and denote the probability density functions of groups {A, B} by fa (), fx y). respectively. Since the data is separable and labels are positively correlated with the feature v, it holds that w" >> Assume that regularization parameter 7 is large enough such that w < 1. From Proposition D.14, it holds that:
aL(w,b) b)
-
f(xv== -1)d
f(x, 1)dæ
ab
Jurg1
b=
When E (0, 1]. observe that the distribution defined in Appendix D.6.1 is suppported on [-1, and therefore both integrals cover the whole distribution. We can therefore write:
@L(w, b)
(a.))~D,lV = -1] -1]] (((,)))D,,,V [1
ab
By construction, for group holds that
P( ly = 1] P(o.))D 1] = a 12--(111 - a) = -(1 - 2a)
and similarly for group B:
P P( -1] -(xxy)
2a
Therefore, for Dp. it holds that:
@L(w,br b)
=p(1 - 20) pppp1 2xx
ab
6=0
The state p (0.5, 0.5) corresponds to p-p = 0.5. For this state, holds that aL(w ab
= 0, and
6=0
therefore b* 0. Since the data separable at = 0, any classifier with w* and b 0 achieves perfect accuracy, and therefore the state p (0.5, 0.5) is beneficial for both groups
37

---

Data Distributions
Soft-SVM (A 10)
Soft-SVM (A = 100)
Soft-SVM (A = 1000)
100%
nocurar
99%
68%
80%
96%
60%
96%
90%
409
-1
Only A
50.50
Only
Only
50-30
Only 2
Orly A1
50.50
Only
Populaation (1))
Populction state
Population (P)
Figure 10:
Varying regularization strength for Soft-SVM classifiers. (Left) Data distributions. (Center Left) Evolutionary prediction game induced by Soft-SVM classifier with À 10. The game has two stable equilibria at the boundaries, and an unstable coexistence equilibrium. (Center Right) Game induced by Soft-SVM classifier with À 100. The game has stable coexistence equilibrium. (Right) Game induced by a Soft-SVM classifier with = 1000. The game has five equilibria: Two stable single-group group equilibria, one stable beneficial coexistence equilibrium (0.5, 0.5), and two unstable coexistence equilibria at p Rs (0.2, 0.8) and z (0.8, 0.2).
To prove stability, we take an additional partial derivative by p:
8 [(w, b)
= 2(1 - 2a)
Op3b
6=0.p00.5
And 0.5, it holds that ((w.b) Opeb
0. Let >> O which is sufficiently small. For
b=0.p=0.5
p' = 0.5 + it holds that @L(w.b)
O, and since L(w, b) is convex, the optimal b" corresponding to p' satisfies b* > 0, and the decision boundary is smaller than O. Assume that the deviation is By definition, in a sufficiently
b=0
sufficiently small such that the new b" is also in the neighborhood small neighborhood of zero, it holds by construction for each group:
0) =1 1
(ccp()tp)) acc ~0.b*
fA 1)dz
1-b0 /w0 o
[ /wo .0]
accp(hp)) acc (w* 0,0² > 0) =
Hence, for a sufficiently small deviation p' = p + holds by the alignment of the triangle distributions that acca (hp')) aco Applying similar argument to opposite deviation p" = 0.5 shows that an opposite relation holds, and therefore the state (0.5, 0.5) is also stable.
D.6.3 Bifurcations Induced by Regularization
In Figure 10, we vary the regularization strength for the distributions specified in Appendix D.6. with imbalance parameter 0.75. We observe that the system transition between three topologically- distinct phases: For weak regularization (A = 10), coexistence is beneficial but not stable. For intermediate regularization
100), coexistence is beneficial and stable. Interestingly, for strong regularization (A = 1000), the game has five equilibria Two stable single-gron grgee- equilibria, one stable coexistence equilibia, and two unstable coexistence equilibria.
D.7 N-NN Classifiers With Label Noise
Another practical approach to loss minimization is to use a class of models that fit the training data perfectly and then interpolate. Notable examples include the nearest-neighhor algorithm, and over-parameterized neural networks (e.g.. 64). Interpolation leads to perfect accuracy on the training set by definition, but often at the price of sensitivity to noise; as concrete example, consider how
1-NN predictions change with the addition single point. The problem is that even in the limit, I-NN cannot express the true p(y]z), and remains sensitive label noise. Our next result shows this can lead to coexistence:
Theorem D.4. For the I-NN learning algorithm, there exist noisy-label data distributions that induce an evolutionary prediction game with stable coexistence.
38

---

Formal proof in Appendix D.7.2. The construction is illustrated in Figure 3 (Center). and described formally Appendix D.7.1. Informally, there are two groups defined symmetrically. Each group is composed of majority and minority classes, and the majority class has label noise. For each group alone, the I-NN algorithm makes perfect predictions on its minority class, and imperfect predictions on the majority class. Since groups are defined symmetrically, training on one group alone leads to better accuracy for the other-leadinn to stable coexistence. Moreover, Lemma D.2 shows that for k-NN with k 2 3, stable coexistence can also be beneficial for both groups.
Definition D.12 C-Neaarst-Neighhorss e.g. [54, Section 19.1]. Let k be an odd positive integer. Given training set S (X x ))) and a feature vector I, denote the nearest neighbors of the training set by N (z). The k-NN classifier returns the majority label y among the members of (z).
D.7.1 Construction
Consider a two-group binary classification setting (K = 2 (-1,1). Denote the two groups by [K] {A, B), and let > 0, B 2 0.5. The data distributions for the two groups are defined symmetrically:
Each group is comprised of a majority subgroup and minority subgroup. The proportion of the majority subgroup / 2 0.5.
One subgroup is initially contains positively-labelee data, denoted by Doss E (X x {1}). Conversely, the other subgroup is initially comprised of data with negative labels, and denoted by Dneg E A (X' x {-1}). The positive subgroup is the majority of group A, and the negative subgroup is majority of group B.
For each group, the labels in the majority subgroup are flipped with probability a. For any data distribution D E A (X' x ))), we denote its noisy version by Da
It is assumed that the positive and negative data distribution have bounded support, and are sufficiently far apart such that the nearest neighbor of any sample is from the same group.
Overall we have:
DA = B8
pos + (1 - -33)nec
DB (1 B)D pas + -3 neg
The construction is illustrated in Figure 3 (Center).
D.7.2 Stable Coexistence
Proposition D.15 (Convex combinations). Let p (1 p.P) E A ((A, and denote p - 2p8 + 3. It holds that:
p(- m)
Dp = 7Dos + (1 -
(1- D)
Proof. For any mixture coefficient g [0, 1], the label-flipped datasets satisfy:
(1 - D + +D = ()-a)a
(11)
The convex combination of the mixture coefficients of corresponding to Dpos and Da Doss is given by:
p1 + p(1 - B) = p - 2p3 = ?
(12)
Plugging eq. (12) into eq. (11), we obtain that the corresponding @ for the positive mixture is:
33 p)
Opos
(1 - p)B + p(1 B3)
and for the negative mixture:
P8
3p
Gneg
p3 + (1 p) (3
Combining the two equations above yields the result.
39

---

Proposition D.16 (Reflection Reelection-exchange symmetry), pet (1 p.p), and let p' =p,11 p). For the prediction setting defined above and for any classifier ht trained on dataset D holds that:
Bs-.p,laccp.l
D JaccD, (h"))
Proof. From symmetry of the definition. Intuitively, BD lacc (h*) can be obtained by reflect- ing Bs-D,laccp, (h*)] across the axis.
Proposition D.17 (Expected prediction of k-NN), Let D2 (X) be distribution over features, let a 0 be noise parameter, and let k be odd number greater or equal 1. Denote the label distribution by D = 2Bernouli((t). and denote the joint feature-label distribution by
D = D& @ Do E A (X' x x)). Lel h : X + {-1,1} be a b-Nearest-Neighhors (k-NN) classifier trained on dataset S D" with n 2 and let X. It holds that:
D [h(x]] +2)((a))
Where (1)(0) the cumulative distribution function (CDF) of a Binomial (k: oi) random variable, taken [
k:
(x(aa) Pr (Binomial (k, a) I
2
Proof. Denote the k-NN training set by D any x E X, denote its k nearest neighbors by ((x) E (X' y)" Denote by Nneg (x) the number of neighbors with negative labels:
Noeg
H(<','') E ((x) v = =-11
Since features and labels in D are assumed to be independent, the number of neighbors with negative labels is binomial random variable:
Negg (x) Binomial a
As a k-NN classifier predicts the label according to the majority label in [x),, the expectation value of the label of æ, taken over the randomness of the training set, depends the cumulative distribution function of Nneg(
Es- [h(x)] = -1 -1+ 2Pr ( 1)
k
--1+20 2Pr (Nnop(x) S
2
And using the definition of dh (ax), obtain:
ES-D= [M(x)] --1 22px(a)
Lemma D.2 (Expected accuracy). For the distributions D DB defined above, let p (1- p. p) (IA,)) and let be an odd integer. Denote by hep the N-Nearest-Neighhor (k-NN) classifier trained data sampled from Dp- Assume that the training set contains feature vectors from both subgroups, and that the supports of the feature vector distributions in the datasets D Dposs Dueg are sufficiently far apart. The expected accuracy of hp with respect to D is:
1 (1-1-200 -1+ 2
8(1p) Ct
Es- (acon, (h²)) =B
2
(13)
1 1 (-1) 11 20h
3p
I-
+ (1-B)
Where óv(a) the cumulative distribution function of Binomial (k, a) variable, taken at
40

---

K-NN with &
K-NN with &
K-NN with
80.0%5
82.0%
BECA
02.5%
Acainay
78.0%
netp
Acoiraty
81.0%
02.0%
80.0%
79.0%
Only
50.50
Only
50:50
Only
Population state (p)
Population state (p)
Population state (p)
Figure 11:
k-NN accuracy for the construction specified in Appendix D.7.1. The plots show alignment between the theoretical curves (Lemma D.2) and empirical simulations (mean accuracies represented by dots). For 3, the game has stable beneficial coexistence.
Proof. Denote the training set by S {1r; v:) Dp. and denote the learned I-NN classifier by hp. For any X. denote its nearest neighbor the training set by i" (x) arg min ((x,x;)). The expected accuracy of hp with respect to D is given by:
(Dr((x)
DD aaccp (h)) = Ds
2
Since the supports of the subgroups originates from the same subgroup as æ. From this we obtain that and (z) are independent given subgroup from which (r, y) was sampled (E {pos, neg)). Thus, for noisy data from the positive subgroup (Dp p. we have:
sufficiently far apart, the nearest neighbor i (x)) always
Es /acc (hi)
= E Dp. (x.))~ pax
+ 9V-
2
1 + Ely] [3)
2
+ (1 22x)
(-1+ 20h
[(1-pp)
2
Where the identity [ly] (1 20) is given by the definition of De the definition of y is given by Proposition D.15, and the identity E (()) Proposition D.17.
(-1 20h ((2)))a) is given by
Similarly, for clean data from the negative subgroup (D (Dneg), obtain:
yil (w)
Eg [acC (h))
E ES (t..))
2
1 + Ely|E] [w)
2
1 (-1) (1 (1-20) 2Φhk
Bp
I-7
2
and jointly:
Es-ny (acc (hi) -3 acc pe.
(h)] + (1 B) [ Dacg. (h)]
And finally, plugging the expected accuracy values calculated above yields the desired result.
Proposition D.18. For = 1, prediction game induced by 1-NN classifier trained dataset with label noise has the following properties:
1. stable coexistence equilibrium exists if and only if (0, (0,1-1) 1/3 2/ Ifa stable coexistence equilibrium exists, then it exists the uniform population state p (0.5, 0.5).
41

---

data distributions induce 2. The overall welfare at the coexistence state p (0.5, 0.5) is lower than the overall welfare extinction states p E {[1,0), (0, 1)).
Proof. By applying Lemma D.2. When k = 1, the d% function satisfies:
(a) = Pr (Binomial(I,a) I 0) = -
Plugging into eq. (13) and using the symmetry argument in Proposition D.16, we obtain:
a (hp=(1.0)) acCB (Mp=(0.)) 2a.((a - 1)
acc (1p= acca (1p-(0.1))
-
accA (k5=(0..0.55) aceB ( (1p=(0.6.00))
-28B(a3 - 1)
For (1), stable equilibrium exists if:
acca ()p=(1.0)) aceB (kp=(1.)))
Plugging in the values calculated above:
2a - -
Which is satisfied when a E (0, 23
For (2), we note that B < 1, and therefore:
accA ( (07=(1.0)) accA (0..00.....5)) = =00(a a3)) a² -3) 0
Proof of Theorem D.4. By Proposition D.18, using e.g. (a. = (0.2,0.8). We observe that the
conditions of the proposition are satisfied for this choice of constants, as a = 0.2 < 0.375 1
=1 28'
2.0.8
D.8 Hard-SVM With Finite Data
All practical learning algorithmss must rely on finite sample of data points for training a classifier. Learnability (in the PAC sense) implies that as data size grows, the sample becomes increasingly representative of the distribution. But the rate at which variance diminishes need not be equally across all regions of distribution. In sense, this means that one region can be learned 'faster' (i.e., with less data) than another. Our final example exploits this to enable coexistence.
Theorem D.5. For the Hard-SVM algorithm, there exist an evolutionary prediction game with beneficial coexistence.
Proof in Appendix D.8.2. The construction is illustrated in Figure (Right), and described formally in Appendix 1.8.1. Informally, the key property we leverage here is asymmetric class variance. In each group, one class has high variance, and the other class has low variance. When the dataset of size n, this creates bias against the high-variance class, and thus convergence towards the optimal classifier at rate O(n 1). When the two groups coexist, this bias is balanced, and the rate of convergence is exponential leading to coexistence that is beneficial, but unstable.
Definition D.13 (Hard-SVM) e.g. [54, Section 15.1]). Given sepearly-eeeraabl training set (x1,,)))" v (Rd x (11, the Hard-SVM algorithm outputs linear classifier sgn(w" x 6*) which maximizes the margin of the decision hyperplane:
(w" b²) = arg max min yı b)
(mr.)::lw]]=1
Remark D.1 (One-dimensional HVrd-SVM). For one dimensional data, denote by x the minimal I with a positive label in the training set by I min; {Iy ys = 1]. Similarly, denote by x the maximal with negative label in the training set by Imax max, {z, -1). When tii 2 Tuass the Hard-SVM decision margin is given by:
Tmax + 2 min
Emargin
(14)
42

---

D.8.1 Construction
Let > 0, and denote by 5(x a) the Dirac delta distribution centered around x: =
X ~ 0.5 (Uniform ((0. 1J) + e) 0.50 -E)
XB Xx 1 X|X
X
-1 X <<
DA is the distribution of tuples (XA, Y|x)), and Dn is defined correspondingly as DB =
(Xn, X|) The distributions illustrated in Figure (Right).
D.8.2 Mutualistic Coexistence
Proposition D.19. For a given positive integer N, Let n Binomial (N, 2). let Il, Ii,..., ~ Uniform ((0, 1]), and let Tmin minke(n) 2t. Then for N 2 15, holds that:
E Miin
N
Proof. Let n Binomial (N, By Hoeffding's inequality. it holds that:
2r S e
Pn P1
N
And for t we obtain:
3N
P
n
2
(15)
4
For any fixed n, let 21.... Fy Uniform ((0, 1]), and denote the first order statistic among {z,} by Tmin. = minke[[] ø4. As each z; is uniform random variable over the unit interval, the first order statistic admits beta distribution:
Tmin, Beta(1, n)
and therefore:
1
E [Tmin, n +
(16)
Now consider the compound variable Tmin = Tmin, where n ~ Binomial ()) Applying the law of total expectation:
E [min]
E, [ Tmin n]
E,, E [min
By eq. (16) we obtain the expectation of Pmin..
En 1
F Pwin
n ++
eq. (15) gives lower bound on the expectation:
3.V
n2 3N
E
[mnin|
1
c
+0.e
3N
+1
Then for N 2 15 it holds that:
F
Fmin
Z
As required.
43

---

Proposition D.20. For the distribution defined above, assume the Hard-SVM classifier is trained on dataset of size 21, and let ε E (0, ). It holds that:
1. When the classifier is trained on for p (1,
1
Blaccp(( (h)] = Eaaacc (h)) - 8n 2. When the classifier is trained for p (0, 1):
Elacc = EE [acc (h)] < 8n
3. When the classifier is trained on for p = (0.5, 0.5):
accp (h.,) = EElacc (h)) Elaccy ( [n]] 2 - -2(1-21)) )
4. Coexistence is beneficial.
Proof. For case (1), the Hard-SVM algorithm trains on data sampled from D Denote the size of the dataset by n and the corresponding Hard-SVM classfier by hn. We note that h, is random variable depending on the dataset. For dataset {z y:) D%, Since the problem one-dimensional,
the Hard-SVN decision margin is given by eq. (14):
max + x
min -
#margin
> 0
2
And the expected accuracy of each group is given by:
E (ha))a(hn) w0] = 2 2
+ (1 Tmargin + E) = x min
1
E
+
4
4
By Proposition D.19 and for n 2 15, it holds that E E[Fmargia]
2 + E, and therefore:
1
1
Elacca(h.) (hx) | w S
4n
8n
8n
Case (2) follows from symmetry.
For case (3), denote by w the bad event in which the training set ( v:) DØ does not contain I; E {-e,=). By the union bound, the probability of this event bounded by 2 (1 4)" and therefore the accuracy for p = (0.5, 0.5) is bounded from below by:
accp (hm) 2 1 2 1-77- 2
Finally, for (4), we note that - 2 (1- - 11--1)"1111 21 8n
1 for all n 2 21.
Proof of Theorem D.5. By Proposition D.20.
D.9 Stabilizing Coexistence
Proof of Proposition 2. Let F(p) be evolutionary prediction game induced by optimal learning algorithm A(p), and let p" be an unstable coexistence equilibrium with full support (isupporl
K). The game induced by A'(p) in the neighborhood of p" is F'(p) F(2p" p). By Lemma D.I, the game F F()) is a potential game, and therefore admits a potential function f(p) = accp(hpp)
Consider the potential function f'( f22pp p). By the chain rule, it holds that:
f - (V p) v(2p' p) = F(2p" - -p) Fp)
And therefore (p) a potential function for the game F'(p) in the neighborhood of p". The equilibrium p" is unstable, and therefore by [52, Theorem 8.2.1] it is not a maximizer of f(p). Moreover, by Proposition D.3 the potential function ((p) accp(hpp) is convex, and as p= is assumed to have full support, it is also global minimizer of s(p). From this we conclude that p* is a global maximizer of (p)), and therefore the equilibrium p stable under the evolutionary game induced by A'
44

---

E Implementation Details
Code. We implement our simulations and analysis Python. Our synthetic-data experiments rely on scikit-learn [47] for learning algorithm implementations, our CIFAR- 10 and MNIST experiments rely on PyTorch [46] and cfvv [38], and our ACSIncome experiment relies on scikit-learn and XGBoost [10]. We use matplotlib [29] for plotting, and mpltern [30] for ternary plots. Code is available at: https: //pptuu com/ htt///ddnnan///eeeee htt//vvoootoorrrrry hatptutiiiirrrrrrrrireee peeddpedeiiioooo -gaees.
single Macbook Pro laptop, with 16GB of RAM, M2 processor, and no GPU. Experiments involving neural networks (Section 6) were run on a dedicated server with an AMD EPYC 7502 CPU, 503GB of RAM, and Nvidia RTX A4000 GPU,
Hardware. Synthetic data simulations
Runtime. A single run of the complete synthetic data pipeline takes roughly 30 minutes on a laptop. A single repetition of each real-data experiment (i.c. sampling h A(p) and computing marginal accuracies) takes roughly 10 minutes.
the Resnet-9 architecture and training code provided in the CIFAR-10 example code in the ffcv Github repository (libffc//ffcr) commit 7885f40), with modifications control the probability of horizontal flips in training and testing. For MNIST, we use the convolutional neural network provided in the MNIST example code in the PyTorch examples Github repository pyttorch/examplee two convolutional layers, and two fully-connected layers, with dropout and max pooling. Training performed for 200 epochs using SGD with learning rate 0.01 and momentum 0.5. For the ACSIncome experiments, we use scikit-learn and XGBoost classifiers with default regularization parameters (LinearSVC,
Architectures. For CIFAR-10,
commit 37a1866). The MNIST network has
XCBCI (esefassi..er..
Replicator dynamics simulation. In the experiments, we use the discrete replicator equation for simulating evolutionary dynamics. The continuous-time replicator equation is:
Pr ph (p) F(p))
where F(p) = Ex Fx(() is the average fitness across the population. our simulations (e.g.. Figures 2, 4), use the discrete-ting replicator equation given by Taylor and Jonker [57, eq. (3)]:
(p( +1 F(p) +1
Pk
++1
Pk
(17)
For the CIFAR-10 replicator simulation (Section 6.1), we discretize the two-group simplex into the grid (),,0), (0.9, 0.1)...., (0, 1)), precompute fitness values (p) each grid point, and use linear interpolation to determine fitness at intermediate states.
Confidence intervals. In both experiments, variability is due to random split within the train and test sets, and due random noise when it is added. For numerical results, report mean confidence intervals at 99% confidence level.
F
Additional Empirical Evaluation
F.1 Effect of Sampling Noise
The prediction games defined in Section 3 associate evolutionary fitness with accuracy in expectation over the training (eq. (2), a reasonable assumption where sampling noise is small. Here we test the robustness of this assumption by quantifying the effect of sampling noise on long-term outcomes.
Method. We assess the effect of sampling noise on long term outcome by fitting a Gaussian process to the outcomes of different sampled training sets, and analyzing the resulting stochastic dynamics for varying initial conditions. We use the raw data from the CIFAR-10 experiment introduced in Section 6. (group accuracies for each training set sample). We fit a Gaussian process (GP) with an RBF kernel with length scale 0.1, using the skicit-learn implementation with alpha regularization parameter 4. To generate stochastic replicator dynamics, replace the deterministic fitness terms in
45

---

CIFAR-IE 10 Gaussian process fit
Stochastic replicator trajectories
Outcomes of stochastic dynamics
Only A
1.0
Popu baton state (p)
Prip" Only B)
0.8
93.0%
0.6
50.50
level
0.4
Oiservations
92.0%
Gaussian process
0.2
(..6
Orly 8-
0.0
Only A
50-50
Only
2000
4000
0.4
0.5
0.5
Population state (p))
Time (4)
Initial prosortion of group D (%)
Effect of sampling noise in the CIFAR-IO setting (Appendix F.I). (Left) Gaussian process fit to the raw observations. Error bars indicate standard deviation of data, and shaded areas indicate standard deviation of the Gaussian process. (Center) Trajectory samples of the induced stochastic replicator dynamics, for different simulated noise levels a. (Right) Distribution of long- term outcomes function of initial condition.
Figure 12:
eq. (17) with samples from the Gaussian process. We simulate different noise levels by multiplying the Gaussian process covariance by constant v 2 O, such that // = 0 coincides with the deterministic dynamics, 7) coincides with the Gaussian process fit to the original data, and simulated five-fold increase in compared to the observed sampling noise. For each we simulate stochastic dynamics for a range of initial states ph E [0.35, 0.65). and measure the distribution of long term outcomes (dominance of group A vs. group B) over repeated realizations.
represents
Results. Gaussian process fit is illustrated in Figure 12 (Left), stochastic trajectories are presented in Figure 12 (Center), and distributions of long-term outcomes presented in Figure 13 (Right). When there no sampling noise (i.e., 7) = 0), group dominates iff ph: 0.5 as expected. With the natural variance of the task (i.e., using the inferred Gaussian process covariance and results are relatively robust, and group B eventually dominates with high probability for all ph > 0.5. When variance is excessively inereased (i.e., 5), outcomes also become noisy, and dynamics can converge to group with some probability even for p1 x 0.55.
1),
F.2 Time to Convergence
For unstable equilibria, a key question is how fast the system transitions to a stable equilibrium state, and what affects this rate. Here measure the time reach (approximate) domination by one group as a function of the initial state.
Method. We extend the analysis of the CIFAR-10 experiment, described in Section 1. To obtain a smooth population game, we use the mean curves of the Gaussian process described in Appendix F.1. For each initial condition, we simulate the replicator dynamics, and record the number of steps until one group is approximately dominant, formally min p < 0.01.
Results. Figure 13 (Center) presents the measured convergence times. Results show that the time convergence is roughly linear in ph for pi E 0011, 0.4) U [0.6, 0.9), but tends to infinity as the initial state approaches the uniform distribution (p) - 0.5). The implication is that for initially balanced populations, natural selection forces are relatively weak, and and higher-order effects (such as sampling noise, or other exogenous forces) can dominate the dynamics.
F3 Sensitivity Analysis for Stabilization
Our proposed stabilization mechanism (Section 5.2) assumes access to the true equilibrium p', which may not be known in practice. Although our experiments in Section 6. use an estimated p² and results comply with our theoretical findings, they do reveal how robust outcomes are to misspecification of p". For this, we present sensitivity analysis quantifying the deviation of the reached state from the estimated equilibrium p".
46

---

CIFAR- 10 GP mean fitness
Time to replicator convergence
Stabilization with estimation errors
0.5
93.2%
convergorcee
6000
Outcome of deramics
93.0%
P
0.6
4000
92.8%
92.5%
0.4
Protent
BCSA
Rio.
92.2%
0.2
Only A
50-50
Only B
0..2
10.4
10.0
0.8
0.4
0.5
0.5
Population state (P)
Initial proportion of group (r2)
ecutlibrium
Figure 13: Empirical analysis of deterministic sensitivity aspects (Appendices F.2, F.3). (Left) Evolutionary prediction game induced by CIFAR-10 with horizontal flips and Resnet-9. Curves are smoothed by fitting Gaussian process, described in Appendix F.1. (Center) Time convergence for different initial conditions. Curve is relatively linear far from the unstable equilibrium, but approach infinity the initial condition approaches the unstable equilibrium. (Right) Sensitivity to cquilibrium estimation errors in steering, showing a linear relation.
Method. We simulate the stabilization method presented in Section 5.2 for varying population states around the uniform population state equilibrium, and measure the resulting long-term outcomes of the dynamics.
Results. Results are presented in Figure 13 (Right), and show a linear relation between the estimated equilibrium and the eventual outcome of the stabilized dynamics. An additional observation is that misspecification affects only the population composition, and not welfare. This is because the algorithm ensures that the returned classifier "as if" the system is under equilibrium, in which accuracy for both groups is the same.
F.4 Three-Group Dynamics for Different Learning Algorithms
To further establish the results of Section 6.3, compare three-group dynamics induced by different algorithms in the same setting.
Method. We follow the same procedure presented in Section 6.3, and compute the dynamics for linear SVM (from scikit-learn), logistic regression (scikit-learn), and an XGBoost classifier, all with default regularization parameters. Replicator dynamics simulated using Equation (17), and basins of attraction are computed by simulating the dynamics until convergence. The initial condition for all algorithms is p (P(TX): P(NY): P(CA)) (0.375, 0.5, 0.125).
Results. Results are presented in Figure 14. For identical initial conditions, each learning algorithm leads qualitatively different outcome: dynamics induced by the linear SVM converge to dom- inance of the Texas (TX) user group, dynamics induced by logistic regression converge towards a coexistence saddle point, and XGBoost dynamies converge towards dominance of the California (CA) user group. In particular, for the logistic regression system, note that the population state evolves along the attraction basin boundary (separatrix). By the definition of the replicator dynamics, states along the attraction basin boundary satisfy acc(cA)((p)) two-phase phenomenon similar to the one presented Section 6.3 (an early balance followed by competition between the remaining groups). In contrast, all interior points in the XGBoost dynamics evolove toward CA domination, precluding the existence of the two-phase phenomenon this setting.
acc(rx) (ktp). and therefore exhibit
47

---

Linear SVM
Logistic Regression
XGBoost
Figure 14: Comparing three- three-group group replicator dynamics across different learning algorithms for the ACSIncome prediction task (Appendix F.4). Background colors represent basins attraction, and red lines represent trajectories starting from the same initial condition.
48


Reviews:
[{"review_id": "H3auZGcRue", "reviewer": "Reviewer_oLKx", "summary": "This paper introduces an interesting framework that the authors name evolutionary prediction games. The motivation is that distributions of trianing data are composed of users who choose to use the platform, which can often depend on the quality of predictions. Thus there is a feedback loop where the prediction model can shape the population of users and the population of users will impact the distribution of data the model continues to be trained on. The authors analyze this setting use a population game, coming from evolutionary game theory. Their Theorem 1 shows that repeatedly training with new data will lead to dominance of the groups that had highest fitness to begin with, essentially allowing them to dominate the platform. Their second theorem provides conditions, however, under which there can be coexistence. Furthermore, the authors provide experimental results on MNIST and CIFAR-10 demonstrating the theoretical results.", "questions": "Although there is discussion in the supplement around related work, I wonder if it would be possible to highlight more in terms of what the author's believe evolutionary potential games will allow us to achieve in terms of gains in production ML systems compared to previous literature?\n\nAlluding to the weakness mentioned above, I wonder if there is any setting perhaps from a fairness benchmark dataset that would be closer to the motivations of the paper?\n\nGiven the results in Theorem 1 and Theorem 2, is there a situation in which the system could first evolve into domination by one group but by retraining grow to the mixed setting with multiple groups besides just changing to assumptions of Theorem 2?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The framing of evolutionary prediction games is really interesting and addresses a real issue faced by platforms that heavily rely on their prediction algorithms for user-retention (eg. TikTok). I really like it. The theorem are the natural type of result a reader would wish to see upon introduction of a new setting such as this, and the technical details are very well explicated in the main text and proven in the supplementary material. The technical results are relevant to the introduced setting but non trivial. Clear proof sketches and helpful interpretations are given for the main two theorems. Additionally, the figures are very clear and helpful. The experiments are simple but illustrative. \n\nThe main weakness that stands out to me is that the experimental results while illustrative of the theoretical results, don't immediately appear to be very representative of the main motivations for this framework in practice. Although it would of course be difficult to obtain a platform's worth of such data, an experiment more closely mimicking such settings would be very compelling.", "was_revised": true, "final_justification": "It is great that the reviewers were able to provide a new experiment from the fairness benchmark dataset. I appreciate the authors directly addressing my question to include results from a fairness setting as well as the more in-depth analyses they provided for their existing experiments. I give high weight to this as my initial perceived weakness of the paper was the representativeness of the experiments. Given that the motivation, positioning, and theory of the paper were already solid and motivated my original positive assessment of the paper, I do believe it certainly deserves my already original positive review of a 5 and I will increase my confidence in this assessment. I recommend Accept.", "rebuttal": "Thank you for the positive and thoughtful feedback! We address your questions below, including new experiments based on your suggestions:\n\n> I wonder if it would be possible to highlight more in terms of what the authors believe evolutionary potential games will allow us to achieve in terms of gains in production ML systems compared to previous literature?\n\n\nEvolutionary prediction games can help designers and operators of ML systems anticipate performative population shifts, and integrate this foresight into their decision-making. One key message is that applying conventional retraining risks favoring some groups over others in the long term, even if unintentionally. Evolutionary prediction games can help understand which groups are at risk, why they are at risk, and highlight possible avenues for interventions, if such are needed. One example is to decide on using a stabilizing learning algorithm (such as the one we propose in Prop. 2) instead of retraining to help steer towards desired states. Another is to target the retention of at-risk groups, either by providing benefits or reducing costs (e.g., via subsidies), in order to improve their fitness. A third is to understand if and when to increase the influx of certain groups, e.g. via marketing efforts, to balance the natural tendencies of the system. We will extend the discussion to emphasize these insights.\n\n\n\n> Alluding to the weakness mentioned above, I wonder if there is any setting perhaps from a fairness benchmark dataset that would be closer to the motivations of the paper?\n\nThank you for this question! Based on your suggestion, **we have conducted a new experiment with the Folktables dataset** [1] used commonly in the fairness literature. We detail them below, and will gladly add these new experiments and a discussion of their results to the paper.\n\nIn more detail, we use the ACSIncome prediction task, and associate groups with data originating from different states in the US, with the idea that firms operate in different states but train a model on data aggregated from all sources. We simulate replicator dynamics with $K=3$ using states NY, TX, and CA, and initialize with uniform $p^0$. We explore both XGBoost and logistic regression as learning algorithms.  \n\nWe use this new dataset to explore the implications of population dynamics on fairness. Results show that population dynamics exhibit two distinct dynamical phases: an initial phase where one group diminishes (while the other two remain balanced with similar accuracies), and a second phase in which the two remaining groups compete (until one of them dominates). Our results here highlight a shortcoming of static fairness definitions: auditing the system towards the end of the first phase will indicate that fairness holds – but only in appearance, since members of the two large groups are easily observed, but the third group has almost vanished. For further discussion please see our answer to Reviewer YgUX. \n\n> Given the results in Theorem 1 and Theorem 2, is there a situation in which the system could first evolve into domination by one group but by retraining grow to the mixed setting with multiple groups besides just changing to assumptions of Theorem 2?\n\n\nOur results state that for an *oracle* learning algorithm satisfying the conditions of Theorem 1, dynamics always converge towards single-group domination (with the only technical exception being the “equivalent groups” case discussed in Appendix D.5.1). This means that dynamics which first evolve towards single-group dominance and then towards coexistence cannot be induced for oracle algorithms, which are characterized by Theorem 1.\n\nHowever, this need not necessarily hold under other learning algorithms. One example is when the algorithm itself is allowed to vary over time. For example, in Section 5.2 we show that a mixed population setting can be induced by increasing the weight of underserved groups in the retraining of the optimal algorithm. It is possible to design a reweighing scheme which only “activates” once the population reaches a certain level of single-group dominance, and the dynamics induced by such an approach seem to resemble the scenario you describe. \n\n--\n\nFinally, we would like to thank you again for the positive and encouraging review, and for the great suggestions. If any questions remain or arise, please let us know!\n\nReferences:\n* [1] “Retiring Adult: New Datasets for Fair Machine Learning”, Ding et al., NeurIPS 2021."}, {"review_id": "ZX6C4veRwO", "reviewer": "Reviewer_Azxr", "summary": "This paper instantiates a model of a population of user participation decisions in a prediction system. The population is broken into types. If predictions about a type are good, more individuals of that type will join the system. If predictions about a type are less good, fewer individuals will participate. The system at a given point in time is described by the type-makeup of the population and different population distributions may yield different predictors (optimized for accuracy). This describes situations like content recommendation systems where user participation decisions and prediction quality may generate positive feedback loops. The model is instantiated as an evolutionary game. The main results in the paper are theoretical. They include:\n- Thm 1: If the accuracy of the predictor is uniform over a hypothesis class, accuracy improves, there exists a stable (attracting neighboring states) equilibrium, one type makes up the whole population in any stable equilibrium, and there may be unstable equilibria with more than two types.\n- Thm 2: If the predictor is optimized by a proxy loss, there exists a equilibrium with multiple types that is stable. The intuition is that the proxy induces\n- Prop 2: If the data used to train the predictive system can be reweighted, unstable equilibria can be made stable.", "questions": "What does it mean to be fitness maximizing? I don’t see where it was defined. Is it the maximum (over p) fitness summed over types?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The analysis seems correct.\n- The area of prediction under dynamics or performativity is interesting and under-explored.\n\nWeaknesses:\n\n- Real platforms seem to have a diversity of users, it seems like the results in the paper are discordant with real, e.g., recommendation systems. What explains this? Is it that the evolution dynamics are not as powerful in real life? Is it explained by the proxy objectives that platforms must use? Are the platforms training on a diversity of data? Are platforms not in equilibrium? I would have liked more grappling with the implications of these results for platform practices or how we understand them.\n- Generally, I did not think the theoretical take-aways were very surprising. Theorem 2 in particular seems weak since it is an existence proof that relaxing optimality of the objective can yield stable multi-group equilibria.\n- I did not feel the related work on evolutionary prediction games was sufficiently detailed. Results in this paper seem like they might be special cases of general analyses of population games. (Intuitively, it seems like the conclusion of Thm 1 should hold quite generally and any setting where growth begets growth should yield these kinds of conclusions.) On the other hand, I am not familiar enough with population games to verify whether this kind of claim is proved elsewhere. The related work says “competitive exclusion” is well-studied, which suggests to me there should be related results.\n- I did not find the experiments particularly informative. It would be nice to highlight qualitative take-aways or more nuances rather than just reporting the type breakdown in the equilibria.", "was_revised": true, "final_justification": "This is a strong paper which addressing the important question of performativity/dynamics in recommendations. The authors formulate this question naturally in terms of evolutionary games and include novel elements that tailor the setting to content/recommendations. Their analysis and discussion support their conclusions and I am pleased to see the additions they propose to make to the paper.", "rebuttal": "Thank you for your detailed review and feedback. Based on your suggestions, we’ve extended our experimental analysis and added a new empirical investigation, which we believe will strengthen the paper - thank you for this suggestion! Please see our response to your questions and concerns below.\n\n> Real platforms seem to have a diversity of users, it seems like the results in the paper are discordant with real, e.g., recommendation systems. What explains this?\n\nThis is an excellent question, but the answer is somewhat nuanced. As we state in the paper, our results should be taken to describe the *tendency* of population dynamics – when selective pressure is the primary force, when the learning algorithm is optimal retraining, absent any interventions, and at the limiting equilibrium. Hence, possible explanations for why a particular system at a given point in time can sustain several groups are:\n* **Time**: dynamics have not yet reached equilibrium, or converge very slowly.\n* **Learning**: the algorithm is not optimal (e.g., uses a proxy or finite data), or is designed to balance groups (e.g., via steering, regularization, or robust training).\n* **Intervention**: system administrators may intentionally intervene to facilitate diversity; for example, consider how firms invest significant resources in marketing (e.g., Meta reports investing ~$2B per year [1]) to ensure a constant influx of users from diverse social groups.\n* **Dynamics**: Other forces are at play, such as lack of alternatives, cross-group influence, rigidity of demand, non-linear heterogeneous utility, etc.\n\nNonetheless, an important but subtle point is that when we observe a system consisting of several groups, **we do not observe other groups that could have been supported**, as they may have already been driven out. Thus, what we perceive as a mixed state may actually be a snapshot of a process in which “weaker” groups have already vanished, with others possibly to follow. This is in fact a likely outcome – **we have added to the paper a new experiment using the Folktables dataset** [2] that shows for $K=3$ how dynamics first diminish one group (while preserving balance between the others), and only then exhibit competition between the two remaining groups. We believe this perspective has important implications on fairness and its regulation.\n\nIn biology, the question of coexistence is also prevalent, despite consensus that natural selection is the primary force at play. The study of coexistence therefore focuses on how it is sustained *despite* natural selection. This has revealed many biological and ecological mechanisms that can facilitate coexistence, such as resource partitioning, environment variation, and stabilization via intermediates. In this work we focus on learning itself as a possible stabilizing mechanism (Sections 5,6), which we view as a first step towards the study of coexistence in this context (as also noted by Reviwer oLKx). We also believe that prediction games in particular, and evolutionary game theory more generally, can aid future research in exposing additional mechanisms that can facilitate diversity in environments with learning.\n\nAs per your request, we will add to the paper a detailed discussion on the practical implications of our results.\n\n> I did not feel the related work on evolutionary prediction games was sufficiently detailed.\n\nWe will gladly extend our discussion of evolutionary game theory in the related work section, and in particular highlight the unique challenges evolutionary prediction games present in this context.\n\n> Intuitively, it seems like the conclusion of Thm 1 should hold quite generally and any setting where growth begets growth should yield these kinds of conclusions.\n\nWhile we agree with the conclusions of “growth begets growth”, proving that oracle learning algorithms imply this property is not trivial. This is due to two main reasons. First, to our knowledge, the structure of evolutionary prediction games is novel, as group fitness is defined implicitly as a solution of a loss optimization problem (see our answer below). Second, our results in Section 5 and Appendices D.7, D.8 show that the conclusion of Theorem 1 is surprisingly “brittle” with respect to the assumptions about the learning algorithm. We show that when the algorithm is limited in either computation, data, or consistency, the “growth begets growth” property doesn’t always hold. More broadly, evolutionary dynamics are very expressive, and able to express complex phenomena such as limit cycles, chaos, and at the extreme even simulate Turing machines [3]. Oracle learning algorithms are themselves complex in many ways, and it is interesting to see that they induce well-behaved competition dynamics.\n\n\n> The related work says “competitive exclusion” is well-studied, which suggests to me there should be related results.\n\nWhile competitive exclusion is a widely-observed biological phenomenon, evolutionary prediction games are unique in that the interaction between groups is induced by the solution (argmin) of a loss minimization problem. The implicit nature of optimization problems introduces complexity into the analysis of our game dynamics, which makes it difficult to draw on existing results. To our knowledge, there are no previously studied games in the evolutionary game theory literature that possesses this structure, and we believe that some of our techniques may be of independent interest.\n\n> (In the experiments) It would be nice to highlight qualitative take-aways or more nuances rather than just reporting the type breakdown in the equilibria.\n\nThis is an excellent idea! As per your request, **we have extended our empirical analysis to include an in-depth analysis of learning dynamics and outcomes** for the existing experiments, as well as an additional experiment on new data (mentioned above). Results include:\n1. **The effect of sampling noise:** As our theoretical analysis considers accuracy in expectation over the training set, it is interesting to understand the effects of sampling noise on long-term outcomes. Here we tested the effect of sample variance (controlled by fitting a Gaussian process to the outcomes of different sampled training sets) on long term outcome (dominance of group A vs. group B) for varying initial states $p^0_B \\in [0.45,0.55]$. When there is no variance, group B dominates iff $p^0_B>0.5$ as expected. With the natural variance of the task (i.e., using the inferred Gaussian process covariance), results are relatively robust, and group B eventually dominates w.p. ~1 for all $p^0_B>0.51$. When variance is excessively increased (e.g., 5-fold), then outcomes also become noisy, and dynamics can converge to group A with some probability even for $p^0_B=0.55$.\n2. **Time to convergence:** For unstable equilibria, a key question is how fast an equilibrium state is reached, and what affects this rate. Here we measured the time to reach (approximate) domination by one group as a function of the initial state $p^0$. For the CIFAR-10 dataset, results show that when $p^0_B\\in[0.6,0.8]$ (or $[0.2,0.4]$), the time to convergence is roughly linear in $1-p^0_B$ (or $p^0_B$). However, as $p^0$ approaches the uniform distribution, time to convergence tends to infinity. The implication is that for initially balanced populations, natural selection forces are weak.\n3. **Sensitivity analysis for stabilization:** Our stabilization mechanism (Section 5.2) requires access to the true equilibrium $p^\\*$, which may not be available in practice. \nOur current experiments use an estimated $\\tilde{p}^\\*$ – and results comply with our theoretical findings – but this does not tell us how robust outcomes are to misspecification of $p^\\*$. For this, we add a sensitivity analysis showing that the deviation of the reached state from the true $p^\\*$ is linear in the estimation error of $\\tilde{p}^\\*$. \nAn interesting observation is that misspecification affects only the population composition, and *not* welfare. This is because the algorithm ensures that the returned classifier acts “as if” under equilibrium, in which accuracy for both groups is the same.\n4. **Additional dataset - Folktables**: We have added experiments on an additional social dataset, namely Folktables [2], which is based on a large collection of US census data. Here features describe individuals and labels describe their yearly income level. We used ‘state’ as a group (with the motivation that geography plays an important role in determining consumer sectors), and focused on three US states: New York, California, and Texas. Here we used both classifiers trained with XGBoost and logistic regression. We computed the induced vector field of the game and plotted both the population composition $p$ and group accuracies over time. As discussed above, our goal here was to showcase the intricate ways in which dynamics can progress in a multi-groups setting.\n\nWe will add these results and a thorough discussion to the final version of the paper using the extra space.\n\n> What does it mean to be fitness maximizing?\n\nAn equilibrium is fitness-maximizing if it pareto-dominates other equilibria with respect to fitness (i.e., the accuracy of each group is higher than its accuracy in any other equilibrium, and thus all groups are better-off if the system reaches this equilibrium). We will clarify this in the paper.\n\n–\n\nFinally, we would like to thank you again for the insightful and positive review! Please let us know if our response helps address your key concerns. If anything remains unanswered, or if additional questions arise, please let us know.\n\nReferences: \n* [1] Meta Platforms, Form 10-K - Annual report, 2024, SEC Accession No. 0001326801-25-000017. “Advertising Expense”.\n* [2] “Retiring Adult: New Datasets for Fair Machine Learning”, Ding et al., NeurIPS 2021.\n* [3] “No-Regret Learning in Games is Turing Complete”, Andrade et al., EC 2023."}, {"review_id": "jsvYTMzUP8", "reviewer": "Reviewer_iKna", "summary": "Under the framework of evolutionary game theory, this paper provides evolutionary prediction games to study the interplay between the accuracy of prediction algorithms and user retention rates. Moreover, a theoretical analysis is provided to investigate the equilibria of the games. This is interesting. However, some details of the evolutionary prediction games should be clarified, like how does a user change the group. In addition, from the perspective of improving the performance of prediction algorithms, this paper lacks a relevant study and explanation.", "questions": "Please see the weaknesses above.", "limitations": "Based on the study of the co-evolutionary dynamics between the prediction algorithms and user behavior, can the authors give some insights into the design of prediction algorithms?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**  \nThis paper originally presents evolutionary prediction games to study the feedback loop between the accuracy of prediction algorithms and user behavior. This provides a new perspective for prediction algorithms.\n\n**Weaknesses**  \n1. The introduction is not easy to follow. The accuracy of sentences should be further improved. For example, in line 19, what kinds of decision-making are included in personalized education? In line 23, what does social outcome mean here? I think the experiments do not focus on any data from real platforms. In line 25, \"But in social settings, ...\", I think this is not about social settings. For any platforms, which collect data from users to train their prediction algorithms, they may exhibit the feedback loop between users and the accuracy of prediction algorithms. In line 35, \"Common to ...\", What does group structure mean here? What does the relation between prediction algorithm and group?\n2. Why do you divide the population into $K$ groups? Is it necessary for the evolutionary prediction games?\n3. In line 104, the group-specific distribution $D_k$ is assumed to be fixed. Meanwhile, users are able to decide to leave a group and reshape the population composition $p$. Thus, once a user leaves the group $k$, the distribution $D_k$ changes. This seems to contradict the assumption.\n4. In lines 111-113, can you give more details about how users change their groups? For example, under what conditions will a user leave their current group? How does a user choose which new group to join?\n4. In line 211, how can you ensure the existence of $A^{opt}$? What are the conditions for the existence of $A^{opt}$?\n6. Theorem 2 states the existence of such an evolutionary prediction game. What are the conditions for the existence? \n7. This paper studies the co-evolutionary dynamics between prediction algorithms and user behavior. Based on the stable states of the dynamics, can you give some insights into the design of prediction algorithms? \n8. In experiments, can you give a comparison between different prediction algorithms?", "was_revised": true, "final_justification": "The authors have explained the rationale behind dividing the population into multiple groups, clarified the process by which individual users make decisions on group selection, and stated that they have added experiments comparing different prediction algorithms. Based on the authors’ response, I now appreciate the significance of studying the co-evolution of prediction algorithms and individual strategies within an evolutionary game-theoretic framework. I have therefore decided to raise my evaluation of the paper.", "rebuttal": "Thank you for your feedback and detailed review. In your review, the “weaknesses” section consists mostly of questions, with one additional question under “limitations”. Our response below fully addresses all of these questions. We hope you would be willing to reconsider your evaluation given our response herein.\n\n> (1.a) In line 19, what kinds of decision-making are included in personalized education? \n\nPersonalized education uses predictions to improve learning experiences. For example, a common task is to construct a personalized learning curriculum for students based on predicted difficulty levels of possible exercises. See [1] for a recent survey.\n\n> (1.b) In line 23, what does social outcome mean here? … In line 25, \"But in social settings, ...\", I think this is not about social settings.\n\nWe use the term “social” as it is used in economics, and within the machine learning literature, as it is used for example in performative prediction (see, e.g., [2,3]). The term “social outcome” refers to the positive or negative effects on society that occur as a result of the interaction with the prediction system. Accordingly, the term “social settings” refers to settings in which the prediction algorithm’s effect on society is significant and should be considered. This includes platforms that collect data from their users to train their prediction algorithms, assuming this process also has an impact on those users.\n\n> (1.c) In line 35, \"Common to ...\", What does group structure mean here? What does the relation between prediction algorithm and group?\n\n“Group structure” means that the population responds to a deployed classifier in a way which can be modeled as a group-wise operation for some partition of the population into groups (e.g. according to demographics, behavioral patterns, social group etc.). We give formal examples for such relations in Appendix C (“Microfoundations”).\n\nThe training data for the prediction algorithm is sampled from the mixture of groups, and the mixture coefficients change in response to prediction quality. Illustration is provided in Figure 1, and formal definitions are provided in Section 3.\n\n> (2) Why do you divide the population into $K$ groups? Is it necessary for the evolutionary prediction games?\n\nWe focus on a finite number of groups $K$ for two main reasons. First, prior work shows that learning dynamics under infinitely-many or a continuum of groups can be considerably more involved (see for example [4], and also Appendix B.1), and therefore we believe that working with a finite number of groups is a reasonable first step. Second, the notion of a finite set of groups is common in the literature, for example in fairness. Note however that our notion of “group” is much more general as it need not be based on some intrinsic, immutable property, and can be quite flexible, as illustrated in Appendix C.\n\n> (3) In line 104, the group-specific distribution $D_k$ is assumed to be fixed. Meanwhile, users are able to decide to leave a group and reshape the population composition $p$. Thus, once a user leaves the group $k$, the distribution $D_k$ changes. This seems to contradict the assumption. \n\nWe make a distinction between the group distribution $D_k$, which is indeed assumed to be fixed, and the actual users from group $k$ that are a part of a sampled set of users, which is not assumed to be fixed (and is very much expected to vary). The meaning of group relative size “increasing” or “decreasing” have different interpretations and manifest differently in different types of dynamics. For example, in replication processes users invite peers from the same group, whereas in imitation users are assumed to copy the behavior of peers that are more successful (see Appendix C). Similarly, some dynamics allow individual users to remain in the system over time, while others assume different users are sampled at each time step. All of these comply with the assumption that there *exists* some fixed group distribution $D_k$ from which data is sampled, and we therefore view this as a reasonable assumption. \n\n> (4) In lines 111-113, can you give more details about how users change their groups? For example, under what conditions will a user leave their current group? How does a user choose which new group to join?\n\nAs described above, these depend on the specific type of dynamics under consideration. Our results are general in the sense that they apply to a large and expressive family of dynamics that adhere to certain global properties, and which includes many varied types of micro-dynamics. For examples and further details please see Appendix C.\n\n> (5) In line 211, how can you ensure the existence of $A^{opt}$? What are the conditions for the existence of $A^{opt}$?\n\nThe oracle algorithm $A^{\\\\mathrm{opt}}$ is a theoretical object that we use as a construct in our analysis. It is guaranteed to exist whenever the corresponding optimization problem has a solution (Equation (4)). This will depend on the choice of model class $\\\\mathcal{H}$. One example is when $\\\\mathcal{H}$ is compact (e.g. a space of parameterized classifiers with bounded norm), which holds for many common classes. Another example is when $\\\\mathcal{H}$ includes all functions, in which case $A^{opt}$ yields the Bayes-optimal classifier.\n\n> (6) Theorem 2 states the existence of such an evolutionary prediction game. What are the conditions for the existence?\n\nA formal constructive proof is provided in Appendix D.6. An illustration of the approach is provided in Figure 3 (Left).\n\n> (7) This paper studies the co-evolutionary dynamics between prediction algorithms and user behavior. Based on the stable states of the dynamics, can you give some insights into the design of prediction algorithms?\n\nCertainly. One central insight is that the natural selection forces induced by learning algorithms tend to push the population towards single-group dominance as the algorithms become better at generalization. Algorithm designers should be aware of this property, and plan intervention methods if single-group dominance is not aligned with the system goals. For example, this can be achieved by employing our stabilizing algorithm from Section 5.2. Another example is to facilitate an exogenous influx of users of diverse groups, e.g. via marketing efforts. We will expand the discussion on this in the final version of the paper.\n\n> (8) In experiments, can you give a comparison between different prediction algorithms?\n\nWe have added new experiments on empirical fairness datasets, including a comparison between different learning algorithms. See our response to Reviewer oLKx.\n\n--\n\nReferences:\n* [1] “A Comprehensive Survey on Deep Learning Techniques in Educational Data Mining”, Lin et al., Data Science and Engineering 2025.\n* [2] “The predictability of social events”, Grunberg & Modigliani, Journal of Political Economy 1954.\n* [3] “Performative Prediction: Past and Future”, Hardt & Mendler-Dünner, to appear in Statistical Science 2025.\n* [4] “Strategic distribution shift of interacting agents via coupled gradient flows”, Conger et al., NeurIPS 2023."}, {"review_id": "v36BAbIsMY", "reviewer": "Reviewer_YgUX", "summary": "This paper introduces evolutionary predictive games, which are a class of evolutionary games in which a classifier and the underlying data distribution co-evolve in a feedback loop with dynamics driven by payoffs defined by average prediction performance. Analysis is carried out to show that, under perfect learning (optimization) and perfect computation of population-level expectations, the evolutionary predictive games monotonically increase in the classifier's accuracy and converge to Nash equilibria with singleton support (meaning only one group of the data distribution remains at play). It is shown that various alternative problem setups can give rise to co-existence of multiple groups at (stable) Nash equilibrium, e.g., by using surrogate loss functions or different learning algorithms. Experiments on MNIST and CIFAR-10 are conducted to empirically corroborate the theory.", "questions": "1. The introduction is very sparse in terms of references. It would be good to support your motivating assertions with citations. For example: \"However, while generally beneficial, promoting accuracy blindly can have unexpected, and in some cases undesired, consequences.\" Can you give references that demonstrate such undesired consequences? For instance, there is a body of influential work that studies accuracy-robustness tradeoffs in ML models that you may want to mention and cite. There are currently no references in the first four paragraphs.\n2. Another important example of where there really should be some specific references cited is the list given in the sentence \"Such dynamics arise across domains:...\"\n3. ‘compete’ Here, you are using single quotes, but before, you used double quotes. It would be good to maintain consistency in your notations and writing style.\n4. Figure 1 is not introduced or discussed at all in the main text of the paper. It would be good to do this, to indicate to the reader when is a natural time to go and look at Figure 1.\n5. Proposition 1: The \"overall accuracy equality fairness criterion in expectation\" property is referred to in Proposition 1, but its definition is deferred to the Appendix. It is fine to defer the proof of Proposition 1 to the Appendix, but it is preferable to include all definitions used in the main text, in the main text, for self-containedness reasons.\n6. \"Overall Accuracy Equality Fairness\": The way you have defined this criterion doesn't quite seem to fully capture \"fairness\" to me. Specifically, how can you say that you achieve fairness if the equality in prediction accuracy only holds for groups that remain at play in Nash equilibrium, unless the Nash equilibrium has full support? For example, if the dynamics converge to a Nash equilibrium with a single index of support, meaning all but one group have effectively been driven out of the game, then how is that fair for the groups that have been driven out?\n7. \"fixed points of the dynamical system coincide with the Nash equilibria\" You may want to mention that this is a property known as \"Nash stationarity.\"\n8. It seems a little misleading in the paragraph of line 169 to claim that all of your three assumptions are satisfied by the dynamics described in Section 1 and Appendix C, and to throw in the \"exception\" that imitative dynamics don't satisfy Nash stationarity into the footnote 1, and to not mention in the main text that the replicator dynamics are imitative dynamics. That is, the replicator dynamics that you seem to focus on do not satisfy your assumption of Nash stationarity, but the reader would need to piece this together themselves by carefully reading footnote 1 and identifying that replicator dynamics are imitative (which you don't explicitly define or mention). If your main theoretical results hold for dynamics that have unstable non-Nash stationary points, then why bother making the Nash stationarity assumption at all?\n9. Line 197: There are two periods in the sentence with footnote 2.\n10. Line 298 typo: \"dynamcis\"", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: After carefully reviewing the theoretical developments of the paper, the submission appears to be rigorous and technically sound. The key claim (namely, that an analysis is provided for the proposed evolutionary prediction games showing that they exhibit competitive exclusion in the idealized setting of unlimited knowledge of the data distribution, and that stable co-existence can arise in the setting of finite data) is supported through the theoretical developments, and corroborated by the experiments. The theoretical and experimental methods used are appropriate for the problem at hand, and the manuscript culminates into a nice, \"complete\" work. The authors do not explicitly discuss weaknesses or limitations of their work (see my \"Questions\" #6 and #8 for specific instances where this could be improved), but they do briefly mention avenues for future work and related discussion in the conclusion of the paper.\n\nClarity: For the most part, the submission is well-written and well-organized. I have enumerated a few small suggestions for improvement to the writing and presentation in the \"Questions\" below. Enough details of the theoretical proofs and experimental setup are provided (in the supplemental materials) to reproduce the results. The primary weakness in the paper's clarity is that the introduction is quite sparse in terms of the number of references, making it potentially difficult for a reader to jump to relevant work that is prerequisite in fully understanding and appreciating the work at hand. See my \"Question\" #1 below for more details.\n\nSignificance: I find the theoretical findings of competitive exclusion in the ideal setting and stable co-existence in other, non-ideal settings, to be significant to those interested in the intersection of evolutionary game theory and ML. The results are likely not very significant to ML as a whole, which is okay. I suspect that other researchers are likely to build upon the theoretical framework and ideas proposed here. The submission does not necessarily address tasks in better ways than previous works, but rather it provides a novel theoretical framework that, to the best of my knowledge, is unique to the community. The theory developed indeed advances our understanding of the proposed evolutionary prediction games, as the conclusions are not immediately or intuitively obvious.\n\nOriginality: Not only does the work introduce a novel problem setup at the interesting intersection of evolutionary game theory and machine learning, but it provides novel theoretical insights about the problem. The authors could better clarify how their work differs from previous contributions by providing more citations throughout the motivations, problem setup, and introduction of the paper (see my \"Question\" #1 below).", "was_revised": true, "final_justification": "I have read the author responses to all of the (also positive) reviews. The authors have very clearly and carefully addressed all of my concerns, and, in particular, have better clarified the key technical assumptions underlying their stability theory, and the nuances behind conventional fairness definitions, how they inadequately capture the notion of groups being driven out of games entirely, and how the contributions of this paper elucidate these inadequacies. With these clarifications and associated revisions in mind, I find this to be a strong theoretical paper (and, although I cannot see the new Folktables dataset experiment, the authors have also convinced me that their empirical evaluations have also strengthened), with deep insights into the connections between evolutionary dynamics and fairness in learning. Therefore, I raise my score from 4 to 5.", "rebuttal": "Thank you for the positive feedback, and for the very constructive and thorough review! We address your questions below:\n\n> (1) The introduction is very sparse in terms of references. It would be good to support your motivating assertions with citations. For example: \"However, while generally beneficial, promoting accuracy blindly can have unexpected, and in some cases undesired, consequences.\" Can you give references that demonstrate such undesired consequences? For instance, there is a body of influential work that studies accuracy-robustness tradeoffs in ML models that you may want to mention and cite. There are currently no references in the first four paragraphs. \n\nThank you for pointing this out, we will gladly add supporting references throughout. One type of unintended consequences that we had in mind are those of concern in recommender systems, media platforms, and online markets, such as: content homogenization, opinion polarization, filter bubbles and echo chambers, hyper-popularization, and diminishing tail consumption. We will add references to papers which connect these phenomena to prediction. Another type are emergent social biases, such as those studied in the fairness literature, which have a well-established connection to accuracy maximization. We did not consider the connection to robustness that you mention, but this certainly makes sense, and we will gladly discuss this point as well.\n\n> (2) Another important example of where there really should be some specific references cited is the list given in the sentence \"Such dynamics arise across domains:...\"\n\nThank you for raising this point. We will expand the list of references.\n\n> (3) ‘compete’ Here, you are using single quotes, but before, you used double quotes. It would be good to maintain consistency in your notations and writing style.\n\nFixed!\n\n> (4) Figure 1 is not introduced or discussed at all in the main text of the paper. It would be good to do this, to indicate to the reader when is a natural time to go and look at Figure 1.\n\nThank you for noticing this. We’ve added a reference at the end of Section 2.\n\n> (5) Proposition 1: The \"overall accuracy equality fairness criterion in expectation\" property is referred to in Proposition 1, but its definition is deferred to the Appendix. It is fine to defer the proof of Proposition 1 to the Appendix, but it is preferable to include all definitions used in the main text, in the main text, for self-containedness reasons.\n\nThanks for this suggestion, we will move the definition to the main body of the paper.\n\n> (6) \"Overall Accuracy Equality Fairness\": The way you have defined this criterion doesn't quite seem to fully capture \"fairness\" to me. Specifically, how can you say that you achieve fairness if the equality in prediction accuracy only holds for groups that remain at play in Nash equilibrium, unless the Nash equilibrium has full support? For example, if the dynamics converge to a Nash equilibrium with a single index of support, meaning all but one group have effectively been driven out of the game, then how is that fair for the groups that have been driven out?\n\nThis is a great question. The incompatibility between traditional fairness definitions and population dynamics *is precisely the point* we are trying to make. We apologize if this was unclear, and will make sure to clarify it.  \n\nOne straightforward implication is that if you train a classifier to comply with fairness constraints, dynamics can cause this guarantee to break over time; this idea has already been explored to some extent in the fairness literature, mostly following [1]. But another more nuanced implication – which you describe precisely – is that a system can *appear* to comply with a fairness definition, but only because some groups were already driven out of the system, and fairness is measured only over the groups that remain.\n\nOur message here is that fairness should consider not only what has happened, but also what *could* have happened (in our case, in terms of which groups to compare). In fact, we think that settings in which a strict subset of groups exists for the majority of the time is very likely, and have included an additional experiment that demonstrates this: Using the Folktables dataset [2], we show that in a three-group setting ($K=3$), dynamics first drive out one group (while keeping the other two balanced), and only then introduce competition between the two groups that remain. Auditing the prediction algorithm for fairness after the first group is driven out provides an appearance of fairness for the two groups that remain, with little ability to identify the preceding existence of the group that was driven out of the system.  \nWe believe this form of “counterfactual” fairness has not received sufficient attention in the literature, and hope our work can promote discussion and motivate future work on this topic.\n\n\n> (7) \"fixed points of the dynamical system coincide with the Nash equilibria\" You may want to mention that this is a property known as \"Nash stationarity.\"\n\nWe agree. We do use the term “Nash stationarity” in Appendix D.2, but will make sure to promote its usage to the main body of the paper for clarity.\n\n> (8) It seems a little misleading in the paragraph of line 169 to claim that all of your three assumptions are satisfied by the dynamics described in Section 1 and Appendix C, and to throw in the \"exception\" that imitative dynamics don't satisfy Nash stationarity into the footnote 1, and to not mention in the main text that the replicator dynamics are imitative dynamics. That is, the replicator dynamics that you seem to focus on do not satisfy your assumption of Nash stationarity, but the reader would need to piece this together themselves by carefully reading footnote 1 and identifying that replicator dynamics are imitative (which you don't explicitly define or mention). If your main theoretical results hold for dynamics that have unstable non-Nash stationary points, then why bother making the Nash stationarity assumption at all?\n\n\n\nThank you for raising this point. Our intent was to simplify the presentation, but this perhaps came at the expense of clarity and precision. Overall we were very careful to explicitly and accurately state any and all assumptions that we make, and so we apologize if this part in particular caused confusion. We clarify our assumptions below, and will update the paper accordingly.\n\nIn more detail, our theoretical analysis inherits the assumptions about dynamics from the convergence guarantees provided in [3], which are proven under the assumptions of either Nash-stationary *or* imitative dynamics (see Appendix D.2 for a summary of the relevant results we rely on). While imitative dynamics indeed do not satisfy the Nash stationary assumption, they do satisfy a similar condition with respect to a slightly-coarser equilibrium concept of the population game – namely, stationary points of imitative dynamics correspond to “restricted equilibria” of the game, a set that in particular includes all Nash equilibria (see, e.g., [3, Section 8.1]). For our purposes, this particular form of deviation from Nash stationarity does not affect our results or analysis.\n\nFor this reason, we chose to focus in the exposition primarily on Nash-stationary, which captures more cleanly the structure we exploit, and in our mind provides better intuition. Nonetheless, we will make sure to revise this section to more precisely state the conditions under which our results hold, in line with the above description.\n\nWe would however like to note that the fact that our results hold for imitative dynamics does not imply that they hold for any dynamics permitting arbitrary unstable non-Nash points. The question of whether results hold also for other dynamics with milder assumptions is very interesting, and we will highlight this in the paper.\n\n> (9,10) Line 197: There are two periods in the sentence with footnote 2.; Line 298 typo: \"dynamcis\"\n\nFixed!\n\n–-\n\nAnd finally, we would like to thank you once again for the positive and constructive review! Please let us know if our response helps address your concerns. If any questions remain, or if additional ideas come up, we would be more than happy to continue the conversation.\n\nReferences:\n* [1] “Delayed Impact of Fair Machine Learning”, Liu et al., IJCAI 2019.\n* [2] “Retiring Adult: New Datasets for Fair Machine Learning”, Ding et al., NeurIPS 2021.\n* [3] “Population Games and Evolutionary Dynamics”, Sandholm, 2010."}]

Author rebuttal:

Meta-review: This paper introduces a novel theoretical framework called evolutionary prediction games for studying feedback loops between ML prediction accuracy and user population dynamics.  Their motivation stems from the idea that prediction models can shape the population of users engaging with an ML platform, and the population of users will, in turn, impact the distribution of data the model continues to be trained. They present two theoretical results on the equilibria of the games, which show that (1) repeatedly training with new data will lead to dominance of the groups that had the highest fitness to begin with, essentially allowing them to dominate the platform, and (2) present certain conditions under which there can be coexistence between different groups. Experiments on toy datasets MNIST (w/ label noise) and CIFAR-10 (w/ augmentations) demonstrate the theoretical findings on equilibria conditions for two user groups in each setting and further demonstrate the effectiveness of the proposed stabilization algorithm for beneficial coexistence between groups. 

All reviewers agree on the significance [oLKx, Azxr, iKna], usefulness of the approach [oLKx, iKna], technical clarity and soundness [oLKx, Azxr, YgUX], and originality [Azxr, YgUX] of the approach. The main limitations of the work are listed as experiments not being close to real-world ML production systems [oLKx, Azxr] and the need for clarification in certain assumptions [iKna, YgUX]. During the rebuttal, the authors claim to have conducted additional experiments on a fairness dataset (Folktables) and explored the implications of population dynamics on fairness with three different user groups derived from real-world US census data. The insights from this experiment confirm the theoretical findings that a false appearance of fairness can emerge between two groups while one is completely driven out by the dynamics of the game. The authors indicate that such implications from their theory can be beneficial for different real-world applications to monitor which groups survive in the long run. All reviewers have expressed that most of their concerns have been successfully addressed by the authors during the discussion period.

The authors must include the newly presented results/insights on the more realistic fairness dataset and update the final version of their paper with the references and clarifications they have addressed in discussion with all the reviewers (additional references, clarification of Nash stationarity assumption, discussion on insights of prediction algorithms, implications on real-world applications and ML production systems).
Decision: Accept (poster)