SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lauren Conger  
California Institute of Technology  
lconger@caltech.edu
Franca Hoffmann  
California Institute of Technology  
franca.hoffmann@caltech.edu
Eric Mazumdar  
California Institute of Technology  
mazumdar@caltech.edu
Lillian Ratliff  
University of Washington  
ratliff1@uw.edu
We propose a novel framework for analyzing the dynamics of distribution shift in real-world systems that captures the feedback loop between learning algorithms and the distributions on which they are deployed. Prior work largely models feedback-induced distribution shift as adversarial or via an overly simplistic distribution-shift structure. In contrast, we propose a coupled partial differential equation model that captures fine-grained changes in the distribution over time by accounting for complex dynamics that arise due to strategic responses to algorithmic decision-making, non-local endogenous population interactions, and other exogenous sources of distribution shift. We consider two common settings in machine learning: cooperative settings with information asymmetries, and competitive settings where a learner faces strategic users. For both of these settings, when the algorithm retransies via gradient descent, we prove asymptotic convergence of the retraining procedure to a steady-state, both in finite and in infinite dimensions, obtaining explicit rates in terms of the model parameters. To do so we derive new results on the convergence of coupled PDEs that extends what is known on multi-species systems. Empirically, we show that our approach captures well-documented forms of distribution shifts like polarization and disparate impacts that simpler models cannot capture.
In many machine learning tasks, there are commonly sources of exogenous and endogenous distribution shift, necessitating that the algorithm be retrained repeatedly over time. Some of these shifts occur without the influence of an algorithm; for example, individuals influence each other to become more or less similar in their attributes, or benign forms of distributional shift occur [Qui+1]. Other shifts, however, are in response to algorithmic decision-making. Indeed, the very use of a decision-making algorithm can incentivize individuals to change or mis-report their data to achieve desired outcomes—a phenomenon known in economics as Goodhart's law. Such phenomena have been empirically observed, a well-known example being in [CC11], where researchers observed a population in Columbia strategically mis-reporting data to game a poverty index score used for distributing government assistance. Works such as [Mil+20; Wil+21], which investigate the effects of distribution shift over time on a machine learning algorithm, point toward the need for evaluating the robustness of algorithms to distribution shifts. Many existing approaches for modeling distribution shift focus on simple metrics like optimizing over moments or covariates [DY10; LHL21; BBS09]. Other methods consider worst-case scenarios, as in distributionally robust optimization [AZ22; LFG22; DN21; Kuh+19]. However, when humans respond to algorithms, these techniques may not
be sufficient to holistically capture the impact an algorithm has on a population. For example, an algorithm that takes into account shifts in a distribution's mean might inadvertently drive polarization, rendering a portion of the population disadvantaged.
Motivated by the need for a more descriptive model, we present an alternative perspective which allows us to fully capture complex dynamics that might drive distribution shifts in real-world systems. Our approach is general enough to capture various sources of exogenous and endogenous distribution shift including the feedback loop between algorithms and data distributions studied in the literature on performative prediction [Per+20; IYZ21; Ray+22; Nar+22; MPZ21], the strategic interactions studied in strategic classification [Har+16; Don+18], and also endogenous factors like intra-population dynamics and distributional shifts. Indeed, while previous works have studied these phenomena in isolation, our method allows us to capture all of them as well as their interactions. For example, in [Zrn+21], the authors investigate the effects of dynamics in strategic classification problems—but the model they analyze does not capture individual interactions in the population. In [IYZ21], the authors model the interaction between a population that repeatedly responds to algorithmic decision-making by shifting its mean. Additionally, [Ray+22] study settings in which the population has both exogenous and endogenous distribution shifts due to feedback, but much like the other cited work, the focus remains on average performance. Each of these works fails to account for diffusion or intra-population interactions that can result in important qualitative changes to the distribution.
Contributions. Our approach to this problem relies on a detailed non-local PDE model of the data distribution which captures each of these factors. One term driving the evolution of the distribution over time captures the response of the population to the deployed algorithm, another draws on models used in the PDE literature for describing non-local effects and consensus in biological systems to model intra-population dynamics, and the last captures a background source of distribution shift. This is coupled with an ODE, lifted to a PDE, which describes the training of a machine learning algorithm results in a coupled PDE system which we analyze to better understand the behaviors that can arise among these interactions.
In one subcase, our model exhibits a joint gradient flow structure, where both PDEs can be written as gradients flows with respect to the same joint energy, but considering infinite dimensional gradients with respect to the different arguments. This mathematical structure provides powerful tools for analysis and has been an emerging area of study with a relatively small body of prior work, none of which related to distribution shifts in societal systems, and a general theory for multi-species gradient flows is still lacking. We give a brief overview of the models that are known to exhibit this joint gradient flow structure: in [DS20] the authors consider a two-species tumor model with coupling through Brinkman's Law. A number of works consider coupling via convolution kernels [FF13; Giu+22; JPZ22; CHS18; DT20; Dou+23] and cross-diffusion [LY22; AB21; MKB14], with applications in chemotaxis among other areas. In the models we consider here, the way the interaction between the two populations manifests is neither via cross-diffusion, nor via the non-local self-interaction term. A related type of coupling has recently appeared in [HPS22; HPS23], however in the setting of graphs. Recent work [Dom+21] provides particle-based methods to approximately compute the solution to a minimax problem where the optimization space is over measures; following that work, [WC23] provides another particle-based method using mirror descent-ascent to solve a similar problem. Other recent work [Lu23] proves that a mean-field gradient ascent-descent scheme with an entropy annealing schedule converges to the solution of a minimax optimization problem with a timescale separation parameter that is also time-varying; in contrast, our work considers fixed timescale separation setting. [GG23] show that the mean-field description of a particle method for solving minimax problems has proveable convergence guarantees in the Wasserstein-Fisher-Rao metric. Each of these references considers an energy functional that is linear in the distribution of each species respectively; our energy includes nonlinearities in the distributions via a self-interaction term as well as diffusion for the population. Moreover, the above works introduce a gradient flow dynamic as a tool for obtaining and characterizing the corresponding steady states, whereas in our setting we seek to capture the time-varying behavior that models distributions shifts. In the other subcase, we prove exponential convergence in two competitive, timescale separated settings where the algorithm and strategic population have conflicting objectives. We show numerically that retraining in a competitive setting leads to polarization in the population, illustrating the importance of fine-grained modeling.
Machine learning algorithms that are deployed into the real world for decision-making often become part of complex feedback loops with the data distributions and data sources with which they interact. In an effort to model these interactions, consider a machine learning algorithm that has loss given by  $L(z,x)$  where  $x\in \mathbb{R}^d$  are the algorithm parameters and  $z\in \mathbb{R}^d$  are the population attributes, and the goal is to solve
$$
\operatorname *{argmin}_{x\in \mathcal{X}}\underset {z\sim \rho}{\mathbb{E}}L(z,x),
$$
where  $\mathcal{X}$  is the class of model parameters and  $\rho(z)$  is the population distribution. Individuals have an objective given by  $J(z, x)$  in response to a model parameterized by  $x$ , and they seek to solve
$$
\operatorname *{argmin}_{z\in \mathbb{R}^{d}}J(z,x).
$$
When individuals in the population and the algorithm have access to gradients, we model the optimization process as a gradient-descent-type process. Realistically, individuals in the population will have nonlocal information and influences, as well as external perturbations, the effects of which we seek to capture in addition to just minimization. To address this, we propose a partial differential equation (PDE) model for the population, that is able to capture nonlocal interactions between individuals on the level of a collective population. To analyse how the population evolves over time, a notion of derivative in infinite dimensions is needed. A natural, and in this context physically meaningful, way of measuring the dissipation mechanism for probability distributions is the Wasserstein-2 metric (see Definition 4). The following expression appears when computing the gradient of an energy functional with respect to the Wasserstein-2 topology.
Definition 1. [First Variation] For a map  $G: \mathcal{P}(\mathbb{R}^d) \mapsto \mathbb{R}$  and fixed probability distribution  $\rho \in \mathcal{P}(\mathbb{R}^d)$ , the first variation of  $G$  at the point  $\rho$  is denoted by  $\delta_{\rho}G[\rho]: \mathbb{R}^d \to \mathbb{R}$ , and is defined via the relation
$$
\int \delta_ {\rho} G [ \rho ] (z) \psi (z) \mathrm {d} z = \lim _ {\epsilon \rightarrow 0} \frac {1}{\epsilon} (G (\rho + \epsilon \psi) - G (\rho))
$$
for all  $\psi \in C_c^\infty (\mathbb{R}^d)$  such that  $\int \mathrm{d}\psi = 0$ , assuming that  $G$  is regular enough for all quantities to exist.
Here,  $\mathcal{P}(\mathbb{R}^d)$  denotes the space of probability measures on the Borel sigma algebra. Using the first variation, we can express the gradient in Wasserstein-2 space, see for example [Vil03, Exercise 8.8].
Lemma 1. The gradient of an energy  $G: \mathcal{P}_2(\mathbb{R}^d) \to \mathbb{R}$  in the Wasserstein-2 space is given by
$$
\nabla_ {W _ {2}} G (\rho) = - \mathrm {d i v} \left(\rho \nabla \delta_ {\rho} G [ \rho ]\right).
$$
Here,  $\mathcal{P}_2(\mathbb{R}^d)$  denotes the set of probability measures with bounded second moments, also see Appendix A.2. As a consequence, the infinite dimensional steepest descent in Wasserstein-2 space can be expressed as the PDE
$$
\partial_ {t} \rho = - \nabla_ {W _ {2}} G (\rho) = \operatorname {d i v} \left(\rho \nabla \delta_ {\rho} G [ \rho ]\right). \tag {1}
$$
All the coupled gradient flows considered in this work have this Wasserstein-2 structure. In particular, when considering that individuals minimize their own loss, we can capture these dynamics via a gradient flow in the Wasserstein-2 metric on the level of the distribution of the population. Then for given algorithm parameters  $x \in \mathbb{R}^d$ , the evolution for this strategic population is given by
$$
\partial_ {t} \rho = \operatorname {d i v} \left(\rho \nabla \delta_ {\rho} \left[ \underset {z \sim \rho} {\mathbb {E}} J (z, x) + E (\rho) \right]\right), \tag {2}
$$
where  $E(\rho)$  is a functional including terms for internal influences and external perturbations. In real-world deployment of algorithms, decision makers update their algorithm over time, leading to an interaction between the two processes. We also consider the algorithm dynamics over time, which we model as
$$
\dot {x} = - \nabla_ {x} \left[ \underset {z \sim \rho} {\mathbb {E}} L (z, x) \right]. \tag {3}
$$
In this work, we analyze the behavior of the dynamics under the following model. The algorithm suffers a cost  $f_{1}(z,x)$  for a data point  $z$  under model parameters  $x$  in the strategic population, and a cost  $f_{2}(z,x)$  for a data point in a fixed, non-strategic population. The strategic population is denoted by  $\rho \in \mathcal{P}$ , and the non-strategic population by  $\bar{\rho} \in \mathcal{P}$ . The algorithm aims to minimize
$$
\underset {z \sim \rho} {\mathbb {E}} L (z, x) = \int f _ {1} (z, x) \mathrm {d} \rho (z) + \int f _ {2} (z, x) \mathrm {d} \bar {\rho} (z) + \frac {\beta}{2} \left\| x - x _ {0} \right\| ^ {2},
$$
where the norm is the vector inner product  $\| x\|^2 = \langle x,x\rangle$  and  $\beta >0$  weights the cost of updating the model parameters from its initial condition.
We consider two settings:  $(i)$  aligned objectives, and  $(ii)$  competing objectives. Case  $(i)$  captures the setting in which the strategic population minimization improves the performance of the algorithm, subject to a cost for deviating from a reference distribution  $\tilde{\rho} \in \mathcal{P}$ . This cost stems from effort required to manipulate features, such as a loan applicant adding or closing credit cards. On the other hand, Case  $(ii)$  captures the setting in which the strategic population minimization worsens the performance of the algorithm, again incurring cost from distributional changes.
2.1 Case (i): Aligned Objectives
In this setting, we consider the case where the strategic population and the algorithm have aligned objectives. This occurs in examples such as recommendation systems, where users and algorithm designers both seek to develop accurate recommendations for the users. This corresponds to the population cost
$$
\underset {z \sim \rho , x \sim \mu} {\mathbb {E}} J (z, x) = \iint f _ {1} (z, x) \mathrm {d} \rho (z) \mathrm {d} \mu (x) + \alpha K L (\rho \mid \tilde {\rho}),
$$
where  $KL(\cdot|\cdot)$  denotes the Kullback-Leibler divergence. Note that the KL divergence introduces diffusion to the dynamics for  $\rho$ . The weight  $\alpha > 0$  parameterizes the cost of distribution shift to the population. To account for nonlocal information and influence among members of the population, we include a kernel term  $E(\rho) = \frac{1}{2}\int \rho W*\rho \mathrm{d}z$ , where  $(W*\rho)(z) = \int W(z - \bar{z})\mathrm{d}\rho (\bar{z})$  is a convolution integral and  $W$  is a suitable interaction potential.
2.2 Case (ii): Competing Objectives
In settings such as online internet forums, where algorithms and users have used manipulative strategies for marketing [Del06], the strategic population may be incentivized to modify or mis-report their attributes. The algorithm has a competitive objective, in that it aims to maintain performance against a population whose dynamics cause the algorithm performance to suffer. When the strategic population seeks an outcome contrary to the algorithm, we model strategic population cost as
$$
\underset {z \sim \rho , x \sim \mu} {\mathbb {E}} J (z, x) = - \iint f _ {1} (z, x) \mathrm {d} \rho (z) \mathrm {d} \mu (x) + \alpha K L (\rho \mid \tilde {\rho}).
$$
A significant factor in the dynamics for the strategic population is the timescale separation between the two "species"—i.e., the population and the algorithm. In our analysis, we will consider two cases: one, where the population responds much faster than the algorithm, and two, where the algorithm responds much faster than the population. We illustrate the intermediate case in a simulation example.
We are interested in characterizing the long-time asymptotic behavior of the population distribution, as it depends on the decision-makers action over time. The structure of the population distribution gives us insights about how the decision-makers actions influences the entire population of users. For instance, as noted in the preceding sections, different behaviors such as bimodal distributions or large tails or variance might emerge, and such effects are not captured in simply looking at average performance. To understand this intricate interplay, one would like to characterize the behavior of both the population and the algorithm over large times. Our main contribution towards this goal is a novel analytical framework as well as analysis of the long-time asymptotics.
A key observation is that the dynamics in (2) and (3) can be re-formulated as a gradient flow; we lift  $x$  to a probability distribution  $\mu$  by representing it as a Dirac delta  $\mu$  sitting at the point  $x$ . As a result, the evolution of  $\mu$  will be governed by a PDE, and combined with the PDE for the population, we obtain a system of coupled PDEs,
$$
\partial_ {t} \rho = \operatorname {d i v} \left(\rho \nabla_ {z} \delta_ {\rho} \big [ \underset {z \sim \rho , x \sim \mu} {\mathbb {E}} J (z, x) + E (\rho) \big ]\right)
$$
$$
\partial_ {t} \mu = \mathrm {d i v} \left(\mu \nabla_ {x} \delta_ {\mu} \big [ \underset {z \sim \rho , x \sim \mu} {\mathbb {E}} L (z, x) \big ]\right),
$$
where  $\delta_{\rho}$  and  $\delta_{\mu}$  are first variations with respect to  $\rho$  and  $\mu$  according to Definition 1. The natural candidates for the asymptotic profiles of this coupled system are its steady states, which - thanks to the gradient flow structure - can be characterized as ground states of the corresponding energy functionals. In this work, we show existence and uniqueness of minimizers (maximizers) for the functionals under suitable conditions on the dynamics. We also provide criteria for convergence and explicit convergence rates. We begin with the case where the interests of the population and algorithm are aligned, and follow with analogous results in the competitive setting. We show convergence in energy, which in turn ensures convergence in a product Wasserstein metric. For convergence in energy, we use the notion of relative energy and prove that the relative energy converges to zero as time increases.
Definition 2 (Relative Energy). The relative energy of a functional  $G$  is given by  $G(\gamma |\gamma_{\infty}) = G(\gamma) - G(\gamma_{\infty})$ , where  $G(\gamma_{\infty})$  is the energy at the steady state.
Since we consider the joint evolution of two probability distributions, we define a distance metric  $\overline{W}$  on the product space of probability measures with bounded second moment.
Definition 3 (Joint Wasserstein Metric). The metric over  $\mathcal{P}_2(\mathbb{R}^d) \times \mathcal{P}_2(\mathbb{R}^d)$  is called  $\overline{W}$  and is given by
$$
\overline {{W}} ((\rho , \mu), (\tilde {\rho}, \tilde {\mu})) ^ {2} = W _ {2} (\rho , \tilde {\rho}) ^ {2} + W _ {2} (\mu , \tilde {\mu}) ^ {2}
$$
for all pairs  $(\rho, \mu), (\tilde{\rho}, \tilde{\mu}) \in \mathcal{P}_2(\mathbb{R}^d) \times \mathcal{P}_2(\mathbb{R}^d)$ , and where  $W_2$  denotes the Wasserstein-2 metric (see Definition 4). We denote by  $\overline{\mathcal{W}}(\mathbb{R}^d) := (\mathcal{P}_2(\mathbb{R}^d) \times \mathcal{P}_2(\mathbb{R}^d), \overline{W})$  the corresponding metric space.
3.1 Gradient Flow Structure
In the case where the objectives of the algorithm and population are aligned, we can write the dynamics as a gradient flow by using the same energy functional for both species. Let  $G_{a}(\rho ,\mu): \mathcal{P}(\mathbb{R}^{d}) \times \mathcal{P}(\mathbb{R}^{d}) \mapsto [0,\infty]$  be the energy functional given by
$$
\begin{array}{l} G _ {a} (\rho , \mu) = \iint f _ {1} (z, x) \mathrm {d} \rho (z) \mathrm {d} \mu (x) + \iint f _ {2} (z, x) \mathrm {d} \bar {\rho} (z) \mathrm {d} \mu (x) + \alpha K L (\rho | \bar {\rho}) + \frac {1}{2} \int \rho W * \rho \\ + \frac {\beta}{2} \int \| x - x _ {0} \| ^ {2} d \mu (x). \\ \end{array}
$$
This expression is well-defined as the relative entropy  $KL(\rho |\tilde{\rho})$  can be extended to the full set  $\mathcal{P}(\mathbb{R}^d)$  by setting  $G_{a}(\rho ,\mu) = +\infty$  in case  $\rho$  is not absolutely continuous with respect to  $\tilde{\rho}$ .
In the competitive case we define  $G_{c}(\rho, x): \mathcal{P}(\mathbb{R}^{d}) \times \mathbb{R}^{d} \mapsto [-\infty, \infty]$  by
$$
G _ {c} (\rho , x) = \int f _ {1} (z, x) \mathrm {d} \rho (z) + \int f _ {2} (x, z ^ {\prime}) \mathrm {d} \bar {\rho} (z ^ {\prime}) - \alpha K L (\rho | \tilde {\rho}) - \frac {1}{2} \int \rho W * \rho + \frac {\beta}{2} \| x - x _ {0} \| ^ {2}.
$$
In settings like recommender systems, the population and algorithm have aligned objectives; they seek to minimize the same cost but are subject to different dynamic constraints and influences, modeled by the regularizer and convolution terms. In the case where the objectives are aligned, the dynamics are given by
$$
\partial_ {t} \rho = \operatorname {d i v} \left(\rho \nabla_ {z} \delta_ {\rho} G _ {a} [ \rho , \mu ]\right)
$$
$$
\begin{array}{l} \partial_ {t} p = \operatorname {d i v} (\rho \nabla_ {z} \delta_ {\rho} G _ {a} [ \rho , \mu ]) \\ \partial_ {t} \mu = \operatorname {d i v} (\mu \nabla_ {x} \delta_ {\mu} G _ {a} [ \rho , \mu ]). \end{array} \tag {4}
$$
Note that (4) is a joint gradient flow, because the dynamics can be written in the form
$$
\partial_ {t} \gamma = \operatorname {d i v} \left(\gamma \nabla \delta_ {\gamma} G _ {a} (\gamma)\right),
$$
where  $\gamma = (\rho, \mu)$  and where the gradient and divergence are taken in both variables  $(z, x)$ . We discuss the structure of the dynamics (4) as well as the meaning of the different terms appearing in the energy functional  $G_{a}$  in Appendix A.1.
In other settings, such as credit score reporting, the objectives of the population are competitive with respect to the algorithm. Here we consider two scenarios; one, where the algorithm responds quickly relative to the population, and two, where the population responds quickly relative to the algorithm. In the case where the algorithm can immediately adjust optimally (best-respond) to the distribution, the dynamics are given by
$$
\partial_ {t} \rho = - \operatorname {d i v} \left(\rho (\nabla_ {z} \delta_ {\rho} G _ {c} [ \rho , x ]) | _ {x = b (\rho)}\right),
$$
$$
b (\rho) := \underset {\bar {x}} {\operatorname {a r g m i n}} G _ {c} (\rho , \bar {x}). \tag {5}
$$
Next we can consider the population immediately responding to the algorithm, which has dynamics
$$
\frac {\mathrm {d}}{\mathrm {d} t} x = - \nabla_ {x} G _ {c} (\rho , x) | _ {\rho = r (x)},
$$
$$
r (x) := \underset {\hat {\rho} \in \mathcal {P}} {\operatorname {a r g m i n}} - G _ {c} (\hat {\rho}, x). \tag {6}
$$