Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training

Authors: anchit  jain
rozhin  nobahari
stefano  sarao mannelli
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:45:50

Abstract: Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations of the data. However, our current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup that models different data sub-populations with a Gaussian-mixture model. We provide an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setup, which we prove to be exact in high dimension.
Notably, our analysis identifies different properties of the sub-populations that drive bias at different timescales and hence shows a shifting preference of our classifier during training. By applying our general solution to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real data, i.e. using CIFAR10, MNIST, and CelebA datasets.

Main paper: 
 introduction: 
 Over the past decade, the problem of assessing the fairness of classifiers has garnered significant attention, revealing that machine learning (ML) systems not only reproduce existing biases in the data but also tend to amplify them [21, 40, 11] . Given the complexity of the ML pipeline, isolating and characterising the key drivers of this amplification is challenging. Recent studies have begun to disentangle the contributions from architectural design choices, including overparameterisation [37] , model complexity, activation functions [5, 12] , learning protocols [43, 13] , post-processing practices such as pruning [19] , and intrinsic aspects of the data like its geometrical properties [38] .Theoretical results in this area (e.g., [37, 38] ) are mostly based on asymptotic analysis, leaving the transient learning regime poorly understood. Due to limitations on computational resources, a trained ML system may operate far from the asymptotic regime and hence existing results may not always apply. Insights from class imbalance literature [43, 12] indicate that classifiers converge faster for classes with more data, but how this applies to fairness, where datasets might be balanced by label but imbalanced by demographics, remains unclear.Our analysis addresses this gap by providing a precise characterisation of the transient dynamics of online stochastic gradient descent (SGD) in a high dimensional prototypical model of linear classification. We use the teacher-mixture (TM) framework [38] , where different data sub-populations are modeled with a mixture of Gaussians, each having its own linear rule (teacher) for determining the labels. Adjusting the parameters of the data distribution in our framework connects models of fairness and spurious correlations, providing a unifying framework and a general set of results applicable to both domains. Remarkably, our study reveals a rich behaviour divided into three learning phases, where different features of data bias the classifier and causing significant deviations from asymptotic predictions. We reproduce our theoretical findings through numerical experiments in more complex settings, demonstrating validity beyond the simplicity of our model.Our key contributions are:• High-dimensional analysis: We demonstrate that in the high-dimensional limit, relevant properties of the classifier, such as the generalisation error, can be expressed using a few sufficient statistics. We prove that their evolution converges to a set of ordinary differential equations (ODEs) that can be solved explicitly in our setting.• Bias evolution characterisation: Using our solution, we characterise the evolution of bias throughout training, showing a three-phase learning process where bias exhibits nonmonotonic behaviour. Specifically:1. Initial phase: The classifier is initially influenced by sub-populations with strong class imbalance. 2. Intermediate phase: The dynamics shifts towards the saliency, or norm, of the samples in a sub-population. 3. Final phase: Sub-population imbalance, or relative representation, becomes the dominant factor.• Empirical validation: We validate and extend our theoretical results through numerical experiments in both synthetic and real datasets, including CIFAR10, MNIST, and CelebA.Altogether, our study reveals a complex time-dependence of learning with structured data that previous theoretical studies have failed to capture. This characterisation is crucial for developing effective bias mitigation strategies, especially under limited computational resources.Class imbalance and fairness. A key element in our study is the presence of heterogeneous data distributions within the dataset. In the context of fairness, these distributions model different groups in a population. Sampling unbalance is particularly critical, as minority groups are often misclassified [9, 20] . However, theoretical studies on group imbalance have been limited to asymptotic analyses [38] , which may not apply in practical settings. Related questions have been explored in the label imbalance literature [22] , where it has long been known [1, 17] that underrepresented classes have slower convergence rate and may even experience increased errors early in training. Our work shows that pre-asymptotic analysis can reveal complex transient dynamics, which is practically relevant when learning slows down or training to convergence is not possible. Similar to our analysis, [12] has shown that supposedly neutral choices, like activation functions or pooling operations, can generate strong biases. In contrast to prior work, our focus on data properties identifies several timescales associated to different data features relevant to bias generation.Simplicity bias. Several studies [31, 16, 41, 10, 32] have highlighted a bias of deep neural networks (DNNs) towards simple solutions, suggesting this bias is a key to their generalisation performance. Simplicity bias also influences learning dynamics: [4, 32, 28, 30, 33] have showed that DNNs learn progressively more complex functions during training, with a notion of complexity often defined implicitly by other DNNs or observations like the time to memorisation. Our results connect with simplicity bias by identifying interpretable properties of the data that make samples appear "simple" to a shallow network. Interestingly, our findings reveal that different phases of learning experience simplicity in different ways, leading to forgetting of previously learned features.Spurious correlations. Simplicity bias can also lead to shortcomings [39] by excessively relying of spurious features in the data, possibly hurting generalisation, especially in out-of-distribution contexts [14] . Theoretical works [29, 37, 18] have identified statistical properties that cause a classifier to favour spurious features over potentially more complex but more predictive features.Various methods have been proposed to address this problem using explicit partitioning of the data [2, 36] ; some approaches implicitly infer subgroups with various degrees of correlation as spurious features. Notably, [26, 42] rely on early stages of learning to detect bias and adjust sample importance accordingly. Our study provides a unifying view of learning in fairness and spurious correlation problems, highlighting the presence of ephemeral biases characterised by multiple timescales during training. This adds complexity to the understanding of learning dynamics and points out potential confounding effects in existing mitigation methods.  
problem setup: 
 Data distribution. We consider a standard supervised learning setup where the training data consists of pairs of a feature vector x x x ∈ R d and a binary label y = ±1. To model subgroups within the data [35] , we assume that the feature vectors are structured as clusters c 1 , . . . , c m , respectively centered on some fixed attribute vectorsv v v 1 , • • • , v v v m ∈ R d . Specifically, x xx is sampled from a mixture of m isotropic Gaussians:EQUATIONwith mixing probabilitiesρ 1 , • • • , ρ m and scalar variances ∆ 1 , • • • , ∆ m .Assuming the entries of v v v j are of order 1 as d gets large, the scaling factor 1/ √ d ensures that the Euclidean norm of the renormalised vector is of order 1. This prevents the problem from becoming either trivial or overly challenging in the high-dimensional limit [25, 24] . We adopt a teacher-mixture (TM) scenario [38] where each cluster has its own teacher rule:EQUATION)This rule is characterised by the teacher vectors w w w j ∈ R d , ensuring linear separability within each cluster. Fig. 1b -d illustrate the data distribution for two clusters with opposite mean vectors ±v v v, which will be the primary case study for our analysis.Model. In this study we analyse a linear model applied to the above data distribution. We aim to learn a vector parameter w w w, referred to as the 'student', such that predictions are given byEQUATIONThe training process involves applying online SGD on the squared loss ε = (y -ŷ) 2 . At the k-th iteration, a feature vector x x x k is sampled from (1), the ground truth label y k and current model prediction ŷk are respectively given by ( 2) and (3), and the parameter is updated as:∆w w w k := w w w k+1 -w w w k = - η 2 ∇ε k (w w w k ) = η √ d (y k -ŷk ) x x x k (4)where η/2 > 0 denotes the learning rate. It is important to note that in this online setting the number of time steps is equivalent to the number of training examples. In our analysis, the model is evaluated by its generalisation error, or population loss, ϵ := E[ε]. 
sgd analysis: 
 We study the evolution of the generalisation error during training with SGD with constant learning rate in the high dimensional setting (i.e. large d). Following a classical approach [34, 8] , we streamline the problem by focusing on a small set of summary statistics, referred to as 'order parameters', which fully characterises the dynamics. As the dimension increases, it can be shown by concentration arguments that the evolution of these order parameters converges to the deterministic solution of a system of ODEs [15, 6, 3] . Notably, in our setting, we achieve an analytical solution of this ODE system. We sketch our main results below, referring to the Appendix for derivations and proofs.In the setup described in Section 2, consider the following 2m + 1 variables:EQUATIONfor 1 ≤ j ≤ m. These variables correspond to key statistics of the student, namely its alignment to the cluster teachers, its alignment to the cluster centers, and its magnitude, respectively. Lemma 3.1. The generalisation error can be written as an average ϵ = m j=1 ρ j ϵ j over the clusters, where ϵ j is a degree 2 polynomial in R j , M j and Q taking the formEQUATIONwhere α j , β j are constants independent of the parameter w w w.We present the derivation of this result and the explicit form of the constants α j , β j in Appendix B.1.Our problem thus reduces to characterising the evolution of order parameters (5) . Using the gradient update of the parameter in Eq. 4 and the notation δ k := y k -ŷk , we can write update equations for the order parameters as follows:∆M k j = η d δ k v v v ⊤ j x x x k √ d , ∆R k j = η d δ k w w w ⊤ j x x x k √ d , ∆Q k = 2η d δ k w w w ⊤ j x x x k √ d + η 2 d 2 (δ k ) 2 ∥x x x k ∥ 2 . (7)We build upon classic results [34, 8] , recently put on rigorous grounds [15, 6, 3] , leveraging the self-averaging property of the order parameters in the high dimensional limit d → ∞. As a result, as the dimension gets large, the discrete, stochastic evolution (7) of the order parameters can be effectively described in terms of the deterministic solution of the average continuous-time dynamics.Let S := (S i ) 1≤i≤2m+1 denote the collection of order parameters. The following lemma shows that the average of the updates (7) over the sample x x x k can be expressed solely in terms ofS k . Lemma 3.2. E[∆S k i ] = 1 d f i (S k ) for some functions (f i (S)) 1≤i≤2m+1 in O(1) as d → ∞.The theorem below states that as d gets large, the stochastic evolution S k of the order parameter gets uniformly close, with high probability, to the average continuous-time dynamics described by the ODE system:EQUATIONwhere the continuous time is given by the example number divided by the input dimension, t = k/d. Formally,EQUATIONwhere P -→ denotes convergence in probability. A proof is provided in Appendix B. We provide the explicit expression of the functions f i in the ODEs (8) in Appendix C, focusing on m = 2 clusters for clarity.Here we present the explicit solution of the ODEs (8) in the case of two clusters (m = 2) with opposite mean vectors ±v v v, as in [38] . Henceforth, we refer to v v v as the shift vector and to the two clusters as the 'positive' and 'negative' sub-populations, with mixing probabilities ρ and (1 -ρ), variances ∆ ± and teacher vectors w w w ± , respectively. The order parameters introduced in Eq. 5 are specifically denoted as M = w w w ⊤ v v v/d, R + = w w w ⊤ w w w + /d, and R -= w w w ⊤ w w w -/d in this setting. Theorem 3.4. In the above setting, solutions to the order parameter evolution take the formEQUATIONEQUATIONEQUATIONwith∆ mix = ρ∆ + + (1 -ρ)∆ -, ∆ 2mix = ρ∆ 2 + + (1 -ρ)∆ 2 -and v = ||v v v|| 2 /d.The remaining constants are less significant and are reported in Appendix D.1 and discussed further in Appendix E. This solution allows us to describe important observables such as the generalisation error (via Lemma 3.1) at any timestep. Fig. 1a plots the theoretical closed-form solutions along with values obtained through simulation when we set d = 1000. Note the remarkable agreement between the analytical ODE solution and simulations of the online SGD dynamics in this high dimensional data limit. In this section, we delve deeper into the solution derived in Theorem 3.4. By examining the exponents in Eqs. 10-12, we can identify the relevant training timescales. Notably, M follows a straightforward behaviour dominated by a single timescale, whereas R ± and Q exhibit multiple timescales, leading to significant implications for the emergence and evolution of bias during training.In the following sections, we analyse increasingly complex scenarios to understand how bias develops and evolves. Parameters specifying these different scenarios are the shift norm v = ||v v v|| 2 /d and relative representation ρ, the subpopulation variances ∆ ± , and the teacher overlap T ± = w w w ⊤ + w w w -/d. For simplicity we fix the teacher norm ∥w w w ± ∥ 2 = √ d, so that T ± is the cosine similarity between the two teachers. 
spurious correlations: 
 The emergence of spurious correlations during training exemplifies a type of bias where a classifier favours a spurious feature over a core one. To isolate the impact of spurious correlation in our model while avoiding confounding effects, we consider perfectly overlapping teachers (w w w + = w w w -) and subpopulations with equal variance and representation (ρ = 0.5, ∆ + = ∆ -). With non-perfectly overlapping clusters v ̸ = 0, we introduce a spurious correlation by adding a small cosine similarity between the shift vector and the teacher, creating a label imbalance-an imbalance between the proportion of positive and negative labels-within each sub-population. The setting is illustrated in Fig. 1b .From Eqs. 10-12, two relevant timescales for the problem are observed:EQUATIONEQUATION)The shortest timescale, τ M , associated with M , indicates that the student first aligns with the spurious feature. By aligning with the shift vector, the student can predict most examples correctly, but not all. The effect of spurious correlations is transient; at t ∼ τ R , the student starts disaligning from the spurious feature and aligns with the teacher vector, eventually achieving nearly perfect alignment. This is illustrated in Fig. 2 , where the student initially picks up on the spurious correlation (green) and achieves almost perfect alignment with the shift vector during intermediate times before aligning nearly perfectly with the teacher (red).In this section, we identify the properties of sub-populations that determine the bias during learning and show how bias evolves in three phases. To quantify bias, we use the overall accuracy equality metric [7] , which measures the discrepancy in accuracy across groups. Intuitively, we aim for equal loss on both groups, considering any deviation from this condition as bias.We first consider a simplified case where we assume that both clusters are centered at the origin v = 0 as shown in Fig. 1c . We will later reintroduce the shift and analyse the transient dynamics it introduces as per the discussion in section 4.1. The zero shift case represents an extreme situation where not only is the classification not solvable if w w w + ̸ = w w w -, but it is also difficult to identify which cluster generated a given data point since the shift provides no information. This setting is particularly suited to analysing the effects of 'group level' features, such as group variance and relative representation, on the preference of the classifier.In this simplified setting, M (t) is always zero and the constants k ± 1 , k 3 , k 4 presented in equations 11 and 12 are zero. Thus, the dynamics only involve two relevant timescales given by τ R in Eq. 14 andEQUATIONFig. 3a illustrates the changing preference of the classifier. Specifically, we observe that the variance of the sub-population is particularly relevant initially and the sub-population with higher variance (red) is learnt faster, i.e. its generalisation error drops faster. However, asymptotically we observe that the relative representation becomes more important wherein the student aligns itself with the teacher that has a higher product of representation and standard deviation (blue), i.e.EQUATIONThus, the network can advantage the cluster with higher variance initially but asymptotically advantage the other cluster if its representation is high enough. This leads to the interesting behaviour shown in Fig. 3 wherein we observe a 'crossing' of the losses on the two sub-populations. A more detailed analysis of the 'crossing' is presented in Appendix E.2.Starting from small initialisation, the initial rate of change of the generalisation error for sub-population + isEQUATIONand analogously for -. The learning rate η must be chosen to be small enough such that the generalisation errors decrease and hence the first term in the brackets must dominate over the 1. SinceR ∞ + /R ∞ -∈ [T ± ; 1/T ± ] (for T ± > 0), the ratio between generalisation error rates is bounded byEQUATIONWhen the teachers are only slightly misaligned-T ± ⪅ 1-the bound is tight and we can see that it is the ratio of the square roots of the variances that determines which cluster is learnt faster initially.As precisely detailed below, the initial bias can substantially differ from the asymptotic bias of the classifier. Indeed, Fig. 3b shows in a phase diagram the existence of 'bias crossing' across a wide range of variances and representations. The transition between the phases that represent a initial preference for the positive sub-population (light red and dark blue) and the phases that represent an initial preference for negative sub-population (dark red and light blue) is approximately given by the line ∆ -= ∆ + = 1, independent of the representation as predicted by Eq. 18. The portion of the dark blue phase just above the white divergent phase marks a 'quasi-divergent' region wherein the generalisation error on the negative sub-population rises even at t = 0 because the learning rate is too large for such high variances. It hence marks a region of impractical behaviour that is only observed with poorly optimised learning rates.Asymptotic preference. In the limit of small learning rates η → 0, the student will asymptotically exhibit lower loss on whichever sub-population's teacher it has better alignment with. Thus, Eq. 16 provides a simple characterisation of asymptotic preference from representations and standard deviations in the small learning rate limit. However, the situation is more complex in the case of finite learning rate, which may disrupt learning in one or both clusters. Without indulging further into the discussion of asymptotic performance, which is not the main goal of the paper, we refer to Appendix E.3 for more in depth and analysis and additional phase diagrams.We now consider the general case shown in Fig. 1d , where the shift is non zero and all three timescales identified so far play a role.As observed in Sec. 4.1, when the shift norm v is large, the effect of spurious correlations becomes significant and the timescale associated with the spurious correlations is the fastest. In general, when v ̸ = 0 we observe an additional phase due to the effect of spurious correlation. In this new first phase, the student advantages the cluster with higher representation and lower variance since the salient information received from this cluster is more coherent and easier to access. More precisely, in high dimensions the shift and the teachers are likely to exhibit a small cosine similarity leading to a class imbalance in the clusters and creating spurious correlation. The amount of label imbalance within a cluster is characterised by the value of α, as detailed in Appendix A. For smaller variances, α takes more extreme values leading to stronger spurious correlation of that cluster with the shift. If a cluster has more positive examples, we would observe a reduction in loss for that cluster if the student aligns with the mean of that cluster (and opposite to the mean if the cluster has mostly negative examples). When both clusters have different majority classes, the direction of spurious correlation for the two are same. However, when the majority classes are the same, we have competing directions for spurious correlation. The expression for M ∞ in Appendix D.1 Eq. D.42 shows that in this case the relative representation comes into play and the mean of the cluster with greater representation and class imbalance will be chosen by the teacher to align with. Fig. 4 shows such a scenario with three phase bias evolution:• The green phase is driven by spurious correlation where the positive cluster is advantaged since it has greater representation and class imbalance. • Then, the red phase is driven by greater variance where the negative cluster is learnt faster as discussed through Eq. 18.• Finally, we observe the orange phase wherein the student starts aligning with the positive cluster as per the asymptotic rule in Eq. 16 .In summary, the student first advantages the sub-population with higher representation and lower variance. Next, it advantages the sub-population with higher variance. Asymptotically, it advantages the sub-population with higher representation and variance. Our analysis thus shows that bias is a dynamical quantity that can vary non-monotonically during training and cannot be characterised by simply the initial and asymptotic values.5 Ablations using numerical simulations 
rotated mnist: 
 We train a 2-layer neural network with 200 hidden units, ReLU activation, and sigmoidal readout activation, using online SGD on MSE loss in a MNIST classification task. Data are centered and the variance is set to 1 following standard pre-processing practices. We consider a variation of the MNIST dataset that mimics the TM model, allowing us to verify our theory when network and the data structure are mismatched. Digits 0 to 4 and 5 to 9 are grouped to form the two subpopulations. With probability p + and p -, digits of both subpopulations are rotated with a subpopulation-specific angle-i.e. Fig. 5a uses angles of rotation θ -= 45 o and θ -= -90 o . The goal of the classifier is to detect rotations.The experimental framework gives a correspondence between parameters of the generative model and properties of a real dataset. We can control relative representation by subsampling, teacher similarity by playing with angle difference, label imbalance by changing the probability of rotation, and saliency by increasing and decreasing the norm of the subpopulation using multiplicative factors ∆ ± . The only parameter that we cannot control is the shift v v v which is a property of the data. Therefore, in order to reproduce the zero-shift case of Sec. 4.2, we remove the label imbalance by setting the probability of rotation p + = p -= 0.5. By properly calibrating the saliency ∆ and the relative representation ρ, it is possible to bias the classifier towards one subpopulation at the beginning of training and the other in the end. This is shown in Fig. 5a where ρ = 0.1 and ∆ + > ∆ -. The saliency difference favours subpopulation + initially while setting ρ small enough advantages subpopulationlater in training. This is precisely what we observe in the plot.In Fig. 5b , we extend the MNIST experiment by varying the average image brightness of subpopulation -, which reflects the group variance in our theory. The results show that greater brightness leads to faster learning in the second phase and an increasing asymptotic preference, consistent with our predictions.Finally, we consider the general fairness case. By creating label imbalance, i.e. setting p + = 0.3 and p -= 0.7, we observe an additional phase of bias evolution, wherein the classifier prefers dense regions with consistent labels. This advantages subpopulationand indeed it is what we see in Fig. 5c . The result of the simulations matches the theory displaying a double crossing phenomenon.We consider the same architecture and pre-processing described for MNIST on a CIFAR10 classification task. We select 8 classes and assign 4 of them to the positive group and 4 to the negative group. Inside each group, 2 classes are labelled as negative and 2 as positive. This simulation framework is similar to the one considered by [5] where the authors used sub-populations with only 2 classes each.The average brightness of the samples in each cluster plays the same role as the parameter ∆ in the synthetic model. Our theory predicts that the classifier will advantage the group with highest average brightness, see Eq. 16 . In order to achieve the same generalisation error on both subpopulations, the less bright group needs more samples (larger ρ). This is shown in Fig. 6a , where the three panels correspond to different assignment of the classes: in the top panel classes are randomly assigned to the two groups; in the middle panel classes are randomly partitioned in two groups and the brighter one is assigned to group -; finally the last panel assigns the brightest classes to groupand least bright to group +. As predicted, we need increasingly high relative representation ρ to achieve a balance in losses at the end of training.When labels are balanced, our theory predicts that the classifier is initially attracted by the larger ∆ and eventually-if the relative representation of the group with smaller ∆ is large enough-it switches and favours the other group. This effect is indeed verified in the CIFAR10 experiments.Starting from the partitioning in Fig. 6a (bottom) with ρ = 0.8, the dynamics is initially attracted by group -before advantaging the other group, giving rise to a crossing that can be observed-among other things-in Fig. 6b .Fig. 6b highlights another prediction of our theory concerning the timescale when ρ becomes relevant.Our theory predicts that ρ should become relevant only in the final stages of the dynamics and indeed the curves are almost perfectly overlapping until for most of training. As already noticed, ρ sufficiently large gives rise to a crossing behaviour as indicated demonstrated in the synthetic model.In Appendix F, we provide additional experiments within the TM model and the CelebA dataset, exploring different architectures and losses. Even under these new conditions, we observe that bias presents different timescales and shows crossing behaviours. 
conclusion: 
 This paper examined the dynamics of bias in a high dimensional synthetic framework, showing that it can be explicitly characterised to reveal transient behaviour. Our findings reveal that classifiers exhibit biases toward different data features during training, possibly alternating sub-population preference.Although our analysis is based on certain assumptions, numerical experiments that violate these assumptions still display the behaviour predicated by our theory.While this paper centered on bias propagation in a controlled synthetic setting, the study of bias in ML systems is a significant issue with profound societal implications. We believe this line of research will have practical impacts in the medium term, aiding the design of mitigation strategies that account for transient dynamics. Future research will further explore this connection, proposing theory-based dynamical protocols for bias mitigation.  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Discussion reported in the conclusion Sec. 6. This paper focuses on theoretical research, however we believe that the phenomena observed and characterised in our framework extend to a broader class of classifiers and datasets, potentially leading to a redesign of aspects of the ML pipeline in order to account for bias as a dynamical quantity.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: literature relevant for this work is properly cited, mostly in Sec. 1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We begin by refreshing the problem description and notation introduced in the main body for the two cluster case (Sec. 3.3) as well as defining some new notation to make the presentations of the results more compact.1. (x x x, y) denotes a training example with x x x ∈ R d and y ∈ {-1, 1}.2. x x x is drawn from a mixture of two Gaussians with means v v v/ √ d and -v v v/ √ d respectively, covariances ∆ + I d×d and ∆ -I d×d respectively. These two Gaussians are henceforth referred to as the positive and negative Gaussians respectively. 3. ρ represents the probability of the data being drawn from the positive Gaussian. 4. ⟨⟩ denotes an average over x, ⟨⟩ ⊕ denotes an average over the positive Gaussian and ⟨⟩ ⊖ denotes an average over the negative Gaussian. 5. w w w + and w w w -denote the teachers for the positive Gaussian and negative Gaussian respectively.w w w is the learnt classifier ("the student").6. The true labels, y, are then given by:• y = sign(w w w + • x x x/ √ d) for the positive cluster;• y = sign(w w w -• x x x/ √ d) for the negative cluster.7. Our predictions are ŷ = w w w• x x x/ √ d.8. The student is trained to minimise L2 loss = (y -ŷ) 2 . 9. The student learns using online stochastic gradient descent. 10. η/2 is the learning rate. 11. ϵ denotes the generalisation error. 12. a • b denotes the dot product between vectors a and b. 13. We now define the following Order Parameters (where only the first 4 change with training): ). This has the consequence that T ± exactly equals the cosine similarity between the two teachers. 15. We also define• Q = w w w • w w w/d; • R + = w w w • w w w + /d; • R -= w w w • w w w -/d; • M = w w w • v v v/d; • T ± = w w w + • w w w -/d; • M * + = w w w + • v v v/d; • M * -= w w w -• v v v/d; • v = v v v • v v v/d.∆ mix = ρ∆ + + (1 -ρ)∆ -and ∆ 2mix = ρ∆ 2 + + (1 -ρ)∆ 2 -. 16.For notational convenience we define:α + = ⟨y⟩ ⊕ = 1 -2Φ -M * + ∆ + , (A.19) α -= ⟨y⟩ ⊖ = 1 -2Φ -(-M * -) ∆ - . (A.20)Note, α + also has an intuitive meaning. It represents the difference between the probability that an example drawn from the positive cluster has positive true label and the probability that an example drawn from the positive cluster has negative true label. It is hence 0 when the positive cluster has equal positive and negative examples, positive when the cluster has more positive examples than negative, negative when the cluster has more negative examples than positive. Similarly, α -represents the difference in these probabilities for the negative cluster.17. Finally, we also defineβ + = 2∆ + π exp -M * 2 + 2∆ + , (A.21) β -= 2∆ - π exp -M * 2 - 2∆ - . (A.22)18. Lastly, we use t to denote continuous time given by (epoch number/d).B Proof of the main theorems B.1 Proof of Lemma 3.1.Denote with ⟨•⟩ j the expectation over samples from cluster j. The generalisation error reads ϵ = m j=1 ρ j ϵ j withϵ j := (y -ŷ) 2 j = y - w w w • x x x √ d 2 j = y 2 j + w w w • x x x √ d 2 j -2 y w w w • x x x √ d j = 1 + (Q∆ j + M 2 j ) -2(α j M j + R j β j ), where the second term comes from: isolating the mean and the definition of M j , and the isotropy of x. The third term comes from the useful identity Integral 1 Eq. C.31, derived in Appendix C.1, and the constants are given byα j = 1 -2Φ -M * j ∆ j , β j = 2∆ j π exp -(M * j ) 2 2∆ j . (B.23)whereM * j := w w w ⊤ j v v v j /d and Φ(x) = 1 √ 2πx -∞ e -u 2 /2 du is the cumulative distribution function of the standard normal.The formula for the generalisation error specializes to the case of two clusters with opposite means asϵ = 1 + M 2 -(2ρα + -2(1 -ρ)α -) M -2ρβ + R + -2(1 -ρ)β -R -+ ∆ mix Q, (B.24)Notably, α ± has an intuitive meaning wherein it represents the difference between the fraction of positive and negatives in a cluster, i.e., α + = ⟨y⟩ c=+ and α -= ⟨y⟩ c=-.B.2 Proof of Lemma 3.2.Explicit computations are carried out in Appendix C.2 below for the case of two clusters.Using the notation of Section 3.2 and assuming Lemma 3.2, we examine the update equations ( 7) written as a stochastic iterative processS k+1 = S k + E 1 d f (S k ) + 1 √ d ξ k d , ξ k d := √ d(∆S k -E[∆S k ]) (B.25)where the expectation is over the new sample x x x k and conditional on the past samples. Let us sketch the scaling argument. Algebraic manipulations similar to those in Section C.2 show thatΣ d = ∇S k⊤ E[Φ k Φ k⊤ ]∇S k (1 + O(d -1 )), Φ k := η(δ k x x x k -E[δ k x x x k ]) (B.28)where ∇ denotes the gradient with respect to the student vector w w w. Recall that S k has 2m components that are linear in w w w (corresponding to the order parameters R j and M j in Eq. 5) and one that is quadratic (corresponding to Q). By making the gradients ∇S k explicit using Eq. 5), we see that at leading order, the matrix entries Σ ij d , 1 ≤ i, j ≤ 2m + 1 take the formΣ ij d = 1 d E[Φ k a a ai Φ k⊤ a a aj ], Φ k a a ai = η(δ k a a a ⊤ i x x x k √ d -E[δ k a a a ⊤ i x x x k √ d ]) (B.29)where the vector a a a i is either one of the teacher vectors w w w j , one of the shift vector v v v j , or the student vector w w w, depending on the entry i = 1, • • • , 2m + 1. As can be shown explicitly as in Appendix C.1 below, Φ k a a ai depend on x x x k only through auxiliary variables w w w ). Finally, the weak convergence of S d (t) t to St implies convergence in probability for the supremum norm on the interval [0, T ] for any T > 0. Specifically, for each⊤ j x x x/ √ d, v v v ⊤ j x x x/ √ d, w w w k⊤ x x x k / √ d,EQUATIONwhere P -→ denotes convergence in probability. This result directly leads to Eq. 9, thereby proving the theorem.In this section we are going to explicitly derive the ODE describing the dynamics of the order parameters. Starting from the discrete updates of the order parameters, Eqs. 7, we are going to consider the thermodynamic limit, d → ∞. As proven in Thm. 3.3, the updates concentrate to their typical value and the discrete evolution converges to differential equations. Therefore, the rest of the section is devoted to performing averages over the Gaussians in order to evaluate the typical values. Before proceeding with the evaluation of Eqs. 7, it is useful to introduce two identities.Integral 1:⟨a • x sign(b • x + c)⟩ = (a • µ)(1 -2Φ -(b • µ + c) √ ∆b • b ) + a • b 2∆ b • bπ exp -(b • µ + c) 2 2∆b • b (C.31) where x is multivariate normal distribution with mean µ and covariance ∆I, and the angular bracket notation indicates average with respect to x.1 = a • x and z 2 = b • x + c, that follow a multivariate normal distribution z 1 z 2 ∼ N a • µ b • µ + c , ∆ a • a a • b a • b b • b .Using the law of iterated expectation, our average can be written as:⟨a • x sign(b • x + c)⟩ = E z2 [sign(z 2 )E z1|z2 [z 1 ]] = E z2 [sign(z 2 )(a • µ + a • b b • b (z 2 -(b • µ + c))] = (a • µ - a • b b • b (b • µ + c))E z2 [sign(z 2 )] + a • b b • b E z2 [z 2 sign(z 2 )]The first expectation follows from the definition of the cumulative distribution function ΦE z2 [sign(z 2 )] = (1 -2Φ -(b • µ + c) √ ∆b • b ).The second term is simply the mean of a folded normal distributionE z2 [z 2 sign(z 2 )] = ( √ ∆b • b) 2 π exp -(b • µ + c) 2 2∆b • b + (b • µ + c)(1 -2Φ -(b • µ + c) √ ∆b • b ).Combining these three expressions we obtain the identity.Integral 2: ⟨a • x b • x⟩ = (a • µ)(b • µ) + ∆(a • b) (C.32)where x is defined as for the previous identity.Derivation. We proceed as in the previous case. Define the auxiliary random variables z 1 = a • x and z 2 = b • x. They follow a multivariate normal distributionz 1 z 2 ∼ N a • µ b • µ , ∆ a • a a • b a • b b • b .Using the law of iterated expectation, our average may be written as:⟨a • x b • x⟩ = E z2 [z 2 E z1|z2 [z 1 ]] = E z2 [z 2 (a • µ + a • b b • b (z 2 -(b • µ))] = (a • µ - a • b b • b (b • µ))E z2 [z 2 ] + a • b b • b E z2 [z 2 2 ] = (a • µ - a • b b • b (b • µ))(b • µ) + a • b b • b (∆b • b + (b • µ) 2 ) = (a • µ)(b • µ) + ∆(a • b).We have now the building blocks to evaluate the expected values of Eqs. 7. We refresh the notation that δ µ = y µ -ŷµ , y µ = sign x x x µ • w w w µ / √ d , and ŷµ = x x x µ • w w w/ √ d. Final step is to take the continuous limit. This is obtained by noticing that the RHS of the equations is factorised by 1/d. Therefore by taking as time unit 1/d and defining time as t = µ/d the discrete updates converge to continuous increments as d → ∞.Student-shift overlap M . ⟨∆M ⟩ = η d ρvα + + ρM * + β + -(1 -ρ)vα -+ (1 -ρ)M * -β --(M (v + ∆ mix )) (C.33)Derivation. Starting from the definition in Eq. 7 forM ⟨∆M ⟩ = η d y x x x • v v v √ d - w w w • x x x √ d x x x • v v v √ d .The first term can be evaluated using integral 1 and the second term using integral 2 yielding the result.Student-teacher + overlap R + .⟨∆R + ⟩ = η d ρ(M * + α + + β + ) + (1 -ρ)(-M * + α -+ T ± β -) -ρ(M M * + + R + ∆ + ) -(1 -ρ)(M M * + + R + ∆ -) (C.34)Derivation.⟨∆R + ⟩ = η d y - w w w • x x x √ d x x x • w w w + √ d = η d ρ y x x x • w w w + √ d ⊕ + (1 -ρ) y x x x • w w w + √ d ⊖ -ρ w w w • x x x √ d x x x • w w w + √ d ⊕ -(1 -ρ) w w w • x x x √ d x x x • w w w + √ d ⊖ .These 4 terms can be computed using integrals 1 and 2 yielding the result.Student-teacheroverlap R -.⟨∆R -⟩ = η d ρ(M * -α + + T ± β + ) + (1 -ρ)(-M * -α -+ β -) -ρ(M M * -+ R -∆ + ) -(1 -ρ)(M M * -+ R -∆ -) (C.35)Derivation. Same as for R + .Self-overlap Q.⟨∆Q⟩ = 2η d ρ(α + M + β + R + ) + (1 -ρ)(-α -M + β -R + ) -M 2 -Q∆ mix + η 2 d ∆ mix + Q∆ 2mix + M 2 ∆ mix -2 (ρ∆ + (α + M + β + R + ) + (1 -ρ)∆ -(-α -M + β -R + )) . (C.36)Derivation. This update requires additional steps with respect to the previous ones.⟨∆Q⟩ = 2η d δ w w w ⊤ j x x x √ d + η 2 d (δ µ ) 2 ∥x x x µ ∥ 2 d .The first term is2η d δ w w w ⊤ j x x x √ d = 2η d y w w w • x x x √ d - w w w • x x x √ d 2 = 2η d M (ρα + -(1 -ρ)α -) + ρβ + R + + (1 -ρ)β -R --M 2 -Q∆ mix .The second termη 2 d (δ µ ) 2 ∥x x x µ ∥ 2 d = η 2 d y - w w w • x x x √ d 2 x x x • x x x d = η 2 d y 2 x x x • x x x d + w w w • x x x √ d 2 x x x • x x x d -2y w w w • x x x √ d x x x • x x x drequires additional steps. We consider the three terms in the expression above, starting from the first oney 2 x x x • x x x d = x x x • x x x d = 1 d ( d i=1 x 2 i ) = 1 d ( d i=1 ρ x 2 i ⊕ + (1 -ρ) x 2 i ⊖ ) = 1 d ( d i=1 ρ(∆ + + v 2 i /d) + (1 -ρ)(∆ -+ v 2 i /d)) = ∆ mix + v/d = ∆ mix + O(d -1 ),Where we used the simplification y 2 = 1 independently of the cluster's teacher. However, the remaining terms require us to split the expectation considering the probability of sampling from each cluster. The second termx x x • x x x d w w w • x x x √ d 2 = ρ x x x • x x x d w w w • x x x √ d 2 ⊕ + (1 -ρ) x x x • x x x d w w w • x x x √ d 2 ⊖ .We begin by analysing the average over the positive Gaussian and split x x x asx x x = v v v/ √ d + xx x such that x x x has zero mean. Then,x x x • x x x d w w w • x x x √ d 2 ⊕ = v v v • v v v d 2 + 2v v v • x x x d √ d + x x x • x x x d w w w • v v v v v v v v v d 2 + 2 w w w • v v v d w w w • x x x √ d + w w w • x x x √ d 2 ⊕Multiplying the terms in the brackets will give rise to 9 terms. We can see that the 3+3=6 termscorresponding to v v v • v v v/d 2 and 2v v v • x x x/d √ dwill tend to 0 in the limit of infinite d due to their scaling. We now analyse the other 3 terms: Term 1:x x x • x x x d w w w • v v v d 2 ⊕ = w w w • v v v d 2 x x x • x x x d ⊕ + O(d -1 ) = M 2 ∆ + + O(d -1 ).Term 2:2 x x x • x x x d w w w • v v v d w w w • x x x √ d ⊕ = 2R x x x • x x x d w w w • x x x √ d ⊕ = 2R x x x • x x x d ⊕ w w w • x x x √ d ⊕ + O(d -1 ) = 0 + O(d -1).Term 3:x x x • x x x d w w w • x x x √ d 2 ⊕ = x x x • x x x d ⊕ w w w • x x x √ d 2 ⊕ + O(d -1 ) = ∆ + (∆ + Q) + O(d -1 ) = Q∆ 2 + + O(d -1 ).Thus finally,x x x • x x x d w w w • x x x √ d 2 = ρ(M 2 ∆ + + Q∆ 2 + ) + (1 -ρ)(M 2 ∆ -+ Q∆ 2 -) = M 2 ∆ mix + Q∆ 2mix .For the the third termy w w w • x x x √ d x x x • x x x d = ρ y w w w • x x x √ d x x x • x x x d ⊕ + (1 -ρ) y w w w • x x x √ d x x x • x x x d ⊖ .As before, we analyse the average over the positive Gaussian first and split x x x into its mean and a zero mean component:y x x x • x x x d w w w • x x x √ d ⊕ = y v v v v v v v v v • v v v v v v v v v d 2 + 2v v v • x x x d √ d + x x x • x x x d w w w • v v v v v v v v v d + w w w • x x x √ d ⊕ .This gives rise to 6 terms. We can see that the 2+2=4 terms corresponding to v v v • v v v/d 2 and 2v v v • x x x/d √ d will tend to 0 in the limit of infinite d due to their scaling. We now analyse the other 2 terms: Term 1:y x x x • x x x d w w w • v v v d ⊕ = M y x x x • x x x d ⊕ = M sign( x x x • w w w + √ d + w w w + • v v v d ) x x x • x x x d ⊕ = M sign( x x x • w w w + √ d + w w w + • v v v d ) ⊕ x x x • x x x d ⊕ + O(d -1 ) = M ⟨y⟩ ⊕ ∆ + + O(d -1 ) = M α + ∆ + + O(d -1 ).Term 2:y x x x • x x x d w w w • x x x √ d ⊕ = y w w w • x x x √ d ⊕ x x x • x x x d ⊕ + O(d -1 ) = ∆ + y w w w • x x x √ d ⊕ + O(d -1 ) = ∆ + R + β + + O(d -1 ).Where the last equality follows using integral 1. Thus:y x x x • x x x d w w w • x x x √ d ⊕ = ∆ + (α + M + β + R + ) + O(d -1).We repeat the same analysis for the negative gaussian and get:y x x x • x x x d w w w • x x x √ d = ρ∆ + (α + M + β + R + ) + (1 -ρ)∆ -(-α -M + β -R + ) + O(d -1).Collecting everything together and taking the infinite dimensional limit:⟨∆w w w • ∆w w w/d⟩ = η 2 d ∆ mix + Q∆ 2mix + M 2 ∆ mix -2 (ρ∆ + (α + M + β + R + ) + (1 -ρ)∆ -(-α -M + β -R + ))Thus,⟨∆Q⟩ = 2η d ρ(α + M + β + R + ) + (1 -ρ)(-α -M + β -R + ) -M 2 -Q∆ mix + η 2 d ∆ mix + Q∆ 2mix + M 2 ∆ mix -2 (ρ∆ + (α + M + β + R + ) + (1 -ρ)∆ -(-α -M + β -R + )) .Continuous limit. Final step of the derivation is taking the termodynamics limit that leads to the ODEs implicitely defined in Thm. 3.3:f M (M, R + , R -, Q) = η ρvα + + ρM * + β + -(1 -ρ)vα -+ (1 -ρ)M * -β --(M (v + ∆ mix )) , (C.37) f R+ (M, R + , R -, Q) = η ρ(M * + α + + β + ) + (1 -ρ)(-M * + α -+ T ± β -) -ρ(M M * + + R + ∆ + ) -(1 -ρ)(M M * + + R + ∆ -) , (C.38) f R-(M, R + , R -, Q) = η ρ(M * -α + + T ± β + ) + (1 -ρ)(-M * -α -+ β -) -ρ(M M * -+ R -∆ + ) -(1 -ρ)(M M * -+ R -∆ -) , (C.39) f Q (M, R + , R -, Q) = 2η ρ(α + M + β + R + ) + (1 -ρ)(-α -M + β -R + ) -M 2 -Q∆ mix + η 2 ∆ mix + Q∆ 2mix + M 2 ∆ mix -2 ρ∆ + (α + M + β + R + ) + (1 -ρ)∆ -(-α -M + β -R + ) . (C.40)In this section we first present the general solutions of the ODEs sketched in Theorem 3.4, then we specialise to the two scenarios discussed in the main text.From the previous section, we have a system of coupled ODEs for the order parameters of the form:dM dt = c 1 + c 2 M, dR - dt = c 3-+ c 4-M + c 5-R -, dR + dt = c 3+ + c 4+ M + c 5+ R + , dQ dt = c 6 + c 7 M + c 8 M 2 + c 9+ R + + c 9-R -+ c 10 Q.This represent a linear system of ODEs which can be solved using standard methods like Laplace transform, leading to Eqs. 10-12. We now report the equations including the exact expression of their coefficients.M :M (t) = M 0 e -tη(v+∆ mix ) + M ∞ (1 -e -tη(v+∆ mix ) ). (D.41) Where, M ∞ = (ρM * + β + + (1 -ρ)M * -β -) + v(ρα + -(1 -ρ)α -) v + ∆ mix . (D.42) R + : R + (t) = R 0 + e -tη∆ mix + R ∞ + (1 -e -tη∆ mix ) + k 1+ (e -tη∆ mix -e -tη(v+∆ mix ) ). (D.43)Where,R ∞ + = (ρβ + + T ± (1 -ρ)β -) + M * + (ρα + -(1 -ρ)α --M ∞ ) ∆ mix , (D.44) k 1+ = M * + (M ∞ -M 0 ) v . (D.45) R -: R -(t) = R 0 -e -tη∆ mix + R ∞ -(1 -e -tη∆ mix ) + k 1-(e -tη∆ mix -e -tη(v+∆ mix ) ). (D.46)Where,R ∞ -= (T ± ρβ + + (1 -ρ)β -) + M * -(ρα + -(1 -ρ)α --M ∞ ) ∆ mix , (D.47) k 1-= M * -(M ∞ -M 0 ) v . (D.48) Q: Q(t) = Q 0 e -tη(2∆ mix -η∆ 2mix ) + Q ∞ (1 -e -tη(2∆ mix -η∆ 2mix ) ) + k 2 (e -tη(2∆ mix -η∆ 2mix ) -e -tη∆ mix ) + k 3 (e -tη(2∆ mix -η∆ 2mix ) -e -tη(v+∆ mix ) ) + k 4 (e -tη(2∆ mix -η∆ 2mix ) -e -tη(2v+2∆ mix ) ). (D.49)Where,Q ∞ = η∆ mix + 2ρβ + R ∞ + (1 -η∆ + ) + 2(1 -ρ)β -R ∞ -(1 -η∆ -) 2∆ mix -η∆ 2mix ,The general fairness case coincides with the general case discussed above (D.1), therefore we limit our discussion to the simplified case with centered clusters.Under the zero shift v = 0, the equations take the simplified form wherein M, v, M * ± are 0, the transient term in R ± vanishes and Q only has one transient term. Specifically:R + (t) = R 0 + e -tη∆ mix + R ∞ + (1 -e -tη∆ mix ), R -(t) = R 0 -e -tη∆ mix + R ∞ -(1 -e -tη∆ mix ), Q(t) = Q 0 e -tη(2∆ mix -η∆ 2mix ) + Q ∞ (1 -e -tη(2∆ mix -η∆ 2mix ) ) + Q trans (e -tη(2∆ mix -η∆ 2mix ) -e -tη∆ mix ).WhereR ∞ + = 2 π ρ ∆ + + T ± (1 -ρ) ∆ - ∆ mix , R ∞ -= 2 π T ± ρ ∆ + + (1 -ρ) ∆ - ∆ mix , Q ∞ = η∆ mix + 2 2 π ρ ∆ + R ∞ + (1 -η∆ + ) + 2 2 π (1 -ρ) ∆ -R ∞ -(1 -η∆ -) 2∆ mix -η∆ 2mix , Q trans = 2 π 2ρ ∆ + (1 -η∆ + )(R ∞ + -R 0 + ) + 2(1 -ρ) ∆ -(1 -η∆ -)(R ∞ --R 0 -) ∆ mix -η∆ 2mix.This section provides insights into the learning dynamics -particularly those relevant to bias evolution -that arise out of the expressions for order parameter evolution. We shall provide intuitive explanations behind the various mathematical terms that appear.Consider first a single cluster centered at the origin-i.e. ρ = 1, v = 0 with variance ∆. In this setting, the minimum generalisation error is achieved when the student perfectly aligns with the teacher and optimises its norm such that Q opt = 2 π∆ , achieving the generalisation error ϵ min = 1 -2 π . Importantly, this is not 0 since the student and the teacher are mismatched -i.e. the student is linear whereas the teacher has a sign(•) activation function. From the equations, we observe that the asymptotic generalisation error when training using online stochastic gradient descent in this setting isϵ ∞ = 1 -2/π 1 -η∆/2 = 1 - 2 π 1 + η∆ 2 + O(η 2 ∆ 2 ) . (E.54)Thus, as the learning rate increases, the generalisation error increases until it reaches the critical learning rate beyond which training is unstable and the loss grows unboundedly. In the single cluster case, Eq. E.54 this is 2/∆ which matches the classical result from convex optimisation [23] . We can similarly find the critical learning rate for two clusters to be 2∆ mix /∆ 2mix by ensuring exponential terms decay to zero in equation 12.E.2 Analysis of teacher alignment (τ R ) and student magnitude (τ Q ) timescalesWe now consider the fairness setting with zero shift as illustrated in Fig. 1c . As discussed in section 4.2, the relevant timescales in this setting areτ R = 1 η∆ mix , τ Q = 1 η(2∆ mix -η∆ 2mix ) ,since M (t) is always zero. Fig. 7 shows the crossing phenomena of the loss curves along with the order parameter evolution and other insightful terms. The alignment of the student is governed by the Figure 7 : The Crossing Phenomenon The left shows the 'crossing' of the loss curves on the negative sub-population in red (higher variance and lower representation) and positive sub-population in blue (lower variance but greater representation) along with the overall loss in purple obtained as a weighted average of the two. It also marks τ R as the dashed vertical line and τ Q as the dotted vertical line. The right side shows the evolution of the order parameters and a transient term. The horizontal blue and red dash-dotted line mark the optimal value of Q for the positive-subpopulation and negative sub-populations respectively. The parameters are v = 0, ρ = 0.8, ∆ + = 0.1, ∆ -= 1, T ± = 0.9, η = 0.1. timescale τ R and the change in its magnitude is governed by the timescale τ Q . Initially, the classifier has a small magnitude and its alignment roughly matches the two teachers which are themselves quite similar (T ± = 0.9). Indeed, we see that the R + and R -have very similar trajectories. However, smaller magnitudes advantage higher variances as discussed in Appendix E.1 (Q opt is inversely proportional to the cluster variance). We mark the optimal values of Q using horizontal lines in Fig. 7 on the left side with blue for the positive sub-population (lower variance) and red for the negative sub-population (higher variance). As the magnitude of the student grows, we observe a sharp drop in the generalisation error on the higher variance sub-population till Q crosses the horizontal red line. Beyond this point, the generalisation error on the higher variance sub-population rises since the magnitude of the student has exceeded the optimal value (horizontal red line) and the generalisation error on the lower variance sub-population continues to fall as the magnitude of the student approaches the horizontal blue line. Finally, an inspection of the timescales reveals that τ Q (vertical dotted line) is less than t R (vertical dashed line) and hence we may expect the student magnitude to saturate before its alignment. However, Q trans , the transient term associated with Q (third line of equation 12), is always negative and hence suppresses the growth of Q initially.In summary, we observe a two phase behaviour. First the student shifts its alignment and increases magnitude leading to a sharper drop in the higher variance generalisation error. Second, we observe that as the student continues increasing magnitude while keeping its alignment fixed, it advantages the lower variance cluster.This section discusses the asymptotic generalisation errors of our classifier when v = 0 as a function of representation and variances. Firstly, as discussed in section 4.2,R ∞ + > R ∞ -⇐⇒ ρ ∆ + > (1 -ρ) ∆ -.Intuitively, one might expect that the asymptotically lower generalisation error is achieved on the population whose teacher has better asymptotic alignment with the student. Indeed, when the learning rate tends to 0, we observe exactly this as illustrated by the two dark phases in Fig. 8 on the left side. However, when the learning rate is greater than zero, we observe more complex behaviour. Fig. 8 (right) shows the emergence two new phases (light red and light blue) wherein the classifier exhibits higher generalisation error on a sub-population despite having better alignment with its corresponding teacher. This behaviour can be traced back to equation E.54 wherein the increase in asymptotic generalisation error due to non-zero learning rates is amplified by the cluster variance. When the learning rate is small (η → 0 + on left side), the cluster which has better alignment with the teacher must also have lower generalisation error. However, for non-zero learning rates (η = 0.1 on right side), behaviour is more complicated leading to the light colored phases where despite better asymptotic alignment with the teacher, the generalisation error is higher. Parameters: η → 0 + (left) vs η = 0.1 (right).Thus, our analysis shows how a large learning rate can also become a source of bias in our classifier by advantaging the sub-population with smaller variance. The goal of this experiment is to show the emergence of different timescales in realist scenarios of relevance for the fairness literature.The CelebA dataset [27] contains over 200k celebrity images annotated with 40 attribute labels, covering a wide range of facial attributes such as gender, age, and expressions. For this experiment, we consider different pairs as the target and group attributes. The task is to predict the target attribute while the group attribute defines the + andsubpopulations.For the model, we select a pretrained ResNet-18 model on ImageNet and add an additional fully connected layer, with only the latter being optimised during training. We use cross-entropy as the loss objective and train via online SGD.We randomly selected target-label pairs, making sure to avoid attributes that are pathologically underrepresented in the dataset and would hinder the significance of the result. In the plots shown in Fig. 9 we show some of the pairs that show a crossing phenomenon. Each panel in Fig. 9 show the accuracy and accuracy gap over the course of training. Notice how the classifier favours sub-populationin the initial phase of training before changing preference.This result shows that bias can change over the course of training even in standard setting. This does not imply that it will always occur and indeed several of the pairs in the dataset do not show a crossing phenomenon. However, understanding when and why this phenomenon occurs can affect the algorithmic choices that we make in our ML pipeline.Figure 10 : Synthetic Data Simulation with alternate Training Protocols We observe the 'doublecrossing' phenomena in not only the loss curves, but also the error curves for the positive subpopulation (blue) and the negative sub-population (red) (left). The shaded areas quantify the standard deviation obtained across 10 seeds. We observe similar behavior when using Adam (middle) and weight decay (right). The data distribution parameters are d = 100, v = 4, ρ = 0.7, ∆ + = 0.1, ∆ -= 1, T ± = 0.9, η = 0.01, α + = 0.471, α -= -0.188In this section we test the validity of the prediction of our model across various networks and training protocols. We consider a data distribution with parameters as detailed in the caption of Fig. 10 . We train a 2-layer MLP with ReLU activation in the hidden layer(s) and sigmoidal activation at the output with 128 neurons in the hidden layer with online SGD using BCE loss. We refer to this as the 'standard configuration' for further comparisons. We sample training and test data from the data distribution and use the test data to obtain estimates of the loss as well as error rates (percentage of test examples misclassified). We visualize these in Fig. 10 on the left.We observe the three phase behaviour predicted by our model. The positive sub-population is initially advantaged more since it exhibits stronger spurious correlation. Then, the negative sub-population is advantaged since it has a higher variance. Finally, as per Eq. 16, the positivesub-population is advantaged once more since it has sufficiently high representation. We not only observe the 'double-crossing' phenomena in the losses, but also in the test errors demonstrating the robustness of our model beyond the linearity and MSE loss assumptions.We also observe similar behavior using Adam optimization in Fig. 10 (center). We note that we had to use a smaller learning rate of 0.001 to keep training stable since Adam leads to faster optimization manifesting in a higher effective learning rate than SGD. When we train using a weight decay penalty of 0.1 (Fig. 10 right), we observe that asymptotically the higher variance cluster is now preferred. This behaviour can be explained using the theoretical model. As discussed in Appendix E, smaller student magnitude advantages the higher variance group and weight decay encourages smaller student weights leading to an asymptotic preference for the negative sub-population. We now train deeper networks with 2 to 5 layers and visualise results in Fig. 11 . The 'double-crossing' phenomena persists across deeper networks. We also train wider networks with 2, 16, 128 and 1024 units in the hidden layer and visualise results in Fig. 12 . The 'double-crossing' phenomena persists Finally, we vary the learning rate as well from our standard configuration and note that when the learning rate is too high, the training is unstable for the higher variance cluster (Fig. 13 left ). For stable training, however, we observe the 'double-crossing' phenomena as usual, just at longer timescales for slower learning rates as predicted by our theory.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: All the simulations were performed on personal laptops. Computational resources needed to reproduce our results are quite available.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper analyzes the training dynamics (with ODEs) of a linear model trained with SGD under an assumed Gaussian mixture data distribution. Notably the paper makes precise how the optimization dynamics affect data subgroups in this specific setup. Among other settings, the presented theoretical results are particularly relevant for learning under fairness considerations, and learning in the presence of spurious features. While other existing works tend to rely on asymptotic analysis, the present work’s goal is to understand  the complex time-dependent nature of bias during training. 

**Review**

All reviewers' comments were positive on the strength of the theoretical analysis in the specific setting considered. Notably, the closed-form time-dependent characterization of various quantities (JTC4), and the prediction of the double-cross phenomenon (CSav) are novel. Due to the assumed Gaussian mixture model, the issues of practical value and significance came up during the discussion. The authors sufficiently addressed these points by providing more experiments on more datasets and network architectures. Another issue raised was that the paper only considers a linear model. On this point, the AC thinks that the value of the contribution is in its full characterization (closed-form solution) of training dynamics under a distribution-model setup, even though that setup may be quite specific. The AC thinks that characterizing training dynamics in the more general case is not easy, and that this contribution can provide a stepping stone for a more complicated data distribution model that is yet to come in future works.

Recommendation: accept.

Reviews: review: paper_summary: paper_summary: In this paper, the authors explore how bias evolves while learning features across different sub-populations. They particularly emphasize the transient phase of learning, which is less understood compared to the early and late phases of the learning dynamics.  To this end, data from various sub-populations are modelled with a Gaussian-mixture model, and this data is labeled by a teacher network.  A linear student it trained with online SGD and its dynamics are studied at a high-dimensional limit. Using closed-form analytic expressions at this limit, the bias evolution is studied in various settings, revealing interesting dynamics that are experimentally validated on real datasets.
strength_weakness: strength_weakness: a) The paper studies only a linear student and it does not accurately capture the current over-parameterised deep learning models. This curtails the significance of the work. 

b) One other weakness is training with online SGD on the whole population loss in comparison of the empirical loss, hence these dynamics might fall short of capturing the real-world training setup. It is also not clear if these closed form analytic expressions are tractable beyond $m=2$.
questions: questions: a) Did the authors try any empirics with over-parameterized students over the Gaussian mixture, does it have any qualitatively different behaviour?
limitations: limitations: The limitations are adequately address in conclusion.
paper_summary: paper_summary: The paper investigates the emergence of bias in the transient phase of learning by gradient descent (after initialization and before convergence). The authors theoretically study a mixture of Gaussians dataset in a teacher-student framework. The paper then analytically characterizes several properties of a linear model applied to this dataset, showing the emergence of different forms of bias at different timescales. The authors then demonstrate these theoretical predictions empirically on rotated MNIST and CIFAR10 tasks.
strength_weakness: strength_weakness: In my view, the main weakness of the paper is that its practical value and significance remains difficult to understand. The authors show that their theoretical predictions line up with empirically observed double crossing phenomena, which is great. However, the significance of this finding is not stated. I would encourage the authors to more explicitly draw a connection between their findings and practical problems. Bias and fairness are areas with major significance in the field, so I am reasonably confident that the papers' contributions will have some practical value, but this needs to be explicitly stated and discussed.

Minor comments:
- Typo on line 314
questions: questions: What are the practical insights that can be drawn from this work? Can the authors elaborate more on what bias-mitigating mechanisms might arise from the paper's observations?
limitations: limitations: Limitations are adequately addressed.
paper_summary: paper_summary: The paper analyzes how different subgroups of the data affect optimization dynamics. The analysis is done in a large-dimension limit by analyzing squared error of a linear student classifier modeling a piece-wise linear teacher one. This analysis provides insights into dynamic loss behavior of loss on different subgroups and which aspects of the data lead cause this.
strength_weakness: strength_weakness: - the student is underpowered in this setup. The teacher is piece-wise linear but the student is purely linear. This fundamentally imposes a trade-off between modeling one subset of the data better than another. This breaks out of the case where models we use these days can perfectly fit any training dataset even with medium-sized datasets like CelebA.

Some missing related work: https://proceedings.mlr.press/v162/pezeshki22a.html, https://arxiv.org/abs/2308.12553

- This is important: What were the hyperparameters used in the various experiments? What happens as you vary each of them (like learning rate and weight decay)? It seems to me that the synthetic model cannot account for any of these parameters, so it feels like there is a gap in what the theory actually explains (especially with the underpowered student).

- (Minor) Adam is the more common optimizer and it is unclear whether the theory in the paper extends to this and if the insights about features translate. Can the authors comment on this?
questions: questions: Is there a way you can modify the data distribution in the real experiments according to what your theory suggests lead to certain behavior, and show that the behavior is now different? This would seal the theory and main reason behind the seen curves.

otherwise, See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The manuscript tries to understand the training dynamics in SGD training, through the lens of synthetic data with sub-populations from the Gaussian-mixture model. The paper leverages high-dimensional analysis to prove that the solution converges to a set of ODEs and thus characterizes the evolution of bias throughout training, namely a three-phase learning process. The study provides a unifying view of learning in fairness and spurious correlation problems, highlighting the presence of ephemeral biases characterized by multiple timescales during training. These theoretical insights are further validated on practical datasets, like CIFAR, MNIST, and CelebA.

In general, the take-away messages of the manuscript look interesting.
strength_weakness: strength_weakness: The reviewer cannot spot significant weaknesses in the manuscript.
questions: questions: * How does the value of $m$ affect Theorem 3.4 and the insights discussed in Section 4?
limitations: limitations: NA
paper_summary: paper_summary: This paper presents a theoretical analysis of bias dynamics in machine learning models during the training process.

1. The authors introduce a high-dimensional teacher-student framework called the Teacher-Mixture (TM) model to study how bias emerges and evolves during stochastic gradient descent (SGD) training of linear classifiers.

2. They provide an analytical solution for the dynamics of key order parameters that characterize the classifier's performance and bias in the high-dimensional limit.

3. The analysis reveals a rich three-phase learning process where the classifier exhibits biases toward different data features at different stages of training:
   - Initial phase: Influenced by sub-populations with strong class imbalance
   - Intermediate phase: Dominated by the saliency (norm) of samples in sub-populations
   - Final phase: Determined by sub-population imbalance (relative representation)

4. The authors show that bias can evolve non-monotonically during training, potentially leading to "crossing" phenomena where the preferred sub-population changes over time.

5. They validate their theoretical findings through numerical experiments on both synthetic data and real datasets (CIFAR10, MNIST, CelebA), demonstrating that the observed phenomena extend beyond the simplifying assumptions of their model.

6. The paper discusses implications for fairness and robustness in machine learning, highlighting how different properties of sub-populations can generate and amplify bias at different timescales.

7. The authors argue that understanding these transient dynamics is crucial for developing effective bias mitigation strategies, especially under limited computational resources.
strength_weakness: strength_weakness: I can not really judge the novelty and soundness of the proposed work but I believe the paper is of high quality. Of course the setting considered is far from „large scale“ but I dont see this as too important at this stage.
questions: questions: Can you link training on MNIST, etc closer to the theory by training again on linear models. Can you make the jump between theoretical and experimental section less severe by e.g. studying linear models trained on different (more difficult and not theoretically motivated) data generation processes? Can you provide examples for linear models where we do not observe crossing phenomena to build up more intuition?
limitations: limitations: Yes

Comments: title: Authors-Reviewers Discussion Summary
reply to the author's rebuttal
Thank you for the rebuttal
comment: As the author-reviewer discussion period comes to a close, we would like to extend our sincere thanks to the reviewers and the area chair for their time and thoughtful feedback.

## New Results

In response to the reviewers' suggestions, **we have conducted an extensive series of additional experiments across a variety of datasets and network architectures**, demonstrating the robustness of our theoretical findings beyond the simplified settings initially considered. As shown in the supplementary experiments included in our rebuttal, our empirical results align remarkably well with our theoretical predictions, providing valuable insights into practical applications. Notably, our experiments reveal robustness across different optimization techniques, including variations in learning rates, Adam optimizer, and weight decay.

## Contributions

A key concern raised by the reviewers was that our theoretical analysis focuses on linear classifiers, which differ significantly from modern deep-learning architectures. We would like to emphasize that most theoretical studies on spurious correlations and data imbalance still employ linear classifiers due to the complexity of these problems [1,2,3,4,5]. **Within the current theoretical landscape**, our paper makes a **significant technical contribution** by providing, for the first time, a closed-form solution of the dynamics and **predicting changes in bias** during training. These insights have practical implications, particularly for debiasing algorithms [6,7], as discussed in our response to Reviewer gHK2.

Moreover, our study reveals that even for linear classifiers, common assumptions can be misleading. Our simulations demonstrate robustness beyond what is captured by theoretical analysis. Empirically, the trend observed [4,8] suggests that increased model complexity tends to amplify bias.

## Conclusion

While we regret that we were unable to engage with all reviewers during the discussion period, we remain hopeful that the forthcoming discussions will lead to a favorable outcome.

Best regards,
The Authors

## References:

[1] Loffredo, E., Pastore, M., Cocco, S., & Monasson, R. (2024). Restoring balance: principled under/oversampling of data for optimal classification. In ICML 2024.

[2] Hermann, K. L., Mobahi, H., Fel, T., & Mozer, M. C. (2024). On the Foundations of Shortcut Learning. ICLR 2024.

[3] Nagarajan, V., Andreassen, A., & Neyshabur, B. (2021). Understanding the failure modes of out-of-distribution generalization. ICLR 2021.

[4] Sagawa, S., Raghunathan, A., Koh, P. W., & Liang, P. (2020). An investigation of why overparameterization exacerbates spurious correlations. In ICML (pp. 8346-8356). PMLR.

[5] Takahashi, T. (2024). A replica analysis of under-bagging. TMLR.

[6] Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., ... & Finn, C. (2021). Just train twice: Improving group robustness without training group information. ICML 2021 (pp. 6781-6792). PMLR.

[7] Yang, Y., Gan, E., Dziugaite, G. K., & Mirzasoleiman, B. (2024). Identifying spurious biases early in training through the lens of simplicity bias. In AISTATS (pp. 2953-2961). PMLR.

[8] Bell, S. J., & Sagun, L. (2023). Simplicity bias leads to amplified performance disparities. In Proceedings of FAccT 2023 (pp. 355-369).
We hope that the clarifications and additional experiments have addressed the reviewer’s  concerns. If there are any remaining questions or points that need further discussion, we would be grateful for any additional feedback.
We hope that the clarifications and additional experiments have addressed the reviewer’s  concerns. If there are any remaining questions or points that need further discussion, we would be grateful for any additional feedback.
Thank you for conducting additional experiments on over-parameterization. The experiments provide some clarity on the relevance of the phenomenon beyond linear classifiers. However, I have decided to keep my score since the theoretical analysis of the phenomenon is limited to linear setting.
Thank you for your thoughtful reply! I'm convinced that the additions made by the authors will clarify the practical significance of the work, and maintain my rating in favor of acceptance. 

The main weakness in my view is that (as echoed by some other reviewers), the theoretical setting is limited, which limits the impact of the work primarily to theoreticians at this stage (and practical experiments would be outside the scope of this paper). Nevertheless, the theoretical analysis is strong, which is why I am in favor of acceptance.
Thank you for your appreciation. Is anything we can add to improve rating and confidence scores?
The reviewer appreciates the authors for providing the rebuttal and will maintain the original score.