Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Authors: Anchit Jain
Rozhin Nobahari
Aristide Baratin
Stefano Sarao Mannelli
Venue: NeurIPS 2024 poster (2024)
Keywords: learning dynamics
online learning
stochastic gradient descent
analytical model
fairness
spurious correlation
Primary area: optimization
TL;DR: We propose a completely analytically tractable framework for studying the evolution of bias of a classifier during training

Abstract: Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations of the data. However, our current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup that models different data sub-populations with a Gaussian-mixture model. We provide an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setup, which we prove to be exact in high dimension.
Notably, our analysis identifies different properties of the sub-populations that drive bias at different timescales and hence shows a shifting preference of our classifier during training. By applying our general solution to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real data, i.e. using CIFAR10, MNIST, and CelebA datasets.

Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Anchit Jain" University of Cambridge aj 6650cantab..aa ac uk
Rozhin Nobahari
Aristide Baratin
MILA Quebec Al institute
Samsung SAIT AI Lab Montreal
Université de Montréal
Stefano Sarao Mannelli
Gatsby & SWC University College London 8. saraomaneelli@ll..eaa. ac.
Abstract
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup modeling different data sub- cotb-ooullttins with a Gaussian-mixtue provide an analytical description of the stochastic gradient descent dynamics of linear classifier in this setting, which we prove be exact in high dimension. Notably, our analysis reveals how different properties of sub-pon cob--ooulatiions influence bias at different timescales, showing a shifting preference of the classifier during training. Applying our findings to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real datasets, including CIFARIO, MNIST, and CelebA
model. We
Introduction
Over the past decade, the problem of assessing the fairness of classifiers has garnered significant attention, revealing that machine learning (ML) systems not only reproduce existing biases in the data but also tend to amplify them [21 40 [1]. Given the complexity the ML pipeline, isolating and characterising the key drivers of this amplification is challenging. Recent studies have begun to disentangle the contributions from architectural design choices, including overparameterisation 337,
model complexity. activation functions [ 12], learning protocols [ 13 post-processing practices
such as pruning [19. and intrinsic aspects of the data like its geometrical properties [88]
Theoretical results in this area (e.g., 37 38 are mostly based asymptotic analysis, leaving the transient learning regime poorly understood. Due to limitations on computational resources, a trained ML system may operate far from the asymptotic regime and hence existing results may not always apply. Insights from class imbalance literature 12) indicate that classifiers converge faster for classes with more data, but how this applies fairness, where datasets might be balanced by label but imbalanced by demographics, remains unclear.
Our analysis addresses this gap by providing precise characterisation of the transient dynamics of online stochastic gradient descent (SGD) in a high dimensional prototypical model of linear
Work done as intern at the Gatsby Computational Neuroscience Unit, University College London
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classification. We use the macchrr-mixture (TM) framework (38), where different data sub- poppuolltssinns are modeled with a mixture of Gaussians, each having its own linear rule (teacher) for determining the labels. Adjusting the parameters of the data distribution in our framework connects models of fairness and spurious correlations, providing unifying framework and a general set of results applicable to both domains. Remarkably, our study reveals a rich behaviour divided into three learning phases, where different features of data bias the classifier and causing significant deviations from asymptotic predictions. We reproduce our theoretical findings through numerical experiments in more complex settings, demonstrating validity beyond the simplicity of our model.
Our key contributions are:
High-dimensional analysis: We demonstrate that in the high- high-dimensional limit, relevant properties of the classifier, such as the generalisation error, can be expressed using a few sufficient statistics. We prove that their evolution converges to set of ordinary differential equations (ODEs) that can be solved explicitly in our setting.
Bias evolution characterisation: Using our solution, we characterise the evolution of bias throughout training, showing three-phase learning process where bias exhibits non- monotonic behaviour. Specifically:
1. Initial phase: The classifier is initially influenced by sub-populations with strong class imbalance.
2. Intermediate phase: The dynamics shifts towards the saliency, or norm, of the samples in sub-population.
3. Final phase: Sub-population imbalance, or relative representation, becomes the domi- nant factor.
Empirical validation:
We validate and extend our theoretical results through numerical experiments in both synthetic and real datasets, including CIFARIO, MNIST, and CelebA.
Altogether, our study reveals a complex time-dependence of learning with structured data that previous theoretical studies have failed to capture. This characterisation is crucial for developing effective bias mitigation strategies, especially under limited computational resources.
1.1 Further related works
Class imbalance and fairness. key element in our study is the presence of heterogeneous data distributions within the dataset. In the context of faimesss, these distributions model different groups a population. Sampling unbalance is particularly critical, as minority groups are often misclassified [9] [20]. However, theoretical studies on group imbalance have been limited to asymptotic analyses 38 which may not apply in practical settings. Related questions have been explored in the label imbalance literature [22), where it has long been known
I7 that underrepresented classes have slower convergence rate and may even experience increased errors early in training. Our work shows that pre-ssympootic analysis can reveal complex transient dynamics, which is practically relevant when learning slows down or training to convergence is not possible. Similar to our analysis, [12 has shown that supposedly neutral choices, like activation functions or pooling operations, can generate strong biases. In contrast to prior work, our focus on data properties identifies several timescales associated to different data features relevant to bias generation.
Simplicity bias. Several studies 31 16
32 have highlighted a bias of deep neural networks (DNNs) towards simple solutions, suggesting this bias is key to their generalisation performance. Simplicity bias also influences learning dynamics: 4 32 28 30 33 have showed that DNNs learn progressively more complex functions during training, with notion of complexity often defined implicitly by other DNNs or observations like the time to memorisation. Our results connect with simplicity bias by identifying interpretable properties of the data that make samples appear "simple" to a shallow network. Interestingly, our findings reveal that different phases of learning experience simplicity in different ways, leading to forgetting of previously learned features.
Spurious correlations. Simplicity bias can also lead to shortcomings 39 by excessively relying of spurious features in the data, possibly hurting generalisation, especially in dut-i--dist
contexts 14] Theoretical works 22 37 18 have identified statistical properties that cause a classifier to favour spurious features over potentially more complex but more predictive features.

---

Various methods have been proposed to address this problem using explicit partitioning of the data 36]; some upproaches implicitly infer subgroups with various degrees of correlation as spurious rely early stages of learning to detect bias and adjust sample importance accordingly. Our study provides a unifying view of learning in fairness and spurious correlation problems, highlighting the presence of ephemeral biases characterised by multiple timescales during training. This adds complexity to the understanding learning dynamics and points out potential confounding effects in existing mitigation methods.
features. Notably, 26 42
1.0
Generalization Error
Order Parameter
0.8
0.6
100000
0.4
50000
100000
epach
(a) ODEs vs simulations
(b) Robustness model
(c) Centered fairness
(d) General fairness
Figure 1: Teacher-Mixture in fairness and robustness. Panel (a) shows the generalisation errors for the subpopulations + (blue) and - (reai-obtained through simulation (crosses) and predicted by the theory (solid lines) for a network with linear activation. The inset shows the same comparison for the order parameters: R4, (blue), R_ (red), M (green), and Q (orange). Panels (b-d) exemplify the different scenarios achievable in the TM model investigated in Sec. 4 Panel (b) represent model for robustness where a spurious feature given by the shift vector can mislead the classifier, see Sec. [.1 Panels (c,d) are instead discussed in Sec. 4.2 and represent two models of faimesss. First, Panel (b) has no shift, = O, allowing us to remove the confounding effects. Finally, Panel (d) shows the general faimess problem.
2 Problem setup
Data distribution. We consider standard supervised learning setup where the training data consists of pairs of a feature vector E IRd and a binary label = +1. To model subgroups within the data [35], we assume that the feature vectors are structured as clusters on some fixed attribute vectors "1.-. Um E Rd. Specifically, x is sampled from a mixture of m isotropic Gaussians:
respectively centered
x ~ Σ Ni,
with mixing probabilities Pl, Pm and scalar variances A..... Am- Assuming the entries of vj of order 1 as d gets large, the scaling factor 1/Vd ensures that the Euclidean norm of the renormalised vector is of order This prevents the problem from becoming either trivial or overly challenging in the high-dimensional limit 24. We adopt teacher-mixture (TM) scenario [38 where each cluster has its own teacher rule:
I E
= sign(w] Vd).
This rule is characterised by the teacher vectors w, E Rdd ensuring linear separability within each cluster. Fig. [T--d illustrate the data distribution for two clusters with opposite mean vectors iv. which will be the primary case study for our analysis.
Model. In this study we analyse a linear model applied the above data distribution. We aim to leam vector parameter w, referred to as the 'student", such that predictions are given by
(<x) w TTx 2/
The training process involves applying online SGD the squared loss (y - At the k-th iteration, feature vector = is sampled from 1. the ground truth label 8² and current model

---

prediction i are respectively given by
2 and 3 and the parameter is updated as:
Aw :- w
17 Ve (w")
Va (v² )**
where / > 0 denotes the learning rate. is important to note that this online setting the number of time steps equivalent the number of training examples. In analysis, the model is evaluated by its generalisation error, or population loss, c == Elc].
SGD analysis
We study the evolution of the generalisation error during training with SGD with constant learning rate in the high dimensional setting (i.e. large d). Following a classical approach [34 8, we streamline the problem by focusing fully characterises the dynamics. As the dimension increases, it can be shown by concentration arguments that the evolution of these order parameters converges the deterministic solution of 6 3 Notably, in our setting, we achieve an analytical solution of this ODE system. We sketch our main results below, referring to the Appendix for derivations and proofs.
small of summary statistics, referred to 'order parameters', which
system of ODEs 15
3.1
Order parameters
In the setup described Section 2 consider the following 2m variables:
R2 w
M₃ =
Vjr
Q @=/lw]?
|w]l
(5)
for 1 j These variables correspond to key statistics of the student, namely its alignment to the cluster teachers, its alignment to the cluster centers, and its magnitude, respectively.
Lemma 3.1. The generalisation where Ej degree 2 polynomial in R., Mj and taking the form
be written
average ΣØ= PJCj over the clusters,
Ej 1- -=,-2a,,, + M3 - 3,R, + QA,
(6)
where aj By are constants independent of the parameter w.
We present the derivation of this result and the explicit form of the constants aj, B3 in Appendix
Our problem thus reduces to characterising the evolution of order parameters (5). Using the gradient
update of the parameter in Eq. 4 and the notation 8 = y g+. we can write update equations for the order parameters as follows:
AMA
77 d
A
W
x Vd
AQR
2n
W x
+
(60)
(7)
Vd
d
Vd
3.2 High dimensional dynamics
We build upon classic results 34 [) recently put rigorous grounds 15 3 leveraging the self-averaging property of the order parameters the high dimensional limit d + co. As a result, the dimension gets large, the discrete, stochastic evolution (7) of the order parameters can be effectively described in terms of the deterministic solution of the average continuous -ittuuuuu---meeeeee dynamics.
Let (S,) gi522m+1 denote the collection of order parameters. The following lemma shows that the average of the updates 7 over the sample x can be expressed solely in terms of S*
Lemma 3.2. 1 = (S))) for some functions (f((5)) in O(I) ) as d + cc.
The theorem below states that as gets large, the stochastic evolution S the order parameter gets uniformly close, with high probability, the average continuous-time dynamies described by the ODE system:
dS,(t) (((()), dl
1 2m + 1,
(8)
where the continuous time is given by the example number divided by the input dimension, = k/d. Formally.

---

Theorem 3.3. Fix a time horizon T > 0. For Sis S 2m + 1.
1 - S((k/d) f 0 as +00
OSkSAT
where S denotes convergence in probability. A proof is provided in Appendix We provide the 2 clusters
explicit expression of the functions fi in the ODEs (8) in Appendix C focusing for clarity.
3.3 Solving the ODEs
Here we present the explicit solution of the ODEs in the case of two clusters (m 2) with opposite mean vectors iu, as [88. Henceforth, we refer to as the shift vector and the two clusters as the 'positive' and 'negative' sub-populations, with mixing probabilities p and (1 p). variances A and teacher vectors W+. respectively. The order parameters introduced in Eq. are specifically denoted as M = w v/d, R+ w w and R_ w w-/d in this setting.
Theorem 3.4. In the above selting, solutions to the order parameter evolution take the form
M(t Moe Anitt +M M - -nvt m
(10)
R4 (1) = R4 - RT - 7/ ) + (e(e-n - MAm -
(11)
(() Que 1122 -
-ng 800(1 -7(22 miz -namminj
+ 12(e 1(24
1222-1120 mix - m
e -1sming) + kssle 1(20 n 2main) - (o(uutmmii)
+ kg(e 1126 122 -n amm"))
- --
(12)
with A mir pA + p) A 2miz ab + (1 م) and 1//1///d
The remaining constants less significant and reported in Appendix D.I and discussed further in Appendix [E] This solution allows us to describe important observables such as the generalisation (via Lemma 3.1 any timestep. Fig- [a plots the theoretical closed-form solutions along with values obtained through simulation when we set = 1000. Note the remarkable agreement between the analytical ODE solution and simulations of the online SGD dynamics in this high dimensional data limit.
Insights
generalization error
1.00
1.0
cosine similarity
In this section, we delve deeper into the solution derived in Theorem 3.4, By examining the exponents in Eqs. 12 we can identify the relevant training timescales. No- lably, M follows straightforward behaviour dominated by single timescale, whereas R. and Q exhibit multi- ple timescales, leading to significant implications for the emergence and evolution of bias during training.
0.75
0.5
0.50
0.25
0.0
10a
10a
10
epoch
In the following sections, we analyse increasingly com- plex scenarios understand how bias develops and evolves. Parameters specifying these different scenarios are the shift norm : = ||/|² and relative representation the subpopulation variances A and the teacher over- lap T+- = : の- w_/d. For simplicity we fix the teacher norm |w 1 ||2 Vd, so that T₁ is the cosine similarity between the two teachers.
Figure 2: Spurious correlations tran- sient alignment. Time-evolution of loss (purple), student-teacher (red) and student-shin (green) cosine similarities. The initial phase (green background) of learning aligns classifier and shift vec- tor before aligning with the teacher (red background), Sec. 4.1 Parameters: v 16, P 0.5. A A. 0.1,T= 1, 0.5. For these parameters. spuri- ous features allow the correct classifica- tion of 90% of the samples.
4.1 Spurious correlations
The emergence of spurious correlations during training exemplifies type of bias where a classifier favours a spurious feature over core one. To isolate the impact of spurious correlation in our model while
5

---

generalization error
0.01
1.0
order parameters
1.5
M
0.1
0.8
1.0
R
1.0
total
0.6
0.5
10.0
0.4
0.0
100.0
100
10
106
100
103
106
0.0
0.5
1.0
epoch
epoch
p
(a) Bias crossing
(b) Crossing phase diagram
Figure 3: The crossing phenomenon. Panel (a) (left side) shows the loss curves of sub-population (in red) and nub-population in blue along with the overall loss (in purple). We observe crossing cause by higher variance but lower representation in sub- uub-population -. The background colours represent the different phases of bias that are characterised by the evolution of the order parameters shown in Panel (a) (right side). Panel (b) shows the presence of the crossing phenomenon in large portion of the parameter space using phase diagram. Blue indicates asymptotic preference for sub-population + and red the opposite. Dark colours indicates regions where bias is consistent across training, while regions in light colours undergo crossing phenomenon, White indicates that learning rate was too high and training diverged. Parameters: 0, 0, = 1,T4 = 0.9, 0.1.
avoiding confounding effects, we consider perfectly overlapping teachers (w+ = w w- and sub- populations with equal variance and representation (p = 0.5, A+ = A-). With non- non-perfectly overlapping clusters * O, introduce a spurious correlation by adding small cosine similarity between the shift vector and the teacher, creating a label imbalance-an an imbalance between the proportion of positive and negative labels within each sub-population. The setting is illustrated in Fig.
From Eqs.
10
two relevant timescales for the problem are observed:
TM
(13)
TR = 1/1A máx
(14)
( A maie
The shortest timescale, TM. associated with M, indicates that the student first aligns with the spurious feature. By aligning with the shift vector, the student can predict most examples correctly, but not all. The effect of spurious correlations transient; t TR. the student starts disaligning from the spurious feature and aligns with the teacher vector, eventually achieving nearly perfect alignment. This is illustrated in Fig- 2 where the student initially picks up on spurious correlation (green) and achieves almost perfect alignment with the shift vector during intermediate times before aligning nearly perfectly with the teacher (red).
4.2 Fairness
In this section, we identify the properties of sub-populations that determine the bias during learning and show how bias evolves in three phases. To quantify bias, we use the overall accuracy equality metric 77], which measures the discrepancy in accuracy across groups Intuitively, we aim for equal loss on both groups, considering any deviation from this condition as bias.
4.2.1 Zero shift
We first consider a simplified case where we assume that both clusters are centered at the origin 0 as shown in Fig. We will later reintroduce the shift and analyse the transient dynamics introduces as per the discussion in section 4.1 The zero shift case represents extreme situation where not only is the classification not solvable if w + w but it is also difficult to identify which cluster generated a given data point since the shift provides no information. This setting is particularly suited to analysing the effects of 'group level" features, such as group variance and relative representation, on the preference of the classifier.

---

In this simplified setting, (t) is always zero and the constants kf k3. ka presented in equations 11 and 12 are zero. Thus, the dynamics only involve two relevant timescales given by TR Eq. and
QQ (/( mar
n 2mir)).
(15)
Fig. Bl illustrates the changing preference of the classifier. Specifically, we observe that the variance of the sub cob-population is particularly relevant initially and the sub-population with higher variance (red) learnt faster, i.e. its generalisation error drops faster. However, asymptotically we observe that the relative representation becomes more important wherein the student aligns itself with the teacher that has a higher product of representation and standard deviation (blue), i.e.
AVD, 2 (1 (1- (\- =
R
(16)
Thus, the network can advantage the cluster with higher variance initially but asymptotically advantage the other cluster if its representation is high enough. This leads to the interesting behaviour shown in observe a "crossing' of the losses on the two sub-populations. A more detailed
Fig. 3 wherein analysis of the 'crossing' is presented in Appendix E.2
Initial dynamics. Starting from small initialisation, the initial rate of change of the generalisation error for sub sub-population + is
= -1 máxe A+ (1
degt dt
2 R T -
(17)
and analogously for -. The learning rate 7) must be chosen to be small enough such that the generalisation errors decrease and hence the first term in the brackets must dominate over the 1. Since / R E 1/T:] (for T+ > 0), the ratio between generalisation error rates is bounded by
degt/ 1=0
1
A A
T+
(18)
deg- /
T
==
When the teachers are only slightly misaligned T 1-the bound is tight and we can see that it is the ratio of the square roots of the variances that determines which cluster is learnt faster initially. As precisely detailed below, the initial bias can substantially differ from the asymptotic bias of the classifier. Indeed, Fig. 3 shows in phase diagram the existence of 'bias crossing' across wide range of variances and representations. The transition between the phases that represent a initial preference for the positive sub- aab-ooullation (light red and dark blue) and the phases that represent an initial preference for negative sub-population (dark red and light blue) is approximately given by the line A_ A_ 1, independent of the representation predicted by Eq. [8] The portion of the dark blue phase just above the white divergent phase marks a 'quasi-divergent region wherein the generalisation error the negative aub-population rises even at = 0 because the learning rate is too large for such high variances. It hence marks a region of impractical behaviour that is only observed with poorly optimised learning rates.
Asymptotic preference. the limit of small learning rates 1 - O, the student will asymptotically exhibit lower loss on whichever sub-population's 's teacher it has better alignment with. Thus, Eq. T6 provides a simple characterisation of asymptotic preference from representations and standard deviations in the small learning rate limit. However, the situation is more complex in the case of finite learning rate, which may disrupt learning in one or both clusters. Without indulging further into the discussion of asymptotic performance, which is not the main goal of the paper, we refer to Appendix E.3 for more in depth and analysis and additional phase diagrams.
4.2.2 General case
We now consider the general case shown in Fig. Td. where the shift is non zero and all three timescales identified so far play a role.
As observed in Sec. 4. when the shift norm v is large, the effect of spurious correlations becomes significant and the timescale associated with the spurious correlations is the fastest. general, when v * () we observe an additional phase due to the effect of spurious correlation. In this new first phase, the student advantages the cluster with higher representation and lower variance since the salient information received from this cluster is more coherent and easier to access.
7

---

generalization error
order parameters
1.0
2
0.8
total
0.6
100
102
104
10c
100
102
104
10
epoch
epoch
Figure 4: Double crossing phenomenon. (Left panel) shows the loss for the two sub-populations (blue and red lines) and the global (in purple). (Right panel) shows the value of the order parameters across time. The behaviour of the order parameters across time provides a precise characterisation and understanding of the different phases. Parameters: 100, 0.75, A+ 0.1, = 0.5, v- 0.03, T+ = 0.9, a = 0.343, a 0.12.
More precisely, in high dimensions the shift and the teachers are likely to exhibit a small cosine similarity leading to class imbalance in the clusters and creating spurious correlation. The amount of label imbalance within cluster is characterised by the value of a, as detailed in Appendix [A] For smaller variances, a takes more extreme values leading stronger spurious correlation of that cluster with the shift. If a cluster has more positive examples, we would observe reduction in loss for that cluster if the student aligns with the mean of that cluster (and opposite the mean if the cluster has mostly negative examples). When both clusters have different majority classes, the direction of spurious correlation for the two are same. However, when the majority classes the same, we have competing directions for spurious correlation. The expression for M in Appendix [DI Eq. D.42 shows that in this case the relative representation comes into play and the mean of the cluster with greater representation and class imbalance will be chosen by the teacher to align with. Fig, shows such a scenario with three phase bias evolution:
The green phase is driven by spurious correlation where the positive cluster is advantaged since has greater representation and class imbalance.
Then, the red phase is driven by greater variance where the negative cluster is learnt faster as discussed through Eq. 18
Finally, we observe the orange phase wherein the student starts aligning with the positive cluster as per the asymptotic rule in Eq. [6
In summary, the student first advantages the sub-population with higher representation and lower variance. Next, advantages the sub-population with higher variance. Asymptotically, it advantages the sub--opulattion with higher representation and variance. Our analysis thus shows that bias is dynamical quantity that can vary non non-monotonically during training and cannot be characterised by simply the initial and asymptotic values.
5
Ablations using numerical simulations
5.1 Rotated MNIST
We train a 2-layer neural network with 200 hidden units, ReLU activation, and sigmoidal readout activation, using online SGD on MSE loss MNIST classification task. Data are centered and the variance is set to following standard pre-processing practices. We consider a variation of the MNIST dataset that mimics the TM model, allowing us to verify our theory when network and the data structure mismatched. Digits to 4 and 5 to 9 are grouped to form the two subpopulations. With probability p+ and p.. digits of both subpopulations are rotated with a aubpopulationnspeciiee angle-iee. Fig. 5 uses angles of rotation 0_ 45 and 0_ -90%. The goal of the classifier is to detect rotations.
The experimental framework gives correspondence between parameters of the generative model and properties of a real dataset. We can control relative representation by subsampling, teacher similarity by playing with angle difference, label imbalance by changing the probability of rotation, and saliency by increasing and decreasing the norm of the subpopulation using multiplicative factors A+. The only parameter that cannot control is the shift which is property of the data.

---

100
10
Accuracy Loss
Accuracy Loss
10"1
10
1.0
1.0
0.5
0.5
10a
10a
10a
10a
Epoch
Epoch
(a) Crossing phenomenon
(b) Double crossing phenomenon
VA_
0.1
A- =.2
VA_ 0.4
V 0.8
VA- 1.6
1.0
Loss
0.5
0.0
10a
10a Epoch
102
10
10
Epoch
Epoch
Epoch
Epoch
(c) Effect on initial learning phase
Figure 5: Numerical simulations MNIST. The figure shows the average (solid lines) and standard deviation (shaded area) 100 simulations run in this framework. In particular the upper plots show the test loss and lower plots the test accuracy for subpopulation + (blue) and (red). Panel (a) an example of crossing phenomenon obtained by imposing V 1, v 0.2, and 0.1. Panel (b) shows the double crossing, obtained by introducing an additional timescale to the previous case by tuning label imbalance. Panel (c) explore the effect of changing A_ while keeping constant A 1.
we remove the label imbalance by setting the probability of rotation p+ = p- 0.5. By properly calibrating the saliency A and the relative representation p. it is possible to bias the classifier towards one subpopulation at the beginning of training and the other in the end. This is shown in Fig. where p = 0.1 and A, > A.. The saliency difference favours subpopulation + initially while setting P small enough advantages subpopulation later training. This is precisely what we observe in the plot.
Therefore, in order to reproduce the zero-shift case of Sec. 4.2
In Fig. Sb. we extend the MNIST experiment by varying the average image brightness of subpopula- tion -, which reflects the group variance in our theory. The results show that greater brightness leads to faster learning in the second phase and an increasing asymptotic preference, consistent with our predictions.
Finally, we consider the general fairness case. By creating label imbalance, i.e. setting p+ = 0.3 and p- 0.7, observe additional phase of bias evolution, wherein the classifier prefers dense regions with consistent labels. This advantages subpopulation and indeed is what we see in Fig. 5t. The result of the simulations matches the theory displaying double crossing phenomenon.
5.2 CIFARI0
We consider the same architecture and pre- pre-processing described for MNIST CIFARIO classifica- tion task. We select 8 classes and assign of them to the positive group and to the negative group. Inside each group, classes are labelled as negative and 2 as positive. This simulation framework is similar to the one considered by [ where the authors used sub- popullationn with only 2 classes each.
The average brightness of the samples in each cluster plays the same role as the parameter A in the synthetic model. Our theory predicts that the classifier will advantage the group with highest average brightness, see Eq. 16 In order to achieve the same generalisation error on both subpopulations, the less bright group needs more samples (larger p). This is shown in Fig. Gb, where the three panels correspond to different assignment of the classes: in the top panel classes are randomly assigned to

---

1.00
1.00
1.0
Test Loss
1.25
0.75
Test Loss
0.9
Test Accuracy
0.7
0.75
1.00
0.50
0.75
0.8
0.6
1.00
0.25
0.7
0.5
0.75
0.00
102
10a
102
10
0.25
0.75
Epoch
Epoch
P
(a) Group brightness
(b) Crossing and dependence
Figure 6: Numerical simulations CIFAR1O. The figure shows experiments of 21. neural network on CIFARIA 10 where classes were grouped together to form the subpopulations. The plots show the average performance-
by loss securacy-achiever over 100 simulations (for Panel (a)) and 10 simulations (for Panel (b), respectively) using the shaded area to quantify the standard deviation. Panel (a) shows the result at the end of training changing relative representation while Panel (b) shows the training trajectories as changes as indicated in the colour-bar, see text for more details.
the two groups; the middle panel classes are randomly partitioned two groups and the brighter one is assigned group -; finally the last panel assigns the brightest classes to group and least bright to group +. As predicted, we need increasingly high relative representation p to achieve balance in losses at the end of training.
When labels are balanced, our theory predicts that the classifier is initially attracted by the larger A and eventually- the relative representation of the group with smaller A is large enough-it switches and favours the other group. This effect indeed verified in the CIFARIO experiments. Starting from the partitioning in Fig. 66 (bottom) with p 0.8, the dynamics is initially attracted by group - before advantaging the other group, giving rise to a crossing that can be observed- other things-in in Fig. (p.
Fig highlights another prediction of our theory concerning the timescale when p becomes relevant. Our theory predicts that should become relevant only in the final stages of the dynamics and indeed the curves are almost perfectly overlapping until most of training. already noticed, sufficiently large gives rise to crossing behaviour indicated demonstrated in the synthetic model.
5.3 Additional numerical experiments
In Appendix F provide additional experiments within the TM model and the CelebA dataset, exploring different architectures and losses. Even under these new conditions, we observe that bias presents different timescales and shows crossing behaviours.
Conclusion
This paper examined the dynamics of bias a high dimensional synthetic framework, showing that be explicitly characterised to reveal transient behaviour. Our findings reveal that classifiers exhibit biases toward different data features during training, possibly alternating sub-population preference. Although our analysis is based on certain assumptions, numerical experiments that violate these assumptions still display the behaviour predicated by our theory.
While this paper centered on bias propagation in a controlled synthetic setting. the study of bias in ML systems significant issue with profound societal implications. We believe this line of research will have practical impacts in the medium term, aiding the design of mitigation strategies that account for transient dynamics. Future research will further explore this connection, proposing theory-based dynamical protocols for bias mitigation.
10

---

Acknowledgements
We thank Andrew Saxe and Simon Lacoste-Julien for numerous discussions and insightful com- Anchit.Anchit Jain is grateful to the Gatsby Computational Neuroscience Unit for the research internship opportunity. This work was supported by the Sainsbury Wellcome Centre Core Grant from Wellcome (21 (219627/2/19/Z) and the Gatsby Charitable Foundation (GAT3755).
References
[1] Rangachari Anand, Kishan G Mehrotra, Chilukuri K Mohan, and Sanjay Ranka. An improved algorithm for neural network classification of imbalanced training sets. IEEE transactions neural networks, 4(6):962-969, 1993.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv:1907 02893, 2019.
[3] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high- dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 1199-1 PMLR, 12-15 Jul 2023.
[4] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In International conference on machine learning, pages 233-242 PMLR, 2017.
[5] Samuel James Bell and Levent Sagun. Simplicity bias leads to amplified performance disparities. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 355-369, 2023.
[6] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling. Advances in Neural Information Processing Systems, 11-25349-253666 2022.
[7] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairnsss in criminal justice risk assessments: The state of the art. Sociological Methods Research, 50(1):3 44, 2021.
[8] Michael Biehl and Holm Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
[9] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
[10] Farzan Farnia, Jesse Zhang, and David Tse. A spectral approach to generalization and optimiza- tion in neural networks. 2018.
[11] Yunhe Feng and Chirag Shah. Has ceo gender bias really been fixed? adversarial attacking and improving gender faimess in image search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11882 11890, 2022.
[12] Emanuele Francazi, Aurelien Lucchi, and Mareo Baity-Jesi. Initial guessing bias: How untrained networks favor some classes. arXiv preprint arXiv:2.306.00 00809, 2023.
[13] Prakhar Ganesh, Hongyan Chang, Martin Strobel, and Reza Shokri. On the impact of machine learning randomness on group fairness. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1789-1800, 2023.
[14] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2(1 ())6))666666,,, 2020.

---

[15] Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks the teacher-student setup. Advances in neural information processing systems, 32, 2019.
[16] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018.
[17] Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from baas-immalanccd data: Review of methods and applications. Expert systems with applications, 73:220-233 239,, 2017.
[18) Katherine Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning. In The Twelfth International Conference on Learning Representations, 2024.
[19] Eugenia lofinoa,, Alexandra Peste, and Dan Alistarh. Bias in pruned vision models: In-depth analysis and countermeasures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24364 24373, 2023.
[20] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548-1558, 2021.
[21] Matthew Kay, Cynthia Matuszek, and Sean A Munson. Unequal representation and gender stereotypes in image search results for occupations. In Proceedings of the 33rd annual acm conference on human factors in computing systems, pages 38 19-33828, 2015.
[22] Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, ((4):221-233) 2016.
[23] Yann LeCun, Léon Bottou, Genevieve Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[24] Mare Lelarge and Léo Miolane. Asymptotic bayes risk for gaussian mixture in semi-supervised setting. In 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP). pages 639- 643. IEEE, 2019.
[25] Thibault Lesieur, Caterina De Bacco, Jess Banks, Florent Krzakala, Cris Moore, and Lenka Zdeborová. Phase transitions and optimal algorithms in high- digh-iiiensioaal gaussian mixture clus- tering. In 2016 54th Annual Allerton Conference Communication, Control, and Computing (Allerton), pages 601-608. IEEE, 2016.
[26] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792 PMLR, 2021.
[27] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 111 2018
[28] Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.
[29] Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the fail- ure modes of diitri--dittiittinnn generalization. In International Conference Learning Representations, 2021.
[30] Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 33, 2020.
12

---

[31]
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
[32] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301-5310. PMLR, 2019.
[33] Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. In International Conference Machine Learning, pages 28843-28863. PMLR, 2023.
[34] David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks, Advances neural information processing systems, 8, 1995.
[35] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[36] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020.
[37] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Perey Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR, 2020.
[38] Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luca Saglietti. Un- fair geometries: exactly solvable data model with faimess implications. arXiv preprint 2rXiv:2205.15935. 2022.
[39] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
[40] Harini Suresh and John Guttag. A framework for understanding sources of harm throughout the machine learning life cycle. In Equity and access in algorithms, mechanisms, and optimization, pages 1-9. 2021.
[41] Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv: 1808. 04295, 2018
[42] Yu Yang. Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. In International Conference on Artificial Intelligence and Statistics, pages 2953-2961. PMLR, 2024.
[43] Han- Jia Ye, De-Chuan Zhan, and Wei-Lun Chao. Procrustean training for imbalanced deep learning. In Proceedings of the IEEE/CVF international conference computer vision, pages 92-102, 2021.
13

---

Appendix
Contents
Problem setup and notation
15
Proof of the main theorems
16
Proof of Lemma
16
B.2 Proof of Lemma
16
B 3 Proof of 3.3
16
Derivation of the ODEs
17
Usetul Averages
17
ODEs
18
Student-shift overlap M
18
Student-teacher + overlap 12+
19
Student-teacher overlap R_
19
Self-overlap Q
19
Continuous limit.
D ODE solutions
23
General
23
23
23
R
23
23
D.2
Spurious correlations setting
24
D.3
Fairness setting
25
Deeper analysis of the learning dynamics equations
25
Single centered cluster
25
Analysis of teacher alignment (TR) and student magnitude (TQ) timescales
25
Asymptotic preference
26
Additional numerical simulations
27
CelebA
27
Simulations on Synthetic Data and Deeper Networks
29
14

---

R4 w W d: A Problem setup and notation
We begin by refreshing the problem description and notation introduced in the main body for the two cluster case (Sec. 3.3) as well as defining some new notation to make the presentations of the results more compact.
1. (z, y) denotes training example with æ E Rd and y E (-1,1).
2. æ is drawn from a mixture of two Gaussians with means v d and -v/ v respectively. covariances A+ Idxd and A Idxd respectively. These two Gaussians are henceforth referred as the positive and negative Gaussians respectively.
3. p represents the probability of the data being drawn from the positive Gaussian.
4. () denotes average over J. (a denotes average over the positive Gaussian and () denotes an average over the negative Gaussian.
5. W+ and denote the teachers for the positive Gaussian and negative Gaussian respectively, w is the learnt classifier ("the student").
6. The true labels, y, are then given by:
y sign( w+ </vd) for the positive cluster;
siggi w- x/ x/Vd) for the negative cluster.
7. Our predictions are
x /d.
8. The student is trained to minimise L2 loss = (y 0)?
9. The student learns using online stochastic gradient descent.
1//2 is the learning rate.
11. denotes the generalisation error.
12. b denotes the dot product between vectors and b.
13. We now define the following Order Parameters (where only the first 4 change with training):
Q = w w d:
R_ = w W /d:
M v/d;
T* W /d:;
M W+ v/d:
M W v/d:
V = V v/d.
14. For algebraic simplicity, we assume |
w ||| vd (and thus, W w+/d = and w_ w- /d= 1). This has the consequence that T+ exactly equals the cosine similarity between the two teachers.
15. We also define mir pA+ + (1 (1-p)4_ and A 2mix =p ()11))
16. For notational convenience we define:
Mi
04 (v) = 2Φ
(A.19)
VA,
((-M)
a_ (v) 2Φ
(A.20)
Note, 04 also has an intuitive meaning. It represents the difference between the probability that an example drawn from the positive cluster has positive true label and the probability that an example drawn from the positive cluster has negative true label. It is hence 0 when the positive cluster has equal positive and negative examples, positive when the cluster has more positive examples than negative, negative when the cluster has more negative examples than positive. Similarly, a. represents the difference in these probabilities for the negative cluster.
15

---

17. Finally, we also define
2A +
-
-
3+
exp
(A.21)
π
2
2A
B_
exp
(A.22)
2A
18. Lastly, we use f to denote continuous time given by (epoch number dd)
B Proof of the main theorems
B.1 Proof of Lemma 3.1
Denote with ()y the expectation over samples from cluster j. The generalisation error reads e Li=1 Picz with
cj = ((v 0)2;; ( w x
21
21
= =(>?), +
w x Vd
w 2
-
y
Vd
Vd
=1+ (Q4, + Mf) 2(a, Mj + R2B,),
where the second term comes from: isolating the mean and the definition of M,. and the isotropy of a. The third term comes from the useful identity Integral Eq. the constants are given by
C.31 derived in Appendix C.I and
Mj
24,
((M)))
a2 = 1 - 1-24
By =V
exp
(B (B.23)
/A
2A,
where Mj = w. vy/d and ((x)
- u- /2 du is the cumulative distribution function of /
1
the standard normal.
The formula for the generalisation error specializes to the case of two clusters with opposite means as
= 1 + M - (2pa - 2(1 - P M
(B.24)
- -2p - 2(1 R_ + A Am*Q.
Notably, a has an intuitive meaning wherein it represents the difference between the fraction of positive and negatives in cluster, i.e., a+ (y) and a (w)
B.2 Proof of Lemma 3.2
Explicit computations are carried out in Appendix C.2 below for the case of two clusters.
B.3 Proof of Thm
Using the notation of Section 3.2 and assuming Lemma 3.2
we examine the update equations
written stochastic iterative process
1 s = S* + E8 f(S")) +
ck Sd
< : vd(As* - -
(B (B.25)
where the expectation is over the new sample x and conditional the past samples. The noise term ck has zero mean [[(5)] = and conditional covariance En == E ck J.
Sd
Define the continuous-tin time rescaled process (a)) as the linear interpolation S(td),
Sa(() = SL + (td - [sd]) - s((d))
(B.26)
Here we leverage existing stochastic process convergence results (e.g., (6) Theorem 2.3I) showing that, if Eg converges to the matrix valued function (S) as d in some appropriate sense, then
16

---

thereby proving the the sequence (a(t) converges weakly as d + 3o to the solution St of the stochastic differential equation:
d$, f(S))dt + (S))dB
(B.27)
where B, a standard Brownian motion in 2 In our case, we can show that Mi (dd))) as d + 00, so that Σ = 0 and Eq. B.27 reduces to the ODE in Eq. 8
Let us sketch the scaling argument. Algebraic manipulations similar to those in Section C.2 show that
Ed = V "e "alp"**TTjvaa J O(d=)))
Th == n=6=" [B(
(B.28)
where V denotes the gradient with respect to the student vector w. Recall that S has 2m components
that are linear in (corresponding to the order parameters R3 and M, in Eq. (5) and one that is
quadratic (corresponding to Q). By making the gradients VS explicit using Eq. 5), that at
leading order, the matrix entries Σ 1 S i.j 2m + take the form
Da n(6 a x
a x 1666
M ー =
TT
(B.29)
vd
Vd
where the vector a; is either one of the teacher vectors Wy. one of the shift vector vy, or the student vector w, depending the entry i 1, 22 + 1. As can be shown explicitly in Appendix [C.] below, Th depend on x* only through auxiliary variables w. z Vd, F x Vd, w* x* /d, which jointly follow a multivariate distribution whose parameters depend on student vector w" only through S and are in O(1) as d > co. As a result, Σ E (dd
Finally, the weak convergence of St(t), to S, implies convergence in probability for the supremum norm on the interval [0, 7' for any T > 0. Specifically, for each 1 < < 2m +
sup (ta St()) P 0. 051ST
(B.30)
where E, denotes convergence in probability. This result directly leads to Eq. theorem.
C Derivation of the ODEs
In this section we are going to explicitly derive the ODE describing the dynamics of the order parameters. Starting from the discrete updates of the order parameters, Eqs. 7 we are going to consider the thermodynamic limit, + co. As proven in Thm. 3.3 the updates concentrate to their typical value and the discrete evolution converges to differential equations. Therefore, the rest of the section devoted to performing averages over the Gaussians in order to evaluate the typical values. Before proceeding with the evaluation of Eqs. is useful to introduce two identities.
C.1 Useful Averages
Integral
(- -cl + by b. bm
2A
(=(b c)2
(a z sign(b x + c)) (a :)(4))1111 24
exp
2Ab-1
(C.31)
where is multivariate normal distribution with mean μ and covariance AI, and the angular bracket notation indicates average with respect to æ.
Derivation. Define the auxiliary random variables 21 x and 22 b x + c, that follow multivariate normal distribution
21
a.
A
a b b b.b]
z2
b
Using the law iterated expectation, our average can be written as:
(a sign(b- + c)) Ex, [sign( 22 Es, [2]]]
17

---

a.b
Ez (sign( (a
b. (22 (b. c))]
= (a. a a.b
b. (b. c)) sign(>2)) + a B (=2))
b.
The first expectation follows from the definition of the cumulative distribution function +
bg [zg)] (1 2Φ
VAb.1
The second term simply the mean of folded normal distribution
(-66 μ + +0²
(b)u++)(11 -24 (= c)
[2g(insign - (V b) exp
(
2Ab
VAb. b
Combining these three expressions we obtain the identity. Integral 2:
(a. = (a. p)(b 14) + ((a
(C.32)
where x is defined as for the previous identity.
Derivation. We proceed in the previous case. Define the auxiliary random variables 21 and 22 = b. x. They follow a multivariate normal distribution
21 22
a
a a
b b b
A
Using the law of iterated expectation, average may be written as:
(a b z) Ex, Ez12111 [21]]
RZz (2 1 a b
b. (22 (b. 1))
a (b 1,) [22] - a a.
(a.
E., 22
b-b
b.b
(a b.b ()) 4) + a.b
a.
Ab (b ()-
b.b
(a ()bb v) + A(a b).
C.2 ODEs
We have now the building blocks to evaluate the expected values of Eqs. 7 We refresh the notation that 5" y 1 yd sign (t w w,./Vd and n x w/ vd.vd. Final step is take the
continuous limit. This is obtained by noticing that the RHS of the equations is factorised by 1 1/d. Therefore by taking as time unit 1/d d and defining time as t = μ/d the discrete updates converge to contimuous increments d -
Student-shift overlap M.
(AM) Derivation. Starting from the definition in Eq. 7 for M
! d (pva+ PMI - -(1- p-l-p)a- + +1
P)M (M(u (MMuuAmmm) )mm)))
(C.33)
(AM) =" (2(((1)))) x
Vd
Vd
Vd
The first term can be evaluated using integral and the second term using integral 2 yielding the result.
18

---

Student-feacher
overlap R+.
(AR+) / (p(MFa+ + + (1- +T19-)
-P(MM + (1- R.A-))
(C.34)
Derivation.
x
I W
(AR+)
y
Vd
vd
+
(
x ++
w x I w
w x I W
p
Vd
+ (1
vd
vd
vd
- - p)
vd vd
These 4 terms can be computed using integrals and 2 yielding the result.
Student-teacher overlap R...
(AR_) 2 (p(MMa + +T+4) + (1 - (1- +
-PMM + R_ (1 (MMM +R-A))
(C.35)
Derivation, Same as for R4.
Self-overlap Q.
(AQ) = 2n (pla+M++) + +,4 + (1 - (00-p)(aaa1 M + 3 R+) M QA mix
+ d ( A mix + +QA 2mi + +M² A miz
172
QQ2min
2 (C00:(..N-.BRR-----AAAAAA------1
(C.36)
Derivation, This update requires additional steps with respect to the previous ones.
2n w
112 ( (6*)2
|
(QQ)
d
Vd
d
d
The first term is
W x 21
2m
2n
w x
y
d
Vd
d
Vd
Vd
2n
(M (pa4 (1 p)a-) ) p R4 + (1 p)3 R
M² QA miz)
d
The second term
1/2
( (88921150/11
)
1/2
w
2
d
d
Vd
d
2
x E
W
2y w
2 E
+
d
Vd
d
Vd
d
requires additional steps. We consider the three terms in the expression above, starting from the first one
2 D x
Σ ( (z-)))
d
d
19

---

d Σ + + ///d) + (1-p)(A. + +z/dd) - A mix + v/d
A mir +0(d))
Where we used the simplification y²² independently of the cluster's teacher. However, the remaining terms require us to split the ex expectation considering the probability of sampling from each cluster. The second term
2
2
x
w
x
x
w x
x 1
w
+
p)
d
d
vd
d
vd
We begin by analysing the average over the positive Gaussian and split as - v/ v/Vd + 2 such that has zero mean. Then,
U U
20 3
8 +
w
U.W +
d2 dvd
@
Multiplying the terms the brackets will give rise to 9 terms. We can see that the 3+3=6 terms corresponding to v/d and 2v 2/dvd will tend to the limit of infinite d due to their scaling. We now analyse the other 3 terms:
Term 1:
21
2
w
w
+ 0
d
d
d
d
M
+ +ddd))
Term 2:
1
w
2
2R
V
2 1
w T
2R
+ Odd
vd @
= 0
Term 3:
W & ②
2
W 5
vd +
d
vd
(+ 0 = Q43 + ))
Thus finally,
21
p(M² A+ +1 PM + +QA2)
d
Vd
M mix 1-Q22mme 2meia
For the the third term
1 r
10
1
+ p)
Vd d
Vd
d
Vd
d
As before, we analyse the average over positive Gaussian first and split x into its mean and zero mean component:
1 2
20
1 u
20 x
2 x
0
y
d
d
d2
dvd
d
+
20

---

This gives rise to 6 terms. We can see that the 2+2-4 terms corresponding to v v.vdd and 20 2/dvd will tend in the limit of infinite due their scaling. We now analyse the other 2 terms:
Term
x
y
M
+
W
M
sign
Vd
d
d
M
sign
W+ +
W
+
+ O(d=l)
vd
d
d
19
8
M (y) @ A+ +0(d))
MM 0(d
Term 2:
x x
w +
w +
i x
+
=
y/
d
d
A+
y
W
+ O
A R 3+ + (dd
Where the last equality follows using integral 1. Thus:
x I
= = (a + 3+ O.
d
Vd
We repeat the same analysis for the negative gaussian and get:
x.w. = - + (1- (L.
Vd
Collecting everything together and taking the infinite dimensional limit:
- 1/2 d (A mix + QQ 2miz + +M2 min - 2 (1--0
(Aw- Aw/d =
Thus,
(AQ) 2n ( (1- M - M Q4 méx)
112 d
+ (Amix 2mir + mizi (a+ M +
(1 (1- 3 R4)))
Continuous limit. Final step of the derivation is taking the termodynamics limit that leads to the
ODEs implicitely defined Thm. 3.3
(M, R4, RR,Q) n poa+ PM1
(1 p)vc + +1- - (M(v A )
(C.37)
(M, (p(Mia+ (1-))++(1 +T+B)
-MMMM + R4A+) (1 PMM + +R+A-).
(C.38)
fR_ (M, R_ - +1 (1-))+(1-((-333 +B_)
-MMMM + +R (1- + +R.A)).
(C.39)
21

---

(M R4, Q) = - - Qa
+ n/² A mir
+ QQA
2mir
+ +M² miz - (a M +,+
+ +11-p)
(C.40)
22

---

D ODE solutions
In this section we first present the general solutions of the ODEs sketched in Theorem 3.4 then specialise to the two scenarios discussed in the main text.
D.1
General case
From the previous section, we have system of coupled ODEs for the order parameters of the form:
dMM
dt c c2,
dR_
dt C3- CM-M + C5 R.,, dR_
dl c3+ + c4+ M cz+ R4:
dQ
dt co + c,M + c. + R+ Cg + c.QQ.
This represent linear system of ODEs which can solved using standard methods like Laplace transform, leading to Eqs. 10 12 We now report the equations including the exact expression of their coefficients.
M
M(I) Moe -oe-tn(v+ in(v+Ammy maxy + Mox (1 --bl(+4mmm))
(D.41)
Where,
Mx. (pM1B+ (1- (- u(pa4 - -(1 ppa))
(D.42)
+ A mix
R4:
(4(() = -00)-R00 -n + R en- 1I (1- (e- mnmm -c-1n(u-Amin)
tn mix
(D.43)
Where,
R (pB+ + (1 p)33) + M1 (1 p)ac_ -Mao)
(D.44)
k1 Mi - -Mo)
(D.45)
R_:
R_(1) Rd e- -nnmmi + R - --t minr
11- (1- -c-tn(utsmin)
(D.46)
Where,
R (T+p3+ + (1 -)) M2 p)cc_ Moo)
(D.47)
A mic
41 MM Mo)
(D.48)
Q:
Q(1) Qoe 11/24 D ammir)
+ Q000 1n(2A -1 220e))
+ ka(e 11/2A - -14
+ ks(e 11(2A padmit -0
+ kale 1122 - -17(2-224 mit)))
(D.49)
Where,
Qxc A már + 2p33 RC20 (1 7)+) + + p)B R 7
2A máe
2main
23

---

Mac (M (nmrr -2) + 2por+ 7/44) - 2(1 p)pp nT-)))
(D.50)
2A 11 2miz
h2 = 2p3+ (1 n1 ( 14 -ki+) ) 2(1 p)3 (1 (1-774 \ - RRº ki
(D.51)
A - 11 2mii
k3 2p3+(1 n )k+ + 2(1 p)3 - )
mis 2mir +
(Mo (M (n Mix
+ -2)
2pcr (1 m)+) p) n--))
A Amiz 11 2miz +
(D.52)
(n m - (M - M
k4
(D.53)
A 2rise 22
D.2 Spurious correlations setting
Under the setting discussed in the Sec. (p 0.5, A, A_ =,TT, 1), we can make the following simplifications:
1. A máe A
2. A 2mür A
3. 04+ - O2
4. 34 = 3_ = B,
5. M M M".
6. R4 R_ R.
The equations then take the form:
(t) Moe (v(++A) + Mc e -b((++)),
R(t) R -nA + R (1 e -md) (e -tn
ind e-(((++)),
Q(t) Qoe (2(22 ))?) + Q00 (1 - 11/(22 -11(23-022)) -122))
+ ke -1n(22 -12 -
+ kg in(2A-1A3 -(n(++A))
+ ka (n e- -tn(204241).
Where,
Ma M 3 vo
A
R50 3 + M (a Moo)
A
41 (M -Mo)
Qa A 2,31300 (1 T))
+ (o 2) + 22 n-))
2A 7/2
2A n
K2 28(1 7/ (Roo
RR -k))
n
28(1 7/ )ki
(Mos - Mo) 2) + 20 - -))))
K3 A 1 +
A 7 +
(AA 2M _2)2
AA2_ + 2v
24

---

D.3 Fairness setting
The general fairness case coincides with the general case discussed above (D.1) therefore we limit our discussion to the simplified case with centered clusters.
Under the shift e O, the equations take the simplified form wherein M, e, MI are 0, the transient term in R vanishes and Q only has one transient term. Specifically:
(+) R.. MMA miz
+ R
(t( R
+ R
112 men.
Q(1) = Qoe -nn(2 mir -
20 11(22 - )))))) + Qtrans tt((2 (e-t// n, A2mi))
-b11 miz
Where
2 PV A+ +
A Amn
R TLPV D+
A
Amir
A miz + 2 PA P//VV - -)++) + 2 -1 nA-)
20
miir '1A 2miíx
2 22pV A A Rº, -1 XR - -R
Qtrans
A miz A 2miz
E Deeper analysis of the learning dynamics equations
This section provides insights into the learning dynamics particularly those relevant bias evolution that arise out of the expressions for order parameter evolution. We shall provide intuitive explanations behind the various mathematical terms that appear.
E.1 Single centered cluster
Consider first a single cluster centered at the origin-i.e. p 1, O with variance A. In this setting, the minimum generalisation error is achieved when the student perfectly aligns with the teacher and achieving the generalisation error mmin =
optimises its norm such that Qopt = 2 .
Importantly, this is not O since the student and the teacher mismatched -i.e. the student is linear whereas the teacher has a sign(.) activation function. From the equations, we observe that the asymptotic generalisation enror when training using online stochastic gradient descent in this setting is
- 2/T (1 (1-2
(1
7 + +0(1)22)
=
1- 7/2
(E.54)
Thus, as the learning rate increases, the generalisation error increases until it reaches the critical learning rate beyond which training unstable and the loss grows unboundedly. In the single cluster case, Eq. [E.4 this is 2/ A which matches the classical result from convex optimisation (23]. We can similarly find the critical learning rate for clusters be 2A mén /A 2miz by ensuring exponential terms decay to zero in equation
E.2 Analysis of teacher alignment (Tx) and student magnitude (TQ) timescales
We now consider the fairness setting with zero shift illustrated in Fig. Tc. As discussed in section the relevant timescales in this setting are
TR
mix
TQ
(22 miz n 2mix)
since M()) is always zero. Fig. shows the crossing phenomena of the loss curves along with the order parameter evolution and other insightful terms. The alignment of the student is governed by the
25

---

Linear Activation
6.3
generalisaionn erfor
0.9
0.8
order parameter
1.0
0.7
0.5
0.6
total
Qrana
0.5
-0.5
R-
R,
10a
10a
10a
10a
10a 102
epech
epoch
Figure 7: The Crossing Phenomenon The left shows the "crossing' of the loss curves on the negative sub-population red (higher variance and lower representation) and positive sub-population in blue (lower variance but greater representation) along with the overall loss in purple obtained as a weighted average the two. It also marks Tn as the dashed vertical line and TO as the dotted vertical line. The right side shows the evolution of the order parameters and transient term. The horizontal blue and red dash-dotted line mark the optimal value of Q for the and negative sub-poue aub-populations respectively. The parameters are 0, p 0.8, A+ = 0.1, A_ 1, = 0.9, 7 = 0.1.
timescale TR and the change in its magnitude is governed by the timescale TQ. Initially, the classifier has small magnitude and its alignment roughly matches the two teachers which are themselves quite similar (T- = 0.9). Indeed, we see that the R_ and R_ have very similar trajectories. However, smaller magnitudes advantage higher variances discussed in Appendix E. (Qopt is inversely proportional to the cluster variance).
We mark the optimal values of Q using horizontal lines in Fig 7 the left side with blue for the positive sub-population (lower variance) and red for the negative sub-population (higher variance). As the magnitude of the student grows, observe sharp drop in the generalisation error the higher variance uub-population till Q crosses the horizontal red line. Beyond this point, the generalisation error on the higher variance sub-population rises since the magnitude of the student has exceeded the optimal value (horizontal red line) and the generalisation error on the lower variance sub-population continues fall as the magnitude of the student approaches the horizontal blue line. Finally, an inspection of the timescales reveals that TQ (vertical dotted line) is less than in (vertical dashed line) and hence we may expect the student magnitude to saturate before its alignment. However, Qtranss the transient term associated with Q (third line of equation 12). is always negative and hence suppresses the growth of Q initially.
In summary, we observe a two phase behaviour. First the student shifts its alignment and increases magnitude leading sharper drop in the higher variance generalisation emror. Second, we observe that as the student continues increasing magnitude while keeping its alignment fixed, it advantages the lower variance cluster.
E.3 Asymptotic preference
This section discusses the asymptotic generalisation errors of our classifier when v = 0 as a function of representation and variances. Firstly, discussed in section 4.2
R
R - / p) A_.
Intuitively, one might expect that the asymptotically lower generalisation error is achieved on the population whose teacher has better asymptotic alignment with the student. Indeed, when the learning rate tends O, we observe exactly this illustrated by the two dark phases in Fig. X the left side. However, when the learning rate is greater than zero, we observe more complex behaviour. Fig. g (right) shows the emergence two new phases (light red and light blue) wherein the classifier exhibits higher generalisation error a sub- population despite having better alignment with its corresponding teacher. This behaviour can be traced back to equation E.54 wherein the increase in asymptotic generalisation error due to non-zero learning rates is amplified by the cluster variance.
26

---

0.01
0.025
E. <t-
R, > R-
0.158
E. <t- R, <R.
0.398
1.0
divergent
2.512
E.
R, > R-
15.84 15.849
39.811
r- >-- R, <
100.0
D.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.2
0.3
0.4
0.5
0.6
0.7
0
0.9
Figure 8: Initial and Asymptotic student preferences We set 0, 0, = 0.1 and study the values of A-. The figure studies only asymptotic preferences under v == 0, = 1, 0.9. When the learning rate is small (1 + 0 on left side), the cluster which has better alignment with the teacher must also have lower generalisation error. However, for non-zero learning rates (1) 0.1 on right side), behaviour is more complicated leading to the light colored phases where despite better asymptotic alignment with the teacher, the generalisation error is higher. Parameters: 0.1 (right).
7 0+ (left)
Thus, analysis shows how large learning by advantaging the sub-population with smaller variance.
also become source of bias in classifier
Additional numerical simulations
F1
CelebA
1.0
0.8
Accuracy
0.8
Accuracy
0.7
0.6
0.4
group
Accuracy Diffrence
Accuracy Difference
Accuracy Difference
0.02
0.00
0.00
-0.01
0.02
0.00
0.02
-0.06
104
102
10a
10a
10a
103
10a
10a
10a
Epoch
Epoch
Epoch
(a) (Eye glass, Bags under eyes)
(b) (Bangs, Blurry)
(c) (Young, Blond Hair)
Figure 9: Numerical simulations in the CelebA dataset. Figure shows the average accuracy (solid lines) and standard deviation (shaded area) of 4 different runs this framework. The top row depicts the test accuracy
the course of training for different pairs of target and group attributes. The bottom row illustrates the difference test accuracies between the + and subpopulations, highlighting the crossing phenomenon observed during training. Panels (a), (b), and (c) depict this for the pairs of target and group attributes of (Eye glass, Bags under eyes), (Bangs, Blurry), and (Young, Blond Hair), respectively.
The goal of this experiment is to show the emergence of different timescales in realist scenarios of relevance for the fairness literature.
27

---

The CelebA dataset [27] contains over 200k celebrity images annotated with 40 attribute labels, covering wide range of facial attributes such as gender, age, and expressions. For this experiment, we consider different pairs as the target and group attributes. The task is to predict the target attribute while the group attribute defines the + and subpopulations.
For the model, we select a pretrained ResNet- 18 model on ImageNet and add an additional fully connected layer, with only the latter being optimised during training. We use cross-entropy as the loss objective and train via online SGD.
avoid attributes that pathologically We randomly selected target-label pairs, making significance of the result. In the plots shown underrepresented in the dataset and would hinder the in Fig. show some of the pairs that show crossing phenomenon. Each panel in Fig. show the accuracy and accuracy gap over the course of training. Notice how the classifier favours sub-population in the initial phase of training before changing preference.
This result shows that bias can change over the course of training even in standard setting. This does not imply that it will always occur and indeed several of the pairs in the dataset do not show crossing phenomenon. However, understanding when and why this phenomenon occurs can affect the algorithmic choices that we make in our ML pipeline.
28

---

F22
Simulations on Synthetic Data and Deeper Networks
Standand Contiguration
AdarT
with Weight Decay
0.73
Text tror
Test Error
Test Error
50
0.5
0.50
107
36"
0.45
bach
Ppoch
Figure 10: Synthetic Data Simulation with alternate Training Protocols We observe the 'double- crossing' phenomena in not only the loss curves, but also the error curves for the positive sub- population (blue) and the negative sub-pou populaptottion (red) (left). The shaded areas quantify the standard deviation obtained across seeds. We observe similar behavior when using Adam (middle) and weight decay (right). The data distribution parameters are d 100, 4, 0.7, A. 0.1, = 1, 0.9, = 0.01, a+ 0.471, a = -01888
In this section we test the validity of the prediction of our model across various networks and training protocols. We consider data distribution with parameters detailed in the caption of Fig. To] We train a 2-layer MLP with ReLU activation in the hidden layer(s) and sigmoidal activation the output with 128 neurons in the hidden layer with online SGD using BCE loss. We refer to this as the 'standard configuration' for further comparisons. We sample training and test data from the data distribution and use the test data to obtain estimates of the loss as well as error rates (percentage of test examples misclassified). We visualize these in Fig. o0 the left.
We observe the three phase behaviour predicted by our model. The positive sub-population initially advantaged more since it exhibits stronger spurious correlation. Then, the negative sub- sub-population is advantaged since it has a higher variance. Finally, as per Eq. 16 the positive- sub-population is advantaged once more since it has sufficiently high representation. We not only observe the 'double-crossing phenomena in the losses, but also in the test errors demonstrating the robustness of our model beyond the linearity and MSE loss assumptions.
We also observe similar behavior using Adam optimization in Fig. [0] (center). We note that we had to use smaller learning rate of 0.001 to keep training stable since Adam leads to faster optimization manifesting in higher effective learning rate than SGD.
When we train using weight decay penalty of 0.1 (Fig. TO right), we observe that asymp- totically the higher variance cluster is now preferred. This behaviour be explained using the theoretical model. As discussed in Appendix E smaller student magnitude advantages the higher variance group and weight decay encourages smaller student weights leading to asymptotic preference for the negative cob-population.
Depth:2 2
Depth:
Deptiic
Depth:
6.7
8.7-
1.0
4.5
2.5
1.6
10332
10000
TA
L0393
0.5
62
1.2
1Ece
4000
5000
30000
2000
4000
9292
8000
20000
2.1
2000
4000
9929
5000
30000
2000
4000
9903
sice
s00ce
Toace
fpech
Spors
Spect
Figure 11: Ablations with Deeper Networks We observe the 'double-crossing" phenomena across deeper networks as well.
We now train deeper networks with 2 to 5 layers and visualise results in Fig. 11 phenomena persists across deeper networks. We also train wider networks with 2, 16, 128 and 1024 units in the hidden layer and visualise results in Fig. 12 The 'double-crossing" phenomena persists
The 'double-crossing'
29

---

dth
8-5
53
2.5
10
$AA
13
62
2000
6190
8000
spect
Eppcs
spocs
Spect
Figure 12: Ablations with Wider Networks We observe the 'double-crossing' phenomena across wider networks well.
Learing Rate: 0.1
Leaming Rate: 0.03
Leaming Rate: 0.003
Leariin Rate: 0.003
Test Erbr
Test Errer
Test Enror
0.70
Test Ererr
0.4
0.4
6.51
0.1-
0000
Figure 13: Ablations across Learning Rates Larger learning rates can lead to instability (left). If training is stable however, we observe the 'crossing' phenomena as usual, just at different time scales due to different speeds of training.
across wider networks.
Finally, we vary the learning rate as well from our standard configuration and note that when the learning rate too high, the training unstable for the higher variance cluster (Fig. (Fig.[3]eett) For stable training. however, we observe the "double-crose couble-crossing' phenomena usual, just at longer timescales for slower learning rates as predicted by our theory.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the model is introduced in Sec. 2 analytical characterisation in provided in Sec. 3 their analysis are provided Sec. and implications are provided in Sec. The appendix complements these sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limits of the applicability of the diseussed in the assumptions of the model, Sec. 2 and additional limitations are reported the conclusions Sec. 6 Furthermore, we
comment on how these limitations may be conservative by performing numerical experi- ments in settings that violate the assumptions of the model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
31

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Proofs and derivations are reported in the appendix for space constraint in Appendix B C D.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the derivation is provided as stated in the previous questions and all the parameters to reproduce our results are reported in the text and/or the caption of the figures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
32

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code in the SM to simulate the dynamics in the TM model and reproduce the analytical solution.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guiide//ddeSabbiii httpsi//annaaeeeeedeeetiiissssaa sww.aanmmsisiooooooooooooooe for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ips.cc/ cttp:/scccpbbbccccuicceeeecc guides/ http:///ww//.......ooooo//o/aaaa
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, see Sec. 1oc.5lamt [5andd Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The variance and number of seeds is reported in the text/caption of the figures. We use shaded area around line to indicate the standard deviation (instead of error bars).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: All the simulations were performed on personal laptops. Computational resources needed to reproduce our results are quite available.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted the paper conform, in every respect, with the httpa//wwww.......a..//ccccaaaaa
NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussion reported in the conclusion Sec. This paper focuses on theoretical research, however we believe that the phenomena observed and characterised in our frame- work extend broader class of classifiers and datasets, potentially leading to a redesign of aspects of the ML pipeline in order to account for bias as dynamical quantity.
34

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: literature relevant for this work is properly cited, mostly in Sec.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
35

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "skO8rs0xhK", "reviewer": "Reviewer_JTC4", "summary": "In this paper, the authors explore how bias evolves while learning features across different sub-populations. They particularly emphasize the transient phase of learning, which is less understood compared to the early and late phases of the learning dynamics.  To this end, data from various sub-populations are modelled with a Gaussian-mixture model, and this data is labeled by a teacher network.  A linear student it trained with online SGD and its dynamics are studied at a high-dimensional limit. Using closed-form analytic expressions at this limit, the bias evolution is studied in various settings, revealing interesting dynamics that are experimentally validated on real datasets.", "questions": "a) Did the authors try any empirics with over-parameterized students over the Gaussian mixture, does it have any qualitatively different behaviour?", "limitations": "The limitations are adequately address in conclusion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a) The paper proposes a theoretical framework to study the class imbalance beyond the asymptotic regime and interesting training dynamics have been analysed. \n\nb) The description of various timescales where generalization error of sub-populations dominated and the analytical characterisation of relevant quantities controlling the timescales is novel. \n\nc) The paper is well written and the main arguments are easier to follow.", "weaknesses": "a) The paper studies only a linear student and it does not accurately capture the current over-parameterised deep learning models. This curtails the significance of the work. \n\nb) One other weakness is training with online SGD on the whole population loss in comparison of the empirical loss, hence these dynamics might fall short of capturing the real-world training setup. It is also not clear if these closed form analytic expressions are tractable beyond $m=2$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our characterisation of the non-asymptotics training dynamics and their constructive comments.\n\n\n**Regarding the Use of Linear Classifiers:** \n\n As the reviewer points out, our theoretical analysis is limited to linear classifiers to ensure analytical tractability and to obtain,  for the first time to our knowledge,  closed-form expressions for the generalisation error curves and order parameters. \n\nHowever, we have also tested the robustness of our theory on more complex architectures. In Section 5 and Appendix F, we provided experiments using 2-layer MLPs and fine-tuned a ResNet-18 on real data sets. Taking your remark and the related remark of reviewer CSav into account, we have now included additional results that demonstrate the behaviour predicted by our theory on deeper and wider MLPs (see the submitted pdf in our global response):  \n\n- We have expanded the experiment in section F.2 as follows: We consider a data distribution with $d=100$, $\\rho=0.7$, $\\Delta_+=0.1$, $\\Delta_-=1$, $T_{\\pm}=0.9$, $v=4$, $\\alpha_+ = 0.471$,  $\\alpha_- = -0.188$. We train a 2-layer MLP with ReLU activation in the hidden layer(s) and sigmoidal activation at the output with 128 neurons in the hidden layer with online SGD with a learning rate of 0.01 using BCE loss. We refer to this as the ‘standard configuration’ useful for further comparisons. We visualise the loss curves and test error rates on the positive (blue)  and negative (red) subpopulations along with standard deviation across 10 seeds in Figure 4 (left panel). \n\n- We further train MLPs of increasing depth (2,3,4,5 layers) with 128 hidden neurons in each layer and visualise the loss and error curves in Figure 6. The figures show the three phase behaviour predicted by our theory - the positive subpopulation (in blue) is learnt faster initially since it exhibits stronger spurious correlation with the shift vector; the negative subpopulation (in red) is learnt faster subsequently due to higher variance and finally the positive subpopulation is asymptotically prioritised once more due to higher relative representation. As such, we also observe the ‘double-crossing’ phenomena discussed in the paper. We stress that this result and the next are obtained by taking exactly the same parameters used in Fig.10 of the paper without exploring the phase space any further. \n\n- We also train MLPs of 2 layers but increasing width (2, 16, 128 and 1024 hidden neurons) and visualise loss and errors curves in Figure 7. We note the presence of the same three phase behaviour across increasing parameterization. Thus, our empirics with over-parameterized students show qualitatively similar behaviour. We hope this further empirical validation helps show the validity of our theory in settings closer to modern regimes.\n\n**Regarding Online SGD with Empirical Loss:**\n\n We would like to clarify that our theory indeed considers with online SGD using the empirical loss. Specifically, at each time step, we sample a new supervised example, calculate the loss for that example, and then update the model parameters using gradient descent. All our empirical experiments also follow this training procedure. \n\nWhile this approach is standard in theoretical analyses due to its analytical convenience, we acknowledge the importance of capturing the dynamics of multi-pass/batch SGD, as we discussed in the 2nd paragraph of our conclusion. In future work, we aim to address this challenge by leveraging results from dynamical mean field theory (e.g., [1]). \n\n**On tractability beyond m=2**:\n\nOur current work focuses on $m=2$ to facilitate the  interpretation of the dynamics and to identify key takeaways. However, we are confident that the equations remain analytically tractable for higher $m$ values. Adding more subpopulations would involve taking weighted averages by representation in all expectations,  similar to the existing analysis. We anticipate that this lead to a system of coupled ODEs similar to what we presented in Appendix D.1. \n\nIf there are any remaining concerns or if further clarification is needed, we would be happy to engage in additional discussion to address them.\n\n**References**: \n\n  [1]  F. Mignacco, F. Krzakala, P. Urbani, L. Zdeborová. Dynamical mean-field theory for stochastic gradient in Gaussian mixture classification. NeuriPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMRQ9Uy3o0", "reviewer": "Reviewer_gHK2", "summary": "The paper investigates the emergence of bias in the transient phase of learning by gradient descent (after initialization and before convergence). The authors theoretically study a mixture of Gaussians dataset in a teacher-student framework. The paper then analytically characterizes several properties of a linear model applied to this dataset, showing the emergence of different forms of bias at different timescales. The authors then demonstrate these theoretical predictions empirically on rotated MNIST and CIFAR10 tasks.", "questions": "What are the practical insights that can be drawn from this work? Can the authors elaborate more on what bias-mitigating mechanisms might arise from the paper's observations?", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**\nThe paper is original; although there are many prior works studying bias and fairness from a theoretical perspective, this paper studies these phenomena in the transient learning regime, which is uncommon. Moreover, the authors select a theoretical setting which can be characterized quite precisely.\n\n**Quality**\nThe quality of the theory in the paper is strong. The theorems appear correct and support the overall claims of the paper. Experiments also seem to be generally conducted well.\n\n\n**Clarity**\nThe paper is generally well-written. The authors do a particularly good job of illustrating figures, with good color choices and error bars when appropriate.\n\n\n**Significance**\nThe paper may have some value to researchers studying bias and fairness from a theoretical angle. Certainly, this paper suggests that the theoretical setting used by the authors and the analytical techniques they use are worth pursuing further.", "weaknesses": "In my view, the main weakness of the paper is that its practical value and significance remains difficult to understand. The authors show that their theoretical predictions line up with empirically observed double crossing phenomena, which is great. However, the significance of this finding is not stated. I would encourage the authors to more explicitly draw a connection between their findings and practical problems. Bias and fairness are areas with major significance in the field, so I am reasonably confident that the papers' contributions will have some practical value, but this needs to be explicitly stated and discussed.\n\nMinor comments:\n- Typo on line 314", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and careful consideration of our paper. \n\n**On the practical significance of our work**  \n\nThank you for highlighting this important point. We plan to address the practical implications of our findings by adding the following discussions to the revised manuscript: \n\n* **Caution Against Implicit Assumptions:** Our results caution against the common implicit assumption in the literature on spurious correlation mitigation that bias remains the same throughout training (e.g., [1, 2]). \n\n* **Identifying Key Drivers of Bias:** Connecting our theoretical insights to actionable practices,  we identify variance–specifically image brightness–as a key factor influencing bias in neural networks, as discussed in our CIFAR 10 experiment (section 5). This finding enriches the literature on simplicity bias by providing a direct explanation for what might make a subpopulation appear ‘easier’ for a model to learn, thereby introducing bias. \n\n* **Expanding Bias Mitigation Techniques:** Most bias mitigation strategies focus on under/oversampling to adjust the relative representation of groups. Our work suggests that e.g., by normalising image data to have similar brightness across subpopulations,  practitioners can address an additional source of bias–group variance. This expands the toolkit available for developing bias-free bias-free classifiers, offering a more nuanced approach to bias mitigation. \n\nWe hope these additions will clarify the practical significance of our work, and we are open to further discussion to ensure your concerns are fully addressed. \n\n**References**\n\n[1] E. Z Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P.  Liang, C. Finn.  Just Train Twice: Improving Group Robustness without Training Group Information. ICML 2021. \n\n[2] Y. Yang, E. Gan, G. K. Dziugaite, B. Mirzasoleiman. Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias. arXiv: 2305.19761.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Le7PQ1rsg", "reviewer": "Reviewer_CSav", "summary": "The paper analyzes how different subgroups of the data affect optimization dynamics. The analysis is done in a large-dimension limit by analyzing squared error of a linear student classifier modeling a piece-wise linear teacher one. This analysis provides insights into dynamic loss behavior of loss on different subgroups and which aspects of the data lead cause this.", "questions": "Is there a way you can modify the data distribution in the real experiments according to what your theory suggests lead to certain behavior, and show that the behavior is now different? This would seal the theory and main reason behind the seen curves.\n\notherwise, See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper provides a different analysis of SGD-optimization.\n- predicts a double-cross phenomenon where different stages of training \n- shows evidence of those double-cross phenomenon in real experiments.", "weaknesses": "- the student is underpowered in this setup. The teacher is piece-wise linear but the student is purely linear. This fundamentally imposes a trade-off between modeling one subset of the data better than another. This breaks out of the case where models we use these days can perfectly fit any training dataset even with medium-sized datasets like CelebA.\n\nSome missing related work: https://proceedings.mlr.press/v162/pezeshki22a.html, https://arxiv.org/abs/2308.12553\n\n- This is important: What were the hyperparameters used in the various experiments? What happens as you vary each of them (like learning rate and weight decay)? It seems to me that the synthetic model cannot account for any of these parameters, so it feels like there is a gap in what the theory actually explains (especially with the underpowered student).\n\n- (Minor) Adam is the more common optimizer and it is unclear whether the theory in the paper extends to this and if the insights about features translate. Can the authors comment on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our analysis and their comments. We feel that these are important comments to improve the clarity of our work and we thank the reviewer for pointing them out, we ran additional experiments (discussed below) to address them.\n\n**Limitations of the model** \n\nOur theoretical analysis is limited to purely linear classifiers to maintain analytical tractability. To address questions about the robustness of the results and whether this may be due to the low capacity of the classifier, we relied on numerical evaluation on deeper networks. We provided some experiments in section 5 and appendix F where we used 2 layer MLPs and finetuned a ResNet-18 on real data sets. \n\nTaking your remark and the related remark of reviewer JTC4 into account, we have now included additional results that demonstrate the behaviour predicted by our theory on deeper and wider MLPs (see the submitted pdf in our global response):  \n\n- We have expanded the experiment in section F.2 as follows: We consider a data distribution with $d=100$, $\\rho=0.7$, $\\Delta_+=0.1$, $\\Delta_-=1$, $T_{\\pm}=0.9$, $v=4$, $\\alpha_+ = 0.471$,  $\\alpha_- = -0.188$. We train a 2-layer MLP with ReLU activation in the hidden layer(s) and sigmoidal activation at the output with 128 neurons in the hidden layer with online SGD with a learning rate of 0.01 using BCE loss. We refer to this as the ‘standard configuration’ useful for further comparisons. We visualise the loss curves and test error rates on the positive (blue)  and negative (red) subpopulations along with standard deviation across 10 seeds in Figure 4 (left panel). \n\n- We further train MLPs of increasing depth (2,3,4,5 layers) with 128 hidden neurons in each layer and visualise the loss and error curves in Figure 6. The figures show the three phase behaviour predicted by our theory - the positive subpopulation (in blue) is learnt faster initially since it exhibits stronger spurious correlation with the shift vector; the negative subpopulation (in red) is learnt faster subsequently due to higher variance and finally the positive subpopulation is asymptotically prioritised once more due to higher relative representation. As such, we also observe the ‘double-crossing’ phenomena discussed in the paper. We stress that this result and the next are obtained by taking exactly the same parameters used in Fig.10 of the paper without exploring the phase space any further. \n\n- We also train MLPs of 2 layers but increasing width (2, 16, 128 and 1024 hidden neurons) and visualise loss and errors curves in Figure 7. We note the presence of the same three phase behaviour across increasing parameterization. Thus, our empirics with over-parameterized students show qualitatively similar behaviour. We hope this further empirical validation helps show the validity of our theory in settings closer to modern regimes.\n\n**Hyperparameters used in experiments** \n\nApologise for not including the learning rate in the empirical experiments and will do so in the camera ready. Note that our synthetic model does however explicitly consider the effect of the learning rate. As shown in Fig.3b, discussed in sec. 4.2 and expanded upon in Appendix E.1 of the paper, the learning rate can act as a source of bias by advantaging the cluster with smaller variance. \n\nFollowing your suggestion, we tested the ‘standard configuration’ with varying *learning rate* (Figure 5 of the included PDF). As predicted by the theory (see sec. 4.2 and Figure 3b of the paper), for high learning rates, the loss of the higher variance group blows up (left panel of Figure 5 of the included PDF). For smaller learning rates, the dynamics remain the same, it is simply the overall speed that changes as we can see in the included figure (for smaller learning rates, it takes longer for the crossings to occur).\n\n*Weight decay* is quite an interesting technique to consider and our analysis offers direct predictions based on the theory. As discussed in Appendix E, smaller student magnitude advantages the higher variance group and indeed when we empirically include a weight decay penalty of 0.1 (Figure 4 right panel of the included PDF), we observe that asymptotically the higher variance cluster is now preferred since the penalty encourages smaller weights.\n\nAs for *Adam*, we see very similar behaviour (Figure 4 centre panel of the included PDF) although we had to use a slower learning rate of 0.001 to prevent losses from blowing up since Adam leads to faster optimization manifesting in a higher effective learning rate than SGD. We fully expect all our key findings to translate. \n\n**Suggested additional experiment** \n\nThis is a great suggestion. We provide results (see submitted PDF) from additional experiments to show how modifying the real data changes behaviour exactly as predicted by the theory. Figure 1 shows an extension of the CIFAR10 experiment discussed in the paper where we now vary the relative representation of the groups (darker colours reflect greater imbalance) and we see that as predicted by our theory, crossings only occur when the higher variance group (red) has lower relative representation. Notice that the dynamics of the initial phase seem almost unaffected by the relative representation and are determined only by the variance as predicted by the theory.\n\nWe also extend the MNIST experiment discussed in the paper and show how increasing variance of the second group (red), controlled through image brightness, leads to faster learning in the second phase in Figure 3. \n\n\n**In conclusion**, we hope these extensions provide strong evidence showing how the factors identified in the theory affect real data experiments exactly as predicted.\n\nWe also thank the reviewer for pointing out additional references that we will include in the further work.\nIf there are any remaining concerns or if further clarification is needed, we would be happy to engage in additional discussion to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnfcDr80R2", "reviewer": "Reviewer_a8nr", "summary": "The manuscript tries to understand the training dynamics in SGD training, through the lens of synthetic data with sub-populations from the Gaussian-mixture model. The paper leverages high-dimensional analysis to prove that the solution converges to a set of ODEs and thus characterizes the evolution of bias throughout training, namely a three-phase learning process. The study provides a unifying view of learning in fairness and spurious correlation problems, highlighting the presence of ephemeral biases characterized by multiple timescales during training. These theoretical insights are further validated on practical datasets, like CIFAR, MNIST, and CelebA.\n\nIn general, the take-away messages of the manuscript look interesting.", "questions": "* How does the value of $m$ affect Theorem 3.4 and the insights discussed in Section 4?", "limitations": "NA", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The manuscript is well-structured, with a clear logical flow.\n* The idea of using a theoretical model, namely a linear model with a mixture of Gaussian distribution, followed by using order parameters to depict the generalization error, is quite interesting.\n* The manuscript leverages the theory model to interpret the training dynamics when spurious correlations and bias are present during the training.", "weaknesses": "The reviewer cannot spot significant weaknesses in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their general appreciation of our work and for their comment.\n\n**Impact of the value of $m$**  \n\n\nThank you for your insightful question. We plan to incorporate the following points into the revised manuscript: \n\n* A higher value of $m$ would introduce additional order parameters for the student-teacher overlaps and student-shift overlaps corresponding to the extra clusters. Despite this added complexity,  we anticipate a similar system of coupled ODEs to what is described in Appendix D.1, and we are confident that the results would remain analytically tractable.\n\n* We expect that the form of the equations in Theorem 3.4 would remain fundamentally unchanged, with the same type of timescales and dynamics discussed in the paper. The constants, such as $\\Delta^{mix}$,  would be adjusted to account for the weighted average across a greater number of clusters. Consequently, the same factors to drive bias –degree of spurious correlation, variance, relative representation– and the three-phase behaviour described in the paper would still emerge.  We are optimistic that the key takeaways from Section 4 will extend to the $m>2$ setting.\n\n* While we believe most conclusions will hold for higher values or $m$, , this is an interesting direction we are currently investigating. In this work, we focused on  $m=2$,  which already shows a rich and sometimes counter-intuitive phenomenology.    However, the $m>2$ scenario introduces additional complexities, such as  varying levels of similarity between groups, which merit further exploration. \n\nWe would welcome any further suggestions or discussion to address this concern more comprehensively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAhAByetND", "reviewer": "Reviewer_43yt", "summary": "This paper presents a theoretical analysis of bias dynamics in machine learning models during the training process.\n\n1. The authors introduce a high-dimensional teacher-student framework called the Teacher-Mixture (TM) model to study how bias emerges and evolves during stochastic gradient descent (SGD) training of linear classifiers.\n\n2. They provide an analytical solution for the dynamics of key order parameters that characterize the classifier's performance and bias in the high-dimensional limit.\n\n3. The analysis reveals a rich three-phase learning process where the classifier exhibits biases toward different data features at different stages of training:\n   - Initial phase: Influenced by sub-populations with strong class imbalance\n   - Intermediate phase: Dominated by the saliency (norm) of samples in sub-populations\n   - Final phase: Determined by sub-population imbalance (relative representation)\n\n4. The authors show that bias can evolve non-monotonically during training, potentially leading to \"crossing\" phenomena where the preferred sub-population changes over time.\n\n5. They validate their theoretical findings through numerical experiments on both synthetic data and real datasets (CIFAR10, MNIST, CelebA), demonstrating that the observed phenomena extend beyond the simplifying assumptions of their model.\n\n6. The paper discusses implications for fairness and robustness in machine learning, highlighting how different properties of sub-populations can generate and amplify bias at different timescales.\n\n7. The authors argue that understanding these transient dynamics is crucial for developing effective bias mitigation strategies, especially under limited computational resources.", "questions": "Can you link training on MNIST, etc closer to the theory by training again on linear models. Can you make the jump between theoretical and experimental section less severe by e.g. studying linear models trained on different (more difficult and not theoretically motivated) data generation processes? Can you provide examples for linear models where we do not observe crossing phenomena to build up more intuition?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "this work provides a theoretical foundation for understanding the complex time-dependent nature of bias in machine learning models during training, which was previously, as claimed by the authors (I can only judge this superficially) not well captured by existing theoretical studies focused on asymptotic analysis.​​​​​​​​​​​​​​​​ \nAs far as I can judge the paper is well written and the experiments, results well described.", "weaknesses": "I can not really judge the novelty and soundness of the proposed work but I believe the paper is of high quality. Of course the setting considered is far from „large scale“ but I dont see this as too important at this stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time reading and assessing our paper. We are happy that the reviewer found our paper well-written and, in the following, we would like to address their main comments touching an important point, i.e. the robustness of our result in more complex settings.\n\nWe extended the experiment on MNIST by now training on a network with no ReLU non-linearities –i.e. linear activations– and visualise the result in Figure 2 of the rebuttal PDF. Compared with Figure 5(a) of the paper, we note that removing the non-linearities leads to similar qualitative behaviour and we still observe a crossing. Giving another confirmation of our theory. \n\nThe reviewer also asked about situations where the crossing does not occur. We can find an instance in Fig.1a of the main paper and, more broadly, looking at the phase diagram of Fig.3b we can notice that all the dark red and dark blue regions of the generative model’s parameters are instances where the crossing does not occur. Our theory shows that this is because the preferred subpopulation of our classifier stays the same across the different phases of dynamics described in the paper.\n\nConnected to this point, we would like to refer to answer to reviewer CSav where we also see that the crossing is not necessarily happening –Fig.1 and Fig.3 of the rebuttal PDF– and it occurs when the data distributions of the groups show characteristics identified by our theory –that depends on the variance and relative representation of the groups as explained in section 4.2 of the paper. We kindly ask the reviewer to refer to CSav rebuttal for further details.  \n\nWe hope we were able to address your concern and once again thank you for your valuable time and remarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments and careful consideration of our paper. Their feedback  has helped us identify ways to improve our presentation and convey our message more clearly. It has also prompted us to conduct additional experiments,  which we hope demonstrate the robustness of our theory and provide clearer explanations of our results. 

We have included a supplementary PDF with additional figures and have explained in detail their relevance in individual responses to the reviewers. However, for the benefit of everyone’s understanding,  we would like to briefly highlight some of the key experiments here.

* **Figure 1**: This figure presents an extended version of the CIFAR10 experiment,  where we systematically vary the relative representation of the groups (darker colours represent greater imbalance against the red group). As predicted by our theory, relative representation primarily affects the later dynamics, while the middle dynamics are governed only by group variance.

* **Figure 2**: We show the learning dynamics on MNIST of a simplified architecture with respect to the one considered in section 5 of the paper where the hidden layer doesn't have non-linearities. The result shows consistent behaviour with Figure 5 of the main text.

* **Figure 3**: Here, we extend the MNIST experiment by varying the average image brightness of the second group, which reflects the group variance in our theory. The results show that greater brightness leads to faster learning in the second phase and an increasing asymptotic preference, consistent with our predictions.

* **Figures 4, 6, 7**: In these figures, we extend the experiment in Appendix F.2,  where an MLP with non-linearities is trained on synthetic data  with BCE loss. Across different optimizers and varying degrees of parameterization, we observe a three-phase behaviour:  the positive subpopulation (in blue) is learned faster initially due to a stronger spurious correlation with the shift vector; the negative subpopulation (in red) is learned faster subsequently due to higher variance,  and finally, the positive subpopulation is asymptotically prioritised once more due to higher relative representation. 

We believe these additional results strengthen the contributions of our paper and we hope they address the reviewers’ concerns. We welcome any additional questions or remarks and sincerely appreciate the reviewers’ time and effort throughout this review process.  

The authors

Meta-review: The paper analyzes the training dynamics (with ODEs) of a linear model trained with SGD under an assumed Gaussian mixture data distribution. Notably the paper makes precise how the optimization dynamics affect data subgroups in this specific setup. Among other settings, the presented theoretical results are particularly relevant for learning under fairness considerations, and learning in the presence of spurious features. While other existing works tend to rely on asymptotic analysis, the present work’s goal is to understand  the complex time-dependent nature of bias during training. 

**Review**

All reviewers' comments were positive on the strength of the theoretical analysis in the specific setting considered. Notably, the closed-form time-dependent characterization of various quantities (JTC4), and the prediction of the double-cross phenomenon (CSav) are novel. Due to the assumed Gaussian mixture model, the issues of practical value and significance came up during the discussion. The authors sufficiently addressed these points by providing more experiments on more datasets and network architectures. Another issue raised was that the paper only considers a linear model. On this point, the AC thinks that the value of the contribution is in its full characterization (closed-form solution) of training dynamics under a distribution-model setup, even though that setup may be quite specific. The AC thinks that characterizing training dynamics in the more general case is not easy, and that this contribution can provide a stepping stone for a more complicated data distribution model that is yet to come in future works.

Recommendation: accept.
Decision: Accept (poster)