On The Impact of Machine Learning Randomness on Group Fairness
Authors: Prakhar Ganesh
Hongyan Chang
Martin Strobel
Reza Shokri
Venue: Submitted to ICLR 2023 (2023)

Abstract: Statistical measures for group fairness in machine learning reflect the gap in performance of algorithms across different groups. These measures, however, exhibit a high variance, between different training instances, that makes them unreliable for empirical evaluation of fairness. What is the cause of this variance, and how can we reduce it? We investigate the impact of different sources of randomness in machine learning on group fairness. We show that the variance in group fairness measures is mainly due to the high volatility of the learning process on under-represented groups, which itself is largely caused by the stochasticity of data order during training. Based on these findings, we show how to manipulate group level accuracy (i.e. model fairness), with high efficiency and negligible impact on the overall predictive power of the model, by changing the data order.

Under review as a conference paper at ICLR 2023
ON THE IMPACT OF MACHINE LEARNING RANDOM- NESS ON GROUP FAIRNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Statistical measures for group faimess in machine learning reflect the gap in performance of algorithms across different groups. These measures, however, exhibit a high variance, between different training instances, that makes them unreliable for empirical evaluation of faimess. What is the cause of this variance, and how can we reduce it? We investigate the impact of different sources of randomness in machine learning group fairness. We show that the variance in group fairness measures mainly due to the high volatility of the learning process under-tepresented groups, which itself is largely caused by the stochasticity of data order during training. Based on these findings, we show how to manipulate group level accuracy (i.e. model faimesss), with high efficiency and negligible impact on the overall predictive power of the model, by changing the data order.
INTRODUCTION
Machine learning models are shown manifest and escalate historical prejudices and biases present
in their training data (Crawford 2013 Barocas & Selbst [2066] Zhao et al. 2017 Abbasi et al. 2019).
Understanding these biases and the following ethical considerations has led to the rise of fair machine
learning research (Chouldechova & Roth 2018 Caton & Haas 2020 Mehrabi et al., 2021). Recent
work in fair deep learning have observed a trend of high variance in fairness measures across multiple training runs (Qian et al. 1021 Amir et al. 2021 Sellam et al. 202 I Soares et al. 2022). usually
attributed to non-determinism training. These results have challenged the legitimacy of existing claims in the literature (Soares al. (0022,, and have even disputed the effectiveness of various bias mitigation techniques (Amir et al. /021//eellam et al. 202T). Thus, reliable extraction of fairness trends in model requires accounting for the high variance avoid lottery winners (see Figure
The naive solution of executing a large number of identical training runs to capture the overall variance creates huge computational demand, and discourages the examination of biases in several rapidly growing forefronts of machine learning research by increasing the resource requirements. Therefore, understanding the actual cause of the high variance in the faimess measures is critical. To the best of our knowledge, we are the first to study fairness variance beyond trivially executing multiple identical training runs. More specifically, we show the following:
We show that the trends of fairness variance observed in literature are dominated by random data reshuffling during training, which causes high fairness variance between epochs even within single training run, while the non-determinism in weight initialization has minimal influence.
We extract an empirical relationship between group representation and instability in group level performance, highlighting higher vulnerability of minority to changing model behavior. Our results attribute the high fairness variance to lower prediction stability for under-representeed subgroups.
We demonstrate immediate dominance of the data order model fairness. A model's fairness is predictable, based on only the most recent training points, irrespective of preceding model behavior.
Based on this information, we propose use the faimess variance across epochs proxy to study the changing model fairness across multiple training runs, thus reducing the computational requirements by significant margin.
Finally, manipulate group level performances (i.e., model fairness) by changing the data order, with a relatively minor impact on the overall accuracy. This manipulation can improve fairness as well reverse the effects of several bias mitigation algorithms within single training epoch.

---

Under review as a conference paper at ICLR 2023
(a) Fairness
(b) Performance
Baseline
Rewcigh
B0 Loss
FairBatch
0%
2%
4%
6%
8%
10%
78%
80%
82%
Average Odds
F1-Score
Figure 1: (a) Fairness (average odds) has a high variance across identical runs due to non-determinism in training. This variance persists even with several state-of-the-ar bias mitigation algorithms (Reweighing (Kamiran & Calders 2012); Equalized Odds Loss (Fukuchi al. 2020); FairBatch (Roh) et al. ).20)), and reliable comparison between these methods can only be made after capturing the overall model behavior, due to intersecting ranges. (b) The performance score (F1 score), however, has significantly smaller range of variance and doesn't 't face similar issues as fairness.
EXPERIMENTAL. SETUP
Datasets and Models We will conduct our investigation on ACSIncome and ACSEmployment tasks of the Folktables dataset (Ding et al.) (021)) and binary classification of the 'smiling' label in CelebA dataset (Liu et al.) 2013), with gender (Female vs. Male) as the sensitive attribute for all datasets. For CelebA, imput features obtained by passing the input image through pre-trained and frozen ResNet-50 backbone and extracting the output feature vector after the final average pooling layer. More details on the datasets are provided in Appendix A
We train a feed forward network on input features with a single hidden layer consisting of 64 neurons and ReLU activation, and train the model with cross-entropy (CE) loss objective for total of 300 epochs at batch size 128, in all our experiments unless specified otherwise. We include additional experiments by changing training hyperparameters, i.c., batch size, learning rate, and model architecture, in Appendix F.4 We use train-val-test split of 0.7 0.1 0.2, and maintain the same split throughout all our experiments, i.e. we do not consider potential non-determinism introduced due to changing train-val-test splits. All our evaluations are performed on the test split. We will focus primarily on the ACSIncome dataset in the main text, while additional experiments on CelebA and ACSEmployment are included in the appendix.
Fairness Metrics and Variance Fairness in machine learning has been interpreted widely into a multitude of definitions. In this paper, we will focus average odds (AO), empirically interpreted as the average disparity between true positive rates and false positive rates of various groups. We also include additional results for equalized opportunity (EOpp) and disparate impact (DI) Appendix For model predictions Rt true labels Y, and sensitive attributes A, average odds can be defined
F.3
as,
1 Σ
AverageOdds
P y, = 0) P(R 1 = y, = 1)}.
(1)
2 y=(0,1)
At the heart of our work is the study of fairness variance across model checkpoints. Unless otherwise specified, variance across multiple training runs refers the variance across final checkpoints at epoch 300 of each training run. Similarly, variance across epochs refers to variance in single training run across checkpoints at the end of every epoch for the last 200 epochs of training (i.e., from epoch 100 to epoch 300). We make this choice as the model has converged to stable accuracy scores before epoch 100 (refer to the training curve in Appendix A for more details).
Non- Non-Determmnism in Model Training In this work, we focus randomness due to stochasticity in the training algorithm, and we set manual seeds at various intermediate locations our code to control the randomness. We refer to the seed set right before building the neural model as the weight
2

---

Under review as a conference paper at ICLR 2023
initialization seed, which influences the randomness in sampling the weight values. Similarly, refer to the seed set right before the first training data shuffling as random reshuffling seed, which influences the data order that will be used as reference for the rest of the training. During training, simply set the epoch number as seed right before reshuffling the reference data order at every epoch. As change the reference data order by changing the random reshufiling seed, this setup allows us to control non-determinism in data order throughout training with a single random seed. Fixing both weight initialization and random reshuffling seed allows us deterministically replicate model training, while changing both seeds simultaneously is similar to the discussion of non-determinism present in literature. Pseudo code detailing the definition of various random seeds is provided in Appendix A We also perform additional experiments on the impact of randomness introduced by dropout regularization in Appendix A
THE SOURCES OF RANDOMNESS
We start by moving past the trivial observation that different training runs lead to different outcomes, and investigate the high fairness variance by studying the intricacies of the training dynamics.
3.1 WRIGHT INITIALIZATION AND RANDOM RESHUFFLING
Neural model training under widely adapted mini-bate stochastic gradient descent (SGD) has two main sources of non-determinism, i.e. weight initialization and random reshuffling. To decouple their impact on fairness variance, we perform the following experiment. We execute three sets of 50 unique training runs each while (i) allowing for both sources of randomness, (ii) changing only the weight initialization while keeping the random reshufiling fixed, and (iii) changing only the random reshuffling while keeping the weight initialization fixed. For ACSIncome, we collect average odds (AO) and F1 score of all training runs in Figure 2
(see Figure 13 for the other datasets).
(a) Normal training
(b) Fixed Reshuffling
(c) Fixed Weight Init
79.8%
Fl Score
79.6%
79.4%
79.2%
79%
6%
7%
8% 9%
6% 7% 8% 9%
6% 7% 8% 9%
Average Odds
Average Odds
Average Odds
Figure 2: Average Odds (AO) variance by (a) allowing for both sources of randomness simultaneously. represents the instability in fairness scores as noted by existing literature. We see a significant drop in variance if we change only (b) the weight initialization while keeping the random reshuffling fixed. However, we observe high range of fairness variance by changing only (c) the random reshuffling, even for fixed weight initialization. These results suggest random reshuffling of the data order as the dominant source of variance in fairness, with little influence from weight initialization.
The large range of fairness scores reported by allowing for both sources of randomness represents the variance observed in existing literature. Interestingly, when these sources are examined separately, the variance due random reshuffling is equivalently large but the variance due weight initialization drops significantly. It is clear that fairness variance originates from data order shuffling, while randomness in weight initialization has minimal impact. To further probe the difference between these two sources of non-determinism, of average odds (AO) across the complete set of 50 training runs for the last 200 epochs in the two isolated settings from the previous experiment, i.e., changing only the weight initialization and only the random reshuffling, respectively. The resulting plots are collected in Figure
plot the median, inter-quartile range and overall range
We find high correlation in fairness scores across training runs with changing weight initialization (average pairwise pearson coefficient zs 0.91), despite high variance between epochs as observed in Figure a). This behavior supports our observations of low variance in fairness scores at the final
3

---

Under review as a conference paper at ICLR 2023
(a) Fixed Data Reshuffling
(b) Fixed Weight Initialization
10%
Average Odds
8%
6%
100
150
200
250
300
100
150
200
250
300
Epoch
Epoch
Median
Inter-quartile range
Overall range
Figure 3: Median, inter-quartile range, and overall range of average odds across different training runs while changing only the weight initialization or the random reshuffling respectively. (a) Changing only the weight initialization while keeping random reshuffling fixed implies that for any particular epoch, all training runs are fed the same data order. Despite different initializations, these models have very little variance across training runs, but high variance across epochs. This highlights the dominant impact of random reshuffling model fairness. (b) Changing only the random reshuffling further supports our claim, as we see high variance even across training runs.
epoch in 2 Furthermore, there is a lack of any reasonable correlation between fairness scores of training runs with changing random reshuffling (average pairwise pearson coefficient > 0.13), with noticeably high variance across both changing epochs and training runs in Figure 3(b). Interestingly,
the high faimess variance across epochs inside single training run in Figure 3 closely matches the variance that we observe across multiple training runs Figure 3(b).
3.2 CHANGING PREDICTIONS AND DATA DISTRIBUTION
In previous experiments, observed high variance in model faimess, and thus in turn it's predictive behavior, even between consecutive epochs. The changing predictive behavior of neural models is surprising, and has been studied extensively in literature (Toneva et al. 2018 Kirkpatrick et al.) 2017: Jagielski et al. 2022] Tirumala er al. 2022). As we are concerned with the fairness of the final decisions made by the model, we focus on change in the model's discrete output class when discussing changing predictions. More specifically, a model is said to have undergone a change in prediction for some input x during epoch i, if Úe (x) o Ú (I), where (x()) is the output class when passing the input x: through the model checkpoint at the end of epoch i.
(a) Data Distribution
(b) Unique Data Points that Changed Prediction
40%
15%
Group Percentage
30%
10%
20%
5%
10%
0%
1,0
150
200
250
300
Epoch
Female (Salary 2 50A) Male (Salary 2 50K) Mule (Sulary < 50K) Female (Salary < 50K)
Figure 4: (a) The underlying data distribution of ACSIncome shows positive labels from group Female as an under-represented minority. (b) Total percentage of unique data points from each subgroup that change prediction across epochs follow the opposite order of their representation in the training data. These results highlight subgroups with least representation, i.c. positive labels from group Female in this setting, being the most vulnerable changing predictions.
4

---

Under review as a conference paper at ICLR 2023
While these changing predictions maintain overall stable average performance, they still have disparate impact on individual groups, the exact characteristics of which are less known. We study this instability by investigating individual data points which change their predictions. We plot the percentage of data points from each group that changed their prediction at least once between epochs 100 and k; where gradually increase the value of k. The results are collected in Figure
Clearly, the trends in percentage of unique data points with changing predictions mimics the distribu- tion of each group in the original dataset, i.e. the groups which are represented the least are the most vulnerable to changing model behavior. In ACSIncome dataset, positive labels from group Female severely under represented, and consequently can be seen to have almost twice the percentage of unique examples with changing predictions than any other group in the dataset. The higher instability of minority an indication of higher uncertainty
the underlying model predictions. Thus, any fairness metric defined on the output of such a model will also reflect this instability, and manifests as variance in existing literature (Amir al. 2021
Sellam et 2021
Soares al. 2022).
DATA ORDER AND FAIRNESS
The previous set of experiments suggest the existence of an intricate relationship between the data order and resulting model fairness. In this section, we dig deeper into this relationship and exploit it to approximate fairness variance without separately training multiple models.
4.1 IMMEDIATE IMPACT OF DATA ORDER
A single epoch of SGD comprises of learning sequence of mini-batcher from the training data, changing the model parameters one gradient update at time. The data order during training governs the order of gradient updates and thus its impact resulting model behavior is quite obvious (Bengio et al. (009). However, our investigation into fairness variance suggests that this impact might be immediate than expected, capable of stably predicting faimess scores irrespective of the model's preceding behavior. To test this, we design an experiment as follows. We sample 1000 unique checkpoints randomly from last 200 epochs of 50 different training runs while allowing both forms of training non-determinism simultaneously. These checkpoints represent variety of models with different weight initialization, training data order and even number of training epochs. We then train each of these checkpoints for exactly one epoch on common randomly chosen data order. We collect fairness variance across checkpoints before and after this single epoch of training in Figure 5
ACSIncome
CelebA
Average Odds
10%
10%
8%
8%
6%
4%
6%
2%
Before
After
Before
After
Figure 5: (Before) Multiple checkpoints taken from training runs with changing weight initialization, training data order, and even number of training epochs, show high range of fairness variance (i.e., average odds variance), as expected. (After) However, these fairness scores are stabilized significantly by training these checkpoints for only single epoch common randomly chosen data order. This shows the immediate impact of data order on model fairness.
Before training on the same data order for a single epoch, these checkpoints represent the complete range of faimess variance previously noted. However, after only single epoch of training on common data order, these models have all moved towards the same fairness score with significantly lower variance. This highlights the immediate impact of data order of model faimess, which is stable based on only the data order of the most recent epoch. An existing line of research suggests the presence of no energy valleys in deep learning loss landscape between minimas of separately trained models (Draxler et al. 2018 Garipov et al. 2018)). which could explain the immediate impact of data order, various checkpoints might be connected in the loss landscape. Another possible explanation
5

---

Under review as a conference paper at ICLR 2023
of models converging to the same faimess scores relies recent work that shows that there is only one functionally unique minima in loss landscape of neural models (Ainsworth et al. 2022). Thus, is possible that different models when fed the same data order for even one epoch move towards functionally common model, with possible permutation symmetries in them.
Epochs are pseudo constructs, used to record multiple passes through the data, and thus expect the influence of data order on model faimesss to have a granular manifestation as the few most recent gradient updates of the model. Furthermore, while we have shown that the fairness variance is indeed reduced, we are yet to show that it is also predictable. To achieve both simultaneously. we extend the experiment above as follows. We start by extracting the data order corresponding the epochs achieving best and worst faimess scores on the validation set in standard training run. Next, we train these checkpoints on exactly batches, taken from the suffix of data order with best and worst fairness scores as described above. This allows us to obtain models which have experienced the same most recent b gradient updates. For example, at O, this setup involves no training and we simply report the variance across the original set of checkpoints, while at b 1070 (i.e., total number of batches in an epoch for ACSIncome), this setup will repeat the previous experiment. We test for b {0, 20, 50, 100, 200} and the results are collected in Figure 6
Best Fairness Order
Worst Fairness Order
Average Odds
10%
10%
8%
8%
6%
6%
0
100
200
20
50
200
Number of Training Batches
Number of Training Batches
Figure 6: Fairness scores (average odds) stabilize as the number of most recent common training batches increase, highlighting the impact of these gradient updates on model faimess. Moreover, stabilizes low high fairness scores based on the corresponding data order from the epoch with best or worst fairness score, showing the predictability of model faireess for any given data order.
As the number of fixed batches b increases, the model fairness gets more stable, and already at b = 200, the model fairness is as stable as seen in previous set of experiments. This shows that is indeed the most recent gradient updates seen by the model which dominate its faimess scores, irrespective of model behavior prior to those updates. Moreover, the trends of stability move towards expected fairness based on the choice of original data order taken from best worst faimess epoch as described earlier, which helps us show the predictability of the impact of data order.
4.2 CAPTURING FAIRNESS VARIANCE IN A SINGLE TRAINING RUN
We now return to original problem of capturing fairness variance without wasting computing resources on a large number of training runs. We saw stability and predictability in fairness scores based on the data order, and thus fairness variance across multiple training runs is simply the randomness in data order across their last epochs. As these data orders are obtained by random shuffling, their distribution across multiple training runs should result in the same empirical range of faimess as their distribution across epochs. Thus, we propose evaluating intermediate checkpoints in a single training run as a proxy for reporting fairness variance across multiple training runs.
We test this by first plotting the distribution of fairness scores for sampling across multiple training runs (three different stopping epochs) and sampling across epochs (three different training runs) in Figure Furthermore, we also test the quality of black swans, i.e. the best models captured, as function of number unique training runs and number of epochs evaluated per training run. For a data point epochs e; seeds s, we perform unique training runs, and evaluate the model for last epochs in each training run, thus accumulating a total of e s checkpoints. These checkpoints are then used to calculate some quality measure, and averaged over 50 repeats to compensate for randomness in choosing training runs. The experiment is performed for all combinations of E [1, 50) and E (0.. We use two different quality measures for of checkpoints described above, (i) the

---

Under review as a conference paper at ICLR 2023
best fairness achieved across all checkpoints, and (ii) the Hausdorff distance (Birsan & Tiba 2005 of the pareto-front (including both fairness and F1 scores) from the best achievable pareto-front, i.c., for 50; 50. The results collected in Figure 7]b).
(a) Fairness Score Distribution
(b) Black Swans
Avera Ocds
7.0
Aruogg 0od6
20 Epachs
40
3.0
Ho Distancee
30
2.5
3 20
2.0
Aeres Seeds
Acress beeeh
Accust Serdh
ACTeSs SeocTs
PCUSE CEECES
AGE BEROTS
20
40
Sempaing
Epochs
Figure 7: (a) Fairness scores (average odds) across multiple training runs and across epochs in single training run have similar empirical distributions. Thus, studying this distribution across epochs provides highly efficient alternative to executing multiple training runs. (b) Quality of black swans (i.e. extremely rare checkpoints) improves with more checkpoints collected, either in terms of fairess (the lowest achievable average odds score) or the trade-off between overall performance and fairness (the hausdorff distance to the best pareto front). This improvement occurs at the same rate, irrespective of sampling checkpoints across multiple random seeds (x-axis) or multiple epochs in single seed (y-axis). But, sampling across epochs is significantly cheaper.
As we can notice from the plot in Figure 7 a),, the empirical distribution of fairness across multiple training runs closely matches the distribution across single training run. We even performed the Kolmogorov-Smiing Serinooo-Smmmoov (KS) test (Massey Jr [95T] to match sampling across multiple runs (at epoch 300) and sampling across epochs (for single training difference in empirical CDF to be only 0.07, with a p-value of 0.712, i.e. the probability that both set of checkpoints black swans in Figure (b) for both quality measures suggests similar trends. For example, when comparing the two extremes, i.e. e 1; 50 and e 50; requiring 50 times less computation. This shows that the commonly used method to capture fairness variance in literature (i.e., e 1; 50) highly inefficient use of computing resources, and one can achieve the same by simply observing model behavior across a single training run. With these experiments, we show direct benefits of evaluating multiple epochs in a single training run, saving huge amounts of resources and time in capturing the overall model variance.
with seed 0) and found the maximum
sampled from the same underlying distribution. Moreover, the quality of
see similar results, with latter
MANIPULATING GROUP LEVEL ACCURACY WITH DATA ORDER
In previous sections, we explored the instability in model fairness and took advantage of our insights to reduce computational requirements for studying this variance. However, the predictability of fairness scores based on the most recent batches implies the possibility that we should be able to finely manipulate the fairness scores, or more precisely the group level accuracy, by simply controlling the data order. We hypothesize that the data distribution seen by the model the most recent batches temporarily change the loss landscape and nudge the group level accuracy behavior towards the chosen distribution. Coupled with the immediate impact data order shown in previous sections, this could allow to manipulate group level accuracy in only single epoch of fine-tuning.
To test this hypothesis, we perform the following experiment. We start by defining the ratio between different groups, to indicate the distribution that will dominate the model behavior. For example, ratio of 10 between two groups will distinctly favor the latter group. while a ratio of 10 will favor the first group. We then form batches in the exact ratio provided until run of data
7

---

Under review as a conference paper at ICLR 2023
points (which will happen for any ratio that is not the exact distribution of the original dataset). The excess data points are then shuffled randomly and placed at the prefix of data order.
Similar to previous experiment settings, we pick the last epoch checkpoint from 50 unique training runs while allowing for both forms of non-determinism (i.c. weight initialization and random reshuffling), and then train them all for single epoch on our custom data order. For our first experiment, we fix all other ratios and only change the ratio between positive and negative labels in group Female. We record the group level accuracy for all subgroups separately, along with the overall model accuracy. We repeat the experiment by now changing only the ratio between positive and negative labels in group Male. The results are collected in Figure 8
100%
100%
80%
80%
Accuracy
Accuracy
8
60%
60%
40%
40%
20%
- F-Pus - ENg
M.Pos
20%
F-Pus - F-Ner
M-Nog c... Overall
M.Nay G Overall
0%
0%
10-1
100
10a
10
100
10
Ratio of F-Pos F-Neg
Ratio M-Pos: : M-Neg
Figure 8: We show how to control group level accuracy by changing data distribution of the most recent gradient updates, tested separately for (a) ratio between positive and negative labels for group Female, and (b) ratio between positive and negative labels for group Male, while keeping other ratios fixed. In only single epoch of training, we able manipulate the group level accuracy trade-off, with relatively small impact on overall accuracy.
It is clear that by manipulating the data distribution seen by the model in the most recent gradient updates, we can control the group level accuracy of the model. While the overall accuracy drops noticeably for extreme ratios, it does not change much in the middle despite significant variance in group level performances. This further strengthens our claim on the dominance of data order on model fairness and group level accuracy. Fairbatch (Roh et al. 2020), a state-of-the-art bias mitigation algorithm, follows similar formulation as our custom data order. They create batches with fixed ratio between groups, and the ratio is optimized at every epoch counter the existing bias in the model. Our results not only explain the success of their algorithm, but also state that instead of regularly adapting the distribution to compensate for the model bias, one can directly use the final desired distribution and the model will adapt to it immediately.
Finally, we extract two special cases from the above experiments, i.e. ratios of 1 and 1 between positive and negative labels of group Female (for ACSIncome dataset), and call them EqualOnder and AdvOrder respectively. While EqualOrder achieves competitive performances with state-of- the-art bias mitigation methods, AdvOrder is capable of derailing the fairness scores of various mitigation algorithms. Note that we can force even worse fairness gaps than achieved by AdvOrder by pushing the ratio its extreme, however that will also noticeably impact the model's s overall accuracy. We perform experiments with three unique setups, standard training and two commonly used bias mitigation algorithms. This includes, (i) Reweighing (Kamiran & Calders 2012). data pre-processing algorithm which weighs every label-group pair based on its representation in the overall dataset, and (ii) Equalized Odds Loss (Fukuchi al. 2020), in-processing loss function to nudge the model towards fair predictions. We incorporate both EqualOrder and AdvOrder in these setups see the influence of data order. The results are collected in Figure g
By simply training with EqualOrder for a single epoch, the baseline models achieve competitive fairness scores to state-of-the- the--rrt bias mitigation algorithms. Reweighing suffers from an unexpected high bias under EqualOrder, as the combination of ideally distributed data order along with increase in the weight of minority data points pushes the model towards significant unfair behavior against the majority. On the other hand, using AdvOrder in the baseline setting can further push the model bias, emphasizing the adversarial power of data ordering. Both Reweighing and Equalized Odds Loss also suffers similar fates, but with controlled damage in latter due to the loss function regularly adapting to the degrading behavior. These results together cement the effectiveness of manipulating group level accuracy by controlling the data order for just single epoch of training.
8

---

Under review as a conference paper at ICLR 2023
80%
20%
Average Odds
75%
15%
Fl Score
70%
10%
65%
5%
60%
0%
Baseline
Reweighing
EO Loss
Baseline
Reweighing EO Loss
No additional training
Single Epoch of EqualOrder
Single Lpoch of AdvOrder
Figure 9: Under baseline training setup, by changing only the data order for just a single epoch of training. EqualOrder gets competitive performance to commonly used bias mitigation methods. Similarly, AdvOrder gets significantly worse fairness than even the baseline. For Reweighing, we see an increase bias even with EqualOrder, as the beneficial weighing of minority creates bias towards the majority. On the other hand, even reweighing is not enough to counter the effects of AdvOrder. Finally, equalized odds loss is capable of dynamically adapting to the model's 's changing predictive behavior, yet we still observe noticeable increase in bias under AdvOrder.
6 RELATED WORK
Variance in Fair Machine Learning Recently, there has been growing awareness of high variance and instability in fair deep learning, associated with non-determinism in model training or underlying implementations (Amir et al., 2021 [Sellam et al. 22011; Soares et al. 2022 Qian et al. [2021) and the challenges it creates in trusting existing results. The high variance fairness scores has even been used to propose a new leave-one-out unfairness metric (Black & Fredrikson 202T)
Soares et al. (2022) refute various trends in literature associated with fairness of large language models by simply executing multiple identical training runs, while (Sellm et al. (202 1 focus on the impact of randomness in pre-training paradigm and find further increase in instability. Amiir et al. (2021) even revisit oftte-of-the-att the-art bias mitigation in clinical texts and show a lack of statistically significant improvement after accounting for non-determinism in training. While existing literature focuses on exploring the impact of fairness variance, instead focus on investigating the source of this non-determinism. Furthermore, we propose to move away from the practice of executing multiple training runs capture this variance and instead provide computationally efficient proxy.
Data Order and Example Forgetting Controlling the data order during training has gained noticeable success in improving convergence speeds (Bengio et al., 2009 Shah et al.) 2020 Sooviny al. 2022 Mohtashami et al.; 2022). ,, while its adversarial capabilities have also been explored
(Shumailov et al., (021). Even under randomly shuffled data order, neural models are known to undergo changes in predictive behavior during training (Toneva al.) 2018 Kirkpatrick et al., 2017 Jagielski et al.) 2022. Tirumala et al. 2022). We extend the discussion on non non-determinism in model behavior to model fairness, and investigate its connections with data order. We focus specifically on the instability in predictions made by the learning model and not its internal state, as are concerned about the disparity group level decisions which highlights the encoded bias.
7 CONCLUSION
Fairness variance in deep learning has raised concerns regarding the reliability of existing results in literature. In this paper, we took a closer look at this variance and proposed a proxy to training multiple models by studying fairness scores inside single training run, which exploits a surprisingly dominant impact data order on model behavior. We even manipulated the data order to minutely control group level accuracy without losing overall accuracy in only single epoch on training. In our work, we focused only on the discrete decisions made by the model, as we were investigating the impact of non-determinism in model training its faireess. However, further extensions of this discussion to trends in the internal state of the learned model can reveal even granular characteristics, and has potential application in similar fields of research suffering from high variance.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
Our work studies various sources of non-determinism in neural model training and its impact on model fairness. Thus, it important address the implementation choices we have made allow easy reproducibility. We provide a minimal psuedo code in Appendix to clearly define various random seeds used in our setup control training non-determinism. The complete training and evaluation pipeline is deterministic outside the choice of these random seeds, i.e., one can perfectly replicate the training by simply fixing these seeds. We also provide our code to train models and perform several evaluations done in the paper supplementary material to allow easy adaptation for future research.
REFERENCES
Mohsen Abbasi, Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasubrammanian. Fairness in representation: quantifying stereotyping representational harm. In Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 801-809. SIAM, 2019.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint MxXi=22209,04336, 2022.
Silvio Amir, Jan- Willem van de Meent, and Byron C Wallace. On the impact of random seeds on the fairness of clinical classifiers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 3880-3822 2021.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. L. Rev., 104:671, 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Currieulum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 41-48, 2009.
T Birsan and Dan Tiba. One hundred years since the introduction of the set distance by dimitric pompeiu. In IFIP Conference on System Modeling and Optimization, pp. 35-39. Springer, 2005,
Emily Black and Matt Fredrikson.
Leave-one-out unfaimess. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 285-295, 2021.
Simon Caton and Christian Haas. Faimess in machine learning: A survey, arXiv preprint arXiv:2010.0405: 2020.
Alexandra Chouldechova and Aaron Roth. The frontiers of faimess in machine learning. arXiv preprint arXiv: 0880000,0000.... 2018
Kate Crawford. The hidden biases in big data. Harvard business review, 1(4), 2013.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309-1318. PMLR, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive ics.uci..dds
Akihiko Fukuchi, Yoko Yabe, and Masashi Sode.
Fairtorch. https://github.r
<bawakate/fairtorch 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint urXiv:2207.00099 2022.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowledge and information systems, 33(1):1-33, 2012.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinstt Bar--aaawin,,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46(25)):68-7777 1951.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Amirkeivan Mohtashami, Sebastian Stich, and Martin Jaggi. Characterizing & finding good data orderings for fast convergence of sequential gradient methods. arXiv preprint arXiv:2202.01838. 2022.
Shangshu Qian, Viet Hung Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah. Are my deep learning systems fair? an empirical study of fixed-seed training. Advances in Neural Information Processing Systems, 34, 2021.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fairbatch: Batch selection for model fairness. In International Conference on Learning Representations, 2020.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D D'mmour, Tal Linzen, Jasmijn Bastings, Tulia Raluca Ture, Jacob Fisenstein, et al. The multiberts: Bert reproductions for robustness analysis. In International Conference Learning Representations, 2021.
Vatsal Shah, Xiaoxia Wu, and Sujay Sanghavi. Choosing the sample with lowest loss makes sgd robust. In International Conference on Artificial Intelligence and Statistics, pp. 2120-2130. PMLR, 2020.
Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems, 34:18021-188 -18032, 2021.
Ioana Baldini Soares, Dennis Wei, Karthikeyan Natesan Ramamurthy, Moninder Singh, and Mikhail Yurochkin. Your faimess may vary: Pretrained language model faimess in toxic text classification. In Annual Meeting of the Association for Computational Linguistics, 2022.
Petru Soviany, Radu Tudor Tonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pp. 1-40, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1929:199991998 2014.
Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics large language models. arXiv preprint arXiv:2205. 10000000..0070. 2022.
11

---

Under review as a conference paper at ICLR 2023
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study example forgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017.
2017 Conference on Empirical Methods in Natural Language Processing, pp.
12

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
Non-Determinism in Model Training
14
B
Datasets
14
Training Curve and Convergence
Dropout Regularization
E
Additional Experiments for All Datasets
17
E.1
Weight Initialization and Random Reshuflling
17
E,2
Changing Predictions and Data Distribution
18
E.3
Capturing Variance a Single Training Run
19
Manipulating Group Level Accuracy with Data Order
F
Additional Experiments Changing Hyperparameters
21
F.1
Weight Initialization and Random Reshuflling
21
F2
Changing Predictions and Data Distribution
24
F33
Manipulating Group Level Accuracy with Data Order
25
Additional Experiments for Other Fairness Metrics
26
1
High Variance Fairness Scores
................................
27
Weight Initialization and Random Reshuflling
.e..............................
27
G.3
Capturing Variance Single Training Run
28
H
10 Raw Training Runs from Figure
29
13

---

Under review as a conference paper at ICLR 2023
A NON-DETERMINISM IN MODEL TRAINING
We provide a minimal pseudo-code to explain our control over training non-determinism and define weight initialization and random reshuffling seeds.
terch mannall (weightt model = MLPModel () # Initialize Neural Model
train_data, .. data_datasett load data with deterministic order train_data shuffle (train_ddata, seeddrrandomreehhfffllgggggge.
trainloader = torch. utulstllls....... Dataloade (train_data, shuffle=True)
for epoch in range (total_epochs)
torch. manual_seed for batch trainloader:
B DATASETS
ACSIncome ACSIncome is one of the five pre-defined tasks in the Folktables dataset Ding et al. (00TT) which was recently collected to improve the older and commonly used UCI Adult Income the subset of Folktables dataset from the state of California, USA for 2018. The dataset contains a total of 195, 665 data points with 10 features each, where each data points represents individual. The task is binary classification predict whether the individual's 's income is above $50, 000. For fairness measures, we use gender (Female vs Male) as the sensitive attribute, which is also one of the 10 input features.
dataset Dua & Graff (2017). More specifically,
List of features: Age, Class of worker, Educational attainment, Marital status, Occupation, Place of birth, Relationship. Usual hours worked per week in past 12 months, Sex, Recoded detailed race code
ACSEmployment ACSEmployment is another one of the five pre-defined tasks in the Folktables dataset [Ding et al. (2021). We use the same subset from the state of California, USA for 2018 as above. The dataset contains total of 378, 817 data points with 16 features each, and the task is binary classification to predict whether the individual is employed or not. For fairness measures, same above,
gender (Female Male) the sensitive attribute.
List of features Age, Educational attainment, Marital status, Sex, Disability recode, Employment status parents, Mobility status, Citizenship status, Military service, Ancestry recode, Nativity, Relationship, Hearing difficulty, Vision difficulty, Cognitive difficulty, Recoded detailed race code, Grandparents living with grandchildren
CelebA CelebA dataset is a large scale celebrity face attributes dataset Liu et al. 2015 which contains a total of 202, 599 images of celebrities with different binary labels each. We focus on the binary classification task of the 'smiling' label in the dataset, while we use the 'gender' label as the sensitive attribute for fairness evaluation. Moreover, we do directly use the images of CelebA dataset, but instead pass them through pre-trained and frozen ResNet-50 backbone He et al. (0016) to extract image representations, which are treated as inputs to our model.
14

---

Under review as a conference paper at ICLR 2023
C TRAINING CURVE AND CONVERGENCE
To understand why we choose to study the model behavior between epochs 100 and 300, we point the reader towards the overall training curve of the model plotted in Figure [0] It is clear that the model has converged by epoch 100, and maintains stable accuracy scores for the last 200 epochs.
(a) Aggregated training curve
(b) Two individual training curves
80%
78%
Fl
74%
100
150
200
250
300
150
200
250
300
Epoch
Epoch
Figure 10: (Left) Aggregated training curve for all 50 training runs with both changing weight initialization and random reshuffling. (Right) Example of two individual training runs.
Even though the accuracy scores have converged, we still find high variance in fairness scores as discussed in main text of the paper. One might suspect this implies that fairness scores could take longer to converge. To check this, we allow single training to run for total of 3000 epochs (as opposed to the standard 300 epochs used in all other experiments in our paper) and collect the fairness scores in Figure [I] It clear that the model fairness does not stabilize by simply increasing the number of epochs, which further supports our hypothesis of never-ending local oscillations due to SGD noise that cause high fairness variance.
10%
Average Odds
8%
6%
4%
500
1,000
1,500
2,000
2,500
3,000
Epoch
Figure 11: single training run extended 3000 epochs.
15

---

Under review as a conference paper at ICLR 2023
D DROPOUT REGULARIZATION
Another notable source of randommess in neural model training is dropout regularization (Srivastaval al., 2014). Dropout regularization randomly drops a certain percentage (known as dropout rate) of connections between consecutive layers in the model at every forward pass during training. We extend our discussion from Section 3 to study the impact of dropout on the trends of random reshuffling seen in Figure More specifically, y, we repeat the experiments while introducing various rates of dropout the training setup. The results are collected Figure
Randomness in dropout regularization under fixed weight initialization and fixed random reshuffling (2)g (h)) introduces minute variance across multiple runs. While it adds noise to previously
noted trends, the variance in fairness across consecutive epochs in single training run, as well as the dominance random reshuffling on fairness variance still clearly noticeable (Figure [2(c), (d), (O, (g)). Interestingly, the overall range of fairness variance does go down with higher dropout rate, but gradually impacts the final achieved accuracy scores (Figure j). (k)) as well as fairness.
No Dropout
Fixed Data Reshuffling
(b) Fixed Weight Initialization
Average Odds
10%
8%
6%
100
150
2000
250
3000
100
150
2000
250
3000
Epoch
Epoch
Dropout Rate 10%
(c) Fixed Reshufting
(d) Weight Initialization
(e) Both
10%
Avenge Odds
895
6%
100
150
200
250
300
100
150
200
250
300
100
150
200
250
300
Epoch
Epoch
Epoch
Dropout Rate 20%
Fixed Data Reshulling
(g) Fixed Weight Initialization
(h) Fixed Both
10%
Averago Odds
6%
150
200
250
300
100
150
200
250
300
100
250
300
Epoch
Epoch
Epoch
Mediane
quartile range
range
Training Curves
Drupout
(j) Dropout
(k) Dropout
80%
Score
76%
50
100
150
200
250
300
50
100
150
200
250
300
50
100
150
200
250
300
Epoch
Epoch
Epoch
Figure 12: Additional experiments with dropout regularization reveal similar trends in Figure These results further highlight the dominant impact of random reshuffling on fairness.
16

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENTS FOR ALL DATASETS
E. WEIGIIT INITIALIZATION AND RANDOM RESHUFFLING
We provide additional results in Figure 1 14 on CelebA and ACSEmployment dataset to show the dominance of random reshufiling on model fairness.
CelebA
(a) Normal training
(b) Fixed Reshuffling
(c) Fixed Weight Init
88%
Score
87%
86%
85%
8%
2%
1%
0%
8%
2%
4%
6%
8%
Average Odds
Average Odds
Average Odds
A ACSEmployment (b) Fixed Reshuffling
(a) Normal training
(c) Fixed Weight Init
81.8%
81.6%
81.4%
4.5%
51%
5.5%
6%
6.5%
4.5% 5%
5.5% 6%
6.5%
4.5%
5.5% 6%
6.5%
Average
Average Odds
Average
Additional experiments on CelebA and ACSEmployment datasets reveal similar trends as seen in Figure Random reshuffling of data order is the dominant source of variance in fairness scores, with very little influence from the weight initialization.
Figure 13:
CelebA
Fixed Reshuflling
(b) Weight Initialization
10%
Avenge Odds
6%
4%
2%
100
150
200
250
300
100
150
200
250
300
Epoch
Epach
ACSEmployment
(a) Fixed Data Reshuffling
Fixed Weight Initialization
79%
Average
5%%
4%100
150
200
150
200
250
300
Epoch
Epoch
Median Inter-quartile range
range
Figure 14: Additional experiments CelebA and ACSEmployment datasets reveal similar trends as in Figure These results further highlight the dominant impact of random reshuffling on fairness.
17

---

Under review as a conference paper at ICLR 2023
E.2 CHANGING PREDICTIONS AND DATA DISTRIBUTION
We show the relationship between data distribution and changing predictions for CelebA and AC- SEmployment Figure 13
The group with least representation maintains being the most vulnerable to changing predictions.
CelebA
(a) Data Distribution
(b) Unique Data Points that Changed Prediction
40%
Group Perconaage
30%
10%
10%
066
100
Epoch
(Smiling)
(Not Smiling)
(Not Smillng)
Fenale (Smiling)
ACSEmployment
(a) Data Distribution
(b) Unique Data Points that Changed Prediction
20%
Group Parcentage
3098
10%
5%
10%
060
100
Epoch
Female (Employed)
Male (Employed)
Male (Unemployed)
Female (Unemployed)
Figure 15: Additional experiments on CelebA and ACSEmployment datasets reveal similar trends as seen in Figure 4 These results highlight subgroups with least representation being the most vulnerable to changing predictions.
We showed that the faimess scores dominated by the most recent gradient updates seen by the model. As a sanity check, we also provide ablation study for experiments in Figure 6 but choose the batches randomly instead of from the prefix. The results are collected in Figure [6 The results show that it is indeed the batches from data order prefix that govern the predictability of model faireess.
Best Fairness Order
Worst Fairness Order
Average Odds
10%
10%
8%
8%
6%
6%
0
20
50
100
200
0
20
50
100
200
Number of Training Batches
Number of Training Batches
Figure 16: Additional experiments by choosing batches, similar to the experiments in Figure 6 but choosing batches randomly instead of the prefix. The results still maintain stability as 6 increases but now they stabilize to random fairness scores instead of best and worst fairness scores as seen in Figure 6 Clearly, its the most recent batches in that order which truly governs the model fairness.
18

---

Under review as a conference paper at ICLR 2023
E.3 CAPTURING VARIANCE IN SINGLE TRAINING RUN
We show the empirical similarity of distribution across multiple training runs and multiple epochs in single training run for additional datasets CelebA and ACSEmmpoyment in Figure 17
CelebA
(a) Fairness Score Distribution
(b) Black Swans
5.0
MM Carage 0d65
Seeds
30
4.0
20
Avrrage 0385
0
40
Epochs
3.0
Sesssoo Juoperis
Sends
2.0
20
Fporh 200)
(Epach 1250)
(Tpoch) 1001
(Sed
(Seed 10
Across (Seed 23
40
Sampling
Epachs
ACSEmployment
(a) Fairness Score Distribution
(b) Black Swans
5.5
40
5.2
30
5.0
20
10:
4,88
pvrengge 0455
6.0
20
Epochs
Hausdorff Distancc
40
30
20
4.5
10
Across Spocis
Acess Epcchs (See)
Acrosa Epocha
0
0,2
(Esech: 2001
Hipoch 2301
(Epach: 3001
(Sceal:
(Seed
Sampling
Epochs
Figure 17: Additional experiments on CelebA and ACSEmployment datasets reveal similar trends as seen in Figure 7 Fairness scores (average odds) across multiple training runs and across epochs in single training run have similar empirical distributions, Thus, studying this distribution across epochs provides a highly efficient alternative executing multiple training runs.
19

---

Under review as a conference paper at ICLR 2023
E.4 MANIPULATING GROUP LEVEL ACCURACY WITH DATA ORDER
We provide additional results in Figure 18 19
highlighting the predictability of model fairness based on the data order, and its impact compared against various state-of ofttte--rtttttttt-- baselines.
CelebA
100%
100%
Accuracy
Accuracy
80%
80%
-- - 6-
- --
M-N
MNN Overall
60%
60%
10-1
100
10a
10
100
10a
Ratio of F-Pos F-Neg
Ratio M-Pos M-Neg
ACSEmployment
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
--
-
E.Neg -
20%
-
- --
MNN
MNe Overall
0%
0%
10-1
100
10a
10
10
10a
Ratio of F-Pos F-Neg
Ratio M-Pos: M-Neg
Figure 18:
Additional experiments on CelebA and ACSEmployment datasets reveal similar trends as seen in Figure X In only single epoch of training, we are able to manipulate the group level accuracy trade-off, with relatively small impact on overall accuracy.
CelebA
90%
10%
88%
Average Odds
8%
Score
86%
6%
84%
4%
82%
2%
80%
0%
Baseline
Reweighing
EO Loss
Baseline
Reweighing
EO Loss
ACSEmployment
80%
Average Odds
30%
Fl Score
70%
20%
60%
10%
50%
0%
Baseline
Reweighing
EO Loss
Baseline
Reweighing EO Loss
No additional training
Single Epoch of EqualOrder
Single Epoch of AdvOrder
Figure 19: Additional experiments on CelebA and ACSEmployment datasets reveal similar trends seen in Figure 9 EqualOrder gets competitive performance to commonly used bias mitigation methods. Similarly, AdvOrder gets significantly worse fairness than even the baseline.
20

---

Under review as a conference paper at ICLR 2023
F ADDITIONAL EXPERIMENTS FOR CHANGING HYPERPARAMETERS
We provide additional experiments for (i) batch size 16 and 1024, deviating from the default batch size of 128 in the main text, (ii learning rate 0.01 and 0.0001, deviating from the default learning rate of 0.001 in the main text, and (iii) model architecture with two hidden layers containing 2048 and 64 neurons respectively, deviating from the single hidden layer architecture used in the main text.
F.1 WEIGHT INITIALIZATION AND RANDOM RESHUFFLING
We provide additional results in Figure 21 for changing batch size, learning rate, and architecture on ACSIncome dataset to show the dominance of random reshuffling on model fairness. With a decrease in batch size, the range of fairness variance increases significantly, but the overall expected trends follow the same behavior as noted in the main text, i.e. a sharp change in fairness scores across epochs under fixed data reshuffling, and high variance even for single epoch across multiple runs under changing data reshuffling (fixed weight initialization). Similar trends can be observed when we increase the learning rate, or provide the training algorithm with bigger neural model. The increase in variance suggests high instability with smaller batch size, higher learning rate, and bigger neural models, all of which expected.
On the other hand, one would expect more stable model behavior with a bigger batch size or a smaller learning rate. While this is indeed the case, the comparison here not fair because these models have not yet converged, as evident by the clear downward trend of fairness scores in Figure 21 Note that compared to these models, the original set of models converged before epoch 100 (see Appendix A).. To further highlight the same, we also show the training curve of all 5 settings compared against the default setup in Figure 20 It is clear that certain hyperparameter settings are not conducive to efficient convergence, even though they might eventually provide more stable fairness scores.
Finally, to understand the difficulty of reaching convergence, we continue training a single instance with batch size 1024 (and learning rate 0.0001) for total of 1000 epochs and compare it against the standard training setup (i.e., with batch size 128 and learning rate 0.001). The results are collected in Figure 23 is clear that a larger batch size (or lower learning rate) significantly slows down the convergence speed. Moreover, it also does not achieve the same accuracy scores previously seen, possibly due to not being able take complete advantage of the noise usually found in mini-batch gradient descent. Thus, using high number of training epochs for a model with hyperparameters that enforce stability (i.e. large batch size or lower learning rate) is inefficient solution to solving the problem of fairness variance.
(a) Standard Training
(b) Batch Size 16
(c) Batch Size 1024
80%
Seoor
78%
76%
74%
50
100
150
200
250
300
50
100
150
200
250
300
50
100
200
250
300
Epoch
Epoch
Epoch
(d) Learning Rate 0.01
(e) Learning Rate 0.0001
(f) Model Architecture = (2048, 64)
80%
FI Seore
78%
76%
74%
50
100
130
200
250
3000
50
1000
150
2010
250
300
50
100
150
200
250
300
Epoch
Epoch
Epoch
Figure 20: Training curve of the five new settings, along with original training curve copied here for easy reference. It is clear that decreasing the batch size, increasing learning rate, or using bigger model architecture results in a faster model convergence but with higher variance even in accuracy scores. On the other hand, models with high batch size or low learning rate tend to not achieve the same accuracy scores and are still improving even at 300 epochs.
21

---

Under review as a conference paper at ICLR 2023
Batch Size 16
(a) Fixed Data Reshuffling
(b) Fixed Weight Initialization
15%
Average Odds
10%
5%
100
150
200 Epoch
250
300
100
150
200
250
300
Epoch
Batch Size 1024
(a) Fixed Data Reshuffling
(b) Fixed Weight Initialization
12%
Average Odds
10%
8%
100
150
200
250
300
100
150
200
250
300
Epoch
Epoch
Learning Rate
0.01
(a) Data Reshufiling
Fixed Weight Initialization
13%
Average Odds
10%
5%
100
150
200
250
300
100
150
200
250
300
Epoch
Epoch
Learning Rate = 0.0001
(a) Fixed Data Reshuffling
(b) Fixed Weight Initialization
Average Odds
12%
10%
8%
150
200
250
300
100
150
200
250
Epoch
Epoch
Model Architecture {2048, 64)
(a) Fixed Data Reshuffting
(b) Fixed Weight Initialization
15%
Average Odds
10%
5%
100
150
200
250
300
100
150
200
250
300
Epoch
Epoch
Median
Inter-quartile range
Overall range
Figure 21: Additional experiments for changing training hyperparameters with experiment setting as Figure These results further highlight the dominant impact of random reshuffling on fairness.
22

---

Under review as a conference paper at ICLR 2023
80%
78%
70%
Fl
74%
72%
Batch Size 128
Butch Size 1024
70%
200
400
600
800
1,000
Epoch
Batch Size 128
14%
Batch Size 1024
Average Oddss
12%
10%
8%
6%
200
400
600
S00
1,000
Epoch
Figure 22: Using higher batch size can over time achieve reasonably stable fairness scores, however the convergence speed is significantly slower. Moreover, the higher batch size does not convergence to the same accuracy scores, and looses a noticeable margin accuracy.
78%
76%
74%
72%
Learning Rate 0.001
Learning Rate
70%
200
400
1,000
Epoch
Learning Rate 0.001
149%
Leamingg Rate 0.0001
Average Odds
12%
1096
8%
6%
200
800
1,000
Epoch
Figure 23: Using lower learning rate can achieve reasonably stable fairness scores, however the convergence speed is significantly slower. Moreover, the lower learning rate does not convergence to the same accuracy scores, and looses noticeable margin of accuracy.
23

---

Under review as a conference paper at ICLR 2023
F.2 CHANGING PREDICTIONS AND DATA DISTRIBUTION
We note changing predictions for all settings described above in Figure 24 Note that since we are focusing on changing training hyperparameters for ACSIncome, the data distribution remains the same as in Figure Ma 4a) (also copied here in Figure 24 for reference).
Same as before, model instability has increased with smaller batch size, higher learning rate, or bigger model architecture, but the trends of vulnerability for various subgroups remains the same. We already know that the models have not converged for higher batch size or smaller learning rate, but we can still see the same trends of higher vulnerability for minority, although they are not as pronounced in already converged models.
Batch Size
Batch Size = 16
Batch Size 1024
Unique Data Points that Changed Prediction
Unique Data Points that Changed Prediction
Group Percentgge
30%
Percentgee
8%
10%
2%
086
0%
100
150
Ppoch
Epoch
Learning Rate
Learning Rate 0.01
Learning Rate 0.0001
Unique Data Points that Changed Prediction
Unique Data Points that Changed Prediction
Group Percenagg
30%
Gimup Ponentagg
8%
20%
10%
2%
1100
150
1000
150
Epoch
Epoch
ACSIncome Data Distribution
Model Architecture {2048, 64)
(a) Data Distribution
Unique that Changed Prediction
10%
Group Perceniage
301%
20%
20%
10%
10%
0%
100
150
200
250
300
Epoch
Female (Salary 50X)
Male (Sulary 50X'
Male (Salury 50K)
Female (Salary 50K)
Figure 24: Additional experiments reveal similar trends as seen in Figure 4 These results highlight subgroups with least representation being the most vulnerable to changing predictions.
24

---

Under review as a conference paper at ICLR 2023
F.3 MANIPULATING GROUP LEVEL ACCURACY WITH DATA ORDER
We provide additional results in Figure 25 highlighting the predictability of model fairness based on the data order. Note that the hyperparameter setting for additional epoch of fine-tuning during group accuracy manipulation is the same as the setup used for training that particular model. For example, when manipulating models which were trained with batch size 16, the single epoch of fine-tuning is also done with batch size 16.
Batch Size 16
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
- F-Poi - ENg e- M.Poo
20%
- F-Pss - F-Ne - M-Pos
0- M.Nag c...
6- M.Na c
0%
10-1
100
10a
10-1
100
10a
Ratio of F-Pos F-Neg
Ratio of M-Pos M-Neg
Learning Rate
0.01
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
- F-Pos -
FNN
e M.Po
20%
- F.Pss - F-Ner - M-Pos
0- c..
0- MNag c
0%
10-1
100
10a
10-1
100
10
Ratio of F-Pos: F-Neg
Ratio of M-Pos: M-Neg
Model Architecture
{2048, 64)
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
- F-Pux - F-g e- M-Pa
20%
- F-Raa - F-Nay - M-Pos
0 M-Neg Ohrril
0- M-Nog C Dorall
0%
0%
10-1
100
10a
10-1
100
10a
Ratio of F-Pos F-Neg
Ratio of M-Pos: M-Neg
Figure 25: Additional experiments reveal similar trends in Figure In only single epoch of training, we are able to manipulate the group level accuracy trade-off, with relatively small impact on overall accuracy.
We also provide results for models with high batch size and low learning rate separately in Figure 26 However, note that not only are these models not converged, but they are also fine-tuned on the same inefficient hyperparameter settings. Thus, the trends here are not comparable, but added for completeness.
25

---

Under review as a conference paper at ICLR 2023
Batch Size
1024
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
F-Pus
M.Pos
20%
FNNer
Neg Orzaall
Overal
MiNeg G.. Overall
0%
0%
10
100
10
10
100
10
Ratio of F-Pos: F-Neg
Ratio of M-Pos: M-Neg
Learning Rate
0.0001
100%
100%
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
F-Pos
M.Pos
20%
F-Pos
M.Ne Ovenall
M.Nq G Ovenll
0%
0%
10
100
10a
10
10
Ratio of F-Pos F-Neg
Ratio M-Pos M-Neg
Figure 26: Additional experiments as seen in Figure
25 but for models that did not converge as noted
earlier. The results are added for completeness.
G ADDITIONAL EXPERIMENTS FOR OTHER FAIRNESS METRICS
repeat the experiment in the main text for two more fairness metrics, Equal Opportunity (EOpp)
and Disparate Impact (DI) Hardt et al. (2016)
For model predictions R, true labels Y, and sensitive attributes A, EOpp can be defined as,
EqualOpportunityy : [P 1|Y 1, - 1| = 1, 1)}.
(2)
Similarly, DI can be defined as,
P(R 1|A P(R 1|A=1)
DisparateImpact == 11 min
min
(3)
P(R = 1|4= 1) P(R = 1|A= 0)
26

---

Under review as a conference paper at ICLR 2023
G.1
HIGH VARIANCE IN FAIRNESS SCORES
We start by repeating the results in Figure D
show high variance in different fairness measures.
(a) Equal Opportunity
(b) Performance
Bascline
Reweigh
FairBarch
0%
2%
6%
8%
10%
12%
78%
80% F1-Sore
82%
Equal Opportunity
(a) Disparate Impact
(b) Performance
Baselin
Reweigh
ED Lass
FairBatch
10%
15%
20%
25%
30%
35% 789
800
92%
Disparate Impact
Fl-Score
Figure 27: Additional experiments with different fairness metrics for setting in Figure [ Clearly, fairness has a high variance across identical runs. Note that the x- x-axis across fairness and accuracy metric not similarly scaled for disparate impact to keep the results readable.
G.2 WEIGHT INITIALIZATION AND RANDOM RESHUFFLING
We provide additional results for EOpp and DI for the experiment conducted in Figure 3 on ACSIn- come dataset. The results collected in Figure 28 show similar trends as seen in the main text.
Equal Opportunity
(a) Fixed Data Reshullling
(b) Fixed Weight Initialization
14%
Avenuge Odds
12%
100
130
200
250
300
100
150
200
250
300
Epazh
Epoch
Disparate Impact
(a) Fixed Data Reshuffting
(b) Fixed Weight Initialization
Average Odds
34%
32%
28%
26%
100
200
Epoch
Epoch
Median m Inter-quariin range
Overall range
Figure 28: Additional experiments with faimess metric EOpp and DP reveal similar trends in Figure These results further highlight the dominant impact of random reshuffling on faimess.
27

---

Under review as a conference paper at ICLR 2023
G.3 CAPTURING VARIANCE IN SINGLE TRAINING RUN
We show the empirical similarity of distribution across multiple training runs and multiple epochs in single training run for fairness measures EOpp and DI in Figure 29
ACSIncome
Equal Opportunity
Disparate Impact
ACSEmployment
Equal Opportunity
Disparate Impact
AQUR 201
ADREE 200
KOOM Souck
JOOH RONO ADAMl
ROUA SeAu
Adjas SOACTe
CelebA
Equal Opportunity
Disparate Impact
Dasora Apprr
200
2008
Figure 29: Additional experiments for fairness metrics EOpp and DI. Fairness scores across multiple training runs and across epochs in single training run have similar empirical distributions. Thus, studying this distribution across epochs provides highly efficient alternative.
28

---

Under review as a conference paper at ICLR 2023
H 10 RAW TRAINING RUNS FROM FIGURE
We plot 10 randomly chosen training runs each for fixed weight initialization and fixed random reshuffling in Figure 30 Figure 31 respectively. As expected, each individual training in both settings has high variance across epochs even after convergence. More importantly, the trends of fairness matches closely across multiple runs for fixed random reshuffling, even though they started from different weight initialization.
10%
Average Oddss
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
My wy Wn Ww
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
8%
6%
100
120
1410
160
180
220
240
260
300
Epoch
Average Odds
10%
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
Figure 30: 10 randomly chosen raw training runs plotted in groups of 2 for fixed weight initialization.
29

---

Under review as a conference paper at ICLR 2023
10%
Average Odds
Wor Mird
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
MM M
MM
M
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
M Man
Mmn
MM
8%
6%
100
140
160
180
220
240
260
300
Epoch
10%
Average Odds
MM
M
M
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
10%
Average Odds
/
MM
M
J/M MMM
8%
6%
100
120
140
160
180
200
220
240
260
280
300
Epoch
Figure 31: 10 randomly chosen raw training runs plotted in groups of 2 for fixed random reshuffling.
30


Reviews:
[{"review_id": "doKcBKlT7o-", "reviewer": "Reviewer_XbQB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper observes that the performance of a deep learning model might exhibit a high variance between different training instances and hence this might lead to a high variance in the performance gap across different groups. The paper proposes a solution based on the reordering of data, to lower the variance.", "summary_of_the_review": "In summary, this paper is outside my area of expertise but based on what I understand, the contributions and novelty are limited.\n\n----------------------------------------\nPost Rebuttal:\n----------------------------------------\nI want to thank the authors for their detailed responses. I have read the other reviews and the rebuttal and I stand by my original assessment.", "strength_and_weaknesses": "---------------------------------\nStrengths:\n---------------------------------\n-- The paper is fairly well-written and it is easy to understand the paper.\n\n---------------------------------\nWeaknesses:\n---------------------------------\n-- The technical contribution is limited. The paper does not provide any algorithms or analysis.\n\n-- While having theory is not always required, I find the empirical analysis of the paper to be insufficient. The paper mainly focuses on one dataset and there should be more empirical evidence to justify the provided solution. More intuition about the proposed solution would also be appreciated. \n\n-- I may have not fully misunderstood the paper, but what is the rationale behind lowering the variance besides the training time? Can the results be repeated and averaged to lower the variance?\n\n---------------------------------\nDisclaimer:\n---------------------------------\n-- I do not an expert on the topic and it is possible that I have missed the major points of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NGmi-0fGP8", "reviewer": "Reviewer_Qe5p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to find empirically validated reasoning for the cause of high variance in fair deep learning, where the fairness metric of interest is average odds (AO), i.e., the average disparity between true and false positive rates. This is generally attributed to randomness (what they call non-determinism) in training, e.g., randomness in data order and weight initialization. A series of empirical evaluations are conducted that investigate the source of this variance. The following assertions are made in light of the empirical evidence:\n-  Variance (wrt to AO) can be attributed more to data shuffling than to weight initialization. \n- A model's fairness is predictable based on the most recent training points.\nThe work also asserts that a proxy for understanding model fairness across runs can be variance in fairness across epochs. Moreover, the work shows that sub-group performance on the fairness metric can be manipulated by rearranging the training data instances. \n", "summary_of_the_review": "The paper empirically shows that the order of data instances in model training can impact the quality of the averaged odds fairness metric, introduces variance in fairness performance across epochs as a proxy for understanding fairness across runs, and shows that sub-group performance on the fairness metric can be manipulated by rearranging the training data instances. Some aspects of the experimental evaluation related to model predictability and recent data instances are unclear. ", "strength_and_weaknesses": "Strengths\n- The topic of this work is important, i.e., what are some of the causes of high variance in fair deep learning?\n- The concept of rearranging data instances to change sub-group performance is technically interesting. \n\nWeaknesses \n- There are other sources of randomness besides weight initialization and the training data, e.g., various regularization techniques and train/fit/validation sets (as the authors mentioned in the text). Can the authors provide reasoning for why only weight init and data ordering are considered in this work? \n- Unfortunately, I do not understand section 3.2 (see `clarity' section for questions) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "71R21MeA8Re", "reviewer": "Reviewer_ofwu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates how a specific fairness metric varies with the weights initialization and the batches data ordering used in the training of neural networks on three datasets. The authors find that the order in which data observations are fed to the networks during training significantly affects the variance in the fairness metric and conduct experiments to corroborate this claim.", "summary_of_the_review": "The paper builds upon an interesting observations but more work is needed in order to support the conclusions drawn in the paper (e.g., those at the end of section 4.2). ", "strength_and_weaknesses": "The authors make an interesting observation. This, however, seems to be only a starting point and more discussion and experiments are needed. I'll detail below some of the reasons behind this consideration. \n\nFirst, section 3 is titled \"Sources of variance\" but the only two sources of variance that are considered are weights initialization and random reshuffling. Batch size (which is fixed, I believe), architectures (two, fixed), and possibly other parameters are not discussed. These parameters most likely influence variance. In particular, as batch size increases, random order of the data will play less of a role. Why hasn't batch size been considered?\n\nSecond, I would expect data reshuffling to possibly play a role in the initial epochs but less in the later ones when the weights have converged to one of the minima. I'm not sure why this doesn't occur in the experiments. For this, it would be useful to see how overall accuracy increases with the number of epochs. Relatedly, I am very surprised that there is the similar amount of variance for a fixed epoch after 100 and 200 epochs. We could look into that by visualizing the curves for each individual model in figure 3 right, rather than the distribution for a given epoch. \n\nThird, the authors consider three datasets and only one measure fairness measure. Does this behavior generalize to other fairness measures? Why was this fairness measure chosen?\n\nFourth, section 5 is based on the idea of reweighing the data distribution to change the fairness properties of the model evaluated on the original data distribution. This idea has been widely studied in the literature (both in the fairness and machine learning in general) and I'm not sure that I fully understand where the novelty is, especially how this affects the . The observed behavior may also not even persist as one tunes the batch size and increases the number of epochs (see https://arxiv.org/abs/1812.03372).\n\nA paper that was not cited but is certainly relevant is https://arxiv.org/abs/2107.10171\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject