Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to find empirically validated reasoning for the cause of high variance in fair deep learning, where the fairness metric of interest is average odds (AO), i.e., the average disparity between true and false positive rates. This is generally attributed to randomness (what they call non-determinism) in training, e.g., randomness in data order and weight initialization. A series of empirical evaluations are conducted that investigate the source of this variance. The following assertions are made in light of the empirical evidence:
-  Variance (wrt to AO) can be attributed more to data shuffling than to weight initialization. 
- A model's fairness is predictable based on the most recent training points.
The work also asserts that a proxy for understanding model fairness across runs can be variance in fairness across epochs. Moreover, the work shows that sub-group performance on the fairness metric can be manipulated by rearranging the training data instances. 

strength_and_weaknesses: Strengths
- The topic of this work is important, i.e., what are some of the causes of high variance in fair deep learning?
- The concept of rearranging data instances to change sub-group performance is technically interesting. 

Weaknesses 
- There are other sources of randomness besides weight initialization and the training data, e.g., various regularization techniques and train/fit/validation sets (as the authors mentioned in the text). Can the authors provide reasoning for why only weight init and data ordering are considered in this work? 
- Unfortunately, I do not understand section 3.2 (see `clarity' section for questions) 
clarity,_quality,_novelty_and_reproducibility: Clarity
- sec 2 first para. Could please explain these datasets in the appendix so that readers do not have to hunt in numerous different works? The datasets are important to the empirical evaluations, and the additional description will make the work more modular. 
- description of the experimental setup is clear 
- sec 3.2 Can you be exact in what it means for predictions to change? Does this mean that before including the data instance, the model would have predicted the instance incorrectly? In light of this confusion I am unable to understand Figure 4b. 
- sec 4 is well-written and has a nice experimental setup

Novelty: I am unfamiliar with work related to this paper but did a quick search on arXiv, and did not find similar work. 

Reproducibility: I believe I could reproduce this work if need be---enough experimental detail has been given throughout the work and in the appendices. 

Small suggestions:
- section 6 second-to-last line: "practise of" to "practice of" 
- section 5 first para last line: "and place at the" to "and placed at the" 
- missing period eqn (1) 

summary_of_the_review: The paper empirically shows that the order of data instances in model training can impact the quality of the averaged odds fairness metric, introduces variance in fairness performance across epochs as a proxy for understanding fairness across runs, and shows that sub-group performance on the fairness metric can be manipulated by rearranging the training data instances. Some aspects of the experimental evaluation related to model predictability and recent data instances are unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper observes that the performance of a deep learning model might exhibit a high variance between different training instances and hence this might lead to a high variance in the performance gap across different groups. The paper proposes a solution based on the reordering of data, to lower the variance.
strength_and_weaknesses: ---------------------------------
Strengths:
---------------------------------
-- The paper is fairly well-written and it is easy to understand the paper.

---------------------------------
Weaknesses:
---------------------------------
-- The technical contribution is limited. The paper does not provide any algorithms or analysis.

-- While having theory is not always required, I find the empirical analysis of the paper to be insufficient. The paper mainly focuses on one dataset and there should be more empirical evidence to justify the provided solution. More intuition about the proposed solution would also be appreciated. 

-- I may have not fully misunderstood the paper, but what is the rationale behind lowering the variance besides the training time? Can the results be repeated and averaged to lower the variance?

---------------------------------
Disclaimer:
---------------------------------
-- I do not an expert on the topic and it is possible that I have missed the major points of the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, and it is generally well-written. I am not an expert in the area so I cannot assess the novelty of this work. It seems like similar observations have been made in a setting with no fairness consideration. In that sense, observing variance in performance when restricting to a subset of data is not surprising. The authors claim that the code is attached but I do not see a link. 
summary_of_the_review: In summary, this paper is outside my area of expertise but based on what I understand, the contributions and novelty are limited.

----------------------------------------
Post Rebuttal:
----------------------------------------
I want to thank the authors for their detailed responses. I have read the other reviews and the rebuttal and I stand by my original assessment.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates how a specific fairness metric varies with the weights initialization and the batches data ordering used in the training of neural networks on three datasets. The authors find that the order in which data observations are fed to the networks during training significantly affects the variance in the fairness metric and conduct experiments to corroborate this claim.
strength_and_weaknesses: The authors make an interesting observation. This, however, seems to be only a starting point and more discussion and experiments are needed. I'll detail below some of the reasons behind this consideration. 

First, section 3 is titled "Sources of variance" but the only two sources of variance that are considered are weights initialization and random reshuffling. Batch size (which is fixed, I believe), architectures (two, fixed), and possibly other parameters are not discussed. These parameters most likely influence variance. In particular, as batch size increases, random order of the data will play less of a role. Why hasn't batch size been considered?

Second, I would expect data reshuffling to possibly play a role in the initial epochs but less in the later ones when the weights have converged to one of the minima. I'm not sure why this doesn't occur in the experiments. For this, it would be useful to see how overall accuracy increases with the number of epochs. Relatedly, I am very surprised that there is the similar amount of variance for a fixed epoch after 100 and 200 epochs. We could look into that by visualizing the curves for each individual model in figure 3 right, rather than the distribution for a given epoch. 

Third, the authors consider three datasets and only one measure fairness measure. Does this behavior generalize to other fairness measures? Why was this fairness measure chosen?

Fourth, section 5 is based on the idea of reweighing the data distribution to change the fairness properties of the model evaluated on the original data distribution. This idea has been widely studied in the literature (both in the fairness and machine learning in general) and I'm not sure that I fully understand where the novelty is, especially how this affects the . The observed behavior may also not even persist as one tunes the batch size and increases the number of epochs (see https://arxiv.org/abs/1812.03372).

A paper that was not cited but is certainly relevant is https://arxiv.org/abs/2107.10171

clarity,_quality,_novelty_and_reproducibility: The paper is clear and the experiments are, in principle, reproducible. I don't think that the work is significantly novel. 
summary_of_the_review: The paper builds upon an interesting observations but more work is needed in order to support the conclusions drawn in the paper (e.g., those at the end of section 4.2).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Uncertainty and unfairness have a close connection, as underprivileged groups suffer, among other issues, from sources of variability that larger groups may not. The paper breaks down the analysis of some of these sources with a focus on the variability of the estimator of fairness metrics.

Strengths: a presentation of a systematic characterization of sources of variability in the pipeline of a machine learning solution that should produce estimates of some fairness metric, in particular those addressing randomness in the optimization process.

Weaknesses: as discussed by reviewers, ultimately I found the experimental setup somewhat ad-hoc, more of an exploratory exposition. It is difficult to interpret variability to one metric without a sense of how much is inherent to the data and how much is inherent to the algorithm. At least I would expect to anchor results on some other learning method with fewer degrees of freedom like XGBoost and try to make sense how much do we even still gain by worrying about deep learning issues of model fitting (for the Folktables examples in particular, I would not be shocked if a generalized linear model is essentially as good as it gets). Also, much of the phenomena observed may boil down to group imbalance in the data, which is not exactly surprising and mitigated by stratified sampling in the mini-batches. I do think this type of study has value, but at the present stage it feels some points open that should be treated earlier than later.

justification_for_why_not_higher_score: As discussed by reviewers, ultimately I found the experimental setup somewhat ad-hoc, more of an exploratory exposition. It is difficult to interpret variability to one metric without a sense of how much is inherent to the data and how much is inherent to the algorithm. At least I would expect to anchor results on some other learning method with fewer degrees of freedom like XGBoost and try to make sense how much do we even still gain by worrying about deep learning issues of model fitting (for the Folktables examples in particular, I would not be shocked if a generalized linear model is essentially as good as it gets). Also, much of the phenomena observed may boil down to group imbalance in the data, which is not exactly surprising and mitigated by stratified sampling in the mini-batches. I do think this type of study has value, but at the present stage it feels some points open that should be treated earlier than later.
Decision: Reject