Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper did a thorough analysis of the effect of software implementation (e.g., floating-point and parallelism) on bias measures. It has been previously shown that FIT runs produce different accuracies! The result is surprising and although I knew that the same FIT runs can produce different outcomes, I didn’t know that the outcome could be that different! And it is a good study for the fairness community to be aware of! 
main_review: The paper did a thorough analysis of the effect of software implementation (e.g., floating point and parallelism) on bias measures. It has been previously shown that FIT runs produce different accuracies! The result is surprising and although I knew that the same FIT runs can produce different outcomes, I didn’t know that the outcome could be that different! And it is a good study for the fairness community to be aware of! 

The paper is very well written and empirical analysis is very thorough ranging from different biases, different architectures, etc.

Some suggestions (minor concerns):
I think it would be great if you report how much accuracies vary (from previous work, e.g., 51), and compare it with fairness measures! Are fairness measures more brittle and experience a wider range of change? 
I think showing how much these measures vary with (1) different train/test split (2) different random seeds, and (3) due to software implementation, would be great (even over a few architectures to just get the sense of different components in variations.  I mean if (1) leads to much higher variation than (3) then studying (3) may not be very interesting. 
You emphasis a lot on the fact that increasing fairness measures will lead to decrease accuracy which is a very well-known fact, I think emphasizing on increase in the variance of accuracy or variance of bias measure is more interesting. 

limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the impact of so-called implementation-level nondeterminism of machine learning frameworks on state-of-the-art fairness methods. Unlike algorithmic nondeterminism, which is caused by, e.g., dropout or random shuffling and can be rendered deterministic by fixing the random seed, implementation-level nondeterminism is a consequence of parallelism and non-associativity of floating-point arithmetic. Thus, even fixed-seed training can lead to different results, and prior work has shown that the accuracy can vary by up to 2.9% across identical runs. This paper empirically demonstrates that implementation-level nondeterminism also impacts many fairness methods and can lead to variations of up to 12.6% in standard fairness metrics.
main_review: **Originality:** The paper is the first to empirically show that implementation-level nondeterminism can negatively impact state-of-the-art fairness methods. These findings are different from those of prior work by Friedler et al. (2019), which this paper needs to cite, showing that fairness methods are sensitive to different training-test splits. The related work is generally cited adequately, but Finding 6 should indicate which 32 papers were considered.

**Quality:** The submission is technically sound. The paper uses various statistical tests to compute the significance of the results. However, the paper could improve the motivation for using these specific tests. For example, multiple tests for effect sizes based on the difference between means exist, but the paper only uses Cohen’s d. Moreover, the paper should repeat the experiments with different fixed seeds to see how the results are affected.

**Clarity:** The paper is generally clearly written and well organized. However, I think the causes of implementation-level nondeterminism should be moved from Appendix E to the introduction to provide a better overview of the problem setting. In addition, there are some mistakes, such as missing words (e.g., L087 most (83.8%) *researchers* are, L103 label *which* should, L105 even *though* the) and typos (e.g., L658 Google *Imgae* Search).

**Significance:** The results are meaningful to the fairness community as they elucidate the need for multiple fixed-seed runs when evaluating the merits of a novel method. However, as the paper mentions in Appendix E, this problem is also easily solvable using, e.g., PyTorch’s deterministic mode. In fact, the paper should conduct the same experiments using the deterministic mode for comparison. For this reason, I do not think that the current contribution is relevant enough to warrant publication at a top-tier conference.

**Questions**

- How does fixed-seed variance compare to the variance induced by varying the random seed?
- Did you run the experiments sequentially for roughly ten months, as indicated on L164, or does this represent the total running time?

**References**

Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary. Evan P. Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. FAT 2019.
limitations_and_societal_impact: The paper adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors investigate the impact of software implementation on the fairness variance in the training of DL models. They perform an empirical study of DL fairness with 27 techniques (22 mitigation techniques and five baselines) on four popular datasets (CelebA, MS-COCO, imSitu, and CIFAR-10S) and three DL networks (ResNet-18, ResNet-50, and NIFR), measured by seven popular bias metrics. Their results show that models considered fair for one specific run might not be fair in another fixed-seed identical training run. There are up to 12.6% fairness variance across identical training runs with identical seeds. Their literature survey shows that while fairness is gaining popularity in AI conferences, only 34.4% of the papers use multiple identical training runs to evaluate their approach.

main_review: # Strengths

. Target an important problem in deep learning.

. Extensive experiments.

# Weaknesses

- Novelty is a bit weak
- Limited scope of the literature survey

This paper describes an empirical study to quantify the impact of software implementation on the fairness and its variance of DL systems. It targets an important problem of Deep learning (DL) systems, which demand fairness. The paper finds that DL software implementations introduce fairness variance: identical DL training runs with a fixed seed produce different results. Such variance could make DL models and networks violate fairness compliance laws, resulting in negative social impact. The authors performed extensive experiments: they evaluated the fairness variance of 27 techniques using seven bias metrics (a total of 189 experiments). Statistical significance of our findings were also analysed.

The novelty of this work is a bit weak as the impact of software implementation on the variance of DL systems has already been described in some recent work.  

For the software implementations, the paper did not consider the impact of various deep learning frameworks, 
such as Tensorflow, Pytorch, MXNet, etc. Do different DL frameworks affect your findings? Furthermore, there are also many other dependent software packages such as Python, SciPy, etc.

The literature survey only covered 32 papers on DL systems’ fairness, from a few conferences (NeurIPS, ICML, AAAI, CVPR, ICCV, ECCV, FAccT). It is not clear if these papers are complete and representative enough.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This is a very empirical paper that documents an interesting and surprising fact: fixed-seed identical training runs (FIT runs) can have surprisingly different fairness characteristics. 

I think this is an important fact that deserves to be known widely in the community and will likely be used to justify a lot of follow up work. The paper also includes a systematic literature survey to argue that this is both a little known fact and one that can affect the validity of some existing results. 

That said, while the specific focus on fairness is novel, and I don't disagree with the authors that more people need to be aware of the variance that can be introduced by various sources of implementation-level non-determinism, I think what the paper claims as implications of these results are mostly things that people should know to do already. For example, the suggestion that people should use statistical tests to check the validity of any proposed improvements is a good one, but it is not one that follows from this research. Given that training algorithms are fundamentally stochastic, applying statistical tests to confirm claimed improvements should be a requirement even if implementation-level non-determinism were not an issue. 
Decision: Accept (Poster)