Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper builds on the "bias amplification" aspect of fairness in machine learning literature i.e. the tendency of models to make predictions that are biased in a way that they amplify societal correlations. The paper claims three major contributions: a metric, discussion about the dependence of bias measurements on randomness, and a normative discussion about the use of bias amplification ideas in different domains. 

Overall I find the metric as the only major contribution of the paper, and below I will explain why.

The BiasAmp metric makes a significant contribution in terms of fixing the drawbacks of the previously proposed metric from Zhao et al. 2017(BiasAmp_MALS). It would be more effective if the work also included a study such as Zhao et al demonstrating how to mitigate the bias as measured by the BiasAmp measure. 

The discussion around--the usage of error bars because of the Rashomon effect seems incomplete and almost trivial. First, in my opinion, it is indeed necessary to have error bars if there are metrics whose measurements vary across different runs--regardless of whether that measure was being optimized on or not. But a more nuanced idea would be either: (a) usually a fairness metric is used as a guidance of whether a specific model is deployable or not, rather than being a property of a training method unless however, an ensemble of a number models trained by the same method is being used in deployment. (b) Fairness metrics may be used as a model selection criterion: since the fairness metric is not being optimized upon but a proxy of accuracy, it just means that, out of the models with equally high accuracy, we need to choose the model with the least bias (as measured by a trustworthy metric). 

I find the discussion around the 'consistency' of different metrics incomplete, where average precision (AP) seems to rank models consistently while BiasAmp and others don't. BiasAmp is a pretty sophisticated metric and because of the use of conditional probabilities of all kinds prone to pitfalls such as Simpson's paradox when measuring bias. I would be curious if it were robustly tested e.g. another experiment such as Fig 4 where the authors control the amount of the bias by tuning at the source of the bias. 

The discussion around the use of bias amplification in terms of prediction problems where the outcome is chance-based is a little confusing and it does not provide a fresher perspective of discussion already in the fairness literature around understanding the significance of inherent 'uncertainty' or 'randomness' in application domains other than vision (and probably language). For example, the Fair ML book (Barocas, Hardt, and Narayanan 2017) does mention uncertainty in the context of such applications (on page 34, 56). 

Overall, I am not very convinced that the paper should be accepted unless fellow reviewers think strongly otherwise. However, I think the paper has the potential to be a more complete and important contribution with a more comprehensive study around the technical contributions and clearer discussion about the normative contributions.
review_title: Paper deals with Bias Amplification aspect of FairML but lacks a more comprehensive study of the metric and discussion of the normative perspectives. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper talks about an interesting aspect in algorithmic fairness meaning bias amplification; however, there are couple of concerns that I will raise below:

1. The paper mainly addresses shortcomings of a previously mentioned simple concept, bias amplification, brought up in Zhao et al 2017. In this regards, I am not sure about extensiveness of novelty and contribution of this paper along with its technical rigor. One suggestion would be to add some theoretical analysis maybe in Validating the metric session.

2. Although Zhao et al's paper is a famous paper in NLP domain, not much attention has not been given in the pure algorithmic fairness and using it as a measure along with other well known measures, such as statistical parity or EO. This makes me even more concerned about an extension to this work.

3. The discussion on error bars and their need sounds like an intuitive concept which authors spent a section on it. Other sections could have been made richer experimentally instead of this section maybe.

4. Analysis of benchmark fairness datasets remains unexplored. Mostly vision datasets are explored, but authors could have done some studies on famous fairness benchmark datasets as well. This can introduce bias amplification concept to the fairness community and make it more comparable to other well known fairness measures and more acceptable to the community.

5. There are some claims that I do not find accurate in the paper as follows:
5.1. "By definition, this problem stems from the algorithms, and can not be attributed to the dataset" -> we can not 100% say this! The bias in the first place is coming from data itself in this case!
5.2. "A trait of bias amplification is that it is not at odds with accuracy, unlike many other metrics ..." -> where is the proof for this? need strong evidence for this claim.

Overall, this paper addresses shortcomings in a previous work based on a simple bias amplification metric. The lack of technical rigor and theoretical guaranties for some claims makes this paper not a strong candidate. Inclusion of some fairness benchmark datasets can make this paper more strong.
review_title: Lack of Technical Rigor

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper proposes a new metric for quantification of bias amplification that aids in interpretability of models. In particular, the paper offers useful insights concerning the meaning of bias amplification across varying contexts or use cases and prescribes the use of confidence intervals for better validation of various fairness metrics.

Positives:

1. Technical Sophistication
The paper throughly examines the shortcomings of a previous metric used for computing bias amplification and proposes a new metric that overcomes the limitation of prior metric while still retaining the desirable properties. In particular, the authors clearly state the benefits of their method as described in Sec 3.4. 
2. Presentation: 
There is sufficient clarity in the paper facilitating easy comprehension. 
3. Experiments and analysis:
This is the biggest strength of the paper. A thorough investigation makes this paper compelling. The authors also discuss the limitations of the proposed metric. The running example (fig 1 ) considered in the paper is interesting, especially given that such stereotypes are not considered often. 

Concerns:

1. The recommendation for confidence interval is justifiable, but lacks a proper grounding. 
2.  The authors use causal notations for bias amplifications with the arrows from A to T or vice versa. While it is ok in terms of notation, it raises the question of how causal claims can be justified by merely thresholding  as opposed to interventions . In this context, more elaboration of the metric (Sec 3.3) would be useful.

Overall comments:

I enjoyed reading this paper, and it offered some interesting insights. I believe it will be valuable in analysis of bias across different use cases in a manner that is interpretable.
review_title: An insightful perspective about bias amplification backed with sound metrics and analysis

Meta-review: The authors study bias amplification [Zhao et al, 2017] and propose an improved metric for measuring it. The authors also discussed normative issues in bias amplification (predicting a sensitive feature), and how to measure amplification when we do not have labels, or where labels correspond to uncertain future events. While the reviewers acknowledged the importance to study bias amplification, normative measures and social context, they raised several important concerns: 

(1) limited technical contributions (R3 and R4) -- see reviewers’ concerns that the metric is the only technical contribution; one possible suggestion is to propose a mitigation strategy based on the proposed metric similarly to [Zhao et al, 2017]; 

(2) ‘the usage of error bars because of the Rashomon effect seems incomplete and almost trivial’ (R3, R4), ‘lacks a proper grounding’ (R1) -- see two suggestions by R3 how to revise; these suggestions have not been discussed in the rebuttal; 

(3) empirical evidence lacks a controlled scenario of tuning the bias source to evaluate consistency (suggested by R3) and is limited in the context of algorithmic fairness benchmarks (R4) -- this has been partly addressed in the rebuttal; 

(4) normative contributions and broader discussions are oversimplified – see R3’s comments and suggestions on how to better position the paper.

Among these, (3) and (4) did not have a major impact on the decision but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed work and were viewed by AC as critical issues. 
A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. It needs technical strength, more empirical studies and polish to achieve the desired goal. We hope the detailed reviews are useful for revising the paper.

Decision: Reject