Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the fairness of deep learning models in classification tasks with a large number of intersectional groups. The paper has two primary contributions:

1. The paper includes an empirical study that shows the inherent difficulty of this setting (i.e., due to the lack of labels), and the limitations of existing approaches.

2. The paper presents a new approach to train deep models to settings with many intersectional groups called Knowledge Distillation of Independent Models as Regularization (DIR).
main_review: ### Strengths

- Focus on an important and understudied problem. 

- Paper presents a new approach to handle intersectional attributes: Knowledge Distillation of Independent Models as Regularization (DIR).

### Weaknesses

- [Major] Paper evaluates methods on datasets where we have no ground truth. These findings should be confirmed on a synthetic dataset where we have ground truth labels (e.g., binary classification task with $k = 1, 2, 4$ binary group attributes).

- [Major] Key findings of the empirical study are based on summary statistics that do not capture or reflect differences in accuracy across intersectional groups (see e.g., Figure 1, Figure 3, Tables 2 and 3). For example:

     (*) The ``reweighted accuracy" with k = 128 groups does account for the performance of all intersectional groups. However, it would be unable to flag instances where an intersectional group has an accuracy of 0\% since any group affects at most 1/128 of the total performance. 

     (*) Measures such as Bias Amplication and "Intersectional Bias Score," are surrogate measures that suffer from their own issues (as discussed by the authors on Page 4). The informative-ness of these issues is also hindered in that they represent population-level statistics.

      I would recommend would be better to evaluate methods in ways that do not involve summary statistics at all (e.g., via a plot showing the distribution of group-specific accuracy). I would also recommend the authors use measures of group-level performance (e.g., worst-case accuracy over groups, # of groups with substantial bias). 

- [Minor]: The paper is missing references to some of the earlier work on this topic, namely: (1) work on training single classifiers with fairness guarantees over an infinite number of intersectional groups [Kearns et al - ICML 2018](https://proceedings.mlr.press/v80/kearns18a.html); (2) work on decoupled training across many intersetional attributes (see e.g., [Ustun et al at ICML 2019](http://proceedings.mlr.press/v97/ustun19a.html)).
summary_of_the_review: This paper studies an important problem. In short, there is little work on intersectional subgroups, even less on deep learning. In this context, I see the empirical study as an opportunity to make a contribution as it can highlight previously unknown issues. 

My recommendation is based on the following issues: (1) key findings are not well-supported by the empirical results (see comment above); (2) the proposed approach DIR is underdeveloped and under-evaluated in the text. I think that both issues could be addressed, but should be subject to another round of peer review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper conducts an empirical analysis of existing bias mitigation methods on two large datasets CelebA and ImageNet where there are multiple sensitive attributes and some unavailable protected labels. The results show the existing can mitigate intersectional bias at scale but the unlabeled methods generalize poorly. This paper further proposes a knowledge distillation of independent models as regularization method (DIR) which is able to augment into other bias mitigation algorithms.
main_review: Strength

1. The empirical analysis of existing bias mitigation methods on two large datasets with multiple sensitive attributes/partially labeled protected attributes is very interesting. It shows the scalability of existing methods and the potential challenges when applying those methods to large-scale datasets.

2. A new method for group-specific models is proposed to augment other mitigation algorithms.


Weakness:

1. The novelty of this work concerns me the most. The empirical analysis is interesting but far from the expectation of the standard of ICLR. In addition, the proposed DIR method is incremental for model simplification of bias mitigation. It would be considered as transferring the existing method **Domain-Independent model** into a smaller one whose inference complexity is less than the original version, through incorporating the knowledge distillation approach. The distillation process requires |G| pre-trained models for |G| groups.

2. The claimed O(1) complexity is implausible because it assumes the availability of G group-specific models. It seems the proposed method is not able to reduce the complexity without G models. ~~In addition, it is unclear how to integrate multiple classifiers $h_g$ into Eq (6) and (7) because $\lambda$ is not model-specific.~~ (Resolved)

3. It is unclear how to integrate the DIR method with unlabeled mitigation algorithms as the empirical results show they are vulnerable to poor generalization.

summary_of_the_review: The empirical analysis is interesting but the incremental contribution concerns me the most. In addition, the proposed DIR method is trivial and contains many uncertainties. I would not root for this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conducts a comprehensive comparison among existing bias reduction methods when there are multiple protected attributes. The authors also propose a knowledge distillation framework for improving fairness on the protected-label-scarce ImageNet dataset.
main_review: Intersectional fairness is a practical and important problem. The authors conduct a comprehensive comparison among existing bias mitigation techniques, which I find valuable. The paper begins reasonably well-written but the clarity deteriorates a bit in Section 3. 

1. My main concern is the fairness metrics considered in this paper. 

First, the authors introduced a notion called worst-case accuracy in Eq. (2) and argued that it was a surrogate metric for equalized odds. However, it is unclear to me why it is an appropriate surrogate. Note that equalized odds measure the *performance disparity* instead of the worst-case accuracy. 

Second, it is unclear to me why the authors computed the surrogate metrics instead of the original measures (e.g., demographic parity) when analyzing the performance of bias mitigating algorithms (see e.g., Fig 2). I understand that training a model with the original fairness measures can be hard. However, once a binary classifier is fixed, verifying demographic parity is just to computing the mean of Bernoulli variables. 

Third, the definitions of the fairness metrics in Section 3.1 are puzzling from a mathematical perspective. 

(1)  the importance-weighted accuracy in Eq. (1) involves 1/|g| inside the expectation. It is unclear what |g| means. Is it the maximal number of subgroups? If so, then Acc_U(h) -> 0 when |g| -> \infty, which does not make any sense. 

(2) the bias amplification in Eq (3) involves Pr(g|h(x)). This conditional probability distribution is unclear to me. First, is h(x) a random variable? If so, the authors should write it as h(X) to be consistent. Second, if h(x) is a random variable, why does the measure s_y only rely on y?

(3) the intersectional bias in Eq (4) is also unclear. Isn’t it \max \min instead of \max \argmin?

I highly suggest the authors carefully revise their definitions of the fairness metrics in Section 3.1! 

2. The primary results in Section 4 are interesting. I wonder if the authors can provide any interpretation of these results. 
summary_of_the_review: Interesting experimental results but the fairness metrics adopted in this paper need to be clarified
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper provides an empirical investigation of deep learning bias mitigation methods, focusing on two problems: intersectionality and missing sensitive attribute labels. The paper evaluates several bias mitigation approaches on the Celeb dataset and on ImageNet, the latter having few sensitive attribute labels. The main takeaways from their empirical analysis is that the best performing mitigation method depends on the context and mitigation methods that operate without using the sensitive attribute do not perform well in terms of intersectionality. The paper also proposes a new distillation-based approach to resolve the runtime complexity problem that models like Domain Independent exhibit. On ImageNet their method significantly reduces bias amplification but there are no differences in group-reweighed accuracy or intersectional bias.
main_review: + The focus of this paper is an important problem: intersectionality in fairness. 
+ The empirical analysis provides insight into how existing methods perform on intersectional groups and highlights fairness challenges that are particularly vexing in the context of intersectionality: runtime complexity and stability. 

- The paper would benefit from a discussion on the hyperparameter tuning. Typically this involves a fairness-accuracy tradeoff which would seem to affect the results. e.g, we can always eliminate intersectional bias by returning a constant classifier. 
-The current definition of the sensitive attributes invokes the “pale male” default. For clarity and inclusivity, I would recommend revising to define the attribute in terms of the attribute, not the value: e.g, “Gender” instead of “Male.” Are each of these binary? It may also be more appropriate to term these sensitive attributes, since protected typically evokes a legal connotation. 
- It would be useful to clarify why we should be concerned about fairness with respect to attributes like “straight hair" or “attractive.” Is this purely illustrative or is there a story the reader should have in mind?
- Accuracy in table 4 seems to be reported on a 0-100 scale but was earlier reported on a 0-1 scale. Why are the reweighed accuracies in Table 4 so low?


Minor:
Define Mean Average Precision. 
Define the error bars in the figures. Are all figures for test sets?
Define synsets
summary_of_the_review: This paper provides an extensive empirical analysis of an important question, how fairness methods perform with regards to intersectionality. The paper would benefit from additional discussion on a few points, like hyperparameter tuning.
review_title: null

Meta-review: The manuscript performs an empirical analysis of existing bias mitigation methods on two large datasets CelebA and ImageNet People Subtree where there are multiple sensitive attributes and some unavailable sensitive attribute labels. The results show that existing methods can mitigate intersectional bias at scale but unlabeled mitigation methods generalize poorly. The manuscript further proposes a knowledge distillation approach which can augment other labeled mitigation approaches.

On the positive aspect, the manuscript studies an important problem: intersectional subgroups on deep learning methods. Reviewers acknowledged that an empirical study on this problem is as an opportunity to make a contribution as it can highlight previously unknown issues.

There are however several major concerns including:
1. Methodological contribution (knowledge distillation) is under-developed, while empirical investigation is interesting but can be further developed;
2. The fairness metrics adopted in this manuscript need to be clarified;
3. A discussion on the hyperparameter tuning, maybe involving a fairness-accuracy tradeoff;
4. The claimed O(1) complexity for the knowledge distillation approach is implausible because it assumes the availability of G group-specific models. This has been clarified in the rebuttal that the claim is only for the inference complexity, and the approach does not improve the training complexity. 

Reviewers also concluded that while the empirical analysis is interesting, the results on CelebA to be of limited use because the sensitive attributes are "purely illustrative." It's not clear that the insights from these illustrative intersectional groups (e.g. big nose & attractive) will hold for groups that are meaningful in a fairness sense.
Decision: Reject