Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles a fair classification problem with an invisible demographic, a situation where the records who have some specific target labels and sensitive attributes are missing. In this setting, the authors introduce a disentangled representation learning framework to make the resultant classifier fair by taking advantage of the additional dataset, context dataset. They demonstrate by the empirical evaluations that the proposed disentangled representation learning algorithm success to mitigate unfair bias by utilizing the perfect dataset, a dataset in which the target label and sensitive attribute are independent. Usually, the perfect dataset is unavailable; hence, they introduce a method to convert the context dataset into the perfect dataset. The authors also show that even if the context dataset is not perfect, the presented method successes to mitigate an unfair bias.

The strong points of this paper are as follows:
- This paper introduces a potentially interesting problem, the invisible demographic. 

The weak points of this paper are as follows:
- The experimental results have a high variance. Hence, they are weak to support the significance of the proposed algorithm.
- The motivation of the proposed method is unclear. Some existing methods already solve most of the crucial situations considered in this paper.
- This paper lacks a comparison with the important related method.
- Presentation is poor. I cannot follow the description of the algorithm.

My recommendation is rejection. The main reason is that I have concerns about suspicious behavior in the experimental results. Also, the proposed method is not well-motivated, and its merit is unclear. 

I am very suspicious about the experimental results. The standard deviations for the fairness metrics shown in Table 1 and Table 2 are considerably high. Why can we believe the successful mitigation of unfair bias of the proposed method from these results? Even if I believe the reported values, due to the large standard deviation, we cannot say the authors' method outperforms the others but can only say it is competitive.  I don't think this is a significant result.

Parts of the invisible demographic problem are already solved. For example, a situation where records in some classes are missing is solved by utilizing semi-supervised learning techniques, e.g., Hsieh et al. Classification from Positive, Unlabeled and Biased Negative Data. In ICML'19. For a situation where the sensitive attributes are missing, there are several works, including
- N. Kallus et al. Residual Unfairness in Fair Machine Learning from Prejudiced Data. In ICML'18.
- A. Coston et al. Fair Transfer Learning with Missing Protected Attributes. In AIES'19.
It is a rare situation where records with a specific combination of the target class and demographic group are missing. These existing methods already solve other cases. Therefore, it is unclear that the proposed method has merits compared to the existing ones.

There is a fair classification method based on disentangled representation learning:
- E. Creager et al. Flexibly Fair Representation Learning by Disentanglement. In ICML'19. 
Because this method and any fair classification methods can apply to the problem tackled by this paper, it is necessary to compare the proposed method with them. I know these methods are not designed to work in the invisible demographic situation; however, it is unclear if they do not work in the situation without empirically evaluating them. 

I cannot understand the introduced objective function in Eq. 10. What is the meaning of $f(z_y \subset \mathcal{X}_{perf})$ and $f(z_y \subset \mathcal{X}_{tr})$? While the function $f$ takes $x$ as its input, it takes a boolean value in Eq. 10. 

What is clustering accuracy? Its definition is missing.

### Minor comments
- While I understand the situation where the whole sensitive attributes are missing, I wonder if it is a realistic situation that a part of the target label and sensitive attributes are missing. Is there a concrete dataset that invisible demographic situation occurs?
- I cannot make sure about the notation of $\mathcal{M}_{y=1,s=0}=\emptyset$. If my understanding is correct, the set $\mathcal{M}$ (omit subscript $y=1,s=0$ because it doesn't work) comprise of whole data points whose target label and sensitive attribute are $y=1$ and $s=0$, respectively. It involves not only the target data points but also unobserved data points available in the world. From this perspective, $\mathcal{M}=\emptyset$ means that there are no people whose target label and sensitive label are 1 and 0, respectively, in the world. In this case, we cannot construct the context and deployment sets that satisfy Eq. 3 or Eq. 4.
- Typo on page 3, first paragraph:   but in in contrast to ->  but in contrast to
review_title: Suspicious experimental results and unclear merit

final_score: rating: 6: Marginally above acceptance threshold
review_content: ############# Summary of contributions ##############

This paper introduces the problem of enforcing group-based fairness for “invisible demographics,” which they define to be demographic categories that are not present in the training dataset. They assume access to a “context set,” which is an additional unlabeled dataset that does contain the invisible demographic categories of interest. They further provide an algorithm for enforcing fairness on these invisible demographics using this context set. 

Specifically, their contributions are:

- Algorithmic: They provide an algorithm for enforcing fairness on these invisible demographics. This algorithm involves first applying clustering methods on the context set to “balance” it, followed by disentangled representation learning and on the “balanced” context set.

- Empirical: They provide experiments on two benchmark datasets (colored MNIST and Adult) comparing their proposed method to multiple baselines. 

############# Strengths ##############

- The paper is organized well, and the problem of “invisible demographics” is described and motivated well using concrete examples. 

- The architecture of the proposed method is documented clearly in Figure 2. 

- Their architecture builds on state of the art techniques such as DeepSets (Zaheer et al. 2017). Using DeepSets, the discriminator in their architecture estimates the probability that a given batch of samples, as a set, has been sampled from one distribution or the other. Preserving the set invariance to permutations is useful here, and different from a typical GAN discriminator.

- The baselines in the experiment section are thorough. It’s useful to see a comparison between their clustering + balancing + disentangling method and the baseline methods of ZSF, which has balancing + disentangling but no clustering, and ZSF + bal. (ground truth), which has ground truth clusters + balancing + disentangling.

############# Weaknesses ##############

- The experiments section does not describe the implementation of the comparison to Hashimoto et al. 2018. Notably, the methodology of Hashimoto et al. 2018 is not specifically meant to enforce equality of acceptance rates, true positive rates, or true negative rates -- it only minimizes the worst case loss over unknown demographics. 

- The authors do not provide any description of hyperparameters tuned, or any use of a validation set for hyperparameter tuning. I could not find this in the appendix either. In fact, on page 7, they say that they “repeat the procedure with five different train/context/test splits”, which suggests no validation set. The parameters for the clustering methods are not given, and I find it hard to believe that no hyperparameters were tuned. Can the authors specifically provide the hyperparameters used, whether/how they were tuned, and any validation methods used (whether it be a validation set or cross validation)? 

- The experiments are all done with binary protected groups: purple vs. green for the colored MNIST dataset, and male vs. female for the Adult dataset. Furthermore, these groups are not hugely imbalanced in the context set to begin with. This makes the clustering task easier. It would be interesting to see experiments with protected groups with more than two categories. For example, in the Adult dataset, the race feature is highly inbalanced, with a very small proportion of examples labeled as Asian-Pac-Islander or Amer-Indian-Eskimo. It would be interesting to see how the clustering techniques compare when the context set includes more than two protected categories, there is initial strong data imbalance between those groups, and the “invisible demographic” has relatively few data examples in the context set. This may not be entirely necessary for acceptance this round, but could be an interesting future experiment.

The notation is in multiple cases unclear/inconsistent, possibly due to typos. Examples listed below:

- In the last paragraph on page 5, the notation and description of the support is confusing and not well defined. First, \mathcal{S} and \mathcal{Y} are themselves sets as defined in Section 2.1. Can the authors more specifically define what they mean by Sup(\mathcal{Y}_tr)? Is this the set of elements from \mathcal{Y} that are contained in the training set? If so, why not just notate this as \mathcal{Y}_tr alone? The additional “Sup” notation is confusing and appears unnecessary. Furthermore, what do the authors mean when they say, “we wish to use Sup(\mathcal{S}_{ctx} \times \mathcal{Y}_{ctx}) \ Sup(\mathcal{S}_{tr} \times \mathcal{Y}_{tr}) as the training signal for the encoder”?

- [Top of page 6: “whenever we have |S| > 1”] -- What does this notation mean? Is this the absolute value of the random variable S? This doesn’t quite make sense given that S was previously stated to be a discrete-valued protected attribute, which could be a vector with p entries. The next statement of this corresponding to the “partial outcomes” setting is thus also unclear.

- [Section 2.2: “c_i = C(z_i)”] -- What is z_i here? Is z_i the vector of (z_s, z_y) for the input features x_i? 

############# Recommendation ##############

UPDATE (after author response): I appreciate the authors' response. The inclusion of the hyperparameters are helpful. I also think it's an improvement that the authors added a comparison to ZSF+bal.(ground truth) to the Adult experiment.

I still have a question about the experimental comparison to Hashimoto et al. (called "FWD" in this paper). Is the version of "FWD" implemented in this paper using exactly the same fairness criterion as in the Hashimoto et al. paper? If so, am I correct in saying that the "FWD" comparison in the experiments section does not directly constrain for any of the measured AR ratio, TPR ratio, or TNR ratio? The authors should clarify this in a later version.

Overall, I'm willing to raise my score to a 6, but still think the paper is borderline. The paper could still use some improvement in covering related work on the problem of fairness where the protected attributes are not fully known (including the references I suggested).

------------- OLDER RECOMMENDATION BELOW -------------

Overall, my recommendation is 5: Marginally below acceptance threshold. The paper states an interesting and practically relevant problem of enforcing fairness with “invisible demographics.” The methodology is overall well documented, and the experimental baselines make sense. However, the implementation detail in the experiments section is severely lacking, including description of hyperparameters/validation methods and implementation details for the comparison to Hashimoto et al. If the authors provide some of these details and answer some of my notation questions, then I would be willing to raise my score.

############# Questions and clarifications ##############

- Why is there no comparison to ZSF+bal. (ground truth) on the Adult dataset? 

- Can the authors clarify what the ZSF alone baseline is doing in the experiments section? It’s not written super clearly in the text. Does ZSF alone simply replace the perfect set in Figure 2 with the context set?

############# Additional feedback ##############

- Below I’ve listed some additional related work in the setting where protected attributes are unknown. This is not factored into the review, as these settings seem different enough and some of these works are recent. 

Lamy et al. Noise-tolerant fair classification. NeurIPS, 2019.

Awasthi et al. Equalized odds postprocessing under imperfect group information. ICML, 2020.

Wang et al. Robust Optimization for Fairness with Noisy Protected Groups. arXiv:2002.09343, 2020

- [page 3: “We can all agree that this sounds unfair”] -- nit: this wording seems unnecessarily strong to me. Let’s not claim that “we would all agree” on something, especially when the meaning of unfair has not yet been defined.

- [page 5]: There appear to be multiple typos in the paragraph following equation (10), where the variables V, Q, K are not written in math mode, and are instead just capital letters in the text.

review_title: Well organized paper on a relevant problem, but lacking in key experiment details.

final_score: rating: 5: Marginally below acceptance threshold
review_content: #Summary

This paper studies zero-shot fairness where the demographic information is partially unavailable, but assuming the existence of a context dataset that contains all labels x all demographics (including the invisible). The paper proposes a disentanglement algorithm that separates information of the label and demographics, under two zero-shot settings: 1) learning with partial outcomes: both labels and both demographics are available, but for one of the demographics only negative outcome is present; 2) learning with missing demographics: one of the demographics is completely missing.

#Pros
- Zero-shot fairness is a very important topic under many practical settings, where the demographic information can be (partially) missing due to sampling bias or privacy reasons.
- The two zero-shot settings presented in this paper are both very interesting, and the paper did a good job decomposing the two scenarios in the methods and experimental section.
- The paper is clearly presented, with careful analysis over each of the proposed component, with proper ablation studies.

#Cons
- The biggest concern I have is the clustering part of the context set into a perfect set. This seems to be a prerequisite for the disentangle algorithm to perform well. However, there is no guarantee over the clustering quality, and this is partially reflected in the experiments (table 1 & 2) as well. For example, while ranking-based clustering achieves reasonable clustering accuracy, k-means seems to be rather bad for certain datasets (e.g., Adult Income). In addition, how does the distribution of the label x demographics on the context dataset affect clustering quality? I can imagine under extreme cases, if the distribution is very skewed (some of the label x demographic has very scarce data), then it is hard to get good clusters, which is very likely to happen in practice if the training distribution is already skewed.
I think some further analysis on this is required, e.g., how the cluster quality differs w.r.t. different retained proportions of each quadrant.

- The experimental results seem to present different trade-offs for the proposed approaches. There doesn't seem to be a single algorithm that has a clear better performance compared to the baselines. E.g., for colored-MNIST ZSF seems to work a bit better, but for Adult Income MIM+bal, and FWD seem to work better. The performance also varies a lot across different fairness metrics as well. 

- Although the topic of zero-shot fairness is very important, the end-to-end setting in this paper is a bit artificial. It requires two things, 1) both label $y$ and demographic $s$ are present in the training data, although some of the quadrants are allowed to be missing; 2) there exists a context set that has all quadrants available for $y$ and $s$, thus can be used for balancing and learning the disentangled representations. I wonder how realistic this setting is in practice. It is very likely that 1) is true in real-world but the requirement of 2) makes the setting a bit constrained, what if during deployment time, no context set is available and online inference (for each incoming individual) is needed? Or, what if I have a context set, but some of the quadrants are also missing, and even worse, the missing quadrants are different from the ones missing in training?

#Over recommendation

I think this paper studies a very interesting problem but some further analysis, e.g., how the distribution over the context data affects the results, and how to make the algorithm work reliably better in practice, is needed. Overall I think this is a borderline paper.


#Minor comments and questions
- In the experiments, for colored-MNIST, a comparable portion for each quadrant is retained for the context dataset, have you tried different retained portions and how does that affect clustering quality? Have you tried some of the more extreme settings (e.g., more skewed distribution over |S|x|Y|) and will you still obtain reasonable clusters?
- I didn't find how the context dataset is constructed for Adult Income, could you provide more information on this?
- How are $\lambda_1$ and $\lambda_2$ chosen in the experiments?
- Some of the error bars in table 1&2 are rather large, could the authors further clarify which set of the results are statistically significant? 
review_title: Interesting work on zero-shot fairness with partial demographics

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
- This paper proposed a problem in algorithmic fairness where labeled examples for some demographic groups are completely missing in the training dataset and still the goal is to make predictions that satisfy parity-based fairness constraints.
- The method developed to solve this problem uses a "context" dataset with unlabeled data but containing individuals from all demographics to construct a 'perfect dataset' and 'disentangled representations'

Quality
- This work appears to have only a superficial understanding of the field of algorithmic fairness, hence proposing a problem that in my opinion is artificial. In the case where a dataset has *zero* labeled examples for some demographic groups, this is such an extreme situation that it is a clear red flag that there is a large bias in the collection process and/or the data collection design was poorly done---what is the justification for continuing to use this dataset as is? Is there any real life scenario where one is forced to use this problematic dataset (this could be irresponsible, even unethical), instead of trying to get labels for the "context set" (which is assumed to be available!) or rethinking the data collection process? Clearly one ought to go back to the drawing board in this imagined worst case situation.


References:
  - Kate Crawford. The hidden biases in big data. Harvard Business Review, 1, 2013. 
  - Kate Crawford. The trouble with bias. NIPS Keynote https://www.youtube.com/watch?v=fMym_BKWQzk, 2017.
  - Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. Datasheets for Datasets


- The authors write: "If the model relies only on the incomplete training set, it is not unreasonable to expect that the model to easily misunderstand the invisibles. We can all agree that this sounds unfair, and we would like to rectify this." without any proof or mathematical argument. "this sounds unfair..." is an unrigorous and uncritical statement that doesn't contribute any deeper insight, nor does it engage with existing work on what "unfairness" constitutes. It also does not explain why the paper's method (to massage a clearly problematic dataset) is any less unfair.


The paper also does not cite some related work on algorithmic fairness with missing demographics, e.g.

Recovering from Biased Data: Can Fairness Constraints Improve Accuracy? Avrim Blum∗, Kevin Stangl†


- I'm concerned that the paper calls the missing demographic groups "the invisibles" and then proceeds to still champion the use of the clearly flawed dataset. The algorithm has only intuitive justifications and so does not convince that the missing demographic groups would not be still somehow disadvantaged or find this procedure extremely unjust. The paper does not discuss any of these problematic aspects.



Clarity
- The lack of theoretical guarantees for the algorithm makes it unclear what assumptions are needed for the algorithm to do something meaningful in 'rectifying' the extremely large missing pieces in the original training dataset.

- The explanation for how a "perfect dataset" is constructed is vague (section 2.2). Since the clusters are not explicitly named (i.e. no labels), how is this a "perfect dataset", defined as one where the labels y and group s are independent? Is there any way to check the independence?



review_title: Problem proposed has weak and/or problematic motivations, unsure about significance of contributions

Meta-review: The paper studies the problem of satisfying group-based fairness constraints in the situation where some demographics are not available in the training dataset. The paper proposes to disentangle the predictions from the demographic groups using adversarial distribution-matching on a "perfect batch" generated by a clustered context set.

Pros:
- The problem of satisfying statistical notions of fairness under "invisible demographics" is a new and well-motivated problem.
- Creative use of recent works such as DeepSets and GANs applied to the fairness problem. 

Cons:
- Makes a strong assumption that the clustering of the context set will result in a partitioning that has information about the demographics. This requires at the very least a well-behaved embedding of the data w.r.t. the demographic groups, and a well-tuned clustering algorithm (where optimal tuning is difficult in practice on unsupervised problems) -- but at any rate, as presented, the requirements for a "perfect batch" is neither clear nor formalized.
- Lack of theoretical guarantees.
- Various concerns in the experimental results (i.e. proposed method does not clearly outperform other baselines, high variance in experimental results, and other clarifications).

Overall, the reviewers agreed the studied problem is new, interesting and relevant to algorithmic fairness; however, there were numerous concerns (see above) which were key reasons for rejection.

Decision: Reject