Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces computational-unidentifiability, a concept for fair representation learning which essentially makes it challenging to identify the sensitive attribute of a sample within cluster, by ensuring that the two distributions remain the same. An associated metric called the fair fretchet distance is proposed. An algorithm for clustering that minimizes the KL-divergence based fairness loss (ensuring the within cluster samples) have low KL divergence under gaussian assumptions is demonstrated to have better fair fretched distance on benchmark datasets with improvement in downstream fairness tasks
strength_and_weaknesses: 1. Definition of distributional independence that is distinct from statistical independence is not clear, and seems unnecessary given that downstream task is mostly only minimizing loss terms in terms of the KL divergence.

2. The choice of KL-divergence to impose a fairness constraint is odd, and very specific. Why isn't a non-parametric distance like MMD preferred?

3. Authors don't provide any intuition on why the fair fretchet distance is a good metric beyond bounding it with the fretched distance.

4. Computational unidentifiability seems to imply that the distribution for the sensitive attributes within cluster are essentially the same. Hence I am a bit concerned about why only a limited clustering algorithm based on KL is proposed. Many distributions don't have closed form estimators, and non-parametric estimates are challenging. On the other hand, scaling MMD seems much easier.

5. More baselines could be added such as the ones authors themselves cite - fair-centroid?

6. Overall clustering being ill-defined, one could also compare all fair representation learning methods that impose direct independence between the learned representation and the outcome (since the goal seems to be to improve downstream classification fairness)


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written. I have some conceptual questions that remain unanswered

Quality: The contribution is interesting overall the paper is addressing an important problem. I do feel the number of baselines is insufficient.

Novelty: Reasonably novel, though some choices made require clarification

Reproducibility: Code is provided, I think experimental details are reasonable for reproducibility
summary_of_the_review: I am concerned that the contribution is lacking some conceptual nuances and choices made for the algorithm are not fully justified. Hence I will review the rebuttal to see if authors are able to address all concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a method for deep clustering which attempts to match the distributions of several sensitive groups within each cluster. They propose a metric/loss based on Frechet distance, and incorporate this into a learning framework. Empirically, they show how this loss can result in clusters with the desired property.
strength_and_weaknesses: I think fair clustering is a useful direction and there are some novel ideas in here, but overall I'm not sure I totally grasp the direction this paper is heading in.

Some feedback:
- Introduction: the introduction makes some claims about novelty that I don't think are supported. The authors introduce "computational-unidentifiability" which they later define as the property that "no external classifier can distinguish which sensitive group the sample is drawn from" (Sec 3, Motivating Problems). They claim that this is a novel approach, and that in general, "fairness concerns in the [unsupervised learning] space have been overlooked". I think these novelty claims are misguided given the literature on adversarial fair representation learning (see Edwards & Storkey [1] for the first instance of these ideas, and Madras et al [2] for a thorough examination of an adversarial fairness metric which encapsulates this idea. There is also a literature on fair representation learning outside of adversarial notions which these novelty claims seem to ignore. Now, as far as I know, the adversarial fair representation learning literature doesn't usually consider clustering, and I believe that application is novel - however, the introduction and novelty claims of the paper should be rescoped accordingly.
- Introduction: the exposition is a little bit confusing. Re: clarity, the introduction really only mentions clustering in passing, instead discussing unsupervised learning/representation learning more generally. However, I think it would be good to focus the introduction much more clearly on clustering, given that this paper is only concerned with clustering.
- Fair Frechet Distance: I'm a little bit confused by the metric Fair Frechet Distance, and which corresponding notion of fairness it is intended to enforce. As far as I can tell, it seems like FFD is intended to minimize distributional distance between groups within a cluster. However, for the notion of a fair representation, and if we are concerned about a downstream classifier identifying these groups, shouldn't we also be concerned about the distribution of groups *between* clusters? I think there should be a little more work done in the paper to justify why this is a good metric for fairness specifically.
- Sec 5: I think reading between the lines I can figure out how the encoder and clustering objective work together, but I think it would be helpful to have some more explanation here - up to this point there is no mention of an encoder, and it would be helpful for those not so familiar with the literature to explain how it is incorporated (rather than just doing clustering on fixed representations).
- Table 2: I'm not quite sure what the takeaway is from the MTFL column - it seems that the proposed methods are much worse than a number of the baselines on FFD. There should probably be a bit more explanation about what is happening here.

Notes:
- Sec 6.1, 2nd paragraph: "desired clustering attribute is gender" - can you explain what a desired clustering attribute is here? 
- some sloppiness in language throughout - for instance, bottom of page 8: "It is noticeable that the representation from DFC is highly identifiable compared with ours". I think what the authors are trying to say is that it is easier to identify, within a cluster, the sensitive attribute of interest from the DFC representations than from their proposed method's. Being a little bit more precise with expressing these notions I think will go a long way to improving the clarity of the paper
- I find section 6.4 a bit confusing - I'm not quite sure what the experimental protocol is here, and what is being compared 


[1] Edwards, Harrison, and Amos Storkey. "Censoring representations with an adversary." arXiv preprint arXiv:1511.05897 (2015).
[2] Madras, David, et al. "Learning adversarially fair and transferable representations." International Conference on Machine Learning. PMLR, 2018.

clarity,_quality,_novelty_and_reproducibility: Clarity: I think the clarity of this paper could be improved greatly, particularly in terms of its proposed notion of fairness
Quality: it's hard to tell exactly the quality of the work - I think there are useful pieces in here conceptually and empirically, but don't really know how to best evaluate the whole of it
Novelty: I think there is novelty in the clustering application of a fair representation learning method, but I think the novelty is greatly overstated by the introduction of the paper
Reproducibility: some experimental details are missing, for instance, how the clustering procedure is run, and its hyperparameters
summary_of_the_review: I think this paper could use some improvement in terms of the communicating the usefulness of the fairness notion they are considering, as well as general clarity improvements around the exposition, method and experiments. As such, I'm recommending a rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to address the fair representation learning method for unsupervised learning. They define a notion of fairness, computational-unidentifiability, which suggests the fairness of the representation as the distributional independence of the sensitive groups. They also propose a new fairness metric, FFD, to quantify the computational-unidentifiability. Then, some comparison experiments are also conducted to validate the effectiveness of the proposed method in various downstream tasks. 
strength_and_weaknesses: Strengths:
1. They propose a novel metric that quantifies fairness in representation space. And they provide rigorous analysis of the theoretical property and complexity of the fairness metric. 
2. They propose a framework for fair representation learning for downstream tasks.
3. They compare the proposed method with the state-of-the-art fair method in the literature (Balance). 

Weaknesses:



clarity,_quality,_novelty_and_reproducibility: 1. The reproducibility of the work is good, according to the description and supplementary materials. 
2. This paper has a clear clarity of definition. 
summary_of_the_review: This paper provides a set of solutions for fair representations, including a novel metric and a framework. Some experiments are also conducted. I think it is good enough for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a notion and metric for fair representations. While there were some strengths to the paper, there was no strong support for accepting it. Instead, reviewers brought up several concerns: 1) Overstated novelty in the introduction. 2) The clarity of the paper is not yet at the level expected for publication. 3) Several choices in the algorithm are insufficiently justified. 4) The paper has insufficient comparisons with baselines. As such, I recommend rejecting this paper.
justification_for_why_not_higher_score: There was no strong support for accepting this paper and several concerns among reviewers.
Decision: Reject