Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the case of group robustness when only some of the group labels are known. They propose a generalized objective for this setting that considers the worst-case possible group assignment subject to the known group labels, and an alternating optimization algorithm for minimizing this objective.
main_review: Weaknesses:
- The main criticism of the paper is that the empirical results are not strong. The simple baselines of ERM and Group DRO (partial) Specifically, on Waterbirds and CelebA, many prior methods (such as JTT, GEORGE) that do not require *any* group labels substantially outperform this paper in terms of worst-group performance. Similarly, on CMNIST, EIIL does not require group labels and outperforms the proposed approach (and I suspect JTT and GEORGE would as well if evaluated).
- While the theoretical results are correct, they are of limited impact. The gap between L_{GDRO} and L_{WDRO}, even the marginal distribution version, is likely to be large because the constraint set quite is large. In particular, for the points without group labels, only the marginal distributions have to approximately match, but given this the assignment of the individual group labels to the individual points is arbitrary, whereas in reality we could use the data features to better predict the probability of a given point having a particular group label. Thus, L_{WDRO} is a fairly pessimistic loss upper bound whose loss landscape may be quite different than that of L_{GDRO}; indeed, this is borne out by the empirical results.

Suggestions:
- While the current results are not strong, perhaps there are settings in which the relatively pessimistic L_{WDRO} loss is indeed preferable to existing baselines such as those mentioned above. Finding such a dataset would be helpful.
- It would also be very helpful to construct a synthetic task on which L_{WDRO} can be proven to be preferable to baselines such as ERM and Group DRO (Partial).

Strengths:
- The ablation studies are well-conducted and provide useful insight.
summary_of_the_review: Overall, the paper has some interesting ideas but unfortunately has limitations, and fairly poor performance in practice. My actual score would be 4, but the rating is unavailable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new variant of DRO, called Worst-Off DRO, to address a ubiquitous real-world setting where group labels are only partially available over the training set. The core idea is to introduce a nested optimization to maximize the worst-off group assignment over the entire training set, given some sensible constraints, hence enabling to alternate the standard DRO optimization. Empirical results are reported to support the proposed methods.
main_review: The proposed method is technically sound, nicely polished, and well presented.

However, I have two major concerns:

1. The paper is motivated by real-world scenarios where only partial group information is available, for example when individuals may choose not to reveal this information due to privacy concerns. These realistic situations should usually result in a *biased* distribution of which samples contain the group annotation, typically due to demographics leading to unknown individual cohorts. However, the entire paper is based on the assumption that group information is *missing completely at random (MCAR)*, in both theoretical section and experimental section. I strongly advise that the authors carefully study other types of missingness, and build ways to understanding the impact of group information availability bias on the learning algorithm proposed. For instance, the algorithm depends on estimating the group distribution $\hat{\mathbf{p}}$ through the available samples, this estimation can be biased if not MCAR, which may render the Lemma 2 unreliable at least. Empirical evidence is also called for when $\hat{\mathbf{p}}$ is misspecified.
2. The paper is missing *very important baselines* from SOTA unsupervised methods for improving worst-group performance, e.g., EIIL/JTT. Ideally, the proposed Worst-Off DRO should perform between EIIL/JTT (lower anchor) and Group DRO (anchor bound), since it requires *more* information than any unsupervised method (admittedly, JTT actually does require a small annotated validation set). But since the paper did not report these comparisons, I can only judge from the numbers across those papers on the same dataset (e.g., CUB), and it seems that Worst-Off DRO performs quite poorly compared to JTT for instance. If this is the case, the authors should list those baselines, and discuss the pros/cons as to the advantages of the proposed methods compare to EIIL/JTT.
summary_of_the_review: The paper is technically sound and well written. However, due to the two major concerns above (unrealistic MCAR assumption and missing important baselines), I can hardly recommend the paper as ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn invariant representations for group membership using partially group labeled data. Thus, they propose the Worst-off DRO, which extends Group DRO by optimizing the loss against the worst-off group assignments in the constraint set. 
main_review: ##########################################

Pros:

+ The theories in this paper are solid and are verified to be right.

+ This paper addresses a more realistic setting about fairness, and their solution seems to make sense.

##########################################

Cons:

- The research question of this paper is clear. However, after intruducing this question, they directly turn to explain Worst-off DRO in detail. This is too abrupt. They should add at least one paragraph about its motivation.

-  They convert the feasible set to $\mathcal{C}_{\bar{p},\epsilon}$ by adding an extra condition, thus giving an error estimation will be better.

- Pseudo label-based methods serve as important solutions for this research question. However, related baselines are missing.

- If possible, more fairness-related datasets, e.g., Arrest, Violent, German, should be included in experiments. (Not necessary in the rebuttal period)

##########################################

Typo:

1. "We optimize for the a worstoff soft group assignment" -- "the" or "a" ?

and so on...

##########################################

Questions during rebuttal period:

Please address and clarify the cons above.

#########################################


summary_of_the_review: This paper works on a more practial fairness scenario and gives a solid solution in principle. However, there still exists some issues about writting and minor technical details. If these issues can be solved well in rebuttal phase, I am willing to support this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of optimization for the Rawlsian criterion with only partial group labels. They propose to optimize according to the worst case of group assignment and also propose a method to constrain the set of group assignment.

main_review: PROS:
- This paper focus on an interesting problem that has both practical and theoretical importance. In general, learning a fairness-aware classifier with weak supervision is an important problem.
- The extension to Group DRO is intuitive and reasonable. As stated in the paper, it naturally forms an upper bound of Group DRO.
- The paper is overall well-written and easy to follow. I can get the point authors want to convey without ambiguity.

CONS:
- How the size of constraint set affects the convergence / efficiency is not stated theoretically / empirically. It is a crucial property for the proposed algorithm to be feasible.
- Estimation of the true marginal distribution plays an important role in constraining the group assignment set, and this should be emphasized in the paper. Corresponding empirical verification for Lemma 2 is missing. Also, how the algorithm will perform if the estimation is wrong, thus the true p is not in the set?

summary_of_the_review: Due to the weakness written in the main review, I choose to give a weak reject.
review_title: null

Meta-review: This paper introduces a novel method for learning distributional robust machine learning models when only partial group labels are available to improve performance of learning algorithms on minority groups. 

Pros: The paper is well motivated and written.  The ideas are interesting.  Most work on distributional robust optimization (DRO) are in unsupervised settings where group information is not available.  They provide an approach for the semi-supervised setting through a constraint set.

Cons: 
The empirical results do not show better performance over unsupervised baselines as pointed out by reviewers.  

The authors claim one of the benefits of their proposed approach is a one-stage approach, in contrast to competing models that require a two-stage approach; hence, allowing their approach to reduce compute time.  It’ll be helpful to strengthen this point by showing time comparisons.

Missing labels in this case due to participants withholding sensitive information is not an MCAR case, but the proposed work makes an MCAR assumption.  It’ll help to add a discussion and point out such limitations of the approach.

Summary:  This paper has novel and interesting ideas, but still has several issues as pointed out by the reviewers before it is ready for publication.
Decision: Reject