Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes bit-rate constrained group DRO, where the assumption is that group identities can be modeled by a simple function class. This allows the framework to improve the overall utility of the learned model by relying less on arbitrarily chosen mis-labeled points (in a DRO framework), but ensure that only once that are also well modeled by the bit-rate constrained function are upweighted. Using the motivation, an objective is proposed to come up with a local minima where both the adversary and the hypothesis model is a neural network. Theoretical analysis provides risk bounds as well as analysis on convergence for online solver is provided. Experimental results demonstrate some benefit of the approach on real world data, including domain shift benchmarks.
strength_and_weaknesses: Strengths:
1. The paper is largely well written and motivated. 
2. The problem setup is well motivated and being robust to unknown but well modeled group shift is an interesting approach to address lack of robustness of ML models to spurious correlations (if modeled as potential groups).

Weaknesses:
Although the paper is well written, there are many clarifications necessary that could improve the paper:
1. I am guessing $z$ in Equation 5 corresponds to latent group identities? It seems to be introduced in a rush without explaining the objective function at all, leaving it to the reader to decipher details.

2. Unclear motivation of why adversary needs to be a deep neural network and then add l-2 regularization to the parametrization. I would've liked to see a simpler set up where an exact solution is available, and then a heuristic proposed when adversary is also a complex model. It is also not clear how this ties to the assumption that the group identity function is actually a simple function.

3. The latent group identification via KL regularization is interesting, but then I am a little confused what happens if the group identities somehow overlap with class labels themselves, guessing no real robust learning can happen then?

4. Comparison of Theorem 5.1 to vanilla group DRO would be more informative.

5. Same overall concern: What is the applicability of theorem 5.4 to the heuristic assumption of adversary being neural network with regularization added in the objective function.

6. Empirical evaluation could also use more insights:
   6.1 Are celebA results worse compared to other datasets for domain-specific reasons? If yes, why is the worst-group test accuracy lower for Bit-rate DRO always?
   6.2 If the benefit of BR-DRO is more visible only with label noise, how practically useful is BR-DRO?
   6.3 What are the groups used by group DRO vs BR-DRO and how is that affected by the choice of regularization, beyond the fact that higher regularization will constrain the function to be less complex.
clarity,_quality,_novelty_and_reproducibility: Clarity:The clarity of the paper can be improved relatively easily, in my opinion, so I will wait for the rebuttal/response of authors.

Quality:The paper is high quality

Reproducibility: I have not thoroughly checked the code, but authors have provided detailed code to reproduce all results, included datasets wherever necessary.
summary_of_the_review: Overall, I am a little unsure about some modeling choices and practical utility of the paper. But I will look forward to the response and update my review accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new method for the problem of distributionally robust optimization. The learning procedure ‘Bitrate-Constrained DRO’ (BR-DRO) provides robustness to distribution shifts along groups realised by simple functions. This procedure is able to match the performance of methods that use group information on training samples, despite not actually using this information. 

In comparison to prior works such as DRO, which generally optimizes for worst-case performance on distributions that lie in an f-divergence ball around the training distribution, BR-DRO further constrains the adversary’s class, while not requiring more information about the underlying group structure. Other approaches to constrain the adversary, the resulting set seems to still contain some mislabelled and hard instances from the majority set. Other works which seem to obtain group-robustness without group information either implicitly assume the group structure, or are vulnerable to label noise. 

Their main theoretical result is an upper bound on the estimation error of the BR-DRO estimator in terms of the KL divergence from the prior. 

They provide experiments comparing BR-DRO to ERM and other group shift robustness methods that do not require annotations for training samples. 
strength_and_weaknesses: Strengths: 
1. BR-DRO has a well-motivated constraint and is amenable to clean theoretical analysis.
2. BR-DRO performs well on the benchmark datasets and recovers the kind of guarantees GROUP-DRO achieves without using the group information. 
3. I think the paper is well-written and clear.

Weaknesses: 
1. As expected, BR-DRO has higher complexity when compared to GROUP-DRO. 
clarity,_quality,_novelty_and_reproducibility: Clarity: I think this paper is well-written. 

Quality: I think this paper is good and vote to accept it. 

Novelty: I think the BR-DRO constraint is novel and elegant. 

Reproducibility: The authors provide code in the supplementary material, but I have not run it to verify their claims. They give a detailed description of their experiments in the `Experiments' section of their paper. 
summary_of_the_review: I think this paper is good and vote to accept it. It introduces a novel constrained version of DRO which is performs as well as GROUP-DRO without the group information. The paper is well-written and provides experiments supporting this choice of modelling assumptions. 

One interesting modelling assumption on the adversary they introduce, is that the adversary can only separate datapoints into potential groups using simple features.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Distributionally robust optimization methods are often overly convservative, while group DRO approaches requires group annotations, which limits its practicality. To address this challenge, they introduce a practical assumption that group shift is captured by high-level features with a simple function. Based on this assumption, the authors propose a BR-DRO, which makes the model to be robust against distribution shifts along group which is realized by simple functions. This helps model not to be overly conservative while being robust to plausible worst cases, resulting in meaningful performance improvement compared to unconstrained DRO.
strength_and_weaknesses: Strength
- The paper address why restricted adversary for DRO do better than unconstrained one, which is supported by theoretical justification under the assumption of simple group shift.
- The proposed BR-DRO framework is empirically validated by extensive experiments.

Weakness
- Several important unsupervised comparisons are missing. Especially, [George] and [BPA] extend Group DRO in an unsupervised way, where they conduct clustering on the feature space and identify latent subclass (minority group). In other words, they also aim to identify potential groups for worst-case robustness without group annotations.
- The formulation of BR-DRO starts from Assumption 4.2, but what is the motivation of the assumption? Also, I think it may be applied well because most of group robustness datasets has binary (bias) attributes, Does this simple group shift assumption still work well in datasets where the bias attribute has multiple values?

[George] No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems, NeurIPS 2020

[BPA] Unsupervised Learning of Debiased Representations with Pseudo-Attributes, CVPR 2022
clarity,_quality,_novelty_and_reproducibility: - The paper writing is well structured.
- This work nicely bridges the gap between unconstrained DRO and group DRO with a practical assumption in a novel way.
- Source codes are provided for reproducibility.
summary_of_the_review: The authors constrains the distributionally robust optimization method under the assumption of group shift, which prevents the model from being overly conservative and pessimistic. The practical relaxation of BR-DRO is novel and well supported by experimental results, though those are not that strong compared to recent debiasing approaches. Additional discussions with other robust optimization approaches can help for better understanding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes bitrate-constrained DRO and makes a connection with the "simple group shift" conception. Specifically, they argue that this constrained DRO can distinguish between a rare group that is realized by simple functions and a rare group of examples that share no feature in common or may even be mislabeled, without knowing the group information in advance, and can thus improve the performance of neural networks by avoiding being too pessimistic. To support their result, they both give a theoretical generalization guarantee as well as many empirical experiments. In addition, they proposed a modified online game-based strategy to solve this optimization problem.
strength_and_weaknesses: Strength: This "simple group shifted" phenomenon is well-motivated and using this well-defined, easy-to-solve DRO to capture such phenomena without knowing group information is a clean framework and has been well-supported by experiments. (But I am not an empirical person so my evaluation of empirical results has low confidence). 

Weakness: 
1\ All the theorem seem a bit incremental. Theorem 5.1 is a classical generalization bound analysis and 5.2 is s RKHS-specialized version. Theorem 5.4 highly relies on the assumption that $W(\gamma)$ is Vapnik-Chervenokis (VC) class of dimension $O(\gamma)$ and other standard techniques from  Abernethy et al. (2018) . But I think it is ok given it is not a purely theoretical paper.
2\ To me, the connection between "simple group shift" and "bitrate-constrained distribution" is a bit weak. Seems to me the group shift can violate the bitrate-constrained distribution and the bitrate-constrained distribution can also imply a non-group shift. Of course, I think the explanation from the paper is a good intuition... 


clarity,_quality,_novelty_and_reproducibility: This paper is well written.
summary_of_the_review: This paper to me is slightly above the acceptance borderline because it provides a clean and well-defined framework, a good intuitive explanation, and convincing experimental results. But like I said above, its theoretical results are still relatively incremental and their justification on the connection between  "simple group shift" and "bitrate-constrained distribution" is a bit weak.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes bitrate-constrained DRO and it makes a connection with the "simple group shift" conception. Theoretical as well as empirical experiments are shown.

All the reviewers agree that the paper is interesting, well-motivated and with a clean theoretical formulation. Hence, I recommend acceptance of this paper.
justification_for_why_not_higher_score: Given that this paper is about a very technical topic and the scores are not very high, I recommend it as a poster.
justification_for_why_not_lower_score: Only one reviewer is giving a score below 6, but the main complaint is about the "practical utility of the paper". Given that this is a mainly theoretical paper, the practical utility should not be the main concern. So, overall I judge this paper as a clear accept.
Decision: Accept: poster