Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a classifier that performs well under the shift of class proportions. The approach involves a distributionally robust formulation (a min-max problem) where the weight vector is constrained in a $f$-divergence ball of radius $\delta$. The optimal post-hoc classifier is shown to be a reweighed classifier. The paper contains a gradient descent-ascent type algorithm to find the optimal post-hoc classifier from an initialization.
strength_and_weaknesses: I experience certain difficulties in assessing the quality of this paper. I believe that the authors did not clarify well the contributions of the paper.

I am relatively familiar with adversarial training and minimax formulation of equation (3). Theorem 1 of this paper is simply an implication of the Nash equilibrium results, which are abundant in the field of adversarial training and robust learning. For $f$-divergence type set $\mathbb{G}(\delta)$, the results of Duchi and Namkoong (2018) and BenTal et al. (2013) have provided much information about the reweighing schemes and how to solve the problem. The gradient descent-ascent algorithm is also widely used to solve min-max problems. Finally, Theorem 3 provides the guarantee only for the ergodic average $\bar f$, but not for the terminal $f^{(T)}(x)$, thus it is of limited usefulness in practice.

From the perspective of a researcher with moderate experience in adversarial learning, I do not feel that I have learned something new with this paper. I hope that the authors can provide strong justifications for their contributions.
clarity,_quality,_novelty_and_reproducibility: Possible typo in the middle of page 4: To my understanding, the Lagrangean function should eliminate the constraint $D(g, u)\le \delta$, thus, the maximization over $g$ in Proposition 1 should be over $g \in \Delta_m$. 
summary_of_the_review: There is not enough evidence about the novelty and the significance of the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes to apply a post-hoc learning process for correcting label (class-prior) shifts between training and test distribution. It utilizes a labelled validation set that has the same label prior distribution as the test data and optimizes a delta-constrained objective (Eq.(1) or (2)). The delta quantifies how much the prior can change from the test/validation class distribution. Using the Lagrangian method, the algorithm alternates between different components. Convergence analysis is provided and experiments on common benchmarks show the superiority of the proposed method, Distributionally RObust PoSt-hoc method (DROPS).
strength_and_weaknesses: Strengths
- Writing is clear
- A simple method that is easy to implement
- Theoretical justification

Weaknesses
- Some experiment details are missing

Detail comments & questions regarding the experiments:

1. What are the sizes of the validation sets? What happens if we fine-tune the model on the validation set?

2. It is mentioned in the tables’ captions that the best performed methods in each setting are highlighted. They are best in terms of what criteria? Did you perform any statistical testing?

3. Figure 1 is concerning. For CIFAR10, why delta_{train}=1 is so good across different eval deltas? If the model is trained based on delta=2, then we should expect it to work best when the evaluation delta is also 2. However, it seems that delta_{train}=1 works best even for evaluation delta = 2. What’s so special about delta_train=1?

4. What’s the performance of DFR for Table 1?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the method seems novel.
summary_of_the_review: The paper proposes a simple method with theoretical convergence justification. Experiments on common benchmark datasets show that the proposed method can work better than alternative methods. Despite some missing details in the experiments, the results are mostly convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied the problem of distributional shifts in classification models and proposed a distributional robust formulation to tackle this problem. The authors showed that the Bayes optimal solution is a post-hoc scaling of the conditional class probabilities, where the scales can be learned via solving a saddle-point problem. They showed the convergence of this post-hoc classifier, and experimentally demonstrated the superior performance of the proposed algorithm on several datasets compared to baselines. 
strength_and_weaknesses: Strength: 

1. The paper is well written. Problem is clearly set up and described. The algorithm is well justified both theoretically and empirically. 

2. Starting from a complicated distributional robust formulation, the resulting post-hoc classifier is yet in a very simple form, which would facilitate its use in many practical problems. 

3. The proposed method does not require knowledge of the class distribution in the test set, which adds to its applicability compared with a class of methods that uses undersampling to match the distribution between train and test.

Weakness: 

1. Even though the authors claimed that their formulation is essentially different from the worst-case optimization (as done in DRO, for example), from what I read, their approach is not very different from DRO. It is still trying to minimize the worst-case loss among a set of distributions centered around some nominal distribution (Eq. (2) and (3)). The conservativeness of the formulation is controlled by some parameter \delta, which is the same with DRO. Could you please clarify what is essentially the difference between your approach and DRO, if any?

2. In Proposition 1, the RHS, should we remove the constraints w.r.t g under max, since we already penalize the constraint violation in the Lagrangian?

3. In Theorem 1, \eta is the ground truth conditional class probabilities, right? Do we assume there is no shift in \eta between train and test? Do we require the estimated \eta to have good properties for your method to work well? I think explanations on the \eta component needs to be added. 

4. In the experiments section under label shifts, can you also compare with the DRO solutions that handle class label shifts? Many DRO models reduce to adding a regularizer to the loss function. I am curious about how your method performs compared with those traditional DRO methods.

5. In practice, how should we pick the parameter \delta to ensure an appropriate amount of robustness?
clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly, and uses easy to understand language to demonstrate complicated mathematical ideas. Even though the formulation itself does not seem to be novel, the resulting post-hoc classifier looks novel to me, and is in a very simple form that can facilitate many practical use cases.
summary_of_the_review: Overall I think this paper is of good quality, and the resulting post-hoc classifier could be potentially valuable to practitioners who are faced with imbalanced classification problem. I like the succinctness of the results regardless of the fact that it was derived from a complicated min-max formulation. The experimental results also look promising. One thing that confuses me is the authors' claim about how their approach differs from DRO (or worst case optimization), which I expect the authors to further clarify. I'd also like to understand in practice, how should we pick the parameter \delta to ensure an appropriate amount of robustness?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In general, this seems to be a clean and neat paper with a simple approach and good empirical performance. The apporach also has a theoretical analysis. Below are some strengths mentioned by the reivewers: 

" The paper is well written. Problem is clearly set up and described. The algorithm is well justified both theoretically and empirically.

Starting from a complicated distributional robust formulation, the resulting post-hoc classifier is yet in a very simple form, which would facilitate its use in many practical problems.

The proposed method does not require knowledge of the class distribution in the test set, which adds to its applicability compared with a class of methods that uses undersampling to match the distribution between train and test."
justification_for_why_not_higher_score: The paper seems to be neat paper, but I suspect the reviewers didn't find it sufficiently exciting to be on the level of spotlight or oral. The approach seems to be also somewhat related to DRO. 
justification_for_why_not_lower_score: In general, this seems to be a clean and neat paper with a simple approach and good empirical performance. The apporach also has a theoretical analysis. Below are some strengths mentioned by the reivewers: 

" The paper is well written. Problem is clearly set up and described. The algorithm is well justified both theoretically and empirically.

Starting from a complicated distributional robust formulation, the resulting post-hoc classifier is yet in a very simple form, which would facilitate its use in many practical problems.

The proposed method does not require knowledge of the class distribution in the test set, which adds to its applicability compared with a class of methods that uses undersampling to match the distribution between train and test."
Decision: Accept: poster