Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents a simple post-training strategy to balance the worst-group and average-case accuracies of a classifier. In addition, it proposes a new metric to summarize this tradeoff. An extension on the simple method is also proposed to achieve better results. The proposed method outperforms several existing baselines and helps to put prior work on group robustness in context.
strength_and_weaknesses: Strengths:
- the proposed method (RS) and its extension (IRS) are both simple and effective
- the robust coverage method is a nice way to summarize the worst-avg accuracy tradeoff and is a strong contribution
- the related works section is written very well.

Weaknesses:
- Evaluation on more datasets would help to strengthen the analysis. In particular, the BalancingGroups codebase (https://github.com/facebookresearch/BalancingGroups) would be a great point for this to add 2 text classification tasks.
- Following the above work, this work should also compare against subsampling baselines (SUBG/SUBY).
- The results are often not presented in the clearest way (elaborated in more detail below).
- In general, it is unclear to me whether this paper thinks RS or IRS should be the main method (there is also a discussion of ARS in the appendix, which seems to have very strong results but confusingly was not even discussed in the main text). It seems that IRS does have strong results on Waterbirds and should be highlighted more - in fact, I would suggest adding IRS to figure 2 and presenting all results (tables 1 & 2) with both RS and IRS. (As a side note, it is also unclear why IRS was not evaluated on GDRO/JTT etc in table 3). This point may seem small, but it has a significant impact on how the results are interpreted overall.
clarity,_quality,_novelty_and_reproducibility: The writing is easy to read, but plots could be significantly better and the results better presented:
- I did not understand what the takeaway from figure 1 was. I would suggest having figure 2 as the main figure 1.
- Which dataset is presented in Figure 1?
- Figure 4 can safely be moved to the appendix - added value is not that much.
- When reading, it was often unclear whether the results with RS referred to IRS or to RS. This is since IRS is presented before any experiments, so in the first part of the experiments it was unclear whether the results were for RS or IRS. To remedy this, one suggestion would be to introduce the IRS method *after* the first set of RS experiments.
- Why are table 4 experiments performed on a completely random dataset used nowhere else in the paper?
- Figure 5 bottom can be safely removed - it does not add any additional info to Figure 5 top. I would suggest using the space to draw the scaling curves for GDRO, JTT, and other important baselines instead.
- There is no discussion of Table 2 in section 4.2.
summary_of_the_review: This paper presents some important insights regarding the tradeoff between worst and average accuracy. It presents a simple method to allow trading off the two. At the same time, this work lacks some experimental rigor (only 2 datasets and missing subsampling baseline) and clarity in exposition (the RS & IRS results are difficult to compare to each other). If these last two points were done well, there would be a nice takeaway along two directions: 1) worst and avg accuracy can be traded off for all models, and 2) RS & IRS are two simple methods that do this with strong empirical performance. Due to the limitations outlined, however, I can only have confidence and clarity in takeaway #1. This is the reason for my rating.

------------------------ UPDATE AFTER REBUTTAL --------------------

My main criticisms with this work are around clarity and the small number of datasets. The authors have addressed clarity somewhat, and importantly have added results on two more datasets - CivilComments and FMOW. The results on CivilComments are just as impressive/notable as those on CelebA. The results on FMOW are a bit muddied, which is due to ERM performing worse than GDRO, so the story here is a bit nuanced but I am glad the authors included the results nonetheless. Given that evaluation has been much improved, I have updated my rating of this paper to accept. That being said, I would like to see a couple things in the camera ready:
- Full evaluation results of all the baselines on the new datasets.
- A figure or table within the main text or appendix that averages the numbers across all datasets (to provide a wholistic view of the method).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper suggests rescaling the output probability to achieve group robustness. The idea is to find the scale factor using the unbiased validation set to minimize the balanced group accuracy on the validation set. Once the scale factor is found, it is applied during the test as a post-process module to the test prediction. 
strength_and_weaknesses: Strength:
- The proposed method is simple yet very effective and can be easily added to previous methods. 
- The paper proposes a new metric to both evaluate average accuracy and worst-group accuracy. 

Weakness:
- It is clear to me that the proposed method is able to maximize the robust accuracy, it remains unclear to me why it can achieve better balance between robust accuracy and worst-group accuracy than the other methods. 
- The method may heavily rely on the validation set to work and requires access to the group information of the validation set which sometimes is not necessarily available. 
- The method might also need a large validation set to work as the validation set is used to estimate the scaling factor and the centroid. It would be nice if an ablation study can be performed for different sizes of the validation set. 


Question:
- Wondering instead of searching the scaling factor, can it be learned?
- does adding a bias factor be more effective than scaling factor along? 
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written and the proposed method is simple yet novel in the spurious correlation literature.
summary_of_the_review: The paper proposes a simple yet novel technique to post-process the network prediction to achieve group robustness. The method is well-motivated and experiments are conducted to validate its effectiveness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a test-time rescaling strategy to address the trade-off between worse-case and average accuracy. The rescaling coefficient is optimized on a validation set to maximize a certain objective function, e.g., the worse-group accuracy. With different rescaling coefficients, the method is claimed to find a Pareto frontier of the two objectives (worse-case and average accuracy). The proposed method is compared with different baselines in the experiment and is shown to be effective on two datasets.
strength_and_weaknesses: Strength:
The paper is clearly written and the proposed method is straightforward. The experiment looks extensive. 


Weaknesses:

The reported baselines in this paper are not consistent with their official results. For example, GroupDRO has 93.5% average accuracy and 91.4% worse-case accuracy with ResNet50, but this paper's Table 2 reports a much lower accuracy for GroupDRO with ResNet18. The result of JTT has the same concern. So I cannot say the experiment is convincing to me.  Please show the experiment using the same backbone with baselines. 

Only two binary classification datasets are used in the main experiment. It will be more convincing if the method works in more challenging tasks like CivilComments, WildCam or FMoW in WILDS benchmark, which contain more classes and groups. The FairFace with 9 classes are used to show the scalability of robust rescaling, but not compared with any baselines. I would like to see a fair comparison with existing baselines on more challenging and real-world datasets.

The title has a very strong statement that class-specific scaling is all you need, which is not supported by the experiment as least for me.
clarity,_quality,_novelty_and_reproducibility: The clarity is good. The overall quality is satisfactory but the experiment is not rigorous thus not convincing. The novelty is just at the borderline, since the test-time rescaling is not new [1]. 

[1] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.


summary_of_the_review: Given the weaknesses, I would like to give a conservative score and encourage the author to do a more rigourous experiment in the next version to show the real effect of robust scaling. Otherwise, I cannot agree with the argument that class-specific scaling is ALL you need.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a rescaling-based postprocessing method that is applied to the output of any machine learning model with the goal of making the predictions more robust. They conduct experiments comparing the proposed method used to postprocess the outputs of several existing robust learning techniques, along with an ERM baseline, on two standard robustness datasets (CelebA, Waterbirds).
strength_and_weaknesses: 
## Major comments

* The paper lacks any motivation for the proposed method. While of course robustness is a worthy goal, why *rescaling* would be a useful or appropriate technique to advance the community toward this goal is not obvious -- and there is no discussion in the paper for why either.

* The method is not well-described. The equation (3) does not fully define the problem being solved. For example, what is really meany be a "scaling coefficient vector" -- does it need to have unit norm? Why do we take arg max in (3)? What are the practical implications of using this method? There is no discussion of this in the paper; lacking any theoretical results, there should be strong empirical or intuitive support for why this is a reasonable postprocessing choice.

* The implementations of "baseline" or prior works are not faithful, and report much lower numbers than those works on the same datasets. For example, JTT reports much higher worst-group and average accuracies on Waterbirds (Table9 listd their best as 88.6% and 93.1% in Table 9, see also Table 7,8; these are lower than in the current paper Table 2) and Group DRO reports mich higher Waterbirds worst-group and average accuracies in Table 3. I am concerned that this is due to poor hyperparameter tuning or model architecture selection; the approach to tuning is not described at all in the paper so it is difficult to tell (if no tuning were performed, this would also be a concern).

* The empirical results are weak. For example, only 4/18 comparisons in Table 2 show an improvement greater than two standard deviations, and only one of the eight comparisons on Table 3 (for the blue "target metric that robust scaling aims to maximize" columns) is greater than two standard deviations.

* Small contribution - beyond the rescaling method, the proposed metric is not a standalone contribution. I think the metric is lacking a thorough discussion or motivation.

# Minor comments

* Figure 1 needs more detail. What dataset/task is this? Are the different points different initializations, hyperparameter configurations? There is very weak evidence of a tradeoff here, but the details are far too murky to draw much of a conclusion; if there were not lines drawn on the graphs I would not conclude there was any relationship.

* Figure 2 also doesn't describe (a) the dataset/task, or (b) what models are used for each method, how they were selected, or give any notion of variability (i.e. Clopper-Pearson confidence intervals). Again, it is hard to learn much without this information.

* It seems Figure 3 is missing any reference to a baseline; can this comparison be made e.g. to ERM? Also, how are we supposed to conclude that any other scaling (s \neq 1) is an improvement based on this plot?

* Other hyperparameters also seem to be selected in an ad hoc manner. Why is k=20 chosen? How does varying the number of clusters impact the experimental results?

# Typos etc.

Page 2: Footnote 1 should probably be in main text.

Page 4: "given n training examples without group annotations" --> without *attribute* annotations (since group is attribute + label)
clarity,_quality,_novelty_and_reproducibility: See above. Reproducibility concerns are particularly around how hyperparameter searches/tuning were conducted, but the final values are reported.
summary_of_the_review: I have several concerns with the paper. In addition to the proposed method being neither well-motivated nor well-described, the experimental results also show only very small improvements (in most cases, within the realm of statistical variation) and over baselines that appear not to be tuned at all and perform far below the original published results of the baseline works. I think it is not ready for publication in its current form.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes  a rescaling-based postprocessing method to control the trade-off between worst-group and average-case accuracies. The experiments show that the proposed method outperforms several existing baselines on two standard robustness datasets (CelebA, Waterbirds).However, the reviewers find that the motivation of this paper is very weak. Reviewers do not understand why the proposed method works. The reported baselines are not consistent with their official results. The datasets used in this paper is too trivial.
justification_for_why_not_higher_score: N/A. 
justification_for_why_not_lower_score: N/A.
Decision: Reject