Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the effects of regularization techniques — including data augmentation (DA), weight decay, and dropout — on per-class accuracies in image classification problems. They demonstrate that when we pick hyper-parameters such as the strength of regularization with regards to the average accuracy across all classes, it does not necessarily result in the fairest model; instead some classes are affected negatively. The experiments tests both in the case of directly training image classifiers as well as finetuning a pretrained classifier on a downstream task. It also explored possible explanations and showed that simple solutions such as selectively applying regularization to certain classes don't mitigate this problem. 
strengths_and_weaknesses: In general the paper is well written, neatly structured, easy to follow, with comprehensive and solid experimental setup. The authors studied an important problem of class fairness: when we apply techniques with the overall average performance across all classes as a target, do we incur silent unfair and unfavorable effects on certain classes? The angle on regularization is unique and novel, and all experiments are well constructed to prove the author's point. 

However, the paper can be improved by shifting the emphasis and reconstructing the experimental presentation structure. 

The experiments focus overly on data augmentation. However I think it is not too hard to give the same weight on the other two: weight decay and dropout, both only lightly experimented or mentioned. Only the first experiment (2.1) is "scientific" enough — with t-test results; the others only show cherry-picked classes. The author should consider at least adding t-tests to all other sections. In addition, I think the author should define a metric of "class fairness" (e.g. the variance or gap between best performing and worst performing classes) and measure/show it across regularization strengths. Most figures in this paper only show cherry-picked classes, and while illuminating, are not scientifically convincing.

The presentation could be tightened, e.g. Fig 1-4 have almost the same setup, and a lot of texts in 2.1 - 2.3 are repetitive; maybe consider merging 2.1 and 2.2, and bringing dropout results in from appendix. Transitions in between sections are a little redundant, and can be made succinct if needed to save space for, say, adding t-test results to other sections and adding dropout.

Specifically, the presentation of results could've been much stronger with the following suggested changes:

 - Consider merging 2.1 and 2.2, moving dropout results from appendix to main sections
 - t-test should be applied to all scenarios, not just DA
 - In addition to showing cherry-picked classes, define metrics for class fairness and show them across regularization strength 
 - should also explain how the class pairs are picked for figs 1-4
 - For each figure, try starting with a succinct title, instead of details on how results are obtained. Like Figure 2:  “Effect of DA on per-class accuracies” Fig 3 “Effect of weight decaly on per-class accuracies.”
 - annotating in figures what every line is. The blue lines in fig 2-4 are not explained
 - Overall, the text of paper used too much italic/underline/bold formatting 
 - “Section” "Figure" etc. before numbers should be capitalized
 - undefined what "DN" is. If it's "Deep Network", I don't think that's a common abbreviation.

questions: Line 37: What about dropout? It is lightly touched in 2.2 but never mentioned in Introduction

Line 44: requires -> require

Line 78-82: consider using footnote for the promise on codebase. Should delete the second part -- it is not quire appropriate to address to reviewers directly in this style in main text.

Line 147: "all the DN parameters except for the ones of batch-normalization layers" -> I think the author try to say only weights and not biases, but this sentence is confusing as parameters include biases.

Line 206: "respect the level-set of f*" -> unclear what level-set means

Line 218-219: since MixUp etc were brought up as possible fixes for label-preserving, should consider adding them to exps to see if they help at all

Line 235-237: "becomes insufficient to predict the correct label on average" -> should define clearly what that means. What's the definition of "unpredictable"? What's the threshold? The vertical dashed line in Fig 5 is never explained. 
limitations: There's no Limitations section in this paper, which is somewhat reasonable as the authors only exposed the problem (effect of regularization on various classes) but didn't propose a solution. However, I still suggest adding a section about potential limitations if a hypothetical solution is proposed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper builds around the fact that data augmentation (DA) can harm the accuracies over certain classes despite improving the accuracies overall and offers discussions on several related topics. Some of the discussions are making importance contributions to the community, while other discussions seem fairly straightforward. 
strengths_and_weaknesses: - strengths
   - the discussion on the effectiveness of class-dependent DA is relevant and potentially will inspire the development of methods for a long way. 
   - the main message that DA can harm the performances for certain classes while improving the overall performances is fairly important. 
   - the paper is backed up with a heavy set of empirical results with seemingly no issues in reproduction. 
- weakness
   - several other messages (other than the main messages) seem very straightforward
   - theorem 1 (the only theorem in the main paper) is not presented rigorously enough for evaluation, which puts up some question for the theoretical rigor of the entire paper. 
questions: The overall paper is a very interesting read and easy to follow. The main message of the paper, that DA will introduce negative effects to certain classes, despite the overall improvement across all the classes, is quite important and inspiring. 


limitations: There are a couple of generic words regarding the societal impact. The authors do not believe discussions of limitations are relevant, which I don't agree with. For example, the empirical study is limited to a couple choice of transformation functions (and datasets), which could potentially limit whether the conclusions obtained are reproduced in other settings. However, the evaluation is not based on this part. Several questions

- The theorem 1 does not seem to be presented in a rigorous manner, thus very hard to evaluate. 
        
    - there is no definition of "level-set" across the paper. I guess it means decision boundaries or the features related to decision boundaries.
    - in the proof, line 599, assumes p is a constant and omits it, but the remaining proof directly assumes $p=1$ instead of omitting it. (dropping the constant does not allow the equality below line 601 to hold, only assuming it to be 1 does)  
    - the proof concludes by saying "last equation is always positive" (line 603), but the entire term is always positive throughout the proof, they are always norms, which are by definition positive. 
    - I do not see how other choices of loss, such as cross-entropy, will be carried out trivially following this proof sketch, despite the authors suggest so (line 197)
    - these concerns together, since theorem 1 is the only theoretical result of this paper, put a significant question on the formal results of this work. 

- While the paper appears to be dense in contents, I feel like some contents are not significant enough to be presented in the main manuscript, such as details in statistical tests, and augmented samples; some can be presented in a more compact manner, such as Figure 5. To conclude, I feel this paper does not contribute a sufficient amount of results as a paper at this tier. 

- Section 3.4 also offers an exciting beginning of the investigation, but the investigation ends very immaturely with a shallow attempt with the most obvious remedy. I would suggest the authors to offer substantially more results following the direction of 3.4. 
    - the authors suggested "no readily and easily implemented solution" available (line 311) but it seems not even all the solutions attempted as the previous sections are customized into class-dependent version and tried out yet.  

Overall, I think this is an exciting read and interesting start of an important investigation, but the results do not seem mature enough yet.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper set up a hypothesis that implicit regularisation via data  augmentation (in the sense of Poggio's stability) and explicit Tiknonov regularisation (L2 is investigated) penalise different classes by different degrees. This implies that the average test accuracy is not a reliable measure of performance across all class alternatives. It also discovers that data augmentation may even not be label preserving, which it studies in the context of imbalance in the class's support in terms of labels, but it also opens up possibilities of adversarial data augmentation. 
strengths_and_weaknesses: The whole set of ideas being investigated esp. the  transferrability to downstream tasks seem to be original although nonetheless a natural extension to the questions posed by Xu and Raghunathan. I would have had more confidence in the hypotheses if 1.the manner in which unfavoured classes were identified was systematised. At the moment, a few hand-picked classes are demonstrated upon. 2. We had quantified support from more datasets and models to rule out bias from them. Otherwise, the paper is delightfully well-explained and well-structured. 
questions: 1. If the plot on the RHS in Fig. 1 is contrived to present an ideal case, it puts the rest of the results in question. If it is, mentioning so would help. 

2. Both the regularisation strategies investigated induce smoothness. It makes you wonder  if classes are being selectively regularized, what happens with a choice of regulariser that does so by design and produces sparse representations. 

limitations: Although the paper keeps prescriptions out of scope, it may have built upon suggestions made in the work of Raghunath et al. on when and how much data augmentation is suitable. 

ALso,  it does not discuss what discuss what proportion of classes are adversely affected by both forms of regularisation, if it is but 1% of classes, the solution may worsen generalisation in the vast majority of classes. Both data augmentation and L2 reguarisation are widely used methods to help with generalisation, and knowng this seems fairly consequential.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the per-class effect of some common regularization techniques including data augmentation (DA), weight decay, and dropout on neural classifiers.  Experiments across several models and two classification tasks are conducted to show that the popular approach of maximizing mean model performance by cross-validation has heterogeneous effects on different classes. The paper combines intuitive theoretical and empirical evidence to show that the class-dependent label-preserving regime may be the cause of the harmful effects on some classes by inducing class-imbalance for augmented data.
strengths_and_weaknesses: Strengths: 

1. Overall the paper is well written and easy to follow.

2. The question of interest is well motivated as regularization techniques are a crucial part of the success of modern neural models which are usually overparameterized.

3. The authors provide a good discussion of several lines of related work.

4. There is a good combination of intuition, math, and experimental validation which offers a well-balanced package.

5. Experimental designs are simple yet well controlled to demonstrate the class-dependent effects of regularization in a clear way.

6. Experiments are conducted with a variety of realistic neural models on large and popular datasets/tasks including ImageNet (supervised classification) and iNaturalist (supervised transfer learning).

7. A simple fix with class-dependent DA is evaluated in an attempt to recover the performance.

Weaknesses:

Minor:

1. Despite the fact that pre-trained models of a diversity of architectures are used to observe the label preserving regimes (and they seem to agree qualitatively), I believe that it should be more clearly stated that the observed label-preserving regimes represent those of a CNN family, which may deviate the “true” regime. 
Some newer models with patchified input such as ConvNext and Swin Transformers may show different behaviors under the same DA transforms. Additionally, one potentially better empirical estimate may involve repeating the experiments with human observers which are arguably closer to “ideal” observers.

2. The paper can be further strengthened if a practical strategy that recovers the harmful effects of DA on certain classes was proposed.

questions: Questions and suggestions along with the contexts can be found in the “weaknesses” section.
limitations: The authors discussed the limitations of the work. Please see weakness 1 for some suggested discussion.
review_title: null

Meta-review: metareview: This paper studies a natural and important question: could techniques that improve the average performance hurt performance on individual classes? The paper provides comprehensive experiments to answer this question in the positive, for data augmentation, weight decay, and dropout. The authors have made significant attempts to improve the paper during the rebuttal phase, including more thorough experimental evaluations and added clarity of Theorem 1. This is a good paper and I recommend acceptance.

One minor comment: there's some inconsistency in terminologies. The title suggests that DA is not a regularization, but in the paper DA is referred to as a type of regularization. Please make it consistent throughout the paper.
confidence: Certain
Decision: Accept