Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores how “difficulty” is model-specific, such that different models find different parts of a dataset challenging. They measure difficulty as a function of both model and data,  and they measure how difficulty affects performance disparity and quantify the effect of the simplicity bias. Their main observations are: 1. They identify difficulty disparity persists in the post-dataset audit setting. 2. They introduce a difficulty amplification factor to quantify how much a model exacerbates difficulty disparity. 3. They evaluate how choices including model architecture, training time, and parameter count impact difficulty amplification.

strength_and_weaknesses: Strength

- The paper performs investigation of the task difficulty and reveals it is a function of both model and dataset.

Weakness
- The main argument of the paper (the task difficulty is not a function of the dataset itself but relies on the model as well) seems to be a known fact as different models have different capacity or generalization ability and their difficulty of solving a task also varies. For example, a deep CNN (e.g, ResNet-152) can perform better than a shallower one on ImageNet and thus the difficulty of solving ImageNet is different for different models. 

- There is no explanation on how the model bias leads to different difficulty disparity. The model bias is also not clearly defined. The experiments are only based on a handful of different architectures/pre-trained models and it is not clear why these models are selected.

- The paper does not provide a solution for precisely estimating the difficulty disparity of a model. Given section 6, it is not still clear about the real-world impact of the difficulty amplification and what practical lessons can be learned from the observations.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. The novelty lies in the observation of “difficulty disparity”. 
summary_of_the_review: The paper presents an investigation on the association of task difficulty and model bias. However, the observations are not quite convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problem of difficulty amplification: trained models exhibit consistent differences in performance on groups that are "easy" and "difficult" even when there are no obvious spurious correlations or dataset imbalance. In particular, they show that the  simplicity bias is a major reason for difficulty amplification. First, they show that data difficulty is model-specific (different models find different aspects of the dataset difficult). Then, the paper shows that models trained on a variant of CIFAR-100 (with easy and hard groups) exhibit difficulty amplification. Section 5 focuses on understanding how the amplification varies as a function of design choices such as model architecture, model width, early stopping etc. Section 6 shows that difficulty amplification can be an issue in practice as well. 
strength_and_weaknesses: Strengths:
- CIFAR-100 setup. The synthetic variant of CIFAR-100 incorporates an intuitive notion of model-specific difficulty and clearly showcases that difficulty amplification can show up in standard image classification datasets. 

- Amplification factor analysis. The analysis on how the amplification factor changes depending on standard design choices such as architecure, early stopping etc is insightful. 

- The paper (especially the first few sections) is well written. 


Weaknesses:
- Main finding not novel: The main finding of this paper (performance disparity is amplified when easy and hard groups are clumped in the dataset) is not new. This paper (https://proceedings.neurips.cc/paper/2020/hash/6cfe0e6127fa25df2a0ef2ae1067d915-Abstract.html) shows that simplicity bias hurts generalization when the dataset comprises a majority subpopulation of "easy" data points and a minority subpopulation of "hard" data points---the drop in accuracy stems from misclassifying the minority subpopulation, even though a model trained only on the minority subpopulations performs well. The authors should clearly contextualize the experiments in this paper vis-a-vis previous findings on this phenomenon.

- Section 3: The findings in this section contradict previous work. There are several works (e.g. http://proceedings.mlr.press/v119/hacohen20a/hacohen20a.pdf and references therein) that show that models with different architectures etc learn similar classifiers + classify points correctly over time in a similar order. On the other hand, this paper says that dataset difficulty is different for different models. So, it would be nice to reconcile these findings with previous works. Is this because the models compared in this paper are too different (e.g, linear models and CNNs)? Also, PLS analysis hard to understand: The paper needs to introduce PLS briefly before it introduces the findings. The way it is written right now is hard to parse.

- Analysis on disparity over time. It is not clear why the paper focuses on difficulty amplification over time. What is the practical implication of understanding how difficulty amplification varies over time? Having some clear motivation would be useful, right now it feels a bit ad-hoc.

- Section 6. This section is too terse. I think the experiment setup and the analysis should be more in-depth given that this is the only experiment showcasing this phenomenon in practice. Most of the discussion can be deferred to an appendix to make room for this. 

Overall, this paper studies an important problem that is not well understood. However, my major concerns are (a) novelty---previous work on simplicity bias has analyzed this phenomenon, (b) section 3 claims contradict previous findings on this topic, (c) limited + unclear analysis on difficulty amplification in practice. 
clarity,_quality,_novelty_and_reproducibility: Please see strengths and weaknesses.
summary_of_the_review: Please see strengths and weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a notion of difficulty disparity between subsets of a dataset and claims that neural networks parameterize functions that better separate groups with lower difficulty. The claim is then validated empirically via a set of experiments on CIFAR-100 and the Dollar Street datasets. Results indicated that difficulty amplification is affected by factors such as model architecture and early stopping.
strength_and_weaknesses: Strengths:
- The paper adds evidence to show that biased predictions are an effect of both datasets and models biases, a often discussed question within the machine learning community;
- The paper is clearly written;
- The authors made a clear effort to convey their message by providing pictures that clearly illustrate the issues tackled in the paper and summarizing the main findings of each experiment.   

Weaknesses:
- The paper contributions are mostly empirical and the scope of the experiments is quite narrow, which makes it difficult to conclude how robust the findings are to other common settings such as neural networks trained with optimizers different from SGD;
- Critical claims to support the findings are not well explained and supported. Importantly, the authors claim that they perform experiments in a setting where models do not pick up on spurious correlations between input and labels, but in no way this claim is supported in the manuscript.
- The proposed measure of difficult amplification relies on training predictors for each particular group in the dataset. I believe this is not practical in many real-world settings where examples from particular groups are very scarce, making it difficult to train and evaluate specialized models.
- Computing the proposed measure of difficult amplification further requires knowledge about group labels so that the dataset can split. This information is not always available in practice and / or shouldn’t even be taken into account when collecting a dataset.

clarity,_quality,_novelty_and_reproducibility: Overall, I believe the main message of the paper is clearly conveyed in the manuscript, albeit limited, the contribution is novel to the best of my knowledge, and most of the information required for reproducing the experiments is reported in the Appendix. However, as mentioned in the previous section of my review, I have several concerns regarding this submission, which I detail below along with questions and suggestions.

- A major flaw I see in this work is the fact that the authors do not provide any evidence to support the claim that the experimental setting in the CIFAR-100 experiments is indeed absent of correlations between group labels and target labels (as claimed in the manuscript in the beginning of Section 4.2). Moreover, it is also important to emphasize that different model classes can potentially capture different spurious correlations which makes it even more difficult to make such a claim. This point exemplifies reasons why I rated the “Correctness” of this submission as 2.
 
- The experiments suggest that SGD-trained neural networks are biased towards less complex solutions. Although I found this conclusion interesting, I’m concerned with how generalizable this finding is to other optimizers often used in the literature and in practice. More specifically, would the conclusions hold in case we train a neural network with Adam or RMSprop? If not, then switching SGD by another optimizer would already be enough to avoid the problem of amplifying disparity? If so, is this problem even worth studying? I believe that all the aforementioned questions render the motivation for this work unclear. This point exemplifies reasons why I rated the “Technical Novelty And Significance” and “Empirical Novelty And Significance” of this submission as 2.
- 
I found the introduced notion of difficulty amplification based on accuracy of models trained on specific groups very limited given that in real-world scenarios it is likely that there will be groups for which data will be scarce (e.g. a rare disease). In case one wants to reproduce such an analysis to diagnose the amplification disparity of a given model class, this makes the possible application scenarios very limited. This point also exemplifies reasons why I rated the “Technical Novelty And Significance” and “Empirical Novelty And Significance” of this submission as 2.


- The findings reported in Section 5 are underexplored and lack depth in the analysis of the results. The authors only report which of the considered aspects affected or not the amplification of difficulty disparity without providing any hints on possible reasons that would explain those findings. I also found it surprising that weight decay does not have an effect on difficulty amplification since it is tightly related to model complexity. This point also exemplifies reasons why I rated the “Technical Novelty And Significance” and “Empirical Novelty And Significance” of this submission as 2.


Minor:
- Typo in page 4: difficuly -> difficulty

summary_of_the_review: This submission is clearly written and aims at shedding light on interesting and relevant aspects of SGD-trained models. My major concerns are related to the lack of support for central claims of the narrative in the paper, the generalization of the findings, and lack of depth in the experimental analysis, which was critical to my score given that the contributions of this work are mostly empirical. All in all, I found that the flaws in the current version of this submission outweigh its merits by a large margin and I believe this manuscript is not ready for publication yet.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces the notions of difficulty disparity and difficulty amplification, which is the ratio of change in the accuracy gap between different groups (easy vs hard groups) when trained in a combined manner versus when trained on those groups individually. The experiments show different models have different difficulty amplification, and it is not just a data-only issue.
strength_and_weaknesses: **Strengths**

[S1] The paper provides a general, intuitive measure for difficulty amplification and the experiments show that this is dependent on the models and not just data.

[S2] If group labels are provided, then the paper can be used by newer works to measure difficulty disparities.

**Weaknesses**

[W1] This work requires group labels, which may not always be available. Recent works on debiasing techniques have realized that this is an unrealistic assumption, so use implicit measures of difficulty e.g., Learning From Failure, Just Train Twice etc. It is unclear how this work could leverage the implicit measures to compute amplification.

[W2] Group underrepresentation is a real-world problem. The model unfortunately does not consider this setting. Furthermore, ‘underrepresentations’ may not be the only source of spurious correlations e.g., even with balanced groups certain factors in the group (not necessarily signal) may be easier to exploit. So, the claim of ‘absent correlations between group labels and target labels’ may require further verification.

[W3] Weight decay is generally used to regularize the complexity of the model (to prevent overfitting), yet the paper surprisingly finds ‘next to no effect of weight decay’. Could you elaborate on this finding? Would it hold in general? One interesting experiment would be to analyze the models from [1], where the authors showed that with increased weight decay (decreasing overparameterization) even simple re-weighting could help debias models.

[1] Sagawa, Shiori, et al. "An investigation of why overparameterization exacerbates spurious correlations." International Conference on Machine Learning. PMLR, 2020.


clarity,_quality,_novelty_and_reproducibility: While most of the writing is clear, I found the following parts a bit confusing:

1. The paper could dedicate a separate subsection/paragraph to specify data-only vs data+model measures of difficulty. The 'PLS analysis' section could be re-titled to indicate what it is (data-only vs data+model) instead of mentioning the technique. 

2. It is unclear how the class means are computed. If they are computed in the embedding space, isn’t that model-dependent?

3. I did not find the terms ‘observed’ and ‘estimated’ intuitive and had to keep looking back at the definitions. Specifying the intuition behind the terms or changing them to more intuitive terms could improve readability.

As far as I am aware, the measure of difficulty amplification is novel and the experiments seem reproducible.
summary_of_the_review: While the measure might be useful, I think the paper needs to address at least these shortcomings: a) study on settings without group labels since that is more realistic b) confirm the weight decay finding better (perhaps using the underrepresented groups) to be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores how different models may find different parts of the dataset difficult. There was a consensus among reviewers that this paper should be rejected. The main reasons for this were limited novelty (and underexplored connections) to prior work and limited experimental evidence. As a result, I recommend to reject this paper.
justification_for_why_not_higher_score: There was a consensus among reviewers that this paper should be rejected.
Decision: Reject