Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tries to shed light on the cause and nature of simplicity bias and proposes a regularization strategy to mitigate the same. 
While we know that deep neural networks (DNNs) are prone to learning simple features (simplicity bias), recent work provides evidence 
that the penultimate features in the DNNs contain a diverse and complex set of features.  The authors try to bridge the gap between these seemingly contradictory observations. They try to show that the features learned by the DNN are highly biased toward simple features. The simple features are replicated to a large degree causing the SGD to converge to max-margin solutions (due to its implicit bias) that primarily rely on simple features. The authors further devise a regularization scheme (FRR) to mitigate simplicity bias. They use FRR-L to retrain only the last layer (while freezing the backbone), and this is followed by retraining the backbone while freezing the last linear layer. This approach enforces the model to learn a rich and diverse set of features, leading to better OOD generalization. They show SOTA results on standard OOD benchmark datasets.
strength_and_weaknesses: # Pros:
1. Their Feature Replication Hypothesis is easy to understand and forms a good attempt to explain simplicity bias (SB)
2. Their Feature Reconstruction Regularizer (FRR) scheme to mitigate SB is also easy to understand and implement
3. Overall, the experiments seem promising

# Cons
1. The authors say that the main reason for SB is the implicit bias of SGD (more specifically, the stochastic component of it) to converge to max-margin solutions. I am not sure that this is the only main reason. Can the authors show an experiment using full-batch gradient descent (which should be easily possible on toy datasets used in the paper like MNIST/CIFAR, colored-MNIST etc) and its impact on SB? If SGD is the main cause for SB, then full-batch gradient descent should perform better in this regard. If full-batch gradient descent also shows SB, then it's questionable whether the feature replication hypothesis sufficiently explains the SB issue observed in DNNs.

2. I'm skeptical about the results and conclusions derived from Table-1. How the correlation analysis was performed is not clear. What is the metric used? The thing that bothers me in Table-1 is there is no reason to believe features in the last layer are disentangled wrt color and shape. There is no reason to assume that color and shape will be disentangled by the feature extractor for free (without any explicit constraints to ensure this). Can you better justify why features must be disentangled here? Apart from this toyish setup, do you have any more concrete realistic datasets/experiments/results to support the feature replication hypothesis? This is concerning as its one of the main
points of the paper, and it's not well justified according to me.

3. Another concern is why Eq-1 in the paper (L_FRR) regularizer results in diverse features. The authors do not provide much reasoning or justification in this regard. Intuitively it appears that this regularizer is trying to enforce the full rank of the W linear weight matrix. I suggest authors explain this in more detail to help the readers understand this. And it's also not clear why the explicit enforcement of a full-rank constraint doesn't work. The authors say its a very stringent constraint, but their regularizer also is trying to learn a transformation
that inverts the W matrix, which is essentially trying to enforce the full rank of W. Some more clarity is needed in this regard; why is there such a stark difference between these two seemingly similar strategies?


4. The authors should mention the effect of lambda in Eq-4,5. An ablation study in this regard will help know the sensitivity of the method wrt lambda. How dependent is the method on the choice of this value, and what is the possible range of suitable values for this hyperparameter? Maybe I missed this point in the main paper. If this is already present, please point me to it.

5. I think there is a lack of in-depth analysis of the cause and reasoning for the simplicity bias and the other observations the authors make in the paper. And, in the end, the authors simply seem to propose a regularization strategy. The results seem promising, but I feel it is not well supported or backed by in-depth analysis or theory, and hence I'm not fully convinced of the overall story being portrayed.
clarity,_quality,_novelty_and_reproducibility: The writing is definitely clear, and the method is reproducible for sure. The method and overall idea are fairly simple, and I'm unsure about the novelty, especially since it is just a regularization strategy that empirically seems to work well, but it is still not backed up well by theory or in-depth analysis of any kind.  
summary_of_the_review: The idea of the paper is to impose a regularization scheme to somehow alleviate SB, but why their method works is not very clear, and some of the claims, like how the stochasticity part of SGD (which is what leads to max-margin solutions) is responsible for SB. And are the simple features actually replicated a lot? The simple colored-MNIST eg in the table-1 is dissatisfactory as the authors assume that the backbone or DNN feature extractor disentangles shape and color for free during training. A single neuron in the feature layer could be correlated with both color and shape. How do the authors justify that each neuron is majorly responsible only for a single attribute (color or shape). Even if this holds true for colored-MNIST, it is seldom the case in more realistic datasets of high resolution like ImageNet, CelebA or AFHQ (especially since there is no explicit enforcement of any such disentanglement constraint). So the core hypothesis in the paper, which is feature replication, itself is questionable and not fully justified. I feel that the paper lacks an in-depth analysis of the problem, and certain observations and claims seem superficial and maybe not fully true. The authors should better justify all this. I'm open to changing my review after the rebuttal if the authors convince me of the overall story of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission hypothesizes that one of the underlying reasons behind OOD failures related to the simplicity bias in neural networks is that the features learned tend to include several replicates of simple features, while complex features are not similarly replicated. This mechanism is claimed to lead to greater emphasis on simpler features, which is responsible for corresponding OOD failures. To prevent this, the submission proposes to add an invertibility criterion to the features-to-logits mapping — imposing invertibility would discourage the nonidentifiability implicit in replicated units in the feature dimensions. This regularizer is shown to result in improvements for OOD benchmarks.
strength_and_weaknesses: The feature replication hypothesis is novel and intriguing, and somewhat intuitive. The empirical results seem to indicate improved performance, suggesting this is an effective method on the whole.

While the hypothesis is interesting, I find the empirical validation to be quite weak. There is just one experiment demonstrating it (corresponding to one dataset, one network architecture, one round of training). In effect, the dataset introduces significant variation in the easier-to-learn colour-features (because of the use of ranges instead of monochromatic colouring, and colouring the entire background instead of just the digit) coupled with restriction of shape-variation by using only two digits, which perhaps naturally encourages a network to learn multiple features for different colour-buckets (to account for cases when the correlation does not hold). Of course, the invariant feature is still shape, and this mode of failure might well arise in real life datasets as well. However, this single, highly-contrived experiment does not really serve as a sufficiently reliable model of reality, in my view, and therefore does not really constitute as empirical validation of the hypothesis.

Since the proposed method results in longer training overall, what happens if the baselines are also trained for just as long? It seems likely from the results in Table 2 that ERM by itself might not improve much, but the other baselines (such as RSC) might.

There seems to be some inconsistencies in the aggregated results in Table 3 vs. the expanded numbers in the Appendix. For example,

 — PACS: Table 3 lists SMA as 87.5 while Table 6 lists it as 95.5

 — VLCS: Table 3 lists SMA as 78.2 while Table 7 lists it as 80.7

 — OfficeHome: Table 3 lists ERM as 66.5 while Table 8 lists it twice, once as 67.6; Table 3 lists SMA as 70.6 while Table 9 lists it as 82.0

 — TerraIncognita: Table 3 lists ERM as 46.1 while Table 9 lists it twice, once as 47.8; Table 3 lists SMA as 59.7 while Table 9 lists it as 59.7

 — DomainNet: Table 3 lists ERM as 40.9 while Table 9 lists it twice, once as 44.0; Table 3 lists SMA as 46.0 while Table 9 lists it as 60.0

 — MIRO numbers aren’t shown in the Appendix-tables

The SMA discrepancies in particular might invalidate the claim that a new state-of-the-art is set.
clarity,_quality,_novelty_and_reproducibility: Clarity: The submission is reasonably clear.

Quality: This is a subjective assessment, and I’d rate the quality as being average to mildly-above average.

Novelty: The hypothesis and proposed method appear to be novel.

Reproducibility: There seem to be sufficient details in the submission for meaningful reproducibility.
summary_of_the_review: While I am not really convinced by the empirical demonstration nor the hypothesis, it seems that the idea of imposing a form of identifiability on the final-layer mapping has a positive effect. It is not very clear to me why this is the case, but I am happy to treat this as an empirical discovery with positive consequences. My initial rating is borderline, while I wait for the authors to figure out the discrepancies in the tables — if it turns out that the results are indeed state-of-the-art, I shall revise my scores.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of countering the existence of bias features in the data, with a hypothesis that the classifier learns the simple features (that are not generalizable in OOD) settings with too many weights, validating this hypothesis in simple MNIST experiment, and then offer some analytical discussions with correspondence to SVM, and compare to some SOTA line experiments. Each part of these above discussions seems reasonably good, but there seem some concerns when these pieces are combined into a coherent manuscript. 
strength_and_weaknesses: - strength
    - the flow of the paper (a hypothesis, validation, theoretical discussion, empirical results) goes very well, and logic of the contributions are expanded nicely. 
    - the paper studies an important problem and build another layer to the current knowledge of the problem. 

- weakness
   - how each pieces of the main logic flow are connected together is elusive to me (see detailed comments below)
   - the authors use DomainBed as an empirical testbed for the results, but not following the standard test protocol that DomainBed introduces: it seems there are missing datasets from the DomainBed protocol. 
     
clarity,_quality,_novelty_and_reproducibility: - Clarity
     - It is unclear to me how each of these major components are connected. 
          - how does the thereory supports the learning of a non-simple features? it seems there are not even features that are simple or not formally defined in the theory? 
          - it's also unclear to me how the proposed method is linked to the hypothesis or the theoretical discussion. The hypothesis or the theory does not seem to touch anything about "recoverable" to me. 
          - the method talks about "recoverable", which is different from the wide-accepted term invertible as the title suggests. 

- Quality
     - the theory discussion argues to connect the linear classifier to max-margin (SVM) classifier, but it seems to me the connection is a direct result by building the SVM loss function into the premise of the theory (the constraint in proposition 3.2)
     - the empirical scope does not seem to be comprehensive enough:
          - if the authors decide to use DomainBed, they need to follow the exact settings used in DomainBed, otherwise, they probably cannot just ignore all the methods that are shown worse than ERM by DomainBed
          - there is only one experiment other than this incomplete usage of DomainBed

- novelty
    - hard to evaluate due to the questions in clarity above. 

- reproducibility: 
     - good. replication of experiments are well performed. 

- other: 
     - the issues of learning the biased classifier from the data and countering it has been studied in multiple lines of works, I will recommend the authors to consider a more comprehensive summary of the methods in the first part of the related work section
         - Learning De-biased Representations with Biased Representations 
         - Learning robust global representations by penalizing local predictive power 
         - invariant risk minimization (it's already cited, but probably also worth some discussion in the related work part)
summary_of_the_review: the overall paper is interesting, especially the several major components. However, the internal connection of these components seems missing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Inspired by the previous observation that the network is able to learn complex features even when it is biasing to simple features. This paper propose to further investigate how much the complex features are learned compared with the simple features. They find that the simple features are replicated multiple times over complicated ones. Based on this observation, they propose a novel regularization term and verify the effectiveness with some experiments.  
strength_and_weaknesses: **Strength**: 
- The paper spot an interesting observation that even though the complex features are learned, the amount of it is significantly less than the simple ones. 
- A feature reconstruction regularization is proposed to enhance robustness.

**Weakness**:
- Lack of comprehensive analysis on the observed phenomenon. 
- The writing of the paper is not clear. 
- More experiments need to be done such as on larger datasets and comparing with other SoTA methods. 
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved:
- In Figure 1, (c) and (d) have exactly the same title, but with different contents
- The author mentioned that, in section 3.1, that the number of learnt color feature, and "90% correlation with the color in the input". The reviewer read through paper and did not find how does the author calculate the correlation between a feature and the input. 
- Following the previous point, how does the number of learnt features are counted? Does the author count each dimension of the feature space as a feature? 
- In proposition 3.2 the loss is a infinity norm minimization problem, while in the proof, somehow the analytic solution for least square is provided. This is very confusing, please provide clarification about this.  


summary_of_the_review: Overall, the authors observe a very interesting phenomenon, based on the observation, a regularization term is proposed. But the reviewer feels more comprehensive analysis can be done and included, and the quality of the current version of the paper is not yet ready for publication in ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: As indicated by the reviewers, the paper has the following strength 

"--- Their Feature Replication Hypothesis is easy to understand and forms a good attempt to explain simplicity bias (SB)
--- Their Feature Reconstruction Regularizer (FRR) scheme to mitigate SB is also easy to understand and implement
--- Overall, the experiments seem promising."


justification_for_why_not_higher_score: The paper doesn't sound exciting enough to be a spotlight, as indicated by the reviewers' scores. 
justification_for_why_not_lower_score: Please see the summary of the strength. 

The AC finds that the concerns of the reviewers with a rating 5 seem to be addressed by the authors (even though the reviewers didn't respond to the authors.)
Decision: Accept: poster