Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies a novel phenomenon of two different pathways in DNN training that determine whether it learns robust or non-robust features (or both). The paper uses experiments based on adversarial training to validate their hypotheses, and also devise a toy model under which their hypotheses hold.

Pros
- Interesting contribution
- Provides both empirical and theoretical evidence for their claims

Cons
- Very poorly written

Overall, I think the insight the paper is studying is very interesting. Better understanding of the training dynamics of DNNs is an important problem that merits further study, and the authors are taking an important step in this direction. I also appreciate that the authors provide both empirical and theoretical evidence for their claims.

However, this paper is currently poorly written and was very hard to follow. First, their experimental methodology is poorly explained -- e.g., Section 3 and the first paragraph in Section 4.1. They do not use precise terminology, for instance saying they “perturb each example to the next class” (I assume this means they compute an adversarial example for a “7” that is adversarially labelled “8”), and “relabel each adversarial example with its target label (I assume this means label “8”, but I am not 100% sure). Furthermore, they say accuracy on the “shifted labels” corresponds to “robust accuracy”, yet they never explain why this should be the case (again, I think I understand after several re-reads, but the lack of explanation makes it hard to be sure).

Along the same lines, they never explain *why* they are using their methodology. For instance, I’m not sure exactly how Section 3 is connected to Section 4. They say the point is to establish that “a neural network being unable to learn a generalizing model as evidence that there are no highly predictive non-robust features”, but I don’t understand why they need to replicate the results of Nakkiran (2019) to make this claim. Maybe they are replicating to be careful (which is good!), but they never say so. In addition, I also don’t understand this claim very well; e.g., what is the purpose of this connection? Is it to establish their pathways? Section 4 similarly lacks explanation of their methodology.

Finally, it would also be helpful to have an explanation of why these two pathways are interesting. I do think there is some intrinsic value in decomposing different learning pathways in DNNs, and I can imagine several possible implications, but concretely pointing out some implications of these hypotheses would be very helpful.

-------------------------------------------------------------------------------------------------------------------------------

Post rebuttal: I appreciate the improvement in clarity of the paper; however, I think some work remains to improve the clarity of the paper, especially regarding why the two pathways are interesting. I think expanding on potential implications of this mechanism would be very helpful, in addition to the addressing the remaining issues raised by the other reviewers.
review_title: Interesting contribution but very poorly written

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper posits some phenomena on neural network training: 1. With some proper regularizing effect, NN training tends to learn predictive robust features (and weakly predictive non-robust features) first and non-robust features next. 2. Without regularization, NN training does a similar thing as case 1 first but does not learn predictive non-robust features and overfits the training examples. 


I find the results vague and hard to interpret. The paper is written in a sloppy way. The intuition is clear but many experimental settings are quite hard to follow. Specifically, the paper constantly omits names of the model or dataset mentioned in the context and refer to them with "it" or very long descriptive language later, which is quite ambiguous or informal. The discussions and results are not very well-organized. It is also not clear what is the main result and contribution. 

The empirical studies seem to be very far away from all the terms "gamma-robust features" and "rho-useful features" defined before. It is tested on some contrived test sets that vaguely correlated with the so-called robust/non-robust accuracy. I also don't understand how the empirical studies verify that with both procedures, they first learn "weakly predictive non-robust features". What even is "weakly predictive non-robust features" and how is it reflected in plots?


The biggest problem is the results do not give us constructive ways to improve the training. Even if the paper could give a strong empirical verification that a network overfits due to the fact that it does not learn predictive robust features without proper regularization, it is unclear to me how we could take action to improve the training procedure. The contributions of the paper are not very obvious to me. 

review_title: hard to interpret the contribution of this paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This work studies the learning dynamics of neural networks in terms of robust and non-robust features. In particular, the authors argue that depending on various factors (e.g. learning rate, data augmentation), neural networks will have learning dynamics according to 1 of 2 pathways. Neural networks will either (1) first learn predictive robust features and weakly predictive non-robust features, followed by predictive non-robust features; or (2) only learn robust features, then overfit the training set, thereby not learning non-robust features. The paper has a good discussion expanding upon the robust/non-robust features model of Ilyas et. al. 2019, interesting experiments measuring what features the models is learning, and a digression that presents further results on non-robust features in different datasets.

Recommendation and Explanation:

I would currently recommend a rejection, but I believe the paper is on the borderline and may be improved after clarifications.

The strengths of this paper to me are the discussion in section 2 regarding the robust and non-robust features model of Ilyas et. al, and the experiments measuring “robust accuracy” (accuracy derived from robust features only) and “Non-robust Accuracy” (accuracy derived from non-robust features only). It was interesting to see the discussion of contaminated robust features and the distinction between weakly predictive and highly predictive non-robust features. The experiment relating adversarial transferability and non-robust features is clear and shows that transferable adversarial examples likely have more non-robust features than non-transferable adversarial examples. Overall, the paper has good ideas and studies a topic that merits further investigation.

The weaknesses of this paper to me are unclear explanations of Li et. al 2019, some confusion I have about Pathway 1 vs. Pathway 2, and problems with the organization of the paper that could be improved (e.g. a definition that was abandoned after its introduction). After reading the paper, I have many more questions regarding what the authors did. In rough order of their appearance in the paper:

Section 2: In Ilyas et. al, don’t their experiments show that models generalize to the original distribution (and not to the distribution with flipped labels)? Does this mean that there are not many “contaminants” being learned? How can you evaluate if a robust feature contains “contaminants”?

Section 2: You propose a new definition to exclude contaminated robust features (which are non-robust features), but I did not see you use it again. Is there a way to leverage this definition to further our understanding?

Section 4.1: A diagram explaining what each type of accuracy means could improve clarity.

Section 4.2: How many of the changes described (LR, weight decay, data augmentation) are necessary for transitioning from Pathway 1 to Pathway 2? Can you perform an ablation study to see if any one factor is the most important?

Section 4.2: Is Pathway 2 ever preferable to Pathway 1? Or does robust accuracy matter less than overall accuracy? Explaining when one might be better than the other would be insightful.

Section 4.3: I feel that using the terminology “easy-to-generalize” (as is written in the Li et. al 2019 paper) as opposed to “well-generalizing” is more clear. “Well-generalizing”/”not so well-generalizing” seems to imply that those features will lead to good/bad generalization accuracy, but the Li et. al 2019 paper describes “easy-to-generalize”/”hard-to-generalize” features as those that are not noisy/noisy, and thus easy/hard to learn from and generalize from. In Li et. al 2019, my understanding is that you can get good generalization accuracy from both easy-to-generalize and hard-to-generalize features; one is simply easier to learn from. Furthermore, Li et. al does not claim that all hard-to-fit features are easy-to-generalize or vice versa; they simply analyze how learning rate matters when the dataset does satisfy those properties. Thus, the authors of this paper need to significantly improve their discussion of Li et. al 2019 and how their work relates to it.

Section 5: Isn’t it unlikely that n<<d in deep learning? If so, this should be mentioned as a caveat.

Section 6: Explaining why the D_rand experiment was not done for earlier sections should be placed earlier, rather than in Section 6.

If these questions can be clarified, I believe that will make the paper better.


Post Rebuttal Update:

After reading the author's rebuttal as well as discussion with other reviewers, I will maintain my score. I do appreciate the changes the authors made, and believe that they improve the paper (especially the new example with cat and dog features and the associated figure). I would encourage the authors to incorporate the remaining feedback, as it will be helpful as well.
review_title: Review - Interesting topic and experiments, paper could benefit from improved clarity and organization

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper reveals interesting phenomenons of the training dynamics of SGD on convolutional neural networks for image classification. From carefully designed experiments and simplified theories, it argues that SGD in this setting follows two different pathways, depending on the regularizations and hyperparameters. In the first pathway where regularizations are applied and initial learning rate is large, the network first learns predictive robust features and weakly-predictive non-robust features, and then learns to incorporate predictive non-robust features. In the second pathway where little or no regularization is applied and initial learning rate is small, the network does the same thing as the first pathway in the beginning, but then starts overfitting to noise. Overall, I think this paper may have discovered something interesting, but I also feel uncertain about the correctness of the messages due to the lack of clearness on the experimental results. I strongly recommend the authors to improve the clearness in the next version.

Strengths:
1. The phenomenon demonstrated in this paper seems to be interesting. The authors found that CNNs with SGD behave differently in using robust and non-robust features under different hyperparameters.

2. Simplified theoretical analysis is provided to justify the findings. But I have not carefully checked the correctness.

Weaknesses:
1.  My major concern is about the clearness of the experimental results, from which I cannot tell whether the conclusions are correct. For all the plots, there is not any written definition for the metrics such as "Shifted Test Acc" and "Transfer % to Path 1", and it is quite difficult to guess the definitions given the complicated setups, which makes me confused and feel quite uncertain about my understanding of the paper. For example, in Figure 2, I can guess what "shift" means, but I am not sure whether these accuracies are evaluated on the clean samples or the adversarially perturbed samples from a certain network, and on which subset are the networks tested. With Figure 2, the authors claim that "the network trained on the examples that transfer generalizes well to the true distribution". If the accuracies are evaluated on clean samples, then I cannot understand why "Test Acc" is higher than "Shifted Test Acc" when the model is trained with "All Examples (66%)". Intuitively, even the training examples are perturbed, the model should have higher "Shifted Test Acc" than "Test Acc", since the adversarial examples should have more robust features for the original class than the shifted class. If it is evaluated on perturbed samples, then the network might have just learned to depend on the adversarial perturbations, since we already know the perturbations for "Non-Transferable (33%)" do not transfer well but those for "All Examples (66%)" transfer well. If Figure 2 cannot justify "a neural network being unable to learn a generalizing model as evidence that there are no highly predictive non-robust features", then all the following experimental results will be questionable. 

2. The definition of robust and non-robust features in Section 2 are confusing, not related to the experimental and theoretical results, and might have some minor issues. First, from $\mathbb{E}[y\cdot f_{NR}(x)]>0$ and $\mathbb{E}[\inf_{\delta\in \Delta(x)} y\cdot f_{NR}(x+\delta)]\ll 0$, we cannot conclude that $\mathbb{E}[\inf_{\delta\in \Delta(x)} y\cdot f_{NR}(x+\delta)]\gg 0$, since we know for $\delta=0$,  $\mathbb{E}[-y\cdot f_{NR}(x+0)]<0$. Second, the sum $f_C$ of the robust $f_R$ and weakly-predictive non-robust $f_{NR}$ does not necessarily satisfy $\mathbb{E}[\inf_{\delta\in \Delta(x)} y\cdot f_{C}(x+\delta)] < 0$. Let $\delta_1=\arg\min_{\delta\in \Delta(x)} y\cdot f_{R}(x+\delta)$, $\delta_2=\arg\min_{\delta\in \Delta(x)} y\cdot f_{NR}(x+\delta)$. Given the conditions, it is possible that $f_{NR}(x+\delta_1)>0$, and $f_R (x+\delta_2)\gg 0$, and therefore it is possible that $\mathbb{E}[\inf_{\delta\in \Delta(x)} y\cdot f_{C}(x+\delta)] > 0$, i.e., the "contamination" does not always happen when robust and weakly-predictive non-robust features are both present, which might also be consistent with Remark 1. Section 2 is too confusing that I do not quite follow the motivation for Definition 1, and cannot find where Definition 1 is used later in the paper. 

3. Figure 1 (b) seems to conflict with Figure 3 (b). Figure 1 (b) shows the robust accuracy is around 80% at convergence, but Figure 3 (b) says the targeted adv success rate is around 80%, meaning the robust accuracy should be around 20%. 

4. The number of hyperparameters tried are not dense enough to show "there is no middle ground". I would suggest testing at least 20 points within the current interval for each hyperparameter including lr and weight decay.

5. The title might be a bit misleading. It only describes the process of pathway 1, not pathway 2. 

---
### Updates after rebuttal
I appreciate the authors' efforts in revising the paper into a more rigorous and illustrative way. However, I still feel the current version difficult to follow, and I am not sure whether the conclusions are really correct or not. The conclusions are drawn from assumptions for empirical observations, but the scale of the experiments is not large enough, and the conditions for the assumptions are not specified rigorously. Therefore, I feel quite reluctant in raising my scores.

My final suggestions for improvements in the future version:

1. Instead of using accuracies on $\mathcal{D}_{shift}$ as a proxy for how much of the model’s performance can be attributed to its learning predictive robust features, showing the accuracy of the model on the shifted dataset under adversarial attacks would make it more convincing to me. 

2. I would prefer the figures to be close to the text that elaborate the figures. For example, Figure 1 is in Page 2, but the details of this complicated figure is not discussed until Sec. 4.2.
review_title: Interesting findings, but I feel it questionable

Meta-review: The paper discusses the dynamics of training neural nets and how they are related to features that are robust and predictive (following Ilyas et al). The reviewers had many comments regarding the presentation of the claim and the validity of the empirical results, as well as their unclear practical implications. The authors have improved the writing somewhat but reviewers still thought the manuscript should be substantially improved so that the claims are clearer and empirical validation is more convincing. 
The authors are also encouraged to discuss their results in the context of results on  inductive bias of deep-learning (e.g., results on NTK, rick-regimes, margin maximization etc). 
  
Decision: Reject