Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the model robustness to noisy labels from the neural network architecture perspective. Specifically, the authors propose to investigate the alignment between neural network architecture and the (noise) label function, and identify the connection between the proposed alignment measure and model robustness to label noise. The alignment hypothesis is one of the key contributions of this paper, i.e., 'model architecture more aligned with (noise) label function --> better model robustness to label noise'. Experiments on both graph neural networks (GNN) and vision tasks support this hypothesis.
main_review: This submission studies the model robustness from the model architecture perspective by investigating the proposed alignment hypothesis.

**Originality**: The proposed alignment measurement is mainly built on Xu et al. [44].

**Quality**: This submission is a complete piece of work and the claims are well supported by the empirical results. The evaluations are reasonable.

**Clarity**: This submission is well-written and easy to follow.

**Significance**: This paper proposes an interesting hypothesis that is well supported by empirical experimental results.


Pros:

1. The 'alignment & model robustness' hypothesis is interesting, and this brings a new perspective to understand the model robustness to label noise.
2. Experiments on both GNN and vision tasks support the proposed hypothesis, and the experiments are well designed.
3. The 'prior knowledge about structures of the target function' argument is inspiring for improving model robustness to label noise for future work.

Cons:

1. The predictive power $P_j(f, \mathcal{N}, \mathcal{C})$ requires clean data, and in the label noise setting it is hard to access a set of clean data.
2. The alignment measurement is not easy to compute. In this submission, several alignment arguments are made by applying prior knowledge. Also, by looking at definition 2, the $\text{Alignment}(\mathcal{N}, f, \epsilon, \delta)$ requires access to the true function/clean label for calculation.

Questions:

1. As mentioned in Line 107-109, the predictive power is evaluated by using linear regression. For classification tasks, are the targets just the one-hot encoding vectors? Also, maybe it is more reasonable to apply logistic regression for classification tasks (which is also a convex problem).
2. For results shown in Figure 3 and Figure 4, could the authors provide the results for models that are randomly initialized? This might provide more information on the alignment between the architecture and the label function.
3. Could the authors provide one or two baselines in Table 1 and Table 2? For example, methods in [12] and [23] (references in the main body).

Overall, I think this paper provides an interesting perspective for understanding the model robustness to label noise. I would suggest `Marginally above the acceptance threshold'. I would also be willing to raise my score if the questions listed above are answered.



======================================[updates]======================================

Thanks for the authors' response. I have read the authors' responses as well as the other reviews. I will stick with my score.

limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the impact of model architecture on the robustness of its representation upon training with noisy labels. The main hypothesis in the paper is that if the model architecture is "aligned" to the underlying task being solved, then it learns such noise robust representations. This hypothesis is tested using two experimental settings, first using graph neural networks and the second on a CNN / MLP architecture. These experiments confirm the hypothesis considered: in the GNN setting, architectures that are able to represent the underlying max node feature task have noise robust representations, and in the CNN / MLP experiments, CNNs performed better at the default CIFAR experiments and MLP at a synthetic task requiring identifying a particular pixel location. 
main_review: Clarity: This paper is very clearly written and reads beautifully. The main hypothesis, the accompanying math and the assumptions made are clearly laid out and easy to understand. The experimental section is also explains all settings in detail, and seems like it should be straightforward to reproduce from the description. 

Originality: I am not an expert on the "learning with noisy labels" literature, but to the best of my knowledge this paper seems to be the first to analyse the importance of the architecture to this noise robustness property, particularly in this line of reasoning regarding the "alignment" of architecture and task. There are no new techniques introduced in this paper, and the only novelty lies in its analysis of the impact of architecture, which I believe is useful.

Quality: This paper is of generally high quality both with regard to theoretical arguments and experiments. Although I see a few issues which I will elaborate below, the claims in the paper are generally well supported by experiments.

Significance: I believe that understanding the inductive biases of model architecture is of high significance to machine learning in general, and the particular tool of function "alignment" is particularly interesting in this regard.

Comments:

1) The definition of alignment in this paper considers target functions which have a naturally decomposable structure as a function composition. For functions without such structure, alignment is the same as sample complexity, and Theorem 1 is a tautology. However it is unclear from the paper which practical problems have such decomposable structure, and where such structure is known in advance. The decomposable structure of the max node degree problem in Figure 1 does not benefit from such decomposability property as the sample complexity of learning Module 1 is the same as learning the entire function (as Module 2 does not require learning). Hence it would be great that the draft clarify on the benefit of the decomposability property, and add some practical examples where such problems might occur (maybe distillation?). If not, then it is better to simplify the definition of alignment to mean only sample complexity, and discard the decomposability property.

2) In Section 4.2, there is a remark that the alignment of the noise function with the network is also an important consideration for robustness to noisy labels. In light of this, does it make sense to update the hypothesis 1, to include an assumption that the noise function is unaligned with the network? 

3) The paper would benefit from adding a section on examining why hypothesis 1 can be true. For instance, it is also plausible that training with noisy labels can completely destroy the internal representations in such a way that they would no longer have useful predictive power. Why does this not happen? Is it that the training loss on noisy samples is non-zero and these samples are effectively not learnt, possibly an artefact of early stopping or other forms of regularization? Or is it due to over-completeness of representations? If not for experiments, a section on speculating on possible reasons for this phenomenon, or providing a proof sketch for Theorem 2 (which seems to hint at over-completeness of representations), might be instructive for future research.
limitations_and_societal_impact: This work has no direct societal impact as it is a fundamental work.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper aims to bridge the connection between the network’s architecture and robustness to noisy labels. I surely agree that this topic is important. However, the presented results are very intuitive rather than theoretical as claimed. The paper does not propose an appliable measurement that has a connection to the robustness and can be obtained empirically (at least qualitatively). The alignment measurement largely follows from existing works and the measurement is not used at all in the presented experiments.
main_review: **Originality** and **Clarity**

The studied topic is novel and important in learning with noisy labels. The paper is clearly written.

**Significance** and **Quality**

The paper claims to propose a formal framework connecting the robustness of a network to the alignments between its architecture and target/noise functions. However, there are several concerns on the significance and novelty.
-	The robustness is measured by the predictive power in representations, which follows from the work [43]. It is simply the expected test loss after training on clean labels. That is, such a measure is a known result rather than a novel contribution.
-	The alignment measure largely follows from the work [44]. It is straightforward to extend Definition 2 (for a deterministic function) to Definition 3 (for a set of all possible functions).
-	The alignment measure cannot be computed in practice (as admitted by the author on line 126), which limited the utility of the proposed framework.
-	The main theorem (Theorem 2) is based on strong assumptions and a simplified noisy setting.
-	The empirical studies have a limited connection to the proposed numerical framework and do not support the main hypothesis (Hypothesis 1). The alignment measure is not directly demonstrated in experiments (since it can not be computed). Instead, the author directly claims that some architectures have higher/lower alignment with the target functions, based on intuition (line 224-227 and line 262-265). Such claims are not convincing and as a result, the experiments fail to draw a connection to the proposed alignment measure.

**Summary**

Given the above comments, the paper fails to present results that could go deeper on the common intuition, i.e., the findings presented in the paper are no more than an intuition: a network with architecture more aligned with the target clean function can be more robust to label noise. In experiments, the alignment is still evaluated based on intuition.

limitations_and_societal_impact: No.
review_title: null

Meta-review: There were a range of opinions on the paper amongst the reviewers. Based on my reading of the paper, the insight is limited and there are serious issues in the way "alignment" is used and presented in the paper. Here are the key reasons for rejection:

1. There is a gap between the notion of "alignment" and what's actually being used to evaluate alignment, which seems to be "representation predictive power"; the former of which is difficult to determine in practice and the latter of which involves training a linear model on top of the second-last layer representation using a set of clean labels. Although the paper shows a result that better alignment implies better representation predictive power, the other direction isn't shown and thus the paper actually seems to claim that better representation predictive power implies better robustness to label noise. In other words, this "alignment" as used in the paper is a qualitative measure and lacks rigor.

2. It is not a surprising result that if the performance of the second-last layer is better, then the model will be more robust to label noise. This appears to come from the fact that you have a better model for the task regardless of how noisy the labels are; and the fact that the representation does better than the model itself comes from the fact that the representation is trained on clean labels while the model is trained on noisy labels. Furthermore, it's already known that DNNs tend to learn useful representations even in the presence of noisy labels.

Decision: Accept (Poster)