Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Deep neural networks become inefficient when the datasets contain noisy class labels. There are two types of robust techniques that are applied in the presence of the noisy labels: a) noise-robust loss functions; b) removing noisy data by detecting them. Most existing detection methods are dependent on the loss values, while such losses may be biased by corrupted classifier. This paper proposes a method that attempts to alleviate this issue by extracting key information from representations, but without using explicit knowledge of the noise rates.

The paper proposes a new method for filtering noisy data. It focuses on each data’s latent representation dynamics and measure the alignment between the latent distribution and each representation using the eigen decomposition of the data covariance matrix. This framework is coined as filtering noisy instances via their eigenvectors (FINE). It provides a detector with derivative-free simple methods with some theoretical guarantees. It provides a provable evidence that FINE allows meaningful decision boundary made by eigenvectors in latent space. It supports the theoretical analysis with various experimental results regarding the characteristics of the principal components extracted by the FINE detector.

It proposes three applications of the FINE: sample-selection approach, semi-supervised learning approach, and collaboration with noise-robust loss functions. It empirically validates that a sample-selection learning with FINE provides consistently more superior detection quality and higher test accuracy than other existing alternative methods such as Co-teaching families, TopoFilter, and CRUST. Further, experimental results show that the proposed methods consistently outperform corresponding baselines for all three applications on various benchmark datasets.
main_review: The proposed technique for detecting noisy data based on the degree of alignment between the representations and the eigenvector of the representations' covariance matrices for all classes is new and meaningful. The theoretical guarantees given in Theorem 1 are not very innovative but nonethless help understand the proposed approach. Further, numerical results for the case of "Sample-selection" shows that the proposed FINE approach consistently outperforms the baseline existing approaches for all the symmetric/assymetric noise settings on CIFAR-10 and CIFAR-100 datasets. These numerical results are the strongest part of the paper. Experiments for the setting of Semi-supervised learning show little gain over the baseline approaches. For all the numerical experiments, authors do compare their approach with multiple state-of-the-art competitive approaches. 

The major limitation is that theoretical contribution of the paper is not very innovative and does not take into account the fact that the representations are learned dynamically using a noisy dataset. So, the original claim of the paper that their approach is different from the existing methods as in the existing methods are biased by the corrupted classifier is not true. Their proposed approach FINE suffers from the same issue as the representations that are used to compute the degree of alignment are latent representations of a corrupted classifier (a classifier leanred using noisy dataset). 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is about learning with noisy labels. The authors propose a method which selects clean and noisy examples by first computing a covariance matrix (for each class separately) based on the learned representations. They then compute the first eigenvector corresponding to the largest eigenvalue from the covariance matrix. The "cleanness" measurement then becomes the inner product between the original representation and the eigenvector. The authors show multiple applications of their approach, such as sample selection, semi-supervised learning etc., and show that their method consistently improves the classification accuracy.
main_review: This paper proposes a simple but effective method based on the eigenvectors of representations. As far as I know, this is a novel approach, and is also supported by theoretical analysis in the paper. The authors also show that it can scale up to the large datasets by computing the eigenvectors on a smaller subset. 

The paper is also well written and easy to understand. I also appreciate the fact that it can be easily combined with various existing learning with noisy labels approaches to further improve the accuracy. Overall, it's a nice paper to read with convincing results. However, I have some minor comments:

- I think the paper can benefit from one baseline to show why the covariance matrix and eigenvectors are needed. More specifically, instead of computing the first eigenvectors, one can compute the class mean vector v_c, where v_c = 1/n_c \sum z_i where all z_i belongs to class C, and n_c is the number of elements in class C. One can then compute the FINE score based on this vector, i.e. f_i = <v_c, z_i>. This comparison would make it more convincing to show why we need to compute the eigenvectors.

- The proposed method brings minimal improvements in the real-world dataset experiments. Can the authors comment on why this is the case? Have they tried other real-world noisy datasets, e.g. Webvision, Food-101N?

- Nitpick: I was initially confused by the terminology "covariance matrix" used throughout the paper. The confusion was mainly due to the lack of notation in the paper, i.e. the reader does not know if z_i is considered as a dx1 vector or 1xd. I assumed that it was it was dx1, which would make the "covariance" matrix dxd, as in PCA. However, after looking at the code (in supplementary material), it seems like that the "covariance matrix" is actually nxn, where n is the number of vectors in a class. Can the authors clarify this? If it is indeed nxn, perhaps calling it a Gram matrix would be better?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes FINE detector to detect samples with noisy labels. Specifically, authors first construct data covariance matrix by using the latent representation of each training sample, then perform eigen-decomposition on the data covariance matrix to generate the first eigenvector for each class. The authors state that the representation of clean samples have larger alignment score with eigenvector than noisy ones. The authors also provide theoretical results to analyze the robustness of FINE detector and shows this detector can be applied to multiple approaches including sample-selection, semi-supervised learning and collaboration with noise-robust loss functions. Experiments are conducted on CIFAR-10, CIFAR-100 and Clothing1M.
main_review: ***Strength***

**1:** The noisy label detector is simple and novel. Necessary theoretical results are also provided to verify its robustness. 

**2:** I especially like the analysis part in Figure 3. It coincides with the theorem to help better understand the FINE detector.

**3:** Experimental results on three applications are impressive. It shows that FINE can be combined with other approaches to further improve network performance.

***Concerns***

**1:** Since the noise rate is unknown, I am worried that FINE is sensitive to the hyper-parameter $\zeta$ in Algorithm 1.  Authors should clarify the value of $\zeta$  for each experiment setting and perform sensitivity analysis on $\zeta$.

**2:** The noise types in the paper are symmetric and asymmetric label noise which are assumed to be feature-independent. I am wondering how FINE performs on a more challenging setting such as feature-dependent label noise.  

**3:** FINE only uses the first eigenvector. If it is possible to use multiple eigenvectors to calculate alignment scores. Is the result better than using only one single eigenvector?
limitations_and_societal_impact: Authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new method for learning with noisy labels. The proposed method belongs to the research direction of data-selection, and aims to better select the clean data out of noisy samples. The key component of the proposed method is a novel operation, named _filtering noisy labels via their eigenvectors_ (FINE). FINE first computes the covariance matrix of the feature representation for the noisy samples. Then the principal eigenvector is extracted from the covariance matrix via eigen decomposition. Afterwards, the similarity score between the feature vector of each sample and the eigenvector is computed. Finally, the computed similarly scores are used to fit a Gaussian Mixture Model, which predicts the probability that a sample is clean or noisy. The proposed FINE approach is simple and computationally feasible. Theoretically it is shown that FINE is able to approximate the eigenvector of clean data with guaranteed accuracy under a few reasonable assumptions. In practice, FINE is demonstrated to be effective by integrating it into three different kinds of label noise learning approaches. The experimental results show that FINE brings (much) performance gain over previous methods across several synthetic and real-world noisy datasets.

To summarize, the contributions of this paper include:

1. A new method, named FINE, is developed to better select the clean data out of noisy dataset. Theoretically it is shown that FINE is able to approximate the true eigenvector of clean data with guaranteed upperbound. Also, it is shown that the data selection accuracy of FINE has guaranteed lower bound in terms of precision and recall.

2. This paper also includes a few illustrative experiments to explain the characteristics of the proposed method. These experiments are helpful.

3. In practice, FINE is integrated into three different kinds of label noise learning approaches, and (significant) performance gain is observed over different datasets, involving the synthetic and real-world noisy data.
main_review: Overall, this paper is well motivated and has a clear organization. I think this paper is novel in terms of technical contribution and experimental evaluation:

1. The proposed data-selection method is reasonably designed, with theoretical analysis and insights. The proof the theory is complete and illustrates the key ideas (yet with some typos and unclear definitions, though). 

2. Besides, to illustrate the characteristics of the proposed method, a series of experiments are conducted (on Page 6). These experiments are interesting, reasonably designed, and helpful in understanding the behavior of the method. In particular, I find Figure 3(a)(b) particularly useful and interesting. Also, the authors show that the proposed FINE can be efficiently computed via a simple approximate yet accurate strategy (i.e., using a subset of the noisy data). Such an experiment validation is useful since it shows the feasibility of FINE on large-scale datasets. 

3. Finally, to show the effectiveness and generality of FINE, the paper investigates the performance gain by integrating FINE into three different kinds of noisy label learning methods. The experimental results on synthetic datasets are impressive. In particular, significant performance gain is achieved under high noise ratios (80% symmetric noise and 40% asymmetric noise).

Besides the above merits of this paper, it also has some weaknesses as follows:

1. The literature review is inadequate. In particular, this paper only mentions and discusses two directions for handling noisy labels: data selection and designing noise-robust loss. Another important direction, i.e., label correction, is ignored. Label correction tries to correct the wrong labels and thus is different from the noise-robust loss and data-selection. Some existing works along this direction include: Li _et al_ (ICCV 2017), Tanaka _et al_ (CVPR 2018), Yi _et al_ (CVPR 2019), Zheng _et al_ (ICML 2020).

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Reference:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Li _et al_. Learning from noisy labels with distillation. ICCV, 2017.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Tanaka _et al_. Joint optimization framework for learning with noisy labels. CVPR, 2018.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Yi _et al_. Probabilistic end-to-end noise correction for learning with noisy labels. CVPR, 2019.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Zheng _et al_. Error-bounded correction of noisy labels. ICML, 2020.

2. The accuracy of the proposed method is heavily subject to the degree of data imbalance. More specifically, to achieve good precision and recall for data-selection, the number of clean samples (i.e., $N_+$) should be much larger than the number of noisy samples (i.e., $N_-$). This phenomenon can be observed from Line 13 of Algorithm 1, Definition 2, Figure 2, and Eq.(21)(22) in the Appendix. More specifically, when $N_- \geq N_+$, the precision and recall would be low, as indicated by the numerator of Eq.(21) and denominator of Eq.(22). This can also be visually inspected in Figure 2. For example, when $N_- \geq N_+$, the principal eigenvector would be skewed towards the noisy samples; as a result, the F-score will be largely hurt because GMM would mistakenly predict many noisy samples as clean. Therefore, only in cases where $N_- \leq N_+$ the proposed method would achieve a good performance. However, in practice this condition may not always hold.

3. While the theoretical results are informative, they are for the binary case. It is unclear if the results would be extended to multi-class scenario.

4. The Definition 2 is not very clear. In particular, "... if the representation $z$ labeled as the same true class belongs to the clean set" is a bit confusing. Does this sentence mean for all the samples whose noisy labels are the same with their underlying clean labels, they are successfully moved to the clean set? I think this sentence should be refined to make it clearer.

5. Figure 3(a)(b) are illustrative. However, it is better to also provide the heatmaps for ground-truth principal eigenvector. This is because in some cases when the computed $u$ is deviated from the ground-truth eigenvector, the heatmaps of $u$ would still appear as Figure 3(a)(b). By providing the heatmaps of ground-truth principal eigenvector, it is clearer to see if the heatmaps of computed $u$ are as bright as those of ground-truth eigenvector, and thereby check if the computed $u$ is accurate enough.

6. A minor concern: In Figure 3(c), although the perturbation values are small for low noise ratio (20%), they increase a lot for a few categories under high noise ratio (80%). Can we still say "FINE has small perturbation values even in a severe noise rate"?

7. For the combination of FINE and Co-teaching, does it mean both the two networks use FINE for data-section within a batch? And one network uses the selected data from another network for training itself? If so, does the batch size need to be large for effectively computing FINE score? Do the two networks still need warmup? Overall I find the description "we train one model with extracted samples by conducting FINE on another model" is unclear. Further clarification is necessary.

8. In Section 4.2.3, it is unclear how to combine FINE with noise-robust loss functions. No descriptions are provided. Does it mean Algorithm 2 is performed here, but the loss function $\mathcal{L}$ is replaced with the noise-robust loss (such as GCE)?

9. In Table 3, it is reported that DivideMix [26] achieves 74.30% test accuracy on Clothing1M. However, after checking the paper of DivideMix, I find it achieved 74.76% as reported in the original paper [26]. Is this a mistake? Overall the performance gain of FINE is marginal on the Clothing1M dataset.

10. There are some typos and grammatical errors in the paper. Besides, some descriptions are unclear. The writing should be further improved. For example:

-  Line 37-38: "loss value between ... and its noisy label". I think this description is confusing. A better one could simply be "the loss value of the noisy classifier". Line 39, "the magnitude", where "the" should be removed. Line 53: "and separates" -> ", and separates" to break the long sentence. Line 92: ", then" -> ", and then". Line 113: "Base on this" -> "Based on this". Line 152: "presented along" -> "presented with". "explore" -> "exploit"? The description "maximize the separation of alignment clusterability" is confusing, and need to be refined. Line 201, the "$v$" should be italic. Line 241: "F-scores of FINE becomes" -> "F-scores of FINE become". "the access of noise rate" -> "the access to noise rate". "and10k"->"and 10k". "methods belong to this" -> "methods belonging to this". "clean and noisy data is" -> "clean and noisy data are".

- For Lemma 3, I think it is more commonly to call it "David-Kahan sin($\theta$) Theorem".

- One typo in Eq.(20). $(\frac{1}{N_+} + \frac{1}{N_+})$ should be $(\frac{1}{N_-} + \frac{1}{N_+})$.

- Line 659, "extension on" -> "extension to".

- Line 699, "there remain little difference as the number of instances is differently used for eigen decomposition". This is misleading, because the time cost changes a lot from 20.1s to 1.1s, when switching from "FINE" to "FINE using 1% dataset".

- Overall, **the writing of this paper should be further improved**.


To summarize, although this paper has several issues, especially those about unclear descriptions and typos, this paper still has a few interesting points. In terms of data-selection, it achieves better performance than the previous works (e.g., CRUST, TopoFilter, etc.) on synthetic datasets. Therefore, I think this paper would potentially meet the standard of NeurIPS after addressing my concerns, and could be **conditionally** accepted.
limitations_and_societal_impact: The authors have addressed some of the limitations. Another limitation of this paper is that, the proposed method is subject to the degree of data imbalance.

This work does not have potential negative societal impact.
review_title: null

Meta-review: The paper proposes a method for identifying noisy examples by looking at their alignment with top eigenvector of the class specific feature covariance matrix. Reviewers have appreciated the simplicity of the method. Author responses were able to address the reviewers' concerns well and the paper is suitable to appear at the conference. 
Decision: Accept (Poster)