Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Performing controlled experiments on noisy data is essiential in understanding deep learning. But there is lack of suitable data. In this paper, the authorsestablish a large benchmark of controlled real-world noise, which is one contribution of it. For this paper, the dataset they established is a main contribution. There is no much contributions in methods. 

Based on the dataset and previous works, the study find something. for example, DNNS generalize much better on real-world noise, and DNNs may not learn paterns first on real-world noisy data, and so on.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ===========
Summary:
This paper introduces two real-world noisy datasets collected from google search based on two existing datasets: Mini-ImageNet and Stanford Cars. The author then conducted a series of experiments comparing 6 existing noisy label learning methods in two training settings: 1) from scratch, and 2) finetuning. Parameters were tuned when necessary. Based on the results, they made a few claims that challenges some of the previous believes in this field. 

===========
My major concerns:
Collection new data using google search then run existing methods has limited contribution. And the new data are related to noisy LABELS only, which does not cover any input noise such as low resolution, abnormal size, black/blank/carton background etc.
The web data searched are of moderate or rather small scales, which doesn’t seem to contribute much to the community. As mentioned in the paper, we already got some real-world datasets such Clothing-1M.
How the new datasets are collected, labeled and mixed are not clear. The numbers do not seem to add up. In text, the authors said they collected 94906/51687 images with annotations. However, in Table 1, there are 39000/8144 in datasets Red Mini-ImageNet/Red Standford Cars. The last paragraph on Page 3, how exactly the negative examples come from? How the “negative” is defined, since you got more than 1 nanotations for each image. “ Following the construction of synthetic datasets, we replace the training images in the original dataset”, refers to which original dataset? The Mini-ImageNet/Red Standford Cars or the web-searched? I don’t quite get it which dataset is creating here. Are the Red noise datasets a mixture of Mini-ImageNet/Red Standford Cars and web-search images, or are they only web-searched images.
The findings are questionable, due to the specific way how the web data are collected. Since the web datasets are created by similarity search based on a pool of seed images (5000), all the searched images are subject to the small number of seed images and the similarity algorithm of google search. Although some filtering was done to reduce test vs training duplication, the entire dataset can be treated as a duplicate of the 5000 seed images, which is roughly 10% of Mini-ImageNet -- the intrinsic noise rate is low. Having this in mind, some of the findings become no surprise at all, for example, 1) DNNs are robust to real-world noise, 2) performance does not decrease much as training progresses, 3) real-world noise is less harmful, and 4) DNN performance does not change much with different robust methods. These are all phenomena of low noise rates.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper contributes a new dataset for testing label corruption robustness. The authors attempted to make the dataset represent "real-world noise." The hamartia is that they used the "search by image" feature from Google, so that "similar" images were what is determined "similar" by a convnet. In this way it is not clear their noise is any more real than what is seen in previous works (such as the Gold Loss Correction paper which used labels from a weak classifier as a source of label noise). The hamartia also makes their experimental findings and takeaways about "real noise" questionable, as it is not clear they are testing real noise but consequences of properties of convnet embeddings. However this paper still contributes a dataset, hence this paper still is some sort of contribution. However, with this flaw, it is not clear that it is enough for ICLR.

Small things:

> robust learning is experiencing a reminiscence in the deep learning era
renaissance?

>  ImageNet architectures generalize well on noisy data when the networks are fine-tuned. Comparing the first and second rows in Fig. 2, we observe that the test accuracy for fine-tuning is higher than that for training from scratch on both Red and Blue noise
This section should cite _Using Pre-Training Can Improve Model Robustness and Uncertainty_ (ICML 2019) since that is a main conclusion of their paper.

> Inception-ResNet-V2 (Szegedy et al., 2017) is used as the default network architectures
Unfortunately they're probably using some form of Inception to compute image similarities, which was how the "real noise" dataset was curated.
review_title: Official Blind Review #1

Meta-review: Thanks for your detailed feedback to the reviewers, which helped us a lot to better understand your paper.
However, given high competition at ICLR2020, we think the current manuscript is premature and still below the bar to be accepted to ICLR2020.
We hope that the reviewers' comments are useful to improve your manuscript for potential future submission.
Decision: Reject