Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes augmenting a training set by using an out of distribution auxiliary dataset to help train a robust classifier through randomly sampling labels for the OOD dataset. Model's trained with this additional data perform better than related work on a variety of noisy classification tasks, including out of distribution detection.
main_review: Strengths:

-This proposed method is straightforward and effective

-A thorough analysis is done as to how ODNL effects models' parameters during training, specifically comparing to the most similar method Outlier Exposure.

-Many different experiments are used to demonstrate the effectiveness of ODNL.


Originality: 
-The proposed method is fairly simple, that is, modifying Outlier Exposure to use random 1hot labels rather than the uniform distribution over all labels.

Correctness: 
-Section 3 and 4 are well written and thoroughly analyze the effects of the proposed method.

Clarity: 
-The paper is well written and easy to follow

Relation to prior work: 
-This work appropriately places itself within the recent literature on noisy label classification.

Reproducibility:
-The source code is provided and the method seems easy to apply to any classifier. However, the used auxiliary dataset is no longer publicly available.


limitations_and_societal_impact: Weaknesses:
-The only weakness is the lack of novelty. This is a simple, yet effective, change to OE. And while a detailed analysis is performed, I would love to see the outlier dataset used more than just for crossentropy loss.

The authors do not address negative societal impact of their work. It could be mentioned how additional data increases compute/energy usage.

After seeing the author's response, and the rest of the discussion, I think the discovery that openset instance improve noisy label classification is definitely of interest to the community.  I've increased my score to reflect such.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides an interesting perspective on how to deal with open-set label noise when learning with noise labels. In the noise label community, open-set noises are usually considered to be harmful for generalization, similar to closed-set noises. However, in this paper, the authors present a new perspective that the open-set noise can help generalization against inherent label noises by using proposed Dynamic Noisy Labels (ODNL) regularization on auxiliary open-set samples. In addition, using analysis from the lens of SGD noise, they empirically show the superiority of ODNL over other related regularizations such as Stochastic Label Noise (SLN) and Outlier Exposure (OE).
main_review: The main contribution of this paper is that it is the first work to explore the benefits of open-set noise in learning with noisy labels. However, the authors have missed a very important paper related to their main argument. 

[1] Lee, Saehyung, et al., "Removing Undesirable Feature Contributions Using Out-of-Distribution Data.", ICLR, 2021.

[1] argues that the open-set noise (i.e., out-of-distribution) can help to improve the generalization of DNNs by using uniform-label regularization on the auxiliary open-set noise samples. I think this uniform-label regularization has basically the same effect for improving the generalization of DNNs with ODNL, except its dynamicity. 
Although [1] is not focusing on the label noise problem, I think the author should refer to the paper and clarify the difference and advantages against it. Also, the statement like “open-set noises are always considered to be harmful to the training of DNNs” should be modified.

Meanwhile, the analysis through SGD noise and comparisons to other regularizations such as SLN and OE are clear. In figure 3, with the convex combination of open-set noise and closed-set noise, the importance of open-set noise is clearly shown.
In the experiment, as shown in Table 1, the proposed method has advantages in its applicability to various existing methods because of its simplicity. Also, it showed a significant performance boost combined with other baselines. The out-of-distribution detection performance was also enhanced by their proposed method.

Although the authors provide insightful analysis and impactful methods that leveraging open-set noise to improve robustness against label noise, they missed out on an important related work that seems to have a very similar design and effect with the proposed one. In this concern, I recommend a score marginally below the acceptance threshold.

limitations_and_societal_impact: An important related work that has a similar design and effect to the proposed method is missing.
There is no detailed description for Figure 2; what is x- and y-axis, how much auxiliary open-set noise data is used.
There might be a notation error in line 161. $f(\tilde{x}, \theta)$ seems to be $\tilde{f}(\tilde{x}, \theta)$.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper empirically shows that open-set noisy labels can benefit the robustness against inherent noisy labels in the training data, and proposes a simple but effective regularization method through introducing open-set data with Dynamic Noisy Labels (ODNL) into the training process. Specifically, the ODNL utilizes the open-set data with randomly generated labels in training, to exploit the extra capacity of neural networks, and thus makes the network avoid the memorization of noises in training data. The model trained with ODNL can achieve a better capability on Out-of-Distribution detection tasks. Extensive experiments show the effectiveness of the proposed method.
main_review: This paper proposed a method for learning with noisy labels called Open-set data with Dynamic Noisy Labels (ODNL). Specifically, the ODNL utilizes the open-set data with randomly generated labels in training, to consume the extra capacity of neural networks, and thus make networks avoid the memorization of noises in training data. The model trained with ODNL can also achieve a better capability on Out-of-Distribution detection tasks. Extensive experiments show the effectiveness of the proposed method.

This work is somewhat novel since it first introduces open-set auxiliary datasets in learning with noisy labels. It also provides comparisons with other methods which deal with learning with noisy labels like label randomization and outlier exposure.
For the quality of this paper, the whole process is technically sound. 

However, there are also some points that need to be improved. 

First, the selection of a closed-set dataset should be reconsidered. Since the 80 Million Tiny Images dataset is collected from the web while CIFAR-5m is generated through Denoising Diffusion Probabilistic Models (DDPM), the comparison of using closed-set or open-set dataset may be not sufficient enough to reach the conclusion because we cannot make sure that images generated form DDPM could have an identical distribution as real data. Secondly, it would be better if the hyperparameter \eta is indicated in the experiment results. Thirdly, a curve of performance for models with different percentages of symmetric noise labels could better illustrate the improvement of the method. Finally, since an intuitive interpretation of this method is that it consumes insufficient capacity and makes the model hard to memorize the noises, a comparison experiment between using the proposed method on a small neural network, like ResNet-18, and a big neural network, like WRN, is quite meaningful. Here we can see that the improvement between standard method and standard method with ODNL in the paper (trained with WRN-40-2) is higher than those when trained with ResNet-34, which is shown in the appendix, under open-set and dependent noise, but lower under asymmetric noise.

And for clarity, this paper is well organized and written clearly. It is a bit of a pity that Table 3 is not close enough to its corresponding context, but that is okay.

As for the significance, the proposed method is simple while the improvement reported in the paper is quite impressive. This method could be a new way for learning with noisy labels if the claim of the ineffectiveness of using a close-set dataset as an auxiliary dataset could be further supported.

limitations_and_societal_impact: The authors have addressed some limitations about the work, like instance-dependent open noisy labels and using a synthesized auxiliary dataset. However, more discussions or possible solutions about the limitation of their method should be listed out, such as: 
(1) The discussion of the ineffectiveness of using closed-set auxiliary data in training. Since the 80 Million Tiny Images dataset is collected from the web while CIFAR-5m is generated through Denoising Diffusion Probabilistic Models (DDPM), it is not sufficient enough to reach the conclusion. 
(2) The intuitive interpretation of consuming ‘insufficient capacity' could be further explained by using models with different depths and widths.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a training method for improving the robustness of deep neural networks against inherent label noise. Specifically, the authors first found that the open-set noise is helpful for the generalization and robustness of a deep neural network, and then proposed the strategy that use an additional training set with open-set noise as the regularization in the training objective. Experiments showed improved performance by the proposed method against inherent label noise.
main_review: The counter-intuitive discovery that the open-set is not harmful but even helpful for the robustness of neural networks is interesting, and worth to be paid more attention to. Based on this, the proposed robust training method is simple to implement yet effective.

However, there still exist some issues to be addressed:

1. Theoretical justification. The authors provided theoretical analysis for the effectiveness of the proposed method from the perspective of SGD noise. However, it seems that this is not that essential, since there can be simpler method to add SGD noise other than introduce the additional data with open-set noise. The authors also explained the effectiveness by claiming that the capacity of the neural networks can be first consumed by the open-set noise and then the inherent noise. However, there is no theoretical guarantee for this.

2. Implementation issue. The proposed method introduces two additional hyper-parameters to be tuned, i.e, the size of the auxiliary dataset and $\eta$. While these hyper-parameters can be tuned by validation as mentioned by the authors, it could increase the computational cost. Besides, as can be observed from Figure 4 can claimed by the authors, the decreasing trend of test accuracy caused by inherent noise can be avoided with a larger $\eta$, which raises a question that what the performance could be if further increasing $\eta$. For the size of the auxiliary dataset, as can be observed from Figure 1 of the appendix, it affects the performance with a relatively large extent, while with no consistent trend, which increases the difficulty in tuning it. The authors also claimed that the proposed method is computational inexpensive, which I do not quite agree, since enlarging the training set can definitely increase the computational complexity.

3. Limited improvement on real world dataset. As can be seen from Table 3, the performance improvement solely by the proposed method is not that significant than solely by DivideMix. Considering the combination of the two methods, DivideMix improves the proposed method more than reverse.
limitations_and_societal_impact: The authors did not mention the limitations of their method. However, in my opinion, the major limitation is the practical implementation issue as mentioned in the "Main Review" part, i.e., additional hyper-parameters and extra computational cost.
review_title: null

Meta-review: UPDATE: The revision from the authors has been reviewed.  After some back-and-forth with the authors to discuss the details of the 300K Random Images dataset that they have chosen to use, the paper has been officially accepted.

----

Reviewers generally appreciated the paper's observation regarding the benefit of open-set label noise as being interesting and non-obvious. The simplicity and effectiveness of the technique, coupled with the theoretical analysis, were also appreciated. Some concerns were however raised on:

(1) choice of hyper-parameters

(2) relation to OAT and other out-of-distribution regularization schemes

Point (1) was convincingly addressed in the response. There was some debate about point (2). From my reading, I do agree that there are conceptual similarities between the proposed technique and OAT. However, I also take the authors' point about the similarity being closer to OE, a baseline which the authors discuss and compare against. Further, even when considered as a variant of OAT for the label noise setting, the present paper contributes to theoretical understanding of such out-of-distribution regularization schemes.

The authors are encouraged to incorporate the reviewers' suggestions, include a discussion of OAT, and contrast their method to other regularization schemes for label noise (Wasserstein Adversarial Regularization (WAR) on label noise, ICLR 2020; Robust early-learning: hindering the memorization of noisy labels, ICLR 2021).

It was discovered late in the review process that this paper makes use of the 80 million tiny images dataset, which has been retracted (https://groups.csail.mit.edu/vision/TinyImages/).  Following the NeurIPS ethical guidelines (https://neurips.cc/public/EthicsGuidelines), this dataset should not be used.  For this reason, the paper is being conditionally accepted.
Decision: Accept (Poster)