Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ===========
Summary:
This paper proposes a new regularization scheme inspired from (virtual) adversarial training to tackle the problem of learning with noisy labels. While based on the adversarial training (AR), it was found that AR does not directly transferable to deal with noisy labels. The author then proposed the Wasserstein version of AR replacing the KL with the Wasserstein distance and its approximate. This gives the proposed  Wasserstein Adversarial Regularization (WAR) which provide considerable robustness improvement on 5 datasets (both classification and segmentation). The correlation between WAR regularization and boundary smoothing is justified both theoretically and empirically with toy examples. The advantage of WAR regularization over existing methods is the flexibility to incorporate intra-class divergence, making it plausible against asymmetric label noise, which is more common in real-world datasets. The authors have done solid work in this paper. The experiment is complete, in terms of the scale, noise settings and comparison to existing works.

I recommend to accept this paper.

Minor suggestions:
It would be interesting to see the performance on the other common type of real-world noise: open-set label noise [1], and may be applied to adversarial training against adversarial examples.
The adversarial regularization was also used in a recent adversarial training paper [2]. A similar idea is adversarial logit pairing which is regularization on logits [3].


References:
[1] Iterative learning with noisy labels. CVPR 2018
[2] Theoretically principled trade-off between robustness and accuracy. ICML 2018
[3] Adversarial logit pairing. arXiv preprint arXiv:1803.06373 (2018).
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
Update after rebuttal:
The rebuttal addresses my concerns. The point raised by reviewer 2 regarding other results on Clothing1M is concerning, but the authors' response is reasonable. Moreover, the proposed method is novel and could be complementary with other methods that achieve good results on Clothing1M. I think this paper would generate good discussion, and I recommend acceptance.

----------------------------------------------------------------------------------------
This paper presents a novel approach for dealing with asymmetric label noise. There are in fact two methods proposed, Adversarial Regularization (AR) and Wasserstein Adversarial Regularization (WAR), both of which derive from the intuition that model smoothing (in the sense of virtual adversarial training) acts as label smoothing when there are noisy labels. The authors prove two statements formalizing this notion and demonstrate the efficacy of AR and WAR in an extensive evaluation.

WAR is an iteration on AR that uses the optimal transport distance between categorical distributions to allow incorporating a cost matrix that encodes class similarities. The authors obtain these similarities from word embeddings of class names.

The evaluation demonstrates that the proposed WAR method achieves state-of-the-art results with the largest gains at high noise levels. In no case does it perform substantially worse than competing methods. Moreover, the AR method proposed as an intermediary to WAR also outperforms all prior work at non-zero noise levels, demonstrating the promise of the underlying approach.

The authors also evaluate on Clothing1M and evaluate on a semantic segmentation dataset for which they synthesize structured, realistic label noise. WAR outperforms prior work on these tasks. Moreover, the semantic segmentation task could act as a useful benchmark for future work.

For these reasons, I recommend acceptance.

Minor points:
Comparing the second and third columns in Figure 1, it looks like lambda has a large effect, so it would be good to know how much tuning lambda requires for the main tasks.

In equation 9, shouldn’t there be a beta in front of the entropy term? This does not break the result.

In equation 10, it looks like you’re multiplying both sides by (1 – epsilon), but you drop the (1 – epsilon) on the left side. This does not break the result.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes to use the optimal transport distance to smooth the output of algorithms dealing with label noise. More specifically, the smoothness is on a controlled distance to the input instances, achieved by using the idea of adversarial learning (or robust learning). Generally, the paper is well-organized, with a good writing, and with sufficient experiments. However, I have major concerns about the technical contribution and the experiment parts.

1. It seems that the authors employed the existing adversarial learning techniques directly to the label noise problem. The technique contribution is quite limited. I think the challenging and interesting part should be about how to learn the metric C. Currently, the authors use the semantic distances between different class labels to set-up C. This intuitively makes sense but not convincing. We expect the authors to put more effort to study C, which may depend on the label noise rates or the geometric information of the instances. An in-depth study of C will make the technical contribution solid.

2. The experiment part looks sufficient but may be problematic. For example, in the subsection about Clothing1M, the authors directly cited the results from others while the experimental setting could be very different. It is not convincing to make the conclusion that the proposed method has the best performance.
review_title: Official Blind Review #2

Meta-review: This article proposes a regularisation scheme to learn classifiers that take into account similarity of labels, and presents a series of experiments. The reviewers found the approach plausible, the paper well written, and the experiments sufficient. At the same time, they expressed concerns, mentioning that the technical contribution is limited (in particular, the Wasserstein distance has been used before in estimation of conditional distributions and in multi-label learning), and that it would be important to put more efforts into learning the metric. The author responses clarified a few points and agreed that learning the metric is an interesting problem. There were also concerns about the competitiveness of the approach, which were addressed in part in the authors' responses, albeit not fully convincing all of the reviewers. This article proposes an interesting technique for a relevant type of problems, and demonstrates that it can be competitive with extensive experiments. ``Although this is a reasonably good article, it is not good enough, given the very high acceptance bar for this year's ICLR. 

Decision: Reject