Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed a novel framework, Uncertainty Mining Net (UMN), to address the problem of learning on limited labeled data with label noise. First, UMN applied the unsupervised variational autoencoder to learn a more representative latent feature representation by involving large amounts of unlabeled data. Second, in the training process, ELBO integrates information from all the training data and sample-wise uncertainty estimated from the predictions of the updating model and its exponential moving average is incorporated to enable the learning process to focus on the data with reliable labels. Experimental results show that UMN outpuroms several state-of-the-art methods on multiple benchmark datasets. 

UMN will be helpful for a lot of real-world products since annotation is expensive and annotation quality is a regular concern. However, it is subjectively uncertain to define whether it is a large or small dataset without considering application context and model complexity. It would be better if the authors can further provide the error rate changes under different number of labeled training data. Such analysis would provide suggestions regarding when it is necessary to implement UMN. On the other hand, it would be nice to explore how tolerant/sensitive the UMN is to the corruption rate to every single class, especially for a multi-classification problem. In the real world, the data within the same class might be labeled incorrectly more than correctly. It would be better to investigate if UMN is able to identify such situation and correct the labels accordingly. 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper describes a method for learning in semi-supervised settings with label noise. This is an interesting topic with a relatively scarce literature. The proposed method works by first postulating a generative model for the labelled/unlabeled/label-corrupted data. The model is then fitted using a standard variational lower bound maximization.

The approach is elegant, and appears to work empirically well. Unfortunately, I do not seem to really understand the rationale behind the main novelty (even at an heuristic level) of the paper which is contained in Section 3.2.1, namely Equation (10) which describes the intensity of the label corruption. Why is it a sensible idea? There is only 3 lines of comments after equation 10, which seems a bit short since this is the crux of the paper. Also, the notations of equation (10) are very confusing (to me) since the function f(.) was used before to denote another quantity.
Minor remarks:
=============

1. page4 was a bit difficult to digest at first reading since the author did not describe in the main text he form (i.e. factorization structure) of the variational distribution q.

2. second line of Equation 14 seems to be wrong, while the 1st and last line seems correct -- the authors may want to double check

3. the term "labeled loss term" was not properly defined, which makes the reading a bit difficult even if one eventually gets what the authors mean

4. In equation (6) I do not understand why f(eps) = log[(C-1)(1-eps)/eps]. Why isnt it equal to log[p(hat(y)|y)] ?

5. notations z_a, z_1, etc.. do not seem to be consistent throughout the text

6. it is quite surprising to me that, in the low data-corruption regime, the method appears to be competitive/better with consistency-based method such as the MT approach.

7. naive question: why directly modeling \eps as a function of z_a or (z_a, y), possibly parametrized by a neural network, a bad idea?

In conclusion, it feels like the method has a lot of potential (in views of the numerical simulations) -- if the authors could clarify the exposition, this could be a very good contribution to the field. The method appears to be conceptually simple (i.e. postulate a generative model + fit it by maximizing the ELBO + one trick to estimate \eps), which is a good thing -- what is missing, I think, is a real discussion of why the proposed manner to estimate \epsilon is sensible.

Edit after reading [1] 
======================
The proposed generative model is same -- the authors should make this very clear in the paper. Although is is acknowledged: "In this work, we implement the idea in (Langevin et al., 2018)" -- this is only in section "4.1 IMPLEMENTATION DETAILS". After reading (1), it is clear that the novelty of the paper us much less than what I had initially thought. The only difference is in Section 3.2.1, and this Section is far from being satisfying.


[1] "A Deep Generative Model for Semi-Supervised Classification with Noisy Labels", Langevin & al
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Update after rebuttal:
The rebuttal addressed a few of my concerns, but there is still a major issue. Namely, UMN is claimed to work on sample-wise label noise, but there are no experiments to support this (note that this is different from non-uniform class-dependent label noise). As fixing this would require large modifications to the paper, I am keeping my score at weak reject.

----------------------------------------------------------------------------------
Summary:
This paper presents a method for training classifiers in the setting of semi-supervised learning with noisy labels. The proposed method, Uncertainty Mining Net, combines the Mean Teacher method of Tarvainen and Valpola with the M-VAE method of Langevin et al. to estimate the trustworthiness of each input-label pair and weigh its contribution to the loss. This is not an obvious use of the Mean Teacher method, and it seems like a nice idea.

The results are good when controlling for architecture, and the setting is important and underexplored, but there are several concerns I have with the paper in its current form and some parts I would like clarified. At present, the paper is borderline, and I will raise my score if these are addressed.

Major points:
The generative process defined in equations 2 and 3 presents a model for input-dependent label noise, but the corruptions in the experiments are conditionally independent of the input given the true label. What is p(y_tilde | y, z_a) supposed to capture when the true noise model in the experiments follows p(y_tilde | y)?

It seems like the proposed approach would have difficulty with non-uniform label noise, but there is no discussion on this. Adding discussion of this would be good.

What is the purpose of z_b? It seems like a redundant variable in the generative process, since it is only used with y to sample z_a, and it seems like z_a could just be sampled from y.

The caption in Figure 1 says, “For simplicity, we omit the optional consistency loss term between the classifiers of θ and θ 0 for unlabeled data in the figure”, but this is never mentioned again. I would find it interesting to see the combined effect of the Mean Teacher consistency loss with the VAE reconstruction loss, since they are distinct approaches to semi-supervised learning. Did you run experiments with the consistency loss?

Minor points:
The writing is full of grammatical errors and typos, including

“we experiment a CNN architecture with 13 convolutional neural network as well as a ResNet-101 architecture”

“the training of UMN also converge faster”

“For the details of the proof, please refer to the appendix section.”
(This quote is from the appendix!)

“eign decomposition”

“for solving learning model”
review_title: Official Blind Review #1

Meta-review: The paper presents an interesting idea but all reviewers pointed out problems with the writing (eg clarity of the motivation) and with the motivation of the experiments and link to the contest. The rebuttal helped, but it is clear that the paper requires more work before being acceptable to ICLR.
Decision: Reject