Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work address the problem of supervised learning from strongly labeled data with label noise. This is a very practical and relevant problem in applied machine learning.  The authors note that using sampling approaches such as EM isn't effective, too slow and cannot be integrated into end-to-end training. Thus, they propose to simulate the effects of EM by a noisy adaptation layer, effectively a softmax, that is added to the architecture during training, and is omitted at inference time. The proposed algorithm is evaluated on MNIST and shows improvements over existing approaches that deal with noisy labeled data.

A few comments.
1. There is no discussion in the work about the increased complexity of training for the model with two softmaxes. 

2. What is the rationale for having consecutive (serialized) softmaxes, instead of having a compound objective with two losses, or a network with parallel losses and two sets of gradients?

3. The proposed architecture with only two hidden layers isn't not representative of larger and deeper models that are practically used, and it is not clear that shown results will scale to bigger networks. 

4. Why is the approach only evaluated on MNIST, a dataset that is unrealistically simple.
review_title: Training with Noisy Labels

final_score: rating: 7: Good paper, accept
review_content: The paper addressed the erroneous label problem for supervised training. The problem is well formulated and the presented solution is novel. 

The experimental justification is limited. The effectiveness of the proposed method is hard to gauge, especially how to scale the proposed method to large number of classification targets and whether it is still effective.

For example, it would be interesting to see whether the proposed method is better than training with only less but high quality data. 

From Figure 2, it seems with more data, the proposed method tends to behave very well when the noise fraction is below a threshold and dramatically degrades once passing that threshold. Analysis and justification of this behavior whether it is just by chance or an expected one of the method would be very useful. 

 
review_title: Interesting paper but lack of experiments

final_score: rating: null
review_content: In your experiments, how well can your EM algorithm recover the "converted
each label with probability p to a different label", from the experiments section?
review_title: Recovering the latent p

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper looks at how to train if there are significant label noise present.
This is a good paper where two main methods are proposed, the first one is a latent variable model and training would require the EM algorithm, alternating between estimating the true label and maximizing the parameters given a true label.

The second directly integrates out the true label and simply optimizes the p(z|x).

Pros: the paper examines a training scenario which is a real concern for big dataset which are not carefully annotated.
Cons: the results on mnist is all synthetic and it's hard to tell if this would translate to a win on real datasets.

- comments:
Equation 11 should be expensive, what happens if you are training on imagenet with 1000 classes?
It would be nice to see how well you can recover the corrupting distribution parameter using either the EM or the integration method. 

Overall, this is an OK paper. However, the ideas are not novel as previous cited papers have tried to handle noise in the labels. I think the authors can make the paper better by either demonstrating state-of-the-art results on a dataset known to have label noise, or demonstrate that a method can reliably estimate the true label corrupting probabilities.

review_title: This paper investigates how to make neural nets be more robust to noise in the labels

final_score: rating: null
review_content: Suppose there are N classes to predict, the proposed methods requires (N+1) sets of softmax with the size of N. For speech tasks, there are normally around 10k output classes or even more. Discussions on the scalability of the method and whether they still work well in those cases would very helpful.   
review_title: Scalability to large number of classes

Meta-review: Reviewers agreed that the problem was important and the method was interesting and novel. The main (shared) concerns were preliminary nature of the experiments and questions around scalability to more classes. 
 
 During the discussion phase, the authors provided additional CIFAR-100 results and introduced a new approximate but scalable method for performing inference. I engaged the reviewers in discussion, who were originally borderline, to see what they thought about the changes. R2 championed the paper, stating that the additional experiments and response re: scalability were an improvement. On the balance, I think the paper is a poster accept.
Decision: Accept (Poster)