Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on the learning-from-crowds problem when there is only one (or very few) noisy label per item. The main framework is based on the Dawid-Skene model. By jointly update the classifier weights and the confusion matrices of workers, the predictions of the classifier can help on the estimation problem with rare crowdsourced labels. The paper discusses the influence of the label redundancy both theoretically and empirically. Results show that with a fixed budget, it’s better to label many examples once rather than fewer examples multiple times.

The model and algorithm in this paper are simple and straightforward. However, I like the motivation of this paper and the discussion about the relationship between training efficiency and label redundancy. The problem of label aggregation with low redundancy is common in practice but hardly be formally analyzed and discussed. The conclusion that labeling more examples once is better can inspire other researchers to find more efficient ways to improve crowdsourcing.

About the technique details, this paper is clearly written, but some experimental comparisons and claims are not very convincing. Here I list some of my questions:
+About the MBEM algorithm, it’s better to make clear the difference between MBEM and a standard EM. Will it always converge? What’s its objective?
+The setting of Theorem 4.1 seems too simple. Can the results be extended to more general settings, such as when workers are not identical?
+When n = O(m log m), the result that \epslon_1 is constant is counterintuitive, people usually think large redundancy r can bring benefits on estimation, can you explain more on this?
+During CIFAR-10 experiments when r=1, each example only have one label. For the baselines weighted-MV and weighted-EM, they can only be directly trained using the same noisy labels. So can you explain why their performance is slightly different in most settings? Is it due to the randomly chosen procedure of the noisy labels?
+For ImageNet and MS-COCO experiments with a fixed budget, you reduced the training set when increasing the redundancy, which is unfair. The reduction of performance could mainly cause by seeing fewer raw images, but not the labels. It’s better to train some semi-supervised model to make the settings more comparable.

review_title: A good paper with relatively pool experiments.

final_score: rating: 7: Good paper, accept
review_content: The authors proposed a supervised learning algorithm for modeling label and worker quality. Further utilize it to study one of the important problems in crowdsourcing - How much redundancy is required in crowdsourcing and whether low redundancy with abundant noise examples lead to better labels.

Overall the paper was well written. The motivation of the work is clearly explained and supported with relevant related work. The main contribution of the paper is in the bootstrapping algorithm which models the worker quality and labels in an iterative fashion. Though limited to binary classification, the paper proposed a theoretical framework extending the existing work on VC dimension to compute the upper bound on the risk. The authors also showed theoretically and empirically on synthetic data sets that the low redundancy and larger set of labels in crowdsourcing gives better results. 

More detailed comments
1. Instead of considering multi-class classification as one-vs-all binary classification, can you extend the theoretical guarantee on the risk to multi-class set up like Softmax which is widely used in research nowadays.
2. Can you introduce the Risk -R in the paper before using it in Theorem 4.1
3. Is there any limit on how many examples each worker has to label? Can you comment more on how to pick that value in real-world settings? Just saying sufficiently many (Section 4.2) is not sufficient.
4. Under the experiments, different variations of Majority Vote, EM and Oracle correction were used as baselines. Can you cite the references and also add some existing state-of-the-art techniques mentioned in the related work section.
5. For the experiments on synthetic datasets, workers are randomly sampled with replacements. Were the scores reported based on average of multiple runs. If yes, can you please report the error bars.
6. For the MS-COCO, examples can you provide more detailed results as shown for synthetic datasets? Majority vote is a very weak baseline. 

For the novel approach and the theoretical backing, I consider the paper to be a good one. The paper has scope for improvement.

   
review_title: Novel approach and a well written paper. Need more details on theoretical guarantees for multi-class settings and detailed analysis on real world data.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method for learning from noisy labels, particularly focusing on the case when data isn't redundantly labeled (i.e. the same sample isn't labeled by multiple non-expert annotators). The authors provide both theoretical and experimental validation of their idea. 

Pros:
+ The paper is generally very clearly written. The motivation, notation, and method are clear.
+ Plentiful experiments against relevant baselines are included, validating both the no-redundancy and plentiful redundancy cases. 
+ The approach is a novel twist on an existing method for learning from noisy data. 

Cons: 
- All experiments use simulated workers; this is probably common but still not very convincing.
- The authors missed an important related work which studies the same problem and comes up with a similar conclusion: Lin, Mausam, and Weld. "To re (label), or not to re (label)." HCOMP 2014.
- The authors should have compared their approach to the "base" approach of Natarajan et al. 
- It seems too simplistic too assume all workers are either hammers or spammers; the interesting cases are when annotators are neither of these.
- The ResNet used for each experiment is different, and there is no explanation of the choice of architecture.

Questions: 
- How would the model need to change to account for example difficulty? 
- Why are Joulin 2016, Krause 2016 not relevant?
- Best to clarify what the weights in the weighted sum of Natarajan are. 
- "large training error on wrongly labeled examples" -- how do we know they are wrongly labeled, i.e. do we have a ground truth available apart from the crowdsourced labels? Where does this ground truth come from?
- Not clear what "Ensure" means in the algorithm description.
- In Sec. 4.4, why is it important that the samples are fresh?

review_title: Well-written paper with extensive experiments, but missing a realistic non-simulated experiment, as well as some comparisons and related work.

Meta-review: This paper provides an important discussion about the relationship between training efficiency and label redundancy. The updates to the paper will improve the paper further. Reviewers found the paper interesting, well written, and addresses and important problem.
Decision: Accept (Poster)