Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The problem of interest is to train deep neural network models with few labelled training samples. The specific assumption is there is a large pool of unlabelled data, and a heuristic function that can provide label annotations, possibly with varying levels of noises, to those unlabelled data. The adopted learning model is of a student/teacher framework as in privileged learning/knowledge distillation/model compression, and also machine teaching. The student (deep neural network) model will learn from both labelled and unlabelled training data with the labels provided by the teacher (Gaussian process) model. The teacher also supplies an uncertainty estimate to each predicted label. How about the heuristic function? This is used for learning initial feature representation of the student model. Crucially, the teacher model will also rely on these learned features. Labelled data and unlabelled data are therefore lie in the same dimensional space. 

Specific questions to be addressed:
1)	Clustering of strongly-labelled data points. Thinking about the statement “each an expert on this specific region of data space”, if this is the case, I am expecting a clustering for both strongly-labelled data points and weakly-labelled data points. Each teacher model is trained on a portion of strongly-labelled data, and will only predict similar weakly-labelled data. On a related remark, the nice side-effect is not right as it was emphasized that data points with a high-quality label will be limited. As well, GP models, are quite scalable nowadays (experiments with millions to billions of data points are available in recent NIPS/ICML papers, though, they are all rely on low dimensionality of the feature space for optimizing the inducing point locations).  It will be informative to provide results with a single GP model. 
2)	From modifying learning rates to weighting samples. Rather than using uncertainty in label annotation as a multiplicative factor in the learning rate, it is more “intuitive” to use it to modify the sampling procedure of mini-batches (akin to baseline #4); sample with higher probability data points with higher certainty. Here, experimental comparison with, for example, an SVM model that takes into account instance weighting will be informative, and a student model trained with logits (as in knowledge distillation/model compression). 

review_title: From modifying learning rates to weighting samples with an estimate of uncertainty in label annotations?

final_score: rating: 7: Good paper, accept
review_content: This paper suggests a simple yet effective approach for learning with weak supervision. This learning scenario involves two datasets, one with clean data (i.e., labeled by the true function) and one with noisy data, collected using a weak source of supervision.  The suggested approach assumes a teacher and student networks, and builds the final representation incrementally, by taking into account the "fidelity" of the weak label when training the student at the final step. The fidelity score is given by the teacher, after being trained over the clean data, and it's used to build a cost-sensitive loss function for the students. The suggested method seems to work well on several document classification tasks. 

Overall, I liked the paper.  I would like the authors to consider the following questions - 

- Over the last 10 years or so, many different frameworks for learning with weak supervision were suggested (e.g., indirect supervision, distant supervision, response-based, constraint-based, to name a few).  First, I'd suggest acknowledging these works and discussing the differences to your work. Second - Is your approach applicable to these frameworks?  It would be an interesting to compare to one of those methods  (e.g., distant supervision for relation extraction using a knowledge base), and see if by incorporating fidelity score, results improve. 

- Can this approach be applied to semi-supervised learning? Is there a reason to assume the fidelity scores computed by the teacher would not improve the student in a self-training framework?

- The paper emphasizes that the teacher uses the student's initial representation, when trained over the clean data.  Is it clear that this step in needed? Can you add an additional variant of your framework when the fidelity score are  computed by the teacher when trained from scratch? using different architecture than the student?
 
 - I went over the authors comments and I appreciate their efforts to help clarify the issues raised.
review_title: Overall, a nice paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose an approach for training deep learning models for situation where there is not enough reliable annotated data.  This algorithm can be useful because correct annotation of enough cases to train a deep model in many domains is not affordable.  The authors propose to combine a huge number of weakly annotated data with a small set of strongly annotated cases to train a model in a student-teacher framework. The authors evaluate their proposed methods on one toy problem and two real-world problems. The paper is well written, easy to follow, and have good experimental study.  My main problem with the paper is the lack of enough motivation and justification for the proposed method; the methodology seems pretty ad-hoc to me and there is a need for more experimental study to show how the methodology work. Here are some questions that comes to my mind:  (1) Why first building a student model only using the weak data and why not all the data together to train the student model? To me, it seems that the algorithm first tries to learn a good representation for which lots of data is needed and the weak training data can be useful but why not combing with the strong data? (2) What are the sensitivity of the procedure to how weakly the weak data are annotated (this could be studied using both toy example and real-world examples)? (3) The authors explicitly suggest using an unsupervised method (check Baseline no.1) to annotate data weakly? Why not learning the representation using an unsupervised learning method (unsupervised pre training)? This should be at least one of the baselines.
(4) the idea of using surrogate labels to learn representation is also not new. One example work is "Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks". The authors didn't compare their method with this one.
review_title: well written, easy to follow, and have good experimental study but lacks enough motivation and justification for the proposed method

Meta-review: This paper introduces a student-teacher method for learning from labels of varying quality (i.e. varying fidelity data). This is an interesting idea which shows promising results.

Some further connections to various kinds of semi-supervised and multi-fidelity learning would strengthen the paper, although understandably it is not easy to cover the vast literature, which also spans different scientific domains. One reviewer had a concern about some design decisions that seemed ad-hoc, but at least the authors have intuitively and experimentally justified them.
Decision: Accept (Poster)