Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper addresses the problem that labelled data is often unavailable in the quantities required to train effective models. It deals with classification problems, and proposes a method to obtain more (but weaker) labels data with minimal involvement from human labellers, by asking them to generalize their labelling decisions into rules and then learning restrictions on those rules to avoid learning incorrectly generalized labels. The motivating observation is that human labellers are often able to make such generalizations in much less time than it would take them to apply that rule to a large dataset themselves. This is an interesting idea, especially for cases where labelling capacity is limited. The point being made about the labelling noise not being random in this situation is an interesting one - it might be worth exploring this notion further on its own also in contexts where the source of the noise is unknown.

The presentation of the implementation the authors choose for their proposed approach is clear, and the implementation is sensible. The experimental section includes comparisons to a number of alternative methods, and the authors find that their method outperforms all others, including recent methods for combining (noisy) rule-based labels and (clean) human-sourced labels.

I would argue for accepting this paper. It studies an interesting question, which if answered has the potential to make access to machine learning solutions to certain types of problem significantly cheaper and therefore more widespread. The experiments are well-chosen and show that, depending on the data available and the task, significant gains can be made using the proposed method.

Some remarks on how the paper could become stronger: The type of problem that can be assessed with the proposed method seems to be fairly specific: most tasks studied are classification of natural language utterances. That is a natural class of tasks, since it is easy to imagine how labellers can formulate rules. However, it would have been very interesting if the authors had found ways to allow for more diversity here.  In general, I have the impression that there are more interesting ideas and results to be found in the direction explored by this paper - what about, e.g., allowing the classifier to add rules of its own?

The paper would benefit from some general editing with regards to appearance; for example, the supplementary material sections continue the regular section numbering, instead of having their own; the images are missing captions, and sometimes have somewhat unorthodox axis tick labelling.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper proposes a novel semi-supervised learning paradigm where the algorithm learns from both clean instance-level labels and noisy rule-level labels, and also a simple but effective algorithm as solution. The proposed algorithm employs a set of latent coverage variables to bridge two kinds of supervisions and uses a soft causal constraint on the coverage variables to denoise the noisy labels. Empirically the paper demonstrates the effectiveness of the proposed algorithm with consistent improvements over several baselines on a wide range of classification tasks.

The idea of using macro-level noisy labels as part of the supervision is novel, and it could potentially trigger a paradigm shift on many research areas in machine learning. The proposed methodology is clean but effective, with extensive experimental support. Therefore I vote for accepting this submission.

Minor problems

(1) Abuse of notation \phi in section 2.
(2) "... from traing the classifier ..." in page 4.


More (further) questions

(1) Since each rule can be regarded as experts or weak learners, how is this work related to learning strong learners from weak learners (boosting/ensemble)?
(2) Is it possible that the algorithm can incorporate more information of the rules, for example, the structure of the logical formulas?
(3) Is it possible to generalize the idea to RL?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In case of a lack of labeled data, human-designed rules can be used to label the unlabelled data. This paper proposes a better rule-based labeling method by restricting the coverage of the rule, which is based on the assumption that the rules can be applied to a local region but can not be 'over-generalized' to the whole sample space. The coverage of the rule is represented by a conditional distribution, which is parameterized as a neural network and jointly learned with the classifier network.

I think this paper is tackling an important problem in machine learning, and the proposed idea is novel and interesting. I vote for weak acceptance because there are still some technical points that are not well-addressed enough:

First, although the intuition of this model makes a lot of sense to me, the construction of the loss function is quite heuristic, with a lot of terms simply summing together, making it hard to judge which components are most important for the final results. A more principled and integrated framework like EM could be more convincing to me.

Second, it seems the unlabelled data is only used in the causal constraint term (the last term in Eqn 5) and it is controlled by a coefficient \gamma. It is a bit unclear to me whether the unlabelled data is fully utilized while it only constraints the causal relation, as one can also use labeled data for constraining the causal relation. Also, why not include labeled data for this constraint regularization?

Another minor question is after the two networks are trained, will you only use the learned classifier for test data, or, do you also use the conditional distribution in the testing phase and compute an expectation of the predicted class? and why?

Also, what's the purpose of section 6 in the appendix?

I general I think the idea of learning a conditional distribution to constrain the use of rules is an interesting and novel idea. The paper can be further improved if the algorithm can be more principled.


---- after reading the response ---

Thanks for answering the questions. I believe some of these explanations can be added to the final version to improve clarity. My score does not change, but overall I advocate to accept this paper.

review_title: Official Blind Review #1

Meta-review: The paper addresses the problem of costly human supervision for training supervised learning methods.
The authors propose a joint approach for more effectively collecting supervision data from humans, by extracting rules and their exemplars, and a model for training on this data.
They demonstrate the effectiveness of their approach on multiple datasets by comparing to a range of baselines.

Based on the reviews and my own reading I recommend to accept this paper.
The approach makes intuitively a lot of sense and is well explained.
The experimental results are convincing. 
Decision: Accept (Spotlight)