Learning from Rules Generalizing Labeled Exemplars
Authors: Abhijeet Awasthi
Sabyasachi Ghosh
Rasna Goyal
Sunita Sarawagi
Venue:  (2020)
Keywords: Learning from Rules
Learning from limited labeled data
Weakly Supervised Learning

Abstract: In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision. We propose a rule-exemplar method for collecting human supervision to combine the efficiency of rules with the quality of instance labels. The supervision is coupled such that it is both natural for humans and synergistic for learning. We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables. The denoised rules and trained model are used jointly for inference. Empirical evaluation on five different tasks shows that (1) our algorithm is more accurate than several existing methods of learning from a mix of clean and noisy supervision, and (2) the coupled rule-exemplar supervision is effective in denoising rules.

Published as a conference paper at ICLR 2020
LEARNING FROM RULES GENERALIZING LABELED EXEMPLARS
Abhijeet Awasthi Sabyasachi Ghosh Department of Computer Science and Engineering
Rasna Goyal
Sunita Sarawagi
Indian Instiute of Technology Bombay
Mumbai, Maharashtra 400076, India
{awasthi, sghosh, goyalrasna, sanasssnnii sunita]ecse.......e with. in
ABSTRACT
In many applications labeled data is not readily available, and needs to be col- lected via pain-staking human supervision. We propose a rule-exemplar method for collecting human supervision combine the efficiency of rules with the qual- ity of instance labels. The supervision is coupled such that it is both natural for humans and synergistic for learning. We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables. The denoised rules and trained model are used jointly for inference. Empirical evaluation on five different more accurate than several existing meth- ods of learning from a mix of clean and noisy supervision, and (2) the coupled rule-exemplar supervision is effective in denoising rules.
tasks shows that (1) our algorithm
INTRODUCTION
With the ever-increasing reach of machine learning, a common hurdle new adoptions is the lack of labeled data and the pain-staking process involved in collecting human supervision. Over the years, several strategies have evolved. On the one hand are methods like active learning and crowd- consensus learning that seek to reduce the cost of supervision in the form of per-instance labels. On the other hand the rich history of rule-based methods (Appelt et al., 1993; Cunningham, 2002) where humans code-up their supervision from such efficient, albiet noisy, supervision (Ratner et al., 2016; Pal Balasubramanian, 2018; Bach et al., 2019; Sun et al., 2018; Kang al., 2018). However, clean task-specific instance labels continue to be critical for reliable results (Goh et al., 2018; Bach et al., 2019) in spite of easy availability of pre-trained models (Sun et al., 2017: Devlin al., 2018).
labeling rules. There is growing interest in learning
In this paper we propose a unique blend of cheap coarse-grained supervision in the form of rules and expensive fine-grained supervision in the form of labeled instances. Instead of supervising rules and instance labels independently, we propose that each labeling rule be attached with exemplars of where the rule correctly 'fires". Thus, the rule can be treated as noisy generalization of those exem- plars. Often rules are coded up only after inspecting data. As a human inspects instances, he labels them, and then generalizes them to rules. Thus, humans provide paired supervision of rules and exemplars demonstrating correct deployment of that rule. We explain further with two illustrative applications. Our examples below are from the text domain because rules have been traditionally used in many NLP tasks, but our learning algorithm is agnostic to how rules are expressed.
Sentiment Classification Consider an instance highly recommend this modest priced cellular phone that human inspects for sentiment labeling task. After labeling as positive, he can easily generalize it to a rule Contains highly recommend' 4 positive label. This rule generalizes several more instances, thereby eliminating the need of per-instance labeling on those. However, the label assigned by this rule on unseen instances may not be as reliable the explicit label on this specific exemplar generalized. For example, it misfires on I would highly zecommend this phone if it weren't t for their poor service.
Code and datasets available http:///ghhhbcccmmmssssssiaihich Froninngg--o--lliii Rules

---

Published as a conference paper at ICLR 2020
Slot-filling Consider slot-filling task on restaurant reviews over labels like cuisine, location, and time. When an annotator sees an instance like: what chinese restaurants in this city have good reviews?, after labeling token chinese as cuisine, he generalizes it to a rule: (.*ese) I. .iian mex i can) restaurants -> (cuisine) restaurants. This rule matches hundreds of instances in the unlabeled set, but could wrongly label a phrase like these restaurants. Our focus in this paper is developing algorithms for training models under such coupled rule-exemplar supervision. Our main challenge is that the labels induced by the rules are more noisy than instance-level supervised labels because humans tend to over generalize (Tessler& & Goodman, 2019) as we saw in the illustrations above. Learning with noisy labels with or without additional clean data has been problem of long-standing interest in ML. (Khetan et al., 2018; Zhang & Sabuncu, 2018; Ren et al., 2018b; Veit et al., 2017: Shen & Sanghavi, 2019). However, we seek to design algorithms that better capture rule-specific noise with the help of exemplars around which we have supervision that the rule fired correctly. We associate latent random variable whether rule correctly 'covers' an instance, and jointly learn the distribution among the label and all cover variables. This way we simultaneously train the clas- sifier with corrected rule-label examples, and restrict over werrgenneralized rules. The denoised rules used during inference to further boost accuracy of the trained model. In summary our contributions in this paper are as follows:
Our contributions (1) We propose the paradigm of supervision in the form of rules generaliz- ing labeled exemplars that is natural in several applications. (2) We design training method that simultaneously denoises over-generalized rules via latent coverage variables, and trains a classifi- cation model with soft implication loss that we introduce. (3) Through experiments on five tasks spanning question classification, spam detection, sequence labeling, and record classification we show that our proposed paradigm of supervision enables an effective synergy between rule-level and instance-leve supervision. (4) We compare our algorithm to several recent frameworks for learning with noisy supervision and constraints, and show much better results with our method.
TRAINING WITH RULES AND EXEMPLARS
We first formally describe the problem of learning from rules generalizing examplars on classi- fication task. Let X' denote the space of instances and y = [1.....K] denote the space of class labels. Let the set of labeled examples be I, [(x1, L,, e1), ..... (x,, en, [n]] where x; E X is an instance, & E Y is its user-pronn providdd label, and e; E (RI.... Rm, 0} denotes that is an exemplar for rule e,. Some labeled instances may not be generalized to rules and for them e; = 0. Also, a rule can have more than one exemplar associated with it. Each rule Rj could be blackbox function Rí : x + {e,, } that takes as input an instance x E and assigns either label E, or no-label. When the ith labeled instance is an exemplar for rule R, (that is, e, R2), the label of the instance l should be é,. Additionally, we have a different set of unlabeled instances U = [xx+1..... XN) ). The cover set H, of rule Rg is the set of all instances in U U T. for which Rg assigns noisy label l An instance may be covered by more than one rule or no rule at all, and the labels provided by these rules may be conflicting. Our goal is to train a classification model Po(y))) using L and U to maximize accuracy on unseen test instances. A baseline solution is to use R2 to noisily label the covered U instances using majority or other consensus method of resolving conflicts. We then train (o(y)x) on the noisy labels using existing algorithms for learning from noisy and clean labels (Veit et al., 2017; Ren et al., 2018b). However, we expect to be able to do better by learning the systematic pattern of noise in rules along with the classifier Pa (y)x)).
Our noise model on R A basic premise of our learning paradigm that the noise induced by a rule R2 is due to mveegeanerl the exemplar(s) seen when creating the rule. And, there exists a smaller neighborhood closer to the exemplar(s) where the noise zero. We model this phenomenon by associating a latent Bernoulli random variable rýi for each instance in the stated cover set H1 of each rule Ry. When rja = 1, rule R, has not weerggeneallieed on X,, and there is no noise in the label E that Rj assigns to x,. When Tji = O we flag an over-generalization, and abstain from labeling x; as E, suspecting it to be too noisy. We call rjs8 as the latent coverage variables. We propose to learn the distribution of ry using another network with parameters o that outputs the probability (r, (x) that r = 1. We then seek to jointly learn Po ( x) and (r((r)1x) to model the distribution over the true label y and true coverage rj for each rule j and each x in Hy- Thus
2

---

Published as a conference paper at ICLR 2020
piá plays the role of restricting rule R, so that ry is not necessarily for all instances in its cover set H
An example We make our discussion concrete with an example. Figure shows two-dimensional X' space with labeled points L denoted as red crosses and blue circles, unlabeled points as dots, and the true labels as background color of the re- gion. We show two rule-exemplar pairs: (x1, V/1 red, R1), (x2, y/2 = blue, R2) with bold boundaries. Clearly, both rules R1, R2 have over-generalized the wrong region. If we train a classifier with many examples H1 U H2 wrongly labeled by rules, then even with a noise tolerant loss function like Zhang & Sabuncu (2018), the classifier (o(y)x x) might be misled. In contrast, what we hope to achieve is to learn the Pyo (r)((x) distribution using the limited labeled data and the overlap among the rules such that Pr(ry x) predicts a value of for examples wrongly covered. Such examples are then excluded from training Pg. The dashed boundaries indicate the revised boundaries of Rys that we can hope learn based on consensus on the labeled data and the set of rules. Even after such restriction, R,s are useful for training the classifier because of the unlabeled points inside the dashed regions that get added to the labeled set.
2₂
Figure 1: Restricting over-generalized rules
2.1 How WE JOINTLY LEARN Pg AND Pjo
In general will be provided with several rules with arbitrary overlap in the set of labeled L and unlabeled examples that they cover. Intuitively, we want the label distribution Po (yl x) to correctly restrict the coverage distribution (ry(xx)), which in turn can provide clean labels to instances in U that can be used to train (o(y))). We have two types of supervision in our setting. First, individually for each of Po(y)x) and (TO(Ty we have ground truth values the (a(y]x] distribution, supervision on is provided by the human labeled data L, and we use these define the usual log-likelihood as one term in our training objective:
and ry for some instances. For
max LL(0) = max Σ
log Po(fs(x;)
(1)
(X:,))EL
For learning the distribution Pje (r)lx) over the coverage variables, the only sure-shot labeled data is that rjx = 1 for any x; that is an exemplar of rule R, and ry; 0 for any E H2 whose label e; different from l,. For other labeled instances x; covered with rules R, with agreeing labels, that is l l, we do not strictly require that Tja = 1. In the example above the corrected dashed red boundary excludes red labeled point to reduce its noise on other points. However, if the number of labeled exemplars
too few, regularize the networks towards more rule firings, by adding a noise tolerant rj = loss on the instances with agreeing labels. We use the generalized cross entropy loss of Zhang & Sabuneu (2018).
LL(Ø)= Σ
(log P (rep 1(x:) +
Σ
log P3o(Tzi 0/x;)
(2)
Σ
Generalized Generaaliea-XXINT((Pooo......
J.K.HHA HHHA 4=t,
Note for other instances x; in Ry 's cover Hy. value of ry is unknown and latent. The second type of supervision is on the relationship between r'ys and y: for each x, E Hy. A rule R2 imposes a causal constraint that when Fjz 1, the label has to be Ez.
ys = ly Vx, E H, We convert this hard constraint into (log) probability of the constraint being satisfied under the (o(x)) and (x)(rx distributions as:
U5 =
(3)
log (r, l -1 Po(G)x)))
(4)
Figure 2 shows surface plot of the above log probability as function of Po(l)xx) (shown axis P(y) in figure) and Pzo(r, 1|x) (shown axis P(r) in figure) for a single rule.
3

---

Published as a conference paper at ICLR 2020
Observe that likelihood drops sharply P(r,)x) x) is close to but P(y = xx) close to zero. For all other values of these probabili- ties the log-likelihood is flat and close to zero. Specifically, when Pjo predicts low values of ry for x, the log-likelihood surface is flat, effectively withdrawing the (x, {2) supervision from training the classifier Po. Thus maximiz- ing this likelihood provides soft enforcement of the constraint without unwanted biases. We call this the negative implication loss.
-0.5
-15
-25
-3.0
-3.5
0.8
0.0
0.2
0.4
0.6
0.2
0.4
0.6
P(y)
We do not need to explicitly model the conflict among rules, that when an x; is covered by two rules R and Rg of differing labels (e₁ + (g), then both rys and Tki cannot be 1. This is because the constraint among pairs (yt, rya) and (ys, Tks) stated in Equation 3 subsumes this one.
P(r)
10
0.0
Figure Negative implication
During training we then seek to maximize the log the above probability along with normal data likelihood terms. Putting the terms in Equations 1, 2 and 4 together our final training objective is:
min LL((0) - (L())- - Σ log(1 PςO(r; = 11x) Pa(e)xx)) 0,0
(5)
j:EEH,ne
We refer to our training loss denoised rule-label implication loss or ImplyLoss for short. The (L()) term seeks to denoise rule coverage which then influence the y distribution via the implication loss. We explored several other methods of enforcing the constraint among y and ry in the training of the Pg and Pjo networks. Our method ImplyLoss consistently performed the best among several methods we tried including the recent posterior regularization (Ganchev et al., 2010; Hu et al., 2016) method of enforcing soft constraints and co-training (Blum Mitchell, 1998).
Network Architecture Our network has three modules. (1) A shared embedding layer that pro- vides the feature representation of the input. When labeled data is scarce, this will typically be pre-trained layer from a related task. The embedding module is task-specific and described in the experiment section. (2) A classification network that models (o(xx) with parameters 0. The embedding of an imput x is passed through multiple non- non-linear layers with ReLU activation, a last linear layer followed by Softmax to output distribution over the class labels. (3) A rule network that models (r = 1 x) whose parameters & are shared across all rules. The input to the network rule-specific and concatenates the embedding of the input instance x, and one-hot encoding of the rule 'j' The input is passed through multiple non-linear layers with ReLU activation before passing through Sigmoid activation which outputs the probability (z 1[x).
Inference During prediction, joint inference over the label y and coverage variables ry provides slight gains over depending solely on Pa (y x). For any test example x, consider the set of rules G' covering such that (1x( > 0.5. Probabilities from the label and coverage variables are combined to obtain a score s(y) for each label as:
sy/lx) = Pa(v/x) + ER,Eg 8(e, (1 + 5(e, * (0(x)
(6)
G
The above can be viewed as a soft voting over the trained classifier Ps and labels provided by rules with uncertain coverage. Because also learned denoise rules along with training the classifier, the labels assigned by the rules have higher precision than original rules.
3 EXPERIMENTS
We compare our training algorithms against simple baselines, existing error-tolerant learning algo- rithms, and existing constraint-baste learning in deep networks.
We evaluate across five datasets spanning three task types: text classification, sequence labeling, and record classification. We augment the datasets with rules, that obtained manually in three
4

---

Published as a conference paper at ICLR 2020
Dataset
[2]
[0]
#Rules
%ovver
Precision
&Conflict
Avg [1,]
#Rules Per In-
Valid
Test
stance
Question
68
4884
68
95
63.8
22.5
124
1.8
500
500
MIT-R
1842
64888
15
14
80.7
2.5
634
1.1
4091
14256
SMS 69
4502
73
40
97.3
0.6
1.3
500
500
You'Tube 100
1586
10
87
78.6
30.2
258
1.9
120
250
Census
83
10000
83
100
84.1
27.5
540
4.5
5561
16281
Table I: Statistics of datasets and their rules. %Cover is fraction of instances in U covered by least one rule. Precision refers miero precision of rules. Conflict denotes the fraction of instances covered by conflicting rules among all covered instances. Avg [H1] | is average cover size of a rule in Rules Per Instance average number of rules covering an instance in U.
cases, from pre-existing public sources in one case, and automatically in another. Table 1 presents statistics summarizing the datasets and rules. A brief description of each appears below.
Question Classification (Li & Roth, 2002): This is a TREC-6 dataset to classify a question to one of six categories: (Abbreviation, Entity, Description, Human, Locatior, Numeric-van luee) The training set has 5452 instances which are split as 68 for L, 500 for val- idation, and the remaining as U. Each example in is generalized as a rule represented by regular expression. E.g. After labeling How do you throw as Description we define a rule
nousewarming party
+ Description.
More rules Table 4 supplementary. Although, creating such 68 generalised rules required 90 minutes, the generalizations cover 4637 instances in U, almost two orders of magnitude more instances than in L! On an average each of our rule covered 124 instances (HH] column in Table I). But the precision of labels assigned by rules was only 63.8%. 22.5% of covered instances had an inter-rule conflict, demonstrating noise in the rule labelings. Accuracy is used as the performance metric.
MIT-R' (Liu et al., 2013); This is slot-filling task on sentences about restaurant search and the task is to label each token as one of (Location, Hours, Amenity, Price, Cui sine, Rating, Other). The training data is randomly split into 200
Dish, Rastaurant.Name sentences (1842 tokens) as L, 500 sentences (4k tokens) as validation and remaining 6.9k sen- tences (64.9k tokens) as U. We manually generalize 15 examples in L. Egg. After inspecting the sentence where can get the highest rated burger within ten mi les and labeling highest rated as Rating, we provide the rule:
* (highly/highlgoooll
highest) (ratee rating/rated).
Rating
to the matched positions. More examples in Table 7 of supplementary. Although, creating 15 generalizing rules took 45 minutes of annotator effort, the rules covered roughly 9k tokens U. Fl metric used for evaluation the default test set of 14.2k tokens over 1.5k sentences.
SMS Spam Classification (Almeida et al., 1): This dataset contains 5.5k text messages labeled spam/not-spam. of which 500 were held out for validation and 500 for testing. We manually generalized 69 exemplars to rules. Remaining examples go in the U set. The rules here check for presence of keywords covers 31 examples on an average and has precision of 97.3%. However, this case only 40% of the unlabeled set is covered by a rule. We report Fl here since class is skewed. More examples in Table 5 of supplementary.
phrases in the SMS gua ranteed gift
spam. A rule
Youtube Spam Classification (Alberto al., 2015): Here the task to classify comments on YouTube videos as Spam or Not-Spam. We obtain this from Snorkel's Github page?. which provides 10 labeling functions which we use as rules, an unlabeled train set which we use as U. labeled dev set to guide the creation of their labeling functions which we use as L,, and labeled test and validation sets which we use in the same roles. Their labeling functions have large coverage (258 on average), and precision of 78.6%.
Census Income (Dua & Graff, 2019): This UCI dataset is extracted from the 1994 U.S. census. lists total of 13 features of an individual such as age, education level, marital status, country of
ggroups.caai..me wtwp://mmm......////ooo////seeee http://ttttt..ooooooomaaaammeett
5

---

Published as a conference paper at ICLR 2020
Datasets
Methods
Question
MIT-R
YouTube
SMS
Census
(Accuracy)
(FI)
(Accuracy)
(F1)
(Accuracy)
Majority (No parameters trained)
60.9 (0.7)
40.9 (0.1)
82.2 (09)
48.4 (1.2)
80.1 (0.1)
Only-L
72.9 (0.0)
73.5 10.3)
90.9 (1.)
89.0 (1,6)
79.4 (0.5)
L+Umaj
1.4 (1.5)
+ 0.0 (0.3)
0.8 (1.9)
3.5 (1.2)
0.9 (0.1)
Noise-tolerant (Zhang al., 2018)
0.5 a..
+ 0.0 (0:))
1.7 0..)
+ 2.9 (1.2)
+ 1.0 (0.2)
1.2R (Ren et al., 2018b)
+ 0.3 (2.
11.4 (1.0)
+ 2.5 (0.5)
+ 2.3 (0.5)
+ 2.9 (0.3)
L+Usnorkel (Ratner et al., 2016)
0.7 (3.0)
+ 0.0 (0.))
2.7 (0.7)
+ 3.5 (1.3)
+ 1.0 (0.4)
Shorkel-Noise-Tolee
1.4 (1.6)
+ 0.0 (0.3)
2.0 (0.7)
2.7 (1.5)
+ 0.2 (0.5)
Posterior Reg. (Hu et al., 2016)
0.8 (1.0)
0.1 (0.4)
2.9 (1:9)
+ 1.8 (1.5)
0.8 (0.5)
ImplyLoss (Ours)
+ 11.7 (1.5)
+ 0.8 10.3)
3.2 (1.1)
4.2 (1.0)
1.7 (0.2)
Table 2:
Comparison ImplyLoss (our method) with various methods (described in Section 3.1) on five
different datasets. The numbers reported for all methods after the double-line are gains over the haseline (Only-
L) that does not use rules at all. Higher is better. NOTE: Numbers in brackets represent standard deviation of the original accuracy and not of gains.
origin etc. The primary task on it is binary classification - whether a person earns more than S50K or not. The train data consists of 32563 records. We choose 83 random data points as L, 10k points as U and 5561 points as validation data. For this case we created the rules synthetically as follows: We hold out disjoint 16k random points from the training dataset as a proxy for human knowledge and extract a PART decision list (Frank& & Witten, 1998) from it as our set of rules. We retain only those rules which fire
Network Architecture Since our labeled data small we depend on pre-trained resources. As the embedding layer we use a pretrained ELMO (Peters et al., 2018) network where 1024 dimen- sional contextual token embeddings serve as representations of tokens in the MIT-R sentences, and their average serve as representation for sentences in Question and SMS dataset. Parameters of the embedding network are held fixed during training. For sentences in the YouTube dataset, we use Snorkel's 2 architecture of simple bag-of-words feature representation marking the frequent uni- grams and bi- bi-grams present in a sentence using a few-hot vector. For the Census dataset categorical features are represented as one hot vectors, while real valued features simply normalized. For MIT-R, Question and SMS both classification and rule-weight network contain two 512 dimensional hidden layers with ReLU activation. For Census, both the networks contain two 256 dimensional hidden layers with ReLU activation. For YouTube, the classifier network is a simple logistic re- gression like in Snorkel's code. The rule network has one 32-dimensional hidden layer with ReI.U activation.
Each reported number is obtained by averaging over ten random initializations. Whenever a method involved
to weigh the relative contribution of various terms in the objective, we used a validation dataset to tune the value of the hyper-parameter. Hyperparameters used are pro- vided in Section C of supplementary.
3.1 COMPARISON WITH DIFFERENT METHODS
In Table 2 we compare our method with the following alternatives on each of the five datasets:
Majority: that predicts via majority vote among the rules that cover an instance. This baseline indicates the stand-alone quality of rules, no network is learned here. Ties are broken arbitrarily for blass-balanced datasets or by using a default class. Table 2, shows that the accuracy of majority is quite poor indicating either poor precision or poor coverage of the rule 3ets.
Only-L:
Here we train the classifier (o x) only on the labeled data L using the standard cross- not utilized at all in this case. We observe
entropy loss (Equation 1). Rule generalisations in Table 2 that even with the really small labeled set we used for each dataset, the accuracy of a classifier learned with clean labeled data is much higher than noisy majority labels of rules. We consider this method as our baseline and report the gains on remaining methods
'Only for the Census dataset the relative accuracy is high because the rules were obtained synthetically through rule-learning algorithm very large labeled dataset to serve as proxy for human's 's generaliza- tion.

---

Published as a conference paper at ICLR 2020
L+Umaj: Next we train the classifier L along with Unuj obtained by labeling instances in U with the majority label among the rules applicable to the instance. Loss corresponding to the examples labeled by rules is weighted follows:
min 0
Σ
log po(4 +Y
Σ
-
log (o (x,)
(X3
GJEL
(X,19))EUmi
The row corresponding to L+Umaj in Table 2 provides the gains of this method over Only-I.. We observe gains with the noisily labeled U in three out of the five cases.
Noise-tolerant: Since labels in Umaj are noisy, we next use Zhang & Sabuncu (2018)'s noise tolerant generalized cross entropy loss on them with regular cross-entropy loss on the clean L as follows:
(1 (x())(x))*
min
Σ
log Po(l)xx))+ +Y
Σ (E))))))U
q
(X,,,))L
Parameter y E [0, 1] controls the noise tolerance which we tune hyper-parameee We observe that in three cases minimizing the above objective improves beyond L+Umaj validating that noise- tolerant loss functions can be useful for learning from noisy labels on Umaj-
Learning to Reweight (L2R) (Ren et al., 2018b): is a recent method for training with a mix of clean and noisy labeled data. They train the classifier by meta-learning to re-weight the loss on the noisily labelled instances (Umaj) with the help of the clean examples (L). This method provides significant accuracy gains over Only-L in three out the five datasets. However, it fails in the multi- class classification task of slot-filling which has a very high class imbalance and rules of smaller coverage.
All the above methods employ no extra parameters to denoise or weight individual rules. We next compare with number of methods that do.
L+Usorrkel: This method replaces Majority-based consensus with Snorkel's generative model (Rameer al., 2016) that assigns weights rules and labels examples in U. Thereafter we use the same approach as in L+Umaj with just Snorkel's soft-labels instead of Majority on U. We also compare with using noise-tolerant loss on U labeled by Snorkel (Eqn:8) which we call Snorkel- Toler-t. Like previous methods, both of these methods provide improvements over Only-L. on three of the five datasets where the rules are less noisy. U+nnoreel performs slightly better than Toise-Tolerant on Umaj.
We next compare with a method that simultaneously leams two sets of networks Pa and Pjo like ours but with different loss function and training schedule.
Posterior Regularization (PR): This method proposed Hu al. (2016) also treats nules as soft- constraints and has been used for training neural networks for structured outputs. They use Ganchev et al. (2010)'s posterior regularization framework to train the two networks in a teacher-student setup. We adapt the same framework and get procedure tribution over y and rys using current Po and Pjs, the teacher uses the constraint in Eq 3 to revise the distributions so and & to minimize KL. distance with the revised distribution. The detailed formulation appear in the Section A of supplementary. We find that this method is better than Only-L in most of the cases and worse than the noise-tolerant method that does not train extra & parameters.
follows: The student proposes dis-
minimize the probability of violations, the student updates parameters
Imply Loss((urs)) Overall our approach of training with denoised rule-label implication loss pro- vides much better accuracy than all the above eight methods and we get consistent gains over Only-L on all datasets. On the Question dataset we get 11.7 points gain over Only-L whereas the best gain by existing method was 0.3. A useful property of our method compared to the PR method above that the training process simple and fits into the batch stochastic gradient training template. In contrast, PR requires special alternating computations. We next perform number of diagnostics experiments to explain the reasons for the superior performance of our method.
7

---

Published as a conference paper at ICLR 2020
Diagnostics: Effectiveness of learning true coverage via Peo An important part of our method is the rule-specific denoising learned
Old precision
Denaised Precision
Percent Suppressed
90
via the Pá network. In the chart alongside we plot the original precision of rules the test
66
data, and the precision after suppressing those
55
rule labelings where (ro (x) predicts in-
stead of I. Observe now that the precision is more than 91% on all datasets. For the Ques- tion dataset, the precision jumped from 64% to 98%. The percentage of labelings suppressed (shown by the dashed line) is higher datasets with noisier rules (c.g. compare Question and SMS). This shows that Pjo is able to denoise rules by capturing the distribution of the latent true coverage variables with the limited (o) loss and indirectly via the implication loss.
11
Question
MIT-R
YouTube
SMS
Canaus
Figure 3: Rule-specific denoising by our method.
Effect of rule precision Rules in the Census dataset are of higher quality in terms of preci- sion as well as coverage. Superior performance of the L2R method on this dataset motivated us to inspect how well our method performs on the same dataset in the absence of high preci- sion rules. We created four new versions of the rule sets by successively removing high preci- sion rules from the original rule set. We ob- serve that our method performs better than L2R when rules have low precision. Because Imply- Loss denoises rules, it is better able handle low-precision rules.
83
Accuracy
75
66
71
75
83
Rule Precision
Figure 4: Effect of rule precision
Role of Exemplars in Rules We next evaluate the importance of the exemplar-rule pairs in learning the Peo and Po networks. The exemplars of rule give interesting new form of supervision about an instance where a labeling rule must fire. To evaluate the importance of this supervision, we exclude the ry likelihood on rule-exemplar pairs from LL(ő), that is, the first term in Equation 2 is dropped. In the table below we see that performance of ImplyLoss usually drops when the exemplar-rul supervision is removed. Interestingly, even after this drop, the performance of ImplyLoss surpasses most of the methods in Table 2 indicating that even without exemplar-rule pairs our training objective effective learning from rules and labeled instances.
Question
MIT-R
SMS
Census
r for rule-exemplar pairs
84.5 (1.5)
73.7 (0.))
93.2 (1.)
81.0 (02)
No ry for rule uule-exemplar pairs
83.8 (0.7)
73.5 (0.5)
93.5 (1.2)
80.8 (0.3)
Table Offect of removing rule-exemplar supervision from LL(Ø)
Effect of increasing labeled data L We in- crease L while keeping the number of rules fixed the Question dataset. In the attached plot we see the accuracy of our method (Imply- Loss) against Only-L., L+Usnorkel and Poste- rior Reg. We observe the expected trend that the gap between the method narrows labeled data increases.
Ours
Posterior Reg.
L+Usnorkel Only-L
Accurscy
4 RELATED WORK
Learning from noisily labeled data has been extensively studied in settings like crowd- sourcing. One category of these algorithms upper-bound the loss function make it robust noise. These include methods like MAE (Ghosh
Figure 5: Effect increasing labeled data
8

---

Published as a conference paper at ICLR 2020
et al., 2017), Generalized Cross Entropy (CE) & Sabuncu, 2018), and Ramp loss (Collobert et al., 2006). Most of these assume that noise is independent of the input given the true label. In our model noise is systematic and instance-dependent.
A second category assume that a small clean dataset is available along with noisily labeled data. This is also true in our case, and we compared with state of the art method in that category Ren et al. (2018b) that chooses a descent direction that aligns with a clean validation set using meta- learning. Others in this category include: Shen & Sanghavi (2019)'s method of iteratively selecting examples with smallest loss, and Veit et al. (2017)'s method of learning a separate network to trans- form noisy labels to cleaned ones which are used to impose a cross-entropy loss on (o)x)). In contrast, we perform rule-specific cleaning via latent coverage variables and flexible implication loss which withdraws y supervision when (ro((ss(x) assumes low values. Another way of relat- ing clean and noisy labels is via an instance-independer confusion matrix learned jointly with the classifier (Khetan et al., 2018; Goldherger & Ben-Reuven, 2016; Han et al., 2018b;a). These works assume that the confusion matrix is instance independent, which does not hold for our case. Tanaka et al. (2018) uses confidence from the classifier climinate noise but they need to ensure that the network does not memorize noise. Our learning setup also has the advantage of extracting confi- dence from a different network. There is growing interest in integrating logical rules with labeled examples for training networks, specifically for structured outputs (Manhaeve et al., 2018; Xu al., 2018; Fischer et al., 2019; Sun et al., 2018; ; Ren et al., 201 20) Xu et al. (2018); Fischer et al. (2019) convert rules on output nodes of network, to (almost differentiable) loss functions during training. The primary difference of these methods from ours is that they assume that rules are correct whereas we assume them to be noisy. Accordingly, we simultaneously correct the rules and use them to improve the classifier, whereas they use the rules is-is to train the network outputs.
A well-known framework for working with soft rules is posterior regularization (Ganchev et al., 2010) which is used in Hu et al. (2016) to train deep structured output networks while harnessing logic rules. Ratner et (2016) works only with noisy rules treating them as black-box labeling functions and assigns a linear weight to each rule based on agreement objective. Our learn- ing model is more powerful that attempts to learn a non-linear network to restrict rule boundaries rather than just weight their outputs. We presented comparison with both these approaches in the experimental section, and showed superior performance.
To the best of our knowledge, our proposed paradigm of coupled rule-exemplat supervision is novel, and our proposed training algorithm is able to harness them in ways not possible by existing frame- works for learning from rules noisy supervision.
CONCLUSION
We proposed new rule-exemplar model for collecting human supervision to combine the scalability of top-level rules with the quality of lestance-level labels. We show that such supervision is natural since humans typically inspect examples to code rules. Furthermore, such coupled examples provide supervision on correct firing of rules which help to denoise rules. We propose train the classifier while jointly denoising rules via latent coverage variables imposing a soft-implication constraint on the true label. Empirically on five datasets we show that our training algorithm that performs rule-specific denoising is better than generic noise-tolerant learning. In future we plan to deploy this framework on other applications where human supervision is a scarce resource.
Reproducibility Code and Data for the experiments available at http://gihbbbcoomaaaaaaaahhhiltt
From- Rules
Acknowledgements We thank the anonymous reviewers for their constructive feedback on this work. This research was partly sponsored by Google India AI/ML Research Award and partly by the IBM AI Horizon Networks IIT Bombay initiative. Abhiject is supported by Google PhD Fellowship in Machine Learning.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 138-143. IEEE, 2015.
Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering, pp. 259-262. ACM, 2011.
Douglas E. Appelt, Jerry R. Hobbs, John Bear, David J. Israel, and Mabry Tyson. Fastus: A finite- state processor for information extraction from real-world text. In IJCAI, pp. 1172-1172 1778, 1993.
Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, Amsterdam, The Netherlands, June 30 - July 5, 2019., pp. 362-375, 2019.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In COLT, 1998.
R. Collobert, F. Sinz, J. Weston, and L. Bottou. Trading convexity for scalability, In ICML 2006, 2006.
Hamish Cunningham. Gate: A framework and graphical development environment for robust nlp tools and applications. In Proc. 40th Annual Meeting of the Association for Computational Lin- guistics (ACL 2002), pp. 168-175, 2002.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2019. URL http://archive.icsuu... edu/ml.
Mare Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. DL2: Training and querying neural networks with logic. Proceedings of the 36th International Conference Machine Learning, pp. 1931-1941, 2019.
Eibe Frank and Ian H. Witten. Generating accurate rule sets without global optimization. In J. Shav- lik (ed.), Fifteenth International Conference on Machine Learning, pp. 144-151. Morgan Kauf- mann, 1998.
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 1 11(Jul):001-2049, 2010.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-FFrst AAAI Conference on Artificial Intelligence, 2017.
Garrett B. Goh, Charles Siegel, Abhinav Vishnu, and Nathan Hodas. Using rule-based labels for weak supervised learning: chemnet for transferable chemical property prediction. In Proceed- ings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &#38; Data Mining, KDD 18, 2018
Jacob Goldherger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances Neural Information Processing Systems, pp. 5841-5851, 20184 8a.
10

---

Published as a conference paper at ICLR 2020
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31, pp. 4536-8546 2018b.
Zhiting Hu, Xuezhe Ma. Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), August 2016.
Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. Adventure: Adversarial train- ing for textual entailment with knowledge-guiee examples. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers). Association for Computational Linguistics, 2018.
Ashish Khetan, Zachary Lipton, and Anima Anandkumar. Learning from noisy singly-labeles data. In International Conference on Learning Representations, 2018. URL https://openrreiew. ses/forum?id-li HUggb00
Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th international confer- ence on Computational linguisticx-Volume 1, pp. 1-7. Association for Computational Linguistics. 2002.
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. Query understand- ing enhanced by hierarchical parsing structures. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 72-77. IEEE, 2013.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Pro- cessing Systems 31, pp. 3749-3759. 2018
Arghya Pal and Vineeth N. Balasubramanian. Adversarial data programming: Using gans to relax the bottleneck of curated labeled data. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 1556-1565, 2018.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer Deep contextualized word representations, arXiv preprint 2018.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data pro- gramming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems 29. 2016.
Hongyu Ren, Russell Stewart, Jiaming Song, Volodymyr Kuleshov, and Stefano Ermon. Learning with weak supervision from physics and data-driven constraints. Al Magazine, 39()):27-38, 2018a.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv: 1803009000, 2018b.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In Proceedings of the 36th International Conference on Machine Learning, pp. 5739- 5748, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017,11 pp. 843-852, 2017.
Haitian Sun, William W Cohen, and Lidong Bing. Semi-supervised learning with declaratively specified entropy constraints. In Advances in Neural Information Processing Systems 31, pp- 4425 4435. 2018
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018
11

---

Published as a conference paper at ICLR 2020
Michael Henry Tessler and Noah D. Goodman. The language of generalization. Psychological Review, 126(3):395 436, 2019.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 839-847, 2017.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In Proceedings of the 35th International Conference on Machine Learning, pp. 5502-5511, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems 31, 2018.
12

---

Published as a conference paper at ICLR 2020
Supplementary Material: Learning from Rules Generalizing Labeled Exemplars
A POSTERIOR REGULARIZATION METHOD
We model a joint distribution Qy ri,..., /x|xx) capture the interaction among the label random variable y and coverage random variables ....... of any instance x. We use to compactly represent ......: Fy- Strictly speaking, when a rule Rj does not cover x, the ryy not a random variable and its value is pinned to O but we use this fixed-tuple notation for clarity. The random variables r' and y impose a constraint on the joint distribution Q: for a x H, when rj = 1, the label y cannot be anything other than Ey.
Vx H, We can convert this into a soft constraint on the marginals of the distribution Q by stating the probability of Eyze, Quy.r 1/x) should be small.
rg - ==1=p y y=
=
min Q
EEE XEH, vFL,
(10)
The singleton marginals of Q along the y and ry variables are tied to the Pe and Pr we seek to Iearn. A network with parameters 0 models the classifier Po (y)xx. and separate network with variables (shared across all rules) learns the Pro(rylx (x) distribution. The marginals of joint Q should match these trained marginals and we KL. term for that:
min Q
Σ ( + Σ (L(Q(),))):
(1I)
XEUUL
JiXEH,
We call the combined KL term succinetly as KL Po) + KI(Q, Pa).
Further the Po and P3 distributions should maximize the log likelihood on their respective labeled data as provided Equation and Equation 2 respectively.
Putting all the above objectives together with hyper parameeassmeters > O, À we get final objective as:
min - -a (LL(( + LL(O)) + KL(Q, KL + + Σ Quu,r, = 1[x) (12) Q
H3 VテL,
We show in Section A. that this gives rise to the solution for Q in terms of Po. Pjs and alternately for Po. Pjo in terms of Q as follows.
Q x Po(ulx) II
Pro(rylx)e- Mirtr,
(13)
jiXEH;
where 5(v &, Ary 1) is indicator function that is 1 when the constraint inside holds, else it Computing marginals of the above using straight-forward message passing techniques we get:
Q(vix) x Po(v)x) II
(1)x( AD(y)t)) + (0)x))
(14)
jxcHH
Q(rk 1|x) x (1) Σ A6 Po(y
II j*kxEH,
(Pzs(1)x)e ( P3s (0lx))
(15)
Thereafter, we solve for and oi in terms of given Q as
min LL LL( Σ Σ 0,0
(o(x)) +
Σ Σ j ree(0.1)
Q(r./x)) log Pio(rs(x))
X:EU vEV
(16)
Here, == This gives rise to alternating optimization algorithm as in the posterior regular- ization framework of Ganchev et al. (2010). We initialize and ó randomly. Then in a loop. we perform the following two steps alternatively much like the EM algorithm (Dempster et al., 1977).
13

---

Published as a conference paper at ICLR 2020
Q Computation step: Here compute marginals Q(y(x) and Q(ryxx) from current Pe and Pje using Equations 14 and 15 respectively for each x in a batch. This computation ttraight-fowwan and does not require any neural optimization. We can interpret the ((y)x) small correction of the Po(y)x) ) so as to align better with the constraints imposed by the rules in Equation 3. Likewise Qrry x) is improvement of current psS in the constraint preserving direction. For example, the expected rj values might be reduced for an instance if its probability of y being Ej is small.
Parameter update step: We next reoptimize the and & parameters to match the corrected Q distribution as shown in Equation 16. This is solved using standard stochastic gradient techniques. The Q terms can just be viewed as weights at this stage which multiply the loss or label likelihood. A pseudocode of our overall training algorithm is described in Algorithm 1.
Algorithm Our Joint Training Algorithm using Posterior Regularization
Input: L, U
Initialize parameters 0, & randomly
for random training batch from U u T. do
Obtain Po (y()x) x) from the classification network.
Obtain Poo y x)]e(m] from the rule-weight network.
Calculate (z(xx) using Eqn 14 and ((Ty1x) using Eqn 15.
Update 0 and & by taking step in the direction to minimize the loss in Eqn 16.
end for Output: 0
A.1
PROOF: ALTERNATING SOLUTION FOR OPTIMIZATION OBJECTIVE IN EQN 12
Treat each Q(y. r) as an optimization variable with the constraint that Σ this constraint with Langrangian multiplier in the objective. Also, define distribution
Lzrr Qyy r) =. We express
Po, = (v(x) II
(ro(r))x)
jixe
easy to verify that the KL terms in our objective 12 can be collapsed as KL(Q; Po.o). The
rewritten objective (call it F(Q, 0, Φ) is now:
(II((0)) + LL(O)) + Σ [KT(Q(y,1x)) Po,o(u,
(17)
+1 Σ M Σ Quv = 1/x) + m1(1 ((w,)))
XEH, N
B.F
Next we solve for ar
0 after expressing the marginals in their expanded forms: e.g.
8Q Qly, rylx) = E, Em
Qgy ri. Fw xx)x). This gives us
aF 0Q(y,r)
log r) log Po (8,
Σyxeh, Ao(y + Eg, = 1) +
Equating it and substituting for Pa.s get the solution for Q(y. r) in Equation 13. The proof for the optimal Py and PS while keeping Q fixed in Equation 17 easy and we skip here.
14

---

Published as a conference paper at ICLR 2020
B LIST OF RULES
We provide a list of rules for each task type.
Rule
Example
Class
(1) (where) < (w++ )(0,1) 11
Where is Trinidad
Location
(waslis) [ + (\$)
(1) (which) what) ["\ (lw+ [0,
What book is the follow-up to Future Shock ?
Entity
(play game moviellookk ["\w] 1$)
{ (") (what) \w> (\w+ [0,
Of children between the Numeric
(part Idivision /
ages of two and eleven
["Aw]*( 1$)
what percentage watch The Simpsons "?
( ('') (who wwo) ["] (w+ [0,
Who invented volleyball ?
Human
(found discovered /aarredmmadee made built
Ibuild invented) ["] x 19)
Table Sample rules for TREC Question Classification. Rule fires if regex matches
Rule
Example
Class Spam
( (1) (free) ["\
Free video camera phones with Half Price line rental for 12 mths and 500 cross ntwk mins 100 txts.
([*Ns]+ + ) (price) [-\w]*
([*)s]+ )* (call) [w]] 19)
Call MobileUpd8 08001950382 or Call2OptOu//674
( (') (guranteed) [" ["\]> ([*\s]+
Great News! Call FREEFONE
Spam
(gift).lgiftt ["\w] x ($)
08006344447 to claim your guaran- teed 000000 CASH or åε2000 gift. sry can 't talk on phone, with parents
(1)) (can't) ["\w]
NotSpam
(ww |0, (talk) ["\w] ($)
( (") (that'ss s) ["\]]
Yeah, that's fine! It's åg6 to get in, that ok?
NotSpam
(w+ [0,1) (fine! finee) ["] ( [$)
Table 5: Sample rules for Spam Classification. Rule fires if the regex matches
Rules
Class
capital-gain 6849
50K
education- num 12 AND
> 50K
marital-status Never-married AND
native-country United-States AND
occupation = Exec-managerial
marital-status Separated AND
50K
hours-per-week < 41
education-num 12 AND
50K
native-country United-States AND
age < 30
Table 6: Sample rules for census dataset. Rule fires if all clauses are True
15

---

Published as a conference paper at ICLR 2020
Rule (T) [w]
Example
Class
any kid friendly restaurants around here
Location
(within near next close nnaaby around around)
(0,
(here peitt/mmlee Imile)
* ["] ( ($)
WordLists:
can you find some chi- nese food
Cuisine
aoisinela=='itaaiaa , 'american',
japanese' 'sspanis' mexx 'mexican', chinese' /vietnamese', Vvegan')
Dalsinelb=('bistee 'delis')
cuisine2=("barbecue'" hhalal' 'vegetarian' bbakery'] ([0-9]+]few under [0-9]+) dollar
need family restaurant with meals under 10 dollars and kids eat where can get the highest rated burger within ten miles
Price
http//hghhllgyyeee beat/ best |
Rating
well hell/htghest/ zagat)
(rate raaing Irated))))))))))))))))))))))))))
(rated) rate |
[0-9] star) ([0-9)] staa)))aataaa)))))
((open opened) (now)late) (popen))
where the nearest italian
Hours
(still (open opened /cleed/clooo clodd cl close))
restaurant that is still open
[[[[open) /aaee//clsee / opened] closed)
([ \w* \w* \w* [0-9]+
(am/pm ((app) m) hhours (hour)
( worbboooorrrooppoooooo romanticl
want to go restaurant within 20 miles that got a high rating and is considered fine dining
Amenity
b foraalsssfaaaaaaaaassuaaa rooftop
(special occasion))
([\a]] \w Awt \w+ ]dining
[\w+ [ 2 (palace cafe |aar| kitchen outback dominoes)
is passims kitchen open at 2
Restaurant
Name
winel sandlihh pasta [burger]
please find me a pub that serves burgers
Dish
peroggis /trgggi///burrtt |burritol
(chicken tikka masala)
appetizerlpizz Ipizza |wine!
cupcake| (onion ring) Itapas
Table 7: Sample rules for MIT-R dataset. Rule fires if the regex matches or sentence contains word found in the provided word lists.
16

---

Published as a conference paper at ICLR 2020
C HYPERPARAMETERS
Across all experiments we use Adam optimizer with default values of 31, 82, and c. Dropout of 0.8 (keep probability) was used in the feed forward layers. All the models were trained for maximum of 100 epochs and early stopping was used based on a validation set. Best model on the validation set was evaluated on the test set. Each experiment was run with 10 random initializations. A list of hyperparameters used in our experiments is provided below.
Noise-
Snorkel-
Post. Reg.
implication
L+Usnorkel
L+Umaj
tolerant
Noise- Tolerant
Question Classification
7
0.001
0.1
0.001
0.1
0.01
0.001
9
0.9
0.6
1r
0,0003
bs
32 (16 for Only-L)
MIT-R 0.01
0.01
0.001
0.1
0.05
0.01
0.6
0.6
Ir
0.0003
bs
64 (32 for Only-L)
Youeubee
0.003
0.5
0.1
0.2
0.5
0.003
0,6
0.6
1r
0.0003
bs
32 (16 for Only-L)
SMS
7
0.1
0.1
0.001
0.3
0.5
0.6
0.6
0.1
br
0,0001
bs
32 (16 for Only-L)
Census
0.5
0.1
0.001
0.1
0.01
0.5
9
0.1
0.6
0.5
1r
0.0001
0.0003
bs
64 (16 for Only-L)
Table 8: Hyperparameters for various methods and datasets. bs refers to the hatch size and ir refers to the learning rate. For Only-L baseline smaller batch size used considering the smaller size of L set.
Question
MIT-R
You
SMS
Census
meta Ir
0.01
0.0001
0.001
0.0001
0.0001
lr
0.0003
0.0001
0.0003
bs
32
64
32
32
64
Table Meta-learning rate, learning rate and batch size
L2R (Ren et al., 2018b) for various datasets
17


Reviews:
[{"review_id": "SyebtskccS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem that labelled data is often unavailable in the quantities required to train effective models. It deals with classification problems, and proposes a method to obtain more (but weaker) labels data with minimal involvement from human labellers, by asking them to generalize their labelling decisions into rules and then learning restrictions on those rules to avoid learning incorrectly generalized labels. The motivating observation is that human labellers are often able to make such generalizations in much less time than it would take them to apply that rule to a large dataset themselves. This is an interesting idea, especially for cases where labelling capacity is limited. The point being made about the labelling noise not being random in this situation is an interesting one - it might be worth exploring this notion further on its own also in contexts where the source of the noise is unknown.\n\nThe presentation of the implementation the authors choose for their proposed approach is clear, and the implementation is sensible. The experimental section includes comparisons to a number of alternative methods, and the authors find that their method outperforms all others, including recent methods for combining (noisy) rule-based labels and (clean) human-sourced labels.\n\nI would argue for accepting this paper. It studies an interesting question, which if answered has the potential to make access to machine learning solutions to certain types of problem significantly cheaper and therefore more widespread. The experiments are well-chosen and show that, depending on the data available and the task, significant gains can be made using the proposed method.\n\nSome remarks on how the paper could become stronger: The type of problem that can be assessed with the proposed method seems to be fairly specific: most tasks studied are classification of natural language utterances. That is a natural class of tasks, since it is easy to imagine how labellers can formulate rules. However, it would have been very interesting if the authors had found ways to allow for more diversity here.  In general, I have the impression that there are more interesting ideas and results to be found in the direction explored by this paper - what about, e.g., allowing the classifier to add rules of its own?\n\nThe paper would benefit from some general editing with regards to appearance; for example, the supplementary material sections continue the regular section numbering, instead of having their own; the images are missing captions, and sometimes have somewhat unorthodox axis tick labelling.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Byg7vz-pFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel semi-supervised learning paradigm where the algorithm learns from both clean instance-level labels and noisy rule-level labels, and also a simple but effective algorithm as solution. The proposed algorithm employs a set of latent coverage variables to bridge two kinds of supervisions and uses a soft causal constraint on the coverage variables to denoise the noisy labels. Empirically the paper demonstrates the effectiveness of the proposed algorithm with consistent improvements over several baselines on a wide range of classification tasks.\n\nThe idea of using macro-level noisy labels as part of the supervision is novel, and it could potentially trigger a paradigm shift on many research areas in machine learning. The proposed methodology is clean but effective, with extensive experimental support. Therefore I vote for accepting this submission.\n\nMinor problems\n\n(1) Abuse of notation \\phi in section 2.\n(2) \"... from traing the classifier ...\" in page 4.\n\n\nMore (further) questions\n\n(1) Since each rule can be regarded as experts or weak learners, how is this work related to learning strong learners from weak learners (boosting/ensemble)?\n(2) Is it possible that the algorithm can incorporate more information of the rules, for example, the structure of the logical formulas?\n(3) Is it possible to generalize the idea to RL?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxefw05KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In case of a lack of labeled data, human-designed rules can be used to label the unlabelled data. This paper proposes a better rule-based labeling method by restricting the coverage of the rule, which is based on the assumption that the rules can be applied to a local region but can not be 'over-generalized' to the whole sample space. The coverage of the rule is represented by a conditional distribution, which is parameterized as a neural network and jointly learned with the classifier network.\n\nI think this paper is tackling an important problem in machine learning, and the proposed idea is novel and interesting. I vote for weak acceptance because there are still some technical points that are not well-addressed enough:\n\nFirst, although the intuition of this model makes a lot of sense to me, the construction of the loss function is quite heuristic, with a lot of terms simply summing together, making it hard to judge which components are most important for the final results. A more principled and integrated framework like EM could be more convincing to me.\n\nSecond, it seems the unlabelled data is only used in the causal constraint term (the last term in Eqn 5) and it is controlled by a coefficient \\gamma. It is a bit unclear to me whether the unlabelled data is fully utilized while it only constraints the causal relation, as one can also use labeled data for constraining the causal relation. Also, why not include labeled data for this constraint regularization?\n\nAnother minor question is after the two networks are trained, will you only use the learned classifier for test data, or, do you also use the conditional distribution in the testing phase and compute an expectation of the predicted class? and why?\n\nAlso, what's the purpose of section 6 in the appendix?\n\nI general I think the idea of learning a conditional distribution to constrain the use of rules is an interesting and novel idea. The paper can be further improved if the algorithm can be more principled.\n\n\n---- after reading the response ---\n\nThanks for answering the questions. I believe some of these explanations can be added to the final version to improve clarity. My score does not change, but overall I advocate to accept this paper.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of costly human supervision for training supervised learning methods.
The authors propose a joint approach for more effectively collecting supervision data from humans, by extracting rules and their exemplars, and a model for training on this data.
They demonstrate the effectiveness of their approach on multiple datasets by comparing to a range of baselines.

Based on the reviews and my own reading I recommend to accept this paper.
The approach makes intuitively a lot of sense and is well explained.
The experimental results are convincing. 
Decision: Accept (Spotlight)