Learning from Explanations with Neural Execution Tree
Authors: Ziqi Wang*
Yujia Qin*
Wenxuan Zhou
Jun Yan
Qinyuan Ye
Leonardo Neves
Zhiyuan Liu
Xiang Ren
Venue:  (2020)

Abstract: While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data, which restricts their applications in scenarios where data annotation is expensive. Natural language (NL) explanations have been demonstrated very useful additional supervision, which can provide sufficient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles. However, directly applying them for augmenting model learning encounters two challenges: (1) NL explanations are unstructured and inherently compositional, which asks for a modularized model to represent their semantics, (2) NL explanations often have large numbers of linguistic variants, resulting in low recall and limited generalization ability. In this paper, we propose a novel Neural Execution Tree (NExT) framework to augment training data for text classification using NL explanations. After transforming NL explanations into executable logical forms by semantic parsing, NExT generalizes different types of actions specified by the logical forms for labeling data instances, which substantially increases the coverage of each NL explanation. Experiments on two NLP tasks (relation extraction and sentiment analysis) demonstrate its superiority over baseline methods. Its extension to multi-hop question answering achieves performance gain with light annotation effort.

Published as a conference paper at ICLR 2020
LEARNING FROM EXPLANATIONS WITH NEURAL EXECUTION TREE
Ziqi Wang', Yujia Qin Wenxuan Chou",Jun Yan", Qinyuan Ye², Leonardo Neves Zhiyuan Liu', Xiang Ren
Tsinghua University', University of Southern California,, Snap Research {ziqi- qinyil 6emmails tainghua. edu: en, (1iuzy)Etsinghann) .edu en, (zhouwenx, yanjun, qinyuany, juuse cdu, (1neves snap. com
ABSTRACT
While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data, which re- stricts their applications in scenarios where data annotation expensive. Natural language (NL) explanations have been demonstrated very useful additional su- pervision, which can provide sufficient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles. How- ever, directly applying them for augmenting model learning encounters two chal- lenges: (1) NI. explanations are unstructured and inherently compositional, which asks for modularized model to represent their semantics, (2) NI explanations often have large numbers of linguistic variants, resulting in low recall and lim- ited generalization ability. In this paper, we propose novel Neural Execution Tree (NExT) framework" to augment training data for text classification using NL explanations. After transforming NL explanations into executable logical forms by semantic parsing, NExT generalizes different types of actions specified by the logical forms for labeling data instances, which substantially increases the cover- age of each NL explanation. Experiments on two NLP tasks (relation extraction and sentiment analysis) demonstrate its superiority over baseline methods. Its extension to multi-hop question answering achieves performance gain with light annotation effort.
INTRODUCTION
Deep neural networks have achieved state-of-the-art performance on wide range of natural lan- guage processing tasks. However, they usually require massive labeled data, which restricts their applications in scenarios where data annotation is expensive. The traditional way of providing pervision is human-generated labels. See Figure [ as an example. The sentiment polarity of the sentence "Quality ingredients preparation all around, and very fair price for NYC" beled as "Positive". However, the label itself does not provide information about how the decision made. A more informative method is allow annotators language so that the annotation can be generalized to other examples. Such an explanation can be "Positive, because the word price directly preceded by fair", which can be generalized other instances like "It has delicious food with a fair price". . Natural language (NL) explanations have shown effectiveness in providing additional supervision, especially in low-resource settings (Srivas-) tava et al. 2017 Hancock et al. 2018). Also, they can be easily collected from human annotators without significantly increasing annotation efforts.
be la-
explain their decisions in natural
However, exploiting NL explanations supervision is challenging due to the complex nature of human languages. First of all, textual data is not well-structured, and thus we have to parse explana- tions into logical forms for machine to better utilize them. Also, linguistie variants are ubiquitous, which makes it difficult to generalize an NL explanation for matching sentences that are semantically
Equal contribution. The order is decided by coin toss. The work was done when visiting USC. 1 Project: hhtt:://int http://nnkk/////// usc. edu/projppppeeee jectteExtt Code: https::/// thub. com/ NNK USC/NEXX

---

Published as a conference paper at ICLR 2020
equivalent but having different word usage. When perform exact matching with the previous ample explanation, it fails to annotate sentences with reasonable price" or " "good deal".
Attempts have been made to train classifiers with NL explanations. SSrvastavaa et al. (2017) use NL explanations as additional features of data. They map explanations to logical forms with a semantic parser and use them to generate binary features for all instances. Hancock et al. (2018) employ a rule- based semantic parser to get logical forms (i.e. "la- beling function") from NL explanations that gen- erate noisy labeled datasets used for training mod- els. While both methods claim huge performance improvements, they neglect the importance of lin- guistic variants, thus resulting in a very low recall. Also, their methods of evaluating explanations on new instances oversimplified (e.g. compari- son/logic operators), making their methods overly confident. In the above example, sentence "Decent sushi a fair enough price" will be rejected because of the "directly preceded" requirement.
Sentence: quality ingredients preparation all
around, and a very fair price for NYC.
What is the sentiment polarity w.r.t. "price"?
Label: Positive
Explanation: because the word "price" directly preceded by fair.
matching from corpus
Sentence: it has delicious food with a fair price.
Matching new instances from raw corpus using natural language explanations.
Figure 1:
To address these issues, we propose Neural Execution Tree (NExT) framework for deep neural networks to learn from NL explanations, as illustrated in Figure 2 Given a raw corpus and a set of NL explanations, we first parse the NL explanations into machine-actionable logical forms by combinatory categorial grammar (CCG) based semantic parser. Different from previous work, we "soften" the annotation process by generalizing the predicates using neural module networks and changing the labeling process from exact matching to fuzzy matching. We introduce four types of matching modules in total, namely String Matching Module, Soft Counting Module, Logical Calculation Module, and Deterministic Function Module. We calculate the matching scores and find for each instance the most similar logical form. Thus, all instances in the raw corpus can be assigned a label and used to train neural models.
The major contributions of our work are summarized as follows: (1) We propose a novel NExT framework utilize NL explanations. NExT is able to model the compositionality of NL explana- tions and improve the generalization ability of NL explanations so that neural models can leverage unlabeled data for augmenting model training. (2) We conduct extensive experiments on two rep- resentative tasks (relation extraction and sentiment analysis). Experimental results demonstrate the superiority of NExT over various baselines. Also, we adapted NExT for multi-hop question answer- ing task, in which it achieves performance improvement with only 21 explanations and 5 rules.
2 LEARNING TO AUGMENT SEQUENCE MODELS WITH NL EXPLANATIONS
This section first talks about basic concepts and notations for our problem definition. Then we give a brief overview of our approach, followed by details of each stage.
Problem Definition. We consider the task of training classifiers with natural language explanations for text classification (c.g., relation extraction and sentiment analysis) in a low-resource setting. Specifically, given raw corpus S = {x; [x: S X and a predefined label set y our goal is to learn a classifier fa : X' - y. We ask human annotators to view a subset S' of the corpus S and provide for each instance E S' a label y and an explanation which explains why x should receive y. Note that | << SS) which requires our framework to learn with very limited human supervision.
Approach Overview. We develop multi-stage learning framework to leverage NI. explanations in a weakly-supervised setting. An overview of our framework is depicted in Fig. 2 Our NExT framework consists of three stages, namely explanation parsing stage, dataset partition stage, and joint model learning stage.
Explanation Parsing. To leverage the unstructured human explanations غ {e 1S' we turn them into machine-actionable logical forms (i.e., labeling functions) (Ratner et al. 2016), which
can be denoted as F (Jg X {0, 13 where 1 indicates the logical form matches the
2

---

Published as a conference paper at ICLR 2020
Raw Corpus NL Explanations
Semantic Parsing
Training Classifier
whather the TERM
negative
def f(x):
Hord Mutching
Mondhed
neturn if word nice) Ts AtMost (RKost (Right g (TEIN)) kun(' tokens]]] else
table
it was a little too hot the sun anyway so our TERM was very nice.
Dato Portitiog
Trahing
Explanation: Positive, because the words "very after
Besliauy nisel la thaostt harta letrlll
Ky: The wine always good, the tapas are always yummeny, especially with the warm TERM.
Mascinng Scare
Caloalttee
Unmatshed
Explaration: Positive, because the word "warm" directly before the TERM.
Sovigg Manching
Selt
Matching
Set
Cauntiog
Figure 2: Overview of the NExT Framework. Natural language explanations are firstly parsed into logical forms. Then we partition the raw corpus S into labeled dataset S, and unlabeled dataset Ji=1- We use matching modules to provide supervision on S.. Finally, supervision from both S, and S,, is fed into classifier.
S, S {x Na
input sequence and 0 otherwise. To access the labels, we introduce function h F - that maps each logical form fj to the label yj of its explanation Cj. Examples are given in Fig. We use Combinatory Categorial Grammar (CCG) based semantic parsing (Zettlemoyer & Collins, 22012] Artzi et al. 2015), an approach that couples syntax with semantics, to convert each NL explanation e, logical form fz.
Following Srivastava et al. 2017), we first compile a domain lexicon that maps each word to its syntax and logical predicate. Frequently-used predicates are listed in the Appendix. For each expla- nation, the parser can generate many possible logical forms based on CCG grammar. To identify the correct from these logical forms, we use feature vector (ff) E Rd with each element count- ing the number of applications of a particular CCG combinator (similar to Zettlemooyr & Collins) (2007)) ). Specifically, given explanation e, the semantic parser parameterized by 0 E Ra outputs a probability distribution over all possible logical forms Zo,- The probability of a feasible logical form can be calculated as:
exp HT ((f)
Po.)) Ep'rez, exp ('")"
To learn 0, we maximize the probability of given e; calculated by marginalizing over all logical forms that match x; (similar to Liang al. (2013)). Formally, the objective function is defined as:
18'1 log
Lpevaer
Σ
Po(lle:)).
IA
When the optimal 8" is derived using gradient-basee method, the parsing result for c, is defined as Ji arg max, Po- (Jle:).
Dataset Partition. After we parse explanations (e:),e] into F {fi) !9) where each corre- sponds to e,, we use F to find exact matches S and pair them with the corresponding labels. We denote the number of instances getting labeled by exact matching as N.. As result, S is partitioned Na N. (x {x}}} y
into a labeled dataset S. {xx),)()) and unlabeled dataset S,, S [x
where No [S] -N..
Joint Model Learning. The exactly matched S, can be directly used to train a classifier while informative instances in S,, are left untouched. We propose several neural module networks, which relax constraints in each logic form fū and substantially improve the rule coverage in S., Classifiers will benefit from these soft-matched and pseudo-labeled instances. Trainable parameters in neural module networks are jointly optimized with the classifier. Details of each module and joint training method will be introduced the next section.
NEURAL EXECUTION TREE
Given a logical form f and sentence x, NExT will output matching u, [0, 1], which indicates how likely the sentence x satisfies the logical form f and thus should given the corre- sponding label h(.). Specifically, NExT comprises of four modules deal with four categories of
3

---

Published as a conference paper at ICLR 2020
Sentence:
SUBJECT was murdered on OBJECT
0.9
03,02,09,,2,004
0,6,1,11111
0.3,02, 02,
0,1,1,1,
Logical Form: (shon( died)
Is AtHost Attost (LofE((B)ECT))))))) And (word died)
Is Betwoon (SUBIECT, (BJECT))
Nun(3 Tokens)) 1))
String Matching
Logical Calculation
Soft
Logical Calculation
Logical Calculation
Deterministic
Counting
Matching
Function
Explanation: words "who died" procede OBJECT by no more than three words and occur between SUBJECT and OBJECT Date_ Death
Figure 3: Neural Execution Tree (NExT) softly executes the logical form on the sentence.
predicates, namely String Matching Module, Soft Counting Module, Deterministic Function Mod- ule, and Logical Calculation Module. Any complex logical form can be disassembled into clauses containing these four categories of predicates. The four modules are then used to evaluate each clause and then the whole logical form in softened way. Fig. shows how NExT builds the execu- tion tree from an NL explanation and how it evaluates an unlabeled sentence. We show in the figure the corresponding module for each predicate in the logical form.
3.1 MODULES NEXT
String Matching Module. Given a keyword query q derived from an explanation and an input sequence [w), W2.--,,Wu W.] the string matching module f=(x, q) returns a sequence of scores [21, .... Sn] indicating the similarity between each token and the query q. Previous work implements this operation by exact keyword searching, while we augment the module with neural networks to enable capturing semantically similar words. Inspired by Li et al. (2018). for token wi, we first generate N₂ contexts by sliding windows of various lengths. For example, if the maximum window size is 2, the contexts Cιo- Ca1. C22 of token w; are [w:]. [w wzy and [wy; Ws+ respec- tively. Then we encode each context Ciy to vector Ze,, by feeding pre-trained word embeddings into a bi-directional L.STM encoder (Hochreiter & Schmidhuber 1997). All hidden layers of Bil.- STM are then summarized by an attention layer (Bahdanau et al. 2014). For keyword query q. we directly encode it into vector zy by bi-LSTM and attention. Finally, scores of sentence and query are calculated by aggregating similarity scores from different sliding windows:
Myy (x, 9)
cos (Ze.) D), fx(x,q)
M(x,
where D is a trainable diagonal matrix, v E R.
No is the trainable weight of each sliding window.
Parameters in the string matching module need to be learned with data in the form of (sentence, keyword, label). To build a training set for learning string matching, we randomly select spans of consecutive words keyword queries in the training data. Each query is paired with the sentence it comes from. The synthesized dataset denoted as {x,, qu k; ) where k;j will take the value of 1 if is extracted from Xij and 0 otherwise. The loss function is defined as the binary cross-entropy loss, as follows.
Naya
Lfind
(k;
q:) + +1 k:) log(1 Sa (xi, q:))).
N syy
k.j
While pretraining with Liind enables matching similar words, this unsupervised distributional method is poor learning their semantic meanings. For example, the word "good" will have rel- atively low similarity to "great" because there is no such training data. To solve this problem, we borrow the idea of word retrofitting (Faruqui et al. 2014) and adopt contrastive loss (Neculoiu et al. 2016) to incorporate semantic knowledge in training. We use the keyword queries in labeling functions as supervision. Intuitively, the semantic meaning of two queries should be similar if they appear in the same class of labeling functions and dissimilar otherwise. More specifically. for query q, we denote queries in the same class of labeling functions as Q+ and queries in different classes of labeling functions as Q_ The similarity loss is defined as:
Tegim = max {(T cos(2,, D,2q, Zqu D))2 + max (cos(zg D, zqp D)2}- E
The overall objective function for string matching module is:
Lstriig- Iijind ? Tegims
4

---

Published as a conference paper at ICLR 2020
Algorithm 1: Learning on Unlabeled Data with NExT
Input:
Labeled data S. = [(xx( :Mx (a unlabeled data S. 1 and logical forms = {
Output: A classifier fe : X - -...
Pretrain String Matching Module in NExT w.r.f. Luteing using Eq. while not converge do
Sample labeled batch B3, [(x]; )) from Sa, and unlabeled batch B.. = (xx )3= from S... foreach x" E B,, do
Calculate a pseudo label yj for xj with confidence uy using NExT and F.
Normalize matching scores {uy} to get {wz]}=] based on Eq. 3
Calculate La using Eq. [ Lu using Eq. + L using Eq. S
Update fo and String Matching Module in NExT W.E.f. Liotal
where y is hyper-parameter. We pretrain the string matching module for better initialization.
Soft Counting Module. The soft counting module aims to relax the counting (distance) con- straints defined by NL explanations. For counting constraint precede object by no more than three words, the soft counting module outputs matching score indicating to which extent chor word (TERM, SUBJECT, and OBJECT) satisfies the constraint. The score is set to if the position of the anchor word strictly satisfies the constraint, and will decrease if the constraint is bro- ken. For simplicity, we allow an additional range in which the score set to la E (0, 1), which is hyper-parameter controlling the constraints.
The deterministic function module deals with deterministic pred- icates like "Left" and "Right", which can only be exactly matched because a string is either right or left of an anchor word. Therefore, the probability it outputs should be either 0 or 1. The Determin- istic Function Module deals with all these predicates and outputs a mask sequence, which is fed into the tree structure and combined with other information.
Deterministic Function Module.
Logical Calculation Module. The logical calculation module acts a score aggregator. It can aggregate scores given by: (1) string matching module and soft counting module/deterministic function module (triggered by predicates such "Is" and "Occur") (2) two clauses that have been evaluated with score respectively (triggered by predicates such "And" and "Or").
In the first case, the logical calculation module will calculate the element-wis wise products of the score sequence provided by the string matching module and the mask sequence provided by the soft count- ing module/deterministic function module. then uses max pooling to calculate the matching score of the current clause. In the second case, the logical calculation module will aggregate the scores of at least one clause based on the logic operation. The rules are defined as follows.
Pl A P2 max(pı + P2 - 1,0), P1 V p2 = min(pi + P2, 1), -p= = 1 -
where p the score of the input clause.
3.2 AUGMENTING MODEL LEARNING WITH NExT
As described in Algo. I in each iteration, sample two batches B, and B,, from S, and S., We conduct supervised learning on B.. The labeled loss function is calculated as:
L 1
Σ
log 1og,(u (x.).
(2)
Na
To leverage B,,, which is also informative, for each instance x") E B,,. modules to compute its matching score with every logical form. The most probable logical form that matched with x5 is denoted y along with the matching score uj- To ensure the scale of the unlabeled loss is comparable to labeled loss, we normalize the matching scores among pseudo- labeled instances in B,, as:
use matching
wy exp(O.u,)
(3)
exp(Orun)
None label (e.g. No Relation for relation extraction and Neutral for sentiment analysis) usually lacks explanations. If the entropy of downstream model prediction distribution over labels is lower than threshold, None label will be given.
5

---

Published as a conference paper at ICLR 2020
Dataset
exps
calegs
avg ops
logie/%
assertion/%
position/%
counting/5
acc/%
TACRED
170
13
8.2
25.8
21.3
21.4
12.4
95.3
SemEval
203
4.2
32.7
15.9
26.3
5.5
84.2
Laptop
40
3.9
0.0
23.8
23.8
17.5
Restaurant
9.6
2.8
25.4
26.1
16.2
88.2
Table I: Statistics for Human-curated Explanations and Evaluation of Semantic Parsing, We report the number of NI. explanations (exps), categories of predicates (categs) and operator compositions per explanation (avg ops) respectively. We also report the proportions of different types of predicates, where logic denotes logical operators (And, Or), assertion denotes assertion predicates (Occur, Contains), position denotes position predicates (Right, Between) and counting denotes counting predicates (MoreThan, AtMost). We summarize accuracy (acc) of semantic parsing based on human evaluation.
where the index of the instance and hyperparameter 0, (temperature) controls the shape of normalized scores' distribution. Based on that, the unlabeled loss is calculated as:
Lu
Σ (x)
w/ log (x)))
(4)
Note that the string matching module is also trainable and plays a vital role in NExT. We jointly train it with the classifier by optimizing:
Liotal = La + a Lu + B Letring:
(5)
where a and /3 are hyper-parameters.
4 EXPERIMENTS
Tasks and Datasets. We conduct experiments on two tasks: relation extraction and aspect-term- level sentiment analysis. Relation extraction (RE) aims to identify the relation type between two entities in sentence. For example, given sentence Steve Jobs founded Apple Inc, we want to extract triple (Steve Jobs, Apple Inc., Founder). For we choose two datasets, TACRED (Zhang et al. (017) and SemEval (Hendrickx et al., 2009) in our experiments. Aspect-term-level sentiment analysis (SA) aims decide the sentiment polarity with regard to the given aspect term. For ex- ample, given sentence Quality ingredients preparation all around, and very fair price for NYC, the sentiment polarity of the aspect term price is positive, the explanation can be The word price is two customer review datasets, Restaurant and Laptop,
directly preceded by fair. For this task which are part of SemEval 2014 Task 4.
Explanation Collection. We use Amazon Mechanical Turk collect explanations for a randomly sampled set of instances in each dataset. Turkers are prompted with list of selected predicates (see Appendix) and several examples of NL explanations. Examples of collected explanations are listed in Appendix. Statistics of curated explanations and intrinsic evaluation results of semantic parsing are summarized in Table I To ensure a low-resource setting (i.e., |S'| <<' |SS), in each experiment only use random subset of collected explanations.
Compared Methods. As mentioned in Sec. logical forms partition unlabeled corpus S into labeled set S2 and unlabeled set S,. Labeled S, can be directly utilized by supervised learn- ing methods. (1) CBOW-GloVE uses bag-of-wordd (Mikolov et al. 2013) on GloVe embeddings (Pennington al. 2014) represent an instance, or surface patterns in NL explanations. It then annotates the sentence with the label of its most similar surface pattern (by cosine similarity). (2)
PCNN (Zeng et al., (015)) uses piece-wise max-pooling to aggregate CNN-generated features. (3)
LSTM+ATT (Bahdanau al. 2014) adds attention layer onto LSTM to encode sequence. (4)
PA-LSTM (Zhang et al., 2017 combines LSTM with an entity-position aware attention to conduct
relation extraction. (5) ATAE-LSTM (Wang al. 2016) combines the aspect term information
into both embedding layer and attention layer to help the model concentrate on different parts of a sentence.
For semi-supervised baselines, unlabeled data S, is also introduced for training. For methods re- quiring rules as input, surface pattern-based rules transferred from explanations. Compared

---

Published as a conference paper at ICLR 2020
TACRED
SemEval
(F)
23.33
33.86
Restaurant
Laptop
CBOW-GloVe (R + +)
34.610.4
48.811.1
(F)
7.7
13.1
CBOW-GloVe (R S)
68.5++2.9
6..5
PCNN (S.)
34.8+0.99
41.811.2
PA-LSTM (S.)
41.3100.8
57.311.5
PCNN (S.)
72.6+1.2
60.9+1.1
BILSTMAATT (S.)
41.4+1.0
58.0+1.66
ATAE-LSTM (S.)
71.1100.4
56,2.13.6
STTT (S)
3..4+1.4
54.1=1.0
ATAF-ISTM (S))
71.4++0.5
1..0+1.4
Self Training (S, S.)
41.7+:1.5
55.2+-0.8
Self Training (S. S.)
71.2.10.5
57.612.1
Pseudo Labeling (S, + S.)
41.51:12
53.51112
Pscudo Labeling (S. S.)
70.9+0044
58.0+1.9
Mean Teacher (Sa Su)
40.810.9
56,01111
Mean Teacher (S. + Sw)
72.0+1.5
62.1+2.3
Mean Teacher (8) + Stu)
25.9+2222
52.2+0.77
Mean Teacher (S: + Ssa)
74.1
61.7+3.7
DualRE (S.- S..)
32.610.77
61.7+009
Data Programming (E S)
71.2.10.0
61.5.0.11
Data Programming (E S)
30.8+:2.4
43.94:2.4
NExT (E S)
75.8++0.8
6,.8+1.9
NExT (E + +S)
4..6110.4
63.5111.
(b) Sentiment Analysis
(a) Relation Extraction
Table 2:
Experiment results on Relation Extraction and Sentiment Analysis. Average and standard de- viation of F1 scores (%) over multiple rums are reported (5 rums for RE and 10 rums for SA). I.F(E) denotes directly applying logical forms onto explanations. Bracket behind cach method illustrates corresponding data used in the method. S denotes training data without labels, & denotes explanations, denotes surface pattem rules transformed from explanations; S. denotes labeled data annotated with explanations, S., denotes the re- maining unlabeled data. St denotes labeled data annotated using same time as creating explanations E, Siz. denotes remaining unlabeled data corresponding to St.
semi-supervised methods include: (1) Pseudo-Labeling (Lee) 2013) first trains classifier on la- beled dataset, then generates pseudo labels for unlabeled data using the classifier by selecting the labels with maximum predicted probability. (2)
(Rosenberg et al.) (005) proposes expand the labeled data by selecting a batch of unlabeled data that has the highest confidence and generate pseudo-labels for them. The method stops until the unlabeled data is used up. (3) Mean-Teacher (Tarvainen Valpola| (017) averages model weights instead of label predictions and assumes similar data points should have similar outputs. (4) DualRE (Lin et al. 2019) jointly trains a relation prediction module and a retrieval module.
Learning from explanations is categorized generated for a downstream classifier. (1) Data Programming (Hancock et al., 2018; Ratner et al. 2016) aggregates results of strict labeling functions for each instance and uses these laeudo-labels to train a classifier. (2) NExT (proposed work) softly applies logical forms to get annotations for unlabeled instances and trains a downstream classifier with these pseudo-labeled instances. The downstream classifier is BiLSTM+ATT for relation extraction and ATAE-LSTM for sentiment anal-
third setting. Explanation-guided pseudo labels are
ysis.
4.1 RESULTS OVERVIEW
Table 2 (a) lists F1 scores of all relation extraction models. Full results including precision and recall can be found in Appendix A.44 We observe that proposed NExT consistently outperforms all baseline models in low-resource setting. Also, we found that, (1) directly applying logical forms unlabeled data results in poor performance. We notice that this method achieves high precision but low recall. Based on our observation of the collected dataset, this is because people tend to use detailed and specific constraints in an NL explanation to ensure they cover all aspects of the instance. As a result, those instances that satisfy the constraints are correctly labeled in most cases, and thus the precision is high. Meanwhile, generalization ability is compromised, and thus the recall is low. (2) Compared to its downstream classifier baseline (BiL-SM++ATT with S.). NExT achieves 4.2% Fl improvement in absolute value on TACRED, and 5.5% on SemEval. This validates that the expansion of rule coverage by NExT is effective and is providing useful information to classifier training. (3) Performance gap further widens when we take annotation efforts into account. The annotation time for & and S are equivalent; but the performance of BiLSTM+ATT significantly degrades with fewer instances in S. (4) Results of semi-supervised methods are unsatisfactory. This may be explained with difference between underlying data distribution of S, and S,.-

---

Published as a conference paper at ICLR 2020
Table 2 (b) lists the performances of all sentiment analysis models. The observations similar to those of relation extraction, which strengthens our conclusions and validates the capability of NExT.
4.2
PERFORMANCE ANALYSIS
TACRED
SemEval
Restaurant
Laptop
Full NExT
45.6.10.
63.511.0
75.8.0.8
62.8_1.9
No counting
44.6:10.9
63.2:0.77
75.6:1..8
62,4=1.9
No matching
41.8.11.1
54.611.2
71.2.1.4
57.0
Drisen:
Lessm
42.5:+100
56.24229
70.7=0.8
59.4=0.7
Lind
43.211..
60.241.9
70.043.5
58.1=2.8
Table 3: Ablation studies on modules of NExT and losses of string matching module. FI score the test set is reported. We remove soft counting module (No counting) and string matching module (No matching) by only allowing them to give 0/1 results.
Figure 4: NExT's performance w.r.t. numher unlaheled instances.
Effectiveness of softening logical rules. As shown in Table 3 we conduct ablation studies on TA- CRED and Restaurant. We remove two modules that support soft logic (by only allowing them to give 0/1 outputs) to see how much does rule softening help in our framework. Both soft counting module and string matching module contribute to the performance of NExT. It can be easily con- cluded that string matching module plays vital role. Removing leads to significant performance drops, which demonstrates the effectiveness generalizing when applying logical forms. Besides, examine the impact brought by T.sim and find Removing them severely hurts the performance, indicating the importance semantic learning when performing fuzzy matching.
Performance with different amount of unlabeled data. To investigate how NExT's perfor- mance is affected by the amount of unlabeled data, we randomly sample 10%, 30%, 50% and 70% of the original unlabeled dataset do the experiments. As illustrated in Fig. 4 our NExT benefits from larger amount of unlabeled data. We attribute it to high accuracy of logical forms converted from explanations.
Superiority of explanations in data efficiency. In the real world, with limited human ppwer, there question of whether is more valuable to spend time explaining existing annotations than just an- notating more labels. To answer this question, we conduct experiments on Performance v.s. Time on TACRED dataset. We compare the results of supervised classfier with only labels as input and NExT with both labels and explanations anno- tated using the same annotation time. The results listed Figure 5 from which NExT achieves higher performance while labeling speed reduces by half.
40
F1 Soore
LSTMMATT
see that
Time/ min
Figure 5: Performance of NExT traditional supervised method. Blue line denotes NExT and dashed line denotes annotating numbers, normal line means performance. Red line denotes tradi- tional supervised method, and dashed line performance, normal line means annotating num- bers.
Performance with different number of explana- tions. From Fig. 6 one can clearly observe that all approaches benefit from more labeled data. Our NExT outperforms all other baselines by large margin, which indicates the effectiveness of lever- aging knowledge embedded in NL explanations. We can also that, the performance of NExT
with 170 explanations on TACRED equals to about 2500 labeled data using traditional supervised method. Results of Restaurant also have the same trend, which strengthens conclusion. Besides
Fig. 6, we conduct more experiments for this ablation study and make 4 results tables, see Appendix A.5 for details.
8

---

Published as a conference paper at ICLR 2020
60
40
70
40
80
35
56
00
Teadner
Piccudo Lobeling 15x
Prauado Label bigiing i5, -5,)
40
100
120
140
160
180
200
220
240
20
30
40
50
80
AEpplenaiooa
REppaaations
(a) TACRED
Restaurant
Figure 6: Performance with different number of explanations. We choose supervised semi-supervised baselines for comparison. We did experiments on TACRED and Restaurant. Gray dashed lines mean the performance with the corresponding labeled data.
4.3
ADDITIONAL EXPERIMENT ON MULTI-HOP REASONING
To further test the capability of NExT in downstream tasks, apply it to WIKIHOP (Welbl et al. 2018 country task by fusing NExT-matched facts into baseline model NLPROLOG (Weber et al, 2019). For brief introduction, WIKIHOP country task requires model to select the correct candi- date ENT-Y for question "Which country is ENT-X in?" given a list of support sentences. As part of dataset design, the correct answer can only be found by reasoning over multiple support sentences.
NIPROLOG is a model proposed for WIKIHOP. first extracts triples from support sentences and entities. For example, "Socrate was born in
treats the masked sentence as the relation between Athens" is converted to (Socrate, "ENTI was born in EN72", Athens), where "ENTI was born in ENT2" is later embedded by SENT2VEC (Pagliardini et al.) 2018 represent the relation between ENTI and ENT2. Triples extracted from supporting sentences are fed into a Prolog reasoner which will do backward chaining and reasoning to arrive at the target statement country(ENT-X: ENT-Y). We refer readers to (Weber et al. 2019) for in-depth introduction of NLProlog NLPROLOG.
Fig. 7 shows how the framework in Fig. adjusted suit NLPROLOG. We manually choose 3 predicates (i.e., located in, capital of, next to) and annotate 21 support sentences with natural language explanation. We get 103 strictly-matched facts (S.) and 1407 NExT-matched facts (S,) among the 128k unlabeled QA support sentences. Additionally, these predicates for the Prolog solver, e.g. located ln(,)e + located in(X,) A located in(Y.).
manually write 5 rules about
Results are listed in Table From the result we observe that simply adding the 103 mtrietly-matched facts is not making notable improvement. However, with the help of NExT, a larger number of structured facts are recognized from support sentences, that external knowledge from only 21 ex- planations and rules improve the accuracy by point. This observation validates NExT's capability in low resource setting and highlight its potential when applied downstream tasks.
Training NLProlog
[S,]]
[S,]
Accuracy
Lobeled
warchell
NLProlog (published code)
0
74.57
Foet Extrectios
Tralniing
+ S.
74.40
+ S. (confidence >0.3)
103
340
74.74
NLProlog
+ S, (confidence >0.2)
103
577
75.26
S., (confidence >0.1)
103
832
75.60
NEa
Lobrleed by HiXT
Table 4: Performance of NLPROLOG when extracted facts are
used as input. Average accuracy over 3 runs is reported. NL- PROLOG empowered by 21 natural language explanations and 5
Figure 7:
Adjusting NEXT
Framework (Fig. LOG.
2 for NLPRO-
rules achieves 1% gain in accuracy.

---

Published as a conference paper at ICLR 2020
5 RELATED WORK
Leveraging natural language for training classifiers. Supervision in the form of natural language has been explored by many works. Srivastava et al. (2017) first demonstrate the effectiveness of NL explanations. They proposed a joint concept
and semantic parsing method for classification problems. However, the method is very limited in that it is not able to use unlabeled data. To ad- dress this issue, Hancock et al.] (2018) propose to parse the NL explanations into labeling functions and then use data programming to handle the conflict and enhancement between different labeling functions. Camburu et al. (2018) extend Stanford Natural Language Inference dataset with NL ex- planations and demonstrate its usefulness for various goals for training classifiers. Andreas et al. (2016) explore decomposing NL questions into linguistic substructures for learning collections of neural modules which can be assembled into deep networks. Hu et al. (2019) explore using NL instructions as compositional representation of actions for hierarchical decision making. The sub- structure of an instruction summarized as a latent plan, which then executed by another model. Rajani et al.) (2019) train language model to automatically generate NL explanations that can be used during training and inference for the task of commonsense reasoning.
suaely-supervised learning. Our work is relevant to weakly-supervised learning. Traditional sys- tems use handcrafted rules (Hearst 1992) or automatically learned rules (Agichtein & Gravano 2000 Batista et al.] 2015) to take rule-based approach. Hu et al. (2019) incorporate human knowl- edge into neural networks by using a teacher network to teach the classifier knowledge from rules and train the classifier with labeled data. [ et al. (2018) parse regular expression to get action trees as a classifier that are composed of neural modules, so that essentially training stage is just a pro- cess of learning human knowledge. Meanwhile, if we regard those data that are exactly matched by rules as labeled data and the remaining as unlabeled data, we can apply many semi-supervised models such as self learning (Rosenberg al. 2005), mean-teacher (Tarvainen & Valpola 2017), and semi-supervised VAE (Xu et al.] (2017). However, These models turn out to be ineffective in rule-labeled data or explanation-labeled data due to potentially large difference in label distribution. The data sparsity is also partially solved by distant supervision (Mintz et al., 2009 [Surdeanu et al.) 2012). They rely on knowledge bases (KBs) annotate data. However, the methods introduce lot of noises, which severely hinders the performance. [Luu et al. (2017) instead propose to conduct relation extraction using annotations from heterogeneous information source. Again, predicting true labels from noisy sources is challenging.
6 CONCLUSION
In this paper, we presented NExT, a framework that augments sequence classification by exploiting NL explanations supervision under a low resource setting. We tackled the challenges of modeling the compositionality of NL explanations and dealing with the linguistic variants. Four types of mod- ules were introduced to generalize the different types of actions in logical forms, which substantially increase the coverage of NI. explanations. A joint training algorithm was proposed to utilize infor- mation from both labeled dataset and unlabeled dataset. We conducted extensive experiments on several datasets and proved the effectiveness of our model. Future work includes extending NExT to sequence labeling tasks and building cross-domain semantic parser for NL explanations.
ACKNOWLEDGMENTS
This research is based upon work supported in part by the Office of the Director of National In- telligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 1199-1951100000 NSF SMA 18-29268, and Snap research gift. The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Govemment. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.
REFERENCES
Eugene Agichtein and Luis Gravano. Snowball: Extracting relations from large plain-text collec- tions. In Proceedings of the fifth ACM conference on Digital libraries, pp. 85-94. ACM, 2000.
10

---

Published as a conference paper at ICLR 2020
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. Broad-coverage eog semantic parsing with amr. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp- 1699-1710, 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
David S Batista, Bruno Martins, and Mário Silva. Semi-supervised bootstrapping of relationship extractors with distributional semantics. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 499-504, 2015.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-smli: natu- ral language inference with natural language explanations. In Advances in Neural Information Processing Systems, pp. 9539-953444 2018.
Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv: 1411.4166, 2014.
Braden Hancock, Martin Bringmann, Paroma Varma, Percy Liang, Stephanie Wang. and Christopher Ré. Training classifiers with natural language explanations. In Proceedings of the conference. Associationfor Computational Linguistics. Meeting, volume 2018, pp. 1884. NIH Public Access, 2018.
Marti A Hearst. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 2, pp. 539-545. Association for Computa-
14th conference on Computational linguistics-Volle tional Linguistics, 1992.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebas- tian Pad6, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Fulure Directions, pp. 94-99. Association for Computational Linguistics, 2009.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 17): 1780, November 1997. ISSN 0899-7667, doi: 10.1 wel116///co..9999990 1997.9.8.77355 httptt/://dx. doi. org/ 10. 1 62 www.///w..... 1991.99999.... 9.........1...
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hicrarchical decision making by generating and following natural language instructions. arXiv preprint /0f1://906.0044., 2019.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2013.
Shen Li, Hengru Xu, and Zhengdong Lu. Generalize symbolic knowledge with neural rule engine. arXiv preprint arXiv: 1808 10326, 2018
Percy Liang, Michael I Jordan, and Dan Klein. Learning dependency-based compositional seman- tics. Computational Linguistics, 39(2):389 2013.
Hongtao Lin, Jun Yan, Meng Qu, and Xiang Ren. Learning dual retrieval module for semi- supervised relation extraction. In The Web Conference, 2019.
Liyuan Liu, Xiang Ren, Qi Zhu, Shi Zhi, Huan Gui, Heng Ji, and Jiawei Han. Heteroge- neous supervision for relation extraction: A representation learning approach. arXiv preprint 17X1v:1007.00166. 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 31111111133 111113119, 19, 2013.
11

---

Published as a conference paper at ICLR 2020
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference Natural Language Processing of the AFNLP: Volume 2-Volume 2, pp. 1003-1011. Association for Computational Linguistics, 2009.
Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. Learning text similarity with siamese recur- rent networks. In Rep4NI.P@ACI, 2016.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embed- dings using compositional n a-gram features. In Proceedings of NAACL-HLT, pp. 528-540, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02336 2019.
Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in neural information processing systems. pp. 3567-3575, 2016.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. 2005.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. Joint concept learning and semantic parsing from natural language explanations. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 1527-1536, 2017.
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher Manning. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 joint conference empir- ical methods natural language processing and computational natural language learning, pp- 455-465. Association for Computational Linguistics, 2012.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: -veehte-veraged consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1204, 2017.
Yequan Wang, Minlic Huang, Xiaoyan Zhu, and Li Zhao. Attention-based Istm for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 606-615 615, 2016.
Weber, P Minervini, Münchmeyer, Leser, and Rocktäschel. NIprolog: Reasoning with weak unification for question answering in natural language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, Florence, Italy, Volume 1: Long Papers, volume 57. ACL (Association for Computational Linguistics), 2019.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. Variational autoencoder for semi-supervised text classification. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1753 2015.
Luke Zettlemoyer and Michael Collins. Online learning of relaxed ceg grammars for parsing to
logical form. In Proceedings of the 2007 Joint Conference Empirical Methods in Natural Lan- guage Processing and Computational Natural Language Learning (EMNLP-CoNLA, pp. 678- 687, 2007.
12

---

Published as a conference paper at ICLR 2020
Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv: 1420, 2012.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. Position-
aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 35-45, 2017.
13

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1 PREDICATES
Following Srivastava et al. 2017). first compile domain lexicon that maps each word to its syntax and logical predicate. Table lists some frequently used predicates in our parser, descriptions about their function and modules they belong to.
Predicate
Description
Module
Because, Separator ArgX, AngY, Arg Int. Token, String
Basic conjunction
Subject, object aspect in each task
Primitive data types
True, False
Boolean operators
And, Or, Not, Is, Occur
Logical operators that aggregate matching scores
Logical Calculation Module
Left, Right, Between, Within
Return True if one string is letürihhheeieeen/vthhnn some runge of the other string
NumherOf
the number words given range
Deterministic Function
AtMost, AtLeast, Direct,
Counting (distance) constraints
Soft Counting Module
More Than, LessThan, Equals
Word, Contains,
Return a matching score sequence for sentence query
String Matching Module
Table Frequently used predicates
A.2 EXAMPLES FOR COLLECTED EXPLANATIONS.
TACRED
OBI-ORGANIZATION coach SUBJ-PERSON insisted he would put the club 's 2-0 defeat Palermo firmly behind him and move forward
(Label) per :eeeloyeyeeof
(Explanation) there is only one word "coach" between SUBJ and OBJ
Officials Mumbai said that the two suspects David Coleman Headley. an American with links of Pakistan, and SUBJ-PERSON. who born Pakistan but is OBI-NATIONALITY citizen both visited Mumbai and several other Indian cities before the attacks, and may have visited some of the sites that were attacked
(Label) per:origin
(Explamation the words "is a" appear right before ORU-NATIONALITY and the word "citizen" is right after OBJ-NATIONALITY
SemEval 2010 Task 8
The SUBJ-O is caused by the OBJ-O of UV radiation by the oxygen and ozone
(Label) Cause-Effect (e2, el)
(Explanation) the phrase "is caused by the" occurs between SUBJ and OBJ and OBJ follows SUBJ
the SUBJ-O are parts of the OBJ-O OBJ-O disregarded by the compiler
(Label) Component
(el,e2)
(Explanation) the phrase "are parts of the" occurs between SUBJ and OBJ and OBJ follows SUBJ
SemFval 2014 Task 4 restaurant
14

---

Published as a conference paper at ICLR 2020
I am relatively new the area and tried Pick a bgel on 2nd and was disappointed with the service and thought the food was overated and on the pricey side (Term: food)
(Label) negative (Explanation) the words "overated" is within words after term
The decor is vibrant and eye eye-pleasing with several semi-priven boths on the right side of the dining hall, which great for date (Term: decor)
(Label) positive
(Explanation) the term is followed by "vibrant" and "eye-pleasing
SemEval 2014 Task 4 laptop
It's priced very reasonable and works very well right out of the box. (Term: works) (Label) positive (Explanation) the words "very well" occur di directly after the term
The DVD drive randomly pops open when in my backpack as well, which is annoying (Term: DVD drive)
(Label) negative
(Explanation) the word "annoying" occurs after term
A.3 IMPLEMENTATION DETAILS
We use 300-dimensional word embeddings pre-trained by GloVe (Pennington et al. 2014). The dropout rate is 0.96 for word embeddings and 0.5 for sentence encoder. The hidden state size of the encoder and attention layer is 300 and 200 respectively. We choose Adagrad as the optimizer and the learning rate for joint model learning is 0.5.
For TACRED, we set the learning rate to 0.1 the pretraining stage. The total epochs for pretraining 10. The weight for Laim is set 0.5. The batch size for pretraining is set to 100. For training the classifier, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight a for Ly is 0.7, the weight 33 for Lettrngg is set 0.2, the weight for Laim set to 2.5.
For SemEval 2010 Task 8, we set the learning rate to 0.1 in the pretraining stage. The total epochs for pretraining are 10. The weight for Leim is training the classifier, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight a for I., is set to 0.5, the weight / for Littrnng is set to 0.1, the weight 7 for L is set to 2.
0.5. The batch size for pretraining is set to 10. For
For two datasets in SemEval 2014 Task 4, we set the learning rate to 0.5 the pretraining stage. The total epochs for pretraining are 20. The weight for Luim is set to 5. The batch size for pretraining is set to 20. For training the classifier, the batch size for labeled data and unlabeled data 10 and 50 respectively, the weight for L., is set to 0.5, the weight 9 for Letring is set to 0.1, the weight y for Lesm is set to 2. For ATAE-LSTM, we set hidden state of attention layer to be 300 dimension.
A.4 FULL RESULTS FOR MAIN EXPERIMENTS
The full results for relation extraction and sentiment analysis are listed Table and Table spectively.
7 re-
15

---

Published as a conference paper at ICLR 2020
TACRED
SemEval
Metric
Precision
Recall
F1
Precision
Recall
F1
LF (E)
83.21
13.56
23.33
83.19
21.26
33,86
CBOW-GloVe (R S)
28.2.10.7
44.9.10.9
34.6.10.4
1.3
51.2.12.2
48.
PCNN (8.)
1..8
2.1911.11
34.8=0..
51.5#19
35.2=1.4
41.8+1.
PA-LSTM (S.))
44.4
38.7+:2.2
41.3-00.8
59.9422.4
54.9-12.2
57.3++1.5
BILSTM+ATT (Sa)
43.8++2.0
39.44:2.6
4..4
60.04:2.1
56.2+11.3
58.0+11.6
SiTSTM+ATT (Si)
42.8+42.6
23.8+22.4
3..4+:.4
54.7+:1.0
53.6+112
54.1+1.0
Data Programming (E S)
45.9:1..8
23.312.6
30.8#2.4
51.313.5
38.8
43.912.44
Self Training (S, + S.)
45.942.3
38.442.77
41.7++1.5
57.342.11
53.3+
55.2+008
Pscudo Labeling (S.. S..)
44.5+1.5
38.9+1.6
41.5+112
53.7422.
53,4+2.2
53,5+1.2
Mean Teacher (S,, + S..)
39.211.77
42.6.11.8
40.810.9
60.8.199
51.91112
56.011.1
Mean Teacher (St + St..)
28.3115.7
25.41558
25.911222
53.113.8
51.6112.4
52.210.77
DualRE (S. + S.)
38.8.14.7
28.6.12.9
32.610.77
64.5110.7
59.2.1..0
61.71009
NExT S)
49.2.10.9
42.4.1.33
45.6
66.31114
61.0.12.2
63.5
Table 6: Full results as supplement to Table 2
Restaurant
Laptop
Metric
Precision
Recall
F1
Precision
Recall
F1
(E)
86.5
4.0
7.7
90.0
7.1
13.1
CBOW-GloVe (R S)
62.8.12.8
75.3.1..1
68.5
13.4.11.1
1.5
6..5111.3
PCNN (S.))
67.112.1
7.80.1.8
72.6111.2
53.1111.0
71.41111
60.911.11
ATAE-LSTM (S.)
65.1110.4
78.4.10.6
71.1110.4
49.0.13.1
66.0.1..4
56.2.13.6
ATAE-LSTM (S))
65.3.10.5
78.910.5
71.40..5
48.9.11.5
55.6
$2.0.11.4
Data Programming (E + S)
65.0+:0.0
78.8+0.0
71.2=000
53,410..1
72.5=0.1
61.5+0.1
Self Training (S, + S.)
65.3
78.4:10.9
71.2--0.5
50.14-188
67.7+22.4
$7.642.11
Pseudo Labeling (Sa + S.)
64.9+
78.0
70.9+10.4
50.41:.66
68.4
58.0+11.9
Mean Teacher (S,, + S.)
68.812.2
75.7
72.01:15
54,411.7
72.3+4.0
62.1
Mean Teacher (S: St.))
68.3110.8
81.010.4
74.1=0.4
55.014.11
7..3
61.743.7
NExT (E + S)
69.6++0.9
83.3+-1.8
75.8+
54.6+1166
73.9++2.3
62.8+1.9
Table Full results as supplement Table b
A.5 PERFORMANCE WITH DIFFERENT NUMBER EXPLANATIONS
As supplement to Fig. 6 we show the full experimental results with different number of explana- tions as input Table Results show that our model achieves best performance compared with baseline methods.
A.6 MODEL--AGNOSE
Our framework model-agnostic as it can be integrated with any downstream classifier. We con- duct experiments SA Restaurant dataset with 45 and 75 explanations using BERT downstream classifier, and the results are summarized in Table [2 Results show that our model still outperforms baseline methods when BERT is incorporated. We observe the performance of NExT is approach- ing the upper bound 85% (by feeding all data to BERT), with only 75 explanations, which again demonstrates the annotation efficiency of NExT.
A.7 CASE STUDY STRING MATCHING MODULE.
String matching module plays vital role in NExT. The matching quality greatly influences the accuracy pseudo labeling. Fig. g we can see that keyword chief executive of is perfectly aligned with executive director of in the sentence, which demonstrates the effectiveness of string matching module in capturing semantic similarity.
16

---

Published as a conference paper at ICLR 2020
TACRED 130
TACRED 100
Metric
Precision
Recall
F1
Precision
Recall
F1
LF (E)
83.5
12.8
22.2
85.2
20.7
CBOW-GloVe (R + S)
26.0++2.3
39.9+55.0
31.2++0.5
2..4+
41.7++3.7
30.7++0.1
PCNN (S.)
41.8:12.7
28.811.8
34.1=1.1
28.2413.
2..2=
24.8:11.9
PA-LSTM (Sa)
44.9+1.77
33.5+22.9
38.3+11.3
39.9+2.1
38.2+1111
1.3
BiLSTM+ATT (S.)
40.1+226
36,2+3,4
37.9+1.1
36.1+0.4
37.6=3.0
36,8+1.4
BiLSTM+ATT (Si)
35.0+9.0
2..4
28.9++2.7
43.3+2.2
23.1+3.3
30.0+3.11
Solf Training (S, + S.)
43.643.3
35.1#2.1
38.7=0.00
41.95599
32.0=7.4
35.5+2.5
Pseudo Labeling (8, + S.)
14.2
34.2+1.9
38.5+0.6
39.7+2.0
34.9=3.3
37.1
Mcan Teacher (S, + S.)
38,8+0.9
3..6
37.1
37444.0
37.4=022
37,3+2.0
Mean Teacher (S) + Si.)
21.1
28.7111.8
24.2.1...8
17.544.7
18.4
17.94..0
DualRE (S, + S.)
34.943.6
30.5+-2.3
32.3+110
40.6444.3
19.1
25.940.6
Data Programming (E S) NEXT (E S)
34.3++6.11 45.3+
18.7+ 39.2+0.3
23.5++4.99
43.5+-.3 43.9+
15.0++2.3
22.2+2.4
42.0+-1.1
36.2+1199
39.6+0.55
Table TACRED results on 130 explanations and 100 explanations
SemEval 150
SemEval 100
Metric
Precision
Recall
Precision
Recall
LF (E)
85.1
17.2
28.6
90.7
9.0
16.4
CBOW-GloVe (R + S)
14.8+11.9
48.6
46.6+1.1
36.0+1.4
40.2+2.0
37.9:t0.1
PCNN (8.)
49.1
36.1+:2.4
41.5+11.4
43.3+1.4
27.9=1.0
33.940.3
PA-LSTM (S.)
58.0+1122
52.5+10.4
55.1
55.2+1.7
37.7=0.8
44.8
BiLSTM+ATT (S.)
59.2#0.44
53.7+1.8
56.3+1.88
54.915.00
40.5=0.9
1..
ATT (S))
47.6442.66
42.0
44.6++2..5
43.74:2.6
37.6+5.00
40.3
Self Training (S, + S.))
53.414.33
47.512.9
50.1
53,242.3
34.2-2.2
11.611.4
Pseudo Labeling (S, + S.))
55.344.5
$1.0:12.3
53.0=1..
47.444466
39.9=3.9
43.14:0,6
Mean Teacher (S. + Sw)
61.8+4.00
49.1422.6
54.6+-0.2
58.5+119
41.8+2.6
48.7+1.4
Mean Teacher (S) + Sia)
40.6+22.0
31.2++4.5
35.24:3.6
32.7+33.0
25.6++3.1
28.6+2.2
DualRE (S. + Su)
61.7++3.0
56.1+:3.0
58.8+33.0
61.6+1..7
39.7
48.
Data Programming (E + S) NEXT (3 + S)
50.9+10.8
27.0+:0.8
35.0+:3.2
28.0+4.1
17.4=5.5
21.0+3.44
68.5+11.6
60.0+11.7
63.7++0.8
60.2+
53.5-+077
56.7+1.1
Table 9: SemEval results 150 explanations and 100 explanations
Laptop
Laptop
Metric
Precision
Recall
F1
Precision
Recall
F1
(E) CBOW-GloVe S)
90.8
9.2
16.8
89,4
9.2
16.8
53.7:10.2
72.910.2
61.8210.2
53.6110.3
72.410.2
61.610.2
PCNN (S.)
53.5113.3
71.01:3.6
61.0:1:.2
15.6.11.9
1.
63.5+:1.5
ATAE-LSTM (S.)
53.5:10.4
71.912.2
61.3:110
53.711.2
72.9
61.911.5
ATAE-LSTM (S))
48.311100
59.515.00
53,2:12.2
54.11:1.4
61.1113.
$7.412.1
Self Training (S, + S.))
51.311..6
68.6.12.7
58.7112.6
51.2.11.4
68.6
58.711.6
Pseudo Labeling (S, + S.))
51.8.11.7
70.3.12.3
59.711.9
52.410.8
70.9
60.31100
Mean Teacher (S. + S.)
55.110.9
74.111..
63.2.111
55,9133.
73.0-12.6
63.2.11.7
Mcan Teacher (S: St.)
55.5+2..
69.342.8
61.6+2.2
58.0+00.7
73.2
64.7+1.0
Data Programming (E S)
53,410.00
72.610.00 75.942..
61.5.00.0
53.510.1 56,9+0.2
72.5.10.1
61.610.1
NEXT (3 +
56.31:1.3
64.6=1177
77.1=0.6
65.5+10.3
Table 10: Laptop results on 55 explanations and 70 explanations
17

---

Published as a conference paper at ICLR 2020
Restaurant 60
Restaurant 75
Metric
Precision
Recall
F1
Precision
Recall
F1
LF
86.0
3.8
7.4
85.4
6.8
12.6
CBOW-GloVe (R S)
63.7.12.3
1..6.11.3
69.1
64.1111.3
76.6.10.1
69.8:10.7
PCNN (8.)
67.0
81.0+1.
73.3=0.9
68.4+10.1
0..
74.9+t.2
ATAE-LSTM (S.)
65.2
78.5+002
71.2-+0.3
64.7:10.4
78.3+-0.4
70.8+10.4
ATAE-LSTM (S))
1..0
1.2
72.7+:100
66.64:2.0
1..
72.140.66
Self Training (S, S.)
65.2++0.2
78.7+
71.3+0.2
65.7+
77.2+
71.0+-0.1
Pseudo Labeling (S, + S.)
64.9+0.6
77.8+1.0
70.8+0.33
64.9++0.9
77.8+112
70.7+1.0
Mean Teacher (S, + Sw)
68.8+2.3
76.0+2.2
72.2+11.3
73.3+33.5
79.2+33.8
76.0+
Mean Teacher (S: + St..)
69.0:10.8
82.011.11
74.910.7
69.210.77
82,6
75.3100.6
Data Programming (E + S) NEXT (E + S)
78.8+0.1 65.0+0.0
71.2++0.0 76.4=0.4
65.0++00
78.8+0.0
71.2+0.0
71.0+1144
82.8+111
71.9
82.8+1.9
76.9+0.7
Table 11: Restaurant results on 60 explanations and explanations
OBI- PERSOE
executive director the SUBI-ORGANIZATION Saint Anselm College Manchester
0
0.4
0.6
0.8
Figure 8: Heatmap keyword chief executive of and sentence OBJ-PERSON, executive director of SUBJ- ORGANIZATION at Saint Anselm College in Manchester. Results show that our string matching module can successfully grasp relevant words.
45
75
ATAE-LSTM (S.)
79.9
80.6
Self Training (S., S..)
80.9
81.1
Pseudo Labeling (S, + S.))
78.7
81.0
Mean Teacher (S. + S.))
79.3
79.8
NExT (E + S)
81.4
82.0
Table 12: BERT experiments Restaurant dataset using 45 and 75 explanations
18


Reviews:
[{"review_id": "Skli-hQx9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "One recent work that comes to mind from ACL 2019: Leveraging Language Models for Commonsense Reasoning (Rajani et al 2019). In that work, they also have human annotators provide explanations (extending the CommonsenseQA dataset), and they show that by training with these explanations, inference is improved even without them. They also train a language model to generate the explanations, and they show that the language model generated explanations improve performance further at inference time. Seems like a reasonable reference to contrast the more structured approach to using explanations like Srivasta et al (2017), Hancock et al (2018), and this work.\n\nI find the method summary beginning with \"Human explanations are first converted to machine-actionable logical forms by a semantic parser\" until the end of that paragraph to be unnecessary. Actually, as I read it, I find myself asking a lot of questions that get answered below. So I would prefer scrapping that method summary and just getting straight into the Explanation Parsing.\n\n\"indicates the the logical form matches\" redundant 'the'\n\nI can't find a definition for LF(E) anywhere, and yet LF(E) is present in many tables. I see that E is mentioned to be the explanations, but this is only in the caption of Table 2 even though the symbol is first used in the first paragraph of Section 4.1. I'm assuming LF logical forms applied directly to explanations, but this should be stated explicitly. Can you elaborate on why it is so dominant on precision In Table 6 and 7 of the Appendix, but low on recall, rather than just saying this is expected in Section 4.1? \n\n\"For keyword query q, we directly encode it into vector z_q by bi-LSTM and attention.\" Can you elaborate on how z_q is constructed? Is it the final state of a forward LSTM concatenated with the final state of a backward LSTM?\n\nWhy is it so essential that you study the setting in which explanations are low-resource? I'm curious to see what would happen with more explanations.\n\nI am surprised that none of the modules or compared methods include any architecture that use a Transformer or a form of contextualized word vectors (McCann et al 2017, Peters et al 2017, Devlin et al 2018). Is there an explanation for this?\n\nI would prefer to see a larger suite of tested tasks given that each of these datasets is quite small. Would any other tasks from benchmarks like GLUE or SuperGLUE be amenable to your approach? The tasks you've chosen limit the scope of this work and leaves the question of whether it would generally improve across a greater variety of tasks, especially tasks that have seen significant improvement using new methods. Your claim would be much stronger if the explanations were shown to be helpful even to pretrained models like BERT when fine-tuned for a specific task. In particular, it would be interesting to see how the benefits of explanations vary for different kinds of tasks and for different training set sizes. \n\n\"In the real world, a more realistic problem is that, with limited human-power, should we just annotate more labels or spend time explaining existing annotations.\"\" I think that you mean \"should we\" makes it sound like this is a question, but there is no question mark, and it makes more sense as a statement. I propose \"...human-power, there is a question of whether it is more valuable to ...\"\n\n\"Explanations prove to be an very efficient form for data annotation.\" should be \"a very\" not \"an very\".\n\n\" a vital rule\" should be \"a vital role\"\n\nI find the first paragraph of Section 4.3 quite abstruse and bare in its explanation of the method used.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkgwWtPCFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Labeling sentences for NLP requires a lot of human effort. In order to tackle this problem, the system proposed in this paper, called NMET, aims at labeling sentences by exploiting the explanations given by humans. First it converts the explanations into logical formulas. This logical formulas are then exploited for partitioning the dataset into two datasets: labeled dataset and unlabeled dataset. Then, NMET relaxes the logical formulas for labeling unlabeled examples by exploiting a neural architecture that uses four modules to deal with different types of predicates. \n\nThe paper is pretty clear and well-written. It can be understood even by non-experts. The proposed approach seems technically sound and pretty novel. Moreover, the experimental results show that the proposed system achieves better performances than traditional label-only supervised models.\nThe only concern that I see in the paper is that the Deterministic Function Module is not explained very well. It is not clear to me its purpose.\n\n[Minor]\nPage 3\n“the the logical form”\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1e1GcC2Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper explores using natural language explanations as auxiliary training data for NLP tasks. It first transforms natural language expressions into a logical form through CCG, and then use a neural module network architecture to label data instances. Experimental analyses are conducted on two tasks -- relation extraction and sentiment analysis, showing that the proposed approach outperforms previous work that incorporates explanations as training data.\n\nOverall, the paper addresses an important issue of how to utilize human explanations as additional supervision source for NLP tasks and shows promising results. Hence, I believe the paper is above the acceptance threshold, and recommend for weak acceptance.\n\nI had concerns on the cost of collecting human explanations, since they are non-trivial to collect. However, the authors provided convincing arguments regarding the data annotation cost in their response, so I do not think this is a major limitation of the method.\n\nHowever, I would also like to note that the paper has a few limitations. The proposed is based on semantic parsing of the natural language explanations into logical forms and is therefore inherently limited by the representation power of symbolic and logical representations. In the two examples shown in Figure 1, the human explanations are very simple and have limited variety, so it is relatively easy to represent them in logical forms. However, on many NLP tasks (such as question answering), the human explanations (in natural language) may often be complicated and difficult to be represented in CCG. Therefore, it is unclear whether the proposed approach can be easily generalized to other tasks.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposing a framework for augmenting classification systems with explanations was very well received by two reviewers, and on reviewer labeling themselves as "perfectly neutral". I see no reason not to recommend acceptance.
Decision: Accept (Poster)