Soft Token Matching for Interpretable Low-Resource Classification
Authors: Federico Errica
Fabrizio Silvestri
Bora Edizel
Sebastian Riedel
Ludovic Denoyer
Vassilis Plachouras
Venue:  (2020)
Keywords: low-resource classification
semantic matching
error boosting
text classification
natural language processing

Abstract: We propose a model to tackle classification tasks in the presence of very little training data. To this aim, we introduce a novel matching mechanism to focus on elements of the input by using vectors that represent semantically meaningful concepts for the task at hand.
By leveraging highlighted portions of the training data, a simple, yet effective, error boosting technique guides the learning process. In practice, it increases the error associated to relevant parts of the input by a given factor. Results on text classification tasks confirm the benefits of the proposed approach in both balanced and unbalanced cases, thus being of practical use when labeling new examples is expensive. In addition, the model is interpretable, as it allows for human inspection of the learned weights.

Under review as a conference paper at ICLR 2020
SOFT TOKEN MATCHING FOR INTERPRETABLE LOW-RESOURCE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose model to tackle classification tasks in the presence of very little training data. To this aim, we introduce a novel matching mechanism to focus on elements of the input, by using vectors that represent semantically meaningful concepts for the task at hand. By leveraging highlighted portions of the training data, simple, yet effective, error boosting technique guides the learning process. In practice, it increases the error associated to relevant parts of the input by given factor. Results on text classification tasks confirm the benefits of the proposed approach in both balanced and unbalanced cases, thus being of practical use when labeling new examples expensive. In addition, the model interpretable, as it allows for human inspection of the learned weights.
INTRODUCTION
Gathering and labeling data is a task that can be expensive terms of time, human effort and resources. When we cannot rely already available datasets, training a model with acceptable performance on few data points annotated by few annotators, becomes critical in many practical applications. This is, indeed, especially important when the data is naturally imbalanced and the demands of gathering samples of the minority class are high. One important domain in which these issues arise is text classification, for example hate-speech (Waseem & Hovy 2016). web spam Castillo et al., 2007) and abuse detection (Mishra et al. 2018).
One effective approach overcome the lack of training data is that of Zaidan et al. (007 which consists of augmenting the few data available with rationales, i.e., highlighted portions of the input. Rationales are usually coupled with feature-engineering to be effective low resource scenarios. An alternative way to deal with data sparsity, especially in text classification tasks, is to use pre- trained language models (LMs) that are fine-tuned on target domain. While this approach has been tested on hundreds of training points (Devlin et al.) 2018 Howard & Ruder 2018). it is not clear how it behaves in an even scarcer setting, as the vast parameter space of an LM might pose a problem. Moreover, fine-tuning model may require a considerable amount of computing power, therefore restricting its applicability.
On the other hand, some embedding- based models represent the input weighted average of words (Kalchbrenner et al., 2014 [hhiikh et al.) 2016)., where the weight is given by a parame- ter called "reference vector". However, these models cannot easily incorporate multiple reference vectors, and they are not interpretable since classification works on unreadable embedding features.
In this paper, propose novel and efficient representation learning model to address the above issues. The underlying idea is to focus on relevant words in the input while being able to generalise semantically similar concepts; this something akin to what human would do in the presence of data scarcity. We therefore introduce two techniques that should coexist to reflect our intuition. First of all, the model to focuses on specific words by computing soft matching probabilities between each word and multiple vectors which represent semantic concepts. Secondly, we guide the learning process to learn important concepts thanks to an error boosting technique that exploits rationales. Basically, it encourages the model to reduce the overall error by improving the prediction associated with highlighted words.
Additionally, by direct inspection of model weights it is possible to understand what words it focuses on; in short, the model is interpretable" Results across a consistent number baselines and three
See Guidotti et al. 2019) for the notion of interpretability we use throughout the paper.

---

Under review as a conference paper at ICLR 2020
datasets also indicate a significant improvement in performance. Interestingly, always outper- form fine-tuned models when little training data is available. Our model can also assist users to train classifier for very specific task. As example, consider training abstract filtering system with rationales provided by the user itself. The model will then learn to filter out papers that are not matching the user's preferences.
The rest of the paper is structured as follows: Section provides an overview of the existing litera- ture, highlighting similar and different ideas; Section 3 formally introduces the problem as well as our model, providing intuition behind our architectural choices; Section + details our experiments and shows our findings, with a thorough ablation study that disentangles the contribution of each part of the model and use case on interpretability; finally, Section (ection[[]summnn summarizes our work.
2 RELATED WORKS
There are different ways in which rationales can be used. Some works generate rationales, while others exploit them to inform the learning process. The method proposed by Lei et al. (2016) tack- les text classification by learning distribution of rationales given the text and distribution of the target class given the rationales. Interestingly, an additional regularization term is added to the loss to produce rationales that short and coherent. The model makes use of high-capacity recurrent neural networks (Schuster & Paliwal 1997). thus it is tested on large amounts of training data to pre- vent overfitting. This work was later refined by Bastings et al. (2019), who proposed a probabilistic version of a similar architecture, where a latent model is responsible for the generation of discrete rationales. The main advantage of predicting discrete rationales is that is possible to constrain their maximum number per sample, thus effectively controlling sparsity. However, it usually requires a large amount of data points to be effective.
The first to exploit rationales in a low resource scenario were Zaidan et al. (0077) and Zaidan & Eisner (2008), by means of a rationale-constrainee SVM (Cortes & Vapnik, 1995) and a proba- bilistic model. Moreover, the latter is realized as log-linear classifier with heavy use of feature- engineering. On the other hand, when rationales are defined on features rather than on samples, one use the Generalized Expectation (GE) criteria (Druck et al. 2007: significantly improve the performance of classifiers.
McCallum et al. 2007 to
Rationales can also be incorporated in the loss function (Barrett al. 2018), where the attention module (Vaswani et al. 2017) on top of an LSTM (Hochreiter & Schmidhuberi 1997) is forced to at- tend relevant tokens in a document. This method was not tested on small datasets. possibly because of the aforementioned issues of high capacity models. A similar approach has been successfully applied by |ao et al. (2018) to the weak supervision problem. However, the model assumes one source domain, with supervised labels, to learn an attention generation module that is then applied to the target domain. In contrast, our method can be built on a given embedding space with mini- mum supervision.
Apart from incorporating prior knowledge in the form of rationales, one can augment neural net- works with: first-order logic (Li & Srikumar 2019: Hu et al. 2016a);; a corpora of regular pressions (Luo et al. 2018); or massive linguistic constraints (Hu et al., 2016b). While generally powerful and effective, all these methods require domain-specific expertise to define the additional features and constraints that are then explicitly incorporated into the network. In a different manner, the SoPA architecture of |Schwartz al. (2018) learns match surface patterns on text through differentiable version of finite state machines. A weighted combination of these patterns is used to classify a document. Instead, BabbleLabble (BL) (Hancock al. 2018) is method for generating weak classifiers from natural language explanations when supervision is scarce. These are then fed to Data Programming (DP) (Ratner et al.] (2166). a probabilistic framework. that outputs a final score. On one hand, BI, works well because it exploits domain-specific grammar to parse explanations; on the other hand, this grammar must be carefully designed by domain experts.
Finally, the Neural Bag Of Words (NBOW) model (Kalchbrenner et al., 2014) takes average of token embeddings and applics a logistic regression to classify a document. Its extension, NBOW2 (Sheikh et al., 2016), computes an importance score for each word by comparing it with single reference vector that is learned. Despite the underlying idea being similar, we propose a different mechanism to focus relevant words.
In the following, we describe the architecture whose only requirements are i) an embedding space of the input, and ii) additional rationales, though the latter is not strictly required. As we shall
2

---

Under review as a conference paper at ICLR 2020
discuss, the model has a strong inductive bias, which is effective when trained with very little data. Hereinafter, we refer to our new architecture with the name PARCUS, which stands for Pattern Representations Continuous Spaces.
THE PARCUS MODEL
Let consider classification task in which very small labelled dataset Dz ((11, r1, v1)..................... (IZ,, TL, v1)] is given, where 2; is an input sample, vy represents the rationale information and y: is the discrete target label. For the purpose of this paper, an input is a set of tokens 2; {- .......... of arbitrary size T, In addition, xi E Rd, where is the size of an embedding space obtained using pretrained model. Finally, we assume that each token in the input has been marked as relevant or not by annotators, i.e., = (r....... r E {0, 1)
Intuition When humans are asked to solve a text classification problem after seeing few examples, they tend to look for very simple patters across the dataset, such as specific words. Nevertheless, humans are also able to generalise to semantically similar concepts; our goal is to design a model that reflects this ability. For example, assume that the word "excellent" is important for classifying a movie review as positive. If we were to work in the character space, a straightforward solution would be match specific (sub-)strings in the input, an instance of the so-called pattern matching technique. Clearly, pattern matching cannot generalize to words that have similar meaning, e.g.. "outstanding"" In this work, we transfer the concept of pattern matching into the embedding space, where semantically similar words are assumed to have "close" representations. We achieve this via mechanism that outputs a probability of soft matching between token and "reference vector", which is learned together with the classification task to capture discriminative "concepts". Differ- ently from bag-of-word methods of Section [ this model easily accommodates multiple reference vectors, hence it can focus many different concepts that are critical for classification.
In order to guide the learning process using the given rationales, it seems sensible to magnify the er- ror for those words that have been marked relevant by annotators. Notwithstanding the simplicity of the idea, the underlying challenge is to effectively embed human knowledge into the reference vectors, which are responsible for the soft matching technique. In other words, the probabilities of soft matching should be highly correlated with the target class.
Finally, we require that classification should be done in such way that user explicitly under- stand which reference vectors are more important for positive or negative prediction. This last step obtained by learning linear combination of the soft matching probabilities.
The next sections describe the proposed model in depth. First, we show how to compute and com- bine soft token matching probabilities, and then we introduce the error boosting technique that in- corporates rationales in the training process. It is worth mentioning that both techniques have been designed to coexist, even though the latter not strictly necessary to train the model.
3.1 SOFT TOKEN MATCHING
We now present the core mechanism that implements soft token matching. Let us define set of parameters P = (pi,,...,PN] PN), Ps: E IRd called protolypes, where N is an hyper-parameter of the model. A prototype plays similar role as the reference vector in Sheikh et al. (2016). Going back to our movie review example, one E P should idcally adapt to be close to the embedding of the word "excellent".
To learn the N prototypes, we employ the cosine similarity metric. Cosine similarity can be seen as a way to measure semantic similarity; its co-domain ranges from -1, i.e., opposite in meaning, to 1, i.e., same meaning, with 0 indicating uncorrelation. Ideally, we would like our prototypes to have near-1 similarity with the relevant tokens in the input. To this aim, we further define a gate activation function g [-1, 1] + [0, 1] that takes the distance between a token x² and a prototype pk and outputs a probability of soft matching:
p(x soft-matches pæ) g(d(a), pk)) = (x/.m)) e [0
(I)
where a is an hhper--paammee parammer.. In practice, the closer to 1 the similarity is, the greater the output of this gated activation, and g(v) 1 e 1. By choosing high value of strongly penalize
3

---

Under review as a conference paper at ICLR 2020
*
x;
P1
0
Alice is married to Bob
Φ1
W - y
3
x
g d
Φ
A(x))
y
yi
y
yi
x
o
y1 = Σ >1rec)
Figure 1: The PARCUS architecture applied to the i-th example of dataset, i.e., "Alice married to Bob", with "married" being highlighted. After an initial phase where rationales are extracted and words are mapped to embeddings 24, we extract features by computing the similarity between the token's 's embedding and the parameters of our model. Then, combine these features with linear layer which outputs per-token predictions. At training time only, predictions are multiplied by a boosting factor fr??). Results are then summed to yield the sentence prediction y/.
tokens that are associated with low similarity scores. For completeness, Section A.1 depicts g(v) for different values of a. Such a technique is important because allows PARCUS to focus on N concepts that semantically different while fostering interpretability. Notice that this method differs from NBOW2 (Sheikh et al. (016), as we use prototypes to compute per-token features rather than importance scores.
3.2 COMBINING PROTOTYPES
Equation D computes the probability of soft matching between token and prototype. Likewise, because have N prototypes, treat probabilities as features associated with that token.
We represent these features as (h)()²) g(d(x) PP)) Yk: 1,..., N. Now that we have no- tion of multiple soft matching probabilities, we combine them via AND/OR logical functions. An approximation of such functions can be straightforwardly implemented through the pseudo- differentiable version of min and mar (Paszke al. 2017):
(AND(E?) = min (1))( vk()
= Ak})
In Scction A.2 we propose fully differentiable version the above equations, though min and significantly speed up convergence (due to the absence of non-linearities).
3.3 INFERENCE
Finally, we need to linearly combine all F features to output token prediction yن Let us define an auxiliary function (omitting the argument zi to make notation less cluttered):
A (xf) (01,... UN, BAND, Φon) [0, 1|
(4)
where square brackets denote concatenation. Then, the token prediction is computed as
y = A(x)) W +
(5)
where W E IR FxC is a matrix parameters (multi-class prediction with C' classes') and the (optional) bias. is worth noticing that these features are yet another strong inductive bias, and that the linear model especially necessary to interpret the model, as detailed Section 4.3 Figure combines all steps of the proposed architecture for token classification.
4

---

Under review as a conference paper at ICLR 2020
Finally, the input prediction is just sum of the individual y!
T
y = o
v
(6)
where o, from now on, represents the softmax activation.
3.4 RATIONALE DRIVEN ERROR BOOSTING
So far, have not made use of rationales, which are of fundamental importance guide the learning process. Intuitively, we would like the prototypes to softly match those tokens that are relevant for prediction. It follows that a boosting approach (Freund et al.] 1999) is not feasible in this scenario, because we want to weight the importance of tokens rather than of whole samples. Instead, the method we propose is simple and efficient, and it effectively exploits prior information. The idea is to boost the error associated with specific tokens' predictions to encourage the model to focus on them. To be more precise, at training time only we modify Equation 6 take into account the prior information follows
T. vi f(r!)).
y1= = ol 2
where ʃ [0, 1] + R is an arbitrary exponential function of our choice that boosts the error, e.g.. [() e"; we leave the extension an adaptive version of for future works. In terms of learning,
f(r) boosts the gradient of highlighted tokens while leaving unchanged the rest (i.e., if r is 0, our f(r)) outputs a multiplicative factor of 1).
Discussion Regularization of the matrix W plays an important role for effective learning. We use LI and L2 regularization terms on W, as done in (Zou & Hastie 2005), for two main purposes. First, the LI term enforces sparsity, which allows features. Secondly, L2 limits the magnitude the weights, hence avoiding over- cver-compensttion of low cosine similarity scores. Consequently, in order to increase features, the model encouraged to make changes to the prototypes rather than to the linear weights; in turn, this translates into particular prototype being "close" to relevant embeddings.
more easily interpret the importance of different
of the soft-matching probability
From a mathematical standpoint, we cannot achieve the same result as Equation 6 means of an additional loss term as done in Lei al. (2016), because gradients would be summed and not multi- plied. Moreover, in our experiments we choose to augment A (xx)) with additional information, such as the probability of "opposite" matching: x = g( ((-((r/, -d(([.p))) Ak E 1,..., N. Specifically, when é (x.) z 1 the token z! and pı have cosine similarity equal to -1, hence they are opposite meaning.
Implementation details PARCUS can be trained by full-gradient descent in an end-to-end fash- ion, from the prototypes to the linear weights. We rely on Pytorch (Paszke et al. (017) implement our model in few lines of code; this reflects the simplicity and strong inductive bias of our approach, which is necessary in the context we consider. The error boosting technique applied by the auto- matic differentiation packages once we implement Equation 7
We conclude with remarks on the model complexity. The total number of parameters O(Nd + FC), which is larger than those needed by simpler models such as logistic regression. Usually, restricted number of parameters serves to counteract overfitting, by limiting the hypotheses space of the model (Vapnik 1998). However, this work tackles the problem from different and novel per- spective, as we prevent the prototype weights from freely changing. Specifically, prototype weights vary in way that depends the given embedding space, because they tend to be close to some token representation. If this had not been the case, we could have simply used an MI.P in place of P. which does not perform well as PARCUS in our experiments.
4 EXPERIMENTS
This section reports the experimental setting as well as our experimental findings. We perform an in-depth analysis our model through ablation studies, in order to clearly separate the contribution
5

---

Under review as a conference paper at ICLR 2020
of prototypes from the error boosting technique. Then, explain what model can learn by direct inspection of its parameters. All code to reproduce experiments is publicly available"
4.1 EXPERIMENTAL SETTING
Datasets We empirically validate our method on three different datasets. First, MovieReview Zaidan et al. 2007) contains balanced positive and negative movie reviews with rationales. Sec- ondly, we use the highly imbalanced Spouse dataset from HHncock et al. (2018), where the task is to tell whether two entities in a given piece of news are married or not. This is much harder task than standard classification, as the same document can appear multiple times with different given entities and background context greatly varies. Finally, we use the Hatespeech Twitter dataset of Waseem & Hovy (2016), which contains short and noisy tweets that can belong to the hate-speeck or neutral class. Datasets' statistics are reported in the appendix for completeness. We manually provide rationales for 60 randomly chosen positive samples of both Spouse and Hatespeech (this process required approximately hour).
Setup The experimental evaluation was carried out by measuring performances on the given test set while varying the number of data points used for training. We used balanced train and validation splits for all models, and the validation set is taken as big as the training one to simulate a real sce- nario. As for Spouse, we used the given validation set fairly compare with the results Hanckkkk et al. 2018). We chose the pre-trained base version of BERT (Devlin et al. 2018) to provide the embedding space to our method and to other neural baselines as well.
We repeated each experiments 10 times with different random train/valla validation splits: however, dif- ferent models have been trained and validated on the same data splits, and we report the hyper- parameters table in the appendix. Moreover, to avoid bad initializations of the final re-training (for the selected configuration), we average test performances over runs. The optimized measure is Accuracy for MovieReview and Fl-score for Spouse and Hatespeech, as the former is perfectly bal- anced. We optimize the Cross Entropy loss using the Adam optimizer (Kingma & Ba 2015) for all the baselines we implement.
Methods To have a fair evaluation with respect to the same embedding space, we train a linear model (Linear) and single-layer MLP that work on token embeddings (MLP), as well as NBOW and NBOW2. Importantly, we also finetune BERT on Spouse and MovieReview. For Spouse, we propose a regular expression that associates specific sub-strings ( ("wife", "husb", "marr" and "knor") to the positive class; ideally, our model should be able to focus on such words while also generalizing. Traditional Supervision (TS) is a logistic regression trained on n n-gram features, whereas BL-DD stands for the BabbleL_ablle pipeline tested on 30 random explanations; results for TS and BL-DDM are taken from HHacock et al. (2088). BL.-DM can explicitly exploit the relational information of the Spouse dataset, hence is strong bascline.
Moreover, we report results of an SVM (Zaidan et al.) (007) and a log-linear model on language features (Zaidan & Eisner, 2008), both of which are specifically designed to exploit additional ratio- nales. On Hatespeech, we compare against a Logistic Regression model based on character n-grams (LR-ngrams), as it was shown to reach state of the art performance (Park & Fung 2017). Finally, we perform a number of ablation studies to isolate the contribution of different techniques: i) an MLP with the error boosting technique; ii) our method without highlights; ii) our method with no logical features; iii) our method with 6₁ features only.
4.2 RESULTS& & DISCUSSION
Table presents our empirical results for all three datasets, Results confirm that the choice of a strong inductive bias indeed benefits performances in very low data regime. On Spouse, our model strongly outperforms other neural baselines and reaches the manually tuned regular expression with just 60 data points (only 30 of them are positive). Moreover, TS needs <50x more data achieve similar performance, whereas 10 datapoints are sufficient to do better than almost all basclines with training size of 300, 30x improvement which does not depend on the chosen embedding space. We also found that TS performs much worse than our linear baseline (hence the need for a fair com- parison on the embedding space). With 300 datapoints, our model without highlights has an average
Link to the code reproduce the experiments omitted at review time

---

Under review as a conference paper at ICLR 2020
Table I: Results for all three datasets. Standard deviation shown in brackets. On Spouse and Hatespeech we report the F1 score, whereas we use accuracy to compare models on MovieReview.
SPOUSE
MODEL/TRAIN SIZE
10
30
60
150
300
3K
10K
TUNED REGEXP
40.48
LINEAR
18.24 (1.3)
20.56 (1.4)
22.46 (1.4)
26.08 (1.1)
26.12 (1.2)
MLP
17.91 (2.4)
20.24 (3.1)
18.34 (0.6)
23.26 (1.2)
24.08 (1.3)
NBOW
21.00 (2.3)
21.84 (1.7)
24.03 (1.0)
27.39 (2.0)
28.15 (1.8)
NBOW2
(2.6)
22.27 (1.9)
25.87 (1.4)
29.62 (1.5)
31.66 (2.1)
BERT+FINETUNIGG
(2.6)
20.19 (2.1)
23.41 (1.2)
32.11 (2.0)
35.45 (3.2)
(Abl.) MLP W. H.
16.71 (1.4)
20.81 (2.7)
20.92 (1.6)
22.72 (1.8)
23.12 (2.0)
(AbL. PARCUS WO H.
31.60 (2.2)
34.24 (2.3)
41.79 (2.1)
44.03 (1.2)
(AbL) PARCUS-Ø1
32.43 (4.5)
34.36 (4.2)
37.76 (2.7)
42.70 (1.0)
41.41 (2.4)
(AbL) PARCUS-NO-LOGIC
(3.40
34.52
36.76
42.66 (1.6)
41.96 (1.9)
TS
15.5
15.9
16.4
17.2
41.8
55.0
BL-DM (30 EXPL.)
46.5
PARCUS
34.03 (4.5)
36.64 (4.3)
40 29 (2.5)
43.67 (1.7)
42.85 (1.6)
MOVIEREVIEW
20
100
LINEAR
60.37 (3.4)
64.03 (3.5)
70.17 (2.0)
(2.6)
80,28 (3.1)
MLP
59.12 (4.1)
62.57 (4.2)
69.65 (2.4)
73.25 (3.8)
80.01 (3.0)
NBOW
62.56
65.7 (4.8)
73.88 (1.6)
77.99(2.0)
81.22 (3.6)
NBOW2
(4.45
64.30 (4.9)
72.88 (1.4)
(4.4)
83.63 (1.8)
BERT FINETUNING
53.52 (2.0)
54.83 (4.9)
59.72 (4.5)
67.73 (4.3)
79.20 (2.5)
(Abl.) MIP W. H.
61.45 (4.6)
63.05 (5.8)
68.87 (7.0)
72.38 (8.5)
74.63 (5.8)
(Abl.) PARCUS WO H.
61.2 (4.3)
64.85 (5.0)
74.25 (2.4)
78.55 (2.3)
84.55 (2.8)
(Abl) PARCUS-ÓE
66.05 (5.7)
(3.5)
77.8 (2.0)
80.7 (3.0)
83.40 (2.4)
(Abl) MARCUS-NO-LOGEE
66.90 (5.9)
67.9 (3.5)
75.5 (4.0)
80.95 (2.4)
83.65 (2.7)
SVM RATIONALES
65.4
75
83,2
LOG-LINEAR RATIONALES
65.8
76
83.75
PARCUS
67.2 (5.5)
70.05 (5.6)
76.55 (2.4)
79.95 (2.6)
83.75 (2.8)
HATESPEECH
10
100
200
LINEAR
37.81 (5.8)
40.8 (6.2)
47.35 (3.4)
51.49 (3.3)
51.86 (2.0)
MLP
42.09 (8.5)
42.94 (6.3)
50.77 (4.2)
51.03
54.62 (4,8)
NBOW
41.90 (6.1)
(5.70
49.07 (3.0)
52.52
56.17 (2.5)
NBOW2
40.20 (6.9)
41.63 (5.5)
47.96 (3.2)
51.52 (2.5)
53.48 (2.5)
AR-NGRAMS
41.48 (9.1)
48.00
49.86 (7.2)
56.83
62.75 (1.5)
PARCUS
43.06 (12.7)
45.25(9.8)
48.32 (6.6)
55.09 (3.0)
56.39 (4.7)
PARCUS (Fasttext)
40.97 (11.1)
46.43 (5.3)
52.46 (4.4)
(5.0)
59.07 (2.3)
F1 score very close to that of BL-DM, which relies domain-specific grammar and parser. In ad- dition, we note that the reported result (46.5) is not averaged over multiple runs; as matter of fact, of random splits achieves comes to experiments on few examples. Overall, we found that the proposed approach can be really helpful when data greatly imbalanced, and outperforms models like BERT that are deemed quite performing when fine-tuned on relatively small datasets (Devlin et al.) 2018 Howard & Ruder 2018). Similar arguments apply to MovieReview, where our model strongly improves over the base- lines. Interestingly, our simple representation learning approach is able to beat the state of the art by a large margin when few data points are available. Here, NBOW and NBOW2 models proved to be strong baselines, as the mean representation of document seems to work well. Genlraenyryl speaking, the gap between performances is more evident as training size is very scarce, even when compared to other baselines that rationales.
of 46.3, indicating the need for robust evaluation when
In light of these results, we performed ablation studies on both datasets to understand the improve- ments are only due prototypes, rationales both. Overall, observe that the strong inductive

---

Under review as a conference paper at ICLR 2020
0.8
Average Posilive Score
0.6
0.0
marrinng
will
husband
fianee
mamied
spouss
fenone
rematried
boyfheedd
martagoo
Word
Figure 2: Top-10 most relevant tokens for positive prediction, averaged on unseen data.
bias represented by prototypes provides consistent improvement with respect to the other models, which is especially evident on the Spouse dataset. Interestingly, the MLP does not benefit from error boosting, which might be explained by the fact that its larger hypotheses space, i.e., unconstrained weights, makes it difficult to diminish the contributions of non-relevant tokens. Because rationales guide the learning process, their are more important in the extremely low resource scenario, but their effect slowly fades as the training size increases; contrarily to our expectations, PARCUS performed even better on larger amounts of training points without rationales.
On Hatespeech PARCUS still performs fairly well on average, but with 200 traning examples it cannot keep up with the logistic regressor of Park Fung (2017). We found two reasons for such behavior: (i) BERT's tokenizer is unable to accurately splitting tweets, due to their noisy nature, and (ii) character n-grams are strong discriminative features for this task (Nobata et al. (0166). and BERT does not use them. To simultaneously solve both issues switched to Fasttext embeddings [Bojanowski et al. (017)) which are also trained on character n-grams and do not need an additional tokenizer. Surprisingly, we observed a significant improvement the gap in performance with the n-grams based model. Therefore, we conclude that choosing the "right" embedding space can make the difference, which does necessarily mean using one of the latest and most powerful language models available.
200 data points, which reduces
4.3 PROVIDING EXPLANATIONS
In this Section show that PARCUS is interpretable. To this aim, train model using N=3 prototypes on 60 examples taken from the Spouse dataset. Then we run the model on unseen data and we inspect the outputs associated with each token. We then rank them important ones, and we observe that the tokens with highest rank correspond to semantic concepts that are relevant for the task. Indeed, the model learned to focus words related to marriage, as well as syntactic variations associated with similar semantics. Moreover, some of the words were not given as part of rationales in the training set. The next step effectively been incorporated in the prototypes, and how the features of Eq. 4 have been weighted. We started by inspecting the magnitude of the linear weights W E R8x2; specifically, if the i-th feature is discriminative for a class c, then the i-th row of W will have the e-th element clearly larger than the others. In our example, we found that Φ1 was important for positive predictions, whereas the other features did not contribute cosine similarity ranking between tokens and the prototype Pl. From the most similar to the least one, obtained: husband; marriage; marrying; wife; married: marry; fiance; wedding; fiancee; and girlfriend. Interestingly, PARCUS has automatically learned to match concepts similar to those provided in natural language form by BabbleI abble Hancock al. (2018).
what the most
show that rationales have
particular class. Therefore, we performed top- 10
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS
We presented a new methodology to perform classification in the low data regime. We coupled soft token matching with error boosting focus on concepts that are important for the task at hand. The model is able to outperform other competitors, including fine-tuned complex models. Moreover, we showed with practical example how humans can interpret the predictions in terms of concepts matching. In conclusion, our model proved to be useful in tasks where gathering data challenging.
REFERENCES
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1903-1913, 2018.
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. Sequence classi- fication with human attention. Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 302-312, 2018.
Joost Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL, 2019.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Carlos Castillo, Debora Donato, Aristides Gionis, Vanessa Murdock, and Fabrizio Silvestri. Know your neighbors: Web spam detection using the web topology. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development information retrieval, pp. 423-430. ACM, 2007.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3)2773-277 1995.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018.
Gregory Druck, Gideon Mann, and Andrew McCallum. Reducing annotation effort using gener- alized expectation criteria. Technical report, Massacusetts University, Department of Computer Science, Amherst, 2007.
Yoav Freund, Robert Schapire, and Naoki Abe. short introduction to boosting. Journal- Japanese Society For Artificial Intelligence, 14(771-780):1612, 1999.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5))93, 2019.
Braden Hancock, Paroma Varma, Stephanie Wang. Martin Bringmann, Percy Liang, and Christopher Ré. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 1884-1895, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers). pp. 328-339, 2018.

---

Under review as a conference paper at ICLR 2020
Zhiting Hu, Xuezhe Ma. Zhengzhong Liu, Eduard Hovy, and Eric Xing. Hamessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 2410 2410-2420, 2016a.
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1670-1679, 2016b.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Compu- fational Linguistics (Volume 1: Long Papers), pp. 655-665, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICIR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, 2016.
Tao Li and Vivek Srikumar. Augmenting neural networks with first-order logic. arXiv preprint arXiv: 2019.
Bingfeng Luo, Yansong Feng, Zheng Wang, Songfang Huang, Rui Yan, and Dongyan Zhao. Marry- ing up regular expressions with neural networks: case study for spoken language understand- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2083,2093,, 2018.
Andrew McCallum, Gideon Mann, and Gregory Druck. Generalized expectation criteria. Computer science technical note, University of Massachuseus, Amherst, MA, 94(95):159, 2007.
Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. Neural tharacter-based composition models for abuse detection. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pp. 1-10, 2018.
Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mchdad, and Yi Chang. Abusive language detection in online user content. In Proceedings of the 25th international conference world wide web, pp. 145-153. International World Wide Web Conferences Steering Committee, 2016.
Ho Park and Pascale Fung. One-step and two-step classification for abusive language detection on twitter. In Proceedings the First Workshop Abusive Language Online, pp. 41-45, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in neural information processing systems, pp. 3567-3575, 2016.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(1 1):2673-2681. 1997.
Roy Schwartz, Sam Thomson, and Noah A. Smith. Bridging cnns, mns, and weighted finite-state machines. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL., 2018
Imran Sheikh, Irina Illina, Dominique Fohr, and Georges Linares. Learning word importance with the neural bag-of-words model. In ACL, Representation Learning NLP (RepL4NLP) workshop. 2016.
NLP
Vlamimir Vapnik. Statistical learning theory wiley. New York, pp. 156-160 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
10

---

Under review as a conference paper at ICLR 2020
Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pp. 88-93, 2016.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using annotator rationales improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguisties; proceedings of the main conference, pp. 260-267, 2007.
Omar F Zaidan and Jason Eisner. Modeling annotators: A generative approach to learning from an- notator rationales. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 31-40. Association for Computational Linguistics, 2008.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2)3001-320, 2005.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 GATING ACTIVATION FUNCTION
The gating activation function of Eq. controls how similar an embedding and prototype have to be in order to significantly contribute as feature. Figure shows different curves for different values of a. bigger a squashes low cosine similarities more, as the case for a 100, therefore acting stricter filter. For completeness, Figure 5 shows the effect of g(-) in R², using 100.
shows
1.0
e-20
a-50
0.8
a=100
0.6
0.2
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Figure 3: Plot of the gating function g(v) a=-1 for different values of a. In practice, the argument v will be the cosine similarity between vectors, hence [[1, 1] + [0,
0.00
0.58
0.88
0.44
0.66
0.00
0.44
-344
0.22
-088
-2
0.00
Cosine similarity
(b) Gated cosine similarity
Figure 4: From left to right: (a) Cosine similarity with respect to the [1.1] vector v and (b) its gated activation for 100. The gate function acts filter on cosine similarity, thus promoting those vectors "close enough" to while penalizing the others.
12

---

Under review as a conference paper at ICLR 2020
A.2
DIFFERENTIABLE IMPLEMENTATION OF AND AND OR
10
1.0
08
0.6
04
0.6
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.0
0.8
1.0
0.6
0.8
1.0
(a) Differentiable AND function
(b) Differentiable OR function (s 2)
1.0
0.8
0.4
0.0
1.0
0.0
0.2
0.4
0.4
0.6
0.2
0.0
(c) Differentiable XOR function (s
Figure 5: From left to right: Cosine similarity with respect the [1,1] vector and its gated activation.
Given the set ΦN) E [o, IId. interpreted as independent "soft matching probabilities", we would like compute their joint probability well as the probability of single matching out of all possible ones. In our experiments, we found a pseudo-difierentabble implementation of min and mar (Eq. and Eq. 3 speed up convergence. Alternatively, we tried fully differentiable version of those two functions as well as the probability of mutually exclusive events, i.c., XOR. We present the equations for N 2, although this can be easily generalized to arbitrary N:
Diff (2D,
AND
Φ1 * Φ2
Diff (2,8) = (0) 2) 2)-22 22-20(1 -11) (0h - 22)-2* 2- 2-22(1 -b2) Diff
( (2)
(xon 02)
Diff Pon (1h, 02)
AND (01,
Diff
where controls how squashed the curve is. Figure depicts all three curves for
13

---

Under review as a conference paper at ICLR 2020
A.3 DATASETS' STATISTICS
In T we report the statistics of the datasets we used. When preprocessing the data with BERT, used maximum sentence length of 128 for token-based methods, and 512 when finetuning BERT.
Table 2: Datasets' statistics.
Train Size
Validation Size
Test Size
% Pos.
No. Rationales
Spouse 22195
2796
2697
8%
60 (Positive)
MovieReview
1800
200
50%
1800
Hatespeech 3430
3430
14.7%
60 (Positive)
As mentioned in Section 4 each sample of the Spouse dataset contains pair of entities as well as the sentence. Therefore, the MLP baseline, ablation studies and our method will make use of an input-mask (at test time only), which reflects the fact that we are not interested in those sentences where both our entities of interest do not appear. is worth noticing that such relational information should be naturally exploited by methods like Babblel which rely on domain-specific gram- mars. While this method should help to improve precision, practice it did not significantly affect performances.
A.4 HYPER PARAMETERS
Jyper-parameter are used to perform model selection, which returns the best configuration for given validation split. We use such configuration to train model the whole training set and then evaluate its generalization performances on the unseen test set. When Data Programming is used, we simply combine model selection of model with that of data programming. Since the experiment is repeated 10 times to avoid lucky/unlucky data splits, model selection is performed 10 times as well. Model assessment, i.e., the measure of performance of our family of models, is evaluated by averaging the 10 different performances the test set. When fine-tuning BERT, mainly follow guidelines reported in Devlin al. (2018).
Table 3: prrper-praaeters tried for model selection.
LINEAR (0.01, 0.001, 0.000T)
MLP/NBOW(2) {0.001, 0.0001}
BERT+FINETUS (0.00002, 0.00003, 0.00005}
DP range(0.001, 0.0001)
OURS
LEARNING RATE
(0.01}
LI
{0.01, 0.001}
L2
(0.1. 0.01, 0.0001) (50, 100, 150)
(0.01, 0.0001) (100, 500)
(0.001. 0.0001) (500)
EPOCHS
(2, 4, 10)
(100,500)
HIDDEN UNITS
{8, 16, 32) 32
BATCH SIZE
32
8
32 {5, 10) [c",5", 10") (10,100)
N
f(r)
MAX SEARCH
14

---

Under review as a conference paper at ICLR 2020
A.5 GRADIENT BOOSTING EFFECT BACKPROPAGATION
For loss & defined on top of Equation and true label D. backpropagation is computed as (ab- stracting from the sample index to simplify notation):
0 I f((()) 0 Σ3 E, s((r))
aely.0) 0)
ae 0)
dy
() By
00
Oy
00
ii y f(((r)))
00
0 3 y f(1)
0 ((A
+ +b) 1(1)))
Σ
00
00
where A are the parameters of the model.
A.6 ROBUSTNESS TO NOISE
45
40
Score
35
30
25
10
20
30
40
50
60
70
80
90
Noise (%)
Figure 6: Robustness to different degrees of rationale noise on the Spouse dataset. From bottom to top, the curves refer to different training sizes: 10, 30, 60, 150, 300. We fix f(r) =
We investigated how injecting random rationales (i.e., ones) in prior information affects the final performances. Clearly, having a 100% noise corresponds to not having rationales all, therefore we rn simple experiment (with fixed hyper-parameters) 10 times on Spouse, with the goal to study robustness noise. Figure shows the result, where different curves stand for different training sizes; influence whereas for 150 (orange curve) and 300 (green curve) data points the effect is negligible. This confirms that there is trade-off where the impact of rationales has little effect the final performances.
would expect, for few data points (blue and red curves) noise has bigger
15

---

Under review as a conference paper at ICLR 2020
A.7 VISUALIZATION OF THE MOST IMPORTANT TOKENS
The importance of tokens in a sentence can be inspected by looking at the prototypes pt, the weight matrix W and the tokens predictions y!- In Figure 7 we show the some of the tokens which are responsible for positive prediction on Spouse unseen data, which is consistent with the results of Section 4.3
alex perkins who is currendly in a relationship with presenter anna richardson, was in a relationship with a man for six years after meeting him school during a theatre production alex told telegraph she long - term relationship with man she only calls' who she met while they both starred in a production of bi lize det carmen in school
fast moving couple: more #dna ba #icca #drin 's estranged husband austin chris pictured back in april 2014 has made some bold accusations in their bitter custody fend over their one - year- old son regurding her gotham co star alex at the centre gotham co- star alex pictured in new york thursday night, is mentioned in court do #uumen litss by chris he claims that be arrived home gather his belongings and found the 37 year old actor had just laken shower and he was playing with my son.".
alex and maxi line av Mis e #ewart divorced in 1976 six years after allegedly had an affuir with chris, the second wille of legendary late- night talk show host johany chris.
alex wondered if kidnapping of the wife of the managing director sun chris, in lagos last week the fault of mimi ##ko
alex. who is chris 's second child with husband life alex and was sigued im 48g modelss july ware simple quarter sleeve dress as she sat front row at the cat #éwalk show
alex the sister of mani d8isha alex who has been found guilty of the murder of pu #är stivi jochi, her boyfriend chris new fiance and is awaiting sentencing.
48ma 8ived. s. district judge alex ordered bog 6-ooma #8lova ,2, taken into custody immediately and her husband, chris 62. returned to prison. where he has been since his arrest.
the night before. kyyic and her boyfriend alex attended alexander wang 's 10 year anniversary show featuring a star stud #sdded mudience with the likes of chris and lady guga.
and ##warr daughter victoria married chris son michacl
alex rarely seen with his famous wife and she often jokes about the lengths he goes to avoid the spotlight, but delly 's hushand has agreed to part of her anniversary party may alex, 73, who owns and operates a road repair company in tennessee, will the stage with chris 69, when she performs at their willow lake plantation home.
close: nicole adopted alex and connor with busband chris 1992 and 1995 respectively prior their separation and subsequent divorce in 2001 pictured together in 2004) ) she added:
pierce is also father to sons alex and christopher from his first marriage to chris, , while their daughter charlotte tragic #tally died in 2013 of #dvar ésion cuncer,
disense that killed her mother, aged just
couple nicole married chris years.
's chris their two kids to be him.
Figure 7: Visualization of the most important tokens for positive prediction. Sentences were tracted from the Spouse dataset. The entities we are interested in are always called "alex" and "chris".
16


Reviews:
[{"review_id": "rylTafBXqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of text classification, especially the settings in which the number of labeled sentences is very small. However, authors assume, annotations of rationales behind the label, i.e. highlighting tokens in a sentence which are important in deciding its label. As per my understanding, this is a big limitation. Second, the proposed model makes inference of class labels just based upon occurrence of words in a sentence, rather than making more sophisticated inferences relying upon sub-sequence patterns at least. \n\nThe idea proposed in the paper is to learn prototype vectors which have high similarity w.r.t. tokens in sentences, especially the highlighted one. I didn't understand the justification for learning such prototypes in the first place.\n\nThis works build upon a workshop paper. \n\nThe idea proposed in the paper, even in the specific problem context considered, are incremental. I don't think that this kind of work aligns with the theme of learning representations. This paper may be suitable for publication in an NLP workshop as the baseline model.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gbbP6TKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After responses:\n\nI read the authors response and decided to stick to my original score mostly because:\n\n1 - I understand that interpretability is hard to define. I also agree with the authors response. However, this is still not reflected in the paper in any way. I expect a discussion on what is the relevant definition used in the paper and how does it fit to that definition. Currently, it is very confusing to the reader.\n\n2 - I understand the authors' response that few-shot learning is a different empirical setting. However, authors also agree that settings are some-what relevant. I really do not see any gain by NOT discussing the few-shot learning literature. At the end, a reader is interested in this work if they have limited data. Moreover, other ways to address limited data issue should be discussed.\n\n-----\nThe manuscript is proposing a few-shot classification setting in which training set includes only few examples. The main contribution is using prototype embeddings and representing each word as cosine distances to these prototype embeddings. Moreover, the final classification is weighted summation of the per-token decisions followed by a soft-max. Per-token classifiers are obtained with an MLP using the cosine distances as features. When the relevance labels are available, they are used in training to boost gradients.\n\nPRO(s)\nThe proposed method is interesting and addressing an important problem. There are many few-shot scenarios and finding good models for them is impactful.\n\nThe results are promising and the proposed method is more interpretable than the existing NLP classifiers. I disagree with the claim that the model is interpretable. However, I appreciate the effort to interpret the model.\n\nCON(s)\nThe model is not interpretable because 1) it starts with embeddings and they are not interpretable, 2) model is full of non-linearities and decision boundaries are not possible to find. In other words, it is not possible to answer \"what would make this model predict some other classifier\".\n\nThe authors should discuss the existing few-shot learning mechanisms. Especially, \"Prototypical Networks for Few-shot Learning\" is very relevant. I also think it can be included as a baseline with very minimal modifications.\n\nThe writing is not complete. The authors do not even discuss how the prototypes are learned. I am assuming it is done using full gradient-descent over all parameters. However, this is not clearly discussed. Implementation details should be discussed more clearly.\n\nSUMMARY\nI believe the manuscript is definitely interesting and has a potential. In the mean time, It is not ready for publication. It needs a through review of few-shot learning. Authors should also discuss can any of the few-shot learning methods be included in the experimental study. If the answer is yes, it should be. If the answer is no, it should be explained clearly. \n\nAlthough my recommendation is weak-reject, I am happy to bump it up if these issues are addressed.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlsCX2CFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose PARCUS (\"Pattern Representations on Continuous Spaces\"), a model which computes a soft-matching probability for all words in an input sequence with so-called prototypes in order to predict a label for the input. Furthermore, for training, PARCUS makes use of rationales. Those are indicators of input importance, and help to boost the loss for relevant tokens.\n\nThe main motivation to use PARCUS is that it works better in a low-resource setting than recent state-of-the-art models for the high-resource case. This is due to it having relatively few parameters and to it having a strong inductive bias. However, the fact that models with less parameters perform better than BERT-based models in the low-resource case is not very surprising. Looking at the experiments, the results on HATESPEECH show less differences between models than for SPOUSE or MOVIEREVIEW.\n\nAnother selling point of PARCUS is that it's interpretable. While neural networks can also be analyzed in different ways, I agree with the authors that this is nice to have.\n\nOverall, the paper seems solid.\n\n==========\n\nUpdate: After reading the other reviews and the responses by the authors, I lowered my score from 6 to 3.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors focus on low-resource text classifications tasks augmented with "rationales". They propose a new technique that improves performance over existing approaches and that allows human inspection of the learned weights.

Although the reviewers did not find any major faults with the paper, they were in consensus that the paper should be rejected at this time. Generally, the reviewers' reservations were in terms of novelty and extent of technical contribution.

Given the large number of submissions this year, I am recommending rejection for this paper.

Decision: Reject