Soft Token Matching for Interpretable Low-Resource Classification

1 INTRODUCTION . Gathering and labeling data is a task that can be expensive in terms of time , human effort and resources . When we can not rely on already available datasets , training a model with acceptable performance on few data points annotated by few annotators , becomes critical in many practical applications . This is , indeed , especially important when the data is naturally imbalanced and the demands of gathering samples of the minority class are high . One important domain in which these issues arise is text classification , for example hate-speech ( Waseem & Hovy , 2016 ) , web spam ( Castillo et al. , 2007 ) and abuse detection ( Mishra et al. , 2018 ) . One effective approach to overcome the lack of training data is that of Zaidan et al . ( 2007 ) , which consists of augmenting the few data available with rationales , i.e. , highlighted portions of the input . Rationales are usually coupled with feature-engineering to be effective in low resource scenarios . An alternative way to deal with data sparsity , especially in text classification tasks , is to use pretrained language models ( LMs ) that are fine-tuned on a target domain . While this approach has been tested on hundreds of training points ( Devlin et al. , 2018 ; Howard & Ruder , 2018 ) , it is not clear how it behaves in an even scarcer setting , as the vast parameter space of an LM might pose a problem . Moreover , fine-tuning a model may require a considerable amount of computing power , therefore restricting its applicability . On the other hand , some embedding-based models represent the input as a weighted average of words ( Kalchbrenner et al. , 2014 ; Sheikh et al. , 2016 ) , where the weight is given by a parameter called “ reference vector ” . However , these models can not easily incorporate multiple reference vectors , and they are not interpretable since classification works on unreadable embedding features . In this paper , we propose a novel and efficient representation learning model to address the above issues . The underlying idea is to focus on relevant words in the input while being able to generalise to semantically similar concepts ; this is something akin to what a human would do in the presence of data scarcity . We therefore introduce two techniques that should coexist to reflect our intuition . First of all , the model to focuses on specific words by computing soft matching probabilities between each word and multiple vectors which represent semantic concepts . Secondly , we guide the learning process to learn important concepts thanks to an error boosting technique that exploits rationales . Basically , it encourages the model to reduce the overall error by improving the prediction associated with highlighted words . Additionally , by direct inspection of model weights it is possible to understand what words it focuses on ; in short , the model is interpretable1 . Results across a consistent number of baselines and three 1See ( Guidotti et al. , 2019 ) for the notion of interpretability we use throughout the paper . datasets also indicate a significant improvement in performance . Interestingly , we always outperform fine-tuned models when little training data is available . Our model can also assist users to train a classifier for a very specific task . As an example , consider training an abstract filtering system with rationales provided by the user itself . The model will then learn to filter out papers that are not matching the user ’ s preferences . The rest of the paper is structured as follows : Section 2 provides an overview of the existing literature , highlighting similar and different ideas ; Section 3 formally introduces the problem as well as our model , providing intuition behind our architectural choices ; Section 4 details our experiments and shows our findings , with a thorough ablation study that disentangles the contribution of each part of the model and a use case on interpretability ; finally , Section 5 summarizes our work . 2 RELATED WORKS . There are different ways in which rationales can be used . Some works generate rationales , while others exploit them to inform the learning process . The method proposed by Lei et al . ( 2016 ) tackles text classification by learning a distribution of rationales given the text and a distribution of the target class given the rationales . Interestingly , an additional regularization term is added to the loss to produce rationales that are short and coherent . The model makes use of high-capacity recurrent neural networks ( Schuster & Paliwal , 1997 ) , thus it is tested on large amounts of training data to prevent overfitting . This work was later refined by Bastings et al . ( 2019 ) , who proposed a probabilistic version of a similar architecture , where a latent model is responsible for the generation of discrete rationales . The main advantage of predicting discrete rationales is that it is possible to constrain their maximum number per sample , thus effectively controlling sparsity . However , it usually requires a large amount of data points to be effective . The first to exploit rationales in a low resource scenario were Zaidan et al . ( 2007 ) and Zaidan & Eisner ( 2008 ) , by means of a rationale-constrained SVM ( Cortes & Vapnik , 1995 ) and a probabilistic model . Moreover , the latter is realized as a log-linear classifier with heavy use of featureengineering . On the other hand , when rationales are defined on features rather than on samples , one can use the Generalized Expectation ( GE ) criteria ( Druck et al. , 2007 ; McCallum et al. , 2007 ) to significantly improve the performance of classifiers . Rationales can also be incorporated in the loss function ( Barrett et al. , 2018 ) , where the attention module ( Vaswani et al. , 2017 ) on top of an LSTM ( Hochreiter & Schmidhuber , 1997 ) is forced to attend relevant tokens in a document . This method was not tested on small datasets , possibly because of the aforementioned issues of high capacity models . A similar approach has been successfully applied by Bao et al . ( 2018 ) to the weak supervision problem . However , the model assumes one source domain , with supervised labels , to learn an attention generation module that is then applied to the target domain . In contrast , our method can be built on a given embedding space with minimum supervision . Apart from incorporating prior knowledge in the form of rationales , one can augment neural networks with : first-order logic ( Li & Srikumar , 2019 ; Hu et al. , 2016a ) ; a corpora of regular expressions ( Luo et al. , 2018 ) ; or massive linguistic constraints ( Hu et al. , 2016b ) . While generally powerful and effective , all these methods require domain-specific expertise to define the additional features and constraints that are then explicitly incorporated into the network . In a different manner , the SoPA architecture of Schwartz et al . ( 2018 ) learns to match surface patterns on text through a differentiable version of finite state machines . A weighted combination of these patterns is used to classify a document . Instead , BabbleLabble ( BL ) ( Hancock et al. , 2018 ) is a method for generating weak classifiers from natural language explanations when supervision is scarce . These are then fed to Data Programming ( DP ) ( Ratner et al. , 2016 ) , a probabilistic framework , that outputs a final score . On one hand , BL works well because it exploits a domain-specific grammar to parse explanations ; on the other hand , this grammar must be carefully designed by domain experts . Finally , the Neural Bag Of Words ( NBOW ) model ( Kalchbrenner et al. , 2014 ) takes an average of token embeddings and applies a logistic regression to classify a document . Its extension , NBOW2 ( Sheikh et al. , 2016 ) , computes an importance score for each word by comparing it with a single reference vector that is learned . Despite the underlying idea being similar , we propose a different mechanism to focus on relevant words . In the following , we describe the architecture whose only requirements are i ) an embedding space of the input , and ii ) additional rationales , though the latter is not strictly required . As we are shall discuss , the model has a strong inductive bias , which is effective when trained with very little data . Hereinafter , we refer to our new architecture with the name PARCUS , which stands for Pattern Representations on Continuous Spaces . 3 THE PARCUS MODEL . Let us consider a classification task in which a very small labelled dataset DL = { ( x1 , r1 , y1 ) , . . . , ( xL , rL , yL ) } is given , where xi is an input sample , ri represents the rationale information and yi is the discrete target label . For the purpose of this paper , an input is a set of tokens xi = { x1i , . . . , x Ti i } of arbitrary size Ti . In addition , x j i ∈ Rd , where d is the size of an embedding space obtained using a pretrained model . Finally , we assume that each token in the input has been marked as relevant or not by annotators , i.e. , ri = { r1i , . . . , r Ti i } ∈ { 0 , 1 } Ti . Intuition When humans are asked to solve a text classification problem after seeing few examples , they tend to look for very simple patterns across the dataset , such as specific words . Nevertheless , humans are also able to generalise to semantically similar concepts ; our goal is to design a model that reflects this ability . For example , assume that the word “ excellent ” is important for classifying a movie review as positive . If we were to work in the character space , a straightforward solution would be to match specific ( sub- ) strings in the input , an instance of the so-called pattern matching technique . Clearly , pattern matching can not generalize to words that have similar meaning , e.g. , “ outstanding ” . In this work , we transfer the concept of pattern matching into the embedding space , where semantically similar words are assumed to have “ close ” representations . We achieve this via a mechanism that outputs a probability of soft matching between a token and a “ reference vector ” , which is learned together with the classification task to capture discriminative “ concepts ” . Differently from bag-of-word methods of Section 2 , this model easily accommodates multiple reference vectors , hence it can focus on many different concepts that are critical for classification . In order to guide the learning process using the given rationales , it seems sensible to magnify the error for those words that have been marked as relevant by annotators . Notwithstanding the simplicity of the idea , the underlying challenge is to effectively embed human knowledge into the reference vectors , which are responsible for the soft matching technique . In other words , the probabilities of soft matching should be highly correlated with the target class . Finally , we require that classification should be done in such a way that a user can explicitly understand which reference vectors are more important for positive or negative prediction . This last step is obtained by learning a linear combination of the soft matching probabilities . The next sections describe the proposed model in depth . First , we show how to compute and combine soft token matching probabilities , and then we introduce the error boosting technique that incorporates rationales in the training process . It is worth mentioning that both techniques have been designed to coexist , even though the latter is not strictly necessary to train the model .

Summary: The authors propose PARCUS ("Pattern Representations on Continuous Spaces"), a model which computes a soft-matching probability for all words in an input sequence with so-called prototypes in order to predict a label for the input. Furthermore, for training, PARCUS makes use of rationales. Those are indicators of input importance, and help to boost the loss for relevant tokens.