Soft Token Matching for Interpretable Low-Resource Classification

1 INTRODUCTION . Gathering and labeling data is a task that can be expensive in terms of time , human effort and resources . When we can not rely on already available datasets , training a model with acceptable performance on few data points annotated by few annotators , becomes critical in many practical applications . This is , indeed , especially important when the data is naturally imbalanced and the demands of gathering samples of the minority class are high . One important domain in which these issues arise is text classification , for example hate-speech ( Waseem & Hovy , 2016 ) , web spam ( Castillo et al. , 2007 ) and abuse detection ( Mishra et al. , 2018 ) . One effective approach to overcome the lack of training data is that of Zaidan et al . ( 2007 ) , which consists of augmenting the few data available with rationales , i.e. , highlighted portions of the input . Rationales are usually coupled with feature-engineering to be effective in low resource scenarios . An alternative way to deal with data sparsity , especially in text classification tasks , is to use pretrained language models ( LMs ) that are fine-tuned on a target domain . While this approach has been tested on hundreds of training points ( Devlin et al. , 2018 ; Howard & Ruder , 2018 ) , it is not clear how it behaves in an even scarcer setting , as the vast parameter space of an LM might pose a problem . Moreover , fine-tuning a model may require a considerable amount of computing power , therefore restricting its applicability . On the other hand , some embedding-based models represent the input as a weighted average of words ( Kalchbrenner et al. , 2014 ; Sheikh et al. , 2016 ) , where the weight is given by a parameter called “ reference vector ” . However , these models can not easily incorporate multiple reference vectors , and they are not interpretable since classification works on unreadable embedding features . In this paper , we propose a novel and efficient representation learning model to address the above issues . The underlying idea is to focus on relevant words in the input while being able to generalise to semantically similar concepts ; this is something akin to what a human would do in the presence of data scarcity . We therefore introduce two techniques that should coexist to reflect our intuition . First of all , the model to focuses on specific words by computing soft matching probabilities between each word and multiple vectors which represent semantic concepts . Secondly , we guide the learning process to learn important concepts thanks to an error boosting technique that exploits rationales . Basically , it encourages the model to reduce the overall error by improving the prediction associated with highlighted words . Additionally , by direct inspection of model weights it is possible to understand what words it focuses on ; in short , the model is interpretable1 . Results across a consistent number of baselines and three 1See ( Guidotti et al. , 2019 ) for the notion of interpretability we use throughout the paper . datasets also indicate a significant improvement in performance . Interestingly , we always outperform fine-tuned models when little training data is available . Our model can also assist users to train a classifier for a very specific task . As an example , consider training an abstract filtering system with rationales provided by the user itself . The model will then learn to filter out papers that are not matching the user ’ s preferences . The rest of the paper is structured as follows : Section 2 provides an overview of the existing literature , highlighting similar and different ideas ; Section 3 formally introduces the problem as well as our model , providing intuition behind our architectural choices ; Section 4 details our experiments and shows our findings , with a thorough ablation study that disentangles the contribution of each part of the model and a use case on interpretability ; finally , Section 5 summarizes our work . 2 RELATED WORKS . There are different ways in which rationales can be used . Some works generate rationales , while others exploit them to inform the learning process . The method proposed by Lei et al . ( 2016 ) tackles text classification by learning a distribution of rationales given the text and a distribution of the target class given the rationales . Interestingly , an additional regularization term is added to the loss to produce rationales that are short and coherent . The model makes use of high-capacity recurrent neural networks ( Schuster & Paliwal , 1997 ) , thus it is tested on large amounts of training data to prevent overfitting . This work was later refined by Bastings et al . ( 2019 ) , who proposed a probabilistic version of a similar architecture , where a latent model is responsible for the generation of discrete rationales . The main advantage of predicting discrete rationales is that it is possible to constrain their maximum number per sample , thus effectively controlling sparsity . However , it usually requires a large amount of data points to be effective . The first to exploit rationales in a low resource scenario were Zaidan et al . ( 2007 ) and Zaidan & Eisner ( 2008 ) , by means of a rationale-constrained SVM ( Cortes & Vapnik , 1995 ) and a probabilistic model . Moreover , the latter is realized as a log-linear classifier with heavy use of featureengineering . On the other hand , when rationales are defined on features rather than on samples , one can use the Generalized Expectation ( GE ) criteria ( Druck et al. , 2007 ; McCallum et al. , 2007 ) to significantly improve the performance of classifiers . Rationales can also be incorporated in the loss function ( Barrett et al. , 2018 ) , where the attention module ( Vaswani et al. , 2017 ) on top of an LSTM ( Hochreiter & Schmidhuber , 1997 ) is forced to attend relevant tokens in a document . This method was not tested on small datasets , possibly because of the aforementioned issues of high capacity models . A similar approach has been successfully applied by Bao et al . ( 2018 ) to the weak supervision problem . However , the model assumes one source domain , with supervised labels , to learn an attention generation module that is then applied to the target domain . In contrast , our method can be built on a given embedding space with minimum supervision . Apart from incorporating prior knowledge in the form of rationales , one can augment neural networks with : first-order logic ( Li & Srikumar , 2019 ; Hu et al. , 2016a ) ; a corpora of regular expressions ( Luo et al. , 2018 ) ; or massive linguistic constraints ( Hu et al. , 2016b ) . While generally powerful and effective , all these methods require domain-specific expertise to define the additional features and constraints that are then explicitly incorporated into the network . In a different manner , the SoPA architecture of Schwartz et al . ( 2018 ) learns to match surface patterns on text through a differentiable version of finite state machines . A weighted combination of these patterns is used to classify a document . Instead , BabbleLabble ( BL ) ( Hancock et al. , 2018 ) is a method for generating weak classifiers from natural language explanations when supervision is scarce . These are then fed to Data Programming ( DP ) ( Ratner et al. , 2016 ) , a probabilistic framework , that outputs a final score . On one hand , BL works well because it exploits a domain-specific grammar to parse explanations ; on the other hand , this grammar must be carefully designed by domain experts . Finally , the Neural Bag Of Words ( NBOW ) model ( Kalchbrenner et al. , 2014 ) takes an average of token embeddings and applies a logistic regression to classify a document . Its extension , NBOW2 ( Sheikh et al. , 2016 ) , computes an importance score for each word by comparing it with a single reference vector that is learned . Despite the underlying idea being similar , we propose a different mechanism to focus on relevant words . In the following , we describe the architecture whose only requirements are i ) an embedding space of the input , and ii ) additional rationales , though the latter is not strictly required . As we are shall discuss , the model has a strong inductive bias , which is effective when trained with very little data . Hereinafter , we refer to our new architecture with the name PARCUS , which stands for Pattern Representations on Continuous Spaces . 3 THE PARCUS MODEL . Let us consider a classification task in which a very small labelled dataset DL = { ( x1 , r1 , y1 ) , . . . , ( xL , rL , yL ) } is given , where xi is an input sample , ri represents the rationale information and yi is the discrete target label . For the purpose of this paper , an input is a set of tokens xi = { x1i , . . . , x Ti i } of arbitrary size Ti . In addition , x j i ∈ Rd , where d is the size of an embedding space obtained using a pretrained model . Finally , we assume that each token in the input has been marked as relevant or not by annotators , i.e. , ri = { r1i , . . . , r Ti i } ∈ { 0 , 1 } Ti . Intuition When humans are asked to solve a text classification problem after seeing few examples , they tend to look for very simple patterns across the dataset , such as specific words . Nevertheless , humans are also able to generalise to semantically similar concepts ; our goal is to design a model that reflects this ability . For example , assume that the word “ excellent ” is important for classifying a movie review as positive . If we were to work in the character space , a straightforward solution would be to match specific ( sub- ) strings in the input , an instance of the so-called pattern matching technique . Clearly , pattern matching can not generalize to words that have similar meaning , e.g. , “ outstanding ” . In this work , we transfer the concept of pattern matching into the embedding space , where semantically similar words are assumed to have “ close ” representations . We achieve this via a mechanism that outputs a probability of soft matching between a token and a “ reference vector ” , which is learned together with the classification task to capture discriminative “ concepts ” . Differently from bag-of-word methods of Section 2 , this model easily accommodates multiple reference vectors , hence it can focus on many different concepts that are critical for classification . In order to guide the learning process using the given rationales , it seems sensible to magnify the error for those words that have been marked as relevant by annotators . Notwithstanding the simplicity of the idea , the underlying challenge is to effectively embed human knowledge into the reference vectors , which are responsible for the soft matching technique . In other words , the probabilities of soft matching should be highly correlated with the target class . Finally , we require that classification should be done in such a way that a user can explicitly understand which reference vectors are more important for positive or negative prediction . This last step is obtained by learning a linear combination of the soft matching probabilities . The next sections describe the proposed model in depth . First , we show how to compute and combine soft token matching probabilities , and then we introduce the error boosting technique that incorporates rationales in the training process . It is worth mentioning that both techniques have been designed to coexist , even though the latter is not strictly necessary to train the model .

Summary: This paper considers the problem of text classification, especially the settings in which the number of labeled sentences is very small. However, authors assume, annotations of rationales behind the label, i.e. highlighting tokens in a sentence which are important in deciding its label. As per my understanding, this is a big limitation. Second, the proposed model makes inference of class labels just based upon occurrence of words in a sentence, rather than making more sophisticated inferences relying upon sub-sequence patterns at least.