WordsWorth Scores for Attacking CNNs and LSTMs for Text Classification

Black box attacks on traditional deep learning models trained for text classification target important words in a piece of text , in order to change model prediction . Current approaches towards highlighting important features are time consuming and require large number of model queries . We present a simple yet novel method to calculate word importance scores , based on model predictions on single words . These scores , which we call WordsWorth scores , need to be calculated only once for the training vocabulary . They can be used to speed up any attack method that requires word importance , with negligible loss of attack performance . We run experiments on a number of datasets trained on word-level CNNs and LSTMs , for sentiment analysis and topic classification and compare to state-of-the-art baselines . Our results show the effectiveness of our method in attacking these models with success rates that are close to the original baselines . We argue that global importance scores act as a very good proxy for word importance in a local context because words are a highly informative form of data . This aligns with the manner in which humans interpret language , with individual words having well-defined meaning and powerful connotations . We further show that these scores can be used as a debugging tool to interpret a trained model by highlighting relevant words for each class . Additionally , we demonstrate the effect of overtraining on word importance , compare the robustness of CNNs and LSTMs , and explain the transferability of adversarial examples across a CNN and an LSTM using these scores . We highlight the fact that neural networks make highly informative predictions on single words . 1 INTRODUCTION . Deep learning models are vulnerable to carefully crafted adversarial examples . The goal of such an attack is to fool a classifier into giving incorrect prediction while the perturbed input appears normal to human observers . The probelm is important from the point of view of robustness as well as interpretability . Thoroughly analyzing different kinds of vulnerabilities in neural networks would help us in creating robust models for deployment in the real world , in addition to throwing some light on the internal working of these models . In this work , we consider text classification , where finding important words in a body of text is the first step towards malicious modification . For this problem , we propose a novel method for calculating word importance . After training a model , we calculate importance scores over the entire training vocabulary , word by word . We further use these importance scores for black box attacks and demonstrate that the attack success rate is comparable to the original methods , particularly for CNNs . Since these scores are global and calculated over the training vocabulary , they can also be used as a tool to interpret a trained model . They provide a measure for comparing different architectures and models beyond training and validation accuracy . Over a single training dataset , we can compare a small CNN to a large CNN , a CNN to an LSTM , or the word importance distribution of one class against another , as we outline in our experiments section . The motivation for our particular algorithm comes from the fact that in a piece of text , most of the time , words and phrases have a strong influence on their own . This gives us a rationale for evaluating a model on single words , in direct contrast to the leave-one-out technique , which involves deleting a word from a document and measuring its importance by the change in model prediction on this modified input . Further , we expect a well-trained network to treat a word approximately the same , irrespective of its location in the input , when surrounding words are removed . Thus a particular word can occur at any position in a document with 200 words and its importance will be roughly the same . We expect a well-trained model to exhibit this behaviour and our experiments confirm this . In summary , our contributions are as follows : • We propose a simple and efficient method for calculating word importance for attacking traditional deep learning models in the black box setting . • We argue that these scores can act as a tool for model interpretation and outline a number of use cases in this context . 2 RELATED WORK . 2.1 ADVERSARIAL ATTACKS ON NLP MODELS : . The idea of perturbation , whether random or malicious , is rather simple in the image domain , where salt and pepper noise can be enough to images to fool models . This kind of noise is hard for humans to detect . However , since text data is discrete , perturbations in text are difficult to quantify . Besides , people easily notice errors in computer-generated text . This places additional constraints for an NLP attack to be counted as successful , where a successful attack is one that forces the model to give an incorrect prediction while a human would make the correct prediction on the input . We limit ourselves to text classification problems , using sentiment analysis and topic classification as examples . We only consider the attack scenarios in which specific words in the input are replaced by valid words from the dictionary . Thus we are not considering attacks in which extra information is appended to input data , or where word replacements purposefully introduce spelling errors . The former take an entirely different approach ; the latter introduce errors and do not preserve semantics . In addition , training a neural network to be robust to spelling errors would stop these attacks . Further , we limit ourselves to black box attacks where the attacker has no information about model architectures and parameters . 2.2 FIND AND REPLACE ATTACKS ON TEXT CLASSIFICATION . Most attacks on text classification solve the problem in two parts ; by locating important words in the input , and by finding suitable replacements for these words . We only consider attacks where substitutions are valid words picked from a dictionary , to avoid introducing grammatical errors , and ignore the case , for example , when spelling errors are introduced in important words . 2.2.1 WHITE BOX ATTACKS . In the white-box setting , where an attacker has full knowledge of the model architecture , gradients serve as a good proxy for word importance . Gong et al . ( 2018 ) use gradient based methods to locate important words . Samanta & Mehta ( 2017 ) use gradients to calculate word importance , with linguistic constraints over substitution words . Lei et al . ( 2019 ) carry joint word and sentence attacks , by generating sentence paraphrases in the first stage , and resorting to greedy word substitutions if the first stage fails . Again , important words are located by the magnitude of the gradient of word embedding . 2.2.2 BLACK BOX ATTACKS . In the black box scenario , where gradients are not available , saliency maps are calculated for words through different methods . Yang et al . ( 2018 ) provide a greedy algorithm which we will outline in detail in the next section . Li et al . ( 2016 ) propose masking each feature with zero padding , using the decrease in the predicted probability as the score of the feature or word , and masking the top-k features as unknown . Alzantot et al . ( 2018 ) and Kuleshov et al . ( 2018 ) propose variations of genetic algorithms . Kuleshov et al . ( 2018 ) replace words one by one until the classifier is misdirected while observing a bound on the number of perturbed features . They run each new iteration on the modified input . For substitution , they used post processed GloVe to find pool of suitable words . They also compute ’ thought vectors ’ for sentences and ensure that these are preserved . Alzantot et al . ( 2018 ) select words by random sampling , where probability of each word being selected is proportional to the number of suitable neighbours for replacement . They use Google 1 billion words language model to ensure that replacements match the context provided by the rest of the input . Ren et al . ( 2019 ) propose a saliency-based greedy algorithm , calculated by deleting words during the search phase and select substitutions from WordNet . Another similar attack model is Jin et al . ( 2019 ) , which has extra semantic similarity checking when searching adversarial examples , and calculates word importance by deleting words . Zang et al . ( 2019 ) propose a particle swarm optimization algorithm for the search problem . Gao et al . ( 2018 ) define different scoring functions where they look at prediction before and after removing a particular word from a subset of input , and perform character level modifications in the second stage . Li et al . ( 2019 ) use the sentence probability directly but once again , when ranking words , they try masking words in a sentence . A common thread among all search methods for black box attacks is erasure or omission , where the effect of a word is observed by comparing the classifier output probability for original input to that for input with this particular word removed or replaced by zero . 2.3 INTERPRETABILITY IN MACHINE LEARNING THROUGH ERASURE . Li et al . ( 2016 ) is a pioneering body of work in the domain of interpretability that highlights the importance of interpreting networks by erasing parts of various layers . This Leave-One-Out method is followed by most interpretation algorithms . For a particular word , they calculate importance score as the average of prediction difference due to erasing this word from all test examples . Feng et al . ( 2018 ) gradually remove unimportant input words so that only the important ones are left at the end . Barham & Feizi ( 2019 ) propose sparse projected gradient descent to generate adversarial examples to improve interpretability . Nguyen ( 2018 ) looks at different methods of local explanations for labels , which include LIME , random feature deletion and first derivative saliency . Kádár et al . ( 2017 ) measure salience of a word by removing it and noting the change in prediction . Jin et al . ( 2019 ) mention deleting a particular word to calculate its importance score . Ren et al . ( 2019 ) use word saliency which is the change in the classifier output if a word is set to unknown . Carter et al . ( 2018 ) find sufficient input subsets while calculating the feature importance by masking words . For calculating word score matrices , Xu & Du ( 2020 ) propose a method which involves masking words . We want to highlight the aspect that all the dominant techniques for interpretation use leave-one-out method for calculating word importance . WordsWorth scores provide a reliable way of calculating feature importance , as shown by attack success rates . Thus , they can be reliably used to interpret a model after it has been trained . When these scores show that a particular word is important or unimportant for predicting a particular class , we can be sure that this is how the model behaves . 3 GREEDY ALGORITHM FOR BLACK BOX ATTACKS . The greedy algorithm mentioned in Yang et al . ( 2018 ) consists of two steps : finding the most important words in a text , and finding the most distracting replacements for these words , with some constraint . For an attack where k features are allowed to be perturbed , the top k important words are picked first , and then replaced one by one . In the first step , greedy finds the most important words by calculating importance scores for each word in the input using leave-one-out technique . The score of a word is the difference in prediction probability for original input and for input with the word removed . The second step of the algorithm includes finding suitable replacement for these words . Throughout this paper we will use their greedy algorithm as a baseline for comparison , since it achieves the highest success rate among all black box methods ( Hsieh et al. , 2019 ) . Greedy uses the pretrained GloVe embeddings and limits the search in second step to within a prespecified distance , to preserve semantics . However , it should be noted that GloVe embeddings do not always provide semantic preserving replacements , and a post-processed form of embeddings would work better , such as the ones used by Kuleshov et al . ( 2018 ) . In our experiments , we use 50-dimensional GloVe embeddings to find replacements for important words . We limit our search to the ten nearest neighbours for each word .

Summary: This paper proposes WordsWorth score (WW score), a score to represent the importance of the word obtained from the trained model. Then, the score is applied to the greedy attack proposed by (Yang et al., 2018). In detail, the greedy attack first tries to search for the most important $k$ words in a text, and then it searches for values to replace the selected $k$ words. This paper uses the WW score to select the $k$ words in the first step.