WordsWorth Scores for Attacking CNNs and LSTMs for Text Classification
Authors: Nimrah Shakeel
Venue:  (2021)

Abstract: Black box attacks on traditional deep learning models trained for text classifica- tion target important words in a piece of text, in order to change model prediction. Current approaches towards highlighting important features are time consuming and require large number of model queries. We present a simple yet novel method to calculate word importance scores, based on model predictions on single words. These scores, which we call WordsWorth scores, need to be calculated only once for the training vocabulary. They can be used to speed up any attack method that requires word importance, with negligible loss of attack performance. We run ex- periments on a number of datasets trained on word-level CNNs and LSTMs, for sentiment analysis and topic classification and compare to state-of-the-art base- lines. Our results show the effectiveness of our method in attacking these models with success rates that are close to the original baselines. We argue that global importance scores act as a very good proxy for word importance in a local context because words are a highly informative form of data. This aligns with the manner in which humans interpret language, with individual words having well- defined meaning and powerful connotations. We further show that these scores can be used as a debugging tool to interpret a trained model by highlighting rele- vant words for each class. Additionally, we demonstrate the effect of overtraining on word importance, compare the robustness of CNNs and LSTMs, and explain the transferability of adversarial examples across a CNN and an LSTM using these scores. We highlight the fact that neural networks make highly informative pre- dictions on single words.

Under review as a conference paper at ICLR 2021
WORDSWORTH SCORES FOR ATTACKING CNNS AND LSTMs FOR TEXT CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Black box attacks on traditional deep learning models trained for text classifica- tion target important words in piece of text, order change model prediction. Current approaches towards highlighting important features are time consuming and require large number of model queries. We present a simple yet novel method to calculate word importance scores, based on model predictions on single words. These scores, which we call WordsWorth scores, need to be calculated only once for the training vocabulary. They can be used to speed up any attack method that requires word importance, with negligible loss of attack performance. We run ex- periments number of datasets trained word-level CNNs and LSTMs, for sentiment analysis and topic classification and compare to state-of-the-ar base- lines. Our results show the effectiveness of our method in attacking these models with success rates that are close to the original baselines. We argue that global importance scores act as very good proxy for word importance in a local context because words are a highly informative form of data. This aligns with the manner in which humans interpret language, with individual words having well-defined meaning and powerful connotations. We further show that these scores can be used as a debugging tool to interpret trained model by highlighting relevant words for each class. Additionally, we demonstrate the effect of overtraining on word importance, compare the robustness of CNNs and LSTMs, and explain the transferability of adversarial examples across a CNN and an LSTM using these scores. We highlight the fact that neural networks make highly informative pre- dictions single words.
INTRODUCTION
Deep learning models are vulnerable to carefully crafted adversarial examples. The goal of such an attack to fool classifier into giving incorrect prediction while the perturbed input appears normal to human observers. The probelm is important from the point of view of robustness as well as interpretability. Thoroughly analyzing different kinds of vulnerabilities in neural networks would help in creating robust models for deployment in the real world, in addition to throwing some light on the internal working of these models. In this work, we consider text classification, where finding important words in body of text is the first step towards malicious modification.
For this problem, we propose a novel method for calculating word importance. After training a model, we calculate importance scores over the entire training vocabulary, word by word. We further use these importance scores for black box attacks and demonstrate that the attack success rate is comparable the original methods, particularly for CNNs.
Since these scores are global and calculated over the training vocabulary, they can also be used as tool to interpret a trained model. They provide a measure for comparing different architectures and models beyond training and validation accuracy. Over single training dataset, we can compare small CNN to a large CNN, CNN to an LSTM, or the word importance distribution of one class against another, as we outline in our experiments section.
The motivation for our particular algorithm comes from the fact that in a piece of text, most of the time, words and phrases have strong influence their own. This gives rationale for evaluating a model on single words, in direct contrast to the leave-one-out technique, which involves deleting word from document and measuring its importance by the change in model prediction on this

---

Under review as a conference paper at ICLR 2021
modified input.
Further, we expect a well-trained network to treat a word approximately the same, irrespective of its location in the imput, when surrounding words removed. Thus particular word can occur at any position in a document with 200 words and its importance will be roughly the same. We expect well-trained model to exhibit this behaviour and experiments confirm this. In summary, our contributions are as follows:
We propose a simple and efficient method for calculating word importance for attacking traditional deep learning models in the black box setting.
We argue that these scores can act as tool for model interpretation and outline a number of use cases in this context.
2 RELATED WORK
2.1 ADVERSARIAL ATTACKS NLP MODELS:
The idea of perturbation, whether random malicious, is rather simple in the image domain, where salt and pepper noise can be enough to images to fool models. This kind of noise hard for humans detect. However, since text data is discrete, perturbations in text difficult quantify. Besides, people easily notice errors in computer-generant attack be counted successful, where a successful attack is one that forces the model to give an incorrect prediction while a human would make the correct prediction on the input.
text. This places additional constraints for an NLP
We limit ourselves to text classification problems, using sentiment analysis and topic classification as examples. We only consider the attack scenarios in which specific words in the input are replaced by valid words from the dictionary. Thus we are considering attacks in which extra information is appended to input data, or where word replacements purposefully introduce spelling errors. The former take an entirely different approach; the latter introduce errors and do not preserve seman- tics. In addition, training neural network to be robust to spelling errors would stop these attacks. Further, we limit ourselves to black box attacks where the attacker has no information about model architectures and parameters.
2.2 FIND AND REPLACE ATTACKS ON TEXT CLASSIFICATION
Most attacks on text classification solve the problem two parts; by locating important words in the input, and by finding suitable replacements for these words. We only consider attacks where substitutions are valid words picked from a dictionary, to avoid introducing grammatical errors, and ignore the case, for example, when spelling errors introduced in important words.
2.2.1 WHITE BOX ATTACKS
In the white-box setting, where attacker has full knowledge of the model architecture, gradients serve as a good proxy for word importance. Gong et al. (2018) use gradient based methods to locate important words. Samanta & Mchta (2017) use gradients to calculate word importance, with linguistic constraints over substitution words. Lei et al. (019) carry joint word and sentence attacks, by generating sentence paraphrases in the first stage, and resorting greedy word substitutions if the first stage fails. Again, important words are located by the magnitude the gradient word embedding.
2.2.2 BLACK BOX ATTACKS
In the black box scenario, where gradients are not available, saliency maps are calculated for words through different methods. Yang et al. (2018) provide a greedy algorithm which we will outline in detail in the next section.
Li al. (2016) propose masking each feature with zero padding, using the decrease the predicted probability the score of the feature or word, and masking the top-k features unknown. Alzantot et al. (2018) and Kuleshov et al.] (2018) propose variations of genetic algorithms. Kuleshov et al. (2018) replace words one by one until the classifier is misdirected while observing bound the
2

---

Under review as a conference paper at ICLR 2021
number of perturbed features. They run each new iteration on the modified input. For substitu- tion, they used post processed GloVe to find pool of suitable words. They also compute 'thought vectors' for sentences and ensure that these random sampling, where probability of each word being selected is proportional to the number of suitable neighbours for replacement. They use Google billion words language model to ensure that replacements match the context provided by the rest of the input. Ren et al.) (2019) propose a saliency- greedy algorithm, calculated by deleting words during the search phase and select substitutions from WordNet. Another similar attack model is Jin et al., (2019), which has extra se- mantic similarity checking when searching adversarial examples, and calculates word inpoddaanaen importance by deleting words.
preserved. Alzzntot et al. (2018) select words by
Zang et al. (2019) propose particle swarm optimization algorithm for the search problem. [aao et al. (2018) define different scoring functions where they look at prediction before and after remov- ing a particular word from a subset of input, and perform character level modifications in the second stage. Li al, (2019) use the sentence probability directly but once again, when ranking words, they try masking words in a sentence.
A common thread among all search methods for black box attacks is erasure omission, where the effect of a word is observed by comparing the classifier output probability for original input to that for input with this particular word removed or replaced by zero
2.3 INTERPRETABILITY IN MACHINE LEARNING THROUGH ERASURE
Li et al. (2016) is pioneering body of work in the domain of interpretability that highlights the importance of interpreting networks by erasing parts of various layers. This Leave-One-On Onee-utneotttt method is followed by most interpretation algorithms. For particular word, they calculate importance score as the average of prediction difference due to erasing this word from all test examples. Feng eet al.) (2018) gradually remove unimportant input words that only the important ones are left at the end. Barham & Feizi (2019) propose sparse projected gradient descent to generate adversarial examples improve interpretability. Nguyen (2018) looks at different methods of local explanations for labels, which include LIME, random feature deletion and first derivative saliency. Kádár et al. (217)) measure salience of a word by removing and noting the change in prediction. Jin et al. 2019) mention deleting particular word to calculate its importance score. [Ren et al. (2019) use word saliency which is the change in the classifier output if a word set to unknown. Carter et al. (2018) find sufficient input subsets while calculating the feature importance by masking words. For calculating word score matrices Xu & Du (2020) propose a method which involves masking words. We want to highlight the aspect that all the dominant techniques for interpretation use leave-one-out method for calculating word importance. Words Worth scores provide a reliable way of calculating feature importance, as shown by attack success rates. Thus, they can be reliably used to interpret a model after it has been trained. When these scores show that a particular word is important or unimportant for predicting a particular class, we can be sure that this is how the model behaves.
3 GREEDY ALGORITHM FOR BLACK BOX ATTACKS
The greedy algorithm mentioned in Yang et al. (2018) consists of two steps: finding the most im- portant words in text, and finding the most distracting replacements for these words, with some constraint. For an attack where k features are allowed to be perturbed, the top k important words are picked first, and then replaced one by one. In the first step, greedy finds the most important words by calculating importance scores for each word in the input using leave-one-out technique. The score of word the difference prediction probability for original input and for input with the word removed. The second step of the algorithm includes finding suitable replacement for these words. Throughout this paper we will use their greedy algorithm as a baseline for comparison, since it achieves the highest success rate among all black box methods (Hsieh et al., 2019).
Greedy uses the pretrained Glo embeddings and limits the search in second step to within a pre- specified distance, to preserve semantics. However, it should be noted that GloVe embeddings do not always provide semantic preserving replacements, and prsc-processed form of embeddings would work better, such the ones used by Kuleshov et al. (2018). In experiments, we use s0-dimensional GloVe embeddings to find replacements for important words. We limit our search to the ten nearest neighbours for each word.
3

---

Under review as a conference paper at ICLR 2021
4 WORDSWORTH SCORES FOR FEATURE IMPORTANCE
For determining importance of individual words text document, we propose WordsWorth scores, which are the prediction scores of cach individual word in the vocabulary. from the trained classifier. Since the CNNs and LSTMs on which we experiment have a fixed input size(limited to 200 words throughout the experiments), for caleulating these scores, the integer representation of the word (from the tokenizer) is appended with zeros and fed to the classifier. This is equivalent to evaluating the classifier piece of text where the text consists of single word. The algorithm for greedy attack using WordsWorth scores is given below.
Algorithm Step 1: Calculate WW scores over V, the training vocabulary Input F, trained CNN LSTM
Input the number of classes in data
Input the size of classifier input
Input V E Rd the training vocabulary having words
Output WW E Rdvp, Wordsworth scores over the training vocabulary
1: for w E V do
Ou. 01, 02. .2.-2..22, 2: define
3: WW (w) = F(r)
4: end for
Algorithm Step 2:
Greedily replace top k words to maximize incorrect class probability Input F, trained CNN or LSTM
Input X Rd text input to be modified
Input k the maximum number of features to be perturbed
Input D E pm=10 the nearest neighbour dictionary with 10 neighbours for each training vocab
word
Input WW E Rd*,, Wordsworth scores over the training vocabulary Output X', the maliciously modified input with maximum k words modified
2: Pick i1, ie.... is such that W (XW(,,) >= WW,,,,). > wW(
3: Initialize X" =X
4: for j E i,, (2,,.,, .., do
5: Wy = X D, = 10 nearest neighbour of wy
6: for E Dj do
7:
X if
X 1
=
W
if =i
8:
end for
9:
X'
argx max F(X)) F(X')I
10: end for
5 EXPERIMENTS
Comparison with two other blackbox attacks: Here we present the performance of del-one (Li et al. 2016) and greedy (Yang et al.) 2018), along with their modified versions, where word importance has been computed through WortsWorth scores for the modified versions. We call the modified versions del_one_wwe_eeeeeeeeeeeeeeeeeee_ ww and greedy www respectively. We also show the AUC scores for original data, named as original, to serve as a baseline.
5.1 SENTIMENT ANALYSIS: IMDB REVIEWS
5.1.1 DATASET AND MODEL ARCHITECTURE
We use the IMDB dataset Maas et al. 201T). which consists of 25000 training reviews and 25000 test reviews of variable length. Each review in the training set has positive/negative label attached
4

---

Under review as a conference paper at ICLR 2021
to it. Training vocabulary size is 5000 and cut each review to 200 words max. We simple CNN as the starting point of our experiments, with 32 dimensional embedding layer, 50 filters and 100 units in dense layer. The input layer uses word2vec embeddings that are learned during training. ReL.U activation is used. The network is trained on 25000 training examples. We use the Adam optmizer with default learning rate of 0.001 and early stopping. Validation accuracy is 88.78%.
We pick 300 examples at random from the test dataset and plot the ROC AUC values versus number of features perturbed for different algorithms. The results for CNN in figure Ileft) show that the modified versions of both algorithms have a performance that is comparable to that of the original versions. The distance between greedy and greedy ww is larger than that between del one and del one ww. This implies that if simply deleting words is the strategy, WordsWorts scores are almost as effective as manually deleting each word one by one and finding the one that contributes most to the model prediction.
5.1.2 RUNTIME COMPARISON WITH BASELINES
WordsWorth scores be calculated over the vocabulary learnt during training once the clas- sifier has been trained. At test time, model evaluation can be replaced by a simple lookup. Thus, for 5000 vocabulary size, WW score calculation takes 5000 model prediction. On the other hand, with 200 word reviews on average, the original paselines(greedd as well as del_one) need 5000 evalua- tions to locate important words just for 25 text examples, because they involve deleting each word in review to calculate its importance only for this particular review. Thus, if word appears in multiple reviews, its importance has to be calculated separately in the context of each review, for greedy and del.one.
attacks are carried out in bulk, WordsWorth evaluations are essentially free after the first 25 re- views. This indicates considerable slashing of computation time and resources. Additionally, WordsWorth score computations use a sparse input, which are more suitable for low-power plat- forms as compared to greedy and del one.
5.1.3 Do GREEDY AND GREEDY WW FIND THE SAME WORDS TO BE IMPORTANT?
During this experiment, when we compared the top ten words found by greedy and greedy www for each test example, 7.3 words were same on average. When we looked the top 5 words, 3.5 were same average. This strengthens the idea that both algorithms choose quite similar set of words for each instance.
5.1.4 LSTM
We repeat the experiment figure (right... Here, similar trends can be observed, with del del__on__we _ww performing close to del oon and greedy performing close to greedy. However, the difference here is larger as compared to CNN, which could be due to the LSTM learning a more robust representation.
LSTM with 100 examples chosen at random and report the results in
CNN with IMDB reviews
LSTM with IMDB reviews
0.
0.8
0.7
AUC scone
AUC score
0.6
0.6
0.
0.4
original
original
0.
greedy
greedy
0.3
greedy.vv
0.2
greedy
del one
del one
0.2
one Ww
del WW
No. of features perturbed
No. features perturbed
Figure 1: Left: AUC score words modified for IMDB Reviews with CNN Right: IMDB Review with LSTM
5

---

Under review as a conference paper at ICLR 2021
5.2 SENTIMENT ANALYSIS: YELP REVIEWS
The Yelp reviews dataset consists of positive and negative reviews. We train CNN with 32 input units, 32 filters and 64 hidden units with Relu activation. We use 83200 training example and 15000 validation examples. The CNN has 89.96% train accuracy, 93. 74% validation accuracy. We use the Adam optmizer with default learning rate of 0.001 and use early stopping. The input layer uses word2vec embeddings that are learned during training. We carry out attacks on 500 random test examples and report results in figure 1efft. Here we show the accuracy of the classifier on all 500 examples as the number of perturbed features increases. Here we have added aanlace and delete random as two additional baselines. Replace random replaces k features chosen at random, whereas relete_random deletes k random features.
delete.random
5.3 TOPIC CLASSIFICATION: AG NEWS
The AG news dataset consists of news related to 4 categories. We train a CNN with 32 input units, 32 filters and 64 hidden units with ReLU activation. It has 96000 training example and 24000 validation examples. We train for 2 epochs with 96.4% train accuracy, 94.8% validation accuracy. We use the Adam optmizer with default learning rate of 0.001 and use early stopping. The input layer uses word2vec embeddings that are learned during training.
We carry out attacks on 500 random test examples and report results in figure Here we show the accuracy of the classifier on 500 examples as the number of perturbed features increases. The results on this multiclass dataset confirm that WW scores are a good proxy for local word importance, The attacks here are untargeted, with the objective being to minimize the correct class probability. Targeted attacks can also be similarly launched using these scores.
CNN with Yelp
CNN with AG News
0.
0.9
0.8
0.8
07:
0.7
Rorrryy
0.6
Accurscy
0.6
0.5
0.5
0.4
replace pplace_randon random greedy
0.4
0.3
delete_random random
0.3
greedy
0.
greedy_wo cnn
0.2
greedy
del one
del one
011
one ww cnn
del WW
0.0
0.0
No. of features perturbed
No. features perturbed
Figure 2: Left: Accuracy vs words modified for Yelp reviews with CNN Right: Accuracy vs words modified for AG News with CNN
5.4 ADDITIONAL EXPERIMENTS ON IMDB REVIEWS
In this section we describe number of other experiments
the IMDB reviews dataset.
5.4.1 IS GREEDY LOWER BOUND FOR ATTACK SUCCESS?
We carried out further experiments by creating a new algorithm that evaluates greedy and greedy _ww for each test example and chooses the best result of both. If greedy and greedy ww were finding different types of vulnerabilities, we would have expected the algorithm to perform better than both. In fact, the algorithm did no better than greedy, and thus greedy appears to be a bound for greedy_ww www. Recall that in greedy, feature deletion is followed by feature insertion, so it does not follow directly that evaluation of input with a feature deleted should perform better than evaluation with everything except the feature deleted.
We hypothesize that the success of greedy attacks is partially explained by Worts scores. Most of the times, greedy is just picking the words with the highest global importance and finding replacements. In some cases it optimizes further, which explains its improved performance

---

Under review as a conference paper at ICLR 2021
greedy We would like to point out that surprisingly high fraction of successful greedy attacks is explained by our single word scores. This suggests that most of the time, the impact a word on the prediction is independent of its context.
5.4.2 THE CASE OF SMALL ARCHITECTURES
To test the algorithm with smaller networks, we train very small CNN (8 dimensional embedding layer,8 filters and 16 units in dense layer) and run our experiments on it. As with all other ex- periments, we use the Adam optmizer with default learning rate of 0.001 and use early stopping. The input layer uses word2vec embeddings that are learned during training. The test accuracy the same as that for the larger CNN, but the model appears to be holding up better to greedy ww attacks, as shown figure Ileft). Compare the results figure | (left), for large CNN. This shows that text classification problems can be easily learned by relatively small CNNS, particularly when the number classes is small.
Small CNN with IMDB reviews
LSTM with IMDB reviews
0.9
0.8
0.8
0.7
AUC score
AUC score
0.6
0.-
0.5
original
0.4
greedy
greedy
greedy.ww
greedy
del one
del one
0.3
one ww cnn
del ww cnn
No. of features perturbed
No. features perturbed
Figure 3: Left: AUC scores for IMDB reviews with small CNN Right: AUC scores for LSTM, with Worth scores from a CNN
The Pearson correlation between Words scores for this model and our main CNN is 0.83. The relatively poor performance of the bigger CNN could be due to overfitting. Since the task of binary classification rather simple, the smaller network could be learning more robust and mean- ingful representations. However, contradictory hypotheses exist for images, such as Oscar Deniz & Bueno (2020).
Here, highlight the fact that robustness to attacks, well as score comparison, could be one interesting way to compare small vs big and deep vs shallow models.
5.4.3 TRANSFERABILITY AND SECURITY
The phenomenon of transferability is well documented in adversarial attacks deep models, where adversarial examples generated for one trained model are often successful in attacking another model (Papernot et al. 2016). To demonstrate the phenomenon of transfcrability, we attack an LSTM with greedy and del_one, and with greedy_w www.yw/woooo __nnn and del_one_ ww .cnn where the WordsWort scores have been calculated through CNN, and the second step in adversarial search is evaluated directly on the LSTM. The correlation between the CNN and LSTM WW scores came out to be 0.88. Results of the attacks are shown figure right).
There is some drop in performance but still a noticeable degree of success. The close alignment of greedy and greedy ww enn shows that the importance scores caleulated through CNN are valid for LSTM too, even though directly using LSTM scores gives better performance. Compare this to figure rright) where LSTM was attacked with WW scores from the LSTM itself. We argue that this close, non-random alignment in figure 3 (right) explains the phenomenon of transferability in general. Features that are important for one architecture are important for another architecture too, when both models have been trained on the same dataset.
Our argument here has two parts: that scores from CNN and scores from LSTM have very high correlation(0.88) and scores from CNN can be used attack LSThhwhich is the transferability
7

---

Under review as a conference paper at ICLR 2021
Sorted WerdsWorth scores for CNN over the entire vocabulary
WordsWorth
0.5
0.4
1000
2000
3000
4000
5000
Words
Figure 4: WordsWorth scores for CNN on training vocabulary on IMDB reviews
phenomenon) with reasonable success rate, and that the former explains the latter.
Additionally, this highlights the aspect that for attacking a black box model, an adversary can train small model locally and it to highlight the vulnerable points of piece of text, while using the black box model to find out substitutes, since the latter requires much fewer number of model evaluations than the former.
INTERPRETING NEURAL NETWORKS THROUGH WORTH SCORES
In this section we show how to use WordsWorth scores for interpreting a model.
6.1 IMDB REVIEWS: LOCATING IMPORTANT WORDS
For the IMDB dataset, we computed the WordsWorth scores over our entire vocabulary (limited to 5000 top words) for the CNN as well as the LSTM. The top ten important words for the CNN are given in the table In this manner, the model designer can directly find the top ranked words associated with each sentiment after training and examine errors in training. For the CNN, the mean WW score is 0.559 and standard deviation is 0.0530. We also include snapshot of the scores for the entire vocabulary for the CNN in figure
6.2 AG NEWS
We computed the WordsWorth scores over our entire vocabulary (limited to 20000 top words) for the trained CNN. The top ten important words for each category are given in the table 3 Here, 'martha" is the 9th most important word for category "Business' and this potentially represents generalization error, one that model designer might want investigate. Using score-based insights to actually improve generalization is direction for future research.
6.2.1 AG NEWS WORD IMPORTANCE SCORE DISTRIBUTION
We also plot the scores for each class for AG News in figure and 7 Different categories have different distributions associated with them. This is an interesting fact and could point to differences in writing style for each category, or a difference word distribution within each category.
6.3 OVERTRAINING AND WORD IMPORTANCE SCORES
We overtrain a CNN on the IMDB reviews dataset and calculate word importance scores over the training vocabulary after every epoch. A few snapshots are in figure 5 Stats related to training are shown in table I Overtraining has a noticeable effect on individual word scores, with score distribution becoming homogeneous throughout the epochs. Again, investigating how word scores evolve can serve as an additional tool for a model designer in addition to training and validation accuracy.
8

---

Under review as a conference paper at ICLR 2021
7 CONCLUSION
We consider the problem of quickly finding important words in a text to perturb in order to maximize the efficacy of black box attacks on deep NLP models, in the context of text classification. For this problem we present WordsWorth, a feature ranking algorithm that performs comparably well to the state of the art approaches, particularly when only small number of feature perturbations allowed, while being orders of magnitude faster by virtue of being essentially a lookup on training vocabulary.
We also use these scores as a tool model interpretation, compare different architectures and give a metric for evaluating performance beyond training and validation accuracy.
We also explain the phenomenon of transferability observed in text adversarial attacks and show that black box attacks can yield valuable information about the training dataset. All in all, we argue that text generated by humans is a highly compact and informative representation of data and the way neural networks interpret language aligns with human understanding.
Overall, we provide a method for evaluating importance in parallel with word erasure techniques. Combining the two techniques would yield even richer insights into the workings of models. Seen another way, Words attacks uncover particular kind of vulnerability in deep models. Our work is the first step in designing a rule based algorithm to attack deep models that deal with text, and the next one would be to look complex interactions. By aligning the performance of rule based al algorithms with empirical methods currently popular in deep learning, we can improve our understanding of these otherwise blackbox models.
REFERENCES
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pp. 2890-2896, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v// D18- 1116. URL https: btLbtts:://wwtt http://wwwww....b...oooc/atttlla
Samuel Barham and Soheil Feizi. Interpretable adversarial training for text. CoRR, abs/1905, 2019.
905. 12864
Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you do this? under- standing black- box decisions with sufficient input subsets, 2018
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies of neural models make interpretations difficult. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. doi: 118855/3v//ds/ 1407. URL
http: / dx. doi. 10. 1ww53/16661/0 -100-111811177777
Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. 2018 IEEE Security and Privacy Workshops (SPW), doi: 10.1109/spw.211..00 20000000.......6...
May 2018.
URL http: / dx http:://xxddii.oooooooooo wtt.:////.....////......
2018 1018.00 00016
Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. Adversarial texts with gradient methods. 2018
Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. On the robustness of self-attentive models. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pp. 1520-1529, Florence, Italy, July 2019. Association BtLhttt::/ //www//wwww/..// welweb.
for Computational Linguistics. doi: 10. 10.18653//1//19-1142 oor//ann axtp/anntceooyy//119-111
Di Jin. Zhijing Jin, Jocy Tianyi Zhou, and Peter Szolovits. Is BERT really robust? natural language attack on text classification and entailment. CoRR, abs/1900. 19932, 2019. URL
http:://axiv www.orwwww........o..eooee www./osg/////ooooooo/...oo.. 9077 11932
Ákos Kádár, Grzegorz Chrupata, and Afra Alishahi. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics. 43(4):761-780, December 2017. doi: 10. 16.1662/C 00300.
wwwww ac web. org wwtp:/tto/aaaooooo.oooo http://anhhoooyyy/1/1o

---

Under review as a conference paper at ICLR 2021
Volodymyr Kuleshov, Shantanu Thakoor, Tingfung Lau, and Stefano Ermon. Adversarial exam- ples for natural language classification problems, 2018. URL htllitpsss//peeeereew..com
Daruu?id=rr123210
Qi Lei, Lingfei Wu, Pin-Yu Chen, Alex Dimakis, Inderjit S. Dhillon, and Michael J. Witbrock. Discrete adversarial attacks and submodular optimization with applications to text classification. In Ameet Talwalkar, Virginia Smith, and Matei Zaharia (eds.). Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 April 2, 2019. mlsys.org, 2019. URL https:: pttp:ps:/pproceee i ngs. ml sys org 2t.ooooo/224.p0
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. Proceedings 2019 Network and Distributed System Secu- rity Symposium. 2019. doi: 10. 1112222/ds..201922232.
URL http: / dx doi. org/o........
14722/ 20/22/dds.... 2019 23138
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. CoRR, abs/1612.08220,
wwtpx/wwwwwgg..///x//xoooo 1612.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Pruceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL hetps:
//www.aclweb... wwtpor///nanooonnnnneeoe.e
Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classifi- cation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers). pp. 1069-1078, New Orleans, Louisiana, June 2018. 8. Association for Computational Linguistics. doi: 10. 18.18653/v1// 1/118-11/111-11 1097. URL htt..httt////// /ww.://wwwwwwwoo.ooo www. ac web wttpbborrrgaattooooo////////
Noelia Vallez Jesus Salido Oscar Deniz, Anibal Pedraza and Gloria Bueno. Robustness to adversar- ial examples can be improved with overfitting. International Journal of Machine Learning and Cybernetics, 2020.
Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, URL http:: org/ abs/ 1605. 07277
2016.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversar- ial examples through probability weighted word saliency. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1085-1097, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18853/11//19-11000 URL hetps: /ww. aclweb org/ wtt.aoooooooooooooe// 1103
Suranjana Samanta and Sameep Mehta. Towards crafting text adversarial samples. CoRR, http::ttt///t../ www./aawwwaaaiaaaaaaaaaaaaaorooa abs/ 1077. 02812
abs/1707.02812, 2017.
Jincheng Xu and Qingfeng Du. On the interpretation of convolutional neural networks for text clas- sification. Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference Artificial Intelligence, 29 August-8 September 2020. Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference Prestigious Applications of Artificial In- telligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications. pp. 2252 2259. IOS Press, 2020. 3233 5233/PAIA200332
doi: 10.3233/FAIA200352 URL https: //ooi. org/10..
Puyudi Yang, Jianbo Chen, Cho-Jui Hsich, Jane-Ling Wang, and Michael I. Jordan. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. CoRR, abs/1805.12316 2018.
abs. /tt....///1....21 12316
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. 2019.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
Sorted WordsWorth scores for neagtive sentiment
Sorted WordsWorth scores for neagtive sentiment
10
07-
WortsWeth sore
WortbWortt score
0.6
0.2
1000
2000
3000
4000
3000
3000
2000
3000
4000
5000
Sorted WordsWorth scores for neagtive sentiment
Sorted WordsWerth scores for neagtive sentiment
0.8
0.8
WerdWoorth scre
WerdSWormh score
06
08
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Figure 5: Overtraining CNN with IMDB Reviews. Top Left: After epochs. Top Right: After epochs. Bottom Left: After 10 epochs. Bottom Right: After 13 epochs.
Table I: Stats for overtraining CNN with IMDB Reviews
epoch
Training loss
Training accuracy Validation Loss Validation Accuracy
0.1678
0.9376
0.3187
0.8725
0.0208
0.9960
0.5919
0,8626
10
8.0602e-04
0.9999
0.8431
0.8625
13
7.6706e-05
1.0000
1.0780
0.8643
Sorted WordsWorth World'
Sorted WordsWorth for category 'Business'
08
0.
0.6
WondWWttth
0.2
01
2500 5000
7500
10000 Wards
12500
25000
17500
2500
5000
7500
10000 Wards
12500
15000
17500 20090
Figure 6: Left: scores for category World. Right: scores for category Business
11

---

Under review as a conference paper at ICLR 2021
Sorted WordsWorth scores for category Sports'
Serted WordsWorth scores for catagory "Tach"
06
09
05
Wendowormh
0.3
04
02
011
02-
2500
5000
7500
10009 Wards
12500
15000
17500
20000
2500
5000
7500
10000
12500
15000
17500 20000
Wards
Figure 7: Left: Scores for category Sports. Right: Scores for category Tech
Table 2: Most important words for IMDB Reviews with CNN.
Positive
Negative
perfect
waste
excellent
worst
rare
poorly
surprisingly
awful
refreshing
disappointing
wonderfully
forgettable
superb
fails
wonderful
disappointment
highly
pointless
outstanding
alright
Table
Most important words for AG News for each class
World
Sports
Business
Tech
dialogue
bes
aspx bpd
hypersonic
cayman
motorsports
voip
adultery
speedway
aeronautic
singel
grenade
nets
retreated
cybersecurity
takers
rockies
alitalia
processors
wangari
gators
mortgages
hacker
constitutional
quarterback
airlines
halo
sudanese
clippers
attendants
phonographic
israel
knicks
martha
healthday
militant
motorsport
opec
browser
12


Reviews:
[{"review_id": "mg4XZRnLvW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is poorly written. This is especially the case for the\nproposed algorithm (the core contribution). This section is very\ndifficult to understand, and notations are awkward. Everything is a\nbit messy. However, the experimental results are quite well presented\n(to be compared with the beginning of the paper).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TSguygLK89s", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new and simple way to determine word importance for black box adversarial attacks on text classification models. Instead of using example-specific measures of importance like recent work (typically expensive to compute), the authors propose to feed individual words from the vocabulary to a trained model and use the model confidences to get global, class-specific importance scores.\n\nWhile being an interesting paper, at a high level I am concerned about several points:\n- The paper is unpolished and at times hard to follow. I do not consider it ready for publication at this stage.\n- There are many easy to implement baselines (feature selection has a rich history) that would have been very interesting to study. Are the WW scores capturing anything that simpler statistical methods do not?\n- Many implementation details are lacking, which could be an issue for reproducibility. For example, the CNN model details are unclear (number of layers, filter sizes, embedding initialization, etc.). Learning rates and other hyperparameters are not mentioned.\n- Why is the vocabulary size only 5000? Why are experiments run on only 500 examples, and are these examples selected?\n- The claim of comparable performance seems slightly exaggerated, as the WW scores perform consistently worse than the baselines. In some cases, the difference seems around 0.1 / 0.15 absolute AUC. Also, providing raw scores (either all scores in Appendix or a subset of the most interesting case in the text) would help readers to quantify these differences.\n- Some of the claims seem overly broad. For determining, say, grammaticality, I would expect greedy to vastly outperform greedy_ww.\n- It is unclear to me how WW scores help with network interpretation. Again I would expect WW scores to correlate with a number of statistical correlation measures.\n\nThere are many formatting issues, with figure numbers, references, equations, etc.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AzecD1-FePN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes WordsWorth score (WW score), a score to represent the importance of the word obtained from the trained model. Then, the score is applied to the greedy attack proposed by (Yang et al., 2018). In detail, the greedy attack first tries to search for the most important $k$ words in a text, and then it searches for values to replace the selected $k$ words. This paper uses the WW score to select the $k$ words in the first step.\n\nStrong points\n+ A simple but effective approach to utilize for the greedy attack\n\nConcerns: \n- The main concern of this paper is that a minor contribution to the current knowledge. Despite the paper stating that this paper is based on the greedy attack (Yang et al., 2018), the contribution of this paper is limited to calculate the word score from the trained classifier and applied it to the greedy attack.\n\n- Another concern about the paper is lack of rigorous experimentation to study the usefulness of the proposed method. This paper does not compare with other score-based approaches. That is, it was not even compared to the tf-idf based score approach.\n\n- The writing should be largely improved. Section 4 is the main part of this paper. In Step 1, this paper represents a word as $d$-dimensional vector. Why does this paper append the zeros in front of the word representation? Does it mean the one-hot vector? If not, some studies or discussions about this representation should be included. In Step 2, Equations are hard to follow, and some are incorrectly written (e.g. case equation and definition of the D’).\n\n- On the same note, readability and completeness of this paper do not meet the standard of conference. The reviewer suggests the authors to review the paper several times before submission.\n\n- This paper states that $F$ is the trained classifier. However, there are no explanations on how to train or what kinds of classifiers were used.\n\n- In Section 5.1.2, this paper states that it covers 5000 vocabularies freely after 25 reviews are processed because reviews are written in 200 words on average. It is definitely incorrect. Because all words in the review are not unique, it takes a longer timer to cache all words in the 5000 vocabularies. Although the proposed method can speed up by looking up the cached score, the performance of the proposed method is lower than the one of the original greedy approach. Some ablation studies or discussions about the relationship between speed and performance would have been useful to understand this. \n\n- Some parameters or data are heuristically selected such as selecting 10 nearest neighbors in step 2, picking 300 examples from test data in IMDB review experiments, and so on. Some form of ablation studies about the parameters would provide appropriateness to readers.\n\n- Furthermore, it would be better to show examples of successful attacks with WW scores.\n\n- In experiments, the AUC score is used for IMDB evaluation and the accuracy is used for Yelp and AG news. Are there any reasons to use different evaluation measures?\n\nMinor comments: \n1. It would be better to write some constants such as $k$ in Section 3 as an italic character.\n2. It will help readers if the authors can explicitly specify which side of the figure, left or right, is explained when the authors are referring to the figure in text.\n\nSome typos: \n1. In Section 5.1.1, greedyww -> greedy\\_ww\n2. In Section 5.4.2, figure ?? -> Figure 1\n3. In Section 5.4.1, $greedy_ww$ -> greedy\\_ww\n4. In Section 5.4.3, figure ??\n5. In Section 6.1, CNN 4 -> CNN at Figure 4\n...\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method for attacking neural NLP models based on individual word importance ("WordsWorth" scores).  This is an interesting, timely topic and there may be some interesting ideas here, but at present the paper suffers from poor presentation which makes it difficult to discern the contribution. Presentation issues aside, it seems that the experimental setup is missing key baselines (an issue not sufficiently addressed by the author response). 
Decision: Reject