Title: Adversarial Examples for Natural Language Classification Problems
Authors: Volodymyr Kuleshov
Shantanu Thakoor
Tingfung Lau
Stefano Ermon
Venue: ICLR 2018

Abstract: Modern machine learning algorithms are often susceptible to adversarial examples — maliciously crafted inputs that are undetectable by humans but that fool the algorithm into producing undesirable behavior. In this work, we show that adversarial examples exist in natural language classification: we formalize the notion of an adversarial example in this setting and describe algorithms that construct such examples. Adversarial perturbations can be crafted for a wide range of tasks — including spam filtering, fake news detection, and sentiment analysis — and affect different models — convolutional and recurrent neural networks as well as linear classifiers to a lesser degree. Constructing an adversarial example involves replacing 10-30% of words in a sentence with synonyms that don’t change its meaning. Up to 90% of input examples admit adversarial perturbations; furthermore, these perturbations retain a degree of transferability across models. Our findings demonstrate the existence of vulnerabilities in machine learning systems and hint at limitations in our understanding of classification algorithms.


Review discussion:
Nice overview of adversarial techniques in natural language classification. The paper introduces the problem of adversarial perturbations, how they are constructed and demonstrate what effect they can have on a machine learning models.
The authors study several real-world adversarial examples, such as spam filtering, sentiment analysis and fake news and use these examples to test several popular classification models in context of adversarial perturbations.
Their results demonstrate the existence of adversarial perturbations in NLP and show that several different types of errors occur (syntactic, semantic, and factual). Studying each of these errors type can help defend and improve the classification algorithms via adversarial training.
Pros: Good analysis on real-world examples
Cons: I was expecting more actual solutions in addition to analysis
Thank you for your generally positive review.
However, since the score that you are giving us (6.0) is not very high, could you please elaborate on your concerns with this paper?
We would be very happy to improve the paper before the final decision period, but your only negative comment is that you were expecting "more actual solutions in addition to analysis". We don’t know how to interpret that.
If you are interested in ways of protecting against attacks, we tried using the method of Miyato et al., which results in a small increases in performance on adversarial examples.
Happy holidays! Since the end of the discussion period is approaching, we wanted to check if you had the chance to read our response. We would like very much to address your concerns and have a discussion about our paper before the January 5 deadline.
Thank you!
This paper proposes a method to generate adversarial examples for text classification problems. They do this by iteratively replacing words in a sentence with words that are close in its embedding space and which cause a change in the predicted class of the text. To preserve correct grammar, they only change words that don't significantly change the probability of the sentence under a language model.
The approach seems incremental and very similar to existing work such as Papernot et. al. The paper also states in the discussion in section 5.1 that they generate adversarial examples in state-of-the-art models, however, they ignore some state of the art models entirely such as Miyato et. al.
The experiments are solely missing comparisons to existing text adversarial generation approaches such as Papernot et. al and a comparison to adversarial training for text classification in Miyato et. al which might already mitigate this attack. The experimental section also fails to describe what kind of language model is used, (what kind of trigram LM is used? A traditional (non-neural) LM? Does it use backoff?).
Finally, algorithm 1 does not seem to enforce the semantic constraints in Eq. 4 despite it being mentioned in the text. This can be seen in section 4.5 where the algorithm is described as choosing words that were far in word vector space. The last sentence in section 6 is also unfounded.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z.Berkay Celik, and Ananthram Swami
Practical Black-Box Attacks against Machine Learning.
Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security
Takeru Miyato, Andrew M. Dai and Ian Goodfellow
Adversarial Training Methods for Semi-Supervised Text Classification.
International Conference on Learning Representation (ICLR), 2017
* I increased the score in response to the additional experiments done with Miyato et. al.
However, the lack of a more extensive comparison with Papernot et. al. is still needed. The venue for that paper might not be well known but it was submitted to arXiv computer science too and the paper seems very related to this work. It's hard to say if Papernot et. al produces more perceptible samples without doing a proper comparison. I find the lack of a quantitative comparison to any existing adversarial example technique problematic.
Thank you very much for your feedback. We see that you raise several issues in your review.
1. We are not evaluating our method on state-of-the-art models
Here, we respectfully disagree: this claim is incorrect.
Our models (except the linear classifier) achieve accuracies of 94.9%-95.3% on the popular Yelp dataset (same as the papers whose models we used). The 2017 state-of-the art is ~97% using a ResNet [1], and the 2016 SOA was 96%. On the widely used IMDB dataset we obtain accuracies of ~92-93%; the state-of-the-art around 96%. (We didn't include IMBD results due to lack of space and similarity to Yelp). On spam detection, we also obtain nearly perfect accuracy.
Finally, there is no standard fake news dataset, but we achieve high accuracy on the one that we use.
Furthermore, our architectures are very modern and date from as recently as last year (see our citations)
[1] Johnson and Zhang, http://www.aclweb.org/anthology/P17-1052
2. We do not take into account the recent work of Miyato et al.
First, note Miyato et al. propose a method for adversarial training, which is very different from adversarial examples (what we study). Adversarial training is at the moment not part of the standard toolkit for classification algorithm, which is why we did not immediately compare to it.
You also mention that the method of Miyato et al.. could be used as a defense.
However, that too is not correct: our adversarial examples arise from large perturbations in embedding space (we replace an entire word); Miyato et al., on the other hand, perform adversarial training in embedding space, which consists in introducing very small perturbations. In the context of Naive Bayes, their method does not even apply (there are no word embeddings in NB)
We confirmed this empirically by testing the method of Miyato et al. on the CNN model. We observed only a small (10%) improvement in accuracy on AEs. We are also currently running additional experiments on every setup. We will report here our final results once they are done.
3. There is no comparison to existing adversarial text generation approaches
We are more than happy to compare to any existing work.
However, the Papernot et al. paper you cite has no mention of text classification at all. The underlying algorithm is gradient-based, and is not applicable to discrete inputs (relative to which we cannot differentiate the model). The Miyato paper you mention does not work well as a defense against our method (see above).
An anonymous commenter mentioned some relevant work; please see also our detailed response to their comment.
4. Extra technical questions and clarifications
The language model we use a tri-gram model. This is a detail that we forgot to mention and that we will add into the paper.
We certainly enforce equation (4) in our algorithm. There is a typo in Algorithm 1 (it should read "Equations 4, 5" instead of "Equation 5"), which we will correct right away. We apologize for any confusion due to this typo.
Apologies, from your comment I realised I cited the wrong Papernot et. al paper for previous work on adversarial text generation. I meant to cite:
Papernot, N., McDaniel, P., Swami, A., & Harang, R. (2016, November). Crafting adversarial input sequences for recurrent neural networks. In Military Communications Conference, MILCOM 2016-2016 IEEE (pp. 49-54). IEEE.
Thank you for pointing us to this paper. We were not aware of it, and we agree that it needs to be cited.
However, the scope of this paper is very limited compared to our work.
1. Most importantly, there is no notion of preserving the meaning of the original sentence.
In other words, Papernot et al. replace words in a sentence without ever looking at whether the new words are related to the originals. In our experience, this would most likely produce non-sensical sentences, and a human would recognize them as such. The paper also offers no evaluation of the constructed AEs besides the accuracy of the classifier, e.g. it doesn't evaluate their coherence, their similarity to the original, their human classification accuracy, and it does not even provide a list of example AEs.
2.
Significantly more limited scope of the experimental setup
Papernot et al. focus on a specific model (recurrent neural networks) and a specific task (sentiment analysis). Our work compares RNNs with word-level CNNs, character-level CNNs, and Naive Bayes. We also look at multiple tasks: sentiment analysis, fake news detection, spam classification. Our algorithm explicitly attempts to preserve of the meaning of the new sentences (and make the adversarial examples difficult to detect). We extensively evaluate our method on Mechanical Turk.
Our algorithm greedily optimizes the objective (score of the wrong class); that of Papernot appears to optimize a linearization of that objective. We are happy to compare against their approach if you think it's important, but we don't see why this would be better than optimizing the actual objective function.
Overall, we think that the paper by Papernot et al. does not prove that *imperceptible* adversarial examples can be constructed for text classification tasks. This is a crucial property of AEs. Our paper, on the other hand, demonstrates that it is possible.
Finally, this paper was published at the "Military Communications Conference". We have never heard of this venue, and suspect that it might be closer to a peer-reviewed workshop in the machine learning community. The page count appears to be shorter, and the scope of the experiments seems to be considerably more narrow (e.g., human evaluation would be a must at ICML/NIPS/ICLR).
As with the three papers below, we think it's a bit unfair to count previous recent publications on the Arxiv, conference workshops, or non-standard conference proceedings against the novelty of our paper. We believe that these papers should be cited and discussed as parallel work, but it is a bit harsh to claim that our work is incremental.
Happy holidays! Since the end of the discussion period is approaching, we wanted to check if you had the chance to read our response. We would like very much to address your concerns and have a discussion about our paper before the January 5 deadline.
Thank you!
The paper shows that neural networks are sensitive to adversarial perturbation for a set of NLP text classifications. They propose constructing (model-dependent) adversarial examples by optimizing a function J (that doesn't seem defined in the paper) subject to a constraint c(x, x') < \gamma (i.e. that the original input and adversarial input should be similar)
c is composed of two constraints:
1. || v - v' ||_2 < \gamma_1, where v and v' are bag of embeddings for each input
2. |log P(x') - log P(x)| < \gamma_2 where P is a language model
The authors then show that for 3 classification problems "Trec07p", "Yelp", and "News" and 4 models (Naive Bayes, LSTM, word CNNs, deep-char-CNNs) that the models that perform considerably worse on adversarial examples than on the test set.
Furthermore to test the validity of their adversarial examples, the authors show the following:
1. Humans achieve somewhat similar accuracy on the original adverarial examples (8 points higher on one dataset and 8 points lower on the other two)
2. Humans rate the writing quality of both the original and adversarial examples to be similar
3. The adversarial examples only somewhat transfer across models
My main questions/complaints/suggestions for the paper are:
-Novelty/Methodology. The paper has mediocre novelty given other similar papers recently.
On question I have is about whether the generated examples are actually close to the original examples. The authors do show some examples that do look good, but do not provide any systematic study (e.g. via human annotation)
This is a key challenge in NLP (as opposed to vision where the inputs are continuous so it is easy to perturb them and be reasonably sure that the image hasn't changed much). In NLP however, the words are discrete, and the authors measure the difference between an original example and the adversary only in continuous space which may not actually be a good measure of how different they are.
They do have some constraint that the fraction of changed words cannot differ by more than delta, but delta = 0.5 in the experiments, which is really large! (i.e. 50% of the words could be different according to Algorithm 1)
-Writing: the function J is never mathematically defined, neither is the function c (except that it is known to be composed of the semantic/syntactic similarity constraints).
The authors talk about "syntactic" similarity but then propose a language model constraint. I think is a better word is "fluency" constraint.
The results in Table 3 and Table 6 seem different, shouldn't the diagonal of Table 6 line up with the results in Table 3?
-Experimental methodology (more of a question since authors are unclear): The authors write that "all adversarial examples are generated and evaluated on the test set".
There are many hyperparameters in the proposed authors' approach, are these also tuned on the test set?
That is unfair to the base classifier. The adversarial model should be tuned on the validation set, and then the same model should be used to generate test set examples. (The authors can even show the validation adversarial accuracy to show how/if it deviates from the test accuracy)
-Lack of related work in NLP (see the anonymous comment for some examples). Even the related work in NLP that is cited e.g. Jia and Liang 2017 is obfuscated in the last page. The authors' introduction only refers to related works in vision/speech and ignores related NLP work.
Furthermore, adversarial perturbation is related to domain transfer  (since both involve shifts between the training and test distribution) and it is well known for instance that models that are trained on Wall Street Journal perform poorly on other domains.  See SJ Pan and Q Yang, A Survey on transfer learning, 2010, for some example references.
Thank you for your feedback! We identified several concerns in your review.
1. Our work is not sufficiently novel.
First, we believe that your claim that our paper has "mediocre novelty" is quite harsh. Especially given that your review does not include any references to related papers.
Our paper explores adversarial examples for natural language classification. If you think this has been done before, could you please provide references? We will be more than happy to compare.
Earlier, an anonymous commenter mentioned 3 references; we discuss these below.
2. We are not evaluating similarity between the adversarial examples and the originals.
First, saying that we "don't provide any systematic study via human annotation" is incorrect: we measured both human accuracy and readability on every model/dataset combination (dozens of experiments in total).
Next, we want to point out that the topic of similarity is more nuanced than it seems. Most often, the algorithm changes irrelevant parts of the input, e.g.:
On Wednesday, Obama raised taxes (fake) -> On Tuesday, Obama raised taxes. (real)
We ordered pasta and it was the worst we ever had (neg) -> We ordered chicken and it was the worst we ever had (pos)
These are still valid similarity-based adversarial examples: i.e., we fool the fake news detection system and succeed in spreading the false news that Obama is raising taxes. What is most important is that humans and machines consistently classify our examples into opposite classes and the examples sound natural to humans.
However, we understand the validity of your concern and we thank you for suggesting this experiment. To address your concern as much as possible, we performed the experiment in question.
We quantified the similarity of adversarial examples via Mechanical Turk. We asked Turkers to rate the similarity of the adversarial examples to the originals on a scale of 1-5, with 1 being completely unrelated, and 5 being identical. Here are the results we compiled so far:
Domain Score Number
News 1 56
News 2 49
News 3 138
News 4 141
News 5 116
Yelp 1 53
Yelp 2 40
Yelp 3 121
Yelp 4 180
Yelp 5 106
Overall, we see that the majority of adversarial examples are similar to the originals.
3. We measure the difference of adversarial examples only in continuous space.
Again, this is incorrect. We optimize a continuous objective;
however we measure and report only metrics that are derived from human experiments (accuracy and readability)
Although we set the maximum fraction of replaced words to 50%, we very rarely reach that number (see examples in the paper). This is just an early stopping criterion. Similarity is enforced via Equations 4 and 5, and the constants there are indeed tight. This can be seen by looking at the similarity of our examples to the originals. We are happy to add an experiment where we vary the threshold, if you think this is important.
4. Other technical issues
The objective J is the score of the target (adversarial) class, and we define it right below  Equation 6. Sorry if this wasn't clear, we will make it more obvious.
The function c is defined right below Equation 3, and is simply a vector of constraints. In our algorithm, we instantiate c with two constraints: a syntactical and a semantic one.
We are going to think of a better name for the syntactic constraint (e.g., fluency as you suggested).
We did not tune any hyper-parameters on the test set (we're not sure what might lead to think that). We chose hyper parameters on the training set (validation would have been slightly cleaner). We did not touch the set test, except for generating the final adversarial examples.
We are certainly not “obfuscating” the work of Jia and Liang. We spend a whole paragraph comparing our work to theirs in Section 3.1.
In brief, they create AEs by adding irrelevant sentences; we create AEs by changing some words to synonyms. We will extend the existing discussion if you think it’s necessary.
Happy holidays! Since the end of the discussion period is approaching, we wanted to check if you had the chance to read our response. We would like very much to address your concerns and have a discussion about our paper before the January 5 deadline.
Thank you!

Meta-review:
role: PC
data: comment: This paper presents a way to generate adversarial examples for text classification.   The method is simple -- finding semantically similar words and replacing them in sentences with high language model score.  The committee identifies weaknesses in this paper that resonate with the reviews below -- reviewer 1 suggests that the authors should closely compare with the work of Papernot et al, and the response to that suggestion is not satisfactory.  Addressing such concerns would make the paper stronger for a future venue.

Decision: Reject