Experiments ::: Manual Evaluation	To verify the validity our generated adversarial examples, we further perform a manual evaluation. For each dataset, using BERT as the target model, we randomly sample 100 successful adversarial examples on which the target model fails, with balanced labels. We blend these adversarial examples with the corresponding original examples, and present each example to three workers on Amazon Mechanical Turk. We ask the workers to label the examples and also rate the grammaticality of the sentences with a scale of 1/2/3 (3 for no grammar error, 2 for minor errors, and 1 for vital errors). We integrate annotations from different workers with majority voting for labels and averaging for grammaticality. Table TABREF35 shows the results. Unlike target models whose performance drops dramatically on adversarial examples, human annotators retain high accuracies with a much smaller drop, while the accuracies of the target models are 0 on these adversarial examples. This demonstrates that the labels of most adversarial examples are successfully preserved to be consistent with original examples.

After adversarial modifications, the performance of the original target models (those without the “-adv” suffix) drops dramatically (e.g. the overall accuracy of BERT on Quora drops from 94.6% to 24.1%), revealing that the target models are vulnerable to our adversarial examples. Particularly, even though our generation is constrained by a BERT language model, BERT is still vulnerable to our adversarial examples. These results demonstrate the effectiveness of our algorithm for generating adversarial examples and also revealing the corresponding robustness issues. Moreover, we present some generated adversarial examples in the appendix. We notice that the original models are more vulnerable to unmatched adversarial examples, because there are generally more replaceable position choices during the generation. Nevertheless, the results of the matched case are also sufficiently strong to reveal the robustness issues. We do not quantitatively compare the performance drop of the target models on the adversarial examples with previous work, because we generate a new type of adversarial examples that previous methods are not capable of. We have different experiment settings, including original example sampling and constraints on adversarial modifications, which are tailored to the robustness issues we study.

Conclusion	In this paper, we propose a novel algorithm to generate new adversarial examples for paraphrase identification, by adversarially modifying original examples with difficult common words. We generate labeled adversarial examples that can be semantically different from original sentences and the BOW similarity between each pair of sentences is generally low. Such examples reveal robustness issues that previous methods are not able for. The accuracies of the target models drop dramatically on our adversarial examples, while human annotators are much less affected and the modified sentences retain a good grammarticality. We also show that model robustness can be improved using adversarial training with our adversarial examples. Moreover, our adversarial examples can foster future research for further improving model robustness.

Adversarial Examples with Difficult Common Words for Paraphrase Identification	Despite the success of deep models for paraphrase identification on benchmark datasets, these models are still vulnerable to adversarial examples. In this paper, we propose a novel algorithm to generate a new type of adversarial examples to study the robustness of deep paraphrase identification models. We first sample an original sentence pair from the corpus and then adversarially replace some word pairs with difficult common words. We take multiple steps and use beam search to find a modification solution that makes the target model fail, and thereby obtain an adversarial example. The word replacement is also constrained by heuristic rules and a language model, to preserve the label and grammaticality of the example during modification. Experiments show that our algorithm can generate adversarial examples on which the performance of the target model drops dramatically. Meanwhile, human annotators are much less affected, and the generated sentences retain a good grammaticality. We also show that adversarial training with generated adversarial examples can improve model robustness.

We aim to adversarially find a word replacement solution that maximizes the target model loss and makes the model fail, using beam search. We generate valid adversarial examples that are substantially different from those in previous work for paraphrase identification. Our adversarial examples are not limited to be semantically equivalent to original sentences and the unmodified parts of the two sentences are of low lexical similarity. To the best of our knowledge, none of previous work is able to generate such kind of adversarial examples. We further discuss our difference with previous work in Section 2.2. In summary, we mainly make the following contributions: We propose an algorithm to generate new adversarial examples for paraphrase identification. Our adversarial examples focus on robustness issues that are substantially different from those in previous work. We reveal a new type of robustness issues in deep paraphrase identification models regarding difficult common words. Experiments show that the target models have a severe performance drop on the adversarial examples, while human annotators are much less affected and most modified sentences retain a good grammaticality. Using our adversarial examples in adversarial training can mitigate the robustness issues, and these examples can foster future research.

Experiments ::: Sentence Pair BOW Similarity	To quantitatively demonstrate the difference between the adversarial examples we generate and those by previous work , , we compute the average BOW cosine similarity between the generated pairs of sentences. We only compare with previous methods that also aim to generate labeled adversarial examples that are not limited to be semantically equivalent to original sentences. Results are shown in Table TABREF38. Each pair of adversarial sentences by  differ by only one word. And in , sentence pairs generated with word swapping have exactly the same BOW. These two approaches both have high BOW similarities. By contrast, our method generates sentence pairs with much lower BOW similarities. This demonstrates a significant difference between our examples and the others. Unlike previous methods, we generate adversarial examples that can focus on robustness issues regarding the distraction from modified words that are the same for both sentences, towards matching the unmodified parts that are diverse for two sentences.

Performance drop on different kinds of adversarial examples with little overlap is not comparable, and thus surpassing other adversarial examples on model performance drop is unnecessary and irrelevant to support our contributions. Therefore, such comparisons are not included in this paper.

Experiments ::: Main Results	We train each target model on the original training data, and then generate adversarial examples for the target models. For each dataset, we sample 1,000 original examples with balanced labels from the corresponding test set, and adversarially modify them for each target model. We evaluate the accuracies of target models on the corresponding adversarial examples, compared with their accuracies on the original examples. Let $s$ be the success rate of generating adversarial examples that the target model fails, the accuracy of the target model on the returned adversarial examples is $1-s$. Table TABREF18 presents the results. The target models have high overall accuracies on the original examples, especially on the sampled ones since we form an unmatched original example with independently sampled sentences. The models have relatively lower accuracies on the unmatched examples in the full original test set of MRPC because MRPC is relatively small while the two labels are imbalanced in the original data (3,900 matched examples and 1,901 unmatched examples). Therefore, we generate adversarial examples with balanced labels instead of following the original distribution.

As for the second example (unmatched), each noun in the first sentence, “Gmail” and “account”, can form replaceable word pairs with each noun in the second sentence, “school”, “management” and “software”. The irreplaceable part determines that the modified sentences are “How can I get $\cdots $ back ? ” and “What is the best $\cdots $ ?” respectively. Sentences based on these two templates are likely to discuss about different things or different aspects, even when filled with common words, and thus they are likely to remain unmatched. In this way, the labels can be preserved in most cases.

Experiments ::: Effectiveness of Paired Common Words	We further analyse the necessity and effectiveness of modifying sentences with paired common words. We consider another version that replaces one single word independently at each step without using paired common words, namely the unpaired version. Firstly, for matched adversarial examples that can be semantically different from original sentences, the unpaired version is inapplicable, because the matched label can be easily broken if common words from two sentences are changed into other words independently. And for the unmatched case, we show that the unpaired version is much less effective. For a more fair comparison, we double the step number limit for the unpaired version. As shown in Table TABREF41, the performance of target models on unmatched examples generated by the unpaired version, particularly that of BERT, is mostly much higher than those by our full algorithm, except for BiMPM on MRPC but its accuracies have almost reached 0 (0.0% for unpaired and 0.2% for paired). This demonstrates that our algorithm using paired common words are more effective in generating adversarial examples, on which the performance of the target model is generally much lower.

Experiments ::: Adversarial Training	Adversarial training can often improve model robustness , . We also fine-tune the target models using adversarial training. At each training step, we train the model with a batch of original examples along with adversarial examples with balanced labels. The adversarial examples account for around 10% in a batch. During training, we generate adversarial examples with the current model as the target and update the model parameters with the hybrid batch iteratively. The beam size for generation is set to 1 to reduce the computation cost, since the generation success rate is minor in adversarial training. We evaluate the adversarially trained models, as shown in Table TABREF18. After adversarial training, the performance of all the target models raises significantly, while that on the original examples remain comparable. Note that since the focus of this paper is on model robustness which can hardly be reflected in original data, we do not expect performance improvement on original data. The results demonstrate that adversarial training with our adversarial examples can significantly improve the robustness we focus on without remarkably hurting the performance on original data.

Among them, back translation still aimed to produce semantically equivalent sentences; the others generated pairs of sentences with large Bag-of-Words (BOW) similarities, and the unmodified parts of the two sentences are exactly the same, so these same unmodified parts required little matching by target models. By contrast, we generate new adversarial examples with targeted labels by modifying a pair of original sentences together, using difficult common words. The modified sentences can be semantically different from original ones but still valid. The generated sentence pairs have much lower BOW similarities, and the unmodified parts are lexically diverse to reveal robustness issues regarding matching these parts when distracted by difficult common words in the modified parts. Thereby we study a new kind of robustness issues in paraphrase identification.

Results also show that the grammaticality difference between the original examples and adversarial examples is also small, suggesting that most adversarial examples retain a good grammaticality. This verifies the validity of our adversarial examples.

Related Work ::: Deep Paraphrase Identification	Paraphrase identification can be viewed as a problem of sentence matching. Recently, many deep models for sentence matching have been proposed and achieved great advancements on benchmark datasets. Among those, some approaches encode each sentence independently and apply a classifier on the embeddings of two sentences , , . In addition, some models make strong interactions between two sentences by jointly encoding and matching sentences , ,  or hierarchically extracting matching features from the interaction space of the sentence pair , , . Notably, BERT pre-trained on large-scale corpora achieved even better results . In this paper, we study the robustness of recent typical deep models for paraphrase identification and generate new adversarial examples for revealing their robustness issues and improving their robustness.

Experiments ::: Target Models	We adopt the following typical deep models as the target models in our experiments: BiMPM , the Bilateral Multi-Perspective Matching model, matches two sentences on all combinations of time stamps from multiple perspectives, with BiLSTM layers to encode the sentences and aggregate matching results. DIIN , the Densely Interactive Inference Network, creates a word-by-word interaction matrix by computing similarities on sentence representations encoded by a highway network and self-attention, and then adopts DenseNet  to extract interaction features for matching. BERT , the Bidirectional Encoder Representations from Transformers, is pre-trained on large-scale corpora, and then fine-tuned on this task. The matching result is obtained by applying a classifier on the encoded hidden states of the two sentences.

Related Work ::: Adversarial Example Generation	For a certain type of adversarial examples, adversarial attacks or adversarial example generation aim to find examples that are within the defined type and make existing models fail. Some work has no access to the target model until an adversarial dataset is generated , , , , , . However, in many cases including ours, finding successful adversarial examples, i.e. examples on which the target model fails, is challenging, and employing an attack algorithm with access to the target model during generation is often necessary to ensure a high success rate. Some prior work used gradient-based methods , , , requiring the model gradients to be accessible in addition to the output, and thus are inapplicable in black-box settings  where only model outputs are accessible. Though, the beam search in  can be adapted to black-box settings. Gradient-free methods for NLP generally construct adversarial examples by querying the target model for output scores and making generation decisions to maximize the model loss.  searched in the solution space. One approach in  greedily made word replacements and queried the target model in several steps.  employed a genetic algorithm.

Moreover, although the adversarial example generation is constrained by a BERT language model, BiMPM and DIIN which do not use the BERT language model can also significantly benefit from the adversarial examples, further demonstrating the effectiveness of our method.

Introduction	Paraphrase identification is to determine whether a pair of sentences are paraphrases of each other . It is important for applications such as duplicate post matching on social media , plagiarism detection , and automatic evaluation for machine translation  or text summarization . Paraphrase identification can be viewed as a sentence matching problem. Many deep models have recently been proposed and their performance has been greatly advanced on benchmark datasets , , . However, previous research shows that deep models are vulnerable to adversarial examples ,  which are particularly constructed to make models fail. Adversarial examples are of high value for revealing the weakness and robustness issues of models, and can thereby be utilized to improve the model performance for challenging cases, robustness, and also security. In this paper, we propose a novel algorithm to generate a new type of adversarial examples for paraphrase identification. To generate an adversarial example that consists of a sentence pair, we first sample an original sentence pair from the dataset, and then adversarially replace some word pairs with difficult common words respectively. Here each pair of words consists of two words from the two sentences respectively.

Methodology ::: Task Definition	Paraphrase identification can be formulated as follows: given two sentences $P=p_1p_2\cdots p_n$ and $Q=q_1q_2\cdots q_m$, the goal is to predict whether $P$ and $Q$ are paraphrases of each other, by estimating a probability distribution where $y\in \mathcal {Y} = \lbrace matched, unmatched \rbrace $. For each label $y$, the model outputs a score $[Z (P, Q)]_{y}$ which is the predicted probability of this label. We aim to generate an adversarial example by adversarially modifying an original sentence pair $(P, Q)$ while preserving the label and grammaticality. The goal is to make the target model fail on the adversarially modified example $(\hat{P}, \hat{Q})$: where $y$ indicates the gold label and $\overline{y}$ is the wrong label opposite to the gold one.

Methodology ::: Algorithm Framework	Figure FIGREF12 illustrates the work flow of our algorithm. We generate an adversarial example by firstly sampling an original example from the corpus and then constructing adversarial modifications. We use beam search and take multiple steps to modify the example, until the target model fails or the step number limit is reached. In each step, we modify the sentences by replacing a word pair with a difficult common word. There are two stages in deciding the word replacements. We first determine the best replaceable position pairs in the sentence pair, and next determine the best substitution words for the corresponding positions. We evaluate different options according to the target model loss they raise, and we retain $B$ best options after each stage of each step during beam search. Finally, the adversarially modified example is returned.

An advantage of using difficult common words for unmatched examples is that such words tend to make target models over-confident about common words and distract the models on recognizing the semantic difference in the unmodified part. Our algorithm explicitly utilizes this property and thus can well reveal such a robustness issue. Moreover, although there is no such a property for the matched case, replacing existing common words with more difficult ones can still distract the target model on judging the semantic similarity in the unmodified part, due to the bias between different words learned by the model, and thus our algorithm for generating adversarial examples with difficult common words works for both matched and unmatched cases.

Related Work ::: Adversarial Examples for NLP	Many methods have been proposed to find different types of adversarial examples for NLP tasks. We focus on those that can be applied to paraphrase identification. Some of them generate adversarial examples by adding semantic-preserving perturbations to the input sentences.  added perturbations to word embeddings. , , , ,  employed several character-level or word-level manipulations.  used syntactically controlled paraphrasing, and  paraphrased sentences with extracted rules. However, for some tasks including paraphrase identification, adversarial examples can be semantically different from original sentences, to study other robustness issues tailored to the corresponding tasks. For sentence matching and paraphrase identification, other types of adversarial examples can be obtained by considering the relation and the correspondence between two sentences.  considered logical rules of sentence relations but can only generate unlabelled adversarial examples.  and  generated a sentence pair by modifying a single original sentence. They combined both original and modified sentences to form a pair. They modified the original sentence using back translation, word swapping, or single word replacement with lexical knowledge.

As the second example in Figure FIGREF1 shows, we can replace words “Gmail” and “school” with a common word “credit”, and replace words “account” and “management” with ”score”. The modified sentences remain unmatched, but the target model can be fooled to predict matched for being distracted by the common words while ignoring the difference in the unmodified context. Following these observations, we focus on robustness issues regarding capturing semantic similarity or difference in the unmodified part when distracted by difficult common words in the modified part. We try to modify an original example into an adversarial one with multiple steps. In each step, for a matched example, we replace some pair of common words together, with another word adversarially selected from the vocabulary; and for an unmatched example, we replace some word pair, not necessarily a common word pair, with a common word. In this way, we replace a pair of words together from two sentences respectively with an adversarially selected word in each step. To preserve the original label and grammaticality, we impose a few heuristic constraints on replaceable positions, and apply a language model to generate substitution words that are compatible with the context.

Methodology ::: Candidate Substitution Word Generation	For a pair of replaceable positions, we generate candidate substitution words that can replace the current words on the two positions. To preserve the grammaticality and keep the modified sentences like human language, substitution words should be compatible with the context. Therefore, we apply a BERT language model  to generate candidate substitution words. Specifically, when some words in a text are masked, the BERT masked language model can predict the masked words based on the context. For a sentence $x_1x_2\cdots x_l$ where the $k$-th token is masked, the BERT masked language model gives the following probability distribution: Thereby, to replace word $p_i$ and $q_j$ from the two sentences respectively, we mask $p_i$ and $q_j$ and present each sentence to the BERT masked language model. We aim to replace $p_i$ and $q_j$ with a common word $w$, which can be regarded as the masked word to be predicted.

Experiments ::: Implementation Details	We adopt existing open source codes for target models BiMPM, DIIN and BERT, and also the BERT masked language model. For Quora, the step number limit $S$ is set to 5; the number of candidate substitution words generated using the language model $K$ and the beam size $B$ are both set to 25. $S$, $K$ and $B$ are doubled for MRPC where sentences are generally longer. The length difference between unmatched sentence pairs is limited to be no more than 3.

And difficult common words are words that we adversarially select to appear in both sentences such that the example becomes harder for the target model. The target model is likely to be distracted by difficult common words and fail to judge the similarity or difference in the context, thereby making a wrong prediction. Our adversarial examples are motivated by two observations. Firstly, for a sentence pair with a label matched, when some common word pairs are replaced with difficult common words respectively, models can be fooled to predict an incorrect label unmatched. As the first example in Figure FIGREF1 shows, we can replace two pairs of common words, “purpose” and “life”, with another common words “measure” and “value” respectively. The modified sentence pair remains matched but fools the target model. It is mainly due to the bias between different words and some words are more difficult for the model. When such words appear in the example, the model fails to combine them with the unmodified context and judge the overall similarity of the sentence pair. Secondly, for an unmatched sentence pair, when some word pairs, not necessarily common words, are replaced with difficult common words, models can be fooled to predict an incorrect label matched.

This yields a set of modified example after step $t$, $\lbrace (\hat{P}^{(t, k)}, \hat{Q}^{(t, k)}) \rbrace _{k=1}^{B}$, as follows: After $t$ steps, for some modified example $(\hat{P}^{(t,k)}, \hat{Q}^{(t,k)})$, if the label predicted by the target model is already $\overline{y}$, i.e. $[Z(\hat{P}^{(t,k)}, \hat{Q}^{(t,k)})]_{\overline{y}} > [Z(\hat{P}^{(t,k)},\hat{Q}^{(t,k)})]_y$, this example is a successful adversarial example and thus we terminate the modification process. Otherwise, we continue taking another step, until the step number limit $S$ is reached and in case of that an unsuccessful adversarial example is returned.

Methodology ::: Beam Search for Finding Adversarial Examples	Once the replaceable positions and candidate substitution words can be determined, we use beam search with beam size $B$ to find optimal adversarial modifications in multiple steps. At step $t$, we perform a modification in two stages to determine replaceable positions and the corresponding substitution words respectively, based on the two-stage greedy framework by . To determine the best replaceable positions, we enumerate all the possible position pairs, and obtain a set of candidate intermediate examples, $C_{pos}^{(t)}$, by replacing words on each position pair with a special token [PAD] respectively. We then query the target model with the examples in $C_{pos}^{(t)}$ to obtain the model output.

Methodology ::: Original Example Sampling	To sample an original example from the dataset for subsequent adversarial modifications, we consider two different cases regarding whether the label is unmatched or matched. For the unmatched case, we sample two different sentence pairs $(P_1, Q_1)$ and $(P_2, Q_2)$ from the original data, and then form an unmatched example $(P_1, Q_2, unmatched)$ with sentences from two sentence pairs respectively. We also limit the length difference $||P_1|-|Q_2||$ and resample until the limit is satisfied, since sentence pairs with large length difference inherently tend to be unmatched and are too easy for models. By sampling two sentences from different examples, the two sentences tend to have less in common originally, which can help better preserve the label during adversarial modifications, while this also makes it more challenging for our algorithm to make the target model fail. On the other hand, matched examples cannot be sampled in this way, and thus for the matched case, we simply sample an example with a matched label from the dataset, namely, $(P, Q, matched)$.

proposed a two-stage greedy algorithm and a method with gumbel softmax to improve the efficiency. In this work, we also focus on a black-box setting, which is more challenging than white-box settings. We use a two-stage beam search to find adversarial examples in multiple steps. We clarify that the major focus of this work is on studying new robustness issues and a new type of adversarial examples, instead of attack algorithms for an existing certain type of adversarial examples. Therefore, the choice of the attack algorithm is minor for this work as long as the success rates are sufficiently high.

From the language model output, we obtain a joint probability distribution as follows: We rank all the words within the vocabulary of the target model and choose top $K$ words with the largest probabilities, as the candidate substitution words for the corresponding positions.

Methodology ::: Replaceable Position Pairs	During adversarial modifications, we replace a word pair at each step. We set heuristic rules on replaceable position pairs to preserve the label and grammaticality. First of all, we require the words on the replaceable positions to be one of nouns, verbs, or adjectives, and not stopwords meanwhile. We also require a pair of replaceable words to have similar Part-of-Speech (POS) tags, i.e. the two words are both nouns, both verbs, or both adjectives. For a matched example, we further require the two words on each replaceable position pair to be exactly the same. Figure FIGREF15 shows two examples of determining replaceable positions. For the first example (matched), only common words “purpose” and “life” can be replaced. And since they are replaced simultaneously with another common words, the modified sentences are likely to talk about another same thing, e.g. changing from “purpose of life” to “measure of value”, and thereby the new sentences tend to remain matched.

We take top $B$ examples that maximize the output score of the opposite label $\overline{y}$ (we define this operation as $\mathop {\arg {\rm top}B}$), obtaining a set of intermediate examples $\lbrace (\hat{P}_{pos}^{(t,k)}, \hat{Q}_{pos}^{(t,k)}) \rbrace _{k=1}^{B}$, as follows: We then determine difficult common words to replace the [PAD] placeholders. For each example in $\lbrace (\hat{P}_{pos}^{(t, k)}, \hat{Q}_{pos}^{(t, k)}) \rbrace _{k=1}^B$, we enumerate all the words in the candidate substitution word set of the corresponding positions with [PAD]. We obtain a set of candidate examples, $C^{(t)}$, by replacing the [PAD] placeholders with each candidate substitution word respectively. Similarly to the first stage, we take top $B$ examples that maximize the output score of the opposite label $\overline{y}$.

Experiments ::: Datasets	We adopt the following two datasets: Quora : The Quora Question Pairs dataset contains question pairs annotated with labels indicating whether the two questions are paraphrases. We use the same dataset partition as , with 384,348/10,000/10,000 pairs in the training/development/test set respectively. MRPC : The Microsoft Research Paraphrase Corpus consists of sentence pairs collected from online news. Each pair is annotated with a label indicating whether the two sentences are semantically equivalent. There are 4,076/1,725 pairs in the training/test set respectively.

Question: How much dramatically results drop for models on generated adversarial examples?
Answer: BERT on Quora drops from 94.6% to 24.1%