Title: ON MEANING-PRESERVING ADVERSARIAL PERTUR-BATIONS FOR SEQUENCE-TO-SEQUENCE MODELS

Abstract: Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance. However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output. Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general. Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs. Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

1. INTRODUCTION:

Attacking a machine learning model with adversarial perturbations is the process of making changes to its input to maximize an adversarial goal, such as mis-classification (Szegedy et al., 2013) or mistranslation (Zhao et al., 2018) . These attacks provide insight into the vulnerabilities of machine learning models and their brittleness to samples outside the training distribution. This is critical for systems sensitive to safety or security, e.g. self-driving cars (Bojarski et al., 2016) .

Adversarial attacks were first defined and investigated for computer vision systems (Szegedy et al. (2013) ; Goodfellow et al. (2014) ; Moosavi-Dezfooli et al. (2016) inter alia), where they benefit from the fact that images are expressed in continuous space, making minuscule perturbations largely imperceptible to the human eye. In discrete spaces such as natural language sentences, the situation is more problematic; even a flip of a single word or character is generally perceptible by a human reader. Thus, most of the mathematical framework in previous work is not directly applicable to discrete text data. Moreover, there is no canonical distance metric for textual data like the 2 norm in real-valued vector spaces such as images, and evaluating the level of semantic similarity between two sentences is a field of research of its own (Cer et al., 2017) . This elicits a natural question: what does the term "adversarial perturbation" mean in the context of natural language processing (NLP)?

We propose a simple but natural criterion for adversarial examples in NLP, particularly seq2seq models: adversarial examples should be meaning-preserving on the source side, but meaning-destroying on the target side. The focus on explicitly evaluating meaning preservation is in contrast to previous work on adversarial examples for seq2seq models (Belinkov & Bisk, 2018; Zhao et al., 2018; Cheng et al., 2018; Ebrahimi et al., 2018a) . Nonetheless, this feature is extremely important; given two sentences with equivalent meaning, we would expect a good model to produce two outputs with equivalent meaning.In other words, any meaning-preserving perturbation that results in the model output changing drastically highlights a fault of the model.

A first technical contribution of the paper is to lay out a method for formalizing this concept of meaning-preserving perturbations ( §2). This makes it possible to evaluate the effectiveness of adversarial attacks or defenses either using gold-standard human evaluation, or approximations that can be calculated without human intervention. We further propose a simple method of imbuing gradient-based word substitution attacks ( §3.1) with simple constraints aimed at increasing the chance that the meaning is preserved ( §3.2).

Our experiments are designed to answer several questions about meaning preservation in seq2seq models. First, we evaluate our proposed "source-meaning-preserving, target-meaning-destroying" criterion for adversarial examples using both manual and automatic evaluation ( §4.2) and find that a less widely used evaluation metric (chrF) provides significantly better correlation with human judgments than the more widely used BLEU and METEOR metrics. We proceed to perform an evaluation of adversarial example generation techniques, finding that constrained substitution attacks do preserve meaning to a higher degree than unconstrained attacks while still degrading the performance of the systems across different languages and model architectures ( §4.3). Finally we apply existing methods for adversarial training to the adversarial examples with these constraints and show that making adversarial inputs more semantically similar to the source is beneficial for robustness to adversarial attacks and does not decrease test performance ( §5).

2. A FRAMEWORK FOR EVALUATING ADVERSARIAL ATTACKS:

In this section, we present a simple procedure for evaluating adversarial attacks on seq2seq models. We will use the following notation: x and y refer to the source and target sentence respectively. We denote x's translation by model M as y M (x). Finally, x and y M (x) represent an adversarially perturbed version of x and its translation by M , respectively. The nature of M and the procedure for obtaining x from x are irrelevant to the discussion below.

2.1. THE ADVERSARIAL TRADE-OFF:

The goal of adversarial perturbations is to produce failure cases for the model M . Hence, the evaluation must include some measure of the target similarity between y and y M (x), which we will denote s tgt (y, y M (x)). However, if no distinction is being made between perturbations that preserve the meaning and those that don't, a sentence like "he's very friendly" is considered a valid adversarial perturbation of "he's very adversarial", even though its meaning is the opposite. Hence, it is crucial, when evaluating adversarial attacks on MT models, that the discrepancy between the original and adversarial input sentence be quantified in a way that is sensitive to meaning. Let us denote such a source similarity score s src (x, x).

Based on these functions, we define the target relative score decrease as:

d tgt (y, y M (x), y M (x)) := 0 if s tgt (y, y M (x)) ≥ s tgt (y, y M (x)) stgt(y,y M (x))-stgt(y,y M (x)) stgt(y,y M (x)) otherwise (1)

The choice to report the relative decrease in s tgt makes scores comparable across different models or languages1 . For instance, for languages that are comparatively easy to translate (e.g. French-English), s tgt will be higher in general, and so will the gap between s tgt (y, y M (x)) and s tgt (y, y M (x)). However this does not necessarily mean that attacks on this language pair are more effective than attacks on a "harder" language pair (e.g. Czech-English) where s tgt is usually smaller.

We recommend that both s src and d tgt be reported when presenting adversarial attack results. However, in some cases where a single number is needed, we suggest reporting the attack's success S := s src + d tgt . The interpretation is simple: S > 1 ⇔ d tgt > 1 -s src , which means that the attack has destroyed the target meaning (d tgt ) more than it has destroyed the source meaning (1 -s src ).

2.2. SIMILARITY METRICS:

Throughout §2.1, we have not given an exact description of the semantic similarity scores s src and s tgt . Indeed, automatically evaluating the semantic similarity between two sentences is an open area of research and it makes sense to decouple the definition of adversarial examples from the specific method used to measure this similarity. In this section, we will discuss manual and automatic metrics that may be used to calculate it.

2.2.1. HUMAN JUDGMENT:

Judgment by speakers of the language of interest is the de facto gold standard metric for semantic similarity and specific criteria for such as adequacy/fluency (Ma & Cieri, 2006) , acceptability (Goto et al., 2013) , and 6-level semantic similarity (Cer et al., 2017) have been used in evaluations of MT and sentence embedding methods. In the context of adversarial attacks, we propose the following 6level evaluation scheme, which is motivated by previous measures, but designed to be (1) symmetric, like Cer et al. (2017) , ( 2) and largely considers meaning preservation but at the very low and high levels considers fluency of the output2 , like Goto et al. (2013) :

How would you rate the similarity between the meaning of these two sentences?

0. The meaning is completely different or one of the sentences is meaningless 1. The topic is the same but the meaning is different 2. Some key information is different 3. The key information is the same but the details differ 4. Meaning is essentially equal but some expressions are unnatural 5. Meaning is essentially equal and the two sentences are well-formed English

2.2.2. AUTOMATIC METRICS:

Unfortunately, human evaluation is expensive, slow and sometimes difficult to obtain, for example in the case of low-resource languages. This makes automatic metrics that do not require human intervention appealing for experimental research. This section describes 3 evaluation metrics commonly used as alternatives to human evaluation, in particular to evaluate translation models.3 

BLEU: (Papineni et al., 2002) is an automatic metric based on n-gram precision coupled with a penalty for shorter sentences. It relies on exact word-level matches and therefore cannot detect synonyms or morphological variations.

METEOR: (Denkowski & Lavie, 2014) first estimates alignment between the two sentences and then computes unigram F-score (biased towards recall) weighted by a penalty for longer sentences. Importantly, METEOR uses stemming, synonymy and paraphrasing information to perform alignments. On the downside, it requires language specific resources.

chrF: (Popović, 2015) is based on the character n-gram F-score. In particular we will use the chrF2 score (based on the F2-score -recall is given more importance), following the recommendations from Popović (2016) . By operating on a sub-word level, it can reflect the semantic similarity between different morphological inflections of one word (for instance), without requiring languagespecific knowledge which makes it a good one-size-fits-all alternative.

Because multiple possible alternatives exist, it is important to know which is the best stand-in for human evaluation. To elucidate this, we will compare these metrics to human judgment in terms of Pearson correlation coefficient on outputs resulting from a variety of attacks in §4.2.

3. GRADIENT-BASED ADVERSARIAL ATTACKS:

In this section, we overview the adversarial attacks we will be considering in the rest of this paper.

3.1. ATTACK PARADIGM:

We perform gradient-based attacks that replace one word in the sentence so as to maximize an adversarial loss function L adv , similar to the substitution attacks proposed in (Ebrahimi et al., 2018b) .

3.1.1. GENERAL APPROACH:

Precisely, for a word-based translation model M ,4 and given an input sentence w 1 , . . . , w n , we find the position i * and word w * satisfying the following optimization problem:

arg max 1≤i≤n, ŵ∈V L adv (w 0 , . . . , w i-1 , ŵ, w i+1 , . . . , w n ) (2)

Where L adv is a differentiable function which represents our adversarial objective. Using the first order approximation of L adv around the original word vectors w 1 , . . . , w n5 , this can be derived to be equivalent to optimizing arg max 1≤i≤n, ŵ∈V [ ŵw i ] ∇ wi L adv .

The above optimization problem can be solved by brute-force in O(n|V|) space complexity, whereas the time complexity is bottlenecked by a |V|d times nd matrix multiplication, which is not more computationally expensive than computing logits during the forward pass of the model. Overall, this naive approach is sufficiently fast to be conducive to adversarial training. We also found that the attacks benefited from normalizing the gradient by taking its sign.

Extending this approach to finding the optimal perturbations for more than 1 substitution would require exhaustively searching over all possible combinations. However previous work (Ebrahimi et al., 2018a) suggests that greedy search is a good enough approximation.

3.1.2. THE ADVERSARIAL LOSS L ADV:

We want to find an adversarial input x such that, assuming that the model has produced the correct output y 1 , . . . , y t-1 up to step t -1 during decoding, the probability that the model makes an error at the next step t is maximized. In the log-semiring, this translates into the following loss function:

EQUATION

3.2. ENFORCING SEMANTICALLY SIMILAR ADVERSARIAL INPUTS:

As a contrast to these previous methods, which don't consider meaning preservation, we propose simple modifications of the approach presented in §3.1 to create adversarial perturbations that are more likely to preserve meaning. The basic idea is to restrict the possible word substitutions to similar words. We compare two sets of constraints:

kNN: This constraint enforces that the word be replaced only with one of its 10 nearest neighbors in the source embedding space. This has two effects: first, the replacement will be likely semantically related to the original word (if words close in the embedding space are indeed semantically related, as hinted by Table 1 ). Second, it ensures that the replacement's word vector is close enough to the original word vector that the first order assumption is more likely to be satisfied.

CharSwap: This constraint requires that the substituted words must be obtained by swapping characters. Word internal character swaps have been shown to not affect human readers greatly (Mc-Cusker et al., 1981) , hence making them likely to be meaning-preserving. Moreover we add the additional constraint that the substitution must not be in the vocabulary, which will likely be particularly meaning-destroying on the target side for the word-based models we test here. In such cases where word-internal character swaps are not possible or can't produce out-of-vocabulary (OOV) words, we resort to the naive strategy of repeating the last character of the word. The exact procedure used to produce this kind of perturbations is described in appendix A.1.

In contrast, we refer the base attack without constraints as Unconstrained hereforth. Table 1 gives qualitative examples of the kind of perturbations generated under the different constraints. 

4. EXPERIMENTS:

Our experiments serve two purposes. First, we examine our proposed framework of evaluating adversarial attacks ( §2), and also elucidate which automatic metrics correlate better with human judgment for the purpose of evaluating adversarial attacks ( §4.2). Second, we use this evaluation framework to compare various adversarial attacks and demonstrate that adversarial attacks that are explicitly constrained to preserve meaning receive better assessment scores ( §4.3).

4.1. EXPERIMENTAL SETTING:

Data: Following previous work on adversarial examples for seq2seq models (Belinkov & Bisk, 2018; Ebrahimi et al., 2018a) , we perform all experiments on the IWSLT2016 dataset (Cettolo et al., 2016) in the {French,German,Czech}→English directions (fr-en, de-en and cs-en). We compile all previous IWSLT test sets before 2015 as validation data, and keep the 2015 and 2016 test sets as test data. The data is tokenized with the Moses tokenizer (Koehn et al., 2007) . The exact data statistics can be found in Appendix A.2. We only keep the 40,000 most frequent word and replace all others with <unk> tokens.

MT Models::

We perform experiments with two common neural machine translation (NMT) models.

The first is an LSTM based encoder-decoder architecture with attention (Luong et al., 2015) . It uses 2-layer encoders and decoders, and dot-product attention. We set the word embedding dimension to 300 and all others to 500. The second model is a self-attentional Transformer (Vaswani et al., 2017) , with 6 1024-dimensional encoder and decoder layers and 512 dimensional word embeddings.

Both the models are trained with Adam (Kingma & Ba, 2014), dropout (Srivastava et al., 2014 ) of probability 0.3 and label smoothing (Szegedy et al., 2016) with value 0.1. During inference, we replace <unk> tokens in the translated sentences with the source words with the highest attention value. The full experimental setup and source code are at redacted_url.

Automatic Metric Implementations::

To evaluate both sentence and corpus level BLEU score, we first de-tokenize the output and use sacreBLEU6 (Post, 2018) with its internal intl tokenization, to keep BLEU scores agnostic to tokenization. We compute METEOR using the official implementation7 . ChrF is reported with the sacreBLEU implementation on detokenized text with default parameters.

4.2. CORRELATION OF AUTOMATIC METRICS WITH HUMAN JUDGMENT:

We first examine which of the automatic metrics listed in §2.2 correlates more with human judgment for our adversarial attacks. For this experiment, we restrict the scope to the case of the BiLSTM model on fr-en. For the French side, we randomly select 900 sentence pairs (x, x) from the validation set, 300 for each of the constraints Unconstrained, kNN and CharSwap. To vary the level of perturbation, the 300 pairs contain an equal amount of perturbed input obtained by substituting 1, 2 and 3 words. On the English side, we select 900 pairs of reference translations and translations of adversarial input (y, y M (x)) with the same distribution of attacks as the source side, as well as 300 (y, y M (x)) pairs (to include translations from original inputs). This amounts to 1,200 sentence pairs in the target side.

These sentences are sent to English and French speaking annotators to be rated according to the guidelines described in §2.2.1. Each sample (a pair of sentences) is rated by two independent evaluators. If the two ratings differ, the sample is sent to a third rater (an auditor and subject matter expert) who makes the final decision. 2 . As evidenced by the results, chrF exhibits higher correlation with human judgment, followed by METEOR and BLEU. This is true both on the source side (x vs x) and in the target side (y(x) vs y M (x)). We evaluate the statistical significance of this result using a paired bootstrap test for p < 0.01. Notably we find that chrF is significantly better than METEOR in French but not in English. This is not too unexpected because METEOR has access to more language-dependent resources in English (specifically synonym information) and thereby can make more informed matches of these synonymous words and phrases. Moreover the French source side contains more "character-level" errors (from Char-Swap attacks) which are not picked-up well by word-based metrics like BLEU and METEOR.

Thus, in the following sections, we report attack results both in terms of chrF in the source (s src ) and relative decrease in chrF (RDchrF) in the target (d tgt ).

4.3. ATTACK RESULTS:

We can now compare attacks under the three constraints Unconstrained, kNN and CharSwap and draw conclusions on their capacity to preserve meaning in the source and destroy it in the target. Attacks are conducted on the validation set using the approach described in §3.1 with 3 substitutions (this means that each adversarial input is at edit distance at most 3 from the original input). Results are reported in Table 3 (on a scale of 1 to 100 for readability). To give a better idea of how the different variables (language pair, model, attack), we give a graphical representation of these same results in Figure 1 . The rest of this section discusses the implication of these results. Adding Constraints Helps Preserve Source Meaning: Comparing the kNN and CharSwap rows to Unconstrained in the "source" section of Table 3 clearly shows that constrained attacks have a positive effect on meaning preservation. Beyond validating our assumptions from §3.2, this shows that source chrF is useful to carry out the comparison in the first place. To give a point of reference, results from the manual evaluation carried out in §4.2 show that that 90% of the French sentence pairs to which humans gave a score of 4 or 5 in semantic similarity have a chrF > 78.

Different Models are not Equal in the Face of Adversity: Inspection of the target-side results yields several interesting observations. First, the high RDchrF of CharSwap is yet another indication of word-based model's known shortcomings when presented with words out of their training vocabulary, even with <unk>-replacement. Second, and perhaps more interestingly, Transformer models appear to be less robust to small embeddings perturbations (kNN attacks) compared to BiL- STMs. Although the exploration of the exact reasons for this phenomenon is beyond the scope of this work, this is a good example that RDchrF can shed light on the different behavior of different architectures when confronted with adversarial input. Overall, we find that the CharSwap constraint is the only one that consistently produces attacks with > 1 average success according to Table 3 . 

5. ADVERSARIAL TRAINING WITH MEANING-PRESERVING ATTACKS:



5.1. ADVERSARIAL TRAINING:

Adversarial training augments the training data with adversarial examples. Formally, in place of the negative log likelihood (NLL) objective on a sample x, y, L N LL (x, y) = N LL(x, y), the loss function is replaced with an interpolation of the NLL of the original sample x, y and an adversarial sample x, y: L (x, y) = (1 -α)N LL(x, y) + αN LL(x, y). Ebrahimi et al. (2018a) suggest that while adversarial training improves robustness to adversarial attacks, it can be detrimental to test performance on non-adversarial input. We investigate whether this is still the case when adversarial attacks are largely meaning-preserving.

In our experiments, we generate x (under the CharSwap constraint) on the fly at each training step. To maintain training speed we apply only one substitution to produce x from x. Although this results in "weaker" adversaries, we find that this makes training time less than 2× slower than normal training. In addition, we compare adversarial training with adversarial perturbations chosen with the gradient-based approach from §3.1, CharSwap-adv to a baseline where the perturbation is chosen uniformly at random, CharSwap-rand. Test performance on non-adversarial input is reported in Table 5 . In keeping with the rest of the paper, we primarily report chrF results, but also show the standard BLEU as well. We observe no significant change in test performance on non-adversarial data. However, a look at the RDchrF of CharSwap attacks on the validation data (Table 6 ) shows that adversarial training clearly has a positive influence on robustness to adversarial attacks.

Hence, we can safely conclude that adversarial training with CharSwap attacks improves robustness while not impacting test performance in this setting. This is likely because CharSwap adversarial inputs are more constrained and less at risk of changing the meaning of the source sentence, thereby creating a spurious training sample (x, y) where y is not an acceptable translation of x.

6. RELATED WORK:

Following seminal work on adversarial attacks by Szegedy et al. (2013) , Goodfellow et al. (2014) introduced gradient-based attacks and adversarial training. Since then, a variety of attack (Moosavi-Dezfooli et al., 2016) and defense (Cissé et al., 2017; Kolter & Wong, 2017) mechanisms have been proposed. Adversarial examples for NLP specifically have seen attacks on sentiment (Papernot et al., 2016; Samanta & Mehta, 2017; Ebrahimi et al., 2018b) , malware (Grosse et al., 2016) , gender (Reddy & Knight, 2016) or toxicity (Hosseini et al., 2017) classification to cite a few.

In MT methods have been proposed to attack word-based (Zhao et al., 2018; Cheng et al., 2018) and character-based (Belinkov & Bisk, 2018; Ebrahimi et al., 2018a) models. However these works sidestep the question of meaning preservation in the source: they mostly focus on target side evaluation. Finally there is work centered around meaning-preserving adversarial attacks for NLP via paraphrase generation (Iyyer et al., 2018) or rule-based approaches (Jia & Liang, 2017; Ribeiro et al., 2018; Naik et al., 2018) . However the proposed attacks are highly engineered and focused on English.

7. CONCLUSION:

This paper highlights the performance of meaning-preserving adversarial perturbations for NLP models (with a focus on seq2seq). We proposed a general evaluation framework for adversarial perturbations and compared various automatic metrics as alternatives to human judgment to instantiate this framework. We then confirmed that, in the context of MT, "naive" attacks do not preserve meaning in general, and proposed alternatives to remedy this issue. Finally, we have shown the utility of adversarial training in this paradigm. We hope that this helps future work in this area of research to evaluate meaning conservation more consistently.

:

Note that we do not allow negative dtgt to keep all scores between 0 and 1.

This is important to rule out nonsensical sentences and distinguish between clean and "noisy" (e.g. typos, non-native speech. . . ) paraphrases.

Note that other metrics of similarity are certainly applicable within the overall framework of §2.2.1, but we limit our examination in this paper to the three noted here.

Alternatively, we could have chosen character-based(Lee et al., 2017) or subword-based(Sennrich et al., 2016) models. For character-based models the formulation would have been largely similar. For subword-based models, additional difficulty would be introduced due to changes to the input resulting in different subword segmentations. While these challenges are interesting, they are beyond the scope of the current work.

More generally we will use the bold w when talking about the embedding vector of word w

https://github.com/mjpost/sacreBLEU

http://www.cs.cmu.edu/ ˜alavie/METEOR/

Figures and Tables:

Figure 1: Graphical representation of the results in Table 3. High source chrF and target RDchrF (upper-right corner) indicates a good attack.

Examples of different adversarial inputs. The substituted word is highlighted.

Table content: <table><tr><td>Original</td><td>Pourquoi faire cela ?</td></tr><tr><td>English gloss</td><td>Why do this?</td></tr><tr><td colspan="2">Unconstrained construisant (English: building) faire cela ?</td></tr><tr><td>kNN</td><td>interrogez (English: interrogate) faire cela ?</td></tr><tr><td>CharSwap</td><td>Puorquoi (typo) cela ?</td></tr><tr><td>Original</td><td>Si seulement je pouvais me muscler aussi rapidement.</td></tr><tr><td>English gloss</td><td>If only I could build my muscle this fast.</td></tr><tr><td colspan="2">Unconstrained Si seulement je pouvais me muscler etc rapidement.</td></tr><tr><td>kNN</td><td>Si seulement je pouvais me muscler plsu (typo for "more") rapidement.</td></tr><tr><td>CharSwap</td><td>Si seulement je pouvais me muscler asusi (typo) rapidement.</td></tr></table>

Correlation of automatic metrics to human judgment of adversarial source and target sentences. " * " indicates that the correlation is significantly better than the next-best one.

Table content: <table><tr><td colspan="3">Language BLEU METEOR</td><td>chrF</td></tr><tr><td>French</td><td>0.415</td><td>0.440</td><td>0.586  *</td></tr><tr><td>English</td><td>0.357</td><td>0.478  *</td><td>0.497</td></tr></table>

Target RDchrF and source chrF scores for all the attacks.

Table content: <table><tr><td/><td>Model</td><td/><td>BiLSTM</td><td/><td/><td>Transformer</td><td/></tr><tr><td/><td colspan="4">Language pair cs-en de-en fr-en</td><td colspan="3">cs-en de-en fr-en</td></tr><tr><td/><td>Original chrF</td><td>45.68</td><td>49.43</td><td>57.49</td><td>47.66</td><td>51.08</td><td>58.04</td></tr><tr><td>Target</td><td colspan="4">Unconstrained 25.37% 25.53% 25.59% CharSwap 24.11% 24.93% 23.60%</td><td colspan="3">25.24% 24.99% 24.68% 21.58% 23.23% 21.74%</td></tr><tr><td/><td>kNN</td><td colspan="3">14.99% 15.59% 15.22%</td><td colspan="3">20.73% 19.96% 18.58%</td></tr><tr><td/><td>Unconstrained</td><td>70.13</td><td>72.39</td><td>74.29</td><td>69.03</td><td>71.93</td><td>73.22</td></tr><tr><td>Source</td><td>CharSwap</td><td>82.65</td><td>84.40</td><td>86.62</td><td>84.12</td><td>85.97</td><td>87.01</td></tr><tr><td/><td>kNN</td><td>78.08</td><td>78.10</td><td>77.62</td><td>74.94</td><td>77.92</td><td>77.88</td></tr></table>

Table 4 contains two qualitative examples of this attack on the BiLSTM in fr-en.

Table content: <table/>

Example of CharSwap attacks on the fr-en BiLSTM. The first example is a successful attack (high source chrF and target RDchrF) whereas the second is not.

Table content: <table><tr><td colspan="2">Successful attack (source chrF = 80.89, target RDchrF = 84.06)</td></tr><tr><td>Original</td><td>Ils le réinvestissent directement en engageant plus de procès.</td></tr><tr><td colspan="2">Adversarial source Ilss le réinvestissent dierctement en engagaent plus de procès.</td></tr><tr><td>Reference</td><td>They plow it right back into filing more troll lawsuits.</td></tr><tr><td>Base output</td><td>They direct it directly by engaging more cases.</td></tr><tr><td colspan="2">Adversarial output .. de plus.</td></tr><tr><td colspan="2">Unsuccessful attack (source chrF = 54.46, target RDchrF = 0.00)</td></tr><tr><td>Original</td><td>C'était en Juillet 1969.</td></tr><tr><td colspan="2">Adversarial source C'étiat en Jiullet 1969.</td></tr><tr><td>Reference</td><td>This is from July, 1969.</td></tr><tr><td>Base output</td><td>This was in July 1969.</td></tr><tr><td colspan="2">Adversarial output This is. in 1969.</td></tr></table>

chrF (BLEU) scores on the original test set before/after adversarial training.

Table content: <table><tr><td>Model</td><td>BiLSTM</td><td>Transformer</td></tr><tr><td>Language pair</td><td>cs-en de-en fr-en</td><td>cs-en de-en fr-en</td></tr><tr><td>Base</td><td>44.21 (22.89) (28.61) (35.28) 49.30 55.67</td><td>45.95 (24.52) (29.90) (36.00) 50.77 56.41</td></tr><tr><td>CharSwap-adv</td><td>44.15 (23.38) (28.55) (35.44) 49.20 55.72</td><td>45.92 (24.67) (29.97) (35.91) 50.73 56.21</td></tr><tr><td>CharSwap-rand</td><td>44.07 (23.01) (28.49) (35.26) 49.14 55.79</td><td>46.04 (24.65) (29.81) (36.19) 50.75 56.72</td></tr></table>

Robustness to CharSwap attacks with/without adversarial training. Lower is better.

Table content: <table><tr><td>Model</td><td>BiLSTM</td><td>Transformer</td></tr><tr><td>Language pair</td><td>cs-en de-en fr-en</td><td>cs-en de-en fr-en</td></tr><tr><td>Base</td><td>24.11% 24.93% 23.60%</td><td>21.58% 23.23% 21.74%</td></tr><tr><td>CharSwap-adv</td><td>21.53% 21.70% 18.77%</td><td>18.75% 19.72% 18.68%</td></tr><tr><td colspan="2">CharSwap-rand 22.29% 22.48% 19.80%</td><td>19.35% 19.32% 17.49%</td></tr><tr><td>5.2 RESULTS</td><td/><td/></tr></table>

Peer review (iclr2019):
 Summary: Proposes a framework for performing adversarial attacks on an NMT system in which perturbations to a source sentence aim to preserve its meaning, on the theory that an existing reference translation will remain valid if this is done. Given source and target metrics for measuring similarity, an attack is deemed successful if the source difference is smaller than the relative decrease in target similarity to the reference. A first experiment measures correlation with human judgements of similarity between original and perturbed sentences, and concludes that chrF is better than BLEU and METEOR for this purpose. Next, standard gradient-based adversarial attacks are carried out, replacing the three tokens that result in the biggest drop in (approximate) reference probability, either 1) with no constraints, 2) constrained to character swaps of the original token, or 3) constrained be among the 10 closest embeddings to the original token. In comparisons on three language pairs from IWSLT, the constrained attacks are found to preserve meaning and yield more successful attacks according to the current framework. The Transformer architecture was also found to deal less well with attacks under the 10-closest embedding constraint. Finally, adversarial training with the character-swap constraint confers some robustness to this attack, without degrading performance on normal text. I think it is a good idea to formalize a method for carrying out and assessing adversarial attacks, but the framework proposed here seems too narrow, as it excludes adversarial inputs that are sensible but not a close perturbation of an existing source/reference pair, or ones that contain varying amounts of noise. It is more difficult to measure output quality for such attacks, but that doesn’t seem like a good reason for excluding them from what is intended to be a general framework. Note also that “more difficult” doesn’t mean impossible, since good attacks can produce severely degraded output that is relatively easy to detect. I found some of the methodology questionable. Limiting source perturbations to character swaps and neighbors in embedding space, then using automatic metrics to measure semantic distance seems both unnecessary and unlikely to succeed. Unnecessary because knowing the class of perturbation already gives you a lot of information about semantic distance. Unlikely to succeed because automatic metrics are too coarse to reliably distinguish among different perturbations. This is particularly obvious in the case of using character ngram distance (chrF) to determine which character swaps preserve meaning best. The experiments that support the viability of automatic metrics in 4.2 do so by measuring correlation with human judgment when the number of perturbed tokens varies from 1 to 3. I think the good correlation is likely due to the metrics being able to detect that, eg, changing 3 tokens makes things worse than changing only one. To be convincing, the experiments would have to be repeated with number of perturbations fixed at 3, to match the setting in the remaining experiments. Apart from the interesting observation about the Transformer’s performance on embedding-neighbor attacks mentioned above, it is difficult to know what conclusions to draw from the experiments. In 4.3 it seems obvious a priori that perturbations intended to be relatively meaning preserving would indeed preserve meaning better than unconstrained ones. Similarly, it is not surprising that character swaps that by design produce an OOV token will cause more damage than choosing a near neighbor in embedding space. In 5.3, training with OOVs (resulting from character swaps) is of course not likely to hurt performance on test sets containing few OOVs, and, as is known from previous work, it will improve robustness to the same kind of noise. A final comment about the experiments is that word-based systems are not state of the art, and it isn’t clear how much we could expect any conclusions to carry over to sub-word models. To conclude, although this is an interesting initiative, both the framework and the methodology need to be tightened up. Details: End of 2.1: this would be easier to interpret if you had previously specified the allowed range for s_src. 3.2 For kNN, being semantically related doesn’t imply that the relationship is synonymy, as would be required for meaning preservation. It also doesn’t imply that the substitution will be grammatical, which could jeopardize meaning preservation even if the words are synonyms. CharSwap seems odd. If you’re just going to replace a work with an OOV symbol in any case, why go to the trouble of swapping characters? No matter what actual semantic shift is caused by the swap, the model will always see exactly the same representation. 4.1 “Following previous work on adversarial examples for seq2seq models (Belinkov & Bisk, 2018; Ebrahimi et al., 2018a)” - this is misleading: Ebrahimi et al only work with classification, and don’t use IWLST. 4.1 Should mention the size of the training sets in this section. Table 1, first sentence, CharSwap example omits “faire”. 4.3, “Adding Constraints Helps Preserve…” last sentence: but here you need to reason in the opposite direction. 5.2 It would be good to also give absolute scores for table 6, so we can judge how much the systems actually benefited, and whether these gains were statistically significant.