TRANSLOWDOWN : EFFICIENCY ATTACKS ON NEURAL MACHINE TRANSLATION SYSTEMS
Year: 2021

Abstract:
Neural machine translation (NMT) systems have received massive attention from academia and industry. Despite a rich set of work focusing on improving NMT systems’ accuracy, the less explored topic of efficiency is also important to NMT systems because of the real-time demand of translation applications. In this paper, we observe an inherent property of NMT system, that is, NMT systems’ efficiency is related to the output length instead of the input length. Such property results in a new attack surface of NMT system—an adversary can slightly changing inputs to incur significant amount of redundant computations in NMT systems. Such abuse of NMT systems’ computational resources is analogous to the denial-ofservice attacks. Abuse of NMT systems’ computing resources will affect the service quality (e.g., prolong response to users’ translation requests) and even make the translation service unavailable (e.g., running out resource such as batteries of mobile devices). To further the understanding on such efficiency-oriented threats and raise community’s concern on the efficiency robustness of NMT systems, we propose a new attack approach, TranSlowDown, to test the efficiency robustness of NMT systems. To demonstrate the effectiveness of TranSlowDown, we conduct a systematic evaluation on three public-available NMT systems: Google T5, Facebook Fairseq, and Helsinki-NLP translators. The experimental results show that TranSlowDown can increase NMT systems’ response latency up to 1232% and 1056% on Intel CPU and Nvidia GPU respectively by inserting only three characters into existing input sentences. Our results also show that the adversarial examples generated by TranSlowDown can consume more than 30 times battery power than the original benign example. Such results suggest that further research is required for protecting NMT systems against efficiency-oriented threats.

heading: 1 INTRODUCTION
text: Recently, Neural Machine Translation (NMT) systems have received massive attention from academia and industry (Bahdanau et al., 2015; Kalchbrenner et al., 2016; Vaswani et al., 2017; Belinkov & Bisk, 2018). NMT systems overcome many weaknesses of traditional phrase-based translation models and can capture long dependencies in sentences; thus, they are widely used in commercial translation systems. For example, Microsoft has deployed the NMT systems in many commercial products since 2016 (Hassan Awadalla et al., 2017; 2018a;b; Gu et al., 2018); Google Translate claims to have translated over 100 billion words daily in 109 languages (Turovsky, 2016; Caswell & Liang, 2020; Pitman, 2021).
For NMT systems, efficiency is critical because of translation applications’ real-time demand (Tu et al., 2017; Guo et al., 2019; Xia et al., 2019). However, it is unknown whether NMT systems are robust against efficiency-oriented adversarial pressure. Despite a rich set of works (Cheng et al., 2020; Jin et al., 2020; Belinkov & Bisk, 2017; Cheng et al., 2019; Wu et al., 2018; Yang et al., 2017) evaluate NMT systems’ accuracy robustness through maximizing the errors, understanding the NMT systems’ efficiency robustness has not received much attention.
In order to study NMT systems’ efficiency robustness, we first need to figure out what factors will affect NMT’s efficiency. In this paper, we observe a natural property of NMT systems, i.e.,, NMT’s computational consumption is volatile for different inputs because NMT systems invoke the underlying decoder with non-deterministic numbers of iterations to generate output tokens (Vaswani et al.,
2017; Liu et al., 2020). This property exposes a new vulnerability of NMT systems, an adversary can design specific inputs to cause enormous computation overhead in NMT systems, thus wasting the computational resources of NMT systems. Based on such observation, in this paper, we investigate: Can adversary make slight modification on textual inputs to increase NMT systems’ computational consumption and decrease NMT systems’ efficiency? If so, how severe the efficiency degradation can be?
New Vulnerability. We consider a new attack surface of NMT models, analogous to the vulnerabilities leading to the denial of service (DoS) attacks (Zhang et al., 2015; Qin et al., 2018) that have plagued the security committee for decades. Specifically, the adversary’s goal is to decrease the victim NMT system’s efficiency (i.e., response latency, energy consumption) with unnoticeable perturbations on the inputs to the system. This attack will result in devastating consequences for many real-world applications. For example, abusing computational resources on commercial machine translation service providers (e.g., Huggingface (Wolf et al., 2020)). An efficiency attack will cause enormous redundant computational resources and affect the service quality of benign users. Furthermore, abusing computational resources on mobile devices or IoT devices might shorten the battery charge cycle and result in the unavailability of the devices.
New Attack. In this paper, we evaluate NMT systems’ efficiency robustness by generating adversarial inputs that consume much greater amount of computation resources than normal inputs on NMT systems. Specifically, we propose TranSlowDown, which is based on the observation that the source of vulnerability is that NMT systems’ efficiency is related to the output length instead of the input length. Specifically, NMT systems iteratively compute the output token until the systems generate the particular end of sentence (EOS) token. Thus, we design a novel algorithm to search for a minimal perturbation to delay the appearance of EOS. Specifically, TranSlowDown can generate both token-level and character-level perturbations. After applying the minimal perturbation on the benign sentences, the probability of EOS of output token will decrease, resulting longer output that costs NMT systems more computational resources.
Evaluation. To evaluate the effectiveness of TranSlowDown, we use TranSlowDown to attack three public-available NMT systems (i.e., Google T5 (Raffel et al., 2019), Facebook FairSeq (Liu et al., 2020), and Helsinki-NLP), and compare TranSlowDown against two accuracy-based attack algorithms. We first measure the floating-point operations (FLOPs) and response latency of the victim NMT systems when translating benign and adversarial examples. Then, we apply I-FLOPs and I-Latency (defined in Equation 4) to quantify the severity of efficiency degradation caused by generated adversarial examples. The evaluation results show: for the token-level attack, TranSlowDown generate adversarial examples increase victim NMT systems’ FLOPs, CPU latency, GPU latency up to 2131.92%, 2403.21%, and 2054.83% respectively with only perturbing two tokens in sentences; for the character-level attack, TranSlowDown generated adversarial examples increase victim NMT systems’ FLOPs, CPU latency, GPU latency up to 1102.86%, 1232.71%, and 1056.88% respectively with only inserting three characters in sentences. We also form a real-world case study to demonstrate the negative impact of the efficiency threat. Specifically, we deploy Google’s T5 on a mobile device and investigate how adversarial examples affect the mobile device’s battery power consumption. The results show that by inserting only one character into the benign sentence, the adversarial examples generated by TranSlowDown can consume more than 30 times battery power than the original benign example. Our results suggest that further research is required for protecting NMT systems against this emerging security threat.

heading: 2 BACKGROUND & RELATED WORK

heading: 2.1 NEURAL MACHINE TRANSLATION SYSTEMS
text: Neural machine translation (NMT) (Vaswani et al., 2017; Liu et al., 2020) systems apply neural networks to approximate the conditional probability P (Y |X), where X = [x1, x2, · · · , xm] is the source token sequences and Y = [y1, y2, · · · , yn] is the target token sequences. As shown in Figure 1, a typical NMT system consists of an encoder fen(·) and a decoder fde(·). The encoder encodes the source input X into hidden representation H , then the decoder starts from a special token (SOS), and iteratively accesses H for an auto-regressive generation of each token yi until the end of sequence token (EOS) is reached. An important observation from the NMT working mechanisms is that NMT will iteratively running the decoder fde to generate a output token until EOS is
reached. Thus, if the generated output sequence is longer, the NMT will consume more computational resources and becomes less efficient. In our example, the NMT system run the decoder four times to generate the output.
2.2 DNN’S EFFICIENCY
Recently, the efficiency of deep neural networks (DNNs) has raised a huge concern because of their substantial inference-time computational costs. To reduce DNN’ inference-time computational costs and make it feasible to applying DNNs for real-time applications, many existing work has been proposed. There are two main techniques: The first type (Howard et al. (2017); Zhang et al. (2018)) of the techniques prune the DNNs offline to identify important neurons and
remove the unimportant neurons. After pruning, the smaller size DNNs could achieve competitive accuracy with the original DNNs but require less computational costs. Another type of techniques Wang et al. (2018); Graves (2016); Figurnov et al. (2017), called input-adaptive techniques, this type of technique dynamically skip a certain part of the DNNs to reduce the number of computations in the inference time. By skipping some parts of the DNNs, the input-adaptive DNNs can balance the accuracy and computational costs. However, recent studies (Haque et al., 2020; Hong et al., 2020) show input-adaptive DNNs are not robustness against the adversary attack, which implies the input-adaptive will not save computational costs under attacks.

heading: 2.3 ADVERSARIAL ATTACKS
text: Existing work on adversarial machine learning has shown that even the state-of-the-art DNNs can be fooled by adversarial examples (Carlini & Wagner, 2017; Athalye et al., 2018). Adversarial examples are the elaborately crafted samples that apply human-unnoticeable perturbations on benign samples to maximize the target DNNs’ errors. Based on prior knowledge about victim DNNs, the generation of adversarial examples could be categorized into white-box and black-box attacks. In the white-box settings (Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2016; Kurakin et al., 2016; Carlini & Wagner, 2017; Jang et al., 2017; Madry et al., 2017; Chen et al., 2018; Rony et al., 2019), the adversary know the victim DNNs’ architecture and parameters and can compute gradients to generate adversarial examples. In the black-box settings (Brendel et al., 2017; Chen et al., 2020; Brendel et al., 2019), the adversary can also exploit a surrogate model for launching the attack.

heading: 3 ATTACK METHODOLOGY

heading: 3.1 THREAT MODEL
text: We consider an adversary who aims to decrease the efficacy of a victim NMT system. The attacker perturbs a benign sentence with unnoticeable perturbations to craft adversarial examples and feeds adversarial examples to the victim NMT. The perturbed adversarial examples will consume more computational resources of the victim NMT systems, thus impairing the translation service or make the service unavailable.
Adversary’s Capabilities. The attacker is able to modify the victim’s input samples to apply the perturbations, e.g.,, By publishing some documents contain adversarial examples on the Internet. We follow existing work (Cheng et al., 2020; Belinkov & Bisk, 2017; Li et al., 2018; Jin et al., 2020) and consider two types of perturbations: (i) token-level perturbation and (ii) character-level perturbation. To ensure imperceptibility, we limit the perturbation size that the adversary can perturb the benign inputs. In Section 4, we evaluate how different will affect the effectiveness of proposed attack.
Adversary’s Knowledge. To assess the security vulnerability of existing NMT systems, we study white-box scenarios, i.e., the attackers know the victim NMT systems architecture, parameters and the tokenizer that tokenize the word. In Section 4.4, we study more practical black-box scenarios,
i.e.,, the attacker leverage the transferability to generate adversarial examples and attack a victim NMT system without any prior knowledge about the victim.
Adversary’s Goals. The adversary’s goal is to decrease the efficacy of a victim NMT system. As we discussed in Section 2, NMT system’s efficacy is related to its’ output length. A longer output length indicates the NMT system costs more computational resources and becomes less efficacy. Thus, the adversary can achieve their goal through increasing the NMT system’s output length.
∆ = argmaxδ Len(F(x+ δ)) s.t. ||δ|| ≤ (1)
Finally, we formulate our problem of generating efficiency adversarial examples as an optimization problem. As shown in equation 1, where x is the benign input, F is the target NMT system, is the maximum adversarial perturbation, and Len(·) measure the length of a output sequence. Our attack TranSlowDown tries to search a perturbation ∆ that maximize the output length (decrease target NMT system F effiency) while smaller than the allowed perturbation (unnoticeable).
3.2 TRANSLOWDOWN ATTACK
Algorithm 1 TranSlowDown Attack Input: Benign input x, victim NMT system F(·), maximum perturbation Output: Adversarial examples x′
1: x′ ⇐ x Initialize x′ with x 2: while ||x′ − x|| ≤ do 3: for each tki ∈ x′ do 4: Compute gi according to equation 2 5: end for 6: TK⇐ Sort(tk1, tk2, · · · , tkm) according to gi 7: L = GeneratePerturbation(TK, x′,F(·)) 8: [tk,∆] = SelectBestPerturbation(L, x′,F(·)) 9: x′ ⇐ replace tk with ∆ in x′
10: end while 11: return x′
Our attack algorithm is shown in Algorithm 1, which iteratively performs the following three main steps: (i) find important tokens, (ii) generate possible perturbation, and (iii) select optimal perturbation, until the generated adversarial examples reach the maximum perturbation.
Find Important Tokens (line 3 to 6):
Given a benign input x = [tk1, · · · , tkm], the first step is to find each tokens importance to NMT systems’ efficiency. As we discussed in Section 2, NMT systems’ efficiency is related to its’ output length, and the output length is determined by the probability of EOS tokens. Then our objective is to decrease the probability of the EOS token to reduce NMT’s efficiency. Formally, let NMT system’s output probability be a sequence of vectors, i.e., F(x) = [p1, p2, · · · , pn]. Then the probability of EOS tokens are [peos1 , p eos 2 , · · · , peosn ]. We seek to find the importance of each token tki in x to this probability sequence. We also observe that the output token sequence will affect EOS’s probability. Thus, we define the importance score of token tki as gi, shown in equation 2.
oi = argmax(pi) f(x) = 1
n n∑ i (peosi + p oi i ) gi = ∑ j ∂f(x) ∂tkji (2)
Where [o1, o2, · · · , on] is the current output token, f(x) is the probability we seek to minimize, tkji is the jth dimension of tk’s embeddings, and gi is the derivative of f(x) to ith token’s embedding.
Adversarial Perturbation Generation (line 7): After identifying important tokens, the next step is to mutate the important token with unnoticeable perturbations. In this step, we get a set of perturbation candidate L after we perturbing the most important tokens in the original input. Following existing work, we consider two kinds of perturbations, i.e.,, token-level perturbation and characterlevel perturbation. Table 1 shows some example of token-level and character-level perturbation under different perturbation size , we color the perturbation with color red.
Isrc,tgt = ∑ j (E(tgt)− E(src))× ∂f(x) ∂tkji
δ = argmaxtgt Itk,tgt; (3)
For token-level perturbation, we consider replacing the original token tk with another token δ. To compute the target token δ, we define token replace increment Isrc,tgt to measure the efficiency degradation of replacing token src to tgt. As shown in equation 3, E(·) is the function to get corresponding token’s embedding, E(tgt) − E(src) is the vector increment in embedding space. Because ∂f(x)
∂tkji indicate the sensitive of output length to each embedding dimension, Isrc,tgt repre-
sent the total benefits of replacing token src with tgt. We search the target token δ in the vocabulary to maximize the token replace increment with the source token tk.
For character-level perturbation, we consider character insert perturbation. Specifically, we insert one character c into token tk to get another token δ. The character-inset perturbation is common in the real world when typing quickly and is unnoticeable without careful examination. Because character insertion is likely to result in out-of-vocabulary (OOV), thus it is challenging to compute the token replace increment as token-level. Instead, we enumerate
possible δ after character insertion to get candidate set L. Specifically, we consider all letters and digits as the possible character c because humans can type these tokens through the keyboard, and we consider all positions the possible insertion position. Then for token tk, which contains l characters, there are (l + 1)× ||C|| perturbation candidates, where ||C|| is the size of all possible characters. Select Best Perturbation (line 8 to 9): After collecting candidate perturbations L, we select an optimal perturbation from the collected candidate sets. As our objective is searching adversarial perturbation that will produce longer output length, thus, we try all adversarial perturbation and select the optimal perturbation that produce the maximum output length.

heading: 4 EVALUATION

heading: 4.1 EXPERIMENTAL SETUP
text: Models and Datasets. As shown in Table 2, we consider the following three public NMT systems as our victim models: Google’s T5 (Raffel et al., 2019), Facebook’s Fairseq Transformer (Ng et al., 2019), and Helsinki-NLP’s H-NLP Translator (Jörg et al., 2020). For each model, we choose a specific testing dataset: (i). T5 is released by Google, it’s first pre-trained with multiple language problems, then fine-tuned on the English-German translation task. We apply English sentences from dataset ZH19 as benign examples to generate adversarial examples; (ii). Fairseq is one of the NMT models that Facebook FAIR submitted to the WMT19 shared news translation task, and it’s based on the FFN transformer architecture (Vaswani et al., 2017). We select Fairseq’s en-de model as our victim model, which is designed for the English-German translation task. We apply WMT19 test data as benign examples to generate adversarial examples; (iii). H-NLP is a seq2seq translation model, the source language is English and the target language is Chinese. We apply the sentences from Tatoeba-test.eng.zho as benign examples to generate adversarial samples.
Metrics. We apply floating-point operations (FLOPs) and response latency to measure victim NMT systems’ efficiency. FLOPs is a hardware-independent metric and is widely used to measure DNNs’ computational complexity. Higher FLOPs mean that the DNN requires more computations to handle an input, which represents less efficiency (Howard et al., 2017; Zhang et al., 2018). Response latency is a hardware-dependent metric, which can measure the victim model’s efficiency on benign and adversarial examples. High response latency indicates that the victim NMT system needs to spend
more computational resources. The higher the response latency, the worse the real-time translation quality. We measure the latency on two hardware platforms: Intel Xeon E5-2660v3 CPU and Nvidia 1080Ti GPU.
To evaluate the effectiveness of TranSlowDown, we measure the FLOPs and latency on benign and adversarial examples respectively. Specifically, we first compute the probability density function (PDF) and cumulative distribution function (CDF) of FLOPs and latency. Then, we define two metrics: I-FLOPs and I-Latency, to evaluate how the adversarial samples affect the victim NMT systems. The formal definition of I-FLOPs and I-Latency shows in equation 4, where x denotes the benign example and x′ represents the adversarial example after perturbing x, FLOPs(·) and Latency(·) are the functions to calculate the average FLOPs and latency per input character. Higher I-FLOPs and I-Latency indicate a more severe slowdown caused by the adversarial example.
I-FLOPs = FLOPs(x′)− FLOPs(x) FLOPs(x) × 100% I-Latency = Latency(x ′)− Latency(x) Latency(x) × 100%
(4)
Comparison Baseline. To the best of our knowledge, we are the first to study the attack efficiency of NMT systems, therefore no existing efficiency attack framework can be applied as the baseline. To show existing accuracy-based attacks can not be applied for evaluating NMT’s efficient robustness, we compare TranSlowDown against two accuracy-based attacks. We choose Seq2Sick (Cheng et al., 2020) as the token-level baseline. Seq2Sick can replace the tokens in benign inputs to produce adversarial translation outputs that are entirely different from benign outputs. Because Seq2Sick only works under token-level, we choose SyntheticError (Belinkov & Bisk, 2017) as the characterlevel baseline. SyntheticError minimizes the NMT system’s accuracy (BLUE score) by introducing synthetic noise. Specifically, SyntheticError introduces four character-level perturbation: Swap, Middle Random, Fully Random, and Keyboard Typo to perturb benign examples to decrease the NMT system’s BLUE scores.

heading: 4.2 TOKEN-LEVEL PERTURBATION RESULTS
text: Effectiveness of Token-level Attack. Figure 2 shows the distribution of H-NLP’s efficiency metrics under token-level (more results for T5 and Fairseq can be found in Appendix A.2). The first and second rows show the PDF and CDF 1 results respectively. In each plot, the blue area denotes the distribution of benign examples, green and red areas represent the distribution of adversarial examples generated from the comparison baseline Seq2Sick and TranSlowDown respectively. It can be clearly observed from Figure 2 that adversarial examples generated by TranSlowDown significantly change the distribution of the victim NMT system’s FLOPs and latency. In contrast, the adversarial examples from baseline have very little effect on NMT’s efficiency. The results for the other two NMT systems in the Appendix show similar trends with Figure 2. From the results, we conclude that: existing accuracy-based attacks can not be applied to evaluate the efficiency robustness of NMT systems. In contrast, our proposed attack, TranSlowDown, effectively generates
1For better presentation, we plot the CDF from one to zero.
Under review as a conference paper at ICLR 2022
H-NLP-C
adversarial examples to slow down NMT systems. Thus, TranSlowDown is effective in evaluating the efficiency robustness of NMT systems.
Severity of Token-level Attack. To quantify the severity of the proposed efficiency attack, we measure I-FLOPs and I-Latency under different perturbation sizes. From the results in Table 3, we have the following observations: (i) For all experimental subjects, the adversarial examples generated by TranSlowDown slow down the victim NMT systems by a large margin compared to the baseline method. For H-NLP, TranSlowDown adversarial examples increase the FLOPs, CPU latency, GPU latency up to 2366.09%, 2669.82%, 2321.93% respectively. (ii) As perturbation size increases, the adversarial examples generated by TranSlowDown can make the victim NMT system slower. However, this observation does not hold for the baseline method. This is because the baseline method is designed to decrease victim NMT’s accuracy, increasing adversarial perturbation may decrease NMT accuracy but does not imply to consume more computational resources. (iii) The adversarial examples generated by the baseline method can not always ensure increasing FLOPs and latency, while TranSlowDown adversarial examples increase FLOPs and latency on all settings. This result is consistent with the second observation and indicates that the baseline method is not suitable to evaluate NMT’s computational consumption robustness.

heading: 4.3 CHARACTER PERTURBATION
text: Effectiveness of Character-level Attack. Figure 3 shows the distribution of H-NLP’s efficiency metrics under character-level (more results for T5 and Fairseq can be found in Appendix A.2). For character-level attack, compared with baseline, adversarial examples generated by TranSlowDown
significantly change the distribution of the victim NMT system’s FLOPs and latency as well. The results for the other two NMT systems in the Appendix show similar trends with Figure 3. From the results, we obtain a similar conclusion with the token-level attack, i.e.,, our proposed attack, TranSlowDown, effectively generates adversarial examples to slow down NMT systems. In contrast, the accuracy-based attack can not achieve this goal.
Severity of Character-level Attack. Similar to Section 4.2, Table 4 presents the severity of character-level adversarial examples. The results show consistency with the token-level attack. For the character-level attack, the adversarial examples generated by TranSlowDown also significantly slow down the victim NMT systems compared to the baseline method. For H-NLP, TranSlowDown adversarial examples increase the FLOPs, CPU latency, GPU latency up to 1224.91%, 2669.82%, 1174.57% respectively. As perturbation size increases, the adversarial examples generated by TranSlowDown can make the victim NMT system slower, which is not reflected in the baseline results. This is consistent with the token-level attack.

heading: 4.4 THE TRANSFERABILITY OF ADVERSARIAL EXAMPLES
text: In this section, we study the transferability of efficacy adversarial examples. Even though whitebox attacks are important to expose the vulnerability, black-box attacks are more practical in the real world because they require less information about the NMT systems. We investigate whether TranSlowDown is transferable under black-box settings.
Specifically, we treat one NMT system as a target and apply another NMT system as the source to generate adversarial examples. We then feed the adversarial examples to the target NMT systems and measure the maximum I-FLOPs. Maximum I-FLOPs indicate the efficiency degradation under the worst scenario, which is important to measure the vulnerability of NMT systems. Notice the source and the target NMT systems in our experiment adopts different model architectures and are trained with different datasets. Thus, if the adversarial examples can increase the FLOPs of the target NMT system, it proves that transferability exists in our attack. The results for token-level attacks are shown in Table 5 (more results in Appendix A.3). From the results, we observe that for
all experimental settings, TranSlowDown generates adversarial examples that increase the target NMT system’s computational FLOPs to a large extend. The results imply that under the worst scenarios, the attackers can generate efficient adversarial examples even without prior knowledge about the victim NMT systems.

heading: 5 REAL WORLD CASE STUDY
text: We conduct a case study to evaluate TranSlowDown’s ability to attack real world mobile devices’ battery power.
Experiment Settings. We select Google’s T5 as our victim NMT model. We first deploy the model on Samsung Galaxy S9+, which has 6GB RAM and a battery capacity of 3500 mAh. After that, we select one sentence from the dataset ZH19 as our testing example; We then apply TranSlowDown to perturb the benign
example with character-level perturbation and obtain the corresponding adversarial example. The benign sentence and the adversarial sentence are shown in Table 6 (More results in Appendix section A.4), we color the perturbation with the color red. Notice the adversarial example insert only one character in the benign sentence. This one-character perturbation is very common in the real world because of the user’s typos. Finally, we feed the benign and adversarial examples to the deployed NMT system and measure the NMT’s energy consumption of translating benign and adversarial examples.
Experiment Results. The mobile’s battery power change is shown in Figure 4. The red line is for adversarial example, and the blue line is for benign example. The results show that the adversarial example increases the mobile device’s battery power significantly more significant than the benign example. Specifically, after 300 iterations, the adversarial example consumes 30% of the battery power, while the benign example only consumes less than 1%. The results show the vulnerability of the efficiency attack for IoT and mobile devices. Recall that the adversarial example used in our experiment only inserts one character in the benign sentence. This minimal perturbation can result from benign user typos instead
of from the adversary. Thus, the results suggest the criticality and the necessity of increasing NMT systems’ efficiency robustness.

heading: 6 DISCUSSION
text: Although there are many accuracy-based defense mechanisms to protect DNNs. However, accuracyoriented defense mechanisms are not applicable for protecting NMT systems’ efficiency robustness because of two reasons: (i) Many existing accuracy defense mechanisms require running the model with the input and use the hidden states of the model to make decision. However, running the input would defeat the purpose of an energy defense as the computation resource is already consumed. (ii) Our results in section 4 show that accuracy-based adversarial examples and efficiency-based adversarial examples belong to different distribution, thus, running accuracy-based detector can not detect efficiency-based adversarial examples successfully.

heading: 7 CONCLUSIONS
text: In this work, we study the robustness of NMT systems against adversarial efficiency attacks. Specifically, we propose TranSlowDown, an attack that introduces imperceptible adversarial perturbations to benign inputs to increase NMT’s computational complexity. We evaluate TranSlowDown on three public available NMT systems, the results show TranSlowDown generates adversarial examples decrease NMT systems’ efficiency. Our study suggests that efficiency attacks are a vulnerable, yet under-appreciated, threat against NMT systems.

heading: A APPENDIX
text: A.1 TOKEN-LEVEL RESULTS
Figure 5 and Figure 6 show the distribution of T5 and Fairseq’s efficiency metrics under token-level respectively. Similar to Section 4.2, adversarial examples generated by TranSlowDown significantly change the distribution of the victim NMT system’s FLOPs and latency while the adversarial examples from baseline have minor effect on NMT’s efficiency.
A.2 CHARACTER-LEVEL RESULTS
Figure 7 and Figure 8 show the distribution of T5 and Fairseq’s efficiency metrics under characterlevel respectively. Similar to Section 4.3, compared with the baseline, the adversarial samples generated by TranSlowDown still achieved higher FLOPs and latency.
A.3 TRANSFERABILITY RESULTS
As we mentioned in Section 4.4, we investigate whether TranSlowDown is transferable under black-box settings. The results for character-level attacks are shown in Table 7, which is consistent with the result obtained in the token-level experiment. The results further prove that the attacker can generate effective adversarial examples even without prior knowledge about the victim NMT system.
T5-W
Under review as a conference paper at ICLR 2022
T5-W
A .4
C A
S E
S T
U D
Y R
E S
U LT
S
In th
is se
ct io
n, w
e pu
tm or
e ge
ne ra
te d
ef fic
ie nc
y ad
ve rs
ar ia
le xa
m pl
es fo
ra ll
ou rv
ic tim
N M
T sy
st em
s.
Ta bl
e 8:
T he
be ni
gn an
d ad
ve rs
ar ia
le xa
m pl
es un
de rd
iff er
en ts
et tin gs A tt ac k Ty pe N M T B en ig n A dv er sa ri al 1
To ke
n
H -N
L P
L et
m e
se e.
哎 m
e se
e
Ia m
he re
. Ia
m go
in g.
H e
is ba
ck .
H e
is 视
Fa ir
Se q
co nfi
gu re
,v er
if y,
an d
tr ou
bl es
ho ot
vt p
co nfi
gu re
,b eb
el us
,a nd
tr ou
bl es
ho ot
vt p
fin ge
rpo
in tin
g po
lit ic
ia ns
an d
ch es
tbe
at in
g na
tio na
lis ts
cl as
sm at
es -p
oi nt
in g
po lit
ic ia
ns an
d ch
es t-
be at
in g
na tio
na lis
ts
in th
e tw
o na
tio ns
w ill
m ak
e ra
tio na
ld is
cu ss
io n
ne ar
ly im
po ss
ib le
in th
e tw
o na
tio ns
w ill
m ak
e ra
tio na
ld is
cu ss
io n
ne ar
ly im
po ss
ib le
.
T 5
T he
C om
m is
si on
’s re
po rt
se ts
ou ta
de ta
ile d
an al
ys is
of al
lt he
re pl
ie s.
T he
C om
m is
si on
’s re
po rt
se ts
ou ta
de ta
ile d
an al
ys is
of al
lt he
re pl
ie s
G ef
ah r
T he
O ct
ob er
E ur
op ea
n C
ou nc
il m
ee tin
g w
ill re
tu rn
to th
e is
su e
of m
ig ra
tio n.
T he
O ct
ob er
E ur
op ea
n C
ou nc
il m
ee tin
g w
ill re
tu rn
to th
e is
su e
of m
ig ra
tio n
no t
on z/
os th
e m
ax im
um le
ng th
of th
e at
tr ib
ut e
is 25
6 by
te s.
on al
lo th
er pl
at fo
rm s
it is
10 24
by te
s. on
z pr
es en
ce os
th e
m ax
im um
le ng
th of
th e
at tr
ib ut
e is
25 6
by te
s. on
al lo
th er
pl at
fo rm
s it
is 10
24 by
te s.
C ha
ra ct
er
H -N
L P
Y ou
ha ve
he ar
td is
ea se
. Y
ou ha
ve he
ar tO
di se
as e.
T hi
s is
m y
qu es
tio n.
T hW
is is
m y
qu es
tio n.
I’ m
a lit
tle co
nf us
ed .
I’ m
a lit
t$ le
co nf
us ed
.
Fa ir
Se q
Pr es
id en
tJ un
ck er
:‘ E
ur op
e ne
ed s
a ge
nu in
e se
cu ri
ty un
io n’
Pr es
id en
tJ un
ck er
:‘ E
ur op
e ne
ed s
a ge
nu in
e se
cu ri
ty un
i( on
’”
st ep
up th
e co
op er
at io
n on
re tu
rn an
d re
ad m
is si
on .
st ep
up th
e co
op er
at io
n on
re tu
rn an
d re
ad m
is 0s
io n.
In th
e re
fu ge
e cr
is is
,T ur
ke y
an d
th e
E U
w al
k to
ge th
er an
d w
or k
to ge
th er
. In
Y th
e re
fu ge
e cr
is is
,T ur
ke y
an d
Y th
e E
U w
al k
to ge
Y th
er an
d w
or k
to ge
Y th
er .
T 5
lo ad
,p er
fo rm
an ce
,o rs
ca la
bi lit
y lo
ad ,p
er f”
or m
an ce
,” or
sc al
ab ili
ty
w ill
he ll
sw al
lo w
yo u
up ?
w ill
he ll
sw al
lo w
$y ou
up ?
sh en
gl io
ilfi el
d us
ed eu
ro na
vy es
30 1
to m
ak e
co at
in gs
to of
fs ho
re tid
al zo
ne .
sh en
gl io
il[ fie
ld us
ed eu
ro na
vy es
30 1
to m
ak e
co at
in gs
to of
fs ho
re tid
al zo
ne .

Review summaries:
This paper considers to attack an NMT system from the perspective of efficiency. At a high level, it aims to slightly modify the input such that the target NMT system outputs a long translation, leading to decreased decoding efficiency. To this end, it proposes a method to guide the input modification such that the probability of EOS is as low as possible. Experiments show that the proposed method outperforms the baseline methods in terms of efficiency attack.  

This paper focuses on the efficiency of neural machine translation (NMT) systems, proposing a novel attack approach to test the efficiency robustness. The attack approach can be divided into three parts: 1) find the most relevant tokens of inference efficiency; 2) generate adversarial perturbations for the found tokens; and 3) select the most influenced perturbations. Experimental results on three publicly released NMT systems show the attack approach can significantly decrease NMT efficiency.

TransSlowDown is an attack scheme that aims to targets the computational resources of NMT systems. The authors show that small modifications to benign input sentences can significantly increase the computation NMT needs to process the input, paving the way for denial-of-service attacks on translation services or attacks on the battery power of mobile devices. TransSlowDown is a white box technique that makes use of gradients of the target NMT system, but can also be adapted to a more practical block box technique where gradients from another NMT system are used.