Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers to attack an NMT system from the perspective of efficiency. At a high level, it aims to slightly modify the input such that the target NMT system outputs a long translation, leading to decreased decoding efficiency. To this end, it proposes a method to guide the input modification such that the probability of EOS is as low as possible. Experiments show that the proposed method outperforms the baseline methods in terms of efficiency attack. 

main_review: Strength:
1. The paper is well-written. 
2. This paper study a new research topic for attacking neural models.  

Weakness:
1. Efficiency attack is a new research topic and thus this paper seems novel. However, for NMT efficiency attack is not as valuable as accuracy attack. As mentioned in the paper, efficiency is dependent on the length of the output sentence, which is controlled not only by the NMT model itself but also by some heuristics (such as length penalty and max-length). More importantly, it is easy to defend the efficiency attack by using some other heuristics. For instance, one can simply use the length ratio between source and target sentences to constrain the beam search algorithm. This paper does not take into account these decoding heuristics. Therefore, the overall contribution of the current version is limited.

2. The efficiency is closely dependent on the length of the output sentence, but the experiments do not report the efficiency about the length. In addition, the paper does not mention details about the beam search algorithm for decoding such as max-length. 

summary_of_the_review: This paper study a new research topic for attacking neural models, i.e., efficiency attack, but it is not as meaningful as accuracy attack.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the efficiency of neural machine translation (NMT) systems, proposing a novel attack approach to test the efficiency robustness. The attack approach can be divided into three parts: 1) find the most relevant tokens of inference efficiency; 2) generate adversarial perturbations for the found tokens; and 3) select the most influenced perturbations. Experimental results on three publicly released NMT systems show the attack approach can significantly decrease NMT efficiency.
main_review: ***Strengths***
1. The paper is easy to follow and the motivation is very clear.
2. The topic is interesting and can attract much attention from industry research.
3. The proposed method is intuitive and well-designed. The part of finding important tokens might benefit other research lines such as the over- and under-translation of NMT.

***Weaknesses***
1. In the paper, the authors repeatedly mention the relationship between NMT efficiency and output length. So, why not show the change of output length after the adversarial attacks? I think the sentence length is a more direct metric to evaluate NMT efficiency and is hardware-independent. 
2. A very simple solution to the attack is applying some constraints to the beam search process of NMT. For example, terminating the decoding when the output is twice the length of the input (please refer to --max-len-b of fairseq-generate). If a simple constraint can solve the issue, the contribution of the paper will be significantly decreased.
3. Since the proposed method is more relevant to practical NMT systems, so, how about the results (Section 4.4) on Google translator or other online systems?

**It would be nice if the authors could add some experimental results of the above weaknesses during the author response.**
summary_of_the_review: The paper is very interesting and the contribution is clear, but the claims are not well supported by the existed experiments. Therefore, I give 5 (weak reject) at this moment and would like to increase/decrease my review score after the author response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an adversarial attack method for NMT, aiming to increase the computation complexity while decoding by introducing perturbations into the input. To achieve that, the authors propose an attacking algorithm that consists of three steps: find the most important source token w.r.t the probability of EOS token, then add token level (by FGSM) or character level perturbations (by greedy search), and finally select the perturbation that produces the longest result.
main_review: Strengths:
- The idea of attacking the computation complexity of NMT model is interesting and practical in industrial applications. 
- The paper is well-written and easy to follow.

Weaknesses:
- The paper considers the white-box attacking scenario, which is less significant than black-box attacking which fits the real-world application more. In my view, the significance of white-box attacking is to find the vulnerable point of the model and then fix it by some corresponding solutions such as adversarial training/regularization loss function. The paper only proposes the attacking method without equipping it with the fixing method, which limits its contribution. Therefore I suggest the authors add this part to show that the proposed method can increase the robustness of the NMT model w.r.t the computation complexity attacking.
- In textual adversarial attacking, a very important condition is that the generated adversarial sample should be imperceptible with the input, in human knowledge. Therefore an indispensable part in related papers is to prove this point, which is missed in this paper, however. I suggest the authors conduct evaluations between the similarity of the source input as well as the generated adversarial sample, in automatic (such as BLEU) and human evaluation (such as grammar/semantic/consistency) metrics.
- The proposed method seems time consuming as nearly all operations are based on greedy search. How long does the method take in experiments? Have you compare the running time with other baselines?

Minor points:
- The insight behind of Equ (2) is not introduced. How the importance is defined? What does each part represents? Needs more explantion.
- In character-level perturbation, if the corrupted result is tokenized into multiple tokens or the UNK token, how to maintain them are still the most important tokens? As the probability may change if the source length is changed.
- How to maintain the corrupted result is in the mamximum permutation \epsilon? 
summary_of_the_review: The paper proposes an interesting idea of a new attacking target of NMT models, but has some major flaws such as limited contribution and inadequate evaluation. Therefore I vote for a rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: TransSlowDown is an attack scheme that aims to targets the computational resources of NMT systems. The authors show that small modifications to benign input sentences can significantly increase the computation NMT needs to process the input, paving the way for denial-of-service attacks on translation services or attacks on the battery power of mobile devices. TransSlowDown is a white box technique that makes use of gradients of the target NMT system, but can also be adapted to a more practical block box technique where gradients from another NMT system are used.
main_review: The robustness of NMT against adversarial input sentences has not received much attention (if any) in the literature. Raising this question and demonstrating an effective attack would be a valuable contribution to the community. The demonstrated impact of TransSlowDown seems indeed impressive first, the paper is clearly written, but I feel that at times it is (perhaps unconsciously) on the verge of overselling the results.

First, all NMT systems in practice use a maximum sentence length, which is usually set to a factor times the source sentence length. This maximum target sentence length is needed to prevent infinite loops when NMT is hallucinating and to avoid OOMs. This factor is usually 2-3. Fig. 2 suggests that this factor was either not used or was set to a very high value. Since TransSlowDown works by increasing the target sentence length, specifying and/or varying this factor is crucial for assessing the results.

Second, T5 and fairseq are pure Transformer models with Transformer decoders. Using self-attention in the decoder results in an inference time complexity of n^2. Therefore, an attack like TransSlowDown that increases the target sentence length is expected to be particularly effective. Most deployed MT systems use recurrency instead of self-attention in the decoder for efficiency reasons, so it would be worth to test the effectiveness of TransSlowDown on architectures with LSTM/GRU decoders.

I still think that the paper has merits, it just needs to be more upfront about the details and the limitations of the results. 

Some pointers to related work are missing. The literature on faithfulness in NMT seems relevant, and in Sec. 2.2 at least two very common techniques to improve the efficiency of neural networks (quantization and distillation) are not cited. 

Minor comments
- i in Sec. 3.2 sometimes runs over the input sentence [1,m], sometimes over the output sentence [1,n]
- Minor typos throughout the paper
summary_of_the_review: The paper explores an important aspect - efficiency robustness of NMT systems - that is clearly understudied in the literature. Some missing details and experimental setups prevent me from giving a stronger acceptance recommendation.
review_title: null

Meta-review: The papers studies a novel problem and proposes an interesting algorithm. That said, the reviewers question the motivation of the paper. That is whether this method presents a viable attack on existing MT systems. The attack is not black box and MT systems often have an output length threshold beyond which the output is trimmed. Given the motivational concerns, I recommend that the paper is revised and resubmitted to other venues.
Decision: Reject