Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 

This paper claims to propose an extension of the Transformer architecture specialized for the mobile environment (under 500M Mult-Adds).
The authors propose their method called "Long-Short Range Attention (LSRA)," which separates the self-attention layers into two different purposes, where some heads focus on the local context modeling while the others capture the long-distance relationship. 
They also demonstrate consistent improvement over the transformer on multiple datasets under the mobile setting. 
It also surpasses the recently developed comparative method called "Evolved Transformer" that requires a far costly architecture search under the mobile setting.
 
This paper is basically well written and easy to follow what they have done.
The experimental results look good.
 
However, I have several concerns that I listed as follows.
 
1,
I am not sure whether my understanding is correct or not, but it seems that the proposed method, LSRA, is not a method specialized for mobile computation.
In fact, in the paper, they say, "To tackle the problem, instead of having one module for "general" information, we propose a more specialized architecture, Long-Short Range Attention (LSRA), that captures the global and local information separately." 
 
There is no explicit discussion that LSRA is somehow tackling the mobile setting.
There is a large mismatch (gap) between the main claim and what they have done.
In other words, LSRA can be simply applied to standard-setting (non-mobile setting). Is there any reason that the proposed method cannot be applied to the standard-setting?
If my understanding is correct, the paper must be revised and appropriately reorganized to clear this gap.
 
2,
I am not convinced of the condition of the so-called "mobile setting (and also extremely efficient constraint)." 
Please provide a clear justification for it.
 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper proposes Mobile Transformer, an efficient machine translation model, which achieves state-of-the-art results on IWSLT and WMT. The Mobile Transformer is base on long-short range attention (LSRA) modules that combine a depthwise convolution branch to encode the local information and a self-attention branch to capture the long-range information.

The main contribution of this paper includes
1. bottlenecks are not beneficial to 1D attention models
2. having both convolution and attention modules in parallel performs better and more efficient than having one of them alone. While LSRA is included in the search space of Evolved Transformer, surprisingly, their searching algorithm doesn't discover it. Evolved Transformer has either two convolution branches or two attention branches in parallel.

The paper is well written and easy to follow. The experiments are quite solid; however, it would be if the authors can report how Mobile Transformer performs on other language pairs or other NLP tasks. 

Questions:
1. Do the attention maps in Figure 3 come from the average of multiple heads or just one of them? 
2. The constraint for the mobile setting is set to 10M parameters. Can you justify why you choose it? In my opinion, memory footprint or inference time on mobile devices can be more realistic. 
3. Regarding the design cost shown in Figure (b). Does the number for Mobile Transformer include the cost of all the experiments you ran to search for your Mobile Transformer? 
4. I wonder if LSRA can also be applied to other tasks such as language modeling or reading comprehension.
5. In terms of inference latency, how much faster Mobile Transformer is compared to Transformer and LightConv?
6. Have you considered having the trade-off between having more parameters in the encoder or decoder?
7. Have you done any analysis on why all tokens in Figure (c) assign high weights to the <EOS> token?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents a new technique (LSRA) improving Transformer for constrained scenarios (e.g., mobile settings). It combines two attention modules to provide both global and local information separately for a translation task. In this manner, the authors place the attention and the convolutional module side by side, thus having different perspectives (globally and locally) of the sentence. They test their approach to 2 common translation benchmarks.

Enhancing deep learning model efficiency is very important, and the authors succeeded in reducing the computation costs and consequently, the CO2 emissions. But the evaluation results are not so impressive and go in line with other previous efficient deep learning approaches for different domains. I’m not an expert in NLP, but overall results of 10-1000x or more wall clock time reduction with <1-5% loss in accuracy are usually obtained for domains that have seen more optimization for mobile deployment (especially mobile-optimized CNNs like MobileNet). LSRA-based appraoch is slightly better than the original version of Transformer and its evolved version. From the latter (ET) authors seem to take the idea of parallel branches for their architecture. Also, adaptive attention span in Transformer models and all-attention layers have already been investigated to make networks more efficient and simpler for longer sentences. Include clearer ablation studies would be also interesting to support their findings and superior performance.

To summarize, the paper is addressing an important and interesting idea. It is, in general terms a nice engineering paper, but I am not sure about whether the developments and results are relevant/novel enough yet at this point to publish at ICLR. 

-------------------------------------------------

Looking at the other comments and the feedback provided by the authors, I have a more positive feeling about the contributions of the paper which are now sufficiently demonstrated. Therefore, I increase my original recommendation to "Weak Accept".


review_title: Official Blind Review #2

Meta-review: This paper presents an efficient architecture of Transformer to facilitate implementations on mobile settings. The core idea is to decompose the self-attention layers to focus on local and global information separately. In the experiments on machine translation, it is shown to outperform baseline Transformer as well as the Evolved Transformer obtained by a costly architecture search. 
While all reviewers admitted the practical impact of the results in terms of engineering, the main concerns in the initial paper were the clarification of the mobile settings and scientific contributions. Through the discussion, reviewers are fairly satisfied with the authors’ response and are now all positive to the acceptance. Although we are still curious how it works on other tasks (as the title says “mobile applications”), I think the paper provides enough insights valuable to the community, so I’d like to recommend acceptance. 

Decision: Accept (Poster)