Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents Long-Short Transformer (Transformer-LS), which integrates a dynamic projection based long-range attention and a local window short-term attention to capture both global and local features. Specifically, 1) the long-range attention is implemented by projecting the Key and Value embeddings into shorter ones (in terms of number of tokens) based on the projection matrix generated from the original Key embeddings and 2) the short-term attention is implemented by dividing the input sequences into disjoint segments. A scale mismatch is identified from the long-range and short-term attention and solved by the proposed DualLN. The experiments are conducted in both NLP and CV tasks.
main_review: > + Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

This work claim 3 contributions: 1) integrating long-range and short-term attention; 2) input-dependent dynamic low-rank projection in long-range attention; 3) DualLN to handle the norm mismatch problem when integrating long-range and short-term attention. Although each of them are not novel enough, such a combination seems to be quite effective in both NLP tasks and CV tasks, which is also valuable.

> + Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

1. A better performance breakdown (ablation studies) on the contritions 1), 2) and 3) mentioned above: Although some ablation studies about the effectiveness of each components are given in Section 3 and 4. It would be better if how each of them contributes to the final performance improvements are given, e.g., how the performance of simply combining the Linformer and the window attention in Big Bird using contrition 1) and 2)? Will the benefit of DualLN (Figure 1) and Dynamic project still exists in CV tasks, which usually have relatively shorter sequences?

2. Some confusing details: What's the f function in Line 162, is it a Linear layer? In Table 3, CvT*-LS-21 seems to have comparable or even better accuracy than CvT*-LS-17 and CvT*-LS-21S with much less FLOPs, any insight or analysis about it?

> + Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

This submission is clearly written and well organized.

> + Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

The results can help the researchers to build more efficient Transformer models w/ the proposed Transformer-LS as the building block.
limitations_and_societal_impact: The authors addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses the quadratic complexity issue in Transformer, and proposes a long-short efficient Transformer model. It aggregates a long-range attention with dynamic projection to model distant correlations and a short-term attention to capture fine-grained local correlations. A dual normalization strategy is used to address the scale mismatch between the two attention mechanisms. The method is applied to both autoregressive and bidirectional models without additional complexity. Experiments on both vision and language benchmarks verify the efficacy of the method.
main_review: Pros:

* This paper is well-written and easy to follow. I enjoy reading this paper. The summarization of the previous efficient Transformer is good.

* The experiments on both vision and language, as well as both autoregressive and bidirectional models, are interesting and comprehensive.


Cons:

* Overall, the novelty of this work is not strong, because the efficiency issue has been addressed by several previous works, which are also discussed in the related work. The long-short idea is also discussed before, such as "Lite Transformer with Long-Short Range Attention" (although the method is different). It is suggested to add more discussions on key differences.

* The main focus of this paper is the efficiency of Transformer. However, the latency (or model inference time) is not reported and compared. Though the FLOPs reduction is clearly demonstrated, yet FLOPs is still the theoretical metric for efficiency measurement. Models may not run fast even when they have smaller FLOPs. I would like to see the comparisons on latency on GPU, CPU, etc. This is very important for practical applications.

* In Sec. 4.3, your re-implementation of CvT gets worse performance, what's the concrete top-1 accuracy? Btw, CvT recently released their official implementation.

* Why you do not run your method on Swin models? It is suggested to add an experiment. Besides hand-designed models, whether the proposed long-short attention can be used in automated search models, such as HAT (https://arxiv.org/abs/2005.14187), BossNAS (https://arxiv.org/abs/2103.12424) and AutoFormer (https://arxiv.org/abs/2107.00651)?

* The improvements on high-resolution images (from 384x384 to 448x448) seems not competitive. 

* In Fig.6 (Appendix), it is a little bit strange why the similarity drops significantly from layer #12 to #13? 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces Long-Short Transformer, an efficient Transformer model that combines local sliding window attention with low rank global attention. This is an attempt in combining the two dominant paradigms of making Transformers more efficient: Sparsifying attention and Low-rank methods. 
main_review: There are two main ideas presented in the paper: 1- Combining sparse local attention with low-rank methods. 2- Using dynamic low-rank projection with a function that depends on the Keys instead of a fixed learned projection. 
The general ideas are very interesting and results are encouraging. However, there are few key weaknesses with the paper:

First, while I found the first idea to be very interesting, the impact of the second idea is less clear as I couldn't find any ablations comparing the full model with the model that combines the local attention with a fixed low-rank projection methods. I.e., What would happen if we replace the dynamic projection with fixed learned projection (similar to the Linformer paper)? 
Similarly, the impact of the local sliding window is also not very clear, what would happen if the queries only attended to global projections? The only place such results exists appears to be on the 3 Long Range Arena tasks. Another useful ablation would have been comparing dynamic projection with fixed projection in the setting without the addition of local sliding window attention.

The second weakness is the general evaluation tasks. I appreciate the experiments on LRA and char language modeling, but there is no evaluation on actual downstream NLP tasks, which makes the effectiveness of the model in real-world applications less clear. Instead of expensive char LM experiments, I would have much preferred experiments on NLP tasks similar to those evaluated by Longformer and Big Bird papers. 

I was a bit disappointed to see some of the tasks of LRA excluded from evaluations. The authors mention that this is because the data is small and tasks are synthetic, but in general other LRA tasks are also not realistic and I was a bit confused about this evaluation setup. About the size of data, does this mean that the model will have harder time training in low-resource setting?

On Robustness results (Table 4), how does a CvT model (without the addition of Long Short Transformer) work? This comparison seems to be missing. 

The paper mentions that dynamic projections are "more flexible and robust to, e.g., insertion, deletion, paraphrasing, and other operations that change sequence length" (L164-165). However, I didn't find any empirical evidence or analyses about this. 

Minor: A figure about the main idea would have been nice to include. 

Update: The authors included new experiments addressing most of the above concerns. I'm updating my score. 



limitations_and_societal_impact: I would have liked to see some discussion on limitations of the work and societal impacts. 
Some that come to mind are potential environmental impacts of training and experimenting with such models at scale (e.g., https://arxiv.org/pdf/1907.10597.pdf).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors focus on reducing the quadratic computation complexity of the self-attention mechanism in the standard Transformer. They combine local and global context through sliding window attention and matrix projections. During aggregating long-range and short-term attention, they employ two sets of Layer Normalizations to make sure that the model will use both long and short range effectively. Experiments show some promising results on various tasks.
main_review: Their model tries to model both the long-range and fine-grained local correlations using two different attentions, both of which have linear computation complexity. Their method is easy to understand but I do think it is not rival to implement this model so it would be great if authors could release their code to the public. Their paper is easy to follow and extensive experiments has shown the effectiveness of their method. I just have a few questions:
1. For aggregating the local and short-range attention, you concatenate their attention matrix. Will this introduce extra computation, could you just simply add them together, and will this hurt the performance much?
2. Will you apply your method for the pre-trained language model to further demonstrate the effectiveness and efficiency of your proposed method?
3. For efficiency, FLOPs are used to compare different models, and could you compare their wall time speed?
limitations_and_societal_impact:  Although they do not explicitly mention the societal impact, I think their method could help us to build the efficient Transformer. However, I still suggest authors add this section to their draft.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed Long-Short Transformer (Transformer-LS), which aggregates sliding window attention (i.e. short-term attention) with linear attention via low-rank projection (i.e. long-term attention), by simply concatenating the two sets of keys.
Due to the scale mis-match of the keys from the long and short term attentions, the authors further proposed the dual layernorm (DualLN) method.

Experiments were conducted on Long-range Arena (LRA), auto-regressive language modeling and ImageNet classification. Transformer-LS achieves improvements over previous efficient transformer models on both accuracy and efficiency.
main_review: The paper is clear and generally well written. 

However, there are some weaknesses:

1. The proposed architecture of Transformer-LS is not novel. The two key components of Transformer-LS, i.e. the sliding window attention and low-rank projection (or similar architectures), have been introduced in existing works. For example, the sliding window attention has been extensive explored in LongFormer and Big-Bird models (with some slight differences), and the low-rank projection approach is a straight-forward extension of LinFormer. Actually, a recent work (Goyal et al., 2021) proposes a more general low-rank projection approach which captures contextual information in the projection, instead of just using a learnable parameter (like W_i^{P} in E.q 3). Thus, the contribution of this work is to combine the two types efficient attention mechanisms and empirically evaluate it on language and vision tasks.


2. The experimental results, especially  those on language tasks, are not entirely convincing.
For the experiments on LRA, they did not follow the default setting in the LRA paper, and the efficiency comparison is indirect. It is highly recommended to follow the default setting in the LRA paper to fairly compare the accuracy with previous efficient transformer models, and directly compare the speed and memory efficiency by steps per second and memory usage.
Moreover, it is highly recommended to conduct more challenging language tasks, including machine translation and pre-trained language models (such as BERT), to demonstrate the effectiveness of Transformer-LS.

References:

Coordination Among Neural Modules Through a Shared Global Workspace. Goyal et al., 2021.

================Updates after reading authors' response=============================

Thanks for the authors' great effort on the response, which mostly addressed my concerns. Thus, I upgraded my score to 6.
limitations_and_societal_impact: NA
review_title: null

Meta-review: This paper aims to advance the efficient transformer by integrating long-range attention and short-term attention. The reviewers identified several risks of this paper. Most of reviewers had concerns about the novelty, as the presented method is a combination of existing techniques. The reviewers also raised questions regarding the architecture choices, evaluation setups for LRA and lack of experiment for downstream NLP tasks. During the rebuttal, as acknowledged by the reviewers, the authors have successfully addressed most of the experimental concerns by adding more results. Despite the fact that the methodology is rather incremental, the empirical results seem quite strong and possibly be impactful to massive CV and NLP downstream tasks. 
Decision: Accept (Poster)