Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new transformer architecture for various vision tasks. The authors revisit the spatial attention design and propose integrating LSA(local self-attention) and GSA(global self-attention) for effective feature modeling. The experiment results have demonstrated the effectiveness of the Twins on many vision tasks, including image-level classification and dense prediction.
main_review: 1. The paper is easy to follow.
2. The idea of this paper is simple but effective. The author proposes to decompose global attention into two separate steps: firstly, applying local self-attention to aggregate features like the swin-transformer. Next, they propose using the summarized key to provide long-range information with low computation costs.  The most important thing of designing a transformer backbone is to alleviate the learning difficulty for the original global transformer, and the success of convolutional-based backbone gives the prior that the local interaction on features can speed up the backbone training. The idea of this paper has incorporated the design of the local window prior in convolution but uses a simple way to obtain long-range information by cross-attention. It is very cool, and I appreciate this idea.
3. The experiments on several tasks has achieved the state-of-the-art performance, and the ablation studies give detailed analysis on the key components of this paper
limitations_and_societal_impact: #193~#199 has figured out the difference with Swin-transofmer. But I wonder why the authors claim the Swin-transformer is more complicated and cannot be optimized for speed on devices? Can the authors provide the evidence to support your claim? And in my opinion, the shifted window in Swin-transformer is also a straightforward idea, resulting in a simple but effective design.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to combine CPE with PVT. In addition, local attention is combined with a subsampled global attention layer so that the model can capture both local details and global relations efficiently. The design is verified on image classification, detection and segmentation.
main_review: 
## Originality
The Proposed Twins-PCPVT simply combines CPE with PVT in each pyramid resolution. This lacks enough novelty, except verifying the effectiveness of CPE. The Twins-SVT combines local windows attention from Swin and global attention from PVT on low resolution feature maps, but this is a relatively more interesting combination.

## Strength
1. The experiments are extensive, on three large scale datasets and tasks.
2. The proposed method is simple and intuitive.

## Weakness
1. Table 5 lacks enough explanation. Does (L, L, L) mean three stages only? In addition, (L, LLG, LLG, G) has fewer parameters than (L, LG, LG, G). Does it mean the total number of layers is fixed?
2. Another important baseline lacks explanation. In table 7, Swin is combined with CPVT, but is it applied to all stages as in Twins, or in the first stage only? What if the relative positional encoding is not removed?
3. The proposed method seems to require strong regularization, e.g. a larger stochastic depth rate. The strong regularization might be due to the small spatial resolution the extra global attention layers are applied. In addition, the modified gradient clipping is claimed as especially important but lacks enough explanation. Is it also due to the extra global layer?
4. It is not clear where the block number (in appendix table 3) comes from. For example, in Twins-SVT-S, 4 blocks are used for global attention. Is it modified from Swin for best result on ImageNet or downstream tasks?
5. Controlled comparison (where only one module is added or removed) is not sufficient for the main Twins-SVT contribution. Although the number of FLOPS and parameters are controlled, there are always more than one difference between compared models. Maybe one could first insert all the PEGs to Swin, then add the global attention PVT layers, and then probably show that shifted window is not necessary with the global attention layers.

### Typos and suggestions
1. Line 252, 160k iterations
2. Line 303, cannot achieve
3. Line 163, "key", change to important so that it is not confused with "attention key"?
4. Maybe emphasize that query is not downsampled in the global attention.

------------------------ Post Rebuttal ------------------------

Thanks the authors for the responses. The rebuttal addresses most of my concern regarding the clarity of the paper and the fair ablation/comparison with Swin, CPE, and RPE. So I tend to keep my original rating of 6.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes two new transformer architectures to address image classification and object detection and segmentation tasks, focusing on the attention mechanisms in each. The first method, Twins-PCPVT, combines the pyramidal subsampling of attention in PVT with conditional positional encoding of CPVT, the second, Twins-SVT, proposes to combine local grouping to manage computational complexity in attention with global subsampling to improve receptive field size, drawing analogies to separable filters.

Both methods are empirically analyzed on relevant datasets (Imagenet1k, MSCOCO, ADE20k) in comparison to state of the art methods (PVT, CPVT, Swin) and show improvements in accuracy and run-time speed.


main_review: I am inclined to rate the paper marginally below the acceptance threshold based on the following analysis:

Strengths:
- The paper addresses an important problem: improving speed and accuracy of transformer architectures by designing attention mechanisms
- The empirical analysis appears thorough and on a wide set of relevant problems and data
- The methods are motivated reasonably well at the coarse level (combine subsampling with conditional encoding, combine local and global grouping, but see the concerns below)
- The empirical analysis shows numerical benefits in accuracy and speed across the tested models and data

Weakness:
- The novelty of the methods and what sets them apart from the state of the art is not clearly highlighted and (at least for me) needed several re-reads. The paper would benefit from outlining more clearly what the novelty and contribution of the paper are.
- It is at times hard to follow the exposure: (a) Some terminology is not explained: Why are the methods termed "Twins", why is one PCPVT, the other SVT (l. 96ff)? The reader is left to interpret. (b) The two methods appear valid on their own, but I do not see unification (l39) beyond addressing attention mechanisms. If the authors see more arguments that the paper leads to a more coherent picture of attention mechanisms, they should point this out more clearly. As it is, the title, abstract (ll. 3-5) and introduction (ll. 39ff) left me with the impression that the paper was aiming towards coherence or unification of attention methods, but I saw little of this evidenced in the remainder of the paper.
- While the high-level motivation appears sound (as pointed out above), the motivation for the individual choices of implementation in the Twins-SVT method (section 3.2) are not clear to me. I would like to understand why these particular choices are suitable, what are other options and why where they not chosen. I understand that the paper is primarily geared towards empirical evidence (versus theoretical motivation and analysis), but I feel that I am not learning much about why and if this is a good choice beyond the empirics.

Further comments/suggestions, these do not impact the review rating:
- I feel that drawing the analogies to separable convolutions or separable filtering in general does not add to the paper. The motivation to bridge fine-grained, local and coarse, global information is fine by itself and does not need the motivation from separable convolutions. 
- The authors indicate deployment frameworks such as ONNX and TensorRT as a motivation from the runtime speed perspective. It would be nice to demonstrated improvements there directly (by showing feasibility and/or runtime improvements at deployment) to substantiate this motivation.


[Discussion period update]
I appreciate that the authors added some details based on my concerns during the review phase. I feel that my initial concerns on outlining the novelty more directly and clarifying "unification" and some of the terminology are reasonably well addressed. My concern on the largely empirical choices in 3.2 is dampened, but not removed. Notwithstanding this, I feel compelled to upgrade my rating. 
limitations_and_societal_impact: The authors do not discuss societal impact, but I do not have any specific concerns.

The authors do not speculate strongly on limitations of the work.
review_title: null

Meta-review: This submission received 3 positive final ratings: 7, 6, 6. 
On the positive side, reviewers appreciated simplicity and effectiveness of the idea, strong empirical performance and clear presentation.
At the same time, some of them initially expressed concerns with overall novelty and motivation of certain design choices.
After an extensive discussion between the authors and the reviewers during the rebuttal period, one of the reviewers upgraded their score (from negative to positive), while others remained unchanged.
Overall, the strengths of this paper outweigh its weaknesses, so the final recommendation is to accept for poster presentation.
Decision: Accept (Poster)