Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents an efficient self-attention mechanism for vision transformer, which simultaneously capture both short and long-range visual dependencies by making the query to attend patches at different granularity levels. The authors demonstrate its performance on image classification and object detection.
main_review: Originality:
- The idea of extracting visual features using multiple granularity levels is well-established. Previous work has also tried to build a hierarchy of features within vision transformer (e.g., PVT). However, it is novel to build a single self-attention head which can attend multiple granularity levels.

Quality:
- The submission is technically sound.

Clarity:
- The paper is well written.

Significance:
- The empirical results are compelling. 

Detailed comment:

1,  In section 3.1, a “patch embedding layer which consists of a convolutional layer with filter size and stride both equal to 4” could not transform input $\frac{H}{4} \times \frac{W}{4} \times (4\times4\times3)$ to output $\frac{H}{4} \times \frac{W}{4} \times d$.

2,  In Table 2, Focal-Base does not achieve better result than Focal-Small. Do you think it’s due to the limited set of hyperparameters that has been tried, or it’s due to the strong induction bias within focal attention may hurt the generalization ability of big model?

3, What is the criterion to organize Table 2? CvT-21 should be placed along with Focal-Tiny.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a focal version of self-attention module for vision transformers. The query is kept at high resolution, but the number of keys is reduced depending on the relative position difference. This proposed attention is implemented within multi-scale windows. It shows good performance on top of recent transformer advances, on ImageNet and COCO.

main_review: ## Strength
1. The proposed focal attention mechanism is original and novel.
2. The idea is clearly presented with equations and illustrations.
3. Training details included for fair comparison and reproducing. Comparison is fair with other methods.
4. Various settings and tasks are explored and compared with solid baselines.

## Weakness
1. Runtime / throughput is not clear. The proposed attention requires extra subsampling layers, key value computation, and more importantly, concatenating keys and values from different resolution windows (according to the code). These might introduce extra runtime cost with the same FLOPs. But this is not sufficiently discussed in the paper.
2. Scaling to larger models. The ImageNet performance of Focal-Base is the same as Focal-Small, and is similar to Swin with the same size. In addition, the improvement over Swin in the Small regime is limited as well. In this case, error bars may be reported to address result variances. However, Focal-Base improves consistently on detection segmentation tasks, given a limited ImageNet Top-1. Does it suggest that more context (larger focal size) is necessary for COCO than ImageNet? (Relates to the discussion in Line 281).
3. What is the intuition behind sharing the same set of query key parameter for both small and large windows? It does save parameters and FLOPs, but the model might be extracting local edges/corners with fine detail and global semantic concepts with coarse context windows. This seems to suggest that different (not shared) parameters might be good for fine and coarse levels. It might be worth ablating that three independent attention could be used for the three levels.
4. The training recipe seems modified slightly from DeiT/Swin, e.g. the drop path rate. Does it affect comparison fairness?

------------------------------------------------------------Post Rebuttal-------------------------------------------------------------------

After reading the author rebuttal and other reviews, I think the weakness 2 and 4 are mostly addressed. Nevertheless, the runtime/throughput is a confirmed weakness according to the authors' results in the rebuttal and is also a shared concern among reviewers. But I think this is ok given other contribution of the paper. In this case, I keep my original rating of 6.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes Focal Transformer, which captures both local and global attention based on their design. It also follows the pyramid architecture so their model can be easily extended to detection and segmentation tasks. The result on image classification detection and segmentation are very promising. 
main_review: Overall, the main contribution of this paper is proposing a new attention module for multi-level trasnformer, which seems efficient and can capture both local and global information. Then the architecture heavily follows PVT and Swin Transformer.  

About performance, it is slightly better than Swin Transformer, which is reasonable because swin do not have global attention.

Advantages:

1. clear paper writing, which is easy to follow.

2. strong performance on several tasks.

3. reasonable story to design an efficient self-attention but consider both global and local region.

Weakness:

1. I have some concerns about the inference speed of this method. If I have a misunderstanding please correct me. Mainly because calculating focal attention seems not very straightforward. I understand the theoretical computation complexity is efficient but it is not clear of real running speed. **It is necessary to report the speed and compare with other methods** such as Swin and PVT, especially on object detection and segmentation which need a high-resolution image as input.

2. In Table2 when scaling model size from small to base, the accuracy stops to increase. So I wonder does this model's performance upper bound is lower than other methods? Because ViT can consistently improve the performance when enlarging the model size even the model size is super large.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new attention mechanism called focal attention to enable efficient long-range interactions in vision transformer. By plugging the proposed module into a multi-scale vision transformer model,  a new focal transformer was designed and its superiority was demonstrated over the state-of-the-art approaches on both image classification and object detection tasks.
main_review: 1. This paper is well written, and has no obvious mistakes.
2. For a vision transformer model, how to reduce the computational overhead while keeping ability of capturing the long-range dependencies is still an important and interesting research topic. 
3. The experiments are conducted on various benchmarks on image classification and object detection, with extensive ablation studies.

limitations_and_societal_impact: 1. There are limited discussions on model efficiency. I suggest the authors to add more experimental comparisons with Sparse Transformer or some other efficient transformer methods. 
2. The focal windows and region partition operations do not bring any additional flops but may be time-consuming.  It should better add throughoutput or latency results in Table2.
3. Figure 4 shows three fine to coarse attention levels with short and long range interaction.  However, in the actual model configuration the attention only has two level and mismatch Figure4. Why dose this happen?  The authors should do ablation study about the number of level including acc and latency. 
4. In table 4 stage1-2, s^1_{w} * s^1_{r} is larger than input resolution, e.g, 7x5>28. How do you handle this situation.
review_title: null

Meta-review: There is a unanimous agreement on the novelty/solid results of the paper and the AC agree with the reviewers. The proposed idea in this paper is interesting, well-motivated, and could make good contributions to the general vision and representation learning communities. In this regard, the AC recommends acceptance as spotlight.
Decision: Accept (Spotlight)