Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Recently local attention based vision transformers achieved state-of-the-art results on various visual recognition tasks. This paper rephrases local attention as a channel-wise spatially-locally connected layer with dynamic connection weights. By analyzing local attention form the view of sparse connectivity, weight sharing and dynamic weight computation, the paper discusses the similarities and differences between local attention and dynamic depth-wise convolution. Motivated by this connection, the paper experimentally compares local attention with depth-wise convolution and its dynamic variants on three vision tasks using the macro architecture of Swin-Transformer. The results show that dynamic depth-wise convolution-based models perform on par or better when compared to local attention while being computationally more efficient.

main_review: Well-written and easy-to-follow paper
Forms interesting connections between local attention and dynamic depth-wise convolution in terms of sparse connectivity, weight sharing and dynamic weight computation.
Experimentally demonstrates that dynamic depth-wise convolution is on par or better than local attention in terms of performance on various vision tasks
The proposed inhomogeneous dynamic depth-wise convolution variant is computationally (slightly) better than local attention module
summary_of_the_review: The main difference between local attention and the inhomogeneous dynamic depth-wise convolution (sec 2.4) is in terms of how the dynamic aggregation weights are computed. The on par or superior performance of I-D-DW-Conv when compared to local attention suggests that dot product-based weight computation (which is core to attention)  is not crucial. Dynamic weights that are predicted using only the query/center pixel features also work well. I think this is an interesting for the community to know.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper connects the local attention and dynamic depth-wise convolution and validates that empirically.

main_review: Strengths

The paper is easy to follow and well-written and builds the connection between local attention and dynamic depth-wise convolution.

Weaknesses

1. Regarding the empirical validation of equivalent, I think simple accuracy is not proper as it summarizes the whole dataset. E.g. prediction disagreement from Swin, mean class accuracy or even centered kernel alignment would be better whether or not the networks act similarly.

2. I am not quite clear about the main difference from Cordonnier et al 2020. The authors pointed out that this work focuses on local attention to depth-wise convolution while the prior art discussed normal convolution and normal self-attention. Nonetheless, the depth-wise convolution is a special case of normal convolution and local attention is a special case of normal attention (local attention is masked attention of normal attention).

3. As depth-wise will slide the window while Swin does not. Why author pick it as an example rather than ViL (Multi-scale vision longformer, Zhang 2021)? ViL adopts both local attention and sliding windows. Or what is the connection to ViL?
summary_of_the_review: The paper qualitatively connects local attention and dynamic depth-wise convolution and they validate this connection empirically.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper reveals the connection between local attention and dynamic depth-wise convolution. It empirically shows that the models based on depth-wise convolution and the dynamic variants with lower computation complexity perform on-par with or slightly better than Swin Transformer for ImageNet classification and other downstream tasks such as COCO object detection and ADE semantic segmentation.

main_review: Strengths:
This work is simple and easy to follow. It raises valuable insights on what is the essential components for the "superior" performance of Swin Transformer. With its clear and precise experiments, we can see that the local attention-based networks, Swin Transformer, and the depth-wise convolution-based networks, perform on par in both classification and other downstream tasks such as COCO object detection and ADE semantic segmentation. 

I think the single above fact is valuable and enough for acceptance, as it helps us "narrow down what features of other models are most valuable" (comes from a Reddit comment for another work ConvMixer). Besides, this work also analyses three individual properties of local attention and depth-wise Conv, showing more valuable details of their connection. 

Weakness:
I am curious about the performance of larger models with the ImageNet 22k dataset, especially for downstream tasks such as detection and segmentation. Based on some current works and my own exps, for the larger size of model and data, attention-based models (Swin, CSwin) do have some advantages on downstream tasks. So I think it would be helpful to include these experiments if the authors have sufficient resources.


summary_of_the_review: I think this paper provides valuable insights on the connection between local attention and dynamic depth-wise convolution and I suggest accepting it without a doubt.
review_title: null

Meta-review: All three reviewers recommend acceptance. The paper introduces an interesting study and insights on the connection between local attention and dynamic depth-wise convolution, in terms of sparse connectivity, weight sharing, and dynamic weight. The reviews included questions such as the novelty over [Cordonnier et al 2020] and the connection to Multi-scale vision longformer, which were adequately addressed by the authors. The findings in this paper should be interesting to the ICLR community.
Decision: Accept (Spotlight)