Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method that increased DeiT performance by 1%, where the proposed method introduces attention to incorporate spatial locality into self-attention heads. By doing this paper claims that local dependencies are captured with masked attention heads along with global dependencies captured by original unmasked attention heads.
main_review: strengths:
- the paper proposes a masking strategy for attention mechanism, to incorporate the spatial locality into self-attention heads.
weaknesses :
- how is the proposed method differs from the multi-scale transformers or hierarchical vision transformers
-  novelty for masking strategy is minimal.
- Can the proposed method be extended to any kind of transformers.
- it hard to understand how the masking strategy addresses the spatial locality-based attention mechanism

summary_of_the_review: In summary the proposed method has few strengths and weaknesses, it would be easy to understand the effectiveness of the proposed method, if the paper provides experiments showing the improvements when masking strategy is introduced to other transformer networks
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to bring locality into the attention module of vision transformers. This locality mechanism is brought by the introduced attention masks. Basically, the attention masks are binary and is likely to restrict the attention to the local field of a token. The local attention mask results in a block matrix before the application of Softmax function. Tokens with zero mask values are given a constant attention value of 1. Thus, those tokens are still involved in the computation. It is claimed that the binary masks is able to keep the global connection when necessary.
main_review: Pros:
1. The paper is well-written.
2. The experimental results shows that improvement of the proposed method compared with the baselines.

Cons:
1. The idea of this paper is closely related to Swin Transformer. Swin transformers bring locality into network architecture by the window attention, which could save computation significantly. Yet, using attention masks does not have this additional benefit.
2. Thus, whether the proposed method could be applied to other transformers with locality in the attention module is questionable.
3. In the experiments, other transformers with locality mechanism (Swin transformer, LocalViT, T2T-ViT) should be compared.

summary_of_the_review: The main concern of this paper is the usefulness of the locality mechanism due to the existence of Swin transformers which are more computationally efficient.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes to introduce attention masks to incorporate spatial locality into part of self-attention heads, while keeping the remaining heads to capture global dependencies. Experiments show improved accuracy on ImageNet in comparison with some baselines.
main_review: **Strength**: 

Incorporating local information efficiently into global self-attention is a promising research direction.

**Weakness**:

1. Writing is very poor, which makes it hard for me to understand what the authors want to express. 

2. Notations are frustratingly messy and I can only list a small part of mistakes here. For example, at the beginning of Section 3, this paper fixes the size of input images, patch size, hidden dimensions, etc. The authors should define symbols clearly and specify the configurations in the experiments. 
In Eq. (1), what are the definitions of Q, K, V?  In Eq. (3), what’s the meaning of index i? I guess it does not represent the layer index here? But in Section 4.2, i indicates the layer index. 

3. The masking strategy, which is the core component of the paper, has serious technical problems.
    - The paper proposes a heuristic strategy to add masks. However, in Transformers, different heads actually encode distinct patterns (e.g., [A1]). However, in this paper, all the heads assume to share the same mask which is not reasonable. For 2 (a), why add masks to all attention heads? For 2(b), if the score is around 0.5, why only add a mask to the 0-th head? More explanations are needed. 
    - The paper analyzes the search space but no search algorithm is proposed. To automatically search the masking strategy is important and it is not OK to leave it for future work.

4. The advantage of the proposed masked attention over the existing efficient Transformers with local-global self-attention (e.g., PVT [Wang el al., 2021a], Swin [Liu et al., 2021a] and Twins [Chu et al., 2021a]) is unclear.  In addition, the paper should compare with these methods.

5. The network configurations for MaiT-XT, Mai-T, MaiT-XS, MaiT-S are not clearly explained. 

6. The paper argues that the proposed masked attention heads can facilitate the convergence but I failed to find any training and validation loss curves to justify this statement. 

7. In Figure 1, why does the green mask correspond to the 72-th patch? In Figure 2, what does ‘-’ mean in the visualized attention mask?

Overall, I think the submission is not complete at the current stage and needs to be significantly improved. 

**References**:

[A1]: On the Relationship between Self-Attention and Convolutional Layers, in ICLR 2020
summary_of_the_review: The advantage of the proposed masked attention over the existing literature is not properly discussed. In addition, paper writting is very bad, with substansive confusing descriptions and notations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to improve the classic vision transformers, or specifically the DeiT model by introducing the Masked Attention Head. Instead of focusing on aggregating global information in the original self-attention heads, this paper introduce local information via the proposed Masked Attention Head into self-attention. Experiments on ImageNet show that the proposed approach does matter for lifting the model performance of the original DeiT model.
main_review: Pros:

- The idea of this paper is interesting. Though there have been many papers discussing the importance of introducing locality into vision transformers, this paper starts from using a mask to improve the self-attention.

- Experiments on DeiT show that the proposed approach performs better than the baselines, especially when two masked heads are used as shown in Table 2.

- Analysis is also given to demonstrate why involving attention mask could benefit training deeper vision transformers.

Cons:

- The arguement in Page 4 seems not correct. The authors claim that "the locality is not intrinsically encoded in the transformer structure."
In my view, it should be the locality is not explicitly introduced in transformers. As shown in the DeepViT paper, lower transformer blocks can indeed capture local information. I suggest the authors to reorganize the presentation here.

- The mathematic symbols should be italic. I think the authors should take a correct attitude towards paper writing.

- Experiments are insufficient. After the DeiT paper, there are a large number of papers working on vision transformers. In my view, taking the DeiT paper as baseline is already out-of-date. It would be better to show results on stronger baselines, such as CaiT, LV-ViT, which do not change the self-attention itself. If based on a stronger baseline, the proposed approach can still perform well, I think it should be a significant contribution.

- The improvement over the baselines is also not significant. Most previous work, like CaiT, LV-ViT, T2T-ViT, improve the DeiT model by more than 1% in terms of Top-1 accuracy. However, the improvement shown in the paper is lower.

- I do think a comparison with other methods for vision transformers should be added. I cannot find any tables showing such a comparison.
summary_of_the_review: Based on the concerns listed above, I think at this moment this paper does not reach the status for publication in ICLR.
review_title: null

Meta-review: The paper presents a masking strategy to introduce the locality bias into the vision transformers. The experiments show the effectiveness of considering such inductive bias. The reviewers agreed on the importance of the research question and the simplicity of the algorithm. MaiT also has a straight-forward sparse attention extension that performs on the complexity of $O(n)$ rather than $O(n^2)$.

The reviewers also listed some common concerns of the paper:

(1) The novelty of such a masking approach is relatively low. I don't think the ALS or the soft masking adding too much contribution to that. Similar ideas have been explored in a number of papers.

(2) Reviewers also raise concerns about the experiments. Inductive biases often help more in small settings (fewer parameters and FGLOPs) and gain less in the large settings. When comparing with the STOA models, I think this is basically the trend shown in the paper as well. While I appreciate the authors’ efforts in including more comparisons, I have to say I really don’t think the performance gain is significant enough especially in the large settings. Needless to say that there are many other ways of encoding the same locality bias into the model.

Based on the reviewers' judgements and my own opinion, I therefore recommend rejection of this paper.
Decision: Reject