Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces orthogonal transformation of tokens and combines it with the idea of interleaving window self attention (WSA) and dilated self-attention (DSA) to capture both local and global interactions without incurring the prohibitive cost of full global self attention. They showed that initialization and optimization of orthogonal matrices can be simplified by using Houholder transformations and leveraging already established optimization techniques to learn these matrices. In addition, the paper also make use of Positional MLPs where MLPs are equipped with depth-wise convolutions to allow for downsampling within the transformer block. 

Authors individually ablated different techniques proposed in the paper and combined the best configurations to achieve top performance on wide range of image tasks.
strengths_and_weaknesses: The main differentiating contribution of the paper is using the Householder transformation trick to introduce orthogonal transformations. This idea is coupled with already existing techniques of window self-attention and dilated self-attention. Authors present extensive experiments ablating each individual technique introduced. The paper is also well written and easy to read and understand. 
questions: line 44: Authors claim that one of the advantages of orthogonal transformations is that the tokens can be separated into linearly independent groups. But, there is no further explanation as to why this linear independency is important or how this contributes to the overall performance of the model.

equation 5: Authors present complexity analysis in comparison with the global self-attention. But, it is only fair to compare against dilated self attention complexity. 
limitations: Given that the paper is just concatenation of already existing ideas with minimal novelty of newly introduced techniques, I am not inclined to accept the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a new efficient self-attention form  (Orthogonal Self-Attention, OSA) that conducts self-attention within groups, where orthogonalization of window tokens are performed first before forming the groups. The final architecture consists of alternative OSA and window self-attention (WSA) as attention, with FFN equipped with depth-wise convolutions. It shows competitive results on ImageNet classification, COCO object detection, ADE20k semantic segmentation. 
strengths_and_weaknesses: Strengths: 
 	The way to construct orthogonal matrix endogenously by multiplying householder reflectors is novel and interesting. The OSA form proposed enabled the token connections within local window and links across windows. 
	The experiments results show competitive performance across different vision tasks, including image classification, object detection, semantic segmentation.

Weaknesses:
	The essentialness of using orthogonal matrix is not studied. 
The whole OSA process 
1.	connects tokens within local windows with local window token orthogonalization, is serves as MLP layer within local windows, except the weight matrix of MLP is naturally orthogonal.
2.	Connects tokens beyond local windows by forming new groups across previous local window. 
3.	Token reverse as the inverse of orthogonal matrix is easy to get, just the transpose of the matrix. 
            Step 2 can be done regardless of the weight matrix of this local window MLP is orthogonal or not. 
	Step 3 is the vital part that only orthogonal matrix weight can perform, I believe this should be studied, which is not presented, for validating the essentialness of using orthogonal matrix rather than just following the form that connects local and connects beyond local windows. 

questions: 1.	As mentioned in weakness, The reversible operation is the novel part, but no ablation on that. Experiments do not show the essentialness of using orthogonal matrix instead of regular one. What if we keep the form (i.e. using non-orthogonal matrix to transform tokens within local windows, and then group them in same way as current strategy to perform self-attention), then no token reverse as inverse of weights is hard to get, maybe replacing multiplying the inverse of A with another new weights A’. 

2.	Line 41 mentioned that tokens have lower resolution in the orthogonal space, can the authors provide more details regarding this? From my understanding, the new tokens obtained from multiplying the orthogonal matrix will not change dimensions, Z’ = AZ, Z’ is (n x c), A is (n x n), Z is (n x c), n = h x w.  The computation saving for attention here is from conducting group-wise attention, which does not seem to be brought from orthogonalization of tokens, but from window and group partitions of tokens. 
3.	How to choose window size for WSA and OTB? Will different size affect the final performance? 



-----post-rebuttal---
Thanks for the authors presenting the ablation study results on the essentialness of orthogonalization during rebuttal. Still, considering that the gain from ortho. sa is limited, 0.3% compared to dilated sa as in Table 6, I'll keep my initial rating as borderline accept. 
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed an efficient Transformer backbone, Orthogonal Transformer, for efficient processing of visual inputs. The backbone is mainly built on the proposed efficient self-attention mechanism, orthogonal self-attention (OSA), to model dependencies between tokens in the orthogonal space. The proposed backbone achieves excellent performance-FLOPs tradeoff on visual tasks.
strengths_and_weaknesses: ### Strengths
The performances of the proposed backbone are superior.
The paper is well written and easy to follow, and the illustration is clear to me.

### Weaknesses
The motivation is not clear to me. Why do other efficient self-attention approaches fail? The author claimed other efficient self-attention mechanisms "lose fine-level details for coarse global self-attention or hurting long-range modeling for local self-attention" (L35-36). However, I cannot find any pieces of evidence to support the claim. In addition, how does the proposed orthogonal self-attention (OSA) work? The author claimed OSA "captures global dependency without losing fine-level details" (L37-38). I have found no evidence to support the claim, either theoretically or empirically.

The contribution is not significant. I believe the only novel part of this paper is the proposed efficient self-attention approach.  The proposed Positional MLP in Sec 3.4 has been studied in PVT v2 [CVMJ 22], CeiT [ICCV 21], and CMT [CVPR 22]. So I would not say this is the novel part. 

The ablation is not sufficient. In Tab 6, the authors compared the proposed OSA with window SA and dilated SA. How about the window SA with shified window like Swin?
questions: The questions about the paper are listed in the Weaknesses.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed an token orthogonalization layer, there self-attention is performed on groups of orthogonalized tokens. The paper then designs the Orthogonal Transformer (OT) which combines the orthogonal self-attention layer with a pyramid Transformer architecture, positional MLPs (adding a depthwise-convolution into the MLP layer), early convolutions, and a novel downsampling method between stages. OT performs well on ImageNet-from-scratch, COCO detection and instance segmentation, and ADE20k semantic segmentation.
strengths_and_weaknesses: Strengths

* I like the proposal of constructing a trainable orthogonalization module from the product of Householder matrices. The orthogonal self-attention layer is fairly simple and general, and I think is a nice contribution.

* The experiment cover multiple tasks, appear thorough, and compare to many state-of-the-art alternatives.

* There is an ablation of the four components of the orthogonal Transformer.

Weaknesses

* I think the main weakness is the large number of moving parts in OT: the combination of both windowed attention and orthogonal attention, addition convolutions at the start and in the middle of the network, and a new downsampling mechanism. The combination of these factors significantly increases the complexity of the network over the original Transformer/Vision Transformer, potentially limiting adoption. The ablation study shows that some of the "minor" components have an equal-or-greater impact than the orthogonal attention layer which is the main "selling point" for OT. E.g. Conv position embeddings improve the scores on all tasks over absolute position embeddings more than orthogonal attention. Therefore, it feels like the orthogonal self-attention layer is not the key driver of performance in OT.

* Given that the network contains many vision-specific components (convolutions), I feel that "Orthogonal Transformer" is over-selling or over-generalizing the network. A name like "Orthogonal Vision/Image Transformer" would make it more clear that this is a vision-specific variant.

* There are a few typos, such as:

"tokens has a lower resolution" -> have

"singe OTB" -> single OTB
questions: Please address my main concern about the orthogonalization layer. Given that the orthogonal self-attention layer is marketed the main contribution of the paper it would be useful to have an experiment that demonstrates the value of this layer in isolation. I think the best way to do this would be to run a vanilla ViT/DINO setup with just the OT layer replacing self-attention, but without any of the additional changes/tricks. This would help determine with the OT layer is useful in isolation, or it requires the other network modifications in addition to be a useful component.

Second, you answered "Yes" to "Did you include the code, data, and instructions needed to reproduce the main experimental result?". I did not see any mention of opensourced code, will code be made available?
limitations: Section F in the appendix dedicated to limitations and societal impact. The societal impact is adequately addressed. The limitations section is brief, but I think it is adequate, and i believe captures the main limitation that the study is restricted only to image-based tasks.
review_title: null

Meta-review: metareview: The paper presents orthogonal attention mechanism for vision transformers. All reviewers found the overall system has good performance and the introduced orthogonal attention has the potential to be widely used. The authors' rebuttal resolves the majority of the questions.

The authors should add their promised additional experiments in the final version.
confidence: Less certain
Decision: Accept