Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper proposes a non-hierarchical transformer model for visual recognition tasks (detection, segmentation). Unlike recently proposed hierarchical methods like Swin Transformer that use an hierarchical transformer architecture, exchanging global information between features is computationally expensive for non-hierarchical transformers. To deal with this challenge, the paper proposes an efficient  Group Propagation Block (GP Block) to exchange global information between high-resolution features. In a GP block, grouped features formed by learnable group tokens and then global information is exchanged between grouped features. Finally, global information in updated grouped features is returned to the image features through the transformer decoder. GPVIT is evaluated on  image classification, semantic segmentation, object detection, and instance segmentation and obtains state-of-the-art performance. Keeping parameters or FLOPs constant, GP-ViT shows improved performance over previous methods in all cases.  

strength_and_weaknesses: Strengths:
1. The paper is clearly written. The method is explained well and carefully evaluated. 
2. The idea of group propagation is intelligent and is effective at reducing computational cost while keeping the architecture simple. 
3. The ablation studies and especially the explanation of how the architecture was built are informative and help explain the contribution of different components of the architecture.
4. The method obtains SOTA performance across tasks, with impressive gains when compared to prior methods with similar number of parameters or FLOPs. 

Questions:
How would you scale up GPViT beyond L3 to obtain better performance? Are there marginal gains to be had on current datasets by further scale up?

clarity,_quality,_novelty_and_reproducibility: See  above. 
summary_of_the_review: The paper proposes a novel, yet uncomplicated, architecture for non-hierarchical transformers for visual recognition tasks and obtains SOTA performance on these tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new transformer-based architecture that enables the extraction of high resolution feature maps from an images, while avoiding hierarchical downsampling in intermediate layers. To deal with the quadratic complexity of normal attention layers, a set of latent group representations is learned that collects information from the high resolution tokens. The latent group representations then mix their information based on an MLPMixer block and in turn the image patch tokens can cross attend to the update group features. Furthermore local attention is used for self-attention of the patch tokens. Using these principle, the complexity is linear in the number of groups and in the number of tokens, allowing a higher number of used input tokens. The resulting architecture is evaluated on several tasks (classification, detection, segmentation) and achieves good results compared to other methods with similar FLOP counts.
strength_and_weaknesses: Strengths:
- While the idea to use such groups to reduce the complexity of attention is not really very novel, the application in this specific architecture is very interesting and show some promising results. Furthermore, a lot of the design choices seems rather general and similar systems could be applicable to other modalities such as point clouds, or potentially even multi-modal data.
- The overall direction of not using intermediate downsampling is interesting and worth investigating.
- The paper is easy to read and quite straightforward to follow.

Weaknesses:
- I think there is quite some rather related work that is not discussed. For example the Perceiver models by DeepMind have a different focus, but they are pretty similar in the underlying idea. Mask2Former also has a similar transformer part, albeit it works on top of a backbone to extract basic features and similar ideas where also explored in "Generative Adversarial Transformers". These are just a few I can recall off the top of my head. I wouldn't be surprised if there are a lot of other similar approaches. I think it is crucial that such related work is discussed with more care. Given that this is one main part of the contributions, the paper loses quite a bit of novelty in my mind. (And just because this uses an MLPMixer instead of vanilla attention does not make the model inherently different!)
- I'm quite sad to see yet another paper that simply claims a method achieves state of the art results on some task, when it's obviously not true. Simply looking at other approaches, it becomes clear that the numbers are far from state of the art. I know the focus here is likely on models with similar FLOP counts, but in most of the sentences where claims about being state of the art are made, this fact is simply omitted. It would be very important to clarify this! And apart from that, it would also be interesting to see how well this type of architecture generalizes to bigger versions of a model with similar FLOP counts than the actual state of the art. This should clearly be clarified more!
- Just looking at parameter counts and FLOP counts is not really giving a clear picture, but the actual throughput of a model is also important. (Have a look at the paper called "The Efficiency Misnomer".) Considering this architecture, I wouldn't be surprised if such a comparison would actually be favorable for the model, but I think it's sad that we don't see such numbers.
- I think the idea of the approach is interesting and it would be great to see a bit more experiments with respect to the design choices that went into the different building blocks. E.g. how many groups should be used, or why is the MLPMixer used instead of normal self-attention? Indeed some of these things are discussed in the appendix, but I think they should be featured more prominently in the main paper in order to highlight the important aspects of this architecture design.
clarity,_quality,_novelty_and_reproducibility: Overall the paper is pretty clear and easy to follow. Especially the overview figure of the architecture clearly explains the main building blocks. Even though I am not aware of approaches that follow the exact same approach, there are quite some very related existing methods that are not discussed here and I think this limits the novelty. Nevertheless a deeper view into this architecture could be interesting, but sadly the results here are more focused on showing "state of the art" performance. As such we also don't gain a lot of novel insights. Given the authors state that code will be released upon acceptance, I assume the results should be reproducible. Furthermore, the results are based on several MM toolkits, potentially further improving the reproducibility.
summary_of_the_review: To me the core approach is interesting, but I'm not very convinced by the FLOP constrained evaluations. Furthermore some important related work is missing. Together these things limit the novelty and contributions quite a bit in my opinion and as such I am leaning more towards rejecting the paper. But I'm interested to see the other reviews and the rebuttal and would be willing to change my mind here.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present GPViT, a non-hierarchival transformer model for visual recognition. The authors argue that by not using a hierarchical approach, and by relying only on high resolution features they achieve better recognition when fine grained details are needed for recognition. The authors introduce the Group Propagation Block to exchange global information tokens which achieves runtime efficiencies under these scenarios. They evaluate their approach on image classification. semantic segmentation object detection and instance segmentation.
strength_and_weaknesses: Strengths:

S1: tested on a variety of tasks

S2: a good discussion on past transformer work, and the paper's place within this past work is discussed

S3: thorough testing and a thorough ablation study is performed. The authors show that their model achieves good accuracy. For imagenet however it is not always clear to me that there is an improvement in terms of the number of parameters used. Discuss this in more detail in the results



Weaknesses:

W1: unless I missed it, will source code be provided. I did not notice a discussion on whether source code will be provided to make it easy for someone to replicate these results.

W2: section 2: clarify what you mean by "inductive biases".

W3: Eq 2: where is the concat operation shown in Figure 2?
clarity,_quality,_novelty_and_reproducibility: See my comments above. Overall the paper is well written. it has a few items that need clarification but i do not think these constitute ground for rejection. The model is described in detail. It is not clear to me whether source code will be provided. A clarification from the authors is needed on this. The results are good and I think the ICLR audience would be interested in the paper.
summary_of_the_review: As I indicated above, the results are good and I think the ICLR audience would be interested in the paper. The effort made to decrease the number of parameters should be appreciated by the ICLR audience.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a novel implementation of a group-based transformation of high-resolution features for general vision transformers. Experiments are comprehensive and the analysis is well-done. 

Common Strengths:
1. Well-written. 
2. A good implementation of using grouped features to save computation of self-attention. 
3. Experiments are comprehensive, and of general interest to the community.

Weakness:
1. Missing important related works.
2. Missing some ablations.
justification_for_why_not_higher_score: The implementation of the proposed group-based self-attention is interesting but not groundbreaking. 
justification_for_why_not_lower_score: Nice implementation and comprehensive experiments would definitely attract the major audience in computer vision.
Decision: Accept: notable-top-25%