Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a novel module to combine the local and global context. The key of this method is to introduce global query tokens into the local context module. The advantage of this method is that it can merge the context information from short and long-term ranges. The authors do experiments for image classification, detection, and segmentation tasks to show the effectiveness of the method.
strength_and_weaknesses: Strength
1. The experiments show this method is effective and can be generalized to many different tasks.
2. The problem is essential for network architecture design and the method is straightforward.
3. The visualization results show the effectiveness of this method.

Weakness
1. The combination of local and global context is a long-standing problem and has been explored by many methods. The method for merging local and global contexts lacks novelty.
2. The improvement between the proposed method and other SOTA methods is minor.
clarity,_quality,_novelty_and_reproducibility: This paper is easy to follow. The novelty of this paper is somehow limited. I think researchers can reproduce this paper easily.
summary_of_the_review: Please follow the previous parts.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposed a new hirerchical transformers architecture. The paper revisit the attention mechanism and the downsampling layers.
The claim of the paper are the following: 

• We introduce a compute and parameter-optimized hierarchical ViT with reparametrization of the
design space (e.g., embedding dimension, number of heads, MLP ratio).

• We design an efficient CNN-like token generator that encodes spatial features at different resolutions
for global query representations.

• We propose global query tokens that can effectively capture contextual information in an efficient
manner and model both local and global interactions.

• We introduce a parameter-efficient downsampling module with modified Fused MB-Conv blocks
that not only integrates inductive bias but also enables the modeling of inter-channel dependencies.

• We demonstrate new SOTA benchmarks for : (1) ImageNet classification with Pareto fronts on
ImageNet-1K for model size and FLOPs (see Fig. 1), and (2) downstream tasks such as detection,
instance segmentation and semantic segmentation on MS COCO and ADE20K, respectively.

strength_and_weaknesses: Strength
- The paper is well written and easy to follow
- There is evaluation on different tasks

Weaknesses:
- Missing comparaison and discussion:

a) The global query tokens claim: The proposed global query token behaviour seem to be quite similar to the attention pattern mechanism in the BigBird paper[1] and also the local global attention in the paper EdgeViT[2]. It is necessary to discuss the contribution of the proposed method to these two papers. Because at present the contribution of the paper on this aspect is not clear.

[1] Zaheer et al., Big Bird: Transformers for Longer Sequences
[2] Pan et al., EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers

b) The compute and parameter-optimized hierarchical ViT and SOTA claim: in Figure 1 and Table 1: Indeed, the EdgeViT paper exploit similar ideas (local-global attention, Hierachical architecture) and seem to have better FLOPs-accuracy and Parameter accuracy trade-off (EdgeViT-S: 11.1M params, 1.9G FLOPs, 81.0 top-1 on ImageNet vs GC ViT-XXT 12M params, 2.1G FLOPs, 79.8 top-1 on ImageNet)  It is therefore necessary to add this architecture to the comparison. Moreover, the paper mention EfficientNetV2[1] because GC ViT exploit some similar blocks and seem also better/similar for the FLOPs-accuracy and parameter accuracy trade-off (EfficientNetV2-M 54M params, 24G flops, 85.1 top-1 on ImageNet  vs GC ViT-L  201M params, 32.6 G FLOPs, 84.6 top-1 ImageNet). So it's important to add also EfficinetNet-v2 in the comparaison.

Given these elements it is probably best to down tone the state-of-the art claims in the paper. Because it is not clear which state of the art it is.

[1] Tan et al., EfficientNetV2: Smaller Models and Faster Training

b) The efficient CNN-like token generator: Many papers have proposed convolution blocks for token generation like LeViT[1] or XCiT[2].
There is no discussion or comparison of performance with the previous approach. It is therefore impossible to evaluate the contribution of the paper on this point.

[1] Graham et al., LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
[2] El-Nouby et al., XCiT: Cross-Covariance Image Transformers

- Overfitting evaluation: For the ImageNet dataset there is no separate test and validation set so it's important to evaluate the level of overfitting by doing evaluation of the models on ImageNet-v2[1].

[1] Recht et al., are we done with ImageNet?

- Missing metrics: There is only params and FLOPs in the different table. It's important to measure other trade-off like Latency and memory consumption. Indeed, the proposed architecture use MBconv which are known to be good on trade-off FLOPs accuracy and parameter accuracy.



clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. Unfortunately, the claims did not seem to be supported and the novelty seems very marginal. The paper seems to be reproducible
summary_of_the_review: The idea of the paper is interesting but unfortunately the contribution does not seem to be justified experimentally. The paper seems to be incremental. There is a lack of discussion and comparison with very similar approaches in the literature.


====== Post rebuttal ====

The rebuttal raised a very important concern: 

The authors report maximum accuracy during training by evaluating at each epoch in order to maximise performance on the ImageNet-1k validation set (there is no separate test set for ImageNet). Although the authors argue that it is a common practice, there is no evidence in the mentioned papers that this is the approach used. Furthermore I consider this approach to be detrimental to the community as it puts scientific understanding behind the optimal performance.

On the 3 logs provided the impact of this practice is important on the log with the bigger model +0.2% and impact the conclusion. The authors no longer have the other logs of their experiments, especially those of the ablation, so it is impossible to know the impact of the method. There is no multi-seed evaluation that shows that 0.1% on ImageNet-v2 is significant although it is claimed in the rebuttal.

In the current state the experiments are mainly designed to optimise the performance of the GC ViT architecture at the expense of an understanding of the impact of each component of the proposed method.

So I will lower my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced GC ViT that can efficiently capture global context by utilizing global query tokens and interact with local regions. 
This paper demonstrate through experiments that this simple modification on traditional ViTs could get SOTA results for image classification on ImageNet-1K dataset and downstream tasks.
strength_and_weaknesses: Strength:
1. the idea is simple and straightforward, from the experiment it seems the improvement is reasonable.

Weakness:
1. The comparison of GC-VIT to other models clearly shows the quantitative improvements in Tab.1. In addition, the ablation study shows the proposed module could be quite plug-able into existing models. It will be more appealing to see how GC module improves existing models by simply plug them into them. This will inevitably add #param  and FLOPS, but it gives more intuitive evidence on the effectiveness. 
clarity,_quality,_novelty_and_reproducibility: There are no problems with Clarify, Quality and Novelty. 
However, since the paper is based on plain empirical/experimental findings. The released code will be critical in verifying the reproducibility of the paper. 
summary_of_the_review: Overall this paper proposed a simple and yet effective module that could add reasonable improvement to current ViT. This work could be useful to the community once the code is released for verification/comparison.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel global context vision transformer built with a simple combination of global context self-attention modules and local self-attention modules to model
both long-range and short-range interactions. Besides, the authors propose to model the inter-channel dependencies with an inverted residual block.
The proposed GC ViT shows encouraging results across image classification, object detection, instance segmentation, and semantic segmentation tasks.

strength_and_weaknesses: > Strengths

✅ The presented idea is very simple and easy to follow.

✅ The overall writing is of high quality, such as Figure 2, Figure 3, Figure 4, Figure 5, and Figure 6 is all of GOOD quality and help the readers to understand the key
idea easily.

✅ The authors provide rich details and conduct rich experiments to verify the effectiveness of the proposed approach.

> Weaknesses

❎ The experimental results are relatively WEAK and the authors are encouraged to verify the advantages over Swin-T based on the very recent object detection and segmentation frameworks
such as DINO-DETR[1], H-DETR[2], and Mask2Former[3]. For example, Mask2Former+Swin-B (pre-trained on ImageNet1K) achieves PQ=55.1, AP=46.7, and mIoU=52.4 on COCO panoptic segmentation, COCO instance
segmentation, and ADE20K semantic segmentation tasks respectively. I would like to expect the authors to present stronger results and release the related code upon acceptance.

[1] https://github.com/IDEA-Research/DINO

[2] https://github.com/HDETR/H-Deformable-DETR 

[3] https://github.com/facebookresearch/Mask2Former 

❎ According to Table 1 & Table 2, the proposed GC Vit backbones require more parameters and GFLOPs than the previous approaches such as CSWin and ConvNeXt. For example, GC ViT-B requires 1018 GFLOPs vs. ConvNeXt-B requires 964 GFLOPs and GC ViT-B only gains 0.2 measured by AP^box.

❎ The authors are encouraged to verify the advantages of the proposed backbone under larger model scales. Specifically, the authors should construct a GC ViT-L and GC ViT-H instead of only comparing the models under moderate scales as scaling up is one of the most important aspects of an important backbone design.
clarity,_quality,_novelty_and_reproducibility: > Clarity

Good.

> Quality

Good.

> Novelty

Good.

> Reproducibility

No code is available.
summary_of_the_review: Designing new versatile architecture is always very important for the whole computer vision community. The overall quality is OK but the results are relatively WEAK to support the claim that the proposed method really outperforms the previous SoTA methods such as Swin-Transformer.
Please carefully address the above-listed weaknesses. I will increase the ratings if the authors could well address these concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces the GC ViT architecture that combines local and global contexts using global query tokens. The architecture includes a parameter-efficient downsampling module. The authors claim that the GC ViT architecture achieves state-of-the-art results on various tasks, including image classification, object detection, instance segmentation, and semantic segmentation.

The reviewers praised the paper for its clear exposition, simple and practical ideas, and comprehensive experimental evaluation.

The reviewers have noted that the proposed approach lacks novelty, as the community has explored similar ideas in previous work, and that the experimental results are relatively weak. The proposed architecture surpasses the comparison points, especially for model variants with small parameter counts, namely VIT-XT / VIT-T / VIT-S. Reviewers have suggested several missing and important comparison points, for instance, EdgeVit and EfficientNet. Regarding EdgeVit, the authors included results in Table 1 but did not include them in Fig. 1, claiming that EdgeVit is specialized for small model ranges. Regarding EfficientNet, the authors have refused to include the comparison. They deemed the comparison unfair, partly because of a mismatch in the number of pretraining epochs. On the other hand, GC ViT was trained for 320 epochs (see logs), while in the paper, the details describe 300, and the authors deemed the difference not having “a considerable impact on the final accuracy.” Reviewer V9pP asked for investigations about the performance of larger models, namely GC VIT-L. The performance of this model has evolved during the rebuttal phase, with the authors editing their comments along the way, making it very hard to judge and track the performance.

I have carefully read the paper, the reviews, the discussion, and all comments on OpenReview and am not making my decision based on partial information. My decision is informed by my own, independent, technical assessment of the work. In its current form, I recommend this paper for rejection. The authors should take the feedback from the reviewing process into account, improve the paper accordingly and submit it to the next venue.
justification_for_why_not_higher_score: The weakness section of the meta review describes this points in details.
Decision: Reject