Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an efficient and general component GrafT for Transformer-based vision architectures. The GrafT can introduce multi-scale features for early-stage layers, thus achieving better results. The authors demonstrate the effectiveness of the proposed GrafT on several tasks and datasets.

strength_and_weaknesses: Strengths
* The description of GrafT is clear and the figures are nice, making the paper easy to understand.
* The authors empirically demonstrate the effectiveness of GrafT on multiple tasks.

Weakness
* The improvement on ImageNet-1K is marginal. In Table 1, Swin-T+ GrafT outperforms RegionViT-S by only 0.1 but introduces 3.4M more parameters. 
* GrafT should generally introduce more FLOPs and parameters for the backbone model.  However, in Table 2, the FLOPs of DeiT-T+GrafT are even smaller than the original DeiT-T.  Moreover, are the training settings for DeiT+ GrafT and DeiT-T fair?
* The ablation in Table 6-(a) is not sufficient. What are the results with more scales? Is scale = 3 the best?
* Generalization performance. The authors only validate the effectiveness of GrafT on Transformer-based architectures. If the multi-scale features are beneficial, I wonder about the performance of GrafT combined with classical CNN models such as VGG and ResNet. 
clarity,_quality,_novelty_and_reproducibility: There are some details in experiments that should be clarified, as described in the weakness. 
summary_of_the_review: I have several concerns claimed in the weakness part, thus I choose marginally below the acceptance. 
I may modify the score according to the author's response.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an add-on component (termed GrafT) that considers global dependencies and multi-scale information throughout the transformer network. The proposed method is evaluated on several fundamental visual recognition problems.
strength_and_weaknesses: Strength:

The proposed method is evaluated on several fundamental vision tasks, although the evaluation on each task is limited.


Weaknesses:

The performance of ViL (Zhang et al., 2021) is much lower than that in the original paper. Please check the experiments and use ViL as the baseline to evaluate the improvement because ViL is the state-of-the-art method.

This paper should add the proposed GrafT to models with different model sizes. In the current version, the proposed method is only evaluated for Swin-T for object detection, instance segmentation and semantic segmentation. It is only evaluated for Swin-T, Swin-S and DeiT-T for image classification, which is insufficient to demonstrate the effectiveness of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The experiments are insufficient to demonstrate the effectiveness of the proposed method.
summary_of_the_review: The insufficient experiments and unfair comparisons are my main concerns. Please see the above comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 	The paper proposes to add a FPN-like parallel branch to existing vision transformers that fuses multi-scale tokens with consecutive downsampling, upsampling and window attention. Design choices of each part are studies empirically and improvement over Swin-T on ImageNet, COCO, ADE20k are reported. 
strength_and_weaknesses: Strength: 
		The idea of adding such multi-scale attention branch in parallel is interesting and design choices of downsampling and upsampling are carefully validated empirically. 

	Weaknesses:
1.	The paper says the method adds minimal complexity. The proposed module actually introduce quite the parameters and flops, 17% #param increase and 14% FLOPs increase when applied to Swin-T, 29% #param increase and 11% FLOPs increase for Swin-S.  And the model performance, compared with RegionViT-S, has only marginal 0.1 improvement, but 10% more #params.
2.	Can authors show the performance when applied to large models, e.g. Swin-B?
3.	Throughputs and memory? As the method adds parallel branch to existing architectures with consecutive downsamling and upsampling, which is hard for parallelization,  can authors show some throughputs comparison w v.s w/o GrafT. Also, each layer will have much more token maps, can authors compare the memory consumption as well? 

clarity,_quality,_novelty_and_reproducibility: The paper has good clarity and the proposed method is novel and validated with ablation studies. Reproducibility is medium as the method involves lots of implementation details. 
summary_of_the_review: Overall, the paper proposes a novel way to fuse multi-scale information in vision transformers, which is to add a FPN-like parallel branch to each of the attention block. It would make the paper better if some concerns regarding computation, throughputs, memory and applying to large models are resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an efficient  add-on for Transformer-based vision architectures, calles GrafT. GrafT allows to introduce multi-scale features for early-stage layers in the network and can thereby lead to improved results. 
The proposed results improve over the presented baseline , yet there have been concerns on the correctness of reported resultsin the first submission, as well as for example  the number of flops and the dependence on the model sizes.
The rebuttal addressed some of these concerns, yet could not fully resolve them in the current form. We strongly encourage the authors to pepare a revised version of the paper including the additional experiments as well as the corrected number to prepare a submission for  future submissions. 


justification_for_why_not_higher_score: Even though the rebuttal added and clarified several aspects, the evaluation regarding model size is only evaluated on one specific architecture. Further the newly reported number of paramaters indicated that GrafT indeed increases the sumber of parameters to achieve better results. An exact ablation with comparable model sizes would be much stronger here.
Decision: Reject