Grafting Vision Transformers
Authors: Jongwoo Park
Kumara Kahatapitiya
Donghyun Kim
Shivchander Sudalairaj
Quanfu Fan
Michael S Ryoo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformers
Multi-scale
Multi-branch
Grafting
Classification
Semantic segmentation
Object detection

Abstract: Vision Transformers (ViTs) have recently become the state-of-the-art across many computer vision tasks. In contrast to convolutional networks (CNNs), ViTs enable global information sharing even within shallow layers of a network, i.e., among high-resolution features. However, this perk was later overlooked with the success of pyramid architectures such as Swin Transformer, which show better performance-complexity trade-offs. In this paper, we present a simple and efficient add-on component (termed GrafT) that considers global dependencies and multi-scale information throughout the network, in both high- and low-resolution features alike. GrafT can be easily adopted in both homogeneous and pyramid Transformers while showing consistent gains. It has the flexibility of branching- out at arbitrary depths, widening a network with multiple scales. This grafting operation enables us to share most of the parameters and computations of the backbone, adding only minimal complexity, but with a higher yield. In fact, the process of progressively compounding multi-scale receptive fields in GrafT enables communications between local regions. We show the benefits of the proposed method on multiple benchmarks, including image classification (ImageNet-1K), semantic segmentation (ADE20K), object detection and instance segmentation (COCO2017). Our code and models will be made available.

Under review as a conference paper at ICLR 2023
GRAFTING VISION TRANSFORMERS
Anonymous authors Paper under double-blind review
FRN
83.5
Grafp
GrafT
Gradt!
Courre
ImageNer top- (%)
83.0
1.4%
Transformer
82.5
Swin GrafT
Swin
Region VIT
82.0
Grofr
FoomFormer
PvT
8L.
TNT
GraPT
TZF
8 x
VIL
Multi-scle tokens
6
10
12
GFLOPs (b)
Figure 1: We introduce GrafT, add-on component which makes of global and multi- scale dependencies arbitrary depths of network. (a) An overview of how GrafT modules are branched-our (or grafted) from a backhone Transformer. Each GrafT may consider multiple scales of features (i.c., token representations), widening a network efficiently while relying on the backbone to perform most of the computations. It can be adopted to both homogeneous (e.g. ViT (Dosovitskiy et al., 2020)) and pyramid (e.g., Swin (Liu et al., 2021)) architectures. (b) Performance-conpplxxtyy trade-off of Swin+GrafT, comparison with previous related methods. GrafT shows considerable performance gains with minimal increment complexity.
ABSTRACT
Vision Transformers (ViTs) have recently become the state-of-the-ard across many computer vision tasks. In contrast convolutional networks (CNNs), ViTs en- able global information sharing even within shallow layers of a network, i.e., among high-resolution features. However, this perk later overlooked with the success of pyramid architectures such as Swin Transformer, which show better performance-comllexin trade-offs. In this paper, present simple and effi- cient add-on component (termed GrafT) that considers global dependencies and multi-scale information throughout the network, in both high- and low-resolution features alike. GrafT can be easily adopted in both homogeneous and pyramid Transformers while showing consistent gains. It has the flexibility of branching- out at arbitrary depths, widening a network with multiple scales. This grafting operation enables us to share most of the parameters and computations of the backbone, adding only minimal complexity, but with a higher yield. In fact, the process of progressively compounding multi-scale receptive fields GrafT en- ables communications between local regions. We show the benefits of the pro- posed method on multiple benchmarks, including image classification (ImageNet- IK), semantic segmentation (ADE20K), object detection and instance segmenta- tion (COC02017). Our code and models will be made available.
INTRODUCTION
Self- Self-attention mechanism Transformers (Bello et al., 2019) has been widely-adopted in language domain for some time now. It can look into pairwise correlations between input sequences, leamn ing long-range dependencies. More recently, following the seminal work Vision Transformers (ViT) (Dosovitskiy et al., 2020), the vision community has also started exploiting this property,

---

Under review as a conference paper at ICLR 2023
showing state-of-the-ar results various tasks including classification, segmentation and detec- tion, outperforming convolutional networks (CNNs) (He et al., 2016; Tan & Le. 2019; Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019; Brock et al.. 2021). Motivated by this suc- cess, many variants of vision transformers (e.g.. Dei'T (Touvron et al., 2021), Cross ViT (Chen et al., 2021), TNT (Han et al., 2021)) emerged, inheriting the same homogeneous structure of ViT (i.e., a structure w/o downsampling). However, due to the quadratic complexity of attention, such a structure becomes expensive, especially for high-resolution inputs and does not benefit from the semantically-riee information present in multi-scale representations.
To address these shortcomings, Transformers with pyramid structures (i.e., structures w/ down- sampling) such as Swin (Liu et al., 2021) were introduced with hierarchical downsampling and window-based attention, which can learn multi-scale representations at a computational complex- ity linear with input resolution. As result, pyramid structures become more suited for tasks such as segmentation and detection. However, still, multiple scales arise deep in to the network due to stage-wise downsampling, meaning that only the latter stages of the model may benefit from them. Thus, we poise the question: what if we can introduce multi-scale information even at early stages of Transformer, without incurring heavy computational burden?
Previous work has also looked into the direction above, both in CNNs (Szegedy et al., 2015a) and in Transformers (Chen et al., 2021; 2022). However, models such as ViT requires carefully tuning the spatial ratio of two feature maps in two branches and Region ViT needs considerable modifications to handle multi-scale training. To mitigate these issues, in this paper, we propose simple and efficient add-on component called GrafT (see Figure I-(a)). It can be easily adopted in existing homogeneous or pyramid architectures, enabling multi-scale features throughout network (even in shallow layers) and showing consistent performance gains, while being computationally lightweight. GrafT is applicable at any arbitrary layer of network. It consists of three main com- ponents: (1) left-right pathway for downsampling, (2) a right-left pathway for upsampling, and (3) a bottom-up connection for information sharing at each scale. The left-right pathway uses series of average pooling operations to create set of multi-scabe representations. For instance, if GrafT is attached to a layer with (56 x 56) resolution, it can create scales of (28 x 28). (14 x 14) and (7 x 7). We then process information at each scale with L-MSA block, local self-attention mechanism (e.g., window-attention)
which becomes global-attentio in the coarsest scale, as window-size becomes the same as the resolution. Next, the right-left pathway uses series of learnable and window-based bi-linear interpolation (W-Bilinear) operations to generate high-resolution features by upsampling the low-resolution outputs of L-MSA- which contains global (or high-level) se- mantics extracted efficiently, at a lower resolution. Such upsampled features are merged with high- the-left, resolution features of the branch-to- traneh-to-the-leftt, which contain lower-level semantics, as also done in Feature Pyramid Networks (Lin et al., 2017). Refer to Figure 2-(b) for detailed view.
GrafT is unique in the sense that it can extract multi-scale information at any given layer of Trans- former, while also being efficient. It relies on the backbone to do the heavy-lifting, by using a minimal computation overhead within grafted branches, in contrast having completely-separate branches as in Cross ViT (Chen et al., 2021). In our evaluations, we show the benefits of GrafT in both homogeneous (ViT) and pyramid (Swin) architectures, across multiple benchmarks: on ImageNet- K (Deng et al., 2009), +3.9% with ViT-T+GrrfTT, +1.4% with Swin-T+GrafTT and +0.5% with, on COC02017 (Lin et al., 2014), +1.1 AP' for object detection and +0.8 APM for instance segmentation with Swin-T+GrafT, and on ADE20K (Zhou et al., 2017) semantic segmentation, +1.0 mIOUs +1.3 mIOu mm with Swin-T+Graff. Figure 1-(b) shows the performance-complexit trade- off Swin-T+GrafT on ImageNet-1K.
GRAFTING VISION TRANSFORMERS
Our goal is to provide multi-scale global information to the backbone transformer from the bottom layer so that high-level semantics from GrafT can help the transformer to construct more efficient features. Since Graft is modular, it can be applied to various transformer architectures. We select two representative transformers, VIT (Dosovitskiy et al., 2020) based homogeneous structure and Swin (Liu et al., 2021) based on a pyramid structure, to show that GrafT is a general purpose module.
2

---

Under review as a conference paper at ICLR 2023
Tokeni
Craff p-a
Receptive Pald of Grafl
Receptive Fielá Gradi
Receptive Graff
FFN
Uiconpling
Mark
GrafT n-11
W-Bilinear
Anti- Linear GELL LN
g42
Interpolation
aliasing
Upramppinn
L-MSA
L-MSA
Downsampping
IN
Chssappiing
LN
GELU
Avg. Pool
(0.00)
Tranfonser
GrafITy
backbone
L-MSA block
Domscumpling
(b)
Figure 2: Detailed view of GrafT. (a): Here, we show the receptive field of attention mechanisms in cach scale and how it changes by merging with information from the corresponding lower-rosolution scale (i.c., from branch-to-the-righu) Multiple scale created with downsampling, processed with local attention (in L- MSA)-whiee becomes global attention in coarsest scale (GrafT)). and merged back with upsampling. When a lower-resolution representation is upsampled and merged, the effective receptive field increases, essentially giving access to efficiently errtinttl-eettratedd extracted global (or larger-local) information. (b): We present all the components and grafting/ merging points in GrafT. We graft prior self atffaattenion block in the backbone, and merge prior FFN so that can reuse the heavy computations. Downsampling (left-right patfeway) uses weight average pooling to create rescr-resolution features, whereas upsampling (right-left pathway) uses learnable window-based biner interpolation (W-Bilinean to upscale. The processing unit within GrafT module is a L-MSA block, which performs local-attention. When merging features to higher-resolution, we element- wise addition.
2.1 OVERALL ARCHITECTURE
The overall architecture of GrafT illustrated in Figure 2-(a). We start with backbone transformer (i.e., ViT or Swin) and simply attach GrafT to some vertical layers of the backbone transformer. For an input image with size of H W x 3, the patch tokens the first vertical layer is W C
after the patch embedding.
GrafT is horizontal pyramid structure which consists of left-right pathway (series of GrafT down- sampling), right-left pathway (series of GrafT upsampling), and bottom-up connection (multiple GrafT L-MSA) as shown in Figure 2-(b). The imput feature GrafT goes through left-right pathway (downsampling), right-left pathway (upsampling), bottom-up connection (L-MSA) and becomes feature having strong high-level semantics at multiple scales which then gets fused into the back- bone transformer. Specifically, for the GraiT attached to the vertical layer at Sth stage, the input feature to the GrafT has the size 4-S-T X - x C' where r = 2 for pyramid structure and r for homogeneous structure. Then, fuse the feature from GrafT the original backbone, which will be described the later section.
2.2 TRANSFORMER-GRAN BLOCKS
In this section, describe each operation in Figure 2-(b). Let Xd,6 as input tensor vertical layer d and horizontal downsampling level in GrafT with the shape: x l xxxxx Hbb Wbb C is the height, width, channels of the feature map dd at horizontal level b respectively.
Left-right pathway (downsampling): The left-right pathway creates serial feature maps at sev- eral scales with the downsampling rate which follows the vertical pyramid downsampling rate. For example, in the first stage of Swin+GrafT, GrafT creates three downsampled feature maps (X1, X2 x3) by downsampling the input feature map X from the backbone with downsampling

---

Under review as a conference paper at ICLR 2023
rate 2. In left-right pathway, use adaptive average pooling for downsampling:
xd A(X4.) p(GELLU(LN(X*4)))
(I)
A is lightweight downsampling function which consists of LayerNorm (LN), GELU, and adap- tive average pooling (p). which is more cost-efficient than other downsampling methods such as cross attention and linear projection (see Table 5). It maps the input Xd to downsampled feature xx
x * + RH 1 where Hb+1 < H W < W. xb+1 is the input at the horizontal downsampling level b b-1. 1. Downsampling happens sequentially over horizontal layers to progressively abstract the fine information in coarse features.
Right-left pathway (upsampling): The right-left pathway hallucinate serial higher resolution fea- ture maps by upsampling low resolution feature maps. These upsampled feature maps enhanced by feature maps from bottom-up connection that retain spatially more accurate activations and lower- level semantics. This enhancement process is similar to FPN (Lin et al., 2017). In right-left path- way, GrafT upsampling uses learnable W-Bilinear (window-ber baase bilinear) interpolation which is more cost-efficient than other upsampling methods such as cross attention and nearest neighbor in- terpolation as shown in Table 5. Learnable W-Bilinear interpolation solves the aliasing problem by embedding anti-aliasing weights, the sigmoid of positional embeddings, in the feature maps. Anti-aliasing weights learns perturbations for each grid in the feature map that can prevent alias- ing effect. W-Bilinear interpolation is adopted to address the semantics discontinuity between local regions. Finally, our upsampling is defined as:
Zd.b+1 = 4b ( (d.++11+ = ((Ean O a(Z T(Z4.+1),
7d.b+1 Mon 2...
((2d.b+))) Smerin
(2)
T is lightweight upsampling function mapping the representation back to the spatial resolution in the previous horizontal level. This function is designed to upsample output features from L-MSA. First, given the tensor Zd.6+1, the output from L-MSA, channel mixing (a) is applied to align chan- nels before interpolation. consists of LayerNorm (LN), GELU, and a linear layer. Next, anti- aliasing embeddings (Eaa) are multiplied to resolve the aliasing problem. The proposed anti-aliasing embeddings are the output of sigmoid function on position embeddings Zp E R xWI+1 xC" It learns to provide perturbations in the spatial dimension that prevents interpolation from suffering the aliasing problem. It is a simpler and lighter method compared to 3x3 convolutions (Lin et al., 2017). Lastly, W-Bilinear interpolation ($) maps 7d.++1 R x
7d.b+1 H''"" can be described as mapping the low resolution feature in each (m, n)th window 2, into high resolution feature in (m, n)tth window um..".. -ay"""" (im, Jn) is a spatial position (1, f) in (m, n)th window where E j E J, m E M, n E N. (Um. 2n,) is a spatial position (u, v) in (m, n n)th window where u E rnI,v v E Tw.J. The Tw the height and width ratio between feature resolutions in two consecutive horizontal levels. Upsampling happens sequentially over horizontal layers to progressively upscale spatial resolution of coarse features.
The local self-attention method from the backbone transformer, limits the self- attention in a local region which causes the discrepancy of semantics the boundary of local re- gions. Therefore, bilinear interpolation is applied in each local region multiple times instead of the entire feature map once. The bottom-up connection merges lower-level feature maps enhanced by L-MSA and ligher-level feature maps upsampled by W-Bilinear interpolation by element-wing addition. The merging process iterates until it generates the high-level feature map that has the same spatial size as the input feature map X from the backbone. In the example above, the coarsest feature map X goes through L-MSA and becomes 73, a feature map having the highest-level se- mantics. 2 is generated by merging upsampled 2 with output feature from L-MSA which applies local self-attention on X2. This process iterates until the finest feature map 2 which has the same spatial size of the X is produced. Zo is then fused into the output feature map from L-MSA in the backbone transformer and proceed to FFN block.
L-MSA in GrafT:
zd.++ xd.+1 + [L-MSA[LN(XdbH)] (. - 2d.b+2
(3)
is a simple and light block that fuses multi-scale features. It uses a standard window-bute bbsed MSA from Swin (Liu al., 2021) encode fine features dd.b+1, It uses simple element- wise addition to
4

---

Under review as a conference paper at ICLR 2023
fuse this fine feature and coarse feature 2d.b42 R 1h xw coming from one deeper horizontal level. Lastly, Skip connection added to produce output Zdb+1 RRH x Wh+
Bottom-up connection with multiple high-level semanties: In GrafT, L-MSA uses local self- attention enhance downsampled feature maps but it limits receptive field within each local region. It is important to exchange information among local regions so that feature can embed not only local structure but also global structure. In the left side of Figure 2, the features at the bottom are the output of GrafT L-MSA where receptive field is limited to each local region. The feature map at b+2 level 2 has a receptive field, blue color line, that covers the entire feature map. The feature map at b 1 level, Zb+1, originally suffers the information discrepancy between local regions due to local receptive field. When Zb+2 is merged into 2b+1, 7 inherits receptive field from 20+2 and this newly added global receptive field provides higher-level semantics that can understand the relation between local regions. 2h, the feature map at level, also originally suffers the information discrepancy between local regions. When 2b+1 merged into 20, 20 inherits receptive fields from both Z Zh+2 and this newly added receptive fields provides multi-scale high-level semantics that can understand the relation between local regions in multiple aspects. Multi-scale receptive fields which are progressively generated from multi-scale features resolve the drawback of local receptive field formed by L-MSA in the backbone transformer. The L-MSAin the original backbone transformer is defined as:
yd.0 = X [L-MSA(X"0) zd11
L-MSA is the local self attention that the transformer in the main branch is using to encode fine feature X E "R xxxx The encoded fine feature element-wise added with coarse feature 2d,1 from the GrafT. Lastly, skip connection is added to produce output y E R xw" xW"xC xC It is interesting that a simple element-wing wise addition successfully fuse the fine feature in the main branch and the coarse feature from GrafT. Thanks to the power of GrafT upsampling method for the robust fusion of the multi-scale features encoded by various MSA. Then, we use the same FFN in the backbone transfomers:
x = yd. -MLP(LN(YY00)
(5)
It is standard transformer block performing channel mixing via LayerNorm (T.N) and MLP on the output of I.-MSA block and adds the skip connection to generate output Xd+1.0 which the input to the next vertical layer.
Computation complexity: Average pooling and bilinear interpolation runs in O(HW) and L- MSA in GrafT follows the complexity of L-MSA in the backbone transformer. Since L-MSA is more complex than @(HW), the complexity of transformer+GrafTT equal to the complexity the pure backbone transformer. For example,
(Swin = ((SWinnGraaTT) = 12HW + HWC
(6)
where H, W is the width and height of feature map and M is the size of window.
EXPERIMENTS
3.1 IMAGENET-IE IK CLASSIFICATION
ImageNet- IK (Deng et al., 2009) is classification benchmark with anno- tations 1000 categories. contains 1.2M training images and 50K validation images. our evaluation, report Top-1 accuracy (%) on single-crop setting along with complexity metrics (measured in Parameters and FLOPs). We train our models with the standard settings: for 300 epochs with 224x224 resolution inputs. We use timm (Wightman, 2019) library for our implemen- tation. We use the original hyperparameter settings for each of the backbone. We use increasing stochastic depth with ratio of 0.25, 0.4 for Swin- Twin-T+GrafT and Swin-S+GrafT respectively.
Dataset and Settings:
5

---

Under review as a conference paper at ICLR 2023
Table I: Comparison GrafT with pyramid architectures on ImageNet-
Model
Params (M)
FLOPs
Acc. (%)
Model
Params
FLOPs
Acc.
(G)
(M)
(%)
PVT-M (Wang et al., 2021)
44
6.7
81.2
PVT-L
81.7
PoolFormer-S36 (Yu al. 2022)
31
5.2
81.4
PoolFormer-M33
56
9.1
82.1
I2T,-14 (Yuan et al., 2021)
22
6.1
81.7
T2T,119
39
9.8
82.2
TNT-S (Han al., 2021)
24
5.2
81.5
14.1
82.9
Swin-T (Iiu 2021)
4.5
Swin-S
ViL-S-RPB (Zhang al., 2021)
25
4.9
82.4
VIL-M
40
8.7
83.5
RigionViT-S (Chen et al., 2022)
31
5.3
82.6
ReT-onViT-M
41
7.4
83.1
Swin-T+Graf] (ours)
34
5.1
(+1.4)
82.7
Swin-S+Graff
9.7
(+0.6) 83.6
CSWin-T
23
4.3
82.7
CSWin-T+GrafT (ours)
29
4.7
(+0.5)
83.2
Results ImageNet-1K: Table
and Table 2 shows the results of the GrafT applied three well- known Transformers that have ho- mogeneous structure and pyramid structure: DeiT, Swin, and CSWin. In the Table ?? Swin-T+Gie TirafT achieves 82.7% top-1 accuracy which 1.4% better than Swin-T. Swin- S+GrafT achieves 83.6% Top-1 accu- racy which is 0.6% better than Swin-S. The Figure I-(b) shows that Region outperforms Swin but Swin+GrafT outperforms RegionViT VIT due the support from GrafT. The gain by the GrafT is drawn by gray arrows. Secondly, CSWin-T+GraTT achieves 83.2% top-1 accuracy, a new SOTA result and it is 0.5% better than CSWin-T. Table shows that DeiT-T+GrafT achieves 76.1% % top-1 accuracy which 3.9% better than DeïT-T. CrossViT outperforms DeiT but DeiT-T+GrafT further outperforms Cross ViT due the support from GrafT. In addition, even though Deif-T+GrafT does not take advantage of the vertical pyramid structure, it outperforms PVT, vision Transformer with the vertical pyramid structure. is replaced by the window-based local attention without window shifting to follow the suggested architecture in Figure 2-(b). We confirm that DeiT with window-bass local attention under- performs the one with the full attention by 2%. Therefore, GrafT delivers 5.9% performance gain (from 70.2% to 76.1 1%) by providing multi-scale high-level information and global receptive fields to the revised DeiT.
Table 2: Comparison of GrafT with homogeneous architectures on ImageNet-IK.
Model
Params (M
FLOPPs (G)
Acc, (%)
D (Touvron et al., 2021)
5.7
1.3
72.2
CiTssViT-9 (Chen al., 2021)
8.6
73.9
PVT-T (Wang 2021)
13.2
75.1
DeiT-T GrafT (ours)
7.9
1,2
(43.9) 76.1
DeiT-T+GrafT, the full attention in the backbone Transformer
3.2 OBJECT DETECTION AND SEMANTIC SEGMENTATION
Dataset and Settings for COC02017: COCO2017 (Lin al., 2014) consists of 118K images. for training. 5K for validation and 20K for testing. We adopt Mask R-CNN(He et al., 2017) framework evaluate StinfT which pretrained on the ImageNet-IK dataset. We use the stochastic depth with ratio of 0.1 and 0.2 for the x (SS) schedule training and 3 x (MS) schedule training and follow the original hyperparameter settings as Swin. Here, (SS) means 12 training epochs with single scale, x (MS) means 36 training epochs with multi-scale.
Table 3: Performance of GrafT on object detection and instance segmentation COCO2017.
Ix (SS)
(MS)
Model
Params
FLOPs
Aph
Apm
AP
Ap
ResNet50 (He al., 2016)
44
260
38.0
34.4
41.0
37.1
PVT-S (Wang et al., 2021)
44
245
40.4
37.8
43.0
39.9
VIL-S-RPB (Zhang al., 2021)
45
277
47.1
42.7
Region (Chen al., 2022)
171
42.5
39.5
46.3
42.3
Swin-T (Liu al., 2021)
264
42.2
39.1
46.0
41.6
Swin-T+GrafT (ours)
275
(+1.1)
43.3
(+0.8)
39.9
(+1.0)
47.0
(+0.9)
42.5
VIL-S-RPB (Zhang al., 2021)
352
48.9
Region ViT-B (Chen et al., 2022)
288
43.5
40.1
47.2
43.0
Swin-S (Liu et al., 2021)
69
354
44.8
40.9
48.5
43.3
Swin-S+ GrafT (ours)
84
373
(+0.2) 45.0
(+0.3)
41.2

---

Under review as a conference paper at ICLR 2023
Results Object Detection and Instance Segmentation: Table 3 shows that Swin-T+GrafT outperforms other models in both object detection and instance segmentation while being trained with 1x schedule and 3x schedule. RegionViT-- outperforms Swin-T but Swin-T+Graff fights back and outperform Region ViT-S again due to the support from GrafT. The multi-level semantics from GrafT plays important role to improve both object detection and instance segmentation.
Dataset and Settings for ADE20K: ADE20K (Zhou et al., 2017) annotates 150 categories for semantic segmentation. It contains 20K training, 2K validation and 3K testing images. In our eval- uations, we use multi-scale mloU as the metric (using scales of [0.5, 0.75, 1.0, 1.25, 1.5, 1.75]x the training resolution) and follow training procedure similar to Swin (Liu al., 2021). We also report model complexity metrics such as parameters, FLOPs (for an input size of 512 2008) and rrame-rate. We use GrafT backbones pretrained on ImageNet-1K K (Deng et al., 2009) for 300 epochs at x24x224 resolution, and finetune it with the decoder at a 512x512 resolution. We choose UperNet (Xiao et al., 2018) as our decoder, and i implement within the mmsegmentation (Con- We use the original hyperparameter settings as Swin.
tributors, 2020) framework.
Results on Semantic Segmentation: Table 4 shows that Swin+GrafT out- perform Swin in both single scale (+1.0) and multi-scale (+1.3) training settings. The multi-level semantics from GrafT is beneficial to segment various sizes of objects.
Table 4: Performance of GrafT semantic segmentation on ADE20K.
Models
Param (M)
FLOPs
mIOU
mIOU MS
(G)
SS
Swin- T (Liu al., 2021)
945
44.5
45.8
Swin-T + Graft
66
955
(+1.0)
45.5
(+1.3) 47.1
3.3 ABLATIONS ON IMAGENET
We conducted the following experiments to better understand GrafT. Swin-T and Deit-T are used in the experiments.
Table 5: Different downsampling and upsampling approaches in GrafT. When ablating downsampling meth- ods, we always consider Learnable W- W-Bilincar interpolation upsampling. and when ablating upsampling methods, use linear projection downsampling.
Params (M)
FLOPs (G)
Through. (im/s)
Acc. (%)
Params (M)
FLOPs
Through. (im/s)
Acc. (%)
Downsampling
Upsampling
(G)
Linear projection
1.3
2260
75.2
Nearest neighbor
1.3
2890
74.8
Cross attention
7.9
1.2
2834
75.4
Cross attention
8.7
1.3
2392
74.4
Average pooling
7.9
1.2
3143
76.1
Learnable bilincar
8.5
1.3
2260
75.2
Downsampling & Upsampling in GrafT: Table 5 shows the performance of DeiT-T+GrraTT on dif- ferent horizontal downsampling and horizontal upsampling approaches. GrafT keeps Iearnable W- Bilinear interpolation an upsampling method while applying various downsampling approaches. The linear projection approach concatenates spatially neighboring tokens into channels and applies linear projection them to reduce the channels. The cross attention approach creates spatially coraser feature by average pooling the input feature. It then uses a coarser feature as query and the input feature in the backbone transformer key and value to perform cross attention. The av- erage pooling approach creates a spatially coarser features by average pooling spatially neighboring tokens in the input feature. The result shows that the average pooling approach is the most efficient downsampling method in terms of memory and computation complexities and speed. GrafT keeps linear projection as a downsampling method while applying various upsampling approaches. Cross attention uses the fine feature in the backbone transformer as query and spatially coarser feature in GrafT as key and value to exchange global and local information. We also tried Nearest neighbor interpolation as the simplest upsampling method. Learnable W-Bilinear interpolation applies anti- aliasing weights to prevent the aliasing effect and applies bilinear interpolation in each local region separately. The result shows that learnable W-Bilinear interpolation achieves the highest accuracy with reasonable complexities and speed. Therefore, we adopt average pooling as a downsampling method and learnalle W-Bilinear interpolation upsample method for GrafT.
7

---

Under review as a conference paper at ICLR 2023
Table 6: Considering different (a) number scales within a GrafT, and (b) number of GrafT's in a model.
(a)
(b)
Model
scales
Parums (M)
FLOPs
(G)
Acc. (%)
(%)
Medel
GrafT:
Params (M)
ELOPs
(G)
Acc. (5)
(9)
Swin-T T (Liu et al., 2021)
0
29
4,5
81.3
DeTF-T (Terroo a al, 201)
0
57
1.3
71.2
65
1.2
(+2.2) 744
Swin-T+ GrafT
33.5
34.0
5.0
(+1.0) 82.3
(+1.4) 82.7
DeTFT GrafT -
11
8
79
73
1.2
1.2
(+3.9) 76.1
(+1.4 75.6
The number of multi-scale features in GrafT: GrafT exploits a horizontal pyramid structure where multi-scale low-resolution features are created. It is important understand whether creating fea- tures having multiple high-level semantics is more beneficial than creating a single-scale feature. Table 6-(a) shows that exploiting features with three different scales achieves +1.4% better accuracy with a small computation increase than exploiting a single-scale feature. For example, at the first stage in Swin where the resolution of the input feature is 56 56, GrafT should create features with the size of 28 x 28, 14 x 14, 7 x 7 to deliver multi-scale global information.
the number of GrafT' in DeiT. The accuracy consistently increases as the number of GraiT increases. Therefore, we attach GrafT all the vertical layers above the first layer in DeiT, Swin, and CSWin achieve the best performance. GrafT is not attached to the first layer in the backbone Transformer because the feature needs to be encoded enough before being used to create coarse features in GrafT.
Performance over the number of GrafT: Table 6-(b) shows the performance
Sharing FFN: Table 7 compares the design of shared FFN vs. separate FFN between GrafT and the backbone transformer. DeiT is used as the backbone transformer this experiment. In the first row, DeiT and GrafT have separate FFN, so the low-level feature in the DeiT and the high-level feature in GrafT are fused after FFN blocks. In the second row, DeiT and GrafT have shared FFN, so the low-level feature in the DeiT and the high-level feature in GrafT are fused before the shared FFN block. The design of shared FFN achieves +1 +1.3% higher accuracy with fewer parameters than having separate FFN. Therefore, we adopt the shared FFN in the GrafT architecture.
Table 7: Sharing the parameters of backbone-FFi
Shared FFN
Params (M)
FLOPs (G)
Acc. (%)
8.2
1.2
74,8
7.9
1.2
(+1.3) 76.1
RELATED WORK
Convolution neural networks (CNNs) have been widely adopted as it have shown promising performance (Krizhevsky et al., 2012: He et al., 2016; Chen et al., 2017; Howard et al., 2017; Sandler et al., 2018; Hu et al., 2018; Huang et al., 2017; Simonyan & Zisserman, 2014; Szegedy et al., 2015b; Tan & Le, 2019) on small-scale dataset such as ImageNet- (Deng et al., 2009). Inductive biases such as translation invariance and locality from CNNs are the key reasons to be trained well from scratch in small-scale dataset. Recently, Transformers (e.g., ViT (Dosovitskiy et al., 2020) or DeiT (Touvron et al., 2021)) achieved comparable results CNNs. First type is the transformer with homogeneous structure like ViT where the number of tokens and channels do not change over the vertical layers. T2T (Yuan al., 2021) proposes progressive tokenization method where spatial structures are preserved. Cross ViT (Chen et al., 2021) creates two branches formulate both local and global information and exchange information. PiT (Heo al.. 2021) and PVT (Wang et al., 2021) successfully applied pyramid structure into transformer by dividing the vertical layers into multiple stages and progressively decrease the number of tokens and increase the channels over stages. Swin (Liu et al., 2021) introduces window self-attention where self-attention is performed in each window and shifting window mechanism exchanges information among windows with pyramid structure. Rig (Chen et al., 2022) creates two branches to formulate local tokens and global tokens like Cross and assign each global token to local tokens in the same region to exchange information. CSWin (Dong et al., 2022) introduces cross-shaped window self- attention where half of the channels is used to create vertical stripes as local regions and the other half is used create horizontal stripes as local regions. In this paper, we propose GrafT, simple
Vision transformers:
8

---

Under review as a conference paper at ICLR 2023
and cost-efficient add-on that provides rich global information backbone transformers where there a lack of communication between local regions because of local attentttention.
Table 8: Approaches to deliver high-lever semantics in terms of their structure or multiple scales.
Model
Vertical structure
MS per layer
Fusion method per layer
ViT-T (DeiT-T) (Touvron et al., 2021) CrossViTA (Chen et al., 2021)
Homogencous Homogeneous
None
None
Cross attention
Learnable W-Bilincar
DeiT-T GrafT
Homogeneous
+ E- Wisc add.
PVT-T (Wang al., 2021)
Pyramid
None
None
TYua et al., 2021)
Homogencous
None
None
PoolFormer- S12 (Yu et al., 2022)
Pyramid
None
None
TNT-S (Han al., 2021)
Homogeneous
LL + E-Wise add.
Swin-T (Liu al., 2021)
Pyramid
None
None
RegionViT-S (Chen et al., 2022)
Pyramid
Cross attention
Learnable W-Bilinear
Swin-T GrafT
Pyramid
E-Wise add.
While pyramid structure transformers (e.g., Swin) learns multi-scale features in hierachical way, high-level semantics is introduced the last layers and transformer cannot receive early guidance on how to efficiently encode low-level semantics by un- derstanding the high-level semantics. Some transformers such as Cross ViT (Chen al., 2021), TNT (Han et al., 2021), Region ViT (Chen et al., 2022) keep two branches to encode low-level se- mantics and high-level semantics from the early stage. However, having two separate branches are detrimental to throughput and requires careful design of choosing which layers to exchange lo- cal and global information and choosing the right size ratio of low-resolution and high-resolution features as mentioned in CrossViT (Chen et al., 2021) the divisibility between local and global tokens. ViL (Zhang et al., 2021) creates global tokens through random initialization in each layer and use them to exchange information between local regions. However, this global token does not contain good inductive bias of local tokens and does not incorporate multiple high-level semantics.
Exploiting multi-scale global tokens:
Table 8 summarizes the difference between GrafT and previous works how deliver high-level semantics. GrafT is unique in the sense that delivers multiple high-level semantics by exploit- ing horizontal pyramid structure and uses simple element-wise addition fuse global information to the backbone transformer. It is applicable to transformers with both homogeneous structure and pyramid structure and improves the performance of transformers without increasing the computation complexity due to the light-weight components as described in 2.2.
5 CONCLUSION
In this paper, we introduced GrafT: an add-on component which can easily be adopted in both ho- mogeneous and pyramid vision transformers, enabling multi-scale feature fusion in arbitrary depths of model. The proposed GrafT branches are designed to be efficient, relying on the backbone to perform heavy computations. fact, gives consistent gains at minimal computation burden. We also validated its effectiveness across multiple backbones and benchmarks including classification, detection and segmentation. In the current work, GrafT is applied to three well-known Transformers: DeiT, Swin, and CSWin. Going forward, we hope that GrafT becomes generally used component for introducing multi-scale features, which particularly benefits tasks with high-resolution inputs.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286-3295, 2019.
Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. High- High-Performance Large-Scale Image Recognition Without Normalization. arXiv, 2021.
Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In International Conference on Learning Representations, 2022. URL https:
/ //openreview. htt///oormmmiddde -
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 357-366, October 2021.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. arXiv preprint arXiv:1707.01622) 2017.
MMSegmentation Contributors. MMSegmentation:: Openmmlab semantic segmentation toolbox
and benchmark. /ttps://githubp com/ cacoo//ppemmmmim ab ab/mnssmmentatiineo. 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 218-24822555 255. leee, 2009.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion (CVPR), pp. 12124-12134, June 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. CoRR, 0bs/20110.1122) 2020. URL https: /tttps::/arxvvv org/ abs 220002202222 11110.10 11929.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing XU, and Yunhe Wang. Transformer in transformer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15908-15919. Cur- ran Associates, Inc., 2021. URL https: /ttp///eoooeeeeeeeeeeeeeeeecccce neurips wttsscccpaaaee papprr/2212 www.r.pppeeeeeeeeee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11936-11944 1945, October 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: M0MMv:/004.000 1704.04611, 2017.
10

---

Under review as a conference paper at ICLR 2023
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), July 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022, October 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 45200 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015a.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015b.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6114. PMLR, 2019.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Intermational Conference on Machine Learning, volume 139 of Proceedingy of Machine Learning Research, pp. 10347- 10357. PMLR, 18-24 Jul 2021. URL https:/ httpss///pcceeedne... mww.iings.m.....m.mmmmmmmmmm.m.m wtt.press//119////// touvron21a. html.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 568-578, October 2021.
Ross Wightman. Pytorch image models. 2019.
http:s////gtthub.comm//
saarrhh-image-modee
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for the European conference on computer vision (ECCV), pp.
scene understanding. In Proceedings the 418-434, 2018.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings the the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 108 10829, June 2022.

---

Under review as a conference paper at ICLR 2023
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp- 558-567, October 2021.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2998-
3008, October 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633-641,220 2017.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Table A: Grafting towards left (upscale from hackbone) or towards right (downscale from backbobe).
Model
left levels
right levels
Params (M)
FL.PPs (G)
Top-1 (%)
DeiT-T (Touvron et al., 2021)
0
5.7
1.3
72.2
1
8.0
1.6
75.1
DeiT-T + GrafT
2
10.0
1.3
75.6
1.2
75.6
GrafT should be used to generate higher-level semanties rather than lower-level semanties: Table A shows the performance over graphting directions. In the experiment, downsampling uses the average pooling and upsampling uses learnable W-Bilinear interpolation for both left and right grapftings. In DeiT-T + GrafT, level of right grafting per layer provides the best accuracy with the least memory and computation complexity. Left grafting is not helpful because the hallucinated higher-resolution feature map in the GrafT may not contain additional local information. It is not helpful to perform 2 levels of right grafting in DeiT because 2nd level of Graft has feature map with the size of 2 x 2 and it is too now-resolution feature map that it does not provide any additional global information.
13


Reviews:
[{"review_id": "lM-IJA93Fr", "reviewer": "Reviewer_EcB8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an add-on component (termed GrafT) that considers global dependencies and multi-scale information throughout the transformer network. The proposed method is evaluated on several fundamental visual recognition problems.", "summary_of_the_review": "The insufficient experiments and unfair comparisons are my main concerns. Please see the above comments.", "strength_and_weaknesses": "Strength:\n\nThe proposed method is evaluated on several fundamental vision tasks, although the evaluation on each task is limited.\n\n\nWeaknesses:\n\nThe performance of ViL (Zhang et al., 2021) is much lower than that in the original paper. Please check the experiments and use ViL as the baseline to evaluate the improvement because ViL is the state-of-the-art method.\n\nThis paper should add the proposed GrafT to models with different model sizes. In the current version, the proposed method is only evaluated for Swin-T for object detection, instance segmentation and semantic segmentation. It is only evaluated for Swin-T, Swin-S and DeiT-T for image classification, which is insufficient to demonstrate the effectiveness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Y3OlIOUodn", "reviewer": "Reviewer_wB1L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\tThe paper proposes to add a FPN-like parallel branch to existing vision transformers that fuses multi-scale tokens with consecutive downsampling, upsampling and window attention. Design choices of each part are studies empirically and improvement over Swin-T on ImageNet, COCO, ADE20k are reported. ", "summary_of_the_review": "Overall, the paper proposes a novel way to fuse multi-scale information in vision transformers, which is to add a FPN-like parallel branch to each of the attention block. It would make the paper better if some concerns regarding computation, throughputs, memory and applying to large models are resolved. \n\n", "strength_and_weaknesses": "Strength: \n\t\tThe idea of adding such multi-scale attention branch in parallel is interesting and design choices of downsampling and upsampling are carefully validated empirically. \n\n\tWeaknesses:\n1.\tThe paper says the method adds minimal complexity. The proposed module actually introduce quite the parameters and flops, 17% #param increase and 14% FLOPs increase when applied to Swin-T, 29% #param increase and 11% FLOPs increase for Swin-S.  And the model performance, compared with RegionViT-S, has only marginal 0.1 improvement, but 10% more #params.\n2.\tCan authors show the performance when applied to large models, e.g. Swin-B?\n3.\tThroughputs and memory? As the method adds parallel branch to existing architectures with consecutive downsamling and upsampling, which is hard for parallelization,  can authors show some throughputs comparison w v.s w/o GrafT. Also, each layer will have much more token maps, can authors compare the memory consumption as well? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Cx0uVyHggt6", "reviewer": "Reviewer_br4Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient and general component GrafT for Transformer-based vision architectures. The GrafT can introduce multi-scale features for early-stage layers, thus achieving better results. The authors demonstrate the effectiveness of the proposed GrafT on several tasks and datasets.\n", "summary_of_the_review": "I have several concerns claimed in the weakness part, thus I choose marginally below the acceptance. \nI may modify the score according to the author's response. ", "strength_and_weaknesses": "Strengths\n* The description of GrafT is clear and the figures are nice, making the paper easy to understand.\n* The authors empirically demonstrate the effectiveness of GrafT on multiple tasks.\n\nWeakness\n* The improvement on ImageNet-1K is marginal. In Table 1, Swin-T+ GrafT outperforms RegionViT-S by only 0.1 but introduces 3.4M more parameters. \n* GrafT should generally introduce more FLOPs and parameters for the backbone model.  However, in Table 2, the FLOPs of DeiT-T+GrafT are even smaller than the original DeiT-T.  Moreover, are the training settings for DeiT+ GrafT and DeiT-T fair?\n* The ablation in Table 6-(a) is not sufficient. What are the results with more scales? Is scale = 3 the best?\n* Generalization performance. The authors only validate the effectiveness of GrafT on Transformer-based architectures. If the multi-scale features are beneficial, I wonder about the performance of GrafT combined with classical CNN models such as VGG and ResNet. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject