Vision GNN: An Image is Worth Graph of Nodes
Authors: Kai Han
Yunhe Wang
Jianyuan Guo
Yehui Tang
Enhua Wu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new \emph{Vision GNN} (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes. ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research.
  
The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-AI-Backbones} and the MindSpore code is available at \url{https://gitee.com/mindspore/models}.

Vision GNN: An Image is Worth Graph of Nodes
Kai Han 1.2+
Yunhe Wang Jianyuan Guo?
Yehui Tang Enhua Wu 1,4
State Key Lab of Computer Science, ISCAS & UCAS
Huawei Noah's Ark Lab Speking University 'University of Macau (kai han, yunhe, wang)Chaawei. com, weh@ios. cn
Abstract
Network architecture plays key role in the deep learning- based computer vision system. The widely-used convolutional neural network and transformer treat the image grid sequence structure, which not flexible capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce new Vision GNN (ViG) architecture level feature for visual tasks. We first split the image to a number of patches which are viewed nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model transform and exchange information among all the nodes. ViG consists two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioncering study of GNN on general visual tasks will provide useful inspiration and experience for future research.
extract graph-
The PyTorch code is available at https://github. com/huawei-noaa SrriicienttAA-BBackboe and the MindSpore code is available at https: //gitee. grgitee.comm/minnapprreeeeeeeeee
Introduction
In the modern computer vision system, convolutional neural networks (CNNs) used to be the de-facto standard network architecture [29] [7.17] Recently, transformer with attention mechanism was introduced for visual tasks 19 and attained competitive performance. MLP-based (multi-layer
perceptron) vision models 49 50 can also work well without using convolutions or self-attention. These progresses are pushing the vision models towards an unprecedented height.
Different networks treat the input image in different ways. As shown in Figure T. the image data
is usually represented as regular grid of pixels in the Euclidean space. CNNs (29 apply sliding window on the image and introduce the shift-invariance and locality. The recent vision transformer or MLP 49 treats the image as sequence of patches. For example, VIT [ divides 224 x 224 image into a number of 16 16 patches and forms sequence with length of 196 as input.
Instead of the regular grid or sequence representation, we process the image in more flexible way. One basic task of computer vision is recognize the objects in an image. Since the objects are usually not quadrate whose shape is irregular, the commonly grid or sequence structures in previous networks like ResNet and ViT are redundant and inflexible to process them. object can be viewed as a composition of parts, e.g., human can be roughly divided into head, upper body,
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Grid structure.
(b) Sequence structure.
(b) Graph structure.
Figure 1: Illustration of the grid, sequence and graph representation of the image. the grid structure, the pixels or patches are ordered only by the spatial position. In the sequence structure, the 2D image transformed to a sequence of patches. In the graph structure, the nodes are linked by its content and are not constrained by the local position.
arms and legs. These parts linked by joints naturally form graph structure. By analyzing the graph. we are able to recognize the human. Moreover, graph is generalized data structure that grid and sequence can be viewed as special case of graph. Viewing image graph is more flexible and effective for visual perception.
Based on the graph representation of images, we build the vision graph neural network (ViG for short) for visual tasks. Instead of treating each pixel as node which will result too many nodes (>10K), we divide the input image number of patches and view each patch as node. After constructing the graph of image patches, we use our VIG model to transform and exchange information among all the nodes. The basic cells of ViG include two parts: Grapher and FFN (feed-forward network) modules. Grapher module is constructed based on graph convolution for graph information processing. To alleviate over-smoothing phenomenon of conventional GNN, a FFN module is utilized for node feature transformation and encouraging node diversity. With Grapher and FFN modules, we build our ViG models in both isotropic and pyramid manners. In the experiments, we demonstrate the effectiveness of ViG model on visual tasks like image classification and object detection. For instance, Pyramid ViG-S achieves 82. % top-1 accuracy ImageNet classification task, which outperforms the representative CNN (ResNet [7]). MLP (CycleMLP (5)) and transformer' (Swin-T (35] with similar FLOPS (about 4.5G). To the best of knowledge, work is the first to successfully apply graph neural network on large-scale visual tasks. We hope our work will inspire the community further explore more powerful network architectures.
2 Related Work
In this section, we first revisit the backbone networks in computer vision. Then we review the development of graph neural network, especially GCN and its applications on visual tasks.
2.1 CNN, Transformer and MLP for Vision
The mainstream network architecture in computer vision used be convolutional network 29
27
Starting from LeNet (29), CNNs have been successfully used in various visual tasks, e.g. image
classification (27]. object detection [2] and semantic segmentation (36]. The CNN architecture
evolving rapidly in the last ten years. The representative works include ResNet [17] MobileNet [1]
and NAS [0]. Vision transformer was introduced for visual tasks from 2020 [4 9
3 4
From
then on, number of variants of VIT (9] were proposed to improve the performance on visual tasks.
The main improvements include pyramid architecture (57] 35 local attention [5] 35 and position
encoding [61] Inspired by vision transformer, MIP is also explored in computer vision 49 50
MI.P can achieve competitive performance and work on general visual tasks like object detection and segmentation.
With specially designed modules 32

---

2.2 Graph Neural Network
The earliest graph neural network was initially outlined in [1 Micheli 338 proposed the early form of spatial-based graph convolutional network by architecturally composite nonrecursive layers. In recent several years, the variants spatial-based GCNs have been introduced, such as [39] [0] spectral-based GCN was first presented by Bruna al. [2] that introduced graph convolution based on the spectral graph theory. Since this time, a number of works to improve and extend spectral- based The GCNs are usually applied graph data, such social
GCN have been proposed [18] 7 26
networks 13 citation networks
45 and biochemical graphs [55].
The applications of GCN in the field of computer vision 63 28
56 25 mainly include point clouds classification, scene graph generation, and action recognition. A point cloud is set of 3D points in space which is usually collected by LiDAR scans. GCN has been explored for classifying and segmenting points clouds 28
58 69 Scene graph generation aims to parse the input image intro a graph with the objects and their relationship. which is usually solved by combining object detector and GCN [63 68 By processing the naturally formed graph of linked human joints, GCN was utilized human action recognition task [24 677. GCN only tackle specific visual tasks with naturally constructed graph. For general applications in computer vision, we need a GCN-based backbone network that directly processes the image data.
Approach
In this section, we describe how transform an image graph and introduce vision GNN architectures to learn visual representation.
LX
Head for recognition
Head
Graph processing
Feature transform
Image
Graph
Network
ViG framework
Figure 2: The framework of the proposed ViG model.
3.1
ViG Block
Graph Structure of Image. For an image with size of X W x 3, we divided it into N patches. By transforming each patch into feature vector x; R., have X = X2, .xy] where D is the feature dimension and i 1,2, N. These features can be viewed as a set of unordered nodes which are denoted V = (U1, 12, vN For each node vs, find its K nearest neighbors ((ss)) and add an edge eji directed from iy to i; for all Uj E (N;)). Then we obtain a graph (V,E) where & denote all the edges. We denote the graph construction process G G(X) in the following. By viewing the image as graph data, we explore how to utilize GNN to extract its representation.
The advantages of graph representation the image include: 1) graph is a generalized data structure that grid and sequence can be viewed as special case of graph; 2) graph is more flexible than grid or sequence to model the complex object as an object in the image is usually not quadrate whose shape irregular; 3) object can be viewed divided into head, upper body, arms and legs), and graph structure can construct the connections among those parts; 4) the advanced research GNN can be transferred to address visual tasks.
composition of parts (e.g., human can be roughly
Graph-level processing. To be general, we start from the features X E NDD We first construct a graph based on the features: g = G(X). A graph convolutional layer can exchange information between nodes by aggregating features from its neighbor nodes. Specifically, graph convolution

---

operates as follows:
g' F(G. W)
(I)
Uudute (Aggregate(Gr Wagg), W Wasdate);
where Wagg and Wapdate are the learnable weights of the aggregation and update operations, re- spectively. More concretely, the aggregation operation computes the representation of node by aggregating features of neighbor nodes, and the update operation further merge the aggregated feature:
X = hx,, ((x,,N(xx,) Wogg), ) W Wapdate);
(2)
where (xx) is the set of neighbor nodes of x/. Here we adopt max-relative graph convolution 30 for its simplicity and efficiency:
g() x" [[x;, (xx x,1i E (xx))).
(3)
h()) x x x W'Waapen update:
(4)
where the bias term is omitted. The above graph-level processing can be denoted as X' GraphConv(X):
=
We further introduce multi-head update operation of graph convolution. The aggregated feature x is first split into h heads, i.e., head', head" .... head" and then these heads are updated with different weights respectively. All the heads can be updated in parallel and concatenated as the final values:
x head W W update: head W update: head" W updatel
(5)
Multi-head update operation allows the model to update information in multiple representation subspaces, which is beneficial to the feature diversity.
ViG block. The previous GCNs usually repeatedly use several graph convolution layers extract aggregated fea- ture of the graph data. The over-smoothing phenomenon
0.5
0.5
in deep GCNs (31 40 will decrease the distinctiveness of node features and lead to performance degradation for visual recognition, as shown in Figure 3 where diversity is measured as X - | with x = arg ming || 1x ||
0.4
Diversity
0.3
[8). To alleviate this issue, we introduce more fea- ture transformations and nonlinear activations in our ViG block.
Vanilia ResOON
- Our VIG
We apply a linear layer before and after the graph convo- lution to project the node features into the same domain and increase the feature diversity. A nonlinear activation function is inserted after graph convolution avoid layer collapse. We call the upgraded module as Grapher module. In practice, given the input feature X NDD, the Grapher module can be expressed
Layer
Figure 3: Feature diversity of nodes as layer changes.
= oGrrphhConv (X W., W) + X,
(6)
where Y RND, Win and W are the weights of mully-connected layers, a is the activation function, e.g., ReLU and GeLU [19], and the bias term is omitted.
To further encourage the feature transformation capacity and relief the over-smoothing phenomenon, utilize feed-forward network (FFN) each node. The FFN module is simple multi-layer perceptron with two fully-connected layers:
Z = o(YW))) +Y,
where Z NND W1 and W2 are the weights of fully-connected layers, and the bias term is omitted. The hidden dimension of FFN is usually larger than D. In both Grapher and FFN modules, batch normalization is applied after every fully-connected layer or graph convolution layer, which is omitted in Eq. 0 and 7 concision. A stack of Grapher module and FFN module constitutes the ViG block which serves as the basic building unit for constructing a network. Based on the graph representation of images and the proposed ViG block, we can build the ViG network for visual tasks as shown in Figure Compared to vanilla ResGCN 3D), our ViG can maintain the feature diversity (Figure the layer goes deeper to learn discriminative representations.

---

3.2 Network Architecture
In the field of computer vision, the commonly-use transformer usually has an isotropic architecture (e.g. ViT 19], while CNNs prefer to use pyramid architecture (i.e., ResNet [17]). To have a extensive comparison with other types of neural networks, we build two kinds of network architectures for ViG, Le., isotropic architecture and pyramid architecture.
Isotropic architecture. Isotropic architecture means the main body has features with equal size and shape throughout the network, such as VIT [9] and ResMLP [50]. We build three versions of isotropic ViG architecture with different models sizes, Le., ViG-Ti, S and B. The number of nodes is set as N = 196. To enlarge the receptive field gradually, the number of neighbor nodes K increases from 18 linearly as the layer goes deep in these three models. The number of heads is set as h 4 by default. The details are listed in Table I
Table 1: Variants of isotropic ViG architecture. The FLOPs are calculated for the image with x24 resolution. 'Ti' denotes tiny, 'S' denotes small, and 'B' denotes base.
Model
Depth
Dimension D
Params (M)
FLOPs (B)
ViG-Ti
12
192
7.1
1.3
ViG-S
320
22.7
4.5
ViG-B
16
640
86.8
17.7
Pyramid architecture. Pyramid architecture considers the multi-scale property of images by
extracting features with gradually smaller spatial size as the layer goes deeper, such as ResNet [17 and PVT [57]. Empirical evidences show that pyramid architecture is effective for visual tasks [57] Thus, we utilize the advanced design and build four versions of pyramid ViG models. The details are shown in Table Note that we utilize the spatial reduction [57] in the first two stages to handle large number nodes.
Table 2: Detailed settings of Pyramid ViG series. D: feature dimension, E: hidden dimension ratio FFN, K: number of neighbors in GCN, H W: input image size. "Ti' denotes tiny, 'S' denotes small, 'M' denotes medium, and 'B' denotes base.
Stage
Output size
PyramidViG-Ti
PyramidViG-S
PyramidViG-M
PyramidViG-B
Stem
X
Convx3
Convx3
Convx3
Convx3
D 48
D 80
D 96
D 128
Stage
H X
E = x2
E 4
x2
E =
x2
E = 4
x2
4
K
K
K 9
K 9
Downsample
H W
8 X
Conv
Conv
Conv
Conv
D = 96
D = 160
192
= 256
Stage
H 8 x
E
x2
E 4
x2
E
x2
E 4
X2
K =
K
K = 9
K 9
Downsample
Conv
Conv
Conv
Conv
D 240
D = 400
D = 384
D = 512
Stage
H x 16
= X6
x6
F. = x16
x18
16
K
K 9
K 9
K 9
Downsample
H 32
x
32
W
Conv
Conv
Conv
Conv
D 384
D
768
= 1024
Stage
H
W
E
x2
E 4
x2
E
x2
E = 4
x2
32
K 9
Head
Pooling & MLP
Pooling & MLP
Pooling & MLP
Pooling & MLP
Parameters (M) FLOPs (B)
10.7
27.3
51.7
92.6
4.6
Positional encoding. In order to represent the position information of the nodes, we add positional encoding vector to each node feature:
X; f- X; + e; where e; E DR The absolute positional encoding as described in Eq. 8 is applied in both iostropic and pyramid architectures. For pyramid ViG, we further include relative positional encoding by
(
5

---

following the advanced designs like Swin Transformer [35] For node i and j. the relative positional distance between them is e/ ej. which will be added into the feature distance for constructing the graph.
Experiments
In this section, we conduct experiments to demonstrate the effectiveness of ViG models on visual tasks including image recognition and object detection.
4.1 Datasets and Experimental Settings
Datasets. image classification task, the widely-used benchmark ImageNet ILSVRC 2012 [43 used in the following experiments. ImageNet has 1.2M training images and 50K validation images, which belong to 1000 categories. For the license of ImageNet dataset, plcase refer to http::/vee //www. image- net. org/ downloadd For object detection, with 80 object categories. COCO 2017 contains 118K training images and 5K validation images. For the licenses of these datasets, please refer to https://cocodataseet org./home
COCO 2017 (34) dataset
Experimental Settings. For all the ViG models, we utilize dilated aggre- gation [30 in Grapher module and set the dilated rate as [7/4] for the 1-th layer. GELU 19 used as the nonlinear activation function in For ImageNet clas- the commonly- used training strategy proposed in DeiT [5]] for fair comparison. The data augmentation includes RandAug- ment (6). Mixup (73), Cutmix (72). random erasing 74] and repeated aug-
Table 3: Training haper-parameters for ImageNet.
(Pyramid)
Epochs Optimizer
300
Adam B7
Batch size
1024
Start learning rate (LR)
2e-3
Eq. 6 and 7 sification,
Learning rate schedule
Cosine
Warmup epochs Weight decay Label smoothing
20
0.05
47
Stochastic path 22
0.1
0.1
0.1
0.3
Repcated augment
20
RandAugment 6
Mixup prob. 773
0.8
ment 20 The details shown in
Cutmix prob. 772
1.0
Table
For COCO detection task, 333 and Mask R-CNN [16] as the detection frame-
Random erasing prob. 74
0.25
take RetinaNet
Exponential moving average
0.99996
works and use our Pyramid ViG as backbone. All the models are trained on COCO 2017 training set
in "Ix" schedule and evaluated on validation set. We implement the networks using PyTroch [4]] and MindSpore [x] and train all our models NVIDIA V100 GPUs.
Table 4: Results of ViG and other isotropic networks on ImageNet.
CNN,
MLP
Transformer,
* GNN.
Model
Resolution
Params (M)
FLOPs (B)
Top-1
Top-5
RcsMI.P-S12 conv3x3
224x224
16.7
77.0
ConNMixer-768832 52
224x224
21.1
20.9
80.2
ConvMixer-1536220 52
224x224
51.6
51.4
81.4
ViT-B/16 9
384x384
86.4
55.5
77.9
DeiT-Ti 51
224x224
5.7
1.3
72.2
91.1
DeiT-S 51
224x224
22.1
4.6
79.8
95.0
DeiT-B
224x224
86.4
17.6
81.8
95.7
ResMLP--224 50
224x224
79.4
94.5
ResMLP-B22 55
224x224
116
23.0
81.0
95.0
Mixer-B/16 49
224x224
59
11.7
76.4
ViG-Ti (ours)
224x224
7.1
1.3
73.9
92.0
ViG-S (ours)
224x2224
22.7
4.5
80.4
95.2
ViG-B (ours)
224x224
86.8
17.7
82.3
95.9

---

4.2 Main Results on ImageNet
Isotropic ViG The neural network with iostropic architecture keeps the feature size unchanged in its main computational body, which is easy to scale and is friendly for hardware acceleration. This scheme widely used in transformer models for natural language processing 53 The recent neural networks in vision also explore it such ConvMixer [49], ViT [9] and ResMLP 50
We
compare our isotropic ViG with the existing iostropic CNNs [50]
transformers 9 [1] and
MLPs [50 49 in Table From the results, ViG performs better than other types of networks. For example, our ViG-Ti achieves 73.9% top-1 accuracy which is 1.7% higher than DeiT-Ti model with similar computational cost.
The pyramid architecture gradually shrinks the spatial size of feature maps as the network deepens, which can leverage the scale-invariate property of images and produce multi-scale features. The advanced networks usually adopt the pyramid architecture, such as ResNet [17), Swin Transformer (35) and CycleMLP [3]. We compare our Pyramid ViG with those representative pyramid networks in T Our Pyramid ViG series can outperform or be comparable to the state-of-the-art pyramid networks including CNN, MLP and transformer. This indicates that graph neural network can work well on visual tasks and has the potential to be basic component in computer vision system.
Pyramid ViG
Table 5: Results of Pyramid ViG and other pyramid networks on ImageNet. Transformer, * GNN.
CNN.
MLP.
Model
Resolution
Params (M)
FLOPs (B)
Top-1
Top-5
ResNet-18 17
59
224
12
1.8
70.6
89.7
ResNet-50
17
59
224x224
25.6
4.1
79.8
95.0
ResNet-152
17
59
224x224
60.2
11.5
81.8
95.9
BoTNet-T3
224x224
33.5
81.7
BoTNet-T3
46
224x224
54.7
10.9
82.8
BoTNet-T3
46
256x256
75.1
19.3
83.5
PVT-Tiny S7
224x224
13.2
1.9
75.1
PVT-Smal
57
224x224
24.5
3.8
79.8
PVT-Medium
224x224
44.2
6.7
81.2
PVT-Large
224x224
61.4
81.7
CvT-13 660
224x224
20
4.5
81.6
CvT-21 660
224x224
32
7.1
82.5
CvT-21 600
384x384
32
24.9
83.3
Swin-T 351
224x2244
29
4.5
81.3
95.5
Swin-S
224x:24
50
83.0
96.2
Swin-B 35
224x224
88
15.4
83.5
96.5
CycleMLP-B2 [
224x224
27
3,9
81.6
CycleMLP-B3 5
224x224
38
6.9
82.4
CycleMLP-BB [5]
224x224
52
10.1
83.0
Poolformer-S12
224x224
2,0
77.2
93.5
Poolformer-S36
224x224
81.4
95.5
Poolformer-M45
224x224
73
11.9
82.5
96.0
Pyramid ViG-Ti (ours)
224x224
10.7
1.7
78.2
94.2
Pyramid ViG-S (ours)
224x224
27.3
4.6
82.1
96.0
Pyramid ViG-M (ours)
224x224
51.7
8.9
83.1
96.4
Pyramid ViG-B (ours)
224x:244
92.6
16.8
83.7
96.5
4.3 Ablation Study
We conduct ablation study of the proposed method on ImageNet classification task and use the isotropic ViG-Ti as the base architecture.
Type of graph convolution. We test the representative variants of graph convolution, including EdgeConv [58], GIN (44, GraphSAGE [13] and Max-Relate GraphConv (30). From table 6 we see that the top-1 accuracies of different graph convolutions better than that of DeiT-Ti.

---

Table 6: ImageNet results of different types of graph convolution. The basic architecture is ViG-Ti.
GraphConv
Params (M)
FLOPs (B)
Top-1
EdgeConv
58
7.2
2,4
74.3
GIN d
7.0
1.3
72.8
GraphSAGE
13
7.3
1.6
74.0
Max- Mlxx-Reltiive GraphConv
30
7.1
1.3
73.9
indicating the flexibility of ViG architecture. Among them, Max-Relative achieves the best trade-off between FLOPs and accuracy. In rest the experiments, we use Max-Relative GraphConv by default unless specially stated.
The effects of modules in ViG. To make graph neural network adaptive to visual task, we introduce FC FC layers Grapher module and utilize FFN block for feature transformation. We evaluate the effects of these modules by ablation study. We change the feature dimension of the compared models to make their FLOPs similar, as have a fair comparison. From Table we can see that directly utilizing graph convolution for image classification performs poorly. Adding more feature transformation by introducing FC and FFN consistently increase the accuracy.
Table 7: The effects of modules in ViG ImageNet.
GraphCony
FC Grapher module
FFN module
Params (M)
FLOPs (B)
Top-1
x
5.8
1.4
67.0
x
4.4
1.4
73.4
X
7.7
1.3
73.6
7.1
1.3
73.9
The number of neighbors. In the process of constructing graph, the number of neighbor nodes K is a hyperparameter controlling the aggregated range. Too few neighbors will degrade information exchange, while too many neighbors will lead to over-smoothing. We the results in Table g We can see that the number of neighbor nodes in the range from 9 to 15 can perform well on ImageNet classification task.
from 20 and show
Table 8: Top- accuracy vs. on ImageNet.
K
3
9
12
15
20
9 18
Top-1
72.2
73.4
73.6
73.6
73.5
73.3
73.9
The number heads. Multi-head update operation allows Grapher module to process node features in different subspaces. The number of heads h in Eq. Eo.Brooss the transformation diversity in subspaces and the FLOPs. We tune from to and show the results in Table 9 The FLOPs and top-1 accuracy ImageNet changes slightly for different h. We select h 4 as default value for the optimal trade-off between FLOOPs and accuracy.
Table 9: Top-1 accuracy vs. ImageNet.
h
2
6
FLOPs / Top-1
1.6B / 74.2
1.4B 74.0
1.3B / 73.9
1.2B 73.7
1.2B / 73.7
4.4
Object Detection
We apply our ViG model on object detection task to evaluate its generalization. To have a fair comparison, we utilize the ImageNet pretrained Pyramid ViG-S as the backbone of RetinaNet [33] and Mask R-CNN [66] detection frameworks, The models are trained in the commonly-used "Ix" schedule and FLOPs calculated with x280x800 input size. From the results in Table 10 we can see that our Pyramid ViG-S performs better than the representative backbones of different types, including ResNet [77, CycleMLP 5 and Swin Transformer (35) on both RetinaNet and Mask
R-CNN, The superior results demonstrate the generalization ability of ViG architecture.

---

Table 10: Object detection and instance segmentation results on COCO val2017. Our Pyramid ViG is compared with other backbones on RetinaNet and Mask R-CNN frameworks.
RetinaNet 1x
Backbone
Param
FLOPs
mAP
APso
AP76
APs
APM
APi.
ResNet50 77
37.7M
239.3B
36.3
55.3
38.6
19.3
40.0
48.8
ResNeX(-101-32x44 [62
56.4M
319B
39.9
59.6
42.7
22.3
44.2
52.5
PVT-Small 57
34.2M
226.5B
40.4
61.3
44.2
25.0
42.9
CycleMLP-B2 5
36.5M
230.9B
40.6
61.4
43.2
22.9
44.4
54.5
Swin-T 35
38.5M
244.8B
41.5
62.1
44.2
25.1
44.9
55,5
Pyramid VIG-S (ours)
36.2M
240.0B
41.8
63.1
44.7
28.5
45.4
53.4
Mask R-CNN 1x
Backbone
Param
FLOPs
AP
AP
AP75
APM
AP
AP75
ResNer50 17
44.2M
260.1B
38.0
58.6
41.4
34.4
PVT-Small S7
44.
245.1B
40.4
62.9
43.8
37.8
60.1
40.3
CycleMLP
46.5M
249.5B
42.1
64.0
45.7
38.9
61.2
41.8
PoolFormer-S22 77
41.0M
40.1
62.2
43.4
37.0
59.1
39.6
Swin-T 35
47.8M
264.0B
42.2
64.6
46.2
39.1
61.6
42.0
Pyramid VIG-S (ours)
45.8M
258.8B
42.6
65.2
46.0
39.4
62.4
41.6
4.5
Visualization
To better understand how ViG model works, visualize the constructed graph structure in ViG-S. In Figure 4 we show the graphs of two samples in different depths (the 1st and the 12th blocks). The pentagram is the center node, and the nodes with the center nodes are visualized as drawing all the edges will be messy. We can observe that our model select the content-related nodes the first order neighbors. In the shallow layer, the neighbor nodes tend to be selected based on low-level and local features, such as color and texture. In the deep layer, the neighbors of the center nodes are more semantic and belong to the same category. Our ViG network can gradually link the nodes by its content and semantic representation and help to better recognize the objects.
color its neighbors. Two
(a) Input image.
(b) Graph connection in the 1st black.
(c) Graph connection in the 12th block.
Figure 4: Visualization of the constructed graph structure. The pentagram is the center node, and the nodes with the same color are its neighbors in the graph.

---

5 Conclusion
In this work, we pioneer to study representing the image as graph data and leverage graph neural network for visual tasks. We divide the image into a number of patches and view them as nodes. Constructing graph based on these nodes can better represent the irregular and complex objects the wild. Directly using graph convolution on the image graph structure has over-smoothing problem and performs poorly. We introduce more feature transformation inside each node to encourage the information diversity. Based on the graph representation of images and improved graph block, we build our vision GNN (ViG) networks with both isotropic and pyramid architectures. Extensive experiments on image recognition and object detection demonstrate the superiority of the proposed ViG architecture. We hope this pioncering work on vision GNN can serve as a basic architecture for general visual tasks.
Acknowledgement
This research supported by NSFC (62072449, 61872345), National Key R&D Program of China and Macau Science & Tech. Fund (0018/219//KK)) We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
[1] James Atwood and Don Towsley. cnffusion-convolutionn neural networks, In NIPS, pages 2016.
[2] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6200 2013.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229 2020.
[4] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
[5] Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mip-like architecture for dense prediction. In ICLR, 2022.
[6] Ekin Cubuk, Barret Zoph. Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtcring. In NIPS, volume 29, 2016
[8] Yihe Dong, Jcan- Bantittt Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML, pages 2793 -2003 PMLR, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[10] Justin Gilmer, Samuel S Schoenholz. Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In JCML, pages 1263-1272. PMLR, 2017.
[11] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A model for learning in graph domains. In IJCNN, volume 2, pages 729 -344 2005.
[12] Jianyuan Guo, Ychui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mip via hierarchical rearrangement. In CVPR, 2022.
[13] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1025-1035, 2017.
[14] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, A survey on vision transformer. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2022.
[15] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In NeurlPS, 2021.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In ICCV, pages 2961-22212 2969. 2017.
10

---

[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pages 770-778, 2016.
[18] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks graph-structured data. arXiv preprint 506.055506.05633.. 2015.
[19] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.0815, 2016.
[20] Elad Hoffer, Tal Ben-Nun. Iay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, 2020.
[21] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704. 04861, 2017.
[22] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, pages 646 6666661.363 Springer, 2016.
[23] Huawei. Mindspore.
http::://www..mmddee
cn/ 2020.
[24] Ashesh Jain, Amir R Zamir. Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In CVPR, pages 5308-5301 5317, 2016.
[25] Yongcheng Jing. Yining Mao. Yiding Yang, Yibing Zhan. Mingli Song, Xinchao Wang, and Dacheng Tao. Learning graph neural networks for image style transfer. In ECCV, 2022.
[26] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages
2012.
[28] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, pages 4558-4567, 2018.
[29] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86( 1):2278-2324, 1998.
[30] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns?
In ICCV, pages 9267-9276, 2019.
[31] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, pages 3538-3545, 2018.
[32] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. In ICLR, 2022.
[33] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[34] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[36] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05100.22 2017.
[38] Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498-511 2009.
[39] Mathias Niepert, Mohamed Ahmed, and Konstantia Kutzkov. Learning convolutional neural networks for graphs. In ICML, pages 2014-2023, PMLR, 2016.
[40] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In /CLR, 2020.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high perforrrro deep learning library. NewrIPS, 2019.
[42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real time object detection with region proposal networks. In NIPS, pages 91-99, 2015.

---

[43] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vixion, 115(3):11-252, 2015.
[44] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner. and Gabriele Montardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61 80, 2008.
[45] Prithviraj Sen. Galileo Namata, Mustafa Bilgic. Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data. Al magazine, 29(3):93-93, 2008.
[46] Aravind Srinivas, Tsung-Y Yi Lin, Niki Parmar, Jonathon Shlens, Picter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, pages 16519-16522 16529, 2021.
[47] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojua. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[48] Ychui Tang, Kai Han, Jiamyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is wave: Phase-aware vision mlp. In CVPR, 2022.
[49] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. In NeurlPS, volume 34, 2021.
[50] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprini urXiv:2105.088
2021.
[SI]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[52] Asher Trockman and Zico Kolter. Patches are all you need? arXiv preprint 0rXiv:2201.0979 2022.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
[54] Aladin Virmaux and Kevin Seaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In NeurlPS, pages
2018.
[55] Nikil Wale, lan A Watson, and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowledge and Information Systems, 14(3):347-375. 2008.
[56] Runzhong Wang, Junchi Yan. and Xiaokang Yang. Learning combinatorial embedding networks for deep graph matching. In ICCV, pages 3056-3065, 2019.
[57] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Deng-Ping Fan, Kaitao Song. Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[58] Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[59] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:21 2021.
[60] Haiping Wa, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. In ICCV, pages 22-31, 2021.
[61] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, pages 10033-10041, 2021.
[62] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In CVPR, pages 1492-1500, 2017.
[63] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. CVPR, pages 5410-5419. 2017.
[64] Keyulu Xu, Weihua Hu, Jure Leskovec. and Stefanie Jegelka. How powerful are graph neural networks?
In ICLR, 2018
[65] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain
approximation for binary neural networks. In NeurlPS, volume 34, pages 25553 2556322555555 2021.
[66] Yixing Xu. Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. In NewrlPS, volume 32, 2019.
[67] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In AAAI, 2018
[68] Jianwei Yang. Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV, pages 7770-685,200 2018
12

---

[69] Yiding Yang, Jiayan Qiu, Mingli Song. Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In CVPR, pages 7074-7083, 2020.
[70] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang
Xu. Cars: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[71] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng. and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[72] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In /CCV, 2019.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, In ICIR, 2018.
[74] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, volume 34, pages 13001-13008, 2020.
[75] Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. ICLR, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] No potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section 4.1 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] The common settings on ImageNet and COCO datasets. (d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See Section 4.1
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "i6uOZ2fKwUH", "reviewer": "Reviewer_aRD2", "summary": "Recently, ConvNets and transformers have achieved state-of-the-art results on various visual recognition tasks. This paper explores graph neural networks (GNN) for visual tasks. By constructing a graph structure from the input image, the paper discusses the differences and advantages of graph structure over grid and sequence structure. GNN is applied on the graph data and FFN is introduced for improving the feature diversity. The obtained ViG backbone can achieve comparable and even better performance than the SOTA ConvNets and transformers.", "questions": "See questions in the weaknesses part.", "limitations": "The limitations and potential negative societal impact were addressed in the paper.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is easy to follow and well-written.\nThe pioneering exploration of GNN as vision backbone is inspiring to more works. This work will much appeal to the community.\nThe experiments on image classification and object detection show the effectiveness of vision GNN.\n\nWeaknesses\n\n- In different layers, will the constructed graph structure be updated?\n- How ViG is used in object detection is not described in detail. Please include the implementation details.\n- Using graph neural networks as a component (although not as the backbone) for image recognition has been investigated in some previous works [1,2]. Please discuss with these works.\n\n[1] Chen, Zhao-Min, et al. \"Multi-label image recognition with graph convolutional networks.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n[2] Shen, Yantao, et al. \"Person re-identification with deep similarity-guided graph neural network.\" Proceedings of the European conference on computer vision (ECCV). 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_tIwhPuP2_o", "reviewer": "Reviewer_CG6g", "summary": "This paper introduces a GNN-based backbone model for image tasks that is mainly built on the GNN layers and FC layers. Basically, it splits an image into patches and takes each patch as a node to construct the graph structure. A GNN-based architecture is utilized to the graph for visual representation learning. To address the degradation of feature diversity, more transformation of feature is added in the network. Extensive experiments of image classification and object detection show that the proposed Vision GNN can outperform representative convolutional networks and transformers with similar number of parameters.", "questions": "1. In the default setting for ImageNet, there are 196 nodes for an image. How does the number of nodes affects the performance of ViG?\n2. In the Mask-RCNN experiment, the overall AP of Pyramid ViG is better than Swin but the AP_75. Could you explain this phenomenon?\n\n", "limitations": "The limitations are addressed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This work is clearly motivated and well written\nThe background of the research, the motivation of graph representation for images and the related work are all clearly stated and summarized.\n+ The first GNN-based backbone for visual tasks\nThe simple yet effective Vision GNN is introduced by adapting the GNN with reasonable modification by patch-based graph construction and adding more node feature transformations.\n+ Extensive evaluation and impressive results\nThis paper demonstrates the power of the Vision GNN model on ImageNet and COCO datasets, outperforming representative CNN and transformer models. The results are impressive and interesting.\n\nWeaknesses:\n- Based on my experience, the inference speed of GNN is not as fast as CNN. It’s a general issue for the researchers to explore for mobile applications of Vision GNN.\n- Typos: In line41: GCN->Grapher. In supplemental material, the output channel of self.fc1 in FFN should be `hidden_channels` rather than `in_channels`.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AVpHDIZwe61", "reviewer": "Reviewer_5KPK", "summary": "In this paper, the authors proposed to represent the image as a graph structure and introduce a graph neural network (ViG) architecture to extract graph level feature for visual tasks. The graph neural network can be aligned with standard vision transformers with many shared micro designs. Images are spilt into patches as nodes in graphs. Each node is connected with its neighborhoods. The ViG network is in a hierarchical feature extraction style like that of Swin Transformer. The authors conducted extensive experiments on image recognition and object detection and comaprable performance with other state-of-the-art methods.", "questions": "It seems that when compared with their ViT-B and Swin-B competitor, either ViG-B or Pyramid ViG-B show obvious advantages on ImageNet classification results. I am curious about whether it's the training tricks rather than the algorithm itself that brings the performance gains.  ", "limitations": "I suggest the authors check whether it's possible to use image intrinsic structures proposed in GraphFPN  (GraphFPN: Graph Feature Pyramid Network for Object Detection)  to guide the feature learning of ViG. It will be very interesting then.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strength***\n1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.\n2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.\n\n***Weakness***\n1. The idea of graph neural network for visual recognition is appealing, but it seems to be great to expoloits image structures to adjust deep network structures. However, it seems that ViG just takes the simplest setting of graph neural network. It looks like a complex verision of vision transformer. The hierarchical structures involved is in a Swin Transformer style. Since there are a lot of techniques in ViT used, such as multihead attention, feed forward network and etc. I am a little confused about whether ViG is just a simplfied version of Swin Transformer. \n2. The feature dimensions. resolutions and other settings listed in Table 2 are very similar to that of Swin Transformer.  So when there are $224 \\times 224$ inputs, ViG outputs $7 times 7$ feature maps. But in Table 5, I don't find obvious advantage of ViG when compared with Swin Transformer.\n3. Another big problem is that Table 3 lists a lot of tricks used in ViG. According to the MAE paper (Masked Autoencoders Are Scalable Vision Learners, CVPR 2022), a vanilla ViT-B model (86M parameters) can get 82.3% top 1 ImageNet accuracy with the Exponential Moving Average trick (EMA). So the proposed VIG doesn't show any advantages compared with the ViT-B model (in Table 1).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2zpOJaJjYLU", "reviewer": "Reviewer_D9bm", "summary": "This manuscript proposes a new kind of backbone named (ViG), which represents the image as a graph and extracts graph-level features for vision tasks. Specifically, the input image is separated into patches as nodes in a graph. Grapher module and FFN module are used to aggregate the information among nodes and transfer feature space. Isotropic and pyramid architectures are proposed to build models of different sizes. The ViG are compared with other SOTA backbones on both image classification task and object detection task.", "questions": "- For Tab.8, what’s the meaning of \"9 to 18\" in the last column?\n\n- The authors compare the ViG with several Transformer-based methods in Tab. 4 in terms of both parameters and FLOPs. What about the real inference time of the proposed method in a standard GPU platform?\n", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- Representing an image as graph is novel and interesting.\n\n- The proposed grapher module and FFN are soundness. Also, the visualization of the graph structure indeed shows that the proposed model has learned meaningful relationships among image patches.\n\n- The ablation studies and experiments on several vision tasks are good. Both isotropic and pyramid architectures show the effectiveness of the proposed ViG model.\n\nCons:\n\n- Some parts of the manuscript are unclear. For example, how to initialize a graph of an image is unclear. It is said the graph is built based on K nearest neighbors, but how to compute the KNN is unclear. Is the KNN constructed based on the position or the similarity of the feature?\n\n- Does the method to compute the KNN influence the performance?\n\n- It would be great if the authors can show more visualization cases of the constructed graph under a more complicated scenario that several objects are involved.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept