Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed to represent the image as a graph structure and introduce a graph neural network (ViG) architecture to extract graph level feature for visual tasks. The graph neural network can be aligned with standard vision transformers with many shared micro designs. Images are spilt into patches as nodes in graphs. Each node is connected with its neighborhoods. The ViG network is in a hierarchical feature extraction style like that of Swin Transformer. The authors conducted extensive experiments on image recognition and object detection and comaprable performance with other state-of-the-art methods.
strengths_and_weaknesses: ***Strength***
1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.
2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.

***Weakness***
1. The idea of graph neural network for visual recognition is appealing, but it seems to be great to expoloits image structures to adjust deep network structures. However, it seems that ViG just takes the simplest setting of graph neural network. It looks like a complex verision of vision transformer. The hierarchical structures involved is in a Swin Transformer style. Since there are a lot of techniques in ViT used, such as multihead attention, feed forward network and etc. I am a little confused about whether ViG is just a simplfied version of Swin Transformer. 
2. The feature dimensions. resolutions and other settings listed in Table 2 are very similar to that of Swin Transformer.  So when there are $224 \times 224$ inputs, ViG outputs $7 times 7$ feature maps. But in Table 5, I don't find obvious advantage of ViG when compared with Swin Transformer.
3. Another big problem is that Table 3 lists a lot of tricks used in ViG. According to the MAE paper (Masked Autoencoders Are Scalable Vision Learners, CVPR 2022), a vanilla ViT-B model (86M parameters) can get 82.3% top 1 ImageNet accuracy with the Exponential Moving Average trick (EMA). So the proposed VIG doesn't show any advantages compared with the ViT-B model (in Table 1).

questions: It seems that when compared with their ViT-B and Swin-B competitor, either ViG-B or Pyramid ViG-B show obvious advantages on ImageNet classification results. I am curious about whether it's the training tricks rather than the algorithm itself that brings the performance gains.  
limitations: I suggest the authors check whether it's possible to use image intrinsic structures proposed in GraphFPN  (GraphFPN: Graph Feature Pyramid Network for Object Detection)  to guide the feature learning of ViG. It will be very interesting then.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript proposes a new kind of backbone named (ViG), which represents the image as a graph and extracts graph-level features for vision tasks. Specifically, the input image is separated into patches as nodes in a graph. Grapher module and FFN module are used to aggregate the information among nodes and transfer feature space. Isotropic and pyramid architectures are proposed to build models of different sizes. The ViG are compared with other SOTA backbones on both image classification task and object detection task.
strengths_and_weaknesses: Pros:

- Representing an image as graph is novel and interesting.

- The proposed grapher module and FFN are soundness. Also, the visualization of the graph structure indeed shows that the proposed model has learned meaningful relationships among image patches.

- The ablation studies and experiments on several vision tasks are good. Both isotropic and pyramid architectures show the effectiveness of the proposed ViG model.

Cons:

- Some parts of the manuscript are unclear. For example, how to initialize a graph of an image is unclear. It is said the graph is built based on K nearest neighbors, but how to compute the KNN is unclear. Is the KNN constructed based on the position or the similarity of the feature?

- Does the method to compute the KNN influence the performance?

- It would be great if the authors can show more visualization cases of the constructed graph under a more complicated scenario that several objects are involved.

questions: - For Tab.8, what’s the meaning of "9 to 18" in the last column?

- The authors compare the ViG with several Transformer-based methods in Tab. 4 in terms of both parameters and FLOPs. What about the real inference time of the proposed method in a standard GPU platform?

limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Recently, ConvNets and transformers have achieved state-of-the-art results on various visual recognition tasks. This paper explores graph neural networks (GNN) for visual tasks. By constructing a graph structure from the input image, the paper discusses the differences and advantages of graph structure over grid and sequence structure. GNN is applied on the graph data and FFN is introduced for improving the feature diversity. The obtained ViG backbone can achieve comparable and even better performance than the SOTA ConvNets and transformers.
strengths_and_weaknesses: Strengths

The paper is easy to follow and well-written.
The pioneering exploration of GNN as vision backbone is inspiring to more works. This work will much appeal to the community.
The experiments on image classification and object detection show the effectiveness of vision GNN.

Weaknesses

- In different layers, will the constructed graph structure be updated?
- How ViG is used in object detection is not described in detail. Please include the implementation details.
- Using graph neural networks as a component (although not as the backbone) for image recognition has been investigated in some previous works [1,2]. Please discuss with these works.

[1] Chen, Zhao-Min, et al. "Multi-label image recognition with graph convolutional networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[2] Shen, Yantao, et al. "Person re-identification with deep similarity-guided graph neural network." Proceedings of the European conference on computer vision (ECCV). 2018.
questions: See questions in the weaknesses part.
limitations: The limitations and potential negative societal impact were addressed in the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a GNN-based backbone model for image tasks that is mainly built on the GNN layers and FC layers. Basically, it splits an image into patches and takes each patch as a node to construct the graph structure. A GNN-based architecture is utilized to the graph for visual representation learning. To address the degradation of feature diversity, more transformation of feature is added in the network. Extensive experiments of image classification and object detection show that the proposed Vision GNN can outperform representative convolutional networks and transformers with similar number of parameters.
strengths_and_weaknesses: Strengths:
+ This work is clearly motivated and well written
The background of the research, the motivation of graph representation for images and the related work are all clearly stated and summarized.
+ The first GNN-based backbone for visual tasks
The simple yet effective Vision GNN is introduced by adapting the GNN with reasonable modification by patch-based graph construction and adding more node feature transformations.
+ Extensive evaluation and impressive results
This paper demonstrates the power of the Vision GNN model on ImageNet and COCO datasets, outperforming representative CNN and transformer models. The results are impressive and interesting.

Weaknesses:
- Based on my experience, the inference speed of GNN is not as fast as CNN. It’s a general issue for the researchers to explore for mobile applications of Vision GNN.
- Typos: In line41: GCN->Grapher. In supplemental material, the output channel of self.fc1 in FFN should be `hidden_channels` rather than `in_channels`.

questions: 1. In the default setting for ImageNet, there are 196 nodes for an image. How does the number of nodes affects the performance of ViG?
2. In the Mask-RCNN experiment, the overall AP of Pyramid ViG is better than Swin but the AP_75. Could you explain this phenomenon?


limitations: The limitations are addressed.
review_title: null

Meta-review: metareview: This paper proposes to explore the graph structure of images by considering patches as nodes, where the graph is constructed by connecting nearest neighbors. Extensive experiments on various visual tasks, i.e., image recognition and object detection have demonstrated the effectiveness of the proposed ViG. All the reviewers agree on the inspiring and promising exploration. The paper is also well-written and the experimental results are impressive. 
confidence: Certain
Decision: Accept