Vision Graph Prompting via Semantic Low-Rank Decomposition
Authors: Zixiang Ai
Zichen Liu
Jiahuan Zhou
Venue: ICML 2025 poster (2025)
Keywords: Graph Neural Networks
Visual Prompting
Primary area: deep_learning->graph_neural_networks
TL;DR: In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures.

Abstract: Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditional grid or sequence-based representations. To efficiently adapt ViG to downstream tasks, parameter-efficient fine-tuning techniques like visual prompting become increasingly essential. However, existing prompting methods are primarily designed for Transformer-based models, neglecting the rich topological relationships among nodes and edges in graph-based representations, limiting their capacity to model complex semantics. In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures. Our core insight reveals that semantically connected components in the graph exhibit low-rank properties. 
Building on this observation, we introduce a semantic low-rank prompting method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-grained semantic dependencies. 
Extensive experiments demonstrate our method significantly improves ViG’s transfer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency.

Vision Graph Prompting via Semantic Low-Rank Decomposition
Zixiang Ai Zichen Liu
Jiahuan Zhou
Abstract
Graph Structure
PCA Components
Vision GNN (ViG) demonstrates superior perfor- mance by representing images as graph struc- tures, providing a more natural way capture irregular semantic patterns beyond traditional grid or sequence-based representations. To effi- ciently adapt ViG to downstream tasks, parameter- efficient fine-tuning techniques like visual prompt- ing become increasingly essential. However, ex- isting prompting methods are primarily designed for Transformer- baadd models, neglecting the rich topological relationships among nodes and edges in graph-based baaph- representations, limiting their ca- pacity to model complex semantics, In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph struc- tures. Our core insight reveals that semanti- cally connected components in the graph exhibit low-rank properties. Building on this observa- tion, we introduce semantic low-rank prompt- ing method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-graine -reigga semantic de- pendencies. Extensive experiments demonstrate our method significantly improves ViG's trans- fer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our code is available at https: //gtthub com/ wtw.:/iaahaan1111/0112200000
I-SNE Visualization
-10
-18
Now- Seighhr Paicha
Neightee Patches
Creter Paicho
Figure I. Visualization of ViG graph structures using PCA and t-SNE. This figure illustrates the graph structures in ViG, where semantically related components exhibit shared principal features and form single compact cluster in i-SNE embeddings. The visu- alization underscores the low-rank nature of semantic information.
grid-based representations in CNNs (LeCun et al., 1998; Krizhevsky et al., 2017) or the sequential tokenization in ViTs (Dosovitskiy, 2020; Liu et al., 2021), the graph struc- ture employed by ViG offers a more natural and flexible approach to modeling semantic patterns within images. In real-world scenarios, semantically related object parts are often distributed irregularly rather than following strict grid- like arrangements. By leveraging graph representations, ViG effectively captures these global interactions, preserv- ing rich semantic information that is difficult to encode in traditional architectures.
1. Introduction
Recent advent of Vision GNN (Han et al., 2022; Munir et al., 2023; 2024) has unlocked new potential by representing im- age patches as a graph structure, facilitating the application of GNN to diverse vision tasks. In contrast to the fixed
When transferring pre-trained Vision GNN (ViG) models to downstream tasks, full fine-tuning is a commonly used adaptation strategy. However, as model sizes continue to increase, this method incurs substantial storage and com- putational overhead. To this end, parameter-efficient fine- tuning techniques, such visual prompting (Jia et al., 2022; Bahng et al., 2022: Han et al., 2023a; Yao et al., 2025), have emerged as promising approach, exhibiting competi- tive performance with significantly fewer trainable param- eters. Despite previous success, existing vision prompting
'annxuuan Institute of Computer Technology, Peking Uni- versity, Beijing, China. Correspondence to: Jiahuan Zhou <ji- ahuanzhou @ wwtpooouuuuuuuuuuurr.mmmm pku.edu.cn>>.
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
methods are predominantly designed for Transformer-based architectures, struggling to adapt ViG models. Directly transferring these methods leads to suboptimal results com- pared to full fine-tuning, due to overlooking rich semantic information embedded in the topological structures within vision graph. Furthermore, current graph prompting tech- niques (Liu et al., 2023; Fang et al., 2023) are mainly devel- oped for applications in social networks and chemical data, unable capture the unique semantic features of visual images, thereby limiting their effectiveness in downstream vision tasks. This highlights the need for a visual prompting method that effectively captures visual semantics within vision graphs.
existing visual prompting techniques, offering high pa- rameter efficiency while achieving results comparable to full fine-tuning.
2. Related Work
2.1. Vision Graph Neural Network
Graph Neural Network (GNN), a critical branch of deep learning, is traditionally designed for processing graph- structured data (Kipf & Welling, 2016b; Li et al., 2019), including chemical structures, social networks, and cita- tion networks. In computer vision, GNNs have been em- ployed in point cloud classification and segmentation tasks by DGCNN (Wang et al., 2019) and Point-GNN (Shi & Ra- jkumar, 2020), as well as in human pose and action recog- nition (Yan et al., 2018). However, these applications are limited to tasks that can be explicitly represented as graphs.
To address this challenge, we propose Vision Graph Prompt- ing, a novel approach designed to capture semantic features within vision graph structures. Our method incorporates low-rank decomposition in the prompts, grounded in the insight that the semantic information within vision graphs primarily resides in the low-rank components of the latent feature space, depicted in Figure 1. This low-rank design effectively preserves global semantic information while min- imizing interference from local details. To adapt the topological nature of vision graphs and efficiently capture relevant features, we introduce three prompt components with varying levels of granularity. First, to capture global semantic dependencies, we introduce Semantic Low-Rank Graph Seeoo))aahh) Prompt by appending trainable virtual nodes and dynamically forming edges with existing nodes, interacting with the original graph. Second, to facilitate the propagation of semantic features between connected nodes, we design Semantic Low-Rank Edge (SeLo-Edge) Prompt, incorporating the low-rank decomposition to filter out residual local details. Finally, to enhance the local fine- grained semantic features, we incorporate Semantic Low- Rank Node (SeLo-Node) Prompt that preserves intrinsic de- tails while intensifying the low-rank semantic information. We conduct series of experiments on downstream vision tasks to validate the superior performance of our approach. Our method significantly enhances ViG's transfer capability on downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our contributions can be summarized as follows:
The advent of Vision GNN (ViG) (Han et al., 2022) marked the first instance of GNN serving as general-purpose vi- sion backbone. ViG segments images into patches and dynamically connects them using the K-nearest neigh- bors algorithm. By leveraging the flexibility of graph structures to model irregular shapes, ViG surpasses grid- based (ResNet) and sequence-based (ViT) models on the ImageNet- 1k (Deng al., 2009) benchmark. Subsequent advancements like Vision HyperGraph Neural Networks (ViHGNN) (Han et al., 2023b) further enhanced ViG by employing hypergraphs to eliminate the constraint of pair- wise node connections. Mobile ViG (Munir et al., 2023) and Greedy ViG (Munir et al., 2024) improved computational efficiency, reducing graph construction overhead while pre- serving representational power. Despite these advances, adapting ViG-based models to downstream tasks still pri- marily depends on full fine-tuning, which is inefficient in terms of both parameter usage and storage. This limitation underscores the importance of exploring parameter-efficient fine-tuning (PEFT) strategies, such as visual prompting, to enhance adaptability and efficiency.
2.2. Visual Prompting
Prompting techniques (Brown et al., 2020; Schick & Schütze, 2020) originated in the field of natural language processing (NLP), where prompts align downstream tasks with pre-training objectives by prepending prefix words to input sequences. In computer vision, prompting has been adapted as a parameter-efficient fine-tuning technique, known as visual prompting, to customize pre-trained vision models for new tasks. Existing visual prompting meth- ods (Li & Zhou, 2025; Liu et al., 2025) primarily target Transformer-basee architectures, operating in either the in- put data space latent feature space. VP (Bahng et al., 2022) learns single prompt applied uniformly across sam-
We introduce Vision Graph Prompting, a novel visual prompting method designed to capture semantic fea- tures within vision graph structures, overcoming the limitations of existing approaches.
We provide a critical insight that semantic informa- tion in vision graphs primarily resides in the low-rank component of the latent feature space, leading to the development of semantic low-rank prompt design.
Through extensive experiments on various downstream tasks, we demonstrate that our method outperforms
2

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
(a) From CUB Test Dataset
(b) From Flowers Test Dataset
Figure 2. Observation VIG graph structures and principal components. The center patch (red star) is randomly selected from the primary semantic region, with neighboring patches (blue dot) linked via edges. PCA is applied to patches from the same image group (a. b), mapping the first three components the RGB color channels. Despite variations in shape, texture, and color, the ViG graph effectively connects semantically related object parts. These connected regions share common principal components, demonstrating a low-rank structure. Background regions filtered
thresholding the first principal component.
ples, implemented as padding around the input image before feeding it into the pre-trained model. VPT (Jia et al., 2022) extends this approach by adding learnable prompt tokens shared across all samples, integrated within the multi-heas self-attention layers of Vision Transformers. Efficiency- driven variants, such as E2VPT (Han et al., 2023a), prune prompts at both token and segment levels to mitigate the influence of redundant prompts while minimizing additional parameters. More advanced methods aim for instance-level adaptability. DAM-VP (Huang et al., 2023) addresses class- level variation by clustering samples and learning separate prompts for each cluster. InsVP (Liu et al., 2024) employs lightweight prompters that generate unique prompts per in- stance, enabling fine-gr -ne-ggained adaptation. These methods have achieved performance on par with, or even surpass full fine-tuning while maintaining high parameter efficiency.
performance compared to full fine-tuning while reducing parameter-tuning costs. To investigate the relationship be- tween vision graph structures and semantic information, conduct series of experiments, illustrated in Figure 2. By visualizing the dynamically constructed vision graph structures and analyzing the corresponding latent features using Principal Component Analysis (PCA), we gain deeper insights into the mechanisms ViG models. This leads to our key observation:
Semantically connected vision graph nodes, with diverse local intrinsic properties, share common PCA components, suggesting that the semantic information within vision graphs predominantly resides in the low-rank components of the latent feature space.
Since the dominant PCA components encode the most sig- nificant variance feature distribution, this alignment im- plies that semantic representations are largely concentrated in subspace of reduced rank. Further details and additional analysis are provided in the Appendix.
Despite these advances, applying existing visual prompt- ing techniques to ViG models has yielded trivial results. Transformer-centric methods overlook the rich topologi- cal relationships inherent in graph-based representations, failing to exploit the structured semantic information en- coded in nodes and edges. This gap highlights the need for prompting strategy that effectively harnesses the unique characteristics of vision graphs for downstream tasks.
4. Vision Graph Prompting
In this section, we detail our method for efficiently fine- tuning pre-trained ViG models, ensuring seamless integra- tion with the original ViG workflow. To capture the low-rank semantic information embedded in the topology of vision graphs, we introduce three types of prompts: SeLo-Graph Prompt, Selo-Edge Prompt, and SelooNode Prompt, each incorporating a semantic low-rank decomposition design. Given pre-trained ViG model, we freeze the parameters
3. Observation and Insight
In this work, we propose a visual prompting approach de- signed to capture the semantic information embedded in vision graph structures effectively, ensuring competitive
3

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
Vision Graph Structure
Prompted Graph Features
Viision GNN Block XN
Image Patches
Semantic Decomposition
GOD Furward Layer
Daststroaa Hoad
Encoder
SeLo-Graph Prompt
Graph Layar
SeLo-Edge Prompt
SeLo-Node Prompt
Graph Nodes
Node Festure
SeLo-Graph Prompt
SeLo-Edge Prompt
SeLo-Node Prompt
Virtual Node Feature
Low-runk
Semantic Feature!
Product
Equation
Equatiin
Equation
Sel e-Craph
SeLo-Edge
Promgt
Prezupl
MLP,
Frozen
Node
Compooed
Vistual Node
Traizable
Promee
Prompts
Figure Pipeline of Vision Graph Prompting via semantic low-rank decomposition. Specifically, we design: (i) SeL.o-Grahh Prompt trainable virtual nodes that dynamically form edges with existing nodes to capture global semantic dependencies, (ii) SeLo-Edge Prompt with low-rank decomposition to enhance feature propagation between semantically connected nodes, and (iii) SeLo-Node Prompt refine fine- fine-grained semantic information while preserving local intrinsic details.
of the backbone, and only the prompts along with down- stream head are fine-tuned. This approach minimizes the computational cost and parameter overhead while enabling effective adaptation to downstream tasks.
(xx)= g(x;) W update:
Following that, the feed-forward layer is simple multi- layer perceptron with two fully connected layers:
X' = o Wt) W2 + X,
(4)
4.1. Preliminaries for VIG
Given an image with the size of [xxxxx3 ViG divides it into N patches. By encoding each patch into a feature vector X; E Rdd, we transform the image into a set of features where d is the feature dimension. These features are viewed as graph of unordered nodes which are denoted as V {u1, U2..... ......... J.
where X, X' E Rxd represent the graph node features, W1 and W2 are the weights of fully-connected layers. And a denotes the activation function. Note that the bias term omitted. As graph convolution layers can exchange informa- tion between nodes by aggregating features from neighbor nodes, feed-forward layers further encourage the feature transformation capacity and relieve the over-smoothing phe- nomenon.
X XI,
). For each node We find its K nearest neighbors N(s))) and add edge eji directed from vj to for all Uy E (V;).. Then we obtain a graph g = EV,E) where & denote all the edges.
4.2. SeLo-Graph Prompt
By viewing an image as graph data, ViG brings GNN models into vision tasks for effective feature extraction. The image graph is processed by a stack of blocks constructed by a graph convolutional layer and feed-forward layer. Graph convolution operates follows:
The vision graph consists of feature nodes and topological edges, where nodes correspond to local image patches, and edges capture semantic relationships between local regions. To effectively model global semantic dependencies within vision graph structures, we propose GrLo- Prompt, set of trainable virtual nodes that dynamically establish edges with existing nodes, enhancing long-range semantic interactions and enriching the contextual understanding of the graph.
X = [x g (x,), W Wagg), W Wupdate);
where x denotes updated node feature, Nxx;) is the set of neighbor nodes of x, g(-) and f(-) denote the aggrega- tion and update operations respectively with the learnable weights W agg and W updatee Taking max-relative graph convolution for simplicity and efficiency, the process can be detailed as:
Given a encoded vision graph G = (V,E) with node features X X2,........ xN] E I Nxd introduce set of low- rank prompt nodes [n. n2,.... nyl E R Mx,,,, where M denotes the number of prompts and is the rank dimension,
g(x;) [x,, max((xx X; Xx EX(())))) W agg

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
chosen such that < d. To align with the feature space of the original graph, we compose these prompts with low-rank graph-level prompt P& E IRrxd, described as:
where Sn denotes the semantic feature of neighbor node Xn and is the reduced rank dimension with < d.
To reinforce semantic coherence within the local neighbor- hood, we compose the semantic feature Sn with low-rank edge prompt matrix P. E IR'x,, propagating the result- ing information to the center node X.. The propagation is formalized as:
[Pi, ,., P [mi; 02,..., nar] Px AR (5)
Next, we compute the cosine similarity between each prompt node P. E Ed and the feature nodes xj E R., dynamically constructing neighborhood (p:))) using the K nearest neighbor selection:
c
Σ [xV. (x
Sn (1 -c:
(9)
((p:)) = {x, E X | E topK(p, x5))- (6)
where Nº (x.) denotes the set of semantic features for the neighbors of x₂) and 3 a blending factor. By applying this semantic edge propagation across all edges, the entire graph benefits from enhanced semantic consistency and richer contextual integration.
By treating the prompts as additional virtual nodes V {UN+I,.... ".++:...,,UN+rr/ and connecting them to their neigh- bors, the constructed edges are:
E = {e 2y E N(u)), E V}.
(7)
4.4. SeLo-Node Prompt
The updated graph G = (VUD,EUE) integrates these prompt nodes and edges, enabling richer global semantic in- teractions. By embedding semantic-aw prompts into the vision graph topology, this design strengthens the model's transfer capability, capturing complex relationships while maintaining parameter efficiency.
Each node within the vision graph corresponds specific image patch, encapsulating diverse attributes such texture, color, shape, and semantic relationships to other regions of the image. These node features inherently combine two es- sential aspects: intrinsic components that capture localized details and semantic components that convey the node's contextual relevance within the broader graph structure.
4.3. SeL.oEdge Prompt
To better exploit this inherent duality, we propose a node- level prompt that explicitly decouples the intrinsic and se- mantic components via semantic low-rank decomposition, enabling finer control over the interaction between local and global information. This design ensures that the intrinsic details are preserved while amplifying the semantic consis- tency across the vision graph, thereby improving the overall representational capacity for downstream tasks.
In the vision graph, edges dynamically encode pairwise rela- tionships by leveraging the similarity between node features, thereby aggregating semantically related regions within the latent feature space. As illustrated in Figure 2, these con- nections effectively capture semantic dependencies between object components, even in the presence of irregular shapes and varied poses, highlighting the structural coherence of the graph representation.
Given the feature representation of node x; E Rdd we decompose the low-rank semantic components S, E R" through the semantic extraction module MLP,, previously utilized in the edge-level prompt. To further consolidate semantic information, we introduce a low-rank nove-level prompt P, E Rrrx which is learned across all graph nodes to capture common features shared among different image regions. Here, r and d respectively denote the rank dimen- sion and feature dimension of the ViG model, with r < d.
To leverage the contextual information embedded in these topological connections, we introduce edge-level prompt, incorporating semantic low-rank decomposition strategy. This design strategically filters out high-frequency local details, which often introduce noise, while promoting the propagation of semantic features from neighboring nodes to central nodes. By refining the edge-level interactions, our approach reinforces the global semantic consistency within the graph structure, leading to more robust and expressive representations for downstream tasks.
Then, the node feature is updated by integrating the refined semantic component back into the intrinsic feature space:
Given a center node x, E Rd, we define its neighborhood (xxx) as the set of nodes with directed edges pointing to Xe, where each node corresponds to a feature vector. A compact semantic extraction multi-layer perceptron MLPP, employed to project each neighbor feature x, into low- rank semantic space, producing:
X + a S; P, + (1 -a) Xi,
(10)
where a is blending hyperparameter that balances the contribution of intrinsic and semantic components. By se- lectively enhancing semantic consistency while preserving critical local details, the node-level prompt enables a more expressive and robust representation of image patches, im-
S = (x,,(xx.) E RR," Xn (xxx),,
5

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
Table 1. Head-tuning adaptation performance of various methods across ten datasets using the pyramid ViG model pre-trained on ImageNet-2Ik, with classification accuracy (%) reported. For reference, state-of-the-ar visual prompting methods applied ViT models pre-trained on ImageNet-222 21k are also included.
Methods
Backbone
DTD
CUB
NABirds
Dogs
Flowers
Food
CIFAR
CIFARIO
GTSRB
SVHN
Average
Full Fine-Tune
ViG-M
73.5
89.0
82.8
81.4
98.5
87.4
89.2
98.6
98.0
91.7
89.0
Full Fine-Tune
VIT-B
64.3
87.3
82.7
89.4
98.8
84.9
68.9
97.4
97.1
87.4
85.8
VPT
VIT-B
88.5
84.2
90.2
99.0
83.3
78.8
96.8
90.7
85.5
DAM-VP
ViT-B
73.1
87.5
82.1
92.3
99.2
86.9
88.1
97.3
90.6
87.9
88.5
Pu
ViT-B
62.5
85.4
83.5
90.3
90.4
82.3
77.9
95.2
93.1
92.9
85.4
InsVP
ViT-B
74.5
89.3
84.6
93.6
99.2
89.5
91.3
98.4
96.1
96.1
91.3
Linear
ViG-M
66.7
76.2
71.3
71.3
81.4
79.2
67.2
89.4
77.4
65.5
74.6
Adapter
ViG-M
68.3
74.4
76.0
66.8
83.5
84.7
82.6
95.5
93.5
81.8
VP
ViG-M
67.4
81.1
74.5
72.0
94.3
77.4
78.6
93.1
89.0
83.6
81.1
VPT
ViG-M
71.4
77.3
76.4
73.1
95.3
81.9
76.3
93.2
79.7
82,4
80.7
DAM-VP
ViG-M
71.8
82.6
77.4
74.2
95.9
82.3
81.5
94.9
91.4
85.1
83.7
InsVP
ViG-M
69.8
85.0
78.0
77.0
96.1
84.1
83.3
95.8
87.6
89.4
84.6
VFPT
ViG-M
72.1
82.3
77.2
75.6
95.9
83.2
82.4
95.4
85.4
86.1
83.6
GraphPrompt
ViG-M
72.5
76.9
69.9
83.6
80.2
81.4
94.2
91.2
92.6
80.6
GPF-Plus
ViG-M
71.0
82.0
77.2
78.2
95.7
80.9
94.5
90.5
83.6
VGP(Ours)
VIG-M
74.8
87.4
80.9
81.7
98.2
89.5
89.7
98.3
98.1
96.9
89.6
proving the graph's 's capacity to handle complex downstream tasks.
update mechanism in graph convolution, reformulated as:
f(x,, P.) = (1 x; g(x;, Pn)) W
uudatte
4.5. Analysis and Discussion
+ Σ 3- (LMP(xxx) Pe,
(12)
In this section, we analyze and discuss how our method promotes the feature extraction of Vision GNNs, effectively capturing the critical semantic information.
x)()(x))
where f(-) denotes prompted update operation. From Equa- tion 12, we can see that our semantic low-rank prompts collectively work together, facilitating both structural adap- tation and feature enhancement. The SeLo-Node Prompt operates on the aggregation process while the SeL-o-dge Prompt enhances the update process of graph convolution. Meanwhile, the SeLooGraph Prompt strengthens both pro- cesses by refining the neighborhood topology, represented by (xx)). Together, these components effectively balance structural adaptation and feature enhancement, reinforcing the model's ability to extract robust semantic representa- tions.
As shown in Figure 3, the SeLo-Graph Prompt dynamically refines original graph structures, Given initial node features X [x1,, X2....,XX, E R xd we extend the graph by appending virtual nodes [n1, n......; ny] thus building new edges E within the vision graph, as for- mulated in Equation 7. This structural adaptation leads to an updated neighborhood relationship. denoted as (N). ef- fectively enriching the connectivity patterns in the graph representation.
Pg E RMxx
Beyond structural refinement, SeLo-Node Prompt and SeLo- Edge Prompt operate at the feature level, modulating the convolutional process rather than altering the graph topology explicitly. With SeLo-Node Prompt P, IRrxd involved, the aggregation operation in Equation is reformulated as:
5. Experiments
We conduct extensive experiments to evaluate the efficacy of our proposed method across diverse domains. Specif- ically, we test on ten image classification datasets using pre-trained Vision GNN (ViG) backbone to validate its performance on vision tasks. Additionally, to demonstrate the generalizability of our approach, we extend it to tradi- tional graph tasks, evaluating nine datasets from the fields of chemistry and biology. The results highlight the adaptability and robustness of our method across different scenarios.
((x,, Pn) = (((1 - a) x; + MLP.(x;) P.), (11)
where g()) represents reformulated aggregation process and g(.) is original version Equation 2. Similarly, SeLo-Edge Prompt P. E R influences the
6

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
Table 2. Adaptation performance on eight chemistry and one biology benchmarks, based on GIN models pre-trained with Edge Prediction.
Methods
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
PPI
Average
Full Fine-Tune
66.56
78.67
66.29
64.35
69.07
79.67
77.44
80.90
71.54
72.72
GPPT
64.13
66.41
60.34
54.86
59.81
63.05
60.54
70.85
56.23
61.80
GPPT (wo ol)
69.43
78.91
64.86
60.94
62.15
82.06
73.19
70.31
76.85
70.97
GraphPrompt
69.29
68.09
60.54
58.71
55.37
62.35
59.31
67.70
49.48
61.20
GPF
69.57
79.74
65,65
67.20
69.49
82.86
77.60
81.57
76.98
74.51
GPF-Plus
69.06
80.04
65.94
67.51
68.80
83,13
77.65
81.75
77.00
74.54
VGP(Ours)
71.53
80.11
68.53
65.65
74.21
82.60
80.69
83.59
80.58
76.39
5.1. Datasets
with consistent data augmentation strategies, in line with previous works to ensure fair comparison.
Vision Datasets. For vision tasks, we employ 10 bench- marks listed in Table 1, covering various categories and diverse distributions, following the approaches of DAM- VP (Huang et al., 2023) and Vn (Liu et al., 2024). The selected datasets include CIFARIO (Krizhevsky et al., 2009), CIFAR (Krizhevsky et al., 2009), DTD (Cimpoi et al., 2014), CUB (Wah et al., 2011), NABirds (Van Horn et al., 2015),
For graph tasks, we compare with traditional graph prompt- ing methods, including GPPT (Sun et al., 2022), GPPT without orthogonal prompt constraint loss, denoted as GPPT (w/o ol), GraphPrompt (Liu et al., 2023), GPF (Fang et al., 2022), and GPF-Plus (Fang et al., 2023). Full fine-tuning results are also reported for reference.
Stanford Dogs (Khosla et al., 2011), Oxford Flowers (Nils- back & Zisserman, 2008), Food (Bossard et al., 2014), GT- SRB (Stallkamp et al., 2012), and SVHN (Netzer et al., 2011). The same data augmentation strategy is adopted across all compared methods, involving randomly resizing the input images to 256 256, followed by cropping them 224 224. More details are provided in the Appendix.
5.3. Quantitative Analysis
Analysis Vision Tasks. As shown in Table 1, our method excels all the existing visual prompting techniques across ten datasets with diverse distributions, achieving an aver- age accuracy improvement of 5.0%. This significant gain can be attributed to our prompt designs at the graph, edge, and node levels, which effectively capture the rich semantic information embedded within the topological graph struc- tures. It is worth noting that our approach even outperforms full fine-tuning on smaller datasets like Food and SVHN while requiring far fewer trainable parameters. This advan- tage comes from our lightweight low-rank decomposition, which efficiently preserves fine-grained semantic informa- tion while maintaining parameter efficiency.
Graph Datasets. To further examine the extendability of the model to traditional graph tasks, we select downstream datasets from the chemistry and biology domains, following GPF-Plus (Fang et al., 2023). For the chemistry domain, we use eight graph classification datasets from Molecu- leNet (Wu et al., 2018) as downstream tasks. For the biol- ogy domain, utilize dataset consisting of 88K labeled protein ego networks, designed for predicting 5000 coarse- grained biological functions, referred to as the PPI dataset. We adopt the challenging scaffold split for the chemistry datasets and the species split for the biology dataset, ensur- ing alignment with prior works (Hu et al., 2019).
Interestingly, when adapting visual prompting methods orig- inally designed for Vision Transformers to ViG models, we observe noticeable performance drop due to their re- liance on the attention mechanism. Among these, data-level prompting methods such as Ins mitigate the issue to some extent. Furthermore, our approach also surpasses graph- specific prompting methods adapted from traditional graph domains. This superiority stems from our key insight into the semantic coherence and low-rank properties inherent to graph-based representations, which are effectively leveraged by our semantic low-rank decomposition prompting design.
5.2. Comparing Methods
As for vision tasks, we compare our method with both vi- sual prompting and graph prompting methods. We also report the full fine-tuning results as baseline. For visual prompting methods, we compare with task-level approaches such VP (Bahng et al., 2022), VPT (Jia et al., 2022), and VFPT (Zeng et al., 2024), the cluster-level visual prompting method DAM-VI VP (Huang et al.. 2023), as well as the latest InsVP (Liu et al., 2024). Additionally, we apply graph- specific prompting methods, including GraphPrompt (Liu et al., 2023) and GPF-Plus (Fang et al., 2023) to vision tasks for comparison. All methods were trained for 100 epochs
Analysis on Graph Tasks. To further validate the gener- alizability of our semantic low-rank prompts, we extend them traditional graph tasks. As shown in Table 2, our method achieves strong performance across graph datasets, surpassing prior approaches on seven out of nine datasets
7

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
Table Ablation study on effects of each semantic low-rank prompt. We report the classification accuracy (%) on the CUB and GTSRB datasets evaluation.
90.9
9k1
SVHN
90-
CUB
(%) Sacmecy
97
89
99.6
$9.4
$9.0
20.0
SeLo-Graph
SeLo-Edge
SeLo-Node
CUB
GTSRB
*6
96.1
88
47.2
$7.4
$8.5
Lincar Probing
76.2
77.4
95.8
87
56.5
36.1
36-
$5.0
81.9
86.9
V
85.3
93.0
94
0.81 0.10 6.30 10.0
9.40
0.50
0.75
85
0U0 6.10 11.20
6.50 0.40 0.50
0.75
87.4
98.1
Blending Factor
Blending Factor |
Figure 4. Ablation on blending factors a and B. The red star marks the best result.
Table Ablation study on rank dimension r. Experiments are conducted on four datasets, with trainable parameters (Param.) reported. Since the size of the downstream head varies with specific datasets, we report the Param. of CUB for reference.
Ablation on Rank Dimension r. In our semantic low-rank prompt design, the rank dimension r is an essential hyper- parameter. As shown in Table 4, we conduct experiments on four datasets to probe the optimal r. The searching scope ranges from to 384, where 384 matches the primary feature dimension of our ViG backbone. Results indicate that both overly small and excessively large r values lead to suboptimal performance. Specifically, very small limits the model to represent high-dimensional semantic information, while an excessively large fails to filter high- frequency local details, disrupting the extraction of low- rank semantic features. Furthermore, trainable parameters increase linearly with the r. To balance performance and efficiency, we set 32.
Rank
(M)
CUB
CIFAR
GTSRB
SVHN
4
1.50
85.8
89.1
97.6
96.3
8
1.70
86.2
89.2
97.7
96.5
16
2.10
86.7
89.5
98.0
96.9
32
2.90
87.4
89.7
98.1
96.9
64
4.49
87.2
89.8
97.9
96.7
128
7.69
86.5
89.6
97.8
96.2
384
20.49
86.1
89.3
97.6
95.8
and attaining a notable 3.67 % improvement over full fine- tuning. Although graph data in the chemistry and biology domains lacks explicit visible semantic connections, we hypothesize that low-rank patterns exist within structural relationships, such as chemical bonds protein interactions, analogous to those in visual objects. By leveraging our low- rank decomposition design, our method effectively captures these latent relationships, yielding significant performance gains on graph-base based tasks.
Ablation Blending Factors and B. We evaluate the impact of blending factors or and B, which respectively control the mix-up ratio of the SeLo-Node Prompt and SeLo- Edge Prompt with the original node features. As shown in Figure 4, experiments on four datasets reveal that both a and achieve near-optimal performance within the range of 0.1 to 0.3. Deviation from this range slightly degrades performance due to an imbalance preserving local intrinsic details and amplifying semantic information. Empirically, we set and 33 to 0.2 for all experiments.
5.4. Ablation Study
Effect of Core Components. Our Vision Graph Prompt- ing (VGP) integrates three key components: SeLo-Graph Prompt, SeLo-Edge Prompt, and SeLo-Node Prompt. As parameter-efficient fine-tuning strategy, VGP freezes tire backbone parameters and optimizes only the prompts alongside a shallow downstream head. To verify the indi- vidual effectiveness of each component, we incrementally incorporate them into a linear probing setup on the ViG back- bone, using CUB and GTSRB datasets for evaluation. The introduction of the SeLo-Graph Prompt yields a notable per- formance gain of 5.7% and 9.5% on the respective datasets. Subsequent incorporation of the SeLo-Edge Prompt and Nodde Prompt further improves the classification ac- curacy, achieving the peak performance on both datasets. It verifies the efficacy of our components in capturing essential semantic information embedded in the topological structure of vision graphs, validating the rationality of our designs.
6. Conclusion
In this paper, we propose novel visual prompting method, namely Vision Graph Prompting (VGP), with semantic low-rank decomposition design. To the best of our knowl- edge, we are the first exploration on prompting ViG models. Our approach is built upon the insight that the semantic information within vision graph structures resides in the low-rank components of latent feature space, enabling both improved performance and parameter efficiency. Extensive experiments on diverse downstream tasks demonstrate that our method outperforms state-of- of-thh-art visual and graph prompting methods, achieving results comparable to full fine-tuning while substantially reducing trainable parame- ters. Furthermore, the adaptation our semantic low-rank design to traditional graph tasks highlights its broader po- tential, offering promising direction for future research.
8

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
Acknowledgments
Han, K., Wang, Y., Guo, J., Tang, Y., and Wu, E. Vision gnn: An image is worth graph of nodes. Advances in neural information processing systems, 35:8291
This work was supported by the National Natural Science Foundation of China (62376011) and the National Key R&D Program of China (224YFA11110000,
2022.
Han, Y., Wang, P., Kundu, S., Ding. Y., and Wang, Z. Vi- sion hgnn: An image is more than graph of nodes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19878-19888, 2023b.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Hu, W., Liu, B., Gomes, J., Zitmik, M.. Liang, P., Pande, V., and Leskovee, J. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 00119995.22222 2019.
Huang. Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., and Yu, N. Diversity-aware meta visual prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10878-10887, 2023.
References
Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, Exploring visual prompts for adapting large-scale models. arXiv preprint ar arXiv:2203.17274 2022.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In Computer vision-ECCV 2014; 13th European ference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pp. 446-461. Springer, 2014.
Khosla, A., Jayadevaprakash, N., Yao, B., and Li, F-F.. Novel dataset for fine-grained image categorization: Stan- ford dogs. In Proc. CVPR workshop fine-grained visual categorization (FGVC), volume 2, 2011.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Kipf, T. N. and Welling, M. Variational graph auto-encoders. arXiv preprint arXiv:1611.07300 2016a.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. arXiv preprint 2016b.
Cimpoi, M., Maji, S., Kokkinos, I,, Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images, 2009.
Deng, J. Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
leee, 2009.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEFE, 86(1 222222224 1998.
Fang, T., Zhang, Y. M., Yang, Y., and Wang, C. Prompt tuning for graph neural networks. arXiv preprint arXiv:2209.152400 2022.
Li, G., Muller, M., Thabet, A.. and Ghanem, B. Deepgens: Can gens go as deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9267-9276, 2019.
Fang, T., Zhang, Y., Yang, Y., Wang, C., and Chen, L. Uni- versal prompt tuning for graph neural networks. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Li, Q. and Zhou, J. Caprompt: Cyclic prompt aggregation for pre-trained model based class incremental learning. Proceedings of the AAAI Conference on Artificial In- telligence, volume 39, pp. 18421-18429, 2025.
Han, C.. Wang, Q. Cui, Y., Cao, Z., Wang, W., Qi. S., and Liu, D. E 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint vxXiv:2307133700 2023a.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the

---

Liu, Z., Xu, K., Su, B., Zou, X., Peng, Y., and Zhou, J. Stop: Integrated spatial-temporal dynamic prompting for video understanding. arXiv preprint arXiv:2503. 15973, 2025. Vision Graph Prompting via Semantic Low-Rank Decomposition
IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Sun, M., Zhou, K., He, X., Wang. Y., and Wang, X. Gppt: Graph pre-training and prompt tuning gener- alize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1717-1727, 2022.
Liu, Z., Yu, X., Fang, Y., and Zhang, X. Graphprompt: Uni- fying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, pp. 117-428, 2023.
Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. Building bird recognition app and large scale dataset with citizen scien- tists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 595-604, 2015.
Liu, Z., Peng, Y., and Zhou, J. Insvp: Efficient instance visual prompting from image itself. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset, 2011.
Wang. Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., and Solomon, J. M. Dynamic graph enn for learning on point clouds. ACM Transactions Graphics (tog), 38 (5): 2019.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05001, 2017.
Munir, M., Avery, W., and Marculescu, R. Mobilevig: Graph-based sparse attention for mobile vision appli- cations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2211- 2219, 2023.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530, 2018.
Yan, S., Xiong. Y., and Lin, D. Spatial temporal graph convolutional networks for skeleton-basee action recogni- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Munir, M., Avery, W., Rahman, M. M., and Marculescu, R. Greedyvig: Dynamic axial graph construction for efficient vision gnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pp.
Yao, Y., Liu, Z., Cui, Z., Peng, Y., and Zhou, J. Selective visual prompting vision mamba. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 22083-22099 2025.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised learning, volume 2011, 1, pp. Granada, 2011.
Zeng. R., Han, C., Wang, Q., Wu, C., Geng, T., Huang, L., Wu, Y. N., and Liu, D. Visual fourier prompt tuning. In NeurIPS, 2024.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint ar arXiv:2001.07676 2020.
Shi, W. and Rajkumar, R. Point-gnn Graph neural network for 3d object detection in point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1711-1719, 2020.
Stallkamp. J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural nenworks, 32: 323-332, 2012.
10

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
A. Appendix
A.I. Analysis on Efficiency
As shown in Table 5, we analyze the efficiency of our Vision Graph Prompting (VGP) in terms of parameter efficiency and computational cost. As parameter-efficient fine-tuning (PEFT) approach, our method aims to reduce trainable parameters, which means much less GPU memory consumption and storage burden, while retaining comparable performance against full fine-tuning and little computational overhead.
Parameter Efficiency. The key contribution of our method lies in its parameter-efficient design. In contrast to full fine-tunine thee-tuning where all parameters are updated, VGP only updates small portion of prompt parameters. Attributed to our semantic low-rank design, our method reduces 94.6% trainable parameters averagely while maintaining comparable even superior performance against full fine-tuning, demonstrated in Table I and Table respectively.
Computational Efficiency. In terms of computational cost, our VGP introduces little additional overhead of textbf3. I % averagely, demonstrated in Table 5. Attributed to our semantic low-rank decomposition design, we lot of computation by reducing feature dimension to low-rank dimension for computing, while extracting critical semantic information and filtering out disruption of noisy local details. Our approach does not require the re-training of the entire model, which helps to mitigate the computational cost typically associated with full fine-tuning.
In summary, the VGP framework provides a highly efficient approach to adapting Vision GNNs for downstream tasks, offering significant trainable parameters reduction and neglectable computational cost, while maintaining competitive performance compared full fine-tuning.
A.2. Implementation Details
Our experiments on vision tasks are based on medium pyramid Vision GNN model pre-trained on ImageNet- 21k (Krizhevsky et al., 2017). With the backbone parameters frozen, only our prompt modules and the task-specific head are trained. Following DAM-VP (Huang et al., 2023), train for 100 epochs for each dataset and incorporate 10 additional epochs for probing the optimal result. We utilize the AdamW (Loshchilov & Hutter, 2017) optimizer for optimization and implement cosine learning rate annealing. The learning rate is set as 0.001 and the weight decay is 0.05, Regarding the graph tasks, we follow the approach of GPF-Plus (Fang et al., 2023), utilizing a widely used 5-layer GIN as the underlying architecture. The GIN model is pre-trained on the chemistry and biology datasets correspondingly with the Edge Prediction strategy (Kipf& & Welling, 2016a).
A.3. Analysis on Semantic Low-Rank Property
In this work, we explore the feature extraction mechanism behind Vision GNN models. By visualizing vision graph structures, we find that semantically related object parts are consistently connected by dynamically constructed edges, regardless of variations in object shape, pose, or local texture. Additionally, we utilize Principal Component Analysis (PCA) to decompose the latent features and visualize the dominant components by mapping them to RGB color channels, as shown in Figure 2. Notably, semantically connected patches exhibit similar PCA components, suggesting an inherent low-rank property.
To provide theoretical foundation for this observation, we recall that PCA is a widely used dimensionality reduction technique that projects high-dimensional data onto a lower-dimensional subspace via a linear transformation, maximizing variance retention.
Principal Component Analysis (PCA) is a common data dimensionality reduction method. maps data from a high- dimensional space to low-dimensional space through linear transformation while preserving much variance as possible. The essence of PCA is indeed the Figenvalue Decomposition (EVD) based on the covariance matrix, which strictly retains rank equivalence.
Given a set of normalized node features X = [x x,1 E N xd, where n is number of nodes and d represents feature dimension, N > d. The corresponding covariance matrix E can be computed as:
Σ = - x X Rdxd
(13)

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
The eigenvalue decomposition of the covariance matrix be formulated
== V A A.
(14)
where
A = diag A1, A2...., Aa),
A .. 2120
(15)
and
V Vi..... Val E Rddd
(16)
Then we can express equivalently as:
== Σ Av Av-.v
(17)
The rank dimension r equals the non-zero number of eigenvalues AІ. In practice, eigenvalues X less than a certain threshold c. can be regarded as zero. Assuming that the reservation made before r principal components, the error term can be estimated by O()++1).
As depicted in Figure 2 and Figure 5, we visualize ViG graph structures alongside their PCA component coefficients. From Figure 2, we see that the semantic parts share similar coefficients of the first three dominant PCA components, which entails most feature distribution variance. Furthermore, we visualize the coefficient magnitude distribution of the PCA components, and an obvious long-tail tail phenomenon can be observed. Setting eπror threshold 0.25, the rank r of connected patches in CUB and Flowers dataset can be estimated as 50 and 60 respectively, significantly lower than the original feature dimension = 768. Based on this semantic low-rank observation, we design our VGP method, achieving both efficiency and efficacy in adapting ViG models.
A.4. Details Vision Datasets
As shown in Table 6, we provide more detailed information for our vision adaptation datasets. Diverse target classes and rich categories have been covered by these benchmarks, verifying the generalizability of our proposed method.
Table Efficiency metrics of our method compared to full fine-tuning across ten datasets downstream vision task. Trainable parameters (Param) and FLOPs reported to evaluate parameter efficiency and computational costs.
Methods
Backbone
Metric
DTD
CUB NABirds Dogs Flowers Food CIFAR CIFARIO GTSRB SVHN
Average
Param. (M)
48.55
48.71
49.54
48.62
48.61
48.6
48.6
48.51
48.54
48.51
48. 48.68(100%)
Full Tine-Tune
ViG-M
FLOPs (G)
8.94
8.94
8.94
8.94
8.94
8.94
8.94
8.94
8.94
8.94
8.94
Param. (M)
2.47
2.63
3.46
2.55
2.53
2.53
2.53
2.44
2.47
2.44
(-61
VGP(Ours)
ViG-M
FLOPs (G)
9.22
9.22
9.22
9.22
9.22
9.22
9.22
9.22
9.22
9.22
9.22
Table 6. Vision dataset statistics used in our downstream adaptation tasks.
Dataset
Target Class
Categories
Train
Val
Test
DTD (Cimpoi et al., 2014)
textures
47
1,880
1,880
1,880
CUB200 (Wah et al., 2011)
birds
200
5,394
600
5,794
NABirds (Van Horn et al., 2015)
birds
555
21,536
2,393
24,633
Stanford-Doge (Khosla et al., 2011)
dogs
120
10,800
1,200
8,580
Oxford-Flowedd (Nilsback & Zisserman, 2008)
flowers
102
1,020
1,020
6,149
Food101 (Bossard et al., 2014)
food dishes
101
60,600
15,150
25,250
CIFAR100 100 (Krizhevsky et al., 2009)
all
100
40,000
10,000
10,000
CIFAR10 (Krizhevsky et al., 2009)
all
10
40,000
10,000
10,000
GTSRB (Stallkamp et al., 2012)
traffic signs
43
21,312
2,526
12,630
SVHN (Netzer et al., 201 I)
numbers
10
58,605
14,652
26,032
12

---

Vision Graph Prompting via Semantic Low-Rank Decomposition
1.0
CUB
coefficients
0.6
0.4
0.0
0
100
200
300
400
500
600
700
800
PCA Component Index
Flowers
0.8
coefficients
0.6
0.4
0.2
0.0
400
700
PCA Component Index
Figure 5. Visualization of VIG graph structures, the first three major PCA components, and the average distribution of PCA component coefficient magnitudes.
13


Reviews:
[{"review_id": "gzKCrcaYOn", "reviewer": "Reviewer_x6eb", "summary": "In this work, the authors present Vision Graph Prompting (VGP), a parameter-efficient fine-tuning method for Vision Graph Neural Networks. The core insight is that semantic information in vision graphs primarily resides in the low-rank components of the latent feature space. The authors propose three semantic low-rank prompting methods: SeLo-Graph, SeLo-Edge, and SeLo-Node prompts, which capture global structural patterns and fine-grained semantic dependencies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n### Q1. Adding LoRA as a Prompt Alternative\nWe supplement additional experiments comparing with LoRA across ten vision datasets. As table shown below, LoRA surpasses traditional visual prompting method (VPT) due to its **low-rank adaptation property** and matches GPF-Plus’s performance. \n\nWhile LoRA focuses on low-rank adaptation within the model’s parameter space, it **does not leverage graph topology**, unable to refine structural relationships, limiting further gains. Our VGP achieves SOTA performance by jointly optimizing visual semantics and graph structures via semantic low-rank prompting.\n\n|Method|*DTD*|*CUB*|*NABirds*|*Dogs*|*Flowers*|*Food*|*CIFAR*|*CIFAR10*|*GTSRB*|*SVHN*|Average|\n|:------:|:------:|:------:|:-------:|:------:|:-------:|:------:|:------:|:-------:|:------:|:------:|:------:|\n|VPT|71.4|77.3|76.4|73.1|95.3|81.9|76.3|93.2|79.7|82.4|80.7|\n|GPF-Plus|71.0|82.0|77.2|78.2|95.7|82.6|80.9|94.5|90.5|83.1|83.6|\n|LoRA|69.7|79.2|77.3|74.0|94.6|83.5|81.4|94.9|90.2|90.8|83.6|\n|**VGP**|**74.8**|**87.4**|**80.9**|**81.7**|**98.2**|**89.5**|**89.7**|**98.3**|**98.1**|**96.9**|**89.6**|\n\n### W1. Implementation Details of Virtual Nodes\nThanks for your reminder. We provide additional implementation details of virtual nodes:\n\n1) The virtual nodes in SeLo-Graph Prompt are initialized using **Kaiming Normal distribution**.\n\n2) The number of virtual nodes $M$ is set to **14**. Ablation experiments is conducted as table below. A smaller number leads to suboptimal prompting effects due to insufficient guidance, while an excessive number does not yield further improvements but incurring additional parameter cost.\n\n|Virtual Node Number $M$|0|3|7|14|28|42|\n|:---------------------:|:--:|:--:|:--:|:------:|:--:|:--:|\n|*CUB*|85.8|86.3|86.9|**87.4**|87.2|86.9|\n|*GTSRB*|93.4|95.5|97.3|**98.1**|98.0|97.6|\n|*SVHN*|95.1|96.0|96.2|**96.9**|96.7|96.8|\n\n### W2. Dealing with Large or Complex Graph\nOur VGP is capable of handling large and complex graph data. In our experiments on chemistry/biology graph datasets in Table 2, the number of nodes can reach **5,000** with **non-uniform edges distribution**, far more complex than the **196-nodes** image graphs. The table below presents the average graph sizes for different chemistry/biology datasets. Even though, our VGP still achieves **seven** SOTA results across nine benchmarks with only **0.15M** parameters, verifying its robustness and generalizability.\n\nAs for number of prompts $M$ in the SeLo-Graph Prompt, we follow the same setting with vision datasets as 14, not specifically tuned for chemistry/biology datasets. Even though the graph are much larger and more complex graphs, our method still excels with other graph prompting methods with a general hyperparameter setting, verifying its robustness.\n\n|Datasets|*BBBP*|*Tox21*|*ToxCast*|*SIDER*|*ClinTox*|*MUV*|*HIV*|*BACE*|*PPI*|\n|:--------:|:----:|:-----:|:-------:|:-----:|:-------:|:---:|:---:|:----:|:------:|\n|Graph Size|776|516|583|741|989|828|893|1074|**5139**|\n\n### W3. Parameter Quantities Comparison\nWe provide a parameter comparison of SOTA methods on the *CUB* dataset, as table shown below. Our method achieves high efficiency, requiring only **2.63M** trainable parameters (**5%** of ViG-M’s full fine-tuning at **48.71M**). This is due to our **lightweight low-rank design**, which avoids large parameter matrices while maintaining strong performance.\n\n|Method|ViG-M|VPT|Ins-VP|GPF-Plus|Adapter|DAM-VP|VGP|\n|:--------:|:---:|:--:|:----:|:------:|:-----:|:----:|:--:|\n|Param.(M)|48.71|1.77|1.83|2.29|3.48|6.24|2.63|\n\n### W4. Whether SeLo-Node Prompt Acts on Virtual Nodes\nYes, the SeLo-Node Prompt acts on all the nodes within graph, including virtual nodes inserted by SeLo-Graph Prompt. We will provide a more explicit description of prompting process in final version for better clarity as below.\n\n1) SeLo-Graph Prompt inserts virtual nodes and builds virtual edges, updating graph structures\n2) SeLo-Edge Prompt refining the edge-level semantic interactions via edges within updated graph\n3) SeLo-Node Prompt intensifies node-level semantic information on each node in updated graph\n\n### Each Component's Individual Contribution\nWe supplement additional ablation experiments on different components combinations as table shown below. While SeLo-Graph Prompt refines graph structures, SeLo-Edge and SeLo-Node Prompt enhance low-rank semantics between and within nodes. Each component contributes to performance gains.\n\n|SeLo-Graph|SeLo-Edge|SeLo-Node|*CUB*|*GTSRB*|\n|:--------:|:-------:|:-------:|:------:|:------:|\n|-|-|-|76.2|77.4|\n|√|-|-|81.9|86.9|\n|-|√|-|82.3|87.5|\n|-|-|√|81.0|86.5|\n|√|√|-|85.3|93.0|\n|√|-|√|85.5|93.3|\n|-|√|√|85.8|93.4|\n|√|√|√|**87.4**|**98.1**|", "claims_and_evidence": "Overall, the claims are well-supported and clear.", "methods_and_evaluation_criteria": "The evaluation criteria, including accuracy and parameter efficiency across diverse datasets, are appropriate and comprehensive.", "theoretical_claims": "I roughly checked the theoretical claims in the article. They're mostly based on existing theories and seem reasonable.", "experimental_designs_or_analyses": "The ablation study in the paper does not include experiments where only SeLo-Edge or only SeLo-Node is used, nor does it show the results of combining SeLo-Graph with SeLo-Node. This limits the thoroughness of the analysis of each component's individual contribution.", "relation_to_broader_scientific_literature": "This work bridges the gap between Transformer-focused prompting techniques and graph-based vision models, contributing to the development of parameter-efficient fine-tuning methods for ViG and potentially other graph neural network applications.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n1. The authors present a parameter-efficient fine-tuning method specifically designed for Vision Graph Neural Networks (ViG), addressing a previously under-explored area.\n\n2. The core insight regarding the low-rank properties of semantic information in vision graphs is well-supported and motivated.\n\n3. Extensive experiments across diverse datasets demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n\n1. Missing detailed information on the implementation of virtual nodes, such as their initialization method and how the number of virtual nodes is determined. \n\n2. The author does not discuss potential issues that may arise when dealing with very large or complex graph structures. Additionally, the author does not clarify the number of prompts M in the SeLo-Graph Prompt, which may also be crucial to the the effectiveness of the method.\n\n3. The proposed method appears to be instance-level, which may result in a significantly larger number of parameters compared to more generic prompt methods. Though a comparison with the full fine-tuning method is given in the appendix, the paper does not provide a detailed comparison of parameter quantities, which is important for assessing the method's efficiency.\n\n4. It is unclear whether SeLo-Node acts on all nodes in the graph or if it includes the virtual nodes from SeLo-Graph. This ambiguity affects the understanding of the method's application scope and its impact on different parts of the graph.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "I'm curious, given the authors' finding that the original graph structure's features have a low-rank decomposition property, could they consider adding LoRA to ViG for fine-tuning as a prompt alternative?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A."}}, {"review_id": "QyptzLoZzd", "reviewer": "Reviewer_kLEh", "summary": "This paper proposes a novel approach called Vision Graph Prompting (VGP), which enables parameter-efficient fine-tuning of the Vision GNN (ViG) model. Additionally, the paper observes that essential semantic information in Vision Graph structures is concentrated in low-rank components and leverages this insight to introduce a Semantic Low-Rank Decomposition-based prompting method. To capture both global and local semantic features within the graph structure, three key components—SeLo-Graph, SeLo-Edge, and SeLo-Node Prompt—are introduced. Experimental results demonstrate that this approach significantly enhances the transfer learning performance of the ViG model while requiring far fewer parameters compared to full fine-tuning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n### Q1. Experiments on Other Graph-based Vision Models\nWe supplement additional experiments on other graph-based vision models, including **MobileViG and GreedyViG**, across six vision datasets with ImageNet-1k pre-trained backbones. As table shown below, our VGP consistently excels other SOTA vision and graph prompting methods, demonstrating robustness across backbones due to our adaptability to diverse graph structures.\n\n|Method|*DTD*|*CUB*|*Flowers*|*Food*|*CIFAR10*|*SVHN*|Average|\n|:------:|:---:|:---:|:-------:|:----:|:-------:|:----:|:------:|\n||||MobileViG|||||\n|GPF-Plus|68.5|81.4|94.3|82.0|94.1|82.2|83.7|\n|InsVP|68.1|84.0|95.2|83.9|95.2|88.9|85.9|\n|**VGP**|71.6|84.9|97.6|88.0|96.8|94.7|**88.9**|\n||||GreedyViG|||||\n|GPF-Plus|69.3|81.7|94.9|82.2|94.5|82.7|84.2|\n|InsVP|68.8|84.1|95.5|84.0|95.5|89.1|86.2|\n|**VGP**|72.1|85.4|98.0|87.3|97.2|94.5|**89.1**|\n\n### Q2. Ablation on Altered Graph Structures\nWe conduct additional ablation studies to analyze the impact of structural modifications in the SeLo-Graph Prompt. Our method **inserts virtual nodes** and **dynamically constructs edges** based on feature similarity, thereby altering the graph structure.\n\nAs table shown below, both virtual node insertion and edge construction enhance feature extraction. While static edge allocation improves performance, **dynamic edge construction based on feature similarity achieves the best results**, as it better captures complex semantic relationships.\n\n|**Ablation**|**CUB**|**GTSRB**|\n|------------------------------------------------|--------|---------|\n|w/o SeLo-Graph Prompt|85.8|93.4|\n|Only insert virtual nodes|86.2|94.1|\n|Insert virtual nodes+static edge allocation|86.6|96.9|\n|Insert virtual nodes+dynamic edge construction|**87.4**|**98.1**|\n\n### Q3. Analysis of Explainability of Prompting Impact\nFig.2 and Fig.5 in paper show that fully fine-tuned vision graph models can recognize semantically related patches and connect them via edges, exhibiting a semantic low-rank property. \n\nOur VGP reinforces this effect through three prompting modules, ensuring **low-rank feature consistency** across connected patches. This mimics the behavior of fully fine-tuned models, effectively linking semantically related regions.\n\nAs shown in Table 1, our method effectively extracts discriminative semantic information, leading to significant performance gains.\n\n### Q4. Experiments in Other Domains\nWe supplement additional experiments on **remote sensing tasks on *EuroSAT* dataset**, which consists of satellite images from Sentinel-2. As table shown below, our method still compels other SOTA prompting methods, even getting comparable results with full fine-tuning with only **4%** trainable parameters.\n\n|Method|ViG-M|Adapter|VPT|InsVP|GraphPrompt|GPF-Plus|VGP|\n|:----------------:|:---:|:-----:|:---:|:---:|:---------:|:------:|:-------:|\n|*EuroSAT* Acc.(%)|92.37|85.24|83.55|87.14|85.50|86.97|**91.98**|\n\n### W1. Statistical Significance and Hyperparameter Selection\nWe run all experiments for **three times** with different seeds and report the highest results. The average standard deviation is **0.3%**, which is significant lower than our **4%** performance gain, confirming the robustness of our results. Detailed statistical significance for each dataset will be provided in final version.\n\nFor hyperparameter selection (r, α, β), we evaluate multiple candidates and select the optimal ones. And we use a fixed set of hyperparameters across all datasets.\n\n### S1,S2,S3. Comments and Suggestions\nWe appreciate your feedback and will address the following in final version:\n\n1) We have corrected typos in the supplementary materials.\n2) We will discuss limitations and failure cases to guide future research.\n3) Additional PCA visualization details are included. Specifically, we compute PCA components for all patch tokens encoded by the trained model. The latent feature space is decomposed into PCA components, where those with large coefficients capture major variance. We map the top three PCA components into RGB channels for visualization, ensuring patches with similar colors share similar PCA component distributions, thus indicating low-rank properties. This approach is similar to visualization of famous DINOv2.\n\n### Graph Prompt Structurally Optimized for ViG\nStandard vision prompting methods (e.g., VP, VPT) operate on **pixel-level** or **token-level** representations without explicit graph structures.\n\nIn contrast, graph prompting methods (e.g., GraphPrompt, GPF) **directly modify node features, insert virtual nodes, or establish new edges**, enabling structured graph-based prompting while overlooking semantic features in vision data.\n\nOur VGP builds upon this principle, explicitly optimizing prompts for graph-structured vision models, incorporating semantic low-rank decomposition strategy.\n\n### Number of Repetitions\nFollowing DAM-VP (Appendix A.2), we train each dataset for **100 epochs**.", "claims_and_evidence": "- This paper visually demonstrates, through Figure 2 and Figure 5, that the primary semantic information of the Vision Graph is concentrated in the lower-dimensional components via PCA analysis. Additionally, the Ablation Study in Table 3 experimentally proves that SeLo-Graph, SeLo-Edge, and SeLo-Node Prompt each contribute to performance improvement. \n- The experimental results in Table 1 and Table 2 further indicate that the proposed method outperforms existing approaches across various benchmark experiments.\n\n- However, there is a lack of experiments comparing the cases with and without the application of low-dimensional decomposition. Therefore, it remains unclear how significant the performance improvement is compared to the basic ViG.\n- It is necessary to provide a more detailed explanation of why the graph prompting technique is structurally optimized for the ViG model.\n- In this paper, the impact of blending factors α and β on performance is discussed, indicating that the optimal values are found within a specific range (0.1 to 0.3). Although this claim is supported by quantitative analysis, more detailed insights should be provided regarding why deviations from this range lead to performance degradation.", "methods_and_evaluation_criteria": "- This paper conducts experiments using various benchmarks (CIFAR, CUB, GTSRB, and chemical/biological graph data) and appropriately analyzes the contribution of each technique through an Ablation Study. Additionally, the evaluation considering the balance between parameter efficiency and performance appears to be well-justified.\n\n- While the paper claims that the proposed method achieves results comparable to full fine-tuning, it does not specify the exact performance evaluation metrics used for comparison. Including metrics such as accuracy, F1 score, or AUC would provide clearer insights into the effectiveness of the proposed method.\n- Additional explanations on the graph datasets should be provided. While it appears that datasets from GPF-PLUS and MoleculeNet were used, there is a lack of detailed descriptions regarding their characteristics (even the appendix does not provide an explanation).\n- It is necessary to verify whether the proposed method demonstrates the same effectiveness in other graph-based vision models (e.g., MobileViG, GreedyViG).\nFurther analysis should be conducted to determine whether the performance of the proposed low-dimensional decomposition method varies depending on the dataset.\n- A clearer analysis of how changes in graph structure affect performance during the prompting process would be beneficial.", "theoretical_claims": "- This paper demonstrates through PCA-based analysis that the semantic information of the Vision Graph is primarily contained in low-dimensional components. By utilizing Eigenvalue Decomposition (EVD) of the Covariance Matrix, it shows that semantic information is concentrated in a few principal components. Furthermore, based on this mathematical foundation, it logically validates the effectiveness of Semantic Low-Rank Decomposition. \n\n- This paper claims that the proposed method effectively captures critical semantic information in Vision GNNs, thereby enhancing feature extraction. While this claim is supported by experimental results, the theoretical justification for how this improvement is achieved through low-rank decomposition and graph adaptation needs to be clearly articulated.\n- The paper explains that extensive experiments demonstrate significant improvements in transfer performance across various downstream tasks. However, the logical connection between the theoretical claims and the experimental results should be further strengthened. In particular, providing a more detailed explanation of how the proposed theoretical framework translates into practical performance gains would enhance the coherence of the paper and establish a clearer pathway from theory to application.", "experimental_designs_or_analyses": "- This paper has appropriately set up comparison groups for the experiments and conducted a comprehensive comparative analysis, including existing prompting techniques (VPT, InsVP) and the graph prompting (GPF-Plus) technique. The Ablation Study verifies the contribution of each proposed technique to performance improvement, ensuring logical validity.\n\n\n- However, providing more explicit information on the number of repetitions (epoch) would enhance the transparency of the experimental design.\n- This paper claims significant improvements in transfer performance, but it does not include detailed statistical analyses or significance testing results. Incorporating such analyses would help assess the robustness of the findings and is essential for demonstrating that the observed improvements are not merely due to random chance.", "relation_to_broader_scientific_literature": "- The paper effectively summarizes the relationship with existing Vision GNN and Vision Prompting research, clearly explaining the differences from Transformer-based prompting techniques\n\n- It is necessary to verify whether the proposed method demonstrates the same effectiveness in other graph-based vision models (e.g., MobileViG, GreedyViG).", "essential_references_not_discussed": "- The paper effectively summarizes existing research, particularly related to Vision GNN and Vision Prompting, and does not appear to have omitted any essential studies that should have been mentioned.", "other_strengths_and_weaknesses": "- The paper proposes the first Vision Graph Prompting technique for the ViG model, demonstrating high originality. It achieves superior performance compared to existing methods while maintaining parameter efficiency. The inclusion of an Ablation Study and various benchmark experiments enhances the reliability of the research.\n\n- It would be beneficial to include additional statistical significance for the experimental results. There are changes in model performance based on hyperparameters (r, α, β), and it may be necessary to include a process for finding the optimal values.", "other_comments_or_suggestions": "- Grammar errors and inaccurate expressions:\n\"textbf3.1%\" → \"3.1%\" (Typo correction needed)\n\"demonstrated in Table reftab-datasets\" → \"demonstrated in Table 6\" (Reference error correction)\n- It is necessary to more clearly articulate the limitations of the study in the conclusion.\nFor example: analysis of the causes of poor performance on certain datasets, suggesting additional research directions, etc.\n- Providing a clearer explanation of the PCA visualization in Figure 5 would be beneficial.\nThe current explanation is somewhat brief, which may make it difficult for readers to easily understand the meaning of the graph.", "questions_for_authors": "- Will the proposed method show the same effect in other graph-based vision models (e.g., MobileViG, GreedyViG)?\n- How will performance change if the structure of the graph is altered?\n- Has the impact of prompting on the model's explainability been analyzed?\n- Is it possible to achieve the same performance improvements in other domains, such as autonomous driving, medical imaging, and remote sensing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fY02WvywOX", "reviewer": "Reviewer_WsNv", "summary": "This paper introduces a novel parameter-efficient method called **V**ision **G**raph **P**rompting (**VGP**) with semantic low-rank decomposition for Vision GNNs. Empirical results demonstrate that the proposed approach achieves impressive performance on both image classification and traditional graph classification tasks. In addition, the paper provides supporting visualization evidence via PCA, which underscores the motivation behind the low-rank decomposition design in the prompts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n### Q1. Cluster Located in Bottom-Right Corner of Figure 1\n\nWe further check the correspondence between the t-SNE clusters and images patches, finding that the bottom-right cluster corresponds to **the bird's reflection on the water**. This observation aligns with the PCA visualizations in Figures 1 and 2, where the bird’s reflection is also highlighted. \n\nInterestingly, the ViG model appears to learn semantic information about reflections as a byproduct of supervision in the bird classification task.\n\nHowever, since the model lacks explicit supervision on background elements, the background features exhibit a sparse distribution in the upper region of the t-SNE figure.\n\n### W1. PCA with Multiple Objects and Complex Backgrounds \n\nIn Figure 2(b) of our paper, the samples from *Flowers* dataset already contains **complex backgrounds with cluttered grass and leaves**, as well as instances with multiple objects (e.g., **the top-right sample with two flowers**).\n\nThe results demonstrate that PCA effectively extracts target objects using trained ViG model’s features, attributed to the **semantic low-rank property** of the ViG's latent feature space. In the final version, we will provide additional visualizations specifically focusing on multiple objects and complex backgrounds.(https://anonymous.4open.science/r/ICML25-anonymous-DF1B/multi-objects-w-complex-backgrounds.pdf)\n\n### W2. Summarizing Proposed Three Modules for Training\n\nCombining Equation 11 and 12 in the paper, we further summarize the prompted ViG model as equation below. The updated modules during training are underlined. Only the three **low-rank prompt matrices**, one **semantic feature extraction MLP** and the **low-rank virtual nodes** are trained, while all other modules in the ViG backbone remain frozen:\n\n$\\hat{f}(\\mathbf{x}\\_i)= (1-\\beta)\\cdot\\mathbf{x}\\_i+\\hat{g}(\\mathbf{x}\\_i, \\underline{\\mathbf{P\\_n}}) \\cdot \\mathbf{W}\\_{update} +\\sum\\_{\\mathbf{x}\\_j \\in \\hat{\\mathcal{N}}(\\mathbf{x}\\_i)} \\beta\\cdot  \\underline{\\mathrm{MLP\\_s}}(\\mathbf{x}\\_j)\\cdot\\underline{\\mathbf{P\\_e}}, ~~~\\hat{\\mathcal{N}}(\\mathbf{x}\\_i)  \\subseteq [\\mathbf{X},~[\\underline{\\mathbf{n}\\_1, \\dots, \\mathbf{n}\\_M}]\\cdot \\underline{\\mathbf{P\\_g}}]$\n\n### W3. Differences between ViT-based and ViG-based Prompts\n\nThe ViT-based prompting methods either prompting on images pixels like VP, explicitly adjusting the RGB channels space, or prompting on image tokens like VPT, functioning via feature similarity-based attention mechanism.\n\nHowever, these methods lack awareness of **graph structures**, such as edge connections between patches. While ViG-based graph prompting methods like GraphPrompt and GPF-Plus explicitly alters graph structures, including modifying node features, inserting new nodes and constructing new edges, thus better leveraging the graph representation.\n\nAs for ViT-based prompting methods outperform ViG-based one on certain datasets, this is likely because vision datasets contain both visual features and latent graph structures. ViT-based vision prompting methods excel in processing **raw vision data**, whereas ViG-based methods are more effective at **graph-based reasoning**. Consequently, each approach has its own advantages, leading to instances where ViT-based prompting achieves superior results.\n\n### W4. Parameter Sizes of the Backbone\n\nWe have provided details of both parameter sizes and computation costs of backbone ViG and our VGP in **Appendix A.1 and Table 5**. The ViG model has **48.68M** parameters averagely and our VGP only has **2.61M** trainable parameters, reducing **94.6%** from full fine-tuning. \n\n### W5. Extending to Other Vision Tasks\n\nWe supplement additional semantic segmentation tasks on *ADE20K* dataset. As table shown below, our method consistently excels other vision and graph prompting methods on semantic segmentation tasks, demonstrating its effectiveness across different vision tasks.\n\n|Method|ViG-M|Adapter|VPT|InsVP|GraphPrompt|VGP|\n|:------:|:---:|:-----:|:--:|:---:|:---------:|:------:|\n|mIoU(%)|47.9|44.2|41.6|42.3|44.4|**47.6**|", "claims_and_evidence": "The authors assert that semantically connected components in the graph exhibit low-rank properties, as evidenced by visualizations produced using PCA and t-SNE. However, I believe this visualization approach may have a critical limitation due to the capacity of PCA in effectively extracting the target object in complex images. Specifically, PCA may struggle to isolate the target object when selecting the top components, particularly in scenarios where images contain multiple objects or intricate backgrounds. Therefore, I suggest providing additional visualization results of PCA components, especially for images with multiple objects and complex backgrounds, to better evaluate the method’s effectiveness under such conditions.", "methods_and_evaluation_criteria": "Overall, the proposed method, VGP, provides an effective solution for leveraging semantic graph information within a low-rank space. Specifically, the semantic low-rank decomposition framework of VGP, including the SeLo-Graph Prompt, SeLo-Edge Prompt, and SeLo-Node Prompt, facilitates both structural adaptation and feature enhancement.", "theoretical_claims": "I have carefully reviewed the theoretical aspects of this paper and did not identify any obvious errors. However, I noticed that the authors did not provide an equation summarizing the proposed three modules for parameter updating. Including such an equation is recommended to enhance the clarity and understanding of the overall method.", "experimental_designs_or_analyses": "The experimental results of the proposed method, as presented in Tables 1 and 2, are promising. However, the authors do not appear to provide sufficient analysis regarding the differences between ViT-based methods and ViG-based ones.\n\nFirstly, the parameter sizes of the selected backbones remain unclear and should be explicitly stated for better comparison. Secondly, it is recommended that the authors analyze why the basic visual prompting method for ViTs (i.e., VPT) outperforms ViG-based prompting methods on certain datasets, such as CUB, NABirds, Dogs, and Flowers. Specifically, the authors should offer a more detailed discussion on the advantages and disadvantages of ViT-based and ViG-based prompting methods. This would help readers better understand the critical discrepancies between these two approaches.", "relation_to_broader_scientific_literature": "The authors validate the effectiveness of the proposed method solely on image classification and graph classification tasks in this paper. However, it remains unclear whether the method can be extended to other vision tasks, such as object detection and segmentation.", "essential_references_not_discussed": "I am not an expert in this field, so I am unsure if there are any related references that have not been cited in this paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The originality of the proposed method stems from the innovative combination of existing ideas, including low-rank adaptation and Vision GNN, which demonstrates a creative and thoughtful approach.\n\n2. The writing in this paper is clear and well-structured, making it easy for readers to understand both the motivation behind the work and the effectiveness of the proposed method.\n\nWeaknesses:\n\nMy major concerns can be found in previous parts of Claims And Evidence, Theoretical Claims, and Experimental Designs Or Analyses.", "other_comments_or_suggestions": "I have no further comments.", "questions_for_authors": "I am curious about the cluster located in the bottom-right corner of Figure 1. It appears to differ significantly from the other clusters. While the patches in the top figure seem to represent the background of the overall image, it is unclear what this particular cluster corresponds to. Could you clarify its meaning or significance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhCQV7nO2n", "reviewer": "Reviewer_53Z7", "summary": "The paper introduces Vision Graph Prompting (VGP), a novel parameter-efficient fine-tuning method tailored for Vision Graph Neural Networks (ViG). The authors propose that semantic information in vision graphs resides primarily in low-rank components of the latent feature space, a key insight derived from PCA-based analysis of graph structures. Building on this, VGP incorporates three prompt types—SeLo-Graph Prompt, SeLo-Edge Prompt, and SeLo-Node Prompt—each leveraging semantic low-rank decomposition to capture global and local semantic dependencies within ViG topologies. The method freezes the pre-trained ViG backbone and fine-tunes only the prompts and a downstream head, achieving performance comparable to full fine-tuning with significantly fewer trainable parameters. Extensive experiments on ten vision datasets (e.g., CUB, Flowers, GTSRB) and nine graph datasets (e.g., BBBP, Tox21, PPI) demonstrate that VGP outperforms existing visual and graph prompting methods, achieving an average accuracy of 89.6% on vision tasks and 76.39% on graph tasks, surpassing full fine-tuning in several cases. The main contributions include the VGP framework, the low-rank prompting insight, and its superior transfer performance across diverse downstream tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n### Q1,W1. Generalizability to Graph Tasks\n\nFrom the efficacy of our method on chemistry/biology graph datasets, we hypothesize **similar latent semantic low-rank patterns** also exist in these graph data. In particular, **chemical bonds** and **protein interaction** structures exhibit structured low-rank properties similar to **semantic regions in images**.\n\nFor example, in *PPI* (Protein-Protein Interaction) dataset, each node in graph represents a type of protein, while edges denote interaction relationships. These interactions are primarily driven by **specific functional groups** such as hydroxyl and carboxyl groups, which are crucial to biochemical reactions, analogous to **low-rank semantic features in vision images**. Conversely, other chemical groups that do not significantly contribute to interactions correspond to **high-frequency local details** in images, which tend to be redundant.\n\nSo when extracting features from these protein graph data for tasks such as protein function prediction, it is essential to **identify the key functional groups** that drive interactions. Since our **VGP model is designed to capture low-rank semantic structures**, it effectively generalizes to chemistry and biology graph datasets, surpassing prior graph prompting methods.\n\nDue to the inherent abstract nature of protein interaction graphs, it is challenging to visualize this similar semantic pattern like 2D images in Figure 2. To illustrate the underlying patterns, we visualize the graph structures and color the nodes with corresponding the first three PCA components. We make comparisons between node features obtained from trained and untrained GNN models on *PPI* dataset. Interestingly, we find that node features from trained GNN models present significantly better low-rank features consistency. The visualization is provided in an anonymous link(https://anonymous.4open.science/r/ICML25-anonymous-DF1B/PPI-PCA.pdf).\n\n### Q2. Choice of Rank $r=32$\n\nOur choice of $r=32$ is motivated by two key considerations:\n\n1) **Trade-off between performance and parameter efficiency**. As shown in Table 4, *CUB* gets near-optimal results with **87.4%** of $r=32$ and **87.2%** of $r=64$ and its estimated rank is about 50, falling between these two values. Besides, *CIFAR* gets peak results when $r=64$ and second-best performance when $r=32$. In aspect of performance, choose $r=32$ or $r=64$ both seems plausible. However, in aspect of parameter efficiency, increasing $r$ from 32 to 64 **nearly doubles the trainable parameters**. So we tend to choose the smaller one, as $r=32$ for an optimal balance between performance and parameter efficiency.\n2) **Consistent hyperparameters across datasets**. To maintain a unified hyperparameter setting and avoid dataset-specific tuning, we adopt consistent $r=32$ across all the datasets. Even though estimated ranks of *CUB* and *Flowers* are 50 and 60 respectively, other datasets like *SVHN* and *CIFAR10* exhibit lower ranks as 18 and 20, as table shown below. To reach a reasonable compromise between these datasets, we set $r=32$ to satisfy majority of datasets.\n\n|               | *DTD* | *CUB* | *NABirds* | *Dogs* | *Flowers* | *Food* | *CIFAR* | *CIFAR10* | *GTSRB* | *SVHN* |\n| :-----------: | :---: | :---: | :-------: | :----: | :-------: | :----: | :-----: | :-------: | :-----: | :----: |\n| estimated $r$ |  36   |  50   |    90     |   30   |    60     |   46   |   55    |    20     |   26    |   18   |\n\n\n\n### W2. Backbone Variety\n\nWe supplement additional experiments based on other representative graph-based vision models, including MobileViG and GreedyViG. The experiments are conducted on six vision datasets and the backbones are pre-trained on ImageNet-1k. As table shown below, our VGP consistently excels other SOTA vision prompting and graph prompting methods, demonstrating robustness across backbones due to our adaptability to diverse graph structures.\n\n|  Method  |  *DTD*   |  *CUB*   | *Flowers* |  *Food*  | *CIFAR10* |  *SVHN*  | Average  |\n| :------: | :------: | :------: | :-------: | :------: | :-------: | :------: | :------: |\n|          |          |          | MobileViG |          |           |          |          |\n| GPF-Plus |   68.5   |   81.4   |   94.3    |   82.0   |   94.1    |   82.2   |   83.7   |\n|  InsVP   |   68.1   |   84.0   |   95.2    |   83.9   |   95.2    |   88.9   |   85.9   |\n| **VGP**  | **71.6** | **84.9** | **97.6**  | **88.0** | **96.8**  | **94.7** | **88.9** |\n|          |          |          | GreedyViG |          |           |          |          |\n| GPF-Plus |   69.3   |   81.7   |   94.9    |   82.2   |   94.5    |   82.7   |   84.2   |\n|  InsVP   |   68.8   |   84.1   |   95.5    |   84.0   |   95.5    |   89.1   |   86.2   |\n| **VGP**  | **72.1** | **85.4** | **98.0**  | **87.3** | **97.2**  | **94.5** | **89.1** |", "claims_and_evidence": "The claims in the paper are generally well-supported by clear and convincing evidence. The primary claim—that VGP achieves performance comparable to full fine-tuning while being parameter-efficient—is substantiated by quantitative results in Tables 1 and 2, showing VGP’s accuracy surpassing or matching baselines across diverse datasets. The assertion of semantic information residing in low-rank components is convincingly supported by PCA visualizations (Figures 2 and 5) and theoretical discussion in Appendix A.3, linking shared PCA components to low-rank properties. Ablation studies (Table 4, Figure 4) further validate the effectiveness of individual components (SeLo-Graph, SeLo-Edge, SeLo-Node) and hyperparameter choices (e.g., rank $r$, blending factors $\\alpha$ and $\\beta$). However, the claim of generalizability to traditional graph tasks (Section 5.3) is slightly weaker due to the lack of detailed analysis on why low-rank properties extend to chemistry/biology domains beyond a hypothesis. While plausible, this claim could benefit from additional evidence, such as a similar PCA analysis on graph datasets, to strengthen its foundation.", "methods_and_evaluation_criteria": "The proposed VGP method and its evaluation criteria are well-suited to the problem of adapting ViG models for downstream vision tasks. The method’s design—introducing low-rank prompts at graph, edge, and node levels—aligns logically with the topological nature of ViG, addressing the limitations of Transformer-centric prompting methods. Using ten vision datasets (e.g., CUB, GTSRB, SVHN) with diverse categories and distributions is a robust choice for evaluating transfer performance, as is the extension to nine chemistry/biology graph datasets to test generalizability. The evaluation metric (classification accuracy) is standard and appropriate for these tasks. The experimental setup, including freezing the backbone and training for 100 epochs with AdamW optimization, is reasonable and consistent with prior work (e.g., DAM-VP, InsVP). However, the choice of a single ViG-M backbone (pre-trained on ImageNet-21k) could be expanded to other ViG variants (e.g., MobileViG) to validate robustness further, though this is a minor concern given the focus on prompting efficiency.", "theoretical_claims": "The paper includes a theoretical claim in Appendix A.3 linking the low-rank property of semantic information to PCA and eigenvalue decomposition (Equations 13-15). I reviewed the correctness of this analysis, which builds on standard PCA principles to argue that semantically connected nodes share dominant components, implying a low-rank structure. The formulation appears mathematically sound: the covariance matrix decomposition and rank estimation based on eigenvalue thresholds are consistent with PCA theory. The error term estimation ($O(\\lambda_{r+1})$) is a reasonable approximation, though it assumes a clear eigenvalue drop-off, which is visually supported by Figure 5’s long-tail distribution. No significant issues were found, but the analysis could be strengthened by quantifying the variance captured by the chosen rank $r$ (e.g., 50 for CUB) to directly tie it to the experimental choice of $r=32", "experimental_designs_or_analyses": "I examined the experimental designs and analyses in Sections 5 and 5.4, including the quantitative results (Tables 1, 2, 5) and ablation studies (Table 4, Figure 4). The design is sound: comparing VGP against full fine-tuning and state-of-the-art prompting methods (e.g., InsVP, GraphPrompt) on diverse datasets ensures a fair and comprehensive evaluation. The ablation studies systematically test core components, rank $r$, and blending factors $\\alpha$ and $\\beta$, with results consistently showing performance improvements (e.g., 5.7% gain from SeLo-Graph on CUB). The statistical validity is supported by the use of standard splits (e.g., scaffold split for chemistry datasets) and consistent augmentation strategies. One minor issue is the lack of statistical significance testing (e.g., confidence intervals) for accuracy differences, which could bolster claims of superiority (e.g., VGP’s 89.6% vs. InsVP’s 84.6% on vision tasks). Additionally, the computational efficiency claim (3.1% FLOPs overhead, Table 5) is plausible but could be clarified by detailing how FLOPs were calculated for prompt operations.", "relation_to_broader_scientific_literature": "The paper’s key contributions align well with trends in parameter-efficient fine-tuning (PEFT) and graph neural networks (GNNs). The use of prompting for vision tasks builds on prior work like VPT (Jia et al., 2022) and InsVP (Liu et al., 2024), extending it to ViG, a graph-based vision backbone introduced by Han et al. (2022). The low-rank decomposition idea echoes techniques in efficient Transformer adaptation (e.g., LoRA, Hu et al., 2021, not cited) but is novel in its application to graph structures. The extension to traditional graph tasks (e.g., MoleculeNet) ties into GNN prompting literature (e.g., GraphPrompt, Liu et al., 2023; GPF-Plus, Fang et al., 2023), offering a bridge between vision and graph domains. The insight into low-rank semantic properties also resonates with dimensionality reduction studies in GNNs (e.g., Kipf & Welling, 2016b), though applied uniquely to vision graphs.", "essential_references_not_discussed": "While the paper cites relevant prior work, two areas could benefit from additional references:\n\nLow-Rank Adaptation: The low-rank decomposition approach shares conceptual similarities with LoRA (Hu et al., 2021, \"LoRA: Low-Rank Adaptation of Large Language Models,\" ICLR 2022), a PEFT method for Transformers. Discussing LoRA could contextualize VGP’s novelty in adapting low-rank ideas to graph structures.\nGraph Compression: The low-rank insight might relate to graph compression techniques like \"GraphSAGE\" (Hamilton et al., 2017, NIPS), which aggregates neighborhood features efficiently. Citing this could clarify how VGP differs from prior graph feature reduction methods.\nThese omissions do not undermine the work but could enhance its positioning within the broader PEFT and GNN literature.", "other_strengths_and_weaknesses": "**Strengths**:\n\n*Originality*: VGP is a pioneering effort in prompting ViG models, creatively adapting low-rank concepts to graph structures.\n\n*Significance*: The method’s parameter efficiency (94.6% reduction) and strong performance (e.g., 89.6% average accuracy) make it highly practical for resource-constrained settings.\n\n*Clarity*: The paper is well-written, with clear explanations of the method (Section 4) and insightful visualizations (Figures 2, 3).\n\n**Weaknesses**:\n\n*Clarity of Generalizability*: The extension to graph tasks (Table 2) is compelling but lacks depth in explaining why low-rank properties hold beyond vision, limiting interpretability.\n\n*Limited Backbone Variety*: Testing only on ViG-M restricts insights into broader applicability across ViG variants.\nMinor Presentation Issue: The abstract could better highlight the low-rank insight as a core novelty, as it currently focuses more on the framework.", "other_comments_or_suggestions": "I have no other comments or suggestions.", "questions_for_authors": "**Generalizability to Graph Tasks**: Your hypothesis suggests low-rank patterns exist in chemistry/biology graph data (Section 5.3). Could you provide PCA or similar analysis on these datasets (like Figure 2 for vision) to confirm this? A positive response with evidence would strengthen my confidence in VGP’s broader applicability, potentially raising my rating from \"accept\" to \"strong accept.\"\n\n**Choice of Rank $r=32$**: Table 4 shows peak performance at $r=32$, but Appendix A.3 estimates $r=50$ (CUB) and $r=60$ (Flowers). Why was $r=32$ chosen over these values? Clarification could resolve this apparent discrepancy, impacting my view on the method’s optimization rigor.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received four positive ratings, with all reviewers generally inclined to accept it. The paper presents Vision Graph Prompting (VGP), a novel parameter-efficient fine-tuning method tailored for Vision Graph Neural Networks (ViG). Different from the prompting methods designed for transforms, the author specifically considers the rich topological relationships between nodes and edges in the visual graph structure, which improves the modeling ability of complex semantics. According to the reviews, this paper is well written, with a well-explained methodology, clear motivation and insightful visualizations that aid understanding. All reviewers recognized the novelty of the proposed method, which innovatively explores efficient parameter fine-tuning for the ViG model. In addition, the authors conduct extensive experiments across different datasets and with different backbone networks to effectively demonstrate the generality and scalability of the proposed method. 
The authors have addressed the concerns raised, resolving most of the doubts. Therefore, the Area Chair (AC) recommends accepting the paper.
Decision: Accept (poster)