APT: Towards Universal Scene Graph Generation via Plug-in Adaptive Prompt Tuning
Authors: Ruikun Luo
Changwei Gu
Jing Yang
Yuan Gao
Jieming Yang
Song Wu
Hai Jin
Xiaoyu Xia
Venue: ICLR 2026 Poster (2026)
Keywords: Prompt Tuning
Scene Graph Generation
Open Vocabulary
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Scene Graph Generation (SGG) is pivotal for structured visual understanding, yet it remains hindered by a fundamental limitation: the reliance on fixed, frozen semantic representations from pre-trained language models. These semantic priors, while beneficial in other domains, are inherently misaligned with the dynamic, context-sensitive nature of visual relationships, leading to biased and suboptimal performance. In this paper, we transcend the traditional one-stage v.s. two-stage architectural debate and identify this representational bottleneck as the core issue. We introduce Adaptive Prompt Tuning (APT), a universal paradigm that converts frozen semantic features into dynamic, context-aware representations through lightweight, learnable prompts. APT acts as a plug-in module that can be seamlessly integrated into existing SGG frameworks. Extensive experiments demonstrate that APT achieves +2.7 improvement in mR@100 on PredCls, +3.6 gain in F@100 and up to +6.0 gain in mR@50 in open-vocabulary novel splits. Notably, it achieves this with less than 0.5M additonal parameters (<1.5\% overhead) and reduced 7.8\%-25\% training time, establishing a new state-of-the-art while offering a unified, efficient, and scalable solution for future SGG research. The source code of APT is available at <https://github.com/CGCL-codes/APT>.

Published as a conference paper at ICLR 2026
APT: TOWARDS UNIVERSAL SCENE GRAPH GENERA- TION VIA PLUG-IN ADAPTIVE PROMPT TUNING
Ruikun Luo 1234, 1204 Changwei Gu 1234 Jing Yang?; Yuan Gao*, Jieming Yang', Song Wu 1231 Hai Jin 1231, Xiaoyu Xia
W212
National Engineering Research Center for Big Data Technology and System
SSrvices Computing Technology and System Lab *Cluster and Grid Computing Lab
School of Computer Science and Technology, Huazhong University of Science and Technology Zhengzhou University
'Royal Melbourne Institute of Technology
(rkluo, gumorming, wusong, hjin/ @huss. edu, cn, syangjingen ngghst123, yuangaohne, yjmlai i 1e @ggmall com, x xiaoyu. x xia xia@rmit. edu.au
ABSTRACT
Scene Graph Generation (SGG) is pivotal for structured visual understanding, yet it remains hindered by a fundamental limitation: the reliance on fixed, frozen se- mantic representations from pre-trained language models. These semantic priors, while beneficial in other domains, are inherently misaligned with the dynamic. context-sensitive nature of visual relationships. leading biased and suboptimal performance. In this paper, we transcend the traditional one-stage v.s. two-stage architectural debate and identify this representational bottleneck as the core issue. We introduce Adaptive Prompt Tuning (APT), universal paradigm that converts frozen semantic features into dynamic, context-aware representations through lightweight, learnable prompts. APT acts plug-in module that can be seam- lessly integrated into existing SGG frameworks. Extensive experiments demon- strate that APT achieves +2.7 improvement in mR @ 100 on PredCls, +3.6 gain in @@ 100 and up to +6.0 gain mR @550 in open-vocabulary novel splits. Notably, it achieves this with less than 0.5M additonal parameters (<1.5% overhead) and reduced 7.8 7.8%-25% training time, establishing new state-of-the-at while offer- ing unified, efficient, and scalable solution for future SGG research. The source code of APT is available at wtlhttp::/// /thttss:/gttthbb wtwp:ttw.c.ococccccccooocccce.tt www.ccCLL codes/AR /PPT
INTRODUCTION
Scene Graph Generation (SGG) stands foun- dational pillar in visual understanding, aiming to represent images graphs of objects and their interrelationships a structural manner. For years, the field has been shaped by two com- peting paradigms: two-stage methods, which exploit robust detector features but suffer from contextual fragmentation, and one-stage meth- ods, which enable end-to-end learning the ex- pense of computational cost and relation gran- ularity. Despite their differences, both meth- ods share a common practice: incorporating seteentttioos-typiccaa
50
SGDet R.50
30
20
Semantic Festure
Gotle
DEIT
Clp-tor:
Leamable hemppt (Durg)
SGTR
EGTR
STSGG
LiMG
Spead
RA-SGG
DRM
PENET
SDsGG
more OVSOTA
RAHP
static, fixed semantic derived from pre-trained language models like GloVe (Pennington et al.) 2014) and BERT (De-
Performance comparison across dif- ferent semantic features settings (including ours)
Figure 1:
based various SGG methods.
vlin et al. 2019)) as semantic priors.
While such features have proven useful in NLP Penninnnnnto et al. (2014); Devlin et al. (019)): Brown et al. (2020) and certain vision-language tasks Lu et al. (2019); Zhou al. (2022b a). their adaptive nature fundamentally limits their effectiveness in SGG, where context sensitivity, relational
Corresponning author.

---

Published as a conference paper at ICLR 2026
nuance, and role-specific semantics are paramount. As illustrated in Figure 1 we systematically compare the performance of various SGG methods under different semantic feature settings. The results reveal a consistent performance gap when models are constrained to use frozen embeddings, underscoring their suboptimal adaptability to visual relational reasoning.
The fundamental limitation lies in the rigidity of these off-the-shelff representations. Whether in one-stage transformer or a two-stage detector, frozen word embeddings remain oblivious to visual context, incapable of distinguishing between fine-grained relations (e.g.. "standing on" v.s. "walking on"), and fail to capture the semantic asymmetry between subjects and objects. For instance, the same "person" " embedding remains identically frozen whether the person is riding a horse or holding phone, which highlights a clear misalignment with the dynamic visual world. More revealingly, we visualize the feature space, as shown in Figure The static semantic space collapses all 'person' instances into single point, regardless of their diverse contexts. In contrast, the visual feature space naturally separates 'person' into clusters based on their relational context (e.g., riding, walking, etc.). This stark contrast visually demonstrates the inability of frozen representations to adapt to visual contexts and the limitation of static word embeddings in capturing relational context.
two- dimensional t-SNE projection of embeddings from four mainstream pre-trained models. It also displays the distance from the central point to the farthest point within each semantic space, alongside the cu- mulative distribution function of pairwise distances. A progressive loosening of the feature cluster is observed when moving from GloVe to BERT and further to CLIIP-tett. Moreover, in Table I we report the silhouette score, participation ratio, the number of principal components required to explain 90% of the variance (PCA @ 90), and a mutual information proxy I(embedding: predicate). This indicates that more powerful pre- trained models encode richer substructures within their representation spaces. However, this internal structure remains misaligned with the fine-grained visual-relational context required by the SGG task. This suggests that simply replacing one frozen model with another does not address the core issue of semantic rigidity. While one-stage models attempt unify detection and relation modeling, their heavy pre-trained backbones and dense attention mechanisms incur prohibitive training costs without fundamentally solving the semantic adaptivity problem.
Figure 3 illustrates
leekine
marling ea maliiine helldine
(a) Frozen GloVe Feature Space (b) Visual Context Feature Space
Figure 2: t- t-S visualization of person instances with visual con- textual features and GloVe embeddings across different relations.
In light of these observa- tions, what the community has overlooked is not the architecture, but the repre- sentation paradigm: the need for a lightweight, uni- versally applicable mecha- nism that injects adaptive semantics into any SGG framework. To this end, transcend the one- two-stage dichotomy and introduce a new unified representation paradigm for SGG: adaptive prompt tuning. Rather than engineering another architecture, we propose a plug-in module that enables any existing SGG one-stage and two-stage, which are transductive, or the inductive open vocabulary setting-to to dynamically modulate pre-trained semantic features in response to visual context and relational roles. Our key idea is both simple and powerful: a set of lightweight, learnable prompts that act as conditional adapters, transforming frozen language model features into context-aware representations without backpropagation through the original pre-trained backbone.
Table I: Diagnostics across pre-trained models.
Embedding
Silbouette
Participation Ratio
PCA@90
l(embedding; predicate) (bits)
GloVe BERT
0.12
9.8
27
0.12
0.18
15.6
32
0.49
GPT
0.22
23. 1
41
0.53
CLIP-texr
0.29
48.7
125
0.57
Extensive experiments validate the generality and effectiveness of our approach. When plugged into leading one- and two-stage models, our prompt module delivers consistent and significant improve- ments, establishing new state-of-the-art results across multiple benchmarks. Importantly, it achieves these improvements with reduced training time and minimal parameter overhead, making it highly
2

---

Published as a conference paper at ICLR 2026
00.5
GoVe
1.65
2.58
3.54
5.28
GPT
CLiP-toxt
0.0
Pairwise distance
(a) GloVe
(b) BERT
(c) GPT
(d) CLIP-text
(c) Pairwise distance
Figure 3: Two-dimensional feature distribution of four mainstream text feature models under t-SNE dimensionality reduction. The number in (a), (b), (c), and (d) represent the distance between the center point and the farthest point, while (e) shows the cumulative distribution function (CDF) of pairwise distance.
practical for compute-efficient research and applications. The main contributions of this paper are summarized as follows:
We identify and diagnose the representational limitation of frozen, fixed semantic representations as a fundamental bottleneck in SGG, transcending architectural categories.
We propose lightweight, universal prompt-based representation paradigm that converts frozen semantic representations into dynamic, context-aware features, compatible with both one- and two-stage SGG frameworks.
Extensive experiments are conducted on Visual Genome, Open Image V6, and GQA, demon- strating that APT improves mean recall (mR @ 100) by up +2.7 PredCIs and boosts harmonic mean (F@ 100) by up to +3.6, while introducing less than 0.5M additional parameters (<1.5% overhead) and reducing training time by 7.8%.25% In open vocabulary settings, APT achieves up to +6.0 gain on mR @ 50 in novel split.
RELATED WORK
Building on our diagnosis of frozen semantic representations as fundamental bottleneck, we now review how this issue manifests across different SGG paradigms.
Two-Stage SGG. Two-stage methods first detect objects and then predict relations between pro- posed regions. Pioneering works like MOTIFS Zellers et al.) (2018) built upon Faster R-CNN de- tectors, leveraging visual features and spatial masks to predict predicates. Subsequent efforts Tang et al. (019) incorporated linguistic priors from pre-trained models (e.g.. GloVe (Pennington et al.) 2014 to enrich object representations, while others Yang et al. (018); Li et al. (2021) employed GNN propagate contextual information between objects. Recent methods such as PE-Net (Zheng et al. 2023), DRM (Li et al., 2024a), and RA-SGG (Yoon et al., 2025) further explore advanced architectures to enhance relational reasoning. A key limitation is their reliance on frozen seman- tic representations. Whether used in label initialization or feature fusion, these fixed embeddings remain insensitive to visual context and relational nuance. Despite strong detectors, their represen- tational capacity remains bottlenecked by non non-adaptive semantic priors.
One-Stage SGG. In contrast, one-stage methods aim unify detection and relation prediction within an end-to-end framework. Models such as Qpic (Tamura et al. 2021), SGTR (Li et al. 2022). EGTR (Im al. 2024), ST-SGG (Kim et al. 2024b). LLM4SGG (Kim et al. 2024c). SpeaQ (Kim et al. 2024a) and HydraSGG Chen et al. (2025) use transformer-bag based architectures to directly predict relation triples from image features. These methods avoid error propagation and simplify training pipelines by jointly optimizing all components. However, these methods often inherit and sometimes exacerbate the problem of semantic rigidity. Many still initialize query embeddings semantic banks using static word vectors, which cannot adapt to visual context. Moreover, their heavy reliance on self-attention over high-resolution feature maps leads to substantial computational overhead, limiting their practicality for large-scale of architectural unity has thus come at the cost both representational flexibility and efficiency.
cesource-constrainge applications. The pursuit
Open Vocabulary SGG. Recent interest in open vocabulary SGG seeks to generalize to unseen objects and predicates. OV-SGG He et al. (2022) proposed a two-stage, prompt-based method to bridge the knowledge gap between base and novel object categories, but its prompt-template-basee
3

---

Published as a conference paper at ICLR 2026
fine-tuning makes it difficult to extend to other paradigms. Methods like Epic Yu al. 2023) PGSG et al. (2024b), SDSGG (Chen et al. 2024a). OvSGTR (Chen et al. 2024b). Space et al. (2025), and RAHP (Liu et al. (222) leverage large pre-trained vision Aaston-language models (c.g., CLIP (Radford et al. ) for zero-shot alignment, while others employ prohabilistic grounding or knowledge distillation. Yet, these methods still largely depend on frozen backbones and semantic spaces. While powerful, CLIP-based features remain generic and are not explicitly tailored to the structured and context-dependent nature of relational prediction. As result, they often struggle with fine-graine grained relational reasoning and exhibit limited adaptability to downstream SGG contexts.
Our work does not propose another architecture in the one- v.s. two-stage divide, nor does it simply replace one frozen model with another. Instead, we introduce a universal plug-in module based on lightweight prompt tuning that can be seamlessly integrated into any SGG framework- whether one-stage, two-stage, or open-vocabulary.
3 ADAPTIVE PROMPT TUNING FRAMEWROK
Fremo
Object
Object Ostactor
APT
Teat
Olips
Wrtde Holt
bastit
Fusion
Iroge
Triee Cunfle
Ead Dre
How
Calculation
Triaer
Delecee
Eiage
Fautures
(x) Two-Stage Mathodk
[b) One- Methods
[c] DV Mathods
(1) APP-eebancee Methods
Figure 4: A comparative illustration of different SGG paradigms versus APT framework. (a) Two- stage methods suffer from fragmented context propagation and reliance on static features. (b) One- stage methods achieve end-to-end learning but at a high computational cost and with semantic rigid- ity. (c) Open- Vocabulary methods leverage large VL models but struggle with fine- -raiind relational reasoning due to generic representations. (d) Our APT framework introduces universal plug-in module that injects adaptive semantics into any SGG backbone, enabling dynamic, context-aware feature modulation for superior performance and efficiency.
3.1 OVERVIEW
The Adaptive Prompt Tuning (APT) framework centers on lightweight, learnable prompts that adapt frozen pre-trained semantic representations into context-aware, task-specific features. As illustrated in Figure 4 APT is designed as a universal plugin that can be seamlessly integrated into both two-stage and one-stage SGG paradigms. The prompts act as conditional adapters, trans- forming frozen word embeddings into dynamic representations that are sensitive to visual context, fine-grained relationships, and the semantic roles of subjects and objects.
Adaptive Prompt Tuning (APT) framework is grounded in the advanced research of continuous prompt learning and model adaptation. Unlike traditional methods that discretely modify input tokens, continuous prompts introduce a set of learnable vector parameters into the model's 's em- bedding space, acting as specific instructions Formally, pre-trained model can be viewed as a function Fo whose parameters 0 are fixed after pre-training. Prompt learning aims to find an optimal prompt P* that enables the frozen model F, to perform best specific downstream task T:
guide the model's behavior Lester et al. (202
p argmin Cr (r(FF((P:))) P
(1)
APT innovatively adapts this paradigm to the multimodal context and structured prediction task of SGG. The prompt P functions not direct prefix language model, but as a feature modu-
4

---

Published as a conference paper at ICLR 2026
lator. It transforms generic, task-agnostic static semantic features Estatic into dynamic features à tailored for the downstream SGG task. This process is analogous to the role of a modem in com- munications: the prompt P. carrying task-specific information from the visual context, "modulates" the original semantic signal, enabling it to convey information more effectively for SGG. From the viewpoint of the Information Bottleneck principle Chi et al. (2022); Yang al. (2023); [ al.. (2000). APT aims to learn an optimal feature representation é that preserves sufficient information about the object identity e while maximally compressing redundant semantic information irrelevant to the current visual relation, and simultaneously incorporating relevant visual contextual informa- tion v. The prompt serves as the adapter achieving this "compression and injection." " 'The objective can be formulated as:
max ((e (ē; estatic U.
In Eq. 2 ((::)) denotes mutual information, and y is the target relation class. The first term requires é to be informative for prediction, while the second term encourages to forget redundant information in Cstatic given the visual context and the target y. The learnable prompt P is optimized through training data to balance this trade-off.
3.2 UNIFIED PLUG-IN PROMPTS
APT operates simple yet powerful principle: for a given semantic concept, it employs lightweight, learnable prompt P to condition the frozen, static embedding (static()) on the current visual context. This process can be universally described by the following equation:
ē(c) = fo (A((((()))) (static(c),
(3)
In Eq. 3 P(c) is the learnable prompt for concept c. A(-) is the aggregation function that reduces the prompt sequence to a single vector. estatic (c) is the frozen pre-trained semantic embedding. Φ(v) is the visual feature projector, encoding relevant visual context v. fe is a small fusion network that generates the final adaptive representation è(c)
The key is that only the prompt parameters P. the projector Φ. and the fusion network fe are learn- able. The pre-trained semantic backbone remains entirely frozen, making APT highly parameter- efficient and preventing catastrophic forgetting.
Detection Prompt Pa: This prompt is applied during the object detection phase. For each object class c, learnable vector Pa(c) E RLaxD is defined, where La is the prompt length and D the feature dimension. The prompt is fused with the pre-trained semantic embedding estatic(c) E R through a dedicated Multi-Layer Perceptron (fe...) to generate an adaptive object representation for the detection head.
Relation Prompt P.:: After objects are detected, relation prediction begins. Here, for each predicate class learnable vector P, (r( E IR LDD is defined. This prompt is specifically designed to capture the nuances of interactions. For a subject-object pair (s, o) with predicted visual features v. and Vo. their adaptive semantie features are generated and fused with visual evidence.
Unified Relation Prompt Par: Since there is no separate detection stage for one-stage paradigm, a single Relation Prompt Pur sufficient and more efficient. This prompt operates the semantic queries or label embeddings that the model uses for final predicate classification. For a potential relation with subject class and object class o, the model dynamically modulates their semantic embeddings. The adapted embeddings are then used by the transformer decoder for cross-altention with visual features.
In both one- and two-stage cases, the pre-trained semantic embeddings (statie()) remain frozen. Only the prompt parameters Pa. P. and the parameters of the light- MLPs (fo) are learned during training. This makes our APT framework highly parameter-efficient and prevents overfitting. The same pre-trained language model can thus be shared across different SGG architectures, with the prompts specializing its knowledge for the task at hand.
3.3
COMPOSITIONAL GENERALIZATION PROMPTER
The VecnnVocabulary (OV) setting demands that the model generalize to unseen object and pred- icate compositions unseen during training. To equip APTframework with this capability, in- troduce a dedicated Compusitional Generalization Prompter (CGP). This module is architected to dynamically synthesize context-aware semantic representations for unseen categories through
5

---

Published as a conference paper at ICLR 2026
structured, multi-stage prompting process. The CGP operates through three specialized sub- modules, which work in concert to achieve robust generalization:
Relational Context Gating (RCG): This component generates role-aware prompt weights by inte- grating visual evidence with initial semantic cues. For subject entity s with visual feature Vg:
W, a (MLP gate (Concat(V;; vs Cstatic (s)))):
The gating vector w, determines the activation of prompt bases, ensuring the modulation is condi- tioned on the immediate visual context of each entity.
Basis Prompt Synthesis (BPS): A set learnable basis prompts XN serves as repository of fundamental relational concepts. The final prompt for entity is synthesized weighted combination of these bases:
N Σ
P (s)
W B',
P (s) (egnP
To obtain a compact pooled prompt we use a normalized, token-weighted pooling with normaliza- tion (I& the basis prompt length):
La P (s) E ERD
p = LayerNorm
(6)
Tb
This allows the model to generate a virtually unlimited variety of tailored prompts from finite set of bases, enabling compositional generalization.
Feature Refinement & Fusion (FRF): This sub-module performs the final integration of the syn- thesized prompt, the frozen semantic embedding, and the projected visual feature:
Cov 8) (Concat (s), Cstatic (s), de (v=)))
The refined feature (s) context-sensitive, semantically grounded, and primed for relational reasoning with unseen concepts.
The CGP module is designed as plug-in component that can seamlessly augment the standard Relation Prompt (P,) in both two-stage and one-stage architectures. All pre-trained embeddings remain frozen. Only the basis prompts B, the gating network, visual projectors Φv. and the fusion learnable parameters, upholding the parameter-efficient nature of
MLPs fo., are introduced the APT framework.
By integrating Relational Context Gating, Basis Prompt Synthesis, and Feature Refinement & Fusion, our CGP module provides a principled and unified solution for open-vocabulary general- ization, ensuring robust performance on both common and unseen compositional queries.
The overall training objective augments the loss L with several prompt and gating regularizers. Formally, the empirical objective expressed as expectation over the data distribution D-is
L E( Las (2,,))))
Xp |
1 + Apal Pactl1? + Apr Pre
(9)
Xa R.DDDD [l|ē
@static 1|]]
(10)
Aorth
|.B.B.B.
(11)
wi log wi
(12)
7 KLL w Uprior)
Am W W.
(13)
where A(-)- 8,,%, 7, and Aorth negative hyperparameters; Upric denotes optional prior (e.g., uniform) over the gating distribution. The KL (entropy) terms serve to encourage sparsity and diversity and to penalize deviations from the prior.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENT
4.1 EXPERIMENT SETTINGS
Datasets. Our experiments are carried out on three publicly available benchmarks: (1) Visual Genome (VG) Krishna et al. (2017) comprises 150 object categories and 50 types of relations. The dataset is partitioned into 57.723 images for training. 5,000 for validation, and 26,446 for testing. (2) Open Images V6 Kuznetsoo et al. (2020) includes 288 entity classes and 30 relation categories. It provides 126,368 training images, 1,8T3 validation images, and 5,322 test images annotated with relational triples. (3) GQA Hudson & Manning (011)) contains 200 distinct entity types and 100 kinds of relations. It offers 52,623 training samples, 5,000 validation images, and 8,209 test images with scene graph annotations. We report only the results on VG due to the page limit.
Evaluation Protocol. The evaluation is con- ducted on three conventional SGG sub-tasks, in- cluding Predicate Classification (PredCls)) Scene Graph Classification (SGCIs), and Scene Graph Detection (SGDet). PredCIs predicts the predi- cate classes given all ground-truth object bound- ing boxes and the object classes. SGCIs aims at predicting the predicate classes given the ground- truth object bounding boxes. SGDet detects all entities and their pairwise predicates given an im- age.
025
RGT
Weight
0.21
0.10
0.2
3.55
alere Rehial to
f.forr atteched
everioypigg
dround
Evaluation Metrics. We evaluate SGG mod- els on the three metrics: (1) Recall@K (R@K) calculates the proportion of top-K predicted triplets that are in ground truth. (2) Mean Re- call@K (mR @K) K) calculates the average recall for each predicate class, which is designed to mea- sure the performance of SGG models under the long-tailed predicate distribution Liu et al. (2019). (3) F@K K calculates the harmonic average of @ K and mR @ K. ST-SGG (Kim et al. (022b) suggests that there is trade-off between R@ K and mR@K. Thus, recent works have focused achieving greater F@K.
Figure 5: AP, net performance comparison per class. The Weight (i.e., right y-axis) represents the frequency ratio in the test data.
Baselines. APT is compared with methods from three categories: (1) Two-stage SGG methods,
including MOTIFS (Zellers et al. 2018), PE-Net (Zheng et al.) (023). DRM (Li et al., 2024 and
RA-SGG (Yoon et al. (2025); (2) One-stage methods including SGTR Li et al. 2022). EGTR (I
al. 2024),ST-SO STGSSGGGII (Kim et al. 2024b), LLM4SGG (Kim et al. 2024c). SpeaQ (Kim et al. 2024a)
and HQSG (Fu et al.
2025); ; (3) Open Vocabulary methods including SDSGG (Chen et al. 2024a),
OvSGTR (Chen al.
2024b) and RAHP (Liu et al. 2025).
Table 2:
Performance (%) of state-of-the-art SGG models with & without APT on Visual Genome Krishna et al. (2017) denotes the results are produced using official code.
Methods
Res00VD0
Predicate Classification
mRoe SCV100
F0:50/100
Scene Graph Classification
miv1000000
F0:50/100
R
Scene Graph Detection
miR 505/100
Fassy100
Ta s00ee mihhot
Modif MwiifZzllles.oo
64.6666.
15.2/16.2
24.6/76.0
8.799.3
14:2/15.0
31.0/35
6.77777
11A/12
Mouif
17.4/88.1
26.478.11
40.3440.8
16.4/7..3
33.3/7.6
9.110.33
13.2/15.1
PC-Nel Zheng et
2023 CoNEEs
65.8167.6
17,71922
27.9/29.9
36.7/37.4
9,6/100
15.0/5.88
27.1/29.8
6,4733
[0.4/11.7
PE: Ne-Nete
67.5189.2
19.3/20.5
29.7/3166
37,2/38.
10.4/10.7
16.2/17.1
28.033.3
7.18.6
11.3/25
DRM [ier 20240
65.8867.6
17.7219.2
27.9/29.9
36.7/37
9,4000
1500158
27.1/29
64773
104/112
DRMFAPT
68.3/70.5
19,4/1..
28.900.5
39.440.1
1L3111
17.2/18.6
29.6/1.7
8.29.7
12.5/13.3
RA-SOG Yeor et
1025
MANTIS
66.188.0
18.2/198
28.4/30.5
9.800666
154/161
27,5/30
68774
1066/222
RA SOGFAFT
66.169.4
18.5/20.2
29.03002
37.5138.1
10.1/11.3
16.1/17.4
28.8/29.8
8.289
11.31255
Ohe-sture millont
SGIR et 2022
59.261.3
30.4/32.5
40.2/42.8
47385355
14,3/66.
20.7/23.1
31.0035
10.7/12.6
15.9186
SCIR
62.363355
32.7/35.3
43.5/45.9
39,840.3
17.1/18.7
22.423.4
33.5/36.8
12.9/14.8
18.4/20.3
ECTR Im etal.
2024
$4,1/56.5
35 35.7/38.2
43.045.6
34 90361
17,008.4
22.9/24.4
27,431.8
13.2/155
17.8208
EGTR-APPT
50.458.55
37.5440.1
45 2542/47.7
36.7/38.6
19.5/203
24.4266.2
29.7/33.5
15.7/169
19.4/2233
13 MASSC Kim of
2024c
62 2364.1
36 2292
45.7/48.6
38.2290.
20.9/22.5
27.0038.6
36.0300
14.4/17
18.5/19
LLMSSSC-7X7
6/666
38.14422
47 47.9550.3
10.1ML6
22.7/24.8
29.5/30.3
28.832.4
16.7/193
20.3/23.6
ST-50G
28
33.4/34.
16.9018.0
22.4/23.8
26.7/30.7
11.6/142
16.2/194
ST-800-XPT
31.334.6
30.9/43.7
36.648.44
20.321.5
36.3/369
30.2/34.4
148/18
19.3/222
SpesQ Kimetal
20244
55.7557.
3019/33.4
59.7/42.4
33. 13.1/44.4
17.5/18.8
22.9/34.3
24.5/29
14.1/165
17.921.
SpenQ-xr1
57.860.8
34.2/36.8
42.5/45.3
36.53776
20.321.7
26.3/27.5
27,5/31
18.0/19.5
20.2/33.3
HQSG 2025
o0w3s
41.5143.2
25.4/27.2
21.8/26.7
HQSG-XP1
58.7151.2
35.1/37.3
43.3/45.4
37.7/58.3
21.9/22.7
26.9/38.7
36.5/90.9
18.2/1177
22.6/28.2

---

Published as a conference paper at ICLR 2026
4.2 COMPARISON WITH BASELINES ON VISUAL GENOME
As comprehensively detailed in Table 2 we evaluate the effectiveness of APT by integrating into diverse set of state-of-the-art SGG models, encompassing both two-stage and one-stage paradigms. The integration of APT consistently enhances the performance of all base models across the three canonical SGG tasks: PredCIs, SGCIs, and SGDet. This universal applicability solidifies our method's s role as a powerful and veneral-purpose plugin for the SGG community.
The most notable improvements are observed on the mean recall (mR @K) metric, which a more robust measure of a model's ability to predict a balanced set of predicates beyond the head classes. For instance, APT elevates the mR @ 100 of EGTR [mm et al. (2024) by +1.9 and +1.9 on PredCIs and SGCIs, respectively. The striking improvement confirms that our adaptive prompts effectively mitigate the inherent bias of static features towards frequent predicates, enabling the models to perform more fairly and accurately on tail categories. This implication can be further explored by referring to Figure 5] which displays the performance APret for each class. Specifically, for the head predicates. EGTR+APT achieves competitive results. For the tail predicates, AGT significantly enhances the performance, particularly in the cases where EGTR struggles to make accurate predictions, such attached to, overlapping predicates. Furthermore, the superior F@ K scores demonstrate that APT does not achieve gains in mR @ K at the expense of R R@ but instead fosters comprehensive and balanced relational understanding.
APT yields substantial gains on both two-stage and one-stage methods. This validates our core hypothesis that the limitation of frozen semantic priors fundamental bottleneck transcending architectural choices. Our method successfully alleviates this bottleneck, empowering diverse archi- tectures with adaptive semantic representations.
Table 3: Performance (%) of sfate-of-the-art Open Vocabulary SGG models with & without APT on Visual Genome Krishna et al. (2017). i denotes that the results are produced using official code.
Novel
Methods
RG/0050/100
20055100000000
[ 20/00100000
R@20/50/100
20/00101100
SDSGG
12.3/16.9/20.2
18.4/25
1/,/2222/31122
17.3/25.3/0.44
SDSGG=APT
19.5/27.3/32.2
10.1/13.2/156
13.4/18.02199
19.4/266/311.1
18.6/26.7/32.3
19.1/27.1/32.3
OvSGTR Chen al. 2024b
19.0/22.9/26.7
12.6/6.4419.7
15.7/19.1/22.7
17.0/20.//33.9
10.9/13.5/16.2
13.4/16.3/19.
OvSGTR TASGTR-PPT
20.0/24.1/27.9
13.4/17.3/20.1
16.8/20.1/23.4
17/2.1.
11./14.3/11.22
14.1/17.1/20.4
SGTR+RAHP Liu atal
2025 AAAUZ
34.6/41.3/4777
16:4/20.5/5 2
22.0/27.4/33 4/33.0
12/4/1..//0..4
9.1/118811.5
10.5/13.4/1.6
SGTR+RAHP4API
33,4/42,0148.4
17.0/1.1126.0
22.7/28.1/38 1/33.8
13.1/16.1/11.J
9.3/12.4/133
11.2/14.018.4
4.3 COMPARISON WITH OPEN VOCABULARY SGG MODELS ON VISUAL GENOME
To evaluate the model's capability to generalize unseen relationships, we follow the common practice CaeiiceeCheneeeee et al. (2024a b) and partition the VG dataset into Base and Novel splits. The Base split contains 70% of the relation categories for training, while the Novel split comprises the remaining 30% of categories that are held out from training. This setting tests the model's true compositional reasoning ability. The results are presented in Table 3
APT improves performance on the Novel split across all base models, which is the core challenge of OV-SGG. This demonstrates that our adaptive prompting mechanism and CGP module effectively unlock the compositional knowledge embedded in pre-trained models, enabling them to generalize to unseen predicate combinations. APT compelling across different OV-SGG architectures, from transformer-based one-stage models to methods incorporating external knowledge, substantiating that the proposed adaptive prompting paradigm addresses a fundamental bottleneck in OV-SGG- the inability of frozen representations to dynamically adapt unseen compositional contexts.
Table 4: Ablation study of APT on VG. i denotes the results are produced using official code.
Predicate Classification
Scene Graph Classification
Model
5@50/000
mR@50/100
F@ 50/100
R@500100
m0/500000
F@ 50/100
Vanilla PE-Net"
64.9/67.2
31.5/33.8
42.4/45.0
37.7/38.7
17.8/18.9
24.5/25.8
+D-Prompt only
65.2/67.1
30.4/32.6
41.0/43.8
38.5/39.4
16.6/17.9
24.0/25.3
+R-Prompt only
64.6/66.7
33.4/36.4
43. 43.6/46.0
38.6/39.5
19.6/20.8
1/.1/26.3
+Full APT
62.2/64.1
36.2/39.1
45.7/48.6
38.2/39.1
20.9/22.5
27.0/28.6
8

---

Published as a conference paper at ICLR 2026
4.4 ABLATION STUDY ON COMPONENTS OF APT
The ablation experiments are conducted based on the two-stage method PE-Net (Z et al. (2023)
framework and open-vocabulary method SDSGG Chen al. (2024a) on VG.
As demonstrated in Table introducing only the Detection Prompt (D-Prompt) improves the model's 's object classification accuracy (a slight increase in R @@K), as it helps generate better context- aware object representations. However, its impact on relational reasoning is limited. Conversely, adding only the Relation Prompt (R-Prompt) yields significant boost in mR @ K, it directly addresses the core problem of predicate discrimination by dynamically modulating features based on relational context. This confirms that the relational prompt is the key to mitigating predicate bias. Though both prompts are beneficial, the R-Prompt is particularly critical for relational reasoning. The complete APT, integrating both D-Prompt and R-Prompt with the MLP fusion, achieves the best performance across all metries, especially mR @ K. The synergistic effect between the two prompts is evident, as they work in concert to provide adaptive semantics from object detection to relation prediction.
In addition, we perform a detailed ablation study on the SDSGG base model to dissect the contribution of each proposed component within our Composi- tional Generalization
Table 5: Ablation study of APT CGP module based on SDSGG Chen al. 2024a) on VG split. t denotes the results are produced using official code.
Model
Bane
Novel
RO2055100
RO2VV51000
Vanills smSnoy
21.1265/221
102/124/188
144/169/17
21,12554/260
344/352/357
357/353/755
-ROG IBPS
22.4/2../222
122/181/136
158/176/184
234/256/263
255/262//28
1/26/
2366/200/366
125/136/140
161/18.1183
23.6/25//26
260/275/278
248/267/22.2
+ROG only
22.8/209/285
185/165/147
170/198/194
258/258/264
273/20//222
254/224//25
-Fill OGP
2333/2/2/36
149/199/85
18.2 26.//2.1/202
46.1/663/275
2887312//477
26.3/366/266
266
Prompter (CGP): the Relational Context Gating (RCG), the Basis Prompt Synthesis (BPS), and the Feature Refinement & Fusion (FRF) modules. Experiments are conducted on the Open- Open-Vocabulary VG split, and results are presented in Table 5
The baseline model achieves mod- est performance, particularly strug- gling on the Novel split ex- pected. The significant gap be- tween Base and Novel mR @K highlights the challenge of gener- alizing to unseen predicate cate- gories with static representations. Introducing only the RCG module brings a noticeable gain, especially on the Novel split, which demonstrates that conditioning the model on visual context is a crucial first step, allowing it to dynamically reweight its features based on the input image, which is vital for generalizing to unseen compositions. Further incorporating the BPS module yields a substantial performance jump, gaining the Novel mR @ 50 an increase of +3.8 over the baseline. The model's generalization ability is greatly enhanced by its capacity to synthesize new prompts from a basis set, effectively generating tailored representations for unseen concepts. Full CGP achieves the best results across all metrics. The FRF module provides a critical non-linear transformation, effectively fusing the synthesized prompts, original semantics, and visual features into an adaptive representa- tion. This results in the highest harmonic mean, indicating a balanced and robust improvement on both Base and Novel splits.
Table 6: Efficiency analysis APT. Performance reported as mR @ 100 on PredCls.
Model EGTR
Orig.
Farameters (M)
+APT
Orig.
Time / Epoch (h)
+APT
A
Performance imk 100)
+APT
42.7
43.1
5.1
4.7
-7.8%
30.9
STSGG
43.5
42.3
-1.2
533
4.7
-11.3%
25.7
28.2
LLMASGG
45.8
43.7
-2.1
5.6
42
-25.0%
22.3
23,8
Speal)
42.1
41.6
-0.5
4,9
4.1
-16.3%
25.7
29.1
RA-SOG
18.7
2.1
-1.5%
DRM
47.1
47.5
+0.4
2.3
2.0
-13.0%
19.0
22.2
PENET
46.8
47.2
+0.4
2.0
1.8
-10.0%
26.5
28.3
4.5 QUANTITATIVE EFFICIENCY ANALYSIS
Beyond performance gains, we quantitatively evaluate the parameter and time efficiency of APT across various SGG models. The results are summarized in T
APT introduces a negligible number of additional parameters, consistently less than 0.5M across all models. This represents an increase of less than 1.5% even for larger models like LLM4SGG Kim
et al. 2024c). This minimal overhead confirms that our prompt- based paradigm is a highly rerrameter-fffiiiee strategy for enhancing model capability, avoiding the need for costly pre-trained backbone fine-tuning. In addition, APT not only improves performance but also significantly re- duces training time per epoch for nearly all models. This efficiency gain is particularly pronounced

---

Published as a conference paper at ICLR 2026
for one-stage methods. We attribute this acceleration to the role of adaptive prompts. By providing well-modulated, context-aware semantic features, the prompts appear to stabilize and accelerate the convergence of the downstream relation prediction. The model requires fewer training iterations to fit the data, as the adaptive representations are more informative and easier to optimize than fixed ones. When considering the performance improvement per unit of computational cost, APT demon- strates an overwhelmingly favorable trade-off. For instance, LLM4SGG+APT achieves a +1.49 gain in performance with a 25% reduction in training time and 4.6% reduction in parameters. This es- tablishes new Pareto frontier in SGG, where our method delivers higher performance at lower computational cost.
DISCUSSION: WHY PROMPT TUNING WORKS IN SGG
The consistent and significant gains delivered by APT across diverse architectures and tasks prompt a deeper inquiry into its theoretical foundation. We posit that the effectiveness of our method stems from its ability to reconcile two fundamental principles in representation learning: acquiring task- sufficient features while maintaining minimal complexity, as guided by the Information Bottle- neck (IB) principle
Yang Yang al. (2023); Chi et (2022); Tishby et al. (2000).
Pre-trained semantic em- beddings are compressed representations of linguistic knowledge, optimized for wide array of language tasks. However, for the specific task of SGG, they constitute over-complete and noisy representation. The entire spectrum of semantic information for a concept like "person" from biographical to literary associations-is is encoded indistinguishably. Directly using these fixed priors forces the SGG model to contend with this noise, as must learn to ignore irrelevant facets of meaning while preserving those pertinent to visual relationships. This violates the IB principle Kawaguchi al. (2023), which seeks representation Z that is minimal (retaining only information relevant for predicting the predicate Y) and sufficient (preserving all information needed for prediction).
Table 7: APT vs. FROZEN (GloVe) embeddings IB proxy metrics
PCA@90%
PCA@95%
Linear CKA
Discretized MI proxy
APT FROZEN
28
0.877
1.96
26
35
1.49
APT act as a lightweight, learnable information filter that dynamically modulates these frozen representations. The prompts, conditioned implicitly the visual context through training, learn to perform a form of feature selection and re-weighting on the frozen embeddings. They sup- press semantic dimensions that are irrelevant or detrimental to the current relational context (e.g.. suppressing literary aspects of a person when the visual context suggests riding relation) while amplifying discriminative dimensions (e.g., amplifying anthropomorphic features). The MLP then non-linearly transforms this modulated signal into the final adaptive representation.
Therefore, the resulting dynamic features can be viewed as closer approximations of the minimal sufficient statistics for the SGG task. They are more sufficient because they are context-aware and tailored for predicate discrimination. They are more minimal because they are stripped of generic semantic noise that hinders generalization. This principled compression of irrelevant information and enhancement of predictive signals explains APT's efficacy in improving model performance and generalization, transcending mere architectural improvements.
We added experiments to test whether an APT semantic representation can more compact while preserving discriminative power. As shown in Table 7 APT requires fewer principal components to achieve an equivalent level of explained variance, indicating that semantic information more concentrated and that APT representations mutual information proxy yields higher values for APT than for FROZEN embeddings, suggesting superior retention of label-relevant information.
more amenable to compression. Furthermore, the
6 CONCLUSION
In this work, we diagnosed the pervasive but overlooked problem of static semantic representa- tions as a fundamental bottleneck in Scene Graph Generation. We proposed Adaptive Prompt Tuning (APT), a novel and unified paradigm that addresses this issue by dynamically modulat- ing frozen features into context-aware representations through lightweight, learnable prompts. APT is architecture-ee aggnossi,,,,,,, serving as an efficient plug-in that enhances both one-stage and two-stage models across standard, long-tailed, and open-vocabulary settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We sincerely thank the AC and reviewers for their constructive and valuable feedback. This re- search was supported in part by the National Science Foundation of China (NSFC) under grant No. 62302131, 62402188 188, the China Postdoctoral Science Foundation under Grant No. 2025T180424, the Postdoctoral Fellowship Program of CPSF under Grant No. GZC20240542, and the Postdoctoral Project of Hubei Province under Grant No. 2024HBBHCXA026
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp. 1877- 1901, 2020.
Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. In NeurlPS, volume 37, pp. 132238-132266 2024a.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. In ICLR, 2025.
Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Chang Wen Chen. Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention. In ECCV, pp. 124, 2024b.
Hyung-gu gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani. Infogen: Representation learning for human skeleton-bart bandd action recognition. In CVPR, pp. 20186-20196, 2022.
Jacob Devlin, Ming-Wee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186, 2019.
Jiawei Fu, Tiantian Zhang, Kai Chen, and Qi Dou. Hybrid reciprocal transformer with triplet feature alignment for scene graph generation. In CVPR, pp. 8953-8963, 2025.
Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV, pp. 56-73, 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Jinbae Im, Jeong Yeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In CVPR, pp. 24238,2 2024.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and liaoyang Huang. How does information bottleneck help deep learning? In ICMI., pp. 16049-16096. 2023.
Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim, and Hyunwoo J Kim. Groupwise query specialization and quality- multi-assignment for transformer-basee visual relationship detection. In CVPR, 2024a.
Kibum Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Adaptive self-training framework for fine- grained scene graph generation. ICLR, 2024b.
Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language models for weakly supervised scene graph genera- tion. In CVPR, pp. 28306-28316 2024c.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
11

---

Published as a conference paper at ICLR 2026
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJCV, 128(7):1956-198) 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
Jiankai Li, Yunhong Wang, Xiefan Guo, Ruijie Yang, and Weixin Li. Leveraging predicate and triplet learning for scene graph generation. In CVPR, 2024a.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In CVPR, pp. 11109-11119, 2021.
Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with trans- former. CVPR, pp- 19486-19496, 2022.
Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open- vocabulary scene graph generation with vision-language models. In CVPR, pp. 28076-28086, 2024b.
Tao Liu, Rongjie Li, Chongyu Wang, and Xuming He. Relation-aware hierarchical prompt for open-vocabulary scene graph generation. In AAAI, volume 39, pp. 5576-5584, 2025.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in an open world. In CVPR, pp. 2537-2546, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining aask-agnostic visiolin-
guistic representations for iision-and-language tasks. In NeurlPS, volume 32, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word
representation. In EMNLP, pp. 1532-1543, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpie: Query-based pairwise human-object interaction detection with image-wide contextual information. In CVPR, pp. 10410-10419, 2021.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning compose dynamic tree structures for visual contexts. In CVPR, pp. 6619-6628, 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Mingjie Xu, Mengyang Wu, Yuzhi Zhao, Jason Chun Lok Li, and Weifeng Ou. Llava-spacesge: Visual instruct tuning for open-vocabulary scene graph generation with enhanced spatial relations. In WACV, pp. 6362-6372, 2025.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. ECCV, pp. 670-685, 2018
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. In Advances in neural information processing systems, volume 36, pp. 60397-60428, 2023.
Kanghoon Yoon, Kibum Kim, Jinyoung Moon, and Chanyoung Park. Unbiased heterogeneous scene graph generation with relation-awar message passing neural network. In AAAI, volume 37, pp- 3285-3294, 2023.
Kanghoon Yoon, Kibum Kim, Jachyeong Jeon, Yeonjun In, Donghyun Kim, and Chanyoung Park. Ra-sgg: Retrieval-augmented scene graph generation framework via multi-prototype learning. In AAAI, volume 39, pp. 1562-9570, 2025.
12

---

Published as a conference paper at ICLR 2026
Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. prommpe language model for fine fine-graines grained scene graph generation in an open world. ICCV, pp. 21560- 21571, 2023.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pp. 5831-5840, 2018.
Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. CVPR, pp. 22783-22799 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pp. 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. IJC, 130(9))2337-2348, 2022b.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
For a better understanding of the main paper, we provide additional details this supplementary material, which is organized as follows:
A depicts the implementation details.
and $ shows performance results and ablation study of APT on Open Image
V6 Kuznetsova et al. (2020) and GQA Hudson Manning (2019).
gives formal mathematical specification of the Relational Context Gating (RCG),
B Basis Prompt Synthesis (BPS), and Feature Refinement & Fusion (FRF) components.
provides the pseudo code of APT.
A IMPLEMENTATION DETAILS
All experiments are implemented in PyTorch and evaluated on a server equipped with 4 NVIDIA A40 GPUs. APT is applicable both one-stage and two-stage models, therefore select repre- sentative one-and two-stage methods to validate the generality and high adaptability of APT. The length of all prompts (Ld, L,) set 6. The basis prompt set B for the open-vocabulary CGP module consists of N 16 bases. These values were determined via a hyperparameter search on the validation set. For all baseline models, we use their officially released code and rigorously fol- low their recommended training protocols and hyperparameter settings to reproduce the results. To ensure fair and controlled comparison, we strictly isolate the variable of interest: the integration of APT module. All other factors, including data preprocessing, augmentation, random seeds, and evaluation metries, are kept identical between the baseline and our APT-enhanced versions.
B ADDITIONAL EXPERIMENTS ON OPEN IMAGE V6
To further validate the generalization capability of APT across diverse data distributions, we conduct extensive experiments the Open Images (Ol) V6 Kuznetsova et al. (2020) benchmark. Unlike VG, OI-V6 features larger-scale real-world imagery with distinct relationship taxonomy, presenting rigorous testbed for evaluating model robustness.
Following the data processing of previous works (Li et al. 2021 Yoon et al. 2023 Kim et al.
2024b), OI-V6 is split into 126,368 train images, 1,813 validation images, and 6,322 images, and contains 301 object classes, and 31 predicate classes. Similar to VG, OI-V6 is divided into two splits: base and novel, with the same proportion in Section
B. COMPARISON WITH BASELINES ON OPEN IMAGE
Our evaluation OI-V6 demonstrates that APT delivers consistent performance gains across all model architectures and evaluation settings, reinforcing its generalizability beyond dataset-specific characteristics. As summarized in Table the integration of APT improves both conventional recall (R @ K) and, more significantly, mean recall (mR @ K) across two-stage and one-stage paradigms.
The results OI-V6, combined with findings VG, provide compelling evidence that APT offers universal and practical solution for enhancing SGG models in diverse real-world scenarios.
B.2 COMPARISON WITH OPEN VOCABULARY SGG MODELS ON OPEN IMAGE
We further evaluate APT's capability in the more challenging Open Vocabulary setting on Open Images V6, where models are required to generalize to unseen predicate compositions. Following the standard protocol, we partition the relationship categories into Base (70%) and Novel (30%) splits, testing the true compositional reasoning ability beyond mere pattern memorization. As shown in Table APT consistently enhances performance across all OV-SGG methods on both Base and Novel splits, with particularly notable improvements the challenging Novel categories. The cross- architectural effectiveness from transformer based OvSGTR to graph-based DoSGG-conffrms that APT addresses fundamental limitation in OV-SGG: the inability of frozen representations to
14

---

Published as a conference paper at ICLR 2026
Table 8: Performance (%) of state-of-the-art SGG models with & without APT on Open Image V6 Kuznetsvva et al. 20200. F@ K is the harmonic mean of mR 50/100 and R R@50/100. i denotes the results are produced using official code.
Mechods
Re:500000
Fradicate Classlication
mive501000
Fe5000000
R@5w100
Scone Graph Classiication
mRe551100
F.250/1000
R@5w100
Serne Graph Detection mRG50/100
F05VV000
Moun Zellens al.
2018
65.2666.7
15.5/6.6
25.0/6.5
38.339.1
8,999.5
14.5/15.2
31.53356
6,877.9
11.3129
Monitbap
67.06886
17.61885
27.078.7
40.5/41.
10.5/11.3
16.6/7.66
33.7/38.0
9.310.5
13,51155
PE-Net Zheng al 2023 c0
66,166..8
17.9199.5
28.1/30
36.9/37.6
9.8/103
15.2/1600
27,4/00
6.574
108011
Pt: PE-Nethrur
67.96995
19,620.8
29.93119
37,533.77
10.50109
16.3/37.3
28.8/31.
7.3888
11.1299
DRM Lieral 20240009977
6622668
18.1/19.7
28 28.3/30.3
37,037.8
9,7/103
15.3066
27.6/30.3
6,6755
10,7/12.0
DRMFAPT
69.0770.7
19.7222.1
29.3/3100
39,640.3
1141118
17.4/18.6
29.9/32.
8.4/.9
12.8/3377
Yoor
MANTES
28.03000
36.8232.5
9.5/1011
15.1/5.9
27.2/300
105011
RA-S0G-AP
66.96995
18.7/30.4
29.1/30.4
37,8/8822
10.2211.4
16.2/17.5
38.9/30.
8.39.11
11,5/12.7
Dnesstue melloat
SGTR ei 2027
59.5161.6
30.7333.2
40.5/43.1
37 638.7
14,516677
21.023.3
31,2/36.
10,9/12.8
16. 8/18
SGIR-WV'
62.6163.8
33.035.66
43.3/46.1
40,040.5
17.3/889
23.1/25.6
33.7/37.0
13.1/15.0
18,6/0.6
43.2/45.88
35.1/36.3
17.2/18.6
23.1/34.6
27.6/22.0
134/1-7
18AV20.
EGTR+xPP
56.6588.5
37.7/0.3
45.4/47.9
9/38.8
19.7/205
24.8/26.4
29.9/33.7
15.9/17.1
19,6/2.5
11 M4SSG
Kim all
2024c
62:464.3
16.4799.3
25.9/48.8
38.4/30.3
21.1/22.7
27.2/38.8
26.2/30.5
14.6/173
18.7/221
LL.MSSS
65.3177.1
38 38.342..4
48.1/50.5
40.344.0
22.9/25.0
29.7/30.5
29.0/32.6
16,9/20.0
20.5/23.8
ST-SQG
Kim 2024
28.2331.7
37.3/41.0
33.825.11
17.1/18.2
22.6/2400
26.931.0
16.4/196
ST-S0C-7PP1
31.5/34.8
144.99
20.5/23.7
26.5/23.1
15.0/18
19,5/22.4
Spea[ Kim 2024:
55.9/58.1
31,1233.6
39.9/42.6
33.334.6
17.7/19.0
23.1/24.5
24,7229
14.3/16.7
18.1/21.2
SpesQ/FFF
58.061.
34,4737.0
42.7/45.5
36.7757.8
20.5/21.9
26.5/27.7
27,7122
18.2/19.7
2114/23.5
dynamically adapt to unseen compositional scenarios. This establishes APT as a universal solution for advancing open-vocabulary visual reasoning.
Table 9: Performance (%) of state-of ofe-o---thhrrrrr Open Vocabulary SGG models with & without APT on Open Image V6 Kuznetsova et al. (2020). i denotes the results are produced using official code.
Methods
R@20V50/000
mR 20/501000000
F@ 20/501100
R 20/50/100
mk 20/5//000/1100
F8 20/50/100
SDSGG Cheo et al.
2024a
N
25.1/32.4/36.8
10.2/14.1116.5
22.3/28.6/32. 14.5/19.//33.0
13.4/18.7/2199
17.1/23.0/6..
SDSGG API
26.2/33.7/38.3
11.3/15.//1..8
15.1/2144/2466
23/6/30,033.8
14.6/201//3.44
18.5/24.7/23
OvSGTR Chen et al.
2024b S00V
21.3/29.0/33.2
14.0/18.2200.
16.9/22.7/26.0
19.2/25.4/29.1
11.8/15.2/179
14.9/19.822.9
OvSGTR-XPP
2
15.1/19,//2..
18.12 18.1//24./27.55
20. 1/26.6630.5
1277/165/18.9
16//21.3/2433
SGTR+RAHP
2025 MANTS
1/50/
1/.4//3.//
26.0/31.9/5500
14.6/19.3/2..8
10.5/14.2/1699
12.2/16.5/193
SCTR+RAHP-APT
40.7/47.2/51.4
20.0/25.1/28.6
27.1/33.1/3..3
15.4/20.2/3.99
11.3/1../1100
13.1/17.5/20..
B.3 ABLATION STUDY OF APT ON OPEN IMAGE
To dissect the individual contributions of APT's components and their synergistic effects, we con- duct comprehensive ablation studies on OI-V6.
As shown in Table TO, we systematically evaluate the impact of individual prompts within the PE- Net backbone. The Detection Prompt alone slightly improves object classification accuracy but shows limited benefits for relational reasoning. In contrast, the Relation Prompt significantly en- hances predicate discrimination, boosting mR @ 100 by +2.6 (34.0-+36.6) underscoring its piv- otal role addressing predicate bias. The full APT integration achieves the optimal balance, with F@ 100 reaching 48.7, demonstrating the synergistic effect between object-level and relation-level adaptation.
Further analyzing the Compositional Generalization Prompter (CGP) in Table IT observe pro- gressive improvements. The Relational Context Gating (RCG) module establishes a foundation by incorporating visual evidence, while Basis Prompt Synthesis (BPS) enables dynamic prompt gener- ation for unseen concepts, increasing Novel mR @50 by +3.9 over the baseline. The complete CGP achieves the highest harmonic mean (F@50: 30.6) on Novel categories, validating our multi-stage prompting approach for open-vocabulary generalization.
C ADDITIONAL EXPERIMENTS ON GQA
C.I COMPARISON WITH BASELINES ON GQA
The evaluation on GQA Hudson & Manning (2019) shows that APT brings stable gains under both two-stage and one-stage paradigms, with more sigmificant improvements on the class-balanced met- ric mR @ K, indicating consistent benefits for long-tailed distribution and cross-scene generalization.
15

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study of APT based on PE-Net on Open Image V6. produced using official code.
denotes the results are
Predicate Classification
Scene Gruph Clussification
Model
R@500000
mR@50/100
F@50/100
R0500000
mR@50/100
F@50/100
Vanilla PE-Net
65.1/67.4
31.7/34.0
42.6/45.2
37.8/38.8
17.9/19.0
24.6/25,9
+D-Prropt only
65.4/67.3
30.6/32.8
41.2/44.0
38.6/39.5
16.8/18.1
24.2/25.5
+R PromPrmmp only
64.8/66 8/68/66.9
33.6/36.6
43.8/46.3
38,7/39.6
19.8/21.0
25.3/26.5
+Full APT
62.4/64.3
36,4/39.2
45.8/48.7
38,4/39.3
21.1/22.7
27.2/28.8
Table 11: Ablation study of APT CGP module based on SDSGG Chen et al. (2024a) on Open Image V6 split. denotes the results are produced using official code.
Model
Base
Novel
R@50
mR@50
R@50
mR
F@50
Vanilla SDSGG"
26.5
12.4
16.9
25.4
25.2
25.3
25.7
26.9
26.7
+RCG BPS only
27.0
14.6
19.2
26.0
29.1
28.9
+Ful CGP
27.4
16.1
21.0
26.4
31.4
30.6
Table 12 summarizes the comparative results of each model under the three settings: Predicate Clas- sification, Scene Graph Classification, and Scene Graph Detection.
Table 12: Performance (%) of -fate-of-the-art SGG models with & without APT on GQA Hudson & Manning (2019). F@ the harmonic mean of mR 50/100 and @ 50/1000. i denotes the results are produced using official code.
Methods
Res50/100
Predicate Classificaion 50V100
F0550/100
R0050/100
Scene Graph Classification
F0:50/100
R
Scene Graph Detection mil 50//5000100
Fo550/100
motheds
Moed Zellers al
2018
628864,2
14.7/15.9
23.8/25
36.1/36.9
8.599.2
14.014.9
28.9/33.1
6.2/7.3
10/4/1233
Moti
64.6/66.1
16.6/17.9
26.2/27.9
38,099.00
10.2/11.0
16.2/7.11
31.1/354
8,599.8
13.2/15.0
PEl Zheog al.
2025
64.065.55
17.2/18.7
27.3/28.9
35.5/36.1
8,1997
15.013.66
26.6/9.4
6,177.0
10.31118
PE-Ne1
65.067.3
19.020.4
29.030.9
36.637.88
10.1/18
16.1/7..0
27.851.1
11.7/13.5
DRM Li 20240 37933
64.366.1
17.519.1
27.429.5
35.8/6.6
9,3/100
152//60
26,8297
64472
10.6/2.1
DRM
67.008.7
19.3/266
29.021.88
38.4/9.1
1101116
17.2/8.1
29.4/1.8
8.29.6
12.71422
RA-SC3 Yooe
20155AANTS
2025
64 185588
17311888
27.2729
35.636.2
9,299
15.1/59
26.4/202
6,069
10.2/116
RA- 1A-50G+3PF1
65,46774
18.5/20.2
28.8/0.7
36.9137.5
10.011111
16.0/7.2
28.1/29.6
7.9/8.8
11.7/12.9
SGTR Lietal 2022
58.260.1
29.3231.
39 19.11
36.2/37.4
13.615.7
20.2/22.1
29.5/34.1
10.3/12.1
16AV18.
SGPRRAPP
61.182.
31.634.1
42.0/44.1
39.09997
16.5/18.1
22.9/4.55
32.4/36.0
12.8/147
18.4/20.7
ECTRR Im al 2024
53,65559
34.7237.0
420044.5
34.2335
164//1.
22.0/33.6
26.831
12.6/14
17.2/20.0
EGTRTAPT
55.8177.7
36 36.8299.2
44.046.4
36.037.9
18.7/19.6
24.3/6.0
29.1/33.2
15.1/16.4
19.1/21.7
M.LMS Kim atal
2024c10022 001344
61,263.1
35. 1/3.
45.4
37. 1/3.2
20.5222.1
26.0227.7
25.2/29
13.7/16.1
18.3/21.3
64.065.8
37.3341.4
47.5/00.6
39.240.9
22.4/24.5
28.5/0.2
27.3/33.8
16.1/19.1
21.2/338
ST:SGI Kim al. 212440
53.0156.7
27
56.2/40.0
33.1/34.5
160017.2
22.3/23.9
25.9/30.1
11.1/137
164/1122
ST-SOCTXP1
58,0616
30.633399
39.5443.
36.2/37.9
19,4200
25.6/26.9
29.5/33.5
14.4/17.7
19.2/22.2
Spen()
20243
35.1257.5
30.2/32.7
39.0941.7
23.0/44.5
23.9/25.3
13.8/16.3
18,020077
Spo004PPT
33.636-2
42.2/44.9
19.6221
26/0/7.3
26.831.0
17.6/9
21.1/23.6
C.2 COMPARISON WITH OPEN VOCABULARY SGG MODELS ON GQA
We further examine APT's effectiveness under the Open Vocabulary setting on GQA, where models must generalize to unseen predicate compositions beyond the training taxonomy. Following com- mon practice, we split relation categories into Base (70%) and Novel (30%) sets to probe true com- positional generalization rather than memorization. As summarized in Table T3 APT consistently improves all OV-SGG baselines across both Base and Novel splits, with more pronounced gains on the challenging Novel categories. Its cross-architecture benefits indicate that APT alleviates key bottleneck in OV-SGG: the rigidity of frozen representations when facing unseen compositions. These results establish APT as general and plug-and and-play solution for open-voca visual reasoning on GQA.
16

---

Published as a conference paper at ICLR 2026
Table 13: Performance (%) of state-of-the-ari Open Vocabulary SGG models with & without APT on GQA Hudson & Manning 2019). f denotes the results are produced using official code.
Methods
Base
Novel
2020011000
20/55000000000
10/00/000
R@2055/100
mnR 205500000000
20/50/100
SDSGG
9.8/13.5/15.9
13.9/19.0/2222
20.4/26.6/30
12.5/17.4/20.6
1/22/22///
SDSGG
24.4/31,4:35.0
10:9/14,7/17.2
15.2/20.6/23.7
21.8/281//11.7
13.8/18.9/221
16.9/23.8/2699
OvSOTR
202/1b 200K224
20.1/27.5/31.8
OvSGTR=API
21.1/28.8/58 8133.
14.3/18.7/21 2
17 17.3/23.5/26.6
18,925 1/28.9
12. 1/5.998888
14:4/19.//2.6
RAHR+RAHP|
37.9/44.1/48.2
18.3/220026.1
25.
11.5/15./188
SGTR+RAHP-AP1
38.9/45.4449.6
19:4/24.2/27.4
26.4/32.1/35.2
14.6/19.4/2.77
10.8/14.9/17.4
12.3/17.2/1999
C.3 ABLATION STUDY OF APT ON GQA
To quantify the contribution of each component and their combined effects GQA, conduct step-wise ablations in both closed- and open-vocabulary regimes.
As shown in Table 14 within the PE-Net backbone, the Detection Prompt slightly benefits object- centric cues with marginal effects on relational reasoning. In contrast, the Relation Prompt is the primary driver for predicate discrimination, yielding clear gains in mR @K and the harmonic mean F@ K. The full integration achieves the best balance between precision and coverage, delivering the highest F@ 100.
We further ablate the Compositional Generalization Prompter (CGP) on SDSGG in Table 15 Re- lational Context Gating (RCG) establishes a Assual-videncc-awaae baseline, while Basis Prompt Synthesis (BPS) enables dynamic prompt composition for unseen relations, progressively improv- ing Novel mR 50. The complete CGP attains the best F@ on Novel, validating our multi-stage prompting for open-vocabulary generalization on GQA.
Table 14: Ablation study of APT based on PE-Net on GQA. denotes that the results are produced using official code.
Predicate Classification
Scene Graph Classification
Model
R@50/100
mR mR@50/100
F@50/100
R@50/100
m
F@50/100
Vanilla PE-Net"
63.4/65.2
16.8/18.2
26.6/28.4
35.2/36.0
9.0/9.6
14.8/15.5
+D-Prompt only
63.9/65.6
16.0/17.3
25.7/27.4
36.0/36.8
8.6/9.2
14.3/14.9
+R. Pr-Prrmpp only
63. 13.2/65.0
18.7/20.9
28.6/30.9
36. 1/37.0
10.7/11.6
16.8/17.6
+Full APT
62.0/63.8
20.0/22.5
30.1/32.7
36.4/37.3
12.2/13.5
18.4/19.7
Table 15: Ablation study of APT CGP module based on SDSGG Chen al. (204a)) on the GQA split. denotes that the results are produced using official code.
Base
Novel
Model
R@
@5@50
F@50
R@50
mR @ 50
F@50
Vanilla SDSGG
30.1
13.5
19.0
26.6
17.4
22.0
+RCG
30.4
14.2
19.7
27.0
18.6
23.1
+RCG + BPS only
30.7
15.7
21.3
27.3
20.9
24.6
+Full CGP
31.1
17.2
23.0
27.7
22.5
26.2
D FORMALIZATION OF THE COMPOSITIONAL GENERALIZATION PROMPTER (CGP)
This appendix provides precise mathematical specification of the CGP module used APT (Re- lational Context Gating, Basis Prompt Synthesis, and Feature Refinement & Fusion).
17

---

Published as a conference paper at ICLR 2026
D.1 NOTATION AND SHAPES
Let D denote the semantic embedding dimension, Dy the visual feature dimension, La the basis prompt length, and N the number of basis prompts. We use the following symbols:
estati (c)
E RD
(frozen class embedding for class c)
V E RD
(visual/context vector)
B E N
(basis prompts)
W. R DxD.
(visual projector)
fo(:): R3D
+ R2D (fusion MLP)
MLP gate (+): R D D-D. D. + R
E (residual scaling, learnable)
D.2 RELATIONAL CONTEXT GATING (RCG)
Given a visual vector and static embedding estatic the gate network produces N real-valued logits followed by softmax obtain convex weights:
MLP gate ([v; @staric]) E R
(14)
au > O (temperature, may be learned or fixed),
(15)
π E A N-1
(16)
exp ((s; + log T;)T)
W Σラ= exp ((s; + log kT)/7)
for i 1, L....,N,
(17)
N
w W-A N-1
wi 2 0, Σ w 1,
(18)
We may also add an entropy regularizer on the gate distribution to control sparsity:
Ront -8
Σ
we log wi:
B 2 0.
(19)
km1
Here [ denotes concatenation.
D.3 BASIS PROMPT SYNTHESIS (BPS)
The CGP synthesizes prompt sequence convex combination of the basis prompts token-wise:
czt(optional poniitoonllppsitionnalll biases) E R, t 1,..., Lt.
exp ((s: + (.)/T)
Wi, Σ exp ((8; + ut)/))
for t = 1, 1,.... Lor
(20)
deaq ogpl
[4]
Whtt Bi[l] E RD,, 1= 1,...., To.
(21)
Msd ogp
D Lol) E R
(22)
To obtain compact pooled prompt we use normalized, token-weighted pooling with normaliza- tion:
La E RDD
p = LayerNorm
(23)
L
The implementation our experiments uses mean- mean-nooling for compact fusion; the sequence- aware variant is also supported via token-level fusion (e.g., cross-attention).
18

---

Published as a conference paper at ICLR 2026
D.4 FEATURE REFINEMENT AND FUSION (FRF)
We project the visual vector to the semantic dimension:
v W. + b, E RR,
(24)
h [P; estatic; v E R
(25)
fo (h) W2 GELU W h + b1) + b2 RD,
(26)
fh(h),
(27)
o (W, ,, bn) (0,1)D
(28)
LayerNorm @smaic + a (g u)) ERP, where a is initialized small (e.g., 0.1), @ denotes element- product.
(29)
19

---

Published as a conference paper at ICLR 2026
E PSEUDOCODE OF APT
The pseudo-code of APT given in Algorithm | and 2
The pseudo-code of the CGP module is
given Algorithm
Algorithm APT: Adaptive Prompt Tuning for SGG
Require: Image I RH#xxx3, static semantic embeddings Estatic E R CxDp; frozen backbone
Brozen
Ensure: Scene graph
1: Z,) Broozen (I)
I/ E RN B BERXI E R class labels; y fI...., C)N // container for adapted features
2: y F arg max. Z
3: A
4:
for E [I,...,NN) .N do
5:
Statti +- Estatic [ye]
(i)
Il (i) static ly(0)) // M/ adaps
RPP. RDD.
6:
v(i)
V.
7:
aduppe
(1)
APTCOREMODULE, FORDEEMA role role general)
statio
R.
e Append "daps A
8:
9: end for
10: A < STACK(A)
11 A E RNxD, // relation prediction
11: G f RELATIONHEAD(A, V,
12: return
Algorithm APTCoreModule
I/ Learnable params: prompts pdst prel E RR kr; projection W. E RD,xD.; fusion MLP fa Require: Static semantic vector Exuiii E Dr; visual vector v E RR: role E
(detection, general subject, object]
Ensure: Adapted semantic vector endapt E Rp
1: if role detectionn then
2:
P t pdet
3: else
4:
P x prel
5: end if
6: p < MEAN(P, dim 0)
7: V + W.
8: h + | Estatic [[
9: Cadapt + (h)
10: return Cadapt
Algorithm CGP: Compositional Generalization Prompter
// Learnable params: basis prompts phasis Bx Lx Dy; gate network go RR D.+ n3 - A A projection W, 2, x refinement MLP So
Require: Static semantic vector Extutic R.. Dy: visual vector R.D.
Ensure: Adapted semantic vector Cadapt D.
I: u f [v|] Esmric]
// u E R P D2 // RCG: gate weights, w B Σn Wh // BPS: synthesized prompt, S E L.
2: w +- (u)
3: S + ΣB Wa Phasis
4: p < MEAN(S, dim = 0)
5: V <- W. v
6: h Ipl || E Extiic || v]
FRF input, h E R3D,
7: Gadapt (h)
8: return eadapt
20


Reviews:
[{"review_id": "gKfzdlZeST", "reviewer": "Reviewer_ND3R", "summary": "This paper identifies a key limitation in current Scene Graph Generation (SGG) approaches—the reliance on frozen semantic embeddings from pre-trained language models such as GloVe and BERT. These static features, while useful in other domains, fail to adapt to the dynamic and context-sensitive nature of visual relationships. To address this issue, the authors propose Adaptive Prompt Tuning (APT), a lightweight and universal plug-in framework that injects learnable prompts to modulate frozen semantic representations into context-aware features. APT can be seamlessly integrated into both one-stage and two-stage SGG architectures, as well as open-vocabulary variants. Extensive experiments on Visual Genome, Open Images V6, and GQA show that APT significantly improves mean recall (up to +6.0 in mR@50 on novel splits) with minimal parameter overhead (<0.5M, <1.5%) and reduced training time (7.8%–25%). The authors further analyze the approach through ablation studies, efficiency evaluations, and theoretical grounding using the Information Bottleneck principle, demonstrating that APT effectively filters semantic noise and enhances contextually relevant representations for relational reasoning", "questions": "The paper identifies frozen semantic embeddings as the core bottleneck in Scene Graph Generation. Could the authors provide additional empirical evidence (e.g., controlled feature ablation or attention visualization) showing how adaptive prompts modulate semantic spaces differently from fine-tuning approaches?\n\nHow stable is APT when applied to different pre-trained language models (e.g., BERT vs. CLIP-text)? Does prompt initialization significantly affect convergence or generalization?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a well-motivated and original problem formulation by diagnosing the representational rigidity of frozen semantic features as a fundamental bottleneck across all SGG paradigms. This diagnosis reflects deep insight beyond superficial performance concerns and unifies multiple architectural directions under a single, representation-level perspective. The proposed APT is elegant and conceptually sound, leveraging prompt tuning to dynamically adapt pre-trained embeddings without retraining large models. The design’s universality and modularity—applicable to both one-stage, two-stage, and open-vocabulary frameworks—demonstrate strong methodological generality. The experiments are comprehensive and convincing, covering multiple datasets, models, and metrics, and include detailed ablations and efficiency analyses that support the claimed advantages. Moreover, the authors’ theoretical explanation via the Information Bottleneck principle adds interpretability and rigor, showing awareness of both empirical and conceptual coherence.", "weaknesses": "While the paper presents solid experimental evidence, it would benefit from stronger empirical isolation of causal effects—for example, disentangling the relative influence of prompt conditioning versus visual feature fusion. \n\nThe description of certain components (e.g., Basis Prompt Synthesis and Feature Refinement) could be more mathematically formalized to ensure reproducibility and conceptual clarity. \n\nAlthough the authors frame APT as a “universal paradigm,” the current validation remains centered around SGG; testing on broader relational reasoning tasks (e.g., visual question answering or video relation detection) would further substantiate its universality. \n\nThe reliance on pre-trained language backbones and learned prompts also raises questions about potential semantic drift or bias amplification, which the paper acknowledges only implicitly. \n\nWhile the Information Bottleneck discussion provides an elegant theoretical perspective, it remains largely qualitative without direct quantitative verification (e.g., mutual information analysis). Addressing these aspects would make the argument more rigorous and strengthen the generalization claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WNTpzxwzx0", "reviewer": "Reviewer_qtN8", "summary": "The proposed method of using learnable text embeddings for SGG is a good direction. The results on benchmarks confirm the effectiveness of this approach.", "questions": "Please refer to the issues detailed in the Weakness part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Focusing on text embeddings is a great way to solve this problem. The paper does a good job of analyzing this, and the method is designed in a way that makes sense. The results are good, and it's also a bonus that the method is efficient.", "weaknesses": "My main concern is that the paper doesn't compare its method to the right baselines. The best embedding it compares against is from CLIP. Why not use embeddings from much more powerful modern models like Qwen-VL? The introduction claims that existing embeddings can't tell the difference between \"standing on\" and \"walking on,\" but I'm not convinced this is true for today's large VLMs. This makes the problem seem less important than it's presented. The authors need to add an experiment comparing their learnable embedding directly against an embedding from a model like Qwen. They must prove their method is actually better.\nThere are also some small problems, including:\n1. The claim that Figure 2 illustrates a \"richer substructure\" from left to right is not visually evident.\n2. Does the structure in Figure 2(b) only appear in the scenario of closed-set SGG? What about for open-vocabulary SGG?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbOjWEjzMx", "reviewer": "Reviewer_mifr", "summary": "The paper proposes APT, a plug-in module for Scene Graph Generation (SGG) that keeps language backbones frozen while learning lightweight prompts to modulate object/predicate semantic embeddings using visual context. Concretely, APT introduces (i) a Detection Prompt and Relation Prompt fused with static word vectors via small MLPs, and (ii) a Compositional Generalization Prompter (CGP) with context gating, basis‐prompt synthesis, and feature refinement to improve open-vocabulary generalization. The authors motivate APT by arguing that frozen embeddings (GloVe/BERT/CLIP-text) are rigid and misaligned with context-dependent visual relations, illustrated via t-SNE plots and discussion of the one- vs two-stage SGG divide. On VG (with appendix results on OI-V6/GQA), they report consistent gains in mR/F across several backbones and claim <0.5M extra parameters and 7.8%–25% training-time reduction per epoch.", "questions": "1. In Table 5, Why do some+APT variants show fewer parameters than their baselines? Provide a breakdown and add inference latency/memory/FLOPs.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear problem framing. The paper articulates a real pain point in SGG—static semantic priors—and gives intuitive illustrations (Figures 2–3).\n2. The idea of drop-in prompts for both one- and two-stage SGG and OV-SGG is straightforward and potentially practical.\n3. Compute friendly. Claims of small parameter overhead and faster epochs are attractive for the community.", "weaknesses": "1. The core idea (i.e., learn prompts on frozen backbones) has strong prior art: VPT (vision prompts), MaPLe (joint vision+text prompts for CLIP), DualPrompt (complementary/global-vs-expert prompts), and adapter-style PEFT (CLIP-Adapter / Tip-Adapter). APT reads like a tailored application of these ideas to SGG, but the paper does not articulate a new principle beyond that specialization. A side-by-side comparison (same budgets) against these methods is missing.\n2. Motivation evidence is mostly qualitative. The \"frozen semantics are rigid\" claim is intuitive, but support rests on t-SNE/intuition. Provide quantitative diagnostics that isolate the bottleneck (e.g., probing tasks for role asymmetry, controlled swaps of subject/object labels, ablate text vs. vision prompts separately).\n3. The information-bottleneck framing is descriptive; provide any measurable prediction (e.g., MI/CKA proxies showing APT features are \"more minimal yet sufficient\") that the experiments then corroborate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9I3rnmYeb", "reviewer": "Reviewer_m1jU", "summary": "This paper points out that the core bottleneck of SGG is the mismatch between pre-trained semantic priors and visual relationship dynamic contexts, rather than a one-stage/two-stage dispute. Propose an APT lightweight plugin that converts frozen features into context sensitive representations, with 0.5M parameters and a training acceleration of 7.8% -25% plug and play. With the highest score increase of+6.0 on PredCls, SOTA is refreshed with<1.5% overhead, providing SGG with a unified, efficient, and scalable new paradigm.", "questions": "See [Weaknesses].", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is written clearly and organized, especially in the introduction section, where the author provides a detailed explanation of the problem of frozen features using three figures.\n\n2. The APT model not only achieves a significant improvement in accuracy relative to SOTA, but also ensures a limited increase in parameter count and saves training time.\n\n3. The relevant work has been thoroughly sorted out, and the source code of the paper has been provided.", "weaknesses": "1. The main objective of this work is to address the shortcomings of the one-stage and two-stage methods, but these shortcomings are not unique to SGG tasks, and the designed method can also be applied to other tasks. This will bring two sub-issues: 1) Have you tried using it on other tasks; 2) The analysis of the specificity of SGG tasks in this work needs to be further explored.\n\n2. The ablation study is comprehensive only under the top-50 setting, while it is incomplete under the top-20/100 setting. In the experimental section, ablation research is the core experiment, and it is strongly recommended to supplement it completely. In addition, it is recommended to try removing the one-stage, two-stage, and Open Vocabulary parts of APT-enhanced methods in sequence to eliminate random factors caused by model combinations.\n\n3. Partial modifications needed: 1) Spelling error in line 045 for \"Fiugre\"; 2) Some images have font sizes that are too small and do not match the main text.\n\n4. One-stage methods lack the latest methods for 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)