Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation
Authors: Lin Li
Chuhan ZHANG
Dong Zhang
Chong Sun
Chen Li
Long Chen
Venue: NeurIPS 2025 poster ()
Keywords: Open Vocabulary Scene Graph Generation
Weakly Supervised Pre-training
Interaction-Centric Knowledge Infusion
Interaction-Centric Knowledge Transfer
Primary area: applications

Abstract: Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt a two-stage pipeline: 1) Infusing knowledge into large-scale models via pre-training on large datasets; 2) Transferring knowledge from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category. This limitation induces critical issues in both stages of OVSGG: it generates noisy pseudo-supervision from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer. To this end, in this paper, we propose an interACtion-Centric end-to-end OVSGG framework (ACC) in an interaction-driven paradigm to minimize these mismatches.  For interaction-centric knowledge infusion, ACC employs a bidirectional interaction prompt for robust pseudo-supervision generation to enhance the model's interaction knowledge. For interaction-centric knowledge transfer, ACC first adopts interaction-guided query selection that prioritizes pairing interacting objects to reduce interference from non-interacting ones. Then, it integrates interaction-consistent knowledge distillation to bolster robustness by pushing relational foreground away from the background while retaining general knowledge. Extensive experimental results on three benchmarks show that ACC achieves state-of-the-art performance, demonstrating the potential of interaction-centric paradigms for real-world applications.

Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulay Scene Graph Generation
Lin Li Chuhan Zhang!?? 1.2, Dong Zhang 122 Chong Sun Chen Li Long Chen
'HKUST AA Chip Center for Emerging Smart Systems Tencent
(1llidy, chuhanzhang, dongz, longchen]@ust... hk, {waynecsun, chaselli)@tennenn..com
http://gtthbcco///UUUU/Lngnnngnn
Abstract
Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leverag- ing the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt two-stage pipeline: 1) Infusing knowledge into large-scale models via pre-training on large datasets; 2) Transferring knowledge from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category. This limitation induces critical issues in both stages of OVSGG: it generates noisy gpeudo-supervistee from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer. To this end, in this paper, we propose an interACtion Caiin-CCetiic end-to-end OVSGG framework (ACC) in an interaction-driven paradigm to minimize these mismatches. For interaction- centric knowledge infusion, ACC employs a bidirectional interaction prompt for robust seeudo-supervision generation to enhance the model's interaction knowledge. For interuction-centri knowledge transfer, ACC first adopts interaction- guuided query selection that prioritizes pairing interacting objects to reduce interference from non-interacting ones. Then, it integrates interactioncconsisten knowledge distillation to bolster robustness by pushing relational foreground away from the background while retaining general knowledge. Extensive experimental results on three benchmarks show that ACC achieves state-of-the-an performance, demon- strating the potential interaction-centro paradigms for real-world applications.
Introduction
Scene graph generation (SGG) [55] aims to map an image into structured semantic representation, where objects are expressed as nodes and their relationships are as edges within the graph. Recently, with the burgeoning large-scale models, egg., languael models (VLMs) and multimodal large language models (MLLMs), OVSGG [14, 29, 9] has emerged as promising area. It pushes beyond predefined categories to support the recognition and generation novel objects and relationships, holding great potential for real-world applications.
Generally, end-to-end VLM-based OVSGG pipeline consists of two phases: Knowledge Infusion and Knowledge Transfer. The former infuses knowledge from large-scale datasets into VLMs via pre-training. This process aims to achieve visual-concept alignment via caption-region comparison. Specifically, due to the lack of region-level information (e.g., bounding box annotations), recent work [14, 63, 9] adopts a weakly strategy to generate (subject, predicate, object)
'Long Chen is the corresponding author.
We primarily discuss VLM-bbsed models due to the high resource demands of MLLM-based approaches.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VLM Pre-training
SFT
Piccdo Superviiiperiiiinn
Objecct Query Candidates
Query Initiattzation
Tencher
Object Detretor
Genersl
Pair
Griste
Konndadter
VE
mon, sunthoard
DETR-
like
Classifler
Decoder
fora pezzee while wan hekl two swyfoandd
Iann mon man.... mon
Guide Pro truining
prompt
TR
Lrasee avon, riding, GT
Speegte Knowlddge
Caplion
surtboard
surfboord
(u) Knowledge Infusion
Knowledge Trunsfer
Figure I: Overview of the end-to-end OVSGG framework challenges. a) Knowledge Infusion, using solely object categories for detection causes ambiguity in associating object pairs (e.g.. identifying the correct "man- surfboard" for the "hold"). b) Knowledge Transfer, vast object query² candidates make misaligned non- interacting objects (e.g.. "man k")) with interacting training target "man" in (man, riding, horse).
triplets with bounding boxes as pseudo-superrisions. As displayed in Figure (a), this framework first extracts semantic graphs from image captions using SGG parsers [41], then grounds objects in the graphs with off-the-shelf object detectors (e.g., Faster R-CNN [39], GLIP [27] and Grounding DINO [35]). The latter transfers knowledge from pre-trained VLMs by refining with task-specific objectives and high-quality annotations during supervised fine-tuning (SFT). Concretely, for specific knowledge, it finetunes part of VLM's parameters [9] or adapts prompt-tuning [14] on SGG dataset with super-suee triplet annotations (c.f. Figure 1(b).. Leveraging these bounding box anno- tations, a DETR-like structure [4, 66, 62] with bipartite graph matching is typically used to align predicted object queries* with ground-truths. Moreover, knowledge distillation (KD) [13, 60, 9] is widely used during SFT, where generalist VLM (teacher) guides the target model (student) to retain general knowledge, allowing robust adaptation to unseen categories open- world scenarios.
Despite notable advancements, prevailing OVSGG frameworks exhibit an object-centric paradigm in both knowledge infusion and transfer, i.e., lack of interaction differentiation between instances within the same object category. For example, the man involved in a holding action and the man without any action are represented in an indistinguishable manner. can amplify mismatches in relation pairs across both pre- tre-training stage (i.e., knowledge infusion) and SFT stage (i.e., knowledge transfer), which induces the following drawbacks: 1 Bringing noisy supervision illustrated in Figure (aa), relying solely on entity categories (e.g., man and surfboard) to detect objects generates large number of candidate pairs. This ambiguity makes hard to associate relation (e.g., "hold") to the proper object pair (e.g., "man-surffoard"). Using mismatched triplets (e.g., man in red and surfboard in pink) further exacerbates the confusion, hindering the training robust SGG models. @ Leading mismatched objects during SFT. Due to the vast object query candidates, non-interacting "man kn query can be mistakenly associated with man in the triplet annotation (man, riding, horse) during bipartite graph matching, as displayed in Figure 1(b). However, the real target is another "man ln* engaged in riding. This mismatch further complicates the relation classification task, making it harder to predict correct interactions.
pre-training. As
In this paper, we propose the interACtion- Ceroon-Centicccc end-to-end OVSGG framework (ACC), which fundamentally rethinks knowledge infusion and transfer through an interaction-driven paradigm. Unlike conventional object-centric approaches that treat all instances uniformly, ACC explicitly models relational dynamics both pre-training and SFT stages to reduce the pervasive mismatch between interacting/non- peeraattinnnonnnnteraatnng object pairs. For interaction-centric knowledge infusion, we devise bidirectional interaction prompt to facilitate visual triplet pseudo-supervision generation, thereby infusing more robust interaction knowledge into pre-trained VL.Ms. These prompts incor- porate interaction tokens that capture contextual dependencies and relational semantics, enabling the grounding model to distinguish interacting objects (e.g., man involved in holding action) from knowledge
non-intenn non-interacting ones through the attention mechanism [49]. For interaction- transfer, to achieve the paradigm shift from object-centric to interaction-centtic knowledge transfer, we first establish interaction -uided query selection, a two-step mechanism to prioritize interacting objects and incorporate relational context into the query selection process, mitigating interference of
*Object queries are learnable embeddings input to its Transformer decoder, each specializing through attention global image features to predict unique object's localization and classification.

---

inactive objects and reducing mismatches in bipartite graph matching. To preserve general knowledge. we further incorporate interaction-consiiite KD realize both point-wise semantic alignment and inter-pair relational consistency among teacher and student. By explicitly modeling the relative dependencies between interaction- seeeactioo-based and non-interaction pairs, this KD strategy enhances the model's s robustness in handling novel triplet combinations and background and avoiding catastrophic general knowledge forgetting [13, 9].
To evaluate ACC, we conducted comprehensive experiments on the benchmark Visual Genome (VG) [19], GQA [16], and PSG [56] datasets to validate its effectiveness in addressing the key challenges of OVSGG. In summary, our contributions are threefold:
We reveal key limitations in existing OVSGG frameworks, i.e., the neglect of interaction-specific characteristics among instances of the same object category during knowledge infusion and transfer, which leads to widespread relation pair mismatches.
We propose an interaction-centric end-to-end OVSGG framework ACC, shifting the paradigm from object-level representations to interaction-driven learning. By explicitly encoding interactions during both knowledge infusion and transfer, ACC enables more accurate scene graph generation and robust generalization to unseen categories.
Extensive experiments on three prevalent SGG benchmarks demonstrate the effectiveness and generalizability of ACC.
2
Related Work
Open- Veee--ocayul SGG (OVSGG). This task bridges the gap between closed-set SGG and real- world requirements by leveraging VLMs or MLLMs to generalize beyond predefined categories [38, 35, 22, 57]. Current approaches fall into two main groups: 1) VLM-based Methods. They primarily rely on contrastive pre-training to align visual and textual embeddings. By comparing visual features of unseen objects/relations and their semantics in common spaces, these models (e.g.. CLIP [38] and Grounding DINO [35]) enable zero-shot generalization. Recent advancements, such as He et al. [14], explore visual-relation pre- pre-training and prompt fine-tuning for OVSGG. Yu et al. [59] leverage CLIP to align relational semantics in multimodal spaces, while Chen et al. [9] use a student-teacher framework to improve open-set relation prediction. Besides, other methods integrate class-level descriptions [25, 20] or scene-level descriptions [6] to enrich the semantic context and enhance the discrimination among different relationships. 2) MLLM-based Methods. They extend the capabilities of VLMs by incorporating auto-regressive language models, predicting objects and relations in an open-ended manner. Specifically, they use the sequential prediction capabilities of MILMs, e.g., BLIP [21] and LLaVA [33], to model scene graphs as structured sequences [24]. For example, PGSG [29] and OpenPSG [65] employ auto-regressive models to iteratively predict open-ended objects and relations. ASMv2 [51] builds on LLaVA [33] with instruction tuming [52]. umifying both object localization [42, 43] and relation comprehension [23, 26]. Despite their power, MLLM-based methods typically require huge computing resources. This work focuses on VLM-based methods and proposes an interaction-centric framework that explicitly models interactions and enhances generalization to novel categories.
Knowledge Infusion and Transfer for Open- Vocabbaayyy Learning. Recent VLM advancements unlock ooca-vocabuyary downstream tasks via two main steps: 1) Knowledge infusion into VLMs (e.g., CLIP [38]) via contrastive learning on large image-text pairs for aligned visual-textual repre- sentations. 2) Knowledge transfer by SFT of pre-trained VI.Ms with task-specific objectives and high-quality annotations for adaptation tion. Within this framework, effectively mining semantic knowledge and leveraging transferable representations has emerged as a key research area improve generalization in open-world settings while reducing computational/annotan encoder with CLIP's visual encoder and employ prompt tuning [30] to adapt image-level represen- tations to region-level tasks for OV object detection. Similarly, Cho et al. [10] fine-tune CLIP for open-vocabulary segmentation by incorporating cost aggregation techniques [15]. Besides, Chen et al. [9] extend this framework to OVSGG and build upon the GroundingDINNG [35] with knowledge distillation to preserve learned knowledge, but still under an object-centric paradigm. Conversely, this work emphasizes interaction-centric knowledge infusion and transfer for robust OVSGG.
tasks like open-vocabulary object detection/seegmenta
costs. For instance, Wu er al. [54] replace the DETR-stE -tyle
3

---

Grunding DINO
Bidirectional Interaction
Rule bele--aasedd
bold
A grap of people pos for pictuee man
sirtboard
mun
fhourdl war wirthoard
by
surfboard
suribound
Caption Parser
neo vertoards.
Vorh Parser
Guide Pre- Pre-truining
Interaction- Centric Knowledge Infusion
Top-K Query
Object Query Candicanes
VE josat
Fatity
+
fyp
Coslds
Internetion- guided
like Decoder
Bipurtiie Minaall
Classifier
Retation: rilieg
torme. (d07
Selection $3.2.1
cistag. aboree
Prompt
Chavalier
Interaction- consistent KD
$5.22
(b) Interaction- Centrrc Knowledge Trausfer
Figure 2: Overview ACC for OVSGG. (a) Interaction Centric Knowledge Infusion: Employs bidirectional interaction prompts and rule-based bounding box combinations for robust pseudo-supervision, empowering the model's s grasp of relational knowledge. (b) Interaction-Centte
Knowledge Transfer: Uses interaction- guided query selection prioritize learning interacting objects, and interaction-consistent KD transfers comprehensive relational insights from pre-trained VLM ensure robust generalization novel categories.
Knowledge Distillation (KD). This strategy trains smaller "student" model replicate the outputs of a larger "teacher" model, commonly used in open-vocabulary learning to transfer knowledge from VLMs. It encourages the student mimic the teacher's enriched hidden space, enabling generalization from base to novel concepts. Prior work [13, 60] explores KD in open-vocabulary object detection by using L1/MSE loss align the student detector's features with the teacher VLM's regional visual features. However, this hard alignment may fail to capture complex feature structures. Later work [2, 37] aligns the similarity structured knowledge. Recent work extends to multi-scall level [50] or bags-of-region level [53], contrasting with InfoNCE loss. This paper adopts an ineraction-consisse KD that combines point- to-point concept retention and structure-aware interaction retention distillation, preserving teacher's knowledge and identifying novel relationships beyond backgrounds.
embeddigds,, aiding in the acquisition of
3 ACC: Interaction-Centri, End-to-end OVSGG Framework
Formulation. Given an image I, SGG aims to construct a structured semantic graph g (V. Each node E V is defined by its bounding box (bbox) and category. while each edge eız E E represents the relationship between v and Uj. In open-vocabulary settings, the label set for nodes and edges is divided into base classes Cn and novel classes CN such that Cp U CN = C and CB n Cy C3 contains seen classes during training, while Cx includes unseen classes that the model is expected to generalize to during inference.
Baseline End-to-End OVSGG Architecture. As illustrated in Figure 2(b), an end-to-end OVSGG framework [9, 36] typically follows dual--enooder-sinlle-eeed architecture [35], involving three
main components:
Visual and Text Encoders. Visual encoder (VE) extracts multi-scale visual features E X. Text encoder (TE) processes textual prompts that concatenate all predefined object and relation categories [63, 9], e.g., " [CLS] man. horse. [SEP] riding. above. [PAD] " derive semantic embeddings for objects T, R- N.xd and relation T, E E. N.xd Here, N.. N., and N,- denote the numbers image, object, and relation tokens, respectively. is the feature dimension.
DiTR-like Decoder. It refines the representations of K object queries {q through self- attention and cross-attention mechanisms, leveraging both visual and text features, ultimately predicting object bounding box coordinates. Besides, global relation query Gret often utilized to capture spatial and semantic dependencies among objects [44,9] 9].
Entity and Relation Classifiers. For open-vocabulary recognition, node features (e.) (derived from refined object queries) and edge features {et) (constructed by combining features of paired

---

objects, potentially augmented with global relation embeddings) are compared against the textual object/relation class embeddings.
The training of such models conventionally relies bipartite graph matching align object queries with ground-truth -runn annotations, minimizing a cost function based on semantic and spatial criteria [4]. Optimization objectives usually include bbox regression loss (L1 Creg and GloU loss Cyiou [40]), cross-entropy based entity and relation classification losses (Cobj and Cret)*. R However, the efficacy of this process is undermined the supervision is noisy (due to object-centric pre-training) or if query-to- target alignment is confounded by non-interacting distractors (due to object-centric SFT). To surmount the limitations imposed by current end-to and-to-end OVSGG designs, ACC introduces a fundamental shift towards an interaction-drie driven paradigm. As illustrated in Figure 2, ACC incorporates interaction- centric knowledge infusion and transfer.
3.1
Interaction-Centric Knowledge Infusion
Addressing the challenge of noisy supervision from object-centric pseudo-labeling (issue 1 in $1), ACC's knowledge infusion stage fundamentally alters how training targets are generated for VLM pre- training. To ensure that pseudo-supervision effectively captures interaction distinctiveness, especially within weakly annotated data, ACC conditions the object detection process on interactional context rather than relying on prompts based on isolated object classes (e.g.. "man. surfboard.").
To be specific, after the semantic graph parsing process, which extracts initial -ubject-rredicat--biee triplets from image captions with language parser [41], employ Grounding DINO [35] as the object detector and design bidirectional interaction prompt to guide the object localization. The bidirectional interaction prompt is constructed by combining two perspectives for each interaction triplet: one reflecting the action from the subject's viewpoint (e.g.. "man hold surfboard") and another from the object's perspective (e.g., surfboard held by man"). The former is directly derived from the components of the interaction triplet, while the latter converses the subject and object with a counter-action (e.g., "held by") generated by the verb parser. This verb parser is typically an LLM (e.g., Llama2 [48] and Qwen [1]) or Python Library.
The pers-peere construction process brings two key advantages: 1) Modeling Context Informa- tion: Through the attention mechanism in the text encoder of Grounding DINO, the bidirectional interaction prompt integrates contextual interaction information into object tokens. As shown in Figure 2(a), the attention mechanism enables the token "man" to absorb relevant interaction semantics, such "hold surfboard", ensuring that the grounded object "man" is correctly aligned with its interaction context. 2) Enhancing Object Role Awareness: By reversing operation, the object (e.g., "surfboard"" of given triplet becomes the syntactic subject of the whole sentence (e.g., "surfboard held by man")) As the central of the rephrased sentence, the syntactic subject receives heightened attention, improving its accuracy in localization.
Furthermore, inspired by [31, 17], we adopt a rule-based combination that combines overlapping subject and object bounding boxes to form triplet supervision by Intersection over Union (IoU) score.
3.2 Interaction-Centrie Knowledge Transfer
The interaction-centric knowledge infused during pre-training ($3.1) provides a strong foundation. Nevertheless, it still faces a mismatch problem during SFT (issue @ in $1). ACC's interaction-centric knowledge transfer is designed to ensure that 1) the selection and refinement of object queries are explicitly guided by interaction potential, and 2) the rich, interaction-focussee pre-trained model is adequately transferred and further enhanced to discriminate between genuine interactions and non-interacting background. This is achieved through ierractionguudde query selection and interaction-consiitenn knowledge distillation.
knowledge from the
3.2.1 Interaction-Guidde Query Selection
To mitigate mismatched object pairs during SFT, interaction-guidee query selection instills an interaction prior into the two-step query generation process to reduce non-interacting candidates.
'Detailed formulations are left in appendix $B.
'The generation process of counter-action is in the appendix $C.
5

---

Step I. This step aims to directly identify the most relevant visual tokens likely to participate in object interactions. Intuitively, the visual features of interacting objects should exhibit strong correlations with both object and relation semantics. To achieve this, for each visual token E V relevance score 8; computed by combining its maximum similarity with object and relation class tokens:
8,
(max(v;T))) ))
(1)
where max(v, (v:T), ) computes the maximum similarity between the visual token v, and all object class tokens in To, while max(v;T,) computes the maximum similarity between V; and all relation class tokens in T,, The parameter 7 [0, balances their contributions. Based on the relevance scores, the top K query indices, denoted as Zg. are selected by the following procedure:
Ix = Topx ((s, i = 1,2,...NNJ3).
(2)
The visual features and the position embedding [62, 35] corresponding to the selected indices Ix are used to initialize queries for further decoding operations.
Step II. Nevertheless, Step I's individual encoding of object and relation tokens struggles to capture deeper interaction semantics and distinguishes among objects. Thus, Step II explicitly models interaction semanties by integrating relational context into the object tokens. Specifically, after the initial forward pass, the model predicts set of visual relation triplets. These triplets are decomposed into interaction pairs (subject, predicate) and (predi cate, object), which serve as interaction- guided prompts. These prompts are encoded via the TE of VLM to get interaction tokens embeddings Tin- The decomposition process has dual advantages: First, the predicates within prompts guide the TE's attention to infuse object tokens with interaction information, enabling the model to capture contextual dependencies and enhance its understanding of relationships. For instance, the token "man" " can incorporate the semantic meaning of the interaction "riding" to obtain "man ln" in Figure 2(b). Second, decomposing triplets into pairs avoids direct interference between object tokens, effectively preserving their unique characteristics. As illustrated in Figure 2(b), "man 4o" and "horse *t" are independently processed, preventing unnecessary dependencies across unrelated categories and maintaining the individual semantics of each object.
For each visual token v,, the interaction relevance score sin is calculated by measuring the maximum similarity with interaction tokens:
* max(v, (v, T I..).
(3)
The query indices set prioritizes the top L tokens with the highest interaction relevance:
zt = Top, ((s," i 1,2, 1,
(4)
However, relying solely on interaction relevance may fail identify objects absent from the initially
predicted triplets yet crucial for comprehensive scene understanding. To address this, the object relevance score se is computed similarly, but using object tokens T.. The remaining K - L query indices are selected based on object relevance, excluding those already chosen:
- r = Topx-n
Zt i 1,2...., No
(5)
The final query indices set combines these two subsets Iς = Ii," UIK r. This two-step strategy effectively reduces non-interacting candidates and mitigates bipartite graph mismatches. Pseudo-code detailing this process is in appendix $D for clarity.
3.2.2 Interaction-Consittrr Knowledge Distillation
Beyond localization and classifica- tion objectives, we adopt interaction- consistent KD to enhance the model's ability to distinguish interacting pairs from background and address catas- trophic forgetting of learned relational semantics [9]. Specifically, it uses the VLM pre- pre-trained in the first stage as the teacher model. The student net-
Teacher
Student
Teacher
Student
LVRD
CARD
Background
Background
(a) VRD
Figure 3: Illustration of interaction KD. work is designed as a pseudo-siane siamese structure of the teacher, initialized with its parameters.
This KD combines visual-concept retention distillation (VRD) and distillation (RRD) align the student model with the teacher's semantic space while maintaining inter-pair relational consistency. The entire loss function contains two complementary objectives:
retention

---

VRD. The first objective draws from [9] ensures that the student's edge features remain point-wise consistent with the teacher's semantic space for negative samples. The loss is defined as:
LVRD
1 W
Σ
les-erll1.
(6)
where e, and θy represent student and teacher's edge features, and W is the set of negative samples.
RRD. While VRD effectively preserves point-wise semantic consistency, it fails to ensure the relative relationships between triplets, i.e., distinguishing interaction pairs from backgrounds (c.f. Figure 3(a)). RRD explicitly models inter-pair relativity [2, 37] by aligning the structure similarity of triplet embeddings between the teacher and student models. The structure similarity matrices for the teacher and student models, M, and Ms, normalized by 1.2 norm:
M2 ej e
ej og
M
||e/
lle eg 112
RRD then aligns these similarity matrices by minimizing the Frobenius norm || || between them:
CARD
M Mrlla.
|Ms
(8)
Final Objectives: Combine localization and classification losses with above complementary objec- tives to achieve point-wise semantic alignment and relational consistency:
L = Lreg + Liio + Lors + Lret + BILYRD + $2CARD-
(9)
The weights 31 and B2 control the importance of semantic alignment and relational consistency.
4 Experiments
4.1 Experiment Setup
Datasets. We evaluated ACC on three SGG benchmarks: 1) VG [19] contains annotations for 150 object categories and 50 relation categories across 108,777 images. Following standard setup [55]. 70% of the images are used for training, 5,000 for validation, and the remaining for testing. For fair comparison, we excluded images overlapping with the pre-training dataset of Grounding DINO [35], retaining 14,700 test images as in [63]. 2) GQA [16] uses the GQA200) split [12, 45], including 200 object categories and 100 predicate categories. We randomly sampled 70% of the object and predicate categories as the base, and more details can be found in the appendix $A. 3) PSG [56] offers 44, 967 training, 1,000 test, and 3,000 validation images (sampled from training), with 133 object and 56 predicate categories. We adopted the same base and novel class splitting in [29].
Settings. Following [9], we compared ACC under two settings: 1) OvR-SGG: Evaluates generaliza- tion to unseen relations while retaining original object categories. Fifteen of 50 relation categories in VG150 are removed during training, with performance measured on "Base+Novel (Relation)" and "Novel (Relation)". 2) OvD+R-SGG: Assesses handling of unseen objects and relations si- multaneously. Both novel objects and relations are excluded during training, evaluated on "Joint Base+Novel", "Novel (Object)", and "Novel (Relation)".
Metrics. We conducted experiments under the challenging Scene Graph Detection (SGDET) protocol [55, 19], which requires detecting objects and identifying relationships between object pairs without GT object labels or bounding boxes. We reported: Recall @K K (R (R@K): The proportion of ground-truth triplets correctly predicted within the top-K K confident predictions. 2) Mean R@K K (mR @ K): The average R @ across all categories.
Implementation Details. Due to space constraints, details are provided in the appendix $A.
4.2 Comparison with State-of-the-Art Methods
We compared ACC with existing state-of-the-art methods, i.e., V [63], OvSGTR [9], and RAHP [36]. The experimental results on the VG dataset [19] under both the OvR-SGG and OvD+R- SGG setups are shown in Table and Table 2, respectively. Notably, ACC consistently outperforms the latest SOTA methods across all metrics. the OvR-SGG setup, ACC surpasses the RAHP (Swin- T) by +1.78% R 100 within the novel relation categories, demonstrating superior generalization
7

---

Table 1: Experimental results of OvR- setting on VG [19] test set.
Nase+Novel (Relation)
Novel (Relation)
Method
Backbone
R@20
R@
R@
R@20
R@50
R@ 100
IMP [55]
CVPR117
12.56
14.65
0.00
0.00
MOTIFS [61]
CVPR118
15.41
16.96
0.00
0.00
VCTREE [47]
19
15.61
17.26
0.00
0.00
TDE [46]
CVPR220
15.50
17.37
0.00
0.00
OpenSGen [18]
I5MR22
18.00
20.50
15.70
17.90
VS [63]
CVPR113
15.60
17.30
0.00
0.00
OvSGTR [9]
POCVV224
Swin-T
20.46
23.86
13.45
16.19
RAHP [36]
AAAF25
20.50
25.74
15.59
19.92
ACC (Ours)
17.49
23.22
27.40
12.90
17.89
21.70
OvSGTR [9]
BOCV'24
Swin-B
22.89
26.63
16.39
19.72
ACC (Ours)
18.77
24.81
29.28
14.72
20.04
24.66
Table Experimental results OvD+R-SGG setting on VG [19] test set.
Joint Base+Novel
Novel (Obj)
Novel (Rel)
Method
Backbone
R@
R@50 R@ 100
R@20
R
R@ 100
R@20 R 50
R@ 100
IMP [55]
CVPR'17
0.77
0.94
0.00
0.00
0.00
0.00
MOTIFS [61]
CVPR'15
1.00
1.12
0.00
0.00
0.00
0.00
VCTREE [47]
CVPR119
1.04
1.17
0.00
0.00
0.00
0.00
TDE [46]
CVPR'20
1,00
1.15
0.00)
0.00
0.00
0.00
V [63]
CVPR'23
5.88
7.20
0.00
0.00
0.00
01.00
OvSGTR [9]
BCCV/24
Swin-T
10.02
13.50
16.37
10.56
14.32
17.48
7.09
9.19
11.18
ACC (Ours)
12.61
17.43
21.27
12.48
17.16
21.10
11,38
15.90
19.46
OvSGTR [9]
BOCV124
Swin-B
12.37
17.14
21.03
12.63
17.58
21.70
10.56
14.62
18.22
ACC (Ours)
13.50
18.88
23.19
13.46
18.84
23.29
12.37
17.50
21.73
and reduced overfitting. With the Swin-B backbone, ACC achieves 29.28% R@ 100, which is higher than OvSGTR across both base and novel relations, further emphasizing its robustness. In the more challenging OvD+R-SGG scenario, ACC continues to outperform the competition. Specifically, on the joint base and novel classes, ACC gains +4.90% and +2.16% R @@ 100 over OvSGTR with the Swin-T and Swin-B backbones, respectively. These results validate ACC's superior performance and robust generalization across both relation and object domains.
4.3 Diagnostic Experiment
To ensure a comprehensive evaluation, we performed a series of ablation studies on the VG dataset [19] in the challenging OvD+R-SOO scenario. More experimental analyses are left in the appendix.
Knowledge Infusion Part. We analyzed the effectiveness of ACC's bidirectional interaction prompt (BIP) for pseudo-supervision generation ($3.1) in Table 3. It can be seen that BIP leads to consistent improvements across all metrics. Notably, when compared to the configuration without BIP, it achieves @ 100 gains of 1.73% on the joint base and novel classes, 1.45% on
Table Ablation studies ($4.3) on BIP.
Method
Split
R@20
R@50
R@ 100
12.61
17,43
21.27
Joint Base+Novell
BIP
16.17
19.55
Ours
12.48
17.16
21.10
Novel (Obj)
w/o BIP
12.36
16.09
19.65
Ours
11.38
15.90
19.46
Novel (rel)
w/o BIP
10.73
14.40
17.83
novel object classes, and 1.63% on novel relation classes, respectively. This demonstrates that BIP effectively improves performance by considering interaction contexts in supervision generation.
Knowledge Transfer Part. We evaluated the efficacy of ACC's interaction-guit guided query selection (IGQS 13.2.1) and interaction-consisten KD (ICKD $3.2.2) in the knowl- edge transfer phase. The re- sults are summarized in Table 4, with the first row representing the baseline OVSGG pipeline with visual-concepp retention distillation from [9]. From this analysis, three key conclusions can be drawn: First, IGQS refines the query selection process. By prioritizing interacting objects and minimizing mismatched assignments, IGQS achieves notable improvements, such as 3.00% @ 100 gains, highlighting its ability to enhance precision by focusing on interacting object pairs. Second,
Table 4; Ablation studies ($ 4.3) on IGQS and ICKD.
Components IGQS ICKD
Joint NovelNovel
Novel (Obj)
Novel (Rel)
R 20 R 50 R@ 100 R R@@2 20 R@50 R 100
R@20 R@ 50 R@ 100
10.02 13.50
16.37
10.56
14.32
17.48
7.09
9.19
11.18
11.37 15.71
19.37
11.43
15.80
19.61
9.84
13.92
17.38
15.67
19.20
11.57
15.65
19.32
10.07
17.32
11.84 16.17
19.55
12.36
16.09
19.65
10.73
14.40
17.83

---

Table 5: Comparison with pre-training methods, All models are pre-trained image-caption data and tested
VG150 [19] test set directly. Our models trained COCO captions used as pre-trained models.
SGG model
Buckbone
Grounding
R@20
R@
R@
LSWS [58]
CVPR22
3.28
3.69
MOTIFS [61l
18
et al. [31]
5.02
6.40
7.33
Uniter [8]
BCCV/20
SGNLS [64]
5,80
6.70
Uniter [8]
BCCV/20
et al. [31]
5.42
6.74
7.62
VS [63
CVPR'23
GLIP-L [27]
5.59
7.30
8.62
OvSGTR [9]
BOCV'24
Swin-T
Grounding DINO [35]
6.61
8.92
10.90
ACC (Ours)
Grounding DINO [35]
7.86
10.81
13.31
OvSGTR [9]
Grounding DINO [35]
6.88
9.30
11.48
Swin-B
ACC (Ours)
Grounding DINO [35]
8,28
11.61
14.33
leveraging interaction-consisten KD with relative-interaction retention distillation ensures relational consistency during training, resulting significant performance boosts. It contributes 2.83% R@ 100 gains, improving the model's ' ability to handle novel classes effectively. Third, the integration of two components yields the best overall performance, with 1.80%~6..55% improvements across all evaluation metrics. However, the improvement is less pronounced than expected, since each strategy prioritizes interacting objects, which may lead to diminishing returns by progressively reducing non-interacting objects. Despite this, the combined results still demonstrates enhanced relational understanding and serve as a valuable tool for improving performance in complex scenarios.
Supervision Analysis. We investigated ACC's impact in the pre-training process (c.f. Table 5).
original
ACC
people, ride, bike
As seen, models pre-trained on COCO [7] cap- tions with ACC variants consistently outperform others. achieving 13.31% % R@ 100 with Swin- T and 14.22% @@@ 100 with Swin-B. These re- sults demonstrate the effectiveness of ACC in the VLM pre-training process.
bikes, nn, boat
In addition, we visualized the object detection results from ACC and the original methods that solely use object categories for detection. As dis- played in Figure 2, the original method produces redundant objects, complicating the identifica- tion of subject-object interactions. For instance, given the "(people, ride, bike)" triplet, the baseline detects multiple instances of "people" and "bike", obscuring the interaction. contrast, ACC leverages bidirectional interaction prompts and attention mechanisms to accurately localize the interaction-relevart objects. A similar enhancement observed in the "(bikes, on, boat)" triplet, where ACC focuses on interaction-relevant entities.
Figure 4: Pscudo supervision generation in ACC.
Query Visualization. To demonstrate the ef- fectiveness of IGQS, we visualized the top-50 selected queries in Figure 5. As seen, the orig- inal approach makes no distinction between instances within the same category, such as "man" or "zebra", resulting in both interact- ing and non non-interacting instances receiving similar number of queries. This indiscriminate query generation increases the likelihood of in- correct matches during bipartite graph matching, as irrelevant regions compete with interaction- relevant instances. Conversely, IGQS prioritizes interacting queries ("man holding" or "zebra lay- ing on" in Figure 5), increasing discrimination among objects with the same categories.
original
holding-guided
IBAI
originul
laying on-guided
Figure 5: Interaction -uided query selection.
Conclusion
This work presented ACC, an interaction-centric OVSGG framework. ACC alleviates current
paradigms' failure to distinguish interacting from non-interacting instances by adopting interaction-

---

centric principles in two key phases. Knowledge infusion uses a bidirectional interaction prompt for robust pseudo-supervision, enhancing interaction understanding; knowledge transfer combines interactiongguidee query selection with interaction-conisste knowledge distillation to mitigate mismatches and irrelevant object interference. ACC shows significant improvements on three main benchmarks. We anticipate that ACC will not only set new standard for OVSGG but also inspire further exploration of interaction-driven strategies in VLMs for more accurate scene understanding.
Acknowledgement. This work was supported by the National Natural Science Foundation of China Young Scholar Fund (62402408). This research was partially conducted by ACCESS AI Chip Center for Emerging Smart Systems, supported by the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government. This research was also supported by the Hong Kong SAR RGC Early Career Scheme (26208924) and sponsored by Tencent WeChat Rhino-Bird Focused Research Program.
References
[1] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[2] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open- open-vocabulary detection. In NeurIPS, volume 35, pages 17799444,4 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. pages 381-389. IEEE, 2018.
[6] Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. NeurlPS, 2024.
[7] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1550.00025,, 2015.
[8] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, pages 104-120, 2020.
[9] Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention. ECCV, 2024.
[10] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Se- ungryong Kim. Cat-seg: Cost aggregation for vocabuacyyylary semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4113-4123, 2024.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4188 2019.
[12] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In CVPR, pages 19427-19438 2022.
10

---

[13] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021.
[14] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV, pages 56-73, 2022.
[15] Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong. Sangryul Jeon, Dongbo Min, and Seungryong Kim. Neural matching fields: Implicit representation of matching fields for visual correspondence. Advances in Neural Information Processing Systems, pages 13512-13526, 2022.
[16] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, pages 6700-6709, 2019.
[17] Kibum Kim, Kanghoon Yoon, Jachyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language models for weakly supervised scene graph generation. In CVPR, pages 28306-28316, 2024.
[18] Zihan Kong and Haiwei Zhang. Opensgen: Fine-gre aaee-graind Station-awarr prompt for open- vocabulary scene graph generation. pages 634-643, 2025.
[19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
[20] Jiaming Lei, Lin Li, Chunping Wang, Jun Xiao, and Long Chen. Seeing beyond classes: Zero- shot grounded situation recognition via language explainer. In ACM MM, pages 1602-1611, 2024.
[21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 PMLR, 2023.
[22] Lin Li, Guikun Chen, Jun Xiao, and Long Chen. Compositional zero-shot learning via progres- sive language-based observations. ACM MM, 2025.
[23] Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18869-18878 2022.
[24] Lin Li, Wei Chen, Jiahui Li, Kwang- Ting Cheng, and Long Chen. Relation-rl: Progressively cognitive chain-of-thowe tf-thught guided reinforcement learning for unified relation comprehension. arXiv preprint arXiv:2504. /4642, 2025.
[25] Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. NeurlPS, 36, 2024.
[26] Lin Li, Jun Xiao, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Long Chen. Nicest: Noisy label correction and training for robust scene graph generation. 46(10):6873-66 6888 2024.
[27] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded sagguageiimage pre- pre-training. In CVPR, pages 10955-10966 2022.
[28) Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In CVPR, pages 19464 19474, 2022.
[29] Rongjie Li. Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In CVPR, pages 28076- 28086, 2024.
[30] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXhv:2101.0119, 2021.

---

[31] Xingchen Li, Long Chen. Wenbo Ma, Yi Yang, and Jun Xiao. Integrating object-aware and interaction-awer knowledge for weakly supervised scene graph generation. In ACMMM, pages 4204 4214,,421,220 2022.
[32] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pages 1999-3007, 2017.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024.
[34] Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 2021.
[35] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. CoRR, abs/2303.05499, 2023.
[36] Tao Liu, Rongjie Li, Chongyu Wang, and Xuming He. Relation-aware hierarchical prompt for opea-locay scene graph generation. In AAAI, 2025.
[37] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pages 1967-3976, 2019.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 0748-8763, 2021.
[39] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurlPS, 28, 2015.
[40] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pages 658-666, 2019.
[41] Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70-80, 2015.
[42] Feifci Shao, Long Chen, Jian Shao, Wei Ji, Shaoning Xiao, Lu Ye, Yueting Zhuang, and Jun Xiao. Deep learning for veakly-supervised object detection and localization: A survey. Neurocomputing, 496: 197-207, 2022.
[43] Feifci Shao, Yawci Luo, Fei Gao, Yi Yang, and Jun Xiao. Knowledge-guided causal intervention for weakly-supervised object localization. 36(11):6477-648, 2024.
[44] Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei
Li. Jiazhen Pan, Sahand Sharifzadeh, Georgios Kaissis. Volker Tresp, al. Relationformer: A unified framework for image-to-gran graph generation. In ECCV, pages 422-439. Springer, 2022.
[45] Gopika Sudhakaran, Devendra Singh Dhami, Kristian Kersting, and Stefan Roth. Vision relation transformer for unbiased scene graph generation. In ICCV, pages 21882-21893, 2023.
[46] Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, pages 3713-3722, 2020.
[47] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, pages 6619-6628, 2019.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0920 2023.
[49] A Vaswani. Attention is all you need. NeurIPS, 2017.
12

---

[50] Luting Wang, Yi Liu, Penghui Du Zihan Ding, Yue Liao. Qiaosong Qi, Biaolong Chen, and Si Liu, Object-aware distillation pyramid for open-vocabular, object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11186-11196 2023.
[51] Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan. Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. In ECCV, pages 471-490. Springer, 2025.
[52] Zhen Wang, Lin Li, and Long Chen. Recent advances in finetuning multimodal large language models. Al Magazine, 46(3):e70025, 2025.
[53] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of 2023.
regions for open- vocabuocyyylary object detection. In CVPR, pages
[54] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7031-704p 2023.
[55] Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 3097-3106, 2017.
[56] Jingkang Yang, Yi Zhe Ang. Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In ECCV, pages 178-196. Springer, 2022.
[57] Gefan Ye, Lin Li, Kexin Li, Jun Xiao, and Long Chen. Zero-shot compositional action recognition with neural logic constraints. In ACM MM, 2025.
(58) Keren Ye and Adriana Kovashka. Linguistic structures as weak supervision for visual scene graph generation. CVPR, pages 8289-8299, 2021.
[59] Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually language model for fine fine-grained scene graph generation in open world. In ICCV, pages 21560-21571, 2023.
[60] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, pages 106-122, Springer, 2022.
[61] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pages 5831-58444 2018.
[62] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2022.
[63] Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang Wen Chen. Learning to generate manguage-supervinn and opea-vocab scene graph using pre-trained visual- semantic space. In CVPR, pages 2915 2-244, 2023.
[64] Yiwu Zhong, Jing Shi, Jianwei Yang. Chenliang Xu, and Yin Li. Learning to generate scene graph from natural language supervision. In ICCV, pages 1823-1834, 2021.
[65] Zijian Zhou, Zheng Zhu, Holger Caesar, and Miaojing Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models. In ECCV, pages 199-215. Springer, 2025.
[66] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to end-to-end object detection. In ICLR, 2020.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We carefully described our contributions in the abstract and introduction. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:In the appendix, we discussed our limitations, societal impact, and directions for future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification: This paper is not about theory.
14

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided details about the methodology and implementation in the main paper and appendix. The code will be publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
15

---

Justification: The code will be publicly available in the future. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co ccc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
CCodeSubmis / ss onnolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experimental setup and details the main paper and appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We each experiment three times and report the average and standard deviation.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] Justification: We introduce the used computer resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp:///eeu...eeeeeeee cc. wttee//ulic//eee d ines?
csGuidel
Answer: [Yes]
Justification: We carefully reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:In the appendix, we discussed our limitations, societal impact, and directions for future work.
Guidelines:
The answer NA
that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out
17

---

that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited related papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
18

---

Justification: The paper does not release new assets Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We describe the usage of LLMs in the appendix. Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
19

---

Please refer to our LLM policy hta.rppss.c//aaffrrooe///2////11 for what should or should be described.
20

---

Summary of the Appendix
To facilitate a deeper understanding of the main paper, we present supplementary material with additional details, organized as follows:
A elaborates on the implementation details.
BB introduces the formulations of training objectives.
SC introduces the counter-action generation prompt.
$D provides the pseudo-code for interaction-gan -uidded query selection.
S offers additional experimental results.
SF presents further qualitative results.
$G discusses our limitations, broader impact, and directions of future work.
A Implementation Details
Pre-training, Our models are trained with a batch size of 3, utilizing four/eight RTX 3090 GPUs for computation. During the supervision generation phase (c.f. $3.1), we employ Llama2-7B [48] to generate counter-actions based on the prompts described in $C. A pseudo-triplet class is annotated when the confidence of the grounding object class exceeds 0.25, and the intersection over union (IoU) between the subject and object is greater than 0.0. During pre-training, we initialize model using the pre-trained Grounding DINO checkpoints provided by [9], keeping the visual backbone (Swin-T or Swin-B) and the text encoder (BERT-base [1I]) frozen. The remaining modules, such as the relation-aware embedding. are initialized randomly. line with [9], we select 100 object detections per image for pairwise relation recognition during training.
Supervised Fine-Tuning. The supervised iin-tuning process is conducted using the same com- putational resources as pre-training. For interaction interaction-guided query selection (c.f. $3.2.1), we adopt the settings from [35, 9], where the total number of selected visual tokens K is set to 900, and the top-ranked interaction tokens L is fixed at 200. The models after the pre-training process are leveraged as the teacher model and serve as the initialization for the student model. The weights 8 and B2 of the loss function LVRD and LRRD are set to 0.1 and 0.5, respectively, to balance different optimization objectives.
Dataset Splits. All entity and relation categories for the GQA dataset [16] are listed in Table S1. For the VG dataset [19], we adopted the splitting protocol from [9]. As for the PSG dataset [56], we followed the splits utilized in [29].
B Training Objectives
As mentioned in $3, the model guided by the bounding box regression loss, entity classification loss, and relation classification loss. This section details their corresponding formulations.
Bounding Box Regression Loss: The primary objective of object localization is to accurately predict the positions and sizes of objects within an image. To achieve this, the model utilizes a combination of LI loss (Creg) and GloU loss (Cgiou) [40], ensuring both precise positioning of the bounding boxes and effective handling of overlaps. The corresponding loss functions are defined as:
No Ibb: balli,
Creg
Σ
No
1=1
(10)
Ainter
(Amin Aunion)
Eyrme
Aunion
Amin
where b, and b; denote the predicted bounding box and GT bounding box, respectively. N₂ is the number of the object's bounding boxes. Ainter represents the of intersection between the predicted and ground truth bounding boxes, Awnion is the union area of the bounding boxes, and Amin is the area of the smallest enclosing box covering both.
Entity Classification Loss: To address the class imbalance the object classification task, the model employs Focal Loss (Cob)) [32], which emphasizes difficult-t-cllassffyy and underrepresented

---

Table S1: The categories spitting of GQA [16].
Split
Relation Categories
Object Categories
parked on, growing on, standing in front of, wearing, standing on, with, looking under, carrying, near, above, covered in, behind, at, using, hanging from, sitting on, llyng in, watching, covering, mounted on, in front of, lying on, standing next to, grazing in, holding, beside, the back of, catching. running on, swimming playing on, lop of, floating talking on, on the bottom of, standing behind. leaning against, covered by, facing. filled with, attached to, sitting next to, next to, worn on, in, on the side of, driving, close to, surrounded by, lying in, hitting, pulling, swinging, touching, cating. throwing. skiing on, driving on. hang on, riding, playing in, crossing, walking with, growing in, sitting in, cutting, feeding, leaning
mountain, cow, people, face, number, pizza, tire, player, pillow, screen. truck, kite, trunk, sock, neck, glove, coat, letter, roof. windshield, desk, paw, leaf, flower, plant, counter, paper, eye. book, branch, lamp, cup, phone, loilet, skateboand, logo, laptop, vehicle, motorcycle, hill. curtain, nose, sheep, bowl, wire, bear, banana, mouth, drawer, shelf, cap, animal, bottle, box, airplane, finger, room, flag, seat, lower, wing, fruit, rock, house, pot, bird, umbrella, surfhoard, lady, tic, fork, vasc, bag. orange, clock, sidewalk, food, sink, cahinet, beach, boat, hasket, helmer, child, racket, post, guy, towel, arm, napkin, bush, bench, person, cone, apple, jacket, fur, air, sign, bus, wrist, frame, floor, dress, street, shoe. ball, girl, ear, boy. broccoli, fence, uniform, hair, sneakers. blanket, zebra, train, camera, sticker, license plate, lid, tomato, pants, girafse, watch, wall, leg. bed, t-shirt, shorts, horse, spots, arrow, field, bread, bicycle, knife, couch, cciling
Base
on the front of. reaching for, flying. of, parked along, talking to, sitting standing by hanging on, covered with. standing near, full of, surrounding, walking in, reflected in, walking down, walking on, contain, below,
ocean, picture, hand, snow, horn, woman, sweater, container, paint, feet, clouds, foot, dirt, faucet, chair, sand, tail, stone, cat, tag, traffic light, keyboard, tree, leaves, elephant, ground, glass, frishec, trash can, word, man, jeans, door, building. sky, table, wheel, pole, collar, hat, cheese, mane, shirt, dog, cord, cake, donut,plate, backpack, mirror, street light, skis, window, grass, water, bike, road, head, cell phone
Novel
printed on, driving down, waiting for, resting on, playing with, standing in, grazing on, by, around, pulled by, beneath
categories. Focal Loss modifies the standard cross-entropy loss by down down-weighting easy examples, focusing the model's attention challenging ones. The formulation is as follows:
Coky -a(1 y.)" log(ve),
(11)
where Ye denotes the predicted probability of the true object class c, a is balancing factor, and is a focusing parameter that adjusts the emphasis on hard examples.
The model's objective is to predict the relationships between objects, aligning predicted relation scores with ground-truth annotations. This is achieved using BCE loss (Cret), which measures the discrepancy between predicted and true relationship probabilities. The BCE loss function is expressed as:
Relation Classification Loss:
Nrst [1i log(ū:)) ) + (1 uı²)) log(1 - ūss),,
Lret
(12)
Nrat
where Vij denotes the GT relation label between the i-th object and j-th object, ūig is the predicted relation probability.
C Counter-action Generation Prompt
In this section, we present the prompt used for counter-action generation (c.f. $3.1) in Figure SI with LLMs, i..,, Llama2 [48]. The prompt is struc- tured into two key components: the example and the question. Example: The example, such as the instance of "ride". serves as a reference for the model to produce contextually rele- vant outputs in an in-context learning framework [3, 34]. This part of the prompt also generated by the LLM, providing model-driven demonstra- tion of the expected output format. Question: The question, i.e. "please generate...", prompts the model to produce corresponding counter-action or related output. This structure ensures that the
Question: Given the action 'ride', please generate its corresponding countcr-action.
Answer: 'be ridden by'. Question: Given the action 'cat', please generate its corresponding counter-action. Answer: 'he caten by'.
Question: Given the action '[relation]'. please gener- ate its corresponding counter-action.
Answer:
Figure S1: Counter-action generation prompt.

---

Algorithm S1 Pscudo-code for Step in Interaction-Guiie Query Selection.
X.v: visual features of all tokena.
I_o: object class tokens.
relation
garma: balancing paranater.
Step1_ QuerySSelection(... scores [
K_r, gamna):
for in range(1en(X_
Calculate the relevance
sin_o [ax(I_v[i) I_ aax sinilarity with object clase tokens sim_r max(I_v[i] x_r.T) a max similarity with relation class tokens (sin_o ganna) (sin_r gamna)) Eq. (1)
.ppre..ppeea(ecore)
Select K visual tokens based relevance I_ top_R(s,
I_X
Algorithm Pseudo-code for Step in interaction-Guided Query Selection.
V_ vieual features of all tokens.
X_in: interaction tokena from text encoder.
__o: object class tokens.
def Step2 harpplpurrSSeeeeennnee interaction_scorer - []
X_in, X_o):
(aene
Compute interactioon relevance score based on interaction tokens. ein_in (ax(X_v1s) k_in.T) nax similerity with interaction tokens
interaction_score apppree.pppnd(ein.in
Select L vieual tokene based interaction relevance acore. I_ top wwweeeeractionnecoreeee
Compute object relevance for remaining tokens.
object_accres for in range(1e((X_v));
if i not in I_L_im:
nax(X_v[i] e X_o.T) similarity with object class tokens @jject_coree.apeuneiin
Sclact top (X-L) tokens based objact relevance.
I_K__, L_ijjiiaa_
K L, I_I_in)
Pinal query the union of both I_ I_K_1_o
I_X
model generate responses specific to the action at hand, supporting more relevant and consistent counter-action generation.
D
Pseudo Code
To make the interaction-guided query selection ($3.2.1) process easier understand, provide pseudo-code for Step and Step Algorithm and Algorithm S2, respectively.
E
More Experimental Results
E.1
Comparison with State-of-the-Arts on GQA dataset
In Table S2, we compared our ACC with the existing SOTA method (i.e., OvSGTR [9]) on GQA [16] dataset under the more challenging OvD+R-SGG setting. The backbones are uniformly set to Swin-T. Notably, ACC consistently outperforms OvSGTR across metrics, demonstrating the universality and effectiveness of our approach.

---

Table S2: Experimental results of OvD+R-SGG setting GQA [16] test set.
Joint Nase+Novel
Novel (Obj)
Novel (Rel)
Method
20
R@
100
R@20
R@50 @
R@20
R@50
100
OvSGTR [9] BCCV224
11.21
15.80
19.14
10.32
14.92
18.76
2.59
5.21
7.40
ACC (Ours)
12.30
16.88
20.63
11.51
16.16
20.57
3.41
6.60
9.80
Table S3: Experimental results of OvR-SGG setting PSG [56] test set.
Joint Base+Novel
Novel (Rel)
Method
R 20
R@
R@ 100
R 20
R@50
R@ 100
SGTR [28]
CVPR'22
14.2
PGSG [29]
CVPR224
18.0
20.2
OvSGTR [9]
BCCK224
15.14
17.76
19.50
5.32
6.93
8.08
ACC (Ours)
16.69
20.01
21.71
6.78
8.78
9.70
Table $4: Extra metrics of OvD+R-SGO setting on VG150 [19] test set.
Base (Obj)
Base (Rel)
Novel (Obj)
Novel (Rel)
Method
R@20 R6 R 100
R@20 R@ R 100
mR 20 mR @ mR @ 100 mR
m @:20 mR 6: mR @ 100
OvSGTR [9]
8.78
11.95
14.79
12.07
16.47
20.09
1.69
2.44
3.06
0.82
1.13
1.47
ACC (Ours)
11.66
16.46
20.35
12.20
16.67
20.57
1.93
2.84
3.61
1.64
2.59
3.38
E.2
Comparison with State-of-the-Arts PSG dataset
Given that the PSG dataset split proposed by [29] exclusively addresses novel relation categories, our evaluation consequently focused the OVD-R-SGG setting. As detailed in Table S3, when compared with other prominent statc-of-the the-art methods (e.g.. SGTR [28], PGSG [29], and OvSGTR [9]), our ACC framework also demonstrates superior performance across all reported metrics.
E.3 Evaluation with More Metrics
We reported both recall of base classes and mean Recall (m@R) in S4. It can be seen that our ACC outperforms the previous SOTA method (i.e., OvSGTR [9]) in both metries. This demonstrates that our approach provides a more comprehensive and powerful generalization capability, enhancing performance across the board, not just for unseen classes.
E.4 Ablation Study on Interaction-Centric Knowledge Infusion
Effectiveness of bidirectional interaction prompt. To investi- gate the bidirectional interaction prompt's sensitivity to the choice of verb parser for counter-action generation, we replaced the de- fault Llama2 parser with two alternatives: smaller Large Language Model (LLLM,, Qwen2.5-
Table S5: Ablation study on the verb parser in counter action generation.
Joint Base+Novel
Method
Verb Parser
Size
R@
R@50
100
ACC (Ours)
Llama2
7B
13.50
18.88
23.19
ACC (Ours)
Qwen2.5
0.5B
13.64
18.99
23.43
ACC (Ours) Pattern (Python Lib)
13.36
18.56
22.64
0.5B [1], and the Pattern (a Python library) under OvD+R- GGG setting on VG test set (Swin-B as backbone). As shown Table S5, ACC sustains high performance even when utilizing smaller LLM or non-LLM parser for this task. This demonstrates the robustness of our bidirectional interaction prompt in generating effective pseudo-supervision across various verb parsing mechanisms.
E.5 Ablation Study on Interaction-Centric Knowledge Transfer
Effectiveness of query selection. We per- formed ablation study of the two-step query selection in IGQS (c.f. $3.2.1), 1), as shown Table S6. The general end-to-end OVSGG pipeline with visual-concept reten- tion distillation as the baseline. The results demonstrate that using single step also yields performance improvements over the baseline, with the best performance achieved by employing both steps simultancously.
Table S6: Ablation on two-step query selection in IGQS on OvD+R-SGG setting of VG150 [19] test set.
IGQS Step II
Joint Basc+Nova
Step
R@20
R@50
R@ 100
10.02
13.50
16.37
11.30
15.71
19.16
11.32
15.70
19.29
11.37
15.71
19.37

---

Table S7: Experimental results of HICO-DET [5] dataset under the OvR-SGG selling.
Joint Base+Novel
Novel (Rel)
Method
R@
R@50
R@ 100
R@
R@50
R@ 100
OvSGTR [9]
RCCV'
34.62
37.39
39.04
22.94
28.48
31.84
ACC (Ours)
35.74
38.58
40.19
24.44
30.77
34.38
Table S8: Inference costs on the VG150 [19] test set.
Method
Training costs (min)
Inference costs (s/l)
OvSGTR [9]
DCCV
68
144499220016449949
ACC
71
0.389677112579444
ACC w/ Step
94
0.64021885790400577
Hyperparameters in ICKD, We conducted ablation study on the hyperparameters (31 and 82)
< K3100
2144
2222
in the ICKD (visual-concept tention distillation and relative- interaction retention distillation). Results in Figure S2 show that increasing 31 (e.g., raising VRD weight) decreases overall perfor- mance, consistent with results in [9]. RRD demonstrates robustness for different hyperparameters. The best performance can be achieved with 31=0, 0.1 and B2 = 0.5, respectively.
08
30
Figure S2: Ablation 81 and Ba in VRD and RRD function under OvD+R-SGG setting on VG test set.
E.6 Comparison on Human-Objes Interaction Detection Tasks
Human-Object Interaction (HOI) detection, particularly on benchmarks like HICO-DET [5], primarily detection task over a set of specific human-centric interactions (i.e., <action, object> pairs). In contrast, SGG addresses more general and compositional challenge: generating <subject, action, object> triplets between any pair of objects. To empirically validate the effectiveness of the proposed ACC, we evaluated ACC and OvSGTR [9] on the HICO-DET benchmark. As shown in Table S7, ACC consistently outperforms OvSGTR, achieving 2.54% absolute improvement in R@ 100 of novel classes. This result is significant: it demonstrates that our model's core principles robust. They excel not only on the general OVSGG task they were designed for, but also the specialized HOI task.
E.7 Computational Overhead
We conducted time analysis on VG [19], with training on the entire dataset and testing on 20 images. We report the value in Table S8 of ACC (w/o and with Step II in IGQS). We would like to claim that: 1) Our Step I in IGQS just introduces minor computational complexity in elementary matrix operations (c.f., Eq. 1). Due the requirement of forward prediction for self-enhancemennt Step II will induce extra computational overhead, but the performance gain brought by Step II is optional.
F
More Qualitative Comparison Results
F1
Grounded Entity Visualization
To evaluate the effectiveness of the proposed bidirectional interaction prompt ($3.1), we visualized the entity grounding results for various types of prompts during the pre-training process in Figure S3:
Object Prompt. Prior methods [9] often rely on concatenating the subject and object entities extracted from a scene graph parser, such as "man. knife".
Interaction Prompt. It incorporates relation triplets into phrase, e.g., "man hold knife."

---

Object Prompt
Interaction Prompt
Bidirectional Prompt
kniis.
hole inlis.
hold kni kand huild by
kid. yellow vest.
xid sear yellow vest.
xid wasr yallow vest. yallow vest ween by kid.
Figure S3: Entity grounding results different prompts.
C: Foad
2: roos
head
8:ware
over
Jt daif
weth
ACC
owsted
hor
wecming
Aps
Yeatiing
shirr
Figure $4: Qualitative Results of OvSGTR and ACC the SGDet task and VG dataset [19]. The dashed line represents the predicted novel categories, and red represents the unreasonable predictions.
Bidirectional Interaction Prompt. This proposed prompt further integrates relation triplets with their corresponding counter-action, forming phrases like "man hold knife. knife held by man.".
From the visualization results, the following observations can be made: 1) Directly adopting object prompts tends to generate redundant bounding box candidates (e.g.. multiple instances of "man" " and "kid" in Figure S3). This redundancy complicates the identification of interacting object pairs. Additionally, some interacting object boxes are missing. For example, the imperceptibly held "knife" is not detected, while the non-interacting "knife" is identified. These limitations result in mismatched relational pairs, which ultimately mislead the subsequent training process. 2) While incorporating interaction prompts significantly reduces the number of redundant object boxes, often over vee-ffocuess the subject (e.g., detecting only the "man" subject bounding box), leading to the omission of critical object boxes. 3) By leveraging bidirectional interaction prompts, both the subject and object bounding boxes are accurately detected under the given relation triplet. This approach not only resolves the redundancy issue but also ensures the inclusion of subtle yet crucial interactions (e.g., correctly identifying the "knife" being held). Consequently, it provides more comprehensive and precise grounding for subsequent training stages.
F.2 Mismatched Examples Visualization.
To intuitively illustrate the chal- lenges stemming from current paradigms, Figure S5 visualizes representative examples of mis- matched relational triplets. As depicted, a common error in- volves triplets such as (man, riding, horse) being incor- rectly assigned. This type of mis-
riding
riding
2:
D: horse
4:
horse
Figure $5: Mismatched relation triplets examples.

---

assignment frequently occurs be-
cause models lacking explicit interaction modeling struggle distinguish the specific "man" instance actively engaged in the "riding" interaction from other, non-interacting "man" instances that may be present the scene. Such visualizations highlight the critical need for interaction-centric approaches to achieve more precise relation recognition.
F.3 Scene Graph Visualization
Figure S4 displays the scene graph predictions generated by OvSGTR and ACC on the Swin-T backbone using the VG dataset. Apparently, the scene graph produced by OvSGTR includes several incorrect and redundant relationships, such as "(number, on, wave)" and "(dog. on, leg)". Instead, our ACC eliminates such unreasonable predictions and can generate easily missing the relationship triplet, such as " "(wave, behind, dog)". Even interactive relationships, like "(dog. has, head)" and "(head, of, dog)", " are accurately captured, showcasing ACC's enhanced capacity to reason over subject-object interactions and identify precise and semantically coherent relationships in complex scenes.
F.4 Failure Cases Analysis
We analyzed the examples in Fig- ure $6 where ACC misidentifies non- interacting object pairs, and find that: 1) For InteacctionCenntee Infusion, it difficult to correctly match small objects (e.g., hat in background) and their related ob- jects through bidirectional interaction prompts. 2) For Interaction Centric Knowledge Transfer, when multiple subject-object pairs with the same relational triplet categories (e.g., (person, riding, horse)) appear in the same image, the model might mistakenly match the subject in one triplet to the object in another triplet.
Knowledge
man wear hat worn by man.
person riding horse
Figure S6: Failure cases.
G Discussion
Limitation Analysis. Our approach employs knowledge infusion and transfer framework $2 for open-vocabulary scene graph generation. While this framework reduces annotation costs and effectively leverages transferable representations from pre-trained vision-language models, it also inherits inductive biases from the teacher model. Like two sides of coin, any biases in the vision- language model toward specific feature traits or classes may propagate to our model. Besides, our method can alleviate mismatched relational pairs, but cannot avoid all mismatches.
Potential Broader Impact. This paper presents work aimed at advancing the field of open vocabulary scene graph generation. By introducing interaction-aware mechanisms, approach enhances the model's ability to recognize novel objects and relationships, improving the robustness and accuracy of scene understanding in real-world applications such as robotics, autonomous systems, and augmented reality. While our work has the potential to drive innovation in these fields, ethical considerations must be taken into account, particularly regarding the fairness and representativeenss of the training data used. Ensuring that our models are inclusive and minimize bias will be crucial to preventing harmful misinterpretations or exclusions practical applications.
Future Work. Our current algorithm is tailored for open- vrcabuuuyyylary scene graph generation, adopting bual--encoderssinlll-deoooo -ineedeeiingee architecture proposed in [9, 36.. It prioritizes base-novel generaliza- tion over real-time performance, which may not fully meet the timeliness requirements of real- wollworld applications. In future work, aim enhance the computational efficiency of our approach to better address these practical demands.


Reviews:
[{"review_id": "7YYAdmxy0j", "reviewer": "Reviewer_Ae9W", "summary": "The paper proposes an interaction-centric end-to-end OVSGG framework to reduce the pervasive mismatch between interacting/non-interacting object pairs. The paper introduces a bidirectional interaction prompt to facilitate visual triplet pseudo-supervision generation, therefore achieving interaction-centric knowledge infusion. Besides, it constructs interaction-guided query selection and incorporates interaction-consistent KD to achieve the interaction-centric knowledge transfer.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The motivation is reasonable. The problem claimed in the paper is a key challenge in scene graph generation. \n2. The paper constructed a transfer-learning-based framework, which is suitable for an open vocabulary setting.\n\n\nWeaknesses:\n1. Although the paper focuses on open-vocabulary scene graph generation, the evaluation on base relation should be included. \n2. Considering that the paper focuses on action-centric scene graph exploration, the authors should include a comparison with the HOI task, which focuses on human interactions. \n3. The motivation claimed in the abstract has little relationship with open vocabulary setting; the authors should claim that the challenge existed in open vocabulary scene graph generation.", "was_revised": true, "final_justification": "After reading all the reviews and responses, I maintain my rating score.", "rebuttal": "We deeply appreciate reviewer Ae9W for the valuable time and constructive feedback. We provide point-to-point responses below.\n\n**Q1:  Evaluation on the base relation.**\n\nAs suggested, we have added the evaluation on **base object and relation categories** in **Table_R 1**. The results clearly show that the proposed ACC significantly outperforms previous SOTA (i.e., OvSGTR) on both novel and base classes. This demonstrates that our approach provides a more comprehensive and powerful generalization capability, enhancing performance across the board, not just for unseen classes. We will incorporate these metrics into the revised manuscript.\n\n**Table_R 1: Experimental results of OvD+R-SGG setting on VG test set.**\n\n| Method | Joint Novel + Base R@20/50/100 |  Base (Obj) R@20/50/100  | Novel (Obj) R@20/50/100 | Base (Rel) R@20/50/100 | Novel (Rel) R@20/50/100 |\n| --- | --- | --- | ---- | ------- | --- |\n| OvSGTR (Swin-T) | 10.02 / 13.50 / 16.37 | 8.78 / 11.95 / 14.79 | 10.56 / 14.32 / 17.48 | 12.07 / 16.47 / 20.09 | 7.09 / 9.19 / 11.18 |\n| ACC (Swin-T) | 12.61 / 17.43 / 21.27 | 11.66 / 16.46 / 20.35 | 12.48 / 17.16 / 21.10 | 12.20 / 16.67 / 20.57 | 11.38 / 15.90 / 19.46 |\n\n**Q2: Comparison with the HOI task.**\n\nThanks for your valuable feedback. We agree that a comparison with the HOI detection task provides valuable context for our work. We will incorporate a detailed discussion clarifying the distinctions between these two tasks in the revised manuscript. In brief, the key difference lies in their scope and objective. **HOI detection**, particularly on benchmarks like HICO-DET [1], is primarily a detection task over a predefined, closed set of specific human-centric interactions (e.g., 600 <action, object> pairs). In contrast, **SGG** addresses a more general and compositional challenge: generating <subject, action, object> triplets between ***any pair*** of objects, making the potential output space combinatorially large (e.g., 150x50x150 potential triplets for VG). \n\nFurthermore, to empirically validate the effectiveness of the proposed ACC, we evaluate ACC and OvSGTR on the HICO-DET benchmark. As shown in **Table_R 2**, ACC consistently outperforms OvSGTR, achieving a **2.54**% absolute improvement in R@100 of novel classes. This result is significant: it demonstrates that our model's core principles are so robust. They excel not only on the general OVSGG task they were designed for, but also on the specialized HOI task. We will incorporate these new compelling results into the revised manuscript.\n\n**Table_R 2: Performance comparison on the HICO-DET [1] dataset under the OvR-SGG setting.**\n\n| Method | Novel + Base R@20/50/100 | Base R@20/50/100 | Novel R@20/50/100 |\n| --- | --- | --- | --- |\n| OvSGTR (Swin-T) | 34.62 / 37.39 / 39.04 | 36.87 / 38.12 / 40.51 | 22.94 / 28.48 / 31.84 |\n| ACC (Swin-T) | 35.74 / 38.58 / 40.19 | 37.70 / 40.11 / 41.35 | 24.44 / 30.77 / 34.38 |\n\n---\n\n[1] Chao YW, Liu Y, Liu X, Zeng H, Deng J. Learning to detect human-object interactions. WACV 2018.\n\n**Q3: Claim challenges in OVSGG in Abstract.**\n\nGood suggestion! We will revise the abstract to clearly highlight the challenges in OVSGG. Below is the updated abstract, with bold text indicating the revised parts:\n\n*Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt a two-stage pipeline: 1) Infusing knowledge into large-scale models via pre-training on large datasets; 2) Transferring knowledge from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category.* ***This limitation induces critical issues in both stages of OVSGG: it generates noisy pseudo-supervision from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer*.**"}, {"review_id": "UktJ8U9xTw", "reviewer": "Reviewer_Y5Z7", "summary": "This paper addresses the task of open-vocabulary scene graph generation by proposing a method that focuses on two key aspects: improving vision-language pre-training and enhancing query selection during fine-tuning.\n\n**Interaction-Centric Knowledge Infusion**\n- During the VLM pre-training phase, the authors introduce interaction-centric supervision by incorporating both traditional <subject–active verb–object> and novel <object–passive verb–subject> triplet forms. \n- This is intended to better encode relational knowledge into the model.\n\n**Interaction-Centric Knowledge Transfer**\n- To address issues such as irrelevant query matching or matching objects without relationships (e.g., background noise), the method employs interaction-guided query selection. \n- This ensures that only queries relevant to interactions are used for object detection.\n\n**Interaction-Consistent Knowledge Distillation**\n- Instead of aligning edge features directly, the paper proposes a relative-interaction retention distillation (RRD) strategy. RRD preserves the relative distances between edge features, encouraging consistency in the relational structure rather than absolute similarity.", "questions": "[Limited Technical Novelty]\n- Is there a specific technical aspect of the proposed method that the authors believe is particularly novel or deserves emphasis?\n- In the query selection module, previous works have also used objectness scores or similar heuristics for selecting relevant queries. To validate the effectiveness of the proposed interaction-centric approach, wouldn’t it be important to conduct ablation experiments using only object similarity (e.g., $\\gamma=1$ in Eq. 1) or only interaction similarity ($\\gamma=0$)? Such experiments would help isolate the contribution of each component.\n- Also, what value of $\\gamma$ was actually used in the experiments? I cannot find the value of $\\gamma$ in both of the main paper and the supplementary and it is important for understanding the model behavior.\n\n[Lack of Reasoning Behind Design Choices]\n- Could the authors provide further justification or clarification for the design choices discussed above—particularly regarding the motivation for passive triplets in knowledge infusion and the decomposition strategy in query selection?\n\n[Missing Reference]\n- The formulation of the Relative-Interaction Retention Distillation (RRD) objective appears to be nearly identical to that of Relational Knowledge Distillation (RKD). I think it is necessary to refer the paper. \n\nMy main concern is that the technical novelty feels somewhat incremental, and the paper currently lacks sufficient reasoning to justify several of its core design decisions. I would appreciate a more thorough explanation to strengthen the motivation and clarity of the proposed approach.", "limitations": "- The paper does not explicitly discuss the limitations of the proposed approach. Including a reflection on potential weaknesses—such as failure cases—would strengthen the paper.\n- No ethical issues are apparent in the current work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n[Well-Structured Paper]\n- The paper is clearly written and easy to follow, with all mathematical formulations presented in a precise and understandable manner.\n- The ablation study is well-designed and effectively demonstrates the contribution of each module to the overall performance.\n\n[Novel Interaction-Centric Method]\n- Unlike prior works that primarily rely on object-centric approaches, this paper introduces multiple interaction-centric techniques that improve performance across various components of the pipeline.\n- The overarching motivation—infusing, transferring, and distilling interaction-aware knowledge—is coherently integrated into the proposed framework, resulting in a well-motivated and unified methodology.\n- The authors successfully adapt existing techniques to the specific demands of the task by reformulating them in an interaction-centric manner, leading to a novel and cohesive overall architecture.\n- In particular, the idea of applying interaction-centric principles to query selection is both intuitive and impactful.\n\n**Weaknesses**\n\n[Limited Technical Novelty]\n- While the proposed methods are presented in an interaction-centric framework, many of the individual components appear to be only incremental extensions of existing techniques.\n- In Interaction-Centric Knowledge Infusion, the bidirectional interaction prompt is essentially implemented by simply adding passive-voice triplets—e.g., <object–verb–subject>—which is a relatively straightforward modification.\n- In Interaction-Centric Knowledge Transfer, the query selection mechanism builds on ideas that have already been widely explored in object detection literature [1, 2] or sementic segmentation literature [3].\n- The Relative-Interaction Retention Distillation formulation is nearly identical to that of RKD (Relational Knowledge Distillation) [4], with no citation or significant deviation in formulation.\n- While repurposing these techniques in an interaction-centric manner offers some novelty, the overall contribution feels somewhat incremental, especially given the lack of deeper reasoning behind these design choices (as discussed below).\n\n[1] Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y. Liu, J. Chen, DETRs Beat YOLOs on Real-time Object Detection, CVPR 2024\n\n[2] Y. Huang, H. Liu, H. Shuai, W. Cheng, DQ-DETR: DETR with Dynamic Query for Tiny Object Detection, ECCV 2024\n\n[3] N. A. Shah, V. VS, V. M. Patel, LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation, CVPR 2024\n\n[4] W. Park, D. Kim, Y. Lu, M. Cho, Relational Knowledge Distillation, CVPR 2019\n\n[Lack of Reasoning Behind Design Choices]\n- In Interaction-Centric Knowledge Infusion\n    - It's unclear why adding passive triplets (e.g., <surfboard–held by–man>) would reduce ambiguity. \n    - If <man–hold–surfboard> is confusing, wouldn't its passive form be similarly ambiguous? \n    - Since the two expressions encode essentially the same semantics, it's unclear how adding both to the embedding space provides a meaningful enhancement.\n\n- In Interaction-Guided Query Selection, \n    - The justification for decomposing triplets into <subject–predicate> and <predicate–object> pairs is not well-supported. \n    - The paper claims that this helps reduce confusion (e.g., \"man–riding\" clarifies the ambiguity in \"man–horse\"), but the reasoning is hard to follow—especially since the decomposed pair may still retain ambiguity without full triplet context.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns in the rebuttal. The paper is sufficiently novel and interesting, so I am raising my score by one level to borderline accept.", "rebuttal": "We deeply appreciate reviewer Y5Z7 for the valuable time and constructive feedback. We provide point-to-point responses below.\n\n**Q1: Incremental extensions of existing techniques.**\n\nSorry for the misunderstanding. Our core contributions lie in proposing a **unified interaction-centric framework**, designed specifically to solve the overlooked yet critical problem of “**relation pair mismatches**” in OVSGG. We would like to clarify two crucial techniques:\n\n1. **Bidirectional Interaction Prompt in Interaction-Centric Knowledge Infusion.** Its novelty lies NOT in the linguistic/semantic transformation, but in **how it strategically leverages the attention mechanism of the grounding model's text encoder** (e.g., Grounding DINO). ****As mentioned in Line 187~190**,** in the passive form “surfboard held by man”, “surfboard” becomes the **syntactic subject**, thereby being more contextualized by “held”. We present the attention variance of the passive form and related discussion in **Q3**.\n2. **Query Selection Mechanism in Interaction-Centric Knowledge Transfer**.  Since both OvSGTR and ACC are based on Grounding DINO, **language-guided query selection** (object-centric) is used as the default. This work further proposed IGQS, an **interaction-centric selection scheme** for OVSGG:\n    - **Objective:** Traditional query selection aims to improve general object detection or segmentation. Our IGQS is explicitly designed to resolve the **interaction ambiguity** among objects inherent in SGG.\n    - **Interaction Modeling:** We devise a unique two-step mechanism for modeling interaction. It first leverages both object and **relation** semantics for an initial ranking. More importantly, the second step is a self-reflective process that uses relations predicted from an initial pass to dynamically construct “interaction-pair” prompts. This interaction-focused query refinement is a novel design for the essential “mismatched” problem in OVSGG.\n    \n    Furthermore, the ablation studies in Table 4 and the qualitative results in Figure 5 demonstrate the effectiveness of IGQS compared with the baseline (language-guided query selection in Grounding DINO). We provide more explanations of the IGQS design in **Q3**.\n    \n    Nevertheless, we agree that IGQS’s query initialization draws high-level inspiration from the DETR-series in detection and segmentation. We will add relevant discussion in the revision. Thanks!\n    \n**Q2: Missing citation of RKD.**\n\nThanks for bringing this excellent work (i.e., RKD [2]) to our attention. We would like to clarify the main difference between RKD and our RRD: \n\n- **Motivation.** While RKD and our RRD use relationship distillation, they are motivated by different core problems. The original RKD was proposed as a general method for transferring knowledge by preserving the inter-sample relations of a teacher model's output. In contrast, our RRD is specifically motivated by a key challenge in OvSGTR, distinguishing meaningful **relational foregrounds** (true novel interactions) from a vast number of **background** object pairs. Our goal is to use relationship distillation as a targeted mechanism to help the model learn the structural differences between true interactions and random co-occurrences.\n- **Technique.** Our RRD is fundamentally different from RKD in that:  The distillation loss is computed on the set of **negative samples ($\\mathcal{N}$)**. By aligning the structural similarity of these background triplets between the teacher and student, we explicitly encourage the student model to push the embeddings for **true, positive relations** away from the dense space of irrelevant **background** pairs.\n\nIn addition, we conduct an experiment to empirically validate the advantage of our RRD. Concretely, we implemented RKD loss within our framework. As seen from **Table_R 3**, our RRD can get higher results compared with RKD. We will add these discussions and cite this work in the revision.\n\n**Table_R 3: Comparison of RKD [2] and RRD (ours) on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).**\n\n|Method|R@20|R@50|R@100|\n|-|-|-|-|\n|RKD|10.36|14.45|17.84|\n|RRD|11.43|15.67|19.20|\n\n[2] W. Park, D. Kim, Y. Lu, M. Cho. Relational Knowledge Distillation. CVPR 2019.\n\n**Q3: Deeper reasoning behind the design choices.**\n\n1. **Passive triplets.** The benefit stems not from new semantic information but from the **asymmetry of the attention mechanism** within the text encoder (e.g., BERT). In the passive form “surfboard held by man”, “surfboard” becomes the syntactic subject, which **absorbs richer contextual semantics from the predicate** (e.g., “held”). This more discriminative textual feature is used to perform cross-attention with the visual patches from the image, which directly improves grounding accuracy. To empirically validate this, we display the attention scores of the active phrase (**Table_R 4a**) and the passive phrase (**Table_R 4b**). As seen, the subject in the passive phrase absorbs more from the predicate (”held”).\n\n    **Table_R 4a: Attention score across different tokens in “man hold surfboard”.**\n\n    |token | man | hold | surf | ##board |\n    |-|-|-|-|-|\n    |man|0.0891|**0.2062**|0.0307|0.0448|\n    |surf|0.0125|0.0160|0.0589|0.2721|\n    |##board|0.0227|0.0324|0.1365|0.0724|\n\n    **Table_R 4b: Attention score across different tokens in “surfboard held by man”.**\n\n    |token|surf|##board|held|by|man|\n    |-|-|-|-|-|-|\n    |man|0.0395|0.0352|0.0514|0.0495|0.0622|\n    |surf|0.0706|0.2585|0.0252|0.0396|0.0186|\n    |##board|0.1642|0.0768|**0.2096**|0.0363|0.0212|\n\n2. **Decomposed strategy in query selection.** The core motivation for decomposing triplets is to **prevent the subject and object queries from interfering with each other** during the initialization phase, ensuring both are properly localized. When using a full <subject, predicate, object> prompt (e.g., “man hold racket”) to guide query selection, the initial queries concentrate on the **union box** of interacting pairs, leading to the **omission of relatively small objects within the union box**. To provide clear evidence, we visualize the query initialization results of the first image in Figure 5 under three different prompts (<subject, predicate, object>, <subject, predicate>, <predicate, object>), and count the number of queries that successfully matched GT boxes (IoU > 0.5). \n    - The composed prompt “man hold racket” led to queries focusing on the **union box** of the man and the racket (8 matched “man”, 3 matched “racket”).\n    - The decomposed subject-predicate prompt “man hold” focused queries on the **subject region** (10 matched “man”).\n    - The predicate-object prompt “hold racket” successfully focused queries on the **object region** (9 matched “racket”).\n    \n    This analysis shows that our decomposed strategy is crucial for more accurate query initialization for both subjects and objects. **Since including an external link in the rebuttal is forbidden, we will add the visualization of query comparison in the revision.**\n    \n    In addition, we conduct an experiment to study the effect of the decomposed strategy. As shown in **Table_R 5**, our decomposed strategy can lead to higher performance.\n    \n\n    **Table_R 5: Ablation study on the effect of the decomposed strategy on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).**\n\n    | Decomposed Strategy | R@20 | R@50 | R@100 |\n    | --- | --- | --- | --- |\n    | ✕ | 10.89 | 15.17 | 18.72 |\n    | ✓ | 11.37 | 15.71  | 19.37 |\n\n**Q4: Technical novelty.**\n\nOur main novelty and contributions lie in proposing a **unified interaction-centric framework** designed to solve the overlooked problem of “**relation pair mismatches**” in OVSGG. The innovation is not just an isolated component, but a cohesive, complete solution that synergistically addresses this mismatch problem:\n\n- During the **knowledge infusion stage**, we enhance the quality of weak supervision signals through our bidirectional interaction prompts.\n- During the **knowledge transfer stage**, we devise interaction-guided query selection and interaction-consistent knowledge distillation to ensure the model focuses on true interactions while retaining its generalization capabilities during fine-tuning.\n\nThis end-to-end, interaction-centric paradigm, from data generation to model optimization, represents our unique contribution to OVSGG.\n\n**Q5: Ablation study of object similarity and interaction similarity.**\n\nThanks for the excellent suggestion. To study the contributions of object and relation similarities in our initial query selection, we conduct a detailed ablation study on the weight hyperparameter $γ$ (cf. Eq. 1). The results in **Table_R 6** clearly show that combining both similarities (0 < $γ$ < 1) outperforms using either component alone. Performance peaks at **$γ$**=0.7 (our default implementation), empirically validating our core hypothesis, i.e., incorporating relational context is critical for identifying the most relevant and interacting object candidates, leading to superior performance.\n\n**Table_R 6: Ablation study of similarity weight $γ$  on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).**\n\n|**$γ$**|R@20|R@50|R@100|\n|-|-|-|-|\n|0.0|10.02|13.50|16.37|\n|0.3|11.10|15.30|18.63|\n|0.5|11.21|15.43|19.04|\n|0.7|11.30|15.71|19.16|\n|1.0|10.68|14.17|17.83|\n\n**Q6: Limitations.**\n\nThanks for your helpful suggestion. We discussed the limitations in **Appendix G:** \n\n*“it also inherits inductive biases from the teacher model. Like two sides of a coin, any biases in the vision-language model toward specific feature traits or classes may propagate to our model. Besides, our method can alleviate mismatched relational pairs, but cannot avoid all mismatches.”* \n\nWe will add failure cases and discussion in the revision. For analysis of failure cases, please refer to **Q7** in **Reviewer E7mD.**"}, {"review_id": "vYbtSmsoXp", "reviewer": "Reviewer_9mXA", "summary": "The paper introduces ACC — an interACtion-Centric open-vocabulary scene graph generation (OVSGG) framework to enhance the OvSGTR open‐world scene graph detector by adding three interaction‐centric modules to reduce mismatched object–relation pairs: (1) Bidirectional interaction prompt: For each ⟨subject, predicate, object⟩ triplet, the paper generates “object → subject” (“surfboard held by man”) prompts via an LLM to deepen interaction semantics in addition to the “subject → object” (e.g. “man holds surfboard”) interaction. (2) Interaction‐guided query selection: The paper scores predicted visual tokens against object and relation class embeddings — keeping only high‐relevance queries and enriching them with contextual relation cues. (3) Interaction‐consistent knowledge distillation: A student–teacher distillation that not only transfers relational knowledge but also explicitly separates interacting object pairs from background noise.", "questions": "1. What is the performance of the model in comparison to the baselines on out-of-distribution data?\n2. How much additional computational overhead does ACC incur compared to OvSGTR?\n3. Could you please report the results that were promised (mR@K, std. deviation) but seem to be absent from the paper?\n4. Could you illustrate using examples or statistics where ACC misidentifies non-interacting object pairs and point to which scene contexts or relation types most frequently lead to those errors?\n5. Additional recent baselines for comparison would strengthen the paper.\n\nIf the authors address these points (with particular emphasis on points 1 to 3), I would be happy to raise my score unless grave concerns are being brought up by other reviewers.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper is well motivated, very well written, and the results look promising.\n* The technical contributions address timely and relevant issues in scene graph generation for improving relation prediction in multimodal LLMs for real-world deployment. I particularly like the bidirectional interaction prompt, which is simple yet rather effective.\n* An evaluation on three benchmark datasets shows that ACC consistently outperforms the OvSGTR baseline. Careful ablations show the benefits of the individual components.\n\nWeaknesses:\n* The paper claims (l. 284f) that it reports the mR@K metric, which would be quite important to assess long-tail capabilities. but I could not find mR@K numbers in the tables of the main paper or the appendix.\n* Similarly, the answer to question 7 in the checklist (l. 659) claims that standard deviations are being reported, but I could not find these in any of the tables.\n* Incorporating contextual information likely increases the computational overhead, yet the paper doesn’t discuss this despite claiming otherwise in the checklist (question 8). The appendix only mentions which GPUs where used, but as far as I could see not how long training and inference take.\n* Only three recent baselines are being compared with in Tables 1 and 2. The remaining baselines are 5+ years old. More comparisons with recent baselines would clearly strengthen the paper.\n* It would be insightful to test ACC versus OvSGTR on out-of-distribution data (e.g., PSG or Open Images) to further evaluate generalization.\n* The key notations introduced in the methodology are not defined in Figure 2, making it harder to connect the text to the overall pipeline.\n\nMinor points:\n* \"ACC achieves R@100 over OvSGTR...\" - incomplete sentence.\n* Figure 5 is quite interesting. Having quantitative results that support this analysis would make the paper stronger.\n* Reference [39] is broken.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns as well as those of the other reviewers. I do not see any remaining critical concerns, hence I am raising my score by one notch. That said, my score should be read as somewhere between 4 and 5 as do believe that that the sum of all the added experiments and clarifications especially regarding the motivation are quite substantial. This means that the final paper will need to be rather different than the paper we reviewed. Whether this is too big of a change, I will leave up to the AC to decide, also in comparison to other papers.\nOverall, while my assessment is positive, it is not a completely clear case given the substantial amount of added information at the rebuttal stage. If the paper is accepted, I count on the authors that all promised additions will be incorporated.", "rebuttal": "We deeply appreciate reviewer 9mXA for the valuable time and constructive feedback. We provide point-to-point responses below.\n\n**Q1: Missing mR@K metric.**\n\nOur apologies. We have reported both recall and mean Recall in **Table_R 7**. It can be seen that our ACC outperforms the previous SOTA method (i.e., OvSGTR) in both metrics. \n\n**Table_R 7:  Experimental results of OvD+R-SGG setting on the VG dataset**.\n\n| Method |  Joint Novel + Base R@20/50/100 | Joint Novel + Base mR@20/50/100 | Novel (Obj) R@20/50/100 | Novel (Obj) mR@20/50/100 | Novel (Rel) R@20/50/100 | Novel (Rel) mR@20/50/100 |\n| --- | --- | --- | --- | --- | --- | --- |\n| OvSGTR (Swin-T) | 10.02 / 13.50 / 16.37 | 1.81 / 2.54 / 3.15 | 10.56 / 14.32 / 17.48 | 1.69 / 2.44 / 3.06 | 7.09 / 9.19 / 11.18 | 0.82 / 1.13 / 1.47 |\n| ACC (Swin-T) | 12.61 / 17.43 / 21.27 | 2.09 / 3.02 / 3.80 | 12.48 / 17.16 / 21.10 | 1.93 / 2.84 / 3.61 | 11.38 / 15.90 / 19.46 | 1.64 / 2.59 / 3.38 |\n\n**Q2: Standard deviations.**\n\nThanks for your keen observation. We have repeated the main experiments five times, and the results (**Table_R 8**) demonstrated high robustness across runs — the standard deviations range from 0.01 to 0.03. Due to the consistently low variance, we initially omitted the standard deviations from the main tables for clarity. However, we acknowledge that this information should have been included in the appendix. In the final version, we will add a dedicated table in the appendix, reporting both the mean and standard deviation for all key results.\n\n**Table_R 8: The 𝐾-times experiment of ACC in OVD+R SGG setting (Joint Base+Novel).**\n\n| No. | R@20 | R@50 | R@100 |\n| --- | --- | --- | --- |\n| 1 | 12.61 | 17.43 | 21.27 |\n| 2 | 12.59 | 17.37 | 21.27 |\n| 3 | 12.64 | 17.45 | 21.28 |\n| 4 | 12.67 | 17.43 | 21.26 |\n| 5 | 12.63 | 17.40 | 21.29 |\n| Avg | 12.628 | 17.416 | 21.274 |\n| Std | 0.0303315 | 0.03130495 | 0.01140175 |\n\n**Q3: Computational overhead.**\n\nGood suggestion!  We conduct a time analysis on VG, with training on the entire dataset and testing on 20 images. We report the mean value in **Table_R 9** of our ACC (w/o and with Step II in IGQS). We would like to claim that: 1) Our Step I in IGQS just introduces minor computational complexity in elementary matrix operations (cf. Eq. 1).  1) Due to the requirement of forward prediction for self-enhancement, Step II will induce extra computational overhead, but the performance gain brought by Step II is optional. \n\n**Table_R 9:  Inference costs on the VG dataset under the OVD+R SGG setting (Joint Base & Novel).**\n\n| Method | Training costs (min) |  Inference costs (s / I) | R@20 | R@50 | R@100 |\n| --- | --- | --- | --- | --- | --- |\n| OvSGTR (Swin-T) | 68 | 0.3871220016479492 | 10.02  | 13.50 | 16.37 |\n| ACC (Swin-T) | 71 | 0.3896771125793457 | 12.35 | 17.12 | 20.96 |\n| ACC (Swin-T) w/ Step II | 94 | 0.6402182579040527 | 12.61  | 17.43  | 21.27 |\n\n**Q4: More comparisons with recent baselines.**\n\nTo address the reviewer's concern, we have included a comparison with a newly released method, OwSGG [3] (June 9, 2025), which adopts a multimodal prompting and embedding alignment strategy to enable pre-trained models such as LLaVA-Next and Qwen to generate scene graphs under an open-world setting.\n\nWe report its performance on the VG dataset under two evaluation protocols: 1) the Open-vocabulary Relation (OvR) setting, and 2) the Open-vocabulary Detection + Relation (OvD+R) setting. The comparison results are shown below:\n\n**Table_R 10a: More experimental comparison results of OvR-SGG setting on VG test set.**\n\n| Method | Novel (Rel) R@50 | Novel (Rel) R@100 |\n| --- | --- | --- | \n| OwSGG (LLaVA-next) | 2.33 | 3.04 |\n| OwSGG (Qwen7b) | 1.15 | 1.67 |\n| OwSGG (Qwen72b) | 2.19 | 3.06 |\n| ACC (Swin-T) | 17.89 | 21.70 |\n\n **Table_R 10b: More experimental comparison results of OvD+R-SGG setting on VG test set.**\n\n| Method | Novel (Obj) R@50 | Novel (Obj) R@100 | Novel (Obj) R@50 | Novel (Obj) R@100 |\n| --- | --- | --- | --- | --- | \n| OwSGG (LLaVA-next) | 2.37 | 3.07 | 2.33 | 3.04 |\n| OwSGG (Qwen7b) | 0.87 | 1.28 | 1.15 | 1.67 |\n| OwSGG (Qwen72b) | 1.88 | 2.73 | 2.19 | 3.06 |\n| ACC (Swin-T) | 17.16 | 21.10 | 15.90 | 19.46 |\n\n[3] Dutta, Amartya, et al. Open World Scene Graph Generation using Vision Language Models. CVPR Workshop 2025.\n\n**Q5: ACC vs. OvSGTR on out-of-distribution data.**\n\nThank you for your valuable suggestion. We have tested both the OvSGTR and ACC models, trained on the VG dataset, on out-of-distribution datasets including PSG (**Table_R 11a**) and HICO-DET (**Table_R 11b**). The results are presented in the tables below:\n\n**Table_R 11a: Test on out-of-distribution PSG dataset.**\n\n| Method | R@20 | R@50 | R@100 |\n| --- | --- | --- | --- |\n| OvSGTR (Swin-T) | 11.30 | 13.87 | 15.81 |\n| ACC (Swin-T) | 12.10 | 15.19 | 17.23 |\n\n**Table_R 11b: Test on out-of-distribution HICO-DET dataset.**\n\n| Method | R@20 | R@50 | R@100 |\n| --- | --- | --- | --- |\n| OvSGTR (Swin-T) | 22.32 | 23.90 | 24.89 |\n| ACC (Swin-T) | 24.53 | 26.02 | 26.84 |\n\nAs seen, our ACC demonstrates a striking performance advantage, significantly and consistently outperforming OvSGTR. This result provides strong evidence for the superior generalization capability of our approach, proving its effectiveness extends well beyond the original training domain. We will incorporate these compelling new results into the revised manuscript.\n\n**Q6: Key notations in Figure 2.**\n\nThanks for your helpful suggestion. **Due to the limitation of rebuttal guidelines, we cannot include an external link / figure here.** We will add refined Figure 2 with key notations in the revision.\n\n**Q7: Typical failure scenarios.**\n\nGood suggestion! We analyze the examples where ACC misidentifies non-interacting object pairs, and find that: \n\n1. For Interaction-Centric Knowledge Infusion, it is difficult to correctly match **small objects** and their related objects through bidirectional interaction prompts. \n2. For Interaction-Centric Knowledge Transfer, when **multiple subject-object pairs with the same relational triplet categories** appear in the same image, the model might mistakenly match the subject in one triplet to the object in another triplet. \n\n**Since including an external link in the rebuttal is forbidden, we will add the visualization and discussion of failure cases in the revision.**\n\n**Q8: Minor points.**\n\nThanks for your insightful comments. We will carefully refine the revision by:\n\n1. Completing the sentence: \"ACC achieves 29.28% R@100, which is higher than OvSGTR across both base and novel relations.\"\n2. Adding quantitative results of Figure 5. We count the number of queries that match the corresponding GT's bbox (IoU > 0.5). The query quantities of matched GT's bbox in Figure 5 are 5 (original) *vs.* 10 (holding-guided) and 4 (original) *vs.* 11 (laying on-guided), respectively. It proves the effectiveness of our IGQS strategy in query initialization.\n3. Correcting all reference formatting issues."}, {"review_id": "Mehxg8KC0X", "reviewer": "Reviewer_E7mD", "summary": "This paper aims to solve Open Vocabulary Scene Graph Generation (OvSGG) task, where novel objects and their relationships are predicted by SGG models. This paper shifts from object-centric to interaction-centric paradigm in OVSGG. The authors proposes bidirectional interaction prompts and interaction-guided query selection to learn subtle visual feature differences between interacting vs. non-interacting instances within the same object category.", "questions": "- Can you incorporate the effect of bidirectional prompt training in generalization study? It seems that the effectiveness can be demonstrated by directly comparing the performance of GLIP and GroundingDINO. I would like to see the effect in the object detection-level, not the relationship-prediction-level.\n\n- If we train the student model with the fully annotated SGG dataset, how can we obtain the open vocab prediction ability? Does the model predict the relationships in the training labels better? (Compared to unseen classes.)", "limitations": "While overall motivation is novel and the proposed method is simple and effective, there are some ambiguous points in the paper. For example, the distillation paradigm is difficult to understand. Why is it necessary for OvSGG.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength\n\n**Potential for Universal Grounding Enhancement.** Learning the interaction-aware visual representation is very important and technically sound. Current grounding methods like GLIP and GroundingDINO has restricted ability of awarding the interactions. The proposed method, bidirectional interaction prompt, could be a fundamental technique to improve all vision-language grounding models (GLIP, Grounding DINO, etc.) beyond just SGG. While the paper focuses on SGG, the approach has broader applicability that wasn't fully explored.\n\n**Clear Problem Formulation.** The authors explicitly models subtle but crucial interaction context differences like \"man holding surfboard\" vs \"man just standing\". Solving the problem through learned representations with new prompts is very novel idea.\n\n**Promising Experimental Result.** The experimental results show that using bidirectional interaction prompt for GroundingDINO greatly improves OvSGG performances on both base and novel relationship classes, achieving 2-5% increases of mR@K. Moreover, the proposed method ACC is fully open vocabulary SGG method, which not only generates unseen relationships but also predicts unseen objects during training, which is demonstrated by the OvD+R-SGG experiment. This implies that ACC is a general important approach that understands the scene without any predefined classes for objects and relationships.\n\n## Weakness\n\n**The necessity of ICKD is difficult to understand.** I cannot catch why student model distillation is necessary since we can train the original models with VRD. The advantage of introducing student model should be clarified in terms of the open vocabulary setting.", "was_revised": true, "final_justification": "After the authors' rebuttal phase, my concerns about the ambiguous point on distillation paradigm have been addressed. Moreover, I have the positive view on this paper, as the proposed ACC improves the generalization of ability of pre-trained Vision Language Model using interaction-centric model training. During the rebuttal phase, authors provided the experiment about the effectiveness of bidirectional prompt on object detection task, demonstrating the broad impact of this paper not only for visual relationship detection, but also the general object detection task. Hence, I am inclined to accept this paper.", "rebuttal": "We deeply appreciate reviewer E7mD for the valuable time and constructive feedback. We provide point-to-point responses below.\n\n**Q1: Generalization study of bidirectional prompt training.**\n\nThank you for your valuable suggestion. To validate the effect of our bidirectional prompt on the model's generalization ability, we conduct a direct ablation study at the object detection level. The results in **Table_R 12** clearly show that using our bidirectional prompt leads to a significant improvement across all key AP metrics. We will incorporate these results into the revised manuscript.\n\n**Table_R 12: Object detection results on the VG dataset.**\n\n| **Bidirectional prompt** |  AP  | AP50 | AP75 | APs | APm | APl |\n| --- | --- | --- | --- | --- | --- | --- |\n| ✕ | 17.91 | 29.20 | 18.17 | 7.19 | 14.92 | 23.63 |\n| ✓ | 19.46 | 31.75 | 19.72 | 7.79 | 16.20 | 25.42 |\n\n**Q2: Student model with the fully annotated SGG dataset.**\n\nThank you for the helpful comments. We would like to clarify the following points:\n\n- **Open-Vocabulary Ability.** This capability originates from the model's fundamental **vision-language** architecture. Unlike a traditional classifier with a fixed output category, our model works by computing a **similarity score between visual features and the text embedding** of ***any*** potential relation. This enables it to perform zero-shot predictions for “novel” classes that were never seen during training.\n- **Performance of Base Classes.** Yes, if fine-tuned on the SGG dataset with only standard SGG losses (w/o KD), the model would indeed learn the “base” relations in the training set very well compared to unseen classes (cf. **row 2 in Table_R 13**). But this comes at a huge cost: the model would **catastrophically forget** its general knowledge, leading to a severe collapse in its ability to recognize “novel” classes. This overfitting to base classes is the central challenge in OvSGG, leading directly to why the teacher-student distillation paradigm is not just helpful, but necessary for OvSGG (cf. **row 3 in Tabel_R 13**).\n\n**Table_R 13: Ablation study on fully SGG annotation and ICKD under OvR-SGG setting.**\n\n| Fully annotated SGG dataset | ICKD | Base (Rel) R@20 | Base (Rel) R@50 | Base (Rel) R@100 | Novel (Rel) R@20 | Novel (Rel) R@50| Novel (Rel) R@100|\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ✕ | - | 1.65 | 2.43 | 3.29 | 12.32 | 16.87 | 20.67 |\n| ✓ | ✕ | 25.41 | 32.56 | 36.68 | 0.91 | 1.61 | 2.34 |\n| ✓ | ✓ | 20.28 | 25.94 | 30.12 | 12.90 | 17.89 | 21.70 |\n\n**Q3: The necessity of ICKD.**\n\nThank you for this question. We would like to clarify that **VRD** also follows a typical **teacher-student** paradigm, which does not operate on a single model in isolation:\n\n- **The Teacher Model** is from the large-scale pre-training (knowledge infusion). It possesses rich, generalizable knowledge that can be applied to **novel categories**.\n- **The Student Model** is initialized with the teacher's parameters, and it is then fine-tuned on the fully-annotated SGG dataset (knowledge transfer).\n\nIn the open-vocabulary setting, if we were to fine-tune the pre-trained model directly (i.e., w/o a teacher's guidance), it would easily overfit to the “base classes” in the SGG dataset. This would severely degrade or even erase its ability to recognize “novel categories”—this is **catastrophic forgetting**. The teacher-student distillation framework acts as a regularizer. The teacher guides the student, forcing its feature space to remain consistent with the teacher's general feature space even as it learns the new task. This preserves the invaluable generalization ability. Within this teacher-student framework, we devise ICKD to optimize the knowledge transfer process:\n\n- **VRD,** ensures the student retains the teacher's understanding of “background” relations (i.e., negative samples), which is fundamental for retaining general knowledge.\n- **RRD,** not only transfers knowledge but also teaches the student the “**relative structure”** among these background relations. This helps the student build a more structured concept of the background, enabling it to more sharply distinguish true novel interactions (the foreground) from the background.\n\nWe have performed an ablation study to provide the definitive proof. As shown in **Table_R 13** and **Sec. E.4 in the Appendix,** when our ICKD is removed, performance on novel relations collapses. This empirically demonstrates that distillation is the critical component that prevents the model from overfitting to base classes, thereby unlocking its true and effective open-vocabulary generalization power."}]

Author rebuttal:

Meta-review: This paper presents an interaction-centric open-vocabulary scene graph generation (OVSGG) framework to better handle the pervasive mismatch between interacting and non-interacting object pairs. It introduces three key modules: bidirectional interaction prompts, interaction-guided query selection, and interaction-consistent knowledge distillation, showing improved performance on novel objects and relationships. 

The paper received mixed reviews with clear strengths and weaknesses. The reviewers appreciated the novel interaction-centric approach to open vocabulary scene graph generation, achieving promising experimental results that consistently improved over the baselines. The clear presentation, well-motivated approach, and unified framework with effective ablation studies were highlighted as positives.
However, the reviewers raised significant concerns about: 1) limited technical novelty, as many components appear to be incremental extensions of existing techniques without proper citations, particularly the Relative-Interaction Retention Distillation; 2) insufficient evaluation, including missing metrics (mR@K despite claims), absence of standard deviations, limited comparison with recent baselines, and no out-of-distribution testing; 3) lack of clear reasoning behind key design choices, such as why adding passive triplets or implementing student model distillation; 4) no discussion of computational overhead.

During the author-reviewer discussion, the authors provided a detailed response including further clarification on design choices and novelty relative to related work, additional evaluations with mR@K metrics, comparisons to recent methods, OOD testing, and analysis of computational overhead. The rebuttal addressed most of the reviewers' concerns, and three reviewers responded positively to the discussion. Specifically, Reviewers 9mXA and E7mD recommended acceptance, while the other two reviewers maintained a positive assessment, conditioned on including the additional information from the rebuttal.

The AC largely concurs with the reviewers' assessment: the paper introduces an interesting and effective approach to OVSGG, and its merits outweigh its weaknesses. The AC also notes that while the added experiments and clarifications from the rebuttal stage are substantial, they can be incorporated in revision. Therefore, the AC recommends acceptance. The authors should revise the manuscript to address the reviewers' feedback and incorporate the points discussed in the rebuttal.
Decision: Accept (poster)