Title: Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open Vocabulary Scene Graph Generation

Weakly Supervised Pre-training

Interaction-Centric Knowledge Infusion

Interaction-Centric Knowledge Transfer
Abstract: Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt a two-stage pipeline: 1) Infusing knowledge into large-scale models via pre-training on large datasets; 2) Transferring knowledge from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category. This limitation induces critical issues in both stages of OVSGG: it generates noisy pseudo-supervision from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer. To this end, in this paper, we propose an interACtion-Centric end-to-end OVSGG framework (ACC) in an interaction-driven paradigm to minimize these mismatches.  For interaction-centric knowledge infusion, ACC employs a bidirectional interaction prompt for robust pseudo-supervision generation to enhance the model's interaction knowledge. For interaction-centric knowledge transfer, ACC first adopts interaction-guided query selection that prioritizes pairing interacting objects to reduce interference from non-interacting ones. Then, it integrates interaction-consistent knowledge distillation to bolster robustness by pushing relational foreground away from the background while retaining general knowledge. Extensive experimental results on three benchmarks show that ACC achieves state-of-the-art performance, demonstrating the potential of interaction-centric paradigms for real-world applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve Open Vocabulary Scene Graph Generation (OvSGG) task, where novel objects and their relationships are predicted by SGG models. This paper shifts from object-centric to interaction-centric paradigm in OVSGG. The authors proposes bidirectional interaction prompts and interaction-guided query selection to learn subtle visual feature differences between interacting vs. non-interacting instances within the same object category.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n**Potential for Universal Grounding Enhancement.** Learning the interaction-aware visual representation is very important and technically sound. Current grounding methods like GLIP and GroundingDINO has restricted ability of awarding the interactions. The proposed method, bidirectional interaction prompt, could be a fundamental technique to improve all vision-language grounding models (GLIP, Grounding DINO, etc.) beyond just SGG. While the paper focuses on SGG, the approach has broader applicability that wasn't fully explored.\n\n**Clear Problem Formulation.** The authors explicitly models subtle but crucial interaction context differences like \"man holding surfboard\" vs \"man just standing\". Solving the problem through learned representations with new prompts is very novel idea.\n\n**Promising Experimental Result.** The experimental results show that using bidirectional interaction prompt for GroundingDINO greatly improves OvSGG performances on both base and novel relationship classes, achieving 2-5% increases of mR@K. Moreover, the proposed method ACC is fully open vocabulary SGG method, which not only generates unseen relationships but also predicts unseen objects during training, which is demonstrated by the OvD+R-SGG experiment. This implies that ACC is a general important approach that understands the scene without any predefined classes for objects and relationships.\n\n## Weakness\n\n**The necessity of ICKD is difficult to understand.** I cannot catch why student model distillation is necessary since we can train the original models with VRD. The advantage of introducing student model should be clarified in terms of the open vocabulary setting.\n\nQUESTIONS:\n- Can you incorporate the effect of bidirectional prompt training in generalization study? It seems that the effectiveness can be demonstrated by directly comparing the performance of GLIP and GroundingDINO. I would like to see the effect in the object detection-level, not the relationship-prediction-level.\n\n- If we train the student model with the fully annotated SGG dataset, how can we obtain the open vocab prediction ability? Does the model predict the relationships in the training labels better? (Compared to unseen classes.)\n\nLIMITATIONS:\nWhile overall motivation is novel and the proposed method is simple and effective, there are some ambiguous points in the paper. For example, the distillation paradigm is difficult to understand. Why is it necessary for OvSGG.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ACC — an interACtion-Centric open-vocabulary scene graph generation (OVSGG) framework to enhance the OvSGTR open‐world scene graph detector by adding three interaction‐centric modules to reduce mismatched object–relation pairs: (1) Bidirectional interaction prompt: For each ⟨subject, predicate, object⟩ triplet, the paper generates “object → subject” (“surfboard held by man”) prompts via an LLM to deepen interaction semantics in addition to the “subject → object” (e.g. “man holds surfboard”) interaction. (2) Interaction‐guided query selection: The paper scores predicted visual tokens against object and relation class embeddings — keeping only high‐relevance queries and enriching them with contextual relation cues. (3) Interaction‐consistent knowledge distillation: A student–teacher distillation that not only transfers relational knowledge but also explicitly separates interacting object pairs from background noise.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper is well motivated, very well written, and the results look promising.\n* The technical contributions address timely and relevant issues in scene graph generation for improving relation prediction in multimodal LLMs for real-world deployment. I particularly like the bidirectional interaction prompt, which is simple yet rather effective.\n* An evaluation on three benchmark datasets shows that ACC consistently outperforms the OvSGTR baseline. Careful ablations show the benefits of the individual components.\n\nWeaknesses:\n* The paper claims (l. 284f) that it reports the mR@K metric, which would be quite important to assess long-tail capabilities. but I could not find mR@K numbers in the tables of the main paper or the appendix.\n* Similarly, the answer to question 7 in the checklist (l. 659) claims that standard deviations are being reported, but I could not find these in any of the tables.\n* Incorporating contextual information likely increases the computational overhead, yet the paper doesn’t discuss this despite claiming otherwise in the checklist (question 8). The appendix only mentions which GPUs where used, but as far as I could see not how long training and inference take.\n* Only three recent baselines are being compared with in Tables 1 and 2. The remaining baselines are 5+ years old. More comparisons with recent baselines would clearly strengthen the paper.\n* It would be insightful to test ACC versus OvSGTR on out-of-distribution data (e.g., PSG or Open Images) to further evaluate generalization.\n* The key notations introduced in the methodology are not defined in Figure 2, making it harder to connect the text to the overall pipeline.\n\nMinor points:\n* \"ACC achieves R@100 over OvSGTR...\" - incomplete sentence.\n* Figure 5 is quite interesting. Having quantitative results that support this analysis would make the paper stronger.\n* Reference [39] is broken.\n\nQUESTIONS:\n1. What is the performance of the model in comparison to the baselines on out-of-distribution data?\n2. How much additional computational overhead does ACC incur compared to OvSGTR?\n3. Could you please report the results that were promised (mR@K, std. deviation) but seem to be absent from the paper?\n4. Could you illustrate using examples or statistics where ACC misidentifies non-interacting object pairs and point to which scene contexts or relation types most frequently lead to those errors?\n5. Additional recent baselines for comparison would strengthen the paper.\n\nIf the authors address these points (with particular emphasis on points 1 to 3), I would be happy to raise my score unless grave concerns are being brought up by other reviewers.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of open-vocabulary scene graph generation by proposing a method that focuses on two key aspects: improving vision-language pre-training and enhancing query selection during fine-tuning.\n\n**Interaction-Centric Knowledge Infusion**\n- During the VLM pre-training phase, the authors introduce interaction-centric supervision by incorporating both traditional <subject–active verb–object> and novel <object–passive verb–subject> triplet forms. \n- This is intended to better encode relational knowledge into the model.\n\n**Interaction-Centric Knowledge Transfer**\n- To address issues such as irrelevant query matching or matching objects without relationships (e.g., background noise), the method employs interaction-guided query selection. \n- This ensures that only queries relevant to interactions are used for object detection.\n\n**Interaction-Consistent Knowledge Distillation**\n- Instead of aligning edge features directly, the paper proposes a relative-interaction retention distillation (RRD) strategy. RRD preserves the relative distances between edge features, encouraging consistency in the relational structure rather than absolute similarity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n[Well-Structured Paper]\n- The paper is clearly written and easy to follow, with all mathematical formulations presented in a precise and understandable manner.\n- The ablation study is well-designed and effectively demonstrates the contribution of each module to the overall performance.\n\n[Novel Interaction-Centric Method]\n- Unlike prior works that primarily rely on object-centric approaches, this paper introduces multiple interaction-centric techniques that improve performance across various components of the pipeline.\n- The overarching motivation—infusing, transferring, and distilling interaction-aware knowledge—is coherently integrated into the proposed framework, resulting in a well-motivated and unified methodology.\n- The authors successfully adapt existing techniques to the specific demands of the task by reformulating them in an interaction-centric manner, leading to a novel and cohesive overall architecture.\n- In particular, the idea of applying interaction-centric principles to query selection is both intuitive and impactful.\n\n**Weaknesses**\n\n[Limited Technical Novelty]\n- While the proposed methods are presented in an interaction-centric framework, many of the individual components appear to be only incremental extensions of existing techniques.\n- In Interaction-Centric Knowledge Infusion, the bidirectional interaction prompt is essentially implemented by simply adding passive-voice triplets—e.g., <object–verb–subject>—which is a relatively straightforward modification.\n- In Interaction-Centric Knowledge Transfer, the query selection mechanism builds on ideas that have already been widely explored in object detection literature [1, 2] or sementic segmentation literature [3].\n- The Relative-Interaction Retention Distillation formulation is nearly identical to that of RKD (Relational Knowledge Distillation) [4], with no citation or significant deviation in formulation.\n- While repurposing these techniques in an interaction-centric manner offers some novelty, the overall contribution feels somewhat incremental, especially given the lack of deeper reasoning behind these design choices (as discussed below).\n\n[1] Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y. Liu, J. Chen, DETRs Beat YOLOs on Real-time Object Detection, CVPR 2024\n\n[2] Y. Huang, H. Liu, H. Shuai, W. Cheng, DQ-DETR: DETR with Dynamic Query for Tiny Object Detection, ECCV 2024\n\n[3] N. A. Shah, V. VS, V. M. Patel, LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation, CVPR 2024\n\n[4] W. Park, D. Kim, Y. Lu, M. Cho, Relational Knowledge Distillation, CVPR 2019\n\n[Lack of Reasoning Behind Design Choices]\n- In Interaction-Centric Knowledge Infusion\n    - It's unclear why adding passive triplets (e.g., <surfboard–held by–man>) would reduce ambiguity. \n    - If <man–hold–surfboard> is confusing, wouldn't its passive form be similarly ambiguous? \n    - Since the two expressions encode essentially the same semantics, it's unclear how adding both to the embedding space provides a meaningful enhancement.\n\n- In Interaction-Guided Query Selection, \n    - The justification for decomposing triplets into <subject–predicate> and <predicate–object> pairs is not well-supported. \n    - The paper claims that this helps reduce confusion (e.g., \"man–riding\" clarifies the ambiguity in \"man–horse\"), but the reasoning is hard to follow—especially since the decomposed pair may still retain ambiguity without full triplet context.\n\nQUESTIONS:\n[Limited Technical Novelty]\n- Is there a specific technical aspect of the proposed method that the authors believe is particularly novel or deserves emphasis?\n- In the query selection module, previous works have also used objectness scores or similar heuristics for selecting relevant queries. To validate the effectiveness of the proposed interaction-centric approach, wouldn’t it be important to conduct ablation experiments using only object similarity (e.g., $\\gamma=1$ in Eq. 1) or only interaction similarity ($\\gamma=0$)? Such experiments would help isolate the contribution of each component.\n- Also, what value of $\\gamma$ was actually used in the experiments? I cannot find the value of $\\gamma$ in both of the main paper and the supplementary and it is important for understanding the model behavior.\n\n[Lack of Reasoning Behind Design Choices]\n- Could the authors provide further justification or clarification for the design choices discussed above—particularly regarding the motivation for passive triplets in knowledge infusion and the decomposition strategy in query selection?\n\n[Missing Reference]\n- The formulation of the Relative-Interaction Retention Distillation (RRD) objective appears to be nearly identical to that of Relational Knowledge Distillation (RKD). I think it is necessary to refer the paper. \n\nMy main concern is that the technical novelty feels somewhat incremental, and the paper currently lacks sufficient reasoning to justify several of its core design decisions. I would appreciate a more thorough explanation to strengthen the motivation and clarity of the proposed approach.\n\nLIMITATIONS:\n- The paper does not explicitly discuss the limitations of the proposed approach. Including a reflection on potential weaknesses—such as failure cases—would strengthen the paper.\n- No ethical issues are apparent in the current work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-05"}

comments: SUMMARY:
The paper proposes an interaction-centric end-to-end OVSGG framework to reduce the pervasive mismatch between interacting/non-interacting object pairs. The paper introduces a bidirectional interaction prompt to facilitate visual triplet pseudo-supervision generation, therefore achieving interaction-centric knowledge infusion. Besides, it constructs interaction-guided query selection and incorporates interaction-consistent KD to achieve the interaction-centric knowledge transfer.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. The motivation is reasonable. The problem claimed in the paper is a key challenge in scene graph generation. 
2. The paper constructed a transfer-learning-based framework, which is suitable for an open vocabulary setting.


Weaknesses:
1. Although the paper focuses on open-vocabulary scene graph generation, the evaluation on base relation should be included. 
2. Considering that the paper focuses on action-centric scene graph exploration, the authors should include a comparison with the HOI task, which focuses on human interactions. 
3. The motivation claimed in the abstract has little relationship with open vocabulary setting; the authors should claim that the challenge existed in open vocabulary scene graph generation.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-08