Title: Compile Scene Graphs with Reinforcement Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: scene graph generation

multimodal LLMs

reinforcement learning
Abstract: Next token prediction is the fundamental principle for training large language models (LLMs), 
and reinforcement learning (RL) further enhances their reasoning performance.
As an effective way to model language, image, video, and other modalities, 
the use of LLMs for end-to-end extraction of structured visual representations, such as scene graphs, remains underexplored.
It requires the model to accurately produce a set of objects and relationship triplets, rather than generating text token by token.
To achieve this, 
we introduce  R1-SGG, 
a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset and subsequently refined using reinforcement learning  to enhance its ability to generate scene graphs in an end-to-end manner.
The SFT follows a conventional prompt-response paradigm, 
while RL requires the design of effective reward signals. 
We design a set of graph centric rewards, including three recall based variants—Hard Recall, Hard Recall+Relax, and Soft Recall—which evaluate semantic and spatial alignment between predictions and ground truth at the object and relation levels.
A format consistency reward further ensures that outputs follow the expected structural schema.
Extensive experiments on the VG150 and PSG benchmarks show that R1-SGG substantially reduces failure rates and achieves strong performance in Recall and mean Recall, surpassing traditional SGG models and existing multimodal language models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes reinforcement learning for fine-tuning multimodal large language models for the task of scene graph detection. They evaluate the proposed method on the VG150 and PSG datasets and show improved performance compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The proposed reinforcement learning method improves over the basic supervised fine-tuning and outperforms the listed baselines.\n+ Employing multimodal large language models for the scene graph generation/detection task is interesting and well-motivated.\n\nWeaknesses:\n- The evaluation is very limited and does not follow the previous work (mean recall at 20/50/100). The paper also lacks comparison to recent SOTA from 2024, which have comparable or better performance to the proposed method.\n- The novelty is limited and incremental (combination of [32,28]). Instead of supervised fine-tuning, the baseline LLM (Qwen2-VL-2b/7b-Instruct) [32] is trained using reinforcement learning [28].\n- The results are inconsistent and differ from the results reported in previous work. Specifically, many works evaluate on the VG150 test set, while this work evaluates it on the validation set. This makes it incomparable to many related works. In Tab. 2, on the PSG dataset, the reported results for PSGFormer are on mR@20, which is not mentioned in the paper. Since the other reported results are different from the ones reported in other works, it is not clear whether those values are also on mR@20 or not. \n- There is a large gap in the number of parameters between the LLM and non-LLM based methods (Millions vs. Billions). Also the authors do not report the number of parameters for the non-LLM based methods, which have been reported and are available in the original works.\n- There are some missing related works that are neither cited nor compared against:\n[a] Lorenz, Julian, et al. \"A Fair Ranking and New Model for Panoptic Scene Graph Generation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[b] Zhou, Zijian, Miaojing Shi, and Holger Caesar. \"VLPrompt: Vision-language prompting for panoptic scene graph generation.\" arXiv preprint arXiv:2311.16492 (2023).\n\nMinor:\n- The authors' response to Q16 on the NeurIPS paper checklist asking about using LLM's is \"No\" and is said to be only for assistance for writing and proofreading, while LLMs are part of the main methodology.\n- The authors mentioned they have discussed the limitations in Sec 4.5. However, there are no discussions on the limitations of the proposed method in this section.\n\nQUESTIONS:\n- Please clarify the exact experimental settings for the results reported in Tab. 1, Tab. 2.\n- Please clarify the contributions other than training Qwen2-VL [32] using RL [28].\n- How does the method perform on VG150 test set compared to the SOTA.\n\nLIMITATIONS:\nNo. There are no discussions on the limitations or negative societal impacts. I suggest the authors discuss the limitations of MLLMs and adapting them for the task of scene graph detection/generation, and potential limitations in ambiguous or over-confident responses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-05-31"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an RL framework for the scene graph generation task. The proposed R1-SGG utilizes three graph-centric rewards to align the prediction with the location and relation ground truth. Furthermore, the format consistency reward is proposed to guide the model output to the expected structural schema. The experiments are conducted on two SGG datasets, and the authors compare R1-SGG with specific models, MLLMs, and the SFT baseline.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The research problem is interesting.\n- This paper is well-written and easy to follow.\n\n# Weaknesses\n- The necessity of using RL for SGG is not convincingly demonstrated by the provided results. First, the AP and recall metrics are lower than those achieved by task-specific models in VG150, which require significantly lower training and inference costs. Second, although the authors present experiments on domain generalization in Table 3, the RL-based approach does not yield meaningful performance under this setting. I find it difficult to see its practical value in this context.\n- The format failure rate is not a particularly important issue in the SGG task, and the observation that RL achieves a lower format failure rate compared with SFT is a surprising discovery. It's confusing why this is emphasized by the authors.\n- The proposed method appears to be a straightforward application of GRPO to the SGG task, without substantial novelty in its adaptation or design.\n\nQUESTIONS:\n- The domain generalization ability is the most important reason for applying RL to SGG from my perspective. However, I'm not convinced by the results in the current manuscript. I recommend that the authors polish this part. For example, it's helpful to understand the RL for SGG by providing an analysis on why the mean recall of R1-SGG is higher than all other baselines.\n- I do not understand why the baseline with zero recall or AP, such as GPT-4o, is presented. This either indicates the baseline is not set correctly or the comparison is meaningless.\n\nLIMITATIONS:\nI do not find the discussion on limitations even in Section 4.5.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present R1-SGG, a framework that turns a multimodal LLM (Qwen2-VL 2 B/7 B) into an end-to-end scene-graph generator. Basically, they firstly do SFT to align with scene-graph JSON output using prompt-response pairs on VG150 and PSG, and then use GRPO to train with the reward, which is the Recall of the triplets and other softer variants\n\nSTRENGTHS AND WEAKNESSES:\nAdvantage:\n1.\tThis paper applies RL to the scene graph task, showing whether RL is able to assist the performance of SGG for MLLM, which has a good motivation.\n2.\tThis paper also ablate the some necessary aspects of RL, such as different reward design, computes/speed balance, sequence length.\n3.\tThe paper is well written and easy to follow.\n\nDisadvantages:\n1.\tInsufficient evaluation metric. I would like to know how well the model predicts the relation and object classes given two objects’ bounding box locations, and the prediction of the relation classes given the bounding box and object class. Seems MLLM is not convenient for such prompt structure changed evaluation? \n2.\tIn Table 4, why is soft recall better at failure rate than hard recall, but not the case for mRecall?\n3.\tDoes the <think> token really being used? Seems that the model is not using <think> token to enclose the thinking process, and attribute to the 2B/7B LLM is insufficient to follow the language format. But there seems no format reward being designed, and also larger LLM such as Qwen 32B is more capable on format/instruct following, perhaps the author could try that base model.\n4.\tSeems that the object detection capability (AP50) is still much weaker than the traditional method, as well as recall@50. There are also works that can further enhance the object detection capability of MLLM using GRPO, showing that the detection could be further improved using RL. So I wonder if it is mainly because of the detection capability is improved, or the relation understanding capability is improved. \n5.\tThe reported recall@50 is much lower than related work. Such as Tab.2 R@50 of paper [1]. \n6.     This paper shows three reward in methods, but I think that should be in ablation section.\n[1]  Learning to Generate Language-supervised and Open-vocabulary Scene Graph using Pre-trained Visual-Semantic Space, CVPR2023\n\nQUESTIONS:\nSee weekness.\n\nI think the main issue is (1) performance, MLLM is limited by object detection capability, which drives the SGG performance less than the traditional method, even if with RL. How to fill the gap would be a key issue.\n\n(2) less analysis on the thinking process using RL. The RL can work by reshaping the prob distribution of the text by rewarding, train with direct metric with RL should generally work, but the emerging of CoT would be a positive factor to improve reasoning capability. I would expect to see the CoT with RL.\n\n(3) Also, I would like the author to further illustrate why the scene graph task is still worth studying, since it is an intermediate result, not the finally output. Previous years people research Scene graph as an intermediate layer for visual representation, but now since MLLM has been so powerful, it can directly generate our target output, such as a caption, QA, instead of an intermediate layer of scene graph. And the performance of the scene graph is always limited since this data is not able to scale up naturally, and the abstraction is purely human-defined with object class and relation class, which are over-simplified.\n\nLIMITATIONS:\nThere is no limitation section of the paper. \n\nSee weakness section", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}

comments: SUMMARY:
The paper proposes a novel reinforcement-learning scene graph generation pipeline to achieve end-to-end generation. It proposes a multimodal LLM trained via supervised fine-tuning and a reinforcement learning based refinement to generate scene graphs. The design of graph centric rewards provides an effective way to evaluate the semantic and spatial alignment between predictions and GT for entities and relationships.

STRENGTHS AND WEAKNESSES:
Strengths:
1. A novel scene graph generation pipeline for language model enhanced visual perception. The proposed reinforcement-learning augmented scene graph generation shows strong perception ability against multimodal language models. 
2. The designed graph-centric rewards well evaluate the semantic and spatial alignment between predictions and GT for entities and relationships, which has not been explored before. 
3. The proposed framework enhances the perception ability for multimodal LLMs in the area of understanding adn reasoning about scene graphs. 

Weaknesses:
1. Considering that relationships in scene graphs may not be exactly matched with MLLM-generated predictions, how do the authors tackle this situation?
2. The open-domain ability should be discussed in the experimental analysis. 
3. The authors built the benchmark based on the scene graph detection task, which is one of three scene graph generation tasks.  In my opinion, the paper mainly focuses on evaluating the ability of generating object relationships rather than scene graph generation. I hope the authors can offer additional evaluation metric specifically for scene relationship generation of MLLM.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-08