Scene Graph Generation with Role-Playing Large Language Models

Authors: guikun  chen
jin  li
wenguan  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:40:57

Abstract: Current approaches for open-vocabulary scene graph generation (OVSGG) use vision-language models such as CLIP and follow a standard zero-shot pipeline – computing similarity between the query image and the text embeddings for each category (i.e., text classifiers). In this work, we argue that the text classifiers adopted by existing OVSGG methods, i.e., category-/part-level prompts, are scene-agnostic as they remain unchanged across contexts. Using such fixed text classifiers not only struggles to model visual relations with high variance, but also falls short in adapting to distinct contexts. To plug these intrinsic shortcomings, we devise SDSGG, a scene-specific description based OVSGG framework where the weights of text classifiers are adaptively adjusted according to the visual content. In particular, to generate comprehensive and diverse descriptions oriented to the scene, an LLM is asked to play different roles (e.g., biologist and engineer) to analyze and discuss the descriptive features of a given scene from different views. Unlike previous efforts simply treating the generated descriptions as mutually equivalent text classifiers, SDSGG is equipped with an advanced renormalization mechanism to adjust the influence of each text classifier based on its relevance to the presented scene (this is what the term “specific” means). Furthermore, to capture the complicated interplay between subjects and objects, we propose a new lightweight module called mutual visual adapter. It refines CLIP’s ability to recognize relations by learning an interaction-aware semantic space. Extensive experiments on prevalent benchmarks show that SDSGG significantly outperforms top-leading methods.

Main paper: 
 introduction: 
 SGG [1] aims to create a structured representation of an image by identifying objects as nodes and their relations as edges within a graph. The emerging field of OVSGG [2, 3] , which broadens the scope of SGG to identify and associate objects beyond a predefined set of categories, has become a research hotspot for its prospective to amplify the practicality of SGG in diverse real-world applications.OVSGG has achieved notable progress due to the success of vision-language models (VLMs) [4, 5] and prompt learning [6, 7] . Existing OVSGG methods adopt a standard zero-shot pipeline [4] , which computes similarity between the visual embedding from query image and the text embedding from pre-defined text classifiers (c.f . Fig. 1a ). One straightforward direction for OVSGG is to use only the category name (e.g., "riding") [2, 8, 9, 3, 10] as the text classifier and perform vision-language Participants: Experts specialized in biology , physics , and engineering .Participants: Experts specialized in biology , physics , and engineering .From a biological standpoint, we can infer that the subject might be holding onto the object's back or using some form of harness for stability during the ride. ... the physics involved, there would likely be motion involved, with the subject and object moving together. ... engineering perspective, any mechanical interactions between the human and the object, like the reins.From a biological standpoint, we can infer that the subject might be holding onto the object's back or using some form of harness for stability during the ride. ... the physics involved, there would likely be motion involved, with the subject and object moving together. ... engineering perspective, any mechanical interactions between the human and the object, like the reins.? To further utilize the learned semantic space of CLIP, one can compute similarities of multiple part-level prompts (e.g., the object of ⟨man, riding, horse⟩ may be described with "with four legs" and "with a saddle"). (c) Instead of using these scene-agnostic text classifiers, SDSGG adopts comprehensive, scene-specific descriptions generated by LLMs, which can adapt to specific contexts by using the proposed renormalization.alignment as in prompt learning to learn the underlying patterns. On the other hand, [11, 12] argue that such methods fail to utilize the rich context of additional information that language can provide.To address this, [12] decomposes relation detection into several separate components, computing similarities by checking how well the visual features of the object match the part-level descriptions 1 .As shown in Fig. 1b , the object of relation "riding" should have four legs, a saddle, etc.Despite these technological advances, we still observe current OVSGG systems lack in-depth inspection of the expressive range of the used text classifiers, which puts a performance cap on them. Concretely, OVSGG models that rely on scene 2 -agnostic text classifiers have the following flaws.First, methods based on category names [2, 3] struggle to model the large variance in visual relations. Using only category names as classifiers [4] does hold water when applied to object recognition [13] .For instance, CLIP shares common visual features across diverse image-text pairs of tigers which encompasses a variety of tiger appearances and corresponding descriptions. Nonetheless, the scenario becomes far more complex when it comes to relation detection. The visual features that define the relation "on" can vary dramatically across scenes, e.g., "dog on bed" vs. "people on road". RECODE [12] proposes to decompose relation detection into recognizing part-level descriptions for both subject and object, hence partially easing the aforementioned difficulty. Yet, it computes similarities for the subject and object separately and does not model the interplay between subjects and objects. Second, part-level prompt based methods uniformly process all descriptions as affirmative classifiers [11, 12] , overlooking the possibility that some text classifiers might be contrary to specific contexts. When querying LLMs for distinctive visual features of subjects and objects to distinguish the predicate "riding", with the subject as "human" and the object as "horse", LLMs provide part-level descriptions of the expected appearance of both entities. All generated descriptions are treated equally as definitive text classifiers. However, these descriptions could potentially be misleading, as LLMs produce them without considering the specific context, even resulting in some descriptions that are wholly irrelevant to the presented image. For example, LLMs typically associate the predicate "riding" with the animal "with four legs". Nonetheless, such associations are indeed irrelevant, as an animal's legs are not always visible in the presented image, or the animal may have only two legs.Filling the gaps identified above calls for a fundamental paradigm shift: moving from the fixed, scene-agnostic (i.e., category-/part-level) text classifiers towards flexible, scene-specific ones. In light of this, we develop SDSGG, a scene-specific description based OVSGG framework that utilizes text classifiers generated from LLMs, complemented by a renormalization technique, to understand scenes from different perspectives. For the textual part, given a scene with specified content, an LLM is assigned distinct roles, akin to experts specializing in biology, physics, and engineering, to analyze descriptive scene features comprehensively (c.f . Fig. 1c ). Such a multi-persona scheme is designed to improve the diversity of the generated scene descriptions as LLMs tend to generate repetitive content. LLM can be queried multiple times to obtain a large number of scene descriptions. Moreover, since not all descriptions are relevant to the presented image (e.g., some parts of the object may not appear), SDSGG is equipped with an advanced mechanism that renormalizes each scene description via opposite descriptions corresponding to the original descriptions. This involves evaluating two vision-language similarities: one for the original scene description and the other for its opposite. The difference between the two similarities is viewed as the self-normalized similarity of the scene description, allowing for flexible control over its influence. For instance, an irrelevant description would yield a self-normalized similarity close to zero, as the two similarity scores of it and its opposite would be very close. By doing so, the generated scene-level descriptions become flexible, scene-specific descriptions (SSDs). For the visual part, we propose a new adapter for relation detection, called mutual visual adapter, which consists of several lightweight learnable modules. The proposed adapter projects CLIP's semantic embeddings into another interaction-aware space, modeling the complicated interplay between the subject and object through cross-attention.With the proposed adaptive SSDs, our SDSGG is capable of: i) adapting to the given context via evaluating the self-normalized similarity of each SSD; ii) alleviating the overfitting problem in OVSGG models [2, 3] that use only one classifier; and iii) naturally generalizing to novel relations by associating them with SSDs. We validate SDSGG on two widely-used benchmarks, i.e. Visual Genome (VG) [14] and GQA [15] . Experimental results show that SDSGG outperforms existing OVSGG methods [3] by a large margin. The strong generalization and promising performance of SDSGG evidence the great potential of scene-specific description based relation detection. 
related work: 
 Scene Graph Generation. Since [1] introduces iterative message passing for SGG, research studies in structured visual scene understanding have witnessed phenomenal growth. Tremendous progress has been achieved and can be categorized into: i) Two-stage SGG [16, 17, 18, 19, 20] , which first detects all objects in the images and then recognizes the pairwise relations between them; ii) Debiased SGG [21, 22, 23, 24, 25, 26, 27] , which focuses on the problem of long-tailed predicate distribution in the current dataset; iii) Weakly-supervised SGG [28, 29, 30, 31, 32] , which investigates how to generate scene graph with only image-level supervision; iv) One-stage SGG [33, 34, 35, 36, 37, 38] , which implements SGG within an end-to-end framework (also exemplified in other relation detection tasks [39, 40, 41] ), discarding several hand-crafted procedures; v) Open-vocabulary SGG, which learns to recognize unseen categories during training by using category-level [2, 8, 9, 3, 10] or part-level prompts [12] .Existing OVSGG frameworks adopt a standard open-vocabulary learning paradigm, i.e., perform vision-language alignment in the pre-trained or random initialized semantic space with supervision of only the category names. One except [12] reformulates OVSGG from recognizing category-level prompts into recognizing part-level prompts, by decomposing SGG into several separate components and computing their similarities independently, in a training-free manner. SDSGG represents the best of both worlds. On the one hand, we point out the drawbacks of the commonly used scene-agnostic text classifiers and introduce scene-specific alternates to understand scenes from different perspectives.On the other hand, SDSGG incorporates a learnable mutual visual adapter to capture the underlying patterns in the dataset and proposes to renormalize text classifiers for adapting to specific contexts.Open-vocabulary Learning. Most deep neural networks operate on the close-set assumption, which can only identify pre-defined categories that are present in the training set. Early zero-shot learning approaches [42, 43, 44] adopt word embedding projection to constitute the classifiers for unseen class classification. With the rapid progress of vision language pre-training [4, 5, 45] , open vocabulary learning [46] has been proposed and demonstrates impressive capabilities by, for example, distilling knowledge from VLMs [47, 48, 49, 50, 51, 52] , exploiting caption data [53, 54] , generating pseudo labels [55, 56, 57, 58, 59] , training without dense labels [60, 61, 62] , joint learning of several tasks [63, 64, 65] , and training with more balanced data [66, 67] .While sharing a very high-level idea of vision-language alignment in open-vocabulary methods, our SDSGG i) explicitly models the context-dependent scenarios and introduces scene-specific text classifiers as the flexible learning targets, and ii) incoroperates a new mechanism for computing self-normalized similarities, thereby renormalizing text classifiers according to the presented image.VLMs Meet LLMs [68] . The big win for VLMs has been all about getting the model to match up pictures and their descriptions closely while keeping the mismatched ones apart [4, 5, 45] . This trick, inspired by contrastive learning from self-supervised learning [69, 70, 71, 72, 73] [11, 12] which processes all part-level prompts as affirmative classifiers, SDSGG provides a flexible alternative via the association between classifiers (i.e., SSDs) and categories, and the renormalizing strategy w.r.t. each SSD. Since the learned semantic space of VLMs may not be sensitive to relations [12] , we design a lightweight mutual visual adapter to project them into interaction-aware space for capturing the complicated interplay between the subject and object. 
methodology: 
 Task Setup and Notations. Given an image I, SGG transforms it into a structured representation, i.e., a directed graph G = {O, R}, where O represents localized (i.e., bounding box) objects with object category information and R represents pairwise relations between objects. For a fair comparison, this work focuses on predicting R given O, i.e., the predicate classification task which avoids the noise from object detection, as suggested by [1, 3, 12] . Our study delves into the intricacies of transitioning SGG from a traditional closed-set setting to an open vocabulary paradigm. This transition enables the system to recognize previously unseen predicate categories (i.e., novel split) by learning from observed predicate categories (i.e., base split) during training.SDSGG follows the standard zero-shot pipeline with VLMs [4] , which computes similarity between the visual embedding v and the text embedding t for each category, and the category with highest similarity is viewed as the final classification result (c.f . Fig. 2a ). For each subject-object pair, v can be derived by feeding cropped patches from the input image I into the visual encoder. The text embedding t used in existing OVSGG frameworks falls into two main settings: i) Each category consists of only one text classifier, i.e., the category name itself. ii) Each category consists of multiple text classifiers w.r.t. subject and object, i.e., part-level descriptions. SDSGG reformulates the text classifiers as scene-specific descriptions which will be detailed in §3.1.Algorithmic Overview. SDSGG is a SSD based framework for OVSGG, supported by the cooperation of VLMs and LLMs. For the textual part (c.f . Fig. 2b ), SDSSG enjoys the expressive range of the comprehensive SSDs generated by LLMs' multi-persona collaboration. This is complemented by a renormalizing mechanism to adjust the influence of each text classifier. For the visual part (c.f . Fig. 2c ), SDSSG is equipped with a mutual visual adapter to aggregate visual features v from I for a given subject-object pair. After introducing how we generate and use SSDs for the text part ( §3.1) and the mutual visual adapter for interplay modeling of the subject and object ( §3.2), we will elaborate on SDSGG's training objective ( §3.3).-normalized imilarity easurement (Eq. )Visual Encoder (Eq. )K V Q Mutual Visual Adapter (MVA)Directional Marker (Eq. ) C n r = 0 ) of both the subject and object extracted from CLIP's visual encoder, our mutual visual adapter (MVA) projects them into interaction-aware space and models their complicated interplay with cross-attention.K V Q K V QC n r =-1 v t 1 p t 2 p t 3 p • • • t N p t 1 a t 2 a t 3 a • • • t N a d 1 a d 1 p d 2Scene-level Description Generation via Multi-persona Collaboration. Using standard prompts to query LLMs is a direct way to generate scene descriptions. For instance, one could straightforwardly prompt LLM with a question like "Imagine there is an animal that is eating, what should the scene look like?". LLM's response would typically sketch out an envisioned scene based on its statistical training on large corpus. However, these responses may not fully capture the scene's complexity, often overlooking aspects such as the spatial arrangement of elements and the background environment.To alleviate this, we draw inspiration from recent advances in LLMs' multi-persona capabilities [83, 84, 85, 86] . Specifically, LLM adopts three distinct roles, mirroring the expertise found in experts specializing in biology, physics, and engineering. This approach allows for a comprehensive discussion of what a given scene entails. Because each query to LLM usually only yields 3-5 sentences of description, we query LLM several times, each time giving LLM a different scene content to be discussed, thus obtaining a large number of scene descriptions. Since these initial descriptions may suffer from noise and semantic overlap, we ask LLM to streamline and combine these descriptions, ensuring more cohesive and distinct scene-level descriptionsD l = {d 1 , d 2 , • • • , d N } and corresponding text embeddings T = {t 1 , t 2 , • • • , t N },where N denotes the number of SSDs and text embeddings T are extracted by the text encoder of CLIP. Due to the limited space, we provide more details and prompts for generating scene descriptions in the appendix ( §D).Association between Scene-level Descriptions and Relation Categories. So far, we have obtained various scene-level descriptions that have the ability to represent diverse scenes. A critical inquiry arises regarding their utility for relation detection, given their lack of explicit association with specific relation categories. To address this, we delineate three distinct scenarios characterizing the interplay between relation categories and scene descriptions: i) certain coexistence (C n r = 1), where a direct correlation exists; ii) possible coexistence (C n r = 0), indicating a potential but not guaranteed association; and iii) contradiction (C n r = -1), denoting an incompatibility between the scene description and relation category. Here C n r denotes the correlation between relation r ∈ R and n th scene description, and is generated by LLMs (prompts are shown in §D). Such a categorization enables us to calculate the similarity for each relation category:EQUATIONwhere ⟨•, •⟩ denotes the cosine similarity with temperature [4] .Scene-specific Descriptions = Scene-level Descriptions + Reweighing. Upon examining text classifiers in depth, we noticed that certain classifiers are contextually bound, e.g., "two or more objects partially overlap each other" may not exist in all scenes. This observation underscores the necessity for a mechanism to evaluate the significance of each text classifier, rather than applying a uniform weight across the board. This is exactly what the term "specific" means. Recall that the similarity measurement between an image and the text "a photo of a cat" alone yields limited insight. However, when juxtaposed with multiple texts, such as "a photo of a cat/dog/tiger", the comparison of similarity scores across these categories reveals which category (cat, dog, or tiger) the image most closely resembles. Inspired by this, we propose the incorporation of an opposite description d n p ( §D) for each raw scene description d n a as a reference point (e.g., "two or more objects partially overlap each other" vs. "each object is completely separate with clear space between them"), resulting in SSDsD s = {(d 1 a , d 1 p ), (d 2 a , d 2 p ), • • • , (d N a , d N p )} and updated text embeddings T = {(t 1 a , t 1 p ), (t 2 a , t 2 p ), • • • , (t N a , t N p )}.Subsequently, the self-normalized similarity is defined as:EQUATIONThe difference in similarity scores, i.e., ⟨v, t n a ⟩-⟨v, t n p ⟩, quantifies the relative contribution of that SSD. By such means, a SSD irrelevant to the presented context will have a minimal effect, as the similarity scores of it (⟨v, t n a ⟩) and its opposite (⟨v, t n p ⟩) would be nearly identical.After introducing how to obtain the text embeddings and how to compute vision-language similarity, one question remains at this point: how to obtain visual embeddings? When given a subject-object pair with bounding boxes from O, there exist various strategies for aggregating visual features for the subject and object within I. For example, traditional closed-set SGG frameworks [16, 17] employ RoI pooling to extract visual features for specified bounding boxes, subsequently fusing these features for further classification. In contrast, the more recent OVSGG framework [12] uses the visual encoder of CLIP to extract visual embeddings of both subject and object. Then, it processes two visual embeddings independently through part-level descriptions. Such an independent approach, however, overlooks the informative interplay between the subject and object.To address this oversight and capture the complicated interactions between subject and object, we introduce a new component: the mutual visual adapter (MVA). MVA is composed of several lightweight, learnable modules designed to fine-tune CLIP's visual encoder specifically for pairwise relation detection. This approach aims to enhance the model's ability to recognize the nuanced interactions that define relationships between subject and object in an image.Regional Encoder. Given an image I and a subject-object pair with bounding boxes (b s and b o ) from O, the initial visual embeddings can be obtained from the visual encoder of CLIP:EQUATIONwhere M denotes the number of patches, Encoder v is CLIP's visual encoder that is kept frozen during training, and Crop represents image cropping.Visual Aggregator. Next, MVA is adopted to aggregate f s and f o by cross-attention and two lightweight projection modules. Let the subject part be the query, and the object part be the key and value. The patch embeddings of object f img o are first projected into low-dimensional, semantic space:EQUATIONwhere Linear denotes a standard fully connected layer. Afterwards, cross-attention is adopted to capture the complicated interplay between subject and object, resulting in an aggregated visual embedding for the given subject-object pair:EQUATION)where AvgPool is the average pooling. LN is the standard layer normarlization. CrossAttn(Q, KV ) denotes the standard cross-attention operation. v os can be computed in a similar way by exchanging the query and key of cross-attention. Combining them together leads to the final visual embedding v = (v so + v os )/2 for final similarity measurement. As such, MVA captures the interplay of subject and object in the projected, interaction-aware space.Directional Marker. One may notice that the structure of MVA is symmetric and has no information about which input branch is the subject/object. This has a relatively small effect on semantic relations, but a significant effect on geometric relations. For instance, after exchanging the location of the subject and object (image flipping), the relation "eating" remains unchanged, while the relation "on the left" would become "on the right". Here we simply incorporate two text embeddings (t s and t o ) of "a photo of subject/object" into MVA and thus update the visual embedding as:EQUATIONwhere Concate denotes concatenation. v os and v can be updated accordingly. Further exploration of directional marker, e.g., incorporating more complex feature fusion modules, is left for future work.A typical training objective in open-vocabulary learning aims to bring representations of positive pairs closer and push negative pairs apart in the embedding space. In SDSGG, the term "positive/negative pairs" is not defined at the category level but at the description level, requiring losses tailored for different relation-description association types. Given a labeled relation triplet, one simplest contrastive loss can be defined as:EQUATION)where α is a scaling factor. However, as for scene descriptions marked by possible coexistence (i.e., C n r = 0), there is no direct target that can be used for training. Inspired by the identical mapping in residual learning [87] , we make the prediction results of MVA close to those of CLIP. As such, MVA can learn the implicit knowledge embedded in CLIP's semantic space. In addition, this regularization term prevents MVA from overfitting to relations in the base split, which is a common problem in open-vocabulary learning. Hence, the loss is further reformulated as:EQUATION)where β is another scaling factor, sim CLIP (I, rel) denotes the vision-language similarity derived from the original CLIP, and λ is a constant scalar and is empirically set to 3e-2.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: The code will be publicly available at https://github.com/guikunchen/SDSGG.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Dataset. We evaluate our method on GQA [15] and VG [14] following [3, 12] .Split. Following previous work [3] , VG is divided into two splits: base and novel split. The base split comprises 70% of the relation categories for training, while the novel split contains the remaining 30% categories invisible during training. For a more comprehensive comparison, we also conduct testing on the semantic set, encompassing 24 predicate categories [16, 12] with richer semantics. base and novel split of GQA [15] are obtained in a similar manner ( §B).Evaluation Metrics. We report Recall@K (R@K) and Recall@K(mR@K) following [3, 22] .Base Models and Competitors. As for the base and novel split, we compare SDSGG with two baselines: i) CLS [4] , which uses only the category-level prompts to compute the similarity between the image and text; and ii) Epic [3] , a latest OVSGG method, which introduces an entangled crossmodal prompt approach and learns the cross-modal embeddings using contrastive learning. In terms of the semantic split, we compare our SDSGG with three baselines: i) CLS [4] , which uses only the category-level prompts; ii) CLSDE [12] , which uses prompts of relation class description; and iii) RECODE [12] , which uses visual cues of several separate components. Since [10] has neither released the detailed split nor the code, it is not included in the comparisons for fairness.Implementation Details. Due to limited space, implementation details are left in the appendix ( §B).We conduct quantitative experiments on VG [14] and GQA [15] . To ensure the performance gain is reliable, each experiment is repeated three times. The average and standard deviation are reported. VG [14] base and novel. Table 1 illustrates our compelling results over the latest OVSGG model, Epic [3] . Since Epic did not release the full code, we report values under the same setting as in the original paper. CLS [4] leverages the original CLIP's checkpoint (w/o fine-tuning) and only the category-level prompts. The performance of CLS on the novel split is better than that on the base split because i) the base split contains more relation candidates to be classified (35 vs. 15) ; and ii) the base split contains head, uninformative relations (e.g., "on" and "has") which are hard to be distinguished by CLIP [12] . By perform contrastive learning on the base split to learn the entangled cross-modal prompt, Epic [3] achieves significant performance gains on the base split (i.e., 22.6% vs.3.2% R@50 and 27.2% vs. 3.9% R@100). However, Epic demonstrates worse performance on the novel split (i.e., 7.4% vs. 18.1% R@50 and 9.7% vs. 22.2% R@100), which indicates its overfitting on the training data. By incorporating SSDs, SDSGG outperforms Epic by a large margin on both the base split and novel split. For instance, SDSGG exceeds Epic by 3.9%/4.4% R@50/100 on the base split and 18.0%/19.9% R@50/100 on the novel split, respectively. The significant performance improvements on the novel set demonstrate the strong zero-shot capability of our approach. In addition, our margins over the CLS are 16.6%∼27.7% R@K and 2.2%∼3.9% mR@K on the base split, and 5.2%∼7.3% R@K and 5.6%∼7.4% mR@K on the novel split, respectively.VG [14] semantic. In Table 2 , we present the numerical results of SDSGG against the latest OVSGG work [12] on the semantic split. By leveraging part-level prompts, RECODE demonstrates superior performance compared to CLS [4] and CLSDE [12] . The introduction of filtering strategies, i.e., RECODE ⋆ , shows substantial improvements. As seen, SDSGG surpasses all counterparts with remarkable gains on all metrics. In particular, SDSGG exceeds RECODE ⋆ by 10.9%/11.0%/9.9% on R@20/50/100, and 6.1%/4.0%/0.6% on mR@20/50/100. Notably, SDSGG achieves impressive performance without relying on additional augmentation or data [10] . Since an increase on mR@K implies the average of the improvements for all categories, yields on mR@K may be relatively lower than that on R@K. Without bells and whistles, SDSGG establishes a new state-of-the-art. GQA [15] base and novel. Since the codes of Epic [3] and RECODE [12] are insufficient to support replication, we only compare SDSGG with CLS on GQA [15] . As shown in Table 3 , SDSGG outperforms CLS by a large margin across all splits and metrics. Since CLS is not trained on the base split, its performance on R@K is relatively low due to the massive annotations of geometric relations that are not semantically rich [12] .Taking together, our extensive results provide solid evidence that SDSGG successfully unlocks the power of LLMs in OVSGG, and SSD is a promising alternative to category-/part-level prompts. Fig. 3 visualizes qualitative comparisons of SDSGG against CLIP [4] on VG [14] . As seen, with the proposed SSDs, SDSGG can generate higher-quality relation predictions even in challenging scenarios. We respectfully refer the reviewer to the appendix ( §E) for more qualitative comparisons.For thorough evaluation, we conduct a series of ablative studies on VG [14] . Textual Part. We first study the effectiveness of our multi-persona collaboration (MPC) for scene-specific description generation ( §3.1) in Table 4 . Here we use the standard prompts to query LLMs for generating descriptions. As seen, without MPC, the performance drops drastically, e.g., 11.8%/11.8% vs. 31.6%/30.0% R@100 on the base/novel split, respectively. This indicates the importance of the text classifiers used as they have impact on both training and testing.While the effectiveness of MPC has been validated, one may wonder: i) Why use these three roles? ii) How to ensure the completeness and quality of generated classifiers? We want to highlight that: i) Different roles are used to increase the variety of descriptions. There is no word on exactly which roles should be used. ii) These open problems are beyond the scope of this work. We leave them for future work. iii) This work makes the first attempt to enhance classifier generation for OVSGG via MPC. iv) The experimental results suggest that the current generated SSDs are good enough for commonly used relation categories. To provide more empirical results, we investigate the impact of the proposed self-normalized similarities and the number of used SSDs in the appendix ( §A). Visual Part. Then, we examine the impact of mutual visual adapter (MVA, §3.2) and directional marker (DM, §3.2) in Table 5 . The 1 st row denotes a strong baseline, i.e., a multi-layer perceptron with comparable parameters to aggregate the visual features. Upon projecting the visual features into interaction-aware space and applying crossattention, we observe consistent and substantial improvements for both R@K and mR@K on both the base and novel split. These results demonstrate the efficacy of our adapter and the necessity of incorporating cross-attention for capturing the complicated interplay between subjects and objects. Since DM is designed for geometric relations with massive annotations [21, 24] , the improvements on R@K are considerable, while those on mR@K are relatively small. See §A for more results. 
conclusion: 
 This work presents SDSGG, a scene-specific description based framework for OVSGG. Despite the previous works based upon category-/part-level prompts, we argue that these text classifiers are scene-agnostic, which cannot adapt to specific contexts and may even be misleading. To address this, we carefully design a multi-persona collaboration strategy for generating flexible, context-aware SSDs, a self-normalized similarity computation module for renormalizing the influence of each SSD, and a mutual visual adapter that consists of several trainable lightweight modules for learning interaction-aware space. Our approach distinguishes itself by using SSDs derived from LLMs, which are tailored to the content of the presented image. This is further enhanced by MVA, which captures the underlying interaction patterns based on the semantic space of VLMs. We expect the introduction of SDSGG will not only set a new benchmark for OVSGG, but also encourage the community to explore the potential of integrating LLMs with VLMs for deeper, contextual understanding of images. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present the experimental setup and details in the main paper and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We run each experiment three times and report the average and standard deviation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We introduce the used computer resources in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the appendix, we discussed our limitations, societal impact, and directions for future work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited related papers.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: For a better understanding of the main paper, we provide additional details in this supplementary material, which is organized as follows:• §A introduces more ablative experiments.• §B details the implementation details.• §C provides the pseudo code of SDSGG.• §D shows the generated scene-specific descriptions and the corresponding prompts.• §E offers more qualitative results.• §F discusses our limitations, societal impact, and directions of future work.• §G presents more experimental results.A More Ablative ExperimentSelf-normalized Similarity. Furthermore, we study the effectiveness of self-normalized similarity for evaluating the influence of each scene-specific description ( §3.1) in Table S1 . Here we remove the opposite description and use only the original scene description as the baseline. As seen, without the reference of opposite description, the performance drops drastically, e.g., 28.3%/25.5%/28.4% vs. 31.6%/30.0%/35.1% R@100 & 12.1%/29.8%/24.7% vs. 14.7%/31.5%/28.6% mR@100 on the base/novel/semantic split, respectively. This indicates the importance of renormalizing similarity scores, and also validates our hypothesis -the absolute value of one single similarity yields only limited insight, while the difference of two similarities provides more information.Table S1 : Ablation studies ( §A) on self-normalized similarity (SNS, §3.1). Method Split R@20↑ R@50↑ R@100↑ mR@20↑ mR@50↑ mR@100↑ Number of SSDs. We first study the impact of the number of used classifiers (i.e., SSDs). The results, presented in Table S2 , reveal a considerable performance enhancement when the number of pairs increased from 11 to 21, affirming the efficiency of SSDs. However, we also witnessed a dip in performance when too many SSDs were employed. This could be due to the unnecessary pairing of a relation category with an excessive number of descriptions (52 descriptions for 26 pairs).Table S2 : Ablation studies ( §A) on the number of used classifiers (Eq. 2). The adopted hyperparameters are marked in red. # Pairs Split R@20 R@50 R@100 mR@20 mR@50 mR@100 11 base 13.919 Scaling Factors. We then study the effectiveness of the scaling factors used in our training objectives ( §3.3) in Table S3 . As seen, the performance remains consistent for those compared hyperparameters. This indicates the robustness of SDSGG to changes in the scale of the training targets. Considering the performance on all metrics together, α and β are set to be 2 and 1e-1, respectively.Number of Attention Heads. We further investigate the impact of the number of attention heads H used in the mutual visual adapter ( §3.2, Eq. 5). As shown in Table S4 , the performance improves Table S3 : Ablation studies ( §A) on the scaling factors used in the training objectives, i.e., α and β ( §3.3, Eq. 8). The adopted hyperparameters are marked in red. α β Split R@20↑ R@50↑ R@100↑ mR@20↑ mR@50↑ mR@100↑ 1. from 32.7% to 35.1% R@100 on the semantic split when increasing the number of heads from 4 to 8, but the number of parameters steadily increase as the number of heads grows. When increasing the number of heads from 8 to 16, we observe some improvements (e.g., 31.6% to 32.8% R@100 on the base split), but also some performance drops (e.g., 30.0% to 29.0% R@100 on the novel split). Consequently, we set H = 8 as the default to strike an optimal balance between accuracy and computation cost.Table S4 : Ablation studies ( §A) on the number of attention heads H used in the mutual visual adapter ( §3.2, Eq. 5). The adopted hyperparameters are marked in red. H Param. Split R@20↑ R@50↑ R@100↑ mR@20↑ mR@50↑ mR@100↑ Margin. Last, we examine the impact of the margin λ in our training objectives ( §3.3, Eq. 8). As shown in Table S5 , the performance remains consistent for those compared values, which indicates the robustness of our learning procedure. λ is set to be 3e-2 by default.Table S5 : Ablation studies ( §A) on the margin λ (Eq. 8). The adopted hyperparameters are marked in red. λ Split R@20 R@50 R@100 mR@20 mR@50 mR@100 5e- The detailed, scene-specific descriptions generated by LLM's multi-persona collaboration are provided in Fig. S1 . As seen, our scene-specific descriptions (21 pairs, 42 descriptions in total) cover different scenes from a comprehensive view, e.g., spatial descriptions ("two or more objects partially overlap each other", "interaction between objects", etc.), environment (background) descriptions ("on a road", "on a flat plane, it should appear balanced with no visible tilting", etc.), semantic descriptions ("belong to animal or human behavior", "may have contact behavior", etc.), appearance descriptions ("might have flat teeth or sharp teeth", "have a curvy body", etc.).Two or more objects partially overlap each other Each object is completely separate with clear space between them.Objects are isolated and have no interaction with each other. Next, we detail each step to generate scene-specific descriptions.❶ Initial Description Generation. This step can be repeated many times to generate a large number (72 after manual selection) of scene descriptions. The prompt is shown in Fig. S2 .❷ Summarizing Descriptions. Since these initial descriptions may suffer from noise and semantic overlap, we ask LLM to streamline and combine these descriptions, ensuring more cohesive and distinct scene-level descriptions. The prompt is shown in Fig. S3 .❸ Description-Relation Association. After obtaining various scene descriptions, a critical inquiry arises regarding their utility for relation detection, given their lack of explicit association with specific relation categories. To address this, we delineate three distinct scenarios characterizing the interplay between relation categories and scene descriptions: i) certain coexistence (C n r = 1), where a direct correlation exists; ii) possible coexistence (C n r = 0), indicating a potential but not guaranteed association; and iii) contradiction (C n r = -1), denoting an incompatibility between the scene description and relation category. Here C n r denotes the correlation between relation r ∈ R and n th scene description, and is generated by LLMs. The prompt is shown in Fig. S4 .❹ Opposite Description Generation. Since classifiers are contextually bound, we generate opposite descriptions to compute self-normalized similarities. The prompt is shown in Fig. S5 .# Input: {scene content to be discussed} # Output: 3~5 descriptions """ Begin by embodying three distinct personas: a biology expert, a physics expert, and an engineering expert. Each expert will articulate a step-by-step approach along with their thought process, considering various hypothetical scenarios relevant to their field of study, and then share their insights with the group. If any expert does not know the answer, he will exit the discussion. Once all experts have provided their analyses, summarize the final generic scene descriptions. The generic scene description involves the specific appearance and the visible action/motion/interaction may appear in the scene (use your imagination here).Here are some examples of generic scene descriptions: {some in-context examples} Here is an example of discussion: Discussion started! Question: Suppose there is ... (this involves a detailed discussion of three roles) Discussion started! Question: Suppose there is {scene content to be discussed}, please give concise, generic scene descriptions of this scene. """ # Input: initial {scene descriptions}, all {relation categories}, {number of final scene descriptions} # Output: final scene-level descriptions """ Here is a text pool that includes a series of descriptive texts: {scene descriptions} Here are all the relation categories: {relation categories} You are asked to pick {number of final scene descriptions} descriptive statements from the text pool that can describe at least two predicates. Think step by step. """ Figure S3 : Prompts for summarizing descriptions.# Input: {scene descriptions}, all {relation categories} # Output: description-relation associations """ For {scene descriptions}, decide whether it is likely to appear in one of the following flat photos where {relation categories} appears. The judgment result is to choose between [certain coexistence, possible coexistence, and contradiction]. When the photo shows that the scene must have a certain description, it is judged as "contradiction". When the photo has little relationship with the description, it is judged as "possible coexistence". When the photo shows that the scene must not appear with a certain description, it is judged as "contradiction". """ We provide more visual results that compare SDSGG to CLIP [4] in Fig. S6 . It can be observed that SDSGG performs robust in hard cases and can consistently deliver more satisfying results, based upon the scene-specific descriptions and self-normalized similarities. Limitation Analysis. Following [12] , currently our algorithm is specifically designed for the predicate classification task [1, 16] to make fair comparisons with relation detection models. Such a pairwise classification task gives the ground-truth annotations of objects in the scene for easier representation learning of relations. However, these annotations are unavailable in real-world applications.Societal Impact. This work points out the drawbacks of existing scene-agnostic text classifiers and accordingly introduces several new modules for both the visual and textual components, leading to a scene-specific description based OVSGG framework that combines the strengths of VLMs and LLMs.Like every coin has two sides, using our framework will have both positive and negative impacts.On the positive side, our work contributes to research on intelligent scene understanding, and thus is expected to eventually benefit industries such as autonomous driving. For potential negative social impact, the reliance on VLMs and LLMs could lead to the perpetuation of biases and inequalities present in the data used in these models' large-scale pre-training stage.Future Work. As mentioned before, the focus of this work is not on object detection. It is interesting to extend our algorithm to handle the object detection task simultaneously by, for example, incorporating set-prediction architectures [88] . Moreover, the design of our multi-persona collaboration stands for an early attempt and deserves to be further explored. In addition, the architectural designs of directional marker and mutual visual adapter certainly worth further explorations, e.g., efficiency [89] , architecture [90, 91] , and adaptive prompting [92, 93] . Furthermore, extending our algorithm to other relation detection tasks [94, 39, 41, 95] may lead to an uniformed relation detection algorithm.Training on the Full Set of Relation. We trained our model with frequency bias [25] on the full set of relations. The results are shown in Table S7 . Different Base/Novel Splits. We trained our model on different base/novel splits to investigate the robustness further. Specifically, we i) change the proportion of the base and novel split and ii) change the categories within the base and novel split (i.e., different No. for the same ratio). The results are shown in Table S8 . Inference Time. Since the renormalization and similarity measurement involve only a few matrix operations that can be omitted from the complexity analysis, we will focus on the inference time of three main modules. The results are shown in Table S9 . Different Personas. The involvement of multiple personas in the discussion process enhances the diversity of generated descriptions. The key point of our multi-persona collaboration is about the "collaboration" rather than a specific persona. Actually, using only one persona can even decrease the diversity of generated descriptions and hurt the performance, as it can only generate descriptions from its own viewpoint without discussion with others. In addition, we changed the system prompt of LLM from the default like "you are a helpful AI assistant" into a persona-specific one like "you are a biologist". We then evaluate the performance of our model with these generated descriptions.The results are shown in Table S10 . Justification: In the appendix, we discussed our limitations, societal impact, and directions for future work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.The terms "description" and "prompt" are used interchangeably, all denoting text classifiers.A scene typically involves objects and their interplay and relationships. 


Decision: The reviewers reached a positive consensus early on, and the authors successfully upheld this by effectively addressing their concerns. In their rebuttal, the authors presented several additional experimental results as requested by the reviewers, many of which further strengthen the paper. The AC also strongly recommends including these results in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper starts by discussing methods for Open-vocabulary Scene Graph Generation (OVSGG) based on the CLIP model, highlighting the issue that current OVSGG methods do not differentiate between various scenes, which limits their effectiveness. The authors introduce SDSGG, a scene-specific description-based OVSGG framework that improves both the textual and visual parts, enhancing the model's open-vocabulary relationship prediction capabilities.
strength_weakness: strength_weakness: 1. The description in Sec3.1, Scene-specific Text Classifiers, of the paper is somewhat confusing. This confusion arises primarily because the text section includes multiple different naming conventions and several distinct modules. It is recommended that this section be rewritten to make it easier for readers to understand. Additionally, the terminology used in this section is inconsistent with that in lines 64~77, leading to comprehension difficulties.
2. For the OVSGG method, it is suggested to also train the model on a full set of relations and compare its performance with conventional SGG methods to ensure that it achieves good performance under standard settings.
3. Is the model robust to different base/novel splits? It is recommended to train and test the model on different base/novel dataset divisions to assess its robustness.
4. It is advised to train and test the model on the PSG dataset as well.
questions: questions: 1. Regarding the selection of multiple personas, the ablation study shows that not using this approach results in a significant performance decrease. My question is, what exactly are the "standard prompts" referred to in line 329 of the document? What would be the effect if only one persona is used, and among the three personas mentioned in the document, which persona demonstrates the most significant performance?
limitations: limitations: N/A
paper_summary: paper_summary: This paper aims to solve the open-vocabulary scene graph generation problem. Previous methods mainly adopt scene-agnostic prompts as text classifiers. The authors argue that using the fixed text classifiers not only struggles to model visual relations with high variance, but also falls short in adapting to distinct contexts. Therefore, the authors propose the scene-specific description based OVSGG framework. They employ an LLM and ask it to play different roles. Besides, they design the mutual visual adapter to encode visual features. Extensive experiments show that the proposed method significantly outperforms top-leading methods.
strength_weakness: strength_weakness: My main concern is Computational Complexity: The proposed framework involves multiple stages, including generating descriptions, renormalizing them, and applying mutual visual adapters. This multi-step process could be computationally intensive, making it less practical for real-time applications or scenarios with limited computational resources.
questions: questions: Please read the weaknesses part.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes SDSGG, a novel open vocabulary scene graph generation(OVSGG) algorithm that leverages the reasoning capability of a LLM to better determine the relations between objects in the scene. It achieves this goal by first prompting a LLM with multiple persona prompts to expand a simple relational predicative to a list of detailed visual descriptions, which are subsequently used to augment the classification process.  It also introduces a novel mutual visual adapter, which better captures the interaction between subjects and objects.  Experiments show that these proposed designs are effective.
strength_weakness: strength_weakness: 1. Prompting the LLM is a key element of the method, however some crucial details are missing. For example, how are the prompts constructed? While the author provided the prompt in Appendix Fig 5, it is unclear how the  "{scene content to be discussed}" is generated. The author did show some examples throughout the paper, but they are not sufficient for the reader to understand the underlying process. In particular, in L167, the author showed example #1 "Imagine there is an animal that is eating, ". In Fig 1c, there is example #2 "Assuming that the scene has a man riding a horse."  These two descriptions have two different granularity, as one only includes the generic concept of "an animal that is eating" while the other has specific class names "man" and "horse". The authors should clearly describe what information is included into the prompt, and discuss the scalability and cost of generating such prompts. I suppose if the prompts are like example #1, they can be generated offline based on predicative label sets. However, if the prompts are like example #2, they need to be generated for every possible triple of (subject, predicative, object) over the label space, or be generated online over possible objects in a scene. It is unclear which is the case.

2. Additional discussions and experiments are required to justify some of the design choices. For example,

  2.1 in eq 8, the loss of descriptions marked by possible coexistence is to make the prediction "close to those of CLIP." (L255). If this is the case, why not directly use CLIP results for these possible coexistence descriptions at inference time (eq 2)?

  2.2 some discussion is needed on if CLIP is good at classifying the generated descriptions. What are the nature of these descriptions and do they fit well with CLIP's pretraining pipeline (i.e. object-level image caption)? As a concrete example, can CLIP properly distinguish descriptions involving counting, such as "with four legs", and "with two legs", mentioned in the examples?

 2.3 what happens if we discard "possible coexistence" descriptions and only use definite coexistence and contradiction? Table8 shows that it is ideal to have a low weight for "possible coexistence"  loss. What happens if we set the weight to 0 and remove it at inference pipeline?
questions: questions: See weakness.
limitations: limitations: The authors discussed limitations and societal impact.

Comments: title: Thanks for your review
Thanks for your review
Thanks for your review
comment: Dear Reviewer qa1m,

We are grateful for your thorough review of our submission and the insightful feedback you've provided. Please let us know if you'd like any further information.

Sincerely yours,

Authors.
I appreciate the effort you put into addressing my concerns in your rebuttal. As a result, I am willing to increase the score by one point.
Dear Reviewer xMLQ,

We sincerely appreciate your time and effort in reviewing our submission and providing valuable comments. Please let us know if you'd like any further information.

Sincerely yours,

Authors.
Dear Reviewer RK4c,

Thank you again for your kind review and comments. Please let us know if you'd like any further information.

Sincerely yours,

Authors.
Dear Reviewer qa1m,

The results of the physicist persona are as follows:

| Method | Base |  | Novel |  |
| --- | --- | --- | --- | --- |
|  | mR @50 | mR @100 | mR @50 | mR @100 |
| Physicist Persona | 4.8 | 6.9 | 9.1 | 14.7 |

We appreciate the constructive feedback you provided, and we will incorporate all relevant results and details into our revision.

Sincerely yours,

Authors.
The authors successfully addressed my concerns.  I especially appreciate the clarifications of Q1. I maintain my recommendation for acceptance.
I am satisfying with the responses. I will keep my original rating.