SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization

Authors: wanhua  li
zibin  meng
jiawei  zhou
donglai  wei
chuang  gan
hanspeter  pfister
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 20:01:22

Abstract: Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. While current methods adopt the paradigm of training a dedicated network end-to-end using labeled image data, they are limited in terms of generalizability and interpretability. To address these issues, we first present a simple yet well-crafted framework named SocialGPT, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework, providing a strong baseline for social relation recognition. Specifically, we instruct VFMs to translate image content into a textual social story, and then utilize LLMs for text-based reasoning. SocialGPT introduces systematic design principles to adapt VFMs and LLMs separately and bridge their gaps. Without additional model training, it achieves competitive zero-shot results on two databases while offering interpretable answers, as LLMs can generate language-based explanations for the decisions. The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired. As we essentially convert a visual classification task into a generative task of LLMs, automatic prompt optimization encounters a unique long prompt optimization issue. To address this issue, we further propose the Greedy Segment Prompt Optimization (GSPO), which performs a greedy search by utilizing gradient information at the segment level. Experimental results show that GSPO significantly improves performance, and our method also generalizes to different image styles. The code is available at https://github.com/Mengzibin/SocialGPT.

Main paper: 
 introduction: 
 Social relationships are of paramount importance in our lives, as they significantly impact our emotional, psychological, and physical well-being. Social relationship recognition aims to categorize the relationships such as friends, colleagues, band members, and so on, that exist between individuals given an input image and the bounding boxes of the two persons of interest [1] . In recent years, social relationship recognition has garnered significant attention [1] [2] [3] [4] due to its wide range of applications, including product recommendation [5] , autonomous systems [6] , and more.Over the past decade, the field of computer vision has witnessed tremendous success [7] [8] [9] [10] [11] [12] in the end-to-end learning framework, which trains a dedicated neural network end-to-end on a customized dataset. Research in social relationship recognition has also followed a similar trajectory [1, 13, 2] . As social relationship reasoning represents a cognitive function that operates at a higher level than visual perception, many methods [6, 3] incorporate rich prior knowledge of social relations into the models. For example, GRM [6] integrated a knowledge graph into its model to leverage the information of contextual objects. GR 2 N [3] and TRGAT [14] exploit the logical constraints among multiple social relationships within the same scene. While these methods have achieved notable results, they are limited in terms of generalization and interpretability. In other words, we cannot trust that the trained models can generalize to arbitrary scenarios, and these models fail to provide the reasons and explanations for their decisions.In this paper, we first present a modular framework with foundation models for social relation reasoning. Recently, we have witnessed the significant success of foundational models [15] . Many Vision Foundation Models (VFMs) can accurately perform basic visual perception tasks such as identifying "what" and "where" in images [16] [17] [18] [19] . On the other hand, the emergence of Large Language Models (LLMs) demonstrates strong reasoning capabilities [20] [21] [22] [23] . Therefore, we present a framework that follows the "perceive with VFMs, reason with LLMs" paradigm. This framework first employs VFMs to convert images into textual data, and subsequently leverages the textual reasoning capabilities of LLMs for relation prediction. In this process, VFMs process visual signals into fundamental facts, and then LLMs analyze these facts to make explainable inferences.Our framework performs visual reasoning for Social relationship recognition using GPT-style LLMs, coined SocialGPT. SocialGPT introduces systematic design principles to guide and adapt VFMs and LLMs for social relationship reasoning. Specifically, in the perception phase, we extract both comprehensive and domain-specific visual information with VFMs, which is further fused into a coherent textual social story with symbol-based object reference and is easily readable. In the reasoning phase, we utilize a structured social relation reasoning prompt, named SocialPrompt, composed of different segments for "system, expectation, context, and guidance" to better instruct LLMs. With the proposed systematic design principles, our SocialGPT provides a strong baseline and achieves highly competitive zero-shot results, compared to the state-of-the-art methods that undergo end-to-end training on full training datasets.Lastly, we observed that LLMs exhibit high sensitivity to prompts during the reasoning process, but the manual prompt design is a time-consuming and labor-intensive task [24, 25] . We propose the Greedy Segment Prompt Optimization (GSPO) algorithm for automatic prompt tuning. As we convert a visual classification task as a generative task of LLMs, automatic prompt tuning for SocialPrompt encounters the long prompt optimization issue. Our proposed GSPO addresses these issues by utilizing gradient information at the segment level for greedy search. Experiments demonstrate that GSPO significantly improves the performance of LLMs. Figure 1 visualizes our paradigm. To summarize, we make the following contributions: 1). We present a simple modular framework with foundation models for social relation reasoning, which provides a strong baseline as the first zero-shot social relation recognition method. 2). To address the long prompt optimization issue associated with visual reasoning tasks, we further propose the Greedy Segment Prompt Optimization, which performs a greedy search on the segment level with gradient guidance. 3). Experiments demonstrate that our method attains very competitive and explainable zero-shot results without additional model training. With GSPO, our method significantly outperforms the state-of-the-art methods. Figure 2 : The framework of SocialGPT, which follows the "perception with VFMs, reasoning with LLMs" paradigm. SocialGPT converts an image into a social story in the perception phase, and then employs LLMs to generate explainable answers in the reasoning phase with SocialPrompt.of model parameters, and exhibit excellent performance along with strong generalization capabilities [15] . The emergence of LLMs [15, 27, 30, 31] has significantly reshaped the field of Natural Language Processing (NLP). ChatGPT and GPT-4 [27] , developed by OpenAI, are among the most famous LLMs. GPT-4, in particular, demonstrates a strikingly close-to-human-level intelligence [32] .Meanwhile, many open-source LLMs like Vicuna [29] , LLaMa [33] , and LLaMa-2 [34] have been developed, and have achieved outstanding performance across various NLP tasks. On the other hand, VFMs [19, 26, [35] [36] [37] [38] have also made significant advancements. CLIP [19] connects images and text, enabling zero-shot image classification [39, 40] . BLIP [41] and BLIP-2 [16] demonstrate strong zero-shot image-to-text generation capabilities. SAM [17] offers a foundation model for image segmentation [42] . While foundation model-based frameworks have been proposed for many other tasks including few-shot visual recognition [43] [44] [45] , visual question answering [46] [47] [48] , and semantic segmentation [49] , our SocialGPT explicitly employs text as the bridge between VFMs and LLMs and then proposes symbol-based referencing to support unambiguous text queries.Social Relation Recognition. Social psychologists have conducted extensive research on social relationships over decades [50, 51] , resulting in several different social theories [52, 53] . Sun et al. [1] followed Bugental's domain-based theory [52] and annotated the PIPA dataset, which has become one of the most popular benchmarks for social relation recognition. Li et al. [13] adopted the relational models theory [53] and contributed the People in Social Context (PISC) dataset. A dualglance model was further proposed to leverage multiple contextual regions. With the well-established benchmarks, numerous end-to-end methods [54, 3, 14, 2] have been proposed, effectively advancing the field of social relationship recognition. Some methods [54, 6] employed knowledge graphs to exploit scene and global contextual cues. Noticing that there usually are multiple social relations on the same image, Li et al. [3] proposed GR 2 N to jointly infer all relations on an image with graph neural networks. TRGAT [14] further considered higher-order constraints for social relations on an image and achieved better results. These methods adopted the end-to-end learning-based paradigm, whereas we propose a modular framework with foundation models. 
<mother-child>: the bond between a mother and her child, embodied by a female being the biological or legal guardian to one or more children. 3. <grandpa-grandchild>:: 
 The bond between a grandfather and his grandchild, depicted by a male being the grandfather to one or more grandchildren. 
socialgpt: 
 Social relation recognition takes an image I and two bounding boxes b 1 and b 2 of two interested individuals as inputs, and requires a model that outputs the social relationship y. We first introduce a modular framework with foundation models for social relation recognition in this section, which provides a strong zero-shot baseline. The pipeline is illustrated in Figure 2 . On a high level, we first use VFMs to extract visual information at different granularities. The raw information is then fused into a coherent social story in textual format, denoted as S, which can be best reasoned with LLMs.The perception objective is to extract essential visual information related to social relation reasoning, in order to connect with text-based LLMs for downstream reasoning. One straightforward approach is to utilize existing image captioning foundation models such as BLIP-2 [16] to generate a caption or GPT-4V [55] to generate an image description. However, a single sentence or general-purpose description may overlook crucial details relevant to social relations present in the images.We construct text-based visual information with VFMs with being both comprehensive and domainspecific as our guidelines. To achieve this, we resort to the state-of-the-art image segmentation tool, the Segment Anything Model (SAM) [17] , and the powerful vision-language foundation model, BLIP-2 [16] , for both identifying important details in the image and describing them in language.In particular, we use SAM to segment the image to obtain all different object masks, and then send individual objects by masking out others to BLIP-2 to obtain descriptions of each object. Together with the image-level caption, we formulate the dense captions covering all objects in the input image.The above gives us a comprehensive description of the image details. However, holistic captions of the image and different objects are not tailored to our task of social relation reasoning. To compensate for the lack of domain-specific information, we ask specific questions related to social identities by using the BLIP-2 dialog functionality to extract more specific information depending on object types. Recent research [54, 1] has shown that the age and gender of individuals, as well as the social scene and activity, are important clues. Therefore, we actively inquire BLIP-2 about these clues. Specifically, when dealing with people objects, we inquire about age and gender details. This information is crucial for distinguishing familial relationships within a family unit, such as father-child and grandmother-grandchild relationships. For image-level captions, we explore the social scenario or event depicted in the picture. This approach allows us to generate task-oriented captions that are tailored to our social relation recognition objective.One could directly input the dense captions and task-oriented captions along with object axes and dimensions into LLMs for social relation reasoning, but the information is fragmented and objects are described in isolation. On the other hand, LLMs perform the best when working with human-readable natural language and they often struggle with arithmetic reasoning tasks [56] [57] [58] . Therefore, we integrate the aforementioned vision information by composing a social story that is complete and coherent. Objects are conveniently referable and described in relative relations, and the full story is easily readable by both humans and LLMs. This will serve as a crucial bridge from visual perception to textual reasoning, providing a solid foundation for the next step of understanding with LLMs.We propose symbol-based referencing for object referral. Multiple individuals and various social relationships coexist in a single image, and bounding boxes b 1 and b 2 are provided for specific relation inquiries in supervised learning settings. However, as we now convert the entire image into textual data and rely on LLMs for analysis, effective referral of individual objects becomes a critical question. Based on SAM segmentation masks, we can naturally derive bounding boxes for each object i as b i = [x i , y i , h i , w i ], where (x i , y i ) is the center coordinate and (h i , w i ) are the height and width. While directly using these coordinates for referrals in the social story and question inquiries is precise, they pose extra challenges for readability and numerical reasoning for LLMs. Instead, we assign symbols to each object to associate with its coordinates in the original image, textual caption, and task-specific features for our social story generation. We use P i to refer to people objects, and O i to refer to other objects. Numerical coordinates will not appear in our social story, and relative positional relations are described with the referral symbols. The symbol-based referring also enables straightforward querying for LLMs. For instance, one can directly inquire LLMs about the social relationship between P 2 and P 3 with natural language and LLMs will easily identify the queried persons associated with symbols. This provides a clear and concise bridge between the object descriptions and the bounding box-based queries, and a similar method can be adopted for a broader range of applications when text-based reasoning is involved for object referral for visual question answering, robotics, etc.Finally, based on the list of isolated image and object descriptions after symbol-based referencing, we instruct an LLM to act as an information fusion tool for generating a coherent social story S in a unified paragraph. The social story tells all the information needed about the visual scene forSocial Story𝑃 # 𝑃 " 𝑃 ! 𝑂 ! 𝑂 " a.Dense Captions with Symbols Input Image Objects with Symbols text-based reasoning, which is highly readable and understandable by humans and LLMs with clear symbol references and information consolidation. An example of extracted perceptual information with symbol associations and the generated social story is depicted in Figure 3 .After obtaining the mapping from image to social story: I → S, we feed both S and bounding box queries (b i , b j ), converted to textual queries q with referencing symbols P i , P j , into LLMs to obtain interpretable answers a. This is to let LLMs output the map from (S, q) to a, which we do by prompting. Since LLM performance is highly sensitive to prompt variations [59, 55] , we design our social relation reasoning prompt with four segments, which we name SocialPrompt.System. This is the system prompt provided by many LLMs to steer their behavior. We utilize it to explicitly define several core rules for our task of social reasoning. We denote it as the o segment.Expectation. This is the instruction that we give to the model to set expectations of the anticipated outcomes. This helps avoid vague or unexpected outputs. To do so, we construct a role assignment and task description prompt, denoted as r, where we explicitly assign the role of a social relation expert to the LLM and provide a detailed elaboration of the task's input and output.Context. This provides sufficient contextual information to help the LLMs understand the background of the problem. As a classification task, we provide specific definitions for each social relationship category, resulting in the prompt segment denoted as c.Guidance. This offers an exemplar to show the LLMs how to respond to a query based on a social story. In-context learning has been proven as an effective means to expand the capabilities of LLMs [60] [61] [62] . We manually construct an in-context example prompt, denoted as e = (S 0 , q 0 , a 0 ), to better guide LLMs in performing social relationship reasoning in the desired format. Here we also guide the model to generate possible explanations for its prediction. While using more in-context examples may potentially further enhance performance, this is beyond the scope of the paper and is left as future work.The final SocialPrompt consists of (o, r, c, e), and is concatenated with a testing story-query pair (S, q) at the end for model predictions. Figure 2 shows the structured excerpts of SocialPrompt, and we put the full prompt into the Appendix. Note that we do not use any training samples provided by a dataset and only employ the foundation models. Consequently, SocialGPT is capable of zero-shot social relation reasoning, while maintaining its interpretability and generalizability. 
greedy segment prompt optimization: 
 Although we have devised well-structured SocialPrompt for social relation reasoning, experiments reveal that different ways of prompt rephrasing and demonstration example variations can significantly impact the LLM reasoning performance. Manually searching for the optimal prompt is time-consuming and labor-intensive, thus automatic prompt tuning is desired. Nevertheless, unlike the prompt optimization methods [63, 64] typically employed in NLP, automatic prompt tuning for SocialPrompt faces two unique challenges: free-form target and long prompt optimization. As we convert a visual classification task into a generative task for LLMs, the model's output space transitions from discrete numerical representations of one-hot labels to unconstrained textual forms. Defining free-form text objectives for SocialPrompt optimization is not well-explored. Meanwhile, as the social story S is a comprehensive description of the image such as in Figure 3 , and task m := Top-k(-z∈D ∇ hw m L(w 1:M ; z)) ▷ Compute top-k promising segment substitutions for b = 0, 1, . . . , K * M -1 do w(b) 1:M := w 1:M ▷ Initialization w(b) i := U i (⌊b/M ⌋), where i = (b mod M ) + 1 ▷ Select one replacement segment w 1:M := w(b ⋆ ) 1:M , where b ⋆ = argmin b z∈D L( w(b) 1:M , z) ▷ Compute best replacement Output: Optimized segments w 1:Mand full label set definitions could be lengthy, our SocialPrompt tends to be very long. This poses additional challenges for automatic prompt tuning methods. To address these issues, we propose a segment-based optimization algorithm, named Greedy Segment Prompt Optimization (GSPO).Tuning Objective. To automate prompt searching, the first step is to define the optimization objective. Ideally, we aim to find the optimal prompt {o * , r * , c * , e * } that maximize the probability of LLMs generating the correct answer a for any given sample z = (S, q). Let's first review the training paradigm commonly used for autoregressive language models [65, 66, 60] , which essentially employ the next token prediction task, i.e., learning p(w n+1 |w 1:n ), where token w n+1 ∈ V, and V represents the token vocabulary. Unlike typical classification tasks where only a one-hot formatted category is predicted, our answers are free-form text, consisting of a sequence of numerous tokens. Constructing the ground truth with free-form text for each sample is challenging. This paper proposes instructing LLMs to begin their response with the predicted class category following a pre-defined template. Formally, we assume that the ground truth answer a for sample z takes the following form: a = [a 0 , a 1 , a 2 , ...], where a 0 denotes the first sentence of a, a 1 is the second sentence, and so forth. We specify a 0 to have the following fixed format: a 0 = "The final answer is str(y)", where str(y) represents the string representation of class label y. Then we can define the objective:EQUATIONwhere the expectation is taken from a collection of training examples, and the probabilities are computed from LLM's next token prediction distributions. Note here the LLM is frozen, and we seek to find the optimal prompt to minimize the above loss. In practice, we employ the same template in our in-context example, making it easy for LLMs to follow a consistent output format. This ensures that the loss primarily stems from LLMs' predictions of tokenized category names rather than category-agnostic sentence formatting. Note that we only construct and supervise the first sentence of the ground truth answer, while the model is free to generate its explanation in the following sentences.Long Prompt Optimization. We optimize over discrete prompt tokens, constrained to a vocabulary V for each token position associated with the LLM. While some discrete prompt optimization algorithms [67, 25, 67] have been proposed in the NLP field, they typically operate on a limited number of tokens. In contrast, as a visual reasoning task, we require long prompts to adequately convey the dense information and provide detailed context. In fact, the number of tokens in our SocialPrompt may well exceed 2K, and conduct token-level optimization results in a search space of 2000 |V| , which is beyond the capacities of current optimization methods as |V| = 32, 000 for many LLMs [33, 34] . We propose to perform segment-level optimization as a surrogate. Formally, suppose the prompt is w with M segments, denoted as w 1:M . In our case we can have M = 4 and directly map the segments to o, r, c, e, respectively. We propose a candidate set W m consisting of alternative prompts for each segment, which we use ChatGPT to generate followed by light manual revisions, and the algorithm searches over the combination of different candidates. For the demonstration example segment e, we also manually select samples from an existing training set as candidates.More specifically, inspired by AutoPrompt [25] , our optimization algorithm considers all possible single-segment substitutions, thereby selecting the segment candidate that minimizes the loss over a batch of training samples. We replace one segment at a time in a greedy manner. In practice, instead of evaluating all possible candidates, we further reduce the search space by calculating the gradients of the one-hot segment indicators for each segment and selecting the top K most promising candidates for that segment. The gradient is computed as:∇ hw m L(w 1:M ) ∈ R |Wm|, where h wm represents the one-hot representation of selecting w m from the set W m . Then the top K promising substitutions with the largest negative gradient are chosen for evaluation. We repeat this process to acquire K candidates for each segment, and we only replace one segment at a time to obtain K * M new prompts. Then the one with the smallest loss over a batch of training samples is chosen. We iterate this process N times to find the best-performing prompt. The entire search process is shown in Algorithm 1.The bond between a grandmother and her grandchild, depicted by a female being the grandmother to one or more grandchildren. Answer: [Yes] Justification: The source codes will be made available to the public. Guidelines: Guidelines: 
settings: 
 Data and Evaluation. We adopt two widely-used benchmarks for social relation reasoning: PIPA [1] and PISC [13] . The PIPA dataset categorizes 16 types of social relationships, including family bonds (like parent-child, grandparent-grandchild), personal connections (friends, loves/spouses), educational and professional interactions (teacher-student, leader-subordinate), and group associations (band, sports team, colleagues). The PISC dataset categorizes social relationships into six types: commercial, couple, family, friends, professional, and no-relation. We follow the standard train/val/test split for both datasets and report the classification accuracy on the test set. Note that the training set is not used for our zero-shot results, but is used for in-context exemplar proposals for our prompt optimization algorithm. For both datasets, we measure classification accuracy as our evaluation metric.Implementation Details. We use two VFM models for visual information extraction -the SAM [17] model for object segmentation, followed by BLIP-2 [41] for dense caption generation. For the social story generation, we employ the GPT-3.5 [55] Turbo model that has empowered ChatGPT. We set the temperature to 0 for greedy decoding to bolster the result's reproducibility. Other generation parameters are otherwise set as default. For subsequent reasoning of social relations based on generated stories, we experiment with both GPT-3.5 and open-source LLMs, including Vicuna-7B/13B [29] and Llama2-7B/13B [34] . All the decoding temperature is set as 0, and we set the maximum context length to 4096 for Vicuna and Llama2 to accommodate our long prompt. For GSPO, we curate M = 15 candidates for each of the four segments within the complete prompt and set K = 3 for candidate selection for N = 500 iterations. One A100 GPU is used for all experiments.Main Results. We compare SocialGPT, using either GPT-3.5 or Vicuna-13B, with previous fully supervised methods and present our results in Table 1 and Table 3 . Here our method does not 3 . Most previous methods used mAP (mean Average Precision) as the metric on the PISC dataset, whereas we opted not to employ this metric due to the disparity between our predictions. Unlike previous methods that output per-class confidence scores, our prediction is the textual outputs from LLMs. Therefore, we still adopt the accuracy metric on the PISC dataset. To report the accuracy performance of other methods, we chose the state-of-the-art methods with publicly available code for reproduction and compared their performance. Table 3 shows that our method attains comparable results to the state-of-the-art GR 2 N model, despite not being trained with any data.Comparison with End-to-End VLMs. Our approach breaks down the social relation reasoning into different phases involving perception tasks with VFMs and reasoning with LLMs, bridged by a coherent textual social story. However, recent advancements in multimodal foundation models (VLMs) provide a straightforward way of reasoning about visual contents, which is simply asking questions about the image to a vision-language model that can respond with an answer directly. We compare SocialGPT with three state-of-the-art end-to-end vision-language foundation models by directly inquiring about social relationships in the image, including BLIP-2 [41] , LLaVA [68] , and GPT-4V [55] , with results shown in Table 4 . We see that the method of querying vision-language foundation models, albeit simple, is still lagging behind our approach of SocialGPT with principled designs and modularized VFMs and LLMs. Our well-designed SocialGPT even outperforms the high-performing GPT-4V by 7.03% in accuracy. These results justify the design principles of our framework with comprehensive perception extraction and coherent language reasoning.Ablation Study. We conduct a series of ablation studies to assess the efficacy of various components at different stages of SocialGPT. Table 2 shows the results with Vicuna-7B on the PIPA dataset. The first part of ablation focuses on the social story generation pipeline. As we use SAM to segment the image for visual perception, removing SAM would disable fine-grained object descriptions (dense captions) in the social story, resulting in an accuracy drop of more than 8%. If we do not acquire the task-oriented captions, there is a performance drop of 1.69%. Next, a crucial component of the social story generation in SocialGPT is the utilization of symbols (P for people and O for others) for effective referral of objects. If we do not use the symbols, but instead replace the object referral with either the direct coordinate or the object-specific caption from BLIP-2 in both the social story and the question, we see the performance drops by 3.90% and 1.75%, respectively. Finally, we fuse the multi-aspect visual information into a cohesive social story. If we bypass the fusion and directly utilize the visual annotations from VLMs, we can see there is a significant performance drop of 16.27%. This indicates that a good textual description of comprehensive visual information is necessary to connect LLMs to reason about social scenes presented in images.We also ablate the SocialPrompt segments in our LLM reasoning phase. We do this by removing each of the segments from the full prompt one at a time, and results are presented in the bottom half of Table 2 . We can see that guidance segmentation, which contains a manually constructed demonstration example of how to reason about social relations based on our social story, has the most influence on the model performance. Without it, the accuracy drops by 18.02%. The system In this captivating image with a resolution of 1024X777, we find ourselves immersed in a lively scene where a group of people are joyfully engaged in playing music. The atmosphere is electric as they come together, their instruments in hand, creating a harmonious symphony. Figure 4 : Visualization results of interpretability. We show the SocialGPT perception and reasoning process. We see that our model predicts correct social relationships with plausible explanations.prompt and expectation segment contributes to the final performance by approximately 1.35% and 2.39%, respectively, and the context segment defining social relationship categories has a lesser contribution with a 0.4% accuracy difference. This is perhaps because the LLMs already have substantial knowledge of common social relationships.As SocialGPT utilizes fixed prompt segments to instruct LLMs for social relation reasoning based on social stories, it might not be optimal with the static prompt design. Our GSPO further tunes the long prompt on the segment level for automatic performance improvements. Table 5 presents the results when applying GSPO on SocialGPT with various LLMs for reasoning, compared with the baseline zero-shot performance. Overall our segment-level prompt tuning with GSPO helps with the classification of all model variants. On PIPA the performance boost is about 2.38% on average, and on PISC it achieves a better gain with about 3.18% on average. These show the efficacy of the proposed GSPO algorithm to efficiently enhance prompt effectiveness. Out of the model variations, Vicuna-13B consistently outperforms other LLMs under our setup. The flexibility of SocialGPT in connecting with different reasoning models makes it more easily benefit from the latest advancements of LLMs without any heavy adaptation.Reasoning Process and Interpretability. We illustrate the perception and reasoning process of SocialGPT as well as the final results in Figure 4 . The people objects are fully segmented from VFMs and associated with symbols, which are then utilized to generate a coherent social story with clear references. By using LLMs for the reasoning on top of textual stories, SocialGPT not only outputs the correct social relations between different objects in the image but also provides plausible explanations behind the reasoning process.Generalization on Different Image Styles. Previous supervised models on social relation recognition heavily rely on annotated images and relations in a specific domain. As a result, these models cannot generalize to unseen image types well. In contrast, our method does not have the limitation of being domain-specific. We apply SocialGPT to novel sketch and cartoon images with various social relations generated by GPT-4V, with results shown in Figure 5 . As shown in the first example, the previous state-of-the-art model GR 2 N [3] fails to generalize as it predicts the relation between P 1 and P 2 as colleagues, but SocialGPT correctly recognizes the classmate relation based on the social scene with detailed explanation. Figure 5 : Results when applying SocialGPT to sketch and cartoon images. The images are generated by GPT-4V. Our method generalizes well on these novel image styles.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 Conclusion. In this paper, we present SocialGPT, a modular framework with foundation models for social relation reasoning, which attains competitive zero-shot results while also providing interpretable explanations. Furthermore, we propose the GSPO for automatic prompt tuning, which further improves the performance. Our approach opens new avenues for exploring the synergy between vision and language models in high-level cognitive tasks and offers a promising direction for future advancements in the field of social relation recognition.Limitations and broader impacts. Due to the modular nature of our approach, the performance of our method is constrained by the performance of the foundation models. If the segmentation model fails, or if the BLIP-2 model generates incorrect captions, or if the reasoning by LLMs is flawed, then our method is also prone to errors. Our method transforms visual problems into language-based reasoning, which could improve accessibility for visually impaired individuals. Meanwhile, our method also inherits biases from the foundation models, thus further research is needed to address them. Automatic classification of social relationships may lead to unintended negative consequences.To mitigate these risks, we can implement strategies such as fairness and bias checks, as well as promote transparent and responsible use of our technology. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our research doesn't train new models. We use open-sourced foundation models, and any safeguards they used can be applied to our method.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have properly cited the original paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: This paper does not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Input Image Default SAM Mask Our SAM Mask In this paper, we employ SAM to automatically segment an image into multiple object masks, which we then use to generate dense captions. However, a challenge arises with SAM's default "segment everything" setting, as it tends to produce over-segmented and fine-grained masks. For instance, a person may be segmented into multiple fragments, including hair, face, hand, arm, and so on. Two examples illustrating this issue are presented in Figure 6 . Creating meaningful captions for these subpart-level regions proves to be challenging and often leads to a loss of overall object perception. This is due to the fact that SAM generates three masks for each point prompt, corresponding to three semantic levels: whole, part, and sub-part. To address this issue, we adopted a two-stage SAM forward scheme. Initially, we employed SAM's default "segment anything" approach to obtain segmented masks, then retained the center points of each mask as point prompts for the second SAM forward pass. This ensures that as much as possible, objects in the image are not missed in the second SAM segmentation stage. For the second SAM segmentation stage, the points obtained from the first stage are used as point prompts, considering only the highest semantic level among SAM's three semantic levels. This approach minimizes over-segmentation and allows our method to focus on semantic at the object level. Subsequently, we apply NMS, threshold filtering, and post-processing to obtain high-quality object-level masks following SAM's methodology [17] . The resulting object masks for our method are displayed in Figure 6 .Social Story Generation. We carefully designed the prompt to guide the LLMs in generating coherent and easily understandable social stories based on dense captions. The system prompt and user prompt are depicted in Figure 7 . To ensure symbol-based referencing, we explicitly instruct LLMs not to rely on coordinates but instead to use symbols for reference. Additionally, we require the generated paragraphs to focus on social contexts.SocialPrompt on the PIPA dataset. The PIPA dataset comprises 16 social relationship categories, including father-child, mother-child, grandpa-grandchild, grandma-grandchild, friends, siblings, classmates, loves/spouses, presenter-audience, teacher-student, trainer-trainee, leader-subordinate, band members, dance team members, sport team members, and colleagues. Figure 8 illustrates the prompt we utilized for the PIPA dataset in the zero-shot setting. We provided a detailed explanation for each category within the prompt. Furthermore, the SocialPrompt includes manually constructed in-context examples.SocialPrompt on the PISC dataset. Figure 9 illustrates the SocialPrompt utilized in the PISC dataset, specifically in the zero-shot setting. The PISC dataset comprises 6 social relation categories:There are some rules: -Pay more attention to the people semantic, which have reference <P>.-No more than 15 sentences.-Only use one paragraph.Figure 7 : The prompt used for social story generation. GPT-3.5 Turbo model is used for caption fusion. The system prompt lists some key rules and the user prompt details the task definition.commercial, couple, family, friends, professional, and no-relation. We have also included the definitions of these six social relation categories within the prompt.SocialPrompt after GSPO. Due to the time and effort-intensive nature of manually designing prompts, this paper introduces the Greedy Segment Prompt Optimization method. For each segment, we employ ChatGPT to generate multiple candidates. As for the in-context examples, we also randomly select several samples from the training dataset. Here, we employ Vicuna-7B [29] for training to obtain the optimized prompts. The optimized prompt on the PIPA dataset is illustrated in Figure 10 , while that on the PISC dataset is shown in Figure 11 .[System Prompt]: You are an expert assistant in recognizing social relationships between people based on textual descriptions.-The answer can only be one of the 16 listed social relationships.-Give the most likely answer and don't refuse to answer.-If can't decide, then randomly select one.-Output the final answer in the first sentence. -The answer can only be one of the 6 listed social relationships.-Give the most likely answer and don't refuse to answer.-If can't decide, then randomly select one.-Output the final answer in the first sentence. [System Prompt]: You are an expert assistant in recognizing social relationships between people based on textual descriptions.-The answer can only be one of the 16 listed social relationships.-Give the most likely answer and don't refuse to answer.-If can't decide, then randomly select one.-Output the final answer in the first sentence.Here is the definition: 1. <father-child>: The bond between a father and his child, characterized by a male being the biological or legal guardian to one or more children. 


Decision: The paper received 2 Borderline Reject, 1 Borderline Accept and 1 Accept ratings. While the authors should incorporate the reviewers' comments in their final version to elaborate the motivation of the storyline and the technical and experimental details, it is a solid contribution to the community. The AC reads the paper, the rebuttal, and the discussions and decides to accept it. Congratulations!

Reviews: review: paper_summary: paper_summary: This manuscript introduces SocialGPT, a modular framework designed to enhance social relation reasoning by combining Vision Foundation Models (VFMs) and Large Language Models (LLMs). SocialGPT utilizes VFMs to convert image content into a textual social story, followed by LLMs performing text-based reasoning. The paper further introduces the Greedy Segment Prompt Optimization (GSPO) algorithm to optimize prompts for LLMs, addressing the challenges of long prompt optimization. The proposed method achieves competitive zero-shot results on social relation recognition tasks and offers interpretable answers.
strength_weakness: strength_weakness: - The approach involves substantial computational resources for both the perception and reasoning phases, potentially limiting accessibility and scalability for some users.
- The experiments, while promising, are primarily conducted on two datasets. Further testing on a broader range of datasets and tasks would strengthen the generalizability of the findings.
- The method assumes that the visual context provided by VFMs is sufficiently detailed and accurate, which might not always hold true in diverse real-world scenarios.
- The compatibility of the proposed method seems to be limited; Table 5 implies that LLaMA2-based SocialGPT performs very poorly compared to Vicuna. The proposed framework may work only for specific types of models.
questions: questions: - How can we evaluate generated social stories? It would be great if the authors could show how GSPO improves the quality of generated social stories.
- How GSPO can be performed without the ground-truth answer? The current formulation in section 4 seems to require the ground-truth to define the loss objective.
- What are the differences between the social story of SocialGPT and social relationships used in baselines? I feel that Image-based text explanation is not new.
limitations: limitations: See the weakness section and question above.
paper_summary: paper_summary: This paper proposes a framework called SocialGPT for social relation reasoning, which combines vision foundation models and large language models. A greeedy segment prompt optimization methods is also proposed to prompt LLM. Experimental results show the effectiveness of the proposed method.
strength_weakness: strength_weakness: --- The paradigm of using VLMs for perceiving and LLMs for reasoning is currently a common solution for multimodal tasks. The main difference of this paper seems to be the use of a generated social story as the representation of visual content. As stated by the authors, LLMs perform best when working with human-readable natural language and often struggle with arithmetic reasoning tasks, which is why they design an additional process to generate social stories. However, the generation of social stories is also done by LLMs, which also suffer from the above difficulties. 

--- The authors propose a candidate set consisting of alternative prompts for each segment and select the best-performing prompt from their combination. The final prompt is obtained by selection rather than generation, which limits the upper bound of the performance on the manually collected candidate set. 

--- The function of SAM is to distinguish individuals in the image and obtain their coordinates. However, in the social story generation phase, the LLM (Large Language Model) discards the coordinates, retaining only the semantic information and losing the positional information. Conducting social relationship reasoning purely based on semantics may be insufficient. For example, in Figure 2, the social relationship is identified as a sibling relationship (brother and sister), but there are two boys in the image, both fitting the given description of "stands out in his vibrant red and green striped pajamas," making it unclear which individual P1 refers to.
questions: questions: --- Is the design of using LLMs for social story generation optimal, and why? Also, have the authors tried other approaches to generate social stories from dense captions instead of using LLMs?

--- In the part of reasoning with large language models, the social relation reasoning prompt is artificially divided into four partitions: System, Expectation, Context, and Guidance, but the motivation and reasonableness of such a design is not elaborated in the paper.
limitations: limitations: Please see the weakness and limitations.
paper_summary: paper_summary: This paper introduces SocialGPT, a modular framework for social relation reasoning that integrates the perception capabilities of Vision Foundation Models (VFMs) with the reasoning capabilities of Large Language Models (LLMs). To optimize prompts, the authors propose GSPO, a segment-based optimization algorithm for automated prompt tuning. Extensive empirical results validate the effectiveness of SocialGPT both quantitatively and qualitatively. GSPO consistently enhances SocialGPT's performance across various LLMs, and case studies demonstrate the framework's generalizability and interpretability.
strength_weakness: strength_weakness: Section 3.2 mentions that using precise coordinates can pose challenges for LLM numerical reasoning. However, it appears in Figure 3 that the objects' positional relations in the social story are inferred from numeric coordinates provided in the dense captions with symbols. Does this coordinate-based inference lead to similar numerical reasoning challenges? Additionally, how are relative positional relations conveyed here using referral symbols?
questions: questions: Please see the weaknesses section above.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: Paper proposes a pipeline method of orchestration pre-trained foundation models to solve the social relationship classification problem. It uses vision models to extract information in text about the scene in the form of caption. Relevant information, i.e. age, gender, general description, of individual persons and objects are also extracted in text form via instance segmentation + masking + captioning. The generated text are then further converted to Social Story with a LLM. With the novel prompt engineering method, GSPO, another LLM will then generate the social relationship from the Social Story.

Experimental results on the challenging benchmarks, PIPA and PISC, indicates its strong performance with zero-shot setup. Extensive ablation studies were also done to evaluate the contributions of the various components. In particular, it clearly showed the merits of the "Social Story" design.
strength_weakness: strength_weakness: 1. Besides the clever design of the pipeline, the direct technical contributions is slightly on the weaker side as there is no obvious technical breakthrough. The proposed GSPO appears to be the main new technique introduced. However, I am not an expert in this area and will defer to other reviewers on its technical novelty and merits.

2. (minor) The use of the generic semantic segmentation model (SAM) may not be the optimal choice. There are much stronger Human Instance Segmentation methods which can replace the paper's custom SAM method. Such methods are specifically trained on person dataset to handle various challenging scenarios unique to human segmentation, e.g. heavy occulsion, human-like objects (e.g. maniquinn).

Ling, E., Huang, D., & Hur, M. (2022). Humans need not label more humans: Occlusion copy & paste for occluded human instance segmentation. BMVC.
questions: questions: 1. Will/has the authors consider using pairwise attributes, besides the individual person attributes. E.g. relative age between pairs (older/younger), same/different clothings for the model? 

2. Why are only 2 attributes (age/gender) used for the person instance? In prior works, other attributes such as wearing uniform are important attributes for certain type of social relationship, e.g. team members?
limitations: limitations: (minor) There may be some unintended negative consequences of automatic classification of social relationship.

Comments: title: Looking forward to your further response before the end of discussion period
Looking forward to your feedback before the end of discussion period
Thank you
Looking forward to your feedback
Looking forward to your feedback
Could AC help to reach out reviewers?
Looking forward to your feedback
Looking forward to your feedback
Thank you and we are looking forward to your post-rebuttal feedback!
comment: Dear Reviewer,

Thank you for your follow-up questions.

As the discussion period is coming to an end, we would like to know if our responses have addressed your concerns. Please let us know if you have any further questions or need additional clarification, and we will be glad to assist.
Dear reviewer,

Thank you very much for your valuable feedback. As the discussion period is coming to an end, we kindly wish to inquire if our response has fully addressed your concerns. Please let us know if there are any additional concerns or questions.
We are glad that all your questions have been well addressed. Thank you again for your positive assessment and for taking the time to review our work. Your invaluable feedback has been pivotal in improving our paper.
Thank you for providing me the opportunity to clarify.

To illustrate the difference, we first present the pipeline:

Our pipeline is: 
> Image -> [Foundation Models] -> Social Story (text form) -> [LLM] -> Prediction

The pipeline of GR2N and TRGAT is: 
> Image -> [Neural Net] -> Social Graph (feature representation) -> [Graph Neural Net] -> Prediction.

The differences are significant: 1) Our social story is a **text-form image description** that can be input into LLMs. In contrast, the social graph in GR2N and TRGAT is a **feature representation** that cannot be input into LLMs. Therefore, the social graph in GR2N and TRGAT cannot work with LLMs. Our method is the first social relation recognition method that supports LLM reasoning. 2) The difference between text form and feature representation indicates a key difference in paradigms: our paradigm supports zero-shot social relation recognition with foundation models, achieving greater interpretability and generalizability, whereas existing social relation methods train a dedicated network end-to-end, limiting their interpretability and generalizability.

As for the "graph" in [1], it has a different meaning than the "graph" in GR2N and TRGAT. The "graph" in [1] is also a text-form image description, not a feature representation. The pipeline in [1] is: 
> Image -> [Foundation Models] -> Social Graph (text form). 

So, the "graph" in [1] is more like our text-based social story and shares the same motivation, but with a different way to achieve it. Therefore, the "graph" in GR2N and TRGAT cannot be used for LLM prompts. 

As we highlighted before, our main focus is not the specific instantiation of the social story; rather, it is to provide a framework for zero-shot social relation recognition using foundation models. The major technical contribution lies in our GSPO, which optimizes the SocialPrompt within our framework.


We hope this detailed explanation addresses your concerns. Please feel free to ask any further questions. We are more than willing to discuss and resolve any additional concerns you may have during the discussion period.

[1] https://arxiv.org/pdf/2311.17076
Thanks for your response. My questions have been well addressed, and I will maintain my score.
I appreciate your response. However, I am still unclear about the differences between the proposed social story (in text form) and the graph-based social relationships used in baselines such as GR2N or TRGAT. Do you think text-based LLM prompts are superior to graph-based ones? For example, graphs can also be used as LLM prompts for reasoning [1]. If the social graphs used in the baselines also work well with LLMs, it seems difficult to argue that the social story contributes significantly.

By the way, I also have read the other reviews that have been posted and their corresponding author responses. I will continue to read other reviews and the authors' responses and engage in further discussions with other reviewers until the end of the discussion period.

[1] https://arxiv.org/pdf/2311.17076
Dear reviewer,

Thank you for the comments on our paper. We have responded to your initial comments. We are looking forward to your feedback and will be happy to answer any further questions you may have.
Dear reviewer,

Thank you for the comments on our paper. We have responded to your initial comments. We are looking forward to your feedback and will be happy to answer any further questions you may have.
Dear AC:

Thank you for efficiently handling our draft. 

As the rebuttal phase is nearing its end, we haven't received any post-rebuttal feedback. 

Could the AC assist in reaching out to the reviewers for a response? 

We greatly appreciate all the efforts you've made!




Thanks

Authors
Dear reviewer,

Thank you for the comments on our paper. We have responded to your initial comments. We are looking forward to your feedback and will be happy to answer any further questions you may have.
Dear reviewer,

Thank you for the comments on our paper. We have responded to your initial comments. We are looking forward to your feedback and will be happy to answer any further questions you may have.
Dear AC and all reviewers:

Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity.

The discussion phase has been on for several days and we are still waiting for the post-rebuttal responses.

We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice.

Best,

Authors