SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization
Authors: Wanhua Li
Zibin Meng
Jiawei Zhou
Donglai Wei
Chuang Gan
Hanspeter Pfister
Venue: NeurIPS 2024 poster (2024)
Keywords: Social Relation Reasoning
Large Language Models
Foundation Models
Prompt Optimization
Primary area: machine_vision
TL;DR: We present SocialGPT, a modular framework with greedy segment prompt optimization for social relation reasoning, which attains competitive results while also providing interpretable explanations.

Abstract: Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. While current methods adopt the paradigm of training a dedicated network end-to-end using labeled image data, they are limited in terms of generalizability and interpretability. To address these issues, we first present a simple yet well-crafted framework named SocialGPT, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework, providing a strong baseline for social relation recognition. Specifically, we instruct VFMs to translate image content into a textual social story, and then utilize LLMs for text-based reasoning. SocialGPT introduces systematic design principles to adapt VFMs and LLMs separately and bridge their gaps. Without additional model training, it achieves competitive zero-shot results on two databases while offering interpretable answers, as LLMs can generate language-based explanations for the decisions. The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired. As we essentially convert a visual classification task into a generative task of LLMs, automatic prompt optimization encounters a unique long prompt optimization issue. To address this issue, we further propose the Greedy Segment Prompt Optimization (GSPO), which performs a greedy search by utilizing gradient information at the segment level. Experimental results show that GSPO significantly improves performance, and our method also generalizes to different image styles. The code is available at https://github.com/Mengzibin/SocialGPT.

SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization
Wanhua Li Zibin Meng 21.2 Jiawei Zhou Donglai Wei Chuang Gan Hanspeter Pfister' Tsinghua University
Harvard University
SStony Brook University
Boston College
MIT-IBM Watson Al Lab UMass Amherst
Abstract
Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. While current methods adopt the paradigm training a dedicated network end-to-end using labeled image data, they are limited terms of generalizability and interpretability. To address these issues, we first present a simple yet well-crafted framework named SocialGPT, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within modular frame- work, providing strong baseline for social relation recognition. Specifically, we instruct VFMs to translate image content into textual social story, and then utilize L.LMs for text-based reasoning. SocialGPT introduces systematic design principles to adapt VFMs and LLMSs separately and bridge their gaps. Without additional model training. it achieves competitive zero-shot results on two databases while offering interpretable answers, as LLMs can generate language-based explanations for the decisions. The manual prompt design process for LLMs at the reasoning phase is tedious and automated prompt optimization method is desired. As we essentially convert a visual classification task into a generative task of LLMs, automatic prompt optimization encounters unique long prompt optimization issue. To address this issue, we further propose the Greedy Segment Prompt Optimiza- tion (GSPO), which performs greedy search by utilizing gradient information the segment level. Experimental results show that GSPO significantly improves performance, and method also generalizes to different image styles. The code is available at hhttps:://// htt//gittuuu thub. com http///nnziiinn/cciaaaaa.
Introduction
Social relationships are of paramount importance in our lives, as they significantly impact our emotional, psychological, and physical well-being. Social relationship recognition aims to categorize the relationships such as friends, colleagues, band members, and so on, that exist between individuals given input image and the bounding boxes of the two persons of interest [[]. In recent years, social relationship recognition has garnered significant attention [1 due to its wide range of applications, including product recommendation
5 autonomous systems 66, and more.
Over the past decade, the field of computer vision has witnessed tremendous success 12 in the end-to-end learning framework, which trains a dedicated neural network end-to-end on customized dataset. Research in social relationship recognition has also followed similar trajectory [1 13 2 As social relationship reasoning represents a cognitive function that operates at a higher level than visual perception, many methods 6 incorporate rich prior knowledge of social relations into the models. For example, GRM
6 integrated a knowledge graph into its model to leverage the
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Seart what soctal
Neural
Socu
Network
The sedinal
SAM
colillngg meason
VFMs
LLMs
Input Image
Label
Input Image
Perception
Prompting
Reasoning
Answer
End-to-end Learning- Based Framework
(b) Modular Framework with Foundation Models
Figure 1: (a) End-to-end learning-basee framework for social relation reasoning. A dedicated neural network is trained end-to-end with full training data. (b) We propose a modular framework with foundation models for social relation reasoning. Our proposed SocialGPT first employs VFMs to extract visual information into textual format, and then perform text-based reasoning with LLMs, using either our manually designed SocialPrompt or optimized prompts.
information of contextual objects. GR [3] and TRGAT [1] exploit the logical constraints among multiple social relationships within the same scene. While these methods have achieved notable results, they are limited in terms of generalization and interpretability. In other words, we cannot trust that the trained models can generalize to arbitrary scenarios, and these models fail to provide the reasons and explanations for their decisions.
In this paper, we first present a modular framework with foundation models for social relation reasoning. Recently, we have witnessed the significant success of foundational models [15] Many Vision Foundation Models (VFMs) can accurately perform basic visual perception tasks such as identifying "what" and "where" in images [9] On the other hand, the emergence of Large Language Models (LLMs) demonstrates strong reasoning capabilities [20/23]. Therefore, we present framework that follows the "perceive with VFMs, reason with LLMs" paradigm. This framework first employs VFMs to convert images into textual data, and subsequently leverages the textual reasoning capabilities of LLMs for relation prediction. In this process, VFMs process visual signals into fundamental facts, and then LLMs analyze these facts to make explainable inferences.
Our framework performs visual reasoning for Social relationship recognition using GPT-style LLMs, coined SocialGPT. SocialGPT introduces systematic design principles to guide and adapt VFMs and LLMs for social relationship reasoning. Specifically, in the perception phase, we extract both comprehensive and comainn-speciff visual information with VFMs, which is further fused into coherent textual social story with symbol- object reference and is easily readable. In the reasoning phase, we utilize a structured social relation reasoning prompt, named SocialPrompt, composed of different segments for "system, expectation, context, and guidance" to better instruct I.MMs. With the proposed systematic design principles, our SocialGPT provides a strong baseline and achieves highly competitive zero-shot results, compared to the state-of-the-an are methods that undergo end-to-end training on full training datasets.
Lastly, we observed that LLMs exhibit high sensitivity to prompts during the reasoning process, but the manual prompt design is time-consuming and labor-intensive task (24 25
We propose the Greedy Segment Prompt Optimization (GSPO) algorithm for automatic prompt tuning. As we convert visual classification task as a generative task of LLMs, automatic prompt tuning for SocialPrompt encounters the long prompt optimization issue. Our proposed GSPO addresses these issues by utilizing gradient information at the segment level for greedy search. Experiments demonstrate that GSPO significantly improves the performance of LLMs.
our paradigm. To summarize, we make the following contributions: 1). We present simple modular framework with foundation models for social relation reasoning, which provides a strong baseline the first zero-shot social relation recognition method. 2). To address the long prompt optimization issue associated with visual reasoning tasks, further propose the Greedy Segment Prompt Optimization, which performs greedy search on the segment level with gradient guidance. 3). Experiments demonstrate that our method attains very competitive and explainable zero-shot results without additional model training. With GSPO, our method significantly outperforms the state-of of off-the-rt methods.
2 Related Work
Foundation Models. Recently, we have witnessed the tremendous success of foundational mod- els [19 29] Foundation models are typically trained on massive data, possess large number

---

RRA
oteee notckier pettored grest
1L-1P-2
BEAP-2
sariped modlin
is gire 1827 yower to thr dromp
poues?
Dense
Perception Phase
Captions
Task-oriented
Symbols
Social Story
Captions
SAM
Reasoning Phase
for ere mere mustieee monertiein setal fendd tertal
The fteet cnti inger
System
nittzoatia activent it Actnd
creaary soctas
Stary
and Liie wes encateenliyy
Expectstion
Mre the dy tatttar
getter
resctiactle father
juslly petteria with pore
Context
attoh are chamctertitte
Gundance
[t. courtio)::
Quers
Input Image
SocialPrompt
Explainable Answer
Figure 2: The framework of SocialGPT, which follows the "perception with VFMs, reasoning with LLMs" paradigm. SocialGPT converts an image into social story in the perception phase, and then employs LLMs to generate explainable answers in the reasoning phase with SocialPrompt.
of model parameters, and exhibit excellent performance along with strong generalization capabil- ities [15]. The emergence LLMs [15 27 30 31 has significantly reshaped the field of Natural Language Processing (NLP). ChatGPT and GPT-4 (27), developed by OpenAI, are among the most famous LLMs. GPT-4, in particular, demonstrates strikingly hose-tt-humaa--leel intelligence [32] Meanwhile, many open-source LLMs like Vicuna (29). LLaMa [33] and LLaMa-2
34 have been developed, and have achieved outstanding performance across various NLP tasks. On the other
hand, VFMs [19 26 35 38 have also made sigmificant advancements. CLIP [99 connects images
and text, enabling zero-shot image classification [39 40]. BLIP
41 and BLIP-2 [16] demonstrate
strong zero-shot image-to-text generation capabilities. SAM 17 offers foundation model for image segmentation 442. While foundation model-based frameworks have been proposed for many
other tasks including few-shot visual recognition 43 visual question answering [46)48), and semantic segmentation [49], our SocialGPT explicitly employs text as the bridge between VFMs and LLMs and then proposes symbol-basee referencing to support unambiguous text queries.
Social Relation Recognition. Social psychologists have conducted extensive research on social relationships over decades [50 [1], resulting in several different social theories [52] [3) Sun er
al. I followed Bugental's 's domain- based theory [52] and annotated the PIPA dataset, which has become one of the most popular benchmarks for social relation recognition. Li er al.
[3] adopted
relational models theory [53 and contributed the People in Social Context (PISC) dataset. dual- glance model was further proposed to leverage multiple contextual regions. With the well-established benchmarks, numerous end-to-end methods [54
2 have been proposed, effectively advancing the field of social relationship recognition. Some methods [54] 6 employed knowledge graphs to
exploit scene and global contextual cues. Noticing that there usually are multiple social relations on K proposed GR`N to jointly infer all relations on an image with graph
the same image, Li et al. neural networks. TRGAT 14] further considered higher-order constraints for social relations on an image and achieved better results. These methods adopted the end-to-en learning based paradigm, whereas we propose a modular framework with foundation models.
3 SocialGPT
Social relation recognition takes an image I and two bounding boxes by and by of two interested individuals as inputs, and requires a model that outputs the social relationship y. We first introduce a modular framework with foundation models for social relation recognition in this section, which provides a strong zero-shot baseline. The pipeline is illustrated in Figure On a high level, we first use VFMs to extract visual information at different granularities. The raw information is then fused into a coherent social story in textual format, denoted S, which can be best reasoned with LLMs.
3

---

3.1 Perception with Vision Foundation Models
The perception objective is to extract essential visual information related to social relation reasoning. in order connect with text-based LLMs for downstream reasoning. One straightforward approach is to utilize existing image captioning foundation models such as BLIP-2 [16] to generate a caption or GPT-4V [55] to generate an image description. However, a single sentence or general-purpose description may overlook crucial details relevant to social relations present in the images.
We construct text-based visual information with VFMs with being both comprehensive and domain- specific as our guidelines. To achieve this, we resort to the state-of-the-art image segmentation tool, the Segment Anything Model (SAM) [17, and the powerful vision-language foundation model, BLIP-2 [6], for both identifying important details in the image and describing them in language. In particular, we use SAM to segment the image to obtain all different object masks, and then send individual objects by masking out others to BLIP-2 to obtain descriptions of each object. Together with the image-level caption, we formulate the dense captions covering all objects in the input image.
The above gives us a comprehensive description of the image details. However, holistic captions of the image and different objects are not tailored to our task of social relation reasoning. To compensate for the lack of -omain--pecific information, we ask specific questions related to social identities by using the BLIP-2 dialog functionality extract more specific information depending on object types. Recent research 154] [ has shown that the age and gender of individuals, as well as the social scene and activity, are important clues. Therefore, clues. Specifically, when dealing with people objects, we inquire about age and gender details. This information is crucial for distinguishing familial relationships within family unit, such as father-child and wooddmother-grandchhe social scenario or event depicted in the picture. This approach allows us to generate task-oriented captions that are tailored to our social relation recognition objective.
actively inquire BLII--2 about these
relationships. For image-level captions, we explore the
3.2 Social Story Generation
One could directly input the dense captions and task-oriented captions along with object axes and dimensions into LLMs for social relation reasoning. but the information fragmented and objects are described in isolation. On the other hand, LLMs perform the best when working with human-rcadable natural language and they often struggle with arithmetic reasoning tasks (56)558. Therefore, we integrate the aforementioned vision information by composing social story that is complete and coherent. Objects are conveniently referable and described in relative relations, and the full story is easily readable by both humans and LLMs. This will serve as crucial bridge from visual perception to textual reasoning, providing a solid foundation for the next step of understanding with LLMs.
We propose symbol-based referencing for object referral. Multiple individuals and various social relationships coexist in a single image, and bounding boxes i and b2 are provided for specific relation inquiries in supervised learning settings. However, as we now convert the entire image into textual data and rely on LLMs for analysis, effective referral of individual objects becomes a critical question. Based on SAM segmentation masks, we can naturally derive bounding boxes for each object as b, = [xs Vt. h4 w.). where (I,, y1) is the center coordinate and (h,, w;) are the height and width. While directly using these coordinates for referrals in the social story and question inquiries is precise, they pose extra challenges for readability and numerical reasoning for LLMs. Instead, we assign symbols to each object to associate with its coordinates in the original image, textual caption, and task-specific features for our social story generation. We use P. to refer to people objects, and O to refer to other objects. Numerical coordinates will not appear in our social story, and relative positional relations are described with the referral symbols. The symbol-based referring also enables straightforward querying for LLMs. For instance, one can directly inquire LLMs about the social relationship between P2 and Pa with natural language and LLMS will easily identify the queried persons associated with symbols. This provides a clear and concise bridge between the object descriptions and the bounding box- box-based queries, and a similar method can be adopted for a broader range of applications when text-based reasoning is involved for object referral for visual question answering, robotics, etc.
Finally, based on the list of isolated image and object descriptions after symbol-based referencing, we instruct an L.MM to act as an information fusion tool for generating a coherent social story S in unified paragraph. The social story tells all the information needed about the visual scene for

---

Deur Catiaia 201
WOTT UPL soll
sarte mate unadee
and doeee
aidl sxpmomcie
Input Emage
Objects with Symbols
Dense Captions with Symbols
Social Skory
Figure 3: An example of social story generation.
text-based reasoning, which is highly readable and understandable by humans and LLMs with clear symbol references and information consolidation. An example of extracted perceptual information with symbol associations and the generated social story is depicted in Figure]
3.3 Reasoning with Large Language Models
After obtaining the mapping from image to social story: I + S, we feed both S and bounding box queries (b, by), converted textual queries q with referencing symbols Ps. Pj. into LLMs to obtain interpretable answers a. This is to let LLMs ouiput the map from (S, q) to a. which we do by we design our social relation reasoning prompt with four segments, which we name SocialPrompt.
prompting. Since LLM performance is highly sensitive to prompt variations [59 55
System. This is the system prompt provided by many LLMs to steer their behavior. We utilize it to explicitly define several core rules for our task of social reasoning. We denote it as the o segment.
Expectation. This is the instruction that we give to the model to set expectations of the anticipated outcomes. This helps avoid vague or unexpected outputs. To do so, we construct a role assignment and task description prompt, denoted r, where we explicitly assign the role of social relation expert to the LLM and provide a detailed elaboration of the task's 's input and output.
Context. This provides sufficient contextual information to help the L.LSs understand the background of the problem. As classification task, we provide specific definitions for each social relationship category, resulting in the prompt segment denoted as
Guidance. This offers an exemplar show the LLMs how to respond to query based on social story. In-context learning has been proven as an effective means to expand the capabilities of LLMs 60 62]. We manually construct an in-context example prompt, denoted as e (Sa, qu. ag). to better guide LLMs in performing social relationship reasoning in the desired format. Here we also guide the model to generate possible explanations for its prediction. While using more in-context examples may potentially further enhance performance, this is beyond the scope of the paper and is left as future work.
The final SocialPrompt consists of (o, r, C, e). and is concatenated with a testing story-query pair (S, q) at the end for model predictions. Figure 2 shows the structured excerpts of SocialPrompt, and we put the full prompt into the Appendix. Note that we do not use any training samples provided by dataset and only employ the foundation models. Consequently, SocialGPT capable of zero-shot social relation reasoning, while maintaining its interpretability and generalizability.
4 Greedy Segment Prompt Optimization
Although we have devised well-structured SocialPrompt for social relation reasoning, experiments reveal that different ways of prompt rephrasing and demonstration example variations can signif- icantly impact the LLM reasoning performance. Manually searching for the optimal prompt is time-consuming and labor-intensire intensive, thus automatic prompt tuning is desired. Nevertheless, unlike the prompt optimization methods [63] 64] typically employed in NLP, automatic prompt tuning for SocialPrompt faces two unique challenges: free-form target and long prompt optimization. As we convert visual classification task into a generative task for LLMs, the model's output space transitions from discrete numerical representations of one-hot labels to unconstrained textual forms. Defining free-form text objectives for SocialPrompt optimization is not well-explored. Meanwhile, as the social story S is comprchensive description of the image such as in Figure B and task
5

---

Algorithm 1 Greedy Segment Prompt Optimization
Input:
Initial segments WI.M, training dataset T, iteration number N
Build the candidate set Wm for each segment 8mm
repeat N times
Randomly sample batch of data D from T for m ,,..., .M do
Um Top-k(- SEEEED V Vhww ((W1:M; z))
> Compute top-k promising segment substitutions b Initialization
for b 0, 1, ...... K M do (b)
w 1:M = W1:M
(b) [. where i (b mod M) +
b Select one replacement segment
W L:M w (b*)
i.M. where b* = argmin, EEED ((w (:M)
Compute best replacement
Output: Optimized segments WI:M
and full label set definitions could be lengthy, our SocialPrompt tends to be very long. This poses additional challenges for automatic prompt tuning methods. To address these issues, we propose segment-based optimization algorithm, named Greedy Segment Prompt Optimization (GSPO).
Tuning Objective. To automate prompt searching, the first step is to define the optimization objective. Ideally, we aim to find the optimal prompt (o", r"' c,, e*) that maximize the probability of LLMs generating the correct answer a for any given sample training paradigm commonly used for autoregressive language models 65 66 60), which essentially employ the next token prediction task, i.e., learning p( W[ | ). where token Wn+1 V, and V represents the token vocabulary. Unlike typical classification tasks where only a one-hot formatted category is predicted, Constructing the ground truth with free-form text for each sample is challenging. This paper proposes instructing L.MSs begin their response with the predicted class category following a pre-defined template. Formally, we assume that the ground truth answer a for sample z takes the following form:
(S.g) s). Let's first review the
answers frre-form text, consisting of sequence of numerous tokens.
[a at a² ...). where a denotes the first sentence of a, a' is the second sentence, and so forth. We specify a" to have the following fixed format: a" = "The final answer is str(y)", where str(y) represents the string representation of class label y. Then we can define the objective:
L(o, r, r,
(x lo,
where the expectation is taken from collection of training examples, and the probabilities computed from LLM's next token prediction distributions. Note here the LLM is frozen, and we seek to find the optimal prompt minimize the above loss. In practice. we employ the same template in our in-context example, making it easy for LLMs to follow a consistent output format. This ensures that the loss primarily stems from L.Mss predictions of tokenized category names rather than category-agnostic sentence formatting, Note that we only construct and supervise the first sentence of the ground truth answer, while the model is free generate its explanation in the following sentences.
Long Prompt Optimization. We optimize over discrete prompt tokens, constrained to a vocabulary V for each token position associated with the LLM. While some discrete prompt optimization have been proposed in the NLP field, they typically operate on a limited number of tokens. In contrast, as visual reasoning task, we require long prompts to adequately convey the dense information and provide detailed context. In fact, the number of tokens in our SocialPrompt may well exceed 2K, and conduct token-level optimization results in search space of 2000|>), which is beyond the capacities of current optimization methods as | 32, 000 for many LLMs 33 34] We propose to perform segment-level optimization as a surrogate. Formally, suppose the prompt is w with M segments, denoted as WI.M- In our case we can have AM 4 and directly map the segments to o, r, C, e, respectively. We propose a candidate set W,.. consisting of alternative prompts for each segment, which and the algorithm searches over the combination of different candidates. For the demonstration example segment e, we also manually select samples from an existing training set as candidates.
algorithms 667 25 67
ChatGPT to generate followed by light manual revisions,
More specifically, inspired by AutoPrompt 25 our optimization algorithm considers all possible single-segment substitutions, thereby selecting the segment candidate that minimizes the loss over

---

Table The comparison results on the PIPA
Table 2: Ablations on components of SocialGPT with Vicuna-7B. The results are obtained on the PIPA dataset with a zero-shot setting.
dataset. ZS stands for Zero-Shot.
Methods
ZS
Ace (%)
All attributes SVM 1
X
57.2
Methods
Acc (%)
Pair CNN
13
X
58.0
SocialGPT
61.58
Dual-Glance 13
X
59.6
Dense Captions
52.63
SRG-GN
X
53.6
Task-oriented Captions
59.89
GRM 6
X
62.3
Symbol + Object Coordinate
57.68
MGR 2
X
64.4
Symbol Object Caption
59.83
Social Story
45.31
GR 3
X
64.3
TRGAT
X
65.3
SocialPrompt Segment (System]
60.23
SocialPrompl Segment (Expectation)
59.19
SocialGPT (w/ GPT-3.5)
64.1
SocialPrompt Segment (Context)
61.18
SocialGPT (w/ Vicuma-13B)
66.7
SocialPrompt Segment (Guidance)
43.56
a batch of training samples. We replace one segment at a time in a greedy manner. In practice, instead of evaluating all possible candidates, we further reduce the search space by calculating the gradients of the one-hot segment indicators for each segment and selecting the top most promising candidates for that segment. The gradient computed as: Vhwn [(W1-M) E E IW where hw... represents the one-hot representation of selecting wm. from the set Wm. Then the top K promising substitutions with the largest negative gradient are chosen for evaluation. We repeat this process to acquire K candidates for each segment, and we only replace one segment at time to obtain K M new prompts. Then the one with the smallest loss over a batch of training samples is chosen. We iterate this process N times to find the best-performing prompt. The entire search process is shown in Algorithm]]
Experiments
5.1 Settings
Data and Evaluation. We adopt two widely-used benchmarks for social relation reasoning: PIPA [ and PISC [13]. The PIPA dataset categorizes 16 types of social relationships, including family bonds (like parent-child, and professional interactions (teacher-student, sander-subordinate) and group associations (band, sports team, colleagues). The PISC dataset categorizes social relationships into six types: commercial, couple, family, friends, professional, and no-relation. We follow the standard train/val/test split for both datasets and report the classification accuracy on the test set. Note that the training set is not used for our zero-shot results, but is used for in-context exemplar proposals for our prompt optimization algorithm. For both datasets, we measure classification accuracy our evaluation metric.
personal connections (friends, loves/spouses), educational
Implementation Details. We use two VFM models for visual information extraction the SAM [17 model for object segmentation, followed by BLIP-2 [4] for dense caption generation. For the social story generation, we employ the GPT-3.5 [53] Turbo model that has empowered ChatGPT. We the temperature to 0 for greedy decoding to bolster the result's 's reproducibility. Other generation parameters are otherwise set as default. For subsequent reasoning of social relations based on generated stories, we experiment with both GPT-3.5 and open-source LLMs, including Vicuma- 7B/13B 29 and LJama2-7B/13B 34.. All the decoding temperature is set as O, and we set maximum context length to 4096 for Vicuna and Ilama2 to accommodate our long prompt. For GSPO, we curate M 15 candidates for each of the four segments within the complete prompt and set K 3 for candidate selection for N 500 iterations. One A100 GPU is used for all experiments.
5.2 Zero-shot Social Relation Recognition with SocialGPT
Main Results. We compare SocialGPT, using either GPT-3.5 or Vicuna-13R, with previous fully supervised methods and present results in Table [ and Table B Here method does not

---

Table 3: The comparison results on the PISC dataset. Previ- ous methods are replicated with open-source code to report the accuracy metric. ZS means Zero-Shot.
Table 4: Comparison with exist-
ing Vision-Language Models on the PIPA dataset, with SocialGPT using Vicuna- 3B model.
Methods
ZS
Acc (%)
Pair CNN 13
X
46.30
Methods
Acc (%)
GRM 6
X
64.18
BLIP-2 4T
35.84
GR
x
64.70
LLaVA 68
45.12
GPT-4V (55]
59.67
SocialGPT (w/ GPT-3.5)
53,43
SocialGPT
66.70
SocialGPT (w/ Vicuna-1381 13B)
65.12
undergo the prompt tuning optimization, performing relation reasoning in a zero-shot fashion without utilizing any training examples. On both datasets, Vicuna-13 13B performs better than GPT-3.5 with our framework. In particular, on PIPA benchmark shown in Table [ SocialGPT achieves the best accuracy compared with all prior supervised approaches, leading the previous state-of-the-art model TRGAT [4] by 1.4%. The results on the PISC benchmark are shown in Table 3 Most previous methods used mAP (mean Average Precision) as the metric on the PISC dataset, whereas we opted not to employ this metric due to the disparity between our predictions. Unlike previous methods that output per-class confidence scores, our prediction is the textual outputs from LLMs. Therefore. we still adopt the accuracy metric on the PISC dataset. To report the accuracy performance of other methods, we chose the state-of-the-arr methods with publicly available code for reproduction and compared their performance. Table [ that our method attains comparable results to the state-of-thhe-at GR model, despite not being trained with any data.
Comparison with End-to-End VLMs. Our approach breaks down the social relation reasoning into different phases involving perception tasks with VFMs and reasoning with LLMs, bridged by a coherent textual social story. However, recent advancements in multimodal foundation models (VLMs) provide straightforward way of reasoning about visual contents, which is simply asking questions about the image to a vision-language model that can respond with an answer directly. We compare SocialGPT with three state-of-the-ar end-to-end vision-language foundation models by directly inquiring about social relationships in the image, including BLIP-2 [11,, LLaVA VA [68], and GPT-4V [55], with results shown in Table 4 We see that the method of querying vision- vision-language foundation models, albeit simple, is still lagging behind our approach of SocialGPT with principled designs and modularized VFMs and LLMs. Our well-designed SocialGPT even outperforms the high-performing GPT-4V by 7.03% in accuracy. These results justify the design principles of our framework with comprehensive perception extraction and coherent language reasoning.
Ablation Study. We conduct a series of ablation studies to assess the efficacy of various components different stages of SocialGPT. Table 2 shows the results with Vicuna-7B on the PIPA dataset. The first part of ablation focuses on the social story generation pipeline. As we use SAM to segment the image for visual perception, removing SAM would disable line-grained object descriptions (dense captions) in the social story, resulting in an accuracy drop of more than 8%. If we do not acquire the task-oriented captions, there is a performance drop of 1.69%. Next, crucial component of the social story generation in SocialGPT is the utilization of symbols (P for people and O for others) for effective referral of objects. If we do not use the symbols, but instead replace the object referral with either the direct coordinate or the object-specific caption from BLIP-2 in both the social story and the question, we see the performance drops by 3.90% and 1.75%, respectively. Finally, we fuse the multi-aspect visual information into a cohesive social story. If we bypass the fusion and directly utilize the visual annotations from VLMs, we can see there is a significant performance drop of 16.27%. This indicates that good textual description of comprehensive visual information is necessary to connect LLMs to reason about social scenes presented in images.
We also ablate the SocialPrompt segments in our LLM reasoning phase. We do this by removing each of the segments from the full prompt one at time, and results are presented in the bottom half of Table 2, We can see that guidance segmentation, which contains manually constructed demonstration example of how to reason about social relations based on our social story, has the most influence on the model performance. Without it, the accuracy drops by 18.02%. The system
8

---

Table 5: Prompt tuning results (aceuracy in %) with GSPO.
PIPA
PISC
Model
SocialGPT
GSPO
A
SocialGPT
GSPO
A
Vicuna-7B
61.58
62.99
+1.41
45.13
49.79
+4.66
Vicuna-13B
66.70
69.23
+2.53
65.12
66.19
+1.07
Llama2-7B
31,91
34.07
+2.16
36.71
38.04
+1.33
Llama2- 13B
37.86
41.27
+3.41
42.74
48.39
+5.65
and -tan (Tee four cueer spitke the wan Nay unet whrbeall betvees
fead feher depine 190h
alre maker
wentestte the curann anddr maneect pranes aru expiorast Jater wyy
soag sayus
clia aiis eas
denteris ensterter the Aeters leving
reacheent and los momeensiy incennor ane sokis one dhe yoing hey A pis Und jock cots ug expertace aocsss and eocaaaaeee
sapie youty plaiig
forher axpportey jehnechll cetrieatpp
th reatinting inar unahiio asgoped
oftn exser dhedded
choter
your
Net recel mosoonkly Seranes and
coptwrr avene son HV/ resty Per rir aits commoeds ertentoe hhppe bioct ders reprentingg wamer ner
derrl emproser
sher repteemax aluvey mase ripertes
ratewssy parwib wothrr
deeaial cranotasonal releoeeete.
Input Image
SAM Mask
Social Story
Interpretable Answer
Figure 4: Visualization results of interpretabilityy. We show the SocialGPT perception and reasoning process. We see that our model predicts correct social relationships with plausible explanations.
prompt and expectation segment contributes to the final performance by approximately 1.35% and 2.39%, respectively, and the context segment defining social relationship categories has a lesser contribution with 0.4% accuracy difference. This is perhaps because the LLMS already have substantial knowledge of common social relationships.
5.3 Long Prompt Optimization with GSPO
As SocialGPT utilizes fixed prompt segments to instruct LLMs for social relation reasoning based on social stories, it might not be optimal with the static prompt design. Our GSPO further tunes the long prompt on the segment level for automatic performance improvements. Table 5 presents the results when applying GSPO on SocialGPT with various LLMs for reasoning, compared with the baseline zero-shot performance. Overall our segment-level prompt tuning with GSPO helps with the classification all model variants. On PIPA the performance boost is about 2.38% on average, and on PISC it achieves a better gain with about 3.18% average. These show the efficacy of the proposed GSPO algorithm to efficiently enhance prompt effectiveness. Out of the model variations, Vicuna- 13B consistently outperforms other LLMs under our setup. The flexibility of SocialGPT in connecting with different reasoning models makes more easily benefit from the latest advancements of LLMs without any heavy adaptation.
5.4 Qualitative Analysis
Reasoning Process and Interpretability. We illustrate the perception and reasoning process of SocialGPT as well as the final results in Figure 4 The people objects are fully segmented from VFMs and associated with symbols, which are then utilized to generate a coherent social story with clear references. By using L.Ms for the reasoning top of textual stories, SocialGPT not only outputs the correct social relations between different objects in the image but also provides plausible explanations behind the reasoning process.
Generalization on Different Image Styles. Previous supervised models on social relation recognition heavily rely on annotated images and relations in specific domain. As result, these models cannot generalize to unseen image types well. In contrast, our method does not have the limitation of being nomain-specifet We apply SocialGPT to novel sketch and cartoon images with various social relations generated by GPT-4V, with results shown in Figure As shown in the first example, the previous tite--of-thearr model G 3 fails to generalize as it predicts the relation between P and Ps as colleagues, but SocialGPT correctly recognizes the classmate relation based on the social scene with detailed explanation.

---

(W0)) outet
Colleegurs
(100) are 21 gwwdee-paattiil
[Sary] oieddi The fast avser sestriptiss the impe, nut chaaly recist ratationstee broeer dus clsseiits The settiry of ctadaete fectcate zs educatiore: provide undicetion /estiia! rasattic
[hee firel ause dsscriptios the the not Loaty secial recatiauiio benwen add is
[Bard]
and cxtia, the proseee pantly nas ruppects faritte contect the Wasrigtten seprartees the playtst dewaver tectcaring
The
thes
Figure 5: Results when applying SocialGPT to sketch and cartoon images. The images are generated by GPT-4V. Our method generalizes well on these novel image styles.
Conclusion
Conclusion. In this paper, we present SocialGPT, modular framework with foundation models for social relation reasoning, which attains competitive zero-shot results while also providing interpretable explanations. Furthermore, improves the performance. Our approach opens new avenues for exploring the synergy between vision and language models in high-level cognitive tasks and offers promising direction for future advancements in the field of social relation recognition.
propose the GSPO for automatic prompt tuning, which further
Limitations and broader impacts. Due to the modular nature of our approach, the performance of our method is constrained by the performance of the foundation models. If the segmentation model fails, or if the BLIP-2 model generates incorrect captions, or if the reasoning by LLMs is flawed, then our method is also prone to errors. Our method transforms visual problems into language-based reasoning, which could improve accessibility for visually impaired individuals. Meanwhile, method also inherits biases from the foundation models, thus further research is needed to address them. Automatic classification of social relationships may lead to unintended negative consequences. To mitigate these risks, we can implement strategies such as fairness and bias checks, as well as promote transparent and responsible use of our technology.
Acknowledgment
This research is supported in part by the NIH grant RO100194969, NIH grant 1U01CA284207, and NSF award 2239688.
References
Qianru Sun. Bernt Schiele, and Mario Fritz. A domain based approach social relation recognition. In CVPR, pages 3481-3490, 2017.
[2] Meng Zhang, Xinchen Liu, Wu Liu, Anfu Zhou, Huadong Ma, and Tao Mei. Multi-granularity reasoning for social relation recognition from images. In ICME, pages (618-1623, 2019. 2019.
[3] Wanhua Li, Yueqi Duan, Jiwen Lu, Jianjiang Feng, and Jie Zhou, Graph-based social relation reasoning. In ECCV, pages 18-34. Springer, 2020.
[4] Haorui Wang, Yibo Hu, Yangfu Zhu, Jinsheng Qi, and Bin Wu. Shifted gen-gat and cumulative-transffmmeee based social relation recognition for long videos. In ACM MM, pages 67-76, 2023.
[5] You-Jin Park and Kun-Nyeong Chang. Individual and group hehavior-based customer profile model for personalized product recommendation. Expert Systems with Applications, 36(2):1932 1939.199999999999111111111 2009.
[6] Zhouxia Wang, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, and Liang Lin. Deep reasoning with knowledge graph for social relationship understanding. In ICCAI, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NewrIPS, 28, 2015.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In ICCV, pages 2961-2262 2969. 2017.
10

---

[10] Wanhua Li, Jiwen Lu, Jianjiang Feng, Chunjing Xu, Jie Zhou, and Qi Tian. Bridgenet: A continuity-aware probabilistic network for age estimation. In CVPR, pages 1454-115,, 2019.
[11] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pages 7262-7272, 2021.
[12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020.
[13] Jumnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Dual-glance model for deciphering social relationships. In ICCV, pages
2017.
[14] Yunfei Guo, Fei Yin, Wei Feng, Xudong Yan, Tao Xue, Shuqi Mei, and Cheng-Lin in Liu. Social relation reasoning based on triangular constraints. AAAI, pages 737-745, 2023.
[15] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jcannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint urXiv:2108.07258 2021.
[16] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language -mage pre- wre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
[18] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In CVPR, pages 20051-20060, 2024.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763 8763. PMLR, 2021.
[20] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of off---fffff thought prompting elicits reasoning in large language models. NeurlPS, 35:24824 24837, 2022.
[21] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint or Xiv:2206. 07682, 2022.
[22] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le. Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
[23] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I. Griffiths, Yuan Cao, and Karthik Narasimham. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
[24] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveyx, 1-35, 2023.
[25] Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Erie Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. EMNLP, pages 4222-4235, 2020.
[26] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusinn models. In CVPR, pages
2022.
[27] OpenAL. Gpt-4 technical report, 2023.
[28] Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language- guided ordinal regression. NeurlPS, 35:35313-355 333255, 2022.
[29] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu. Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
11

---

[30] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.011 010066 2022.
[31] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[32] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg. et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint Xxx::2333.12712, 2023.
[33] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 122..3399 13971, 2023.
[34] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 092888, 2023.
[35] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, In ICCV, pages 9650-9660, 2021.
[36] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint drXivv2304.07193, 2023.
[37] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang. Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang. Jenq-Neng Hwang, et al. Grounded language-image pre- pre-training, In CVPR, pages 10965-10977 2022.
[38] Aditya Ramesh, Mikhail Payloy, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-8831. PMLR, 2021.
[39] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337-233 2348, 2022.
[40] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICI.R, 2023.
[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified Dision-language understanding and generation. In /CML, pages 12888 12900. PMLR, 2022.
[42] Evan Ling, Dezhao Huang, and Minhoe Hur. Humans need not label more humans: Occlusion copy paste for occluded human instance segmentation. In BMVC, 2022.
[43] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang. Hanqiu Deng, Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In CVPR, pages 15211-15222, 2023.
[44] Peng Gao. Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. IJC,, pages 1-15, 2023.
[45] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816 16816-16825, 2022.
[46] Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. In CVPR, pages 14974-14983, 2023.
[47] Jiaxian Guo, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Boyang Li. Dacheng Tao, and Steven Hoi. From images to textual prompts: Zera-shot visual question answering with frazen large language models. In CVPR, pages 10867-10877, 2023.
[48] Dídac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023.
[49] Chaohui Yu. Qiang Zhou, Jingliang Li, Jianlong Yuan, Zhibin Wang, and Fan Wang. Foundation model drives weakly incremental learning for semantic segmentation. In CVPR, pages 23685-22 23685-23694, 2023.
12

---

[50] Sheldon Cohen. Social relationships and health. American psychologist. 59(8):676, 2004.
[51] Hope R Conte and Robert Plutchik. A circumplex model for interpersonal personality traits. Journal of personality and social psychology, 40(4))701, 1981.
[52] Daphne Blunt Bugental. Acquisition of the algorithmss of social life: a domain-based approach. Psycholog- ical bulletin, 126(2):187, 2000.
[53] Alan P Fiske. The four clementary forms of sociality: framework for a unified theory of social relations. Psychological review, (9(4);689, 1992.
[54] Arushi Goel, Keng Teck Ma, and Cheston Tan. An end-to-end network for generating social relationship graphs. In CVPR, pages 11186-11 195, 2019.
[55] Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07833 2023.
[56] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398, 2023.
[57] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? In NAACL, pages 2080-2094, 2021.
[58] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. arXiv preprint arXiv:2302.04770 2023.
[59] Jules White, Quchen Fu, Sam Hays, Michael Sandbom, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv: /3382, 2023.
[60] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 33:1877-1900 2020.
[61] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP. pages 11048-11064, 2022.
[62]
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In NAACL, pages 2655-26711 2022.
[63] Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy
Ba. Large language models human-level prompt engineers. In ICLR, 2023.
[64] Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael 7eng. Automatic prompt optimization with "gradient descent" and beam search. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, EMNLP, pages 7957-7968. 2023.
[65] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018
[66] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18))9 2019.
[67] Andy Zou, Zifan Wang. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.1500 2023.
[68] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
13

---

A
More Implementation Details
Input Image
Default SAM Mask
Our SAM Mask
Figure 6: The comparisons of the default SAM masks and our SAM masks.
In this paper, we employ SAM to automatically segment an image into multiple object masks, which we then use to generate dense captions. However, challenge arises with SAM's default "segment everything" setting, as tends to produce over-segmented and fine-grained masks. For instance, person may be segmented into multiple fragments, including hair, face, hand, arm, and examples illustrating this issue are presented Figure 6 Creating meaningful captions for these subpart-level regions proves to be challenging and often leads loss of overall object perception. This is due to the fact that SAM generates three masks for each point prompt, corresponding to three semantic levels: whole, part, and sub-part. To address this issue, adopted two-stage SAM forward scheme. Initially, we employed SAM's default "segment anything" approach to obtain segmented masks, then retained the center points of each mask as point prompts for the second SAM forward pass. This ensures that as much as possible, objects in the image are not missed in the second SAM segmentation stage. For the second SAM segmentation stage, the points obtained from the first stage are used as point prompts, considering only the highest semantic level among SAM's three semantic levels. This approach minimizes over-segmentation and allows our method to focus on semantic at the object level. Subsequently, we apply NMS, threshold filtering, and post- poot-processing obtain high-quality object-level masks following SAM's methodology [17]. The resulting object masks for our method are displayed in Figure
Two
B
Prompts
Social Story Generation. We carefully designed the prompt to guide the LLMs in generating coherent and easily understandable social stories based dense captions. The system prompt and user prompt are depicted in Figure 7 To ensure synbol-based referencing, we explicitly instruct LLMs not rely coordinates but instead use symbols for reference. Additionally, require the generated paragraphs to focus on social contexts.
SocialPrompt the PIPA dataset. The PIPA dataset comprises 16 social relationship categories, including father-child, mother-child, classmates, loves/spouses, presenter-audience,, teacher-student, trainer-traince, leader-subordinate, band members, dance team members, sport team members, and colleagues. Figure g illustrates the prompt utilized for the PIPA dataset in the zero-shot setting. We provided a detailed explanation for each category within the prompt. Furthermore, the SocialPrompt includes manually constructed in-context examples.
friends, siblings,
SocialPrompt on the PISC dataset. Figure illustrates the SocialPrompt utilized in the PISC dataset, specifically in the zero-shot setting. The PISC dataset comprises 6 social relation categories:
14

---

[Systen Prompt]: You are an expert En generating only one naturally fluent and flawtess paragraph based sust follow Illustrate the spatial relationship and depict the interaction between different people. not use coordinate to describe.
use symbols <o..> and <P..> when referring objects people.
[User Prompt]: Here have 4-tuple [xt,yI,,,h] to depict the position box that frames the objects or persons, where [x1,y1] neans the coordinate of the upper Left corner of the box and [w,h] neans the width and Length of the (coordinnte::(d..11.. <genders: (gender text of the people])". The structure of objects semontic is Like "(csymbol>:(O... object]]".
The structure of people semantic ccaption=(captionn text of the people], <oge>:[oge text of the people],
Like "fesynbol>[(P..).
only informotive and nature paragraph based on the given information (,, and following rutes:
<oopiinn:[caption
6. Image Resolution: (width))(heighht)
Image Caption: (caption)
c. Inoge Scene: (image caption scene)
d. People and Objects
Semantic: (regian semontic)
There sane rules:
people semantic, Depict the spatial relationships between individuals and objects, as well as the spotial relationships between people.
Pay nore
reference
Must use symbols <o..> and <P..> when referring to objects and people.
coordinates [x1, [x1,ya], [w,h] nunbers show position informotion
object.
Pay more ottention to the social scene and describe the social event in detail. Exploin how each person event.
object
No more than 15 sentences. Only paragraph.
Figure 7: The prompt used for social story generation. GPT-3.5 Turbo model is used for caption fusion. The system prompt lists some key rules and the user prompt details the task definition.
commercial, couple, family, friends, professional, and no-relation. We have also included the definitions of these six social relation categories within the prompt.
SocialPrompt after GSPO. Due to the time and iffort-intensive nature of manually designing prompts, this paper introduces the Greedy Segment Prompt Optimization method. For each segment, we employ ChatGPT to generate multiple candidates. As for the in-context examples, we also randomly select several samples from the training dataset. Here, we employ Vicuna-7B [2 for training to obtain the optimized prompts. The optimized prompt on the PIPA dataset is illustrated in Figure while that the PISC dataset is shown Figure
15

---

(Systen Proupt]: You are expert asststont in recogntatng sociot relationships betwees people based on textual descriptions.
You eust Follow these rules:
The unswer can only be one of the 16 Listed sacial relotionahips.
dive the sost Libely unswer and dun't refuse to answer.
if can't 't decide, then randonly select one.
output the first onswr to the first sentence.
(User Proent): As social relations expert, you have the shill to accurately identify the category of social relatianships portraped in teage based oo text description, Your expertise covers distiact types of social relationshps, with each pair of individuals folling under one of these 26 categories. Using the pravided information, you drow inferences deternine the most (teely type of soctal relationship dentcted to an targe, Your final output should be one of 16 distiact types of soctal relattonships, defined as follows: (efother-ctloo, esiiiiggs) (clessnttes), clavers/spouses) greserter-evieenee <teacher-studetty acebers), <dance toos conbers), esport tean scebers), ccolleggues)).
cesthr-child, -groddp-grmmntlo. cgrando- mmomioo-grenachttee ofriends),
treinee, <Leader
cband
Here definition:
-father-childe the relationship betwees a futher and his child, reforring to sole who becones biological or Legal father couther-chile> relationship
referring fesale sgrandpa- gromdchtio: The relstionship octween grandfather and his grandchtla, referring a sole becones the gronsfather ore or granschttaren.
biolagical tegol
ore or eore oitidres.
wrremba-grrndhhlddss grandchitdres.
relationship betwenn grondauther and grandchild, referring fooule
the individunls establish intimste connection, usallyy
grandaother
offllndss: relationship betwren
esperiences, backgrounds.
individuls cclasswrtess: relattonshrip between students study
porests bloed relutions.
clovers/spuuses: The rosmtic relationshtp between individuals, Listeners, clients.
taclude worriage relattonship.
aufterc: relationship between
(usually prefesstonal)
delivers&
presentation
(istaners, where teacher (usually prefesstonal)
stacher-students: The relationship between
teports knowladge, skills, and values the student.
11. ctruiner-trsiine trginee:: the relationship betwees trainer and one core trainess, where the troiner ixports specific knowledge, shills, and techeiques.
12. cleader-subrrieette
The relationabip between Leader and their subordinates, where the Leader halds managerial position in organization or institution, guiding and directing the activities of their subordinntes.
13. coard nonberss: The relationship between austctans or stagers who fore group to perfors euste togethor.
14. caance teoe rumbers:: The relationship betwem doncers sho form group to perfore donce routines together.
15. esport teon resbers:: The relotionship betwesn othietes who fore tean to compete in various sports.
16. ccolloogues:: The recationship oetween individuels who work in the seme organizatton or compony.
(3. image description]
In tnis tange with a resotution of 18282768, capttwcting scene unfolds on sun-kissed beach, Captured to the frame are a wasan and a young giri, their presence adding sense of joy and tranuritty to the serene surroundings. The young girl, desated or PI, con be seen steting the sandy ground, her tanocent curtostty shintag through her bright eyes. Hearby, the Hosan, referned to as P2, grocefully bends her leg, tating in the beauty of the shoretine. The composition shilifully portrays the spottal relationship between individulss and objects, as sell as the spotial relationships between people. Vartous objects entonce the scene's 's vibrast nerrotive. az depicts a suited aen confidently holding a cell phone, exuding professional curo. the focused ottention on the people sccuntic contisves with another shot of f3, the youg girl leisurely seated on the gowdery sand. Naving on to other objects that add chare and interest to the taage, 02 presents a girl sitting with her lags crossed, shile 0s shascoses a content blace eat resting on wonden floor. black and shite dag proudly stonds next to white visuel narrative extends to 05, festuring a box aderned with picture of two chéldren, evozing sense of imocence and playjulness. 06 offers vibeast red and white ba set against a contrasting Moch bockground, injecting a touch of shiteyy. The colorful ensemble continues with 07, capturing the striking tmage of a blue and btack best with a vihrant red bas. 68 showcases shite and plne table adonved with a picture of sosos, adding a festeine touch to the beach scene. Lestly, os and ane portray white house and a white picturesque environeent Through careful ottentioe to
in br, creating a delightful controxt. the
garage respectively, both complevented positions and the objects threselues, this image offectively conveys a sense of hurvany and istrigur withie the bounds of the beach setcing.
roofs, completing
(2. Question]: grandencLas, cgrandes grandchcLd,, ofriends), (stalings), celassastes>, doveers/pouses,, sgresenter auffonces, Scsecher-studentt stretner-tretseee, cleoder sipordinntes eband acebers), odonce tese oesbers), csport tean seebers), ccollongues>).
Libely relationships
only from (sfather-chiidd, coother-childs, egrandpa-
(3. Answer]: [The ansver chil)). Bosed on setting and the activities depicted, Likely inference relotionship. aother-child relationship. considering one inference could "ather-child" relotionship. dysesic.
dynovic young girl
ucesn, young together
sutting could suggest faxily outing,
differmnce suggests
Figure& 8: The SocialPrompt on the PIPA dataset.
16

---

[Systen Proxpt]; You are on expert assistant in recogntzing sectal relottonships between people based on textunt descr descriptions.
You sust follow these rules:
The ansver can only one of the Listed sacial relationships.
Géve the sost Lihely answer and don't 't refuse to answer.
can't decide, then randoely select
Output finel
first
[User Proept]: Possessing expertise in sociol relations, you hold the profictency to correctly cotegorize the social relationships depicted in lmage, anolyzing textuol description. Your hnoxledge unique lndiviuuls probable type unique relationship categories, outlined: (#frtends), cosely-- sprofessional>, ccomercial>, ene-relationships).
social categories. the provided details, derive relationship portrayed image. fisst ccouples,
relationships, with
conclusions
fall
definition:
1. ofriends); bend between individuals rooted in sutuot respect, shared experiences, and a genuine Liking for each other, often encospassing corpantonship and trust.
conrection grounded in Lineage or Legat bindings, Like wedlock or guordtanship, where individuals uphold favitiol comniteent or shore tanol ties.
2. restoyy
3. ccouples: Ae Intimote wnton betwen two people, warked by deep offection, autust understanding, and shared aspirations for the future.
sprofesstonal:: consection foreed through occupational
pursuits, collaboretions, where iedividwels business interactions, transactians, vutuot
forces to achieve autsol abjectives or enhance professional standiag.
:comercia(>: bond forged bustnes--oriented espirations.
parties
collaborote realize
coe-relationhiip: absence of any discensible Link engagement between individuals parties, suggesting commnolities, respansiailitien or offiliations.
(3. Lmage description]:
In the bustl tng scone of porade, group of police officers horseback coptiuates the attention of the crowd. Asong then, weeun weoring hot and scorf (<B1>) stands tall, exuding confidence. Close by, weean purple scorf and black jacket scene. In surgrising tuist, police officer. police officer in a hat and sunglosses (<P3>) odds on etr of systery to the police
(<P2>) comeands authority
joy, adding vearing in police ontfore rides shateboard (sN)), showcastng his youthful spirit, Antther in cootness. sodiles bridles
(cP52),
(1862) hot and
of
syebalizes
police force, group people standing
(s0bs) sigrifies ecement
"Washiagton Police covaraderte force. aon jocket Jeans standing opainst blach background (x063) black jacket and proudly represents
the
blue yellos
engagement of croud. non white scarf (<08>) adds touch of style to the event. a blue and white striped chair with a notching back (c09) provides a resting place for weary officers. tastly, the flee of France displayed on a flopole (<010)) sysbel tzes the internationat cooperation and solidarity within the police force. Together, these individunts and objects create a vibrent and dynaxte ateosphere, showcasing tng the dedication and diversity of the police officers in this parade.
intrigue. person holding up bunch of stichers (1075) suggests
[2. Question]:
what the eost Libely soctol relationships between and P6? sprofess ionals, ccamercials, ino-relationship...
from (sfriends), sosbery- ccouple>,
group police officers, including and P6, partictoating. They both described police officers, indicating profess tonal relationship. focus of description is oe connection rather personal rocantic
Answer]: flnal
sprofessionols). description portrays scene
presence parads, suggesting shared profess evidence supgest fartlial, conserctal, friendship relationship between P5 and PG. Therefore, the aost Lthely social relationship between thee is professtonal one.
Figure 9: The SocialPrompt on the PISC dataset.
17

---

[Systee Proupt]: You are an expert assistont in recogetzing soctal relattonshíps between people bosed or textunl descriptions,
You must fallow these rutes:
The onswer can only be one of the I6 tixted social relationships.
Give the east Libely anser and don't refuse to smaeer.
If con't decide, then randaxly select one.
Output the finol anser tn the first sentence.
specialist in social relations, you possess cupability to preciscly detereine nature of social retatConships shown taage free tortual description. The range of your expertise encospazses i5 urique cutsportes of social relationships, with each dep of individuals cotegorized
(user Proept]:2n your roie
one these. free the teforsation given, you deductions relationsnip type you conctude should wong folcowtng sradde--rodattlee efriends), csthiingss,
the probeate type of soctal relationshpp teage displays.
16 urique categortes: [efather-ohlo, cetter-chidd, sgranpa- ggrenga-grandoot grandottd,, s (teecher-stsdett, ctraiser-treinee, clesser-sdorddaattee edunce scebers), esport aesbers), ccollengues)).
Hore the deflettion:
biologtcst Legal guordiar 2. caother-childo: the band between nother and her child, seboditd by fieate being the biotogicol or Legal guardian one aore chtidren.
1. cfathor-chtlo:: The band between fatber and his child, characterized
children.
3. cgrandpa tyrando--rondchhhLD: grondchELd:: The bond betscen a grandfather and hiss grandchild, depicted by o mote being the grondfather to one gore grandchttdren.
4. sgrundbe-rrnddhhildb:: The band between grondeather and her grondchild, depicted by a fonole bring the grandesther to ane or
sore grundehildrmn.
5. ifriends:: The bond between tso or more individials who faster close connection, often stewning from comson interests, shared exper Lences, or steilar bachgrourds.
6. cstblings>: The band between two or sore Individunts who have comson fortlial ties through either biological or Legal porentage,
7. colasseateso: The band betueen students who shore the acadewic journey in the sour class setting.
racantic bond betwees bes individsala, encosposaing iiging onion that may extend curital relatiorabip. 9. cproaenter-cudieeee The interactive band betwess speaker and group of Liateners, shereis the speaker, ofter a professional
8. clovera/sppses:::
iveisers costent wessoges
attertive autimce.
18. steacher-stulee The educational bond between teacher and one or wore students, where the teacher, often professional, student.
bonulldgee troiine--roiines:
truiner
providing specific
Inouledge,
sxnnder-sulrdisstee the Hierarchical band
Leader their subordiontes,
Lender, sonageriaa position,
navigates orchestrates the activitles
subordiaates within organization institution.
chand areherss: creative band between eustetons
who
create perfora sustc collective.
14. edence teen owsherss: The rhytheic bond betaren doncars who calloborate to chareagraph and perfore dance rastines unit. export tean membera:: the cospetitive band between athletes who cruiganut into teun striving to achieve in varioux sport ing events.
15. coollengues:: The professional bond between individonls who shore common working enviromennt within orgarization
(3. image description]:
In neeting race, two cen in suits are engaged in a professional discussion. The first san, <PI>, stands on the Left vide of the rown, avaring u suit with a red tie. He is in his mid-5ts and has a sorn seile on his foce. The second son ep2b, stunds on the right side, vearteg o suit and tie, He is in hts carly 285 and gesturing with his hards as he speats. In front of thee, there is chate, c02, which adds touch of color to the scere. Bebtin the sen, there is a woll with the words "the best way to get your," " <Bl>, rephastring the iaportance of the certing. on the wall, there is also red heart, cob, sysbaltzing passion and dedication. additionally. there is red berst on the flaor, there are pair red shows cos,, which complaamt the averoll aesthetic. The cussination of these abjects creates visurlly appesting they shabe hands, speotttring nutual respect cellamoratton. Overall, exchange between two indtvidupls professtonal setting, with the onjects enhancing the andtance sattigg depth cosporttion.
blace trie, cot, placed on norry table, adding a touch of style to rose.
professtonel. steesptere for the secting. The ietoraction between
<P2,, is evident scene captures significont nosent teteraction
een,
overall
Questton]:
what the Eteely soctøl relationships
P2? only free (efather-chilss, ousther-hhidd,, grandchtids, cgrandas grandchtids, ofrtends), estoctngs), cclassartes), <louesssppuees> ggnesnter-quuteeees <teacher-students, teas
<trotaer-trsteee, Keoder-subordesste,, chand arabers>, <dance cenberss,
cooleenge)))).
(3. Answer]: [The finel asswer is coolooguess]. dased on the descrtption of the taage, the gost Libely soctøl relationshio between Pl and P2 that they are collesgues. The setting of acetinn room and their professtonal otttre sugpest vore-related context. The descrtatton esphastzes thetr professtonal discusstor and the interactton between then, Indtcating professtonal relotionship nather than personal or foetlts! one.
Figure 10: The prompt after GSPO on the PIPA dataset.
18

---

(Systen Proupt]: You are shilled otde in identifying social connections dwong individuots through textuet depictions.
You are bound these guideltnes:
Your respanse sust solety full within one the specified social relations. without abstaining.
Furnish the proboble
In fedectsion, aohe randos chotce froe List. opening
Deliver ultieate
(User Proept]: Pessessing expertise in social. relations, hold the proficiancy correctly categorize the secial urique of social anolyzing its textuel description. Your hooueedge provided relationships depicted tn individuals aligning relationships, conclurions ascertata unique should fall sprofesstonals, <comercials, <no-relattonshipp))
probable type of soctal relationship betng portrayed as teage. finet deterstnation relationship categories, as outlined: (Ifriends), cfuvily-ewabers), ccouples,
of
Here is the definttion:
1. ofriende:; A bond between individunts rooted in vutuel respect, shared experiences, and genuine Liking for each other, often encospassing conpantonship and trust.
2. sjaxily-rebbess:
connection grounded in Lineage or Legal bindings, libe wedlock or guardtanship, where individuals uphold farilial comntteent or shore gonerationst ties.
3. <couples: As intioute usion between people, earked by deep offection, vutuot understunding, and shared aspirations the future.
sprofessionats: consection
autuol objectives through accupational dealings, pursuits, collaborations,
individuals join transactions, eutuel ventures, parties forged bustaess-oriented esptrations. finorcial
forces
standing.
business
collaborote
6. ano-relationship:: absence of any disceriible Link engagement between individuals parties, suggesting no comvonotities, responsiiiiities or affiliations.
(3. tmage description]:
In this scene at strport, womon with blach bag stands in the center, surrounded by diverse group of people and shirt and jeans, identified purple shirt and jeans, identifted epb, stunds nearby. on the right, soe in black shirt and red hut, idmitified as white striped shirt, identified <ea, holds stands
objects. to her left, person
Leans against astal bax, while girl is
<PI>, corries blace suitcase with yellow while
attached. Senied foreground, scean bending red yellow stripe, son black airport, with the indivisuals engaged verious activities person object contributes the bustling etvosphere of and the objects representing the Luggage and personal belongings of the trovelers.
Logagge
(dentifted outfit, idettified stonding briefcase
black jusps box, strap,
background,
sorgeetiaall.. The objects
pick up frishes,
green yellow stripe, wollet
long tie, son standing is of window,
(2. Question]:
what are the rest (thely sociol relationships between P4 and P5? Choose only one from (sfrtends), sobers, ccouples, sprofessionals, (comurctat), cno-relationsh(pp..
Answer]: fieel
ano-retationshhpp.. descriation portrays individunts standing specific indicatiin of reletionship likely relationship between
diverse They Therefore,
individuals
any explicit interaction
reletionship.
Figure 11: The prompt after GSPO on the PISC dataset.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in the Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not involve theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included an Algorithm to clearly demonstrate how to reproduce our method. We will also release the code.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
The source codes will be made available to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc///uuii wwwpc//ggue////eeeeeeooc CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included it in the Experiments Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: It is not included in all previous work in this field.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included it in the Experiments Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte::///eeris..eeeee Bhttghhrtrrr////rrpp......////tp
Answer: [Yes]
Justification: Our research conducted in the paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impact in the Conclusion Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research doesn't train new models. We use open-sourced foundation models, and any safeguards they used can be applied to our method.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited the original paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwprspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "AZl67WqVdY", "reviewer": "Reviewer_9pM3", "summary": "This manuscript introduces SocialGPT, a modular framework designed to enhance social relation reasoning by combining Vision Foundation Models (VFMs) and Large Language Models (LLMs). SocialGPT utilizes VFMs to convert image content into a textual social story, followed by LLMs performing text-based reasoning. The paper further introduces the Greedy Segment Prompt Optimization (GSPO) algorithm to optimize prompts for LLMs, addressing the challenges of long prompt optimization. The proposed method achieves competitive zero-shot results on social relation recognition tasks and offers interpretable answers.", "questions": "- How can we evaluate generated social stories? It would be great if the authors could show how GSPO improves the quality of generated social stories.\n- How GSPO can be performed without the ground-truth answer? The current formulation in section 4 seems to require the ground-truth to define the loss objective.\n- What are the differences between the social story of SocialGPT and social relationships used in baselines? I feel that Image-based text explanation is not new.", "limitations": "See the weakness section and question above.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The GSPO algorithm provides an efficient method for optimizing long prompts, significantly improving the performance of LLMs in social relation reasoning tasks.\n- SocialGPT achieves competitive zero-shot results on PIPA and PISC datasets, demonstrating the effectiveness of the proposed approach without additional model training.\n- By leveraging LLMs for reasoning, SocialGPT can generate language-based explanations for its decisions, enhancing the interpretability of the results.", "weaknesses": "- The approach involves substantial computational resources for both the perception and reasoning phases, potentially limiting accessibility and scalability for some users.\n- The experiments, while promising, are primarily conducted on two datasets. Further testing on a broader range of datasets and tasks would strengthen the generalizability of the findings.\n- The method assumes that the visual context provided by VFMs is sufficiently detailed and accurate, which might not always hold true in diverse real-world scenarios.\n- The compatibility of the proposed method seems to be limited; Table 5 implies that LLaMA2-based SocialGPT performs very poorly compared to Vicuna. The proposed framework may work only for specific types of models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Please find our responses to specific queries below. We hope that our responses will reflect positively on your final decision.\n\n**W1: Substantial computational resources**\n\nOur method leverages multiple foundation models, which may initially appear computationally intensive compared to traditional social relation methods that use dedicated networks. However, the paradigm shift introduced by foundation models offers a significant advantage: they enable solving multiple downstream tasks with just one set of models using different prompts, rather than training a separate model for each task. Using a single set of foundation models for diverse tasks, as demonstrated by the success of ChatGPT, is not only economical but also enhances scalability and accessibility. Our method facilitates deployment in scenarios where intelligent agents are expected to handle multiple tasks simultaneously. Therefore, our approach ultimately presents a more scalable and accessible solution, aligning with the future directions of intelligent systems deployment.\n\n**W2: The experiments are conducted on two datasets**\n\nOur choice to conduct experiments on two datasets (PIPA and PISC) aligns with common practices in the field. Previous SOTA methods such as Dual-Glance, GRM, SRG-GN, GR2N, and TRGAT have utilized these datasets, either individually or in combination, as benchmarks.\n\nAs there are no other commonly used social relation datasets, following the suggestion, we further collected and annotated a small dataset consisting of 100 samples from diverse real-world scenarios on our own. We then directly tested the zero-shot performance on this dataset, and the results below show that our method significantly outperforms the previous open-sourced SOTA method, GR2N.\n\nTable R1-1. Results on a new dataset\n|Method |GR2N|Ours(GPT-3.5)|Ours(Vicuna-13B)|\n|-|-|-|-|\n|Accuracy (%)|48.0|63.0|62.0|\n\n**W3: Detailed and accurate visual context**\n\nWhile converting all information from an image into text with VFMs can be challenging, obtaining **social-related** visual information sufficient for later reasoning is easier and feasible. Our SocialGPT introduces several designs to facilitate such a process: generating dense captions with SAM to cover all objects, inquiring social-related questions through BLIP-2 to obtain task-oriented captions, and generating social stories to depict the image from a social relations perspective. With these designs, our SocialGPT, without any fine-tuning on social relation datasets, achieves better performance on the PIPA and PISC datasets compared with SOTA methods that are specifically trained on these social datasets. This illustrates the effectiveness of the proposed framework. Furthermore, compared with previous methods trained on social datasets, foundation models usually provide better generalization in diverse real-world scenarios, which is also validated by the results in Table R1-1.\n\n**W4: The compatibility**\n\nThe performance of our method is directly influenced by the reasoning ability of LLMs, as we use LLMs for final social relation prediction. Table R1-2 shows that with a more advanced LLM version, Llama 3 achieves performance comparable to that of other LLMs. These findings illustrate that our framework is adaptable and can effectively leverage advancements in LLM technology, showing the compatibility of our method.\n\nTable R1-2. Results with different LLMs\n|LLM|GPT-3.5|Vicuna-13B|Llama-3-8B|\n|-|-|-|-|\n|PIPA|64.1|66.70|67.82|\n|PISC|53.43|65.12|65.34|\n\n**Q1: Evaluate social stories**\n\nTo evaluate the quality of generated social stories, we adopt another LLM (GPT-4) as the judge. We input two different social stories for the same image into this judge LLM and ask it to decide which one describes a more vivid and detailed social story. Our social stories are generated using the prompts shown in Figure 7 of our main paper. We compare our results with a baseline method that instructs LLMs to fuse dense captions without our carefully designed prompts. The results in Table R1-3 show that our generated social stories are of high quality.\n\nTable R1-3. Social story evalutaion\n|Method|Baseline|Ours|\n|-|-|-|\n|Preference|0.13|0.87|\n\n> It would be great if the authors could show how GSPO improves the quality of generated social stories.\n\nWe want to clarify that GSPO is not used for improving social stories; instead, it is used to find a better SocialPrompt (see Figure 2). This improved prompt can better instruct LLMs in social relation reasoning. We have included the SocialPrompts before and after using GSPO in Figures 9-11 of our main paper.\n\n**Q2: Ground-truth answer**\n\nWe want to clarify that GSPO needs the ground truth class labels y but does not have textual ground truth since it deals with LLM outputs. This is the \"free-form target\" challenge, as mentioned in our paper. Our solution involves constructing the textual ground truth that starts with \"The final answer is str(y)\" and only supervising the first sentence of the LLM's output with it. More details and strategies to complement this design are presented in lines 194-226 of our main paper.\n\n**Q3: Image-based text explanation**\n\nSorry, we do not fully understand this question and are unsure what is meant by \"social relationships used in baselines\". Could you please elaborate further? We are more than happy to answer your question. We try to answer it below based on our current understanding.\n\nA social story is a textual paragraph depicting the social context of an image, while social relationships are the target categories, represented by a one-hot numerical vector without any textual explanations in previous social relation methods. Image-based text explanation is not the main focus of this paper. Instead, our paper aims to provide a zero-shot baseline for social relation recognition using foundation models. Building on this framework, the GSPO is proposed to automate the prompting process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C3PEiWKQXi", "reviewer": "Reviewer_zGW2", "summary": "This paper proposes a framework called SocialGPT for social relation reasoning, which combines vision foundation models and large language models. A greeedy segment prompt optimization methods is also proposed to prompt LLM. Experimental results show the effectiveness of the proposed method.", "questions": "--- Is the design of using LLMs for social story generation optimal, and why? Also, have the authors tried other approaches to generate social stories from dense captions instead of using LLMs?\n\n--- In the part of reasoning with large language models, the social relation reasoning prompt is artificially divided into four partitions: System, Expectation, Context, and Guidance, but the motivation and reasonableness of such a design is not elaborated in the paper.", "limitations": "Please see the weakness and limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "---The paper is well organized and written. \n\n---The idea of combining VFMs and LLMs is reasonable.", "weaknesses": "--- The paradigm of using VLMs for perceiving and LLMs for reasoning is currently a common solution for multimodal tasks. The main difference of this paper seems to be the use of a generated social story as the representation of visual content. As stated by the authors, LLMs perform best when working with human-readable natural language and often struggle with arithmetic reasoning tasks, which is why they design an additional process to generate social stories. However, the generation of social stories is also done by LLMs, which also suffer from the above difficulties. \n\n--- The authors propose a candidate set consisting of alternative prompts for each segment and select the best-performing prompt from their combination. The final prompt is obtained by selection rather than generation, which limits the upper bound of the performance on the manually collected candidate set. \n\n--- The function of SAM is to distinguish individuals in the image and obtain their coordinates. However, in the social story generation phase, the LLM (Large Language Model) discards the coordinates, retaining only the semantic information and losing the positional information. Conducting social relationship reasoning purely based on semantics may be insufficient. For example, in Figure 2, the social relationship is identified as a sibling relationship (brother and sister), but there are two boys in the image, both fitting the given description of \"stands out in his vibrant red and green striped pajamas,\" making it unclear which individual P1 refers to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments! Below we address the detailed questions. We hope that our responses will reflect positively on your final decision.\n\n**W1-1: Common solution**\n\nWe are fully aware that leveraging foundation models for vision tasks is a growing trend, which also motivates our work. We would like to emphasize that while our SocialGPT provides a simple zero-shot social relation baseline using foundation models, our major technical contribution lies in our GSPO. It addresses the long prompt optimization issue, automatically optimizing the SocialPrompt.\n\n**W1-2: Difficulties for social story generation**\n\nRecent studies [1][2] on LLMs have shown that decomposing complex tasks, such as mathematical reasoning, into sub-tasks for multi-step reasoning, effectively reduces difficulties, and improves performance. Therefore, instead of directly using LLMs for social reasoning with raw, dense captions, we first integrate these captions into a coherent social story. This story conveys positional information through text rather than numerical coordinates, and then we perform social reasoning with the social stories. This two-step decomposition has reduced the reasoning difficulties and achieved higher performance. We conducted ablation experiments on social stories, and the results in Table R2-1 show that the introduction of social stories greatly improved performance.\n\nTable R2-1. Ablation on social story\n|social story|PIPA|PISC|\n|-|-|-|\n|without|45.31|37.42|\n|with|61.58|45.13|\n\n[1] Decomposed prompting: A modular approach for solving complex tasks, ICLR 2023\n\n[2] Least-to-most prompting enables complex reasoning in large language models, ICLR 2023\n\n**W2: Upper bound**\n\nThere are many works on automatic prompt tuning for LLMs; one popular direction is performing greedy exhaustive searches or selections over tokens. These methods [3][4] are well-recognized in the NLP field. They typically perform well and do not face any upper-bound issues. For example, AutoPrompt [3] identifies a candidate set of the top-k tokens and performs greedy selection, which has garnered over 1500 citations. Our GSPO further proposes performing a greedy search by utilizing gradient information at the segment level to address the long prompt optimization issue. \n\nWe want to clarify that, besides the example segment, the candidate sets for other segments are generated by ChatGPT and are not manually collected. For the example segment, the candidates can be chosen from the entire training set, thus GSPO is also not constrained by a limited candidate set. The results in Table 5 of our paper also validate the effectiveness of our GSPO.\n\n[3] Autoprompt: Eliciting knowledge from language models with automatically generated prompts, EMNLP 2020\n\n[4] Automatically Auditing Large Language Models via Discrete Optimization, ICML 2023\n\n**W3: Losing positional information**\n\nSorry for the misunderstanding. We want to clarify that in our social story generation phase, we explicitly instruct LLMs to preserve positional information, rather than lose it. We discard only the numerical coordinates and use textual sentences to describe positional information. The corresponding prompts \n> Illustrate the spatial relationship and depict the interaction between different people \n\nand \n> Depict the spatial relationships between individuals and objects, as well as the spatial relationships between people\n\nare shown in Figure 7 of our paper. \n\nAdditionally, our generated social story could include very detailed positional cues, as exemplified in Figure 3 of our paper: \"At the center of the image, a young <P1> boy\", \"To his right, a middle-aged <P2> man\", \"On the left side of the frame, a <P3> woman\". As for the example in Figure 2, it actually contains weaker positional information due to the performance fluctuations of LLMs: \"gathered\", \"Sitting beside him is a girl, <P2>\". While this is not sufficient to reconstruct the original image details, it is adequate for LLM reasoning, which is our primary requirement.\n\nTo further verify the effectiveness of our position-related prompts in the social story generation phase, we compared scenarios with and without these prompts. The results in Table R2-2 show the effectiveness of our positional prompts.\n    \nTable R2-2. Ablation on positional prompts\n|Method|w/o pos|w. pos|\n|-|-|-|\n|Accuracy (%)|58.32|61.58|\n\n**Q1: Other approaches to generate social stories**\n\nWe considered and evaluated two alternative approaches alongside our LLM-based method: 1) BLIP-2: instructing BLIP-2 to generate social stories. 2) Concatenation: combining all dense captions into a single paragraph. Table R2-3 shows that the LLM-based method attains the best performance.\n    \nTable R2-3. Results of different social story generation approaches.\n|Method|BLIP-2|Concatenation|LLM|\n|-|-|-|-|\n|PIPA|43.85|45.31|61.58|\n|PISC|40.17|37.42|45.13|\n\n**Q2: Four partitions**\n\nThe Four partitions are well motivated and supported by the recent research progress in LLM. Research [5] shows the system prompt \"establishes the foundational context for the model’s responses\". Study [6] demonstrates that assigning roles to LLMs can enhance their performance, which illustrates the importance of the Expectation prompt. According to [7], incorporating background knowledge benefits LLMs reasoning, strongly motivating the use of the Context prompt. As for the Guidance prompt, using in-context examples has become a common practice to improve LLM performance [8].\n\nBesides the above evidence, Table 2 of our paper provides a detailed ablation, validating their effectiveness.\n\n[5] Jailbreaking GPT-4v via self-adversarial attacks with system prompts, arXiv:2311.09127 (2023).\n\n[6] Better Zero-Shot Reasoning with Role-Play Prompting, NAACL 2024.\n\n[7] Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering, ACL Workshop 2023.\n\n[8] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEZ8r6aBRK", "reviewer": "Reviewer_RdU3", "summary": "This paper introduces SocialGPT, a modular framework for social relation reasoning that integrates the perception capabilities of Vision Foundation Models (VFMs) with the reasoning capabilities of Large Language Models (LLMs). To optimize prompts, the authors propose GSPO, a segment-based optimization algorithm for automated prompt tuning. Extensive empirical results validate the effectiveness of SocialGPT both quantitatively and qualitatively. GSPO consistently enhances SocialGPT's performance across various LLMs, and case studies demonstrate the framework's generalizability and interpretability.", "questions": "Please see the weaknesses section above.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized, with a logical flow and clear explanations of each step.\n- The proposed SocialGPT framework innovatively combines perception from VFMs with reasoning from LLMs, achieving competitive zero-shot performance and offering potential explanations for its reasoning process.\n- Extensive experiments, ablation studies, and case studies comprehensively evaluate the framework's effectiveness.", "weaknesses": "Section 3.2 mentions that using precise coordinates can pose challenges for LLM numerical reasoning. However, it appears in Figure 3 that the objects' positional relations in the social story are inferred from numeric coordinates provided in the dense captions with symbols. Does this coordinate-based inference lead to similar numerical reasoning challenges? Additionally, how are relative positional relations conveyed here using referral symbols?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and the positive assessment of our work! We are happy that you find our framework **innovative** and our experimental evaluation **comprehensive**. Below, we detail our responses to the review concerns.\n    \n**W1-1**\n\n> Does this coordinate-based inference lead to similar numerical reasoning challenges?\n\nThank you for the question! Indeed, using precise coordinates for social reasoning in LLMs poses substantial challenges as it requires the model to understand spatial relationships and perform social reasoning simultaneously, which can be complex.\n\nTo mitigate these challenges, we adopt a two-step approach in our methodology: first, we generate textual social stories by converting coordinates into descriptive textual spatial information; subsequently, we use these stories for social reasoning. This decomposition simplifies the cognitive load on the LLM by separating spatial understanding from social reasoning.\n\nRecent research [1][2] on LLMs supports this approach: they have demonstrated that breaking down complex tasks into simpler, manageable sub-tasks for multi-step reasoning significantly enhances LLM performance by reducing cognitive demands. \n\nOur empirical results further validate this method. As shown in Table 2 of our paper, the performance of our proposed method surpasses that of the coordinate-based method, underscoring the effectiveness of our strategy in reducing the difficulties associated with numerical reasoning in social contexts.\n    \n[1] Decomposed prompting: A modular approach for solving complex tasks, ICLR 2023\n\n[2] Least-to-most prompting enables complex reasoning in large language models, ICLR 2023\n    \n**W1-2**\n> How are relative positional relations conveyed here using referral symbols?\n\n\nThank you for the question! To address this issue, we instruct the LLM to describe spatial relationships among objects and people using textual descriptors instead of relying on precise coordinates. This is achieved by using specific prompts that focus on textual descriptions rather than numerical data. \n    \nFor instance, we use prompts such as: \"Depict the spatial relationships between individuals and objects, as well as the spatial relationships between people\", \"Must use symbols <O..> and <P..> when referring to objects and people\", and \"Do not use coordinates [x1,y1,w,h], [x1,y1], [w,h] or numbers to show position information of each object\". These prompts guide the LLM to generate social stories that describe spatial relationships. The full set of prompts used for generating these social stories is detailed in Figure 7 of our paper, providing a comprehensive view of our methodology.\n    \nFigure 3 in our main paper shows an example of how the generated social stories describe the relative positional relations: \"At the center of the image, a young <P1> boy\", \"To his right, a middle-aged <P2> man\", \"On the left side of the frame, a <P3> woman\". We observe that LLM can reason the relative positional relations with our carefully designed prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "49Gle3BO6p", "reviewer": "Reviewer_aaEq", "summary": "Paper proposes a pipeline method of orchestration pre-trained foundation models to solve the social relationship classification problem. It uses vision models to extract information in text about the scene in the form of caption. Relevant information, i.e. age, gender, general description, of individual persons and objects are also extracted in text form via instance segmentation + masking + captioning. The generated text are then further converted to Social Story with a LLM. With the novel prompt engineering method, GSPO, another LLM will then generate the social relationship from the Social Story.\n\nExperimental results on the challenging benchmarks, PIPA and PISC, indicates its strong performance with zero-shot setup. Extensive ablation studies were also done to evaluate the contributions of the various components. In particular, it clearly showed the merits of the \"Social Story\" design.", "questions": "1. Will/has the authors consider using pairwise attributes, besides the individual person attributes. E.g. relative age between pairs (older/younger), same/different clothings for the model? \n\n2. Why are only 2 attributes (age/gender) used for the person instance? In prior works, other attributes such as wearing uniform are important attributes for certain type of social relationship, e.g. team members?", "limitations": "(minor) There may be some unintended negative consequences of automatic classification of social relationship.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Paper proposed a novel method to solve the challenging social relationship classification problem. The proposed method cleverly combine several state-of-the-art foundation models in a logical, intuitive, and yet non-obvious design to achieve state-of-the-arts experimental results.", "weaknesses": "1. Besides the clever design of the pipeline, the direct technical contributions is slightly on the weaker side as there is no obvious technical breakthrough. The proposed GSPO appears to be the main new technique introduced. However, I am not an expert in this area and will defer to other reviewers on its technical novelty and merits.\n\n2. (minor) The use of the generic semantic segmentation model (SAM) may not be the optimal choice. There are much stronger Human Instance Segmentation methods which can replace the paper's custom SAM method. Such methods are specifically trained on person dataset to handle various challenging scenarios unique to human segmentation, e.g. heavy occulsion, human-like objects (e.g. maniquinn).\n\nLing, E., Huang, D., & Hur, M. (2022). Humans need not label more humans: Occlusion copy & paste for occluded human instance segmentation. BMVC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and the positive assessment of our work! Below, we detail our responses to the review concerns.\n\n**W1: Technical novelty of GSPO**\n    \nThank you for acknowledging the \"clever\" design of our pipeline. We agree that our major technical contribution lies in our proposed GSPO. Our SocialGPT transforms a visual classification task into a generative task of LLMs, presenting unique challenges in long prompt optimization that cannot be effectively tackled by existing methods. To address this issue, our GSPO method innovatively performs a greedy search utilizing gradient information at the segment level. The experimental results further validate the effectiveness of our proposed GSPO algorithm.\n    \n**W2: Human instance segmentation methods**\n\nThank you for your suggestion! In response, we incorporated the model suggested in [1] into our pipeline to directly compare it with our custom SAM method. The experimental results, presented in Table R4-1, indicate that the SAM model outperforms [1], achieving higher accuracy. The reason for SAM's superior performance is its ability to segment not only human figures but also non-person objects. This capability is particularly beneficial for social relation recognition, which relies on contextual object cues in addition to human figures. We will cite [1] and detail these additional experiments in the revised manuscript to provide a comprehensive view of our methodological choices.\n    \nTable R4-1. Results with different segmentation methods\n|Segmentation| [1] |SAM|\n|-|-|-|\n|Accuracy (%)|57.26|64.10|\n\n[1] Humans need not label more humans: Occlusion copy & paste for occluded human instance segmentation, BMVC 2022.\n    \n**Q1: Pairwise attributes**\n\nThank you for your valuable suggestion! We conducted experiments by adding one more relative attribute to our method. Specifically, we extended the queries to BLIP-2 to include not only individual age and gender attributes but also relative age or clothing differences between pairs.\n\nThe experimental results are summarized in Table R4-2. While the inclusion of relative age did not significantly alter the performance, suggesting that our original age attribute effectively captures relative age information, the addition of clothing attributes enhanced our accuracy by 1.03%. This indicates that incorporating complementary attributes can indeed enhance the performance of our model.\n    \nTable R4-2. Results with more attributes\n|Attributes| Ours | + relative age | + clothing |\n|-|-|-|-|\n|Accuracy (%)|64.10|63.98|65.13|\n\n**Q2: Other attributes**\n\nThank you for your question! As demonstrated in Table R4-2, incorporating additional attributes such as clothing information indeed enhances the performance of our framework. We would like to highlight that the primary focus of our paper is on developing a foundational model-based framework for zero-shot social relation reasoning and addressing the long prompt optimization issue within this framework. The decision on the number and type of attributes to generate task-oriented captions was driven by the desire to establish a simple baseline, which is not the main focus of our work. We initially selected age and gender as they are commonly referenced attributes that provide foundational insights for our zero-shot social relation reasoning baseline.\n\nWhile we acknowledge the potential benefits of incorporating more diverse attributes, we opted to limit the scope of this initial study. We believe that exploring a broader array of attributes presents an exciting direction for future research, potentially leading to further improvements in performance.\n\n\n**L1: Unintend negative consequences**\n\n\nThank you for raising this important concern. We recognize that automatic classification of social relationships can indeed lead to unintended negative consequences. To address this, we will expand our discussion in the broader impacts section of the paper.\n    \nIn this expanded discussion, we plan to outline potential risks and propose mitigation strategies, such as implementing fairness and bias checks, and promoting transparent and responsible usage of our technology. We appreciate this suggestion and believe that addressing these concerns thoroughly will enhance the quality and ethical standing of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received 2 Borderline Reject, 1 Borderline Accept and 1 Accept ratings. While the authors should incorporate the reviewers' comments in their final version to elaborate the motivation of the storyline and the technical and experimental details, it is a solid contribution to the community. The AC reads the paper, the rebuttal, and the discussions and decides to accept it. Congratulations!
Decision: Accept (poster)