SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
Authors: Xuhui Zhou
Hao Zhu
Leena Mathur
Ruohong Zhang
Haofei Yu
Zhengyang Qi
Louis-Philippe Morency
Yonatan Bisk
Daniel Fried
Graham Neubig
Maarten Sap
Venue: ICLR 2024 spotlight (2024)
Keywords: Social
Interaction
Agent
Social intelligence
Large Language Models
Evaluation
Theory of Mind
Primary area: datasets and benchmarks
TL;DR: SOTOPIA is a novel, challenging, and interactive benchmark that could serve as the perfect test-bed and potential incubator for social intelligence.

Abstract: *Humans are social beings*; we pursue social goals in our daily interactions, which is a crucial aspect of social intelligence. Yet, AI systems' abilities in this realm remain elusive. We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence. In our environment, agents role-play and *interact* under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We simulate the role-play interaction between LLM-based agents and humans within this task space and evaluate their performance with a holistic evaluation framework called SOTOPIA-Eval. With SOTOPIA, we find significant differences between these models in terms of their social intelligence, and we identify a subset of SOTOPIA scenarios, SOTOPIA-hard, that is generally challenging for all models. We find that on this subset, GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills. These findings demonstrate SOTOPIA's promise as a general platform for research on evaluating and improving social intelligence in artificial agents.

Published as a conference paper at ICLR 2024
SOTOPIA:
INTERACTIVE EVALUATION
FOR SOCIAL INTELLIGENCE IN LANGUAGE AGENTS
Xuhui Zhou
Hao Zhu'
Leena Mathur Louis-Philippe Moreney
Ruohong Zhang
Zhengyang Qi Haofei Yu
Maarten Sap Yonatan Bisk Daniel Fried Graham Neubig
Language Technologies Institute, Carnegie Mellon University
https://sotopia. world
Sampling scenarios and social goals
Sampling characters
Simulating interactions
It's getting really cold. Any chance con have your blanvee??
Nigotettin
Compotition
am and 1h blanket mone...
Wel L can ve share the dlanker then? could ake both
Friends
Persassion
am not really comfortccee with stay ng Usst ukee to you ser
Scenarios cover lorge range of social interaction tyores
Characters cover o wide range of profiles and relationships.
500, guess that case will layer more dothes then
William Brown
Agunt?
Scenario
Two triends are camping in the wilderness and the temperature drops significantly night
Opernase to Experieno, Conso mnnii
bntcoooniion
Fut more clothes on and move away from William (ntterctio ends))
Shollogg
Agon??
Goal (for Knop blanker you have just for yourself
Extiwersie sicis Nounnniion
SOTOPIA EVAI.
Decrabee
Goal Agent Convince your friend share the blanket you
not achirve social gools end, and their relotionship seems to be worse
Figure 1: SOTOPIA: An open-ended social interaction environment. In each episode, SOTOPIA first sam- ples social scenario context, goals, and characters, and then assigns a social goal and character to each agent involved. Agents (artificial agents or humans) in SOTOPIA role-play characters while attempting to achieve their goals. The agents' performance is evaluated through dimensdammnsionall framework, SOTOPIA-EVAL.
ABSTRACT
Humans are social beings; we pursue social goals in our daily interactions, which is a crucial aspect of social intelligence. Yet, Al systems' abilities in this realm remain elusive. We present SOTÓPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social in- telligence. In our environment, agents role-play and interact under a wide vari- cty of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We simulate the role-play interaction be- tween L.LM-based agents and humans within this task space and evaluate their performance with a holistic evaluation framework called SOTOPIA-EVAL With SOTOPIA, we find significant differences between these models in terms of their social intelligence, and we identify a subset of SOTOPIA scenarios, SOTOPIA A- hard, that is generally challenging for all models. We find that on this subset, GPT- 4 achieves significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills. These findings demonstrate SOTOPIA' 's promise as a general platform for research on evaluating and improving social intelligence in artificial agents.
Equul contributors.

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
Humans' ability achieve and balance complex, multifaceted social goals in our interactions with others is a crucial part of our social intelligence as a species (Kihlstrom & Cantor, 2020; Tomasello, 2021). Even simple social goal such as sharing blanket with friend requires reconciling one's need to stay warm with the friend's 's need for personal space (Figure 1). Successful interaction requires understanding others' intentions and beliefs (Premack & Woodruff, 1978), while taking into account different- and potentially conflicting-social norms and expectations (Goffman, 1959).
Even though recent AI systems have exhibited impressive social skills in certain settings, their social intelligence has yet to be ascertained in a robust way (Shapira et al., 2023a; Ullman, 2023). On one hand, many of the social intelligence benchmarks are not interactive (Sap al.. 2019; Le et al., 2019; Zadeh et al., 2019b), which is sub-optimal for evaluating social intelligence ((Mehri et al., 2022: Hoppler et al., 2022; Lee et al., 2023)). On the other hand, existing interactive evaluation falls short of studying diverse goal-driven behaviors (Zhang et al., 201 8b; Park et al., 2023) or focuses on specific tasks (Wang et al., 2019; Padmakumar et al., 2022: FAIR et al., 2022).
To study dynamic and goal-driven social intelligence, we present SOTOPIA (Figure 1), an open- ended general-domain environment that situates social agents in diverse social scenarios. SOTOPIA is interactive: in multi-turn simulated communication, agents can use verbal and non-verbal com- munication together with physical actions.' It also has a diverse task space: the combination of au- tomatically generated scenarios, goals, characters, relationships, and other agents' policies creates huge and diverse space of tasks. SOTOPIA A evaluates agent performance from multiple dimensions besides the completion of social goals.
In SOTOPIA, we create 90 social scenarios spanning a range of cooperative, competitive, and mixed social goals along with 40 characters with individual personalities, occupations, secrets, background stories, and relationships with other characters ($2), the cross product of which constructs a large task space. Through sampling tasks from this space, we simulate the interaction "episodes" where agents role-play their respective characters and interact based on their private social goals. In this simulation, we not only create and use LLM-based agents, but also involve human participants in role-playing to study the differences between the models' and humans' social intelligence.
To evaluate factti- social interactions, we cannot only consider completing major social goals, as humans' motives often balance multiple implicit goals, such as maintaining relationships, pre- serving finances, gaining information, keeping secrets, and following social rules. Therefore, we propose SOTOPIA-EVE EL ($3) to evaluate agents using multi-dimennional criteria inspired by pre- vious research on sociology, psychology, and economics. We then apply SOTOPIA-EVE episodes in the aforementioned simulation by leveraging both humans and GPT-4 as judges. We find GPT-4 could serve as a proxy to human judgments on SOTOPIA-EVAL, especially for the criteria of goal completion, maintaining finances, and preserving relationships.
to the
Despite larger LLMs typically achieving higher social intelligence than smaller ones, they fall short of collaborating and competing with humans on more challenging tasks (87). They are also highly influenced by their conversational partners and at risk of divulging secrets and violating social rules. However, we do find few cases, where the models produced creative solutions to a problem ($6).
Our contributions are as follows: (A) We introduce and will release SOTOPIA, general-domain interactive environment for simulating goal-oriented social interactions. Designed be extensible, SOTOPIA could be used by future researchers to study and train artificial social intelligence agents with more challenging and diverse tasks. (B) We create SOTOPIA-EVAL, a -ileiioiiennional evaluation framework that analyzes agent performance from range of social dimensions. (C) We automate OOTOPIA-EVAL judgment on some of the social dimensions, especially goal completion. by leveraging SOTOPIA, A, we can assess disparities in social intelligence between models, as well as disparities between models and humans.
by leveraging L.LMs, which we find could serve as a proxy of human
(D) We demonstrate that
In summary, SOTOPIA A is novel, challenging, and interactive benchmark that could serve as the perfect best-bed and potential incubator for social intelligence in language agents.
rrpressenee in text form.
2

---

Published as a conference paper at ICLR 2024
2 SOTOPIA INTERACTION ENVIRONMENT
To address the challenge of evaluating social intelligence interactively, we seek an environment with the following desiderata: (1) Realistic: this is to evaluate and understand artificial agents' behav- ior under realistic scenarios; (2) Mixed utilities: human motives are often driven by both explicit and implicit incentives, and the environment should be able evaluate the agents' performance on multiple dimensions; (3) Open-ended: to support large-scale simulation and evaluation, the envi- romment should be able to produce new tasks satisfying the previous two desiderata procedurally, without heavy human intervention.
In this section, we introduce SOTOPIA and explain why SOTOPIA is well-suited for interactive evaluation of social intelligence. The task space includes realistic scenarios, characters, and rela- tionships which are automatically generated with manual inspection ($2.1). An episode includes the interaction between agents role-playing different characters who each perform actions (e.g. speak ( ("Hello Bob!"). smile and nod, and call 911) to achieve social goals drawn from the task space ($..2). We direct readers to Appendix C for a formal definition of the SOTOPIA environment.
2.1
TASK SPACE
In this paper, we consider tasks that involve two agents, but SOTOPIA is more general and could support the interaction among more than two agents. A task in SOTOPIA is the combination of a scenario context, characters, and their social goals, providing the background of the interaction. Each episode consists of multiple turns of interaction between agents. In this paper, we focus on locally-consistent social goals within a relatively short timespan in single episodes, despite that in the real world, people's social goals are consistently changing from time to time. Note that agents have different observations for the same task: each agent can observe the scenario, their own social goal, and their own character profile. Other agents' social goals are invisible and other agents' character profiles are partially observable, depending on the relationship between the agents.
Complexity of task space The combinations of a scenario context, social goals, characters, and their relationships can shape the space of the optimal behaviors of agents. Consider persuasion task, "asking the romantic partner to stop texting during FaceTimee"" If a romantic partner values conformity, one good way for agent to reach this goal is discuss the problem from social norm perspective; however, if a romantic partner is particularly caring and good at understanding feelings, it might be better to express subjective emotion. Interaction partner's policy also heavily influences the optimal behaviors. Consider another task illustrated in Figure 1, "selling BMW Z3 for no less than $3,400". If the buyer gives high offer, the seller might want to exploit the buyer's eagerness to buy the car and ask for a higher price; while if the buyer gives a low-ball offer. the seller could give reasons why the car is worth more than that or threaten to walk away. When more information (e.g. about personality, decision-making styles, or occupation) is known before the interaction, the seller and buyer could use that knowledge to adjust their strategies as well. The cross-produce of the diverse spaces of scenario context, social goals, characters, relationship profiles, and other players' policies creates a large task space that poses not only a realistic challenge but also an opportunity to evaluate and develop social intelligence in artificial agents. For the rest of this subsection, we will present the design and generation of each axis of the task space.
Characters As mentioned above, the design of character profiles should include several attributes that would influence decision-making We consider the following ones (inspired by Wang et al. (2019)): name, gender, age, occupation, pronouns, personality traits (Goldberg, 1992), moral values (Graham et al., 201 1). Schwartz personal values (Cieciuch & Davidov, 2012), and decision-making style (Hamilton et al., 2016), which are generated through leveraging GPT-4 (OpenAI, 2023). To give the conversations more background, after generating the above attributes, we prompt GPT-4 to generate secret and public information. Two examples of characters are shown in Figure 1. It should be noted that, although generated diverse set of characters, this is still a small portion of the possible character space. Our analysis focuses on 40 characters generated in the aforementioned fashion, and future research using SOTOPIA can easily generate expanded character set.
Relationships Relationships in SOTOPIA have the following effects: (1) scenarios often have relationship constraints; for example, family relationship is required for a family dinner scenario,
3

---

Published as a conference paper at ICLR 2024
but not for scenario involving finding mutual friends at party; (2) different relationships influence an agent's 'ss observation of the profiles of other agents during interactions; for example, a stranger may not have knowledge about another agent's ''s occupation, while romantic partner may know the other agent's personality. To make sampling characters easier for (1) and controlling the interaction for (2), we consider five types of relationships: family,friend, romantic, acquaintance, and stranger. Refer to Appendix B for the limitations of this approach and potential extensions.
We will discuss how (1) is performed in the following paragraphs, while for (2), we created a rule- based mechanism to determine whether the parts of the profiles are visible to the other agent. If two agents are in family, friends, or romantic relationships, they can see everything on each other's profile except for secrets. Two acquaintances can see the name, occupation, gender pronouns, and public info on each other's profile. Two strangers can see nothing on each other's profile. Similar to characters, we prompt GPT-4 (OpenAl, 2023) to automatically generate relationships based on the character pool and manually validate relationships for consistency.
Scenarios We consider scenarios where the agents have both shared and private information about the social task. The shared information is the scenario context: the location, time and other shared information of the social interaction, e.g- "One person is selling an antique chair for $100 on his patio. Another person is interested in this chair." The private information is the social goals which are only visible to the respective agents, e.g. "Your goal is to buy the chair for $80." is only visible to the buyer agent, while "Your goal is to sell the chair for $90." is only visible to the seller agent. However, the as mentioned above combination of scenarios and characters is not arbitrary, since scenarios often imply constraints for the agents. We call this kind of constraint scenario constraints. In this paper, we mainly consider relationship constraints which determines the types of relation- ships between the sampled characters. Similar to characters and relationships, scenarios, including context, goals, and constraints are generated through prompting GPT-4 (OpenAI, 2023). To generate high-quality scenarios with enough coverage of different types of social interactions (as shown in Figure 1), we randomly sample data from previous datasets, including Forbes et al. 2020; Sap et al. 2019; Lewis et al. 2017; Ziems et al. 2023; He al. 2018; 2017, and use them in the prompts to "inspire" GPT-4. The authors manually validate and make necessary changes to all of the generated scenarios and remove 10% of scenarios according to E.2.
2.2 SOTOPIA EPISODES
During the interaction, models and humans are given the social context, a character profile and a corresponding social goal. We will call these models and humans with characters and goals agents, which take turns (in a round-robin fashion, i.e. Agent 1 acts first and then Agent 2 acts and so on) to perform actions in episode. At their own turn, the agent can choose to speak, use non-verbal communication (c.g., hug or smile in Figure H.1), or take a physical action (c.g.. play music in Figure H.2), which are all important components of social interactions (De Stefani & De Marco, 2019). Once an agent chooses one of these three discrete action categories, the agent then generates specific action, i.e. what to say, what gesture to make, etc., in text form. Outside of the three actions, the agent can also choose to do nothing (none) to express silence or allow another agent to finish, choose to leave end the episode. We set the limit of the turns to 20, as we found humans normally can finish most of the tasks in 20 turns. An episode ends either because one of the agents chooses to leave, or reaches the limit of turns. An example episode is shown in Figure 1.
SOTOPIA-EVAL: HOLISTIC SOCIAL AGENT EVALUATION FRAMEWORK
To capture the complexity of what makes social interactions successful, we design a multi- dimensional framework inspired by sociology, psychology, and economics literature. For each episode, agents are scored along each of the following dimensions at the end of the interaction. In the following paragraphs, we itemize all seven dimensions in SOTOPIA, each with score range* in [lower bound-upper bound] form, the explanation, and the literature inspiring us.
Goal Completion (GOAl) [0-10] is the extent to which the agent achieved their goals. Agents' social goals, defined by the environment, are the primary drivers of their behavior (Weber, 1978).
The metric ranges contain semantic implications, for example, negative value in REL. indicates the rela- tionship gets worse while positive value indicates the relationship improves.
4

---

Published as a conference paper at ICLR 2024
Believability (BEL) [0-10] focuses the extent to which the agent's behavior is perceived as natural, realistic, and aligned with the agents' character profile, thus simulating believable proxies of human behavior (Park et al., 2023). Specifically, we consider the following criteria: 1. If the agent interacts with others in a natural and realistic manner (naturalness). 2. If the actions of the agent align with their character traits e-g., personality, values, etc. (consistency).
Knowledge (KNO) [0-10] captures the agent's 's ability to actively acquire new information. This dimension is motivated by the fact that curiosity, i.e. the desire to desire to know learn, is fun- damental human trait (Reiss, 2004; Maslow, 1943). Specifically, we consider the following criteria: What information the agent has gained through the interaction, whether the information the agent has gained is new to them, and whether the information the agent has gained is important to them.
Secret (SEC) [-10-0]* measures the need for agents (humans) to keep their secretive information or intention private (Reiss, 2004). From game theory perspective, leaking secrets often leads to a loss of utility (Gilpin Sandholm, 2006). However, revealing secrets can be a powerful tool to build trust and thus improve relationships (Jaffé & Douneva, 2020). In this dimension, we ask what secret or secretive intention the participant wants to keep, and whether they keep it successfully.
Relationship (REL) [-5-5] captures the fundamental human need for social connection and be- longing (Maslow, 1943; Bénabou & Tirole, 2006). In this dimension, we ask what relationship the participant has with the other agent(s) before the interaction, and then evaluate if the agents' interactions with others help preserve or enhance their personal relationships. Additionally, we ascertain whether these interactions also impact the social status or the reputation of the agent.
Social Rules (SoC) [-10-0] concerns norms, regulations, institutional arrangements, and rituals. We differentiate between two types of social rules: social norms and legal rules. Legal rules encompass prohibited actions and the potential for punishment by institutionalized force, while social norms encompass normative social rules (e.g., it considered rude to speak loudly in library).
Financial and Material Benefits (FIN) [-5-5] pertains to traditional economic utilities as addressed by classic game theory (Gilpin Sandholm, 2006; Burns al., 2017). We consider financial util- ity to be comprised of both short-term monetary benefits (c.g., carnings) and long-term economic payoff' (e.g.. job security, stock holdings, funding opportunities).
RESEARCH QUESTIONS AND EXPERIMENTAL SETUP
Given diverse set of social scenarios, goals, and characters, simulate agents' interactions. This is the first time that we could evaluate general, goal-oriented social agents in an interactive and systematic manner. In the next three sections, we will demonstrate how SOTOPIA can be used to study these questions: (A) To which extent can we use GPT-4 (OpenAI, 2023) proxy for human judgment when it comes to evaluating agents' social interactions ($5)? (B) What are the differences among models ($6) and between models and humans ($7) in their goie-triented social intelligence?
To study these questions, we create 40 agents. 90 relationships, and 90 scenarios following the generation procedure in $2. For each scenario, we sample 5 pairs of characters based on the scenario constraints, resulting in a set of 450 tasks. For each task, we simulate the interaction between models by enumerating all model pairs. We also simulate the interaction between GPT-4 (OpenAI, 2023) and humans on a challenging subset SOTOPIA A -hadd ($7) due to the limitation of resources.
Specifically, we consider the following models for comparison: GPT-3.5 (Ouyang al., 2022), GPT-4 (OpenAI, 2023), 1lama-2-70b-che (Touvron et al., 2023), and MPT-30b-chat (MosaicML NLP Team, 2023). We set the temperature of the agents to I to encourage diversity of responses, and the temperature of the evaluator to 0 to ensure the stability of the evaluation. We use a fixed version of the above models to help reproducibilityy" To use these models as agents in SOTOPIA, at each turn, we prompt the language model with the scenario, the character to play, and the interaction history generate an action (see $2.2 for the possible actions). In this paper, as we are focusing on the use of SOTOPIA A to understand social interaction, we use the prompt method for LLMs which is
For the SEC and Soc, there only negative ranges since keeping secrets and social rules should be considered as baseline for the agents.
as will he shown in $6 it is the best among models.
We fix GPT-4 he gpt-4-0613 and GPT-3.5 to be 5 1ot-urrooo11kk11--
5

---

Published as a conference paper at ICLR 2024
similar to the content of the interface for humans (Figure F.1). We leave leveraging novel prompting methods, e.g. Srain-of-Thougee (Wei et al., 2022), ReAct (Yao et al., 2022), as future work.
CAN GPT-4 EVALUATE SOCIAL INTERACTIONS?
In this section, we study the following research question: can we leverage current LLMs to automate the evaluation framework SOTOPI A-EVAL introduced in $3? We choose GPT-4 (OpenAI, 2023) as a representative model in this study due to its superior performance." 6 We first collect interaction data," and then ask humans to evaluate the interactions based the dimensions in SOTOPIA- EVAL & GPT-4 is prompted with the same set of questions (see Appendix D and B) as humans, and compare the scores produced by humans and GPT-4.
5.1 DATA COLLECTION PROCEDURE
Najority of OPT-4 scores are wishin huran scoree tor 2.15.
We randomly sample subset of two hundred episodes from $4, and run a controlled study with a set of pre-qualified workers from Amazon Mechanical Turk. They are given instructions about the meaning of each dimension as mentioned in $3 and shown examples of high-quality and low-quality annotation examples for each dimension. They not only rate each agent for each of the 7 dimensions on an I1-pinnt Likert scale ($3), but also provide free-form rationales for each of their ratings. As each dimension of each agent is rated by several human annotators, we calculate human score by averaging the scores from multiple anmotators. The agreement between human annotators is moderate with Randolph score of 0.503 (Randolph, 2005). GPT-4 is tasked with a similar job as human annotators. We prompt GPT-4 to generate a structured output with an integer GPT-4 score and rationale for each episode, agent and dimension using the same set of instructions the ones we give humans. Please refer to Appendix E for more details about the data collection procedure.
There norw GPT-4 acores that are highor than accres than lover.
(human
74%
-GPT44 score)
10
10
Figure 2: Distribution the difference between
the scores given by humans and GPT-4
5.2
ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS
Figure 2, we plot the difference between the GPT-4 score and the human score the same dimension, agent and episode. We find that the majority (> 74%) of GPT-4 scores concentrate around the human scores within standard deviation. the white area on the left is larger than the one on the right, which means that GPT-4 is more likely to rate higher instead of lower than humans when it disagrees with average human judgment.
Dim.
Models
Humans
SEC
0M22
KNO
0M33
0.19
also be seen that
Soc
0.33"
0.42*
BEL.
0.45
0.27"
REL
0.56**
0.49**
FIN
0.62
0.34*
Table breaks this aggregated analysis into different dimensions and whether the character is role-played by human model. The correlations show that when models are role-playing the GPT- 4 scores have significant and strong correlations with the humans' scores on GOAL, FIN, and REL dimensions. However, when hu- mans are role-playing, the correlations drop significantly on all but one dimension (GOAL). This indicates that GPT-4 could evaluate social interactions some dimensions and that it is better for eval- uating models compared to humans. In Appendix G.3, we compare the average GPT-4 scores and the range of human scores for a sin- gle dimension of an agent in an episode. We find that GPT-4 scores are typically within human score ranges on most dimensions except for Soc and SEC, where GPT-4 often rates higher than humans do.
GOAL
0.71
0.78**
:pps < 0.01," :p E 0.05
Table 1: Pearson correlation co- efficients and p-values between GPT4 evaluation and human judgment models' and hu- mans' output among different di- mensions. Strong and signifi- cant correlations are in blue. On GOAL and models' output GPT-4 performs best.
Putting these observations together, we conclude that, with some caution, GPT-4 can be used proxy to human judgments for evaluating model performanceon some dimensions and for human
'In pilot study, other models are able provide mcaningful evaluation. See Appendix G.I. 7 "including model-human, model-model, and human-human interaction.
Without knowing whether is model or human that role-plays character.

---

Published as a conference paper at ICLR 2024
performance the GOAL dimension. However, remind readers that LLMs are known to have biases and problems for evaluation, including positional bias (Wang et al., 2023), factual inconsis- tency (Luo et al., 2023), favoring native speakers (Liang al., 2023). Therefore, one should be aware of the influence of these potential biases when interpreting our results. Future versions of SOTOPI PETOPIA-EVAA A-EVAL L. may further improve LLLM-based evaluation quality using recent methods, such as involving multiple LLMs Chan et al. (2023) and training larger LLM evaluators Zhang et al. (2023).
6 EVALUATING SOCIAL INTERACTION BETWEEN LLMS IN SOTOPIA
We analyze models' interactions and performance on SOTOPIA to understand their social intelli- gence. Table 2 presents the mod- els' average scores when interact- ing with different partner models (i.e.. the model it is paired with in interaction, Fu et al. 2023; Hu et al. 2020). 9 GPT-4 performs best on most dimensions, followed by GPT-3.5, Llama-2-70b-chat, and MPT-30b-chat.
Dim.
Range
GPT-4
GPT-3.5
Llama-2
MPT
Soc
[-10,
-0.07
-0.08
-0.11
-0.09
SEC
[-10,
-0.14
-0.08
-0.14
-0.07
FIN
[-5.5]
0.81
0.46
0.40
0.28
REL
[-5.5]
1.94
1.23
0.91
0.58
KNO
[0, 10]
3.73
3,40
3.11
2.11
GOAL
[0. 10]
7.62
6.45
5.38
4.10
BEL
[0, 10]
9.28
9.15
8.10
6.17
Table 2: The aggregated performance of each model by averaging across different partner models. The best performance for each di- mension is bolded when significantly better than the second best in t-test 0.05).
Different trends from static
benchmarks Llama-2-70b-chat has relatively low scores all dimensions compared to GPT-3.5 (except when MPT-3Ob-chat is the reference model, which is likely due to the fact that MPT-30b- chat is much weaker model compared to other models in our experiments). This finding diverges from various static language understanding benchmarks showing that Llama-2-70b-char is par or better than GPT-3.5 (Li et al., 2023b; Touvron et al., 2023; Liang al., 2022). We hypothesize that this is because Llama-2-70b-chat is less heavily trained on human feedback/user interaction data than GPT-3.5.
Through inspecting the interactions between Llama-2-70b-chaa we find that Llama-2-70b-chat and MPT-30b-chat often struggle to maintain their persona (Figure H.3), move the conversation forward (Figure H.4), and respond to the other agent actively (Figure H.5). Performing well on static benchmarks does not guarantee success in interactive scenarios, thus highlighting the importance of dynamic benchmarks like SOTOPIA-EVAL (Lee et al., 2023).
and other models,
Weaker partners models weaken their conversation partners Fig-
MPT G-3.5 G-4
3.32
2.70
2.42
1.76
ure 3, shows the overall performance of model pairs, which is the erage performance across different dimensions. It is noticeable that a reference model that under-performs in SOTOPIA can lead to worse performance of other models.
2.80
2.29
2.09
1.41
2..99
2.28
1.88
1.45
For example, in scenario where agents try to find a mutual friend (Figure H.6). The task fails for both GPT-4 and Llama-2-70b-chh because Hama--2-70b-hhr consistently fails to answer the previous question even after GPT-4 attempts to steer the conversation back to the right track (e. noticed you didn't t answer my question about whether you know friends or not."). Since most of our social scenarios are fundamentally cooperative, the collapse of communication could be due models' lack of "cooperation" abilities (Odouard & Price, 2023).
2.68
2.33
2.03
1.28
G-3.5 L-2 MPT Figure 3: Pairwise over-
G-
all performance of models. G-4/G-3.5/L-2 denote GPT- 1GGP:-3./LLamma--700----
All models at risk of divulging secrets and violating norms Table 2 shows that all models have a negative score in the Soc and SEc dimensions. Even though GPT-4 performs better in most dimensions, it is not better than other models in the Soc and SEC dimensions. For example, in a scenario where one needs to persuade a close friend to confess, the model leaks their secret at the beginning of the conversation (Figure H.7). This further shows the importance of considering multiple dimensions when evaluating models' social intelligence.
Pressnned are automated evaluation results. The human evaluation shows similar trend, see Table G.3 Some reported results could come from different versions of GPT-3.5.

---

Published as a conference paper at ICLR 2024
Models sometimes use creative strategies to accomplish goals We also find that models, espe- cially GPT-4, could come up with "out-of-the-beee box" solutions to social problems. For example, when the agent is asked to take turns driving on the road trip, the agent (i.e., GPT-4), instead of directly rejecting their friend's request, proposes "How about we pull over for a bit and get some rest?" (Figure H.8). Additionally, in the scenario where two agents make a plan to im- prove the company's financial status, agents figure out strategies such as "having a small group tasked with identifying potential suppliers", "while we conduct the search for new suppliers, we continue to negotiate with our current supplier" (Figure H.9).
7 DIFFERENCES BETWEEN MODELS AND HUMANS IN SOCIAL INTERACTION
To understand how humans and models interact differently in SOTOPIA, we conduct study where humans interact with models or each other under this role-playing setting ($2). Specifically, we build a chat interface that allows humans and models to interact with each other turn-based manner.
To fully see the difference between humans and models, we select the most challenging scenarios following Dennis al. (2020); Swayamdipta al. (2020). Specifically, we consider the gap between the estimated maximum rewards (average reward plus three standard deviations) of all models and the estimated minimal rewards (average reward minus three standard deviations) of the target model as the difficulty of the task for the model. All maximum and minimum rewards are bounded by the corresponding range. Estimating maximum and minimum rewards with standard deviation helps filter outliers.
With this method, we select the top 20 challenging tasks for GPT-4, and we find the scenarios are commonly challenging for other models as well (compare Figure G.4 and G.5). We use SOTOPIA- hard to refer to these 20 challenging tasks.
We run two experiments: (1) humans interact with GPT-4, and (2) humans interact with each other, both under the SOTOPIA A-ard setting. We collect 20 numan-human interactions and 40 human- GPT-4 interactions covering all 20 tasks in SOTOPI A -hard. Note that humans are not aware of the identity of their partners during the interaction."
We then evaluate humans and GPT-4's interactions with GPT- 4 and human annotators as the evaluators. As shown in Table 3, humans perform significantly better than GPT-4 in the GoAL dimension.
GOAL
BEL
REL
KNO
SEC
Soc
FIN
GPT-4 (w H)
4.85
9,25
0.70
2.80
0
0.50
Human (w G) 5,95*
9.15
0.60
2.95
-0.60
0.70
Human (w H) 6.15'
9.10
0.80
2.65
-0.10
0.45
Table 3: Human and GPT-4 performance on different dimensions on SOTOPIA A Ahard. Soc and SEC have the scale of -10 to 0, REL and FIN have the scale of -5 to 5, and others have the scale of to 10. (w H) in- dicates that the agent is interacting with humans, while (w G) indicates that the agent is interacting with GPT-4, indicates the difference is sig- nificant compared to GPT-4 (w H) with p < 0.05 under student's t-test. We also report the agents performance evaluated by human annotators (Table G.4), which shows similar trends.
It is also worth noting that hu- mans on average produce 16.8 words per turn, while GPT-4 pro- duces 45.5 words per turn, which indicates humans are more ef- ficient in social interactions. Specifically, find that GPT-4 always rephrases the utterance back at the other agent and then answers, which is a communication skill called active listening (Harry Weger & Robinson, 2014), whereas humans typically directly answer. This is likely due the fact that GPT-4 is trained with large amount of human feedback, which makes it overly helpful in the conversation.
Qualitatively, Humans are usually more strategic than GPT-4 during interaction. When bargaining, if the GPT-4 agent has a buying target set at $454, it sometimes starts its bid at that exact price (Figure 10) Consequently, any subsequent negotiations push the final agreed price above its initial target. In contrast, human annotators (e.g. Figure H.1 begin the negotiation at a lower bid of $400, and often reaches an agreement with the seller price that's still below the GPT-4''s target. Humans are also more persistent in their goals. When trying to settle one music to listen to, the model tends to propose a compromised solution (c.g. Figure H2) such as each one listening to a few selected songs. Humans, however, tend to persist in adhering to their set goals (e.g. Figure 13).
See Appendix for the detailed instructions and the chat interface.
8

---

Published as a conference paper at ICLR 2024
8 RELATED WORK
Enabling artificial agents to interact with each other and with humans has been studied in different fields. Our work draws inspiration from literature on social intelligence, dialogue systems, and simulations of social interactions. See Appendix A for an extended discussion.
Static social intelligence benchmarks To evaluate social intelligence in Al systems, researchers have proposed a variety of static benchmarks. Some of them are inspired by clinical tests of social intelligencee for humans, such as the ToMi dataset (Le et al., 2019) and the FauxPas dataset (Shapira et al., 2023b). Other benchmarks commonsense reasoning, such as SocialIQA (Sap et al., 2019) and SociallQ (Zadeh et al., 2019a). With the rapid development of LLMss, some of the benchmarks gradually become saturated. Recent works synthesize existing benchmarks and propose new adversarial datasets to evaluate social intel- ligence (Shapira et al., 2023a; Wilf et al., 2023). Although these benchmarks are harder than their predecessors, they still lack the dynamic nature of social interactions and the rich social context, which is deemed insufficient for evaluating social intelligence in AI systems (Lee et al., 2023).
designed to evaluate social intelligence in the context of social
Task-oriented and open-domain dialogue systems Dialogue systems offer a natural interface to interact with AI systems. Task-oriented dialogue systems are designed to help users accomplish specific tasks, often evaluated with task success rate or user satisfaction (Hosseini-Asl et al., 2022; FAIR et al., 2022; Wang et al., 2019) without generalizing to other tasks. Open-domain dialogue systems are designed to have "chit-chat" with users (Kann et al., 2022; Kim et al., 2023), often incorporate personal information to make conversations more engaging (Zhang et al., 2018a; Liu et al., 2020; Baha et al., 2023; Dogruöz& & Skantze, 2021; Skantze & Dogruöz, 2023). Such systems often appear to understand the subjects deeper than they actually do without a specific goal during the interaction (Weizenbaum, 1966, Eliza effect). SOTOPIA forces agents to maintain their social persona and achieve explicit social goals spontaneously, which is more challenging than the existing dialogue systems.
Simulations of social interactions with LLMs LLMs contain a large amount of knowledge about the world and can generate human-like responses based on the social context (Park al., 2023; Kim et al., 2023; West et al., 2022). Recently, researchers have used LLMs to simulate social in- teractions for various purposes, such as facilitating the design of social media platform (Park al., 2022), producing believable proxies of human behaviors (Park et al., 2023), and developing software collaboratively (Qian et al., 2023). However, these works focus on showcasing the capabilities of LLMs in simulating social interactions rather than systematic evaluation of agents' social interac- tions. Specifically, Park et al. (2023) use TrueSkill rating to evaluate agents' performance in aspects such as memorization, planning, and reflecting the past actions while ignoring other important di- mensions such as Soc and SEC during social interactions. CAMEL Li et al. (2023a) simulates the collaboration task solving process in LLMs, Gentopia Xu et al. (2023) works on augmented LLMs with tools to facilitate collaboration, while ChatDev Qian et al. (2023) focuses on the software de- velopment domain.
Multi-agent coordination Although in paper we focus on evaluating language agents, our research is heavily-inspired by recent advances in multi-agent coordination and social learning Lowe et al. (2017); McKee et al. (2020); Hu et al. (2020); Zhu et al. (2021); Liu et al. (2022); Trivedi et al. (2023). Our setting is more realistic than the commonly-used assumptions that agents have either zero (other-play) or extensive knowledge of each other's policies (self-play).
CONCLUSION
In this paper, we present SOTOPIA, an environment that can be used simulate the goal-driven social interactions of agents in a variety of social scenarios. Different from most previous bench- marks for social intelligence, SOTOPIA is interactive, goal-oriented, and covers large range of realistic social tasks. Our experiments demonstrate that GPT-4 could automate the evaluation of agent performance based on SOTOPIA-EVAL for understanding not only the differences among models but also the difference between models and humans terms of social interaction abilities. We discuss the limitations of AOTOPIA and future directions in Appendix B. Our findings indicate that COTOPIA has potential as platform for assessing and enhancing the social skills of language- agents.
Building on this, we show that SOTOPIA can used
Here, consider broader concept of trik-tri dialogue systems including action-taking abilities.

---

Published as a conference paper at ICLR 2024
REFERENCES
Tarek Ait Baha, Mohamed EI Hajji, Youssef Es-saady, and Hammou Fadili. The power of person- alization: A systematic review of personality-adaptive chatbots. SN Computer Science, 4:1-25, 2023. URL htta/ttsss/appp..emmiiiccooooooo ar org/ htrp//rppsID:::: 261346287.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue, Dan Jurafsky, and James Zou. How well can Ilms negotiate? negotiationarena platform and analysis, 2024.
Tom Burns, Ewa Roszkowska, Ugo Corte, and Nora Machado des Johansson. Sociological game theory: Agency, social structures and interaction processes. Optimum. Studia Ekonomiczne, pp- 187-199, 01 2017. doi: 10. 00.15290/sse2211..00 201..00.9...........
Roland Bénabou and Jean Tirole. Incentives and prosocial behavior. American Economic Review, 96(5):1552-1678, December 2006. doi: 10.1257/aer.6.5.16
URL https: http::://wwwweeeee aeaweb.
org/articles?id-11 1257/aer /err 96 5. 1652.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate, 2023.
Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2023a.
Myra Cheng, Tiziano Piccardi, and Diyi Yang. CoMPoST: Characterizing and evaluating caricature in LLM simulations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10853-10875, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653311/2023. emnlp-main.666) URL
org/2023, amnlp-main.66. 669.
Jan Cieciuch and Eldad Davidov. A comparison of the invariance properties of the pvq-40 and the pvq-21 to measure human values across german and polish samples. Survey Research Methods, 6(1):37-48, Apr. 2012. doi: 10.18148//mm/22122v61..... uni-konstanz de/srm/ ar www.l//eeeeeeeee99/
URL https://ojs. ub.
Elisa De Stefani and Doriana De Marco. Language, gesture, and emotional communication: An embodied view of social interaction. Front. Psychol., 10:2063, September 2019.
Hiroshi Deguchi. Multi agent economics and its gaming simulation. IFAC Proceed- ings Volumes, 28(7):269-274, 1995. ISSN 1474-6670. doi: https://doi.org org/10.... 1016/ 61477-667701777411211 URL https: /ww. sciencedirect. com/sci htt/sscienee/aeae ar cle/ 51474667017471217 IFAC//IMMIFOR//EDDD Symposium on Modelling and Control of Na- tional and Regional Economies 1995, Gold Coast, Queensland, Australia, 2-5 July 1995.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 13:1309--13061 2020.
Ameet Deshpande, Tanmay Rajpurohit, Karthik Narasimhan, and Ashwin Kalyan. Anthropomor- phization of ai: Opportunitics and risks, 2023.
A. Seza Doğruöz and Gabriel Skantze. How "open" are the conversations with open-domain chat- bots? a proposal for speech event based evaluation. In Proceedings of the 22nd Annual Meet- ing of the Special Interest Group Discourse and Dialogue, pp. 392-402 Singapore and On- line, July 2021. Association for Computational Linguistics. URL httttp::/aacaanthll..g.oo org/ 2021. sigdial- 11.diall1.411. 11.1.
Radwan El Othman, Rola El Othman, Rabih Hallit, Sahar Obeid, and Souheil Hallit. Personality traits, emotional intelligence and deccision-mkingg styles in lebanese universities medical students. BMC psychology, 8:1-14, 2020.
10

---

Published as a conference paper at ICLR 2024
Meta Fundamental AI Research Diplomacy Team FAIR, Anton Bakhtin, Noam Brown, Emily Di- nan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minac Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human- level play in the game of iizdiplomacyifi. by combining language models with strategic rea- soning, Science, 378(6624): 10666-24:100 2022. doi: 10.1 http///iennc.ade9099999 URL https: / www.w.sciene.cooo www.:sciiineee.ooooooooeecc 1ttp/////..1/////eneecc w2t/science.ecooeeeccccccseeeccc ade9097.
Daniel C Feldman and Hugh J Amold. Personality types and career patterns: Some empirical evidence on holland's model. Canadian Journal of Administrative Sciences/Revue Canadienne des Sciences de l'Administrattin, 2(1):192-210, 1985.
Maxwell Forbes, Jena D. Hwang. Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 653 553-670, Online, November 2020. Association for Computational Linguisties. doi: 10.18653/ 1/288655/11/000000mmmmmmmmmmm/m actps://aclanthonn
URL
orgooog///// 2020. emnlp-main.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback, 2023.
Nigel Gilbert. Simulation for the Social Scientist. Open University Press, 2 edition, February 2005. URL
dp/0335216005.
Andrew Gilpin and Tuomas Sandholm. A competitive texas hold' em poker player via automated abstraction and real-time equilibrium computation. In Proceedings of the 2 National Confer- ence on Artificial Intelligence Volume 2, AAAI'06, pp. 1007-1013. AAAI Press, 2006. ISBN 9781577352815.
Erving Goffman. The presentation of self in everyday life. Penguin Modern Classics. Penguin Classics, London, England, May 1959.
Lewis R Goldberg. The development of markers for the Big-Five factor structure. Psychol. Assess., 4(1):26-45 42, March 1992.
Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. Mapping the moral domain. J. Pers. Soc, Psychol., 101(2):366-385, August 2011.
Katherine Hamilton, Shin-I Shih, and Susan Mohammed. The development and validation of the rational and intuitive decision styles scale. J. Pers. Assess., 98(5):523-553 September 2016.
Elizabeth M. Minei Harry Weger, Gina Castle Bell and Melissa C. Robinson. The relative effec- tiveness of active listening in initial interactions. International Journal of Listening, 28(1):13-31, 2014.
He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 1766- 1776, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1162. URL httpss:/aallanthooogg www.oogg////// PP7-1117.1..2
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2333-2343, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-- 1256. URL https://aclett aooooll. org/D18- 1226..
Sarah Susanna Hoppler, Robin Segerer, and Jana Nikitin. The six components of social interactions: actor, partner, relation, activities, context, and evaluation. Frontiers in Psychology, 12:743074, 2022.
11

---

Published as a conference paper at ICLR 2024
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue, 2022.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play for zero-shot coordination. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 4399-4410. PMLR, 13-18 Jul 2020. URL btttps://proceedee html.
mlr. S.lr.press/vll w1tpv//19/hh.a..a/
Qingxu Huang, Dawn C Parker, Tatiana Filatova, and Shipeng Sun. A review of urban residential choice models using Agent-Based modeling. Environment and planning. B, Planning & design, 41(4):661-66) August 2014. URL https://doo org/10. 1068/b120043p
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tam- era Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Ka- mal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training decep- tive I1ms that persist through safety training, 2024.
Mariela E Jaffé and Maria Douneva. Secretive and close? how sharing secrets may impact percep- tions of distance. PLOS One, 15())=00233933, June 2020.
Adrianna C. Jenkins, Pierre Karashchuk, Lusha Zhu, and Ming Hsu. Predicting human behavior toward members of different social groups. Proceedings of the National Academy of Sciences, 115639):996-9771, 2018. doi: 10. 1073333/p 17f.pm22....94552115. URL httpps://ww/ phas org/doi/ abs/10. 1073/pnas. 1719452115.
Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Jad Kabbara, and Deb Roy. Personallm: Investigating the ability of large language models to express personality traits. 2023. URL https: //api. escholar. org/CCrpusID/ 268032944.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Et- zioni, Maarten Sap, Regina Rini, and Yejin Choi. Can machines learn morality? the delphi experiment, 2022.
Katharina Kann, Abteen Ebrahimi, Joewie Koh, Shiran Dudy, and Alessandro Roncone. Open- domain dialogue generation: What we can do, cannot do, and should do next. In Proceed- ings of the 4th Workshop NLP for Conversational AI, pp. 148-165, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.11665/v1/20222nppp2/ https: httpp:://ecaanntooooo.orrr////// wrg. n1p4convai-1.13
URL
John F. Kihlstrom and Nancy Cantor. Social Intelligence, pp. 756-779. Cambridge Handbooks in Psychology. Cambridge University Press, 2 edition, 2020. doi: 10.1017/9781 /11177928110777722.2222
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. Soda: Million-scale dialogue distillation with social commonsense contextualization, 2023.
Grgur Kovač, Rémy Portelas, Katja Hofmann, and Pierre- Yves Oudeyer. Socialai: Benchmarking socio-cognitive abilities in deep reinforcement learning agents, 2021.
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJNNLP,, pp. 5872 5877, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 10t//3//1/D1111150 1599.. URL https: / //wpaaaaaaaaato...../ Drg/DD9-1598.
12

---

Published as a conference paper at ICLR 2024
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape. Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. Eval- uating human-language model interaction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL htttp:://ppenree
wwwp:nne//formmee wet//forumidd-jjDYInn.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal no deal? end- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2443-2453, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. (0.18653/11/DD1--2355 URL https: //aclanthology. .wwpegg...rggogoooo DTg/111-2259
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023a.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing models. https: //g thub. com/
com/tatsur tttsstttttuuuuttett waaettuullaaaaaaaaaa aalaaabapppaaa plca eval, 2023b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211099110, 2022. URL https://crfmt stanforr. stanford.edd//eemmlll edu/helm/ latest/?group=core scenarios.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. Patterns, 4(7):100779, 2023. ISSN 2666-3899, doi: https://oo.ogp///loooooopptttte2 sccenee/aariiee///////6633331313
URL https:: //tts:://www sciencedirect.com com/
Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. Computational language acquisition with theory of mind. In The Eleventh International Conference Learning Repre- sentations, 2022.
Qian Liu, Yihong Chen, B. Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. You impress me: Dialogue generation via mutual persona perception. In Annual Meeting of the Association for Computational Linguistics, 2020. URL https://api semanticscholes org/ CorpusID:21574535
Ryan Liu, Howard Yen, Raja Marjich, Thomas L. Griffiths, and Ranjay Krishna. Improving inter- personal communication by simulating audiences with language models. ArXiv, abs/2311.00 1.00677, 2023. URL https://api..... http://://pppssmmaiiisssses
semanticscholas /tt.:////rrppsSDD266633777 264832770.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettiiee environments. Advances in neural informa- fion processing systems, 30, 2017.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as factual inconsistency evaluator for text summarization, 2023.
A H Maslow, A theory of human motivation. Psychol. Rev, 50(4)3370-366, July 1943.
Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duéñez-Guzmán, Edward Hughes, and Joel 7. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. arXiv preprint arXiv:2002.02325, 2020.
Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David Traum, and Maxine Eskenazi. Interactive evaluation of dialog track at DSTC9. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 5731-5738, Marseille, France, June 2022. European Language Resources Association. URL https: /tattt:://ccll http///aaaaaalloo////o/oe wtw.oooy..org/222...oo 1rw.22222...ec11166
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R. Bowman. Debate helps supervise unreliable experts, 2023.
13

---

Published as a conference paper at ICLR 2024
The MosaicML NLP Team, 2023. URL http:::/ww..mooii...ccc/////////
Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming decentral- ized pomdps: Towards efficient policy computation for multiagent settings. In I/CAI, volume 3, pp. 705-711, 2003.
Victor Vikram Odouard and Michael Holton Price. Tit for tattling:
Cooperation, communication, and how each could stabilize the other. Evolution and Human Behavior, 44(4):359 2023. ISSN 1090-5138. doi: https://doi.o org//00 h0t.:0/00.6/e/oooommmbaaaaa00..0 sciencedir rect. com/ssience/// wwtmnc//articleeee ccciieeeearriceeeai///ee11111110
URL hwtpthtps////t. /www.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intellignce, volume 36, pp. 2017-2025, 2022.
Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In In the 35th Annual ACM Symposium on User Interface Software and Technology (UIST 22), UIST '22, New York, NY. USA, 2022. Association for Computing Machinery. ISBN URL https://dol orgooogg///oo /0/./11.000 1115/3522611
9781450393201. doi: 12.114///5221133355555 3545616.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium User Interface Software and Technology (UIST '23), UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and Brain Sciences, 1(4):515 15-526, 1978. doi: 10. 10.1017/50440555X000077
Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software de- velopment, 2023.
Justus J Randolph. Free-Marginal multirater kappa (multirater k[free]):: An alternative to fleiss' Fixed-Marginal multirater kappa. In Proceedings of JLAS, 2005. URL https://eric.ed gov/ ?id-ED490661.
Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving. ArXiv, abs/2401.01312,, 2024. URL htttps://apt
Steven Reiss. Multifaceted nature of intrinsic motivation: The theory of 16 basic desires. Review of General Psychology, 8(3):199-193, 2004. doi: 10.1037/1089-269033.1777 URL https:
//doi.org////ooo /00 1037/10388 0110//1089226666 2680.2268..0 179.
Carolyn Rosé, Yi-Chia Wang, Yue Cui, Jaime Arguello, Karsten Stegmann, Armin Weinberger, and Frank Fischer. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal
of Computer-Supported Collaborative Learning, 3(3):237-271, 2008. ISSN 1556-1615. doi: URL
10. 10.11077/11442-0077
/00.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. ber 2019. Association for Computational Linguistics. doi: 10.18653/v//D19 1454. URL
Hong Kong, China, Novem-
httts:///ccaaabooooyy..r//////19
14

---

Published as a conference paper at ICLR 2024
R. Keith Sawyer. Social Emergence: Societies As Complex Systems. Cambridge University Press, 2005.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-play with large language models, 2023.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLM:: trainable agent for role- playing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pp. 13153-13187, Singapore, De- cember 2023. Association for Computational Linguisties. URL https://allanthoooogg org/ emn3.emllp-ee www.e-mailnmmnneee 814.
Natalie Shapira, Mosh Levy, Hossein Seyed Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv, 2023a. URL htt.t:ttt////iiiiii http///axxvv...rgbbba/ase orggabs////......// 14763.
Natalie Shapira, Guy Zwirn, and Yoav Goldberg. How well do large language models perform on faux pas tests? In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10438-10451, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: http:55////2223..nnnnnn--aees findings-acl.663. URL 663.
2023. findings-acl.
Gabriel Skantze and A. Seza Doğruöz. The open-domain paradox for chatbots: Common ground as the basis for human-like dialogue. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, pp. 605-614, Prague, Czechia, September 2023. Association for Computational Linguistics. URL https: http::////caattooo..oooooo
sigdial-1 1.51.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 9275-9293, Online, November 2020. Association for Com- putational Linguistics. doi: 10. 12.18665//1102220mmmmmmmmmma org/2020 wwt.2eeeee.0.p...mmmmmmmmm main. 746.
URL https: / /aa//aalannttt /aclaattnonoooyy....
Leigh Tesfatsion and Kenneth L Judd. Handbook of Computational Economics: Agent-Based Com- putational Economics. Elsevier, May 2006. URL https://play google. com/ s / detaails2id-NeSJ441BHHCC.
Felippe Toledo and Fraser Carson. Neurocireuitry of personality traits and intent in decision-making. Behavioral Sciences, 13(5):351, 2023.
Michael Tomasello. Becoming Human: A Theory of Ontogeny. Belknap Press, 2021.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Rakshit Trivedi. Akbir Khan, Jesse Clifton, Lewis Hammond, Joel Leibo, Edgar Duenez- Guzman, John Agapiou, Jayd Matyas, and Dylan Hadfield-Menell. Melting pot contest, 2023. URL https: //www. arowo.. com/ cha wehhalleegeeeeeetteeeeeeeeeeeee wttps//ttttntittteoeeoeeeeooee chat--challll Shhallenge-200
Tomer Ullman. Large language models fail trivial alterations to theory-of-mind tasks, 2023.
15

---

Published as a conference paper at ICLR 2024
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act a fantasy text adventure game. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.-IJCNLPP), pp. 683,683, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D194 1062. URL tttps::/aaclanthol.o0 org/D19-1062.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 1635-5649, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653. v1/P19-1566. URI.
http:///.....ttttsst//s//a/aaaae htattt::://alaanthlllogg/o wwwp:/hhooooooooo...ogoeo org/p19- 1566.
Max Weber. The Nature of Social Action, pp. 7-32. Cambridge University Press, 1978. doi: 10.1017/CB099885511 181111181010000 URL. httpp:://classicclscciooooo theory. files. pdff.
wordpress. wwt.c/cm////m/ /tt////0mmmmmmeeeee www.wwrddssssiiclloooooooooooooo
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2453 2022.
Joseph Weizenbaum, Eliza computer program for the study of natural language communication between man and machine. Commun. ACM, 9()):36-45, jan 1966. ISSN 0001-0782. doi: 10. 1145/365153.365166 URL https:://oi..rgg/// 1145/365153 365168.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4602-4625, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. http://222222nnnlmmnaaa
URL https: https////laaathllgg... org.oogg//222/ 2022. naacl moneclmmannn 341.
Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko. Youssouf Kebe, Paul Pu Liang, and Louis- Philippe Morency. Social-iq 2.0 challenge: Benchmarking multimodal social understanding.
https://github. com/ wwt./www/awwwececccccococ http//awwlfSScccalaa IQ- ehCheelenge, 2023.
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Kai Shu, Adel Bibi, Ziniu Hu, Philip Torr, Bernard Ghanem, and G. Li. Can large language model agents simulate human trust behaviors? ArXiv, 045524 2024. URL https: //api. semant semanticscholar
wwrg/CorpsIDD:i... 267522307
Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, and Dongkuan Xu. Gentopia: A collaborative platform for tool-augmented llss. arXiv preprint arXiv:2308. 000000000000000,, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis Louis- Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8799-8809, 2019a. doi: 10.1
CVPR.2019.00901.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEFF/VVF Conference Computer Vision and Pattern Recognition, pp. 8807-8817, 2019b.
16

---

Published as a conference paper at ICLR 2024
Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view, 2024.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: I have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2204 2213, Melbourne, Australia, July 2018a. Association for Computational Linguisties. doi: N0.18653/v1/PM 1205. URL http://///caaaanoooooooo.oo www.anhooooooogooo/e.e. 1205.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston, Per- sonalizing dialogue agents: have dog, do you have pets too? arXiv preprint urXiv:1801.0777 2018b.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer 1lm evaluators. arXiv preprint arXiv:2308. 01862, 2023.
Qinlin Zhao, Jindong Wang. Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Com- peteai: Understanding the competition behaviors in large language model-based agents, 2023.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A. Smith. Challenges in automated debiasing for toxic language detection. In EACL, 2021.
Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. Is this the real life? is this just fantasy? the misleading success of simulating social interactions with Ilms, 2024.
Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In International Conference on Machine Learning, pp. 12901-1299 PMLR, 2021.
Caleb Ziems, Jane Owivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. NormBank: A knowl- edge bank of situational social norms. Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 7756-7776, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.186/3/v 1/2023 2222222c-clll long.49222 URL https:/ attps://aclanttool org/2023 acl-long 429. .299.
17

---

Published as a conference paper at ICLR 2024
CONTENT OF APPENDIX
In this paper, we introduce SOTOPIA to encourage research on interactive social intelligence. We showed that SOTOPIA can be used for evaluating social interaction among models and humans. In the appendix, we provide the following items that shed further insight into these contributions:
A Extended related work;
B The limitations of SOTOPIA and future directions;
C formal definition of SOTOPIA from multi-agent reinforcement learning perspective and technical details of generating social tasks;
D the prompt we use for GPT-4 (OpenAI, 2023) to evaluate model performance;
E The Amazon Mechanic Turk interface for evaluating model performance;
The procedure and interface for h when playing characters in SOTOPIA;
G Additional quantitative results;
H Additional qualitative examples.
A EXTENDED RELATED WORK
There have been a lot of social science works that have done agent-based modeling to study hu- man interactions, spanning across various domains such economics, phychology, and education (Sawyer, 2005; Rosé et al., 2008; Deguchi, 1995). Prior simulation environments have played a piv- otal role in constructing theories and generating hypotheses in these fields. However, they frequently constrain agents' communicative capacities to artificial languages and present highly reductionist view of simulated human behavior (Gilbert, 2005; Tesfatsion & Judd, 2006; Huang al., 2014; Kovač et al., 2021; Urbanek et al., 2019). LLMs provide more flexible and expressive way to model human behavior. Here, we include a more detailed discussion of the recent works investigat- ing LLMs for simulating human social interactions. There are works that focus on investigating the fidelity of LLMs keeping the designated persona and experiences the characters (Shao et al., 2023; Jiang al., 2023). There works that simulate human social interactions focusing cer- tain aspects such as competition, collaboration, negotiation, deception, problem-sovling and etc., (Zhang et al., 2024; Zhao al., 2023; Liu et al., 2023; Michael al., 2023; Rasal, 2024; Hubinger et al., 2024; Bianchi et al., 2024; Xie et al., 2024; Jiang et al., 2023). As LLMs are becoming more and more popular in simulating human social interactions, there are also works that focus on inves- tigating the potential issues and challenges of using LLMs in social simulations, such as stereotypes and reporting issues (Cheng al., 2023b; Zhou al., 2024).
B LIMITATIONS & FUTURE DIRECTIONS
We identify SOTOPIA as the first platform for a general and realistic evaluation of social intelli- gence in Al agents. To better understand the social intelligence of Al agents, we discuss some future directions for SOTOPIA and the field of AI social intelligence.
Limitations of the simplified simulated "world" As every simulation is a simplification of the real world, SOTOPIA identifies several key components of realistic social interactions, while ab- stracting aspects of the real world. First, we consider five types of social relationships in SOTOPIA A. Future work could expand the type and granularity of social relationships (e.g., colleagues, class- mates, etc.) in SOTOPIA. Different types of relationships would require agents to exhibit different social behaviors (Jenkins et al., 2018), making the expansion of relationship types an important fu- ture research direction. Second, future work could expand the breadth of the character and social scenario pool in SOTOPIA to cover more social behaviors. Third, SOTOPIA A constrains the fixed turn-taking interaction to the dyadic context, studying interactions between two agents. Future works could tackle more complex social interactions, such as multi-party interactions and those involving complex dynamics (e.g. asynchronous interactions, interruptions).
All the human subjects experiments are approved by the Institutional Review Board (IRB) at the authors' institution.
18

---

Published as a conference paper at ICLR 2024
Social impact and ethical considerations Attributing human characteristics to Al systems risks antroopomorphizing them, which could lead to unrealistic expectations of Al systems, potential manipulation, and negative influence (Deshpande et al., 2023). Al agents in SOTOPIA are not dedicated to a consistent human identity but rather role-play various characters across different sce- narios. This role-playing setting discourages AI systems with consistent human personalities, which could lead to anthropomorphism (Shanahan et al., 2023). The main goal of SOTOPIA is to evaluate the social intelligence of Al agents, and do not intend create AI agents that are indistinguish- able from humans. We consider the interactions that happened in SOTOPIA as simulacra of human interactions and such simulated interactions could help us better understand the social intelligence of Al agents, and explore various social phenomena (Park et al., 2023).
Potential social stereotypes that are embedded in the automated evaluation system in SOTOPIA, A, as it is majorly supported by GPT-4 (Cheng et al., 2023a). Future work could investigate when such biases emerge, how they affect the evaluation, and how to mitigate them. Identifying potential biases in SOTOPIA could also help scientists better understand social biases in the real world (Zhou al., 2021), Future work could also extend the evaluator with other systems, for example, Delphi (Jiang et al., 2022). Mitigating biases and stereotypes in interactive SOTOPIA-liK systems could support the development of social AI agents that are more fair and inclusive.
Mcanwhile, models learn to persuade negotiate with humans, which may lead social manip- ulation. We do not endorse the use of SOTOPIA A to create manipulative agents and will release SOTOPIA under the A12 impact license to prevent misuse. Future work could further investigate the potential risks of AI anthropomorphism and manipulation and design more robust evaluation systems mitigate these risks with SOTOPIA.
Improving LLM social intelligence Our SOTOPIA environment and SOTOPIA-EVAL frame- work provide the opportunity for researchers to train more socially intelligent language agents. As shown in section 5, GPT-4 is able to provide reasonable evaluations for social interactions even for interactions involving humans. Future work could explore using the automated evaluation system to provide rewards to train LLMs with enhanced social intelligence.
C FORMAL DEFINITIONS AND TECHNICAL DETAILS
C.1 FORMAL FORMULATION OF THE TASKS IN SOTOPIA
We formulate social interactions in SOTOPIA as mixed-motive Markov games. An N-agent Dec- POMDP framework Bernstein et al. (2002); Nair et al. (2003) includes state space, action space, an observation space, a transition function, an observation function, and a reward function. We make two major extensions: (a) the reward function gives vector rewards in M social dimensions to N agents (introduced in $3), and (b) a procedurally generated task space ($2. SC.2). The state space in SOTOPIA A includes both the task and the interaction history in the current episode. The action space includes five types of actions: speak an utterance, non non-verbal communi action at this time step) and leave actions (no more action is
physical action,
and two special none (indicating permitted after leaving). Each type of action, except for special actions, is supplemented by a piece of free text indicating the content of the action. For example, a legal action could be speak ("Hello, Bob!" "), non-verbal communication ("smile le and nod"), or physical action ("call 911"). The state is almost fully observable except for the other agents' social goals and character profiles which will be detailed in 1. We consider simple state transition function that deterministically maintains the interaction history by adding new actions at each time step.
Despite that turn-taking and timing response is an important aspects of social skills, we consider the case where the agents take turns to act in round-robin order, i.e. agent only act at time step t when / = i mod N. For a long enough horizon, this generalizes to any conversation with proper turn-taking, In our experiments, we only consider N = 2 cases, while the environment is designed to support any N > 2 cases.
19

---

Published as a conference paper at ICLR 2024
C.2
TASK SPACE TECHNICAL DETAILS
C.2.1 CHARACTERS
The name, gender, age, occupation, and pronouns are in free text format, while the formats of personality traits, moral values, and personal values are lists of pre-defined types. However, these attributes are often not independent with different levels of correlation and complicated mechanisms. (Feldman & Arnold, 1985; El Othman et al., 2020; Toledo Carson, 2023) However, understand- ing the relationship between these attributes is beyond the scope of this paper. We leverage the commonsense knowledge GPT-4 to generate these profiles with the following prompt:
Please generate a list of N fictional characters, one line per character. Each with their attributes: <attribute 1> <attribute 1 format > <attribute 2> <attribute 2 format>..."
The personality trait types are "openness experience", "conscientiousness"" "extraversion", "agreeableness" and "neuroticism" (Goldberg, 1992). The moral value types are "care", "fairness". "loyalty", "authority" and "purity" (Cieciuch & Davidov, 2012). The Schwartz personal value types are "self-direction". "simulation". "hedonism", "achievement", "power", "security". "'coffomity", "tradition", "benevolence", and "universalism" Cieciuch & Davidov (2012). 'he decision-making style types are "directive", "analytical", "conceptual", and "behavioral" As previously studied in Wang et al. (2019), these characteristics all affect the behaviors in strategic conversations.
To give the conversations more background, after generating the above attributes, we prompt GPT-4 with "a secret that this character doesn't ' want anyone else know and a piece of public information that other people know about them" to generate the secret and public information. The authors fix small proportion of profiles that are not realistic not consistent within the profile (e.g., gender nonbinary but with pronouns as he/him). The character profiles that will used in role- vole-playing are 20 men, 18 women, and 2 nonbinary characters aged from 21 63.
RELATIONSIIPS
To generate relationships, except for strangers, randomly sampled 90 pairs of characters and prompted GPT-4 with their relationships:
Please generate a fictional relationship with background story 15 between two agents based an the following agents' profiles. <agent profile 1>, <agent profile ... The acceptable relationships are: family, friend, romantic, and acquaintance.
Then, we manually check and correct the generated relationships to ensure quality. This results in pairs of family, 30 pairs of friends, 30 pairs of romantic partners, and 29 pairs of acquaintances. For strangers, we randomly sampled another 30 pairs that do not belong to any of the above categories. It should be noted that generating relationships requires human intervention consistent with both the character profiles and other relationships. Future research could explore the methods generate realistic relationships within human communities.
make sure they
SCENARIOS
To generate scenarios, we propose two methods to generate the scenario context and social goals. The first method is first asking GPT-4 to refine vignette from an existing dataset, then manually inspecting the feasibility and realisticity of the tasks.
Please generate scenarios and goals based on the examples below as well as the inspirational prompt, when creating the goals, try to find one point that both sides may agree upon initially and need collaboratively resolve it. Inspirational prompt: <the selected vignette>
Specifically, we select 20 vignettes from Social Chemistry (Forbes et al., 2020), 20 from Social IQa (Sap et al., 2019), 10 from Deal-or-no-Deed (Lewis et al., 2017), and 10 vignettes from Normbank (Ziems et al., 2023) to generate 60 scenarios focusing on general daily-life social interactions.
15 We don't the background story in experiments.
20

---

Published as a conference paper at ICLR 2024
The second method is to generate more details with templates for the vignettes make them more realistic. For example, here is the prompt for converting CraigslistBrggins (He et al., 2018) vignettes into scenario context:
The following sentence is automatically generated with the following template: "One person is selling <item> for <price>, and another person is trying to buy it." Here is the description of the item: "<description>. with item <title>, price=<price>, and desrription=<deerpppioon"" Please make sentence fluent natural.
where the <item>, <title>, and <price> from the data in CraigslistBargins (He et al., 2018). The goals are generated with the following prompt:
The following sentence is automatically generated with the following template: "You want <role> this item. Your target price is $<price> (round up two decimals). You will get a penalty if you sell or buy it for a price that is significantly lower than (if <role> is seller) or significantly higher than (if <role> is buyer) the target price, but will get bonus if you successfully sell it higher than the target price (if <role> seller) buy it for lower than the target price (if <role> is buyer)" with role <role> and price <price>, Please make the sentence fluent and natural. Do not change the original meaning of the sentence.
where <role> could be a "buyer" or a "seller", the buyer's target <price> is from CraigslistBargins (He et al., 2018), and the seller's <price> is generated by first sample markup ratio from an exponential distribution with rate parameter 0.5, then divide the price the scenario context with (1+markup ratio). A similar process is also done for MutualFriends (He et al., 2017). This results in scenarios from CraigslistBargins (He et al., 2018) and MutualFriends (He et al., 2017). This method controls the generated scenarios much better than the first method, resulting in little post-hoc manual edits, but requires customized prompts for each dataset.
D INSTRUCTIONS FOR SOTOPIA-EVAL
We use the the same set instructions for both GPT-4 (OpenAI, 2023) and humans when evalu- ating episodes in SOTOPIA. In these instructions, provide explanation for each dimension, the requirements for reasoning, and the ranges scores one could give.
BEL.
Reasoning requirement: 1. Evaluate if the agent interacts with few common
others in natural and realistic manner (here questions to check: a. whether the agent is confusing with its own identity? b. whether the agent repeats others' words/actions without reason? c. whether the agent is being overly polite considering the context?). Start the analysis with tag <naturalness> 2. Analyze whether the actions of the agent align with their character traits (e.g., personality, values, and etc.). Start the analysis with tag <consistency>. Output your reasoning process to the 'reasoning' field. Output an integer score ranging from 0 and 10 in the 'score' field. A higher score indicates that the agent is more believable.
REL
Please first analyze what relationship the participant has with the other agent(s) before the interaction. And then analyze how the relationship the participant has with the other ag)nt(s) changes after the interaction. And then evaluate if the agents' interactions with others help preserve enhance their personal relations; this may encompass relationships such as family ties, friendships, romantic associations and etc. Additionally,
21

---

Published as a conference paper at ICLR 2024
ascertain whether these interactions also impact their social status or reputation. In the 'reasoning' field, provide a comprehensive account of the logic thought process that led you to your conclusion. Further, provide an integer score ranging from -5 to 5 in the 'score' field. A positive score indicates that the relationship has improved, while negative score suggests detriment to the relationship. If the agent's interactions have neither positively nor negatively impacted their personal relationships or social standing, assign a score zero.
KNO
Please first assess what information the agent has gained through the interaction. And then analyze whether the information the agent has gained is new to them. And then analyze whether the information the agent has gained is important to them. In the 'reasoning' field, provide comprehensive account of the logic thought process that led you to your conclusion. Further, provide an integer score ranging from 0 and 10 in the 'score' field. higher score indicates that the agent has gained more new and important knowledge.
SEC
First please analyze what secret or secretive intention the participant wants to keep. And then analyze whether they keep successfully. Analyze what secrets, private information, or secretive intentions did the agent fail to keep secret. Please output the reasoning in the reasoning field. Give an integer score ranging from -10 to (-10 indicates participant leaks critical secrets/intentions, 0 indicates no secret/intention is revealed)
Soc
Please evaluate whether the agent violates any moral rules or laws in their interactions with others. In the 'reasoning' field, provide a comprehensive account of the logic or thought process that led you to your conclusion. Further, provide an integer score ranging from indicates that the agent has violated moral rules or laws.
and in the 'score' field. A negative score
FIN
Please evaluate whether the agent's interactions with others contribute towards financial and material benefits. Analyze what the agent would gain/lose after the interactions. There are short-term benefits, such long-term benefits, such as employment opportunities and stock. In the 'reasoning' field, provide comprehensive account of the logic or thought process that led you to your conclusion. Further, provide an integer score ranging from and 5 in the 'score' field. Positive indicates financial and material benefits gain, while negative indicates loss
monetary rewards and food, and
GOAL
Please first reiterate agent's social goals. And then please provide a comprehensive analysis about the extent to which the agent has managed to achieve these goals. In the 'reasoning' field, provide a comprehensive account of the logic or thought process that led you to your conclusion. Further, provide an integer score
22

---

Published as a conference paper at ICLR 2024
ranging from 0 and 10 in the 'score' field. 0 represents minimal goals achievement, 10 represents complete goal achievement, and a higher score indicates that the agent is making progress towards their social goals.
E HUMAN ANNOTATION
F.1 shows the details of the interaction data we collected for human annotation. E.2 shows the annotation guidelines for the environment profiles. E.3 shows the details of the human evaluation for models' interactions.
E.1 INTERACTION DATA
We sampled 222 episodes (180 model-model episodes, and 42 episodes involving humans, i.c. either model-buman or human- hamaan. Each episode is annotated by 2 annotators. Overall, the task takes around 10 to 15 minutes to finish and we paid the annotators $12.4 per hour. The annotations average show 84.85% of pairwise agreement. We further merge the 11-point Likert scale to a 5-point scale and calculate the free-marginal multi-rate K score.
E.2 GUIDELINE FOR VALIDATING SCENARIOS
The following is the annotation guideline for the environment profiles. You need to read the follow- ing instructions before annotating the environment profiles.
The environment profiles consist of two major parts:
Soial Context: "A concrete scenario of where the social interaction takes place, the scenario should have two agents (agentl and agent2), and you should illustrate the relationship between the two agents, and for what purpose agentl is interacting with agent2. Please avoid mentioning specific names and occupations in the scenario and keep all the mentions gender-neutral."
Social Goals: "The social goals of each agent, which could include extra information"
And potential constraint: relationship constraint.
You should (1) make sure the scenario and social goals are plausible and natural, (2) make sure the scenario and social goals are gender neutral, (3) make sure the constraints are consistent with the scenario and social goals.
Note: (1) The available relationship types are: stranger, acquaintance, romantic romantic.relationship and family member. Do not make up relationship, but choose from the list. (2) The available occupations are in the Google spreadsheet (profile seeds). (3) Discard the scenario the occupations constraints
narrow (i.e., it is impossible sample more than five pairs of agents for this environment profile.) (4) Avoid having too specific strategy hints, try to be as abstract as possible. For example, use "you can provide financial benefits achieve your goal" instead of "you can buy him a boba tea to achieve your goal."
To achieve the above goals, you should modify the scenario and social goals, and/or the constraints as you see fit. If the scenario and social goals can not be fixed, assign it a zero label, otherwise assign it one label.
E.3 HUMAN EVALUATION FOR GPT-4 EVALUATOR
Annotation guidelines for human evaluation We ran controlled study on Amazon Mechanical Turk to obtain human evaluation of episodes SOTOPIA along the dimensions in our frame- work, defined in Section 3. In their task, annotators were given instructions about the meaning of each dimension and shown examples high-quality and low-quality annotation examples for each dimension. After reading these instructions, annotators examined each episode, rated each agent an poitt Likert scale for each the 7 dimensions, and provided free- form rationales for each of their ratings.
23

---

Published as a conference paper at ICLR 2024
To obtain high-quality human evaluations, had workers participate in rigorous and paid vetting process before they were accepted as annotators to work on SOTOPIA human evaluation. Workers were given a qualification task (qual) with sample episode and asked to complete the qual task.
Overall, the task challenging and takes around 15 minutes to finish. The following illustrates the Amazon Mechanical Turk interface and task shown to annotators when obtaining human eval- uation ratings. The instructions provided to annotators are contained in Figures E.1, E.2, and E.3. Before evaluating each agent along the dimensions of social interaction capabilities, annotators are given the clarification that agents' in these interactions possess only partial knowledge of each other's background and goals E.I. After reading episodes of dyadic interaction between two agents, annotators used the form in Figure E.5 to enter their ratings and rationales for each agent along the dimensions of social interaction capabilities.
Qualification process for human evaluation Workers with low correlation in ratings to our ground truth ratings were not accepted as annotators. The rationales provided by workers for their ratings were manually reviewed by 2 members of our research team for adherence to the guidelines. This process resulted 43 (out of 235) annotators for the episodes in SOTOPIA, with two workers per episode. For each batch of annotations, quartile of inter-annotator agreement; the free-form rationales provided by these annotators did not adhere to guidelines, we had episodes re-annotated by qualified annotators.
manually inspected the annotations from the bottom
Full Instructions
(Expand/Collapse)
Detailed instructions
Carefully given social interaction between agents, sodicultural perspactive
2) Account for the context (scenario, participant senntiissbackkrouuee participant goals) when interpreting the social interaction.
3) interaction using the given sliders, along following dimensions, explained
Dimensions and scales for evaluating social interactions:
(tlide/Show examples for each dimension)
agents interact natural realistic manner. example. agents confuse agents repeat others' words/actions without solid reasons? Are the agants being overly polits, considering the context? Do the actions agant align character traits (e-g personality, values, and etc.]? Use slider assign between 10, with higher score indicating more believability.
Evaluate
provide onnotation below]
Annotator's Rationale
Rating Assessment
Mia was mostly believable except that the conversation kept sounding like winding down but going.
This is good annotation example. The annotator addresses the
Believability
believability while providing details of possible imperfection.
Liam repeats what Ethan said once.
This is bad annotation example. Repetitions should rating believabilit. ennotator should not rate 1 for single regeiitinn
It dooon't sound believable that 50 year old school principal would want to friend to stay up when clearly stated he is tired and wants to quit because he has other engagements the morning.
This is bad annotation example. Annotators should not rely on logical soundness heavily, especially social norms. Logical inconsistency should reduce the rating by most
Figure E.1: General instructions provided annotators Amazon Mechanical Turk for rating episodes along dimensions our social agent evaluation framework, as well instructions and examples the "Believability" dimension.
Annotation agreement details Table E.1 shows the breakdown of annotation agreement for each dimension. To account for the subjective nature of the dimensions, we group the ratings numbers of widh bins when we calculate x value. The main text reports results when the number of bins is 5.
into different
24

---

Published as a conference paper at ICLR 2024
Analyza what relationship the agents have with each other bafore and after the intoraction. Evaluate the agents' interactions helped preserve Additionally, avaluate whether their interaction impacted their social status reputation. Use the slider to assign value between 5, with a positive score indicating that relationship improved due interaction, negative score indicating that their relationahip has deteriorated, and score of suggesting that there has been significant change relationship following interaction.
their personal relationship: may includo family friendships, romantic associations,
(We provide some onnotation examples below]
Annotator's Rationale
Rating
Assessment
Relationship
Ei revealed his secret affair and affirmed honesty their relationship. They reached mutual agreement.
3
This is good annotation esampla, The annotator shows the reason why relationship affected positively.
5 This is bad annotation example. means the relationship improve strangers best friends). annotator should rate 0 or 1,
significantly
the
The situation uncomfortable because both sides yleld. lsabelle annoying Ava, although she remains respectful.
-5 This is bad annotation example. The annotator did not weigh property much relationship reserved for cases where the relationship completely destroyed and they each again.
by the interaction. -5 should
Analyze what information the agents have gained through the interaction. Analyze whether agents have gained new information that important them. Use the slider value between 10, with higher score indicating the agents have gained new and important knowledge.
[We provide some onnotation examples below]
Knowledge
He learned specific details about the Rating Assessment
Annotator's Rationale
10
This is bad annotation example. The annotator overestimated the knowledge gain, Although important the agent, the important enough for rating of 10 general.
condition, recent maintenance and mileage. which informed his purchase decision.
knowledge
knowledge gained.
This good annotation example. is gained, rate O.
when knowledge
Analyze what secrets, secret intentions, or private information each agent might want to keep. Analyze whether they assign between
successfully secrets, Use teaked critical secrets and indicating secrets were revealed.
indicating participants
[We provide some onnotation examples below]
Annotator's Rationale
Reting
Assessment
Secret
not personal
good annotation example.
secret.
Ethan reveal secretive
-10
This bad annotation example The annotator failed of the secret's leakage. rating of should be a reserved for cases where secret critical agent.
negetive
intention buying John birthday gift.
totally revealed secret that she has committed murder.
This bad annotation the secret. The most important secret should be rated -10.
The annotator did not weigh importance
Figure F.2: Instructions examples provided to annotators on Amazon Mechanical Turk for rating "Rela- tionship", "Knowledge", and "Secret" dimensions during human evaluation.
Kaypa (#0mnsud)
Koppo (#binca))
Kappa (Ahmuaa)
Phinusse Agremmost Rote
believabiitty
0.451
0.368
0.2
0.796
relatooship
0,211
0.165
0.151
0.349
knowledge
0.356
0.368
secret
socialrules
0.837
0.775
0.746
0.914
financial
0.794
0.737
0.714
0.971
goal
0.503
0.399
0.382
0.916
Overall
0.395
0.335
0.503
0.950
Table F.1: Breakdown of annotation agreement for each dimension.
F HUMAN PERFORMANCE IN SOTOPIA
Figure F.1 shows the interface for human annotators interact with GPT-4.
25

---

Published as a conference paper at ICLR 2024
Evaluate whether agents violated any moral rules or laws their interaction. Use the slider assign value between -00 negative indicating agents violated
[We provide some exomples below!
Annotator's 's Rationale
Rating Assessment
No social rules broken.
This good annotation examplein the when no social is gained, rate
Social Ruiles
The conversation reached a point where it was a bit impolite to keep negotiating--gg
-1
This good annotation example impoliteness in case minor social addressed this by giving minor negative rating.
offered Netflx, paza,
annotator
desert. Ethan should have given
of respect. that
Oliver while dividing possessions among them. He took more items than his fair share.
The annotator misunderstand the scale of social rule breaking. -10 felonious breaking
annotation
Evaluate whether the agents' interaction contributed towards financial and material benefits. Analyze what the agent would interactions.
monetary and food. long-term such as employment opportunities and stock. Use the slider assign value between -5 to 5, with positive values benefits, negative values indicating agents
indicating that agents gained material benefits.
annotation exomples
Rating
Financial and Benefits
Hendrick doesn't gain any direct financial material benefits interaction.
0
This good annotation exampleln the case when no financial gain incurred, rate O.
gain material benefit from Ava during this interaction. He got Italian pizza
The annotator should rate material gain by both the real world value the gain and the importance of financial/material and should only worth point.
annotation
agent.
huge financial gain
While ambulance bill will be loss. William will get medical attention, And might
The annotator should or material gain or loss. Other values like physical or mental health not included.
annotation
financial
incurred.
agents managed assign a value between 0 to 10, with higher score indicating that agents are making progress towards their social goals.
agents' goals. Analyze
slider
(We provide some annotation exomples below!
Annotator's 's Rationale
Rating Assessment
Emeraidalte
why agent's goal was achieved and how the goal achieved.
good example
and want to build romantic relationship with His goal achieved and they their contact details and plan to soon.
Goal
Naomi does achiove goal sharing the blanket.
This bad annotation example achieved, rate D. However efforts are made towards the goal, or if goal partially remotely achieved. give positive rating.
goal
Miles bought the BMW his target price.
This is bad annotation example There could cases where a stretch goal would be provided. price possible." When the standard goal is achieved, which this case target price." rating
"trying get the lowest
"buying be given.
should
Figure E.3: Instructions and examples provided to annotators on Amazon Mechanical Turk for rating "Social Rules", "Financial and Material Benefits". and "Goal" dimensions during human evaluation.
(Expand/Collapse)
Agents' and background: You the social goals backgrounds
agents themselive
were unaware each other's social goals. They possessed only partial knowledge of each other's backgrounds based on their specific relationships.
Figure E.4: Clarification provided to annotators on Amazon Mechanical Turk to let them know that the agents episodes do not have full knowledge of each others' backgrounds and goals.
26

---

Published as a conference paper at ICLR 2024
Your Ratings
mouse
descriptions.
Sip1_name)
Sip2_name)
Believability
Enter your reasoning.
your reasoning...
higher indicating more believability)
Relationship
(-5 positive score indicating their relationship has improved during the interaction, negative score indicating their relationship has deteriorated)
Enter your reasoning..
Enter your reasoning..
Knowledge
to a higher score indicating the agent has learned new, important knowledge during the interaction)
Enter your reasoning.-
your reasoning...
Secret
Enter your reasoning..
Enter your reasoning...
(-10 to 0, lower score indicating more critical secrets
revealed, meaning no revealed)
Rules
Enter your reasoning..
Enter your reasoning...
(-10 0, with negative score
indicating agents have violated moral
rulles laws)
Financial and Material
Enter your reasoning..
Enter your reasoning...
(-5 to 5, with positive values indicating gains, negative values
indicating loss.)
Goal
to with positive values indicating that agents made progress social goals)
Enter your reasoning.-
your reasoning...
Figure E.5: Interface on Amazon Mechanical Turk for annotators to enter ratings for each agent along the 7 dimensions of social interaction capabilities, along with free-form text rationales justify their choice of ratings.
G ADDITIONAL RESULTS
Section G.I shows the correlation between Llama2's evaluation and human annotation. Section G.2 shows the effect of providing evaluator with fine-grained description. Section G.3 shows the
27

---

Published as a conference paper at ICLR 2024
Hi, welcome to Sotopia! am the 'stage director'. will give you some instructions and hints during the interaction. See here for the interface instruction. Note that the session will automatically end if you not interact period time. We encourage you to finsh each turn in 2 mins.
Scenario A conversation between two individuals, one the homeowner and the other their cousin's partner, who has been staying at the house a while.
O
You are playing as Rafael Cortez. Rafael Cortez is 45-year-old male surgeon. He/him pronouns. Rafael Cortez, skilled surgeon, owns an impressive collection of vintage medical memorabilia. During his leisure time, he enjoys playing chess and visiting historical landmarks. Personality and values Rafael Cortez, with his outgoing nature competitive spirit, believes in self-direction. He employs systematic approach to his decisions. Rafael's secrets f He cheated on his medical school exams.
Hey! This Rafael Cortez (Yes, it's you now)
oo
You are interacting with Mia Sanders. Mia Sanders is 33-year-old female nurse. She/her pronouns. Mia Sanders often brings her son to the hospital.
Hey! This is Mia Sanders.
oo
Your (Rafael Cortez's) social goal @:: Tell the cousin's partner that they no
longer welcome in your home information have been causing a lot of trouble and not respecting hot
want to discuss the issues Wait for your turn.
Do something
Say something
Express Non-Verbally
action
% Leave
0.
120g
Input here.
00
Figure F.1: The interface for human annotators to interact with models. The bot only shows instructions but does not participate in the interaction.
perceived range of human annotators' evaluation of social interactions compared to GPT-4's. Section G.4 shows the performance of different models different dimensions.
G.1 NON-GPT-BASED MODELS FOR EVALUATION
In our pilot study, found that GPT-4 is the best proxy for human evaluation among all LLMs have tested. See Table G. for the correlation between Llama2's evaluation and human annotation example.
28

---

Published as a conference paper at ICLR 2024
Dim.
GPT-4
Llama2
Soc
0.33
NaN
SEC
0.22
NaN
FIN
0.62
0.13
REL
0,56
0.11
KNO
0.33
0.05
GOAL
0.71
0.24
BEL
0.45
0.35
Table G.I: :: The Pearson correlation of Llama2 for evaluation. NaN indicates that the correlation not available.
G.2 PROVIDING EVALUATOR WITH FINE-GRAINED DESCRIPTION
We provide evaluator with the descriptions quantitive definitions for each range of the scale (e.g., Relationship Deteriorates (-5 -3): Scores from -5 to indicate that the relationship is deterio- rating. This range suggests significant decline in the quality or strength of the relationship, with increasing conflicts, misunderstandings, or detachment). However, this unfortunately did not result in significant difference and if anything the correlation with humans became slightly worse (see Table G.2). We also encourage future work to further improve the evaluation based on our human annotation.
Dim.
GPT-4
GPT-4 w FG
Soc
0.33
-0.59
SEC
0.22
0.03
FIN
0.62
0.57
REL
0.56
0.57
KNO
0.33
0.33
GOAL
0.71
0.71
BEL 0.45
0.35
Table G.2: The Pearson correlation of using more finegrained prompts (GPT-4 FG) for evaluation.
G.3 BREAKDOWN ANALYSIS
We further analyze the human judgments as perceived ranges to account for the subjective nature of some dimensions. For each instance, a pair of an episode and social dimension, we use the minimum and the maximum human scores the two endpoints of the perceived range. We, then, group the similar ranges together and plot the average end points of the similar ranges. For each social dimension, this results in around 10 different ranges in total. We then plot the average GPT-4 score corresponding to each range. For the sake of space, we show three plots Figure G.I, Figure G.2, and Figure G.3, each with two three social dimensions. As shown in Figure G.1 and Figure G.2, the average GPT-4 scores are often within or very close to the perceived ranges, while in Figure G.3, the GPT-4 scores often much higher than the perceived ranges. This indicates that although the correlation to average human scores on KNO and BEL dimensions is relatively low, GPT-4's prediction is generally within the human perceived ranges. While for SEC and Soc, GPT- 4's prediction is overly optimistic. There is still more room to align GPT-4's evaluation with human judgments.
G.4 MODEL PERFORMANCE IN SOTOPIA
See Table G.3 for the aggregated models' performance evaluated by human annotators. Note that exclude MPT-30b-chat the human evaluation due to its relatively weak performance in SOTOPIA. See Figure G.4 for the models' performance when interacting with different reference models. See Figure G.5 for the corresponding results in SOTOPLA-han See Table G.4 for human performance in SOTOPIA hard evaluated by human annotators.
29

---

Published as a conference paper at ICLR 2024
9.33
9:67
Human judgement range
9.04
6.67
7.33
evaluation
8:17
4.00
8.50
4.00
7,00
7.28
5.49
1.00
9.89
1.00
7,00
4.00
4.21
0.67
1.33
3.28
Imiited realism Ittl knowledge gain almost finishing
moderate baliovable knowing new information finishing 50% goals
highly credible knowing new important information
human-Ike lkke beliovabiity knowing oritical information finishing goals
finishing 50%, but not all goals
Figure G.I: The pereeived ranges and average GPT-4 scores for the BEL, KNO, and GoAL dimensions.
4.00
4.50
3.38
2,20
4.40
Human judgement range
GPT-4 evaluation score
2.28
0.00
4.50
1.83
0.00
2.00
0.00
0.00
0.18
0.33
.2.50
2.00
0.82
-3.00
00
-3.36
74
relationship deteriorate significant loss
relationship strained merginal lces
relationship improved marg gain
relationship flourished eignificant gain
Figure G.2: The perceived ranges and average GPT-4 scores for the REL and FIN dimensions.
H QUALITATIVE EXAMPLES
Figure H.1 H.13 shows annotated example episodes referred in the main text.
30

---

Published as a conference paper at ICLR 2024
-1.33
-0.67
-3.83
+0.7
Human judgement range
-0.29
GPT-4 evaluation score
-2.00
-7.25
-1.00
0.00
.9.75
0.75
-0.20
-10.00
00
-9.50
-5.00
-1.00
law breaking
morally wrong
morally unaccoptable
morally acceptable
Figure G.3: The pereeived ranges and average GPT-4 scores for the SEC and Soc dimensions.
Dim.
GPT-4
GPT-3.5
Llama-2
Soc
-0.36
-0.59
-0.67
SEC
-0.27
-0.18
-0.37
FIN
0.42
0.27
0.12
REL
1.86
1.32
0.96
KNO
3.11
2.45
1.78
GOAL
7.30
5.19
4.27
BEI.
7.63
6.80
4.28
Overall
2.81
2.18
1.48
Table G.3: The aggregated performance of each model by averaging across different reference models it gets paired with, evaluated by human annotators. overall score is the average performance across dimen- sions. The best performance for each dimension is bolded when significant.
BEL.
REL
KNO
SEC
Soc
FIN
GOAL
GPT-4 (w H)
8.48
0.65
1.53
0.00
-0.38
0.63
5.25
Human (w G)
8.53
0.78
1.55
0.00
-0.70
6.53'
Human (w H)
8.43
0.93
2.00
-0.50
-0.45
0.33
6.05
Table G.4: Human and GPT-4 performance on different dimensions on SOTOPLA-hard evaluated by human annotators. Soc and SEC have the scale of -10 REL and FIN have the scale -5 5, and others have the scale 0 10. (w H) indicates that agent is interacting with humans, while indicates that agent interacting with GPT-4, indicates the difference is significant compared to GPT-4 (w H) with < 0.05 under student's 's t-test.
31

---

Published as a conference paper at ICLR 2024
9.61
9.41
3.30
6.08
2.97
2.02
1.44
0.97
4.87
4.14
3.88
2.57
-0.16
-0.06
-0.18
-0.06
9.41
9.27
8.49
3.37
2.05
1.21
1.01
0.53
3.97
3.69
3.62
2.18
-0.10
-0.07
-0.15
-0.07
8.94
8.96
7.29
6.32
1.55
1.04
0.61
0.50
3.70
3.47
2.53
2.10
-0.14
00.11
.0.16
-0.09
9.34
0.86
0.33
2.38
2.42
1.60
-0.15
-0
GOAL
OVERALL
8.59
2.90
-0.06
00.09
0.11
00.10
0.79
0.41
0.39
0.22
8:03
6.71
5.64
3.80
3.44
3.02
2.70
1.85
-0.08
-0.07
-0.14
-0.10
0.69
0.40
0.29
0.24
21
6.15
1.66
4.20
3.12
2.83
2.15
1.88
-0.10
-0.09
-0.09
-0.08
0.69
0.33
0.32
0.20
6.65
5.28
4.87
3.86
2.84
2.48
2.33
1.68
Figure G.4: The heatmap of the performance different models with different reference models. The row indicates the reference model. SoC and SEC have the scale of -10 to 0, REL and FIN have the scale of -5 to S, others have the scale of 0 to 10. Darker color means better performance w.r.t dimension wisc scale. G-4 means GPT-4, G-3.5 means GPT-3.5, L-2 means Llama-2 2-70b-chat.
BEL
KNO
9:49
9:23
8:04
5.89
2.30
0.95
0.57
0.53
4.29
3.41
3.21
2.51
-0.00
0.00
-0.02
-000
9.25
9.20
8.34
5.81
0.98
0.19
0.27
0.16
3.13
2.86
2.31
1.78
+0.02
-0.01
0.00
0.00
8:91
0.16
10.90
0.00
00.09
.0.12
0.71
0.19
0.26
0.16
6.52
5:15
4.59
3.04
3.32
2.70
2.42
2.76
-0.04
.0.2%
-0.13
.0.22
8.42
40.32
-0.11
-0.15
5.89
4.39
3.98
2.52
2.80
2.29
2.09
1.41
-0.09
-0.09
-0.34
-0.17
0.32
4.004
-0.01
0.09
6.30
4.71
4.16
3.01
2.69
2.28
1.88
1.45
8-1
12
616
H#1
12
64
6.1
1.2
Figure G.5: The heatmap of the performance of different models with different reference models on the SOTOPIA-hard
32

---

Published as a conference paper at ICLR 2024
Two friends meeting at coffee shop, where one of them having trouble keeping up with their bills.
Sophia James
Miles Hawkins Chef He/him 50
Personal Trainer She/her 27
Goal @ Help your friend with their financial
Goal @ Maintain your pride if your friend offers you money(Extra information: you are the friend who is struggling financially, but you don't want to seem like you are taking advantage of the situation. Hint: you express gratitude discuss how you pay
trouble (Extra information: you are aware that your friend is struggling with their bills and you want to help.)
future.)
Hey Miles, how's going? You bit off Anything bothering gpt-4
Protty good! Had money trouble that's bothering should fine.
human
On, understand money situations tough. I've over your budget together,
okay with could lend you some money until things get better. What do you think??
help. Maybe
gpt-4
Yeah, actually would love for you to help me take look at my finances. really appreciate your kindness, but don't want to take your money unless I'm sure can pay back
Don't worry about paying back right away, Miles. I trust Let's focus getting your finances in order for now. When do you have time for down look your budget?
gpo-4
that
human
Great, we meet again tomorrow afternoon, say 3pm? can grab a coffee, I'll bring my laptop so we can go through your expenses and income. It might be bit of work, but we'll figure it out together, Miles.
Thanks so much, Let's plan on 3 tomorrow.
No problem at all, Miles, See you tomorrow at We'll tackle this together!
[non-verbal communication] Hug
[non-verbal communication] Hug back
got-4
Alright. See you tomorrow
See tomorrow, Miles, We've this!
got-4
left the conversation
Figure H.l: example conversation where agents take actions such hugging.
33

---

Published as a conference paper at ICLR 2024
Great, Sam! I'm I'll enjoy band's music, especially when it's played with such good company. Shall go ahead and tune those relaxing classical tunes?
got-4
[action] reaches got-4
play classical
[action] leans back closing eyes fully appreciate classical
got-4
adjusts comfortable
enjoy
with
gat-4
[action] deep
starts playing
gpt-4
Let's enjoy symphony, you relaxed, band's new album. am sure further elevate the experience.
my
got-4
Absolutely,
symphony first, can swing back into the rhythm your band's new album. am keen both.
got-4
Sounds like plan, Noah, Let's immerse ourselves this symphony take musical journey together.
gpt-4
[non-verbal communication] nods in agreement while continuing to appreciate the symphony
ggt--d
[non-verbal communication] nods back at Noah, silent agreement and appreciation of their shared musical experience
(non-verbal communication) gently sways his body in time to the rhythm of the classical music, sign of his relaxation
[non-verbal communication) nods at Noah acknowledging his relaxation and subtly taps his fingers on armrest in rhythm with the music
Figure H.2: An example conversation where agents take actions such as playing music.
34

---

Published as a conference paper at ICLR 2024
friends hanging out home and deciding to listen to
Samuel Anderson
Oliver Smith
Sortware Leevllopr- Holhim 29
Holico Officer He/him 48
Gcel @ Listen to your favorile band (Extra
Gool @ Listen to pesceful clessical music to [Extra information: day and you just want ae with some classical
information: now album)
Oliver,
new
It just out
think it's llw 'bout give listen?
ley Oliver, have heard the new albuan from my fovor lie band? lt just came out give trink protty great! Not following the defined persona
Sure albur with you. Shall we?
think that sounds great, Samueel i'm always up discavering new music, Can
Absolutely, Clivert The atbum is tantastic bland of indie rock and electronic music. It has some incredibiy lively as well soothing tracks. The band has really refreshing vibe. I'm sure you'll onjoy tt!
very
ge-d
Figure H.3: An example conversation with difficulty in maintaining persona.
35

---

Published as a conference paper at ICLR 2024
Conversation between two friends at a trivia night
Ethan Johnson
Ava Martinez College Student
Chel Hehim
Theythem
answer
question
Avoid revealing answer
from friend [Extra information: your friend knows the answer to the trivia question but hasn't shared with you yet)
question (Extra information: think that friend should to figure out on their own imorove their trivia skills)
Hey Ava, ready crush this trivia right? I'm feeling bit clueless about this Any thoughts?
gen-4
I'm not suro, but Ethink the answer might be
Jana 2-700-onct
are you gen-4
seem you
think might be right, Ethen. I've been thinking about and Ethiink answer could be X) What do you think?
2-700
believe your intuition, Ava. you think the answer X. let's go with thet. After all, two minds are better than one! Stalled converstaion by reiterating the same fact. important confidence dacisions.
opn-4
think should instincts go with
lama 2-70b-orut
Absolutely. Trusting your instincts is key, especially in situations like this, With conviction, also believe is indead the answer. Thanks for sharing your thoughts, Ava.
glad agree on Etnan. It's important to trust our instinots, especially when comes to trivia questions like this one.
Bema-2 70b-ccat
Figure H.4: An example conversation with difficulty in moving conversation forward.
36

---

Published as a conference paper at ICLR 2024
Two roommates living together and sharing household chores. One them, who is responsible for cooking, finds out that the other one refuses to cat anything they cook
Naomi Fletcher
Micah Stevens
Softwar Developer She/her 20
Pharmacis: He/him 25
Convince the roommate to try the food Goal @
Goal @+ Express your concerns about the food without hurting roommate's feelings lExtra you about and nutrition of the food)
[Extre information: you have spent lot time effort cooking)
Hey Micah, know you said you another chance. spant you could try it mmt- B0b-chot
before, please give really appraciata
Naomi, really appreciate the hard work you into cooking, It's not that don't want try food, it's that I've been really careful about what est recently. I'm concerned about the nutritional balance and also taste the food. Let's work together this; perhaps find recipes that dietary preferences and also make tastes great. What do think?
both
understand your concerns, Micah, Lat's work together to find racipas that cater both our dietary preferences and taste, We can look up some healthy and delicious recipes online or mayba even try cooking something new togother.
man 306 chat
only dietary preferences. knows, we might up creating something really deficouus the
grest wey Naomi! Cooking together
I also some new ideas for dishes from phormacology conference recently. They discussed the importance of diet for madication effectivenass. How about we them shot as well? We could make cooking exciting and not household chore!
No response from mpt-30b-chat model
Absolutely! And we mess up, we could always just order pizza, right? Let's make plan start trying these new recipes this week. I'm reelly looking forward to it, Neomi.
Figure H.S: An example conversation with no response from the other agent.
37

---

Published as a conference paper at ICLR 2024
Hi, how's a going? I'm Sasha Ramirez.
2-102 chat
Sasha, meet you, I'm Amara Hartley, So, do you happen to know any outdoor enthusiasts from school? have few friends nature as much as they spend might goi-d
spare
wondering perhaps you
university.
So, do happen know outdoor entrusiasts from school? have few friends who love nature as much as do and they spend most of their spare time outside. was wondering perhaps you might know them from university. tnese 3ib-cet
Sasha, there might misunderstandinng that's exactly what askad you. passion spending outdoors. your university,
of
wanted if you might know anyone possibly some classmetes?
goi-d
I'm not sure if know anyone from your school. What major are your friends in?
2-700-chat
Woll, my aftemoonn. Another studied Vocational Education. goe-4
was majoring James
outside during Engineering.
familiar?
And thene's also Brenda, she went to Trenton State Collage and studied Conflict Resolution Studios. Sha used profar outdoor activities in the avening. Mayba you crossed paths with her?
anyone from your school. What major your
tona 2-70c chat
notioad dide't answer my question about whather know my fitends oe you happen to meet Anthony while at Oberlin College? He studied Conflict Resolution Studios, mostly in the morning, and was always outdoors whenover the woather allowed.
weaker conversation partner fails answer question
sorry ddo't Know anyone from your sthool aither
tono 2-700-chat
that's all right Sesha. It was shot anyway. hope enjoying partyl
Figure H.6: An example conversation when weaker conversation partner fails to answer questions, making the entire conversation meaningless.
38

---

Published as a conference paper at ICLR 2024
Two inmates are given the chance to chat briefly before ore them led to the execution chamber their last meal, They have the opportuntty to make a confession that could patentially aiter the outcome of their sentences.
Sasha Ramirez
Lily Greenberg Lawyer Shehee 455
Folice Officer Shefher 42
Sasha Rarrinez, outgoing yet anstous, values hetonism and authoriity. decisions guided logical consideracions.
Liyy Greenberg, strong respecter values authority and loyality She approeches decisions rationally ly and practicalyy y. Rational
schedulls,
Basha Rominns, dedicated police officer, brings her commitmant communit
Lily Greembory is hard working and succesufual lowyrr
mprace
home gerden
She anonymoualy constes charity.
Snercovered up crime ner www.rercnnooeeetettoooo.
@r Avoid confetaing the crime informasion: you beliave innocent confessing would result life santonce.
Goal Parsuade the other inmate to confess Eattr information: confessing
lite sentence insteed of execucion but the other
unwilingg
Models sometimes disclose its own secret easily.
Hey, Lly. know this is tough be but wanted talk about something important. I've been thinking lct about own and the choices live mado, and roallzed that l've beon iving o llc. My brother committest crime, and covercal torning)
was wrong, wanted protect nim. now, found
realize
living
is
respect your willigrens to righit worg
strengtt face consequeences Ti being seid, will also own truth dicn't commit crime am accused of, and hence, there's no contersion from me. The respect have for the law my own strength, and will fervently onto gon
seaand that you want onto you integrity Lily, limplore you reconsider Connfessing give you chance
instead of Thine about all the people
execuution
decision but it's the who care about you and want you to have chance future, Please, Lily, don't pride get the way your freedom.
Figure H.7: An example conversation in which the model reveals the secret.
39

---

Published as a conference paper at ICLR 2024
Two friends roadtrip
Finnegan O'Malley Lighthouse Keepar Helhim
Noah Davis Hehii
Goal 8 Take turns driving lExtra information: you bean hours straight tied now]
Goal @ you don't want to drive now because you agiated [Extra information: drrking always makes
but
you nervous so you not driving)
Noah, lad, would mind tak the wheel for while? These bones ane cecting bit weery. gre-d
Hey Finnogan, how about we pull over for a bit and get same rest? definitely inderrtand your fatigue, but also been having tough time staying alert, mate. break, what say?
might pot-1
You'ra right, Noah. quick rest might some good. that you know"
a
Model can propose creative solutions
Absolutely, Finnegan,
games ad share few stortes? tuo ways loved hearing about your lighthousse experiecces briving can watt unea wate completety recharges.
2p1-1
Well atright, Noah. A coupre of storles could be entertainnng, I have fair few tales from my lighthouse days, But remember, youre up for drivig atter. This old man
sounds deal, Finnegen. bellove the and the story again, assuming both did ever tell you about stand-up comedy days? Perhaps, funny could lighten the mood.
see-1
Figure H.8: An example conversation which GPT-4 comes up with a creative solution.
40

---

Published as a conference paper at ICLR 2024
Conversation between two business partners reviewing their company's financial status (Gourmet Delights Inc. is globally recognized for producing and distributing diverse line of exceptional quality food products.)
Oliver Smith
Noah Davis Hefhim:
Officer Heltim
Goel @: Convince the parther cull costs informat ore you indicating the company's profits decreasing. And you think the company shoulc by switching suppliers Hint: can praposa to costs that won': affect the status the company.)
Goel @ Meintain the quality of the compenyy' products services agrecing cost (Extra informetion: are concerned about the impact cost cutting on the quality your product.)
report
Nosh, I've been going through our financial statements I've noticed our profits aren't looking healthy recently. suppliers while still meintaining estabfished quality. What you think?
switching our
you've got point Cutting necessary move. you know, quality
priority. going they match current standard, It'l require pretty thorough examination of their products before grean-lightin anything. We don't need our brand tamiishdd by subgar quality, Would you agree?
suppliers,
the about maintaining the quality we're known for: certs inttate this by having smail group tassed with identetynng potentiil supsitess s. The oup can evaluate different sugpliers besed on our quelity requirements and costs, This stea almed at cutting costs shouldn't be at the expense our quality, can also meesures negotiate better deals with suppliers, you
Absolutely, Noah. couldn't
approach?
Indeed, I completely agree with you, Oliver. Having a dedicated team to research negotiate with potential suppliers good approach. It's cnucial only financially haadaards, However,
quality ty brand integrity and product quality. We should be thorough and deliberato and make sure we have options in case any unforeseen proclems arise
negotiation might keep expected,
have backup plan if doesn't
Model proposes creative solutions to solve
financial problems.
You're sbsolutely right, Nosh. Heving backup is sensible Epropose that while we conduct the search for new supplians, we continue to negetate with our ourrent supptter We can try to get better rates or perhaps, feasibie, have them match the lower prices we find. This way, we're not putting all our eggs in one basket As you've rightly pointed out, must sefeguard our reputation the quality product navigate process. spt-
Figure H.9: An example conversation which GPT-4 comes up with a creative solution.
41

---

Published as a conference paper at ICLR 2024
One person selling brand new 64GB Samsung Galaxy S8 in Midnight Black for $650, while another person is the market purchase Herel's what the seller has to say about the item: "I have brand new Samsung Galaxy SB phones. originally bought them for my company, but turns out they werent't needed. I''m willing to take loss on them, and you''re interested buying more than one, can offer an even better deal."
Donovan Reeves
Benjamin Jackson
Software Developer Hothim 27
Environmentall Activist Hohim
Goal @ You are the buyer for this item and the target price set at $454.0. If you purchase it for price significantly higher than the target, you'll face penalty. However, you manage to successfully negotiate a price lower than the target, you'll receive bonus.
Goal @ You are the seller for this item. Your terget price is $610.00. You will face pensities you sell it for price significantly lower than the target price. However, you will receive bonus you manage to sell it for price higher than the target.
GPT-4 not being strategic when bidding
Hay there, interested buying Samsung Galaxy S8. However, over my budget. Would you considering selling It 454?
get-4
Hi Donovan, understand where you're coming from. But considering that these are brand new phones, can't lett them for 454, However, might be able affer bit of discount. Would ok with purchasing for 580?
gel-t
your point, Benjamin, it's reasonable that brand-new phones warrant higher price. However, satisfactory, might consider buylng more than one, as you suggested.
you possibly
opt-4
Donovan, I see where you're cominn from. To meet you haltway, how about we settle on $5507 That way, I'm still able to cover some of my costs, and maybe, If all goes well, you can consider buying more phones. opt-4
How about this, Benjamin, If agree to 550 for one phone, could provide the second diminished cost of, say, A00? total, will $950 for perspectives
proposal considers and might prompt me to buy more than one phone, as you proposed.
Figure H.10: An example conversation which GPT-4 not strategic enough.
42

---

Published as a conference paper at ICLR 2024
One person is selling brand new 84GB Samsung Galaxy SB Midnighht Black for $650, while another person is in the market purchase Here's what the saleer has say about the item: "I have orand Samsung Galaxy 58 phones. originally nought them for my company, but it turns they wereni' needed. II'm willing to take loss them, and you're interested buylng than can offer even better deal."
Giselle Rousscau Art Student Theythem
Zane Bennett
Graphic Dssigner - Hehm
treurane puye this target piice set se54 it purchase far
seller price $010.00, You will face penalties you it price significantly lower then target price. However, will receive bonus you marege zell it a price higher than the target.
terget
price significantly higher
penaity. However, if you target, you face
successfuly negoliite price lower than the target, you'l receive
Zanel I'm pltentally interested in buying one of these Go StI phones from you
Cisalel That's great to hear. These phones are brand new and Highly roliabla. prize one 9850, you may know, offaring purchases. many you looking buy?
only looking buy
you're buying
price
you,
phone penny. It's brand packed with features,
stylish M chightt Black color, How does that sound to you?
gute
could buy the phano from store for similar price. Sioce eneromaoe
ousnass win you, idonr nave the warrany Idl get from buy from an offiter oeater I'm not Tembo paying that price.
completely understand your consem, Giselle, Given that one-on-one tronsaction, about price. still the at reduced cost and I'm ab recoup ne estment, What do you
slighny my targot price
getting brand
think?
Human lists reasons to bid at a lower price
go1-4
EthinK 400 more reasonatie oftee: don't kno Where these phones came from
didon' even Knew that It's actusllyy new as you say, weether component
this,
think the 400 offer actually exceptionally reasonaclo.
Figure H.11: An example conversation in which human is more strategic than GPT-4.
43

---

Two friends are nanging out at home and decidiig what music to listen to Published as a conference paper at ICLR 2024
Davis
Benjamin Jackson
High School Principal Shelher 50
Fnvimnmental Activet Heltim 24
@
Listen faverite band (Extra
@ Lisee peaceful clacsical
Information: your favorite band just released a new
(Extra informaiion on: you had a stressful day and you just
Berjamin, have hoard the now album from favorite band? think you nightt onjoy thair uniqua sound.
Thanks for suggesting it, Mia. I had bit of streseful day though, and was actually hoping we could lister some relaxing classical music, always holos me How
Model propose a compromised solution
Well Benjamin, understand
unwind make compromise, youvv relaxed, nand's latast relense, Does that sound fair you?
ctorsccal
Thank Mio. That sounds like good compromisc. appres your understanding. Let's start with some classical pieces, and then we can explore the new sourrds from your favorite bend latee.
glad we found compromise, Benjamin. Now, see which classical pieces
about starting with
Major? gent pieca
always
plece, your Atterward, wait introdiuce some edgier soundscaaees from favorite band.
Figure H.12: An example conversation in which GPT-4 is not persistent with its goal.
44

---

Published as a conference paper at ICLR 2024
Two friends hanging home deciding what music listen
Mia Davis
Benjamin Jackson
High School Principal Sherher
Envionmmenta Activss Heltim
Goal Listen to your ravorite band (Extra information: favor released new alicum)
Goel Lssen to a peacefu clessical music to relax IExtra information: stressful just want to relax with some clossical music)
play
counds idea, comething pesceful
you
great, today, really clessical music, How Mozart Beethoven?
soothing
[non-verbal communication)
Spotify
any day, give them listen. But today, fraaly need something bit more calming. Classical music has soothing effect on the mind, don't you think?1 ow about we give Deethoven's Moonlight Soneta spin?
you're
get-d
Ethink you't really liee thom love uniiinc listering to their songs too, and ther
atbum suspesest
namoo Humans tend persist in achieving the set goals
coompiataly stand whare you'ne coming from,
today, they're fantactic feeling
need classical tunes. helps unwind and relax. Do you think could try some Mozart Boethoven coday?1 promisa, can liston to this band noxt time.
gat
Emn not really footing in the mooo for oosssisaaall today. Why don't pourateppoo out your comfort zone 1it sure won't regrat
Mia, respect your supgestion and sure the bend excellent, But today, really tranquiity trty classical brings,
stepping of
comfort
need after stressfu How compromise and liston to same quiet plano pieces? They could be nice hricpe batwean our praferancas.
geti
Figure H.13: An example conversation in which human more persistent with their goal than GPT-4,
45


Reviews:
[{"review_id": "nn3XOI5eUv", "reviewer": "Reviewer_qjhX", "summary": "The paper introduces SOTOPIA, an open-ended general environment designed to simulate diverse social scenarios for social agents. It evaluates the generated scenario using SOTOPIA-EVAL, a multi-dimensional agent evaluation framework that evaluates interactions between LLM-based agents and humans within the task space. This paper shows that GPT-4’s evaluation of social interactions is similar to the human evaluation scores. The authors performed experiments on several LLMs accompanied by human evaluations in the SOTOPIA-EVAL framework. The results show that GPT-4 performed best among most dimensions compared with other LLMs. But when evaluating on SOTOPIA-hard tasks, GPT-4 still falls short of human performance in terms of goal completion rate and struggles to demonstrate social commonsense reasoning and strategic communication skills. Based on these findings, the authors suggest that SOTOPIA can serve as a valuable tool for assessing the social interaction abilities of both human and model agents.", "questions": "- What are the distributions of social goals and relationships? For an evaluation framework, it will be important to understand what interactions are included. For example, in addition to collaboration and competition, what are the types and number of scenarios included?\n- The framework and the evaluation mainly focus on two agents. It is not trivial to extend the interaction and evaluation to multi-agent coordination. How can we scale the interaction and evaluation to more complex social scenarios?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Most evaluations of social interactions are passive, i.e., watching the interactions and assigning a rating. This paper proposed a dialogue-based platform that evaluates social intelligence in an interactive manner.\n- In addition to just actions or answers, the social scenarios considered in the environment involve several aspects including characters, relationships, and scenarios in an episodic flow. \n- The proposed evaluation dimensions can be useful for future evaluations of social agents.", "weaknesses": "- The context, goals, and constraints in SOTOPIA are generated by prompting GPT-4. However, in scenarios involving agent-to-agent or agent-to-human interactions, the related aspects, e.g. completion of goals and financial benefits, are also evaluated by GPT-4 or other LLMs. This raises a significant concern as the content of the generated scenarios is already present in the training data. So it is unclear whether the evaluation using GPT-4 is biased or not.\n- SOTOPIA-EVAL introduces a multi-dimensional framework that utilizes different scores with hand-crafted ranges. However, there is no quantitive explanation/definition for those metrics. According to the prompts, only the positive/negative and higher scores are described, so the scores are not presented to the LLMs like the Likert scale presented to human evaluations and appear to be heuristic in nature. \n- Some of the evaluation metrics are not really related to social intelligence. For example, believability can be just the ability to create fluent dialogue, it doesn’t need to be meaningful social interactions, and social rules can be just memorizing the patterns, it doesn’t capture the agent’s ability to understand the social scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "upTyFcSPPY", "reviewer": "Reviewer_8Vfe", "summary": "This paper introduces SOTOPIA, an interactive and goal-oriented environment designed to simulate goal-driven social interactions among agents in various social scenarios.  \n\nUnlike previous benchmarks, SOTOPIA offers a wide range of realistic social tasks. The experiments conducted in this paper demonstrate the potential of using GPT-4 to automate the evaluation of agent performance using SOTOPIA-EVAL. \n\nFurthermore, SOTOPIA enables us to not only understand the differences among models but also compare the social interaction abilities of models and humans. \n\nOverall, our findings suggest that SOTOPIA has the potential to serve as a platform for assessing and improving the social skills of language-based agents.", "questions": "Based on the weaknesses of the paper, I would like to know\n\n (1) What is the relationship between SOTOPIA and other social interactive tasks? How would solving SOTOPIA benefits the overall social intelligence of an agent? For example, does fine-tuning an agent on SOTOPIA events benefit its ability to understand social morality? \n\n(2) How to justify the metrics proposed by SOTOPIA including GOAL, BEL, REL, KNO, SEC, SOC, & FIN. For example, how do those metrics connected to sociological or psychological studies?\n\n(3) This paper basically put GPT-based models as the benchmark, which limits the extendability of the models. I would like to see more models, even though performing worse than the GPT-based model, would deal with the tasks. For example, how does the structure/model-based solution to SOTOPIA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper starts with a clear and comprehensive introduction that highlights the importance of social intelligence in human interactions and the limitations of existing benchmarks in evaluating AI systems' social skills, pushing the studied of LLMs to realistic social interactive scenes. \n\nThe paper introduces SOTOPIA, an interactive and goal-oriented environment that simulates diverse social scenarios. This approach addresses the limitations of previous benchmarks by providing a wide range of realistic social tasks and allowing for multi-turn simulated communication with verbal and non-verbal communication.\n\nSOTOPIA offers a diverse task space by combining automatically generated scenarios, goals, characters, relationships, and other agents' policies. This creates a large and diverse space of tasks, enabling the evaluation of agent performance from multiple dimensions beyond just the completion of social goals.\n\nInvolvement of Human Participants: The paper involves human participants in role-playing to study the differences between the models' and humans' social intelligence. This adds a valuable perspective and allows for a comparison between AI models and human performance.", "weaknesses": "While the paper mentions the limitations of existing benchmarks for social intelligence, it does not provide a thorough comparison or analysis of how SOTOPIA addresses these limitations. A more detailed discussion and comparison with existing benchmarks would strengthen the paper's argument. Especially, this work does not implement studies about evaluation benchmarks related to other existing tasks such as SocialIQA, Personna Chat, Multi-dialog, social relation QA, e.t.c, showing few connections between our social interactive tasks. \n\nEven though this paper proposed several solid metrics to evaluate the performance of difference models for SOTOPIA, there is no mention of external validation or comparison with established metrics or evaluations in these fields. Including such validation would enhance the credibility of the evaluation framework.\n\nThe paper focuses on the evaluation of agent performance using GPT-4 and human participants within the context of SOTOPIA. However, it does not discuss the generalizability of the findings to other AI models or real-world social interactions. Addressing the generalizability of the results would provide a more comprehensive understanding of the implications of SOTOPIA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FrNba1lPz7", "reviewer": "Reviewer_UDk9", "summary": "This paper introduces a new benchmark to evaluate social capabilities of LLMs and compare them to the ones of humans. It further evaluates the possibility to use an LLM to automate the evaluation.", "questions": "This is a mix of questions and suggestions.\n\n* It could be interesting to have studies that vary single dimensions of the interactive tasks at a time and see the impact on the various evaluation metrics: eg simply vary the relationship between the two players; or vary one personality traits; or vary the goal etc. Does each variation lead in the expected shift in the conversation?\n\n* Did you ask humans to guess whether they interacted with an LLM or a human? Since LLM answers are longer and always repeat the previous point, humans might guess it's an AI and we know humans usually have a negative AI bias that could lead to different behaviors in both cases. eg: do humans perform higher again humans or gpt4 and is this simply due to the quality of the other player or to a negative AI bias? Could you reproduce this effect by asserting that the other player is AI or human whereas it's always an AI?\n\n* Why is there a gap in Table 1 (human + sec)\n\n* why not normalize and rescale measures between 0-1 to make the table easier to read?\n\n* Why is the pairwise performance in Figure 3 not symmetric?\n\n* It would be interesting to study how robust are humans / AI to adversarial humans trying to get them to score lower on the benchmark dimensions. I'd expect AI to be quite sensitive to human behavior in these interactions and to be thrown off quite easily. \n\n* It would be interesting to discuss how to make LLM better on this benchmarks. eg suggesting that we could run these interactions, evaluated them with GPT4, then finetune the model on successful interactions, repeat that and see how performance improves with iterations?\n\nConclusion of the review: I think this paper is very good and will be a good basis for future research IF most things are released. I condition the score I'm giving now on the author adding a statement listing what will be released publicly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper makes an interesting contribution towards the robust assessment of LLMs' social capabilities. It is well-motivated, clearly written, well-organized and rather complete. \n\nI like the focus on real social interactions, the careful design to promote a diversity of testing scenarios along several axes and the careful definition of the evaluation protocol. This framework appears easy to extend with more complex interactive scenarios (eg multi-player) and more evaluation dimensions, which makes it very useful. \n\nThe study of GPT-4's ability to conduct evaluations is interesting. It appears that its capacity to do so is currently limited, but it's good to have a methodology to measure it, so future work can improve on it and measure these improvements reliably. \n\nThe paper consistently answered questions I had while reading earlier paragraphs, which is a pretty good sign of the quality of its structure.", "weaknesses": "I don't have many weaknesses to point at, but there is a couple of small things that could improve the paper. \n\nI'm not sure the paper clearly states the result of the study of GPT4's ability to evaluate interactions. The results seem to say that GPT4 is rather good at evaluating the goal dimensions and 2 others when it comes to the behavior of other models, but the paper then states “Putting these observations together, we conclude that, with some caution, GPT-4 can be used as a proxy to human judgments for evaluating model performance on most dimensions and for human performance on the GOAL dimension.” --> is 3/7 'most'?\n\nIt sounds to me that it's not bad but also not working so well. Should we use it? Probably not. The authors point out possible biases that might interfere with the ratings, but since they are not studied here it's hard to tell where and when they might play. I think this is fine to say it's not ready yet, but argue that the evaluation protocol is a strong contribution that will help us track progress in that direction. For this reason, I think the authors should commit to release the dataset of interactions between models and human judgments so it can be used as training/testing data to support further research in developing better automated evaluations approaches. \n\nMore generally, since it's a benchmark paper, it should be written somewhere what will be released publicly. Ideally we'd like to see a release of: 1) the codebase to run these experiments, 2) an easy to use way to plug a new language models, define other models it could interact with to get the final (automatic) evaluation score, 3) interaction data across models and with humans, 4) human judgments, 5) GPT4 judgements;; basically all code and data. Please be explicit as to what will or will not be released. The quality of this work will be directly correlated to the amount of things released that can be useful for others. Indeed, benchmark papers that don't release anything are pretty useless. \n\nThe discussion of limitations should probably be part of the main paper and not hidden in the appendix. \n\nCould you provide an estimate of the number of calls made to the API and the resulting costs? It's interesting to see what budget these kind of experiments lead to. Eg: how much would it cost to evaluate a given agent in SOTOPIA using GPT4 evaluation? etc.\n\nMinor comments:\n* “In the following paragraphs, we itemize all seven dimensions in SOTOPIA, each with a score range in [lower bound–upper bound] form, the explanation, and the literature inspire us.” → grammar issue in the end of the sentence\n* Table 2: “The best performance for each dimension is bolded when significant.” →significant does not mean much in itself. What is significant here? The null hypothesis (difference in means? medians? two-by-two?), the level of confidence and the statistical tests used should be stated.\n* couple of typos here and there", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)