How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks

Authors: Yusen Wu, Junwu Xiong, Xiaotie Deng
Primary area: datasets and benchmarks
Keywords: LLM, Multi-user, Social agent, Evaluation

Abstract:
Expanding the application of large language models (LLMs) to societal life, instead of primary function only as auxiliary assistants to communicate with only one person at a time, necessitates LLMs' capabilities to independently play roles in multi-user, multi-turn social agent tasks within complex social settings. However, currently the capability has not been systematically measured with available benchmarks. To address this gap, we first introduce an agent task leveling framework grounded in sociological principles. Concurrently, we propose a novel benchmark, How Social Is It (we call it HSII below), designed to assess LLM's social capabilities in comprehensive social agents tasks and benchmark representative models. HSII comprises four stages: format parsing, target selection, target switching conversation, and stable conversation, which collectively evaluate the communication and task completion capabilities of LLMs within realistic social interaction scenarios dataset, HSII-Dataset. The dataset is derived step by step from news dataset. We perform an ablation study by doing clustering to the dataset. Additionally, we investigate the impact of chain of thought (COT) method on enhancing LLMs' social performance. Since COT cost more computation, we further introduce a new statistical metric, COT-complexity, to quantify the efficiency of certain LLMs with COTs for specific social tasks and strike a better trade-off between measurement of correctness and efficiency. Various results of our experiments demonstrate that our benchmark is well-suited for evaluating social skills in LLMs.

Decision: reject
Reviewer scores: 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730616268715

thread_type:
official_review

<thread_object>
creator: reviewer_xrb5

<content>
summary:
This work proposes a novel benchmark named as How Social Is It (HSII) to assess LLM's social capabilities in different social agent tasks that are constructed from the news dataset. Typically, with a careful four stage design including format parsing, target selection, target switching conversation, and stable conversation, such a framework can evaluate the communication and task completion ability of LLM in interactive multi-user multi-turn complex social tasks. Moreover, to dive deeper into the CoT method on enhancing LLM's social performance, this work proposes CoT-complexity to quantify the efficiency of CoT to conduct social tasks.

strengths:
(1) Unlike previous social simulation work that evaluates conversations along specific dimensions, this study introduces a four-step hierarchical framework for assessing multi-user tasks and social capabilities. Additionally, the design of the social agent goes beyond simple LLM-based conversation; it draws on foundational sociological theories, which decompose social relationships into distinct components for richer interaction modeling.

(2) The HSII-Dataset is grounded in a news dataset, rather than relying solely on general daily-life scenarios. This approach ensures realistic and complex social contexts. Furthermore, clustering analysis of the dataset confirms that it reflects both the similarity and diversity present in the original source news categories, enhancing its relevance.

(3) This work introduces a novel metric, CoT-complexity, that captures the importance of reasoning abilities (often overlooked) in social scenario conversations. CoT-complexity provides a precise measure of social task complexity, focusing on the critical role of reasoning within these interactions.

weaknesses:
(1) The dataset construction pipeline includes several elements that are not explained clearly, which makes understanding them difficult. For example, the concept of "background conversation" is mentioned twice within the pipeline but lacks a concrete definition, leaving its purpose and role ambiguous. Additionally, there are no clear examples showing the difference between a "golden response" and a generated response. This absence of examples makes it challenging to grasp what specifically distinguishes a high-quality, or “golden,” response from those generated by the models. Furthermore, details on how the agents were collected and how their profiles and conversations were generated are missing. These are critical aspects that need more clarity to fully understand the dataset’s construction process and its assumptions. Moreover, the figures included in the paper are too small and difficult to read, making it harder to visually interpret the data and results.

(2) The comparison of model performance lacks depth, as the results only indicate that Llama3-8b outperforms the other two models on metrics like R3, R4, and HSII, without providing any analysis as to why these differences exist. There’s no discussion of what specific dimensions of the models' abilities might account for Llama3-8b's superior performance, particularly on the HSII score, which measures social intelligence. This leaves open questions about what it means for a model to excel in one dimension versus another. Including case studies that compare model-generated responses with the golden responses would offer valuable insight into how these models differ and what these differences imply in practical terms.

(3) The paper does not include human-based evaluations to verify the reliability of the HSII metric. Without assessing whether HSII scores correlate strongly with human judgment, it’s challenging to assert that HSII accurately measures social intelligence. Adding a component that validates HSII through human analysis would lend greater credibility to claims about its accuracy and its suitability as an evaluation metric in this context.

questions:
(1) I would like to understand what is the concrete output during each stage of the evaluation dataset construction pipeline. Additionally, what is the concrete output of the each step of the HSII evaluation framework. The social task example in the appendix is incomplete and confusing.

(2) I would like to see what is changed after adding CoT into the social scenario understanding stage, especially the technical details about how the reflection is made.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692328374

thread_type:
official_review

<thread_object>
creator: reviewer_w2ok

<content>
summary:
This paper introduces a novel assessment method known as the Core Sentiment Inventory (CSI) for evaluating the emotional tendencies of large language models (LLMs). Inspired by the Implicit Association Test (IAT), the CSI aims to provide a more reliable and effective way to assess the implicit emotional characteristics of LLMs. It addresses the limitations of traditional psychometric methods, such as model reluctance and inconsistency.

strengths:
•  The CSI can effectively quantify the emotions of models, reflecting the emotional tendencies of LLMs.

•  It effectively reduces the issues of reluctance and inconsistency in traditional psychological scales.

•  The use of representative neutral words in constructing the CSI reduces the potential emotional orientation of the words themselves, better reflecting the internal emotional associations of LLMs.

•  It explores the impact of different language environments on the emotional tendencies of LLMs.

weaknesses:
•  The constructed CSI is only used to assess the emotions of LLMs and has not been extended to other psychometric fields, such as personality or moral decision-making.

•  It does not introduce the calculation methods for consistency rate and reluctance rate.

questions:
•  Discuss how CSI can improve existing psychological scales designed for humans.

•  Further explore the unintentional biases that language models may have towards everyday concepts as mentioned in section 4.1.

•  On line 424, it might be: “e.g., five positive words, four neutral words and one negative words, and so on.”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730866846308

thread_type:
official_review

<thread_object>
creator: reviewer_ecpx

<content>
summary:
This paper attempts to introduce a multi-turn, multi-user conversational dataset where the users are conversing in a complex social scenario, that can be used to evaluate the capabilities of LLMs to participate in a complex social scenario. To this end, the authors create the conversational dataset from news articles using a capable LLM (GPT-4) in a step-by-step process. Additionally, they introduce an evaluation metric for measuring the performance of LLMs in this scenario. The authors conduct evaluation experiments on 5 open-source and 1 proprietary LLMs in two settings: simple prompt-based and Chain-of-thought based. Results show that contemporary LLMs lag behind human performance on this dataset.

strengths:
**Important Topic**: The authors propose to measure the ‘social intelligence’ of LLMs by putting them in a multi-party conversational scenario. This is a plausible application of LLMs in our society and hence, needs appropriate methods for research and evaluation. Overall, this paper attempts to tackle an important problem.

weaknesses:
**Poor Writing & Representation**: Unfortunately, the writing and presentation of this paper is prohibitively sub-standard such that it has been hard to understand the dataset and evaluation metrics contributions being claimed by the authors in the paper. I highly suggest a thorough revision of the content in this paper, for grammatical errors as well as better understanding of the reader.


**Dataset Details**: The paper completely lacks a representative example of the HSII dataset created in this paper. Without this example, it is difficult to understand the structure of the dataset and its value. Table 3 seems to be an example from the dataset but the format is strange and the unique characters involved in this conversational scenario are unclear. Moreover, any other dataset statistics such as the size of the dataset, the topics from news domain used for creating the dataset, distribution of number of unique characters in each sample, length of conversation, comparison to existing multi-party chat datasets etc. is missing from the paper. There are some preliminary analyses of the dataset using clustering and a figure in the Appendix (Figure 5) which is an uninformative way to represent information from clustering. Further, the paper does not mention whether this dataset is/will be publicly available at any point.


**Evaluation Details**: The evaluation metric presented by the paper is similarly hard to understand in the absence of details about the dataset itself. The pipeline in Figure 2b makes some sense. It seems that the responses are being measured along three axes i.e., target selection, response quality and long-term adherence. The mathematical symbols used in Definition 1 are misleading as $n$ is being used to represent dataset size (which is unclear) and then $n_i$ symbols are being used for other purposes throughout the evaluation process.


**Results**: Keeping in mind that the dataset and evaluation metrics are unclear, it is hard to rate the reliability of the results presented in this paper.


**Motivation based Sociology and Dialogue**: This paper iterates the motivation behind these experiments several times throughout the introduction, as being based out of sociology and dialogue literature. However, there isn’t sufficient discussion of this motivation with appropriate details from that literature.

questions:
- What is the size of the dataset and how does a typical sample look like?
- What are the broad news topics used for creating this dataset?
- How is the human evaluation experiment performed for this dataset and what are the details of the said experiment?
- Is this dataset going to be made publicly available at any point?
- What is the motivation behind defining the HIIS evaluation metric in the manner laid out in Equation 1?

That being said, I would be convinced by nothing short of significant rewriting of this paper to improve my scores.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes the HSII benchmark for assessing LLMs' capabilities in multi-user, multi-turn social interaction tasks, with a focus on task-leveling and social intelligence evaluation. While addressing an important problem, the paper suffers from significant weaknesses. The methodology is poorly articulated, with insufficient explanations for dataset construction, evaluation metrics, and critical elements like "golden responses." The figures are unclear, and human evaluations to validate the proposed metrics are absent. The experimental analysis lacks depth, failing to explain observed model performance differences.

Strengths include tackling an important area and introducing a structured evaluation framework. However, the methodological gaps and lack of clarity undermine its contributions. Decision: reject, with encouragement to address the weaknesses and resubmit.