SNS-Bench: Defining, Building, and Assessing Capabilities of  Large Language Models in Social Networking Services
Authors: Hongcheng Guo
wangyue
Shaosheng Cao
Fei zhao
Boyang Wang
Lei Li
Liang Chen
Xinze Lyu
Zhe Xu
Yao Hu
Zhoujun Li
Venue: ICML 2025 poster (2025)
Keywords: Benchmark
Social Network Services
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: With the rapid advancement of Social Networking Services (SNS), the need for intelligent and efficient interaction within diverse platforms has become more crucial. Large Language Models (LLMs) play an important role in SNS as they possess the potential to revolutionize user experience, content generation, and communication dynamics. However, recent studies focus on isolated SNS tasks rather than a comprehensive evaluation.
In this paper, we introduce SNS-Bench, specially constructed for assessing the abilities of large language models from different Social Networking Services, with a wide range of SNS-related information. SNS-Bench encompasses 8 different tasks such as note classification, query content relevance, and highlight words generation in comments. Finally, 6,658 questions of social media text, including subjective questions, single-choice, and multiple-choice questions, are concluded in SNS-Bench. Further, we evaluate the performance of over 25+ current diverse LLMs on our SNS-Bench. Models with different sizes exhibit performance variations, yet adhere to the scaling law.
Moreover, we hope provide more insights to revolutionize the techniques of social network services with LLMs.

SNS-Bench:
Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services
Hongcheng Guo 1
Yue Wang" 2 Shaosheng Cao²
Fei Zhao 3 Boyang Wang' Lei Li* Liang Chen
Xinze Lyu Zhe Xu
Yao Hu 3 Zhoujun Li
Abstract
In recent years, the development of Large Language Models (LLMs) has significantly advanced our ability to understand and generate content across various tasks (OpenAI, 2023; DeepSeek-Al, 2024; Touvron et al., 2023; Yang et al., 2024a; Zhao al., 2023), including text generation, sentiment anal- ysis, dialogue systems, etc. Social Networking Services (SNS), key platforms for modern information dissemina- tion, have become major venues for human communication, expression of opinions, and emotional transmission. Con- tent on SNS often embodies personal emotions and social- cultural expressions (Yang et al., 2024b; Jin et al., 2024). Therefore, understanding contents requires not only accu- rate text interpretation but also deep exploration of the emotions and cultural contexts embedded within.
With the rapid advancement of Social Network- ing Services (SNS), the need for intelligent and efficient interaction within diverse platforms has become more crucial. Large Language Models (LLMs) play an important role in SNS as they pos- the potential to revolutionize user experience, content generation, and communication dynam- ics. However, recent studies focus on isolated SNS tasks rather than a comprehensive evalua- tion. In this paper, we introduce BNN-BENCH, specially constructed for assessing the abilities of large language models from different Social Networking Services, with wide range of SNS- related information. SNS-BENCH encompasses 8 different tasks such as note classification, query content relevance, and highlight words genera- tion in comments. Finally, 6,658 questions of social media text, including subjective questions, single-choice. and multiple-choice questions, are concluded in SNS-BENCH. Further, we evalu- ate the performance of over 25+ current diverse LLMS on our -ENNH. Models with dif- ferent sizes exhibit performance variations, yet adhere to the scaling law. Moreover, we hope pro- vide more insights to revolutionize the techniques of social network services with LLMs. https: //github... com/ HC- Guo SNS- Bench.
Despite the success of LLMs across various domains (Xu et al., 2023; Li et al., 2023b), evaluating their performance in SNS remains challenging (Bandura). Recent studies have explored LLMS in specific SNS applications (Jin et al., 2024; Liu et al., 2024; Alhamed et al., 2024a; Qi. 2024; Yang et al., 2024b; Chen et al., 2024), but these efforts focus on iso- lated tasks rather than providing comprehensive evaluation of SNS capabilities. Existing benchmarks lack systematic frameworks, standardized metrics, and datasets that capture the diversity of social media interactions, particularly emo- tional understanding, social interaction, and cultural context. A comprehensive evaluation is crucial for assessing how well LLMs navigate the nuanced and dynamic nature of SNS, ensuring their effectiveness in real-world applications.
To address this gap, we introduce SNS-BENCH, bench- mark specifically designed to evaluate the diverse abilities of LLMs on Social Networking Services. SNS-BENCH contains 8 SNS-related tasks (e.g. Note Classcification, Sen- timent Analysis, and Comment Interaction). Besides, we propose specific metrics for each task. This holistic eval- uation enables us to gain deeper insights into the social attributes of LLMs in real-world social contexts, thereby advancing related applications. The data is sourced from a social platform with over 300M users (we have provided valid data license), covering a wide range of domains such as daily life. entertainment, and social issues. includes various text types, such notes, posts, comments, and
1. Introduction
"The true art of socializing is not merely conversing with others, but establishing resonance with them." " Carl Jung
'CCSE, Beihang University *Nanjing University 'Xiaohongshu Inc. "The University Hong Kong "Peking University. Correspon- dence to: Shaosheng Cao <caoshaosheng @ xiaohongshu.com>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Benchmarking Large Language Models in Social Networking Services
conversations.
Evaluation Benchmarks for Network Tool Using. Many tool-use benchmarks have been proposed, some including network tools, but they face significant limitations. First, these benchmarks struggle to assess complex social me- dia services involving multiple network tools (Huang et al., 2024; Li et al., 2023b; Patil et al., 2023; Tang et al., 2023). Second, many rely heavily on GPT models, resulting sub- jective, unstable outcomes and high costs (Qin et al., 2023; Tang et al., 2023). Finally, critical aspects such multi-user interaction comments on SNS are often overlooked, leading to incomplete evaluations (Zhuang et al., 2023; Li et al., 2023b; Patil et al., 2023).
We conduct experiments on SNS-BENCH with more than 25+ LLMs, with the following main contributions and find- ings:
We introduce SNS-BENCH, the pioneering bench- mark meticulously crafted for social networking ser- vice (SNS) environments. Comprising distinct tasks, it spans an extensive array of sub-domains, posing formidable challenges to contemporary LLMs in the arenas of intricate social interactions and note under- standing within the SNS landscape.
In contrast, SNS-BENCH addresses these gaps by encom- passing diverse SNS-related scenarios and introducing multi- dimensional evaluation metries. With automated evaluation and metrics like FI, semantic similarity and ANLS, SNS- BENCH provides a more comprehensive and reliable assess- ment framework.
The comprehensive evaluation is conducted spanning different sizes and series. Generally, closed-source models outperform open-source ones. Nevertheless, the performance gap between the leading closed-source model and the top open-source model is quite small, only about 1 1%.
Models perform more poorly in tasks that involve un- derstanding complex emotions or long notes, compared with more straightforward tasks such as selecting hash- tags for notes.
3. BENCH
Overview: In Section 3.1, we outline the specific capabil- ities of LLMs across various SNS-related scenarios, with particular emphasis on their performance. Section 3.2 details the pipeline used for curating benchmark data. In Section 4, we evaluate 25+ LLMs and provide analysis. Figure 1 presents an overview of 8 tasks in
2. Related Work
Recent advances in LLMs have driven interest in their ap- plications within social networking services (SNS). Our work aligns with two key areas: (1) LLMs in social services, including misinformation detection, rumor analysis, and mental health assessment, and (2) the need for systematic benchmarks to evaluate LLM capabilities in SNS.
3.1. Defining
Defining the capabilities of large language models in So- cial Networking Services (SNS) is crucial. The core skills for SNS (Zeng et al., 2024b; Alhamed et al., 2024b; He et al., 2024) can be summarized into for parts: note com- prehension, information retrieval, sentiment and intent analysis, and personalized recommendation. Below, we outline these capabilities and the corresponding designed tasks. The detailed instruction prompts are in Appendix A.2:
LLMs in Social Services Recent work has explored LLMs in specific SNS tasks (Zhou et al., 2024; Zeng al., 2024c; Törnberg, 2024; Yang et al., 2024b; Bandura; Chen et al., 2024). MM-SOC (Jin et al., 2024) evaluates MILMS' ability to handle social media tasks, which mainly focuses on misinformation detection and hate speech recognition. Liu et al. (2024) propose an LLM-empowered rumor de- tection approach teach L.MMs to reason over important clues in news and comments. Prior work has demonstrated the effectiveness of LLMS in extracting evidence and an- alyzing mental health signals from social media content from Reddit posts (Alhamed et al., 2024a). Such analysis research (Qi, 2024) provides empirical evidence for evalu- ating their capabilities in social networking services. Com- prehensive evaluation of LLM capabilities across diverse SNS aspects remains limited. Existing benchmarks lack systematic frameworks, standardized evaluations, and di- verse datasets. ENS-BENCH addresses these gaps by open- sourcing a framework to define, evaluate, and assess LLM capabilities across multiple dimensions of social networking services.
Note Comprehension. Models must understand and ana- lyze textual content within SNS, identifying key information. The designed tasks are: (1) Toxo-Taax,nmy. Categorizes notes based on content, with single-choice and multiple- choice subsets. (2) Note-MRC. Tests reading comprehen- sion by rephrasing queries, evaluating relevance, extracting key information, and providing reasoning, with simple and complex levels.
Information Retrieval Effective search functionality is essential in SNS. Models should interpret query intent and retrieve relevant information from large datasets, delivering the most accurate results. Tasks: (3) Note-NER. Extracts named entities from note content. (4) Note-Query Matches queries to content, assessing query-based answers and thematic relevance.
2

---

Benchmarking Large Language Models in Social Networking Services
search...
Trending
Recomended
Subscriptions
[TACK] Single Chosse
Note
tity Lil miated sitret
seal dis
Notte
candl coc
cil tap
hao
Single-chhle
Hultipte-Chskee
Nofe
daan
?
Expla ai
Topic
[FAsd]
[TAGH]
Remths vilenne Ven't dature comee
Hucttple-llpp
Compltex
Note- Taxonoay
Note
Noto-MRC
Note-QueryCore
Note Note-Hahhag
StG. Bunch
V 1002
9US-Bench
1010
Mote
Question
Comments
D.
BD
2B
select wordls highlighting
Extract entity according to Question
trigger Benderr
discuesions?
nevel Giserry on note
enttty
entity
Notes
Noto-MER
No Gender
Note QueryGen
NoTo-CHLM
9US-Bench
O 1100
9VG-Benthh
O 1101
905-Bench
O 1110
SKS-Bench
O 1111
Figure Overview of ENS-BENCH. It contains 8 tasks including Note-Taie Taxonomy, Note Note-Quere QueryCorr, Note- hate-Hashtag, NoRe-MRC, Note-NER, Note-Gender, Note- Mote-QueryGee, and Note-CHLW.
Sentiment and Intent Analysis. Models should detect emotional tones (e.g., positive, negative, neutral) and un- derlying intent, especially in sensitive topics like gender discussions. They must identify emotional bias and assess hot-button issues, providing understanding their societal impact. (5) Note-Gender. Determines if content is likely to attract attention from both genders, focusing on gender- sensitive topics. (6) Note-CHLW. Highlights significant words in the comment section, identifying terms that war- rant more attention.
tation. (4) Quality Control. (5) Expert Review. The quantity distribution of tasks is described in Table 1.
Collection To evaluate the capabilities of large language models in Social Networking Services (SNS), we use data from a major social platform with over billion users ! hosting diverse user-generated content across various top- ics. The data undergoes internal company review to ensure compliance and integrity.
We collect user notes, which is the fundamental sharing unit in the platform including the main text, image descrip- tions, tags, and comments, covering topics such as fashion, travel, food, and lifestyle. During the interaction, other users may leave comments, and the note-owner will cate- gorize the notes with different tags. For evaluating infor- mation retrieval and matching, we gather query data and corresponding search results spanning product searches and topic-based queries. User interaction data, including likes, comments, and saves, is compiled to assess personalized rec- ommendation tasks. Additionally, sentiment labels (positive, negative, neutral) and topic tags (e.g., gender discussions, social issues) are included for sentiment and intent analysis, ensuring comprehensive and representative dataset.
Personalized Recommendation. Models should deliver tailored content recommendations based on user interests, behavior, and past interactions. For instance, the model should suggest relevant content aligned with user pref- erences. Tasks: (7) Note-Hashtag, Selects appropriate hashtags from given list based on content, in single- and multiple-choice formats. (8) Note-QueryGent Generates more effective search queries based on note content.
3.2. Building
In this section, following the previous benchmark construc- tions (Chen et al., 2024; Xu et al., 2023; Zhong et al., 2023), we outline the five steps used to build SNS-BENCE and present detailed distribution analysis.
To ensure data diversity and representativeness, we sample across the following standard:
3.2.1. 1. DATA CONSTRUCTION
(1) Topic Diversity, Notes are collected from a wide range of categories, including fashion, beauty, health, travel, and
The construction of the SNS-BENCH comprises five steps in Figure 2: (1) Data Collection. (2) Processing. (3) Anno-
'The recent popular REDnote platform.
3

---

Benchmarking Large Language Models in Social Networking Services
search...
Follow
Collection
1 step:
2 step:
3/ stee:
De-identiiication
De- re-sensitizaaiin
Task-gnesttie Process
SNS
Note
Data Processing
4th sstep: Data Annotation
SNS SNS-Bench
Q Question
sth.ttep: Expert RevioN
stt.ptep Quality Control
Instruction
Expt
LLR
Data
Curation Pipline
construct SNS-Bench using five-step procedure, culminating in neticulous expert review for final construction. ALLM Benchwork #SNS #Curation Steps
100
101
110
100
Figure 2: Pipeline for data curation in SNS-BENCH. Mainly five steps: (1) Data Collection. (2) Processing, (3) Annotation.
(4) Quality Control. (5) Expert Review.
food, to ensure comprehensive topic coverage.
ployed (Hendrycks et al., 2021; Taylor et al., 2022).
(2) User Diversity. Data includes users different ages, genders, locations, and interests, capturing diverse user needs and perspectives.
Data undergoes several filtering steps to ensure quality. (1) politically sensitive, inappropriate, offensive content removed. (2) user identifiers and personal information deleted for privacy. (3) irrelevant or low-quality entries, such as those with severe spelling errors, poor grammar, advertisements, or spam, are excluded.
(3) Time Span. To reflect the dynamic nature of social networks, we focus on real-time content, such as newly published notes and recent comments. Additionally, we include data from 2022-2025 to evaluate the performance over time.
Certain tasks necessitate additional measures. For the Note- Taxonomy task, labels are standardized to eliminate dupli- cates and ensure consistency. For the Note-Gender task, content with extreme overly emotional sentiments is fil- tered out. These preprocessing strategies ensure the data is high-quality and tailored to the needs of specific tasks.
To meet the specific requirements of each task, collect diverse and targeted data. For Note-Taxonomy, notes with varied topic tags are used to evaluate classification capa- bilities, while Core-QueryCorr relies on user search logs with annotated queries to assess query-content matching precision. Note-Hashtag utilizes notes with multiple tags to test tag relevance selection, and Note-MRC includes notes of varying complexity measure reading comprehension adaptability. Note-NER focuses on entity-rich notes for en- tity extraction accuracy, and Note-Gender are from gender- related discussions. Note- Pote-QueryGent involves optimizing search terms based on note content, and Note-CHLW con- tains notes with extensive comments to identify key terms effectively. This diverse and carefully curated dataset en- effectively evaluate the performance of
Data Annotation In SNS-BENCH, each data point under- goes a rigorous and systematic annotation process to ensure high-quality labels. 20 annotations are conducted manually to guarantee precision and consistency. For multiple-choice questions, incorrect options are derived from similar but in- correct answers within the notes. Specifically, for the Note- NER task, we train proprietary Named Entity Recognition (NER) model (Wu et al., 2024) to automatically annotate entities, followed by a human review validate and re- fine the annotations, ensuring accuracy and reliability. In cases of disagreement, majority-vot principle is applied. Similarly, for the Note-CHLW task, we train FastText (Bo- janowski et al., 2017) classification model to preliminarily identify potential highlight words, which are then manually reviewed to ensure their relevance and accuracy.
sures that LLMs in variety of tasks across SNS.
Preprocessing The collected data consists of various for- mats, primarily text and images. Images are converted to text using Optical Character Recognition (OCR) tools (Li et al., 2023a), if the image is not text-intenssie. we will remove For challenging cases, manual parsing em-
More details about our crowdsourcing are in Appendix

---

Benchmarking Large Language Models in Social Networking Services
Quality Control To ensure stringent data quality stan- dards, we adopt dual-validation approach that combines GPT-4 (OpenAI, 2023) scoring with meticulous manual val- idation. This process ensures the integrity and reliability of the data while enhancing its overall quality. Using GPT-4 for scoring, we design tailored prompts specific to our dataset, strategically crafted to enable GPT-4 to evaluate and rate the data based on predefined quality criteria. This auto- mated scoring mechanism efficiently identifies and filters out low-quality data instances, while also flagging potential issues and areas for improvement. we adopt FairEval (Wang et al., 2023) to eliminate positional bias. For details on the prompts, please refer to Appendix A.I.
comprehensive coverage.
Verb-Noun Pairs (Figure 4). This figure illustrates com- mon verb-noun pairs in social media interactions. Core verbs like take, have, and pay form key collocations (take photo, have time, pay attention), while peripheral verbs (solve, share, watch) add diversity. Nouns related to time (time, today), interaction (message, comment), and actions (use, fry) enhance contextual depth.
Length Distribution (Figure 5). Instructions are mostly under 200 words, peaking at 0-25 words ( (1,600 samples), Few exceed 1,200, reflecting detailed cases. Responses are shorter, typically 10-50 words, rarely surpassing 300, aligning with social media's concise communication style.
Human Verification Simultaneously, the dataset under- goes rigorous manual validation. A team of expert review- ers (20) conducts an in-depth assessment of each data en- try. This process involves cross-validation, where each data point is independently reviewed by at least three different reviewers. Their evaluations focus on content accuracy, coherence, and adherence to domain-specific knowledge. In cases of disagreement, a majority-vote principle is ap- plied, with the final decision reflecting the consensus of the reviewers.
4. Experiment
4.1. Evaluation Settings
We conduct experiments on source and closed-source LLMs. Uniform prompts are ap- plied across all LLMs in Appendix A.2. Experiments are conducted on 128 NVIDIA H800 GPUs with the OpenCom- pass codebase (Contributors, 2023).
using nearly open-
Evaluated Models Evaluated models include open-source models and closed-source models.
3.2.2. STATISTICS
Table I: Statistics of SNS-BENCI.
Meta Llama 3 Series (Touvron et al., 2023): This includes Llama-3,2-3B. Llama-3.2-33 3-3---.------nttuct,, Llama- 3. I-B, Llama-3. 1-8--Istrucct and Llama-3.3-70B. Instruct.
Statisties
Value
Total Questions
6,658
Note-Taxonomy (Single)
1,205
Note- MiteTaxonomy (Multiple)
800
Qwen2.5 Series (Yang et al., 2024a): This includes Owen-2.5-1.5B, Qwen-2.5-1.5B IIsstuct, Qwen-2.5- 7B, -rven-2.--B--Instrrie Qwen-2.5-32B, Qwen-2.5- 32B-Instruct, Qwen-2.5-72B, Qwen2.5-72B-Innnt
Pnte-QueryCorr (Explain)
144
Note-Query nre-QueryCorr (Topic)
800
Note-Hashtag (Single)
800
Note-Hashtag (Multiple)
795
Note-MRC (Complex)
227
Other Open-source Models: Phi-4 (Abdin al., 2024), Internim (Cai et al., 2024), GLM-4 (Zeng et al., 2024a), Yi-1.5 (Young et al., 2024) DeepSeek- v3 (DeepSeck-Al et al., 2024)
Note-MRC (Simple)
800
Note-NER
517
Note-Gender
193
Note-QueryGen
80
Hote-CHLW
297
Closed-source Models: GPT4 (OpenAI, 2023), Gem- ini (Anil et al., 2023), Claude-3.5 (Anthropic, 2024).
Avg. instruction length (in words)
793
Avg. answer length (in words)
37
Total Input Tokens
4,553,773
4.2. Evaluation Protocol
Total Output Tokens
1,017,477
We choose different metrics for tasks in
Word Cloud (Figure 3). This showcases diverse topics, from consumer themes (Food, Fashion, Beauty) to lifestyle interests (Home, Fitness, Music). Key terms like Topic, Life, Education, Travel highlight broad discussions, while interaction-related words (Comment, Query, Recommen- dation) reflect engagement patterns. The mix of practical (Business, Study) and leisure (Art, Travel) topics ensures
For classification tasks including Note-Taxonomy (Sin- gle), Note-QueryCore (Explain), Note-Gender, and Note- Hashtag. We calculate the accuracy of the final classifica- tion. Thus, the formula is:
1 N
Accuracy =
Σ I(ūs = yz).
(1)
N
1=1
5

---

Benchmarking Large Language Models in Social Networking Services
We1
levelevers yone yone g
Cat We1
Nerratioe year
u
Wen homen Lime
will service
plase
really
Sport suay
day
Educatiinn Dec Cutes
rim fir good girl
University
sui
Style three Los
first body take
Title
school
g
sone
and
Cont
@Exam
Tani
Material
store
Food
photo
Sumber
Heproigs
free Home tnness nake
small
fft Arr-engNusic ingg
18 Cpeople ma 1oppp
Trave need rice Pet
word
love
deve
every
exper
lohg
Daily
Sencell
ist
Co
two
W/ Art
Query
faceuu
Today worksee
Lness
skin
Figure 3: The word cloud of SNS- BENCH.
Length distribution of instruc- tions and responses in ENS-BENCH.
Figure 5:
Figure 4: Top 50 Verb-Noun structures in SNS-BENCH instructions.
where N is the number of samples, di is the predicted label, is the true label, and I is an indicator function that is 1 when the predicted label matches the true label, and O otherwise.
For Note-QueryGen, we compute the average of ANLS (Normalized Levenshtein Distance) (Marino et al., 2019) and semantic similarity scores. Semantic similarity com- puted using Sentence-BEs (Reimers & Gurevych, 2019). The similarity is then determined by the cosine similarity between these embeddings. The overall score formula is follows:
For the multi-label task Note-NER, compute the F1 score:
(Multiple) and
F1 2 Precision Recall
Precision Recall
ANLS + SS 2 (sf
Score
(7)
| ny
v
Precision
Recall w
(3)
ANLS max s/. ref))
(8)
Here, 9 represents the predicted label set, denotes the true label set, and indicates the size of a set.
SS = cos (s))) E(ref)),
(9)
For Note-QueryCore (Topic) task, we calculate the macro- F1 for all categories, which defined:
where SS Semantic Similarity. D(s, ref) representing the Levenshtein distance between the generated string s and the reference ref, and |-| denoting the length of the string. where E()) represents the Sentence-BERG embedding function, and cos(:,) denotes the cosine similarity between the two embeddings.
Macro-Fl
Σ Fl,
(4)
C
i=1
where C'is is the number of categories, and F1, is the Fl score for category i.
For the Note-CHLW task, we first calculate the success ratio SR based on the predicted label list. Then, we compute the score between the predicted and true labels. The final score given by:
For Note-MRC (Complex) task, we first gain the success ratio after parsing inference results. Then, we compute Fl score and extract match (EM), score between the predicted and true answers. The final score is calculated as:
Score = x SR.
(10)
Fl + EM
Score
x SR,
(5)
where SR success ratio. This measures whether the LLMs successfully follow the format instruction.
4.3. Main Results
4.3.1. 1. OVERALL EVALUATION
For Note-MRC (Simple), we first calculate the success ratio SR. Then, we gain the average score of the FI, BLEU and
The benchmark results highlight key insights into LLM performance on SNS-related tasks.
ROUGE scores. The final score is defined as:
Closed-Source Models Excel. Proprietary models consis- tently outperform open-source counterparts, benefit from
(F1 BLEU ROUGE) SR,
Score
(6)
6

---

Benchmarking Large Language Models in Social Networking Services
superior training resources and optimization techniques. Claude-3.5-oonne leads with average score of 61.98% closely follows GPT-44-2024-005-1 (61.50%).
Instruction Following Analysis We analyze instruction following across tasks assess capability of LLMs in han- dling structured requirements in diverse social media con- texts. Figure 6 shows the success ratio of instruction fol- lowing across Note-MRC (Complex), Note-MRC (Simple), Vote-CHLW and Note- Shote-uueryCor.. Larger models generally perform better. Task difficulty varies, with complex read- ing comprehension being the most challenging, Notably, large models still struggle with structured output (GLM-4), particularly list generation (I-), performing significantly below peers and highlighting the impact of strict formatting requirements.
Task Complexity Varies. Tasks like Note-Gender and Note-Hashtag show strong results across models, suggest- ing alignment with current LLM capabilities. In contrast, Note-MRC (Complex) and Note-Query QueerueerGenn pose significant challenges, with even top models struggling.
Scale Enhances Performance. Among open-source mod- els, larger architectures like Qwen=2.5-72B-Instrr achieve the best average score (61.27%), underscoring the impor- tance of model size in tackling diverse SNS tasks, as shown Table 2.
In-depth Analysis in Note-MRC (Complex) To evaluate the complex reasoning capability of large language models on social media content, we analyze their evidence selection and justification performance in the Note-MRC (Complex) task. Instruction is in Appendix A.2. Figure tracks model performance on the complex reading comprehension task. Open-source model Deepseek- V3 leads with an F1 score approaching 40%, followed closely by closed-source models like GPT-4o and Gemini series.
4.3.2. TASK-SPECIRE EVALUATION
Our evaluation across eight tasks demonstrates varied model performance patterns. We can conclude: (1) Imbalance be- tween tasks. Different models excel in different tasks, high- lighting a lack of uniform strength across all SNS-related challenges. Phi-4-148 14i-4-14Baa and Deepseek- V3 achieve strong performance in structured tasks such as Note-QueryCorr, while Claude-3.5-Sona leads in tasks requiring contextual understanding like Note-Gender and Note-CHLWW Llama- 33-70B-Instruct show noticeable trade-offs, performing exceptionally well in retrieval and taxonomy task. (2) Task Complexity Variability. Note-MRC (Complex) and Note-QueryCe remain among the most challenging tasks, as models struggle with deeper comprehension and multi-step reasoning. Note- Taxonomy "Single-Choice and Note-Hashtag (Single-Choice) exhibit higher scores across most models, indicating these tasks are more approach- able for L.MSs. (3) Performance Stability. Deepseek- V3 and Phi-4-14B demonstrate consistency across related tasks, which maintain balanced performance across Note- Taxonomy, Note Mote-Hash and Note-QueryCont GLM-4- 9B-Chat shows fluctuating performance, excelling in Note- Taxonomy but underperforming in Note-MRC, suggesting that retrieval-hees -havv tasks require additional optimization.
Bedides, we observe that different models exhibit distinct preferences when initially assessing relevance in Table 3. GeT-series models tend to be more cautious in their judg- ments, whereas Qwen-serics models are comparatively less conservative in determining relevance.
Analysis of Different Metrics in Note-QueryGer Since query generation demands lexical precision and semantic un- derstanding, analyzing these metrics reveals capabilities of models in query comprehension. Figure 8 compares model performance across ANLS, Semantic Similarity, and their average. Semantic Similarity shows little variance between models, limiting its ability to differentiate performance. To address this, we average ANLS and Semantic Similarity, balancing lexical precision and semantic understanding.
5.2. Visualization
In this part, we are curious about the model preference in choosing options.
5. Analysis
In this section, we first give deeper analysis in three as- pects: Basic mnstruction-ollooinng Complex Reasoning (Nofe-MRC), and Generation (Note-QueryGen), Then provide the visualization on Note Note-QueryGen (Topic) to understand the model preference.
5.1. Error Analysis
Figure 9: Confusion matrices for Note-Querye (Topic) performance of Claude-3.5-Someee Deepseek- V3, GPT-4o, and Qwen-2.5-32B. IInstrcct..
We are curious about the root of errors, and examine the
basic abilities at higher level. Thus we dive into ba- sic instruction-followinn complex Reasoning (Note-MRC)),
and generation (Note- QueryGen).
Figure 9 presents confusion matrices for Note- -ute-QueryGen
7

---

Benchmarking Large Language Models in Social Networking Services
Table 2: Results of different models on the SNS-BENCH. We utilize green (1st), blue (2nd), and yellow (3rd) backgrounds to distinguish the top three results within both opence and slorc- models.
Moddit
Note Tase-aay
NoteHaoking
Sate-QurrCCer
NRc-BRC
Nate
Note-Genkr
NoteCHUR
Nate-parrlenn
Sigle-Chsiee
Meltipe-Hep
Segle Cleice Mhtigle-Chsie
Pxpleis
Togle
Comple
Soople
Oyen-iarer Large Largaage Moast (7.58
23.48
33:30
21.83
Qyee Seavee Eorge
Tangsige Mi-AN (78+)
12.92
37.01
32.37
Ioamltm2..5-16 Libar
49330
15.30
711.3
57.04
22.34
-11.20
19:55
08.33
20774
23/93
06..5
28.44
44.33
43.33
Laam--3 lusttct
43.73
33775
37.34
18.30
06.35
66.03
66.53
24.30
35..1
13.70
04.34
33.27
07:00
79.58
26.55
38.90
04.53
(etcniiee2 5.30B. Clut
$3.65
40.25
05.00
17:88
09.54
75.71
57.64
63.27
5.25
55.85
30.33
75.03
00.90
76.34
64.00
46.35
Qwea 7BB-aautt
19:75
54.17
45.41
57.52
10.38
Syart Lany
M4deer (528+)
2-11
12:50
28.33
18.38
76-74
43:00
31.97
41:31
26.25
Y64.5.34..0
$4.89
13.58
75.34
58.33
33.50
56.55
23.54
Lare 3-573.3--eeee Iestret
69.75
65.12
02.34
13.02
92.93
63.21
63.99
37.53
50.76
35.40
19.36
31.38
56.09
91.19
33.55
46.41
56.45
Owen 330 Awarac
56.68
63.12
90
11.12
76.50
50.51
59:03
05.30
35.97
71.11
$5.04
54.51
90.53
58.64
15.65
58.99
Ownn 5778 Ivaraa
40.00
56.75
63.63
16.25
9110
85-13
63.00
0.53
50.41
35:01
71.43
51.97
51.17
90.33
03.34
66.37
61.23
Dapoock-V 93
62.16
71.38
07.23
$7:25
$3.99
$6.39
50.23
36.39
47.71
48.07
75.26
6007
56.00
90.35
40.45
46.03
01.90
Searr Largy Lasuangg Madel (APG
GP 401----i--000---000
55.44
$6.30
35.30
13.12
$1.54
62.41
65:25
+2.51
53.35
41.36
06.32
54.95
55.22
30.31
52.54
42.84
$6.56
09:11
70.90
65.98
11.50
$4.11
56.11
66.53
76.08
52.13
38.30
72.41
55.81
55.00
06.53
74.20
01.01
59.33
72.25
67.30
18:62
M6.00
63.09
18:35
41.08
50.83
55:33
194.4
25.15
06.94
99.48
55.36
Note-MRC (Compled)
NoteMMt (imeae)
Aot-CEM QuerCer
Figure 7: Fl score in Note-MRC (Com- plex).
Figure 8: Comparison of ANLS, Seman- tic Similarity, and average scores on the Note-QueryGen.
Figure 6: Success ratio (SR) of LLMs in instruction following ability.
Table 3: Model Preferences in Note-MRC (Complex).
3). However, models commonly struggle with moderate relevance levels (scores and 2), suggesting challenges in capturing nuanced relationships between queries and con- tent. More visualization results are in Appendix
Note-MRC (Complex)
Models
Precision
Recall
F1
Dwwn-2.5-72B-Insttee
34.74
94.29
50.77
GPT-4o-2024-11-22
36.69
88.57
51.88
6. Conclusion
Deepseek-V3
40.29
80.00
53.59
We present SNS-BENCUI, a comprehensive benchmark for evaluating LLMs in Social Networking Services (SNS). Unlike prior work on isolated tasks, SNS-BENCH offers structured assessment across eight diverse SNS-related tasks, covering 6,658 questions in multiple formats cap- ture the complexity of social interactions. We evaluate over 25 LLMs, findings highlight strengths, limitations, and areas for improvement in SNS- related tasks.
across four leading models: Claude-3.5-Sonnee Deepseek- V3, GPT-4o, and Qwenn225-33B-IIsttrue The matrices reveal distinct prediction patterns for query-note relevance scores from -1 to 3. High concentration along the diagonal indicates generally accurate predictions, particularly for ex- treme relevance scores (3 and -1). All models show strong performance in identifying highly relevant matches (score
8

---

offers comprehensive Benchmarking Large Language Models in Social Networking Services
Impact Statement
We introduce SNS-BENCII, a benchmark for evaluating language models' social networking capabilities. While Al integration in social platforms offers benefits, it also raises concerns about privacy, authenticity, and social dynamics. SNS-BENCH provides transparent assessment metrics to better understand Al's impact on human communication.
By systematically evaluating social understanding, SNS- BENCH helps identify potential risks and informs safe- guards for responsible Al deployment. We take precautions protect user privacy and remove sensitive information during dataset curation.
Ultimately, our goal is to foster Al development that en- hances, rather than undermines, authentic social interactions. SNS-BENCH is designed to improve Al safety and relia- bility social networking while preserving user trust and community well-being.
Limitations
Evaluation. While
assessment social networking tasks, it has certain lim- itations. The benchmark primarily focuses on text-based interactions, potentially overlooking multimodal aspects. Additionally, our metries may not fully capture the cultural and contextual nuances that shape social media communica- tion.
Empirical. A key limitation is the dataset's ' geographic and linguistic scope, which predominantly English-bars based and may not fully represent global social networking behaviors. Furthermore, the benchmark focuses on contemporary so- cial media formats, which may not account for emerging trends.
Theoretical. Our evaluation emphasizes practical metrics but lacks theoretical guarantees on their ability to capture the full complexity of human interactions. As social media evolves, some tasks may become less relevant, and the link between benchmark performance and real- world effective- requires further study.

---

Benchmarking Large Language Models in Social Networking Services
References
Cai, Z.., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z.,, Chen, Z.,, Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q.. He, C., Hu, Y., Huang, T., Jiang. T., Jiao, P., Jin, Z., Lei, 7.,, Li, J., Li, J., Li, I.., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong. J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu. J., Qu, Y., Shang, F., Shao, Y., Song, D., Song, Z., Sui, Z., Sun, P., Sun, Y., Tang. H., Wang, B., Wang, G., Wanga J., Wang. J.. Wang, R., Wang. Y., Wang, 7.,, Wei, X., Weng, Q. Wu, F., Xiong, Y., Zhao, X., and et al. Internlm2 technical report, CoRR, abs/2400..1297, 2024. doi: 10.48850/ARRXI12245...
Abdin, M. I., Aneja, J., Behl, H. S., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P.. Lee. J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang. X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang. Y. Phi-4 technical report. CoRR, abs/2412.08905 2024. doi: 10.48550/ ARXIV.2412.08905. URL https: http:://oo..ogg/// 18550/arXiv.2412. 089200
Alhamed, F., Ive, J., and Specia, L. Using large lan- guage models (LLMs) to extract evidence from pre- annotated social media data. In Yates, A., Desmet, B., Prud hommeaux, E., Zirikly, A., Bedrick, S., MacAvaney, S., Bar, K., Ireland, M., and Ophir, Y. (eds.), Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pp. 232-237, St. Julians, Malta, March 2024a. Association for Computa- tional Linguistics. URL htwps::pp////.. http:///aaaaattttaaaa// org/ 2024. wwwssch-s..... 1.22/.
URL https: /tt.:/1..88555 48050/8550//20 2403....22... 172977
Chen, Z., Wu, J., Zhou, J., Wen, B., Bi, G., Jiang, G., Cao, Y., Hu, M., Lai, Y., Xiong. Z., and Huang, M. Tombench: Benchmarking theory of mind in large language models, In Ku, L., Martins, A., and Srikumar, V. (eds.). Pro- ceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACI. 2024, Bangkok, Thailand, August / 1-16, 2024, pp- 15959-15983, Association for Computational Linguistics,
Alhamed, F., Ive, J., and Specia, L.. Using large language models (Ims) to extract evidence from pre-annotated social media data. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pp. 232 2024b.
2024.
Contributors, O. Opencompass: A universal evaluation platform for foundation models. https: /ts::///gt ///ittttt../////// com/ /ttp//////pppppcpaaassssss 2023.
apencompass.
Anil, R., Borgeaud, S., Wu, Y., Alayrac, J., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lilli- crap. T. P., Lazaridou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y. Doherty, R., Collins, E., Meyer, C., Ruther- ford, E., Moreira, E., Ayoub, K., Goel, M., Tucker, G., Piqueras, E., Krikun, M., Barr, I., Savinov, N., Danihelka, I., Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M., Syg- nowski, J., and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/23 11005, 2023.
DeepSeek-AL Deepseck-v2: A strong, economical, and efficient mixture-of-expprts language model, 2024.
DeepSeek-AI, Liu, A., Feng. B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F, Dai, F., Luo, F,, Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L.., Liang, J., Guo, J. Ni, J., Li, J., Wang. J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L.,, Wang, L., Zhang, L.,, Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang. Q. Zhu, Q., Chen, Q., Du, Q. Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R. Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., and Zeng W. Deepseek-v3 technical report. CoRR, abs/2412. 194377 2024. doi: 10.48550/ARXIV.244111 2412. 1943..
Anthropic. Claude 3.5 sonnet. Anthropic News,
June 2024. URL https: http://ww.tttttrreteeee anthropic. com/ 3ale::llleee--------
Bandura, A. Social learning theory. Prentice-Hall
URL https: /
htapss://books../ /ttp///oos///////////..o.o.oso.. www.loooooooooooooo hww..opp...o..ooo.oooooooooooooo books? id=mipbjgEACcAU.
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. En- riching word vectors with subword information. Trans. Assoc. Comput. Linguistics, 5:135-148 146,1 2017. doi: 10. 1162/TACL 1.005 URL attps:/ //oiiooorrggoooooo/ 10.1162/tacc 00051.
URL https:
/ //doi. 10.
He, L., Omranian, S., McRoy, S., and Zheng, K. Using large language models for sentiment analysis of health-related
10

---

Benchmarking Large Language Models in Social Networking Services
social media data: empirical evaluation and practical tips. medRxiv, pp- 2024-03, 2024.
Qi, The impact of large language models on social media communication. In Proceedings of the 2024 7th International Conference on Software Engineering and Information Management, ICSIM '24, pp. 165-170, New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400709197. doi: 10.1145/ 3647722.3647749. URL https: //doi. org/10. 1145/3647722.364
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the MATH dataset. In Van- schoren, J. and Yeung, S. (eds.), Proceedings of the Neu- ral Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong. X., Tang, X., Qian, B., Zhao, S., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. Facilitating large language models to master 16000+ real-world apis, 2023.
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang. Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., and Sun, L. Metatool benchmark for large language models: Deciding whether to use tools and which to use. In The Tweifth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https::/ ptttps://ppenreviieee htwp:/rvviie..n//oooooeeeeo hwt.:/rrnttrrr id RO-Roggggag...
Toolllm:
Reimers, N. and Gurevych, I. Sentence-bert. Sentence embeddings using siamese bert-networks. In Inui, K., Jiang, Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3980-3990. Association for Computational Linguis- tics, 2019. doi: 10.18653/V1/D19- 1410. URL https: / //doi. 10. 1 (1188633vv11/ 9-4100.
Jin, Y., Choi, M., Verma, G., Wang, J., and Kumar, S. MM- SOC: Benchmarking multimodal large language mod- els in social media platforms. In Ku, L..W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 6192- 6210, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/0244 findings-acl.370. URL https: /tc///acllc /awaaaaaaanoooyy.... org/202/ 4. ....4...ii findings-acl. 370/
Tang, Q.. Deng. Z., Lin, H., Han, X., Liang, Q., and Sun, L. Toolalpaca: Generalized tool learning for lan- guage models with 3000 simulated cases. arXiv preprint arXiv:2306.0530// 2023.
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z.., and Wei, F. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAA/ Conference on Artificial Intelli- gence, volume 37, pp. 13094-13102, 2023a.
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E.,, Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. 2022.
Törnberg, P. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, pp. 08944393241286477 2024.
Li, M., Song, F. Yu, B., Yu, H., Li, Z., Huang, F., and Li, Y. Api-bank: A benchmark for tool-augmented lims, 2023b.
Liu, Q.. Tao, X., Wu, J., Wu, S., and Wang, L. Can large language models detect rumors on social media? arXiv preprint arXiv:2402.03916 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.00220 2023.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. OK-VQA: A visual question answering benchmark re- quiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3195-3204. Computer Vision Foundation/ IEEE, 2019.
Wang, P., Li, L.,, Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui. Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023.
OpenAI. Gpt-4 technical report. PREPRINT, 2023.
Wu, C., Ke, W., Wang, P., Luo, Z., Li, G., and Chen, W. Consistner: Towards instructive ner demonstrations for lims with the consistency of ontology and context. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 19244,19222, 2024.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv: 2305. 15334, 2023.

---

Benchmarking Large Language Models in Social Networking Services
Xu, Q. Hong, F., Li, B., Hu, C., Chen, Z., and Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv: 2305.16504, 2023.
Zhou, H., Hu, C., Yuan, Y., Cui, Y., Jin, Y., Chen, C., Wu, H., Yuan, D., Jiang, I.., Wu, D., et al. Large language model (Ilm) for telccommunications: A comprehensive survey on principles, key techniques, and opportunities. arXiv preprint arXiv:2405. 10825, 2024.
Yang, A.. Yang, B., Hui. B., Zheng, B.. Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024a.
Zhuang. Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. Toolqa: A dataset for Ilm question answering with exter- nal tools. arXiv preprint arXiv:2306. 13304, 2023.
Yang, D., Ziems, C., Held, W., Shaikh, O., Bernstein, S., and Mitchell, C. Social skill training with large lan- guage models. CoRR, abs/2404.04204, 2024b.
Young, A., Chen, B., Li, C., Huang, C.,, Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q. Yue, S.. Yang, S., Yang. S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by 01.at ai. CoRR, 22.22443344555 2024.
Zeng, A., Xu, B., Wang, B., Zhang, C.,, Yin, D., Rojas, D., Feng. G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng. J., Gui, J., Tang, J., Zhang, J., Li, J.,
Zhao, L., Wu, L., Zhong, L., Liu, M., Huang, M., Zhang, P., Zheng, Q.. Lu, R., Duan, S., Zhang, S., Cao, S., Yang, S., Tam, L., Zhao, W., Liu, X., Xia, X., Zhang. X., Gu,
X., Lv, X., Liu, X., Liu, X., Yang. X., Song, X., Zhang,
X., An, Y., Xu, Y. Niu, Y., Yang. Y., Li, Y., Bai, Y., Dong,
Y. Qi, Z., Wang, Z., Yang, Z., Du, Z., Hou, Z., and Wang,
Z. Chatglm: A family of large language models from GLM-1300 to GLM-4 all tools. CoRR, abs/2406.12799
2024a. doi: 10.48550/ARRXI..2406..1 12793. URL https:
//doi . org. 48540 arxiv. 2406..2793.
Zeng, J., Huang, R., Malik, W., Yin, L., Babic, B., Shacham, D., Yan, X., Yang. J., and He, Q. Large language mod- els for social networks: Applications, challenges, and solutions. arXiv preprint arXiv:240..02575 2024b.
Zeng, J., Huang, R., Malik, W., Yin, L., Babic, B., Shacham, D., Yan, X., Yang, J., and He, Q. Large language mod- els for social networks: Applications, challenges, and solutions. arXiv preprint arXiv:2401.02575 2024c.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang. X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C. Chen, Y.. Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J., and rong Wen, J. A survey of large language models. ARXIVORG, 2023. doi: 10.
18550/arXiv.2303.11
Zhong, W., Cui, R., Guo, Y., Liang. Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation mod- els, 2023.
12

---

Benchmarking Large Language Models in Social Networking Services
Appendices
Within this supplementary material, we elaborate on the following aspects:
Appendix A: Prompt Details
Appendix B: Crowdsourcing Details
Appendix C: Case Details
Appendix D: More Evaluation Details
Appendix E: Hard Case Details
Appendix F: Confusion Matrix Details
13

---

Benchmarking Large Language Models in Social Networking Services
A. Prompt Template
A.I. Prompt for Quality Control
Prompt Template used for Quality Control
You
data grader. You will grade the data provide according to my requirements, explain the reasons, and then give piece of higher-quality data based on this piece of data.
Please help me rate the following dialogue data and explain the reasons. Require:
1. Scoring perspective: whether the problem belongs to the field of social network services; whether the description
clear; whether the answer is accurate; whether the language is coherent;
Point scale: 5-point scale, point: very poor; points: slightly poor; points: barely qualified; 4 points: usable; excellent. Please rate the answer.
points:
3. Format: You can only return parsable json format data, other content, For example: "score": 4, "reason": "".
Among them, score represents the score for this question, reason represents the reason for the score, and states the advantages and disadvantages of the data.
4. All reasons are written in reason.
14

---

Benchmarking Large Language Models in Social Networking Services
A.2. Prompt Templates for Instructions
Prompt Template used for Note- Note-Taxonomy SSingle-Choice
Given SNS note's title and content, and list of candidate categories, please select the most suitable category from the candidate category list. Here are some examples:
SYSTEM:
Input
Note title: Need help, is this car dash cam
Note content: Bought DV for 250, came with battery and charging cable, and it's very light, wondering if it's really dash cam[crying R][crying R]#DV[topic]# Candidate Categories
Car knowledge, Trending, Car lifestyle, Car accessories, Motorcycles, Car shopping, Car modification, New energy & smart, Car culture, Driving test learning, Other automotive
Answer Car accessories
Inpul
Note content:
Traveling Henan Understanding China Xinxiang South Taihang Tourism Resort: Let's follow the lens to see the sea of clouds in South Taihang TTTaveinngeeeaattneeesssnnnnn #HoeelndHHenan #Home- landHenaanYe.MMediMMaaixx #May 1hhinaaTooiimmDD #XinxiangSouthTaiaan TourismResort sonrnossso #ScaOfClouds
Candidate Categories
Travel VLOG, Shopping, Food exploration, Places go, Travel guide, Travel records, Hotels & B&Bs, Attraction experience, Travel tips, Living abroad, Indoor leisure
Answer Travel records
Please output the answer (in the same format as the examples above) DIRECTLY after 'Answer:', WITHOUT any additional content.
HUMAN:
Input:
{content)
Candidate Categories:
{candidates)
Answer:
15

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-Taxooomy (MultipleHopp
SYSTEM:
Given a SNS note's title and content, and lists of candidate primary, secondary, and tertiary categories, please select the most suitable categories from each level.
Answer in the format: Primary category Secondary category Tertiary category.
Here some examples:
Inpul:
Note titl: The super disappoining AHC facial cleanser for me
Note content: During National Day trip to Hong Kong, several colleagues asked me to help buy AHC facial cleanser, saying was especially good. followed the trend and bought one, used it in the hotel that night x. After washing, my eyes were particularly painful, even a tiny bit made me feel like was going blind good @.. Now full bottle is sitting at home, won't repurchase. Bought for about 70 HKD equivalent about 60 RMB. The Innisfree green tea cleanser is still better for an affordable cleanser
so where exactly is
Candidate Primary Categories:
("Music', 'Beauty.. &Mother& Baby', 'Outdoor', 'Humanities', 'Photography', 'Gaming', 'Art', "Trends', 'Enter- tainment', Fiim&&V 'Kids', 'Career', ]Fodd
Candidate Secondary Categories:
['Education Daily', 'Skincare', "Other Pets', 'Handicraft', "Resource Sharing', 'Exhibition', 'Personal Care', 'Shoes', 'Mobile Games', 'Relationship Knowledge', 'Meals', 'Places', 'Language Education', 'Makeup'] Candidate Tertiary Categories:
[' I'Cycling Records', Playlist Sharing', 'Cleansing', 'Other Shoes', 'Fruit', 'Selffstudy',, 'Instrument Playing', 'JK',
"Beverage Review", "Tablet', 'Career Development', 'Boutique', 'Other Campus', 'Driving Safety"]
Answer: Beauty Skincare Cleansing
Input
Note title: Day8
Note content: Forgot post yesterday's (nnterday\nMMakkee nakiing up for it hobgyyCchhhesssnssiiiliighhhhhhh [topic] DDailyCaligraphyyhhckn [opic]
Candidate Primary Categories;
A'Aimee, 'Beauty,, 'Gaming'. 'Humanities', 'HomeeDecorationn 'Health', 'Relationships' 'Social Science',
CCrreer' Art', "Education', 'Life Records']
Candidate Secondary Categories:
('Science', 'Car Lifestyle', 'Running', 'Other', 'Culture', 'Weight Loss Medicine'. 'Parks', 'Fashion', 'Acces- sories', 'Weight Loss', 'Music Sharing', 'Finger Gaming']
Candidate Tertiary Categories
[' ('Bags', Fruit', Leisure Guide', 'Sheet Music Sharing', 'Font Design', 'Calligraphy', 'Weight Loss Tutorial', 'Snack Review', LLfe Science', 'Swimwear', 'Ball Sports', 'Skincare Collection')
Answer: Humanities Culture Calligraphy
Please output the answer (in the same format as the examples above) DIRECTLY after Answer:', WITHOUT any additional content.
HUMAN:
Input:
{content]
Candidate Primary Categories:
Srandidatespprimryy
Candidate Secondary Categories:
Laadiddate_secondarr
Candidate Tertiary Categories:
(candidates
(candidates_tertirr)) tertiary)
Answer:
16

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note- Has-taa (Single-Choice
Give you a SNS Note title and content, along with a list of candidate topic tags, please select the most relevant tag from the list for the note.
SYSTEM:
Below are some examples:
Input:
Note Title: Enfj Big Swords Will Love Life BGM Recommendations
Note Content: Compiled some BGMs for everyone, if you have any recommended songs, please leave them in the comments section, let's s exchange and share together
Candidate Topic Tags:
National Peace and Tranquility, Outfit for Pear-Shaped Body, High EQ Small Tricks, Vienna Guide
Answer High EQ Small Tricks
Inpul
Note Title: Must-Have # at the Sanya Beach
Note Content: On a summer beach, how can you do without this blue tie-dye dress! An \nSize XL, perfectly fits plus-size sisters! 2 An AnA high-end, laid-back style, loose design, wear it and instantly look whiter and slimmer! An AnThis dress is not only fashionable but also of great quality, must-have item for beach vacations! I \n AnEven the most picky of you will fall in love with it!
Candidate Topic Tags
National Trend Short Sleeves, Beach Outfit, Miracle Warm Pictures, Badminton Uniform Outfit
Answer: Beach Outfit
Please output the answer (in the same format as the examples above) DIRECTLY after 'nnwwer'' WITHOUT any additional content.
HUMAN:
Input:
{content]
Candidate Topic Tags:
(candidates)
Answer:
17

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note Hate-Has (Multppe-Chane Choice)
SYSTEM:
Given a title and content of a SNS Note, and a list of candidate topic tags, please select all the tags that match the note from the list of topic tags.
Here are some examples:
Input:
Note Title: Deposit Money in Wuhan
Note Content: Squat for a Wuhan customer manager \n Deposit for years!!!
Candidate Topic Tags:
September Birthday, DIY in Guangzhou, Savings Check-in, Wuhan, Henna Tattoo, Nap Time, Vancouver Mercedes, Must-Have Outdoor Gear, Bank, Liu Xiang, Savings, 5D Stereoscopic Mask, Sam's Club, Non-Cyan Earring Recommendations
Answer: Wuhan, Savings, Savings Check-in, Bank
Input:
Note Title: First Time Making Handmade Balls, Afraid of Sealing Issues!!
Note Content: Love handmade balls so much! Never thought making them myself would be so much fun! But the glue for sealing is really hard to handle. Use too little and it might not be sturdy, use too much and might burn [Cry R][Cry RJ. How long should wait before peeling off the film? @Gill interested check my page, switched account, will be sharing more textured balls \552eu (5 balls)
Candidate Topic Tags:
CPA Exam Review, After Being a Mom, Recruiting Agents, Manicure Wholesale Collaboration, First Time Making Handmade Balls, Korea Trip. Shanghai Yuanxing Huayu Real Estate Group Co., Ltd., Handmade Ball, Pinching Ball Fun, Popular Mask Recommendations, Resume Coaching, Handmade Ball FX, Handmade Ball Sharing, New Driving Style, Pinching Sharing, Basic Skills Competition for Class Teachers, Handmade Ball Original
Answer Pinching Sharing, Handmade Ball Sharing. Handmade Ball FX, Handmade Ball Original, First Time Making Handmade Balls, Handmade Ball, Pinching Ball Fun
Please output the answer (in the same format as the examples above) DIRECTLY after 'nnwwe:'' WITHOUT any additional content.
HUMAN:
Input:
{content)
Candidate Topic Tags:
{candidates)
Answer:
18

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-QueryCor (Explain)
Below a question and a document. Please determine if the given document can answer the given question. If it can, reply with "Yes", otherwise reply with "No". Only reply with "Yes" or "No", no additional explanations are needed. Here some examples:
SYSTEM:
Question
Is the distorted face in the back camera of an iPhone the real you? Document :
Revealed Is the you in the original camera really you? Distorted features? Facial asymmetry? Poor skin condition? Skewed face? Why do you see all these when you open the original camera on your phone? And then you get anxious: Am I really that ugly? Actually, it's the case. The original camera on your phone does indeed make you look worse. Let me explain one by one about appearance anxiety #hoteamerafriendly Answer No
Question:
Is the com used in cultural playthings real cor??
Document
Light Macaroon Beautiful, beautiful All real corn grown, nature's colorful gifts currently popular cultural plaything corn (naturally grown) undyed
Answer Yes
HUMAN:
Question:
{question
Document:
(document)
Answer:
19

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-QueryCor (Topic)
SYSTEM:
You are a relevance scoring robot. You will receive query and a SNS note content, and you will score the relevance between the note and the query. Before scoring. will first introduce you to the concepts of requirements and topics, and then provide you with rating levels to choose from.
First, will introduce you to the concepts of requirements and topics.
Requirements
In search engines, users express their search intent through queries. However, due factors such as knowledge level and cognition, the input queries may not accurately describe users' true needs. Therefore, search systems need accurately understand the real needs behind queries to retrieve the content users need. Generally, queries can be divided into "Precise Need Queries" and "General Need Queries" based on requirements.
Their definitions and differences are as follows:
Precise Need Queries: queries with clear and unique intentions.
e.g. how to treat knee pain, how to cook tomato and eggs
* General Need Queries: queries that may have multiple intentions, which can further divided into two categories according primary and secondary needs:
** Primary Needs: The most direct and basic expectations when search, covering common needs. Usually what users first think of and most want to get answers immediately.
Secondary Needs: Some additional needs around the primary needs, specific needs. Although these needs aren't common to all users, they are very important to some users.
Topics
Can be understood as core issues or involved entity objects, and notes' topics can be viewed from different angles, examples as follows:
Example 1:
618 must-haves !! Sharing coppiness-boostinn bathroom fixtures
Abstract perspective: Home bathroom essentials
Specific perspective: Introduction of bathroom cabinet, smart toilet, shower head
Example 2:
Five easy-to-raise dogs that don't smell
Abstract perspective: Introduction to casy-to-raise dogs
Specific perspective:
Introduction of 5 dogs: Schnauzer, Poodle, Bichon Frise, Shiba Inu, Pomeranian
Based on the matching degree between "query topic" and "note topic", they can be divided into categories:
& Category Complete topic match: The note's topic matches the query topie For example:
Query=home bathroom recommendations, Note=618 must-haves !! Sharing cappiness-boosting bathroom fixtures Query=which dogs are easy to raise, Note=Five easy-to-raise dogs that don't smell
Category 2: Partial topic match: Part of the note's topic matches the query topic For example:
Query=smart toilet, Note=618 m happiness- hoppiness-boosting bathroom fixtures Query=poodle, Note=Five easy-to-raise dogs that don't smell
Category 3: No topic match: The note's topic doesn't 't match the query topic at all For example:
Query=ceiling decoration recommendations, Note=618 must-haves!! happiness- -booiissss--oooiigg bathroom fixtures
20

---

Benchmarking Large Language Models in Social Networking Services
Query=are kangaroos easy to raise, Note=Five easy-to-raise dogs that don't 't smell
Available Rating Levels
$ 3: Meets primary needs, complete topic match (relevant content 2 80%)
Examples: Query=basketball Note=Who invented basketball?
2: Meets primary needs, partial topic match (relevant content between 10%-80%) Examples: Query=Schmauzerr vs Poodle comparison Note=Five casy-to-raise dogs that don't ion't smell [Meets secondary needs, complete topic match (relevant content 2 80%)]
Query=Apple Note=After becoming famous, Fan Bingbing's 's most regrettable movic "Apple"! (Meets secondary needs, partial topic match (relevant content between 10% -V%))
Query=Xiaomi su7 Note=Help! Should I choose su7 Mercedes c260
1: Low satisfaction level, relevant content less than 10% Examples:
Query=23-week glucose tolerance Note=Peking Union Medical College Hospital International Birth Record
Prenatal Care [Low satisfaction level, only mentioned]
Query=Can gold purchased Note=Today's gold price-May 18, 2024 <Up> [Low satisfaction level, extremely specific need]
Query=How play badminton Note=About sharing racket with my bestie to play badminton
[Low satisfaction
level, helpful for query with matching topic]
Query=Quick weight loss exercise Note=100 reps daily, anytime anywheredhblly fat thigh fat reduction effective
0: Doesn't 't meet needs, has some connection, keywords match Examples:
Query=Gemini wants to break up Note=How long does it take for Gemini forget ex [Doesn't 't meet needs, has some connection, keyword match] Query=Chaotiammen Hotpot Note=Haidilao please stop posting social media
-1: Doesn't t meet needs, no connection at all Examples:
QuazynAmazing girls Note=These three makeup schools are not recommended, ordinary girls can't afford
...
HUMAN:
The Query and note you received are: {content)
Please output the rating level, choosing from the five numerical levels: -1, 0, 2, Output only the numerical rating, without any additional content.
Answer:
21

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-MRC (Complex)
SYSTEM: Task: First rewrite the Query, then perform reading comprehension.
Reference Content Format
The reference content consists of three parts: Query, Doc, and Level
Query is the user's search term with question-asking intent; Doe is the main text part of the image note, wrapped in <Doc> and </Doc>; Level defines the expected answer type, indicating what type of answer the user needs.
Task Requirements #
Step Based on Query and Level, rewrite the Query into specific answer requirements, please output the rewritten rewrite_query. Here are some examples for the first step: [Query=World's most beautiful museums recommendations, Level=museum names] => [Names of the world's most beautiful museums]
2. [Query=Coody Coffee recommendations, Level=drink names] [Delicious drink names Coody Coffee]
3. QQueryQQinddao food, Level=dishes => [Delicious dishes in Qingdao]
4. (Query=Games similar to Number Bomb, Level=game names] ] => [What games similar to Number Bomb]
5. [Query=Birthday cake recommendations in Licheng District, Jinan, Level=bakery cake names] => [Delicious birthday cakes from bakeries in Licheng District, Jinan] [Query=Guangzhou Yuexiu District food recommendations, Level=restaurant dishes] =>
Delicious restaurant
dishes Yuexiu District, Guangzhou]
[Query=Hiking survival bracelet recommendatiinns Level=brand model] => [Brand and model recommendations for hiking survival bracelets]
8. [Query=Must-buy in Japan, Level=brand model] => [Brand and model of Japanese specialty products]
9. [Query=iPhone 11 photography recommendations, Level=model] iiPhone 11 models good for photography] => => 1000 useful products for desks]
10. [Query=1000 desk essentials, Level=product names
11. [Query=Changsha special breakfast recommendations, Level=restaurants => [Names of restaurants with
special breakfast in Changsha]
12. QQuryyNanjing surrounding tours, Level=cities) (Cities suitable for tourism around Nanjing]
13. [Query=Cheap clothing places in Shenzhen, Level=shopping locations] => [Names of shopping locations for cheap clothes in Shenzhen
14. [Query=Harbin bathhouse recommendations, Level=bathhouse names] => [Names of worthwhile bathhouses in Harbin]
Step Evaluate the relevance between Doc and rewrite_query. paying attention to important qualifiers in rewrite rewrite_query, such as time and location modifiers. If Doc misses important qualifiers, it can be considered irrelevant. Analyze the relevance in 1-2 sentences.
Step 3 If Doc is relevant to rewrite_quee rewrite_query, extract all answers (Options) from Doc that can answer the user's question and put them in AnswerList. Options must match the answer type required by rewrite_quens _uerry Specifically, rewrite_query requires "brand model", ensure each answer Option includes both brand nameänd " "model": if this cannot be satisfied, consider unextractable. If rewrite query requires "restaurant dishes ensure each answer Option includes both "restaurant name" and " dish name". Others like "bakery cake names" " follow the same pattern. If Option cannot meet the required answer type. consider it unextractable, and AnswerList will be empty list []-
Step 4 For each answer Option in AnswerList, extract a related description (Reason) from Doc. Format: ("Option": "xxx", "Reason": "xxx", ...]. Doc irrelevant or has no answers, return an empty list [].
### Notes ###
Answer Options must follow the original text, non-continuous extraction is allowed.
2. Reasons must be extracted from the original text, modifications allowed.
3. Answer Options should typically be specific entity nouns names that can be directly searched through search
22

---

Benchmarking Large Language Models in Social Networking Services
engines; avoid extracting vague descriptions like "this hidden gem hotel" or "rice noodles from the shop near XXX bus station".
Doc may have various article structures, including common parallel point structure with leading words. When extracting answer Options, avoid including leading words like "Figure 1", "Figure 2", ......"""" "Shop 1", "Shop 2", etc.
5. Reason length should not exceed 100 english character widths.
## Output Format # #
<relevance <Televncce_nnalsiisz analysisz...</rlevve ..<eelevanceeaanllsss>>> "...""""""""""" ... .../</Answerl.st>>
<Answerl.ist>(...
<Result> "Option": "xxx", "Reason": "xxx"... </Result>
HUMAN:
Query:
(query]
Level:
[level)
Doc:
<Doc>
{doc)
</Doc>
Output:
23

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-MRC (Simple)
SYSTEM: You are a reading comprehension master: will provide you with a user's note article, including a title and content, where the title may be empty "". will also give you a user's question.
Please Think according to the Following Steps
Step I, combine the note's title and content determine if the user's question can be answered. If there are specific entity words qualifiers in the question, they must be satisfied count having answer. Step 2, if Step determines that the content answer the user's question, then extract the answer from the original note content and output the extracted original content.
Content Requirements for Answers
Extract sentences that can answer the question, cross-sentence extraction is allowed, but must be complete sentences or paragraphs, multiple consecutive sentences must be extracted continuously, and preserve special characters like.
If the middle part of consecutive sentences irrelevant the answer, can be skipped for cross-sentence extraction; if the answer long (more than 120 characters), it can be extracted by points.
Content must come from the original text, no summarization allowed; do not extract content unrelated to answering the user's s question.
Format Requirements for Answers
Output the answers in list form, where each value in the list is a sentence or paragraph that can answer the question.
If no answer can extracted, return an empty list [].
Do not output any content other than the answer list.
HUMAN:
Here the note title and content: [content)
User's question: {query)
Answer:
24

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-NER
SYSTEM:
Extract entity information from the Passage according to the Question requirements.
Note that the extracted answers should maintain the same case as in the passage. Here are some examples:
Passage
212 In stock Japanese DIOR 2023 Fall new single color flame gold blush 100/625/343
Question Please find all 'season' entities
Answer: Fall
BUTOO tattoo stickers English series colored tattoo stickers waterproof female long-lasting niche high-end tattoo stickers
Passage:
Question
Please find all 'stylee entities
Answer high-end, niche
Passage mimm collection slim t-shirt women's 's versatile solid color curved hem short sleeve v-neck fitted hot
girl top
Question Please find all 'style (for clothing and accessories)' entities
Answer short sleeve, fitted, curved, v-neck
Passage Early autumn 2022 new sweater plus-size slimming outfit high-end top skirt autumn winter short skirt
two-piece set
Question Please find all 'category' entities
Answer skirt, sweater, short skirt
Please output the answer (in the same format as the examples above) DIRECTLY after 'Answer:', WITHOUT any additional content.
HUMAN:
Passage:
{content)
Question: (question)
Answer:
25

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-Gender
SYSTEM: According to the following note content fields, determine whether the content is likely to trigger discussions of interest to both men and women. includes such content, reply "Yes,' otherwise reply 'No. Only reply with 'Yes" or 'No," without any additional explanation.
These contents mainly include the following aspects:
Stereotypes about men and women;
Gender bias phenomena, gender inequality in occupations, sports, interests, etc.;
Fertility-related content: including views on childbirth, parenting experiences, etc.;
Marriage-related content: including matchmaking, weddings, post-marriage life, mother-in-law relationships, marital breakdown, prenuptial property, etc.;
Romance-related content: emotional experiences, arguments, scumbag men/women, cheating, etc.;
Sex-related content: including sexual violence, sexual harassment, discussions on sexual knowledge, etc.;
And some related social events. If the text mentions these topies, it is likely to trigger discussions of interest to both men and women.
Here are some examples:
Note Content
Title: When deliberately sleep in separate beds from my boyfriend to see his reaction .nCategory: Emotions-Daill Life n'Text: In the end it really scared me # Love ['Topcc] # #Boyfriend [TTppcc #Couple Daily ['Tppic] When deliberately sleep separate beds from my boyfriend... of content)
Answer: Yes
Note Content
Title: The Young Lady's Meng'
AnCatgory:: Hlumanities-Reading BBay Food [TTppcc # Accessories
Share [Topicc] # Anarr: (reer....rest of content)
Answer: No
HUMAN:
Please provide your judgment the following content.
Note Content:
{content)
Answer:
26

---

Benchmarking Large Language Models in Social Networking Services
Prompt Template used for Note-CHLW
SYSTEM:
Please select words worth highlighting from comments. These words should be meaningful and specific entities, issues, or descriptions that spark users' search interest.
For life trivia without meaning or search value and content containing numbers, please select carefully. Here are extraction examples:
1)
Inpul
Give me transparent Bluetooth speaker link; Return ['
'trnnpparet Bluetooth speaker'
2) Inpul
Is there tutorial for Huawei Smart Screen; Return
[' Huawei Smart Screen tutorial' ]
3) Inpul Is there any specific fitness and weight loss APK?; Return:
fitness and weight loss APK'
4) Inpur
Noodles have calories but the sauce does; Return [' noodles have no calories but sauce does'
5) Input
How's Peking duck; Return
Peking duck'
HUMAN:
Comment:
{comment)
Possible Highlight Word Options List: (possible_list) list)
Please provide list words you think are suitable for highlighting. If you think none are appropriate, please output an empty list [].
Answer:
Prompt Template used for Note-QueryGen
Assume you are an internet user, your task is: based the input image/text or video note content, generate 1 novel search term related to the core entity words in the note.
SYSTEM:
Output Format
The return result is single line string, recording the generated search term.
Output Requirements You need to fully understand the input note information, ensure that the given search terms are complete, novel and interesting, with the possibility of being targeted in-depth exploration, and can bring incremental information to the current note;
2. Ensure that the given search terms are fluent, without expression problems (including incomprehensible, repetitive, typos, word order, etc.);
Good search terms can be questions (such as how to xx, what is xx, how to xx etc.), or appropriate compound words (xx tips, xx tutorials, xx outfits, xx recommendations, xx collection, xx selection), where xx comes from the note content.
HUMAN:
Please generate search terms for the following note based on the above instructions.
Input:
[content)
Search term:
27

---

Benchmarking Large Language Models in Social Networking Services
B. Crowdsourcing
In conducting our study, we identified several potential risks to participants. Firstly, there a risk to privacy and confidentiality. as participants are required to share personal information. To mitigate this, all data will be anonymized and stored securely, with access restricted to authorized personnel only. Secondly, there may be psychological risks, such as discomfort stress during the tasks. To address this, we have included detailed instructions and debriefing sessions to ensure participants feel supported throughout the process. Additionally, participants have the right to withdraw from the study at any time without penalty. Lastly, while there are no significant physical risks associated with our procedures, we will monitor participants for any signs of distress and provide appropriate support. We pay each participant an hourly rate of $10. The primary participants we recruit are college students with master degree (Age ranging 23-28).
We give the performance of models, and the human check rate around 12% in Table 4.
Table Task Performance Metrics
Task Name
Base Model Accuracy on SNS-BENCH
Note-NER
88.3% Hote-CHLW
87.6%
C. Cases in SNS-BENCH
For each task, we provide an example in the following content.
The Case Note Tate-Taxoomy Choice)
SYSTEM:
(See Appendix A.2)
HUMAN:
Input:
Note Title: Happiness for Just 9.9, You Deserve the Wonderful Duck Camera at the Imperial Ancestral Temple!!
Note Content: The Wonderful Duck Camera is a! It's to the extent that want to show it to a plastic surgeon to get my face done like this! [Face Pulling H] This year's birthday wish is made in advance: to look like the photo! [Shy R][[Shy Riln [TTopic] # #AIProfilePhoto [Topic] # #QingHuan [TTopic] #
Candidate Categories: :
Photography Equipment, Portrait Photography, Photography Skills
Answer: Portrait Photography
28

---

Benchmarking Large Language Models in Social Networking Services
The Case of Hote-Taxonony (Multiple-Hopy
SYSTEM: (See Appendix A.2)
HUMAN:
Input:
Note Title: Slim and slender arms solve 80% of dressing problems! Note Content: #Slim Arms ['Tppic] # #Eliminate Bingo Wings [TTpic] # #Strongest Slim Arms [Topic] # #Bingo Wings [Topic] # #Slim Arms and Slim Legs [Topic] # #Arm Shaping Tutorial [Topic]+ #
Candidate Primary Categories
[' Business & Finance', 'Trends', MMedical Health', Fitness
Weight Loss',
Social Sciences',
Emotions', Outdoors', Sports', Urban Travel'
Candidate Secondary Categories;
[' Film & Video Montage', Weight Loss', Educational Daily', Exhibition', Football', Maternity Experience', Posture Correction', Anime', Immersive Activities') Candidate Tertiary Categories:
[' Skiing Equipment & Apparel', Anniaal Science', OOthe Fitness', OOter', 'Learning Methods', Architectural Design', Cullural Collectibles', Other Music Performances', Weight Loss Tutorials'
Answer Fitness & Weight Loss Weight Loss Weight Loss Tutorials
The Case of Note-Hashtag (Single-Choice)
SYSTEM: (See Appendix A.2)
HUMAN:
Inpul
Note Title: Completely Crazy 18.9/6 Bottles!! Secretly Placing an Order Behind the Boss's Back Hurry and Stock Fino, you really!!' nnScc delicious NFC coconut water, and it's even sold cheaply?! Nnoo extra additives! The ingredient list only has coconut water! n O added sugar, O fat and low calories's nA box of 200g :PPrffcct for babies on a weight loss diete AnEven workers can easily handle it, lightweight and portable, enjoy anytime! Don't It miss this opportunity! Rush now!
Up'.n Note Content:
is less than 39 calories!! nPerfect
Candidate Topic Tags:
#Collagen Bomb, Wool Snatching, Kids' Clothing, Sad Puppy Answer Wool Snatching
The Case Note- HashtHa (Multiple-Chool Choice)
SYSTEM: (See Appendix A.2)
HUMAN:
Input
Note Title: Making Almond Caramel Today 4An Note Content: Maate'nMade by my sisters, not by me, but can eat it [Selfie R]
Candidate Topic Tags:
Caramel Almond Rice Boats, Cream Bars Squeeze, Palau, Glory New School Season Flying to New Life, Gray Industry, Caramel, Good for Insomnia, Cell Light, Hand Account, Small Cities Also Have Beautiful Scenery, Officially Announced Female Love, Precise Location Leak Detection
Answer: Caramel Almond Rice Boats, Caramel
29

---

Benchmarking Large Language Models in Social Networking Services
The Case Note- Note-QueryCor (Explain)
SYSTEM: (See Appendix A.2)
HUMAN:
Question
What books to buy for primary school teacher qualification certificate exam
Document:
Primary School Teacher Qualification Books Teacher Recruitment Exam [Toppc] #O At that time, I bought books from Zhonggong Education for the written exam and books about interviews from Fenbi for the primary school math. Now don't need them anymore. (About 90% new) Q About the books: actually, the content doesn't 't change much every year. The exam questions are quite stable. The key areas focus on are multiple-choice questions. You don't need to memorize lot, just get familiar with the content. spent over 300 yuan Fenbi offline class for the interview, which guaranteed refund if didn't 't pass. I didn't 't pass the first time, but got refund quickly. The second time, practiced on my own and passed. Remember to sign up for the interview soon as the written test results are out. Answer Yes
30

---

Benchmarking Large Language Models in Social Networking Services
The Case of Note- Note-QueryCor (Topic)
SYSTEM: (See Appendix A.2)
HUMAN:
# Class 3
The Query and note you received are
\nnuury:: How to draw a simple apple sketch Seatch'nnonn nnooe: Cute simple sketches; Simple sketch Draw red apple using the number 5 #CuteSimpleSketch [Tppcc] # #SimpleSketch [Topcc] # #Draw ADrawsSSimleeketth [ [Topic # #Christmas [Topic] # Write the number five, connect with curved lines, draw leaves on it. Add details and color in, and the red apple done. An
Please output the rating level, choosing from the five numerical levels: -1, 2, Output only the numerical rating, without any additional content.
Answer 3
Class
The Query and note you received are
'nQQuer: Zhejiang Middle School Entrance Examination nNot:: Zhoushan middle school admission quota for Zhoushan Middle School. Zhoushan 2023 Zhoushan Middle School admission quota Dinghai Second Middle School 63 students Nanhai 49 students Putuo Wuling 42 students Putuo Chengbei 38 students Zhoushan First Junior High School Donggang Middle School. 27 students 25 students Zhoushan Second Junior High School 13 students Greentown Yuhua 12 students 9 students Dinghai Third Middle School 9 students Jintang Middle School Dinghai Sixth Middle School students Baiquan Middle School 3 students n
Please output the rating level, choosing from the five numerical levels: -1, 2, Output only the numerical rating, without any additional content.
Answer
# Class
The Query and note you received are
'' NAQQuury Apartment Orientation nNote: Chengdu Post-95s My Own Small Apartment Finally for Rent 7-11; Apartment; IKEA; Furniture; Railway South Station Rental; Rent My Own Small Apartment! Brand new furniture! A full-apartment rental near IKEA at Railway South Station, agency fee, brand new furniture!! Vintage cream style, fully equipped appliances, utilities included, ready to move in Houuse Details: The community is from Zhonghai, 50 sqm one-bedroom unit facing south to the central courtyard with great natural light, open kitchen, independent bathroom, spacious balcony, and an elevator. * @Transportation: Railway South Station/Shenxianshu, direct access to subway lines 1, 5, and 7, underground parking/curbside parking/shared bikes, convenient with buses and subways. @SSurounding Facilities: Right downstairs there's 7-1 1/Red Flag Chain Store/Wu Dong Feng, walking distance IKEA furniture/CapitaLaad Mall/T-MART/HHrn Fresh/City First Hospital/Huaxi Hospital/Southern Station Park, located near Railway South and Financial City business districts. Super convenient for dining, shopping, entertainment, and medical needs. * #ChengduuRenaalsTToic #RailwaySoutthStatinnn (ooic) AaaaaniialCityRenttsssS/il/ #IKEA[Topic]# BaapnngghiCCiiiiggaayyyoooooo
#ChhongduDneeBdddoooeennnna//// #ChennghuOneBBedroom/Tppppe
Please output the rating level, choosing from the five numerical levels: -1, 0, 1, 2, Output only the numerical rating, without any additional content.
Answer
31

---

Benchmarking Large Language Models in Social Networking Services
Class
The Query and note you received are
AnQuery: Shabby House Coffee Second Store Store\nNote NNoot: Chengdu Store Exploration Vintage French Style
Cafe niceland Coffee; Cafe; Vintage; Vintage Style; Chengdu Store Exploration; Store Exploration; French Style; Food Store Exploration Store Name: @niceland cafe Nai Island Coffee * Address: No.45, Yixue Alley, Jinjiang District, Chengdu * * Strolling near Chunxi Road with friends * Happened to discover this vintage-style cafe * The decor is very romantic and vintage French Castle Style * The dim lighting adds unique atmosphere * Every corner is perfect for photosia * * Recommended special coffee * The names are very unique * My friends and I ordered the Cinderella and Hana Hime series Full of romance and sweetness
The combination of coffee and refreshing fruity aroma * Is blended very subtly Here, you not only enjoy coffee during the day But also have drink with friends in the evening * It's perfect place to visit during the hot summer days Weekeeed[tooppd #CChenguuSSorexxplortoooionono CccpraaccoooooppppiccCccCeeeaaee #ChengduCaff((tppie) #ChengduS- #CChnngduuhoooograpyyyyyyyo [topi] #ChengduDDinnknnglppief (Sppc #Coffee[topicc|# #ExploreWithMe(tpppe # |00StoeeEpplozziiinPnnnoooggge
Please output the rating level, choosing from the five numerical levels: -1, 2, Output only the numerical rating, without any additional content.
Answer 0
# Class -1
The Query and note you received are
(nQuuur::: Hands swollen from gaming' nnoote: Help! My arm got red and swollen badly after getting the hepatitis B shot. Hepatitis B; Hepatitis B vaccine; vaccine It's already the third day, and my arm is still very swollen [crying emoji] Then went to see doctor, and they pressed my and said it's fine [crying emoji]. They said some people have it worse. If it feels itchy in few days, it means it's healing [crying emoji] #HepatitisB [topic] # #HepatitisB Vaccine [topic] # # #acccin [topic] # AHHepatitssBAntibod [topic] # #HepatitisSB [topic] # #HepatitisB TripleTesst [topic] ] # #TripleTestAlleegaaa
ttopic] | #\n"
Please output the rating level, choosing from the five numerical levels: -1, 0, 2, Output only the numerical rating, without any additional content.
Answer -1
32

---

Benchmarking Large Language Models in Social Networking Services
The Case of Note-MRC (Complex)
SYSTEM: (See Appendix A.2)
HUMAN:
Query:
Dog Hair Drying Hands-Free Gadget
Level:
Product Name
Doc
<Doc>>
Hair Dryer's APet Hair Dryer Neck Mount Stand, Pet Grooming Detangling Brush Hands-Free, Groomers no longer need to hold the hair dryer with their neck, super useful! Pet Groomer </Doc>
Oulput
<rewrite qureiigguereeeeeeeee of the hands-fres gadget for drying dog hair</rewrite
relevant to rewrite_quent because Doc mentions the pet hair dryer neck mount stand that frees up
Hair Dryer Neck Mount
Stand")</AnswerlList int.sseerrsssstnnccResssool// "Option": "Pet Hair Dryer Neck Mount Stand", "Reason": "Pet
Hair Dryer Neck Mount Stand, Pet Grooming Detangling Brush Hands-Free Free, Groomers longer need to hold the
hair dryer with their neck, super useful!" |<Ressllt>
The Case of Note-MRC (Simple)
SYSTEM: (See Appendix A.2)
HUMAN:
Here is the note title and content
2 years and 10 months old, first time riding bicycle at Decathlon AnInitially bit unaccustomed, only able to pedal half circles. AnAfte a while, got the hang of it, awesome. \n14 inches, perfect for a 95 cm tall baby at 2 years and 10 months Wnth training wheels, no fear of falling, yyds'v An n#Everry Second Worth Recording/Topicitt #Children's ss Balance Bike/Toppcci# #Children's 's Bicycle[Topic]# #Boy's Happiness(Topic #Decathlon/Topicitt #Cute Kid's Daily Life[Topic]# #Decathlon Children's Bicycle[Topic]#
User's 's question:
Can a 2-year-old ride bicycle
Answer: ("Iiitially a bit unaccustomed, only able to pedal half circles. AnAfter a while, got the hang of it, awesome.", ", "14 inches, perfect for a 95 em baby at years and 10 months n With training wheels, no fear of falling, yyds")
33

---

Benchmarking Large Language Models in Social Networking Services
The Case of Note-NER
SYSTEM: (See Appendix A.2)
HUMAN:
Passage:
Please identify all color' entities
Question Korean Lipstick Liquid Matte Velvet Dirty Orange 220 Pumpkin Color Whitening Lipstick 909 116
Answer: Pumpkin Color, Dirty Orange
The Case of Note-Gender
SYSTEM: (See Appendix A.2)
HUMAN:
Please provide your judgment on the following content.
Note Content
Title: Yang Li's Stand-up Comedy: Have a Younger Brother (Continued) ese AnCategory: Entertainment-Otin Entertainment ' \nBody: d*0 Yang Li's stand-up comedy is truly heart-wrenching in the latter half. Mom left the pig farm to my brother, and it's hilarious how women help women. \n'n Yang Liwei, China's 's first astronaut, once said, 'Our journey into space a journey of no return.' Similarly, your devotion your brother a one-way street, with no turning back. \n'na you have such lively sister and brother?' she teased, SoSnanduUoom Topics #Siblings #Comedy
Answer Yes
The Case of Note-CHLW
SYSTEM:
(See Appendix A.2)
HUMAN:
Comment
What does the tag inside mean, you can see when buying the XO album, didn't understand what means [cry]\n'v
Comment:
Possible Highlight Word Options List [' ('ta' XO album']
Please provide list of words you think are suitable for highlighting. If you think none are appropriate, please output an empty list [].
Answer
XO album')
34

---

Benchmarking Large Language Models in Social Networking Services
The Case of Note- Note-QueryGen
SYSTEM: (See Appendix A.2)
HUMAN:
Please generate search terms for the following note based the above instructions.
Inpul:
Note Title: Empty, \nMulti-level classification label for the note: Home Furnishing/Home Appliances/Kitchen Appliances, AnMain content: Many people say that under-sink water purifiers are unhygienic, so I installed a direct drinking water machine. Now they say tea bar machines are good to use. Can's 't I just buy another tea bar machine? #renovationvlog[toiip #ddningeabinet(tppcchtt \nCove image OCR information: Did fall into another pit? His have been water in my brain when decorating at that time, AnPopular comments: just feel that the pipe the under-sink water purifier is long, and this pipe is not very hygienic. Just boil water with electric kettle. use this thing; Electric kettle [slyR]; What do you think of this combination [slyR].
Search term Which is better, tea bar machine under-sink water purifier
D. More Detailed Results
We provide detailed results Note-MRC (Simple) in Table 5.
E. Hard Case in Note-Gender
The Hard Case in Note-Gender
SYSTEM: (See Appendix A.2)
HUMAN:
Please provide your judgment on the following content.
Note Content
Title: All Celebrities Are Innocent, Please Let Them Go nnaatgory: Entertainment Celebrity Entertainment News's \BBody: #XXX-Celebrity [TTpicc #Celebrity News [TTpic]# #Thank You to All Celebrities Above [Topic]* asr: Don't 't choose me, I'm the one mocked for having bowl in the fridge by the entire XXX-Show team. Don't 't choose me, I' the one mocked for smoking and having peaceful eyes. Don't choose me, I' I'm the one mocked for being too melodramatic because of my nice voice. Don't I choose me, I'm the one called 'water queen' for winning an award. Don't t choose me, I'm the one mocked for plastic surgery before and after debut. Don't t choose me, I'm the one called 'rreen tea' without reason. I'm the one who got blacklisted for screaming excitedly after winning a game XXX-Show.
Answer
Gold
GPT-4o
Deepseck-V3
Claude
Gemini
GLM-4-Plus
Hamma3.-70B-Instreee
Qwen2.5-72B Dwenn.5577B-Insttuuc
Phi-4-14B
Yes
No
No
No
No
No
No
No
Yes
F. Confusion Matrix Details
All confusion matrix results are in Figure 10.
35

---

Benchmarking Large Language Models in Social Networking Services
OFT
Oow
The
Figure 10: Confusion matrices showing classification performance across different models on Cote-QueryCorr task.
36

---

Benchmarking Large Language Models in Social Networking Services
Note-MRC (Simple)
Models
Success-Ratio
F1
BLEU
Rouge-1
Rouge-2
Rouge-L
Overall
Open-Soue Source Large Language Models (1,58+)
Qwen2.5-1.5B
41.87
78.74
30.51
47.59
38.23
41.80
19.84
1..m2.5-1.5BIInstmue
51.25
73.97
23.42
44.83
34.61
39.04
22.13
Llama-3.2-3B
57.50
76.09
27.46
43,90
33.78
37.89
25.20
Llama-3.2-3B-Inste
62.30
84.75
29.36
46.19
36.50
40.12
29.52
Phi-3.5-mini-Instne mintsrrtii--nsteeet (3.82B)
72.75
86.42
42.52
65.05
58.07
59.86
45.38
Open-Source Large Language Models (7B+)
Internlm-3-8-IIsstie
23.88
73.98
55.82
73.49
67.38
70.01
16.27
Llama-3.1-88
37.75
83.82
34.89
55.04
46.85
49.81
20.42
GLM-4-9B-Chat
43.88
71.53
47.53
67.98
62.18
63.51
27.44
Qwen2.5-7B
71.25
81.56
42.23
62.75
55.65
57.99
42.78
1-88-Itrcct
87.00
9.76
49.62
68.57
63.36
65.85
44.74
Internlm-2.5-7B-Chh
82.12
85.87
40.85
60.96
52.59
55.19
48.53
1ntemlm-225-20B-Chaa
92.88
88.84
41.90
60.88
$2.03
55.10
$5.49
Qwcn2.5-7B-Instree
98.62
84.96
41.67
64.99
58.93
60.17
61.29
Phi-4-14B
93.75
88.08
59.73
77.13
72.32
73.15
69.45
Open-Source Large Language Models (32B+)
Llama-3.3-70B-Innn Instruct
29.37
36.06
65.39
78.04
74.65
75.44
19.36
Yi-1.5-34B-Chat
38.12
85.18
56.90
73.80
68.98
70.03
27.06
Qwen2.5-32B
84.75
72.33
50.29
68.50
63.10
64.52
54.02
Qwen2.5-72B
76.75
82.87
51.20
69.51
64.35
65.70
51.21
Owen2.5-32B-Inaee
98.25
75.67
60.90
77.77
73.54
74.00
71.11
Qwen2.5577BBhnstrrue
98.25
91.64
57.34
75.08
70.65
71.31
71.92
Closed-Source Large Language Models (API)
Claube335--omme2222241022
87.00
91.98
60.83
77.36
72.79
73.10
GPT-4o-mini-20244077-1
99.38
58.92
57.52
75.84
71.41
72.52
66.82
Doubao-Pro-32k
92.62
89.43
61.49
77.96
72.89
74.05
69.62
Gemini-2.0--ae P.-
97.38
90.14
55.03
74.20
69.65
70.28
69.97
GPT-44-2024-05-13
97.00
84.64
58.23
76.07
71.59
72.51
70.43
Gemini- 1.5-pro
97.88
89.69
$7.50
75.17
70.94
71.91
71.49
GLM-4-Plus
96.75
90.57
60.13
77.03
72.92
73.54
72.41
GPT-44-2024-11-22
97.62
88.64
59.99
77.22
72.67
73.37
72.61
Deepscek-Vi
96.25
89.63
62.78
78.80
74.55
74.81
73.26
Table 5; Detailed results of Note-MRC (Simple)
37


Reviews:
[{"review_id": "wNWRUTM9b5", "reviewer": "Reviewer_hXUu", "summary": "This paper presents SNS-Bench, to access LLMs on different social networking services. It includes 8 tasks such as query content relevance. It evaluates 25+ LLMs and provide further insights.\n\n## update after rebuttal\nI maintain my score in support of the work after rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Review 1**:\n\n**R1.Q1: Font size in Table 2.**\nThank you for your helpful suggestion. We will adjust the font size and optimize the layout of Table 2 (e.g., using multiple rows for tasks if needed) to improve readability in the revised version.", "claims_and_evidence": "The central claim is the current LLMs are not performing ideally in SNS tasks. Table 2 well supports this claim.", "methods_and_evaluation_criteria": "Yes, it evaluates 25+ leading LLMs including Claude, GPT, Llama, Qwen etc.", "theoretical_claims": "There is no theoretical claims if the reviewer understands correctly.", "experimental_designs_or_analyses": "Yes, the experiment design is sound (this is a benchmark paper, with reasonable and comprehensive choice of LLMs.)", "relation_to_broader_scientific_literature": "Related benchmarks do not benchmark complex scenarios with multiple network tools or heavily rely on GPT models. The paper well addresses these two problems.", "essential_references_not_discussed": "The reviewer believes the paper provides a good list of references.", "other_strengths_and_weaknesses": "The paper presents a good problem - evaluating LLMs in SNS tasks.", "other_comments_or_suggestions": "The font of Table 2 is too small. Please consider using more than one row for one task.", "questions_for_authors": "Please see above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46R00jnKZC", "reviewer": "Reviewer_mRje", "summary": "This paper aims to advance LLM models for Social Networking Services(SNS) by introducing a comprehensive benchmark SNS-BENCH derived from a social media platform, addressing the limitation of studying SNS in isolated tasks in prior work. The benchmark includes eight distinct tasks, such as note classification, sentiment analysis and personalized recommendations, providing evaluation across various realistic dimensions. The authors evaluate over 25 LLMs on SNS-BENCH, providing insights into model performance across different categories. One main result shown is that closed-source models generally outperform open-source ones, but with a relatively small margin, and tasks involving complex emotions and long-text understanding remain challenging for current LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Review 2:**\nThank you for your insightful feedback. \n\n**R2.Q1: Isolated evaluation of SNS tasks.**\nWe acknowledge the concern regarding task isolation in SNS-Bench and appreciate the opportunity to clarify our design rationale.\n1. Task-Specific Evaluation Necessity:\nSocial networking services (SNS) involve diverse interactions (e.g., content comprehension, sentiment analysis, recommendation), each requiring distinct capabilities. Isolated evaluation allows us to:\n  - Pinpoint strengths/weaknesses: For instance, models may excel at hashtag selection (structured tasks) but struggle with complex reasoning in Note-MRC (Figure 7).\n  - Guide targeted improvements: Task-specific metrics (e.g., F1 for Note-NER, ANLS for Note-QueryGen) reveal granular performance gaps (Table 5).\n2. Holistic Insights via Aggregation:\nWhile tasks are evaluated independently, our analysis synthesizes cross-task patterns (Section 4.3). For example:\n  - Closed-source models consistently outperform open-source ones (Table 2), suggesting general superiority in SNS contexts.\n  - Tasks requiring emotional/cultural understanding (Note-Gender) exhibit higher variance, highlighting challenges in social nuance (Section 5.1).\n3. Real-World Generalization:\nThe benchmark’s diversity—8 tasks spanning 6,658 questions with varied formats (Table 1)—ensures broad coverage of SNS scenarios. By testing isolated but representative tasks, we simulate the multifaceted demands of real-world platforms (e.g., handling both classification and generation).\nWe agree that future work could explore interdependencies between tasks (e.g., joint training). However, our current design prioritizes diagnostic clarity, enabling actionable insights for model development. Thank you again for your valuable advice. We will emphasize this rationale in the revised manuscript.\n\n**R2.Q2: Lack of creativity in evaluation framework.**\n\nWe appreciate the insightful feedback. While our benchmark adopts standard evaluation metrics for individual tasks, its core value lies in the authenticity and diversity of real-world SNS scenarios. Unlike synthetic or simplified datasets, SNS-Bench captures nuanced user behaviors (e.g., informal language, cultural references) and task interdependencies (e.g., sentiment influencing recommendations) from actual SNS platforms. This fidelity enables a more grounded assessment of LLMs’ practical utility in social contexts. Future work will build on this foundation to incorporate interactive and adversarial dynamics.\n\n**R2.Q3: Citation and figure readability issues.**\n\nThank you for your careful review. We will address both points in the revised manuscript:\n1. Citation update: The reference to Bandura will be updated to include the publication year (e.g., \"Bandura, 2001\").\n2. Figure improvements: Figures 4–9 will be resized to ensure all text (axis labels, legends, annotations) is clearly legible in the final version.", "claims_and_evidence": "Claims made in this work are overall well-supported. The comprehensiveness of the SNS-BENCH benchmark is well illustrated by its diver source and different tasks for SNS. This dataset itself is well-structured, paired with standard metrics for each subtask. Experiments are thorough in terms of including a variety of popular LLMs from both open-source and close-source communities. However, one caveat is that the authors motivate the idea to shift from focusing on isolated SNS tasks, but this SNS-BENCH benchmark proposed in this submission suffers from isolated evaluation on each SNS task after breakdown, albeit the source data is realistic and situated. This makes some claims on the ability of generalizing the conclusion beyond individual tasks less convincing.", "methods_and_evaluation_criteria": "The benchmark includes a rigorous annotation process with both automated and human verification, ensuring high data quality. Evaluations on each separate task are fair and standard, including widely-used metrics from QA and text generation literature (F1, BLEU,etc.)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental setup and analysis are well-documented, with clear descriptions of the evaluated models, the computational resources, and the experimental protocol. This work thoroughly compares the performance of different models and analyzes task-specific challenges. However, the discussion reads as a bit brief, likely due to limited space. Moving some graphs to the appendix would help. This work can be further strengthened by providing analysis into the social and online dimensions of SNS tasks, such as how different models cope with user diversity and time dynamics. These dimensions are naturally introduced by the source data into this benchmark, but the evaluation and analysis do not explicitly address those challenging aspects.", "relation_to_broader_scientific_literature": "The study emphasizes the necessity of a comprehensive SNS-specific benchmark, and highlights gaps in existing LLM capabilities regarding social interaction in the online culture.", "essential_references_not_discussed": "This paper is well-referenced.", "other_strengths_and_weaknesses": "One of the key weaknesses of this submission is its lack of creativity in designing a novel evaluation framework. The approach taken primarily involves deriving a benchmark from real-world user data and breaking it down into individual tasks with standard evaluation metrics. While this is a reasonable and common methodology, it does not push the boundaries of evaluation in the Structured Neural Search (SNS) domain. The field would greatly benefit from more innovative evaluation paradigms that account for the complex nature of real-world SNS challenges, particularly those involving user interaction and diversity.", "other_comments_or_suggestions": "022: “in SNS remains challenging (Bandura).” → include year for this citation.\nFigure 4,5,6,7,8,9: words in the graph are a bit too small to read.", "questions_for_authors": "Have you considered evaluating models under adversarial settings, simulating malicious users in online communities? \nThe dataset may have biases due to its reliance on a single social media platform, so I’m curious about further analysis on this aspect.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tHkR24YKWf", "reviewer": "Reviewer_ZKq6", "summary": "The paper introduces SNS-BENCH, a comprehensive benchmark for evaluating large language models in social networking service tasks. It covers eight diverse tasks—from note taxonomy and sentiment analysis to query generation and entity recognition—using a dataset of 6,658 questions sourced from a major social platform. The study presents detailed experimental results across 25+ LLMs, highlighting performance variations and a scaling law that informs both the strengths and limitations of current models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R3.Q1: Multimodal limitation.**\nWe sincerely appreciate the reviewer’s constructive feedback. The reviewer is absolutely right to highlight the importance of multimodal interactions in modern SNS platforms. In fact, we are already working on SNS-Bench-V2, which will incorporate image-text pairs (e.g., OCR-extracted text from note images, user-generated captions) and multimodal tasks (e.g., visual hashtag recommendation, sentiment analysis). This extension will align with real-world SNS content while maintaining our focus on nuanced social understanding. We will explicitly discuss this direction in the revised manuscript’s Broader Impact section.\n\n**R3.Q2: Deeper analysis and insights.**\nThank you for prompting us to clarify our findings. Beyond aggregated results (Table 2), our analysis reveals critical task-specific and generalizable insights about LLMs’ social capabilities:\n\n**1. General Insights:**\n\nEven state-of-the-art models struggle with SNS-unique demands:\n  - Creativity: Underperform in Note-QueryGen (lexical diversity) and Note-CHLW (novelty detection) due to rigid training objectives.\n  - Depth: Struggle with Note-MRC (Complex) (evidence extraction) and Note-QueryCorr (Topic) (intent granularity), revealing gaps in social context modeling (Section 5.1).\n\n**2. More Task-Specific Insights:**\n\n  - Note-Taxonomy: Small models (e.g., Qwen-1.5B) favor single-choice tasks (accuracy: 27.5%), while larger models (e.g., Llama3-70B) excel in multi-hop reasoning (65.12%), proving scale aids hierarchical reasoning (Table 2).\n  - Note-Hashtag: Most models (32B+) perform better on single-choice (Qwen-72B: 86.25%) than multi-choice (84.60%), except Claude-3.5, which leads in multi-choice (88.58%), suggesting superior multi-label adaptability (Figure 9).\n  - Note-MRC (Simple): Gemini/Qwen excel at binary relevance judgment (F1: ~90%) but fail to precisely extract answers (BLEU: ~55%), whereas DeepSeek-V3 achieves the best content extraction (ROUGE-L: 74.81%), highlighting a trade-off between relevance judgment and granular retrieval (Table 5).\n\nThese observations underscore that SNS challenges require both social and technical innovation—a theme we will emphasize in Section 6. We will also add a new subsection (5.3 Model Behavior Analysis) to consolidate these insights with supporting visualizations (e.g., confusion matrices for Hashtag tasks).", "claims_and_evidence": "The paper claims that SNS-BENCH offers a systematic and robust framework to assess LLMs’ capabilities in SNS contexts and that model performance improves with scale, with closed-source models generally outperforming open-source counterparts. Extensive experimental results, quantitative metrics (accuracy, F1, etc), and detailed analyses across eight tasks support these claims. The evidence appears convincing, though some claims would benefit from further discussion on dataset representativeness.", "methods_and_evaluation_criteria": "The paper employs a multi-step data collection and annotation process, ensuring diversity and quality through de-identification, manual review, and expert validation. Tailored evaluation metrics are defined for each of the eight tasks, including accuracy, F1 score, and so on. These methods and criteria are well-aligned with the goals of assessing LLM performance in complex, real-world SNS scenarios.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design benchmarks over 25 LLMs using standardized prompts and metrics across eight distinct SNS tasks on a large-scale dataset. The analyses compare performance variations based on model size and type, providing clear insights into task-specific challenges and strengths. \n\nOverall, the design is methodologically sound and the analyses are thorough.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths include the paper’s comprehensive and well-structured benchmark, detailed experimental analysis, and clear evaluation criteria that address diverse SNS tasks. \n\nA notable weakness is the reliance on text-based interactions might overlook multimodal aspects inherent to modern social networking.\n\nAnother miner concern is the lack of insights or detailed analysis. The current analysis is more akin to a summary to the results.", "other_comments_or_suggestions": "Expanding the discussion on future extensions—such as incorporating multimodal data—could further enhance the paper. Overall, the work is solid and offers valuable insights into the evaluation of LLMs in social networking contexts", "questions_for_authors": "Please check above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y6FcKvvACN", "reviewer": "Reviewer_KXw4", "summary": "This paper introduces a benchmark dataset SNS-Bench to evaluate LLMs' capabilities in social networking service. The dataset consists of 8 NLP tasks centering around user postings that compiled from the REDnote social networking platform. The authors evaluate over 25+ closed and open sourced LLMs on SNS-Bench and show that their performance generally adhere to the scaling law.\n\n## update after rebuttal\nThe authors addressed my concerns regarding using OCR to convert post images to texts, and the performance without translating Chinese texts into English. However, my main concern about the work, that the proposed datasets and tasks lack the sense of \"personalized recommendation\" and \"social networking\". While the Author did try to demonstrate the \"personalization\" concept in the Note-Hashtag task, the provided example only shows how they select post tags to \"characterize\" the post author based on that single post, without utilizing any  personal attribute and behavioral histories. Further more, the \"social networks\" of the users (i.e., the connections and interactions of the user in question) are ignored in these datasets and tasks, making the claim of bench marking for \"Social Networking Services\" questionable. As such, I am maintaining my overall rating.\n\nAnd my apology that I replied the authors rebuttal as an official comment, which is not visible to the authors.My rebuttal comment is pasted below:\n\"\"\"\nThank the authors for demonstrating the \"personalization\" concept in the Note-Hashtag task. The provided example, however, looks like to select post tags to \"characterize\" the post author based on that single post. I think we need a clear definition of \"personalized recommendation\" as the rebuttable ground. In my humble knowledge, the core of \"personalized recommendation\" is to tailor recommendation for each individual based on their personal attribute and behavioral histories, and I don't see such characteristics in the proposed tasks, and neither do I see how \"social networks\" (i.e., the connections and interactions of the user in question) play a part in these tasks.\n\nAdmittedly, the LLMs have deeply reshaped the landscape of many research fields, and my opinion about \"personalized recommendation\" and \"social networks\" tasks may be too shallow and conservative. I would be glad to learn how the authors perceptive these concepts with the application of LLMs.\n\"\"\"", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R4.Q1: Broader Social Networking Context**  \n\nThank you for the insightful observation. The construction does incorporate key social interactions:  \n\n1. **Data Collection Pipeline (Section 3.2)**  \n   - User engagement metrics reflecting community response  \n   - Note categories and tags chosen by creators based on audience reception  \n   - Comment sections capturing direct user interactions  \n\n2. **Example: Note-Hashtag**  \n   - Candidate tags include the original creator’s selections (personal preference)  \n   - Community-generated popular tags for similar content (crowd preference)\n\n**R4.Q2: Personalization**\n\nWe give the example on personalization: \n```\n{\"content\":\"Note Title: 108 Exercise + Reading Classics\\nNote Content: Since discovering the Seed Law in October 2023, I’ve practiced 108 exercises and recited classics for over 7 months. My health has improved—better complexion, fewer wrinkles, and improved sleep. I've also experienced increased energy and financial gains.\\nAs Seed Dad says, only action brings results.\\nI hope everyone persists—each breakthrough leads to a better self!\\nEvery drop of sweat is proof of our transformation!\",\"candidates\":\"Enhance Energy,Grapefruit Jasmine,Must Visit Xinjiang,Cat Internal Deworming,Physics Tutoring for International Students,Stele Forest Museum,Sydney Spa,Buccellati Gardenia,108 Exercise,Tongue Piercing,Seed Law,Sharing What I Find Interesting,Women's Growth,Guojin Building\",\"answer\":\"Seed Law,108 Exercise,Enhance Energy,Women's Growth\"}\n```\n\n1. The tags (*\"Seed Law\"*, *\"108 Exercise\"*) are derived from:  \n     - The author's consistent personal practice (7 months)  \n     - Tangible health benefits (specific improvements listed)  \n     - Community validation (implied by their selection)  \n2. The selected tags represent:  \n   - A personal transformation journey (*\"Women's Growth\"*)  \n   - Proven techniques with real impact (*\"108 Exercise\"*)  \n   - A philosophical alignment (*\"Seed Law\"*)  \n3. This demonstrates how we capture personalization:  \n   - Creator-defined preferences (original tags)  \n   - Community-endorsed interests (popular tags)\n\n**R4.Q3: Platform limitations**\nWe appreciate the concern about REDnote's note format. In our ongoing work (SNS-Bench-V2), we are addressing this through:\n1. Multi-platform expansion:\n- Collecting data from Twitter/X  and Instagram.\n- Designing unified task formulations that work across platforms.\n2. Beyond note content:\n- Incorporating threaded conversations.\n- User history.\n- Cross-post interactions.\n\n**R4.Q4: Visual loss in OCR**\nWe take the visual element concern seriously and have implemented:\n1. Strict quality control:\n- Human verification for all OCR conversions (Appendix B)\n- Automatic filtering of low-confidence OCR results\n\n**R4.Q5: Typo and Tasks**\nWe will fix the typos and rewrite the detailed definition of tasks.\n\n**R4.Q6: Results on Chinese Version**\nMost cases are English, with Chinese translated into English (GLM4 + manual review). To provide more results, we have translated all English cases into Chinese (GLM4 + manual review). We will release the Chinese data.\nThe average Chinese results:\n||Note-Taxonomy|Note-Hashtag|Note-QueryCorr|Note-MRC|Note-NER|Note-Gender|Note-CHLW|Note-QueryGen|**SNS-Bench**|\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n|Llama-3.2-3B-Instruct|10.50|6.48|23.25|13.22|4.05|64.25|24.81|32.84|22.42|\n|Qwen2.5-1.5B-Instruct|12.09|34.06|42.42|17.61|42.55|68.91|50.69|32.43|37.59|\n|Phi-3.5-mini-instruct(3.82B)|32.84|62.09|40.91|25.16|38.30|54.40|26.47|33.21|39.17|\n|Phi-4-14B|57.16|63.60|44.74|41.17|46.18|89.64|27.1|31.43|50.13|\n|Glm-4-9b-chat|53.09|80.21|42.11|27.75|56.69|88.08|37.76|37.51|52.90|\n|Qwen2.5-7B-Instruct|47.90|77.54|43.52|54.37|62.51|89.12|36.20|37.24|56.05|\n|Qwen2.5-32B-Instruct|66.65|84.08|51.38|64.87|65.59|90.67|39.00|35.78|62.25|\n|Qwen2.5-72B-Instruct|68.29|87.83|55.90|59.66|65.75|92.23|49.01|39.65|64.79|\n|Deepseek-v3|74.26|91.39|57.22|62.83|74.2|93.26|40.93|35.07|66.14|\n|GLM-4-Plus|71.42|89.71|52.67|61.49|68.88|93.26|32.88|36.03|63.29|\n|Gemini-1.5-pro|70.27|87.88|48.39|60.42|70.18|90.16|34.26|37.36|62.36|\n|GPT-4o-2024-05-13|69.18|80.28|55.02|65.11|70.52|91.19|48.04|39.56|64.86|\n\n(PS. Llama-3.2-3B-Instruct and Phi-4-14B with a performance drop.)\n\n**R4.Q7: Social Networking & Personalization**\nSNSbench captures social networking dynamics through:\n1. Interaction Signals: Tasks use data with implicit social context—comments, user-generated tags, and replies. For example, Note-CHLW identifies highlight words from actual discussions, mirroring how platforms detect trending topics.\n2. Personalization: Most tasks reflect personalized SNS behaviors, for example:\n- Note-QueryGen: Generated queries mimic how users personalize searches based on interests (e.g., converting a skincare note into \"best vitamin C serums for sensitive skin\").\n\n**We kindly invite you to review our responses and reconsider your assessments. Thank you for your time and consideration!**", "claims_and_evidence": "- The paper claims that SNS-Bench is designed to assess LLMs' capabilities in social networking services. However, the eight benchmark tasks primarily align with conventional NLP tasks centered around user notes (postings), without explicitly modeling the broader social networking context, such as user interactions, network structures, or temporal dynamics.\n- Section 3.1 defines Personalized Recommendation as a task category, stating that models should deliver tailored content based on user interests, behavior, and past interactions. However, the proposed tasks, Note-Hashtag and Note-QueryGen, do not incorporate any personalization mechanism. They focus on general content tagging and query generation rather than adapting recommendations based on user-specific preferences or engagement history.", "methods_and_evaluation_criteria": "No, the proposed benchmark does not consider how a user's social networking may affect their engagement with the media generated by their social networks.", "theoretical_claims": "N.A., this is a benchmark paper, no theoretical claim.", "experimental_designs_or_analyses": "Other than the missing of user's social networking context, the experimental design and analysis are solid and comprehensive.", "relation_to_broader_scientific_literature": "The proposed benchmark dataset is related in applying LLMs in social networking tasks.", "essential_references_not_discussed": "not in my awareness", "other_strengths_and_weaknesses": "Strengths:\n- The paper evaluates a diverse set of both open-source and closed-source LLMs on the proposed benchmark, providing a broad comparative analysis.\n- The SNS-Bench provides a framework to evaluate LLMs's capabilities in social media related tasks.\n\nWeaknesses:\n- The benchmark tasks are designed around the note format of the REDnote app, which may limit their generalizability to other social networking platforms with different content structures and interaction dynamics.\n - The conversion of user note images to text using Optical Character Recognition (OCR) may miss critical visual elements that contribute to the meaning of note, potentially affecting the fidelity of content understanding.", "other_comments_or_suggestions": "Typo:\n\"Large Language Models (LLMs) play an importent role in SNS...\" --> \"important\"\n\nSuggestion:\nImprove justification for task selection. Some tasks, like Note-Gender, seem less well-defined in terms of real-world SNS applications.", "questions_for_authors": "- As the REDnote platform are mostly a Chinese community, did you translated the notes into English or you only selected notes in English? If you you performed translation, is there any performance difference before and after the translation?\n- Could you explain where the concepts of \"social networking\" and \"personalization\" may apply in the benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This authors presents SNS-BENCH, a benchmark designed to evaluate LLMs across a diverse set of social networking service (SNS) tasks. The benchmark covers eight task types derived from a real-world platform, and is used to evaluate over 25 LLMs. The authors also provide a detailed performance analysis, identifying specific challenges such as emotion understanding and multi-hop reasoning.

The motivation is timely and well-supported, and the breadth of tasks and models included makes the benchmark a useful contribution to the evaluation landscape for LLMs in social contexts. The authors’ responses clarified key design decisions, including the rationale for evaluating tasks independently and how implicit social and personalized signals were incorporated. 

That said, one reviewer raised valid concerns about the lack of explicit modeling of social networks and user histories, and about the narrow definition of personalization in the current setup. These concerns remain partially unresolved and point to directions for future work.

After reviewing the paper, author response, and discussion, I believe that the benchmark provides a valuable foundation for assessing LLMs in SNS-like scenarios, even if its scope is not yet comprehensive. I encourage the authors to further clarify their definitions and continue extending the dataset to better capture network dynamics and personalization in future iterations.
Decision: Accept (poster)