STICKERCONV: Generating Multimodal Empathetic Responses from Scratch (2024)

Abstract: Stickers, while widely recognized for enhancing empathetic communication in online interactions, remain underexplored in current empathetic dialogue research, notably due to the challenge of a lack of comprehensive datasets. In this paper, we introduce the Agent for STICKERCONV (Agent4SC), which uses collaborative agent interactions to realistically simulate human behavior with sticker usage, thereby enhancing multimodal empathetic communication. Building on this foundation, we develop a multimodal empathetic dialogue dataset, STICKERCONV, comprising 12.9K dialogue sessions, 5.8K unique stickers, and 2K diverse conversational scenarios. This dataset serves as a benchmark for multimodal empathetic generation. To advance further, we propose PErceive and Generate Stickers (PEGS), a multimodal empathetic response generation framework, complemented by a comprehensive set of empathy evaluation metrics based on LLM. Our experiments demonstrate PEGS’s effectiveness in generating contextually relevant and emotionally resonant multimodal empathetic responses, contributing to the advancement of more nuanced and engaging empathetic dialogue systems1.

heading: 1 Introduction
text: Increasing research indicates that utilizing stickers in online chats can effectively alleviate stress, augment personal happiness, and notably boost empathy (Kariko and Anasih, 2019; Akram and Drabble, 2022; Jiang et al., 2023). Prior studies on stickers primarily concentrated on sentiment analysis (Ge et al., 2022) and recommendation systems (Gao et al., 2020; Zhang et al., 2022), overlooking their vast potential in empathetic response generation. Most empathetic response generation tasks focus solely on textual modality (Fu et al., 2023; Lee et al., 2022; Jiang et al., 2023), yet stickers
1https://github.com/ZhangYiqun018/StickerConv

heading: It seems like you're really enjoying the party! The way you describe it as a kaleidoscope of colors and laughter, swirling together to create an atmosphere of pure joy, is so vivid and beautiful. It's like being caught in a whirlwind of
text: in chats convey more abundant and intuitive emotional information, enhancing the expressiveness and emotional depth of responses.
Integrating stickers with textual communication and interspersing stickers within the dialogue can yield more varied and superior-quality empathetic replies. A primary challenge in integrating stickers into empathetic response generation is developing a high-quality dataset to support this innovative multimodal communication. To address this, we leverage large language model (LLMs) for dataset construction. LLMs, with their extensive world knowledge and text processing capabilities (OpenAI, 2023; Touvron et al., 2023a,b), demonstrate near-human annotation abilities (Wang et al., 2023d; Pangakis et al., 2023). However, applying LLMs directly have limitations in empathetic tasks, excelling in responding to explicit human instructions but lacking proactivity, a critical aspect of empathy (Wu et al., 2023; Yin et al., 2023). Em-
7707
pathy necessitates understanding others’ emotions and the ability to actively express support and understanding (Zech and Rimé, 2005; Sharma et al., 2020). To mitigate this, we introduce a multi-agent system based on LLMs, Agent for STICKERCONV (Agent4SC). This system, through inter-agent interactions, utilizes stickers to simulate human-like dialogue scenarios. It not only generates text responses but also strategically selects suitable stickers, thereby effectively enhancing empathy.
Based on Agent4SC, we build a multimodal empathetic dataset, STICKERCONV, that comprises 12.9K dialogue sessions and 5.8K unique stickers. STICKERCONV boasts an average of 5.22 stickers per dialogue session, mirroring the sticker usage patterns observed in human communication. Figure 1 depicts an example of conversations in our dataset. To the best of our knowledge, this is the first multimodal empathetic dialogue dataset, with the particular utility of sticker as non-textual modal information to better facilitate empathy.
Although Agent4SC effectively generates multimodal empathetic responses, it is limited by expensive inference costs and specific sticker databases. To further advance the research on multimodal empathetic dialogue, we develop an end-to-end multimodal empathetic response generation framework, PEGS, with the ability to PErceive and Generate Stickers. Beyond the general ability to generate textual empathetic responses, PEGS receives multimodal inputs and autonomously generates stickers based on the emotional and contextual aspects of the dialogue at the appropriate moment. Furthermore, to simulate human communications on social media in the real world, our model supports interleaved multiple image and text inputs.
Misalignments between the modal quantities of predicted and golden responses can distort evaluation outcomes, and empathy is difficult to quantify due to its subjective nature. To address this, we propose a novel method for evaluating multimodal empathetic responses, focusing on empathy, consistency, and ranking. Utilizing the extensive world knowledge and anthropomorphic abilities of LLMs, this approach provides solid support for assessing multimodal empathetic replies.
In conclusion, the main contributions of this work are as follows: • We introduce an LLM-based multi-agent system,
Agent for STICKERCONV (Agent4SC), which integrates stickers into empathetic dialogues, ensuring contextual consistency, variety, and em-
pathy aligned with human interactions. Using Agent4SC, we create a multimodal empathetic dialogue dataset, STICKERCONV. • We design PErceive and Generate Stickers (PEGS), a multimodal empathetic response generation framework that intuitively incorporates stickers based on the emotional and contextual dynamics of the dialogue. PEGS adeptly processes multimodal inputs, generating empathetic textual responses and using stickers appropriately to enhance these responses. • We propose a method for assessing multimodal empathetic responses. It leverages LLM to evaluate the quality of these responses, with a specific focus on empathy, consistency, and ranking.

heading: 2 Related Work

heading: 2.1 Empathetic Response Generation
text: Empathetic response generation focuses on enabling machines to understand and respond to human emotions. The foundational EMPATHETICDIALOGUES dataset (Rashkin et al., 2019) and subsequent innovations like the empathetic listener mixture model (Lin et al., 2019) have significantly advanced this area. Large Language Models (LLMs) like ChatGPT have been explored for empathetic response generation, although their application remains limited (Lee et al., 2022; Zhou et al., 2023b). Challenges persist, especially in leveraging multimodal information for richer emotional engagement and in accurately evaluating empathetic responses due to their subjective nature. These obstacles highlight the ongoing need for research in effectively integrating LLMs and multimodal data into empathetic dialogue. Additionally, the subjective nature of empathy complicates its quantitative assessment (Fu et al., 2023; Lee et al., 2022), posing a further obstacle to the field’s advancement.

heading: 2.2 Large Multimodal Models
text: LLMs, such as ChatGPT (OpenAI, 2023), LLaMA (Touvron et al., 2023a,b), demonstrate powerful capabilities in dialog interaction and instruction following, and recent researches have extended LLMs to multimodal domains. Flamingo (Alayrac et al., 2022) exhibits promising zero-shot and few-shot multimodal understanding capability by adding a cross-attention layer to connect the frozen vision encoder with the LLM. BLIP (Li et al., 2023b; Dai et al., 2023), MiniGPT-4 (Zhu et al., 2023; Chen et al., 2023) and LLaVA (Liu et al., 2023b,a) bridge
the frozen vision encoder and the LLM through a small intermediate model. Koh et al. (2023) for the first time explore the mapping of the output of LLMs into the input space of the vision decoder, empowering LLMs with image generation capability. GILL (Koh et al., 2023), MiniGPT-5 (Zheng et al., 2023) align LLMs to frozen vision decoders through an encoder-decoder transformer, while SEED (Ge et al., 2023) employs a learnable Q-Former. In contrast to previous works which favor realistic images, our target is to generate stickers, which are abstract and exhibit distinct emotional tendencies.

heading: 2.3 LLM-Based Agents
text: LLM-based agents mark a major leap in AI, leveraging their capabilities for tasks like reasoning and interaction, as shown in recent studies (Wang et al., 2023b; Li et al., 2023a). They find uses across various domains, such as software engineering (Qian et al., 2023) and scientific inquiry (Boiko et al., 2023), highlighting their versatility. These agents can imitate complex human actions, partake in social interactions (Park et al., 2023; Tu et al., 2023), and replicate intricate scenarios like elections (Argyle et al., 2022), debates (Wang et al., 2023a), and consumer patterns (Wang et al., 2023c), illustrating their capacity to emulate human social dynamics. A notable innovation is their use in generating AI training data. Studies (Wang et al., 2023d; Peng et al., 2023; Zhou et al., 2023a) highlight their efficiency and cost-effectiveness in producing highquality training datasets, transforming AI model development.

heading: 3 Agent for STICKERCONV
text: Confronted with the pivotal challenge of a lack of datasets for multimodal empathetic response tasks, we craft our own dataset utilizing large language models (LLMs). Nevertheless, LLMs experience difficulties in grasping nuanced human emotions and initiating actions beyond explicit directives. These limitations render LLMs and large multimodal models (LMMs) less proficient in the generation of multimodal empathetic responses. In response to these issues, we introduce Agent for STICKERCONV (Agent4SC), a multi-agent system predicated on LLM, devised to mimic human conversational patterns. Figure 2 presents the overview of Agent4SC. By integrating multiple modules and the strategic use of stickers, Agent4SC aims to gen-
erate emotional and varied empathetic responses, thereby overcoming the inherent deficiencies of LLMs in empathetic engagements.

heading: 3.1 Profile Module
text: The profile module, comprising Persona and Situation components, underpins its foundation by defining users’ personality traits and behavioral patterns for empathetic interactions. Persona outlines users’ character traits, backgrounds, and experiences, while Situation details their current circumstances and emotional states.
To enrich and diversify user profiles, we initially create profiles with varied emotional distributions, then expand to 2,000 unique profiles using SELFINSTRUCT (Wang et al., 2023d) method. This approach results in a profile archive with a broad emotional spectrum, as shown in Figure 5, aiming to enhance the system’s ability to simulate humanlike responses in empathetic dialogues.

heading: 3.2 Tool Module
text: To adapt the SER30K dataset, with its 1,887 themes and 30,739 emotion-tagged stickers (Liu et al., 2022), for human-like sticker use in dialogues, we transform it into a tool. Integrating it into the agent system faced several hurdles: the inappropriateness of many stickers for empathetic dialogues, the lack of detailed content or emotional analysis in the single-emotion labeled stickers, and the contrast between the extensive SER30K collection and the limited, personalized collections humans use.
To make stickers an effective tool, we refine the SER30K through filtering, annotating, and knowledge extraction (detailed in Appendix B.1). Each sticker is enriched with three pieces of information: emotion, description, and recommendation. Subsequently, SER30K is segmented into smaller, emotionally balanced vector databases. A critical action, "Retrieve", is then designed for agents to efficiently use these databases by searching for stickers based on the communication context and desired emotion, with the system suggesting the top-K closest matches for selection.

heading: 3.3 Memory Module
text: In empathetic interactions, response quality hinges on communication coherence and personality trait consistency. Agents must learn and give feedback by observing during interactions and ensure longterm personality consistency. Following the Generative Agent (Park et al., 2023), we use short-term
updates of core traits from profiles and observations for immediate memory. For long-term memory, continuous interactions and self-reflection help maintain consistency.

heading: 3.4 Plan Module
text: The Plan module orchestrates agent behavior in empathetic interactions. To ensure that agents mimic human behavior in using stickers, we specially design three key actions to ensure the timely use of stickers, their coherence with the context, and their effective role in empathy. • Intention: Directed by the Profile module, the
agent assesses the use of emotive stickers, considering interaction history and observed data. • Query: When intending to use a sticker, the agent describes its emotion, shaped by the Profile module, observations, and the intent for its use. • Select: After querying its sticker database with the crafted query, the agent retrieves the top-K most relevant stickers. It organizes these stickers into a list detailing their emotion, description, and usage recommendation, from which it selects the most appropriate sticker.

heading: 3.5 Action Module
text: The Action module serves as the window and bridge for communication with the external world. It is responsible for receiving instructions and inputs from other parts of the system and executes
two main actions based on this information: • Message: The agent generates text responses
based on the agent’s Profile and Memory module. • Sticker: The agent engages in sticker usage as
dictated by the Plan module.

heading: 3.6 Manager Agent
text: The Manager Agent, a critical component in enhancing the performance of the Agent4SC, plays a significant role in ensuring interaction quality and dialogue consistency. This agent comprises two main modules: (1) Quality Assurance: This module plays a key role in maintaining high dialogue standards by evaluating sticker relevance to enrich conversations, regulating sticker usage to avoid misuse, and conducting quality checks focused on response length appropriateness and redundancy avoidance. (2) Process Control: This module takes charge of guiding the dialogue’s progression. It leverages historical interactions as a reference to make informed decisions on the optimal timing for concluding dialogues.

heading: 3.7 The STICKERCONV Dataset
text: Leveraging Agent4SC, we propose the construction of a multimodal empathetic dialogue dataset, STICKERCONV. In our dataset creation, we define two roles: User, the dialogue initiator with a profile from 2,000 generated personalities, and System, acting as a listener and empathizer. To mimic the
human use of stickers, the sticker dataset is divided into 100 vector databases, each ensuring a consistent emotional distribution of stickers. The top-K of Tool module is set to 10. For each dialogue session, both User and System access a randomly chosen vector database, mirroring human sticker usage and enriching sticker diversity.
STICKERCONV consists of 12,931 dialogue sessions, 67,505 stickers (unique 5.8K stickers) and 2K user personalities. Each session averages 5.22 stickers and 5.49 dialogue turns. Table 1 shows the statistical breakdown of the datset, segmented into training, validation, and test splits. Emotional label distribution analysis, as illustrated in Figure 4, highlights the differences in sticker usage between User and System, reflecting their unique roles. The frequency of sticker usage in the dataset is 0.42, which is close to the frequency with which humans use stickers (Yang et al., 2023). Our dataset is comparable in size to PhotoChat (Zang et al., 2021). To our knowledge, it is the first multimodal empathetic dialogue dataset, uniquely integrating stickers as non-textual elements to enhance empathetic communication more effectively. Further analysis of the STICKERCONV is detailed in the Appendix A.

heading: 4 PEGS
text: We devise a multimodal empathetic response generation framework, PEGS, with the ability to
PErceive and Generate Stickers. Figure 3 illustrates the architecture of our framework. With different strategies to generate images, we derive three models based on this framework: PEGSRet/Gen/RAG, which denote the retrieval, generation, and retrieval-augmented generation methods to provide images. Technically, we utilize ViT-G/14 from EVA-CLIP (Fang et al., 2023), QFormer from BLIP-2 (Li et al., 2023b), and a linear layer to encode images. Vicuna (Chiang et al., 2023), a widely used language model in LLMs, is employed for language modeling. Stable Diffusion (SD) (Rombach et al., 2022) is employed as the image decoder for sticker generation.

heading: 4.1 Multimodal Input Perception
text: With reference to existing works (Li et al., 2023b; Dai et al., 2023; Zhu et al., 2023), we convert the
multimodal inputs into feature vectors that can be solved by LLM. Specifically, each text token is embedded into a vector et ∈ R1×d, while each image is first encoded by a pre-trained vision encoder, and then an aligned feature vector ev ∈ R32×d is obtained via Q-Former and a linear projection layer.

heading: 4.2 Multimodal Output Generation
text: Expanding Vocabulary We extend the vocabulary V with an additional visual tokens set Vimg = {[IMG1], [IMG2], . . . , [IMG{r}]}. We denote the original word embedding matrix as E ∈ R|V |×d. For the embeddings matrix E∗ ∈ R|V ∗|×d of the extended vocabulary V ∗ = V ∪ Vimg, the embeddings Eimg ∈ Rr×d of the added special tokens are randomly initialized and the embeddings E of the original token are preserved:
E∗[0 : |V |, :] = E (1)
We split visual tokens into two sets, where the front k tokens for image retrieval and the back r−k tokens for image generation:
Vret = {[IMG1], . . . , [IMG{k}]} (2) Vgen = {[IMG{k + 1}], . . . , [IMG{r}]} (3)
where Vret is used in PEGS-Ret and PEGS-RAG and Vgen is used in PEGS-Gen and PEGS-RAG.
Text Generation Receiving multimodal inputs, the target is to generate joint sequences of text tokens and visual tokens {[IMG{i}]}ri=1. Specifically, the generated token can be represented as U = {u1, . . . , uk}, where ui ∈ V ∗. The loss function Llm is defined as:
Llm=− k∑
i=1
log p(ui|s, u1, . . . , ui−1; θ, Eimg) (4)
where s = {e(1)m , e(2)m , . . . , e(l)m } and m ∈ {t, v} denoting the modality.The original LLM weights θ are kept frozen, and we only update Eimg.
Image Retrieval For image retrieval, PEGS aligns the hidden states hret corresponding to Vret into the retrieval space by contrastive learning (Chopra et al., 2005). Wt ∈ Rd×e and Wi ∈ Rp×eto bridge the semantic gap and adjust the dimension. Cosine similarity is used to measure the similarity of the projection vectors:
sim(x, y) = (W Tt hret(x)) T (W Ti νϕ(y))
∥W Tt hret(x)∥∥W Ti νϕ(y))∥ (5)
where νϕ is the image encoder. The projection vectors are used to minimize the InfoNCE loss (Oord et al., 2018), which consists of text-to-image (t2i) and image-to-text (i2t) loss in a batch of N text-image pairs (xi, yi):
Lt2i=− 1
N
N∑
i=1
( log
exp(sim(xi, yi)/τ)∑N j=1exp(sim(xi, yj)/τ)
) (6)
Li2t=− 1
N
N∑
i=1
( log
exp(sim(xi, yi)/τ)∑N j=1exp(sim(xj , yi)/τ)
) (7)
Lret = 1
2 (Lt2i + Li2t) (8)
Lret is the loss used to optimize the projection layers for retrieval.
Image Generation We align the hidden states hgen corresponding to the output visual tokens Vgen into the input space of the image decoder. Specifically, we connect them through a feature mapper module, containing two linear layers and a 4-layer encoder-decoder transformer model with a learnable queries feature q. For the given image caption c (and its emotion e if available), our target is to minimize the MSE loss between their embeddings derived from the frozen pre-trained SD text encoder η and the projected representations:
Lgen = ∥θMapper(hgen, q)− η(c, [e])∥22 (9)
Retrieval-Augmented Image Generation Intuitively, continuing to do generation on retrieved images can extend the diversity of images while maintaining image quality, thus we explore retrievalaugmented generation. Specifically, we retrieve an image serving as a latent representation cI for augmenting the generation process. During image generation, hgen remains utilized as a condition.

heading: 4.3 Joint Learning
text: Pre-trained LLMs excel in text interactions yet struggle in empathetic conversations. We further utilize the constructed STICKERCONV for jointly fine-tuning of the entire model to achieve the capabilities of multimodal perception and generation. We train our model in an end-to-end manner, using LoRA (Hu et al., 2021) to synchronize the update of a limited number of parameters in the LLM with the input linear projection layers and the feature mappers. The overall loss function L can be represented as:
L = λ1 × Llm + λ2 × Lgen + λ3 × Lret (10)
where λ1, λ2, and λ3 represent hyperparameters. Lgen includes emotion e as an input. For PEGSRet, Lgen = 0, and for PEGS-Gen, Lret = 0. Implementation details can be found in Appendix C.

heading: 5 Evaluation Metrics

heading: 5.1 Text Metrics
text: To comprehensively evaluate the fluency, diversity and accuracy of dialogue generation, we utilize an array of broadly recognized text metrics, comprising BLEU (Papineni et al., 2002), Distinct-n (Dist-n) (Li et al., 2016) , ROUGE-L (ROU_L) (Kingma and Ba, 2017), METEOR (MET) (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015), BERTScore (BERTS) (Zhang et al., 2020).

heading: 5.2 Multimodal Metrics
text: We use MM-Relevance (MMr) (Feng et al., 2023) to assess the relevance between the predicted multimodal response and the golden response. However, MMr overlooks the frequency of multimodal replies, a critical aspect considering that text responses within a modality often show higher similarity than across modalities. This could result in models that rarely engage in multimodal responses receiving inaccurately high MMr scores. To address this issue, we impose penalties on models that are more inclined to generate text-only replies:
f -MMr. = (1− α(1− f)) · MMr. (11)
where α represents the penalty coefficient. In our experiments, we set α to 0.8. f denotes the relative frequency at which the model produces multimodal responses, f ∈ [0, 1].

heading: 5.3 LLM-based Metrics
text: LLMs are capable of grading similarly to humans, providing scores for both textual and sticker outputs, thus enabling a comprehensive multimodal evaluation system.
We introduce three LLM-based metrics: (1) Empathy: We assess empathy in model responses, both textual (Empathy-text, EMP-txt) and multimodal (Empathy-multimodal, EMP-mm), averaging scores from five independent scorings to reduce randomness and bias. (2) Consistency: Based on the context, we assign consistency scores for textual and sticker responses, marked as Consistency (CON), employing the same scoring method. (3) Rank: We compare responses of different models to the same context. The ranker organizes responses based on quality, empathy, and consistency, averaging across many possible ranking combinations to ensure fairness and objectivity.

heading: 5.4 Human Metrics
text: Considering the subjectivity of empathy and the complexity of multimodal responses, we establish seven detailed manual evaluation metrics: Sticker Generation Quality (StiGQ), Empathy Sticker (Es), Empathy Text (Et), Consistency (Con), Fluency (Flu), Informativity (Inf), Sticker Diversity (StiD).
Metrics are scored on a 1 to 5 scale, with higher scores denoting better performance. For fairness and consistency, all models are evaluated in identical contexts. The evaluation panel consists of five members who score models anonymously, unaware if they are assessing a baseline model. The human evaluation process randomly selects 100 dialogue sessions from the STICKERCONV test set. Information about the human evaluators is provided in the Appendix D.2.

heading: 6 Experiments
text: Experiments are conducted on STICKERCONV. We perform response predictions for all turns of each dialogue and consider all previous turns as context.

heading: 6.1 Implementation Details
text: Our models are pre-trained on LAION115M, a subdataset of LAION400M (Schuhmann et al., 2021),
and fine-tuned on STICKERCONV. More implementation details and hyperparameter settings are provided in Appendix C. The training procedure is conducted on 2 NVIDIA A6000 48G GPUs.

heading: 6.2 Baselines
text: We benchmark against Vicuna-7B (Chiang et al., 2023) and ChatGLM3-6B (Zeng et al., 2023) models, employing two experimental paradigms: Text Fine-tuning We fine-tune the models using text data from the STICKERCONV training set and then make predictions on the test dataset. Tool Learning We fine-tune the models with multimodal data from the STICKERCONV training set. We use stable-diffusion as a tool to teach the models how to utilize it for generating sticker responses (Tang et al., 2023). To ensure fairness, the tool models employ the same SD model as PEGS.

heading: 6.3 Result Analysis
text: Table 2 reports the results for text metrics. PEGS’s performance on the Dist-n is slightly lower than that of the Vicuna. This may be attributed to PEGS expanding Vicuna’s vocabulary to facilitate sticker perception and generation, impacting PEGS’s text diversity to a certain degree. In our experiment, the training objective of tool learning includes both text generation and tool calling. These two objectives may interfere with each other. As a result, the tool model’s text capabilities fall short of the text fine-tuning model’s. PEGS’s end-to-end structure integrates multimodal inputs and outputs, streamlines tasks with unified training objectives, and yields the best text results. PEGS-Gen utilizes fewer special tokens (32), thereby outperforming PEGS-Ret and PEGS-RAG in text metrics. These findings corroborate the efficacy of PEGS framework in generating text responses of high quality and accuracy.
Table 3 displays the results for multimodal metrics, where Freq represents the relative frequency of stickers replies from each model. Although Vicuna-tool achieves a high MMr (72.48), this result is largely due to its lower Freq (0.141), which
may not accurately represent its multimodal interaction capabilities. F-MMr is more robust than MMr because it accounts for Freq. PEGS excels in f-MMr, showcasing that its end-to-end structure, integrating text and stickers, achieves high consistency in multimodal reply generation.
Building on this analysis, f-MMr is demonstrated to be a more reasonable and robust metric, as evidenced in Table 3, which considers the relative frequency of multimodal responses. The observed frequencies of multimodal responses among all models are notable: Vicuna-tool’s frequency is exceptionally low at 0.141, while others are significantly higher, exceeding 0.8. In contrast, the System’s sticker usage frequency in the STICKERCONV is 0.42, providing a benchmark for comparison. These variances highlight a critical need for future research to focus on adjusting the multimodal response frequency and timing of models to better align with natural human interaction patterns.
Table 4 presents the evaluation results for the LLM. Relative to the baseline model, the two toollearning models outperform the text model in ranking. This underscores the pivotal role of stickers in enhancing empathetic communication. According to the results from LLM-based metrics, PEGS can produce empathetic texts while ensuring high consistency, and utilize stickers to enhance emotional resonance. Table 3 and PEGS-Gen’s EMPmm (4.353) indicates that the quality of the multimodal response directly correlates with the extent to which it enhances empathy. To ensure accuracy, scoring for each metric was repeated five times, and averages were calculated to determine the final scores. The results indicate that the EMP-mm of PEGS significantly outperforms the two tool models, likely due to its end-to-end framework that ensures greater consistency between text and stickers. This integration underlines the positive impact of text-sticker relevance on empathy, affirming the effectiveness of multimodal communication strategies in empathetic interactions.
Table 5 shows the result of human evaluation.
PEGS notably outstrips the two baseline models across most evaluative indicators, save for Flu, thus corroborating the efficacy of the PEGS framework. The Flu of PEGS (<4.5) is lower compared to ChatGLM3-tool (4.58), likely because its high Inf (>4.09) results in a slight reduction in Flu. According to Inf and Et, the amount of information is a crucial factor influencing empathy. PEGS-RAG excels in sticker diversity (3.8) but records the lowest score in sticker generation quality (3.37). This observation suggests that although the RAG strategy enhances sticker variety, it might also reduce the quality of sticker generation in this experiment. Analysis of StiGQ, Es, and StiD reveals that multimodal responses can enhance empathy, and this enhancement is positively correlated with the responses’ quality. When considering all evaluation of human metrics, PEGS-Gen stands out remarkably, partly because of its minimal use of special tokens and partly due to its proficient sticker generation capability (owe to end-to-end structure). These outcomes highlight PEGS-Gen’s exceptional ability to produce high-quality, emotionally resonant, and diverse multimodal responses.

heading: Metric StiGQ. Et. Es. Con. Flu. Inf. StiD.
text: Table 6 shows the results of the inter-rater reliability analysis conducted on the five scorers involved in the manual evaluation, using the Intraclass Correlation Coefficient (ICC) (WEIR, 2005) as the assessment criterion. High consistency was observed among human scorers in metrics like StiGQ, Es, Flu, and Inf. The ICC values for Et scores are moderate, likely due to the subjective nature of assessing empathy through text, which can vary widely based on individual life experiences. Similarly, the moderate ICC for Con may arise from the task’s multimodal context, where the interaction between text and stickers can influence the assessment. The ICC for StiD is poor (less than 0.4), primarily because it is a global metric with a
single score per model, leading to greater volatility. Table 7 presents the Spearman’s and Kendall’s correlation coefficients between various automatic metrics and human annotations. The EMP-txt metric calculates the correlation with the Et of human metrics, whereas EMP-mm corresponds to the Es. CON is matched with the Con, and Rank correlation is determined by averaging all scores from human evaluations. The f-MMr matches the average of Con, Flu, Inf, and StiD. Both CON and f-MMr exhibit the highest consistencies under two coefficients, achieving 0.87 and 0.90 for Spearman and 0.74 and 0.80 for Kendall, respectively. These results emphasize the precision and uniformity of these metrics in mirroring human assessments. EMP-mm metric exhibits substantial correlations, suggesting a significant alignment with human perceptions within these domains. Calculating the correlation between the results of LLM-based metrics and the results of human metrics demonstrates that with carefully designed prompts, LLM can effectively evaluate empathy tasks.
Further analysis is presented in Appendix D. We collect the case study in Appendix E.

heading: 7 Conclusion
text: We explored the concept of multimodal empathetic response and created the STICKERCONV, the first dataset specifically designed for multimodal empathetic dialogue. We developed the Agent for STICKERCONV, a sophisticated LLMbased multi-agent system capable of simulating human-like interactions using stickers, thereby creating multimodal empathetic responses. Building upon the STICKERCONV, we developed PErceive and Generate Stickers (PEGS), an advanced multimodal empathetic dialogue framework. This framework adeptly perceives and generates stickers, effectively enhancing the communication experience. Furthermore, we established a comprehensive set of evaluation metrics for multimodal empathy tasks based on LLM. We are confident this work will be a valuable asset in advancing research in the field of multimodal empathetic dialogue systems.

heading: Limitations
text: Although our study has achieved certain advancements, it is not without its limitations. Firstly, STICKERCONV is exclusively in English, implying that our model and evaluations might not be directly translatable to other linguistic contexts. Secondly, although we investigate the retrieval-augmented generation (RAG) variant of the PEGS, hoping to enhance the quality and diversity of sticker generation, the RAG version has demonstrated instability in generating quality stickers in practice. Despite these issues, the pursuit of diversity and quality in sticker generation means that the explorations into the RAG version remain valuable. It suggests a promising research trajectory: how to elevate the stability and variety of the generated content by refining algorithms or amalgamating additional technological solutions. Finally, from our experiments, PEGS and two baseline tool-model exhibit abnormal frequencies in the usage of stickers. Aligning the model’s sticker usage frequency and timing with human-like interaction patterns is an important area for future exploration.

heading: Ethics Statement
text: The STICKERCONV dataset, produced by the large language model (LLM), therefore sidesteps privacy concerns. Nonetheless, LLMs may inherently harbor biases, and both STICKERCONV and PEGS could inadvertently magnify these biases in their application. Despite our efforts to mitigate these biases by incorporating a diverse range of personality information into Agent4SC, it is important to acknowledge that biases intrinsic to LLMs cannot be completely eradicated. This inherent bias is an ongoing challenge that needs to be addressed continuously through iterative improvement and monitoring of the model. PEGS reveals its robust capability in creating multimodal empathetic replies, adept at grasping human emotions and generating pertinent multimodal content. Such abilities, if exploited, could potentially manipulate user emotions. To forestall these risks, we have implemented a mechanism for detecting NSFW content in the generation of multimodal content. It is imperative to underscore that, despite our model’s proficiency in aiding the understanding and response to human emotions, it is not intended to entirely supplant human emotional communication. Excessive dependency on AI for empathetic interactions might erode human empathic abilities and the cultivation
of emotional intelligence.

heading: Acknowledgements
text: We extend our sincere appreciation to all the anonymous reviewers whose insightful comments and constructive suggestions greatly enriched this work. This research is supported by the National Natural Science Foundation of China (No.62272092, No.62172086, and No.62106039).

heading: A Dataset Statistics
text: In our dataset creation, it is important to note that vector databases numbered 1 to 80 are exclusively employed for the creation of training and crossvalidation sets, whereas those numbered 81 to 100 are reserved for the construction of the test set. We configure the agent’s "select" action’s top-K parameter to 10.
Figure 4 shows the emotional distribution of stickers used by the User and the System. It reveals a striking trend: users have a significant preference for stickers that convey negative emotion, in contrast to the system. The system predominantly utilizes stickers to express neutral and positive emotion. This comparison not only reflects the distinct emotional expression preferences between users and the system but also highlights the system’s active and supportive role in interactions. Figure 10 and 11 present two examples from STICKERCONV.
Figure 5 showcases the emotion distribution within user profiles in the Agent for STICKERCONV. We have retained the raw emotional labels generated by the Agent4SC. The chart illustrates the percentage share of each emotion, offering insight into the prevalence of affective expressions within the dataset. Increasing personality distributions enhances the Agent4SC’s diversity, thereby helping to mitigate the inherent bias issue in LLMs. Figure 9 shows a profile example across various emotions.
Figure 6, a word cloud, visualizes the 200 most prevalent emotion-related words extracted from the STICKERCONV. The prominence of each term in the cloud is indicative of its frequency within the dataset. This figure serves to underscore the linguistic diversity and the emotional range encapsulated in the dataset.
B Implementation Details of Agent4SC

heading: B.1 Tool Module
text: Before advancing to the follwing process, we altered the background of the SER30K stickers from black to white. The detailed process of building the Tool module is as follows:
Filter Leveraging the LLaVa-v1.5-13b (Liu et al., 2023a), we craft a nuanced Chain of Thought (CoT) (Wei et al., 2023) procedure to meticulously scrutinize each SER30K sticker. This approach entails the model initially summarizing the sticker’s content, subsequently assessing its relevance to casual conversation, and finally determining suitability through iterative responses. Each sticker undergoes multiple evaluations to ensure accuracy.
Annotation Filtered stickers undergo a second CoT analysis to answer 5 questions about their content, emotions, usage contexts, tone, and any comedic or satirical elements. This detailed annotation process provides deep insights, structured as Q&A pairs for further use.
Knowledge Leveraging GPT-3.5-Turbo, we extract critical information from the Q&A pairs, generating a comprehensive understanding of each sticker, including its description, emotion, and usage recommendation.
Split We segment the filtered dataset into smaller vector databases, incorporating the original emotional labels alongside freshly acquired knowledge. Each database maintains an emotional sticker distribution that aligns with that of the original dataset.
Following the process described, we obtain a sticker database containing 10,648 stickers. The
metadata for each sticker includes the original emotion label, the description of emotion, the description of content, and usage recommendations. The sticker database is then evenly distributed into 100 smaller vector databases based on emotional balance.

heading: B.2 Manager Agent
text: During dataset creation, we limit conversation turns to a maximum of six. The manager agent is programmed to review the conversation every two turns, and to verify the consistency of each sticker used with the conversation context. We observe that with more conversation turns, the likelihood of the agent repeating previous statements increased. Upon detecting such repetition, the manager agent will transition the agent’s LLM to GPT-4 to prevent further repetition.

heading: C More Implementation Details of PEGS

heading: C.1 Pre-training
text: Multimodal Perception Our effort is built on BLIP-2 (Li et al., 2023b), which has captured vision-language knowledge from numerous aligned image-text pairs. Zhu et al. (2023) have proven the effectiveness that freezing the pre-trained vision encoder, Q-Former, and LLM, while only pre-training
a linear projection layer. We train the input linear projection layer using LAION115M.
Multimodal Generation Distinguished from realistic images, stickers have an abstract nature and rich emotional expressions. Generic text-to-image models perform well in various scenarios, but they are flawed in sticker generation. Constrained by limited sticker data, there are not enough imagetext pairs to turn the base text-to-image model to the style of stickers, and to pre-train for multimodal generation. As a remedy, we adopt a variant of CuteYukiMix2, a SD model trained on cute cartoon data, as the pre-trained text-to-image model. In the pre-training phase, our objective is to align the representations with the text embeddings, so we train our model on the captions from LAION115M.
The AdamW optimizer is adopted, and the cosine annealing scheduler is used to adjust the learning rate. Table 8 shows the hyperparameter settings in the pre-training phase.

heading: C.2 Fine-tuning
text: Multimodal Generation Pre-trained diffusion models (focusing on obvious features, such as char-
2https://civitai.com/models/28169? modelVersionId=163923
acters’ bodies, costumes) are insensitive to sentiment conditions, and it is difficult to directly perform end-to-end fine-tuning to learn emotion elements, we inserted a supervise fine-tuning (SFT) procedure on the image-text pairs with emotion to efficiently cause them to focus on emotion elements. Table 9 shows the hyperparameter settings for this phase.
Joint Learning We utilize the constructed STICKERCONV for fine-tuning of the entire model to achieve the capabilities of multimodal perception and generation. We train our models in an end-toend manner. The AdamW optimizer is adopted, and the cosine annealing scheduler is used to adjust the learning rate. Table 10 shows the hyperparameter settings in the joint-learning phase. In this work, λ1 = 1, λ2 = 1, λ3 = 1 in Eq 10.

heading: D Further Result Analysis
text: Table 10 presents the hyperparameters for finetuning baselines and PEGS. The end-to-end architecture of PEGS necessitates significant graphics memory during training, limiting it to smaller batchsize (4) and maximum lengths (768). Conversely, the baseline model accommodates larger batchsize (16) and maximum lengths (4096). These parameters suggest that PEGS can be further optimized for use on devices with larger graphics memory.

heading: D.1 Analysis of Multimodal Metrics
text: Table 11 shows the range and standard deviation of the ratio between f-MMr and MMr under different α values. As the α value changes from 0.6 to 1.0, the range and standard deviation of the f-MMr to MMr ratio both show an upward trend. Lower alpha values (e.g., 0.6) tend to encourage more text responses, while higher α values (e.g., 1.0) significantly favor multimodal responses. Based on this, we set α to 0.8 as a balance point, as it not only encourages multimodal responses but also maintains the discriminability of the scores.
In this paper, we do not use traditional image generation evaluation metrics, such as FID (Heusel
et al., 2017), IS (Salimans et al., 2016). These metrics are applied to evaluate the quality and diversity of realistic images. FID and IS are obtained by pre-trained Inception v3 (Szegedy et al., 2016) (pre-trained on realistic images) after certain processing. However, sticker possesses an abstract style and characteristics that do not match the application domains of FID and IS. Based on the above facts, we evaluate the quality and diversity of image generation by human evaluation.

heading: D.2 Analysis of Human Metrics
text: We note that Table 5 demonstrates an anomaly whereby the stickers generated by PEGS-RAG have a higher diversity but lower quality compared to PEGS-Gen. The primary factor is that the opensource nature and scale of the sticker dataset directly affect the performance of our image decoder. Due to the limited open-source sticker data available, we cannot train a sticker-specific image decoder from scratch; instead, we chose a pre-trained SD in a cartoon style, and then fine-tune it on the sticker-text data. PEGS-RAG performs well in terms of flexibility and diversity, as it generates content based on the retrieved stickers. However, the generation quality is limited by the training data. With the increase of open-source sticker data, our RAG model will improve its generation quality while maintaining diversity.
Figure 12 shows the complete questionnaire of human evaluation, which includes the detailed description and scoring criteria for each metric. We specifically recruited independent evaluators for the human evaluation, who were not involved in any other aspect of this work. The five scorers are graduate students in computer science, specializing in research on emotional reasoning or empathetic dialogue, with a thorough understanding of empathy tasks. During scoring, they are unaware of which model serves as the baseline and which is the PEGS.

heading: E Case Study
text: To further investigate the quality of the multimodal responses predicted by our proposed framework, we show an example of STICKERCONV test data in Figure 7. As we have seen, the textual responses generated by multimodal baselines and our models are consistent with the dialogue context in terms of content and emotion, demonstrating the effectiveness of fine-tuning based on STICKERCONV. All
models provide visual responses (stickers) within the given context. The difference is that PEGS has more pronounced emotional tendencies, which facilitates empathy.
To illustrate more straightforwardly the performance of our model in multimodal empathic conversations, we show several cases of user-chatbot interaction. As shown in Figure 8, these cases demonstrate the ability of PEGS to empathize with users and perceive the stickers provided by users, along with generating multimodal responses with vivid stickers. For the positive dialogue context, our model generates positive stickers to enhance interactivity and emotional expression. For the negative dialogue context, our model reassures the user by an appropriate sticker (the hugging behavior in Figure 8 right).

heading: F Prompts
text: All {format_instructions} in the prompt utilize the response of langchain3’s response_schema function implements format control. For detailed formatting information, please refer to our project’s open-source repository: https://github.com/ZhangYiqun018/StickerConv.

heading: F.1 Sticker Process
text: Figure 13 shows the prompt of sticker process. For each sticker, we subject it to a process of filtering, annotation, and knowledge extraction, ultimately yielding three pieces of information: description, emotion and recommendation.

heading: F.2 Sticker Agent Chat
text: Figure 14 shows the chat prompt, the summary encompasses the persona of the user or system, the current status, and the core traits generated based on memory. Figure 15, 17, 18 show the process of used sticker.

heading: F.3 Manager Agent
text: Figure 16 shows the prompt of manager agent. By using various "format instructions", this prompt can serve multiple purposes, including quality and consistency checks, among others.

heading: F.4 LLM-Based Scorer
text: Figure 19 illustrates the template for the preference rank scorer. This scorer integrates the outputs from all baseline models as well as PEGS, constituting
3https://www.langchain.com/
the {response_list}. It then proceeds to rank these responses and provide reasoning for the assigned rankings.
Figures 20 depict the templates for the consistency scorer. It is utilized for scoring purely textual responses.
Figures 21 and 24 present the templates for the empathy scorer. The methodology for processing responses of different modalities aligns with the approach described for the consistency scorer.

heading: F.5 Joint Learning
text: Figure 22 shows the prompt template for joint learning. We employed an instruction template similar to Vicuna v1.5. For every dialogue that uses a sticker, we append an <IMG> identifier.

heading: F.6 Tool Learning
text: Figure 23 presents the instruction prompts for tool learning. To instruct the LLM in utilizing the SD model for sticker generation, we transform the STICKERCONV to tool format. When the model requires invoking the SD model, it outputs "ASSISTENT Action" along with the corresponding prompt ("ASSISTENT Action Input"). We then feed this prompt into the pre-configured SD model and relay the outcomes back to the LLM as "ASSISTENT Observation".

heading: G Costing
text: Extracting knowledge from labeled stickers in Llava using GPT-3.5-turbo, which includes stickers in SER30K dataset, stickers generated by tool models and PEGS, costs approximately $130.
The construction of 2000 user profiles for Agent4SC with GPT-3.5-turbo incurred a total cost of $5.5.
Constructing the STICKERCONV dataset with GPT-3.5-turbo incurred a total cost of $438.53, covering training, validation, and testing sets, at an average of $0.03 per dialogue session.
According to our test set, evaluating metrics other than rank for each comparison model costs $22. Combining all model responses with the evaluation Rank metric costs about $60.

heading: Empathy Scorer (multimodal)

heading: Future Work
text: Aligning the model’s sticker usage frequency and timing with human-like interaction patterns is an important area for future exploration