Integrating Stickers into Multimodal Dialogue Summarization: A Novel Dataset and Approach for Enhancing Social Media Interaction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 08:53:37

Abstract: With the popularity of the internet and social media, growing number of online chats and comment replies are presented in the form of multimodal dialogues that contain stickers. Automatically summarizing these dialogues can effectively reduce content overload and save reading time. However, existing datasets and works are either unimodal text dialogue summarization, or articles with real photos that respectively perform text summaries and key image extraction, and have not simultaneously considered the multimodal dialogue automatic summarization tasks with sticker images and online chat scenarios. To compensate for the lack of datasets and researches in this field, we propose a brand-new Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS) task and dataset. It consists of 5,527 Chinese multimodal chat dialogues and 14,356 different sticker images, with each dialogue interspersed with stickers in the text to reflect the real social media chat scenario. MCDSCS can also contribute to filling the gap in Chinese multimodal dialogue data. We use the most advanced GPT4 model and carefully design Chain-of-Thoughts (COT) supplemented with manual review to generate dialogues and extract summaries. We also propose a novel method that integrates the visual information of stickers with the text descriptions of emotions and intentions (TEI). Experiments show that our method can effectively improve the performance of various mainstream summary generation models, even better than ChatGPT and some other multimodal models. Our data and code will be publicly available.

Main paper: 
 introduction: 
 As social media platforms such as WeChat and TikTok continue to advance, people have become accustomed to inserting stickers into online conversations to express their thoughts [15] . Automated summaries of these multimodal dialogues can allow users to quickly obtain key information without reading through cumbersome chat logs or comment dialogues, thereby reducing memory usage.However, due to the existence of various images, this task has become challenging. These images, commonly referred to as stickers, emoticons, emojis or memes 1 , can vividly express the user's emotions [9] and behavioral intentions [28] . Therefore, it is necessary to consider both the dialogue text and stickers when summarizing. As shown in Figure 1 , key contents reflecting speakers' feelings and actions such as "work out", "apology", and "go alone" can only be gleaned from the stickers. Without the stickers, it is impossible to create a complete summary without missing crucial information.Yet, existing multimodal summarization tasks are mostly confined to news [19] , teaching [27] or open domain [22] video summaries, or separate summaries of articles and images [44, 45] . There is little research on generating text summaries from both text and images. Not to mention in dialogue scenarios, which makes multimodal dialogue summarization even more difficult due to the specificity of data format, colloquialism and arbitrariness of text and images, and scarcity of suitable datasets.To fill the gap in this field, we propose the Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS) task, which focuses on researching automatic summarization of chat dialogues on social media that include text and stickers. We also propose MCDSCS dataset for research on this task. First, we select informative and meaningful sentences from a large amount of social media chat text, use suitable COTs [38] to generate dialogues with GPT4 from these sentences, and carry out automatic and manual screening. Afterwards, we download over 30k sticker images from various social apps, as well as from websites like Baidu and Microsoft, then manually select and insert them into dialogues by adding or replacing text. Finally, we use the most advanced GPT4-Vision model, combined with automatic and manual reviews, to get summaries that take into account multimodal information. The resulting MCDSCS dataset, as shown in Figure 1 , consists of 5.5k multimodal dialogues containing Chinese chat text and stickers. With the exception of a few short dialogues, most dialogues contain two or more stickers, with a total of 14.4k different sticker images.Meanwhile, we propose a new method, which adds Text descriptions of Emotions and Intentions (TEI) of stickers, to handle multimodal dialogue summaries. It integrates sticker information by inserting text descriptions that show the emotions and intentions expressed by the stickers into the original dialogues. Compared to methods that only consider dialogue text or multimodal models which integrate visual features, our method enables baseline models such as the zero-shot LLM (ChatGPT) and fine-tuned models (Bart, T5, etc.) to achieve better performance, which is even better than that of Vision Large Language Models (VLMs).Our main contributions are as follows:• We propose a new task MCDSCS. To the best of our knowledge, it is the first to perform text summarization on multimodal dialogues including stickers. • We propose the MCDSCS dataset, the first multimodal Chinese dialogue summary dataset that includes a large number of sticker images. • We propose a novel method, TEI, which outperforms other methods, models and VLMs on the MCDSCS task. 
related work 2.1 dialogue summarization: 
 At present, most dialogue summarization dataset is text-based. Gliwa et al. [11] constructed a dataset SAMSum by manually selecting and annotating available chat conversations. Chen et al. [4] extracted over 13k dialogues from online English listening and speaking exams and manually annotated the summaries. MEDI-ASUM [43] collected extensive media interview transcripts from NPR and CNN, the summaries and topic descriptions of which were used as summaries for each dialogue. In terms of Chinese dialogue summarization, CSDS [21] excerpted some dialogue from JD customer service dataset JDDC [3] , and summaries were made from both the user and customer service perspectives. They all collected existing dialogues in text form and obtained summaries manually. Currently there are many effective methods for text dialogue summarization. Most popular generative models have been researched for dialogue form text summarization and have achieved good results [6, 17, 24, 37] . Khalifa et al. [16] explored four different challenges of dialogue summarization: handling and differentiating parts of the dialogue belonging to multiple speakers, negation understanding, reasoning about the situation, and informal language understanding. Liu et al. [23] explicitly incorporated co-reference information into summarization models, solving the difficulties brought about by complex co-reference links in the dialogue. However, these methods can't effectively solve the task of dialogue summarization that involve images.Lying down, not feeling comfortable, so I'm not going The multimodal field has become increasingly popular in recent years. Video summarization tasks in various fields have received extensive research [1, 19, 22] . The summarization tasks of text and images are mostly in the form of multimodal summarization with multimodal output (MSMO) [44] , obtaining the summary of the text and the most important image among all. Zhu [45] proposed a multi-modal target function guided by multi-modal references, utilizing the loss of summary generation and image selection, and proposed a novel evaluation metric based on a joint multi-modal representation for MSMO. HCSCL [41] is a Hierarchical Cross-Modality Semantic Correlation Learning Model for learning the intra-modal and inter-modal correlations existing in multimodal data, along with a new dataset for the MSMO task, which includes related image annotations and image object label information. Different from MSMO, the MCDSCS task requires a text summary as output to summarize the multimodal content, which is more concise and suitable for online conversation scenarios. The How2 dataset [32] is a multimodal video tutorial collection with text subtitles and translations. Palaskar et al. [27] used the text and video frames in the How2 dataset to generate text summaries, which is similar to our task. The MREDDITSUM dataset [26] includes posts with images and text comments on the Reddit website, and proposes a cluster-based multi-stage summarization (CWS) method to generate text summaries of post content and images. However, since the images, content, and comment text in the posts can be independent of each other, and the text and images in dialogues are continuous and interrelated, the CWS method of staged summarization is not applicable to the MCDSCS task.Furthermore, the sticker images in MCDSCS come in a variety of styles and contain a lot of metaphors and irony, which significantly differ from the real photos and videos in other multimodal datasets. This also brings considerable challenges. 
the mcdscs dataset 3.1 dataset construction: 
 As shown in Figure 2 , we outline the complete process of constructing the MCDSCS dataset, starting from sentences to textual ) and sticker insertion (step 2), and how stickers can impact the summary (step 3). The English translations are provided in brackets. In the summary, orange text corresponds to information that can be obtained from both the original text and the replaced sticker, while pink text corresponds to content that can only be derived from the newly added sticker.dialogues, further to multimodal dialogues, and ultimately obtaining the overall summary. We collect over 15k Chinese chat sentences from the CSMSA [10] and xiaohuangji 2 datasets, both of which contain a large number of Chinese chat sentences. In order to obtain rich and meaningful dialogues, we require that the source sentences should not be overly simple and should contain sufficient information. We construct a syntactic dependency tree [36] for each sentence to select sentences that are rich enough in content to form a dialogue. Any sentence with a syntactic tree depth less than 2 is considered to lack sufficient content for dialogue construction and filtered out. In this way, we obtain about 7k sentences.We decide the number and length of dialogue turns based on the depth of the syntactic tree of sentences, such as 3 corresponds to short and 5 corresponds to medium. See Section 1 in Supplementary for detailed statistics. Then we add "Social Media" to the COT to control dialogue scenario, and incorporate characteristics of chat such as "Colloquial" and "Natural". We call on the API of GPT4 to generate dialogue for each sentence. We also add detection of whether it contains sensitive or violative information, and automatically filter out inappropriate dialogues during the generation process. For the generated dialogues, we ask GPT4 whether the dialogue has logical errors and whether the content is in line with social media chat. We manually modify or filter out dialogues that are illogical or poorly evaluated, so as to minimize the hallucination issues that LLM may cause. Eventually, after multiple rounds of screening, we obtain 5.5k chat dialogues. Figure Section 2 in Supplementary shows a typical COT example.Dialogue-To-Multimodal-Dialogue. We download a total of more than 30k stickers from software like QQ, WeChat, TikTok, as well as from Baidu and Microsoft search engines. We employ five Chinese language professionals who have experience with dataset annotation. Each annotator is randomly assigned 1.1k dialogues and 6k stickers. They are instructed to add at least two stickers to each dialogue with five rounds or more, and to try to avoid duplicate stickers to increase the diversity of stickers. The method of inserting stickers is shown in step 2 in Figure 3 . Sticker1 represents replacing the original text with a corresponding sticker, and Sticker2 indicates adding a suitable sticker directly while maintaining the coherence and fluency of the dialogue. After inserting the stickers, the five annotators exchange the multimodal dialogues they annotated and communicate to modify any inappropriate stickers. Finally, within a 2 https://github.com/aceimnorstuvwxz/dgk_lost_conv/tree/master/results total of 5.5k dialogues, we add stickers 14.7k times. After excluding duplicate stickers from the statistical data, a total of 14.4k distinct stickers are used.We call the GPT4-Vision-Preview's API to generate a summary that encapsulates the dialogue containing text and stickers. We then ask GPT4 whether it have taken into account the information from stickers, and give the obtained feedback and the original data back to the annotators simultaneously to determine if the summary needs adjustment. The results show that less than 3% of the data (158 pieces) need manual summary rewriting, mostly because the input stickers don't comply with openai policy. For this small portion of data, our annotators manually write multimodal summaries referring to the rest auto-generated summaries, and make modifications based on suggestions from the other four annotators. It also verifies that our obtained summaries nicely blended the key information from each modality. Our COT for getting summaries and a detailed example are shown in Section 3 in Supplementary.Figure 3 shows a typical example of how we construct the data. In step 1, we generate a dialogue from a single sentence using GPT4 and incorporate constraints on dialogue length, scenario, and characteristics through COT.Step 2 shows how we add stickers to construct multimodal dialogues. Step 3 illustrates the impact of adding stickers on the summary. "laugh happily" can be derived from both the sticker and the original text, while "congratulate and praise" is additional multimodal information introduced after inserting stickers, which differs from the summary of the original textual dialogue. This highlights the importance of stickers for the multimodal summarization and underscores the challenge and value of the MCDSCS task. Due to privacy protection and the difficulty in preservation, it is hard to excerpt chat dialogues from social media on a large scale. Therefore, the scale of MCDSCS is not large compared to other dialogue and news datasets. However, it is the only dataset that utilizes auto-generation for dialogue and summary acquisition. This method is very flexible and can be applicable to the construction of many kinds of datasets, saving a lot of time on manual collection and summarization. The average text length of MCDSCS is the shortest because many contents expressed in text are replaced by stickers, and it also aligns with the concise and colloquial characteristics of social media chats. This also explains why the text compression rate of MCDSCS is relatively high, as the summary still contains a lot of information derived from analyzing stickers.Statistics. Table 2 illustrates the distribution of stickers in our dataset. Over 90% of dialogues contain two or more stickers, with the maximum number of stickers in a dialogue being 11. We also research the occurrence positions of stickers in dialogues. It can be observed that a minority of dialogues start with a sticker, a certain number of dialogues end with a sticker, while almost all dialogues feature stickers interspersed throughout the conversation. The above statistics also reflect to some extent the real usage of stickers in social media platforms.Extractive-Oracle ROUGE, as a metric for measuring the Abstractiveness of summaries, has been used in many summarization datasets. We list the comparison of Extractive-Oracle ROUGE values between the MCDSCS dataset and several datasets in Table 3 . The scores for these datasets all come from their respective papers [7, 8, 26] . Our dataset is far more abstractive than others, which is a significant feature of autogenerated summaries, and also demonstrates that much of the content in our summaries comes from sticker images. This also aligns well with the casual nature of social media chats.We randomly select 300 images from stickers used by each annotator, totaling 1500 stickers. We calculate the CLIPScore and RefCLIPScore [13] for each image with their [29] , compute the relevance between text and images. RefCLIPScore, on top of this, also calculate similarity, adaptability between images and text, and their relevance with the provided references. We utilize the TEIs of stickers obtained in Section 4.3 as reference text for RefCLIPScore. We also randomly replace each sticker with 100 other stickers and calculate the scores with the current context and summary.Table 4 shows the scores, which indicates that the stickers we add have a close and logically interwoven relationship with their accompanying contextual text. Although the scores with the summary is slightly lower, considering that the summary contains more than one sticker and textual information, such results are still good. The high RefCLIPScore further confirms the close connection between the TEIs of images and the contextual dialogue and summary. When stickers are replaced by other random stickers, all scores decrease. This also indicates the excellent compatibility between the stickers added to our dataset and the original dialogues.Similar to prior works, we manually evaluate the Fluency, Consistency, Relevance, and Coherence of MCD-SCS, the Chinese dialogue summarization dataset CSDS, and multimodal post dataset MREDDITSUM, of which the task format is most similar to ours. We divide the evaluation into dialogues (posts) and summaries within the dataset. Fluency assesses the smoothness of each individual sentence. Consistency evaluates whether the dialogue (post) is consistent or contradictory and whether the summary aligns with the original data. Relevance assesses the relevance of the dialogue (post) to its scenario and whether the summary content is pertinent to the original data. Coherence evaluates the overall quality and logicality of all text and images.Considering the different sizes of the datasets, we randomly select 200 pieces of data from each dataset. We employ 30 graduate students majoring in literature to individually score these data on the four metrics, with scores ranging from 1 to 5. A higher score indicates better quality, up to a maximum of 5 points. Each reviewer is a Chinese speaker and has a good level of English and has passed the CET-6 exam with high scores.Table 5 shows the average human scores. Our dialogues and summaries exhibit the highest fluency and coherence, which reflects that the text in MCDSCS is fluent and the images are appropriate, effectively reducing the awkward chats and irrelevant responses in dialogues. Since CSDS consists of e-commerce customer service dialogues, the content tends to be singular, resulting in higher scores for consistency and relevance. However, the consistency and relevance of our summaries of multimodal dialogues are also excellent. This is also due to the powerful multimodal performance of GPT4, which makes the quality of the summaries very high. MREDDITSUM scores lower on fluency and consistency because posts and comments on the site often feature internet slang, and images, posts, and comments are relatively independent of each other, frequently resulting in disordered comment replies. 
experiments 4.1 task description: 
 The MCDSCS task, as shown in Figure 1 , requires a multimodal multi-turn dialogue as input, where each turn may consist of either a text sentence or a sticker image. The model is then required to output a single text sentence summarizing the key content of all the text and stickers in the entire dialogue.LEAD-𝑛 [33] selects the first 𝑛 utterances. LONGEST-𝑛 [11] selects the longest 𝑛 utterances. EXT-ORACLE-𝑛 [25] can be considered as an upper bound on algorithmically extracted summaries by selecting the 𝑛 utterances with the highest ROUGE scores computed against the summary. LexPageRank [5] ranks dialogue utterances by PageRank algorithm and extracts utterances in order until the length of the summary reaches the limit.In LEAD, LONGEST, and EXT-ORACLE, we set 𝑛 as 3, 3, and 2, respectively, which are chosen after experimenting with 𝑛 ranging from 1 to 5 to find the optimal selections. 4.2.2 Text-only Generative Models. We select several pre-trained language models widely used for text summarization: BART [18] is an encoder-decoder Transformer model pretrained on a large corpus using a denoising autoencoder task. We use the large version of BART. T5 [30] is a versatile pre-trained transformer-based model, formulating language tasks as a unified text-to-text problem. We use the base version of T5. Pegasus [40] is a pre-trained abstractive text summarization model, employing a pre-training objective called "Gap Sentences". We use the large version of Pegasus. LLaMA [35] is a collection of foundation language models ranging from 7B to 65B parameters. We use the LLaMA-v2-7B version with the Low-Rank Adaption (LoRA) [14] .(VG-Bart and VG-T5) [39] are used for the How2 video multimodal summarization task, with an extra visual layer added to the model to handle video embeddings. We use the CLIP model to extract feature embeddings for all stickers as a substitute, and train the visual layer using the sticker images and their text descriptions. Experimental setups for finetuned models are in Supplementary.Zero-Shot LLM. We select ChatGPT (GPT3.5), for zero-shot experiments, since ChatGPT has achieved remarkable results in most natural language processing (NLP) tasks and benchmarks [34] .4.2.5 Zero-Shot VLMs. We select LLaVA3 , Yi 4 and Qwen 5 as the VLM baselines. The model versions and experimental methods are described in Section 4 in Supplementary. Since VLMs can independently process sticker images, our TEI method is not applicable, so they are only used for comparison experiments.Sticker images can succinctly and vividly convey the speaker's emotions and intentions [10, 31] . The same text, paired with different stickers, can convey completely different overall information, as illustrated in Figure 4 . Simply looking at the text doesn't reveal the speaker's emotional inclination. However, when coupled with an sticker representing sadness, the multimodal content can be analyzed to see that it's complaining getting only 80 points in a We input stickers into the GPT4-Vision model, asking it to describe the sticker in terms of emotions and intentions it expresses. We replace the stickers with these descriptions in dialogues.We calculate the CLIPScore of all stickers and their TEI. Considering the conciseness of TEI, which typically consists of only a few words, the result (62.12) is already quite commendable, proving that the TEIs we got are closely related to the stickers.We refer to previous summarization work and use ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) [20] , Bleu-score with smooth-ing_function [12] and Meteor-score [2] 6 as evaluation metrics. 
results and analysis 5.1 extractive results analysis: 
 From Table 6 , it can be seen that the TEI method does not perform well for the LEAD and LONGEST algorithms, since these two algorithms use formalized methods to extract sentences and do not consider the text content. While the TEI is actually aimed at enhancing the understanding of the image and related to the 6 Next, we will use R1, R2, RL, Bl and Met as abbreviations. final summary, direct copying is not advantageous. For the EXT-ORACLE and LexPageRank, these two algorithms approximate the upper limit of extractive summary. Because these methods compute the relationship between the text and the final answer, their extraction is relatively intelligent and can filter out some irrelevant text. Thus, the results are improved. Extractive results are generally superior, reflecting that the summaries in our dataset closely align with the original dialogue text and stickers.Table 6 shows the scores of all generative models including Chat-GPT. It can be seen that adding TEI to each model will improve its performance significantly, making it superior to the baselines. Moreover, after incorporating TEI, performance of the Bart and T5 models outperform their own Vision-Guided models that has added visual layers and image features. We hypothesize that this is because directly embedding image features may introduce noise brought about by the background, lines, and irrelevant texts. In contrast, after being converted into TEI, the model can more easily understand the multimodal emotions and intentions from a textual perspective. In addition, apart from the base version of the T5 model, which may have a smaller model size and fewer parameters, the performance of other models after fine-tuning with TEI greatly surpasses that of ChatGPT, even though the performance of ChatGPT also significantly improves with TEI. This indicates that the TEI method is also highly effective in zero-shot scenarios. Additionally, comparing the experimental results of VLMs in Supplementary shows that VLMs do not perform well in multimodal dialogue summarization tasks involving stickers. This further demonstrates the challenging nature of the MCDSCS dataset.The overall results show that replacing stickers with TEI in dialogues yields effectiveness, indicating that the main role of stickers in social media is to express user's emotions and intentions. TEI is deemed to be an appropriate approach to fuse stickers and text. A asked B about a restaurant, expressing skepticism about B's praise for it. B acknowledged that the food quality there was high. A suggested going there together next time. B agrees and expresses great anticipation.A asked B if he had been to that restaurant. B replied that the restaurant is very good because of the high quality of the dishes. A suggested going there together next time. B agreeed.A asked B if he had been to the restaurant. B praised the restaurant for being delicious. A was a bit surprised by this, and B said it was because of the high quality of the dishes. A wanted to go together and B happily agreed.A asked B about a restaurant. B said it's quite delicious. A was a bit doubtful why. B praised that the food quality there was very high. A invited B to go together next time, and B expressed eagerness. Figure 6 displays summaries generated by some well-peoformed models for one instance in MCDSCS. The Bart model only summarizes the text, omitting all visual information and neglecting the emotional tendencies of speakers. The result significantly improves after incorporating TEI, which takes into account the visual information. VG-Bart mainly introduces visual features. Compared to the questioning expressed through text in Sticker-1, VG-Bart places more emphasis on the character's surprised expression. Also, it overlooks the intention conveyed by Sticker-2 and Sticker-3. Chat-GPT can fluently obtain a complete summary of the dialogue text, but it also misses the multimodal information of stickers.We randomly select 30 instances and corresponding summaries generated by the different models. We recruit 100 undergraduate students majoring in Chinese Language and Literature to vote to select the summary they believe that best met the evaluation criteria for each instance. Referring to existing work [26, 42] , we choose fluency, faithfulness, and overall as the evaluation criteria. Fluency represents whether the summary is fluent and natural, faithfulness measures the extent to which the summary conforms to the dialogue, and overall represents the overall user preference. As shown in Figure 5 , the summaries generated by our method received the majority of preferences in terms of faithfulness and overall quality. At the same time, it also outperforms other methods in terms of fluency. We believe this indicates that our method can get summaries that are more consistent with social media scenarios and better reflect the true feelings of speakers. 
conclusion and future work: 
 The use of stickers in social media dialogues is increasing. To address this, we propose the MCDSCS task and dataset, comprising 5.5k multimodal dialogues, 14.4k sticker images, and reviewed autogenerated text summaries. We also introduce the TEI method, which integrates text descriptions of stickers' emotions and intentions into dialogues, outperforming baseline models, multimodal models, and ChatGPT. Future work includes applying our dataset construction method to create multimodal dialogue datasets in various languages and tasks, and developing improved multimodal fusion methods for better sticker information extraction and integration. 
 
 Appendix: In this paper, we collectively refer to them as "stickers".https://github.com/haotian-liu/LLaVAhttps://github.com/01-ai/Yihttps://github.com/QwenLM/Qwen 


Decision: This paper introduces a novel Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS) task and dataset, addressing an important gap in current dialogue summarization research. The reviewers generally praised the novelty and importance of the task, the thorough dataset construction process, and its potential impact on future research. While some limitations were noted by different reviewers, including the need for comparisons with more recent models and questions about the TEI method's novelty (reviewer vRpX, Vqn3), the authors' rebuttal appears to have addressed many concerns satisfactorily. Notably, Reviewer 9wao upgraded their assessment from Borderline Accept to Weak Accept after the rebuttal, recognizing the dataset's importance for future research. The primary strength of this paper is its dataset contribution. I believe that it has the potential to be helpful for the ACM MM community. Given the overall positive reception, the contribution of the dataset, and the authors' good rebuttal, I recommend accepting this paper for publication at ACM Multimedia.

Reviews: review: paper_summary: paper_summary: This paper proposes a multimodal chat conversation summary with stickers (MCDSCS) task and dataset. The dataset consists of 5527 Chinese multimodal chat conversations and 14356 different emoticon images. The author intersperses emoticons in each text conversation to reflect the real social media chat scene, providing relevant data for multimodal conversations.
strength_weakness: strength_weakness: 1.Presentation: The purpose of the article is well presented. Currently widely used dialogue systems, such as GPT-4, lack dialogue summary datasets that include both expression images and text. The necessity is well argued. The structure and layout of the paper are good, and the stickers are vivid.

2.Novelty:  By expanding stickers in existing conversations, the single-modal dialogue corpus is expanded to a multimodal dialogue corpus, and the dataset construction method is novel.
limitations: limitations: 1.Novelty:  The TEI method proposed in the paper has no formal expression such as formulas, nor does it have a model architecture flowchart or a comparison with other existing methods. The proposed model solution may not be novel enough.

2.Experiments: There is little discussion on experimental analysis, and there is a lack of hyperparameter experimental analysis and related theoretical descriptions. For example, last year's Gemma model and this year's Llama-3, it is recommended that the author can demonstrate the superiority of the method in the latest model.

3.Flexibility:  Most of the image examples cited in the paper have text descriptions. For the dataset builders, if there is no text expression, there is a high probability that errors will occur when embedding the dialogue. If only an OCR is used for text recognition, will it be easier to recognize, and then retrieve the text from the image and text pairs with text descriptions, so as to achieve the situation where the image and text appear at the same time in the dialogue? The practicality of building from the dataset does not seem strong.
The dataset proposed in this paper adds 147,000 stickers to a total of 5.5k dialogues. Objectively speaking, the amount of data is insufficient for a large model dialogue corpus, especially for the training stage of a large multimodal dialogue model.

4.Reproducibility：The supplementary materials of the paper do not provide simple case codes, such as trained model files, and it is questionable whether they can be reproduced.
paper_summary: paper_summary: The paper proposes a new task called Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS) and introduces a corresponding dataset. The task focuses on generating text summaries for chat dialogues on social media that include both text and sticker images. The authors construct the MCDSCS dataset by generating dialogues using GPT-4, manually inserting sticker images into the dialogues, and generating summaries using GPT-4-Vision. The dataset consists of 5,527 Chinese multimodal dialogues containing 14,356 different sticker images. The authors also propose a novel method called TEI (Text descriptions of Emotions and Intentions) to incorporate the visual information from stickers into the summary generation process.
strength_weakness: strength_weakness: 1. The paper addresses a new and important task in multimodal dialogue summarization, which has not been extensively explored in previous research. The MCDSCS dataset is the first of its kind, providing a valuable resource for this task.
2. The authors employ a clever approach to automatically generate dialogues and summaries using large language models, which can be scaled to construct larger datasets with minimal manual effort.
3. The proposed TEI method is intuitive and effective, as it addresses the key role of stickers in conveying emotions and intentions in social media conversations.
4. The paper provides a thorough evaluation of the proposed method against various baselines, including extractive algorithms, generative models, and even the powerful ChatGPT model.
limitations: limitations: 1. The paper claims to have very good results. While they have done experiments with chatGPT, they did not use any of the SOTA and easily available baselines like CogVLM, LLaVA, etc.
2. The paper does not clearly define or justify the choice of evaluation metrics used, which can make it difficult to compare the results with other studies or tasks.
paper_summary: paper_summary: This paper proposes a new task called Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS), which aims to summarize a dialogue with several inserted stickers. The authors also build a new dataset for this task by first sampling informative sentences from a Chinese corpus, asking GPT-4 to complete the dialogues based on the initial sentences, and finally manually selecting and inserting stickers into the dialogues. The authors benchmark multiple summarization systems on this dataset and propose a TEI method that can help improve performance.
strength_weakness: strength_weakness: 1. The major strength of this paper is that the construction process of the newly proposed dataset is elaborated. The authors take many important factors into account, like the privacy issue, how to select good initial sentences, etc. They also have well-performed human quality checks in every step of this pipeline. All of the above makes the new dataset very reliable, and it contributes significantly to the multi-modal dialogue community.
2. Many meaningful statistics for this dataset are provided so that we can have a direct understanding of how this new dataset differs from other related ones.
3. The paper is well-written and easy to follow.
limitations: limitations: 1. One major weakness of this paper is that the authors do not include a sufficient number of models in their benchmark. For example, it is necessary to include VLM models like LLaVA, Qwen-VL, etc.
2. The proposed method, TEI, is quite trivial. I think it is more like an oracle one since it utilizes extra information generated by an advanced VLM (GPT-4-V), so the improvement is predictable.
3. Why, in Table 4, do the CLIPScore/RefCLIPScore not significantly drop when the stickers are switched to random ones, if we assume the original ones are highly related to the text?
paper_summary: paper_summary: This paper proposed a novel task and dataset, namely Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS). The dataset is generated using the GPT-4 model through chain-of-thoughts design and manual review to produce dialogues and extract summaries. Additionally, this paper proposes a novel approach that combines the visual information of stickers with text descriptions of emotions and intentions (TEI) to guide the language model in better adapting to the multimodal summarization task. Experimental results show that TEI improves the performance of the summary generative model in multiple metrics.
strength_weakness: strength_weakness: 1) The paper is simple and easy to understand.
2) The proposed method improves performance across multiple metrics.
limitations: limitations: Weakness:
1) The Chinese summary and the English summary in Figure 1 do not convey the same meaning.
2) From Figure 1 and Table 1, it is evident that the quality of the proposed dataset is not high.
3) There is insufficient analysis corresponding to the data in Table 2, and the information that can be derived from Table 2 is unclear.
4) The abstractive in Section 3.3 refers to what, and why is it considered that the more abstract the text, the better? Section 3.3 does not explain this.
5) In Table 5, why are Con. and Rel. bolded in the CSDS dataset?
6) In Section 4.3, what does TEI stand for? Why is there a mention of "their TEI" in line 692?
7) What does "the result (62.12)" mean in line 694?

Typos:
1. Line 495: Abstractiveness -> abstractiveness.
2. Figure 2: sentences->Sentences, Chain-Of-Thought Prompts ->Chain of Thought Prompts.
paper_summary: paper_summary: This is a paper focusing on the multimodal chat dialogue summarization task that integrates sticker information. The paper proposes a new task of multimodal chat dialogue summarization containing stickers  and contributes a corresponding dataset. In the dialogue, the intent and emotional information contained in the stickers can help understand the overall intent of the chatters and the emotional direction of the conversation. This part of the information is a very important part of the input for chat dialogue summarization. The perspective of the paper is interesting and reasonable.
strength_weakness: strength_weakness: 1.The paper's approach of enhancing multimodal dialogue summaries with the intent and emotional information of multimodal dialogue stickers is novel.
2. The dataset provided by the paper contributes to the research of tasks related to this field, and the statistical introduction and analysis of the dataset are clear and comprehensive.
3.The framework and dataset of this task can be extended to various social multimodal interaction scenarios involving stickers, such as posting and replying on social platforms with sticker images.
limitations: limitations: 1.The dataset analysis is thorough, and the motivation for using stickers' intent and emotional information is sound. However, the methodology is comparatively straightforward and lacks distinctive features.  The authors could summarize the essence and innovation of the method in 1-2 sentences; currently, it appears to simply describe stickers information obtained from GPT-4.
2. The models used in sections 4.2.2 and 4.2.3 seem relatively outdated. The performance of some recent open-source llms on this task should be more noteworthy compared to models like BART and T5.
paper_summary: paper_summary: This paper proposes a brand-new Multimodal Chat Dialogue Summarization Containing Stickers (MCDSCS) task and dataset. At the same time the authors also propose a novel method that integrates the visual information of stickers with the text descriptions of emotions and intentions (TEI). Experiments show that the proposed method can improve the performance of various mainstream summary generation models.
strength_weakness: strength_weakness: (1）It takes a lot of work to construct a new multimodal Chinese dialogue summary dataset MCDSCS.
(2) This paper proposes a baseline model to evaluate the dataset MCDSCS.
limitations: limitations: (1) The research is mainly based on Chinese social media platforms, and the stickers may contain specific cultural backgrounds and contexts, which may limit the generalization ability of the model and its application in other languages or cultural contexts.
(2) The paper proposes a new method for integrating visual information, but does not compare in detail with other existing methods of multimodal conversation summarization, especially those that also consider image and text information. The specific comparison with other mainstream multimodal dialogue summarization technologies, especially those that integrate image processing, should be described in detail, and the advantages and innovations of this method should be demonstrated.
(3) The article is too general in describing the technical details of the algorithm and model, and lacks the necessary theoretical support and explanation of parameter settings.