Perceive before Respond: Improving Sticker Response Selection by Emotion Distillation and Hard Mining

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 13:11:57

Abstract: In online chatting, people are increasingly favoring the use of stickers to supplement or replace text for replies, as sticker images can express more vivid and varied emotions. The Sticker Response Selection (SRS) task aims to predict the sticker image that is most relevant to the history dialogue context. Previous researches explore the semantic similarity between context and stickers, while ignoring the role of both unimodal and cross-modal emotional information. In this paper, we propose a “Perceive before Respond” training paradigm. We perceive the emotion of stickers through a knowledge distillation module, which acquires emotion knowledge from the existing large-scale sticker emotion recognition dataset and distills it into our framework to enhance the understanding of sticker emotion. To further distinguish stickers with similar subject elements within the same topic, we perform contrastive learning at both inter-topic and intra-topic levels to obtain discriminative and diverse sticker representations. In addition, we improve the hard negative sampling method for image-text matching based on cross-modal sentiment association, conducting hard sample mining from both semantic similarity and sentiment polarity similarity for sticker-to-dialogue and dialogue-to-sticker. Extensive experiments verify the effectiveness of each proposed component. Ablation experiments on different backbone networks demonstrate the generality of our approach. The code is provided in the supplement material and will be released to the public.

Main paper: 
 introduction: 
 With the rapid development of instant messaging applications, stickers are widely used in online chats. Compared to simple emoticons (e.g., emojis), stickers are more expressive and superior in conveying strong emotion, positivity, and intimacy [22] . It plays an important role in assisting people to express emotions and regulating the emotional tendencies of the conversation [5, 45, 57] . As shown in Fig. 1 (a), the Sticker Response Selection (SRS) task uses the historical context of multi-turn dialogue to recommend the sticker. With the popularity of sticker usage, SRS receives widespread attention from researchers [4, 22, 30, 33, 40, 45, 55] . It contributes to users engaging in vivid, expressive online chat and holds promising prospects for developing anthropomorphic intelligent robots [1, 16] .Compared to general text-image retrieval tasks, the dialogue and stickers in SRS exhibit stronger emotional relevance [22] . We extract the sentiment polarity scores of stickers and dialogues separately to To better model and leverage emotional information, three challenges need to be addressed in SRS. (1) The varied visual concepts across different topics present a challenge for understanding emotions. As shown in Fig. 2 (a), stickers with different semantic content may express the same emotion, while those with similar semantic content may convey different emotions. This is known as the Affective Gap for visual emotion analysis [56] . Additionally, existing SRS datasets lack emotion annotations, which exacerbates the difficulty of extracting emotional information from stickers. (2) Modeling discriminative representations for stickers is challenging. Due to the distinctive organization structure of stickers [32] , images under the same topic are usually composed of the same subject elements and the extracted features are similar (e.g., Fig. 2 (b) ). This can hinder the SRS model from selecting appropriate stickers for the history dialogue, especially if the candidate stickers all belong to the same topic.(3) Most related works [12, 55] consider the SRS as a textimage matching problem, where the quality of negative sampling plays a crucial role in the performance of the multimodal model.Taking the negative sampling of dialogue-to-sticker as an example, as shown in Fig. 1 (c), the random selection approach leads to a uniform distribution of easy negatives and hard negatives. A widely used improvement approach is to select samples with high semantic similarity as hard negative samples [3, 24, 25] . However, in the SRS task, the difficulty of sample discrimination is also closely related to emotion. To further improve the quality of negative sampling, emotions are valuable which has been ignored in previous works.In this work, we propose a "Perceive before Respond" (PBR) training paradigm. To identify the emotions conveyed by stickers, we introduce an Emotion Knowledge Distillation (EKD) module, which utilizes emotion knowledge obtained from the large-scale sticker emotion recognition dataset SER30K [32] . Inspired by the ClusterFit algorithm [52] , we cluster the image features in SER30K, and call the centroid vector of each cluster Emotion Anchor. To describe images with diverse styles and semantic content in each emotion category, we set 𝑀 Emotion Anchor for each emotion category. For a sticker image, we compute its similarity to the Emotion Anchors to obtain emotion pseudo-labels. The image encoder learns emotional information guided by these pseudo-labels (e.g., Fig 2 (c) ). To better model the discriminative representations of stickers, we design a Topic-level Semantic Contrastive Learning (TSCL) module consisting of intra-topic and inter-topic two parts. For intra-topic contrastive learning, negative samples are selected within the same topic, aiming for the model to focus on the different local emotional features among stickers with similar subject elements. For inter-topic contrastive learning, negative samples are chosen from different topics, intending for the model to learn diverse representations of stickers across various topics (e.g., Fig 2 (d) ). Based on the consistency between the semantics and emotions of stickers and dialogue, we designed a Polarity-based Hard Sample Mining (PHSM) module. For hard sample mining in the text-image matching problem, we consider cross-modal sentiment polarity similarity in addition to semantic similarity. Samples with higher similarity in both semantic and sentiment are considered hard samples for the model to discriminate. Assigning higher weights to such samples enables the SRS to mine more informative negatives.The main contributions of this work are three-fold: (1) We propose the EKD module and TSCL module to exploit the emotional and semantic content of stickers and improve the quality of sticker features. (2) To the best of our knowledge, this is the first time that both dialogue and sticker emotions have been utilized in the SRS task. We design a PHSM module to mine hard samples by integrating the semantic and sentiment similarity between modalities. (3) The experimental results show that our method boosts the performance over the previous state-of-the-art methods by a large margin. Extensive ablations also verify the effectiveness of each designed module. 
related work: 
 Sticker Response Selection. With the development of online chat, the SRS task getting increasing attention in recent years [11-13, 23, 48, 55] . Laddha et al. [23] propose a sticker recommendation method that first predicts the next message, then replaces the predicted text with a sticker. Gao et al. [12, 13] propose a large-scale real-world dialogue dataset with stickers. The deep interaction network is used to match stickers and dialogue history, while the fusion network is used to fuse features and output the match score. As the extension work of [12] , [13] considers the user's sticker preferences by additionally recording the user's historical dialogue information with a key-value memory network. Fei et al. [11] model sub-tasks such as text generation and sticker prediction as general sequence generation. A unified generation network is then applied to retrieval stickers. Wang et al. [48] design a multimodal encoder for dynamic GIFs, training in a similar way to CLIP [41] .Multi-turn Dialogue Emotion Analysis. To recognize the sentiment of dialogue, some studies model the contextual dependencies between utterances. Li et al. [28] propose a bidirectional sentiment recurrent framework for contextual modeling and sentiment classification via a two-channel classifier. Yang et al. [53] introduce curriculum learning to train conversations from easy to hard. Ma et al. [34] detect the sentiment of dialogues from word and utterance level views. Some other studies focus on modeling interactions between speakers. Majumder et al. [35] model the speakers individually and use a recurrent neural network to track the state of each individual. Ghosal et al. [14] present the Dialogue Graph Convolutional Network to model dialogue context by exploiting both self-speaker and cross-speaker dependency. Ishiwatari et al. [21] add position encoding to relational graph attention networks, which capture both the dependencies between speakers and the sequential relationships between utterances.Knowledge Distillation. Knowledge distillation enables the small student model to learn knowledge from the large teacher model [19] . The offline distillation with the flexibility to choose pre-trained teacher models is more concerned with knowledge transfer. For knowledge design, Hinton et al. [19] use soft labels predicted by teacher models as category probabilities to train student models. Romero et al. [43] use the middle layer features of the teacher model as hints to distill a deeper and narrower student model. Zhang et al. [54] propose wavelet knowledge distillation to extract only the high-frequency band of images after discrete wavelet transformation.In terms of loss function design for feature alignment, Li et al. [27] align the output of 1 × 1 convolutional layers at the end of each block of the student network and the output of the teacher network to achieve fast fine-tuning. Mirzadeh et al. [36] add Teacher Assistant to mitigate the gaps between the teacher model and the student model.Hard Sample Mining. Informative hard samples can improve the model's performance [6] . Sample selection and sample reweighting are commonly used methods. Robinson et al. [42] prefer to select informative negative pairs with similar representations in contrastive learning. Lin et al. [31] reduce the weight of simple samples to solve the foreground and background imbalance problem in object detection. Different tasks have different measurements of sample difficulty. For example, long sentences are usually considered hard samples in NLP [39] , while images containing more objects are considered hard samples in semantic segmentation [49] . In image-text retrieval models [25] , hard samples are typically defined as samples with high contrastive similarity. For the SRS task, we introduce the sentiment polarity of dialogue and sticker, measuring the sample difficulty from both sentiment and semantic views. 
methodology 3.1 framework overview: 
 As shown in Fig. 3 , our method mainly consists of an image encoder, a text encoder, and a multimodal encoder. We will first explain the problem definition of SRS and then introduce the details of each component.Problem Definition. Given a dialogue history consisting of utterances from multiple users and a set of candidate sticker images, the Sticker Response Selection task aims to comprehend both the semantic and sentiment information in the dialogue history and select the best matching sticker. Formally, for the given dialogue 𝑈 = {𝑢 1 , ..., 𝑢 𝑁 𝑈 } with 𝑁 𝑈 utterances and 𝑁 𝑆 candidate sticker images 𝑆 = {𝑠 1 , ..., 𝑠 𝑁 𝑆 }, our goal is to train a ranking model 𝑓 that assigns the highest score to the ground truth sticker:EQUATIONNote that each candidate set has only one ground truth sticker, and assume that its index is 𝑝𝑜𝑠. Image Encoder. For a fair comparison, we employ three backbone networks as image encoders, i.e., Inceptionv3 [44] , ResNet [18] , and ViT [10] . For each input 𝑈 and 𝑆 mentioned above, where Text Encoder. We also use four different backbone networks to obtain the dialogue representation, i.e., LSTM [20] , GRU [7] , Transformer [47] , and BERT [9] . For the dialogue history, we combine each utterance and use the [SEP] token to indicate their separation. Then we extract the text representations W = {𝑤 𝑐𝑙𝑠 , 𝑤 1 , ...𝑤 𝑁 𝑇 }, 𝑤 𝑖 ∈ R 𝐶 𝑇 , where 𝑁 𝑇 is the number of words, 𝐶 𝑇 is the dimension of word embedding. Specifically, for LSTM and GRU, we also use the average of all word features as 𝑤 𝑐𝑙𝑠 .𝑠 𝑖 ∈ R 𝐻 ×𝑊 ×3To obtain compact multi-modal features, following [25] , we employ contrastive learning to align visual and textual representations before fusing them. We first map the features of vision and language modalities to a common embedding space:EQUATIONwhere𝑊 𝐼 ∈ R 𝐶 𝑉 ×𝐶 , 𝑊 𝑇 ∈ R 𝐶 𝑇 ×𝐶 .For image-to-text alignment, suppose a mini-batch has 𝐾 samples, and for the 𝑠 𝑝𝑜𝑠 in the 𝑖-th sample we combine it with 𝑈 𝑖 to form a positive pair. The remaining 𝐾 -1 dialogues are combined with 𝑠 𝑝𝑜𝑠 to form the negative pairs. We use the InfoNCE loss [37] to optimize our model, which is defined asEQUATIONFor text-to-image alignment, the negative samples of text 𝑈 𝑖 are not other stickers from the same batch, but negative sample stickers from the candidate set 𝑆 𝑖 :EQUATIONwhere both v and ŵ are [CLS] tokens in Equ. (3) (4) , and we omitted the subscript 𝑐𝑙𝑠 for simplicity. Multimodal Encoder. The multimodal encoder uses the last six layers of the BERT. The difference between it and the text encoder is the application of the Cross Attention (CA) operation in each layer. The CA is similar to Multi-head self-attention, but the Key and Value embeddings are from the image, and Query embedding is from the text. Finally, we use the [CLS] token output from the multimodal encoder to predict the matching score between each sticker and dialogue. We model that as a binary classification task and optimize it using cross-entropy loss:EQUATION)where P is the predicted probability, 𝑛𝑒𝑔 denotes the negative samples, and we will elaborate on the negative sample selection strategy in the subsequent sections.Sticker images have potential emotion information. We extract sticker emotion knowledge from a recently proposed sticker emotion recognition dataset SER30K [32] and transfer them to our image encoder. Formally, we chose ResNet [18] In the training phase, we first extract a feature representation V 𝑒 for each new sticker image using the teacher model mentioned above. Then its dot product result with Emotion Anchor is processed by softmax as the emotion pseudo label. To achieve emotion knowledge distillation, we minimize the KL divergence between the image encoder output and the emotion pseudo label:EQUATIONwhere 𝑊 𝐸 ∈ R 𝐶 𝑉 ×7𝑀 is a linear transformation.Stickers under the same topic have similar subject elements, which hinders the model from learning discriminative features. To address the challenge, we propose an intra-topic semantic contrastive learning to pull apart the image representations under the same topic. On the other hand, image representations under different topics should be naturally diverse, thus we also designed the inter-topic semantic contrastive learning branch to enhance such diversity. 1) Intra-Topic Semantic Contrastive Learning. As mentioned in Fig. 2 , sticker images under the same topic have similar subject elements, and traditional contrastive learning based on semantic labels hinders the model from learning discriminative features. However, stickers in the same topic usually express diverse emotions. we perform contrastive learning within the topic to pull apart image representations while improving the ability of the model to capture local emotional features.For the sticker images under the same topic, we perform two parallel data augmentations and get the output of the image encoder as V 𝑎𝑢𝑔1 and V 𝑎𝑢𝑔2 . We first project the [CLS] token features 𝑣 𝑎𝑢𝑔1 , 𝑣 𝑎𝑢𝑔2 of each sticker to a semantic representation space using a nonlinear transformation 𝜓 (i.e., MLP with a ReLU function). Then we utilize the images under the same topic as negative samples and define the learning objective of intra-topic semantic contrastive learning as follows:input :𝑆𝐼 𝑀 𝐼 2𝑇 ∈ R 𝐾 ×𝐾 , 𝑆𝐼 𝑀 𝑇 2𝐼 ∈ R 𝐾 ×𝐾 𝑁 𝑆 , 𝑃 𝑇 ∈ R 𝐾 , 𝑁 𝑇 ∈ R 𝐾 , 𝐸𝑀𝑂 ∈ R 𝐾 ×7𝑀 inEQUATIONIn this manner, we pull apart stickers with similar subject elements in the semantic space, guiding the model to learn discriminative emotion representations.2) Inter-Topic Semantic Contrastive Learning. Sampling negative stickers solely from the same topic is monotonous. We do not only aim for discriminative sticker representations to be learned but also strive for diversity. To enhance the diversity of negative samples, we introduce inter-topic contrastive learning. In contrast to intra-topic, inter-topic contrastive learning selects candidate stickers under other topics in the same batch as negative samples.EQUATIONThe intra-topic and inter-topic contrastive learning together compose the TSCL, which not only guides the model focus on discriminative regions but also ensures diverse negative samples of contrastive learning.In this section, we propose a hard sample mining strategy based on the consistency of sentiment polarity. Stickers perform the role of emotion regulation in the dialogue. The sentiment polarity of dialogue and sticker tends to be consistent as verified in Fig. 1 . From the process of image-text alignment (i.e., Equ. (3-4)), we can obtain the semantic similarity between image and text modalities:EQUATIONFor each dialogue, we employ an off-the-shelf Chinese text sentiment analysis library, named cnsenti1 , to extract the sentiment polarity score of each utterance. And sum the positive and negative scores for each utterance as the score of the dialogue, denoted as 𝑃 𝑇 and 𝑁 𝑇 . For each sticker, we extract the polarity score using the image emotion distribution obtained from Section. 3.2 (i.e., 𝐸𝑀𝑂 = V 𝑒 𝐸). Then, we select negative samples based on the semantic similarity scores of sticker and dialogue as well as the sentiment polarity scores. We define a sample to be a hard one if the semantic similarity between the samples is greater and the difference in sentiment polarity is smaller. The above selection strategy is summarized in Algorithm 1.In the training phase, our overall objective function is the summation of the above-mentioned formulas: (10) where 𝛼, 𝛽, 𝛾 are trade-offs between the individual objective functions. We set 𝛼 = 𝛽 = 𝛾 = 0.5 empirically.L 𝑡𝑜𝑡𝑎𝑙 = L 𝐼 2𝑇 + L 𝑇 2𝐼 + L 𝑆𝐷𝑀 + 𝛼 L 𝐸𝐾 𝐷 + 𝛽 L 𝐼𝑛𝑡𝑟𝑎 + 𝛾 L 𝐼𝑛𝑡𝑒𝑟 ,In the inference phase, we discard the Emotion Knowledge Distillation and the Topic-level Semantic Contrastive learning modules, and the model outputs ranked scores for each 𝑠 𝑖 based on the input 𝑆 and 𝑈 . 
experiments 4.1 dataset and metrics: 
 StickerChat [12] . The StickerChat dataset collects data from an online chat application. It has 3, 516 sticker topics containing a total of 174, 695 stickers. The 20 utterances before each sticker image are used as the history dialogue to build a dialogue-sticker pair. StickerChat contains 320, 168 pairs for training, 10, 000 pairs for validation, and 10, 000 pairs for testing respectively. The negative samples are constructed by 9 stickers which are randomly sampled from the ground truth sticker topic. DSTC10-MOD [11] . The DSTC10-MOD contains 45000 opendomain conversations with a total of 307 stickers. Following [55] , we only adopt the Chinese version of the DSTC10-MOD. Note that since the test set is not publicly available, we use the validation set to evaluate our methods and comparison methods.Metrics. Following the previous works [12, 58] , we use the 𝑅 𝑛 @𝐾 (𝐾 = 1, 2, 5) and Mean Average Precision (𝑀𝐴𝑃) to evaluate the proposed method. 𝑅 𝑛 @𝐾 measures whether the ground truth sticker is ranked in the top 𝐾 of the 𝑛 candidates. 𝑀𝐴𝑃 measures the mean of the average precision of the total samples. In the following tables, all metrics are shown in percentages, and we omit the % for brevity.During testing, there are 10 stickers in each candidate set, of which 9 negative samples are randomly selected from the set containing the correct sticker. In all of our experiments, we follow the same negative sample construction manner as [12] . When using LSTM [20] , GRU [7] , and Transformer [47] as text encoder, we use pre-trained word2vec embeddings [26] , and the max text length 𝑁 𝑇 is set to 256. The Transformer approach follows the setting of [12] . For BERT [9] , we use the first six layers of pre-trained bert-base-chinese 2 , and the max text length 𝑁 𝑇 is set to 512. For image features, we use the ALBEF [25] pre-trained ViT-B/16 [10] for encoding. For Incep-tionv3 [44] and ResNet [18] , we use the model parameters provided by timm [50] that is pre-trained on ImageNet [8] . The input image is randomly cropped and then resized to 128 × 128. The dimension of the text features 𝐶 𝑇 and image features 𝐶 𝑉 is set to 768, and the dimension of common embedding space 𝐶 is 256. We use the AdamW optimizer with an initial learning rate of 10 -4 and decay to 10 -5 by the cosine schedule. The batch size is set to 4, for a total of 200, 000 training iterations. And validation is performed every 5, 000 iteration. All the experiments are performed on NVIDIA GTX 3090 using PyTorch [38] .Table . 1 shows the comparison results. We follow the selection of the comparison methods by [12] . For a fair comparison, we divided 2 https://huggingface.co/bert-base-chinese the different methods according to the backbone network of image and text encoders. Meanwhile, in order to analyze the effect of external knowledge (i.e., SER30K and cnsenti), we additionally compared the methods after removing the EKD and PHSM modules (denoted as Ours), and the complete methods are denoted as Our*.Compared with the methods designed for visual question answering (i.e., Synergistic [15] and PSAC [29] ), Our method is specifically designed modules for SRS task, and considers emotional association between dialogue and stickers, thus achieving better performance. SMN [51] , DAM [58] , MRFN [46] and LSTUR [2] are the representative approaches for recommendation task, which are more focused on modeling text as well as semantic information contained in the conversation. However, SRS needs not only to reason about dialogue semantics but also to perceive cross-modal emotion information. That makes them unsatisfactory in terms of sticker selection. SRS [12] is designed for the SRS task. CLIP [41] and ALBEF [25] are generalized pretrained multimodal language models commonly used for retrieval tasks. The previous method achieved best performance of 76.8% on 𝑀𝐴𝑃 and 67.0% on 𝑅 10 @1. However, they also ignore emotional information. Our approach utilizes a more concise framework that extracts emotional information from both stickers and sticker-dialogue, resulting in a great performance, i.e., 2.4% in 𝑀𝐴𝑃 and 2.3% in 𝑅 10 @1. In addition, our methods achieved superior performance compared to other methods that used the same backbone network, where Ours* improved 𝑀𝐴𝑃 by an average of 2.32% and 𝑅 10 @1 by 2.86% over Ours on the five network combinations. It shows that emotional information is crucial for SRS and our method has excellent generalization capabilities across different networks.To the best of our knowledge, no previous SRS methods have been validated on both StickerChat and DSTC10-MOD. This may be due to significant differences between the two datasets, such as the number of stickers and the modality of the history dialogue (the history dialogue in DSTC10-MOD contains replyed stickers), which leads to deviations in the basic structure of the model. Therefore, on DSTC10-MOD, we adopt the SOTA method MMBERT [55] as the backbone and compose it with our proposed modules. Table . 2 shows the experimental results of our method on the DSTC10-MOD. Since the stickers in DSTC10-MOD do not have topic annotations, only EKD and PHSM are used in the comparison. And the PBR using only two modules surpasses the previous method. It is worth noting that there are only 307 stickers in the dataset. We suppose the model can learn more diverse emotional information to perform better when the sticker scale increases. In Table . 3, we compare the effect of the proposed modules. The ViT+BERT architecture is used as our "Base" model. It can be seen that the EKD, Intra-Topic, Inter-Topic, and PHSM modules all improve the performance compared to the base model. The TSCL module improved the performance on 𝑀𝐴𝑃 by 0.82% and on 𝑅 10 @1 by 1.41%, and the EKD module achieved a performance increase of 1.28% on 𝑀𝐴𝑃 and 1.99% on 𝑅 10 @1. We believe that the performance improvement of the aforementioned two modules can be attributed to the utilization of more effective sticker representations in our learning process. To verify the hypothesis, in Fig. 4 we visualize the sticker representations using t-Distributed Stochastic Neighbor Embedding (t-SNE). We can see that the sticker features extracted by the Base architecture are densely clustered and lack discriminative properties. The TSCL strategy allows sticker features to be dispersed in the semantic space, but images with the same sentiment polarity are still blended together. The EKD module allows the encoder to learn emotional knowledge and separate stickers from sentiment polarity. In the first five rows, we use semantic similarity to select negative samples for the matching task, while the sixth row employs the PHSM strategy, which selects hard samples based on the semantic similarity and differences in sentiment polarity of stickers within a batch. In the sixth row, we use the emotion pseudo label extracted by ResNet for hard sample mining (i.e., V 𝑒 𝐸), and the last two rows use the emotion labels predicted by the model (i.e., V𝑊 𝐸 ). We finally achieve a model performance of 79.19% in 𝑀𝐴𝑃 and 69.45% in 𝑅 10 @1 using all designed modules, which outperforms the Base model by 2.48% and 2.44%, respectively.Table . 4 shows the performance of the model with different cluster numbers 𝑀. The difference in 𝑀𝐴𝑃 between the highest point 𝑀 = 60 and the lowest point 𝑀 = 80 is 0.65% and the difference in 𝑅 10 @1 is 1.04%, which indicates the robustness of the model at different 𝑀. Furthermore, we found small variations in model performance at 𝑀 <= 50 and larger fluctuations after 𝑀 > 60. We conclude that this is due to the small sample size of some emotion categories in the SER30K dataset (e.g., 211 disgust, 826 fear, etc.). As the number of clusters rises, the clustering effect of these categories decreases, which in turn affects the cluster centroids.Table . 5 shows the performance comparison of different distillation methods. The table shows that feature distillation has the worst results, We conclude that since emotion is high-level semantic information, it cannot be fully understood using feature distillation. The logit distillation performed better on 𝑀𝐴𝑃 and 𝑅 10 @1 metrics. And our EKD module achieved the best performance, outperforming logit distillation by 0.65% and 0.75% on 𝑀𝐴𝑃 and 𝑅 10 @1, respectively. The number of soft target dimensions generated for each sample by logit distillation is equal to the number of emotion categories. Due to the topic structure of stickers, multiple characters will exist under the same emotional category. Therefore we use the ClusterFit approach to generate more fine-grained soft targets whose dimensionality is not restricted to emotional categories. It makes the sticker representations learned by our EKD module more diverse.Since our model can perceive the emotion of the stickers, the emotion intensity of the sample should influence the performance of the model. To verify this hypothesis, Fig. 5 illustrates the performance variation of the comparative method SRS [12] and our method as the sample sentiment intensity changes. We use continuous scores between 0 and 1 to discript sentiment intensity, where 0 represents negative and 1 represents positive. The left part shows the impact of dialogue sentiment intensity on model performance. SRS performance exhibits irregular fluctuations, whereas the accuracy of ours remains relatively stable. The right part shows the impact of sticker sentiment intensity on model performance. SRS still lacks a regular trend, while ours demonstrates a noticeable V-shaped pattern.When the sentiment intensity is intense, the 𝑀𝐴𝑃 and 𝑅 10 @1 of the model reach a maximum of 80.58% and 72.05%, respectively. But when the sentiment intensity is dim (i.e., in the middle interval), the 𝑀𝐴𝑃 and 𝑅10@1 of the model reach a minimum of 74.15% and 61.15%. This indicates that our model pays more attention to the emotion of stickers, and it is more likely to make the right selection when the sentiment intensity of stickers is strong. Furthermore, the lack of a clear relationship between the performance variation of SRS and sentiment intensity suggests that the previous methods do not effectively capture emotional information. Emotions need to be explicitly modeled to achieve optimal utilization.Fig. 6 shows the results of the visualization on the validation set. In the dialogue, sentences that contain sentiment tend to receive higher attention scores from the model. In stickers, regions that are given more attention by our model are typically critical for expressing emotions. For example, in the ground truth stickers, our model focuses on the gesture and surrounding symbols in (a), while in both (b) and (c), facial expressions are the main focus. In addition, there is a strong consistency between dialogue sentiment and sticker sentiment. The stickers with higher scores mostly have the same sentiment as the dialogue. It indicates that our model not only pays close attention to the sentiment conveyed in dialogues and stickers but also considers the cross-modal sentiment association. It was observed that some image sentiments are predicted incorrectly, which we attribute to the insufficient quality of the pseudo-labels generated by the teacher model. A better teacher model may be able to improve the model's performance. 
conclusion: 
 In this paper, we explore the utilization of emotional information on the SRS task. Intuitively, we need to perceive the emotion of the dialogue and sticker before sending the appropriate sticker. It is challenging to perceive and utilize unimodal and cross-modal emotional information. We consider both emotional and semantic information and design an Emotion Knowledge Distillation module as well as a Topic-level Semantic Contrastive Learning module to obtain discriminative and diverse sticker features. Besides, we improve the hard negative sampling method through the Polarity-based Hard Sample Mining module, which constructs hard negatives based on semantic similarity and sentiment polarity differences during the training phase. Extensive experiments prove the validity of each designed component. Our work provides an emotional perspective on the SRS task, and we also expect to facilitate future research on sticker recommendations.  
 
 Appendix: https://github.com/hiDaDeng/cnsenti 


Decision: This paper proposes a novel "Perceive before respond" (PBR) training paradigm to address sticker response selection (SRS), introducing an emotion knowledge distillation (EKD) module, topic-level emotion contrastive learning (TSCL), and a polarity-based hard sample mining (PHSM) method. The approach effectively leverages unimodal and cross-modal emotional information, with comprehensive experimental evaluations demonstrating its robust performance. However, the innovation of the EKD and contrastive learning modules appears limited, as similar challenges have been addressed in prior work. Additionally, the efficacy of visual representations and emotional categorization under different contexts remains inadequately validated. The reported improvements from EKD and PHSM modules are marginal, raising questions about their practical impact. Comparative experiments utilize outdated methods, and more recent multimodal large language models (MLLMs) should be included for a fair assessment. To address these concerns in the camera-ready version, it is recommended that a more detailed innovation analysis be provided, failure case studies should be included, and user subjectivity in sticker preferences should be considered.

Reviews: review: paper_summary: paper_summary: This paper proposes a "Perceive before respond" training paradigm to address the challenge of sticker response selection. This paradigm has three main contributions:
1. The paper introduces an emotion knowledge distillation module to extract sticker emotion knowledge from the SER30K sticker emotion recognition dataset, using the K-Means method to address the issue of single features inadequately describing emotion categories.
2. The authors propose topic-level emotion contrastive learning, including both intra-topic and inter-topic emotion contrastive learning.
3. The paper introduces a polarity-based hard sample mining method to improve the quality of negative sampling.
strength_weakness: strength_weakness: 1. This paper proposes a novel training framework that effectively assists models in utilizing both unimodal and cross-modal emotional information.
2. A hard sample mining method based on sentiment polarity is introduced to improve the quality of negative sampling.
3. The authors conducted comprehensive experimental evaluations and ablation analyses to demonstrate the effectiveness of the framework and the efficacy of individual components.
limitations: limitations: The methods used in the emotion knowledge distillation module and the contrastive learning module within the framework lack sufficient innovation
paper_summary: paper_summary: The paper presents a "Perceive before Respond" (PBR) paradigm for Sticker Response Selection (SRS), enhancing emotion understanding through knowledge distillation and improving sticker representation via contrastive learning and cross-modal sentiment association. However, there are some significant issues.
strength_weakness: strength_weakness: 1. The authors explore the utilization of emotional information on the SRS task.

2. The motivation and methods are easy to understand.

3. Experimental results demonstrate its robust localization and question-answering performance.
limitations: limitations: 1. The core contribution of the paper lies in addressing the three challenges in the sticker selection task, the novelty is limited, in fact, some work has already solved these existing challenges. For example, [1] has already pointed out that stickers vary greatly in style, leading models to learn robust representations for the stickers following various distributions. 

2. The author claims in the second challenge that the visual representations extracted from stickers under the same topic are very similar, which would lead to the model's incorrect predictions. I think this idea is intuitive and has not been well verified. The existing MLLMs would not mistakenly judge a smiling face and a crying yellow one in Figure 2 (b). Moreover, the same sticker can also convey different emotions. For example, crying stickers can express sadness in one context and be used to convey being touched in another.

3. Currently, the efficacy of generation models is widely acknowledged. Does the author consider generating stickers directly with their proposed methods, like [2] [3].

4. The novelty of the method is limited and not convincing; it merely involves a straightforward combination of existing modules. The ablation study results show that without the EKD and PHSM modules, which the authors highlighted in the Introduction as addressing key challenges, there is no significant improvement in the experimental outcomes (i.e., the results of R$_10$@1 only improve from 68.42 to 69.38, as shown in Table 3). This does not substantiate the effectiveness of the method or the validity of the challenges presented.

5. Upon a closer look at the comparative methods in the experiments, they are all too outdated. MMBERT merely utilizes Bert for this task in a straightforward manner, and ALBEF is a backbone model the authors use. Such a comparison is quite unfair. Methods based on the latest Multimodal Large Language Models (MLLM) should be considered for comparison.

6. What does the MMBERT* mean in Table 2? The authors should describe this in the caption of the table.

[1] Ge F, Li W, Ren H, et al. Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline[C]//Proceedings of the 29th International Conference on Computational Linguistics. 2022: 6795-6804.

[2] Zhang Y, Kong F, Wang P, et al. StickerConv: Generating Multimodal Empathetic Responses from Scratch. ACL 2024.

[3] Wang H, Lee R K W. MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation[C]//Proceedings of the ACM on Web Conference 2024. 2024: 4642-4652.
paper_summary: paper_summary: The paper introduces a novel approach, "Perceive before Respond" (PBR), for improving Sticker Response Selection (SRS) in online chatting. It addresses the challenge of predicting the most relevant sticker image in response to a dialogue by leveraging emotional information from stickers and dialogues. The proposed approach includes an Emotion Knowledge Distillation (EKD) module, a Topic-level Semantic Contrastive Learning (TSCL) module, and a Polarity-based Hard Sample Mining (PHSM) module. These modules enhance emotion comprehension, diversity, and discriminative features of stickers, leading to improved sticker response selection.
strength_weakness: strength_weakness: 1. The paper introduces a novel approach, PBR, which considers emotional information from stickers and dialogues, enhancing the understanding and selection of stickers in online chatting.
2. The EKD module effectively distills emotion knowledge from a large-scale sticker emotion recognition dataset, improving emotion comprehension in the model.
3. The TSCL module improves the diversity and discriminative features of stickers by considering intra-topic and inter-topic contrastive learning.
4. The PHSM module introduces a novel strategy for hard sample mining based on sentiment polarity, improving the quality of negative samples for the model.
limitations: limitations: 1. The paper mentions that existing SRS datasets lack emotion annotations, which may limit the generalizability of the proposed approach. It would be beneficial to explore how the approach performs on datasets with varied emotional annotations.
2. The paper provides extensive experiments to verify the effectiveness of each proposed component. However, it would be helpful to include a more in-depth analysis of the limitations of the approach, such as failure cases or scenarios where the approach may not perform as well.
3. While the approach aims to improve sticker response selection based on emotional relevance, it may be challenging to interpret or explain why certain stickers are chosen over others. User preferences for stickers can be subjective and may not always align with the model's predictions, which could impact user satisfaction and engagement in online conversations.
4. The reliance on the SER30K dataset for emotion knowledge may limit the generalizability of the approach to other sticker datasets or domains.
5. The paper focuses on the SRS task, and it's unclear how well the proposed method would perform on other related tasks or in real-world scenarios with varying sticker usage.