Dual-path Collaborative Generation Network for Emotional Video Captioning

Authors: cheng  ye
weidong  chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:49:25

Abstract: Emotional Video Captioning (EVC) is an emerging task that aims to describe factual content with the intrinsic emotions expressed in videos. The essential of the EVC task is to effectively perceive subtle and ambiguous visual emotional cues during the caption generation, which is neglected by the traditional video captioning. Existing emotional video captioning methods perceive global visual emotional cues at first, and then combine them with the video features to guide the emotional caption generation, which neglects two characteristics of the EVC task. Firstly, their methods neglect the dynamic subtle changes in the intrinsic emotions of the video, which makes it difficult to meet the needs of common scenes with diverse and changeable emotions. Secondly, as their methods incorporate emotional cues into each step, the guidance role of emotion is overemphasized, which makes factual content more or less ignored during generation. To this end, we propose a dual-path collaborative generation network, which dynamically perceives visual emotional cues evolutions while generating emotional captions by collaborative learning. The two paths promote each other and significantly improve the generation performance. Specifically, in the dynamic emotion perception path, we propose a dynamic emotion evolution module, which first aggregates visual features and historical caption features to summarize the global visual emotional cues, and then dynamically selects emotional cues required to be re-composed at each stage as well as re-composed them to achieve emotion evolution by dynamically enhancing or suppressing different granularity subspace’s semantics. Besides, in the adaptive caption generation path, to balance the description of factual content and emotional cues, we propose an emotion adaptive decoder, which firstly estimates emotion intensity via the alignment of emotional features and historical caption features at each generation step, and then, emotional guidance adaptively incorporate into the caption generation based on the emotional intensity. Thus, our methods can generate emotion-related words at the necessary time step, and our caption generation balances the guidance of factual content and emotional cues well. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module.

Main paper: 
 introduction: 
 The widespread popularity of self-media platforms, such as TikTok, Twitter and Facebook, allows everyone to express their opinions and emotions through the created video content. Meanwhile, the created videos are easy to arouse a wide range of emotional responses from viewers due to their vividness and variability. Under such circumstances, in the face of the increasing number of videos online, emotion-based video understanding tasks have attracted more and more attention, including video emotional recognition [5, 14, 22, 47] , video advertising recommendation [7, 9, 18, 22, 28] , and emotional video captioning (EVC) [1, 34, 35, 41] . The EVC task is an emerging hot topic in multimedia (vision-and-language) communities, which requires not only understanding the factual video contents, but also recognizing the complex emotion cues contained in the video and incorporating the emotion semantics to generate captions.The EVC task is the extension of the traditional video captioning task [10, 42, 50, 51] . Most existing video captioning methods focus on mining factual semantics, such as identifying visual objects and their attributes, associating their relationships, and translating them into text descriptions. However, they neglect the intrinsic emotions conveyed in the video contents, which makes the generated sentences a bit boring and soulless [1] . In contrast, merely modeling factual semantics in EVC task is insufficient to fill the affective gap between videos and emotions. For example, for the caption "the little girl loses one baby teeth but feels a bit of delightful". The vision factual semantic for "lost teeth" and the emotional semantic "delightful" lead to a gap. Thus, the essential of the EVC task is to effectively perceive subtle and ambiguous visual emotional cues during the caption generation.To fill the affective gap between videos and emotions, existing emotional video captioning methods make their effort to perceive emotion by global emotion learning in the initial stage, and then combine it with video features to generate emotional descriptions. Despite the promising progress, they neglect two characteristics of the EVC task. Firstly, their methods neglect the dynamic subtle changes in the video's intrinsic emotions, making it difficult to meet the needs of complex scenes with various emotions. For example, as shown in Figure 1 , the emotion of the girl in the video changes from frustrated to happy. Existing methods overlook capturing these subtle emotional changes at different time periods of the video and fail to generate accurate multi-emotional related descriptions. Secondly, as they incorporate emotional cues into each generation step, the guidance role of emotion is overemphasized, which will make factual content more or less ignored when generating descriptions. Moreover, when previous EVC methods predict the wrong global emotions, their methods inevitably generate descriptions that are not closely related to the factual contents and have nothing to do with the correct visual emotion cues.To fill the research gap, we propose a Dual-path Collaborative Generation Network to adaptively generate emotional captions word-by-wordly while dynamically perceiving subtle emotion evolutions. Our dynamic emotion perception is not independent of the caption generation. At each generation step, in the dynamic emotion perception path, the subtle emotional evolution from the previous step is perceived according to the combination of historical caption features and visual features; whereas, in the adaptive caption generation path, our method adaptively generates emotional captions based on the evolved emotions. The proposed two paths promote each other by our collaborative learning, leading to a promising performance in emotional captions generation.In the dynamic emotion perception path, our method perceives subtle emotional evolutions for each generation step. Thus, on the one hand, our dynamic emotion perception can correct the emotion bias of previous steps, helping to generate the correct emotion related words at the necessary step; whereas, on the other hand, our method can adapt to real-world scenarios with diverse and changeable emotions, providing the most accurate emotional guidance for each generation step. Specifically, we propose a dynamic emotion evolution module, which first aggregates visual features and historical caption features to summarize the global visual emotional cues, and then dynamically selects emotional cues required to be re-composed at each stage as well as re-compose them to achieve emotion evolution by dynamically enhancing or suppressing different granularity sub-space's semantic.In the adaptive caption generation path, the captions generated by our method are not biased towards emotional cues or factual content. Instead, our methods perceive the correct emotions and adaptively generate emotion-related words at the necessary generation steps, making our captions vivid and diverse. To be specific, an elegant emotion-adaptive decoder is designed, which firstly leverages the alignment of the perceived evolved emotions and historical caption features to estimate the emotion intensity at the current generation step. If the perceived evolved emotions and historical captions are harmonious and unified, it means that emotionrelated words need to be generated at this step, and vice versa. Then, our method adaptively integrates the perceived emotions into the caption generation with the guidance of the estimated emotion intensity. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module in both quantitative and qualitative terms.In short, our main contributions are summarized as follows:• We proposed a novel Dual-path Collaborative Generation Network for emotional video captioning, which dynamically perceives emotion cue evolution while generating emotional captions. The proposed two paths promote each other by collaborative learning, facilitating the emotional captions generation performance.• In the dynamic emotion perception path, a dynamic emotion evolution module is designed, which dynamically recomposes emotion features to achieve emotion evolution at different generation steps, helping to correct the previous emotion bias and providing customized and accurate emotion guidance for caption generation.• In the adaptive caption generation path, we propose an emotion adaptive decoder that utilizes the alignment of evolved emotion and historical caption to evaluate the emotional intensity for the current state, which helps our method generate emotion-related words at the necessary time step, and balances the guidance of factual content and emotional cues well. • Experiments on three public datasets (e.g., EVC-MSVD, EVC-VE, and EVC-Combined) demonstrate the superiority of our framework, e.g., improving the latest records by +7.2%, +6.8% and 6.5% w.r.t. emotion accuracy, CIDEr and CFS, respectively, on EVC-VE. 
related work 2.1 visual emotional analysis: 
 In recent years, instead of only focusing on factual semantic, many researchers have attempted to understand the emotional cues contained in visual elements [6, 19, 37, 44, 57] . Existing visual emotional analysis can be divided into several categories: 1) emotion classification task based on categorical emotion states from the psychological view [29, 48, 54] . Yang et al. [52] explore the relationship between scenes and objects to build emotional graphs, and leverage GCN for reasoning to achieve emotional representation. Zhang et al. [55] integrate hierarchical information of images to generate high-quality emotional representation. 2) emotion recognition from human activity e.g., facial expression and action [8, 13, 15, 23, 25] . Wang et al. [43] utilize a self-attention mechanism over mini-batch and a careful relabeling mechanism to suppress uncertainties for facial expression recognition. Zhao et al. [58] propose a label distribution learning method as a novel training strategy for robust facial expression recognition. 3) emotion distribution learning from complex emotion analysis [20, 36, 49] . Yang et al. [53] propose a circular-structured representation based on emotion circles to promote emotion distribution learning. Jia et al. [20] employ a local low-rank structure to capture the local label correlations implicitly. In video-based emotion analysis, Mittal et al. [31] explicitly model the temporal causality using attention-based methods and Granger causality for time-series emotion prediction for multimedia content. Zhao et al. [56] leverage a hierarchical attention mechanism to achieve end-to-end video emotion recognition. Motivated by these works, we integrate emotional cues mining into the video captioning task to generate vivid descriptions.Early works usually leverage pre-defined emotion categories (e.g., anger, disgust, fear, happiness, sadness, or surprise) to output emotionally customized captions, while ignoring the emotional cues elicited by visual content. To break this, the emotional video captioning task has attracted interest in the community, which firstly performs emotional analysis on the visual content, and then inputs the emotional representation into the decoder to generate emotional descriptions. Wang et al. [41] release a new dataset of emotional video captioning. They design two independent prediction networks for fact flow and emotion flow, then use a weighted average for the output probabilities of the two modules to generate descriptions. Song et al. [34] propose a contextual attention network to recognize and describe the fact and emotion in the video by semantic-rich context learning. Song et al. [1] propose a novel treestructured emotion learning module to achieve explicit emotion perception. Song et al. [35] incorporate visual context, textual context, and visual-textual relevance into an aggregated multimodal contextual vector to enhance video captioning. However, existing works ignore the fact that emotions vary slightly across a video stream. Thus, in this work, we study dynamically changing emotion representations to generate fine-grained emotional captions. 
proposed method: 
 The framework of our proposed method is illustrated in Figure 2 , which mainly consists of three parts: 1) Video and Emotion Feature Extraction, 2) Dynamic Emotion Perception Path and 3) Adaptive Caption Generation Path.Video Encoder. The EVC task aims to generate emotion-dominated descriptions of input video streams. Following the setting of previous work [1, 34] , we firstly down-sample the given video to get a set of frames, and then, we leverage 2-D CNN (e.g., ResNet [17] pre-trained on ImageNet [11] ) and 3-D CNN (e.g., 3D-ResNeXt [16] pre-trained on Kinetics [3] ) to extract appearance features and motion features, respectively. Next, we utilize the Transformer block to further encode appearance features and motion features to obtain the final video features 𝑉 ∈ R 𝑁 ×𝑑 𝑣 , which can be formalized as: where [; ] is the concatenate operation, 𝐹 𝑎 and 𝐹 𝑚 are appearance features and motion features, respectively. The obtained video features 𝑉 capture factual semantics, e.g., objects and scenes, etc. Emotion Encoder. Following [1] , we leverage the tree-structured emotion learning to generate hierarchical emotion representations. Firstly, we obtain the video emotion category representations. Given video feature 𝑉 and emotion category dictionary 𝐷 𝑐 = {𝑐 𝑖 } 𝑁 𝑐 𝑖=1 , where 𝑐 𝑖 denotes the 𝑖-th category such as "love" and 𝑁 𝑐 denotes the number of emotion category. The emotion category representations 𝐹 𝑐 can be encoded by a Transformer block as:EQUATIONEQUATIONEQUATIONand the emotion category distribution 𝑃 𝑐 over 𝐷 𝑐 is predicted based on 𝐹 𝑐 by average pooling and linear mapping operation. We learn fine-grained emotion word representations based on 𝑃 𝑐 by masked attention mechanism. Given 𝑃 𝑐 and emotion word dictionary 𝐷 𝑤 = {𝑤 𝑖 } 𝑁 𝑤 𝑖=1 , we only consider the top-K emotional categories, and select the emotional words contained in these K categories to generate the emotion mask 0-1 matrix 𝐴 ∈ R 𝑁 ×𝑁 𝑤 . Then, we impose the mask 𝐴 onto 𝑉 and 𝐷 𝑤 through a transformer block to obtain the emotion word representation 𝐸 ∈ R 𝑁 ×𝑑 𝑒 :EQUATIONwhere 𝐸 is the global emotional cues based on video features and hierarchical emotion tree structure, which also is the initial state 𝐸 0 of our dynamic emotional perception path. Then, the emotion word distribution 𝑃 𝑤 over 𝐷 𝑤 is predicted based on 𝐸.In the dynamic emotion perception path, our method can adapt to complex scenes with diverse and changeable emotions, and in each generation step, it can perceive subtle emotional evolution and provide the most accurate emotional guidance for generation. Specifically, given video features 𝑉 ∈ R 𝑁 ×𝑑 𝑣 , the caption features of the historically generated captions 𝑌 𝑡 -1 ∈ R 𝑡 -1×𝑑 𝑡 , and current emotion features 𝐸 𝑡 -1 ∈ R 𝑁 ×𝑑 𝑒 , our module finds 𝐸 𝑡 -1 's diverse semantic information by element-level and subspace-level, and finally aggregates this information to reorganize features to obtain evolved emotion representations 𝐸 𝑡 ∈ R 𝑁 ×𝑑 𝑒 . 1) Dynamic Factor Generation. In the process of generating captions for videos, historical caption features represent the current generation state, while video content features reveal subtle emotional changes. We employ a weighted pooling approach on the caption and video combined features to effectively capture these changes in the current generation state. Specifically, we first project the video features into the same embedding space as the caption features and combine them to obtain the combined features 𝐶:EQUATIONwhere 𝑊 𝑣 and [; ] are learnable parameters and the concatenate operation, respectively. Then, we leverage the weighted pooling to reconstruct the combined features into the same embedding space with the emotion feature to obtain the extended feature 𝑌 𝑡 -1 :EQUATIONwhere𝑊 1 𝑐 ∈ R 𝑑 𝑡 ×𝑀 and𝑊 2 𝑐 ∈ R 𝑑 𝑡 ×𝑑 𝑒 are learnable parameters and 𝑀 is a hyper-parameter that controls the size of the feature space. Compared to the original feature, the extended features not only include the simple caption status, but also consider the changes in the emotion of the video content, which provides accurate support for element-level emotional evolution.2) Element-level Emotion Evolution. After obtaining the extended features, we construct the element-level emotion evolution by selecting the emotion sub-features that need to be reorganized. Human emotions are complex and changeable, and different caption generation stages should have different emotional focuses. Thus, it is necessary to eliminate the negative impact of irrelevant emotional elements. In order to select irrelevant emotional elements, we firstly calculate the cross-modality relevance to align the current emotion features 𝐸 𝑡 -1 and the extended features 𝑌 𝑡 -1 . Then, we concatenate the aligned extended features and emotion features, and project them to an N-vector to obtain the dynamic emotion factors 𝑅 𝑏𝑒 . Finally, we leverage 𝑅 𝑏𝑒 to get element-level evolved emotion feature 𝐸 𝑡 -1 which required to be re-composed next:EQUATION)EQUATIONEQUATIONwhere𝑊 𝑅 ∈ R 𝑑 𝑒 ×𝑁 is a learnable weight, and ⊙ is the element-wise multiplication. By selectively reorganizing the emotion elements based on their relevance to the dynamic emotion factors, the model can focus on the most relevant emotional aspects at each time step. for each 𝑘 𝑗 as:EQUATIONThen, the semantic information of each subspace is captured by the attention mechanisms, which can be formalized as:𝑂 𝑗 = Tanh(Atten 𝑒 (𝐸 𝑡 -1 , 𝑌 𝑡 -1 )| 𝑄:𝐸 𝑡 -1,{𝐾,𝑉 }:𝑌 where ⊙ is the element-wise multiplication. Reshape operation turns the emotion features back to the original size 𝑅 𝑁 ×𝑑 𝑒 . Thus, the semantics of different subspaces are reorganized to varying degrees. To merge the semantic information of different subspaces to generate new emotional features, we leverage another attention to generate the weights for each subspace 𝑅 𝑎𝑓 𝑡 ∈ R 𝑁 𝑘 :EQUATION)EQUATION𝑅 ′ 𝑗 = Atten 𝑤 (𝐸 𝑡 -1 , 𝑌 𝑡 -1 )| 𝑄:𝐸 𝑡 -1,{𝐾,𝑉 }:𝑌EQUATION)EQUATIONFinally, we apply the weighted value to each reorganized subspace features to generate evolved emotional features:EQUATIONwhere 𝑅 𝑎𝑓 𝑡,𝑗 is the 𝑗-th element of the 𝑅 𝑎𝑓 𝑡 . By capturing emotional changes at each generation step, our Dynamic Emotion Perception Path ensures that the generated captions are emotionally relevant and synchronized with the video content. Moreover, the multi-step nature of the emotion evolution modules allows our methods to the fine-grained analysis of emotional dynamics, enabling the generation of captions that accurately reflect the emotional nuances of the video content.There are different demands of emotional cues for different stages in emotional caption generation. To this end, in our adaptive caption generation path, we propose the emotion adaptive decoder (EAD) which adaptively adjusts the guidance intensity of emotional semantics when generating captions to avoid the loss of factual semantics caused by overemphasis on emotional semantics.1) Emotion Intensity Estimation. The historical caption features help determine whether emotional guidance is needed for the next generation step. For example, when generating adjectives or adverbs (e.g., surprised and angrily) in sentences, emotional features have strong guiding significance. Instead, when generating nouns (e.g., bike and hospital), factual semantics in video features have stronger guiding significance than emotional features. Thus we propose the emotion intensity estimation module that leverages caption features to estimate emotion guidance intensity at the current generation step. Mathematically, at the 𝑡-th time-step, given caption features 𝑌 𝑡 -1 ∈ R (𝑡 -1) ×𝑑 𝑡 and evolved emotion features 𝐸 𝑡 ∈ R 𝑁 ×𝑑 𝑒 , we firstly obtain emotion correlated text features T 𝑡 𝑒 by calculating the normalized relevance score to align each generated word 𝑦 𝑡 -1 𝑗 onto each emotion element 𝑒 𝑡 𝑗 . Then, we apply the gating unit to T 𝑡 𝑒 to obtain the emotion guidance intensity 𝑔 𝑡 . 𝑟 𝑡 𝑖,𝑗 = Tanh(𝑊 𝑟 𝑦 𝑡 -1EQUATIONEQUATION)EQUATIONwhere 𝑦 𝑡 -1 𝑖 represents the 𝑖-th features of the caption features 𝑌 𝑡 -1 , and 𝑒 𝑡 𝑗 is the 𝑗-th features of emotion features 𝐸 𝑡 . 2) Adaptive Caption Generation. Since previously generated words also help predict the next word, we obtain the visually correlated text feature T 𝑡 𝑣 like T 𝑡 𝑒 . At the decoding step t, we aggregate all the sufficient semantics and use emotion guidance intensity to adapt the features. Finally, we obtain the probability distribution 𝑃 (𝑦 𝑡 ) ∈ R 𝐷 of the next word 𝑦 𝑡 , which is defined as:EQUATIONEQUATIONEQUATION)where Decoder is a standard attention-based LSTM to generate the caption sentence step by step, 𝑊 𝑝 and 𝑏 𝑝 are learnable parameters.To ensure both the factual and emotional semantic correctness of the generated captions, we leverage two objective functions to guide the correct generation direction. Due to the importance of emotional words in captions, we leverage emotion-focused crossentropy loss L 𝑒 that adds a penalty term on emotional words, which ensures the correctness of generated emotional words:EQUATIONwhere 𝛽 is a hyper-parameter that control the level of punishment while 𝑦 𝑡 is a emotional word, like "happily". Additionally, the initial distribution of emotions is critical to the direction of emotion evolution. Thus we calculate the hierarchical initial emotional classification loss L 𝑐𝑙𝑠 based on 𝑃 𝑐 and 𝑃 𝑤 :EQUATIONwhere 𝑌 𝑐 and 𝑌 𝑤 denote the ground-truth emotion category and word, respectively. Finally, our model can be trained by minimizing the weighted sum of these two losses:EQUATIONwhere 𝜆 𝑒 and 𝜆 𝑐𝑙𝑠 are two hyper-parameters that aim to control the balance of two losses. 
experiments 4.1 datasets and evaluation metrics: 
 Dataset. We experiment on three public emotional video captioning benchmarks, e.g., EmVidCap-S [4] , EmVidCap-L [21] , and EmVidCap [41] . EmVidCap-S is built via additionally embedding emotion words into the caption sentences of traditional video captioning dataset MSVD [4] , which contains 240/134 videos and Evaluation Metrics. To evaluate the accuracy of emotion in generated sentences effectively, following previous work [41] , we consider the emotion word accuracy Acc sw [41] and emotion sentence accuracy Acc c [41] . Additionally, to measure the semantics information of generated captions, we use the standard metricse.g., BLEU [32] , METEOR [2] , ROUGE [26] , and CIDEr [38] . Moreover, there are two hybrid metrics BFS [41] and CFS [41] that combine the emotion evaluation with BLEU and CIDEr metrics, respectively.For a fair comparison, we extract appearance features by leveraging ResNet-101 [17] or ResNet-152 [17] and motion features using 3D-ResNext-101 [16] . Besides, we test the model with the modern CLIP features [33] . The maximum length of captions is set to 15. We build an overall vocabulary that contains all words in the corpus. The vocabulary sizes for the EVC-MSVD, EVC-VE, and EVC-Combined datasets are 9,637, 13,980, and 14,034, respectively. For emotion learning, there are 𝑁 𝑐 = 34 emotion categories and 𝑁 𝑤 = 179 emotion words. All word embeddings are extracted by GloVe. The feature dimension is set to 𝑑 𝑣 = 𝑑 𝑡 = 𝑑 𝑒 = 300. For dynamic emotion perception, the extension parameter is set to 𝑀 = 100 and the subspace number list is set to {2, 3, 5, 6, 10} to construct a complete semantic space since the dimension of emotion features is 𝑑 𝑒 = 300. The hidden size of the caption decoder is set to 512. We set the penalty parameter 𝛽, the objective function weight 𝜆 𝑒 and 𝜆 𝑐𝑙𝑠 to 0.1, 1 and 0.2. We adopt the Adam optimizer with a learning rate of 7e-4, and the batch size is set to 128.Comparison with State-of-the-Art Methods. As shown in Table 1 , we present the comparison of our model and SOTA methods on three EVC datasets with three different encoder settings, i.e., ResNet-152, ResNet-101+3D-ResNext-101, and CLIP. We evaluate the performance of our model from three types of metrics. Our proposed model outperforms the state-of-the-art methods for all metrics, which demonstrates the superiority of our method.The results of ablation studies on EVC-VE dataset to discuss on the effectiveness of our proposed components. A stands for our dynamic emotion perception; whereas, B represents the adaptive caption generation. "×" mark in B component means to leverage the decoder used in compared methods [1] . Accuracy Evaluation. Firstly, we evaluate the accuracy of our model in predicting video emotion categories. We observe that our method achieves the best performances on emotion metrics Acc sw and Acc c across the three datasets. For instance, on the EVC-VE dataset, the proposed method improves the performances by 4.8%/5.0% over EPAN [1] with ResNet-152 features on Acc sw and Acc c . On EVC-MSVD and EVC-Combine datasets, the improvements are 2.8%/5.8% and 3.9%/2.9%, respectively. With ResNet-101+3D-ResNext-101 features, the improvements are more obvious, i.e., 4.1%/5.6% and 5.1%/5.7% on EVC-VE and EVC-Combine dataset, since motion features can better capture changing emotional cues in videos. The remarkable improvements indicate that the emotion vectors can accurately capture changing emotions in videos.Semantic Evaluation. Semantic metrics can reflect the quality of the captions generated by the model. We observe that our method achieves the best performance on multiple semantic metrics across the three datasets. It is worth noting that when using the modern CLIP features, our proposed method significantly outperforms the state-of-the-art methods. For instance, on the EVC-VE dataset, our model surpasses EPAN by a large margin of 9.8%, 15.7%, and 19.0% on BLEU-4, METEOR, and CIDEr, respectively. This substantial improvement demonstrates the effectiveness of leveraging CLIP features in generating more accurate, vivid, and diverse captions.Hybrid Evaluation. In addition, we consider combining the emotion evaluation with BLEU and CIDEr metrics respectively to investigate the performance of our model. From the results in the last two columns of Table 1 , our model achieves the best results on the BFS and CFS metrics with modern CLIP features across EVC-VE, EVC-MSVD and EVC-Combine datasets. These show that our dualpath collaborative framework can mine the changing emotional cues implicit in the video and generate captions adaptively.To further demonstrate the superiority of our proposed dual-path collaborative generation network, we conduct ablation studies for two perspectives discussion.Discussion on our proposed two path. To further investigate the contribution of each path in our method, we conduct ablation studies, and the results are shown in Table 2 . We first observe that the dynamic emotion perception could improve the effect by 4.9%, 4.9% on emotion prediction accuracy, which demonstrates that the dynamically changeable emotional cues in the video are precisely captured. However, its improvement on the BLEU metric is slight, and especially, there is even a decline on the BLEU-1 metric. A possible reason for these observations is that traditional caption generation framework incorrectly exploits dynamically changing emotion representations, leading to ambiguity in the generated descriptions. Thus, we propose the adaptive caption generation module in this work, which could dynamically estimate the intensity of emotion representations based on the alignment of the historical caption features and video features. Then, based on the estimated emotion intensity, we generate words adaptively by emotional guidance at different stages. To be specific, it can be seen that the adaptive caption generation path improves the performance by 0.9%, 1.3%, 1.7%, 1.1%, 5.1%, and 6.8% on semantic metrics and by 2.3% and 6.5% on hybrid metrics.Discussion on our emotion evolution methods. In addition, we conduct ablation studies about the element-level and subspacelevel evolution in dynamic emotion perception. The results are illustrated in Table 3 . We firstly observe that the element-level evolution only brings a slight improvement, i.e., 0.8, 0.8 on Acc sw and Acc c , respectively. A possible reason is that the element-level evolution in our perception path serves as the global evolution without any strict restrictions, which means it avoid our methods falling into the local-optima. In contrast, the subspace-level emotion evolution explores multi-granular semantic information by reorganizing the dimensions of emotion features. Thus, it brings a remarkable improvements, i.e., 3.6, 4.2 on Acc sw and Acc c .In order to qualitatively demonstrate the effect of our model, we make a visualization comparison with the state-of-the-art method EPAN on the EVC-VE dataset. Firstly, as shown in Figure 5 , in case (A), EPAN incorrectly predicts the emotion word from "happily" to "frustrated". It is worth noting that "frustrated" emotion is quite opposite of the action "smile". Thus, since it incorporates the emotion into the caption generation for all time steps, the emotional cue "frustrated" is overemphsised, making it unable to perceive the "smile" action in the video and predict the factual word "smile". Instead, our method predicts the emotion evolution at each time step. Thus, our method successfully predicted the "smile" word in the caption generation by predicting the correct evolved emotions "happily". Secondly, in both two cases, since EPAN only perceive the global emotion, such as "frustrated" in case (A) and "happy" in case (B), their methods fail to predict the other emotion words in caption generation. For example, in case (B), the emotions changes to "boringly" as the video showing that the boy eat soullessly. However, EPAN generates wrong caption "child eating happily". Instead, our model correctly identified the emotional evolution from "happily" to "boringly" between the woman and the child in the video. Thus, we generates "a woman is happy to see" and "her child eats bread boringly" correctly with two different emotion guidance. Both two observations further demonstrate the effectiveness of our method.To further evaluate the generalization performance, we apply our method to the emotional image captioning task [12, 24, 40, 45, 46] on Senticap [30] dataset. Senticap dataset marks the positive/negative emotional polarity labels of the images based on the MS-COCO [27] dataset, and rewrites the description with emotions. We directly test the performance of our model on the Senticap dataset. The results are shown in Table 4 . It can be seen that our method achieves the best performances on all metrics, i.e., 3.8%, 7.8%, 4.0% on BLEU-2, METEOR, and BFS, respectively. Although compared with videos, images lose the temporal information and are less likely to have dynamic changes in emotion, our dynamic emotion perception module can still contribute to emotional bias correction. Even if the initial emotion perception is wrong, our dynamic evolution module help to correct the emotional bias during the caption generation. In the step where emotion-related words need to be generated, the correct emotion can be perceived and guide the word generation. 
conclusion: 
 In this paper, we propose a novel Dual-path Collaborative Generation Network for emotional video captioning, which dynamically perceives visual emotional cues evolutions while generating emotional captions by collaborative learning. The two paths promote each other and significantly improve the emotional caption generation performance. To be specific, a dynamic emotion evolution module, including element-level emotion evolution and subspace-level emotion evolution, is designed in the dynamic emotion perception path, which dynamically recomposes emotion features to achieve emotion evolution at different generation steps, helping to correct the previous emotion bias and provide customized and accurate emotion guidance for each generation step. Besides, an emotion adaptive decoder is proposed in the adaptive caption generation path that estimates the emotion intensity for each step, helps to generate emotion-related words at the necessary time step, and balances the guidance role of the factual content and emotional cues. Extensive experiments on three challenging datasets demonstrate the superiority of our method and each proposed module. 
 
 Appendix: Our code is publicly available at https://github.com/kyrieye/MM-2024. 


Decision: All reviewers are satisfied with the responses and the contributions in the filed, I recommend its acceptance.

Reviews: review: paper_summary: paper_summary: This work proposes a new method for the Emotion Video Captioning (EVC) task called Dual-path Collaborative Generation Network (DCGN). The authors emphasize capturing the emotional shifts within the video for better emotion-captioning. To do so, DCGN consists of two paths- A dynamic Emotion path, which captures the subtle emotional evolution considering the old captioning and visual context, and an Adaptive Captioning path, which generates the caption considering the evolved emotion cue from the dynamic emotion path. Details of both the branches are well explained. DCGN is shown to be superior over prior works on three public video EVC datasets and has also proven effective for Emotional Image Captioning (EIC) tasks. Sufficient experiments, and ablation studies are performed to justify the design choices.
strength_weakness: strength_weakness: 1. All the modules of DGCN are well explained, and the intuition behind each module is included.
2. Authors conducted appropriate ablation studies to justify the design choices.
3. Qualitative result reflects the importance of a dynamic emotion path, which is able to correctly the characters and their emotions.
limitations: limitations: 1. Considering the abilities of recent Large Vision Language Models (LVLMs) or Large Multi-Modal Models (LMMMs) like GPT-4o, I encourage authors to include such models as a baseline as well.
2. The paper is too wordy and not easy to read. E.g. abstract lines 27 to 34 are one continuous line that is tough to follow. I encourage authors to simplify the language wherever possible. Many claims/explanations made in the Introduction section get cleared when the reader reaches the dedicated section.
3. In section 4.2, please also comment on the resources required to use DCGN.
4. Authors have used ACMM 2017 template. Please transfer the contents to 2024 template.
paper_summary: paper_summary: This work proposes a dual-path collaborative generation network for emotional video captioning. It combines dynamic emotion perception and adaptive caption generation, which can effectively perceive emotional changes in videos, and balance emotion and factual content in subtitle generation, so as to realize the optimization of emotional video subtitle generation.
strength_weakness: strength_weakness: 1. This paper proposes a dual-path collaborative generation network, which combines dynamic emotion perception and adaptive caption generation to achieve more accurate, vivid and diversified subtitle generation, which is innovative.

2. The experiment is sufficient. The author not only uses multiple data sets to prove the performance of the proposed model, but also uses different types of ablation experiments and a variety of indicators to prove the advantages of the proposed module
limitations: limitations: 1. A large number of experimental results have been obtained in the paper. In the "Semantic Evaluation" part of section 4.3 of the experiment, the performance improvement data described by the author may be wrong. It is suggested that the author re-check the calculation accuracy of each data in the paper.

2. In the "Discussion on our proposed two path" part of section 4.4 of the experimental part, the author's last sentence of performance improvement did not clearly indicate the corresponding specific indicators, which is easy to confuse readers. It is suggested that the author point out the corresponding indicators in detail.

3. The author used "EmVidCap-S, EmVidCap-L, EmVidCap" when introducing the data set in Section 4.1 of the experiment, but the data set name in Table 1 is "EVC-MSVD, EVC-VE, EVC-Combine", which is easy to be misunderstood by readers outside the neighborhood.
paper_summary: paper_summary: This paper proposes a novel method for emotional video captioning (EVC). The motivation of this paper is to deal with two challenges in EVC: (1) It is required to capture subtle changes of emotions during the video, which results in diverse  captions. (2) It is required to assign different weights to emotions and factual contents in different steps to avoid overemphasizing emotions when generating captions. To solve these problems, this paper designs a dual-path collaborative generation network, which includes a dynamic emotion module to achieve emotion evolution at different generation steps and an emotion adaptive decoder to balance the factual content and emotional cues.
strength_weakness: strength_weakness: 1. This paper proposes a novel method to deal with emotional video captioning (EVC). The proposed method involves two modules, one of which is a dynamic emotion module to achieve emotion evolution at different generation steps, and the other of which is an emotion adaptive decoder to balance the factual content and emotional cues.
2. The proposed method is superior to current SOTAs on EVC with a remarkable improvement.
3. The representation of this paper is clear.
limitations: limitations: 1. As far as I know, there have existed many advanced video captioning methods these years, like some cross modal transformers[1]. But this paper only compares with one video captioning method which was proposed in 2018. It is possible to question the value of this paper if current advanced video captioning works can perform well on EVC.
2. For experiment of emotional image captioning on SentiCap dataset, there lacks  emotional image captioning methods like[2,3] which makes the conclusion unconvincing.
3. It is better to include some details in this paper for reader understanding. For example, the calculation metric of BFS and CFS, which is not common in video captioning.

[1] VideoBERT: A Joint Model for Video and Language Representation Learning, CVPR, 2019

[2] Senticap: Generating image descriptions with sentiments, AAAI, 2016

[3] Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences, MM, 2023
paper_summary: paper_summary: In this paper, the authors propose a novel Dual-path Collaborative Generation Network for emotional video captioning, which dynamically perceives visual emotional cues evolutions while generating emotional captions by collaborative learning. The two paths promote each other and significantly improve the emotional caption generation performance. Specifically, a dynamic emotion evolution module, including element- and subspace-level evolution, is designed in the dynamic emotion perception path, which dynamically recomposes emotion features to achieve emotion evolution at different generation steps, helping to correct the previous emotion bias and providing customized and accurate emotion guidance for each generation step. Besides, an emotion adaptive decoder is proposed in the adaptive caption generation path that estimates the emotion intensity for each step, helps to generate emotion-related words at the necessary time step, and balances the guidance role of the factual content and emotional cues. Extensive experiments on three challenging datasets demonstrate the superiority of the proposed method and each proposed module.
strength_weakness: strength_weakness: - Novelty: This paper introduces a novel collaborative learning framework to integrate dynamic emotion perception into the emotional video caption generation. Moreover, an innovative dynamic emotion evolution module is proposed, which helps to obtain fine-grained emotion representation by two cascade evolution modules including element- and subspace-level evolution, thereby providing more accurate emotion guidance for caption generation at different time steps.
- Quality: This paper has an exhaustive comparison with SoTA using multiple natural language generation metrics and emotion recognition accuracy metrics. The proposed methods showed significant gains on all three prevailing EVC datasets. The authors also present several qualitative results to investigate how the proposed approach perceives accurate emotions for each generation step.
- Clarity: This paper is well-structured and easy to follow.
- Significance: Leveraging collaborative learning to perceive the subtle emotional cues to boost emotional video captioning is important for multimedia communities. The emotion-based multimedia understanding is a hot topic today and yet under-researched.
limitations: limitations: 1. Please quantify the experimental results in the Abstract.
2. Please revise your qualitative results. The existing results can only show that the proposed method works better in the case of multiple emotions, but does not illustrate the superiority of the single emotion case.
3. Some occasional writing typos need to be revised:
    - In line 382, “this” should be “these”.
    - In line 441, “mechanisms” should be “mechanism”.