Sentiment-Oriented Sarcasm Integration: Effective Enhancement of Video Sentiment Analysis with Sarcasm Assistance

Authors: junlin  fang
wenya  wang
guosheng  lin
fengmao  lv
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 14:39:25

Abstract: Sarcasm is an intricate expression phenomenon and has garnered increasing attentions over the recent years, especially for multimodal contexts such as videos. Nevertheless, despite being a significant aspect of human sentiment, the effect of sarcasm is consistently overlooked in sentiment analysis. Videos with sarcasm often convey sentiments that diverge or even contradict their explicit messages. Prior works mainly concentrate on simply modeling sarcasm and sentiment features by utilizing the Multi-Task Learning (MTL) framework, which we found introduces detrimental interplays between the sarcasm detection task and sentiment analysis task. Therefore, this study explores the effective enhancement of video sentiment analysis through the incorporation of sarcasm information. To this end, we propose the Progressively Sentiment-oriented Sarcasm Refinement and Integration (PS2RI) framework, which focuses on modeling sentiment-oriented sarcasm features to enhance sentiment prediction. Instead of naively combining sarcasm detection and sentiment prediction under an MTL framework, PS2RI iteratively performs the sentiment-oriented sarcasm refinement and sarcasm integration operations within the sentiment recognition framework, in order to progressively learn sarcasm-aware sentiment feature without suffering the detrimental interplays caused by information irrelevant to the sentiment analysis task.
Extensive experiments are conducted to validate both the effectiveness and scalability of our approach.

Main paper: 
 introduction: 
 Sarcasm detection has gained significant attentions in recent years due to the potential impact of sarcasm on the sentiment analysis task [20, 21, 25, 32, 38, 55] . The key factor of sarcasm detection is to extract incongruous features which can suggest the distinction between the conveyed sentiment and the author's intended sentiment [25, 32, 55] . The contradictory sentiments arising from such incongruity can considerably disrupt the sentiment analysis task.Recent research [15] has consistently shown that highly advanced Large Language Models (such as ChatGPT [31] ), face challenges when it comes to detecting genuine sentiments conveyed in sarcastic samples. Therefore, it is crucial for sentiment analysis systems to handle sarcasm which is usually overlooked by existing works.In practice, sarcasm can become more prevalent within multimodal inputs like videos [12, 24, 29, 39, 42, 56] , which convey contradicting sentiments among different modalities. As shown in Figure 1 , the text conveys a positive sentiment, while the video demonstrates a negative sentiment which reflects the intrinsic sentiment of the character. Existing studies implement multimodal sarcasm detection by modeling the incongruity information within different aspects of multimodal contents [4, 21, 25, 32] . However, there is a lack of investigation on how sarcasm information can be effectively utilized to enhance the sentiment analysis task.To this end, an intuitive way is to utilize the Multi-Task Learning (MTL) framework by jointly training the sarcasm detection and sentiment analysis tasks, which has been studied by Chauhan et al. [4] and Liu et al. [26] . The MTL framework is expected to enhance sentiment prediction by learning shared representations across the two tasks. However, as will be shown in Section 5, after a Well thank God your livelihood doesn't depend on it. comprehensive evaluation, it turns out that the MTL framework impedes sentiment prediction over non-sarcastic inputs which can be easily identified using a single sentiment classifier (see Table 1 ). This phenomenon implies that the sentiment analysis task suffers the negative interference of the sarcasm detection task in the MTL framework, which is a universally present challenge in MTL [13, 52] .Motivated by the above observation, we propose the Progressive Sentiment-oriented Sarcasm Refinement and Integration (PS2RI) mechanism which boosts video sentiment prediction capability with sarcasm-aware information. Figure 2 shows the model overview of the proposed PS2RI approach. The core idea of PS2RI is to iteratively perform the sentiment-oriented sarcasm refinement and sarcasm integration operations within the sentiment recognition framework, in order to discard irrelevant signals of sarcasm features during the sarcasm-aware sentiment learning process. To this end, our model first utilizes two separate encoders to respectively extract sarcasminformed features and sentiment-related features from videos. Then a Sarcasm-Aware Sentiment Learning (SASL) module is stacked on top of sentiment-related features, with the purpose of producing sarcasm-aware sentiment representations. SASL is implemented via an iterative framework in order to progressively explore the deep interactions between two tasks. Specifically, SASL iteratively propagates information between the two tasks by first refining sarcasm features using sentiment information, and then fusing the refined sarcasm features obtained in the previous step into the sentiment feature learning process. Within each iteration, the sarcasm refinement operation is introduced to obtain sentiment-oriented sarcasm features, to reduce the negative interference caused by direct incorporation of sarcasm features. Compared to the MTL framework [4] which shares features across tasks, PS2RI can alleviate the negative interference of sarcasm detection by discarding irrelevant signals of sarcasm features via sentiment-oriented sarcasm learning before fusing them into the sentiment learning module, thus attentively propagating effective signals between sarcasm and sentiment throughout the learning process (as shown in the supplementary). The superiority of PS2RI is verified via extensive experiments on different video sentiment analysis benchmarks.To sum up, the contributions of this work can be summarized as follows:• We conduct a comprehensive study on how to improve sentiment analysis with the assistance of sarcasm detection and discuss the drawbacks of existing MTL-based methods. • We propose the PS2RI framework which effectively alleviates the negative interference caused by the interaction of sentiment analysis and sarcasm detection; and enhances the performance of sentiment prediction significantly. • We conduct extensive evaluations to demonstrate the effectiveness and scalability of our proposed PS2RI framework. 
relate work 2.1 video sentiment analysis: 
 Video sentiment analysis is primarily focused on identifying the sentiment polarity of humans from videos. Given the presence of different information across various modalities, such as language, acoustic, and visual, video understanding requires fusing data from different modalities [6, 10, 19, 30, 43, 45, 60, 64] . In general, existing video sentiment analysis methods can be divided into global-level fusion and element-level fusion. Specifically, the former strategy implements multimodal fusion over global-level feature vectors extracted from each individual modality and does not consider the cross-modal interactions between elements [17, 35, 42, 43, 48, 50, 51] .Compared to global-level fusion, the element-level fusion methods can achieve a more thorough inter-modal interaction by modeling the fine-grained feature fusion with word-level information considered [22, 23, 28, 45, 49, 65] . However, the above works mainly focus on multimodal fusion, ignoring a further investigation from the sentiment recognition aspect. As indicated by recent works [21, 25] , sarcasm information usually makes a negative interference on the sentiment analysis task.Besides the standard sentiment recognition task as introduced above, there also exit various sentiment-related recognition problems, such as metaphor detection [53, 61, 62] , humor recognition [5, 11, 33, 54] , and sarcasm detection [12, 24, 29, 39, 42, 56] . In this work, we mainly focus on the multimodal sarcasm detection problem. Sarcasm detection aims to capture the contradictory sentiment information present in samples, which has garnered significant attentions due to its effect on causing a shift or complete reversal in the expressed sentiment, thereby greatly impacting sentiment analysis [32, 38, 55] . Early sarcasm detection works primarily focuses on text-only samples, with the core idea being to identify incongruous patterns within texts [9, 37, 41] . Recently, there has been a surge of interest in multimodal sarcasm detection, driven by the growing fascination people have with posting and browsing multimodal information [40] . Compared to text-based sarcasm detection, multimodal sarcasm detection can be more difficult to identify considering the misalignment across different modalities.Existing works on multimodal sarcasm detection focus on exploring incongruity information within different aspects of multimodal contents by building graph neural networks [21, 25] or modeling cross-modal discrepancies and intra-text incongruities [32] . Recently, multimodal sarcasm detection has also been explored for videos [2, 63] . However, the above works study the sarcasm detection task independently and there are currently limited studies that explore how sarcasm information enhances the sentiment analysis task, which holds greater importance in practical application scenarios. There are some studies that acknowledge the interaction between sentiment analysis and sarcasm detection, employing a MTL framework to jointly train the sarcasm detection and sentiment analysis tasks, leading to good advances in sentiment analysis [4, 26] . Nevertheless, their primary emphasis resides in amalgamating features from these two tasks, lacking a more comprehensive evaluation for the sentiment analysis task. Moreover, by conducting a comprehensive evaluation, we find that the MTL framework can only make a marginal improvement for sentiment analysis, and even hurts the performance on non-sarcastic examples. 
problem definition & preliminary 3.1 problem definition: 
 The task of video sentiment analysis involves three different modalities, i.e., language (l), vision (v), and acoustic (a) modalities. Following Chauhan et al. [4] , we denote a sample in training data as 𝑋 = {(𝑈 , 𝐶), 𝑌 𝑠𝑎𝑟 , 𝑌 𝑠𝑒𝑛 }, where 𝑈 and 𝐶 correspond to the target utterance and associated historical contextual dialogues, respectively. 𝑌 𝑠𝑎𝑟 denotes the binary sarcasm label and 𝑌 𝑠𝑒𝑛 denotes the sentiment label. To be specific, we use 𝑈 𝑚 ∈ R 𝑇 𝑢 𝑚 ×𝑑 𝑚 , where 𝑚 ∈ {𝑙, 𝑣, 𝑎} refers to a specific modality, to denote the utterance feature from modality 𝑚. 𝑇 𝑢 𝑚 and 𝑑 𝑚 denote the utterance length and feature dimension, respectively. Similarly, the context feature can be denoted by 𝐶 𝑚 ∈ R 𝑇 𝑐 𝑚 ×𝑑 𝑚 , where 𝑇 𝑐 𝑚 denotes the context length. The core idea of this problem is to integrate data of the three modalities and learn sarcasm-aware sentiment representations which are reinforced by sarcasm information.The cross-modal attention operation fuses the source modality information into the target modality [46] . We denote 𝑋 𝑠 ∈ R 𝑇 𝑠 ×𝑑 𝑠 the source modality feature and 𝑋 𝑡 ∈ R 𝑇 𝑡 ×𝑑 𝑡 the target modality feature. The cross-modal attention mechanism treats the source modality as keys and values to be attended using the target modality as the query:EQUATIONwhere 𝑊 𝑞 ,𝑊 𝑘 and 𝑊 𝑣 are transformation matrices. 𝑌 𝑠→𝑡 ∈ R 𝑇 𝑡 ×𝑑 𝑡 is the resulting representation in the target domain. For ease of illustration, we use CA to denote the cross-modal attention process as shown in Eq. ( 1) with multiple attention heads. Note that CA can also be used for fusing information from two source modalities to the target modality.The Modality-guided Trimodal Fusion (MTF) strategy is introduced in [23] which aims to integrate information from other modalities into each unimodal feature representation. Specifically, MTF consists of multiple layers, and we denote by MTF 𝑛 𝑚 the 𝑛-th layer for modality 𝑚 ∈ {𝑙, 𝑣, 𝑎}. Given input features 𝑍 𝑛-1 𝑙 , 𝑍 𝑛-1 𝑣 , 𝑍 𝑛-1 𝑎 from three modalities at the (𝑛 -1)th layer, we denote byEQUATIONthe output feature for modality 𝑚 after trimodal fusion. More specifically, MTF 𝑛 𝑚 is computed by firstly applying CA between the target modality (e.g., 𝑚 = 𝑙) and the other two modalities using Eq. ( 1). Then a gate unit is applied to combine the faetures obtained from two cross-modal attention operations i.e., 𝑌 𝑣→𝑙 = CA(𝑍 𝑛-1 𝑣 , 𝑍 𝑛-1) and 𝑌 𝑎→𝑙 = CA(𝑍 𝑛-1 𝑎 , 𝑍 𝑛-1), as proposed in [23] .In order to combine features from three modalities as the final representation, a gate [28] is further applied on top of the last MTF layer by weighing the contribution from each modality 𝑚:EQUATIONwhere 𝑍 𝑁 𝑚 is the output of the last MTF layer i.e., MTF 𝑁 𝑚 . 𝑈 ⊤ are learnable parameters, 𝑊 𝑚 and 𝑏 𝑚 are the parameters of a linear function. For ease of illustration, we use 𝑍 = F (𝑍 0 𝑙 , 𝑍 0 𝑣 , 𝑍 0 𝑎 ) to refer to the fusion output given by Eq. ( 2) and (3). 
methodology 4.1 model overview: 
 Figure 2 shows the overall architecture of our proposed PS2RI framework. Given a video input consisting of textual, acoustic and visual features, the PS2RI model first utilizes two separate encoders (i.e., Sarcasm Feature Encoder and Sentiment Feature Encoder) to respectively produce sarcasm-informed features and sentiment-related features. A Sarcasm-Aware Sentiment Learning module is then stacked on top of sentiment-related features, with the purpose of producing sarcasm-aware sentiment representations for enhanced sentiment recognition. SASL is implemented via an iterative framework to progressively explore the deep interactions between two tasks. Within each iteration, SASL utilizes a Sentiment-Oriented Sarcasm Refinement (SOSR) operation to refine sarcasm features guided by relevant sentiment information, which is then followed by a Sarcasm Integration (SI) operation to merge the sentimentoriented sarcasm feature into the sentiment feature learning process. After implementing the above progressive refinement and integration process, SASL will result in sarcasm-aware sentiment features, which can lead to enhanced sentiment recognition performance. The PS2RI framework provides an effective way to alleviate the negative interference caused by the detrimental interplay between the sentiment analysis and sarcasm detection tasks (as shown in the supplementary).In this work, we adopt the same feature extraction step used in [11] to extract features from each modality. To extract acoustic features, we preprocess acoustic frames using COVAREP [7] which is capable of processing Melcepstral coefficients, fundamental frequency, voiced/unvoiced segments, normalized amplitude quotient, quasi open quotient [14] , glottal source parameters [8] , harmonic model, phase distortions, and the formants. For the visual modality, facial Action Unit (AU) features and rigid/non-rigid facial shape parameters are extracted using OpenFace 2 [1] . For the textual modality, we use the ALBERT tokenizer [16] to convert language utterances, following the previous work [11] . Formally speaking, we denote by 𝑋 𝑙 = [𝐶 𝑙 ; 𝑈 𝑙 ], 𝑋 𝑣 = [𝐶 𝑣 ; 𝑈 𝑣 ] and 𝑋 𝑎 = [𝐶 𝑎 ; 𝑈 𝑎 ] input feature for the language, vision and acoustic modality respectively, where 𝐶 and 𝑈 indicate context and utterance respectively. We denote 𝑋 𝑚 ∈ R (𝑇 𝑢 𝑚 +𝑇 𝑐 𝑚 ) ×𝑑 𝑚 one of the three modalities, where 𝑚 ∈ {𝑙, 𝑣, 𝑎}.Well thank God your livelihood doesn't depend on it. .....Refine Refine Sentiment PolaritySarcasm Feature Encoder. The Sarcasm Feature Encoder aims to discriminate sarcasm-related information from inputs without any interference from the sentiment aspect. To achieve that, we first process the multimodal input using their respective encoders. Following [11] , for textual input 𝑋 𝑙 , we use a pre-trained ALBERT encoder composed of 12 layers, together with a linear transformation layer, to produce the text representation𝐻 𝑠𝑎𝑟 𝑙 = Linear 𝑠𝑎𝑟 𝑙 (Albert 𝑠𝑎𝑟 (𝑋 𝑙 )), where 𝐻 𝑙 ∈ R (𝑇 𝑢 𝑚 +𝑇 𝑐 𝑚 ) ×𝑑 𝐻and 𝑑 𝐻 is the output dimension of the alignment layer which maps different modalities features into the same space. Following [11] , we apply a multi-layer transformer encoder [47] 2) and (3). We train this module with the supervision of the sarcasm label 𝑌 𝑠𝑎𝑟 by minimizing the cross-entropy loss between 𝑌 𝑠𝑎𝑟 and the predicted score 𝑌 ′ 𝑠𝑎𝑟 = 𝑓 (𝑍 𝑠𝑎𝑟 ), where 𝑓 is composed of a 1D max pooling layer and a linear layer. As a result, this sarcasm feature encoder can produce sarcasm-discriminative features.Sentiment Feature Encoder. Similar to the Sarcasm Feature Encoding module, we use another set of parameters to process the multimodal inputs, and denote them using the superscript "sen" which indicates sentiment: . Unlike the sarcasm feature encoder, the sentiment feature encoder does not consider the trimodal fusion of the three modalities and leaves the fusion process to the following Sarcasm-Aware Sentiment Learning procedure, due to the computation overhead and model complexity aspects. We treat the above generated features 𝐻 𝑠𝑒𝑛 {𝑙,𝑣,𝑎} as sentimentrelated features, which will then be input into the SASL module to generate sarcasm-aware sentiment representations.With a well-trained sarcasm feature extractor, we stack the SASL module on top of sentiment-related features 𝑍 to produce sarcasmaware sentiment representations by effectively incorporating useful sarcasm features. Within each iteration, two interaction block are involved: a Sentiment-Oriented Sarcasm Refinement (SOSR) block and a Sarcasm Integration (SI) block. Initially, the SOSR block takes the output feature 𝑍 𝑠𝑎𝑟 generated from the Sarcasm Feature Encoding module and provides a refiement given input features coming from the sentiment module. This update infuses original sarcasm-only features with relevant sentiment information which progressively orients sarcasm features towards the sentiment space, reducing the task gap. On the other hand, the SI block receives the updated sarcasm features and integrates them into the representation of each modality adaptively, thus merging relevant sarcasm features into the sentiment learning process. We alternate these two blocks progressively throughout multiple layers to reinforce the propagation between two task signals. Figure 3 shows the detail process of PS2RI. The algorithm which displays the information flow across the PS2RI procedure, is shown in the supplementary.Sentiment-Oriented Sarcasm Refinement. We use SOSR 𝑗 to denote the SOSR block at the 𝑗-th layer. SOSR 𝑗 receives sarcasm features 𝑍 𝑗 -1 𝑠𝑎𝑟 and unimodal sentiment features 𝐻 𝑗 -1 𝑚 (𝑚 ∈ {𝑙, 𝑣, 𝑎}) from the previous layer as its input and produces sentiment-oriented sarcasm featuresEQUATIONAt the 1st layer, we initialize unimodal sentiment features as 𝐻 0 𝑚 = 𝐻 𝑠𝑒𝑛 𝑚 and sarcasm features as 𝑍 0 𝑠𝑎𝑟 = 𝑍 𝑠𝑎𝑟 . Next, we illustrate how SOSR 𝑗 is computed. Specifically, we merge unimodal sentiment features 𝐻𝑗 -1 𝑚 into a unified representation 𝑍 𝑗 -1𝑠𝑒𝑛 via the gate unit introduced in Eq. ( 3).Then a cross-attention layer introduced in Section 3.2 and a self-attention layer introduced in [47] are used to infuse sarcasm features with multimodal sentiment representation 𝑍 𝑗 -1 𝑠𝑒𝑛 by attending to relevant sentiment features:EQUATIONwhere LN, 𝑓 𝐹 𝐹 and SA 𝑗 represent the layer normalization operation, the feed-forward operation and the 𝑗-th self-attention layer, respectively. This completes the operation of Eq. ( 4). , where 𝑚 ∈ {𝑙, 𝑣, 𝑎}:EQUATIONSpecifically, the function SI 𝑗 𝑚 includes a cross-attention layer and a self-attention layer to fuse sentiment-oriented sarcasm features𝑍 𝑗 𝑠𝑎𝑟 into unimodal sentiment representations 𝐻 𝑗 -1𝑚 , similar to Eq. ( 5). The difference with Eq. ( 5) is that we replace the input 𝑍  
experiments: 
 Following the previous works [3, 4] , we first demonstrate the superiority of our proposed approach on two standard benchmark datasets (i.e., MUStARD [2] and MUStARD++ [36] ) which contain both sarcasm and sentiment labels. For in-depth analysis, we split both MUStARD and MUStARD++ datasets into two groups: sarcastic samples (Subset 1) and non-sarcastic samples (Subset 2).Considering that the sarcasm label is usually difficult to annotate, we also validate the scalability of the proposed SASL mechanism on wider settings where sarcasm labels are not provided. Specifically, we conduct experiments by employing the SASL mechanism in conjunction with state-of-the-art video sentiment recognition models on CMU-MOSI [59] and CMU-MOSEI [58] , which are two extensively used benchmarks in video sentiment analysis. As these two datasets do not have sarcasm labels, we directly use the sarcasm feature extraction module trained on MUStARD++ to generate sarcasm features for their samples.The detailed descriptions for the datasets involved in our experiments are listed as follows. MUStARD [2] comprises 3.68 hours of human conversations and provides 675 video samples (50% of which are sarcastic samples) from popular TV shows such as Friends, The Big Bang Theory, The Golden Girls and Sarcasmaholics Anonymous. Its predetermined data partition has 539 samples in the training set, 68 samples in the validation set, and 68 samples in the testing set. Castro et al. [2] manually annotate the sarcastic/non-sarcastic label for each example, while Chauhan et al. [4] re-annotate the MUStARD examples with sentiment labels. Therefore, MUStARD contains both 3-class sentiment labels (i.e., positive, neutral and negative) and 2-class sarcasm labels (i.e., sarcastic and non-sarcastic). Each sample in MUStARD consists of a single sentence with accompanied visual and acoustic segments. In agreement with the prior work [36] , we adopt the Weighted-Average evaluation metrics, including Precision, Recall and F1-score to evaluate the performance. MUStARD++ [36] is a sarcasm dataset consisting of 1,202 samples of video clips (50% of which are sarcastic samples). Its predetermined data partition has 962 samples in the training set, 120 samples in the validation set, and 120 samples in the testing set. Ray et al. [36] extend the samples in MUStARD [2] and re-annotate them with 10-class sentiment labels and 2-class sarcasm labels. As in the MUStARD dataset, each sample in MUStARD++ consists of a single sentence with accompanied visual and acoustic segments. Moreover, each sample also comes with a corresponding context in three modalities. The performance metrics are the same to the ones used in the MUStARD dataset. CMU-MOSI [59] is made by 2,199 samples of short video clips. Following the previous works [6, 48, 51] , we split the dataset into 1,284 training samples, 229 validation samples and 686 testing samples. Each sample in the dataset is labeled with a sentiment label ranging from -3 (strongly negative) to 3 (strongly positive). The acoustic and visual sequences are extracted at the receiving frequency of 12.5 and 15 Hz, respectively. As in the previous works [6, 48, 51] , the performance is evaluated by the 7-class accuracy (i.e., Acc 7 ), binary accuracy (i.e., Acc 2 ) and F1 score. CMU-MOSEI [58] is a dataset consisting of 22,856 samples of movie review video clips which are collected from the YouTube platform. According to the predetermined data partition of CMU-MOSEI, we Following Hasan et al. [11] , we use different learning rates for language, acoustic and visual encoders, as well as other blocks in our approach. Specifically, the learning rate for both acoustic and visual encoders is set to 3e-3, while the learning rate for the language encoder and other blocks is set to 1e-5. Adam is adopted as the optimizer. During the training process, the learning rate decreases according to the cosine decay policy. The batch size is set to 64 in the training process. The dropout rates is set to 0.4. The training epoch is set to 50. We use early-stopping with patience of 10 to avoid overfitting. Following Hasan et al. [11] , we utilize a 12layer ALBERT, an 8-layer transformer and a 1-layer transformer to respectively process the text, visual and acoustic modalities within sarcasm/sentiment feature encoders. The number of MTF layer within the sarcasm feature encoder is set to 4. The PS2RI module utilizes 4 iterations, i.e., 𝐽 = 4. The hidden-layer dimension is set to 192, i.e., 𝑑 𝐻 = 192. The models are trained on a 3090 GPU.The hyper-parameters are determined on the validation set. In experiments, we report results averaged on 5 runs with different random seeds.In this paper, we design four sets of experiments to answer four research questions, through which we progressively study the promoting relationship between sarcasm detection and sentiment analysis:• RQ1: Can sarcasm information contribute to sentiment analysis by directly utilizing the MTL framework? • RQ2: Does the proposed framework achieve performance superiority overall and on specific data splits, compared to existing baselines? • RQ3: Can the proposed SASL mechanism bring performance improvement when applied to existing video sentiment recognition models? • RQ4: Can the proposed SASL mechanism achieve good robustness on wider benchmarks where the sarcasm labels are not provided?Next, we detail the answer for each question and discuss the experimental results. Answer to RQ1. For RQ1, we resort to a single model (i.e., the fusion function F consisting of 4 MTF layers) introduced in Section 3.3 as the base model, which processes the multimodal input by incorporating modality-wise interactions. We use F -SEN to denote the model that is only trained using the sentiment supervision. To examine whether sarcasm information contributes to sentiment analysis by directly utilizing the MTL framework, we implement F -MTL which includes two classification heads on top of F -SEN for generating sarcasm and sentiment predictions and is trained with both sarcasm and sentiment supervisions. For in-depth analysis, we partition the testing set into two separate sub-sets: one containing sarcastic samples (Subset 1) and the other containing non-sarcastic samples (Subset 2). Table 1 shows the performance of F -SEN and F -MTL on different data splits.From Table 1 , we can see that F -MTL can only achieve a marginal improvement over F -SEN (i.e., 54.86% vs. 54.12% in terms of F1 score). Moreover, this improvement is solely achieved from Subset 1 containing sarcastic samples. When it comes to Subset 2 containing non-sarcastic samples, F -MTL is significantly inferior than the F -SEN base model. This observation shows that the MTL framework only improves sentiment predictions over sarcastic samples, while causing negative interference for non-sarcastic samples which can be easily identified by a single sentiment recognition model. Hence, we speculate that the MTL framework which requires representation sharing among tasks may bring noisy information negatively affecting the sentiment prediction task. This calls for a more effective strategy to integrate sarcasm information into the sentiment prediction task. Answer to RQ2. For RQ2, we compare our framework PS2RI with existing state-of-the-art video sentiment analysis baselines, including TFN [57] , LMF [27] , MulT [44] , MISA [12] , MAG [34] , F -MTL [4] , GSLM+ [3] and DMD [18] . For fair comparisons, we use ALBERT as the textual encoder for all the baselines and report the performance of PS2RI and the compared baselines averaged over 5 independent runs with different random seeds in Table 2 . As shown in Table 2 , PS2RI outperforms all the baseline models with a large performance gain. Specifically, we can see that PS2RI can obtain consistent performance improvement over both sarcasm samples (Subset 1) and non-sarcasm samples (Subset 2). This observation shows that PS2RI has indeed found an effective way to use sarcasm information to enhance sentiment task, which eliminates the negative impact of the MTL framework. Answer to RQ3. For RQ3, we conduct experiments by applying the proposed Sarcasm-Aware Sentiment Learning (SASL) mechanism to other existing video sentiment recognition models (including the typical MulT [44] baseline and the state-of-the-art DMD [18] baseline). Specifically, the SOSR layer and SI layer are added before the each layer of the MulT model and DMD model. Table 3 shows the corresponding results averaged over 5 different runs. We can see that our proposed SASL mechanism also achieves consistent performance improvement based on other baselines, which demonstrates the scalability of the SASL mechanism. Answer to RQ4. For RQ4, we conduct experiments by applying the SASL mechanism to the MulT [44] and DMD [18] models on CMU-MOSI and CMU-MOSEI, which are two extensively used benchmarks in video sentiment analysis. As these two datasets do not contain sarcasm labels, we directly adopt the sarcasm feature extraction module trained on MUStARD++ to generate sarcasm features for their samples. Table 4 shows the corresponding results averaged over 5 different runs. We can see that our proposed SASL mechanism can also lead to consistent performance improvements on both CMU-MOSI and CMU-MOSEI benchmarks. This observation clearly shows the robustness of our approach on settings where the sarcasm labels are not provided.Ablation Study. We report the ablation study results on the MUS-tARD and MUStARD++ benchmarks in Analysis on Sarcasm Features. In order to further investigate the sentiment-oriented sarcasm refinement mechanism of the proposed approach, we respectively train two sentiment classification heads on vanilla sarcasm features 𝑍 𝑠𝑎𝑟 extracted from the SFE module and sentiment-oriented sarcasm features Ẑ𝑠𝑎𝑟 obtained from the last layer of SOSR. The performance on Ẑ𝑠𝑎𝑟 is significantly worse than that on 𝑍 𝑠𝑎𝑟 (i.e., 46.15% vs 50.42% in terms of weighted-F1 on MUStARD), which validates that our approach can effectively discard information irrelevant to the sentiment analysis task from sarcasm features, resulting in sentiment-oriented sarcasm features to enhance the sentiment task.Iteration Numbers within SASL. Furthermore, we test the performance by varying the iteration number within SASL. For each number, we also conduct experiments with SOSR blocks removed (i.e., directly integrate the original sarcasm feature extracted from the SFE module into sentiment features). The corresponding results are shown in Figure 4 . In general, the proposed PS2RI framework can achieve gradually increasing performance when more interactive iterations are utilized to learn sarcasm-aware sentiment feature. However, for PS2RI w/o SOSR, the performance improvement is limited when more than 2 SI blocks are utilized. This observation further demonstrates that modeling sentiment-oriented sarcasm features is important for effectively improving sentiment recognition. Moreover, a performance drop is observed when the iteration number is greater 4. This observation indicates that the sentimentoriented sarcasm information can be well modeled by using 4 iteration numbers and more network parameters will lead to overfitting.Case Study. To illustrate the assistance of sarcasm information, we provide case studies which express sarcastic intention in Figure 5 . Take the top sample of Figure 5 as example, the word "like" in the text and the smiling face in the last video clip convey the positive sentiment, while the confused face in the first four video clips conveys the negative sentiment. The conflict sentiments constitute the sarcastic expression. Without the assistance of sarcasm information, the F -SEN base model fails to recognize their sentiment polarities.On the other hand, by explicitly modeling sarcasm-aware sentiment features, the PS2RI approach can effectively recognize the true sentiment.Comparison with Equal Parameter Number. In the Results section of the main paper, we implement the proposed SASL mechanism on the MulT and DMD models and achieve a clear performance improvement. To elucidate the attained performance improvement more distinctly, we augment the original MulT and DMD models with external layers, maintaining an equal number of model parameters with the SASL variants. From Table 6 , we can see that the models with additional layers exhibit inferior performance compared to the original ones. This observation emphasizes that the performance improvement of our approach cannot be solely ascribed to the additional parameters introduced by the SASL mechanism. 
conclusion & future work: 
 In this study, we investigate how to enhance sentiment analysis through the assistance of sarcasm detection. To this end, we discuss the limitations associated with the current MTL-based approaches and propose the PS2RI model to iteratively merge sarcasmrelated features into the sentiment learning process. In order to alleviate the negative interference of sarcasm detection, we introduce Sentiment-Oriented Sarcasm Refinement blocks which model sentiment-oriented sarcasm features by attending sarcasm features to sentiment features and Sarcasm Integration blocks which incorporate sentiment-oriented sarcasm features into sentiment features. By applying the SASL mechanism to CMU-MOSI and CMU-MOSEI datasets, we demonstrate the robustness of our approach in settings where sarcasm labels are absent. However, there still exist domain gaps between the source and target datasets. In future work, we will address the challenge by integrating domain adaptation methods into the training phase. 
 
 Appendix:  


Decision: This paper proposes a novel framework called PS2RI for incorporating sarcasm information into video sentiment analysis. Reviewers acknowledge the innovative approach and extensive experiments. Some questions remain such as the lack of deep insights into the framework's effectiveness, unclear distinctions between key components, insufficient qualitative analysis, and questions about the generalizability of the approach. Two reviewers increased their scores after the rebuttal, where the overall assessment lean towards accept. Therefore, despite the concerns, the topic studied in this paper has good potential, and accept is recommended. The authors are encouraged to address the reviewers' concerns thoroughly in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The study recognizes the significance of sarcasm in video sentiment analysis, which is often overlooked. This focus provides a unique perspective and fills a gap in the current research.The proposed PS2RI framework iteratively performs sentiment-oriented sarcasm refinement and integration, which allows for a progressive learning of sarcasm-aware sentiment features. This approach avoids the detrimental interplay between sarcasm detection and sentiment analysis tasks that can occur in traditional multi-task learning frameworks.
strength_weakness: strength_weakness: 1. Extensive experiments are conducted to validate both the effectiveness and scalability of the PS2RI framework. This ensures that the results are robust and applicable to a wide range of scenarios.
2. The authors give detialed presentation of feature process and experiement settings.
limitations: limitations: 1. The title of this paper is strange and the authors may reconsider the proper title that can represent the overall innovation.
2. Sarcasm is a complex linguistic phenomenon that can be difficult to detect, especially in multimodal contexts such as videos. The study's performance in sarcasm detection and refinement may therefore be limited.
3. While the study demonstrates the effectiveness of the PS2RI framework, it does not provide deep insights into why and how it works. Additional qualitative and interpretative analysis could help to further strengthen the approach.
paper_summary: paper_summary: This paper aims to address the issue of overlooking sarcasm in video sentiment analysis. The proposed approach, named Progressively Sentiment-oriented Sarcasm Refinement and Integration (PS2RI), focuses on modeling sentiment-oriented sarcasm features to enhance sentiment prediction. PS2RI iteratively performs sentiment-oriented sarcasm refinement and sarcasm integration within the sentiment recognition framework, aiming to progressively learn sarcasm-aware sentiment features without suffering from detrimental interactions caused by irrelevant information. Extensive experiments are conducted to validate the effectiveness and scalability of the proposed PS2RI framework.
strength_weakness: strength_weakness: 1.	PS2RI represents a novel approach to incorporate sarcasm information into video sentiment analysis. The iterative refinement and integration of sarcasm features within the sentiment recognition framework is a creative solution to the issue of detrimental interactions between sarcasm detection and sentiment analysis.
2.	PS2RI covers both sentiment-oriented sarcasm refinement and sarcasm integration, demonstrating a comprehensive approach to addressing the challenges of sarcasm in sentiment analysis. The iterative nature of the framework allows for progressive learning of sarcasm-aware sentiment features.
3.	The paper is comprehensive, providing ample discussion and analysis. The Methodology and Experiment sections are detailed, and the language, figures, and tables throughout the paper are logically clear.
limitations: limitations: 1.	Is there a distinction between the Sarcasm-Aware Sentiment Module (SASM) and Sarcasm-Aware Sentiment Learning (SASL) in the paper? The author is expected to provide further clarification on this matter.
2.	The paper lacks a table outlining the sample label division of the datasets. The author is encouraged to include this information for completeness.
3.	The Case Study is described in a rather concise manner. It is requested that the author analyze the characteristics of the cases presented in the case study and explain why the proposed model is effective in identifying genuine sentiment.
4.	I am unclear about the explanation in the Answer to RQ4. The CMU-MOSI and CMU-MOSEI datasets do not have sarcasm labels, yet the SASL mechanism is based on sarcasm detection. Even though the sarcasm feature extraction module trained on MUStARD++ is applied to these datasets, the final labels are not sarcasm versus non-sarcasm. What is the significance of robustness in this context? The author is requested to provide a detailed and supplementary explanation on this matter.
paper_summary: paper_summary: The research paper delves into enhancing video sentiment analysis by integrating sarcasm information, a commonly overlooked aspect in current studies. It introduces the Progressively Sentiment-oriented Sarcasm Refinement and Integration (PS2RI) framework to capture sentiment-oriented sarcasm features for enhanced sentiment prediction. The study underscores the challenges of Multi-Task Learning (MTL) methods in managing sarcasm detection and sentiment analysis simultaneously. Through iterative refinement of sarcasm features with sentiment information and their integration into sentiment learning, PS2RI effectively mitigates the negative impact of sarcasm detection on sentiment analysis. Extensive experimental results confirm the efficacy and scalability of the proposed method, underscoring the significance of incorporating sarcasm into video sentiment analysis for more accurate predictions.
strength_weakness: strength_weakness: Following are the strengths of the work:
1. Innovative Framework: The paper introduces the novel Progressively Sentiment-oriented Sarcasm Refinement and Integration (PS2RI) framework, which effectively addresses the challenges of incorporating sarcasm information into video sentiment analysis. 

2. Iterative Approach: The iterative nature of the PS2RI framework, where sarcasm features are refined using sentiment information and integrated into sentiment learning, is a notable strength. This iterative process allows for the progressive learning of sarcasm-aware sentiment features, ultimately improving sentiment prediction accuracy.

3. Empirical Validation: The study conducts extensive experiments to validate the effectiveness and scalability of the proposed PS2RI framework. The paper demonstrates the superiority of the new approach over conventional MTL methods, emphasizing the importance of considering sarcasm in video sentiment analysis for more accurate predictions.

4. Clear Communication: The clarity in presenting the methodology and findings enhances the paper's readability and understanding for readers interested in sentiment analysis and sarcasm detection in videos.
limitations: limitations: Following are the points where the research work requires further discussion:
Theoretical Foundation: While the paper introduces a novel framework for integrating sarcasm information into sentiment analysis, it would benefit from a deeper exploration of the underlying principles governing sarcasm detection and sentiment analysis. Strengthening the theoretical background through illustrative examples would bolster the author's motivation and rationale.

Generalizability: The paper could expand its discussion on the generalizability of the proposed PS2RI framework across different datasets or domains. Addressing potential limitations or challenges in applying the framework to various scenarios would enhance the paper's practical relevance and usability.

Methodology: How were features extracted for the three modalities? The explanation of feature preprocessing lacks sufficient justification. Moreover, why was a particular research paper [Hasan et al., AAAI, 2021] chosen as the foundation for this work? This aspect requires a more thorough discussion.

Implementation Details: There are a few missing citations, and the rationale behind adopting specific implementation settings from a previous research paper [Hasan et al., AAAI, 2021] needs clarification.

Qualitative Analysis: Although the proposed framework demonstrates its superiority quantitatively, the qualitative analysis is lacking depth, especially regarding the case study (Figure 5), which does not effectively illustrate the superiority of PS2RI. Furthermore, the discussion on error analysis of the current model is absent.

Real-world Applications: The paper would benefit from a discussion on the real-world applications and implications of the PS2RI framework. Exploring potential enhancements to the framework in future research could offer valuable insights for researchers in the same field.
paper_summary: paper_summary: The paper is about the incorporation of sarcasm information to enhance video sentiment analysis. It proposes the Progressively Sentiment-oriented Sarcasm Refinement and Integration (PS2RI) framework, which focuses on modeling sentiment-oriented sarcasm features to improve sentiment prediction in videos. The framework iteratively performs the sentiment-oriented sarcasm refinement and sarcasm integration operations within the sentiment recognition framework, progressively learning sarcasm-aware sentiment features without suffering from detrimental interplays caused by irrelevant information.
strength_weakness: strength_weakness: The framework of this model is easy to understand, and its effectiveness has been verified through extensive experiments in the paper.

The proposed model in the paper achieved the best results on four datasets.
limitations: limitations: The training process for the two encoders is unclear.

The extensive use of methods from paper [1] for feature preprocessing and encoding raises some concerns, as I have noticed that paper [1] originates from a different multimodal domain. Applying these methods directly to the field of video sentiment analysis may not necessarily yield the optimal results.

I noticed that as the hyperparameter Iteration Number increases, the performance does not seem to reach a ceiling. Can it be further improved?

There are misspelled words and errors in the mathematical notation or symbols in the paper, such as in Step 5 of Algorithm 1.

Please consider providing access to the code utilized in the paper to facilitate validation and reproducibility.

I will consider revising the review scores.

[1] Humor Knowledge Enriched Transformer for Understanding Multimodal Humor. AAAI.