MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System
Year: 2023

Abstract:
Multi-modal sarcasm detection has attracted much recent attention. Nevertheless, the existing benchmark (MMSD) has some shortcomings that hinder the development of reliable multi-modal sarcasm detection system: (1) There are some spurious cues in MMSD, leading to the model bias learning; (2) The negative samples in MMSD are not always reasonable. To solve the aforementioned issues, we introduce MMSD2.0, a correction dataset that fixes the shortcomings of MMSD, by removing the spurious cues and re-annotating the unreasonable samples. Meanwhile, we present a novel framework called multi-view CLIP that is capable of leveraging multi-grained cues from multiple perspectives (i.e., text, image, and textimage interaction view) for multi-modal sarcasm detection. Extensive experiments show that MMSD2.0 is a valuable benchmark for building reliable multi-modal sarcasm detection systems and multi-view CLIP can significantly outperform the previous best baselines.

heading: 1 Introduction
text: Sarcasm detection is used to identify the real sentiment of user, which is beneficial for sentiment analysis and opinion mining task (Pang and Lee, 2008). Recently, due to the rapid progress of social media platform, multi-modal sarcasm detection, which aims to recognize the sarcastic sentiment in multi-modal scenario (e.g., text and image modalities), has attracted increasing research attention. Specifically, as illustrated in Figure 1, given the text-image pair, multi-modal sarcasm detection system predicts sarcasm label because the image shows a traffic jam that contradicts the text “love the traffic”. Unlike traditional sarcasm detection task, the uniqueness of multi-modal sarcasm detection lies in effectively modeling the consistency and sarcasm relationship among different modalities.
∗Equally contributed authors. † Corresponding author.
Thanks to the rapid development of deep neural networks, remarkable success has been witnessed in multi-modal sarcasm detection. Specifically, Schifanella et al. (2016) makes the first attempt to explicitly concatenate the textual and visual features for multi-modal sarcasm detection. A series of works have employed attention mechanism to implicitly incorporate features from different modalities (Cai et al., 2019; Xu et al., 2020; Pan et al., 2020). More recently, graph-based approaches have emerged to identify significant cues in sarcasm detection (Liang et al., 2021, 2022; Liu et al., 2022), which is capable of better capturing relationship across different modalities, thereby dominating the performance in the literature.
While current multi-modal sarcasm detection systems have achieved promising results, it is unclear whether these results faithfully reflect the multi-modal understanding ability of models. In fact, when a text-modality only model RoBERTa is applied to multimodal sarcasm detection, its performance significantly surpasses the state-of-theart multi-modal model HKE (Liu et al., 2022) with 6.6% improvement (See Detailed Analysis § 4.3). Such observation suggests that the performance of current models may heavily depend on spurious cues in textual data, rather than truly capturing the
10834
relationship among different modalities, resulting in low reliability. Further exploration reveals that the characteristics of the MMSD benchmark (Cai et al., 2019) may be the cause of this phenomenon: (1) Spurious Cues: MMSD benchmark has some spurious cues (e.g., hashtag and emoji word) occuring in an unbalance distribution of positive and negative examples, which leads to the model bias learning; (2) Unreasonable Annotation: MMSD benchmark simply assigns the text without special hashtag (e.g., #sarcasm, etc.) as negative examples (i.e., not sarcastic label). We argue that this construction operation is unreasonable because sentence without #sarcasm tag can also express the sarcastic intention. Take utterance in Figure 2 as an example, the utterance without #sarcasm tag still belong to the sarcastic sample. Therefore, further chasing performance on current MMSD benchmark may hinder the development of reliable multi-modal sarcasm detection system.
Motivated by the above observation, we shift our eyes from traditional complex network design work (Cai et al., 2019; Liang et al., 2021, 2022) to the establishment of reasonable multi-modal sarcasm detection benchmark. Specifically, we introduce MMSD2.0 to address these problems. To solve the first drawback, MMSD2.0 removes the spurious cues (e.g., sarcasm word) from text in the MMSD, which encourages model to truly capture the relationship across different modalities rather than just memorize the spurious correlation. Such operation can benefit the development of bias mitigation in multi-modal sarcasm detection studies. To address the second problem, we directly make our efforts to re-annotate the unreasonable data. Specifically, given each utterance labeled with “not sarcastic” in the MMSD, we use crowdsourced workers to check and re-annotate the label. This correction process results in changes to over 50% of samples in original MMSD.
In addition to the dataset contribution, we propose a novel framework called multi-view CLIP, which can naturally inherit multi-modal knowledge from pre-trained CLIP model. Specifically, multi-view CLIP utilizes different sarcasm cues captured from multiple perspectives (i.e., text, image and text-image interaction view), and aggregates multi-view information for final sarcasm detection. Compared to previous superior graphbased approaches, multi-view CLIP has the following advantages: (1) multi-view CLIP does not
require any image pre-processing step for graph building (e.g., object detection); (2) multi-view CLIP does not require any complex network architecture and can naturally make full use of knowledge in VL pre-trained model for multi-modal sarcasm detection.
Contributions of this work can be summarized as follows:
• To the best of our knowledge, we make the first attempt to point out the behind issues in current multi-modal sarcasm benchmark, which motivates researchers to rethink the progress of multi-modal sarcasm detection;
• We introduce MMSD2.0, a correction dataset removing the spurious cues and fixing unreasonable annotation, for multi-modal sarcasm detection, which takes a meaningful step to build a reliable multi-modal sarcasm system;
• We propose a novel multi-view CLIP framework to capture different perspectives of image, text and image-text interaction, which attains state-of-the-art performance.
To facilitate future research, all datasets, code are publicly available at https://github.com/ JoeYing1019/MMSD2.0.

heading: 2 MMSD2.0 Dataset Construction
text: MMSD2.0 aims to solve two main shortcomings of MMSD, which consists of (1) Spurious Cues Removal (§2.1) and (2) Unreasonable Samples Re-Annotation via Crowdsourcing (§2.2). This section provides MMSD2.0 dataset construction in detail and the overall process is illustrated in Figure 2.

heading: 2.1 Spurious Cues Removal
text: In our in-depth analysis, we observe that the spurious cues come from two resources: (1) hashtag words and (2) emoji words. Therefore, we will remove the two resources, respectively.
Hashtag Word Removal. In MMSD, as shown in Figure 3(a), we observe that distribution of hashtag word number in positive sample and negative sample is obviously unbalanced. As seen, the number of hashtag words in positive sample is on average more than 1 while less than 1 in negative samples in train, validation and test set. In other word, model only need to learn spurious correlation (hashtag word number) to make a correction
prediction rather than truly understand multi-modal correlation in sarcasm detection.
To address the issue, we remove hashtag words from text in the MMSD dataset. This allows the model to capture image features and using them to guide the final prediction, rather than relying on the hashtag word number as a spurious cue.
Emoji Word Removal. Similarly, we find that the distribution of emoji words between positive and negative samples is also unbalanced. Specifically, as shown in Figure 3(b), only 19.3% of which exist in both positive and negative samples while the rest 80.7% of emoji words only appear in one type of sample (e.g., positive sample or negative sample). This indicates that model can simply use emoji word distribution as a priority for predicting
rather than truly capturing multi-modal cues. To tackle this issue, we remove all the emoji words in text to force the model learning truly multi-modal sarcasm features rather than relying on spurious textual cues.

heading: 2.2 Unreasonable Samples Re-Annotation via Crowdsourcing
text: This section consists of two stages for reannotation: (1) sample selection stage to choose the unreasonable samples and (2) re-annotation via crowdsourcing stage to fix the unreasonable samples according to the sample selection stage.
Sample Selection Stage. MMSD simply considers the samples without special hashtags like “#sarcasm” as negative sample (i.e., not sarcasm). In this work, we argue that the process is unreasonable because samples without #sarcasm tag can also express sarcasm intention. Therefore, we select all negative samples in the MMSD dataset (over 50%) as potentially unreasonable samples for further processing.
Re-annotation via Crowdsourcing. For all selected samples in sample selection stage, we directly re-annotate them via crowdsourcing by hiring human experts. Given a sample, each annotator is required to annotate with labels among: (i) Sarcasm: if the sample express sarcasm intention; (ii) Not Sarcasm: if the sample not express sarcasm intention; (iii) Undecided: if the sample is hard to decide by the annotator.
After the whole annotation is done, the samples with label Undecided is re-annotated again by three experts to decide the final annotation.

heading: 2.3 Quality Control and Data Statistics

heading: 2.3.1 Quality Control
text: To control quality of the annotated dataset, we conduct two verification methods.
Onboarding Test. Before the whole annotation work, we require all the annotators to annotate 100 reasonable samples selected by model and the annotation result will be checked by 3 experts. Only those who achieve 85% annotation performance can join the annotation process.
Double Check. We randomly sample 1,000 annotated samples and ask another new annotator to re-annotate sarcasm label. Then, we calculated the Cohen’s Kappa (McHugh, 2012) between the previous labels and new labels. Finally, we get a kappa of 0.811 indicating an almost perfect agreement (Landis and Koch, 1977).

heading: 2.3.2 Data Statistics
text: Table 1 provides a detailed comparison of the statistics of the MMSD and MMSD2.0 datasets. We can observe that the distribution of positive and negative samples in MMSD2.0 is more balanced.

heading: 3 Approach
text: This section first describes the basic architecture for CLIP (§3.1) and then illustrates the proposed multi-view CLIP framework (§3.2).

heading: 3.1 CLIP
text: CLIP (Contrastive Language-Image Pretraining) (Radford et al., 2021), a powerful vision-andlanguage pre-trained model, has shown remarkable success on various VL downstream tasks (Conde and Turgutlu, 2021; Fu et al., 2022; Zhou et al., 2022). As shown in Figure 4 , CLIP contains a visual encoder V and a text encoder T where former usually adopts ResNet (He et al., 2015) or ViT (Dosovitskiy et al., 2021) and the latter employs transformer (Vaswani et al., 2017) as backbone. After representation acquired by T and V, a dot-product function (T(·) ·V(·)) is further used to calculate the similarity of the given image and text
and the highest score indicates the corresponding text and image is matched.

heading: 3.2 Multi-View CLIP
text: We introduce a novel multi-view CLIP framework for multi-modal sarcasm detection. Our framework consists of text view (§3.2.1), image view (§3.2.2) and image-text interaction view (§3.2.3), which explicitly utilize different cues from different views in CLIP model, thereby better capturing rich sarcasm cues for multi-modal sarcasm detection. The overall framework is shown in Figure 5.

heading: 3.2.1 Text View
text: A series of works (Xiong et al., 2019; Babanejad et al., 2020) have shown textual information can be directly used for performing sarcasm detection. Therefore, we introduce a text view module to judge the sarcasm from text perspective.
Given samples D = { (x(i),y(i)) }ND i=1
where {x,y} is a text-image pair, CLIP text encoder T is used to output encoding representations T :
T = (t1, . . . , tn, tCLS) = T(x), (1)
where n stands for the sequence length of x. Then, text-view decoder directly employ tCLS for multi-modal sarcasm detection:
yt = softmax(WtCLS + b), (2)
where yt is output distribution.

heading: 3.2.2 Image View
text: Image information can be considered as a textual supplement feature for sarcasm detection (Schifanella et al., 2016) , which motivates us to propose an image CLIP view module to detect the sarcasm from image perspective.
1.4 修改
Specifically, image view CLIP first leverage visual encoder V to generate image representation:
I = (vCLS,v1, . . . ,vm) = V(y), (3)
where m denotes the number of image patches. Similarly, vCLS is used for prediction:
yv = softmax(WvCLS + b), (4)
where yv is output distribution.

heading: 3.2.3 Image-text Interaction View
text: Modeling relationship across text and image modality is the key step to multi-modal sarcasm detection. We follow Fu et al. (2022) to use a transformer encoder (Vaswani et al., 2017) for sufficiently capturing interaction across different modalities.
Specifically, given the yielded text representation T and image representation I , we first concat them using F = (vCLS,v1, . . . ,vm, t1, . . . , tn, tCLS) = Concat(T , I). Then, we apply different linear functions to obtain the corresponding queries Q, keys K and values V , and the updated representation F̂ can be denoted as F̂ = softmax ( QK>√
dk
) V ,
where dk is the mapped dimension. Given the updated image-text representation F̂ = (v̂CLS, v̂1, . . . , v̂m, t̂1, . . . , t̂n, t̂CLS), we use keyless attention mechanism (Long et al., 2018) to further fuse the image-text interaction feature f by calculating:
f = ptt̂CLS + pvv̂CLS, (5)
pt,pv = softmax(W (t̂CLS, v̂CLS) + b). (6)
Finally, f is utilized for prediction:
yf = softmax(Wf + b), (7)
where yf is output distribution.

heading: 3.3 Multi-view Aggregation
text: Given the obtained yt,yv,yf , we adopt a late fusion (Baltrusaitis et al., 2019) to yield the final prediction yo:
yo = yt + yv + yf , (8)
where yo can be regarded as leveraging rich features from different perspectives of text view, image view and image-text interaction view.

heading: 3.4 Model Training
text: We use a standard binary entropy loss for image, text, and image-text interaction view, and a joint optimization for training the entire framework:
L = ∑
i∈{t,v,f} (ŷi · log(yi) + (1− ŷi) log(1− (yi)), (9)
where ŷi is the gold label. It is worth noting that we can directly use the final gold label for training image-view CLIP and text-view CLIP, which does not bring any annotation burden.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: We conduct experiments on MMSD (Cai et al., 2019) and MMSD2.0. We implement our model
Model MMSD MMSD2.0
Acc.(%) P (%) R (%) F1 (%) Acc. (%) P (%) R (%) F1 (%) Text-modality Methods
TextCNN (Kim, 2014) 80.03 74.29 76.39 75.32 71.61 64.62 75.22 69.52 Bi-LSTM (Graves and Schmidhuber, 2005) 81.90 76.66 78.42 77.53 72.48 68.02 68.08 68.05 SMSD (Xiong et al., 2019) 80.90 76.46 75.18 75.82 73.56 68.45 71.55 69.97 RoBERTa† (Liu et al., 2019) 93.97 90.39 94.59 92.45 79.66 76.74 75.70 76.21 Image-modality Methods ResNet (He et al., 2015) 64.76 54.41 70.80 61.53 65.50 61.17 54.39 57.58 ViT (Dosovitskiy et al., 2021) 67.83 57.93 70.07 63.40 72.02 65.26 74.83 69.72 Multi-Modality Methods HFM (Cai et al., 2019) 83.44 76.57 84.15 80.18 70.57 64.84 69.05 66.88 D&R Net (Xu et al., 2020) 84.02 77.97 83.42 80.60 – – – – Att-BERT (Pan et al., 2020) 86.05 80.87 85.08 82.92 80.03 76.28 77.82 77.04 InCrossMGs (Liang et al., 2021) 86.10 81.38 84.36 82.84 – – – – CMGCN (Liang et al., 2022) 86.54 – – 82.73 79.83 75.82 78.01 76.90 HKE (Liu et al., 2022) 87.36 81.84 86.48 84.09 76.50 73.48 71.07 72.25 Multi-view CLIP 88.33 82.66 88.65 85.55 85.64 80.33 88.24 84.10
Table 2: Main Results. Baseline results on dataset MMSD are taken from Liu et al. (2022). Results with - denote the code is not released. Results with † stand for that we re-implement the model.
based on the Huggingface Library (Wolf et al., 2020) and adopt clip-vit-base-patch32 as backbone. We use AdamW (Loshchilov and Hutter, 2019) as optimizer to optimize the parameters in our model. The batch size and epoch for training are 32 and 10, respectively. The learning rate for CLIP is 1e-6 and for the other parts is 5e-4. We select all hyper-parameters from the validation set. All experiments are conducted at Tesla V100s.

heading: 4.2 Baselines
text: Following Liu et al. (2022), we explore three type of baselines: (i) text-modality methods, (ii) imagemodality methods and (iii) multi-modality methods.
(i) For text-modality methods, we employ (1) TextCNN (Kim, 2014) (2) Bi-LSTM (Graves and Schmidhuber, 2005) (3) SMSD (Xiong et al., 2019) and (4) RoBERTa (Liu et al., 2019).
(ii) For image-modality methods, our exploration including: (1) ResNet (He et al., 2015) and (2) ViT (Dosovitskiy et al., 2021), which are widely used backbone in CLIP.
(iii) For multi-modality methods, we compare multi-view CLIP with the following state-of-theart baselines: (1) HFM (Cai et al., 2019) is a hierarchical fusion model for multi-modal sarcasm detection; (2) D&R Net (Xu et al., 2020) proposes a decomposition and relation network to model crossmodality feature; (3) Att-BERT (Pan et al., 2020) applies two attention mechanisms to model the textonly and cross-modal incongruity, respectively; (4) InCrossMGs (Liang et al., 2021) is a graph-based model using in-modal and cross-modal graphs to
capture sarcasm cues; (5) CMGCN (Liang et al., 2022) proposes a fine-grained cross-modal graph architecture to model the cross modality information; (6) HKE (Liu et al., 2022) is a hierarchical graph-based framework to model atomic-level and compositionlevel congruity. For a fair comparison, CMGCN and HKE are the versions without external knowledge.

heading: 4.3 Performance on MMSD
text: Following Liu et al. (2022), we adopt accuracy (Acc.), F1 score (F1), precision (P) and recall (R) to evaluate the model performance.
Table 2 (left part) illustrates the results on MMSD. We have the following observations: (1) Text-modality methods achieve promising performance and RoBERTa even surpasses multi-modality approaches, which indicates the sarcasm detection model can only rely on the text features to make a correct prediction, supporting the motivation of our work; (2) In multi-modality approaches, multi-view CLIP attains the best results, demonstrating the effectiveness of integrating features from different modality views.

heading: 4.4 Performance on MMSD2.0
text: Table 2 (right part) presents the results on MMSD2.0. As seen, we have the following observations: (1) Compared with the MMSD dataset, performance of all the baselines has decreased in different degrees on MMSD2.0 dataset. Especially the single-modality model RoBERTa’s performance decrease by 16.24% on F1 score. This indicates that MMSD2.0 successfully removes the spurious
cues and can be used for a more reliable benchmark; (2) Additionally, the previous graph-based baselines such as CMGCN and HKE do not make good performance on MMSD2.0, even worse than the Att-BERT. We attribute it to the fact that such graph-based approaches highly rely on the spurious cues (e.g., hashtag and emoji word) when constructing the text semantic dependency graph, which cannot be accessed in MMSD2.0; (3) Lastly, multi-view CLIP not only attains the best results in multi-modality approaches but also in both text and image modality approaches, which further verifies the effectiveness of our framework.

heading: 4.5 Analysis
text: To understand the multi-view CLIP in more depth, we answer the following research questions: (1) Does each modality view contribute to the overall performance of the model? (2) What effect do the training strategies of CLIP have on the model performance? (3) What impact do different interaction approaches have on the model performance? (4) Does the multi-view CLIP approach remain effective in low-resource scenarios? (5) Why does the multi-view CLIP work?

heading: 4.5.1 Answer1: All Views Make Contribute
text: To Final Performance
To analyze the effect of different modality views of our framework, we remove the training object of text view LT , image view LV and image-text interaction view LF separately.
Table 3 illustrates the results. We can observe the accuracy decrease by 1.46%, 1.95% and 3.20% when removing LT , LV and LF , respectively. As seen, our framework attains the best performance when combining all these views. This suggests that all views in our framework make contribute to the final performance. It is worth noting that the removal of LF leads to a significant performance drop, which suggests that modeling the multi-modality interaction features is crucial for multi-modal sarcasm detection.

heading: 4.5.2 Answer2: Full Finetuned Gains The Best Performance
text: To investigate the influence of training methods of backbone network CLIP, we conduct experiments on different training combination of V and T.
The results are shown in Figure 6. It reveals that full finetuning of CLIP leads to the best performance. An interesting observation is that freezing all of CLIP leads to almost the same performance as freezing only T or V. This can be attributed to the fact that the text and image representations in CLIP are aligned, and only training a single part can break this alignment property.

heading: 4.5.3 Answer3: Transformer Interaction Fuse
text: Cross-modal Information Deeper
To further verify the effectiveness of our framework, we explore different interaction methods in text-image interaction view, including cross attention (Pan et al., 2020) and the MLP approach which concatenates information from different modalities and uses feedforward layer to fuse them.
Figure 7 illustrates the results. We can observe that our transformer interaction is more efficient than other two interactions. We attribute it to the fact that the transformer interaction is able to fuse the information from different modals in a more
depth way, hence attaining the best performance.

heading: 4.5.4 Answer4: Multi-view CLIP can Transfer to Low-resource Scenario
text: To explore the effectiveness of multi-view CLIP in low-resource scenario, we experiment with different training sizes including 10%, 20% and 50%.
The results are shown in Figure 8, indicating that the multi-view CLIP approach still outperforms other baselines in low-resource settings. In particular, when the data size is 10%, the multi-view CLIP approach outperforms the baselines by a large margin in the same setting, and even surpasses Att-BERT when trained on the full dataset. We attribute this to the fact that the knowledge learned during the CLIP pre-training stage can be transferred to low-resource settings, indicating that our framework is able to extract sarcasm cues even when the training corpus is limited in size.

heading: 4.5.5 Answer5: Multi-view CLIP Utilize Correct Cues
text: To further explain why our framework works, we visualize the attention distribution of visual encoder V to show why our framework is effective for multimodal sarcasm detection.
As shown in Figure 9, our model can successfully focus on appropriate parts of the image with sarcasm cues. For example, our framework pays more attention on the no-fresh beans in Figure 9(a), which as the important cues to contradicted with “fresh” in text. Meanwhile, our framework focus on the bad weather regions in Figure 9(b), which also an important cues incongruent with “amazing” in text. This demonstrates that our framework can successfully focus on the correct cues.

heading: 5 Related Work
text: Sarcasm detection identifies the incongruity of sentiment from the context, which first attracts attention in text modality. Early studies focused on using feature engineering approaches to detect in-
congruity in text (Lunando and Purwarianti, 2013; Bamman and Smith, 2015). A series of works (Poria et al., 2016; Zhang et al., 2016; Xiong et al., 2019) explored deep learning network (e.g., CNN, LSTM and self-attention) for sarcasm detection. Babanejad et al. (2020) extended BERT by incorporating into it effective features for sarcasm detection. Compared to their work, we aim to solve sarcasm detection in the multi-modal scenario while their work focus on text sarcasm detection.
As the rapid popularization of social medial platform, multi-modal sarcasm detection attracts increasing research attention in recent years. Schifanella et al. (2016) explored the multi-modal sarcasm detection task for the first time and tackled this task by concatenating the features from text and image modalities respectively. Cai et al. (2019) proposed a hierarchical fusion model to fuse the information among different modalities and resealed a new public dataset. Xu et al. (2020) suggested to represent the commonality and discrepancy between image and text by a decomposition and relation network. Pan et al. (2020) implied a BERT architecture-based model to consider the incongruity character of sarcasm. Liang et al. (2021) explored constructing interactive in-modal and crossmodal to learn sarcastic features. Liang et al. (2022) proposed a cross-modal graph architecture to model the fine-grained relationship between text and image modalities. Liu et al. (2022) proposed a hierarchical framework to model both atomiclevel congruity and composition-level congruity. In contrast to their work, we make the first attempt to address the bias issues in traditional MMSD dataset, towards to building a reliable multi-modal sarcasm detection system. In addition, we introduce MMSD2.0 to this end, aiming to facilitate the research. To the best of our knowledge, this is the first work to reveal the spurious cues in current multi-modal sarcasm detection dataset.

heading: 6 Conclusion
text: This paper first analyzed the underlying issues in current multi-modal sarcasm detection dataset and then introduced a MMSD2.0 benchmark, which takes the first meaningful step towards building a reliable multi-modal sarcasm detection system. Furthermore, we proposed a novel framework multi-view CLIP to capture sarcasm cues from different perspectives, including image view, text view, and image-text interactions view. Experimental results show that multi-view CLIP attains state-of-the-art performance.
Limitations
This work contributes a debias benchmark MMSD2.0 for building reliable multi-modal sarcasm detection system. While appealing, MMSD2.0 is built on the available MMSD benchmark. In the future, we can consider annotating more data to break through the scale and diversity of the original MMSD.
Ethics Statement
All data of MMSD2.0 come from the MMSD dataset Cai et al. (2019), which is an open-source dataset available for academic research. Our annotation process was carried out by annotators who were postgraduate students at Chinese universities and they were paid properly.

heading: Acknowledgments
text: This work was partially supported by the National Natural Science Foundation of China (62006062, 62176076, 62236004, 61976072), National Key R&D Program of China via grant 2020AAA0106501, Natural Science Foundation of GuangDong 2023A1515012922, the Shenzhen Foundational Research Funding (JCYJ20220818102415032), the Major Key Project of PCL2021A06, Guangdong Provincial Key Labo-ratory of Novel Security Intelligence Technologies 2022B1212010005.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
7 A2. Did you discuss any potential risks of your work? This work introduces a correction dataset to fix behind issues in current multi-modal sarcasm benchmark and proposes a novel framework multi-modal sarcasm detection, which may not contain potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2 MMSD2.0 Dataset Construction and Section 3 Approach

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1 Introduction

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section Ethics Statement
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? All the data are based on publicly avaliable and used safely by previous works.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We will provide the documentation of our code and models in our code repo.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2.3 Quality control and Data Statistics
C 3 Did you run computational experiments? Section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 Experiments
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 Experiments

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 Experiments
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section Ethics Statement

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section Ethics Statement

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section Ethics Statement

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section Ethics Statement
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section Ethics Statement

heading: Future Work
text: In the future, we can consider annotating more data to break through the scale and diversity of the original MMSD