Multi-modal Sarcasm Generation: Dataset and Solution
Year: 2023

Abstract:
As an interesting and challenging task, sarcasm generation has attracted widespread attention. Although very recent studies have made promising progress, none of them considers generating a sarcastic description for a given image as what people usually do on Twitter. In this paper, we present a Multi-modal Sarcasm Generation (MSG) task: Given an image with hashtags that provide the sarcastic target, MSG aims to generate sarcastic descriptions like humans. Compared with textual sarcasm generation, MSG is more challenging as it is difficult to accurately capture the key information from images, hashtags, and OCR tokens and exploit multi-modal incongruity to generate sarcastic descriptions. To support the research on MSG, we develop MuSG, a new dataset with 5000 images and related Twitter text. We also propose a multi-modal Transformer-based method as a solution to this MSG task. The input features are embedded in the common space and passed through the multi-modal Transformer layers to generate the sarcastic descriptions by the auto-regressive paradigm. Both automatic and manual evaluations demonstrate the superiority of our method. The dataset and code will be available at github.com/lukakupolida/MSG.

heading: 1 Introduction
text: Sarcasm is a type of emotional expression that indirectly expresses contempt, shows irritation, or demonstrates humor. As a typical task on sarcasm, Sarcasm Generation (SG) is proposed to generate a sarcastic message for a given literal input (Joshi et al., 2015), which can express a variety of communicative intent such as evoking humor and diminishing or enhancing critique (Burgers et al., 2012). It can impact many downstream applications such as personalized dialog systems (Cho et al., 2022)
*: Corresponding Author
and news comment generation (Yang et al., 2019). Since SG is proposed, a surge of follow-up studies have been conducted (Peled and Reichart, 2017; Mishra et al., 2019; Chakrabarty et al., 2020; Oprea et al., 2021, 2022). Notably, the aforementioned SG studies have only been investigated in the textual field so far.
However, nowadays social platforms usually leverage multi-modal data where visual information is integrated with the text, making the analysis of uni-modal data in isolation and limitation. Therefore, research on multi-modal sarcasm is crucial and imperative. Studies on multi-modal sarcasm can be categorized into threefold: Multi-modal Sarcasm Detection (Cai et al., 2019; Pan et al., 2020; Xu et al., 2020; Liang et al., 2021, 2022), Multimodal Target Identification (Wang et al., 2022), and Multi-modal Sarcasm Explanation (Kumar et al., 2022; Desai et al., 2022). Unfortunately, there has been no research touching Sarcasm Generation facing multi-modal information until now. We hope general artificial intelligence to learn creativity and associative skills, so learning to generate sarcastic descriptions towards multi-modal inputs like
5601
humans deserves to study deeply. Therefore, we propose a Multi-modal Sarcasm Generation task (MSG), which aims to generate sarcastic descriptions on social platforms like humans for a given image with the help of hashtags (cf. Figure 1). Compared with textual sarcasm generation, MSG is more challenging. Firstly, human expressions on social platforms are usually stylized with too many verbalized expressions like abbreviations and interjections. Secondly, accurately capturing information from the key visual regions which may contribute to the sarcasm remains a question. Finally, the incongruity between images and generated Sarcastic descriptions reflects human creativity, imagination, and associative skills, which are hard for machines to learn and construct.
To support the studies on MSG, we develop MuSG, a new dataset consisting of 5000 images and related sarcastic descriptions. We manually collect samples with clear sarcastic target from Twitter API and two existing multi-modal sarcasm detection datasets (Schifanella et al., 2016; Cai et al., 2019). The descriptions come with hashtags (the tokens with a ‘#’ to indicate the topic of Twitter) that point the way to sarcasm generation (cf. Figure 1 #bankholidayweekend). The images contain OCR tokens information that can provide an associative context for sarcasm generation (cf. Figure 1 Did somebody say Sunday FUNDAY). With the well-formed dataset MuSG, researchers can easily conduct studies on MSG.
Consequently, as shown in Figure 1, we design a Multi-modal Transformer-based model (MTMSG) as a strong baseline for the proposed MSG task. Concretely, we first model spatial, semantic, and visual reasoning relations between multiple OCR tokens, hashtags, and visual features. Further, we map all the modality-specific features to the same reference and utilize the self-attention mechanism (Parikh et al., 2016) to capture the relationships between them. Finally, we combine the vocabulary with OCR tokens which usually contain sarcastic intent to capture the incongruity for sarcasm generations. With the ability to capture the intra- and inter-modality incongruity, our model is thus capable of effectively generating sarcastic descriptions.
Our contributions can be summarized as follows:
• To the best of our knowledge, we are the first to investigate the Multi-modal Sarcasm Generation task, which aims to generate sarcastic descriptions like humans for a given image
with the help of hashtags.
• We develop MuSG, a new dataset consisting of 5000 image-text pairs for Multi-modal Sarcasm Generation. To our knowledge, it is the only dataset that can be applied to this task and evaluated automatically.
• We benchmark MuSG with a multi-modal Transformer-based model which can be served as a strong baseline.
• Empirical results show that our MTMSG outperforms all comparison models on all automatic evaluation metrics. We also perform extensive human evaluations to measure the Creativity, Sarcasticness, Coherence, and ImageText Relation of generated descriptions.

heading: 2 Related Work

heading: 2.1 Textual Sarcasm Generation
text: Recently, sarcasm generation has attracted tremendous attention in the field of natural language processing. The studies can be roughly categorized into twofold: Joshi et al. (2015) and Oprea et al. (2021, 2022) generate sarcasm with a response generator, while Peled and Reichart (2017), Mishra et al. (2019), and Chakrabarty et al. (2020) generate sarcasm with a paraphrase generator. However, these studies concentrate only on the generation of sarcasm in the textual domain, till now there has been no relevant effort on Multi-modal Sarcasm Generation. Applying machines to think and further imagine like humans is a creative and challenging task, fulfilling our imagination for the future of general artificial intelligence. Accordingly, we strongly believe that our proposed MSG will lead to a deeper understanding and expression of individuals’ intent on social media.

heading: 2.2 Research on Multi-modal Sarcasm
text: With the rapid development of mobile Internet, research on multi-modal sarcasm has come into focus. Schifanella et al. (2016) pioneer the multi-modal sarcasm and propose a dataset to collect 10000 sarcastic posts from Twitter, Instagram, and Tumblr. Cai et al. (2019) and Castro et al. (2019) extend the research and develop richer and better-formed datasets based on Twitter and conversational audiovisual utterances, respectively. Since then, a surge of studies have been conducted on multi-modal sarcasm, which can be roughly divided into three
categories: Multi-modal Sarcasm Detection (Pan et al., 2020; Xu et al., 2020; Liang et al., 2021, 2022), which aims to detect whether the input sample is sarcastic; Multi-modal Sarcasm Target Identification (Wang et al., 2022), which aims to extract sarcasm targets from both texts and images; Multi-modal Sarcasm Explanation (Kumar et al., 2022; Desai et al., 2022), which aims to generate a natural language sentence to explain the intended irony in the sarcastic posts. However, there is no research considering Multi-modal Sarcasm Generation. Leveraging the multi-modal information to create sarcastic descriptions will increase the variety of responses for intelligent conversational agents and further serve downstream applications such as personalized dialog systems (Cho et al., 2022) and news comment generation (Yang et al., 2019). Therefore, it is crucial to study the Multimodal Sarcasm Generation task.

heading: 3 Dataset and Metrics
text: In this section, we describe how the new dataset is constructed and how performance is evaluated.

heading: 3.1 Dataset
text: Since Twitter text contains varieties of sarcastic descriptions and intent detection in Twitter is a problem worth investigating, we focus on the Twitterbased dataset. We retrieve posts by querying hashtags to collect potential sarcastic samples. To create a well-formed and high-quality dataset MuSG for the MSG task, we collect publicly available Twitter posts using Twitter API and two existing multi-
modal sarcasm detection datasets (Schifanella et al., 2016; Cai et al., 2019) to obtain 5000 samples that have clear sarcasm targets. For text data, we remove external links and mentions (@email); We remove strange and meaningless symbols such as the token emoji-x and other special tokens which are hard to understand (♠); We also remove text with more than 40 words, because if the text is too long, it is much more difficult to generate by machine with the same inputs. For image data, we remove text-based images (images consist of text only), images with number-based OCR tokens (OCR tokens in the images consist of numbers only), images with too many visual objects, and images with low resolution. The MuSG dataset is randomly split into 3536/723/741 (using 5:1:1 split) as Train/Valid/Test in the experiments.
Further, we conduct a comprehensive statistical analysis of our collected MuSG dataset as follows: First, the content categories of Twitter contain six major categories: politics, sports, games, dining, life, and others, with politics and life accounting for the largest share, reaching half of the total (cf. Table 1). Second, we count the subjects of sarcastic sources, of which 27.3% originate from both images and hashtags, 32.4% from hashtags only, and 40.3% from images only, so we can conclude that most of the sarcasm can be generated with the help of images and hashtags (cf. Table 2). Third, we count the size of the sarcasm targets, and we find that 25.8% of the sarcasm targets are small targets, making the MSG task more challenging (cf. Table 3). Finally, we also count the sentence style of the
ground truth, in which 56.3% are declarative sentences, 40.9% are imperative sentences, and only 2.8% are interrogative sentences or other sentences, which represents that sarcastic descriptions express a definite emotion in most cases (cf. Table 4).

heading: 26.4% 8.7% 5.4% 13.4% 28.4% 17.7%

heading: 3.2 Evaluation Metrics
text: We evaluate the generated descriptions both quantitatively (with standard automatic evaluation metrics) and qualitatively (with human evaluation metrics). For automatic evaluation metrics, we apply the Microsoft coco caption evaluation, which includes BLEU (B1, B2, B3, and B4) (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014), ROUGE (R1, R2, and R_L) (Lin, 2004), CIDErD (Vedantam et al., 2015), and SPICE (Anderson et al., 2016). For human evaluation metrics, we propose a set of 4 criteria to evaluate the generated descriptions: 1)Creativity (“How creative are the generated descriptions?”), to judge if the generated descriptions are novel and attractive; 2)Sarcasticness (“How sarcastic are the generated descriptions?”), to judge the degree of sarcasm (including irony and humor); 3)Coherence (“How coherent are the generated descriptions?”), to judge if the generated descriptions are fluent and further easy to understand; 4)Image-text Relation (“How relevant are the images and the generated descriptions?”), to judge if the generated descriptions are highly correlated with the given images.

heading: 4 Methodology
text: In this section, we describe our proposed MTMSG, a Multi-modal Transformer-based model for MSG. The input to this task is an image and the hashtags of the corresponding Twitter text, while the output is the generated descriptions that need to compare with the original Twitter text (Ground Truth). Yet, demonstrated by Pan et al. (2020), the information of OCR text usually provides the context of sarcasm, which may contribute to sarcasm generation. Therefore, we leverage the information from visual objects, hashtags, and OCR tokens for MSG.
The architecture of the proposed MTMSG is illustrated in Figure 2. Specifically, we first embed the three modalities in the same reference, and then feed them into a multi-modal Transformer to achieve intra- and inter-modality interactions. Finally, our models learn to generate sarcastic descriptions through iterative decoding with the help of a dynamic pointer network. In the decoding process, we leverage the previous output to predict the next generated word in an auto-regressive manner.

heading: 4.1 Uni-modal Feature Embedding

heading: 4.1.1 Hasgtag Embedding
text: Given a hashtag as a sequence of K words, we utilize FastText (Bojanowski et al., 2017) as the feature extractor to get the 300-dimensional vector xftk (k = 1, · · · ,K), which is a word embedding with sub-word information. Finally, we project the vector to a 768-dimensional semantic space to make sure the features from different modalities are embedded in the same reference. The final hashtag embedding xhtagk is obtained by:
xhtagk = LN(W1x ft k ), (1)
where W1 is the learnable parameter and LN denotes layer normalization.

heading: 4.1.2 Visual Object Embedding
text: Given an image, we apply pretrained Faster R-CNN (Ren et al., 2015) as the detector to obtain the appearance feature xfrm of m-th visual object. Further, to leverage the spatial information of each object, we investigate a 4-dimensional location feature by xbm = [xmin/W, ymin/H, xmax/W, ymax/H]. Then we can obtain a list of 768-dimensional vectors xobjm as follows:
xobjm = LN(W2x fr m ) + LN(W3x b m), (2)
where W2 and W3 are learnable parameters, and LN denotes layer normalization.

heading: 4.1.3 OCR Token Embedding
text: For OCR token embedding, following the M4CCaptioner (Sidorov et al., 2020), to get a rich representation of OCR tokens, we leverage FastText (Bojanowski et al., 2017), Faster R-CNN (Ren et al., 2015), PHOC (Almazán et al., 2014) as the feature extractors to get sub-word feature xft, appearance feature xfr, and character-level feature xp, respectively. Given a set of N OCR tokens, the location feature of the n-th token is represented as xbn = [xmin/W, ymin/H, xmax/W, ymax/H]. Then the final OCR token embedding xocrn is projected to a 768-dimensional vector as:
xocrn = LN(W4x ft n +W5x fr n +W6x p n)+LN(W7x b n), (3)
where W4, W5, W6, and W7 are learnable parameters, and LN denotes layer normalization.

heading: 4.2 Multi-modal Transformer
text: After extracting uni-modal feature embedding from three modalities, we concatenate the features and feed them into the multi-modal Transformer. In the multi-modal Transformer, the features fully interact to exploit intra- and inter-modality incongruity. Besides, the previous step of decoding output xdect−1 is also embedded and fed into the Transformers. Finally, the multi-modal Transformers obtain feature vectors as output: [zhtag, zobj , zocr, zdect−1] = MMT ([xhtag, xobj , xocr, xdect−1]), where MMT denotes multi-modal Transformer.

heading: 4.3 Sentence Decoder
text: The sentence decoder takes the feature embedding output of multi-modal Transformers as input, predicts the score for each word, and selects the predicted word of each time step. We generate sarcastic descriptions through the auto-regressive paradigm. Remarkably, the OCR tokens detected in images usually contain intent information for capturing the multi-modal incongruity, while they are usually not involved in the common word vocabulary, so it is inappropriate to make predictions based on the fixed vocabulary. Therefore, we adopt different classifiers for common vocabulary and OCR tokens as follows:
yt = argmax(f(zdect−1), fDPN (z dec t−1, z ocr)), (4)
where f indicates the linear classifier for common vocabulary, and fDPN indicates the dynamic
pointer network(Vinyals et al., 2015). The captioning loss is computed by:
L = −log ˙T∑
i=0
(P (yt|(y0:T−1, zhtag, zobj , zocr))), (5)
where y0:T−1 denotes the generated sequence.

heading: 5 Experiments

heading: 5.1 Comparison Models
text: Due to the multi-modal nature of the input corpus, we compare our proposed MTMSG model with three categories of strong models adapted for the MSG task as follows:
1)Image-modality models: These models only leverage the visual features to generate sarcastic description, including ViT (Dosovitskiy et al., 2020), a powerful visual Transformer (with BART as the decoder); and BLIP (Li et al., 2022), a pre-trained image captioning model finetuned on this task.
2)Text-modality models: These models only leverage the hashtags and OCR tokens to generate sarcastic descriptions, including Transformer (Vaswani et al., 2017) and Chandler (Oprea et al., 2021), a very recent effort that generates sarcastic response to a given textual utterance.
3)Multi-modality models: These models utilize the information from images, hashtags, and OCR tokens for MSG, including MFFG (Liu et al., 2020), a multi-stage fusion mechanism with a forget fusion gate (both RNN and Transformer variants of MFFG); and MMT (Tang et al., 2022), a multi-modal Transformer for multi-modal learning.

heading: 5.2 Experiment Settings
text: For visual objects, we extract 100 object appearance features with the dimension 2048. Besides, we apply Google OCR API to detect sufficient OCR tokens with bounding boxes. The number of OCR tokens for each image is limited to 30 at most. We set the layer number of the multi-modal Transformer to 4 and the number of self-attention heads to 12. Following BERT-base (Devlin et al., 2019), we adopt default settings for other parameters. For MSG, we tokenize the text on whitespace and filter the special symbols that the model cannot recognize. The fixed common vocabulary has 11554 words. Furthermore, we train the model for 160 epochs on a single 3090Ti GPU, and the batch size is set to 64. We adopt the Adam optimizer, the initial learning rate is 1e-4 and declined to 0.1 times every 50 epochs. We monitor the CIDEr-D
metric to choose the best model and evaluate it on the test set. Finally, we average the experimental results of our MTMSG over ten runs to ensure the statistical stability of the experimental results.

heading: 6 Experimental Results and Analysis

heading: 6.1 Main Results
text: Table 5 presents the comparative generation performances on the dataset. The experimental results illustrate that our model achieves the best performance across all of the competing strong baselines adopted for this task. Specifically, our model obtains BLEU scores of 21.37 (+2.75), 13.32 (+5.68), 8.67 (+4.63), and 6.37 (+3.73) on B1, B2, B3, and B4, respectively. Similarly, we exceed 4.42, 3.38, and 4.35 Rouge points at 26.44, 11.38, and 24.12 on R1, R2, and R_L, respectively. Our model also achieves improved performance on METEOR 16.05 (+3.1), and CIDEr-D 48.6 (+13.4).
Moreover, we can draw the following conclusions: 1) Notably, our proposed MTMSG outperforms existing strong baselines on all of the evaluation metrics (Significance Test, all p-value<0.05), which demonstrates the effectiveness of our proposed multi-modal Transformer-based model for MSG. 2) Models based on text modality perform better than the models based on image modality, which indicates more sarcastic information lies in the hashtags and OCR tokens. 3) Multi-modal models achieve much better performance than the unimodal baselines overall, which implies that leveraging the information from images, hashtags, and OCR tokens is efficacious for MSG.

heading: 6.2 Human Evaluation
text: We also perform the human evaluation for assessing the quality of the MSG. We randomly select 200 samples from the test set. Given the provided
information (image, hashtags, and OCR) together with the description generated by ours and the other 4 strong baselines (BLIP, Chandler, MFFG, MMT). Each criterion is rated from 1 (not at all) to 5 (very). We employ 5 evaluators to independently score the generated sarcastic descriptions from the five methods and the ground truth. Figure 3 shows the instructions released to the evaluators.
The comparative results are shown in Table 6. To measure the inner-annotator agreement, we calculated Fleiss’ kappa and all the results show fair agreement (0.2 ≤ κ ≤ 0.4). From the diagram, we can obtain the following conclusions: 1) BLIP is superior in the research of image caption, so this method works well on the metrics of Coherence
and Image-Text Relation. Notably, it works even better than the ground truth, which further demonstrates that the Twitter text posted by humans is creative and full of imagination. 2) For Coherence and Sarcasticness, our proposed MTMSG performs better than any other baselines, which demonstrates that our model meets the basic target of the MSG task. 3) Moreover, we can see that all the baselines adopted for this task show poor performance on the metrics of Creativity and Image-Text Relation, which illustrates that more studies on improving the quality of the generated descriptions are needed. 4) Overall, the results strengthen that our model is superior to all the other baselines. However, a significant gap in performance still remains between humans and machines, which demonstrates that our proposed MSG task is challenging and worth further in-depth research.

heading: 6.3 Ablation Study
text: We conduct an ablation study on the effectiveness of the three input modalities. Table 7 gives the
experimental results of the ablation study. From the perspective of input corpus, we can draw the following conclusions: 1) Note that the removal of hashtags (w/o htag) significantly degrades the performance, which verifies the significance of guiding the direction of sarcasm generation as hashtags indicate the topic of Twitter; 2) Since the context of sarcastic information resides in the OCR tokens, it is hard to understand the sarcastic intent without the OCR tokens. As a result, removing the OCR tokens (w/o OCR) also leads to considerable performance decline; 3) Besides, from the results of w/o visual, we can conclude that the visual features are also beneficial to MSG; 4) Moreover, from experimental results which only utilize one modality at a time to observe how much modality-specific information contributes to the generation, we can conclude that textual information plays a major role in sarcasm generation. More remarkably, concentrating on our models, we can get the conclusions as follows: 1) Dynamic pointer network can predict the copying score between the decoding output and each OCR token. Since the OCR tokens and the common word vocabulary are usually complementary, the removal of the pointer network (w/o pointer) leads to apparent performance degradation; 2) Finally, without leveraging the iterative decoding method (w/o iterative), we only decode for one step. The sharp performance degradation indicates the iterative decoding strategy can improve the quality of generated sarcastic descriptions.

heading: 6.4 Case Study
text: We present some examples to analyze the performance of our model. Specifically, in Figure 4 (a), we can find that without hashtags and OCR tokens, our model can only describe a man in a black car. The hashtag and OCR token provide the timing factor Sunday and FUNDAY may guide the intention of sarcasm, which demonstrates the information from hashtags and OCR tokens is necessary for sarcasm generation. Yet MMT simply concatenates the visual information with the timing factor Sunday, and the generated description seems not to be sarcastic. Even sometimes the hashtag is not so useful, like RTX Austin in Figure 4 (b), it just tells the place of the scene. We can also understand the main intent of the sarcasm from the OCR tokens (behind schedule). Associated with the image, we can understand that the flight is behind schedule, which can help generate a sarcastic description.
While MMT still generates a literal description of the image (plane at the airport). Similarly, in Figure 4 (c), from the hashtag, we know that the sarcastic target is about speed, and from the OCR token, we can get that the intent is to express that speed is slow; in Figure 4 (d), from the hashtag, we know that the sarcastic target is about Monday morning, and from the OCR token, we can get that the intent is to express the hope of optional Monday.
If we understand the intent, we can easily get better sarcastic descriptions instead of literal descriptions like MMT. All the examples demonstrate that with the help of proposed images, hashtags, and OCR tokens, our model is capable of generating sarcastic descriptions like humans to some degree.

heading: 7 Conclusion
text: In this paper, we introduce a novel task of Multimodal Sarcasm Generation, aiming to generate sarcastic descriptions like humans. To address the task, we investigate a new dataset, MuSG, containing 5000 images with corresponding sarcastic descriptions. Further, we propose a strong baseline, MTMSG, to benchmark the MuSG dataset. Machine evaluation metrics demonstrate that our proposed MTMSG outperforms various comparison baselines. Moreover, the human evaluation shows that our proposed MSG task is challenging and worth further in-depth research. We consider that MSG opens a new avenue in the domains of sarcasm understanding and generation. In the future, we will explore the detection of key information from the images and the understanding of the intent from the OCR tokens.
Limitations
To better understand the limitations of our proposed MTMSG, we also perform a qualitative error analysis of the incorrectly generated samples. We randomly select 100 incorrectly generated descriptions and find that our model might incorrectly generate those samples mainly due to the misunderstanding of the necessary intent information from the images and OCR tokens. The statistical results reveal that 37% of the incorrectly generated descriptions are caused because the main part of the sarcasm might lie in the images (eg. Figure 5 (a)), while the other 63% error cases are attributed to the failure of our model in capturing the intent information directly from the OCR tokens (eg. Figure 5 (b)). Specifically, in Figure 5 (a), if we want to generate better descriptions, we need to capture the fine-grained visual attribute feature happy from the image; In Figure 5 (b), we need to understand the intent information from the OCR tokens that ban guns can make us feel safe when we have lunch in the restaurant. Therefore, to address the above issues in the future, we will further explore the fine-grained key information in the images to help guide the MSG. Besides, we will explore a language interpreter to further understand the key information contained in the OCR tokens.

heading: Acknowledgements
text: We thank anonymous reviewers for their valuable comments and thoughtful suggestions. This work was supported by the National Natural Science Foundation of China (62276072, 62076100,and 62261003), the Guangxi Natural Science Foundation (No. 2022GXNSFAA035627), Guangxi Scientific and Technological Bases and Talents Special Projects (Application No. 2022AC21300, 2022AC21254), the Open Research Fund of Guangxi Key Laboratory of Multimedia Communications and Network Technology, and the Open Research Fund of Key Laboratory of Big Data and Intelligent Robot (SCUT), Ministry of Education.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, in the Section "Error Analysis”, we describe the limitation of our work.
7 A2. Did you discuss any potential risks of your work? No, our work does not involve ethical issues.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, from the section of the Abstract and Introduction, reviewers can easily get the main idea of our work.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, from the section of Reference, Related Work, "Dataset and Metrics”, and Abstract.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, from the section of Reference.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, we only utilized artifacts that are publically avaliable.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, from the section of Related Work.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Yes, from the section of "Dataset and Metrics”.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Yes, from the section of Abstract and "Dataset and Metrics”.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, from the section of"Dataset and Metrics”.
C 3 Did you run computational experiments? Yes, from the section on Experiments and Experimental Results.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes, from the section of Experiments.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, from the section of Experiments.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, from the section of Experiments and Experimental Results.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, from the section of "Dataset and Metrics” and Experiments.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, from the section of Experimental Results "Human Evaluation” part.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Yes, from the section of Experimental Results "Human Evaluation” part.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Yes, from the section of Experimental Results "Human Evaluation” part.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Yes, from the section of Experimental Results "Human Evaluation” part.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No, there is no ethics review board being involved.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No, we only ask some evaluators for human evaluation.

heading: Future Work
text: In the future, we will explore the detection of key information from the images and the understanding of the intent from the OCR tokens. Limitations
To better understand the limitations of our proposed MTMSG, we also perform a qualitative error analysis of the incorrectly generated samples. We randomly select 100 incorrectly generated descriptions and find that our model might incorrectly generate those samples mainly due to the misunderstanding of the necessary intent information from the images and OCR tokens. The statistical results reveal that 37% of the incorrectly generated descriptions are caused because the main part of the sarcasm might lie in the images (eg. Figure 5 (a)), while the other 63% error cases are attributed to the failure of our model in capturing the intent information directly from the OCR tokens (eg. Figure 5 (b)). Specifically, in Figure 5 (a), if we want to generate better descriptions, we need to capture the fine-grained visual attribute feature happy from the image; In Figure 5 (b), we need to understand the intent information from the OCR tokens that ban guns can make us feel safe when we have lunch in the restaurant. Therefore, to address the above issues in the future, we will further explore the fine-grained key information in the images to help guide the MSG. Besides, we will explore a language interpreter to further understand the key information contained in the OCR tokens