Dynamic Routing Transformer Network for Multimodal Sarcasm Detection
Year: 2023

Abstract:
Multimodal sarcasm detection is an important research topic in natural language processing and multimedia computing, and benefits a wide range of applications in multiple domains. Most existing studies regard the incongruity between image and text as the indicative clue in identifying multimodal sarcasm. To capture cross-modal incongruity, previous methods rely on fixed architectures in network design, which restricts the model from dynamically adjusting to diverse imagetext pairs. Inspired by routing-based dynamic network, we model the dynamic mechanism in multimodal sarcasm detection and propose the Dynamic Routing Transformer Network (DynRT-Net). Our method utilizes dynamic paths to activate different routing transformer modules with hierarchical co-attention adapting to cross-modal incongruity. Experimental results on a public dataset demonstrate the effectiveness of our method compared to the stateof-the-art methods. Our codes are available at https://github.com/TIAN-viola/DynRT.

heading: 1 Introduction
text: Sarcasm is a widely used figurative language to give the ironic expression in our daily life, which typically means the opposite of what it really wants to express (Joshi et al., 2017). As an important step to analyze people’s opinions and sentiments in communication, sarcasm detection benefits a wide range of applications such as natural language dialogue (Tepperman et al., 2006), public opinion mining (Riloff et al., 2013) and social media analysis (Tsur et al., 2010). With the rapid growth of multimodal user-generated content, multimodal sarcasm detection has gained increasing research attention in recent years (Cai et al., 2019; Xu et al., 2020; Pan et al., 2020; Wang et al., 2020; Liang et al., 2021; Pramanick et al., 2022; Liang et al.,
*Corresponding author
2022; Liu et al., 2022), and has become an important research topic in natural language processing and multimedia computing.
The sarcastic clues of multimodal contents are mainly relevant to the incongruity across image and text (Xu et al., 2020; Pan et al., 2020; Wang et al., 2020; Liang et al., 2021; Pramanick et al., 2022; Liang et al., 2022; Liu et al., 2022). Existing studies model this characteristic of incongruity between image and text with various approaches, including decomposition and relation network (Xu et al.,
2468
2020), attention mechanisms (Wang et al., 2020; Pan et al., 2020), graph-based methods (Liang et al., 2021, 2022), and optimal transport method (Pramanick et al., 2022). In addition, external knowledge is also introduced to boost the performance of multimodal sarcasm detection (Liu et al., 2022).
As it is shown in multimodal samples in Figure 1, there are diverse kinds of sarcastic image-text pairs. In some cases, the image and text express the incongruous meaning with local segments, where visual regions or objects are contrastive to the meaning of words or phrases in the text, as those in Figure 1 (a) and (b). In other cases, the feelings implied in the image and text respectively are totally opposite, as those in Figure 1 (c) and (d). To detect these sarcastic image-text pairs, current approaches mainly focus on modeling the cross-modal incongruity. However, these methods rely on static networks to capture the characteristic of incongruity, which use fixed architectures on different kinds of inputs, thus lacking the flexibility to adapt to diverse image-text pairs.
To tackle this problem, the dynamic aspect of incongruity between image and text should be considered. One possible solution is to model dynamic mechanism with a routing-based dynamic network, where a series of modules can capture the incongruity between image and text dynamically via selecting one or more most suitable modules according to different image-text pairs. Existing routing-based method in multimodal dynamic networks (Zhou et al., 2021) performs routing only on single-modality data, which is insufficient to model the dynamic image-text incongruity in cross-modal sarcasm detection. Therefore, we extend the existing routing scheme to multimodal setting with dynamic network design, aiming to better model the dynamic mechanism for multimodal sarcasm detection.
In this paper, we propose a novel Dynamic Routing Transformer Network, namely DynRTNet, whose router helps model route on dynamic routing transformer modules with hierarchical co-attention adapting to cross-modal incongruity prevalent in diverse image-text pairs. The main contributions of our work are as follows:
• We identify the diversity of image-text sarcastic pairs, and for the first time, model crossmodal incongruity with dynamic network design, which focuses on the dynamic mechanism for multimodal sarcasm detection.
• We propose a dynamic routing transformer network via adapting dynamic paths to hierarchical co-attention between image and text conditioned on multimodal samples, which is capable of capturing cross-modal incongruity dynamically.
• Experimental results on a public dataset demonstrate the effectiveness of our proposed method for multimodal sarcasm detection.

heading: 2 Related Work

heading: 2.1 Image-text Sarcasm Detection
text: Traditional sarcasm detection mainly studies the sarcastic information in textual utterances (Zhang et al., 2016; Tay et al., 2018). With the prevalence of social media, many people tend to express their thoughts with sarcasm using both textual and visual messages online. Early studies utilize simple fusion methods of visual and textual information for multimodal sarcasm classification, such as concatenation of textual and visual embeddings (Schifanella et al., 2016) or hierarchical fusion representation of modalities (Cai et al., 2019). As multimodal sarcasm is often associated with an implicit incongruity between image and text, some studies capture this basic characteristic to detect multimodal contrast from various perspectives, such as modeling cross-modality contrast and semantic association simultaneously (Xu et al., 2020) or modeling intra-modality and inter-modality incongruity using attention mechanisms (Wang et al., 2020; Pan et al., 2020).
To represent more explicit incongruous relations, recent studies employ graph convolution networks to construct in-modal and cross-modal graphs for this task (Liang et al., 2021, 2022). Furthermore, Pramanick et al. (2022) utilize self-attention to model the intra-modal relation and optimal transport to model the cross-modal relation for multimodal sarcasm detection. In addition, Liu et al. (2022) explore external knowledge resources like image captions to enhance the model performance for image-text sarcasm detection.
Despite the promising results achieved for imagetext sarcasm detection, existing approaches rely on fixed architectures in network design. And thus, the computation mechanism to capture the cross-modal incongruity is static, which hinders the model from dynamically adjusting to diverse multimodal samples.

heading: 2.2 Multimodal Dynamic Networks
text: Multimodal dynamic networks have shown good performance on multimodal tasks (de Vries et al., 2017; Perez et al., 2018; Zhou et al., 2021; Qu et al., 2021), which can be roughly divided into two categories: dynamic parameters and dynamic architectures. A typical model with dynamic parameters adapts its weights based on different inputs in the inference stage. For example, Perez et al. (2018) propose a model to adjust the parameters of ResNet conditioned on the text information for visual reasoning. Dynamic architectures adapt the network depth and width or perform routing according to different inputs. For example, Zhou et al. (2021) design a data-dependent routing scheme called Transformer Routing (TRAR) to dynamically select image attentions for visual question answering.
Routing-based method has the potential to dynamically identify cross-modal incongruity via activating different modules dynamically conditioned on different image-text inputs. However, the current work TRAR only performs routing on singlemodality data. To better model the dynamic mechanism in cross-modal sarcasm detection, we extend the existing routing scheme to multimodal setting with dynamic network design.

heading: 3 Method
text: Figure 2 shows the overall architecture of our proposed dynamic routing transformer network
DynRT-Net, which is composed of three components: encoding, dynamic routing transformer, and classification. We first encode the text and a paired image into multimodal features respectively via two pre-trained models. Then, we feed them into the dynamic routing transformer to route on hierarchical co-attention dynamically and learn crossmodal incongruity, resulting in the routed features with cross-modal information. Finally, we feed the routed features and image features into the classifier for multimodal sarcasm classification.

heading: 3.1 Encoding
text: Text Encoder To train our model from a good start of text embeddings, we use the pre-trained model RoBERTa (Liu et al., 2019) as the text encoder, which has implicitly acquired world knowledge from the large-scale dataset. We first split the text into a sequence of tokens Text = {[CLS] , w1, . . . , wn−1}, where [CLS] denotes the global token and n is the length of all the tokens. After that, we feed Text into RoBERTa and get text features T ∈ Rn×dt , which are represented by
T = RoBERTa (Text) = [t1, t2, . . . , tn] , (1)
where ti ∈ Rdt is the text embedding of i-th token wi in the text and dt is the dimension of text embedding.
Image Encoder To train our model from a good start of image embeddings, we use a pre-trained Vision Transformer (ViT) model (Dosovitskiy et al.,
2021) as the image encoder, which has recently achieved excellent performance. We first split an Image ∈ RH×W×C into a sequence of m flattened 2D patches, where H , W and C denote the height, width, and the number of channels of the image. After that, we feed Image into ViT and get image features I ∈ Rm×dv of patches, which are represented by
I = ViT (Image) = [e1, e2, . . . , em] , (2)
where ej ∈ Rdv is the image embedding of j-th patch in the image and dv is the dimension of image embedding.

heading: 3.2 Dynamic Routing Transformer
text: Previous approaches (Xu et al., 2020; Pan et al., 2020; Wang et al., 2020; Liang et al., 2021; Pramanick et al., 2022; Liang et al., 2022; Liu et al., 2022) capture the incongruity between image and text for multimodal sarcasm detection in a static manner, and thus are unable to dynamically adjust to diverse image-text pairs. To fill this gap, we propose the Dynamic Routing Transformer (DynRT), which performs routing on hierarchical co-attention of two modalities to capture cross-modal incongruity adapting to different image-text inputs.

heading: 3.2.1 Routing Space
text: In the Dynamic Routing Transformer, we feed the textual and visual embeddings to several DynRT
layers, which can be calculated as
Tk = DynRTk(Tk−1, I), k ∈ [1,K], (3) where Tk is the output of k-th DynRT layer, T0 = T is the input of the first layer, K is maximum index of DynRT layers, and the output of the last DynRT layer TK is the final routed features.

heading: 3.2.2 Dynamic Routing Transformer Layer
text: Unlike the previous dynamic method TRAR (Zhou et al., 2021), which performs routing on attention grids of one modality, our DynRT layer routes on hierarchical co-attention of image and text conditioned on different inputs (see Figure 3 for a detailed comparison). Our DynRT layer is composed of a multi-head co-attention routing (MHCAR) module (pink rectangle in Figure 3 (c)), a multi-head self-attention (MHA) module and a feed-forward network (FFN), where a residual connection and a normalization layer (LN) (Ba et al., 2016) follow each module. The k-th DynRT layer can be formulated as
T rk−1 = LN(MHCARk(Tk−1, I) + Tk−1), (4)
T ak−1 = LN(MHAk(T r k−1) + T r k−1), (5)
Tk = LN(FFNk(T a k−1) + T a k−1), (6)
where k ∈ [1,K] is the index of DynRT layers, Tk ∈ Rn×dt is the output of k-th DynRT layer, T rk−1 ∈ Rn×dt and T ak−1 ∈ Rn×dt are the output of MHCAR module and MHA module respectively.
The MHCAR in k-th DynRT layer performs h heads of attention functions in parallel with the hidden dimension dh (dh = dt/h) which are concatenated and then projected, resulting in the final values of the MHCAR, which is calculated as MHCARk(Tk−1, I) = concat ( [headki ] h i=1 ) OkT , (7) where concat(·) is the concatenation operation, OkT ∈ Rdt×dt is the projection matrix and every head headki ∈ Rn×dh is calculated by a coattention routing (CAR) function, which routes on co-attention (CA) functions with different coattentions:
headki = CAR k i (Tk−1, I)
=
pk−1∑
j=0
αkj CA k i,j(Qi,j,k,Ki,j,k, V k i,j , A j)
=
pk−1∑
j=0
αkjσ
( Qi,j,kK
i,j,k√ dh ⊗Aj
) V ki,j ,
(8)
where σ(·) denotes the softmax function, αkj is the routing probability weight of j-th CA function with one kind of co-attention mask Aj between image and text, pk is the number of CA functions in k-th layer (we set pk = k in our model), Mi,j,k = Qi,j,kK i,j,k ∈ Rn×m is the attention matrix between two modalities in headki , Qi,j,k = Tk−1W Q i,j,k,Ki,k = IW K i,j,k, V k i,j = IW Vi,j,k, W Q i,j,k ∈ Rdt×dh , WKi,j,k ∈ Rdv×dh and W Vi,j,k ∈ Rdv×dh are parameter matrices, K i,j,k denotes the transpose of matrix Ki,j,k, and ⊗ denotes element-wise matrix product. The hierarchical coattention mechanism and construction of Aj will be presented in the following section 3.2.3. The prediction of αkj is controlled by a router, which will be presented in the following section 3.2.4.
To reduce the computation of the routing process in Eq. (8), we follow Zhou et al. (2021) to redefine the headki as
headki = σ
⎛ ⎝Qi,kK
i,k√ dh ⊗
pk−1∑
j=0
αkjA j ⎞ ⎠V ki . (9)

heading: 3.2.3 Hierarchical Co-attention
text: We first describe how to construct the co-attention mask matrix Aj in Eq. (8)(9). Aj restricts the region of the image that text can see in the CA function. The s-order sliding window with a small patch of (2s+1)× (2s+1) grid traverses every patch of the image to get mask vector vsl ∈ Rm (l ∈ [1,m]), whose visualization is shown in Figure 4. We construct As by stacking the vector vsl for n times (n is the length of tokens) from vs1 to v s m circularly:
As = [vs1, v s 2, . . . , v s n] ∈ Rn×m. (10)
Specifically, A0 is an empty mask matrix, i.e. a matrix of all the ones, which gives words or global token [CLS] the opportunity to see the whole image.
To model the cross-modal incongruity in diverse image-text pairs gradually, we then design the hierarchical co-attention via making the kinds of co-attention masks diverse progressively with the increase of DynRT layers, the architecture of which is shown in Figure 2. In the k-th layer of DynRT, the group of co-attention mask matrices in Eq. (8)(9) that router can route on is Gk = [A
0, A1, . . . , Apk−1], where pk = k is the number of mask matrices in k-th DynRT layer (pk also equals to the number of CA functions in Eq. (8)(9)).

heading: 3.2.4 Router
text: The routing probability αk = [αk0 , α k 1 , . . . , α k pk−1] for k-th DynRT layer can be obtained by the router conditioned on the input, which is calculated as
αk = σg (MLP (APool (I))) ∈ Rpk , (11)
where σg(·) is Gumble Softmax (Zhou et al., 2021) with temperature t, APool(·) is a 1D adaptive average pooling over all the embeddings of patches in the image, MLP is a two-layer multilayer perceptron with hidden dimension dm, and pk is also the number of co-attention mask matrices in the k-th DynRT layer where αk works in Eq. (8)(9).

heading: 3.3 Classification
text: Finally, we project the image features I and routed features TK into global embeddings and predicts sarcastic tendency, which can be formulated as
Ig = Mean(I), (12)
Tg = Mean(TK), (13)
yg = Wg(LN(Ig + Tg)) + bg, (14)
ŷ = Softmax(Woyg + bo), (15)
where Mean(·) is the average function on all the patch embeddings in I and all the token embeddings in TK , Ig ∈ Rdv and Tg ∈ Rdt denote global embeddings of image and text respectively, LN(·) is the layer normalization , yg ∈ Rd is the global multimodal embedding (considering dv = dt = d
in our model, we omit the process of projecting embeddings of two modalities into the same dimension), Wg ∈ Rd×d, bg ∈ Rd, Wo ∈ Rdp×d and bo ∈ Rdp are trainable parameters, Softmax(·) is the softmax function, ŷ ∈ Rdp is the predicted probability of all the possible labels, and dp is the number of possible labels (i.e. sarcastic and nonsarcastic).

heading: 3.4 Optimization
text: We optimize our model with cross-entropy loss, which is most commonly used in classification:
L = − N∑
i=1
y i log ŷi, (16)
where y is the ground truth and ŷi is the probability of predicted label for i-th image-text pair.

heading: 4 Experiments

heading: 4.1 Dataset
text: We evaluate our method on the Multimodal Sarcasm Detection (MSD) dataset (Cai et al., 2019), which is the only benchmark dataset for multimodal sarcasm detection. Cai et al. (2019) collect original image-text pairs from Twitter and randomly divide this dataset into the training set, development set, and test set with the ratio of 80%:10%:10%. The statistics of the MSD dataset are shown in Table 1. Cai et al. (2019) further discard tweets with regular words (sarcasm, sarcastic, reposting, irony, ironic, jokes, humor, humour and exgag) and URLs, and replace mentions with a certain symbol 〈user〉. For a fair comparison, we use the MSD dataset after the above data preprocessing for experimentation, following the convention of all the previous studies.

heading: 4.2 Experimental Settings
text: The values of hyper-parameters are shown in Table 2. More information about experimental settings is shown in Appendix B.

heading: 4.3 Baseline Methods
text: We compare our method with existing unimodal baselines and representative methods for multimodal sarcasm detection.
Image-modality methods. The baseline methods using the image information for sarcasm detection are as follows:
• ResNet (Cai et al., 2019) uses the image embedding of the pooling layer of ResNet (He et al., 2016) for sarcasm classification;
• ViT (Dosovitskiy et al., 2021) is a pre-trained vision model based on Transformer architecture, which achieves excellent results.
Text-modality methods. The baseline methods using text information for sarcasm detection are as follows:
• TextCNN (Kim, 2014) is a network based on CNN for textual classification;
• Bi-LSTM (Liang et al., 2022) is a Bi-LSTM network for textual classification;
• SIARN (Tay et al., 2018) employs the attention mechanism for textual sarcasm detection;
• SMSD (Xiong et al., 2019) proposes a selfmatching network for sarcasm detection;
• BERT (Devlin et al., 2019) is a classical pretrained language model;
• RoBERTa (Liu et al., 2019) is an optimized BERT pre-trained language model.
Multimodal methods. The representative methods employing both image and text for sarcasm detection are as follows:
• HFM (Cai et al., 2019) fuses the information of text, image, and image attributes with a hierarchical network;
• D&R Net (Xu et al., 2020) uses a decomposition network and a relation network to exploit the contrastive and relative relationship between image and text;
• IIMI-MMSD (Pan et al., 2020) utilizes selfattention and co-attention mechanisms to model the intra-modality and inter-modality incongruity between image and text;
• Bridge (Wang et al., 2020) proposes a bridge layer based on RoBERTa and ResNet to capture the relationship between two modalities;
• InCrossMGs (Liang et al., 2021) utilizes a graph-based model to capture sarcastic relations between image and text;
• MuLOT (Pramanick et al., 2022) employs self-attention to learn intra-modal correspondence and optimal transport to learn crossmodal correspondence;
• CMGCN (Liang et al., 2022) proposes crossmodal graphs based on attribute-object pairs of image objects to capture sarcastic clues;
• Hmodel (Liu et al., 2022) models both atomiclevel incongruity and composition-level congruity with attention mechanism and graph neural networks respectively;
• HKEmodel (Liu et al., 2022) incorporates image captions as the external knowledge to enhance the ability of Hmodel to detect multimodal sarcasm, which is the state-of-the-art model in multimodal sarcasm detection.

heading: 4.4 Main Results
text: Following Liang et al. (2022), we use accuracy and macro-average F1-score as the evaluation metrics. Table 3 shows the comparative results of the representative methods and our method, which demonstrate that our proposed method outperforms all the baseline methods and achieves significant gains compared with the state-of-the-art method. For unimodal methods, text-modality methods achieve better performances than image-modality methods, which shows that textual information provides more sarcastic clues within modality than visual information. Compared with unimodal methods, multimodal methods perform better, which indicates that cross-modal interaction is important to capture
multimodal sarcastic meanings in image-text pairs. The pre-trained models, which have learned large world knowledge related to background information of the multimodal sarcasm, help recent methods achieve significant improvements compared with HFM and D&R Net, which use shallow networks to model the interaction between image and text. IIMI-MMSD, Bridge, InCrossMGs, MuLOT, CMGCN and Hmodel provide multiple perspectives to capture the implicit incongruity in imagetext pairs for cross-modal sarcasm detection and achieve gradually improved performances. However, their architectures are static and inflexible, leading to computing redundancy and lacking the adaptability to diverse image-text pairs. In contrast, our method gains a great increase via adapting dynamic paths to hierarchical co-attention of image and text with dynamic network design. In addition, our method also performs better than HKEmodel, which uses external knowledge to enhance the performance. This result further verifies the effectiveness of our simple and dynamic method in capturing the cross-modal incongruity between image and text.

heading: 4.5 Ablation Study
text: We conduct the ablation study to evaluate the impact of different components in our proposed model, using the following variants:
• DynRT-Net (pk = K): sets the pk in each
DynRT layer as K, which connects the same four DynRT layers with four co-attention mask matrices to replace DynRT layers with hierarchical co-attention in our model;
• DynRT-Net (pk = K − k + 1): sets pk as K − k + 1, which reduces the number of the types of co-attention mask matrices from four to one with the increase of DynRT layers;
• - DynRT, + TRAR: replaces the DynRT layer in our model with another routing-based scheme TRAR layer;
• - DynRT, + Standard Transformer: replaces the DynRT layer with the standard multimodal transformer layer;
• - DynRT, + Concatenation: removes DynRT layers in our model and feeds the concatenation of classification vectors of text encoder and image encoder to the final classifier;
• - Dynamic attention, + mean attention: replaces the dynamic attention scores predicted by the router with the average distribution of attention scores in every DynRT layer;
• - Dynamic attention, + fixed attention: replaces the dynamic attention score for the empty co-attention mask matrix with 1 and replaces the dynamic attention scores for other types of co-attention mask matrices with 0 in every DynRT layer.
Table 4 shows the results of the ablation study. We first extensively explore different ways of arrangement of co-attention mask matrices which are controlled by the parameter pk in k-th DynRT layer. In our model, the kinds of co-attention mask matrices increase progressively with the rising of DynRT layers (pk = k). When we connect the
same four DynRT layers with four types of coattention mask matrices, the performance reduces on both metrics. When the number of the types of co-attention mask matrices decreases with the increase of DynRT layers, the performance drops. The above variants show the effectiveness of our hierarchical co-attention, as increasing the types of co-attention mask matrices with the rising of DynRT layers gradually increases the degree of diversity of the model, which benefits the process of learning the cross-modal incongruity according to diverse image-text pairs.
To evaluate the effectiveness of DynRT, which we design for multimodal sarcasm detection, we replace DynRT with other multimodal modules. Replacing DynRT with another routing-based dynamic scheme TRAR leads to a drop in performances, indicating that performing dynamic routing on unimodality only is insufficient to detect multimodal sarcasm. Using the standard multimodal transformer layer to replace our DynRT layer gets rid of the dynamic ability, thus performing worse, which further shows the advancement of our proposed dynamic module in modeling crossmodal incongruity. Ablating all the DynRT layers with the concatenation of classification vectors of text encoder and image encoder sharply slashes the results, which directly shows the advantage of our proposed DynRT.
To verify the effectiveness of dynamic attention predicted by the router in our model, we directly replace the dynamic attention scores with average probability or use fixed attention only focusing on empty mask matrices, leading to poorer performances, as the router predicts dynamic attention scores to balance the co-attention between image
and text for detecting sarcastic incongruity according to different inputs. Besides, we can see that the variants with dynamic design perform better compared with the variants with static design, which further verifies the necessity to model cross-modal incongruity with the dynamic mechanism adjusting to diverse inputs for multimodal sarcasm detection.

heading: 4.6 Hyperparameter Analysis
text: To analyze the impact of the number of DynRT layers in our model, we experiment on varying the layer of DynRT from 1 to 6. The results are shown in Figure 5. In Figure 5, we can see that our model performance improves with the increase of DynRT layers in the first three layers, and then the performances drop slightly in the layers 4-6. The results indicate that, with more layers of DynRT, the ability of our model improves first, but with the further increase of layers, DynRT-Net encounters the performance bottleneck. Thus, we use the model with 4 layers of DynRT in the main experiment, which is relatively stable and achieves the best results for multimodal sarcasm detection.

heading: 4.7 Case Study
text: To further verify the adaptability of DynRT-Net, we visualize the learned attentions between text tokens and image patches in different DynRT layers. From the results in Figure 6, we can see that the tokens of objects are unable to focus on corresponding image regions in the first few layers, while their attentions move to corresponding image regions
gradually with the increase of layers, which shows that our model learns semantic alignment relations between the image and text gradually. Specifically, in the 4th layer, the tokens of objects, such as park in Figure 6 (a) and cup in Figure 6 (b), can focus on the related image regions.
Moreover, the tokens which express sarcastic meanings can concentrate on the image regions which express inconsistent concepts in the 4th layer, thus verifying that our model can dynamically capture the incongruity between image and text. Specifically, in Figure 6 (a), the car takes two parking spaces, and great in the text expresses the sarcastic meaning, which has a higher attention score for the parking space in the image. Likewise, in Figure 6 (b), thanks and awesome in the text have higher attention scores with the region of the leaky cup in the picture.

heading: 5 Conclusion
text: To model the cross-modal incongruity that is adjustable to diverse image-text pairs, we propose the dynamic routing transformer network DynRT-Net to activate different modules with hierarchical coattention for multimodal sarcasm detection. This dynamic mechanism in network design can help capture the sarcastic clues in accordance with different image-text inputs. Experimental results on a public dataset demonstrate the effectiveness of our proposed method. Our future work shall explore diverse types of co-attention between image and text to further improve the adaptability of our method.
Limitations
Our work has some limitations. The design of the co-attention in our method can be improved. Currently the design of co-attention in our method is limited to four types, which affects its adaptability. In addition, due to the fact that there is only one publicly available dataset in multimodal sarcasm detection, we conduct our experiments based on it. This has limited the evaluation of the generalization of our method.

heading: Acknowledgements
text: This work is supported in part by the Ministry of Science and Technology of China under Grants #2022YFB2703302 and #2020AAA0108401, and National Natural Science Foundation of China under Grants #62206287, #11832001 and #72293575. We thank all the anonymous reviewers for their valuable comments.

heading: A License of Scientific Artifacts
text: The license for RoBERTa is MIT License. The license for ViT is Apache-2.0 license. We were unable to find the license for the Multimodal Sarcasm Detection dataset from the original paper (Cai et al., 2019) and the online resources1.

heading: B More Details of Experimental Settings
text: We train all the models on GeForce RTX 2080 Ti GPUs. For each run, the model giving the best performance of macro-F1 in the development set is used for the test set. We provide details of the best model parameters in Table 2. We resize the image to the resolution of 224 × 224 pixels and use vit-base-patch32-2242 with 7× 7 grids for the visual embedding. We use the first layer of robertabase3 for the text embedding. The dropout rate for classifier is 0.5. We optimize our model by Adam (Kingma and Ba, 2015) with learning rate e−6 and weight decay 0.01, we train our models for 15 epochs with mini-batch size of 32. All experimental results reported are the averaged scores of five runs with different random seeds. The number of total parameters in our model is 238,289,140. The training time for our model is about 40 minutes.
1https://github.com/headacheboy/data-of-multimodalsarcasm-detection
2https://github.com/rwightman/pytorch-image-models 3https://huggingface.co/roberta-base
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6 Limitations
7 A2. Did you discuss any potential risks of your work? Our work focuses on multimodal sarcasm detection, which is a classification problem. It won’t evoke potentially harmful effects like generating fake profiles in other tasks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.1, Section 4.1, Section 4.2 Appendix B
3 B1. Did you cite the creators of artifacts you used? Section 3.1, Section 4.1, Section 4.2 Appendix B
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix A
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The pretrained models we used are released under a specified license MIT License and Apache-2.0 license. The data is sufficiently anonymized (like replacing mentions with a certain symbol <user> ) to make the identification of individuals impossible without significant effort.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use a publicly released dataset from previous work which has removed information that names or uniquely identifies individual people or offensive content.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Appendix B
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2 Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4, Appendix B
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.2, Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? No response.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? No response.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

heading: No response.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: No response.

heading: Future Work
text: Our future work shall explore diverse types of co-attention between image and text to further improve the adaptability of our method. Limitations
Our work has some limitations. The design of the co-attention in our method can be improved. Currently the design of co-attention in our method is limited to four types, which affects its adaptability. In addition, due to the fact that there is only one publicly available dataset in multimodal sarcasm detection, we conduct our experiments based on it. This has limited the evaluation of the generalization of our method