MoBA: Mixture of Bi-directional Adapter for Multi-modal Sarcasm Detection

Authors: yifeng  xie
zhihong  zhu
xin  chen
zhanpeng  chen
zhiqi  huang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-24 13:03:34

Abstract: In the field of multi-modal learning, model parameters are typically large, necessitating the use of parameter-efficient fine-tuning (PEFT) techniques. These methods have been pivotal in enhancing training efficiency for downstream tasks in almost all situations. However, directly applying PEFT methods struggles to fully address the intricate demands of multi-modal tasks, such as multi-modal sarcasm detection (MSD), which demands the extraction and comparison of cues from different modalities. MSD, particularly when reliant on textual and visual modalities, faces challenges in identifying sarcasm's incongruity. This issue often arises from the lack of intermodality interaction during tuning, resulting in a disconnect between textual and visual information. In this paper, we introduce a novel approach called Bi-directional Adapter (BA), designated as MoBA. This approach is designed to minimize training parameters while enhancing the model's ability to interpret sarcasm across modalities. By facilitating an exchange between textual and visual information through a low-rank representation, our method adeptly captures the nuances of sarcastic expressions with a reduced number of training parameters. Our empirical studies, carried out on two publicly accessible and emerging datasets, demonstrate that our model substantially improves sarcasm detection accuracy. These findings indicate that our approach provides a more reliable and efficient solution to address the complexities of MSD.

Main paper: 
 related work 2.1 multi-modal sarcasm detection: 
 Multi-modal sarcasm detection (MSD) is an area of research that has gained significant attention due to its importance in understanding human communication, particularly on social media platforms, where text and images are often used together to convey messages. The initial approach to the MSD task utilized both textual and visual information [41] . Subsequently, a significant effort established an MSD benchmark and introduced a hierarchical fusion model to integrate these modalities [3] . Most recently, advancements have been made by utilizing the capabilities of pre-trained CLIP [39] models to perform MSD, achieving state-of-the-art results [38] .Despite significant advancements, MSD remains a challenging task, and relies on full fine-tune for the pre-trained model, which is too resource intensive. Additionally, whether the model can truly understand the signals of sarcasm between different modalities is also open to question. In response to these challenges, we propose MoBA, a low-rank approach that utilizes bidirectional adapters for efficient interaction between the two modalities, further enhancing the sarcasm detection capabilities. This method aims to overcome the limitations of previous approaches, improving both the efficiency and effectiveness of multi-modal sarcasm detection.Parameter-efficient fine-tuning (PEFT) has become a vital area of research in deep learning, particularly due to the increasing scale of pre-trained models. One of the initial methods to achieve this efficiency is the use of adapter modules [19] . These small, trainable layers are strategically inserted between the layers of a pre-trained model. During fine-tuning, only the parameters of these adapter modules are updated, leaving the original model parameters unchanged. Another approach to improving the efficiency of parameters is prompt tuning, which involves adding a few trainable tokens (prompts) to the input sequence [25] . These prompts are optimized during fine-tuning, directing the pre-trained model to produce outputs that align with the target task while maintaining fixed model parameters. Low-rank factorization techniques also contribute to parameter efficiency by applying matrix decomposition to reduce the parameter space of the neural network weights [20] . This allows for efficient fine-tuning by optimizing a smaller subset of factors rather than the entire set of weight matrices, which simplifies the model architecture without compromising performance. Despite these advances, these methods still require some retraining. As a result, we aim to explore alternative strategies for MSD that may circumvent or minimize the need for extensive retraining.The mixture-of-experts (MoE) model, first introduced by [22] , represents a pioneering approach in machine learning, aiming to divide complex tasks into simpler sub-tasks that can be solved by specialized models, termed "experts". Over the years, this framework has seen significant developments and applications across various domains. With the rise of deep learning, the MoE has been particularly influential in the realm of neural networks. It addresses the challenges posed by diverse data types by utilizing contextual cues or input features to direct information to the most pertinent experts. The application of this approach has led to an improvement in the model's efficacy [7, 12, 42] . Moreover, the MoE concept has been adapted for multi-modal learning, where it adeptly processes different data sources such as text, images, and audio [4, 35] . In these contexts, MoE models promote the integration of insights from multiple modalities, thereby enhancing the predictive accuracy and coherence of the model's outputs across diverse input types.Recently, there has been a surge in research focusing on MoE models within the context of low-rank approximations. The fusion of low-rank techniques with MoE is especially prominent in fields where models are required to handle high-dimensional data efficiently [13, 20, 49] . By enabling more streamlined and resourceefficient computations, low-rank MoE methods pave the way for deeper and more intricate ensemble learning strategies. Overall, MoE has shown robust performance across a diverse range of tasks. In this study, we explore the application of MoE within the context of adapters for the MSD models. Our approach presents distinct advantages over the traditional, complex retraining modules employed in previous methodologies. By utilizing MoE, we provide a more efficient and effective means of enhancing model performance through low-rank techniques. 
methodology: 
 Our proposed module can be seamlessly integrated into the MSD models, effectively freezing most of the training parameters. Within this framework, MoBA remains trainable, adept at extracting information from both modalities for bidirectional interaction. In the following, we discuss the relevant details.For each input sample 𝑋 from the training dataset, there are two modalities to consider: text (T) and visual (V). The inputs from these two modalities are processed through encoders to obtain their respective embeddings. These embeddings are subsequently fused to produce a predicted outcome. The process involves training a model, denoted by 𝑓 (•), to perform the MSD task. The objective of the task is to determine whether a given sample contains sarcasm. Given the multi-modal inputs, the model predicts an outcome represented by 𝑌 = 𝑓 (𝑇 , 𝑉 ) ∈ {0, 1}, where 𝑌 = 1 indicates the presence of sarcasm in the sample, and 𝑌 = 0 indicates its absence.As shown in Figure 2 , we predominantly freeze the parameters within the model. Upon being introduced to the pre-trained models, inputs from the text and visual modalities are transformed into their respective embeddings. Subsequently, we integrate our novel module into the model framework.This module is built on the foundation of transformer-based encoder layers [46] . It is designed to deeply encode the embeddings, enhancing their representational richness. In our configuration, MoBA inputs the features before normalization and outputs at the residual addition. Each transformer layer includes two MoBAs, designed to enhance feature integration. Notably, the multi-head attention mechanisms of the transformer layers remain frozen, with the sole exception of the first layer, which is adaptable for tuning.After obtaining the new information feature E 𝑖 ∈ {𝑇 ,𝑉 } , our module seamlessly integrates with existing models, facilitating effective fusion of modal information. To optimize this process, we experiment with three different fusion techniques:(1) Concatenate fusion: This method combines features by concatenating text and visual embeddings, represented as 𝑌 = 𝑐𝑜𝑛𝑐𝑎𝑡 (E 𝑇 , E 𝑉 ). This technique is simple but effective in different scenarios; (2) Guided weight fusion: A guided weighting mechanism calculates weights for each modality,𝛼 𝑖 = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (W 𝑖1 • 𝑡𝑎𝑛ℎ(W 𝑖2 ) • E 𝑖 ).The final output is a weighted sum, 𝑌 = 𝛼 𝑖 E 𝑖 , rendering it particularly suitable for tasks where the precise modeling of the interplay between multiple modalities is crucial;(3) Attention fusion: This approach employs an attention mechanism, where attention weights are computed asW 𝑇 , W 𝑉 = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (W(E 𝑇 , E 𝑉 )), and the output is an attention-weighted combination, 𝑌 = W 𝑇 E 𝑇 + W 𝑉 E 𝑉 . This method significantly enhances the interactivity between modalities and is especially well-suited for contexts within multi-modal learning where such interactions are effective.The fusion will not impact the interaction within the ABC. Furthermore, the effects of our three distinct fusions will be experimented in Table 2 to verify their influence. Following the previous works [38] , we employ the cross-entropy loss to optimize the MSD task:L = ŷ𝑙𝑜𝑔(𝑦) + (1 -𝑦)𝑙𝑜𝑔(1 -ŷ),where 𝑦 is the ture label and ŷ is our predict value.The core element of our module is the MoBA. This model is characterized by its fully trainable parameters. As depicted in Figure 3 , the MoBA is meticulously crafted to ensure the smooth transition of features between different modalities. This intricate architecture is instrumental in bolstering the model's proficiency in understanding and processing multi-modal information. It is important to note that the dimensions remain unchanged before and after passing through the MoBA. To ensure that the lengths of the two modalities entering the MoBA are identical, we apply zero-padding to the modality features after they have passed through the embedding.Bidirectional Adapter. Bidirectional Adapter (BA) is engineered to independently process inputs from each modality before serving as the output for another modality, thereby fostering interaction and synergy between modalities. Initially, the hidden dimension 𝑑 𝑒 is reduced to 𝑑 𝑡 through a down-projection layer, where 𝑑 𝑡 is significantly smaller than 𝑑 𝑒 . This reduction is crucial for managing computational complexity. Subsequently, the embeddings pass through both a linear projection layer and an up-projection layer, which collaboratively restore the hidden dimension to 𝑑 𝑒 . The output from this streamlined, low-rank structure is then used to interact with features from the other modality. This configuration utilizes specialized low-rank matrices to facilitate the interactions between modalities, which are particularly skilled at detecting subtle cues that are indicative of inter-modal sarcasm. By doing so, it markedly improves the model's capacity to accurately interpret and process complex multimodal data.Mixture of Expert for BA. Contrasting with the aforementioned method, MoBA introduces a more intricate structure. It enhances its design by integrating the BA within a Mixture of Experts (MoE) framework, which allows for flexibility in the number of experts across each layer. MoBA utilizes multiple sets of low-rank matrices, with each set identified as a BA expert, comprising a total of 𝑛 BAs throughout the entire model. The routing module, denoted as G, is employed to direct each input token to the appropriate BA experts. This module calculates a weight vector through a linear transformation and then applies a softmax layer to distribute weights among the BA experts effectively. This dynamic weighting mechanism ensures that each input is processed by the most relevant experts based on the task.The architecture of MoBA is meticulously crafted for adaptability, allowing for the fine-tuning of the number of BA experts to enhance performance. The incorporation of the MoE strategy within the BA framework significantly improves the model's efficacy, providing a customized approach to MSD. This sophisticated structure not only adapts to the complex demands of MSD but also optimizes the interaction and processing capabilities of the model. The formulas of the whole MoBA are as follows:EQUATIONEQUATIONEQUATIONEQUATIONHere, E denotes the initial embedding serving as the input, E ′ is the updated embedding following transformation and interaction, and E ′′ is utilized as the final embedding for subsequent processing. The variables A, L, B, and W represent the weight matrices within the adapter. 
experiments: 
 To evaluate the effectiveness of our proposed module, we conducted experiments on two benchmark datasets and compared our method with state-of-the-art sarcasm detection techniques. In this section, we describe the datasets, experimental setup, results, and analysis. [38] , we adopt several key metrics to assess our model's performance: accuracy (Acc.), precision (P), recall (R), and the F1-score (F1). These metrics enable us to conduct a comprehensive and detailed evaluation, shedding light on various aspects of our model's effectiveness in accurately identifying and classifying instances in MSD task.To establish the efficacy of our proposed method, we conduct an extensive comparison with a diverse range of cuttingedge and traditional approaches. This comprehensive analysis is designed to provide a robust evaluation of our method's performance by benchmarking it against a variety of established baselines:• Text modality: We selected influential models that have significantly advanced natural language processing. Text-CNN [24] provides a robust baseline with its ability to capture local text dependencies. Bi-LSTM [15] enhances sequence models by effectively capturing temporal dependencies through its bidirectional structure. BERT [10] revolutionizes language understanding with its deep bidirectional text representation and pre-training approach. • Image modality: We choose models that have been pivotal in the advancement of computer vision. Resnet [18] overcomes the vanishing gradient problem with its deep residual network architecture, forming the basis for many subsequent innovations. ViT [11] repurposes the transformer architecture for image analysis, setting new benchmarks in image classification.• Multi-modal modality: For multi-modal tasks, where the integration of different types of modality is crucial, we select a variety of models that have shown excellence in handling such complexity. HFM [3] , or Hierarchical Fusion Model, is a multi-modal model that effectively combines information from different modalities through a hierarchical fusion approach. Att-BERT [36] introduces attention mechanisms to the BERT model, allowing for more focused and contextaware representations in multi-modal scenarios. InCross-MGs [29] utilizes intersection and multiplication of modalityspecific graph representations to capture cross-modal dependencies. This method has shown to be effective in tasks where the relationships between modalities are complex. CMGCN [30] , leverages graph convolutional networks to integrate information from different modalities in a structured manner. HKE [32] is a model that focuses on the alignment and integration of keypoints across modalities, providing a structured approach to multi-modal fusion. Lastly, Multiview CLIP [38] extends the CLIP model to handle multiple views of data, enabling a more nuanced understanding of the relationships between text and images.4.1.4 Experimental Setup. Our proposed method, alongside the baseline models for comparison, was developed using the PyTorch framework [37] . For embedding the inputs, we employed the CLIP as the backbone. The transformer encoder layers within our proposed model consists of 3 layers. We optimized the parameters of our model using the AdamW optimizer [33] . The training was conducted with a batch size of 64 and over 10 epochs. To ensure a fair and equitable comparison with the existing models, we meticulously followed the training configurations as described in their respective original publications. This included adhering to specified loss functions, batch sizes, and learning rate schedules, providing a consistent basis for comparison and ensuring that the evaluation of our proposed model's performance was conducted under comparable conditions. All computational experiments were conducted on a single V100 GPU. 1 The results of our experiments are presented in Table 2 , which showcases the performance of the three fusion methods mentioned above plus our proposed MoBA, against the baseline model for performance. The detailed examination of the results yields several observations: (1) Multi-modal models consistently outperform the single-modal models, confirming the hypothesis that integrating information from multiple modalities typically results in more robust and accurate models. This is particularly true in complex tasks requiring nuanced understanding, such as sarcasm detection. ( 2) MoBA demonstrates that it is possible to achieve significant performance improvements while using considerably fewer training parameters. This not only makes the MSD models more efficient, but also reduces the computational cost associated with training.(3) Our model not only meets but surpasses the performance of existing models. Specifically, MoBA achieves superior results compared to the original state-of-the-art on the MSD dataset, recording scores of 88.40% accuracy, 82.04% precision, 88.31% recall, and 84.85% F1score. These improvements are particularly noteworthy with our fusion techniques, which more effectively leverage the strengths of both textual and visual modalities. The success of our approach highlights its potential to deliver insightful and valuable outcomes through sophisticated inter-modal interactions.As shown in Table 3 , we evaluate the performance of each tuning method by conducting ablation experiments on the same MSD dataset. For these evaluations, we selected some tuning methods:(1) Full-parameter fine-tuning: This traditional method updates all model parameters for the specific task, typically achieving high performance by allowing complete data fitting. However, it can be computationally expensive and risks overfitting, especially with large models and datasets. (2) Adapter tuning: Using methods described in [19] for Bert-based or/and ViT encoders, and another technique from [14] for the CLIP encoder, this strategy involves inserting a small trainable network into each layer of a pre-trained model while keeping the original parameters unchanged. This approach improves the efficiency of the parameters and adaptability to new tasks. (3) LoRA: Described in [20] , this method incorporates trainable low-rank matrices parallel to existing weights, reducing the need for extensive retraining and minimizing parameter updates while aiming to preserve performance. Although less resourceintensive, LoRA may not capture task complexity as effectively as full-parameter fine-tuning. These methods allow us to test the effectiveness of different tuning strategies on the same model.The results indicate that while full-parameter fine-tuning often delivers the highest performance, its substantial computational demands and potential for overfitting restrict its widespread use. In contrast, the LoRA and adapter modules offer improved parameter efficiency, though they may slightly underperform in comparison. However, our proposed MoBA matches or even exceeds the performance of full-parameter fine-tuning while also maintaining computational efficiency. This is particularly evident in its ability to handle complex multi-modal data effectively. MoBA is designed to process and integrate information from various modalities using an innovative fusion strategy coupled with a parameter-efficient fine-tuning mechanism. It significantly reduces the number of trainable parameters while preserving the model's capability to adapt to each specific modality.4.4.1 The Impact of Component. Our analysis begin by examining the role of the MoE and bidirectional adapter mechanism within the MoBA framework. The MoE approach facilitates dynamic allocation of inputs to the most appropriate adapters. In contrast, we designate a scenario where a BA is used without the MoE feature as "w/o MoE". Additionally, we assess the significance of the bidirectional nature of the adapters in MoBA. The BA is designed to process inputs from one modality and transfer them to another, thereby enhancing their interaction. To evaluate the impact of this feature, we modify the adapter to maintain input within its original modality after processing, termed as "w/o Bi".As shown in Table 4 , we compare these configurations against two baseline models -HKE and Multi-view CLIP -on the MMSD2.0 dataset. Our findings underscore the crucial contributions of both the MoE and BA to the MoBA's performance in MSD tasks. Eliminating these key components results in noticeable performance declines; for example, under the F1 metric, removing MoE leads to decreases of 1.66% and 1.94%, while omitting the bidirectional feature results in reductions of 0.81% and 3.49%, respectively. This analysis confirms the efficacy of our proposed MoBA. By integrating these essential elements, the MoBA effectively captures and processes multi-modal cues, leading to a more accurate and robust sarcasm detection model. As shown in Figure 4 , MoBA, when integrated with Multi-view CLIP, consistently surpasses the performance of the baseline model under these constrained conditions. This can be attributed to MoBA's ability to enhance interactions within the model, thus boosting its overall capabilities. This notable improvement highlights our model's stability and reliability across varied levels of available resources, showcasing significant performance gains even with limited training data. This robustness is particularly valuable in scenarios with uneven distributions of training and testing samples, confirming MoBA's effectiveness in optimizing resource usage while maintaining high performance standards. To assess the generalizability of our proposed MoBA across different tasks, we conducted a comprehensive evaluation using multiple datasets and various sentiment classification scenarios. We utilized several well-known multimodal sentiment analysis (MSA) datasets, including MOSI [56] and MOSEI [55] . Additionally, we tested MoBA against three established MSA models: self-MM [17] , MMIM [16] , and ConFEDE [54] .As detailed in Table 5 , the results confirm that MoBA exhibits strong generalizability across various MSA tasks. For instance, integrating MoBA with the three aforementioned MSA models on the MOSI dataset resulted in performance improvements of 0.90%, 0.87%, and 0.78% in two-class accuracy (Acc-2) and 0.74%, 0.80%, and 1.31% in seven-class accuracy (Acc-7), respectively. Overall, this generalizability analysis for sentiment analysis reveals that MoBA, originally developed for MSD, also shows considerable promise for broader applications in other multi-modal contexts. These findings will inform further enhancements of the model, aiming to improve its performance across a broader spectrum of multi-modal learning. hyperparameter at a time during our experiments. The results are detailed in Figure 5 . The number of experts in the MoE setup is vital for the model's ability to specialize and manage diverse data effectively. Increasing the number of experts typically enhances performance; however, excessively high numbers can lead to overfitting and unnecessarily high computational costs. In contrast, too few experts might not sufficiently address the complexity needed for effective decision-making across different inputs. Thus, identifying an optimal number of experts is crucial to balance model complexity with efficiency. Similarly, the depth of the transformer, indicated by its number of layers, profoundly influences the model's capacity to understand and process input features. More layers generally provide a deeper and more nuanced comprehension, which is advantageous for complex tasks that require detailed contextual and relational analysis, such as sarcasm detection. Nevertheless, more layers also mean more parameters, longer training times, and an increased risk of overfitting, especially with limited data. Therefore, it is important to carefully adjust the number of layers to maintain a balance between depth and operational efficiency.Our findings indicate that there is a critical threshold where further increasing the number of experts or layers does not yield substantial performance gains and may even harm performance due to the aforementioned issues of complexity and overfitting. These insights are essential for optimizing our model to ensure it is not only effective but also resource-efficient. 4.4.5 Error Analysis. Figure 6 presents an error analysis of our experimental findings. This analysis indicates that a considerable number of errors stem from samples with significant textual content within the images. For example, the image in example (a) is purely textual, which challenges our model due to its lack of mechanisms for directly handling the interplay between text and image modalities. To mitigate this, we performed an additional experiment incorporating Optical Character Recognition (OCR) results into our model. The experiment results show that there has been improvements in performance. Additionally, the image in example (b) includes not only crucial textual information, but also critical expressive details from the individual depicted. This complexity underscores the need to better capture the subtle interplay between visual expressions and textual information. We investigated methods to refine the model's ability to interpret these complex interactions, aiming to enhance its capacity to identify underlying sarcastic nuances. Improving this capability could significantly advance the model's effectiveness in recognizing sarcasm across diverse modalities. 
conclusion: 
 In this paper, we presented a novel approach for multi-modal sarcasm detection that combines the low-rank and MoE design concepts. This innovative strategy boosts the performance of models engaged in multi-modal sarcasm detection by effectively combining the strengths of both MoE and BA to facilitate dynamic inter-modal interactions. Our experimental findings consistently show that the MoBA enhances the model's accuracy and robustness while also preserving computational efficiency. MoBA adeptly captures and integrates complex inter-modal interactions, resulting in substantial improvements in sarcasm detection across a variety of datasets Moreover, the generalization and scalability of MoBA indicate its potential applicability in other areas of multi-modal learning. 
: 
 1 IntroductionIn the rapidly evolving landscape of artificial intelligence (AI), of artificial intelligence, the strategy of pretraining models has emerged as a critical foundation for developing specialized AI capabilities. This approach has achieved significant success in natural language processing (NLP) [40, 44, 57, 58] , computer vision (CV) [5, 11] , and multi-modal modeling [6, 27, 39] . Demonstrating remarkable efficacy across domains, this method enhances model performance in terms of accuracy and efficiency on downstream tasks. Subsequently, various downstream tasks involving multiple modalities are to show improved performance after fine-tuning with the pre-train model. For instance, sarcasm-a prevalent and intricate mode of communication-presents a significant challenge in the realm of multi-modal learning. Accurate detection of sarcasm is vital for numerous applications, including sentiment analysis [9, 17, 51] , social media monitoring [1] , and conversational systems [21, 26, 43, 50] . Traditional sarcasm detection methods primarily rely on textual features, which may not capture the full spectrum of information required for accurate detection. With the increasing availability of multi-modal data, including text, images, audio, and videos, researchers have started exploring the potential of integrating these diverse data sources to enhance multi-modal sarcasm detection (MSD) performance [3, 45, 52, 59 ]. This advancement demands that models possess the ability to recognize the cues across modalities accurately. The complexity of the task requires more power of the multi-modal model.Pretraining involves training a model on a large corpus of data to learn broad and generalizable representations, which can then be adapted to specific tasks, including MSD, through a process known as fine-tuning. The pretrain-finetune paradigm, involving the initial training of models on large-scale datasets to learn broad, generalizable representations followed by task-specific fine-tuning, is a common approach. Some studies in multi-modal sarcasm detection have adopted this strategy, attempting to learn the incongruity between textual and visual modalities by merging features from multiple sources. However, fully fine-tuning these models for complex tasks requires training numerous parameters, which can be resource-intensive [2, 8, 47, 48] .Therefore, finding new ways to efficiently transfer existing foundation models to downstream tasks without incurring excessive costs becomes an important challenge in the field. The Parameter Efficient Fine-tuning (PEFT) method aims to solve these problems. By fine-tuning only a select few additional model parameters and freezing the majority of the pre-trained model's parameters, PEFT significantly cuts down on computational and storage demands. This method has been demonstrated to outperform traditional finetuning in settings with limited data and exhibits superior generalization [23, 28, 34] . It is versatile enough to be applied across different modalities. With PEFT, only a few training weights are introduced to the top layer of the pre-trained model. This allows the same pre-trained model to be repurposed for multiple tasks by adding minimal weights, thereby eliminating the need to full finetune the entire model. In short, PEFT methods enable achievement of performance comparable to that of full-parameter fine-tuning while requiring only a limited number of trainable parameters. However, the effectiveness of existing PEFT methods in handling the complexities of MSD remains less than optimal.Consequently, we introduce a novel approach known as the Bidirectional Adapter (BA) and propose a plug-in, termed MoBA, which is designed for seamless integration into MSD models to significantly enhance their performance. As illustrated in Figure 1 , MoBA achieves impressive results with a minimal number of training parameters. Our approach directly addresses the challenges of large-scale modeling by refining both pre-training and fine-tuning techniques to suit the complexities of MSD. We utilize BA to dynamically extract relevant features from changes in each modality. This method allows each modal branch to learn prompt information from the alternate modality and combine it with the feature information of its own modality, thereby improving the model's representation capabilities. Furthermore, MoBA adopts a mixture-of-experts (MoE) style approach. A wealth of research and experimental evidence has shown that MoE can significantly enhance the model's capacity to tackle various downstream tasks through fine-tuning with extensive instructional data [12, 53] . Therefore, we have also leveraged MoE to optimize BA's performance. Moreover, MoBA can be appended to the pre-trained model without adding numerous trainable parameters, maintaining efficiency in model adaptation.The main contributions of this paper are as follows: (1) To our knowledge, this is the first work to apply PEFT to the MSD task. Our innovative module is designed for plug-and-play integration into existing MSD models, significantly reducing the need for trainable parameters by freezing most of them and utilizing only a select few for fine-tuning. (2) We introduce a novel approach, termed MoBA, specifically tailored for the MSD task. This approach combines the PEFT method with a MoE strategy. It adeptly merges multi-modal information in a dynamic fashion, facilitating effective cross-modal interactions through a straightforward and efficient framework. (3) Our experimental findings highlight MoBA's exceptional performance on MSD benchmarks, showcasing significant advancements over other PEFT approaches and previous baselines. We provide a comprehensive analysis, supported by extensive experimental data, to underscore the effectiveness of our proposed method. 
 
 Appendix:  


Decision: This paper proposes a Bidirectional Adapter approach, with the goal of minimizing the number of parameters of models addressing the task of multi-modal sarcasm detection.

Regarding this paper's strengths, reviewers acknowledge the effectiveness of the proposed approach (2AtH, S7wp, usLt), the extensive experiments (2AtH), and that it is well-written (usLt).

Among its weaknesses, some reviewers highlight the reduced significance of the contribution (2AtH) and insufficient experimental comparisons, in particular with full model fine-tuning (S7wp, usLt). 
Reviewers also mentioned a weak discussion of the implications of attaching MoBA to other models/downstream tasks (2AtH, S7wp). 

The authors' rebuttal addressed some of the reviewers' concerns, providing new results to further strengthen experimental evaluation. 

Therefore, to sum up, this paper proposes an interesting and relevant approach but in its current form, experimental evaluation missed some critical aspects and in-depth discussions. Therefore, I suggest this work be accepted if there is space, provided that authors accommodate the clarifications and results provided in the rebuttal, in the paper final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel approach called MoBA (Mixture of Bidirectional Adapter) for multi-modal sarcasm detection (MSD). The key points are:

- MoBA combines parameter-efficient fine-tuning (PEFT) techniques with a mixture-of-experts (MoE) framework to efficiently adapt pre-trained models for MSD while reducing training parameters.
- It introduces a Bidirectional Adapter (BA) that facilitates interaction and information exchange between textual and visual modalities through low-rank representations. This allows the model to better capture sarcasm cues across modalities.
- MoBA is designed as a plug-in module that can be integrated into existing MSD models. It significantly reduces trainable parameters by freezing most of the pre-trained model and only fine-tuning a few.
- Extensive experiments on two MSD benchmark datasets show that MoBA substantially improves sarcasm detection accuracy while using much fewer training parameters compared to other state-of-the-art methods.
- Ablation studies confirm the effectiveness of the MoE and BA components in MoBA. The approach also demonstrates strong performance in low-resource settings and good generalizability to multi-modal sentiment analysis tasks.
strength_weakness: strength_weakness: Overall, the technical aspects of the paper seems to be sound. The introduction of the bidirectional adapter in the form of the MoBa is moderately novel, and from the experimental results do seem to achieve the goals for which it was designed. The use of a MoE setup is a good idea, since this enables cross-referencing of different perspectives which will inevitably result in the case of multimodal inputs, as is the case here. Numerical experiments seems to be quite impressive, as they suggest superior performance compared to some large models such as BERT.
limitations: limitations: On the other hand, the idea of PEFT seems to be quite ad-hoc, and is a standard technique for reducing the complexity of training a deep model; this does not seem like a significant contribution (or should not be designated as such). MoBa, although a nice design which should facilitate its integration into other models, seems to me like a repackaging of modality alignment modules in other multimodal setups. At least, I am not sure how MoBA operates differently from such alignment modules. In addition, it is unclear to me what 'FusionN', where N=1,2,3, means in the Main Results section. As I understand it, MoBA is a standalone module, which needs to be integrated with another deep model for downstream tasks. Hence it is important to state clearly what these fusion models are.
paper_summary: paper_summary: This paper directly applies LoRA struggles to fully address the intricate demands of multi-modal sarcasm detection task.
strength_weakness: strength_weakness: This paper captures the nuances of sarcastic expressions with a reduced number of training parameters.
limitations: limitations: 1) The comparison of the basic LORA and the proposed MoBA is missing. The effectiveness cannot be confirmed.
2) The publicly Twitter Multi-Modal Sarcasm dateset should be conducted for evaluate the results comprehensively.
paper_summary: paper_summary: This paper introduces a novel multimodal sarcasm detection method, named MoBA (Mixture of Bidirectional Adapter). The proposed approach aims to enhance both the efficiency and accuracy of multimodal sarcasm detection (MSD) tasks. By integrating low-rank representation and a mixture of experts (MoE) style, MoBA stands out in its ability to minimize training parameters while maintaining high performance.
strength_weakness: strength_weakness: Innovation in PEFT Application: MoBA represents a pioneering effort in applying Parametric Efficient Fine-tuning (PEFT) techniques to the domain of MSD. This is significant as it addresses a crucial need to reduce the number of training parameters by freezing the majority of pre-trained model parameters and fine-tuning only a select few. This not only makes the model more efficient but also less resource-intensive.

State-of-the-Art Performance: Experimental results demonstrate that MoBA achieves state-of-the-art performance across multiple benchmarks. The integration of bidirectional adapters and the mixture of experts allows for more effective cross-modal interactions, which is pivotal in detecting sarcasm that often relies on nuanced and context-dependent cues from both text and images.

Efficiency and Scalability: By utilizing a low-rank adaptation strategy, MoBA significantly cuts down on the computational load and storage requirements. The model’s design ensures scalability, making it applicable to a wide range of tasks beyond sarcasm detection, as evidenced by its performance in other multimodal contexts.
limitations: limitations: Training Time Discussion Needed: While the focus on PEFT is commendable, the paper would benefit from a more detailed discussion on the training time implications. PEFT techniques generally aim to reduce computational overhead, but an explicit comparison of training times between MoBA and traditional full-fine-tuning methods would provide a clearer picture of its efficiency gains.

Complexity of Bidirectional Interaction: The paper introduces a sophisticated mechanism for bidirectional interaction between modalities, which, while innovative, could potentially add complexity to the implementation. Further analysis of this aspect in practical deployment scenarios would be beneficial.Definitely suitable
paper_summary: paper_summary: The paper presents MoBA, a novel Bidirectional Adapter approach aimed at improving multi-modal sarcasm detection (MSD) involving textual and visual data. By enhancing intermodality interaction through a low-rank representation, MoBA efficiently reduces training parameters to about 3.29 million and significantly boosts sarcasm detection accuracy. Empirical studies on two public datasets confirm MoBA's effectiveness, highlighting its potential as a reliable and computationally efficient solution for complex multi-modal tasks.
strength_weakness: strength_weakness: 1. This paper is well-written and well-organized.
2. MoBA significantly enhances multi-modal sarcasm detection by improving intermodality interaction with a low-rank representation, resulting in both increased accuracy and reduced computational resource requirements.
limitations: limitations: 1. The authors state that the devised MoBA achieves impressive results with a minimal number of training parameters. Authors should compare computational complexity with baselines and some variants with their base model, such as finetuning CLIP.
2. Is there a large gap between MoBA and full fine-tuning? 
3. Some important baselines are overlooked, such as A Semantic Enhancement Framework for Multimodal Sarcasm Detection, Debiasing Multimodal Sarcasm Detection with Contrastive Learning, and Mutual-enhanced Incongruity Learning Network for Multi-modal Sarcasm Detection.
4. Typo is existed in line 545. There should be a space between “layers” and “We”. “The transformer encoder layers within our proposed model consists of 3 layers.We optimized the parameters of our model using the AdamW optimizer”
5. CCS CONCEPTS is not correct.