Aspects are Anchors: Towards Multimodal Aspect-based Sentiment Analysis via Aspect-driven Alignment and Refinement

Authors: zhanpeng  chen
zhihong  zhu
wanshi  xu
yunyan  zhang
xian  wu
yefeng 2024 zheng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 06:22:14

Abstract: Given coupled sentence image pairs, Multimodal Aspect-based Sentiment Analysis (MABSA) aims to detect aspect terms and predict their sentiment polarity. While existing methods have made great efforts in aligning images and text for improved MABSA performance, they still struggle to effectively mitigate the challenge of the noisy correspondence problem (NCP): the text description is often not well-aligned with the visual content. To alleviate NCP, in this paper, we introduce Aspect-driven Alignment and Refinement (ADAR), which is a two-stage coarse-to-fine alignment framework. In the first stage, ADAR devises a novel Coarse-to-fine Aspect-driven Alignment Module, which introduces Optimal Transport (OT) to learn the coarse-grained alignment between visual and textual features. Then the adaptive filter bin is applied to remove the irrelevant image regions at a fine-grained level; In the second stage, ADAR introduces an Aspect-driven Refinement Module to further refine the cross-modality feature representation. Extensive experiments on two benchmark datasets demonstrate the superiority of our model over state-of-the-art performance in the MABSA task.

Main paper: 
 introduction: 
 Multimodal aspect-based sentiment analysis (MABSA) has attracted increasing attention in recent years [9, 16, 29] . Typically, MABSA includes three downstream tasks: Multimodal Aspect Term Extraction (MATE), Multimodal Aspect-oriented Sentiment Classification (MASC), and Joint Multimodal Aspect-Sentiment analysis (JMASA) [12, 20, 27] . Specifically, MATE aims to identify aspect terms from a text-image pair [39] ; MASC aims to classify the sentiment of each aspect term [34] ; while JMASA aims to jointly extract the aspect terms and predict their sentiments. Therefore, JMASA refers to the end-to-end solution combining MATE and MASC. For example, as depicted in Figure 1 , the target of JMASA is to analyze the multimodal information and output the aspect-polarity pairs (Klay Thompson, POS), (Warriors, POS) and (TrailBlazers, NEG), while MATE and MASC only focus on identifying the aspect terms and classifying the sentiment, respectively.Existing works [17, 38] propose to align the global image feature with the text feature to derive cross-modality feature representation. Despite the performance improvement, they ignore the fact that only some specific visual regions are relevant to the aspect terms, and vice versa. Therefore, the information simply extracted from the global features could introduce the noise correspondence problem (NCP) [11] , thereby limiting further performance improvement. Motivated by this, the recent work AoM [41] firstly extracted the aspects from the sentence, and achieved aspect-guided crossmodal interaction to alleviate the NCP between image and text. Although AoM has achieved improved performance, it still suffers from two limitations: (i) Dependency on pre-trained tools: AoM Aspects & Sentiment (Klay Thompson , POS) (Warriors, POS) (TrailBlazers, NEG) Figure 1 : An example of the MABSA task from Twit-ter2017 [34] , including the text, aspects and sentiments.highly depends on the pre-trained phrase parsing tool Spacy 1 for noun extraction. However, the model fails when it comes to extracting nouns that are not well pre-trained. (ii) Noisy correspondence misalignment: AoM may mistakenly align aspect terms with irrelevant visual elements because the relevant ones are missing in the visual information, resulting in noisy correspondence misalignment. Take the case in Figure 1 for example, the player in the middle contributes to the correct detection of positive sentiment towards 'Klay Thompson' and 'Warriors'. However, the negative aspect term 'TrailBlazers' is not shown in the figure, which leads to the failure of detection.To effectively handle the challenge of NCP while avoiding the aforementioned drawbacks, we propose Aspect-driven Alignment and Refinement (ADAR). As shown in Figure 3 , ADAR includes two core modules: Coarse-to-fine Aspect-driven Alignment Module (ADAM) and Aspect-driven Refinement Module (ADRM). We opt for Optimal Transport (OT) at the coarse-grained level instead of pre-trained tools to address the first limitation. As a critical component in many downstream applications [1, 3, 38] , OT has been employed to facilitate cross-modal retrieval tasks involving matching images with text descriptions and vice versa. As depicted in Figure 2 , by computing pairwise similarities and using transport strategies, OT provides a robust tools-free framework to align between visual and textual representations considering the potential aspects as anchors. In ADAM, we first establish a flexible coarse-grained alignment between image and text by optimizing the goal of OT. To mitigate NCP, we directly project the visual representations towards textual feature space by employing the barycenter-based strategies [7] , while previous work [22, 38] solve the problem by optimizing different loss functions.To address the second limitation, based on the output of OT, we further introduce an adaptive filter bin that deals with the case of noisy correspondence misalignment at the fine-grained level. With the adaptive filter bin activated, each visual aspect can be aligned with relevant textual aspects or discarded based on a pre-selected threshold, leading to a precise alignment of existing visual elements instead of noisy correspondence misalignment. Subsequently, in the ADRM module, textual representations that already emphasize aspects can implicitly reason the complementary relations between images through meticulously crafted cross-modal attention, leading to accurate predictions. Extensive experiments conducted on two benchmarks across three tasks demonstrate the effectiveness and robustness of ADAR. The comprehensive ablation analysis reveals how the proposed modules and methods complement each other. For instance, ADAM achieves a coarse-to-fine alignment by coupling the OT method and the adaptive filter bin. Furthermore, our method demonstrates superior performance compared to LLMs on MASC. Overall, our main contributions in this work are three-fold:• We present a novel two-stage framework to address NCP under the guidance of aspects. To the best of our knowledge, this is the first work that introduces Optimal Transport to address NCP in MABSA. • We design a coarse-grained aspect-driven alignment module with a fine-grained adaptive filter bin mechanism and an aspect-driven refinement module to effectively align and refine features between images and texts. • Extensive experiments on two benchmark datasets show that ADAR significantly outperforms previous models. Further analysis verifies the advantages of our model. 
related work: 
 Multimodal Aspect-based Sentiment Analysis As social media becomes increasingly rich with multimodal user posts, researchers have discovered that images provide valuable supplementary information for aspect term extraction [2] and sentiment analysis [29, 40] . Consequently, Multimodal Aspect-based Sentiment Analysis (MABSA) has garnered widespread attention and study. Within MABSA, two distinct sub-tasks have emerged: Multimodal Aspect Term Extraction (MATE) and Multimodal Aspect-oriented Sentiment Classification (MASC). MATE focuses on extracting all relevant aspect terms from a sentence with cues from the accompanying image, while MASC is concerned with predicting the sentiment polarities associated with these aspects.In MABSA, which targets the nuanced analysis of fine-grained entities and their sentiments within a post, models are challenged to accurately apply pertinent visual information to the targeted aspect of each pair. To address cross-modal alignment, Ling et al. [17] developed a generative multimodal architecture grounded in BART. This architecture is designed for both vision-language pre-training and downstream MABSA tasks. Meanwhile, Yang et al. [33] introduced a dynamic approach for controlling the influence of visual information on different aspects. This approach is based on the principle that the lower the confidence in text-only predictions, the greater the reliance on visual inputs. Furthermore, Zhou et al. [41] crafted an aspect-aware attention module coupled with a Graph Neural Network, which is aimed at discerning aspect-relevant multimodal content, considering both semantic and sentiment perspectives.While existing methods make significant strides, they tend to overlook the misalignment between the visual and textual features. This oversight can lead to the introduction of irrelevant noise during the modeling process, as these methods lack fine-grained aspectdriven alignment. Optimal Transport Originally, Optimal Transport was introduced to quantify the distance between two probability distributions. Its application has recently garnered considerable attention across various domains, such as sequence alignment [19] , domain adaptation [31] and document matching [37] . These applications highlight OT's capability in handling structured data and incorporating additional structural information beyond classic formulations. This has proven particularly useful in NLP tasks [1, 3, 25, 38] like unsupervised word translation, sentence similarity, domain adaptation, ontology alignment, etc. However, it's noteworthy that while the potential of OT is significant, its application in aligning images and text specifically for MABSA presents a novel challenge. This is because MABSA requires not just traditional modalities alignment but also a focus on aspect-driven alignment.In this work, we propose a novel aspect-driven framework for MABSA. This framework comprises two key components: the Coarse-to-fine Aspect-driven Alignment Module and the Aspectdriven Refinement Module. The former is designed to align modalities through optimal transport, ensuring that the features align closely with the relevant aspects. The latter is aimed at effectively enhancing the interaction between the different modalities, ensuring a more coherent and integrated multimodal modeling. 
method 3.1 preliminaries: 
 Task Definition Following previous work [17, 32, 41] , we model the three downstream tasks on the same BART-based framework. Formally, when presented with a sample that encompasses an image denoted as 𝑉 , and a sentence containing 𝑛 words, represented as 𝑳 = {𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 }, the outputs of JMASA, MATE, and MASC are formulated as:• 𝑌 𝐽 𝑀𝐴𝑆𝐴 = {𝑎 𝑠 1 , 𝑎 𝑒 1 , 𝑠 1 , . . . , 𝑎 𝑠 𝑖 , 𝑎 𝑒 𝑖 , 𝑠 𝑖 , . . . , 𝑎 𝑠 𝑘 , 𝑎 𝑒 𝑘 , 𝑠 𝑘 }, • 𝑌 𝑀𝐴𝑇 𝐸 = {𝑎 𝑠 1 , 𝑎 𝑒 1 , . . . , 𝑎 𝑠 𝑖 , 𝑎 𝑒 𝑖 , . . . , 𝑎 𝑠 𝑘 , 𝑎 𝑒 𝑘 }, • 𝑌 𝑀𝐴𝑆𝐶 = {𝑎 𝑠 1 , 𝑎 𝑒 1 , 𝑠 1 , . . . , 𝑎 𝑠 𝑖 , 𝑎 𝑒 𝑖 , 𝑠 𝑖 , . . . , 𝑎 𝑠 𝑘 , 𝑎 𝑒 𝑘 , 𝑠 𝑘 },where 𝑎 𝑠 𝑖 , 𝑎 𝑒 𝑖 , and 𝑠 𝑖 denote the start index, the end index, and the sentiment polarity of the 𝑖-th aspect term in the tweet. The variable 𝑘 indicates the total number of aspects. Specifically when dealing with the MASC task, the start and end indexes are given during inference. Feature Extractor The initial word embeddings are derived from the pre-trained BART [14] , selected for its exemplary capabilities in textual representation. For the visual units, we employ preprocessing via ResNet [5] following Yu et al. [35] .Despite the efficacy of the hidden states in capturing the characteristics of multimodal information, the correlations between different modalities remain relatively weak because of the noisy correspondence problem (NCP), thereby limiting performance. To address this, we design a coarse-to-fine module to align visual features to textual features by leveraging optimal transport with an adaptive filter bin mechanism.Coarse-grained Aspect-driven Transport To better utilize the contextual information, we first transfer the visual space to the textual space, driven by aspects from textual inputs. Let 𝑯 𝑉 = {𝒉 𝑉 1 , . . . , 𝒉 𝑉 𝑛 } ∈ R 𝑛×𝑑 and 𝑯 𝐿 = {𝒉 𝐿 1 , . . . , 𝒉 𝐿 𝑚 } ∈ R 𝑚×𝑑 denoteEQUATIONwhere⟨𝑻, 𝑺⟩ = 𝑡𝑟 (𝑻 ⊤ 𝑺), Π = {𝑻 ∈ R 𝑛×𝑚 + | 𝑻 1 𝑚 = 𝝁, 𝑻 ⊤ 1 𝑛 = 𝝂 }and 1 signifies an all-one vector. The terms 𝝁 and 𝝂 refer to the probabilistic simplex measures corresponding to 𝑯 𝑉 and 𝑯 𝐿 , respectively. Considering that each image or text is independently sampled, we adopt equal weighting approach for each instance following Su and Hua [23] , designated by 𝝁 = 1 𝑛 1 𝑛 and 𝝂 = 1 𝑚 1 𝑚 . The optimal transport matrix, 𝑻 * , is then derived by minimizing the overall cost by Sinkhorn fixed point iterations [6] ,EQUATIONWith the optimal transport matrix 𝑻 * determined, we proceed to map 𝑯 𝑉 to Ĥ𝑉 for better utilizing the contextual information, which is facilitated by employing barycenter-based strategies,EQUATIONwhere Δ 𝑇 is an adjustable transport parameter [21] . To facilitate the projection of out-of-sample examples that were not encountered during the learning process of 𝑻 * , we initialize Δ 𝑇 using the Xavier Uniform distribution [8] with the same dimensions of 𝑻 * . Fine-grained Adaptive Filter Bin Although optimal transport is capable of aligning modalities effectively, texts and images in real-world scenarios often contain redundant or irrelevant information. Such noise can impede the efficacy of optimal transport in precisely aligning each visual unit with corresponding textual aspects. Motivated by Sarlin et al. [22] , we introduce an innovative approach to noise reduction through the implementation of a finegrained Adaptive Filter Bin (AFB), which is seamlessly integrated into the existing cost matrix 𝑺 𝑖 𝑗 with simply one additional row and column, Ŝ𝑖,𝑚+1 = Ŝ𝑛+1,𝑗 = Ŝ𝑛+1,𝑚+1 = 𝑞, (4) 𝜇 = 1 𝑛 1 𝑛 2: 𝜈 = 1 𝑚 1 𝑚 3: 𝑺 𝑖 𝑗 = ||𝒉 𝑉 𝑖 -𝒉 𝐿 𝑗 || 2 2 , ∀𝑖 ∈ [1, 𝑛], 𝑗 ∈ [1, 𝑚] 4: 𝑻 * = arg 𝑚𝑖𝑛⟨𝑻, 𝑺⟩ # Derived from the Sinkhorn Algorithm 5: Ĥ𝑉 = diag(1/𝝂)((𝑻 * ) ⊤ + Δ 𝑇 )𝑯 𝑉where Ŝ𝑖,𝑗 = 𝑺 𝑖,𝑗 , 𝑖 ∈ [1, 𝑛] and 𝑗 ∈ [1, 𝑚]. The hyper-parameter 𝑞 is strategically set to correspond to the top 20% cost associated with the original aligned image-text pairs. With the mechanism activated, each visual aspect can be either aligned with the relevant textual aspect or relegated to the bin. Consequently, 𝑞 effectively functions as a threshold, serving to exclude elements that are not alignable and thus maintaining a more noise-reduced environment for the execution of the transport assignment, i.e., T * := T1:𝑛,1:𝑚 . Now, we update the multimodal features as follows,EQUATIONEQUATIONwhere 𝜆 1 is the hyper-parameter to control the contribution from OT and [; ] is the concatenation operator for matrices.Upon achieving effective alignment of feature distributions across various modalities through ADAM, the feature spaces still suffer from NCP and await a fine-grained interaction between visual and textual modality. To learn a refined cross-modality representation that contains relevance-suppressing information and complements language features, we introduce an Aspect-driven Refinement Module (ADRM) to enhance the learning of a refined multimodal representation.Inspired by Liu et al. [18] , we first utilize Ĥ𝑉 𝑗 as the query, and 𝑯 𝐿 𝑗 serves as both the key and value in this process. The interaction is conducted as follows,EQUATIONwhere 𝑗 ∈ {1, 2}, 𝑾 𝑄 𝑉 and 𝑾 𝐾 𝐿 are learnable parameters of linear transformations. Specifically, we initialize 𝑯 𝑟𝑒 𝑓 𝑖𝑛𝑒 0 with Ĥ , denoted as 𝑯 𝑟𝑒 𝑓 𝑖𝑛𝑒 0 = Ĥ . Subsequently, the cross-modality features are updated as follows,EQUATIONwhere 𝑾 𝑉 𝐿 is a learnable parameter and 𝑯denotes the 𝑗th ADRM layer output. After a fine-grained interaction between visual and textual modality through ADRM, we free the feature representation from NCP and help the BART decoder to jointly learn the multimodal input.Following previous works, all subtasks are designed as index generation tasks and predict the token probability distribution with the BART decoder as below,EQUATION)EQUATIONEQUATIONEQUATIONwhere 𝜆 2 and 𝜆 3 are the hyper-parameters to control the contribution from the two modules. H 𝐿 is the textual part of H . 𝑬 denotes the embeddings of input tokens. 𝑪 𝑑 means the embeddings of the [positive, neutral, negative, <eos>]. Finally, the loss function is defined as,EQUATIONwhere 𝑂 is the length of Y, and X denotes the multimodal input. 
details of the sinkorn algorithm: 
 The Sinkhorn algorithm, as introduced by [6] , addresses this issue by reinterpreting transport problems through a maximum-entropy perspective. It introduces an entropic regularization term to the classic OT problem, streamlining the process. This modification not only maintains the optimal solution as a meaningful distance measure but also significantly accelerates the computation. The Sinkhorn algorithm achieves this by employing matrix scaling techniques, offering a computation speed several orders of magnitude faster than traditional transport solvers. The original Sinkhorn distance is 𝑑 𝑺 = ⟨𝑻, 𝑺⟩. With the Sinkhorn algorithm, for 𝜆 ∈ (0, ∞), we consider a Lagrange multiplier for the entropy constraint of Sinkhorn distances and obtain the dual-Sinkhorn divergence 𝑑 𝜆 𝑺 as follow,EQUATIONwhere𝑻 𝜆 = arg 𝑚𝑖𝑛 𝑻 ∈Π ⟨𝑻, 𝑺⟩ -1 𝜆 ℎ(𝑻 ).The divergence 𝑑 𝜆 𝑺 can be computed for a much cheaper cost than the original distance 𝑑 𝑺 . Subsequently, the optimal regularized transport 𝑻 𝜆 is computed as follow,EQUATIONwhere 𝑷 = 𝑒 -𝜆𝑺 . 𝜼 1 ∈ R 𝑛 and 𝜼 2 ∈ R 𝑚 are two non-negative scaling vectors updated iteratively with Sinkhorn's fixed point iteration:(𝜼 1 , 𝜼 2 ) ← (𝝁./(𝑷𝜼 2 ), 𝝂 ./(𝑷 ⊤ 𝜼 1 )).Finally, we get optimal transport distance through ⟨𝑻 𝜆 , 𝑺⟩. Additionally, the tradeoff hyper-parameters, 𝜆 1 , 𝜆 2 and 𝜆 3 , are set at 0.5, 1 and 0.5, respectively. 
comparative baselines: 
 In our experiments, we compare our proposed model with different methods on three types of tasks. Methods for Textual ABSA 1) SPAN [10] aimed to identify opinion targets along with their corresponding sentiments. 2) D-GCN [4] effectively modeled the dependency relations among words by utilizing a dependency tree structure. 3) BART [32] addressesed seven ABSA subtasks within a unified framework. Methods for MATE 1) RAN [28] focused on aligning text with corresponding object regions. 2) UMT [36] incorporated text-based entity span detection as a supplementary task. 3) OSCGA [30] emphasizes the alignment between visual objects and entities. Methods for MASC 1) ESAFN [35] represented an entity-level sentiment analysis approach that leverages LSTM technology. 2) TomBERT [34] utilized BERT to derive aspect-sensitive textual representations effectively. 3) CapTrBERT [13] is designed to convert images into text and construct auxiliary sentences for enhanced fusion.Methods for JMASA 1) RpBERT-collapse [24] proposed a model for MATE, utilizing collapsed labels to encapsulate aspect and sentiment pairs. 2) JML [12] stands as the pioneering joint model for MABSA, featuring an auxiliary cross-modal relation detection module. 3) VLP-MABSA [17] undertaked five task-specific pre-training tasks to effectively model aspects, opinions, and their alignments. 4) CMMT [33] implemented a gating mechanism to regulate the contributions of multimodal information during inter-modal interactions. 5) DTCA [38] cross-modal alignment on two auxiliary tasks to enhance performance. 6) AoM [41] designed an aspect-aware attention module and an aspect-guided GNN to detect aspect-relevant multimodal contents.The results of our ADAR and competitive baselines for the JMASA, MATE and MASC are shown in Table 2 , 3 and 4 respectively, from which we have the following observations: Results for JMASA (i) It is obvious that our proposed ADAR surpasses all text-based models, primarily due to richer information from different modalities. (ii) In terms of multimodal approaches, AoM outstrips previous methods on Twitter2015, largely attributable to its aspect-aware attention module. On Twitter2017, the DTCA, with its dual auxiliary tasks of text-only extraction and text-patch alignment via optimal transport, enhances cross-attention performance and proves more effective. However, we focus on aspectdriven transport and assignment instead of contrastive loss. Among all evaluated methods, ADAR demonstrates superior performance across both datasets, albeit with a marginal 0.2 absolute percentage point deficit in Recall compared to DTCA on Twitter2017. This notable performance gain is largely attributed to our aspect-driven alignment and refinement method, which skillfully aligns disparate modalities using aspect-driven optimal transport and augments their synergy through a dual-layer refinement process.Results for MATE and MASC Table 3 and Table 4 present the outcomes for the MATE and MASC evaluations, respectively. Mirroring the pattern observed in the JMASA task, it is evident that our ADAR approach consistently delivers superior performance on both datasets, with the sole exception being the Recall metric for Twitter2015. These findings further underscore the broad efficacy of our methodology. 4 : Results comparison for MASC. We report the average results of three runs with different random seeds. ♣ denotes the results from Ling et al. [17] . The best results are in bold and the second best ones are underlined. † denotes our model significantly outperforms baselines with 𝑝 < 0.05 under t-test.To validate the efficacy of each component in ADAR, we conducted a set of ablation experiments on JMASA task, and the results are reported in Table 5 . Effect of ADAM On Twitter2015, disabling ADAM solely leads to a significant reduction in performance. With a 7.44% drop on F1, it underscores the critical role of the aspect-driven transport method in effectively aligning different modalities for a further settlement of NCP in ADRM. Effect of ADRM Removing ADRM solely results in a substantial decline of 5.06% on F1 in Twitter2015 performance. This highlights the importance of the refinement process following alignment, as it is essential for facilitating interaction between modalities and enhancing feature information richness in an aspect-driven way. Effect of AFB As an auxiliary module for ADAM, ablating AFB highlights its proficiency as a simple yet effective approach to noise reduction, thereby making a beneficial contribution to the overall system's performance. Table 5 : Results of ablation experiments.Layer Number of ADRM We carried out a series of experiments to ascertain the optimal number of layers for the Aspect-driven Refinement (ADRM) module. The performance outcomes associated with varying layer counts are detailed in Table 6 . With a single ADRM layer, we observed comparatively weaker performance, attributed to insufficient fitting of cross-modality features.Conversely, employing three ADRM layers resulted in overfitting, paradoxically leading to lower performance than with a single layer. Consequently, in light of our comparative analysis, we have opted for a two-layer configuration for the ADRM module.Hyper-parameter 𝜆 1 and 𝑞 To effectively leverage the aligned information following optimal transport, we conducted experiments with different settings of the hyper-parameter 𝜆 1 and 𝑞.As depicted in Figure 4 , a peak at 𝜆 1 = 0.5 is observed in both datasets. The performance is weak due to an underutilization of aligned features at 𝜆 1 = 0.25. As 𝜆 1 increases beyond the optimal value, there is also a performance decline, due to the introduction of noise through excessive reliance on optimal transport. To mitigate this, we implement an adaptive filter bin to eliminate irrelevant information with a threshold 𝑞. The model achieves the best performance on both datasets when 𝑞 is set to 0.2. We observe that setting 𝑞 higher than 0.2 leads to a decline in performance, mainly due to mistakenly filtering of relevant information. It is also worth noting that the performance on the Twitter2017 dataset was not as strong as on Twitter2015 when 𝑞 is set to 0.4 or 0.5. This can be attributed to the greater number of aspects per image-text pair in Twitter2017. When combined with a strict filtration policy, it results in reduced effectiveness due to the increased interference among aspects. The above analysis demonstrates the effectiveness of our proposed coarse-to-fine aspect-driven alignment module. In recent advancements, Large Language Models (LLMs) have demonstrated remarkable capabilities in various Natural Language Processing (NLP) tasks, leveraging their advanced language understanding and generation skills [15, 26] . To ascertain the competitive edge of our model on MABSA, comparative analyses were conducted against prevalent LLMs, including VisualGLM-6B, GPT3.5, and GPT4V. Considering the inherent design limitations of LLMs in aspect identification and output structures, we confined the comparison to the MASC task to maintain evaluative fairness. To conduct the experiments, we design two prompt templates. As shown in Figure 5 , we setup the role for LLMs and define the task. Subsequently, we provide the image, text and aspects to acquire the outputs. Specifically, when experimenting with GPT3. only the text and aspects, owing to its absence of multimodal capabilities. The comparative results, as depicted in Table 7 , demonstrate the superior performance of our ADAR, which achieves notable performance despite utilizing fewer parameters than LLMs.It is imperative to acknowledge the constraint of GPT3.5 that it solely processes textual inputs. Despite its commendable generative prowess, GPT3.5's performance in multimodal tasks was comparatively subdued. This outcome reveals the significant potential and necessity of integrating and leveraging cross-modal information to enhance the accuracy and applicability in multimodal NLP tasks. Thus, GPT4V shows a better performance on MASC with its powerful multimodal capability. However, our ADAR with fewer parameters and training time, while not reaching the same level of complexity or depth as GPT4V, demonstrates a notable efficiency on MASC.To better learn and improve future work on MABSA, we conduct experiments on accuracy of sentences with different characteristics for JMASA on Twitter2015 and Twitter2017 test set. As shown in Table 8 , the two test set are different in aspects and sentiments distribution. Twitter2015 focus on one aspect and single sentiment circumstance, while Twitter2017 contains more sentences with multiple aspects and sentiments. From Figure 6 , we observe that the trend appears to be similar for all three methods in both datasets. The accuracy is higher when only one aspect is mentioned and tends to decrease as the complexity of the sentence increases with multiple aspects and multiple sentiments. For Twitter2015, our ADAR generally outperforms DTCA and AoM across all three characteristics, indicating it handles complexity better due to the aspect-driven method. For Twitter2017, the same pattern holds, although the differences in accuracy between ADAR and the other two methods are less pronounced. ADAR shows a consistent performance advantage over the other two methods, which indicates it has a better handling of sentences with multiple aspects and sentiments. Additionally, we experimented to analyze our method's computation efficiency compared to SOTA models. As shown in Table 9 , our method outperforms AoM and approaches VLP-MABSA in inference time per step due to the Sinkhorn algorithm we use. As described in Section 4, this algorithm reinterprets transport problems through a maximum-entropy perspective and introduces an entropic regularization term to the classic OT problem, streamlining the process. Thus, while our method's complexity only lies in the theoretical aspects of optimal transport, it is easy and efficient to apply in practice.Latency/Inference Time per Step VLP-MABSA 0.484s AoM 0.563s Ours 0.487s  
conclusion: 
 In this paper, we proposed the Aspect-driven Alignment and Refinement model crafted for the task of MABSA, which seamlessly unifies the Coarse-to-fine Aspect-driven Alignment module and the Aspect-driven Refinement module. Notably, our model addresses NCP from the perspective of aspects. Empirical analysis on two extensively utilized datasets substantiates the efficacy of our approach. 
 
 Appendix: Despite the notable superiority of our proposed method over existing SOTA approaches, it is imperative to acknowledge and address several challenges in future research endeavors. Firstly, predicting emotions of tweet posts, which are related to contemporary issues, is hindered by a lack of requisite external knowledge. Secondly, our ADAM method exhibits limitations in dealing with missing visual elements. Acting as a double-edged sword, the adaptive filter bin effectively removes irrelevant multimodal entities but also overlooks the missing information. These limitations present a critical area for further investigation in our subsequent research efforts. 


Decision: The paper received mixed scores. The average score is on the positive side, however most of positive scores did not consider the rebuttal. In general, there still are concerns.

Reviews: review: paper_summary: paper_summary: In this paper authors propose a method named Aspect-driven Alignment and Refinement (ADAR) for Multimodal Aspect-based Sentiment Analysis (MABSA). In particular, this framework is able to extract scene subjects and perform sentiment analysis for each subject based on a picture of the scene and on a descriptive caption. 
The method differs from previous works in some aspects, such as the adoption of a dedicated alignment module based on Optimal Transport to tune the alignment between textual and visual features and an additional refinement module to further refine cross-modal features.
The proposal is evaluated on two datasets, demonstrating improvements in terms of Precision, Recall and F1 compared to existing works. An extensive configuration and ablation study is proposed to evaluate the impact of each choice made in the experiments.
strength_weakness: strength_weakness: - The paper is well written
- Experimental evaluation is vast and deeply detailed
- I appreciate the addition of unconventional but interesting sections that unveils useful insights, such as the hyper-parameter impact analysis and the comparison with general-purpose LLMs.
limitations: limitations: - In Table 2, I assume "P", "R" stand for Precision and Recall. I understand they may have been contracted for spacing reasons but a legend in the caption is necessary for clarity.

- The performance gains stand in the range of 1-2% compared to top-performing baselines, which is not an impressive result compared to the complexity of the method.

I find the paper to be in-theme with the conference and the work is worth publishing. However, I am not convinced by the quantitative improvements, especially when we compare it with the complexity introduced in the framework and when we consider that the most modern dataset dates back to 2017.
paper_summary: paper_summary: This paper introduces the ADAR model to perform MABSA using a two-stage coarse-to-fine alignment framework. The idea is interesting, but some issues need to be addressed.
strength_weakness: strength_weakness: Give a new framework for the MABSA, the idea is good and interesting.
The paper's writing is good with the proper organization.
limitations: limitations: Initial Word Embedding Impact: Does the initial word embedding affect the experimental results? Specifically, did the compared models (e.g., those mentioned in references [14] and [32]) also use the same initial word embeddings or visual embeddings?
 There is no evaluation metric provided to demonstrate how well the alignment between image and text is achieved. This is important since the paper claims to have a superior alignment framework.
The improvement shown in the experiments seems limited compared to the current state-of-the-art methods.
paper_summary: paper_summary: This paper mainly addresses the problem of poor alignment between visual content and text description in Multimodal Aspect-based Sentiment Analysis tasks. This article proposes a two-stage framework that introduces the Coarse to fine Aspect driven alignment module to learn coarse-grained alignment between text and vision, and the Aspect driven Refinement module to refine multimodal feature representations. This article conducted a series of experimental verifications on the proposed framework, and the experimental verification results showed the effectiveness of the framework.
strength_weakness: strength_weakness: This paper is innovative to some extent, and it is the first work to introduce Optimal Transport into MABSA to solve the NCP problem.
The experimental section of this article is rigorous, sufficient, and complete. This article conducts experiments on two datasets, Twitter2015 and Twitter2017, using multiple evaluation metrics such as ABSA, MATE, MASC, and JMASA, and compares them with current advanced models to verify that the proposed method has good performance. At the same time, ablation experiments were conducted to verify the effectiveness of each module. This article also compared it with the currently popular LLM and found that its performance can even surpass LLM with fewer parameters. This article also analyzes the errors in framework prediction, providing ideas for future work.
limitations: limitations: The adaptive filter bin proposed in this article may filter out some useful information, causing the model to move from one side to another, which may decline the performance of the model.
paper_summary: paper_summary: This paper focuses on the noisy correspondence problem (NSP) in the multimodal aspect-based sentiment analysis task and proposes an Aspect-Driven Alignment and Refinement framework. It first devises a coarse-to-fine aspect-driven alignment module to learn the alignment between visual and textual features. Then, an aspect-driven refinement module is used to refine the feature representation. Experiments on two benchmarks across 3 tasks show that the proposed method outperforms the other counterparts.
strength_weakness: strength_weakness: -	This paper focuses on an interesting problem and proposes a new method to solve it.
-	The paper is overall well-written and the storyline is clear.
-	Extensive experiments show the effectiveness of the proposed method.
limitations: limitations: -	The proposed framework is only evaluated on the BART-based models. The paper will be more convincing if the framework can work well on the other LLMs or MLLMs.
-	The compared methods are weak, as the newest compared method is AoM published in ACL2023. I suggest to compare with more recent counterparts, e.g., “WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge” (arXiv 2024).
-	Some claims are not well confirmed by the experiments. For example, there is a lack of experiments and analyses to investigate whether the proposed methods can effectively alleviate the NSP problem. 
-	The proposed methods are relatively complex. More experiments on the efficiency of proposed methods are required.