Aspect-Based Multimodal Mining: Unveiling Sentiments, Complaints, and Beyond in User-Generated Content

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 13:48:07

Abstract: Sentiment analysis and complaint identification are key tools in mining user preferences by measuring the polarity and breach of expectations. Recent works on complaint identification identify aspect categories and classify them into complaint or not-complaint classes. However, aspect category-based complaint identification provides high-level information about the features of products. 
In addition, it is also observed that the user sometimes does not complain about a specific aspect but expresses concern about specific aspects in a respectful way. Currently, uni-modal and multimodal studies do not differentiate between this thin line between complaint and concern. 
In this work, we propose the task of multimodal aspect term-based analysis beyond sentiments and complaints. It comprises of two sub-tasks, \textit{viz} (i) classification of the given aspect term into one of the four classes, \textit{viz.} praise, concern, complaint, and others, (ii) identification of the cause of praise, concern, and complaint classes. We propose a first benchmark explainable multimodal corpus annotated for aspect term-based complaints, praises, concerns, their corresponding causes, and sentiments. Further, we propose an effective technique for the joint learning of aspect term-based complaint/concern/praise identification and cause extraction tasks (primary tasks) where sentiment analysis is used as a secondary task to assist primary tasks and establish them as baselines for further research in this direction. Sample dataset has been made available at: \url{https://anonymous.4open.science/r/MAspectX-327E/README.md} The whole dataset will be made publicly available for research after acceptance of the paper.

Main paper: 
 introduction: 
 Online user reviews have become a critical source of information across a broad spectrum of domains, including e-commerce, hospitality, healthcare, and entertainment [11, 16, 35] . Nowadays, users are posting both visual and textual information in order to better express their opinions and experiences. Organizations can improve their products or services by mining rich-source textual and visual content. Additionally, it allows other customers to make more informed decisions about their purchases.Sentiment analysis and complaint detection play an important role in accomplishing these objectives [12, 21, 33] . Sentiment analysis classifies the polarity of a review into one of the three classes, viz., positive, negative, and neutral [17] . However, due to limited information captured by sentiment analysis systems, researchers proposed the task of complaint detection, which identifies breaches of expectation (complaints) from user-generated reviews. Complaint identification is a binary classification problem, categorizing review as a complaint or not a complaint [21] .There have been works on complaint identification at the document or the sentence levels [4, 8, 12] . In the prior works, authors have also identified the severity [8, 32] and cause [26, 27] of complaints at the sentence level. An attempt has been made towards analyzing complaints at a fine-grained level [25] , using a multimodal corpus (text + image) for binary complaint identification of each aspect category described in the review. Although this corpus provides vital information at the aspect level, it neither provide specific and fine-grained information about specific features or attributes (aspect terms) nor their cause for particular aspect . For example, the customer may complain about the neck size but can be happy with the arm size in the same review. However, authors in [25] only provide information that the user is complaining about the size. It does not reveal whether the user complains about neck or arm size (aspect terms) and the cause for their dissatisfaction. This missing information can mislead and hinder the organizations in their decision-making processes, as there can be multiple aspect terms falling under one aspect category.Apart from expressing satisfaction (praise) or dissatisfaction (complaints), we also observe that sometimes reviewers do not complain about a specific aspect but express a feeling of worry or interest about something, often with the intention of improving an issue. We call it a concern. Concern is usually expressed in a constructive and respectful way, focused on finding a solution or improvement or making suggestions to others. For example, Be careful with what size you order; those who are looking for comfort can order for 1 size larger than usual. Here, the user is suggesting to be careful with what size they order, hence expressing a concern for size. Distinguishing between concerns and complaints can help in managing customer relationships. Addressing concerns promptly and constructively can enhance customer satisfaction, while efficiently resolving complaints can help the dissatisfied customers.However, to the best of our knowledge, current studies (uni-and multimodal) do not consider this differentiation between concerns and complaints and also do not provide complaint analysis and corresponding cause extraction at the aspect term level. To bridge this gap, we propose a new task, namely, aspect term based multimodal analysis beyond sentiments and complaints. Our proposed task consists of two sub-tasks, multimodal aspect term based analysis beyond sentiment and complaint (MAspectX) and finding the cause of class predicted in first sub-task (MAspectX-Cause). Given the review and image, MAspectX task aims to classify the given aspect term into one of the four classes, viz., complaint, concern, praise, and others. Given the review and image, MAspectX-Cause task aims to extract the cause behind the class MAspectX for a given aspect. The cause or span of the MAspectX class can vary. It could be a complete sentence, phrase, or even a single word. Few examples are presented in Figure 1 .As there is no such corpus available, we first build a multimodal corpus (MAspectX) using large vision models to solve both the tasks [15, 31] . Further, To assist the MAspectX task, the dataset is also labeled for sentiment labels for every aspect term mentioned in the review. This task is known as aspect term sentiment analysis (MATSA) [19, 20] . Figure 1 shows a few example review texts and images with the corresponding MATSA, MAspectX, and MAspectX-Cause labels. In the examples, it is observed that text and images together provide a better understanding of the labels compared to only uni-modal.The key contributions of our current work are as follows:• We propose a new task multimodal aspect term based analysis beyond sentiment and complaint, MAspectX, to classify the aspect terms into one of four classes, viz. praise, complaint, concern, and others. • We propose multimodal aspect term based cause extraction task, MAspectX-Cause to identify the rationale or causes behind the complain/concern/praise/other classes for every aspect term.MAspectX-Cause tasks in a semi-supervised fashion using multi-step few-shot prompting. • We build a multi-task system to jointly solve MAspectX, MAspectX-Cause and MATSA tasks in parallel. • The proposed model outperforms the state-of-the-art methods and can serve as a benchmark for further research in this direction. 
related work: 
 Previous works on complaint detection identify the complaints at the document or the sentence levels using feature-based machine learning models [4, 12] and transformers-based models [22] . Additionally, the multitask model has been proposed to aid in complaint analysis by incorporating polarity and affect information [28] . Efforts have also been directed towards identifying the severity [8, 32] and cause [26, 27] of complaints. However, all these attempts to identify severity and cause at the sentence level. In the existing literature, complaints have been categorized on the basis of several factors, including the responsible department, product hazards, degree of urgency, and risks [2, 13, 36] . Recently, attempts have been made towards analyzing complaints at a fine-grained level [25, 29] , using a multimodal corpus (text + image). This corpus is annotated for complaint and non-complaint labels for each aspect category described in the review. However, this data does not provide any specific or more fine-grained information about the specific features or attributes (aspect terms) and corresponding cause (explainability). Aspect-based complaint analysis (ABCA) shares similarities with the well-established domain of aspect-based sentiment analysis (ABSA). However, it is crucial to recognize that despite their connections, ABSA and ABCA are distinct concepts with different objectives, scopes, and outputs. While, ABSA helps in understanding how customers feel about various aspects of a product or service [10, 34, 37] , ABCA, on the other hand, focuses on identifying and understanding complaints made by customers.Our work addresses the following limitations of current unimodal and multimodal studies: (i) current studies lack the capability to conduct complaint analysis at the aspect term level; (ii) They fail to differentiate effectively between complaints and concerns; (iii). existing studies do not provide the cause for complaints at a fine-grained level.To address these limitations, we propose new tasks, MAspectX and MAspectX-Cause to classify the aspect terms into one of the four classes and identify the cause behind them. In addition, we leverage the information from the sentiment task to enhance the performance of our proposed tasks. 
resource creation: 
 To create our corpus, we scraped reviews from Amazon's clothing domain, specifically targeting multimodal reviews in English2 . We follow a multi-step approach to annotate our corpus to reduce the manual annotation efforts. Verifying existing labels is generally quicker and more cost-effective than starting the labeling process from scratch. It reduces the need for annotators to label entire datasets and instead focuses on confirming or correcting existing annotations.The recent success of pre-trained large-scale models motivates us to utilize them for data annotations [15] . Multi-step few-shot prompting is used to assign weak labels to the dataset, where we use the Large Language and Vision Assistant Pretrained Transformer (LLaVA) model for prompting 3 . It uses a relatively small set of in-context examples (about 10 random samples).We have three tasks, namely MATSA, MAspectX, and MAspectX-Cause. As a first step, we first extract all possible aspect terms present in the review sentence. The output of this step is passed as input to the MATSA, MAspectX, and MAspectX-Cause tasks. Hence, three distinct prompts have been designed, each tailored for a specific step of the annotation process.Step 1: Extraction of Aspect Terms: We construct the following prompt template to ask LLaVA about the possible aspect terms mentioned in the review sentence.F1 [Given the review R], which specific aspect terms are possible mentioned?Here, 𝐹 1 are the few shot examples provided for the ATE task (consisting of text and image). Let, 𝐴 be the answer given by LLaVA,There was a nearly inch hole in the kurti. Quality of palazo is good.MAspectX: complaint, praise[There was a nearly inch hole in the kurti], [Quality of palazo is good]The material is thick so it's not transparent at all. I think it's worth the price. containing all possible aspect terms (𝑎𝑠𝑝) mentioned in the review 𝑅. If no aspect term is mentioned in 𝑅, the value of 𝐴 is no mention.Step 2: MATSA:. For a given aspect term 𝑎𝑠𝑝 𝑗 (obtained from step 1), we construct the following prompt template to ask LLaVA about the possible sentiment class of aspect term among the possible three classes, viz. positive, negative, and neutral.𝐹 2 [Given the review S], classify the aspect 𝑎𝑠𝑝 𝑗 into one of the three classes, positive, negative, and neutral.Here, 𝐹 2 are the few shot examples provided for the ATSA task (text and image).Step 3: MAspectX and MAspectX-Cause: For a given aspect term 𝑎𝑠𝑝 𝑗 (obtained from step 1), we construct the following prompt template to obtain the possible class of 𝑎𝑠𝑝 𝑗 among the 4 classes, praise, complaint, concern, and others and cause behind this classification.𝐹 3 [Given the review R], classify the aspect 𝑎𝑠𝑝 𝑗 into one of the four classes: complaint, concern, praise, others, and extract the phrase responsible for the predicted class from the review.Here, F3 are the few shot examples for MAspectX and MAspectX-Cause tasks (text and image). Suppose there are 𝑘 number of aspect terms in the sentence, then steps 2 and 3 are repeated for each of the aspect terms (𝑎𝑠𝑝 𝑗 ).To maintain the quality of the labels, we manually analyzed the labels obtained using a multi-step few-shot prompting method. We also observed a few errors in the labels. The majority of the errors were in the identification of aspect terms. For example, sometimes, LLaVA remembers the aspect terms provided in the few-shot context. Hence, it assigns the same aspect term to the current sentence, regardless of whether the review is talking about that attribute or not. For example, Input: Neck size is perfect, shoulder size is big.Output: Neck size, shoulders, price.Here, price is not part of the current sentence, but LLaVA has assigned it because of the few-shot context. Therefore, to ensure the annotation quality of the dataset, we engaged three linguists with sufficient subject knowledge and experience in the construction of supervised corpora. Two have doctoral degrees, and one has completed his master's. Linguists are asked to manually verify the annotations of every step. After the first step of annotation, linguists have verified the aspect term labels, so missing aspect terms can be added to the labels, or additional terms can be removed. This step ensures the quality of labels obtained at subsequent steps.After verification of first step labels, second and third step were performed to reduce the error rate. Guidelines along with some examples were explained to the linguists before starting the verification process. We follow the guidelines used in SemEval aspect level sentiment analysis task for verification of the aspect terms and our MATSA task [19] . We use the complaint definition from an earlier linguistic study: "A complaint presents a state of affairs that breaches the writer's favorable expectation" [3] . In addition to satisfaction (praise), we introduced a new class concern. MAspectX-Cause is a portion of text that express why the customer feels satisfaction, dissatisfaction, expressing a concern or others. We attained an overall Fleiss' kappa [30] score of 0.75 and 0.79 among the three linguists for MATSA and MAspectX tasks, respectively. For MAspectX-Cause task, we measure the quality of annotations using macro-F1 following earlier research [23, 27] and it comes out to be 0.78. These agreement scores indicate that the annotations can considered as reliable. The use of weak labeling prior to actual annotation reduced the annotation time and also helped in achieving the good quality labels. A few samples along with their labels are shown in Figure 1 .Data Distribution: The MAspectX dataset contains a total of 4,966 aspect terms with 3303 positive, 981 negative, and 682 neutral aspect instances. Detailed class-wise distribution of aspects for MATSA and MAspectX tasks are shown in Table 1 . There is a strong dominance of the positive/praise classes among the other classes. Instances for complaint/concern classes are comparatively less than other classes, which is in line with other works [24] .  
methodology: 
 We develop a multimodal multi-task model for MATSA, MAspectX, and MAspectX-Cause tasks for a given aspect term (MMAspectX ).The system employs a shared fusion mechanism to infuse the features from the multimodal inputs. Detailed architecture of our proposed MMAspectX is depicted in Figure 2 . The review text and aspect term are given as input to the text encoder (BERT), and the corresponding image is fed into the visual encoder (ResNet). Further, AutoEncoder component enhances the model's ability to capture subtle nuances and opinions expressed in the input text. Finally, the shared fusion component integrates information from both textual and visual modalities to provide a comprehensive understanding of the input data.To encode the textual modality, our textual encoder uses BERT (Bidirectional Encoder Representation from Transformers) [5] . BERT encoder has been pre-trained on two unsupervised tasks using a large Wikipedia corpus to capture the contextual relationship between words and sentences. Input to BERT encoder is review [SEP] 𝑎𝑠𝑝 𝑗 , where 𝑎𝑠𝑝 𝑗 is the 𝑗 𝑡ℎ aspect of the multimodal review.We employ ResNet 4 to capture rich visual cues from the images. To simplify the problem and maximize the utility of the embedding space, we partition the embedding dimensions and image data into distinct groups. Each learner is responsible for creating a unique distance metric using a subspace of the original embedding space and a subset of the training data. By segmenting the network's embedding layer into D consecutive slices, we isolate D distinct learners within the embedding space. Once these individual learner solutions converge, we aggregate them to reconstruct the entire embedding space. This merging process involves recombining the slices of the embedding layer that correspond to the D learners. To ensure consistency in the embeddings produced by these diverse 4 https://github.com/josharnoldjosh/ResNet-Extract-Image-Feature-Pytorch-Python learners, we conduct fine-grained tuning across the entire dataset. However, when merging the embeddings, "shattered gradients problem" occurs, where the gradients resemble white noise, hindering training performance. To mitigate this challenge, residual weights are used [1] to provide the gradients with some spatial structure, which facilitates training, as illustrated in Figure 2 .This part of the architecture enhances the model's ability to capture subtle nuances and opinions expressed in the input text. We leverage Context-Free-Grammar-Noun-Adjective-Pairs (Context Free ANP) to extract adjective-noun pairs from the utterances. This approach effectively allows our model to identify textual concepts.The ANP features, obtained through Context-Free ANP, are then fed into an auto-encoder to generate a latent representation. To infuse both textual and class semantic knowledge into the ANP representation, we employ an adversarial loss [38] , as elaborated in the "Training and Inference" section. This adversarial loss is designed to disentangle syntax (captured by ANP) from semantics (captured by contextual character embeddings). This disentanglement enhances interpretability and offers better control over the learned representations.The Fusion component integrates information from both textual and visual modalities to provide a comprehensive understanding of the input data. To effectively fuse the knowledge from both modalities, we employ a shared fusion technique as shown in Figure 3 . This technique uses cross-attention to capture the relationship between both modalities, preserving their unique characteristics. 𝑍 :EQUATIONwhere D denoted by 𝐷 = 𝑥 𝑎 + 𝑥 𝑡 shows the concatenated features'(image, text) dimension. For the given multimodal review (𝑀 𝑟 ), the combined feature representations (Z) are now used to focus on the uni-modal feature representations 𝐹 𝑎 , and 𝐹 𝑡 . The combined features (Z) and the shared correlation matrix (M) between the image features are given by:EQUATION, where 𝑊 𝑧𝑎 ∈ R 𝐿 * 𝐿 is the learnable weight matrix across image and shared image, and text features. Similarly correlation matrix for text features is defined as:EQUATIONThe shared correlation matrices 𝐶𝑜𝑀𝑎𝑡 𝑎 , and 𝐶𝑜𝑀𝑎𝑡 𝑡 for the image, and text modalities offer a semantic indicator of importance both within and between modalities. Within the same modality, there is a high correlation between the matching samples and the other modalities, as indicated by a greater correlation coefficient of the shared correlation matrices 𝐶𝑜𝑀𝑎𝑡 𝑡 , and 𝐶𝑜𝑀𝑎𝑡 𝑎 . In order to improve the performance of the system, the suggested strategy effectively takes advantage of the complimentary nature of the image, and text modalities (i.e., inter-modal relationships) and intramodal relationships. The attention weights of the image and text modalities are computed following the computation of the shared correlation matrices.We use several learnable weight matrices corresponding to features of the separate modalities to compute attention weights for the modalities because the dimensions of shared correlation matrices and the features of the associated modality vary. The learnable weight matrices 𝑊 𝑐𝑎 and 𝑊 𝑎 are used to combine the shared correlation matrix 𝐶𝑜𝑀𝑎𝑡 𝑎 and the matching image features 𝐹 𝑎 , the following formula is used to calculate the attention weights for the image modality:EQUATION, where 𝐴𝑡𝑡𝑒 𝑎 represents the attention map of image modality. Similarly, for the textual modality, the equation is:EQUATIONThe attended characteristics of the image and text modalities are computed using the attention maps. These characteristics are attained by: 𝑋 𝐴𝑡𝑡𝑒 𝑎 ,𝑎 = 𝑊 𝐴𝑡𝑡𝑒 𝑎 + 𝐹 𝑎 , 𝑋 𝐴𝑡𝑡𝑒 𝑡 ,𝑡 = 𝑊 𝐴𝑡𝑡𝑒 𝑡 + 𝐹 𝑡 (6) Finally, the shared features of attended features of image and text are obtained by: 𝑋 𝐴𝑡𝑡𝑒 = [𝑋 𝐴𝑡𝑡𝑒 𝑎 ,𝑎 ; 𝑋 𝐴𝑡𝑡𝑒 𝑡 ,𝑡 ]Adversarial Loss: Our goal is to reduce the gap between the discriminative capability of the text (𝜃 (𝑥) representing 𝑆𝐴(𝑡)) and the rich opinion structural information encapsulated in the feature 𝜙 (𝑙 𝑝𝑜 ). This is accomplished by employing an adversarial constraint designed to deceive the discriminator network D, thereby making the output features of 𝐴(𝜃 (𝑥)) as similar to the ANP features as feasible. L 𝑎𝑑 = E 𝑦 (log D (ℎ(𝑦)) -E 𝑦 (log D (𝜃 (𝑦))Joint Loss: We train our model by incorporating a blend of the four diverse loss functions. L 𝑗𝑜𝑖𝑛𝑡 = L 𝑎𝑑 + L 𝑐𝑎𝑢𝑠𝑒 + L 𝑠𝑒 + L 𝑐𝑐𝑝 . In this context, L 𝑐𝑎𝑢𝑠𝑒 denotes the loss associated with MAspectX-Cause, which is calculated using binary cross-entropy. Additionally, L 𝑠𝑒 and L 𝑐𝑐𝑝 represent the losses related to MATSA and MAspectX tasks, respectively, both of which are cross-entropy losses. 
experiments and results: 
 In this section, we delve into the experiments we conducted, present the results we obtained, and provide a comprehensive analysis.Our proposed model is developed using PyTorch5 , a Python-based deep learning framework. We employ the base version of BERT obtained from the huggingface transformers6 package. All experiments are executed on an NVIDIA GeForce RTX 2080 Ti GPU. Our experiments run for 4 epochs, and we report the average scores after conducting 3 runs of the experiments to account for the inherent non-determinism associated with Tensorflow GPU operations. We utilize several evaluation metrics for the MAspectX-Cause task, including Full Match (FM), Partial Match (PM), Hamming Distance (HD), Jaccard Similarity (JS), and Ratcliffe-Obershelp Similarity (ROS). For MATSA and MAspectX tasks, we assess macro-F1 and Accuracy. We split the MASpectX dataset into an 80-20 ratio for training and testing. The best model is selected based on the performance on the validation set (10%) Baselines: Our multimodal multi-task framework MMAspectX combines MATSA, MAspectX, and MAspectX-Cause into single system. Due to lack of suitable multimodal aspect based baselines with similar objectives (discussed in Section 2), chosen baselines includes the following systems: (i) Aspect category based multimodal complain detection system (ABCD) based on BiGRU with self attention layer [25] , initially designed for binary classification task for aspect category based complaint detection (complaint vs non-complaint); (ii) SpanBERT [9] , MT-BERT (multi-task BERT) [18] , and the Cascaded Multi-task System with External Knowledge Infusion (CMSEKI) using SenticNet [6] . We have extended these systems by including image and aspect term information in the input. Image and textual modalities are fused through concatenation in all these baselines. In line with the MMAspectX method, we have expanded the capabilities of these benchmark models to accommodate our multi-task scenario. More specifically, we have augmented them by adding two additional linear layers on top of the hidden-state outputs (for MATSA and MAspectX ). For the output layer of the cause extraction task, we have incorporated a sigmoid activation function with a predetermined threshold value of 0.4. We have also compared our approach using few shot prompting using LLama textual model (Large Language Model Meta AI) [31] and LLava (multimodal) [14] models.We investigate how various multimodal factors influence the tasks we are examining. The results of our proposed MMAspectX framework for MAspectX (F1), MATSA (F1), MAspectX-Cause (ROS) are depicted in Table 2 . The bimodal configuration produces the most favorable results compared to unimodal networks (only text). Results illustrate the inadequacy of textual information alone in capturing all labels, necessitating the integration of image data to enhance feature representation. Our findings align with previous research [7] . Table 3 illustrates the effectiveness of multi-task learning. It shows that single tasks and dual tasks models perform inferior to models where all three tasks are solved jointly.As depicted in Table 4 , it is evident that CMSEKI stands out as the top-performing baseline. This result is in line with expectations, given that CMSEKI leverages common-sense knowledge from external sources to better comprehend input information. However, it is noteworthy that the proposed MMAspectX method consistently outperforms the CMSEKI model across all metrics, demonstrating a 3.92% and 2.14% improvement in F1 for the sentiment, and complain task, respectively, and a 3-point increase in the Ratcliffe-Obershelp Similarity (ROS) score. The relatively low performance observed in SpanBERT [9] highlights the challenges that even the powerful language models face in tackling critical tasks, such as cause extraction, particularly in situations involving complaints where training data may exhibit increased complexity. Furthermore, our observations indicate that user sentiment significantly enhances the performance of the MMAspectX method across the tasks under consideration. We perform the paired T-test (significance test), which validates that the performance gain over all the baselines is significant with 95% confidence (p-value<0.05). The LLama and LLava models also perform inferior to our proposed approach. It indicates that task-specific fine-tuning helps to learn domain information, which results in improvement.As shown in Table 4 , we perform an ablation study on the MAspectX dataset to analyze the performance of the different components in our proposed framework. The values of the metrics for the is omitted. The performance drop is more profound when either one, two, or all the components are removed. This affirms that the involvement of the shared fusion, adjective-noun pair, and embedding clustering of the utterances significantly contribute to the effectiveness of the three proposed tasks.To manually evaluate the MAspectX-Cause task, 200 model-extracted spans are randomly selected and assessed based on established criteria. For each instance, we present the responses (produced by models and ground-truth by humans) to our annotators. The human raters hold post-graduate degrees in science and linguistics and have prior annotation experience in text mining tasks. The assessment includes evaluating the spans for Fluency and Adequacy on a five-point scale, ranging from unacceptable (0) to excellent (5).• Fluency: This metric evaluates the grammatical correctness of a sentence. • Adequacy: Employed to ascertain if the generated response is meaningful and pertinent to the preceding conversation.Additionally, the assessment considers the Informativeness of the extracted span. The Informativeness metric ranges from 0 (lack of information) to 5 (highly informative reasoning).The results of the human evaluation, as presented in Table 6 , align with the experimental findings discussed in Table 4 . These findings affirm the superior performance of the MMAspectX model in comparison to the existing baselines. It is noteworthy that MMAspectX consistently outperforms the baselines across various manual evaluation metrics. The generated utterances are not only fluent but We conduct a thorough examination of predictions made by different systems, as illustrated in Table 5 . In the top row, human annotators have identified tokens, referred to as 'causes,' which serve as representations of the causes for labels, such as MATSA and MAspectX. The subsequent four rows display tokens extracted by various models. It is evident that the proposed MMAspectX model excels in accurately recognizing examples as instances of MATSA and MAspectX, providing high-quality causal spans. In contrast, the SpanBERT model, while correctly capturing a partial causal span, misclassifies the labels for both MATSA and MAspectX. 
conclusion: 
 We introduced novel tasks, multimodal aspect term based analysis beyond sentiment and complaint (MAspectX) and finding the cause of class predicted in the first task (MAspectX-Cause). We presented a multimodal dataset, MAspectX, that encompasses user complaints, praises, concerns, their associated sentiments, and the underlying causes behind all the classes at the aspect term level. This would immensely benefit organizations and enterprises in helping manage customer relationships by solving issues promptly and efficiently.We have developed a benchmark setup to create MAspectX corpus in a semi-supervised way using multi-step few-shot prompting followed by manual verifications of all the labels. Furthermore, we introduce a framework MMAspectX to solve MATSA, MAspectX, and MAspectX-Cause tasks jointly. Our results demonstrate that the proposed model MMAspectX can effectively share knowledge among all three tasks and outperforms various baselines. 
 
 Appendix: Like most studies, this study has some limitations that could be addressed in future research. Our current study is focused on only a single cause of the MAspectX task. However, multiple causes can be present in the review that can provide more detailed information. These limitations could be addressed in the future by incorporating multiple causes in the dataset. The scope of our current work is confined to the English language, specifically within the clothing domain. In the future, our efforts will focus on enhancing our work in code-mixed languages and other domains. We would investigate ways of sharing knowledge between code-mixed and English languages to gauge the performance.This study has been evaluated and approved by our Institutional Review Board (IRB). The images and review text used to create the dataset for this study do not have any copyright clauses attached to them. Furthermore, the dataset will be shared (upon acceptance) via various channels for the main purpose of facilitating research and educational purposes. Sample dataset has been made available at: https://anonymous.4open.science/r/ MAspectX-327E/README.md The whole dataset will be made publicly available for research after acceptance of the paper.https://www.amazon.inhttps://llava-vl.github.io/https://pytorch.org/https://huggingface.co/docs/transformers/index 


Decision: This paper presents a novel multimodal aspect-based analysis task that goes beyond traditional sentiment analysis by distinguishing between complaints, concerns, and praise. The authors introduce a new dataset and propose a joint learning model to address this task. While reviewers acknowledge the paper's contributions and innovative approach, they also point out several limitations, including the need for more methodological details, concerns about dataset bias, and questions about generalizability beyond the specific domain studied. The rebuttal appears to have addressed some concerns, leading one reviewer (ZVtd) to increase their score, while another decreased theirs. Despite these mixed reactions, the paper's novel task formulation and dataset contribution seem to outweigh its limitations. 4/5 reviewers have shown good interest and have leaned towards the acceptance of the paper. Given the potential impact on the field of multimodal sentiment analysis and my own insights into this work, I recommend accepting this paper, with the expectation that the authors will address the reviewers' concerns in the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a new task called "Multimodal Aspect Term-Based Analysis beyond Sentiment and Complaint (MAspectX)" and a related task "Multimodal Aspect Term-Based Cause Extraction (MAspectX-Cause)". The authors introduce a multimodal dataset (text and images) annotated for aspect term-based complaints, praises, concerns, their corresponding causes, and sentiments. They present a multi-task framework (MMAspectX) that jointly learns the proposed tasks, leveraging sentiment analysis as an auxiliary task. The paper claims to address the limitations of existing studies that lack fine-grained, aspect term-level analysis and the differentiation between complaints and concerns.
strength_weakness: strength_weakness: - The proposed tasks, MAspectX and MAspectX-Cause, address a gap in existing research by providing a more fine-grained analysis of aspect terms and distinguishing between complaints and concerns, which can be valuable for customer relationship management.
- The paper describes a semi-supervised approach using few-shot prompting and manual verification to create a multimodal dataset for the proposed tasks, addressing the lack of such resources.
- The proposed MMAspectX framework jointly learns the primary tasks (MAspectX and MAspectX-Cause) and the auxiliary sentiment analysis task, leveraging the complementary information from different modalities.
- The paper presents a thorough evaluation of the proposed framework, including comparisons with baseline methods, ablation studies, and human evaluations, demonstrating the effectiveness of the approach.
limitations: limitations: - The current work focuses on identifying a single cause for the MAspectX task, while multiple causes may exist in a review, limiting the level of detail and interpretability.
- The dataset and experiments are confined to the English language and the clothing domain, which limits the generalizability of the proposed approach to other domains and languages.
- While the paper introduces the concept of cause extraction, it does not provide in-depth analysis or visualization of the learned representations or the decision-making process, which could enhance interpretability and transparency.
- The paper compares the proposed approach primarily with baseline methods adapted from other tasks, as there are no directly comparable methods for the proposed tasks. A more comprehensive comparison with related tasks or approaches could provide additional insights.
paper_summary: paper_summary: The paper addresses a significant gap in the existing literature by proposing a novel method for multimodal aspect-based analysis that distinguishes between complaints, concerns, and praise. This differentiation is particularly innovative as it goes beyond traditional sentiment analysis, providing more granular insights into user-generated content.
strength_weakness: strength_weakness: 1. The introduction of the MAspectX and MAspectX-Cause tasks is commendable. The approach of using multimodal data (text and image) for classifying and extracting the cause behind each classification demonstrates a thorough and methodical advancement over unimodal analyses. This could potentially lead to more accurate and context-aware systems.
2. The creation of a new multimodal corpus specifically annotated for aspect term-based analysis is a significant contribution. This not only facilitates the specific tasks proposed but also sets a benchmark for future research in the area. Additionally, the use of multi-step few-shot prompting for annotation suggests an innovative approach to dataset creation, which could be of interest to other researchers dealing with similar challenges.
limitations: limitations: 1.  While the paper introduces innovative tasks and a novel dataset, the description of the methodology lacks sufficient detail for full reproducibility. For instance, the specifics of the machine learning models used, feature extraction methods, and the exact nature of the multimodal integration are not thoroughly explained. More technical details would enhance the reader's understanding and the scientific robustness of the paper.
2. The paper would benefit from a more comprehensive validation section. While it claims that the proposed model outperforms state-of-the-art methods, there is limited discussion on the comparative analysis, such as the metrics used for benchmarking, statistical significance of the results, and the performance across different types of data (text-heavy vs. image-heavy).
3. The discussion section lacks an in-depth analysis of the limitations and potential biases inherent in the proposed methods and dataset.
paper_summary: paper_summary: The paper propose the task of multimodal aspect term-based analysis beyond sentiments and complaints to distinguish between concern and complaints, including classification of the given aspect term into one of four classes and identification of the cause of these classes. An effective technique is used for the joint learning of aspect term-based identification and cause extraction tasks(primary tasks), where sentiment analysis is used as a secondary task.
strength_weakness: strength_weakness: 1.The paper proposes a new task named aspect term-based multimodal analysis beyond sentiments and complaints to bridge the gap between the differentiation between concerns and complaints and provide complaint analysis and corresponding cause extraction at the aspect term level.

2. It also builds a multimodal corpus (MAspectX) to solve both the tasks.
limitations: limitations: 1.Figure2 and Figure3 are not very clear for understanding

2.The section of training and inference is not clear enough for understanding.

3.The section of experimental results and analysis is too brief.
paper_summary: paper_summary: This paper proposes a novel task for multimodal aspect term-based analysis, which goes beyond traditional sentiment and complaint detection by introducing categories such as praise and concern. The authors develop a new benchmark dataset, MAspectX, and introduce a joint learning model (MMAspectX) for this task. The model aims to classify aspect terms into four classes (complaint, concern, praise, and others) and identify the causes behind these classifications. The experimental results show that the proposed model outperforms several baselines, demonstrating the effectiveness of incorporating multimodal information and joint learning.
strength_weakness: strength_weakness: (1) The paper introduces a novel task that extends traditional sentiment and complaint analysis by adding new categories like praise and concern.

(2) The creation of the MAspectX dataset, which includes multimodal (text and image) data annotated for aspect term-level complaints, praises, concerns, and their causes, is a significant contribution.

(3) The proposed MMAspectX model effectively integrates multiple tasks (sentiment analysis, aspect term classification, and cause extraction), demonstrating superior performance over existing baselines.

(4) The authors conduct extensive experiments and comparisons with various state-of-the-art models, providing strong evidence for the effectiveness of their approach.

(5) The paper includes a thorough analysis of the model's performance, including ablation studies and human evaluations, which enhance the credibility of the results.
limitations: limitations: (1) The dataset shows a strong dominance of positive/praise classes, which could bias the model's performance. How does the model perform on a more balanced dataset? Can you provide additional experiments to address this concern?

(2) While the authors employ manual corrections to ensure annotation quality, the reliance on large language models for initial weak labeling could introduce biases. 

(3) You mentioned that the combination of text and images provides a better understanding of labels compared to a single modality, but I don't see a comparison with a single modality in Figure 1. Can you illustrate this difference?

(4) Can you modify Figure 2 to make the whole picture look more coordinated?

(5) Can you add some papers on the use of multimodal corpora (text + images) to analyze complaints at a fine-grained level, and need to analyze these methods?

(6) Why did you choose LLaVA as the aspect term extraction model? What are the different advantages of this model compared with other models?

(7) Notations and descriptions of the chart may need to be more detailed to ensure that the reader understands the information expressed in the chart even without reading the full text.

(8) Open-source code is recommended to ensure research transparency and reproducibility.

(9) While the paper compares the proposed model with several baselines, it does not include a comparison with some potentially relevant recent approaches in the field of multimodal sentiment and complaint analysis.

(10) The paper primarily focuses on standard metrics like F1-score and accuracy. Including more diverse evaluation metrics, such as precision, recall, and more detailed error analysis, could provide a more comprehensive understanding of the model's performance.

(11) While the paper mentions limitations, it lacks a detailed discussion on future work directions to address these limitations and further improve the model.

(12) The paper could provide more insights into how the relationships between text and image modalities are leveraged by the model and the impact of these relationships on performance.
paper_summary: paper_summary: The authors identified a gap in existing sentiment analysis and complaint recognition research, noting that the distinction between user concerns and complaints is not adequately refined. To address this, they propose a new task: aspect term-based multimodal analysis beyond sentiments and complaints. This task comprises two sub-tasks: multimodal aspect term-based analysis beyond sentiment and complaint (MAspectX) and identifying the cause of the class predicted in the first sub-task (MAspectX-Cause). For this new task, the authors use LLaVA to help construct a new multimodal dataset called MAspectX. Finally, they introduce the MMAspectX model and validate its effectiveness in addressing the new task using the new dataset.
strength_weakness: strength_weakness: a. The authors introduce a novel task along with the creation of a new dataset, offering a valuable resource for the multimodal sentiment analysis and complaint recognition community. This initiative not only enriches the available resources but also enhances the interpretability of tasks within this domain.

b. Key resources (sample dataset, source code) are available.
limitations: limitations: a. The authors would benefit from providing a more detailed definition and description of the inputs and outputs of the new task. This would greatly aid researchers in understanding and following the task effectively.

b. The paper requires thorough verification of writing details and diagrams：
(1) It appears that Z in Eq. (2) and Eq. (3) might serve as one of the subscripts of W.
(2) Throughout the paper, the subscript a denotes the image modal features, whereas in Figure 3, the subscript i is used for the image modal features. It is recommended to maintain a consistent subscript notation for clarity and coherence.
(3) According to Eq. (2) and Eq. (3), the steps to compute CoMat_a and CoMat_t should be symmetric. However, the computational flow depicted in Figure 3 does not appear to be symmetric.
(4) In Figure 2, there is a loss L_re, but in the final calculation of the joint loss, only L_se is included. What is the role of L_re?
(5) In Table 5, the text corresponding to SpanBERT and CMSEKI for the first case is incomplete.

c. In lines 394-395, how can the embedding dimension be divided into different groups with the image data? Isn't the embedding dimension a scalar?

d. What is the specific calculation process of  L_{re}? What is the motivation behind its design?

e. In Section 4.3, what is the theoretical support for the statement "separation of syntax and semantics enhances interpretability"?

f. In lines 624 to 626, the ratio of the training set to the test set is 8:2. Where is the validation set obtained from? In other words, what data did the author use for validation?

g. A key issue is that the MAspectX-Cause task is a generation task. So, how can it use a similar Linear layer to produce output as in the MATSA and MAspectX classification tasks? The author should provide an explanation of the decoding process for the MAspectX-Cause generation task, even if it's just a brief overview.

h. The experimental details need to be refined, such as the dropout rate, learning rate, early stopping, and embedding dimension.