Caption-Aware Multimodal Relation Extraction with Mutual Information Maximization

Authors: zefan  zhang
weiqi  zhang
yanhui  li
tian  bai
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:20:11

Abstract: Multimodal Relation Extraction (MRE) has achieved great improvements. However, modern MRE models are easily affected by irrelevant objects during multimodal alignment which are called error sensitivity issues. The main reason is that visual features are not fully aligned with textual features and the reasoning process may suppress redundant and noisy information at the risk of losing critical information. In light of this, we propose a Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM). Specifically, we first generate detailed image captions through the Large Language Model (LLM). Then, the Caption-Aware Module (CAM) hierarchically aligns the fine-grained visual entities and textual entities for reasoning. In addition, for preserving crucial information within different modalities, we leverage a Mutual Information Maximization method to regulate the multimodal reasoning module. Experiments show that our model outperforms the state-of-the-art MRE models on the benchmark dataset MNRE. Further ablation studies prove the pluggable and effective performance of our Caption-Aware Module and mutual information maximization method. Our code will be public soon.

Main paper: 
 introduction: 
 Relation Extraction (RE) as a significant study in information extraction, aims to detect potential relations among entities in the unstructured text and plays an important role in various applications [12, 32, 39, 43, 50, 59, 60, 62] . Previous studies mostly focus on extracting information from a single textual modality [19, [24] [25] [26] [27] . With the popularity of multimodal learning and deep learning [34, 46, [52] [53] [54] 66] , research abilities that solely on text become limited [65, 67] . Multimodal Relation Extraction (MRE) methods are proposed to significantly assist text-based models by using images as additional inputs [6, 20, 48] .Early methods [3, 33, 35, 61] encode the text through RNN and the image through CNN, studying how to incorporate the feature of the whole image into a text representation. Many works [55, 57, 65] further validate that object-level visual fusion is more specific and important for MRE. Recently, cross-modal pretraining seems promising [18, 42] . Despite the remarkable results achieved by these methods, there are still some challenges:Error sensitivity issues. The multimodal reasoning and fusion process will be interfered with by error sensitivity issues, which means that irrelevant objects are incorporated into textual features and directly harm the multimodal reasoning process. This problem arises from inadequately detailed and accurate descriptions of visual entities during multimodal interaction, leading to the model capturing incorrect entity relations. As shown in Figure 1 , to infer the correct relation between "Meghan Markle" and "Harry", previous methods may capture the wrong person due to the lack of detailed descriptions of "Meghan Markle" and "Harry", which may affect the selection of model results.ModelCritical information loss issues. The multimodal reasoning process needs to eliminate the influence of irrelevant image objects and improve the efficiency of relation extraction [6] . However, current methods [3, 6, 33, 35, 61] often overlook the preservation of critical information during multimodal reasoning, potentially leading to the loss of critical information during the suppression of redundant and noisy data.In our paper, we propose a Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM) to achieve the fine-grained multimodal alignment between images and text. Specifically, to tackle error sensitivity issues, the Caption-Aware Module introduces detailed image captions through Multimodal Large Language Model (MLLM) [4] , which has the instruction-following ability and the vision-understanding ability in vision and language. Subsequently, we design a crossattention module to achieve fine-grained multimodal alignment from sentence level and word level. Meanwhile, to tackle the loss of crucial information during multimodal reasoning, we introduce a Mutual Information Maximization method, which is often used as a measure of the correlation between two random variables in information theory. This method preserves task-related crucial information during the multimodal reasoning and supervises different features respectively to ensure the integrity of crucial information. Overall, we summarize the main contributions as follows:1. We propose a Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM). The CAMIM introduces detailed image captions for fine-grained multimodal alignment.2. We introduce the idea of Mutual Information to preserve crucial information during multimodal reasoning. To the best of our knowledge, this paper is the first work to introduce mutual information into multimodal relation extraction.3. We evaluate our methods on the MRE dataset and demonstrate their superiority compared to previous state-of-the-art baselines and pluggability. 
related work 2.1 multimodal relation extraction: 
 Relation Extraction (RE) [7, 31, 58] has gained much attention, recently. Previous studies mainly focus on extracting relations from single text modality [21] [22] [23] . Because visual features from images could provide more clues for reasoning, Multimodal Relation Extraction has been proposed and gained more attention. Recently, several studies on multimodal relation extraction aim to utilize relevant images for extracting better relations.In the early stages, many works [3, 33, 35, 61] propose to encode the text through RNN and encode the image through CNN, then establish the implicit interaction between two modalities. Yu et al. [55] , Zhang et al. [57] propose leveraging regional-based image features to represent objects in the image, exploiting fine-grained semantic correspondences based on Transformer. Li et al. [30] propose a finegrained multimodal alignment approach with Transformer, which aligns visual and textual objects in representation space. Wang et al. [48] propose to retrieve textual evidence from the knowledge base constructed based on Wikipedia. However, most methods ignore the issue of interference from irrelevant objects in the image.For the multimodal alignment, Sun et al. [42] propose RoBERT to learn a text-image similarity score and filter out the irrelevant visual representations. Chen et al. [6] propose a visual prefix-guided fusion mechanism to remove irrelevant objects. For more fine-grained alignment, Hu et al. [18] propose entity-object and relation-image alignment pretraining tasks to improve MRE performance.Although these methods make continuous progress, due to the limited description of entities in the text, there is still a problem of being easily affected by irrelevant objects during the process of establishing associations with images, which can affect the selection of model results. Hence, we propose to introduce a detailed image caption to assist the model in completing the alignment operation and facilitate the model's reasoning ability.In information theory, Mutual Information (MI) is often used as a measure of the dependency between two random variables. Early methods, Tishby and Zaslavsky et al. [44] first propose the application of information-theoretic objectives in deep neural networks. However, at that time it may not be feasible, and variational inference provides a natural approach to approximate this problem.To narrow the gap between traditional information theory principles and deep learning, Alemi et al. [1] propose the Variational Information Bottleneck (VIB) framework, which approximates the information bottleneck (IB) constraints and enables the application of information-theoretic objectives to deep neural networks. Since then, Amjad and Geiger et al. [2] and He et al. [15] demonstrate the effectiveness of maximizing mutual information in various contexts. However, it is almost impossible to directly estimate mutual information in high-dimensional space, many studies attempt to approximate the true values using variational bounds [5, 8, 38] . Recently, Han et al. [14] propose a framework that hierarchically maximizes the Mutual Information in unimodal input pairs and between multimodal fusion results and unimodal input in the field of multimodal sentiment analysis. Due to the excellent performance of mutual information in many fields [14, 36, 40, 45] , we propose to introduce the idea of mutual information into MRE to maximize the preservation of crucial information during multimodal reasoning. 
methodology: 
 The Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM) is designed to achieve fine-grained multimodal alignment and fusion. The whole framework is shown in Figure 2 . Specifically, the Caption-Aware Module incorporates the detailed captions by Multimodal Large Language Model (MLLM) [4] to achieve multimodal alignment from word level and sentence level. In addition, Mutual Information is applied to preserve crucial information. In the following sections, we will provide a detailed description of each module.Firstly, the image contains several visual objects associated with entities in the text, which can provide more semantic knowledge to assist information extraction. Secondly, global image features may express abstract concepts and often serve as weak learning signals.Therefore, we use object-level visual data provided by Chen et al. [6] as a supplement to the global image.3.1.1 Visual Features. We leverage the visual grounding toolkit to extract local visual objects with top m salience [51, 57] , and rescale the original image and object images to 224 × 224 followed by Chen et al. [6] as the original images V and object images O.Meanwhile, we leverage the translation image data generated from textual information provided by Zheng et al. [64] , which is also processed through the visual grounding toolkit to obtain the translation original images V 𝑇 and translation object images O 𝑇 .As shown in Figure 2 , for the multimodal relation extraction task, we input the original images V and object images O into ResNet50 [16] for encoding and then obtain the original images feature 𝐹 𝑉 and object images feature 𝐹 𝑂 . We define Image Feature 𝐹 𝐼 to represent them uniformly, such as Eq.( 1):EQUATIONSimilarly, we leverage ResNet50 to encode the translation original images V 𝑇 and translation object images O 𝑇 , and then obtain the translation original images feature 𝐹 V 𝑇 and translation object images 𝐹 O 𝑇 . We define the Translation Feature 𝐹 𝑇 to represent them uniformly, such as Eq.( 2):𝐹 𝑇 = 𝐹 V 𝑇 , 𝐹 O 𝑇 .(2)3.1.2 Textual Features. Since the Translation Images are derived from textual data, we directly leverage textual data as a detailed description of the Translation Images. We believe that by concatenating with sentence-level features, the model can learn complete semantic information, thereby achieving coarse-grained alignment. Meanwhile, to learn fine-grained crucial information, we leverage word-level features which can ensure the model selects the correct information between different modalities. Hence, we leverage BERT [11] to encode the text and obtain Sentence-Level Text Feature 𝑇 𝑆 and Word-Level Text Feature 𝑇 𝑊 . However, due to the lack of detailed descriptions related to the text in the image, we further leverage a Multimodal Large Language Modal named QianWen from Alibaba Cloud and input original EQUATIONwhere 𝑀𝐿𝐿𝑀 (•) denotes the Multimodal Large Language Modal and 𝑝𝑟𝑜𝑚𝑝𝑡 denotes the information that we want to generate. Finally, we also encode caption C using BERT to obtain Sentence-Level Caption Feature 𝐶 𝑆 and Word-Level Caption Feature 𝐶 𝑊 .The main target of the Caption-Aware Module is to achieve better alignment between different modalities. We leverage a two-level alignment method, one is sentence-level feature alignment, and the other is cross-attention-based word-level feature alignment. Sentence-level features are used to align with the overall entity relations in the images, while word-level features are utilized for fine-grained alignment with visual entities.To achieve alignment between different modalities, we leverage the Multi-Layer Perceptron (MLP) to reduce the dimension of the Image Feature 𝐹 𝐼 , and then concatenate it with the Sentence-Level Caption Feature 𝐶 𝑆 to obtain the Sentence-Level Original Feature 𝑂 𝑆 :EQUATIONwhere 𝑀𝐿𝑃 denotes the MLP layer and ⊕ indicates the concatenation operation. Similarly, the Translation Feature 𝐹 𝑇 is obtained through the MLP layer, which is then concatenated with the 𝑇 𝑆 to obtain the Sentence-Level Generate Feature 𝐺 𝑆 :EQUATION)By concatenating with sentence-level features, the model learns complete semantic information and achieves coarse-grained alignment. Therefore, we leverage word-level features to learn finegrained crucial information and achieve better alignment results.Word-Level Features Alignment. We design a cross-attention fusion method for the fusion of Sentence-Level Features and Word-Level Features, respectively. As shown in Figure 3 , we first perform self-attention on the Sentence-Level Original Feature 𝑂 𝑆 and the Word-Level Caption Feature 𝐶 𝑊 separately to obtain 𝐻 1 𝑆 and 𝐻 1 𝑊 , and then use cross attention to guide the other features to obtain 𝐻 2 𝑆 and 𝐻 2 𝑊 :EQUATIONEQUATIONwhere Q 𝑙 , K 𝑙 , V 𝑙 denotes the same level feature as query/key/value.EQUATIONEQUATIONwhere 𝑚 and 𝑛 denote different level features. In 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡 (•), the first input is as query, and the second input is as key and value. Then, we concatenate the different outputs separately and process them through normalization and linear layers to obtain the final Original Feature 𝑂 𝐹 :EQUATIONwhere 𝐿𝑁 denotes the layer normalization operation and 𝑀𝐿𝑃 𝐻 denotes the MLP layer.On the other hand, We leverage the same method to calculate the Sentence-Level Generate Feature 𝐺 𝑆 and Word-Level Text Feature 𝑇 𝑊 and obtain the final Generate Feature 𝐺 𝐹 :EQUATIONEQUATION)EQUATION)We believe that fine-grained alignment between different modalities can be achieved through the Caption-Aware Module. Then, we feed the Original Feature 𝑂 𝐹 , Generate Feature 𝐺 𝐹 , and Word-Level Text Feature 𝑇 𝑊 together into the Fusion Module, ensuring that the model learns crucial task-related information from the text.We first perform cross-attention calculations on the Original Feature 𝑂 𝐹 and Word-Level Text Feature 𝑇 𝑊 to obtain the Original Fusion Feature 𝑂 𝑓 𝑖𝑛𝑎𝑙 . Then, we use the same method to calculate the Generate Feature 𝐺 𝐹 and Word-Level Text Feature 𝑇 𝑊 to obtain the Generate Fusion Feature 𝐺 𝑓 𝑖𝑛𝑎𝑙 :EQUATION𝐺 𝑓 𝑖𝑛𝑎𝑙 = 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡 (𝐺 𝑆 ,𝑇 𝑊 ) .Finally, we concatenate 𝑂 𝑓 𝑖𝑛𝑎𝑙 and 𝐺 𝑓 𝑖𝑛𝑎𝑙 through the MLP layers 𝑀𝐿𝑃 𝑓 𝑖𝑛𝑎𝑙 and obtain the final fusion feature is 𝐹 𝑓 𝑖𝑛𝑎𝑙 : EQUATION)EQUATIONEQUATION3.4.2 Mutual Information. To capture modality-invariant clues between different modalities and ensure the integrity of crucial information related to the task, we leverage MI maximization between 𝑂 𝑓 𝑖𝑛𝑎𝑙 and 𝐺 𝑓 𝑖𝑛𝑎𝑙 . Inspired by Oord et al. [37] and Han et al. [14] , we leverage a score function 𝑠 (•) that acts on the normalized prediction and truth vectors to gauge their correlation:EQUATION)EQUATIONwhere ∥•∥ 2 denotes the Euclidean Norm that we obtain unit-length vectors by dividing it because the model intends to stretch both vectors to maximize the score without this normalization. Similarly, we incorporate this score function into the Noise-Contrastive Estimation framework [13] by treating all other representations of that modality in the same batch as negative samples:EQUATIONwhere 𝐺 𝑖 𝑓 𝑖𝑛𝑎𝑙 denotes one representation of the Generate Fusion Feature 𝐺 𝑓 𝑖𝑛𝑎𝑙 in a batch and 𝐺 𝑗 𝑓 𝑖𝑛𝑎𝑙 denotes other representations of the 𝐺 𝑓 𝑖𝑛𝑎𝑙 in the same batch.We ask the Generate Fusion Feature 𝐺 𝑓 𝑖𝑛𝑎𝑙 to reversely predict representations of the Original Fusion Feature 𝑂 𝑓 𝑖𝑛𝑎𝑙 to pass more modality-invariant information to 𝐺 𝑓 𝑖𝑛𝑎𝑙 and improve the performance of the model. Finally, we obtain the loss L𝑐𝑒 between the two features.Finally, we calculate the weighted sum of all these losses to obtain the main loss for the final:EQUATIONwhere 𝛼 are hyper-parameter that control the impact of MI maximization, meanwhile, L𝑟 denotes the final loss of the relation extraction task. 
experiments: 
 In the following section, we conduct experiments to evaluate our method on a multimodal relation extraction task MRE.We evaluate the model on MNRE [65] , which contains 12,247 / 1,624 / 1,614 samples in train / dev / test sets, 9,201 images, and 23 relation types. In the case of MRE, a correct extraction of the relation between two entities occurs when the predicted relation type aligns with the gold standard. We adopt Accuracy, Precision, Recall, and F1 as the evaluation metrics. For fair comparisons, our method leverages ResNet50 [16] as the visual backbone and BERTbase [11] as the textual encoder.For the textual encoder of our CAMIM model, we leverage the BERT-Base default tokenizer with a max length of 128 to preprocess data. For the visual of our CAMIM model, we leverage ResNet50 to encode the original images and object images from Chen et al. [6] and translation images from Zheng et al. [64] .All optimizations are performed with the AdamW optimizer with a linear warmup of learning rate over the first 10% of gradient updates to a maximum value, then linear decay over the remainder of the training. And weight decay on all non-biased parameters is set to 0.01. We set the number of image objects m to 3.Remarkably, our model performs based on both HVPNeT [6] and TMR [64] , which are previous state-of-the-art methods. On the one hand, we leverage the same settings as HVPNeT, in which we fix the batch size as 32 and learning rates as 3e-5. We train the model for 30 epochs and do an evaluation after the 8th epoch. Moreover, the dimension of the hidden states d is set to 768. The prompt length and prompt dimension remain consistent with the HVPNeT settings. On the other hand, we fix the batch size as 16 and learning rates as 2e-5 which are consistent with TMR. In particular, we train the model for 8 epochs. The dimension of the hidden states d is set to 768. For both HVPNeT and TMR, we leverage the same captions and mutual information methods. Specifically, we set the caption length to 40 and uniformly use a bert-base-uncased encoding. For the calculation of mutual information, we set the hyper-parameter 𝛼 to 0.1 which is analyzed in 4.5.We compare our method with the following baselines for a comprehensive comparison. The baselines consist of two categories: Text-based RE methods that traditionally leverage merely the texts of MRE data.• BERT [11] : It is the first fine-tuning-based representation model that reduces the need for many heavily-engineered taskspecific architectures.• PCNN [56] : It devises a piecewise max pooling layer to capture structural information between different entities.• MTB [41] : It builds task-agnostic relation representations solely from the entity-linked text.Multimodal RE methods that leverage both text and image contents of MRE data.• MoRe [48] : It injects knowledge-aware information into multimodal studies using multimodal retrieval.• MEGA [65] : It employs an efficient alignment strategy for textual and visual graphs to classify textual relations more precisely.• IFAformer [30] : It proposes a method with an implicit finegrained multimodal alignment based on Transformer.• HVPNeT [6] : It treats visual representations as visual prefixes that can be inserted to guide textual representations of errorinsensitive prediction decisions.• TSVFN [63] : It combines the powerful modeling capabilities of graph neural networks and transformers networks to fully fuse critical information between visual and textual modalities.• MMIB [9] : It introduces Information Bottleneck to remove noise in different modalities and aligns multimodal data.• MRE-ISE [49] : It introduces a novel idea of simultaneous information subtraction and addition for multimodal relation extraction.• MRE [20] : It uses cross-modal retrieval for obtaining multimodal evidence to improve prediction accuracy and synthesize visual and textual information for relational reasoning.• PROMU [18] : It enables the extraction of self-supervised signals from massive unlabeled image-caption pairs to pretrain multimodal fusion modules.• TMR [64] : It implements multimodal versions of back-translation and high-resource bridging, which provide a multi-view to the misalignment between modalities.The experimental results of our CAMIM model and all baselines on the MNRE testing set are presented in Table 1 . It is easy to see our method outperforms other SOTA methods.Firstly, we can find that incorporating the visual features is generally helpful for the relation extraction task by comparing the SOTA multimodal approaches with their reliance on pure text-based baselines. Due to the short and ambiguous characteristics of texts in social media, it is difficult to identify entities and their relations in a limited context.Secondly, our model outperforms HVPNeT and TMR, which leverage hierarchical visual representations or multimodal versions of back-translation and high-resource bridging on the MRE task. The former is a typical baseline in the field of multimodal relation extraction. We add Caption-Aware Module and Mutual Information without changing any parameter settings. Specifically, we retain the method of visual prefixes and only add captions to interact with visual features during the prefix processing to achieve better alignment. At the same time, we calculate the mutual information between the final features and visual information to ensure the model preserves crucial information related to the task. As shown in Table 1 , compared with the results of HVPNeT, the F1 score increases from 81.85% to 84.66%. Similarly, we further add the Caption-Aware Module and Mutual Information to the newest SOTA method TMR. We retain the backtranslation data processing and align the caption with the original images. At the same time, since the translation images are derived from textual data, we leverage text as the caption of the translation images. Additionally, to ensure the crucial information between the original images and the translation images remains unchanged, we leverage mutual information calculation for them. As shown in Table 1 , compared with the results of TMR, the F1 score still increases from 89.05% to 90.94%.Therefore, the results indicate that our method, which introduces detailed captions of the whole image, and mutual information that can capture modality-invariant clues among modalities, is helpful for relation extraction tasks and can achieve stable and excellent results. Note that our method is pluggable and can be applied to other MRE models.In this section, we conduct extensive experiments with the variants of our model to analyze the effectiveness of each component. As shown in Table 1 , our baseline methods contain HVPNeT and TMR. Hence, we ablate our Caption-Aware Module and Mutual Information on both of them. 4.5.1 Captions generated by different MLLMs. We test different captions generated by the six latest MLLMs (LLaVA-NeXT [28] , QWenVL-Plus [4] , CogVLM2 [47] , MiniCPM V2.5 [17] , among them, CogVLM2 and MiniCPM V2.5 are open-source multimodal models at the GPT-4V level.) and VLMs (BLIP-2 [29] , Instruct-BLIP [10] ). We invite graduate students with strong English proficiency to check the accuracy of the generated captions in the test set.1)The accuracy of the generated captions is shown in Table .2. Due to the poorer language ability of BLIP-2, the accuracy of the generated caption is lower, and there are often some grammar errors. In MLLMs, the most common mistakes are hallucinatory problems. Additionally, the model with the captions generated by MLLMs performs better than those generated by VLMs because of the more detailed captions. Even with a few hallucination mistakes, our model could select the most relevant information related to the head and tail entities.2)The performance of MLLMs in our models. The captions generated by QwenVL-Plus are of appropriate length and contain substantial content, which has already achieved a notably good effect. 4 (b), we conduct ablation studies of caption alignment methods on HVPNeT. We can intuitively observe that the method of aligning captions using both word-level features and sentence-level features simultaneously gains better results. However, using any level of feature alone does not gain a satisfactory effect. Therefore, it can be illustrated that both wordlevel features and sentence-level features contain crucial semantic information and can assist the model in reasoning.In Table 1 , the results indicate that the introduction of mutual information can stably improve the performance on two baselines.Furthermore, as shown in Figure 4 (a), we conduct ablation studies of mutual information based on TMR (O&G denotes mutual information calculation between the Original features and Generate features, (O⊕G)&F: Visual concatenation result and Final fusion feature, T&F: Text feature and Final fusion feature, (O⊕G)&T: Visual concatenation result and Text feature.). We can observe significant differences in the calculation results of mutual information between different features. The MI result between original features and generated features is the most obvious which means that there may lose some critical information during reasoning. The poor performance of mutual information between different modalities suggests that visual and textual modalities contain different-level semantic information, highlighting the need for joint reasoning after alignment.Summarily, mutual information can assist the model in capturing modality-invariant clues between different modalities, and the more significant the semantic information differences contained in modalities, the better the ability of the model to capture them. Finally, as shown in Figure 4 (c), we conduct ablation studies of hyperparameter 𝛼 in {0.05, 0.1, 0.3, 0.5}, which is used in mutual information calculation. We can easily observe that the best effect is achieved when the value is 0.1.We conduct a case study of the Caption-Aware Module (CAM) and Mutual Information Maximization (MI) with HVPNeT, as shown in Figure .5.In case 1, it is difficult for the model to understand the exacted relation between the head and tail entities, and it is easy to find the correct relation under the guidance of the given caption.In case 2, the model struggles to identify Dematha's attributes and picks the wrong relation without CAM. The detailed caption, specifically "Mr. Pham, a school maintenance worker at St. Matha's school" guides the model to the right relation. Yet, irrelevant details abound. Lacking MI, the model may get distracted by these irrelevant features. Here, the excessive indoor setting in the caption could mislead the model to choose "place-of-residence" without MI to preserve critical information within different modalities. Concisely, CAM ensures multi-grained alignment as the model might not comprehend all entities. Meanwhile, MI is necessary to retain critical information across modalities since not every caption sentence is relevant. 
conclusions: 
 In this paper, to solve the error sensitivity issues and the critical information loss issues, we propose a Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM). The Caption-Aware Module introduces detailed image captions generated from the Multimodal Large Language Model (MLLM) to achieve fine-grained alignment between visual and textual information. The Mutual Information Maximization method is designed to preserve crucial information between different modalities. Experiments show that our CAMIM model outperforms the state-of-the-art multimodal relation extraction models. 
 
 Appendix:  


Decision: This paper proposes a Caption-Aware Multimodal Relation Extraction Network with the mutual information maximization technique. They first generate detailed image captions using LLM for fine-grained multimodal alignment between visual entities and textual entities. The mutual information maximization technique is designed to preserve crucial information between different modalities. Experiments on the MNRE dataset validate the effectivess of the method. 

(+) On the positive side, the reviewers found the method to be novel and effective and appreciate the clear motivation and good results.

(-) On the negative side, there are still some concerns on the quality of generated captions, insufficient discussion of mutual information maximization, insufficient evaluation on just one datasets and weak presentation.

Several technical questions and suggestions were raised by the reviewers. The authors have taken these into consideration. Overall, all reviewers finally agreed to accept the paper after the rebuttal. Therefore, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach to Multimodal Relation Extraction (MRE). The proposed CAMIM model addresses critical issues in MRE, such as error sensitivity and critical information loss, by incorporating detailed image captions and employing Mutual Information Maximization. The model utilizes a Caption-Aware Module (CAM) to align fine-grained visual and textual entities and ensures the preservation of crucial information through mutual information techniques. Experiments on the MNRE benchmark dataset show that CAMIM outperforms state-of-the-art models, and ablation studies confirm the effectiveness and pluggability of its components.
strength_weakness: strength_weakness: (1) Introducing detailed image captions generated by a Multimodal Large Language Model (MLLM) for fine-grained multimodal alignment is a novel approach. This method effectively reduces error sensitivity by minimizing the influence of irrelevant objects.

(2) The use of Mutual Information Maximization to preserve crucial information during multimodal reasoning is well-motivated and theoretically sound. 

(3) The paper provides extensive experiments demonstrating the superior performance of CAMIM over existing state-of-the-art models on the MNRE dataset. The results are backed by quantitative metrics.

(4) Ablation studies are conducted to validate the individual contributions of the Caption-Aware Module and mutual information techniques, proving their effectiveness and pluggability
limitations: limitations: (1) The effectiveness of the CAMIM model heavily relies on the quality of the generated captions. The paper assumes that the MLLM can consistently produce accurate and detailed captions, but it does not address potential issues arising from incorrect or ambiguous captions. An analysis of the quality of generated captions is urgently required.

(2) Lacks details on the mutual information maximization method and how it benefits the overall performance of the proposed architecture. A case study or visualization may strengthen the contribution of this module.
paper_summary: paper_summary: This paper introduces a method named "Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization" (CAMIM), which is tailored to mitigate the challenges of error sensitivity and loss of key information in multimodal relation extraction (MRE). CAMIM harnesses the sophisticated image captions generated by Multimodal Large Language Models (MLLMs) **and incorporates a mutual information maximization approach to facilitate granular alignment and inferencing between visual and textual modalities**. Experimental evidence showcases CAMIM's supremacy over the extant avant-garde MRE models on the MNRE benchmark dataset, with subsequent ablation studies validating the efficacy of its caption-aware component and mutual information maximization strategy.
strength_weakness: strength_weakness: *   This paper conducts valuable research on the impact of irrelevant targets and redundant and noisy information on RE tasks in the context of MRE, making it a valuable research point.
*   The paper demonstrates clear logic and smooth writing.
*   The experimental section of the paper provides comprehensive comparisons with baselines, yielding results that are competitively strong.
limitations: limitations: - The novelty introduced in the paper seems an accumulation of efforts, with enhancing modality alignment through multi-granularity image-text alignment for better downstream task representation always being a focal point in works such as VLMs.
- Where does the advantage of captions generated by LLMs lie over those produced by traditional VLMs? The paper's reference to what it calls MLLM, possibly Qwen-VL, seems to only be described in section 3.1.2, which is rather confusing. Has the author compared the captioning capabilities against currently popular open-source MLLMs? The absence of solid experiments and analysis in this area leaves an impression of trend-chasing.
- In section 3.4, the author employs the concept of Mutual Information Maximization for loss constraint during the classification stage, yet its contribution appears unclear in the ablation study. Claiming this to be the first time mutual information has been introduced to multimodal relation extraction is hard to swallow, as contrastive learning has been extensively researched in MRE and mNER, essentially embodying the core idea of MIM.
paper_summary: paper_summary: This paper presents a multimodal relation extraction network that addresses error sensitivity and critical information loss issues. The proposed approach combines a caption-aware module with a mutual information maximization method. Experimental evaluation shows the superiority of the proposed approach compared to previous state-of-the-art methods.
strength_weakness: strength_weakness: 1. The primary strength of the method lies in its performance. Experimental evaluation demonstrates its superiority compared to other state-of-the-art methods.

2. The proposed method improves the error sensitivity issues and the critical information loss issues existing in the Multimodal Relation Extraction task.

3. The provided ablation results convincingly showcase the effectiveness of the proposed approach, specifically the caption-aware module and mutual information.
limitations: limitations: 1. The case analysis lacks the individual application of the Caption-Aware Module and Mutual Information, resulting in changes.

2. The impact of inappropriate translation image data generated from textual information on the model's performance.

3. In the section 4.2, does "do an evaluation after the 8th epoch" imply selecting the final model based on its performance on the test set?