Shapley Value-based Contrastive Alignment for  Multimodal Information Extraction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 17:25:26

Abstract: The rise of social media and the exponential growth of multimodal communication necessitates advanced techniques for Multimodal Information Extraction (MIE). However, existing methodologies primarily rely on direct Image-Text interactions, a paradigm that often face the significant challenges due to semantic and modality gaps between images and text. In this paper, we introduce a new paradigm of Image-Context-Text interaction, where large multimodal models (LMMs) are utilized to generate descriptive textual context to bridge these gaps. In line with this paradigm, we propose a novel Shapley Value-based Contrastive Alignment (Shap-CA) method, which aligns both context-text and context-image pairs. Shap-CA initially applies the Shapley value concept from cooperative game theory to assess the individual contribution of each element in the set of contexts, texts and images towards total semantic and modality overlaps. Following this quantitative evaluation, a contrastive learning strategy is employed to enhance the interactive contribution within context-text/image pairs, while minimizing the influence across these pairs. Furthermore, we design an adaptive fusion module for selective cross-modal fusion. Extensive experiments across four MIE datasets demonstrate that our method significantly outperforms existing state-of-the-art methods. Code will be released upon acceptance.

Main paper: 
 introduction: 
 The exponential growth of social media platforms has initiated a new phase of communication, characterized by the exchange of multimodal data, primarily texts and images. This diverse landscape necessitates advanced techniques for multimodal information extraction (MIE) [10, 39, 45] , which primarily aims to utilize auxiliary image inputs to enhance the performance of identifying entities or relations within the unstructured text. To the best of our knowledge, the majority of previous methods of MIE mainly concentrate on the direct interaction between images and text. These approaches either (1) encode images directly and employ efficient attention mechanisms to facilitate image-text interactions [4, 30, 34, 37, 41, 43, 44] , or (2) detect finer-grained visual objects within images and use Graph Neural Networks or attention mechanisms to establish interactions between text and objects [9-11, 17, 18, 42, 45, 46, 51, 53] . Despite the advancements made by these methods, this direct Image-Text interaction paradigm still suffers from the simultaneous existence of semantic and modality gaps. The semantic gap refers to the disparity between the meaning conveyed by the text and the actual content depicted by the image. For instance, in Figure 1 , the word "Rocky" could refer to the dog in the image but could also be a name for a cat or a person. Furthermore, even the direct alignment of the word "dog" with an image of a dog remains a challenge due to the representation inconsistency, resulting in a modality gap. The presence of these two gaps weakens the connection between images and text, potentially leading to erroneous predictions of entities or relations.To mitigate the semantic and modality gaps, we propose an approach that relies on an intermediary, rather than direct interactions between texts and images. Given that text is the primary source of information extraction and images serve a supporting role, we suggest the use of a textual intermediary to bridge these gaps. As large multimodal models (LMMs) demonstrate impressive results on instruction-following and visual comprehension [26, 27] , and excel at generating descriptive textual context for images, we employ the powerful generative ability of LMMs to create an intermediate context for reducing the burden of aligning image and text directly. As shown in Figure 1 , with the divide-and-conquer strategy, we only need to model the connections between the generated context and the original text or image, forming an Image-Context-Text interaction paradigm.The core challenge now is aligning the context-text pairs and context-image pairs in the hidden feature space, respectively, where the alignment with text bridges the semantic gap, and the alignment with images addresses the modality gap. In this scenario, contexts, texts, and images can be considered as three kinds of elements in the hidden space trying to achieve the optimal collective semantic and modality overlaps. Drawing inspiration from cooperative game theory, we adapt the concept of the Shapley value [14] , which provides a equitable mechanism for assessing each individual's contribution towards collective utility. In this paper, we propose a novel Shapley Value-based Contrastive Alignment (Shap-CA) method, which leverage a contrastive learning strategy to construct coherent and compatible representations for these three elements based on their Shapley values. Within this framework, each element is considered as a player, and the collective utility of a set of elements is defined as the total semantic or modality overlap. Shap-CA initially calculates the average marginal contribution of each player to the collective utility to estimate their Shapley value. Intuitively, contexts contributing significantly to the overall overlap (i.e., having a larger Shapley value) should have a higher probability of forming a true pair with the text or image. Consequently, a contrastive learning strategy is then employed to enhance the interactive contribution within context-text/image pairs, while minimizing the influence across these pairs. Through this process, Shap-CA not only strengthens the intrinsic connections within each pair but also accentuates the disparities between different pairs, effectively bridging the semantic and modality gaps. Furthermore, we design an adaptive fusion module to obtain the informative fused features across modalities. This module assesses the relevance of each modal feature to the bridging context, strategically weighting their importance to achieve a finer-grained selective cross-modal fusion. Finally, a linear-chain CRF [22] or a word-pair contrastive layer is employed for prediction.Overall, the main contributions of this paper can be summarized as follows:• We are the first to introduce the Image-Context-  
related work: 
 Multimodal Information Extraction Multimodal Information Extraction (MIE) is an evolving research domain primarily aimed at enhancing the recognition of entities and relations by utilizing supplemental image inputs. This field can be primarily subdivided into three critical tasks: Multimodal Named Entity Recognition (MNER), Multimodal Relation Extraction (MRE) and Multimodal Joint Entity-Relation Extraction (MJERE). In particular, the tasks of MNER [37, 39] and MRE [10, 51] are concerned with identifying entities and relations separately, while the MJERE task [45] aims to extract entities and their associated relations jointly. Most of existing methods are concentrated around the paradigm of direct Image-Text interaction [9, 30, 34, 37, 42-45, 51, 53] . Wang et al. [41] designed a fine-grained cross-modal attention module to enhance the cross-modal alignment. Sun et al. [37] , Xu et al. [43] , and Liu et al. [28] focus on quantifying and controlling the influence of images on texts through gate mechanisms or text-image relationship inference. To facilitate the alignment between visual objects and text, Zhao et al. [50] and Yuan et al. [45] propose a heterogeneous graph network and an edge-enhanced graph neural network, respectively. Additionally, there are some other studies [18, 39] that employ external knowledge, such as machine reading comprehension [18] , to foster model reasoning. Regardless of these substantial advancements, these approaches still fail to address the potential dual gaps existing between images and texts due to the representation inconsistencies. In this study, we aim to comprehensively consider these semantic and modality gaps and propose Shap-CA, which leverages the bridging context to mitigate these gaps and achieve a more coherent and compatible representation.Large Multimodal Models Large multimodal models have recently gained substantial traction in the research community [2, 23] . Similar to the trend observed with large language models, several studies have indicated that scaling up the training data [6, 12, 49] or model size [6, 31] can significantly enhance these large multimodal models' capabilities. Moreover, visual instruction tuning can equip large multimodal models with excellent instruction-following, visual understanding, and natural language generation abilities [27] . This advancement empowers these models to excel in interpreting images according to instructions and generating informative textual contexts. However, their open-ended generative characteristics have resulted in less than satisfactory performance when directly applied to information extraction tasks [38] . 
methodology 3.1 task definition: 
 Given an input text 𝒕 = {𝑡 1 , . . . , 𝑡 𝑛 𝑡 } with 𝑛 𝑡 tokens and its attached image 𝑰 , our method aims to predict the output entity/relation labels 𝒚. The format of the labels 𝒚 is task-dependent. Specifically, for MNER, they are sequential labels. For MRE and MJERE, they are word-pair labels [45] .The architecture of Shap-CA is shown in Figure 2 . Initially, we utilize a LMM, e.g., LLaVA-1.5 [26] , to extract the textual bridging context from the image. Subsequently, we employ a pretrained textual transformer to extract features from the text and the context. In parallel, an image encoder is used to derive visual features from the image. Following this, we apply a Shapley value-based contrastive alignment to construct more coherent representations. The adaptive fusion module is then employed to obtain the informative features across modalities. Finally, these comprehensive representations are fed into a CRF or a word-pair contrastive layer for prediction. Context Generation Given an image 𝑰 , we utilize a pretrained LMM to generate a textual context as the bridging context 𝒄 = {𝑐 1 , . . . , 𝑐 𝑛 𝑐 } with 𝑛 𝑐 tokens.Textual and Visual Encoding In order to acquire contextualized textual features, we concatenate the input text 𝒕 with the bridging context 𝒄. Following this, we employ a pretrained textual transformer to extract both sentence-level and token-level features:EQUATIONwhere 𝒙 𝒕 , 𝒙 𝒄 ∈ R 𝑑 represent the sentence-level features, while 𝑯 𝒕 ∈ R 𝑛 𝑡 ×𝑑 , 𝑯 𝒄 ∈ R 𝑛 𝑐 ×𝑑 denote the token-level features of the input text and context, respectively. Simultaneously, we employ an image encoder to extract the visual features of the image 𝑰 :EQUATIONwhere 𝒙 𝒗 , 𝑯 𝒗 denotes the global visual feature and the regional visual features of the image, respectively.To construct more coherent representations, we leverage the Shapley value to perform both context-text alignment and context-image alignment. The Shapley value offers a solution for the equitable allocation of total utility among players based on their individual marginal contributions, and has widespread application [15, 16, 20] . We are the first to utilize the Shapley value for multimodal alignment through contrastive learning.In the context of a cooperative game, suppose we have 𝑘 players, represented by 𝑲 = {1, . . . , 𝑘 }, and a utility function 𝒖 : 2 𝑘 → R that assigns a reward to each coalition (subset) of players. The Shapley value of player 𝑖 is then defined as [19] :EQUATIONThe Shapley value essentially quantifies the average marginal contribution of a player to all potential coalitions (subsets). We detail the context-text alignment as an example as follows.Shapley Value Approximation. In the context-text alignment, inputs are a mini-batch of 𝑘 context-text pairs {(𝒙 𝒄 𝑎 , 𝒙 𝒕 𝑎 )} 𝑘 𝑎=1 . Here, we view the 𝑘 bridging contexts as players, denoted as 𝑲 = {1, . . . , 𝑘 } for simplicity. These players, or contexts, collaboratively contribute to the semantic comprehension of a specific text feature. Consider the 𝑗-th pooled text feature, 𝒙 𝒕 𝑗 , and a selected subset of context players, denoted as a coalition 𝑺 ⊆ 𝑲 . The central idea is based on an assumption: if all the contexts within the subset 𝑺 and the text 𝒙 𝒕 𝑗 form positive pairs, the utility of 𝑺 for 𝒙 𝒕 𝑗 would be represented by the expected semantic overlap between them. This utility captures the collective semantic relationships between the text and the contexts within the coalition, as formalized by:EQUATIONHere, sim(𝒙 𝒕 𝑗 , 𝒙 𝒄 𝑖 ) denotes the semantic overlap between the text and each context (i.e., individual semantic contribution), measured by cosine similarity. The weight 𝑝 𝑖 , computed through a softmax operation with a temperature of 𝜏, models the cooperative behavior among different contexts by normalizing these individual semantic contribution. This approach intuitively suggests that the stronger the semantic overlap a context shares with the text (i.e., the larger the semantic contribution), the more likely it is to form a true pair with the text in real-world situations. From this perspective, the utility of the coalition can be interpreted as an expectation over the semantic overlaps of each context within 𝑺 with the text, where the expectation weights are given by the likelihood of each context-text pair being positive. This method naturally prioritizes contexts that have a higher degree of semantic overlap with the text, thereby refining the overall semantic understanding.However, as indicated by Eqn. 3, the computation of the Shapley value requires an exponentially large number of computations relative to the size of the mini-batch, which poses a challenge during training. To address this, we extend Monte-Carlo approximation methods [8, 33] to our training setting for estimating the Shapley value. We present the detailed algorithm in Alg. 1. It begins with a random permutation of 𝑘 context players and an initial stride of 𝑠, which is set to batch_size/2. At each iteration, the algorithm scans each player in the current permutation and calculates the marginal contribution when the player is added to the coalition formed by the preceding players. This marginal contribution serves as an unbiased estimate of the Shapley value for that player, improving with each iteration. Subsequently, the algorithm updates the permutation through a cyclic shift of the current stride and halve the stride for the next iteration. This gradual reduction in stride results in more stable marginal contributions over time, as the size of the subsets for each player changes less dramatically on average. The process continues until the stride falls to zero. 𝑠 = 𝑠/2 ; end Contrastive Learning. After acquiring the approximated Shapley value { φ1 (𝒖 𝑗 ), . . . , φ𝑘 (𝒖 𝑗 )}, we introduce a context-to-text contrastive loss that aims to maximize the average marginal semantic contribution from each context player to the text with which it forms a true positive pair (i.e., the text from the same pair), while simultaneously minimizing the contributions between the true negative pairs (i.e., all other texts not paired with this context).EQUATIONIn a symmetrical manner, we can treat the 𝑘 pooled text as players, and derive the text-to-context contrastive loss L 𝑡 2𝑐 . The semantic loss L 𝑠𝑒𝑚𝑎𝑛𝑡𝑖𝑐 is the average of these two contrastive losses: EQUATIONEQUATIONTo facilitate a finer-grained fusion across modalities, we develop an adaptive attention fusion module. This module dynamically weights the importance of different features in two modalities, based on their relevance to the context that connects them. Given the representations 𝑯 𝒕 , 𝑯 𝒄 , 𝑯 𝒗 obtained from Section 3.3, we initially employ linear projection to transform them into a set of matrices: a query matrix 𝑸 𝒗 ∈ R 𝑛 𝑣 ×𝐷 , a key matrix 𝑲 𝒕 ∈ R 𝑛 𝑡 ×𝐷 , a value matrix 𝑽 𝒕 ∈ R 𝑛 𝑡 ×𝐷 , and a context matrix 𝑪 ∈ R 𝑛 𝑐 ×𝐷 . Subsequently, we calculate the bridging term 𝑩 as follows:EQUATIONThis bridging term is then employed to dynamically modify the content of queries 𝑸 𝒗 and keys 𝑲 𝒕 based on their relevance to the bridging term:EQUATIONwhere ⊙ represents the Hadamard product, 𝒈 𝑞 ∈ R 𝑛 𝑣 ×𝐷 and 𝒈 𝑘 ∈ R 𝑛 𝑡 ×𝐷 are gating vectors that are used to capture the relevance between the text (or image) and the bridging context:EQUATIONFollowing this, we acquire the image-awared text features 𝑴 𝑡 ∈ R 𝑛 𝑡 ×𝐷 using the gated cross-attention:EQUATIONIn addition to the image-awared text features derived from Eqn. 11, we also incorporate each token's part-of-speech embeddings 𝑴 𝑝𝑜𝑠 ∈ R 𝑛 𝑡 ×𝑑 1 , and positional embeddings 𝑴 𝑝 ∈ R 𝑛 𝑡 ×𝑑 1 , to enrich the information available for decoding. For MNER, a widely used CRF layer [1, 21] is employed to predict label sequences. For MRE and MJERE, we propose a word-pair contrastive layer to predict word-pair labels [45] .CRF layer Initially, we employ a multi-layer perceptron to integrate the three channels of features:𝑴 = MLP 1 (𝑴 𝑡 ; 𝑴 𝑝𝑜𝑠 ; 𝑴 𝑝 ) ∈ R 𝑛 𝑡 ×𝑑 2 (12)Subsequently, we feed 𝑴 = {𝑴 1 , . . . , 𝑴 𝑛 𝑡 } into a standard CRF layer to derive the final distribution 𝑃 (𝒚|𝒕). The training loss for the input sequence 𝒕 with gold labels 𝒚 * is measured using the negative log-likelihood (NLL):EQUATIONWord-Pair Contrastive layer For each word pair (𝑡 𝑖 , 𝑡 𝑗 ), we initially employ three multi-layer perceptrons to separately obtain the three channels of pair-wise features:EQUATION)where The final distribution is refined by contrasting the two distributions:EQUATIONwhere 𝜆 is the refinement scale. We use the cross-entropy loss for the input sequence 𝒕 and the gold word-pair labels 𝒚 * :EQUATIONIn summary, our framework incorporates two self-supervised learning tasks and one supervised learning task, resulting in three loss functions. Considering the semantic loss L 𝑠𝑒𝑚𝑎𝑛𝑡𝑖𝑐 , the modality loss L 𝑚𝑜𝑑𝑎𝑙𝑖𝑡 𝑦 from Sec. 3.4, and the prediction loss L 𝑚𝑎𝑖𝑛 from Sec. 3.6, the final loss function is defined as follows:EQUATIONwhere 𝛼, 𝛽 are hyperparameters. 
experiments: 
 We conduct our experiments on four MIE datasets and compare our method with a number of previous approaches.Experiments are conducted on 2 MNER datasets, 1 MRE dataset and 1 MJERE dataset: Twitter-15 [48] and Twitter-17 [44] for MNER 1 , MNRE 2 [52] for MRE, and MJERE 3 [45] for MJERE, noting that the last dataset and the task share the same name. Statistical details of these datasets are shown in Table 1 . Model Configuration In order to fairly compare with the previous approaches [13, 17, 40-43, 45, 46, 51] , we use BERT-based model with the dimension of 768 as the textual encoder. For visual encoder, we experiment with the ViTB/32 in CLIP 4 and ResNet152 models as potential alternatives. We find ResNet152 with the dimension of 2048 to provide the most effective and consistent results in our settings. To generate the texual descriptive contexts of images, we use LLaVA-1.5 [26] , a newly proposed visual instruction-tuned, large multimodal model 5 . The prompt we employ is shown in Table 2 and the sampling temperature is set to 1.0. We apply the spaCy 6library of the en_core_web_sm version to parse the given sentence and obtain each word's part of speech. The dimensions of positional embeddings and part-of-speech embeddings are 100.Training Configuration Shap-CA is trained by Pytorch on single NVIDIA RTX 3090 GPU. During training, the model is finetuned by AdamW [29] optimizer with a warmup cosine scheduler of ratio 0.05 and a batch size of 28. We use the grid search to find the learning rate over [1 × 10 -5 , 5 × 10 -4 ]. The learning rate of encoders is set to 5 × 10 -5 . The learning rates of other modules are set to 3 × 10 -4 , 1 × 10 -4 , 1 × 10 -4 and 1 × 10 -4 for Twitter-15, Twitter-17, MJERE and MNRE, respectively. The refinement scale 𝜆 in Eqn. 17 is set to 1.0. An early stopping strategy is applied to determine the number of training epochs with a patience of 7. We choose the model performing the best on the validation set and evaluate it on the test set. We report the average results from 3 runs with different random seeds.We compare Shap-CA with previous state-of-the-art methods, which primarily fall into two categories. The first group of methods only consider text input, including BERT-CRF [13] , MTB [7] , ChatGPT and GPT-4. Secondly, we consider several latest multimodal methods, including OCSGA [42] , UMGF [46] , MEGA [51] , MAF [43] , ITA [40] , CAT-MNER [41] , MNER-QG [17] , EEGA [45] , and MQA [38] . Table 3 provides a comprehensive comparison of our proposed method with baseline approaches across various modalities, including text-based methods and previous state-of-the-art multimodal methods. Through this comparative analysis, we get several noteworthy findings as follows: (1) Superior Performance of Shap-CA: Our method, Shap-CA, demonstrates significant superiority across all datasets compared to baseline approaches. Notably, in tasks involving entity or relation extraction, such as Twitter-15, Twitter-17, MJERE e , and MNRE, Shap-CA consistently outperforms the best competitor by substantial margins of 1.33%, 1.07%, 0.7%, and 1.05% in terms of F1 scores, respectively. These results underscore the effectiveness of our proposed Image-Context-Text paradigm in enhancing information extraction tasks. (2) Effective Handling of Complex Tasks: In the most challenging task, MJERE j , which requires simultaneous extraction of entities and their associated relations, Shap-CA achieves the highest F1 score of 55.95%. This outcome highlights Shap-CA's efficacy in managing complex MIE scenarios, where accurate identification of both entities and relations is crucial. (3) Remarkable Adaptability: In contrast to other models that are typically specialized in one or two specific tasks, Shap-CA demonstrates exceptional adaptability by consistently delivering state-of-the-art performance across a diverse range of tasks, which underscores the model's ability to adapt to various environments and tasks, highlighting its flexibility in managing different information extraction challenges. (4) Enhanced Performance with Visual Information: Incorporating visual information from images generally enhances the performance of MIE when comparing state-of-the-art multimodal methods to their text-only counterparts. This observation suggests the integration of visual information provides a more holistic understanding of the input, underlining its value of improving the accuracy of information extraction tasks, particularly in scenarios where text contents alone may be insufficient for prediction. (5) Performance of Large Models: Contrary to the common belief that larger models possess superior generalization abilities, our experimental results show that methods based on LLMs (e.g., ChatGPT, GPT-4) and LMMs (e.g., MQA) perform less well than our proposed method and other previous multimodal approaches. This suggests that the current large models, despite their extensive capabilities, might not yet be fully optimized for information extraction tasks. This potential discrepancy could arise from their open-ended generative nature and pre-training scenarios that are misaligned with the specifics of MIE. 
discussion and analysis 5.1 ablation study: 
 We conduct a comprehensive ablation study to further analyze the effectiveness of our method. The results of these model variants are presented in Table 3 [36] , a widely used approach in the field of contrastive learning [5, 32, 35] . As shown in Table 3 , Shap-CA apparently outperforms the variant without the Shapley Value on all datasets, which validates our proposed alignment method's ability to construct more coherent multimodal representations for MIE.Effectiveness of the Word-Pair Contrastive Layer We further delve into the impact of our proposed word-pair contrastive layer by evaluating the performance of Shap-CA w/o word-pair Contrastive.The results show that contrasting the distributions before and after the integration of the three channels can effectively enhance the model's predictive power. Contribution of Images Since the context of Shap-CA is generated from images, a natural question that arises is whether the context can completely replace the original image. In Table 2 , we compare Shap-CA with its image-removed variant and the baseline on the MJERE dataset. The baseline approach simply concatenates the text and context before feeding it into a BERT-base model.Experimental results demonstrate that while the textual context information derived from the images is beneficial, it cannot fully replace the contribution of the images themselves to the model's performance.How Alignments Affect Performance In this section, we further discuss the influence of alignments on our model's performance on the MJERE dataset. Specifically, we examine two key coefficients, 𝛼 and 𝛽 in Eqn. 19, which represent context-text alignment and context-image alignment, respectively. As shown in Figure 3 , the model's performance exhibits a consistent pattern of variation in response to changes in 𝛼 and 𝛽. The red line draws the performance trend of changing 𝛼 when fixing 𝛽 as 0.4. 𝛼 = 0 means w/o contexttext alignment, resulting in the poorest performance due to the inability to effectively bridge the semantic gap. When 𝛼 = 0.2, the model reaches its optimal performance. However, when 𝛼 exceeds 0.2, the alignment task may interfere with the main task, leading to declining performance. The blue line draws the performance trend of 𝛽 with the fixed value of 𝛼 (i.e., 0.2), which is similar to that of 𝛼. The peak performance is reached when 𝛽 is 0.4.In practical situations, robustness to a range of contexts from different generators is crucial, as these contexts may vary significantly in quality. To investigate this, we conduct experiments to assess the impact of employing various context generators. Specifically, we examine the results produced by five different context generators when applied to the MJERE dataset. These generators include traditional image captioning models: BU [3] and VinVL [47] , and newly developed large multimodal models: BLIP-2 [24] , LLaVA-1.5 [26] , and GPT-4. As shown in Table 5 , the performance disparity across these context generators is minimal. Notably, the use of GPT-4 as a context generator achieves the best overall performance. This observation implies that our method is robust to the variability inherent in different context generators, maintaining consistent performance regardless of the quality of the generated context.The scale of a model is a critical aspect in real-world applications. We present a comparative analysis of the total parameters and performance of our proposed method, Shap-CA, alongside other existing state-of-the-art multimodal approaches, whose results are achieved by their official implementations. As shown in Table 6 , Shap-CA, with a parameter count of only 170.7M, is the most lightweight model among those evaluated, suggesting a reduced computational burden. Generalization Table 7 presents a comparison of our method's generalization ability against several previous state-of-the-art approaches. This experiment is implemented by training on the source dataset while testing on the target dataset. For instance, T-17→T-15 denotes that the model is trained on the Twitter-17 dataset and then tested on the Twitter-15 dataset. As the results shown, our method Case Study In Figure 4 , we present two cases that highlight the effectiveness of our proposed method in bridging both semantic and modality gaps, challenging for EEGA [45] which follows a direct Image-Text interaction paradigm. The first case discusses Stephen Curry's achievements in the NBA. Although EEGA recognizes the player, it incorrectly extracts "NBA" as an entity, leading to an inaccurate "present_in" relation between "Stephen Curry" and "NBA". In contrast, our method, utilizing the context featuring a "trophy" and "celebration", makes an accurate prediction, which denotes the context is helpful to mitigating the semantic and modality gaps.The second case is more nuanced, involving a scene with Elizabeth Taylor and George Stevens. Though EEGA can identify both persons, it fails to infer the intimate relationship implied by the image and text, predicting a mere "peer" relation. However, our method, leveraging the context that emphasizes "intimacy and connection", accurately predicts the relation "couple" between "Elizabeth Taylor" and "George Stevens". 
conclusion: 
 In this paper, we introduce a novel paradigm of Image-Context-Text interaction to address the challenges posed by the semantic and modality gaps in conventional MIE approaches. In line with this paradigm, we propose a novel method, Shapley Value-Based Contrastive Alignment (Shap-CA), which performs both context-text and context-image alignments. Shap-CA first employs the Shapley value to access the individual contributions of contexts, texts, and images to the overall semantic or modality overlaps, and then applies a contrastive learning strategy to perform the alignment by maximizing the interactive contribution within context-text/image pairs and minimizing the influence across these pairs. Furthermore, Shap-CA incorporates an adaptive fusion module for selective crossmodal fusion. Our experiments demonstrate that Shap-CA significantly outperforms previous state-of-the-art approaches across four MIE datasets. In our analysis, we demonstrate Shap-CA's robust performance in the face of varying context quality, its high model efficiency and strong generalization ability.  
 
 Appendix: https://github.com/openai/CLIPhttps://github.com/haotian-liu/LLaVAhttps://github.com/explosion/spaCy 


Decision: The paper's primary strength lies in its novel application of Shapley Value for multimodal alignment via contrastive learning. This innovative approach, bridging concepts from game theory with multimodal tasks, has the potential to contribute significantly to the field of multimodal information extraction. Additionally, reviewer ETQ2 highlighted the intuitive nature of using image captions as contexts to bridge semantic and modality gaps.
However, several limitations were raised by reviewers. Most notably, reviewer qjzp, who provided the most detailed analysis, raised concerns about the limited performance improvements, especially considering the use of LMMs. This reviewer also pointed out potential technical issues, such as interference between text and context encodings, and questioned the motivation behind the Adaptive Fusion module. These technical concerns are particularly impactful.
After careful consideration of the strengths and weaknesses, I recommend Acceptance for this submission.

Reviews: review: paper_summary: paper_summary: The Shapley Value-based Contrastive Alignment (Shap-CA) method introduces a new paradigm of Image-Context-Text interaction for multimodal information extraction. Shap-CA utilizes large multimodal models to generate contexts and introduces the Shapley Value concept to evaluate the individual contribution of each context to mitigate semantic gaps.
strength_weakness: strength_weakness: This work is the first study to utilize Shapley Value for multimodal alignment via contrastive learning, referencing knowledge from game theory to multimodal tasks.
limitations: limitations: 1.The motivation for introducing the Adaptive Fusion module is less clear, which is not strongly associated with the use of LLM-generated contexts as a bridge and the innovativeness of the Shapley Value. Further explanation is needed from the authors.
2.The text are concatenated with context and fed into BERT for textual feature encoding in Eqn. 1, which can cause potential interference between the feature encodings of the text and context due to the sequential contextual relevance of BERT itself. Does this affect the calculation of the cosine semantic similarity between the two in Eqn. 4.
3. The performance enhancement of this work is not particularly significant. Specifically, several competitive works, including ITA, MNER-QG, and EEGA, have not utilized a LMM. Table 3 presents a comparison of LMMs that is limited to MQA, and it lacks a performance benchmark for LLAVA 1.5 or other LMMs, which are utilized for context generation.
4. Please carefully check the details in the paper. Specifically, the symbols corresponding to part-of-speech embeddings and positional embeddings in 3.6 are written backwards. This will affect the reader's understanding.
paper_summary: paper_summary: This paper introduces a method for multimodal information extraction. It adopts LMM to generate a context for each image, which serves as a bridge between image and text to narrow both the semantic and modality gaps. Subsequently, it applies the Shapley value concept from cooperative game theory to refine the connections between modalities and adaptively fuse them. Experiments on four popular datasets validate the effectiveness and generalization of the proposed method.
strength_weakness: strength_weakness: 1.	The paper is well-motivated. Using captions of images as contexts to bridge the semantic and modality gaps intuitively makes sense.
2.	It is clever to apply Shapley value concept from game theory. The experiments demonstrate its superiority compared to contrastive learning in aligning modalities.
3.	The comprehensive experimental results sufficiently demonstrate the advantages of the proposed method.
limitations: limitations: 1.	The necessity of using LMMs as the context generator is not particularly convincing. It can be observed in Tab. 5 that the improvements brought by LMMs are limited. Comparing LMMs with more advanced expert models (such as OFA) might strengthen the results.
2.	Lack of visualization elements (such as heatmap) on how the generated contexts improve the model’s perceptions of images and texts.
3.	It appears that there is an oversight in line 728. The authors may have mistakenly written Table 4 as Table 2.
paper_summary: paper_summary: This paper adopts the context as a bridge for the image-text interaction. Furthermore, a Shapley Value-based Contrastive Alignment is proposed to align the context-text and context-image pairs.
strength_weakness: strength_weakness: 1. The idea of image-context-text interaction is novel and effective;
2. The paper is well-organized and written.
limitations: limitations: 1. The performance improvements are not significant compared with previous works shown in Table 3, I'm not familiar with MIE, is that acceptable?
2. The figures in the paper are better to use vector images.
paper_summary: paper_summary: This paper proposes Image-Context-Text interaction with LLM for multimodal information extraction tasks to bridge semantic and modality gaps. A Shapley Value-based Contrastive Alignment method is proposed to align context-text and context-image pairs. Contrastive learning and adaptive fusion are used to enhance features. Experiments on four MIE datasets demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. The idea of generating context using LMM for bridging the semantic and modality gap is interesting.
2. The introduction of Shapley Value-based Contrastive Alignment is novel.
3. Experimental results achieve SOTA.
4. This paper is well-written and well-organized.
limitations: limitations: 1. The context generated by LMMs can be seen as a kind of wild knowledge, which may be unfair to compare the proposed methods with previous methods directly. It would be better to integrate the context to text modality of previous methods for comparison.
2. Are the backbones of the compared methods the same in Table 1? It would be better to list the backbone.
3. Is the context needed in inference?