Few-Shot Joint Multimodal Entity-Relation Extraction via Knowledge-Enhanced Cross-modal Prompt Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 05:18:03

Abstract: Joint Multimodal Entity-Relation Extraction (JMERE) is a challenging task that aims to extract entities and their relations from text-image pairs in social media posts. Existing methods for JMERE require large amounts of labeled data.
However, gathering and annotating fine-grained multimodal data for JMERE poses significant challenges. Initially, we construct diverse and comprehensive multimodal few-shot datasets tailored to the data distribution. To address the insufficient information in the few-shot setting, we introduce the \textbf{K}nowledge-\textbf{E}nhanced \textbf{C}ross-modal \textbf{P}rompt \textbf{M}odel (KECPM) for JMERE. 
This method can effectively address the problem of insufficient information in few-shot setting by guiding a large language model to generate supplementary background knowledge.
Our proposed method comprises two stages: (1) a knowledge ingestion stage that dynamically formulates prompts based on semantic similarity and employs self-reflection to refine the knowledge generated by ChatGPT; (2) a knowledge-enhanced language model stage that merges the auxiliary knowledge with the original input and utilizes a transformer-based model to execute JMERE tasks. We extensively evaluate our approach on a few-shot dataset derived from the JMERE dataset, demonstrating its superiority over strong baselines in terms of both micro and macro F$_1$ scores. Additionally, we present qualitative analyses and case studies to elucidate the effectiveness of our model.

Main paper: 
 multimodal information joint extraction.: 
 Advancements in independent extraction methods facilitate joint entity and relation extraction from multimodal data. A typical approach is the pipeline framework, widely employed in unimodal domains [2, 17] . In the pipeline method, MNER methods initially identify named entities, followed by MRE methods to classify relationships among these entities. However, this approach is heavily dependent on the precision of the MNER step, potentially propagating errors [9] . Furthermore, pipeline methods are unable to leverage implicit connections between MNER and MRE tasks to enhance their performance. To overcome this limitation, Yuan et al. [40] proposed the Joint Entity-Relation Extraction Task (JMERE) to capture implicit connections between MNRE and MRE. This work also proposed word-pair relation tagging to extract all elements of JMRER in a single step, reducing error propagation.In this paper, we focus on the investigation of joint multimodal entity-relation extraction in few-shot setting. This choice aims to align with real-world applications that frequently offer only a restricted number of labeled data. We are the first to focus on JMERE in a multimodal few-shot scenario. Besides, We propose a knowledge-enhanced cross-modal prompt model to address the few-shot scenario, utilizing dynamic prompts and knowledge reflection, to obtain background knowledge from LLMs for improved performance FS-JMERE.Established few-shot learning approaches have been applied in unimodal information extraction (e.g., Named Entity Recognition (NER) [3, 31, 35] and Relation Extraction (RE) [4, 5, 13] ). The works in Few-shot Information Extraction (FS-IE) can be divided into two categories: the first category uses limited data for supervised training. Fritzler et al. [3] applied prototype networks [24] to address the few-shot NER (FS-NER) task. Inspired by nearest-neighbor inference in few-shot learning [31] , Yang et al. [35] employed supervised feature extractors for FS-NER. Within the few-shot RE (FS-RE) task, Gao et al. [4] introduced attention mechanisms to augment prototype networks [24] , and Han et al. [5] incorporated relationship description information as a means of distinguishing between complex and straightforward tasks. Liu et al. [13] incorporate relationship description with support sample representations, resulting in notably superior results. Nevertheless, this approach fails to address the issue of insufficient information in few-shot setting.The second category focuses on pre-training or integrating external information to address the challenges of the few-shot setting. In the FS-NER task, Huang et al. [8] proposed a specialized noisy supervised pre-training approach, while Wang et al. [28] explored model distillation techniques to enhance few-shot NER performance. In the FS-RE task, Qu et al. [19] developed a comprehensive strategy involving the construction of a global relationship graph from Wikidata, which effectively facilitated the learning of posterior distributions of relation prototype vectors. Additionally, Yang et al. [33] aimed to improve model performance by introducing entity concepts.However, constructing extensive unlabeled datasets and knowledge necessitates a significant human effort. Such an approach contradicts the initial goal of addressing the challenges posed by the few-shot setting. Besides, these methods have primarily been focused on the unimodal domain. Their direct application in the Compared with the above methods, We use LLMs as an implicit knowledge repository for the heuristic generation of additional textual knowledge. In contrast to the labor-intensive task of constructing extensive unlabeled datasets and knowledge repositories, this approach offers more inexpensive and precise knowledge. In addition, we introduce a supplementary knowledge generation method based on semantic similarity. This approach facilitates the dynamic generation of prompts, producing more contextually appropriate examples to guide ChatGPT in producing the requisite auxiliary background knowledge. 
our proposed model: 
 Following the JMERE task setting, the i-th input sample comprises a sentence denoted as 𝑋 𝑖 𝑇 and a corresponding image represented as 𝑋 𝑖 𝐼 . The JMERE task aims to extract entities, including their respective types, while also determining the relationships that exist between these identified entities. The resulting output is structured as a collection of quintuples represented by 𝑌 = {(𝑒 1 , 𝑡 1 , 𝑒 2 , 𝑡 2 , 𝑟 ) 𝑐 } 𝐶 𝑐=1 , where (𝑒 1 , 𝑡 1 , 𝑒 2 , 𝑡 2 , 𝑟 ) 𝑐 pertains to the 𝑐-th quintuple comprising distinct textual entities 𝑒1 and 𝑒 2 . These entities are associated with their respective entity types 𝑡 1 and 𝑡 2 , as well as the relationship denoted by 𝑟 between the two entities within the task context. Fig. 3 presents the overall architecture of the proposed method, which comprises a conceptually straightforward two-stage framework. The Knowledge Ingestion stage is the crucial part of our proposed method. By constructing a cross-modal heuristic prompt, we use this prompt to guide ChatGPT to acquire initial predictions and background knowledge related to the JMERE sample. In the subsequent step, we aim to integrate additional supplementary knowledge into the language model, such as T5, to enhance the model's capacity to extract entities and relations between entities. We combine the aforementioned knowledge to augment the input text. This augmentation involves integrating type-based prompts and structured image descriptions as inputs to the language model, thereby further enhancing its capabilities. It is worth noting that we did not utilize visual object features (e.g., Mask R-CNN), because we found that incorporating these features did not lead to performance improvement. Detailed experimental setups and descriptions can be found in Section 3.7 Case Studies.This stage aims to generate auxiliary background knowledge from ChatGPT relevant to the given text and image, enhancing the downstream model's capacity to comprehend the context. This stage is divided into three steps: Human Prompts, Prompt Selection, and Knowledge Reflection. Subsequently, we describe each step in detail.To select suitable prompts for a given sample, we initially devised a human prompt set 𝑃 = {𝑝 1 , 𝑝 2 , • • • 𝑝 𝑀 } specifically designed for the JMERE task, where 𝑀 represents the number of predefined human prompt sets. Each human prompt 𝑝 𝑚 includes not only the original text 𝑝 𝑚 𝑇 and its corresponding image description 𝑝 𝑚 𝐼 , but also encompasses two distinct external knowledge components, as illustrated in Figure 3 To ensure the relevance of prompts to the given sample, we developed a multimodal prompt-aware module to select appropriate examples from the human prompt set. As an essential component of multimodal knowledge extraction, the JMERE task necessitates the integration of textual and visual information. Therefore, we initially combine the original text 𝑝 𝑚 𝑇 and its corresponding image description 𝑝 𝑚 𝐼 as input into a language model M a (such as T5-large) to encode fused representations for each human prompt,EQUATIONwhere 𝑧 𝑚 ∈ R 𝑑 ℎ represents the high-dimensional representation of the m-th human-prompt and 𝑑 ℎ denotes the output dimension of the language model. Similarly, for the i-th JMERE sample, we obtain its image description X𝑖 𝐼 through image 𝑋 𝑖 𝐼 and its high-dimensional representation 𝑞 𝑖 ∈ R 𝑑 ℎ using the method in Eq. 1,EQUATIONIt is crucial to recognize that instances closer in high-dimensional space are likely to share more similar entity types and object information. Therefore, KECPM utilizes cosine similarity to compute the high-dimensional similarity between each JMERE sample and each predefined human prompt. Subsequently, KECPM selects the top-K most similar predefined human prompts as candidate prompts to be sent to ChatGPT for generating refined auxiliary knowledge.EQUATIONwhere I 𝐾 𝑖 represents the top-K indices in human prompt set 𝑃 fori-th JMERE sample. Thus, the ultimate input for ChatGPT to query its knowledge comprises the in-context 𝐶 𝑖 , which is defined as follows: 1 : The statistics of the FS-JMERE dataset. Here #S denotes the number of sentences. The #AL is the average length of the sentence. The elements in the Top 3 and the Last 3 Relations represent the relation type and their corresponding frequencies.EQUATIONwhere 𝐷 represents a predefined head prompt, which describes the JMERE task in natural language according to the requirements, and 𝑆 𝑖 is a text template for i-th JMERE sample, which is denoted as, 𝑆 𝑖 = [Context :𝑇 𝑒𝑥𝑡 ; Image :𝐼𝑚𝑎𝑔𝑒𝐶𝑎𝑝𝑡𝑖𝑜𝑛; Answer :] 2.1.3 Knowledge Reflection. The composed knowledge prompts 𝐶 𝑖 send into ChatGPT. We encourage ChatGPT the autonomy to make its own judgments for each input sample by leaving the answer field blank, allowing ChatGPT to generate responses. However, we have observed that responses from ChatGPT are not always definitive, which could potentially lead to inaccuracies in the provided knowledge. Despite ChatGPT subtly hinting at its possession of relevant background knowledge, this may not guarantee correctness. As illustrated in Fig. 2 , although ChatGPT did not explicitly identify "Greece (location)", it can be implicitly inferred from its reasoning that ChatGPT can recognize "Greece" as a city. Therefore, we encourage ChatGPT to reflect more deeply and acquire knowledge that is more pertinent and beneficial [22] . However, in practice, we have observed that while reflection can empower large language models to self-correct, it occasionally leads to meaningless responses, introducing considerable noise into the process. As shown in Fig. 4 , for instance, we receive responses like The entities, relations, and reasoning provided above are correct, even inputting a description such as, "If it is correct, just repeat your answer".To address these limitations, we propose a knowledge selector to choose meaningful auxiliary knowledge. After N self-iterations in ChatGPT involving human interactions, we collect N responses and process them using the same language model as Eq. ( 1). Consequently, the resulting representations of the ChatGPT responses (Auxiliary Knowledge) set are denoted as 𝑅 = {𝑟 1 , 𝑟 2 , • • • , 𝑟 𝑁 }. We utilize cosine similarity to compute the high-dimensional similarity between each sample and every item of auxiliary knowledge, as shown in Eq. (3). Subsequently, we designate the most similar auxiliary knowledge as the final auxiliary knowledge, denoted as,𝐹 𝑖 = arg Top1 𝑗 ∈ {1,2,•••𝑁 } (𝑞 𝑖 ) 𝑇 𝑟 𝑗 𝑞 𝑖 2 𝑟 𝑗 2 (5)where 𝐹 𝑖 is the final auxiliary knowledge for i-th input sample.In this stage, we integrate auxiliary knowledge denoted as 𝐹 1 𝑖 obtained from ChatGPT with the original input 𝑞 𝑖 . These combined inputs are fed into a transformer-based language model, such as T5, to generate both entities and their corresponding relationships. To formalize this process, we modify our training objectives to facilitate the generation of entity and relation information. To enhance the readability and interpretability of the generated outputs, we convert each element of the JMERE quintuples, two entities 𝑒 1 and 𝑒 2 with their corresponding types 𝑡 1 and 𝑡 2 and their relationship 𝑐, into a language sketch. For example, as shown in Fig. 3 , the ground truth of the given sample is "The relation between TV3 (organization) and Keadilan (organization) is subsidiary", where TV3 (organization) and Keadilan (organization) are two entities with their corresponding entity types, and the subsidiary describes their relation. It is important to note that when a sample contains multiple quintuples, we perform template filling for each quintuple, subsequently concatenating these templates using ";". Furthermore, the final formalized input adheres to the following format, 𝐼 = [Typ :𝑇𝑦𝑝𝑒𝑃𝑟𝑜𝑚𝑝𝑡; Image :𝐼𝑚𝑎𝑔𝑒𝐶𝑎𝑝𝑡𝑖𝑜𝑛; Knowledge :𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒; Text :𝑇 𝑒𝑥𝑡]EQUATIONwhere LM represents the Language Model and each element ℎ 𝑖 ∈ R 𝑑 ℎ in 𝐻 represent each token representation of input 𝐼 , where 𝑑 ℎ is the dimension of the language model. To optimize the model parameters, KECPM uses the language modeling loss as the loss function for the input sequence with ground-truth labels,EQUATION)where |𝑦| is the length of output sequence 𝑦. 
experiments 3.1 few-shot datasets: 
 We conduct experiments on few-shot multimodal datasets built according to the distribution of relation categories from JMERE [40] . This dataset is currently the solely available resource for joint multimodal entity-relation extraction. In constructing datasets for FS-JMERE, a crucial consideration is to select a diverse set of samples that comprehensively cover various relation categories. We accomplish this by sampling data based on the distribution of relation categories within instances, aligning with the original distribution of relations. Detailed statistics of the dataset can be found in Table 1 . For each dataset, we utilize a random sampling approach with three different seeds: {19, 67, 97}. This process results in the creation of three distinct sets of few-shot training and development datasets.It is important to note that each split is performed three times. Subsequently, we report the average performance and standard deviation derived from 9 training runs (3 × 3), providing a more robust evaluation of our model.Our baseline methods include joint entity-relation extraction in unimodality and multimodality. the implementation details for each method are described below.• Tplinker [29] is a one-stage method designed for the joint extraction of entities and overlapping relations. It accomplishes this by transforming the task into a Token Pair Linking problem, addressing specific questions regarding token positions and relations. • BiRTE [20] represents a bidirectional extraction framework that leverages two complementary directions, incorporating a shared encoder component and an affine model for relation assignment. Moreover, it introduces a shared-aware learning mechanism aimed at alleviating convergence rate disparities. • SpERT [1] employs a span-based approach to identify entities and relations within sentences, to improve the multiword entity extraction.• AGBAN+MEGA is an enhanced iteration of AGBAN, originally introduced in [46] . It utilizes adversarial learning and a bilinear attention network to enhance the fusion and synchronization of entities with their corresponding visual objects, thus improving the performance of the MNER task. • UMGF+MEGA is an enhanced version of UMGF described in [41] , which integrates regional image features to represent objects and utilizes fine-grained semantic correspondences via transformer and visual backbones. This enhancement is motivated by the recognition that informative object features are more crucial than the entire image for MNER task. • EEGA is the state of the art to address the JMERE task. It leverages the interaction between the MNER and MRE tasks via word-pair relation tagging. Additionally, it introduces an edge-enhanced graph alignment network that effectively aligns text entities and objects by utilizing edge information across graphs.To evaluate the performance of triplet extraction, we employ a set of evaluation metrics consistent with the JMERE setting [40] . These metrics include precision (P), recall (R), and micro F 1 -score (Mic-F 1 ). Additionally, considering the imbalance in relationship labels within the dataset-where the top 3 labels account for 50% of the test set samples while the least 3 labels account for only 1%, we use the macro-F 1 score as an additional evaluation metric. This metric enables us to assess the model's effectiveness in learning from few-shot samples. The computation of macro-F 1 (Mac-F 1 ) is presented as follows:EQUATIONwhere F 1 (𝑖) represents the F 1 score for the i-th relation type, with 𝑁 representing the total number of relation types.During the knowledge ingestion stage, we utilize the OFA model [27] , a multimodal pre-trained model used for converting images into corresponding text descriptions. Additionally, we employ T5large as our feature extractor, facilitating the fusion of image captions and textual data. Furthermore, we utilize ChatGPT with the gpt-3.5-turbo version and set the sampling temperature to 0. In the knowledge-enhanced language model stage, we employ the AdamW optimizer [14] to minimize the loss function. To determine the optimal learning rate, we conduct a grid search within the range of [2𝑥10 -4 , 2𝑥10 -6 ], ultimately setting it to 5x10 -5 . Additionally, we incorporate a warm-up linear scheduler to regulate the learning rate. We set the maximum sentence input length to 500, while the mini-batch size is configured to be 6. Our model undergoes training for 30 epochs, and we select the model with the highest micro F 1 -score on the development set for evaluation on the test Table 2 presents the JMERE results on the few-shot setting, unveiling several crucial observations. Firstly, multimodal models, exemplified by AGBAN+MEGA, UMGF+MEGA, and EEGA, only achieved performance similar to unimodal methods like Tplinker and BiRTE, and even fell below the performance of SpERT, which was specifically designed for joint entity-relation extraction in the few-shot setting. This indicates that while the aforementioned multimodal models can obtain some visual features to benefit information extraction, they typically demand a significant number of data to align and fuse different modalities. This requirement contradicts few-shot setting, making it challenging to extract crucial information from visual features. Furthermore, the SpERT model achieved notable performance in terms of Micro F 1 but under significant decline in Macro F 1 metrics, indicative of its struggle to effectively learn from long-tail data due to data imbalance. Moreover, our proposed model consistently outperforms all baseline models in both Micro F 1 and Macro F 1 metrics, showcasing its performance with improvements of 2.27 % and 1.69% in the respective metrics. Several factors contribute to the outstanding performance of our model: firstly, the utilization of an image caption-based approach enables the effective extraction of image information without the need for additional data training; secondly, the incorporation of refined auxiliary knowledge from ChatGPT equips the model with essential and relevant information, thereby enhancing overall performance and mitigating the impact of long-tail data; lastly, the employment of self-reflection0 1 2 3 4Response Iterations N and knowledge selection further enhances the accuracy and relevance of knowledge information.Ablation studies were conducted to assess the effectiveness of each prompt in the proposed model: the original text, image caption, and three different methods for obtaining knowledge (Directly, Reflection, and Selected). The results of the ablation experiments are presented in Table 3 . In comparison to using only the original text (T), the inclusion of Image Captions (T+I) notably improved the results, yielding a 0.5% increase in Micro F 1 and a 0.35% increase in macro F 1 . This demonstrates that image information can offer insights into entity types. Moreover, the auxiliary knowledge of direct responses from ChatGPT (T+I+K 𝐷𝑖𝑟𝑒𝑐𝑡𝑙 𝑦 ) led to a substantial improvement in Macro F 1 (0.51%) and Micro F 1 (0.36%). This observation suggests that augmenting with auxiliary knowledge significantly impacts the performance, particularly in smaller sample sizes. Furthermore, through engaging ChatGPT in self-reflective (T+I+K 𝑅𝑒 𝑓 𝑙𝑒𝑐𝑡𝑖𝑜𝑛 ), we acquired higher-quality responses, resulting in an enhancement across both metrics. Nevertheless, when contrasted with our proposed model KECPM(T+I+K 𝑆𝑒𝑙𝑒𝑐𝑡𝑒𝑑 ), there remains an opportunity for further improvement, particularly in the Macro F 1 (0.67%). This highlights the effectiveness of the proposed knowledge selector in filtering hallucinatory knowledge derived from multi-turn reflections. Finally, we attempted to incorporate visual object features (OF) from Mask R-CNN into the input of the language model, similar to previous work [34] , to include more fine-grained information. However, this integration did not result in performance improvement. We consider that aligning features across different modalities from distinct pre-training datasets (text from T5, objects from Mask R-CNN) demands a significant number of data, which is inappropriate in few-shot setting. Text: RT @beINSPORTS:• Lopetegui insists @lukamodri10 is n't going anywhere . # LaLiga # RMCF Captions: a fan reacts to a commentator during a game against sports team Refined Knowledge:The relation between Lopetegui (person) and lukamodric10 (person) is "peer". Reasoning: Lopetegui is the former manager of Real Madrid and Luka Modric is a professional football player who played for Real Madrid during Lopetegui's tenure as manager.As both of them were associated with Real Madrid at the same time, they can be considered as peers.Label: (Lopetegui(per) lukamomdri10(per) peer)EEGA: (Lopetegui(per) LaLiga(per) peer)SpERT: (Lopetegui(per) lukamomdri10(per) peer)KECPM: (Lopetegui(per) lukamomdri10(per) peer)   Text: RT @HammyEnd : # Chambers, # Fulham on loan : # ffc # coyw Captions: olympic athlete signing a paper during a press conferenceThe relation between Chambers (person) and Fulham (organization) is "member of". Reasoning: Chambers is Calum Chambers, an English professional footballer who plays as a right-back or centre-back for Fulham FC in the English Football League Championship. He is currently on loan from Arsenal Label: (Chambers(per) Fulham(per) member of)SpERT:(Chambers(per) Fulham(per) peer)(Chambers(per) HammyEnd(org) member of) (Chambers(per) Fulham(org) member of)Text: RT @VisitGreecegr : Good morning, Nafplion # Greece # VisitGreece # Travel Captions: a castle in the middle of a body of waterThe relation between Nafplion (location) and Greece (location) is "place of residence". Reasoning: Nafplion is a city in Greece, located on the northeastern coast of the Peloponnese peninsula. It is a popular tourist destination known for its historic architecture, including a Venetian fortress and a Greek Orthodox church. Nafplion is one of the most picturesque and romantic towns in Greece, attracting visitors from all over the world. The results, as shown in Table 4 , Firstly, the ChatGPT with fewshot settings has a performance gap with our proposed model. We have observed that ChatGPTgenerates entity and relationship types not present in the FS-JMERE pre-defined, such as "building." This underscores the importance of Stage 2 in our proposed method, as it effectively aligns with the FS-JMERE task while leveraging background knowledge from ChatGPT. Moreover, the PS module can notably enhance the quality of auxiliary knowledge generated by ChatGPT and improve the model performance. Besides, an optimal number of human-prompt instances can further refine the quality of auxiliary knowledge. However, merely incorporating human prompts without consistent improvement in performance may not suffice. Supplying more than 5 prompts (K=10) to ChatGPT may decrease the quality of auxiliary knowledge. In practice, it has been observed that an excessive number of manual examples introduces noise into the generation process of ChatGPT, potentially disrupting its original reasoning process and leading to the replication of manual examples.Finally, our proposed model integrates N iterations of ChatGPT responses, enhancing knowledge accuracy through self-reflection. Consequently, we investigate the impact of varying iterations of selfreflection (𝑁 ) on performance, as depicted in Fig. 5 . When 𝑁 = 0, which indicates the absence of self-reflection, the model exhibits the lowest performance, underscoring the efficacy of self-reflection in rectifying incomplete knowledge for improved performance. However, when 𝑁 exceeds 2, the model's performance gradually declines. Despite the inclusion of the proposed Knowledge Selector, which filters knowledge for relevance to the input text, multiple rounds of self-reflection also introduce more noise in responses.Through three case studies depicted in Figure 6 , we demonstrate how auxiliary refined knowledge enhances the predictive performance of the model. In the first case, only EEGA incorrectly identifies LaLiga as a person, attributed to the presence of only humans in the image, leading to erroneous information fusion. Our model benefits directly from refined auxiliary knowledge, which accurately provides entity recognition and relationships. In the second case, the brevity of the text makes acquiring information about Fulham challenging, resulting in incorrect labeling by both SpERT and EEGA. However, our proposed model, with refined auxiliary knowledge included, effectively identifies the entity's type from such supplementary information.The third case emphasizes the importance of integrating and fine-tuning auxiliary knowledge for the FS-JMERE task. While refined auxiliary knowledge correctly identifies entities Greece and Nafplion and their respective types, it incorrectly identifies the relationship between Greece and Nafplion. In the JMERE task, entities labeled as a person and an organization cannot have a member of relationship. By integrating this knowledge and fine-tuning the model, our method uncovers potential mappings between entity types and relationships, correcting errors and aligning with the FS-JMERE task requirements more effectively. 
conclusion: 
 In this paper, we propose a Knowledge-Enhanced Cross-modal Prompt Model (KECPM) for Few-shot Joint Multimodal Entity-Relation Extraction (FS-JMERE), which can effectively handle JMERE in low-data scenarios by guiding a large language model to generate auxiliary background knowledge from text and image. To our knowledge, we are the first to focus on Multimodal Entity-Relation Extraction in the few-shot setting. Our model consists of two stages: firstly, a knowledge ingestion stage that dynamically creates prompts based on semantic similarity and uses self-reflection to refine the knowledge generated by ChatGPT; then, a knowledgeenhanced language model stage that integrates the auxiliary knowledge with the original input and employs a transformer-based model to perform JMERE. We have conducted extensive experiments on a few-shot dataset constructed from the JMERE dataset and demonstrated that our model outperforms strong baselines in terms of micro and macro F 1 scores. We have also provided qualitative analysis and case studies to illustrate the effectiveness of our model. For future work, we plan to explore more ways to generate and select high-quality auxiliary knowledge from ChatGPT, such as using reinforcement learning or adversarial learning. We also intend to apply our model to other multimodal information extraction tasks, such as multimodal sentiment analysis.  
: 
 from social media, primarily composed of images and text, to generate structured knowledge. However, previous works treated these tasks independently and restricted the exploration of their interconnections [32, 37, [43] [44] [45] [46] . To address this problem, Yuan et al. [40] introduced a Joint Multimodal Entity-Relation Extraction (JMERE) task, aiming to enhance performance by integrating bidirectional interactions between the two subtasks. JMERE involves simultaneously extracting entities and their potential relations. However, previous JMERE work usually extracted features separately using pre-trained models (e.g., BERT for textes and ResNet for images), and employed multi-modal fusion layers for interaction, which typically require substantial training data [38] . Nevertheless, extensive multimodal data collection and annotation demand significant time and labor investments [47] . Furthermore, in real-world applications, access to labeled data is often constrained, and diverse domains necessitate specific datasets, presenting a challenge for data collection efforts. Thus, we first explore the JMERE task in few-shot setting, aiming to advance the practical applications of multi-modal information extraction. To our knowledge, we are the first to focus on JMERE in a multimodal few-shot scenario (FS-JMERE). However, in few-shot setting, aligning different modalities and conducting information extraction between modalities pose heightened challenges compared to full datasets setting [38] . For instance, as illustrated in Fig. 1 , discerning the relationship between TV3 and MCMC based solely on the concise text "RT @saladinMY: TV3 with the MCMC." proves challenging, despite supplementary images hinting that both entities may belong to an organization such as television stations. An intuitive approach involves utilizing a large number of external Twitter data for pre-training, which can enhance the model's capacity for information extraction and relevant information storage [38, 39] . However, this approach requires extensive pre-train datasets and corresponding training times. Furthermore, some unimodality approaches aim to extract relevant information from sources such as knowledge graphs (e.g., Concept-Net [25] ), to acquire background knowledge on entities, aiming to enhance the model's capability to extract entities or their relationships [10, 26] . However, these retrieval-based methods may encounter difficulties in locating relevant information and retrieving an excess of irrelevant information from knowledge graphs [21] . As shown in Fig. 1 ConceptNet, and their absence introduces some irrelevant concepts knowledge, such as movie and custom. This information is not beneficial for extracting the entities of TV3 and MCMC and their relationship.Inspired by [21] and [11] using Large Language Models (LLMs) as an external knowledge repository for specified tasks, we aim to obtain relevant background knowledge from LLMs, such as Malaysian television managed by Malaysian Communications and Multimedia Commission, as depicted in Fig. 1 section (b). Such background knowledge can effectively address the challenge of insufficient information in few-shot setting without the need for additional pre-training. However, acquiring high-quality knowledge from LLMs relies on selecting appropriate prompt examples [12, 36] . Although one simplistic approach involves employing fixed contextual prompts, this method may generate prompts with very low relevance to given samples and obtain low-relevance responses from LLMs [36] . Yang et al. [36] indicated that all example selection strategies exhibit markedly inferior performance in comparison to the oracle strategy, which leverages the similarity of groundtruth answers. Furthermore, we have observed that responses from ChatGPT are not always definitive, potentially leading to inaccurate knowledge. As illustrated in Fig. 2 , ChatGPT did not explicitly identify "Greece (location)" with the first response. However, it will correct itself and provide correct knowledge, after self-reflection [22] . Thus, it is crucial to guide LLMs in generating the necessary knowledge or addressing new tasks effectively by constructing more suitable contextual examples and responses [21, 30] .To address above challenges, we propose the Knowledge-Enhanced Cross-modal Prompt Model (KECPM) for Few-shot Joint Multimodal Entity-Relation Extraction (FS-JMERE), as shown in Fig. 3 . Initially, to create a few-shot training and development dataset, we sampled data according to distinct relationship types' data distribution. To address the insufficient information in FS-JMERE, we propose a method to acquire relevant knowledge from ChatGPT, which employs a dynamic prompt creation method and a knowledge selector to guide ChatGPT in generating more relevant auxiliary background knowledge. Formally, we manually annotate a limited set of human samples as candidate prompts, which comprises two primary components. The first part entails identifying the named entities and their relationships within the sentences. The second part involves offering comprehensive justifications by considering both the image and text content, along with relevant knowledge. Subsequently, KECPM employs a multimodal similarity-aware module to select human samples and integrates them into prompt templates fitted for the JMERE task, thereby introducing relevant background knowledge. To further improve the relevance and usefulness of knowledge generated by ChatGPT, we enable multiple rounds of self-reflection [22] , allowing it to rectify errors. Subsequently, we employ a knowledge selector to choose more accurate and relevant knowledge. This approach leverages ChatGPT's few-shot learning capability and its capacity for self-reflection within in-context learning. The auxiliary knowledge from ChatGPT's heuristic method is integrated with text and image captions, enhancing input for downstream models without requiring additional modality alignment for the FS-JMERE task. The main contributions of this study are summarized as follows:• We focus on joint multimodal entity-relation extraction (JMERE) in the few-shot setting. To our knowledge, we are the first to focus on it and build the few-shot dataset by taking into account the distribution of relation categories. (1) Human Prompts ……. …….Text: @allkpop : ' Produce 48 ' group IZONE to … to meet with AKB producer Caption: a group of women wearing pink ties and shirts Answer: The relation between AKB (Organization) and Japan (Location) is locate at Reasoning: Janpan is a asian country. AKB48 is a large female idol group in Japan produced by Yasushi Akimoto.Knowledge-aware Prompt Query Text(2) Prompt Selection entities [43] [44] [45] . Zheng et al. [45] collected the first MNRE dataset from social media. Zheng et al. [44] proposed a graph alignment method that aligns structures from different modalities, like text syntax trees and object scene graphs, aiding in entity-object correspondence. To enhance entity relation extraction using visual data, Zhao et al. [43] introduced the two-stage visual fusion network (TSVFN), using multimodal fusion for improved relation extraction.However, these methods treated MNER and MRE tasks independently, thereby limiting the exploration of their interconnections. 
 
 Appendix:  


Decision: The paper proposes a Knowledge-Enhanced Cross-modal Prompt Model for Joint Multimodal Entity-Relation Extraction in few-shot settings. The method addresses the challenge of insufficient data by guiding a large language model to generate supplementary background knowledge through dynamic prompts and self-reflection, which is then integrated with the original input using a transformer-based model. Extensive experiments demonstrate its superiority over strong baselines in terms of both micro and macro F1 scores. Reviewers praised the innovative approach, comprehensive experiments, and effective integration of text and image information. However, concerns were raised about the limited use of image content, reliance on the quality of initial samples and generated prompts, potential introduction of noise through self-reflection, and lack of discussion on efficiency and resource consumption. The authors provided satisfactory responses to some concerns during rebuttal. Despite these limitations, the overall quality and significance of the work justify acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces the Knowledge-Enhanced Cross-modal Prompt Model (KECPM) for JMERE. This method effectively addresses the problem of insufficient information in few-shot settings by guiding a large language model to generate supplementary background knowledge. The proposed method comprises two stages: (1) a knowledge ingestion stage that dynamically formulates prompts based on semantic similarity, guiding ChatGPT to generate relevant knowledge and employing self-reflection to refine this knowledge; and (2) a knowledge-enhanced language model stage that merges the auxiliary knowledge with the original input, utilizing a transformer-based model to align with JMERE’s required output format. The authors extensively evaluate the approach on a few-shot dataset derived from the JMERE dataset, demonstrating its superiority over strong baselines in terms of both micro and macro F1 scores. Additionally, the authors present qualitative analyses and case studies to elucidate the effectiveness of our model.
strength_weakness: strength_weakness: This paper we introduces a new Knowledge-Enhanced Cross-modal Prompt Model (KECPM) for JMERE.

Extensive experiments demonstrate that the proposed approach can obtain better performance.

This work presents qualitative analyses and case studies to elucidate the effectiveness of the model.
limitations: limitations: Some symbols are used without definitions, which makes certain parts difficult to follow.

The experimental analysis can be enhanced to provide more insights.

Missing references:

Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View

Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction
paper_summary: paper_summary: The paper address the challenge of extracting entities and their relations from text-image pairs in social media posts with minimal labeled data. This task, known as Joint Multimodal Entity-Relation Extraction (JMERE), typically requires large datasets, which are difficult to gather and annotate. The authors propose a novel method called the Knowledge-Enhanced Cross-modal Prompt Model (KECPM) to effectively perform JMERE in a few-shot setting.

The KECPM approach consists of two main stages:

1. Knowledge Ingestion Stage: This stage dynamically generates prompts based on semantic similarity and uses ChatGPT to generate supplementary background knowledge. The knowledge is refined through self-reflection to improve accuracy.
2. Knowledge-Enhanced Language Model Stage: In this stage, the auxiliary knowledge is merged with the original input and fed into a transformerbased model to align with JMERE's output requirements.

The authors evaluated their model on a few-shot dataset derived from the JMERE dataset, demonstrating its superiority over existing baselines in terms of micro and macro F1 scores. The approach showed significant improvements in both precision and recall, highlighting the effectiveness of integrating background knowledge generated by large language models.
strength_weakness: strength_weakness: 1. The introduction of a knowledge-enhanced crossmodal prompt model is a novel way to address the lack of data in few-shot learning scenarios. The dynamic prompt generation and knowledge refinement through self-reflection are particularly innovative.
2. The authors conducted extensive experiment on a few-shot dataset, showing that their model outperforms strong baselines in both micro and macro F1 scores.
3. The paper successfully integrates text and image information, addressing the complexity of multimodal data processing in social media contexts.
limitations: limitations: 1. It seems that the model performance is highly dependent on the quality of the initial few-shot samples and the generated prompts. How to evaluate their performance while the author emphasizes the final result？
2. While the self-reflection mechanism is designed to refine knowledge, it can also introduce noise. This noise can lead to inaccuracies and affect the overall performance of the model, particularly if the reflection process does not yield high-quality knowledge consistently. 
3. typo error: the authors use x instead of \times in line 690.
paper_summary: paper_summary: This paper build the few-shot dataset on joint multimodal entity-relation extraction(JMERE) task in the few-shot setting and propose a Knowledge-Enhanced Cross-modal Prompt Model (KECPM) which can effectively handle JMERE in low-data scenarios by guiding a LLM to generate auxiliary background knowledge from text and image. The results show that the model outperforms strong baselines in the JMERE in few-shot setting.
strength_weakness: strength_weakness: 1.	Build the few-shot dataset by taking into account the distribution of relation categories.
2.	Propose a knowledge-enhanced cross-modal prompt model to address the few-shot scenario, utilizing dynamic prompts and knowledge reflection, to obtain background knowledge from LLMs for improved performance in FS-JMERE.
limitations: limitations: 1.	The paper doesn't effectively utilize the content within the images, merely converting them into text using captioning tools. Moreover, the accuracy is reliant on the performance of the captioning tool. Ablation study also indicate that the results don't improve significantly with the addition of images.
2.	With the introduction of additional knowledge, the results don't seem to offer significant advantages compared to the baseline method.
paper_summary: paper_summary: This paper concentrates on the task of Joint Multimodal Entity-Relation Extraction (JMERE), proposing a novel approach that incorporates large language models. The effectiveness of this method is validated through detailed experiments. While the task is innovative and the experiments are thorough, the architectural design lacks significant innovation. Additionally, there is a need for further discussion on efficiency and resource consumption to enhance the practical applicability of the proposed method.
strength_weakness: strength_weakness: 1. The paper focuses on a current hot topic, integrating large language models for entity-relation extraction, which is highly relevant.
2. The experiments in the paper are comprehensive, including ablation studies and case studies that effectively demonstrate the method's efficacy.
limitations: limitations: 1. While combining large models with traditional methods enhances accuracy, efficiency and resource consumption are equally critical for practical applications. This paper does not discuss the method's efficiency or resource utilization, which are essential factors to consider. It is recommended to include this analysis.

2. The architecture's utilization of multimodal information is somewhat limited, as is the comparison of cross-modal information. It would be worthwhile to explore the effectiveness of native large multimodal models on this task.

3. It is suggested that additional experiments be conducted to explore the improvements brought about by using different large model foundations.

4. The overall clarity and presentation of the manuscript could be enhanced. Additionally, some figures require improvement; for example, Figure 6 is distorted due to stretching, and the arrows in Figure 2 are confusing and do not clearly contribute to understanding the core contributions of the paper. Revising these figures is recommended.