Generative Multimodal Data Augmentation for Low-Resource Multimodal Named Entity Recognition

Authors: stephen  curr
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:34:44

Abstract: As an important task in multimodal information extraction, Multimodal Named Entity Recognition (MNER) has recently attracted considerable attention.
One key challenge of MNER lies in the lack of sufficient fine-grained annotated data, especially in low-resource scenarios.
Although data augmentation is a widely used technique to tackle the above issue, it is challenging to simultaneously generate synthetic text-image pairs and their corresponding high-quality entity annotations.
In this work, we propose a novel Generative Multimodal Data Augmentation (GMDA) framework for MNER, which contains two stages: Multimodal Text Generation and Multimodal Image Generation.
Specifically, we first transform each annotated sentence into a linearized labeled sequence, and then train a Label-aware Multimodal Large Language Model (LMLLM) to generate the labeled sequence based on a label-aware prompt and its associated image.
After using the trained LMLLM to generate synthetic labeled sentences, we further employ a Stable Diffusion model to generate the synthetic images that are semantically related to these sentences.
Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed GMDA framework, which consistently boosts the performance of several competitive methods for two subtasks of MNER in both full-supervision and low-resource settings.

Main paper: 
 introduction: 
 Recent years have witnessed an exponential growth of multimodal user posts on various social media platforms such as Twitter, Facebook, and Instagram. As a large amount of multimodal content often contains much important structured information such as named entities and their relations that are crucial for multimodal knowledge graph construction, multimodal information extraction has attracted increasing attention in recent years [23, 25, 43] . As a fundamental task in multimodal information extraction, Multimodal Named Entity Recognition (MNER) aims to extract the named entities mentioned in an image-text pair and classify them into pre-defined types, such as person (PER), location (LOC), and organization (ORG) [41] . For example, given the multimodal tweet in Figure 1 , an MNER system is expected to extract two entities, i.e., Stephen Curry and NBA, and their corresponding entity types are PER and ORG, respectively.Existing approaches on the MNER task primarily include sequence labeling-based methods [4, 32, 38, 40] , index generationbased methods [39] , paraphrase generation-based methods [30] , and in-context learning-based methods [3, 5] . Due to the emerging demand for multimodal knowledge graph construction, Yu et al. [39] recently introduced an extension task of MNER named Grounded MNER (GMNER). As shown in Figure 1 , the goal of GMNER is to extract named entities, entity types and the bounding boxes of their grounded visual objects from text-image pairs. To address the GM-NER task, existing studies mainly focus on either using a pipeline approach to decompose the task into several subtasks and solve them one by one [22, 26] or proposing an end-to-end approach to directly generate the entity-type-object triplets [30, 39] .One key challenge of the aforementioned methods is their heavy reliance on annotated data. As illustrated in Figure 1 , both MNER and GMNER tasks require fine-grained annotation of textual named entities and their entity types, while GMNER further requires annotating the bounding box of visual objects that are corresponding to the named entities. In real applications, it is often time-consuming and costly to obtain such human annotation, which hinders the effectiveness of existing MNER and GMNER models in many lowresource scenarios.One attractive solution to address the data sparsity issue is to automatically generate annotated data by data augmentation (DA). Existing DA methods for NER can be summarized into two groups: 1) using rule-based methods such as word replacement, shuffling, and cropping to obtain similar sentences [11, 29] ; 2) using generationbased methods to directly generate the labeled sentences [12, 44] .However, all these DA methods solely focus on the textual modality, which cannot be directly applied to generate labeled text-image pairs. Compared with these text DA methods for NER, data augmentation for MNER and GMNER is more challenging for several reasons. First, it is necessary to generate both text and images, and each text-image pair should be semantically related. Second, each generated text-image pair is required to have the textual and visual entity annotations.To address these challenges, in this paper, we propose a twostage Generative Multimodal Data Augmentation framework for MNER named GMDA, which contains a Multimodal Text Generation stage to produce synthetic sentences with labeled entities and a Multimodal Image Generation stage to generate the corresponding image for each synthetic labeled sentence. Specifically, given a training sample, we first transform the input text and its entity labels into a linearized sentence, and then devise a Label-aware Multimodal Large Language Model (LMLLM) based on one of the representative MLLMs InstructBLIP [10] , which is trained to generate the linearized sentence based on the input image and an entity label-aware instruction. During inference, for each training sample, we feed the input image and its entity label-aware instruction to the trained LMLLM and use a probability-based sampling strategy to generate a synthetic labeled sentence in an autoregressive manner. Based on the synthetic labeled sentence, the Multimodal Image Generation stage further employs a widely used latent diffusion model named Stable Diffusion [28] to generate a corresponding synthetic image conditioning on the synthetic sentence-based prompt and the original image.The main contributions of our work can be summarized as follows:• We propose a novel Generative Multimodal Data Augmentation framework named GMDA, which can generate a large number of text-image pairs with fine-grained entity annotations for both MNER and GMNER tasks. • Under the GMDA framework, we devise a Label-aware Multimodal Large Language Model (LMLLM) to generate synthetic labeled sentences, followed by employing a latent diffusion model to generate the synthetic image for each labeled sentence.show that the proposed GMDA framework consistently boosts the performance of several competitive methods in both fullsupervision and low-resource settings. 
related work 2.1 multimodal named entity recognition: 
 Multimodal Named Entity Recognition (MNER) aims to recognize named entities in text and classify them into predefined categories based on text-image pairs. Pioneering works [23, 25, 41] focus on fusing visual information for improved word representation learning. With the use of the multimodal transformer architecture, a variety of attention-based mechanisms [1, 6, 7, [35] [36] [37] [38] are designed to model the interactions between textual and visual modalities.In addition, converting images into natural languages [32] and retrieving external knowledge [21, 31] are used to enhance the textual information. Different from sequence labeling-based methods mentioned above, machine reading comprehension (MRC)-based methods [2, 19] , in-context learning-based methods [3, 5] , index generation-based methods [39] , and paraphrase generation-based methods [30] have been recently adapted to the MNER task. Due to the emerging demand for multimodal knowledge graph construction, Grounded MNER (GMNER) is introduced as an extension of MNER, which aims to extract named entities, entity types and the bounding boxes of their grounded visual objects from textimage pairs. Existing end-to-end approaches [30, 39] formulate the GMNER task as multimodal index generation [39] and paraphrase and visual object generation [30] , while pipeline approaches decompose GMNER into MNER, visual entailment and visual grounding task to solve one by one [22] . However, all these methods above heavily rely on a large amount of annotated data, which requires the fine-grained annotation of named entities and their entity types for MNER and the annotation of the bounding box of visual objects that are corresponding to the named entities for GMNER. Since obtaining such human annotation is time-consuming and costly, this work aims to propose an effective data augmentation method for MNER to enrich the annotated text-image pairs, particularly in low-resource settings.Data Augmentation (DA) aims to increase the training data via slight changes of existing training data [34] , which is widely used in various NLP tasks, especially in low-resource scenarios. In the literature, rule-based techniques were commonly employed, and typical approaches include word and mention replacement, segments shuffling [11] , cropping, span rotation [29] , random deletion [34] , and subject/object inversion [24] . However, slight alterations in words may potentially disrupt the fluency of the sentence or compromise the coherent interpretation of annotation tags associated with labeled words. Thus, in recent years, many generation approaches [12, 44] have been proposed to enhance the diversity and preserving the label integrity in sentences.For image data augmentation, Copy-Paste [15] is a simple but useful data augmentation method in object-aware tasks. For visionlanguage representation, MixGen [16] linearly interpolates images and concatenates text sequences to generate a new training sample. Nonetheless, data augmentation in MNER is more challenging due to the necessity of maintaining the semantic relatedness between image-text pairs and generating fine-grained entity annotations in both text and images. Therefore, this work aims to propose a new multimodal data augmentation method to generate paired multimodal synthetic data to address these challenges. 
methodology: 
 In this section, we first introduce the task formulation and the overview of the proposed Generative Multimodal Data Augmentation (GMDA) framework, and then present the details of each module of GMDA.Given a sentence with 𝑛 words 𝒔 = (𝑤 1 , . . . , 𝑤 𝑛 ) and an associated image 𝒗 as the input, the goal of the MNER task is to extract a set of multimodal entity tuples:EQUATIONand the goal of the GMNER task is to further extract the bounding boxes of the corresponding visual objects for each entity: 𝒚 = {(𝑒 1 , 𝑡As mentioned before, the proposed GMDA framework consists of two stages, i.e., Multimodal Text Generation and Multimodal Image Generation. As shown in Figure 2 , in the first stage, we linearize the labeled sentence in each sample into a natural sequence, and then train a Label-aware Multimodal Large Language Model (LMLLM), which decodes the linearized sequence based on the image and the entity label of each sample as the inputs of the encoder. With the trained LMLLM, a probability-based sampling strategy is employed to generate a synthetic labeled sentence for each training sample in an autoregressive manner. As shown in Figure 3 , in the second stage, a widely used latent diffusion model is employed to generate a corresponding synthetic image conditioning on the synthetic sentence-based prompt and the original image.Given a sample 𝒔 𝑖 , 𝒗 𝑖 , 𝒚 𝑖 in the training set D, the goal of this stage is to generate a new sentence 𝒔 𝑔 𝑖 together with its entity labels 𝒚 𝑔 𝑖 . To ensure that the generated sentence is relevant to the original image and the generated entity labels have a high quality, we propose an encoder-decoder based conditional generation technique, which generates the linearized labeled sentence from the original image 𝒗 𝑖 and the original label 𝒚 𝑖 .3.3.1 Linearized Labeled Sentences. Firstly, we perform sentence linearization [12] by converting the labeled sentence into a natural sequence, in which the token labels are inserted before their corresponding words. For example, in Figure 2 , the BIO tags of Stephen, Curry and NBA (i.e., B-PER, I-PER, and B-ORG) are inserted before each word. Note that since the tag O frequently occurs, we remove it to keep the linearized sentence more fluent.To generate the linearized sentence conditioning on the image and the entity labels, we propose a Label-aware Multimodal Large Language Model (LMLLM) based on a widely used MLLM named InstructBLIP [10] .Label-Aware Instruction. To guide the MLLM to better generate the linearized sentence, we first design an entity label-aware instruction, which contains the task description and the entity labels as "Generate a text segment for a tweet based on the image, incorporating the following entities: 𝑒 𝑗 which is a(n) 𝑡 𝑗 ", where 𝑒 𝑗 and 𝑡 𝑗 denote the 𝑗-th entity and its type, respectively. For example, in Figure 2 , we linearize all labeled entities into a natural language sentence "Stephen Curry which is a(an) person, NBA which is a(an) organization". This implies that the entities are Stephen Curry and NBA, with their entity types being PER and ORG, respectively. Image Encoder. To enable the image encoder of the pre-trained InstructBLIP model to acquire task-specific knowledge, we integrate LoRA adapter layers [18] for parameter-efficient fine-tuning. Specifically, given an input image 𝒗, we feed it to the image encoder with LoRA adapters to obtain the image representation H 𝑣 :EQUATIONwhere 𝜃 img LoRA is the set of parameters to learn in LoRA. Q-Former. The lightweight Querying Transformer (Q-Former) consists of an image transformer and a text transformer as submodules that share the same self-attention layers. LoRA adapter layers are also added to the attention layers as follows:EQUATIONwhere Q and E 𝑷 refer to the learned queries and text embedding of label-aware instruction. In cross attention layers, H 𝑣 is regarded as the key and value while concat(Q, E 𝑷 ) is regarded as the query. LLM Encoder. In contrast to InstructBLIP which solely finetunes the Q-former to align textual and visual features, our goal is to fine-tune the LLM to generate the linearized labeled sentence. Thus, we incorporate LoRA layers into the LLM for parameter-efficient fine-tuning. Specifically, the textual embeddings E 𝒑 of the labelaware instruction and the visual embeddings E 𝒗 are concatenated together and then fed into the LoRA-based LLM encoder to derive the hidden representation of the multimodal input:EQUATIONLLM Decoder. The representation of the encoder H 𝑒 is then fed to the LoRA-based LLM decoder to model the probability distribution of the linearized labeled sentence, denoted by 𝒙. Specifically, at the 𝑖-th step, the probability distribution of the output token 𝑝 (𝒙 𝑖 ) is calculated based on the encoded representation H 𝑒 and the previous decoder output 𝒙 <𝑖 as follows:EQUATIONEQUATIONwhere h 𝑖 ∈ R 𝑑 is the hidden representation of the 𝑖-th step, W ∈ R 𝑑 × | V | and b ∈ R | V | areEQUATIONwhere 𝑁 denotes the total number of samples, 𝑀 indicates the length of the linearized labeled sentence, |V | is the size of vocabulary, and 𝒕 𝑗 𝑖 refers to the ground-truth label distribution of the 𝑖-th word in the linearized labeled sentence of the 𝑗-th sample.After training the Label-aware Multimodal Large Language Model, we utilize it to generate synthetic labeled sentences.As shown in Figure 2 , given the image and entity labels of a training sample, we feed its corresponding label-aware instruction and visual representations into the multimodal encoder of the trained LMLLM. For the LLM decoder, its initial input token is the 
 token representing the beginning of the decoded sentence, and the subsequent tokens are generated in an autoregressive manner based on a probability-based sampling strategy.When sampling the next token, we use top-k [13] and top-p (nucleus) [17] sampling strategies to generate a synthetic labeled sentence. Firstly, the top-k method is utilized to retain the top-k tokens with the highest probabilities, and the sampling space is denoted as follows:EQUATIONwhere V (𝑘 ) 𝑖 represents the vocabulary set comprising 𝑘 candidate tokens at time step 𝑖. Subsequently, within this subset of tokens, the top-p method is employed to retain those tokens with cumulative probabilities reaching the top-p threshold 𝑝𝑝:EQUATIONwhere V 𝑖 represents the vocabulary set at time step 𝑖, encompassing all possible tokens that could occur in the sequence. Let𝑝𝑝 ′ = 𝑥 ∈ V 𝑖 𝑝 (𝑥 𝑖 |x <𝑖 ).The original distribution is re-scaled to a new distribution as follows:EQUATION)As the candidate tokens in V 𝑖 are predicted with higher probabilities, the generated sentence typically exhibits fluency and maintains proximity to the original training sample. Furthermore, owing to the inherent randomness in the sampling process, GMDA can sample different tokens as the next token, which enriches the diversity in the generated text.The above process of token generation will be stopped when the next token is predicted as 
. After decoding a linearized labeled sentence, we can extract its entities 𝒚 𝑔 based on the embedded labels within the sentence and obtain the synthetic sentence 𝒔 𝑔 .In this stage, our goal is to generate a corresponding image for each synthetic sentence. As the synthetic sentence is semantically related to the original image, we employ both the synthetic sentence and the original image as inputs for synthetic image generation, which can guide the model to refer to the original image during generation.Specifically, given the synthetic sentence 𝒔 𝑔 and its original image 𝒗, we utilize Stable Diffusion [28] , a popular latent diffusion model, to generate a corresponding synthetic image 𝒗 𝑔 . Firstly, given an original image 𝒗 ∈ R 𝐻 ×𝑊 ×3 in the RGB space, we feed it into the encoder of the variational autoencoder (VAE) in the Stable Diffusion model to obtain the latent representation: where E refers to the VAE encoder and 𝒙 ∈ R ℎ×𝑤×𝑐 . At a specific time 𝑡 0 ∈ (0, 1), the Gaussian noise with a standard deviation denoted as 𝜎 2 (𝑡 0 ) is added to 𝒙, resulting in a perturbed latent representation 𝒙 𝒕 for denoising.EQUATIONSecondly, given the synthetic sentence 𝒔 𝑔 , we design a task instruction containing 𝒔 𝑔 as "A photo of 𝒔 𝑔 ", which is used to guide the generation of the synthetic image. Next, a CLIP text encoder [27] is utilized to project the task instruction to an intermediate representation, which is then mapped to the intermediate layers of the conditional denoising autoencoder. With the conditional denoising autoencoder, we can infer the denoised latent variant: x ∈ R ℎ×𝑤×𝑐 and employ the decoder of the VAE model D to generate the synthetic image:EQUATIONwhere 𝒗 𝑔 ∈ R 𝐻 ×𝑊 ×3 denotes the synthetic image.It is worth noting that for the GMNER task, it is required to annotate the bounding box of each entity. In our preliminary experiments, we observe that the synthetic image is generally similar to the original image, and thus propose to replicate the ground-truth bounding boxes in the original image to cover the corresponding regions in the synthetic image, which are regarded as the bounding box annotation of each entity in the synthetic sentence.To improve the quality of the synthetic labeled text-image pairs, we apply the following post-processing steps for data filtering: 1) We remove the text-image pairs whose text has less than 5 words. 2) We use the original training set D to train a base MNER or GMNER model and employ it to make predictions on the synthetic text-image pairs. If the predicted labels from the base model are inconsistent with the labels in the synthetic data, these text-image pairs will be removed. 3) We further remove the redundant textimage pairs with identical token and label sequences.After data filtering, we can obtain a set of synthetic labeled dataD 𝑔 = 𝒔 𝑔 𝑖 , 𝒗 𝑔 𝑖 , 𝒚 𝑔 𝑖 𝐾 𝑖=1, which is then combined with the original training set for model training in each downstream task. [41] for the MNER task and Twitter-GMNER [39] and Twitter-FMNERG [30] for the GMNER task. The basic statistics of each dataset are presented in Table 1 . Note that Twitter-FMNERG is an extension of the Twitter-GMNER dataset, which extends the four coarse-grained entity types to 51 finegrained entity types.In addition to evaluating the GMDA framework on the standard full-supervision setting, we also construct three low-resource settings by randomly sampling 10%, 20%, and 40% data from the full training and development sets as the training and development sets. The whole test set is kept for model evaluation. 
implementation details.: 
 For the Label-aware Multimodal Large Language Model (LMLLM), we employ the pre-trained In-structBLIP model released by Dai et al. [10] , in which we adopt FlanT5-XL(3B) [9] as the LLM and ViT-g/14 [14] as the image encoder. For the LoRA [18] adapter in LMLLM, we adopt a rank of 8 and a dropout rate of 0.1. The batch size and the learning rate are set to 2 and 5e-5, respectively. For Multimodal Image Generation, we utilize the pre-trained Stable Diffusion v1.5 model [28] , with a strength and guidance scale set to 0.8 and 10, respectively. All the models are implemented with PyTorch, and the Adam optimizer is adopted in the training stage. We run all the experiments on an NVIDIA RTX 3090 GPU.Following previous studies [39, 41] , we use Precision (Pre.), Recall (Rec.), and F1 Score (F1) as the evaluation metric for both MNER and GMNER tasks. The formula for computing the F1 Score is presented below:EQUATIONwhere Pre. refers to the proportion of correctly predicted entity tuples among all predicted entity tuples, and Rec. denotes the proportion of correctly predicted entity tuples among all the groundtruth entity tuples. Note that for entity and type predictions, the predictions are regarded as correct only if they exactly match the ground-truth labels. For the object prediction in GMNER, if the object is groundable, we regard the prediction as correct when the maximum IoU score between the predicted object and all the ground truth bounding boxes exceeds 0.5; otherwise, if the object is not groundable, the prediction is correct only if it is None.To evaluate the effectiveness of our GMDA framework, we consider two data augmentation methods for comparison: 1) Mix Generation (mixGen) [16] is a multimodal data augmentation method, which generates new imagetext pairs by linearly interpolating images and concatenating text sequences from two existing image-text pairs. 2) Easy Data Augmentation (EDA) [34] is a text-only data augmentation method, which randomly performs one of the following operations on the original sentences: synonym replacement, random insertion, random swap, and random deletion. Since EDA can solely generate the synthetic sentence without incorporating a synthetic image, we pair the synthetic sentences with their corresponding original images to obtain synthetic text-image pairs.To show the effectiveness of the synthetic labeled data generated by GMDA, we adopt several competitive methods for MNER and GMNER as the base models: 1) PGIM [21] is the current state-of-the-art method on the MNER task, which leverages ChatGPT as an implicit knowledge base to generate auxiliary knowledge to enhance the performance of MNER.2) MMT5 [30] formulates the MNER task as a paraphrase generation task and employs a pre-trained Seq2Seq model VL-T5 [8] to generate the entity-type pairs based on the textual and visual inputs. 3) H-Index [39] is a hierarchical index generation framework for GMNER, which generates the entity-type-region triplets in a hierarchical manner with a pre-trained Seq2Seq model BART. 4) TIGER [30] is a T5-based multimodal Generation framework for GMNER, which directly generates the paraphrased target sequence containing entity-type-region triples from an image-text input pair.In addition to the aforementioned base models, we further consider a number of representative MNER and GMNER methods for comparison in the full-supervision setting: 1) UMT [38] is a unified Transformer framework for MNER, which captures the intermodal interactions. 2) UMGF [40] is a unified multi-modal graph fusion approach for MNER. 3) MNER-QG [19] is an end-to-end MRC-based MNER method with query grounding. 4) R-GCN [42] is a relation-enhanced Graph convolutional network for MNER. 5) CAT-MNER [33] is a Transformer-based MNER framework, which refines the cross-modal attention with expanding entity label words. 6) ICL-MNER [3] explores the potential of the in-context learning paradigm for few-shot MNER. 6) GVATT-RCNN-EVG [23] , UMT-VinVL-EVG [38] , UMGF-VinVL-EVG [40] and ITA-VinVL-EVG [32] are sequence labeling-based multimodal approaches for GMNER, which stack the EVG model over existing MNER methods introduced by Yu et al. [39] .In Table 2 , we compare the results of two multimodal data augmentation methods, i.e., mixGen and the proposed GMDA model in different low-resource settings.Based on the first six rows of Table 2 , we can observe that the two multimodal data augmentation methods can generally bring improvements to the performance of the corresponding base models, especially in the extremely low-resource setting, i.e., only with 10% training data. Secondly, by comparing mixGen and GMDA, it is clear that using the augmented data generated from GMDA generally performs better. For example, in the 10% setting, when employing MMT5 as the base model, GMDA outperforms mixGen by 2.56 absolute percentage points in the F1 score. Lastly, for our GMDA framework, we can find that its performance improvement over PGIM is much smaller than that over MMT5. This is because the PGIM model leverages ChatGPT to generate auxiliary knowledge, which can perform much better than MMT5 in low-resource scenarios. Nevertheless, GMDA still improves the performance of PGIM by 1.7, 1.05, and 0.9 percentage points in 10%, 20%, and 40% settings, respectively. 2 , the comparison results on the GMNER task show similar trends to those on the MNER task. Specifically, for both H-Index and TIGER, using the data generated by GMDA brings consistent improvements in the F1 score in all the low-resource settings. For both Twitter-GMNER and Twitter-FMNERG datasets, the most significant improvements occur in the 10% setting, with an average F1 improvement of approximately 2% across different base models. Moreover, we find that in some cases, mixGen hardly yields any improvement. One possible reason is that linearly interpolating images and concatenating text sequences may disrupt the data distribution and thus introduce slight noise. In comparison to mixGen, our GMDA framework shows a more significant and consistent improvement, indicating that the augmented data generated from GMDA can well complement the original data. These observations demonstrate the efficacy of the proposed data augmentation framework in low-resource settings.In Table 3 and Table 4 , we further compare the results of different models in the full-supervision setting.Firstly, we can see from Table 3 that the base model MMT5 performs much better than most baseline MNER methods except CAT-MNER, while the other base model PGIM achieves the best performance among all the baseline methods. Secondly, when using synthetic data generated by mixGen, the F1 score even decreases, indicating that its data augmentation strategy may bring much noisy data. Lastly, it is easy to find that EDA and GMDA generally bring further improvements to the performance of MMT5 and PGIM. Concretely, EDA only obtains a very minor improvement (i.e., 0.06%) on both MMT5 and PGIM, while GMDA achieves 1.02 and 0.47 percentage points improvement in F1 score based on MMT5 and PGIM, respectively.Similar to the performance trend in the MNER task, we can observe from Table 4 that the proposed GMDA framework consistently enhances the performance of H-Index and TIGER on both Twitter-GMNER and Twitter-FMNERG datasets. In contrast, it is clear that in the full-supervision setting, the two data augmentation baseline methods, i.e., EDA and mixGen, even lead to a slight performance drop on the GMNER task, probably because their augmented data introduces excessive noise. This further demonstrates the advantage of GMDA over existing data augmentation methods.4.5.1 Ablation study. To investigate the effectiveness of each component in the proposed GMDA framework, we choose to conduct an ablation study of GMDA on the Twitter-GMNER dataset in a 10% low-resource setting by using H-Index as the base model. Specifically, we compare the full GMDA model with its three ablations: 1) removing the Multimodal Text Generation stage; 2) removing the Multimodal Image Generation stage; 3) following the common practice to only fine-tune the Q-Former without adding LoRA adapters in the LMLLM. As shown in Table 5 , we can see that all the components in GMDA play an indispensable role in the overall performance. Firstly, removing the Multimodal Text Generation stage will significantly drop the performance, which shows that text data augmentation can increase the diversity of the sentences and is essential to GMDA. Secondly, discarding the Multimodal Image Generation stage also leads to a performance drop, which indicates that generating the synthetic image for each synthetic sentence can reduce the noise that arises from inconsistencies between the synthetic text and the original image. Lastly, only fine-tuning the Q-Former of the LMLLM also leads to a decrease in performance. This indicates the necessity of incorporating LoRA adapters for parameter-efficient fine-tuning.To better understand the GMDA framework, we conduct a qualitative analysis of two synthetic examples generated from GMDA. For comparison, we show the synthetic data generated from mixGen and GMDA in Table 6 .For the Original Sample A, since mixGen essentially concatenates it with another sample mentioning building entities, the synthetic image tends to be noisy, containing unclear persons and buildings.In contrast, since our GMDA method revolves around the entity mentioned in the original sample, its synthetic sentence not only mentions the original entity, but also contains a different context, and the synthetic image is also quite relevant to the synthetic sentence.Similarly, for the Original Sample B, the mixture sample generated from mixGen contains both PER entities and OTHER entities, which makes the mixed image unclear and may introduce noise to the original data. By contrast, the sample generated from GMDA focuses on keeping the original entity in the generated sentence and preserving the important visual regions in the original image. Moreover, the GMDA method also generates a new entity London-BurlesqueTheater and assigns a OTHER label to the entity, which shows the diversity of the synthetic data generated by GMDA.To evaluate the quality of sentences generated by GMDA, we conduct additional experiments in full-supervision settings and report the results in Table 7 .Diversity. Generating diverse contexts for entities can generally improve the model's robustness in entity recognition. To show the diversity of the generated data, we propose to measure the cosine similarity between the synthetic sentence and the original sentence, and calculate a Diversity score [20] denoting the percentage of unique n-grams in all the synthetic sentences. It can be observed from the first six rows of Table 7 that for EDA and mixGen, the synthetic sentences are generally similar to the original sentences and the percentage of unique n-grams is relatively low, while our GMDA method can generate diverse synthetic sentences with many unique n-grams, mainly due to the probability-based sampling strategy.Perplexity. To evaluate the coherence of synthetic sentences, we further calculate the perplexity1 of data generated from each compared method based on a pre-trained language model GPT-2 with 𝑒 as the base of the exponential function. In the last four rows of Table 7 , we can observe that the perplexity of our GMDA framework is generally close to that of mixGen and the original sentences, while significantly lower than that of the EDA method. This shows that performing word replacement or insertion may disrupt the fluency of the original sentence and lead to high perplexity. In contrast, the Label-aware Multimodal Large Language Model in our GMDA method can help generate coherent sentences.These observations demonstrate the superiority of the proposed GMDA framework over existing data augmentation methods regarding the diversity and fluency of the generated sentences. 
conclusion: 
 In this paper, we proposed a Generative Multimodal Data Augmentation framework named GMDA to simultaneously generate synthetic text-image pairs and their corresponding high-quality entity annotations. GMDA contains two stages, i.e., a Multimodal Text Generation stage to generate synthetic labeled sentences with a Label-aware Multimodal Large Language Model (LMLLM) and a Multimodal Image Generation stage to generate the corresponding image for each synthetic sentence. Experiments on three benchmark datasets show that our GMDA framework consistently boosts the performance of several competitive methods for two subtasks of MNER in both standard and low-resource settings. Further analysis demonstrates the advantage of GMDA over existing data augmentation methods in terms of data diversity and fluency.  
 
 Appendix: https://huggingface.co/docs/transformers/perplexity 


Decision: The paper presents Generative Multimodal Data Augmentation to address the challenge of insufficient fine-grained annotated data in low-resource Multimodal Named Entity Recognition.
Comparisons with existing data augmentation methods demonstrate the effectiveness of the proposed text and image generation,
showing superior performance. The paper is well-written and the proposed method offers insightful thoughts to other researchers in the field. All reviewers lean towards accepting this paper.

Reviews: review: paper_summary: paper_summary: This work addresses the challenge of insufficient fine-grained annotated data in Multimodal Named Entity Recognition (MNER), particularly in low-resource scenarios. The authors propose a Generative Multimodal Data Augmentation (GMDA) framework that includes two stages: Multimodal Text Generation and Multimodal Image Generation. Initially, annotated sentences are converted into labeled sequences, and a Label-aware Multimodal Large Language Model (LMLLM) is trained to generate these sequences using label-aware prompts and associated images. Then, a Stable Diffusion model generates synthetic images related to the generated sentences. Experimental results on benchmark datasets show that the GMDA framework significantly improves the performance of MNER methods in both full-supervision and low-resource settings.
strength_weakness: strength_weakness: 1. The experiments were comprehensive. 
2. Good results were achieved on all three datasets.
limitations: limitations: 1. The authors did not provide source code for reproducibility. 
2. There is a lack of deeper analysis and explanation in comparison with other methods to highlight the advantages of this approach. 
3. The authors did not use a large model with a large number of parameters. Therefore, did they consider the effects of full fine-tuning or the impact of different PEFT methods on the model's performance before ultimately choosing LoRA?
4. In Table 3, are the results highlighted in bold all SOTA results?
paper_summary: paper_summary: This paper proposes a novel Generative Multimodal Data Augmentation (GMDA) framework for Multimodal Named Entity Recognition (MNER) to address the challenge of insufficient fine-grained annotated data, particularly in low-resource scenarios. Experimental results on three benchmark datasets show that GMDA effectively enhances the performance of several competitive MNER methods in both full-supervision and low-resource settings.
strength_weakness: strength_weakness: - The paper introduces a novel Generative Multimodal Data Augmentation (GMDA) framework, addressing the challenge of generating synthetic text-image pairs with high-quality entity annotations. This is a significant advancement in the field of Multimodal Named Entity Recognition (MNER) and Grounded Multimodal Named Entity Recognition (GMNER).
- Detailed comparisons with existing data augmentation methods and various MNER and GMNER baselines provide strong evidence of GMDA’s superior performance.
- The paper is clearly written, with a detailed explanation of the proposed GMDA framework, including its components and the experimental setup.
limitations: limitations: - Using the teacher forcing method during the training of the Label-aware Multimodal Large Language Model might result in lower quality generated sentences. Should strategies like Beam Search or Curriculum Learning be used to improve this?
- The ablation study on the effectiveness of each module should be conducted under diverse settings. Currently, it is only performed for H-Index with GMDA on the Twitter-GMNER dataset in the 10% low-resource setting.
paper_summary: paper_summary: The paper presents a novel framework called Generative Multimodal Data Augmentation (GMDA) for addressing the challenge of insufficient fine-grained annotated data in low-resource scenarios for Multimodal Named Entity Recognition (MNER). MNER is a crucial task in extracting structured information from multimodal content, such as text and images, found on social media platforms. The GMDA framework comprises two stages: Multimodal Text Generation and Multimodal Image Generation. It uses a Label-aware Multimodal Large Language Model (LMLLM) to generate labeled sequences based on prompts and associated images, followed by a Stable Diffusion model to generate semantically related synthetic images. The paper demonstrates the effectiveness of GMDA through experiments on three benchmark datasets, showing performance improvements for MNER in both full-supervision and low-resource settings.
strength_weakness: strength_weakness: 1. The approach proposed in this paper has brought certain improvements over previous data augmentation strategies on many baseline methods, demonstrating the effectiveness of the two-stage data generation process. 

2. The writing is fluent and there are no parts that are difficult to understand.
limitations: limitations: 1. The proposed scheme in this paper appears to require a significant consumption of additional resources, such as training resources for Large Language Models (LLMs) and inference resources for Stable Diffusion. Moreover, the authors also suggest the need to train a base Multimodal Named Entity Recognition (MNER) model to serve as a classifier for synthetic images, which seems to make the proposed solution overly complex. Compared to methods like MixGen that do not require additional resource consumption, the costs associated with this approach in terms of various aspects need further clarification.

2. The paper mentions that when the original image contains object box annotations, it will 'replicate the ground-truth bounding boxes in the original image to cover the corresponding regions in the synthetic image.' Does this operation imply that the corresponding areas in the synthetic image are refilled by the corresponding areas from the original image? If so, this could lead to visual discontinuities at the filled boundaries. For instance, in Figure 3, the original bounding box for Curry does not fully cover Curry's portrait position in the generated image, which could result in the repetition of some features of the person, such as the right arm.

3. In the framework illustrated in Figure 2 of the paper, the output of the Large Language Model (LLM) seems to differ little from the original text input. If the entity types and positions are known in the original text (which is the case in the training set), it is entirely possible to generate the corresponding target text using some heuristic rules. However, this is obviously not feasible in the test set. So, I would like to ask the authors whether they expect the LLM to learn to automatically perform Named Entity Recognition (NER) on the input text? If that's the case, then in the context of MNER, it could be possible to have the LLM complete this task without the need for a two-stage data augmentation approach to enhance the richness of the images.
paper_summary: paper_summary: The authors proposed a new data augmentation method with Multimodal Text Generation and Multimodal Image Generation for the task of MNER and GMNER. Extensive experiments show the proposed framework is effective.
strength_weakness: strength_weakness: The proposed data augmentation framework is well-motivated. The authors address the challenges properly. The paper is well-writen and easy to follow. The proposed framework could bring some insights for other researchers in the same filed.
limitations: limitations: The improvements under some specific settings (like in the 40% setting in Table 2) are not that much.

I have one important clarification question. In Section 3.5 (Line 512), the authors use the OORIGINAL training dataset D for filtering the synthetic data. Does that mean under the setting of 20%, you instead use the whole real training dataset? If so, this research cannot be regarded as under the LOW-RESOURCE setting because of the data leak. The authors should compare the proposed method with baselines which are trained using the whole training datatset. The final score would be changed regarding the answer of this question.

In addition, how many data are generated based on one real image-text pair? And how many data are left after filter? Does the proposed method sensetive to the number of generated data used? I hope there would be some empirical evidences regarding this.