Self-Adaptive Fine-grained Multi-modal Data Augmentation for Semi-supervised Muti-modal Coreference Resolution

Authors: li  zheng
boyu  chen
hao  fei
fei  li
shengqiong  wu
lizi  liao
donghong  ji
chong  teng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 17:38:23

Abstract: Coreference resolution, an essential task in natural language processing, is particularly challenging in multi-modal scenarios where data comes in various forms and modalities. Despite advancements, limitations due to scarce labeled data and underleveraged unlabeled data persist. We address these issues with a self-adaptive fine-grained multi-modal data augmentation framework for semi-supervised MCR, focusing on enriching training data from labeled datasets and tapping into the untapped potential of unlabeled data. Regarding the former issue, we first leverage text coreference resolution datasets and diffusion models, to perform fine-grained text-to-image generation with aligned text entities and image bounding boxes. We then introduce a self-adaptive selection strategy, meticulously curating the augmented data to enhance the diversity and volume of the training set without compromising its quality. For the latter issue, we design a self-adaptive threshold strategy that dynamically adjusts the confidence threshold based on the model’s learning status and performance, enabling effective utilization of valuable information from unlabeled data. Additionally, we incorporate a distance smoothing term, which smooths distances between positive and negative samples, enhancing discriminative power of the model’s feature representations and addressing noise and uncertainty in the unlabeled data. Our experiments on the widely-used CIN dataset show that our framework significantly outperforms state-of-the-art baselines by at least 9.57\% on MUC F1 score and 4.92\% on CoNLL F1 score. Remarkably, against weakly-supervised baselines, our framework achieves a staggering 22.24\% enhancement in MUC F1 score. These results, underpinned by in-depth analyses, underscore the effectiveness and potential of our approach for advancing MCR tasks.
Our codes: https://anonymous.4open.science/r/SLUDA.

Main paper: 
 introduction: 
 Coreference resolution (CR), which identifies all mentions referring to the same entity, plays a crucial role in many downstream applications, such as relation extraction [11, 49] , question answering [5, 46, 55] , and sentiment analysis [15, 24, 53, 56, 57] . However, in real-world scenarios, particularly in social media, data often comes in various forms and modalities [16, 28, 29, 43, 44] , including text, images, rather than pure text. Therefore, some recent advancements have transferred to multimodal coreference resolution [17, 18] , where coreference occurs not only between the entities in text but also the objects in image, as shown in Figure 1 (a). Compared to the text-based coreference resolution, Multimodal Coreference Resolution (MCR) presents even greater challenges due to the substantial semantic gap between different modalities and the scarcity of annotated data.Existing works have made commendable efforts in MCR. Goel et al. [18] proposed a multimodal pipeline and utilized weak supervision to identify coreference chains. In contrast, another study by Goel et al. [17] argued that weak supervision alone is unable to resolve the ambiguity of multiple instances of the same object class. To address this issue, they introduced semi-supervised learning and employed cross-modal attention to integrate image region features with textual features. Despite these advances, current methods in front of the image there is a person wearing a cap and she is sitting on the blanket. there is a cloth on the wooden object. there is another person holding a wooden stick in her hand. there are a few objects in a cover. there are some objects. there are doors. there are windows on the wall. in the background of the image there is a person holding some object in her hand. there is a net on the tables. there is a helmet on the bike. in MCR still tied to model optimization and extracting information features. Data, as a key dimension that can significantly affect model performance, has been far less explored. Therefore, we summarize the challenges facing current research from a data-centric perspective in two aspects:• Scarcity of labeled data. The manual annotation of large-scale fine-grained multimodal datasets is costly, resulting in a scarcity of extensively labeled data required for research on multimodal coreference resolution. This scarcity poses significant challenges in training MCR models. Currently, automatic data expansion [8, 23] is widely used as a practical technique to alleviate the data scarcity problem. Xiao et al. [47] applied the stable diffusion [35] to generate high-quality images with a given text input to solve the image captioning task, but the method is limited to coarse-grained expansion of text-image pairs, ignoring fine-grained objects, entities, etc. For the coreference resolution task, existing methods [10] mainly focus on textual coreference resolution data augmentation, while the MCR data augmentation remains largely unexplored. Therefore, how to adaptively expand high-quality fine-grained multimodal labeled data is the essence of successful MCR.• Under-exploitation of unlabeled data. On the other hand, despite addressing the fundamental issue of scarcity of labeled data through adaptive data augmentation, the untapped potential of unlabeled data remains underexplored. In semi-supervised learning, existing methods [17] set a fixed threshold to ensure the quality of pseudo-labels for unlabeled data. Unfortunately, a fixed high threshold (e.g., 0.9) poses several limitations: (1) Loss of useful information: By using a fixed high threshold, many unlabeled data that provide valuable insights are filtered out, potentially discarding useful samples. (2) Limitation on the quantity of training data: With a fixed high threshold, only a limited number of unlabeled data are selected, restricting the scale of training data. (3) Ignoring the model's learning status: A fixed threshold ignore the dynamic nature of training process as it cannot dynamically adjust the threshold based on the model's learning status. Therefore, it is crucial for the MCR task to effectively harness valuable information from unlabeled data based on the learning status of the model in order to train more robust and accurate MCR models.Taking into consideration the two aforementioned aspects, we propose a novel framework: Self-adaptive Labeled and Unlabeled multimodal Data Augumentation (SLUDA), to improve semi-super vised MCR. Firstly, as depicted in Figure 1 (b), we utilize a text CR dataset [32] and a diffusion model [27] , as well as randomly generated bounding boxes, to generate text-image pairs and object-entity pairs for fine-grained data expansion. Subsequently, we design a self-adaptive selection strategy to filter and select the generated multimodal data and combine it with existing labeled data for model training. This enables us to increase the quantity and diversity of data while ensuring quality, achieving the first goal of data augmentation for labeled data. Next, we adopt a self-adaptive threshold strategy to fulfill the second goal of fully tapping into the unlabeled data. By dynamically adjusting the confidence threshold based on the model's learning status and performance, we estimate the threshold based on the exponential moving average (EMA) of confidence scores from unlabeled data. This allows for the utilization of a lower threshold during the early stage of training, and helps the model to accelerate convergence. As training progresses, the threshold gradually increases to filter out unreliable pseudo-labels and enhance the accuracy of the model. Furthermore, considering that pseudo-labeled data are unavoidably noisy than manuallylabeled data, we introduce a distance smoothing term. It smooths the distances between positive and negative samples, aiding the model in learning more discriminative feature representations and improving the utilization of unlabeled data.To verify the effectiveness of our model, we conduct experiments on the benchmark CIN dataset [18] . The results demonstrate that our framework significantly helps the state-of-the-art model to further improve its performance by at least 9.57% on MUC F1 score and 4.92% on CoNLL F1 score. Most strikingly, compared to the weakly-supervised baselines, ours outperforms their MUC F1 score by 22.24%. Further ablation experiments demonstrate that each component of our framework is essential. Additionally, we conduct extensive experiments to demonstrate the effectiveness of adaptive selection and utilization of labeled and unlabeled data in our framework. Moreover, we focus on smoothing distance differences between positive and negative samples, promoting more accurate and reliable feature representation learning. Our main contributions are summarized as follows:• We propose self-adaptive expansion and selection techniques to enrich the training data with generated high-quality MCR data, effectively addressing the issue of labeled data scarcity.• We devise a self-adaptive threshold strategy to leverage unlabeled data, achieving a balance between quality and quantity. Moreover, we introduce a distance smoothing policy to enhance the discriminative ability of feature representations.• Our extensive experimental results on the CIN dataset demonstrate that our SLUDA framework achieves the state-of-theart performance and outperforms the best baseline with large margins. 
related works 2.1 multimodal coreference resolution: 
 Within the multimodal learning community [12-14, 26, 42, 45, 54] , there has been growing interest in multimodal coreference resolution. Several studies [17] [18] [19] have been conducted on this topic. Goel et al. [18] introduced the CIN dataset, which addresses the problem of coreference resolution in long narratives within visual scenes. They proposed a multimodal pipeline and utilized weak supervision to learn to identify coreference chains. Goel et al. [17] proposed a semi-supervised training approach based on a fixed threshold to effectively learn from unlabeled sets. However, despite achieving partial success, these methods suffered the issue of scarce labeled data and failed to fully leverage the potential of unlabeled data.Semi-supervised learning is an effective paradigm that utilizes a large amount of unlabeled data along with limited labeled data. Various methods [9] have been proposed in the field of semi-supervised learning, such as pseudo-labeling and consistency regularization.In pseudo-labeling, the models [34, 48] use unlabeled samples with high confidence as training targets, thereby reducing the density of data points at the decision boundary. In consistency regularization, the models [2, 4] are self-supervised on unlabeled data, providing additional training signals. To mitigate the confirmation bias in pseudo-labeling, Goel et al. [17] proposed a threshold-based technique to ensure the quality of pseudo-labels, where only unlabeled data with confidence above the threshold are retained. While promising results have been achieved, a fixed high threshold can lead to the disregard of a large number of ambiguously predicted unlabeled examples, especially in the early and mid-training stages.Multimodal data augmentation is the process of creating additional training data with high quality and diversity. It has been widely applied in various machine learning tasks [1, 20, 50, 58] . Recently, Wang et al. [40] proposed a framework for paired cross-modal data augmentation, which generates an infinite amount of paired data to train cross-modal retrieval models. Xiao et al. [47] utilized stable diffusion models for text-to-image generation, expanding the training set with high-quality image-caption pairs. In this paper, we employ a diffusion model [27] to augment fine-grained labeled data, addressing the issue of scarce labeled data, enhancing the diversity and richness of the MCR dataset, and improving the generalization and performance of MCR models. 
methodology: 
 In this paper, we propose a Self-adaptive Labeled and Unlabeled multimodal Data Augumentation (SLUDA) framework that fully utilizes data from two perspectives to address the MCR task. The architecture is shown in Figure 2 . First, we generate fine-grained image-caption pairs and adaptively selects high-quality generated MCR data for labeled data augmentation. Then, we design a selfadaptive threshold strategy, considering factors such as data quality and model confidence, and introduce a distance smoothing term to enhance the utilization of unlabeled data. Finally, with preprocessed multimodal labeled and unlabeled data, we employ two modality-specific Variational Autoencoders (VAEs) to learn latent representations for different modalities and perform the MCR task.Let (𝐼, 𝐶) represent an image-caption pair, where 𝐶 describes the image 𝐼 as shown in Figure 1 (a). We denote the set of 𝑝 mentions as 𝑀 = {𝑚 1 , 𝑚 2 , ..., 𝑚 𝑝 }, and the image 𝐼 contains 𝑞 regions denoted as 𝐼 = {𝑟 1 , 𝑟 2 , ..., 𝑟 𝑞 }. The objective of multimodal coreference resolution is to identify text mentions that refer to the same entity and associate each mention with a specific region in the image.In the task of multimodal coreference resolution, a large number of fine-grained annotated image-caption pairs are typically required.Existing MCR datasets, such as CIN [18] , demand human annotators to label entity mentions referring to the regions in the referenced image, construct coreference entity chains, and draw bounding boxes in the image. This annotation process is not only labor-intensive but also time-consuming. Furthermore, the collected images and annotated mentions may suffer from incompleteness and lack diversity, which limits the generalization capabilities of models trained on such datasets. To address the challenge of scarce labeled data, we propose a novel approach that leverages existing text-based coreference resolution datasets in conjunction with a diffusion model to generate images, and then adaptively select high-quality multimodal data. By doing so, we mitigate the limitations posed by data scarcity in the MCR task. Labeled Data Generation. The diffusion models [36, 47, 51 ] have been proposed and applied for data augmentation, which is a generative model that generates diverse samples by iteratively diffusing noise signals. For the task of grounded text-to-image generation, Li et al. [27] introduced GLIGEN, which builds upon existing pre-trained text-to-image diffusion models and extends their capabilities to incorporate grounding inputs as conditions. We employ GLIGEN and leverage the widely-used text coreference resolution dataset English OntoNotes 5.0 [32] as the foundational dataset for data augmentation. Given the text as the caption 𝐶 = {𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 }, a set of mentions 𝑀 = {𝑚 1 , 𝑚 2 , . . . , 𝑚 𝑝 }, and their randomly generated corresponding bounding boxes 𝐵 = {𝑏 1 , 𝑏 2 , . . . , 𝑏 𝑞 } as input, we aim to generate the image 𝐼 :EQUATIONwhere 𝐼 = {𝑟 1 , 𝑟 2 , ..., 𝑟 𝑞 } represents the image 𝐼 with 𝑞 regions. We discard images that contain NSFW (Not Safe for Work) content. Then, we pair the generated images with the corresponding caption to form an augmented labeled dataset.Self-adaptive Selection. Due to the uncertain quality of the generated images, we employ a self-adaptive selection strategy to filter the data based on the quality assessment of the image-caption pairs, ensuring both the quality and quantity of the training set. Specifically, since CLIPScore [21] is used to assess whether the generated image can match and correlate with the given textual description, we evaluate the generated image-caption pairs using Original Labeled Data D 𝑙' in front of the image there is a person wearing a cap and she is sitting on the blanket. there is a cloth on the wooden object. there is another person holding a wooden stick in her hand. there are a few objects in a cover. there are some objects. there are doors. there are windows on the wall. in the background of the image there is a person holding some object in her hand. there is a net on the tables. there is a helmet on the bike. As training progresses, we filter out a portion of low-scoring generated data based on the model's learning status to maintain the quality of the labeled data. The learning status of the model is estimated using the predicted scores, which are then smoothed using Exponential Moving Average (EMA) [31] . EMA calculates the current score as a weighted average of the current and previous EMA values, providing a more stable reflection of the model's learning status. The filtering ratio 𝜏 𝑟 𝑡 is defined as follows:EQUATIONwhere 𝛼 1 is the smoothing factor (0 ∼ 1), 𝑜 𝑟 𝑡 represents the prediction accuracy at time 𝑡, 𝛽 serves as a control parameter for the filtering intensity. When the EMA value is higher, indicating better overall predicted scores and superior performance of the model in handling the task, we filter out a larger proportion of generated data to ensure the quality of labeled data. Conversely, when the model's learning status is poorer, we filter out a smaller amount of generated data, allowing for the retention of a larger number of samples for training. This approach enables a balance between the quality and quantity of the training set throughout the training process, effectively leveraging the labeled data to enhance the model's performance. This generated and adaptively filtered labeled dataset is denoted as 𝐷 𝑙 ′′ , which is merged with the labeled dataset 𝐷 𝑙 ′ from the multimodal dataset CIN. Therefore, the final labeled dataset is 𝐷 𝑙 = 𝐷 𝑙 ′′ + 𝐷 𝑙 ′ .From the perspective of unlabeled data, existing method [17] fails to fully utilize the potential of unlabeled data, resulting in the waste of unlabeled data and a decrease in the robustness of MCR task. Moreover, unlabeled data often contains noise, and reducing the impact of noise on the model while promoting more accurate and reliable feature representation learning is crucial for MCR. Thus, we propose self-adaptive threshold and distance smoothing techniques to effectively leverage unlabeled data and mitigate noise within it.Self-adaptive Threshold. Considering that the fixed high threshold method results in the wastage of numerous samples, the samples with ambiguous predictions contribute negligibly during the training phase. Hence, to harness unlabeled data effectively, we propose a self-adaptive threshold strategy that reflects the learning status of the model. The strategy estimates the effectiveness of learning based on the model's prediction confidence and subsequently calculates the threshold 𝜏 𝑡 using the exponential moving average of the confidence. The threshold 𝜏 𝑡 is defined and adjusted as follows:EQUATIONwhere 𝑡 represents the time step of training, 𝑜 𝑡 represents the prediction accuracy at time 𝑡, and 𝛼 2 is the smoothing factor (0 ∼ 1). By employing this strategy, we can use a lower threshold in the early stages to leverage unlabeled data and accelerate the model convergence. As the training progresses, we gradually increase the threshold to filter out unreliable pseudo-labels and improve the model's accuracy. For unlabeled data, we focus on using crossentropy loss with a confidence threshold for entropy minimizationbased pseudo-labeling:EQUATIONwhere 𝑝 (•) is the output probability from the model and I is the indicator function for confidence-based thresholding with 𝜏 being the threshold. Distance Smoothing. To address the issue of noise in unlabeled data, we introduce a distance smoothing term to smooth the distances between positive and negative samples. This approach aims to facilitate the model's ability to learn more discriminative feature representations and maximize the utilization of unlabeled data. First, we compute the cosine similarity scores between each pair of samples to determine whether they are positive or negative samples. For each mention 𝑚, we compare its similarity score with another mention 𝑚 ′ and use a predefined threshold to determine their labels. If the similarity score exceeds the threshold, it is labeled as a positive sample. Otherwise, it is labeled as a negative sample.Next, for each pair of positive samples (𝑚 𝑖 , 𝑚 𝑗 ), we calculate their Euclidean distance 𝑑 𝑖 𝑗 = ||𝑚 𝑖 -𝑚 𝑗 ||. Similarly, for each pair of negative samples (𝑚 𝑖 , 𝑚 𝑘 ), we calculate their Euclidean distance 𝑑 𝑖𝑘 = ||𝑚 𝑖 -𝑚 𝑘 ||. To smooth the distance differences between positive and negative samples, we introduce a smoothing function 𝑔(𝑑 𝑖 𝑗 , 𝑑 𝑖𝑘 ) defined as:EQUATIONThis smoothing function maps the distances between positive and negative samples to a smooth value between 0 and 1, providing a smoothing effect by reducing the distance disparity. Finally, we incorporate the distance smoothing term into the loss function to encourage the model to learn discriminative feature representations and effectively utilize unlabeled data. The formula for the loss function is as follows:EQUATIONwhere P is the positive set, N is the negative set. The parameter 𝜆 controls the separation between positive and negative samples. We design this loss function to narrow the gap between positive examples and widen the gap between positive and negative examples. The smoothing function 𝑔(𝑑 𝑖 𝑗 , 𝑑 𝑖𝑘 ) has a value close to 1 when the difference between 𝑑 𝑖 𝑗 and 𝑑 𝑖𝑘 is large, increasing the difference in distance between positive and negative samples. When the difference between 𝑑 𝑖 𝑗 and 𝑑 𝑖𝑘 is small, the value of 𝑔(𝑑 𝑖 𝑗 , 𝑑 𝑖𝑘 ) is close to 0, decreasing the distance difference between positive and negative samples. By employing the distance smoothing method, we are able to smooth the distance disparities between positive and negative samples, reduce the impact of noise on the model, and facilitate more accurate and reliable feature representation learning, thereby enhancing the model's performance.The details of our model are shown in Figure 3 . For fair comparison, following Goel et al. [17] , we utilize BERT [7] as the text encoder to obtain contextualized word representations and Faster RCNN [33] as the visual encoder to obtain region representations of images. To effectively integrate textual and visual features, we employ two modality-specific modality-specific Variational Autoencoders (VAEs) to obtain latent representations of the two modalities' features. Then, by applying Product of Experts (PoE) [22] to the latent representations of both modalities, we obtain the multimodal representation.Text Feature Extraction. We employ BERT to encode each word in the narrative caption 𝐶 = {𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 }, as mentioned in Goel et al. [17] . We prepend a [CLS] token and append a [SEP] token to each sentence, resulting in𝐶 0 = {[𝐶𝐿𝑆], 𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 , [𝑆𝐸𝑃]}.We then concatenate them together as the input to BERT to generate contextualized token representations. The mention embeddings 𝑀 = {𝑚 1 , 𝑚2, . . . , 𝑚 𝑝 } are computed by averaging the corresponding word embeddings.Image Feature Extraction. Following Goel et al. [17] , we employ Faster RCNN to extract region representations of the image 𝐼 = {𝑟 1 , 𝑟 2 , . . . , 𝑟 𝑞 }. Each region 𝑟 𝑖 is represented by a d-dimensional joint embedding 𝑟 𝑖 ∈ R 𝑑 , which encompasses its visual, semantic, and spatial features.Multimodal Feature Fusion. Previous research [17] has treated text and image features equally, mapping the concatenated features of both modalities to the same latent representation. However, there exist mismatches between text and image, introducing noise into the models used for prediction. To address this, we utilize modalityspecific VAEs to map the features of the two modalities into their respective latent representations with internally correlated distributions. This enables the model to adaptively learn its specific data representation for each modality, better capturing the intrinsic relationships and structures among the features of each modality, helping to address the mismatch between text and image. The encoder of each VAE includes dense layers that map the input features to mean vectors 𝜇 and standard deviation vectors 𝜎.For the text modality, the text feature 𝑐 is inputted into the encoder of the text-VAE to parameterize the mean vector 𝜇 𝑐 and the standard deviation vector 𝜎 𝑐 . We can approximate the true posterior distribution 𝑝 (𝑧 𝑐 |𝑐) using these parameters, where the distribution of 𝑧 𝑐 can be represented asEQUATION𝑐 ). For the visual modality, the image feature 𝑟 is also inputted into the encoder of the image-VAE to obtain an approximation of the true posterior distribution 𝑝 (𝑧 𝑟 |𝑟 ), where the distribution of 𝑧 𝑟 can be represented asEQUATION𝑟 ). Based on the assumption of conditional independence of multimodal latent representation, the latent distribution 𝑝 (𝑧 𝑚 |𝑐, 𝑟 ) of the multimodal representation can be simplified as the combination of two separate latent distributions 𝑝 (𝑧 𝑚 |𝑐) and 𝑝 (𝑧 𝑚 |𝑟 ). Thus, we apply the product-of-experts (PoE) to estimate the multimodal latent distribution:EQUATIONWe assume that the latent representations are independent and follow Gaussian distributions parameterized by mean and standard deviation. Therefore, the distribution of 𝑧 𝑚 can be represented as 𝜎 2 𝑐 + 1 𝜎 2 𝑟 ) -1 .The latent variable 𝑧 𝑚 of the multimodal representation can be computed as 𝑧 𝑚 = 𝜇 𝑚 + 𝜎 2 𝑚 ⊙ 𝜖 , where 𝜖 ∼ 𝑁 (0, 𝐼 ). Training. After obtaining preprocessed multimodal labeled and unlabeled data, we feed them into our model to learn latent representations of different modalities and perform coreference resolution. We calculate the score for the multimodal mention pair (𝑧, 𝑧 ′ ) as𝑆 𝑚𝑐𝑟 = 𝑧•𝑧 ′ |𝑧 | |𝑧 ′ | .This score measures the likelihood of coreference between 𝑧 and 𝑧 ′ . A higher score indicates a stronger indication of coreference, whereas a lower score suggests non-coreference. We train the coreference scorer by utilizing the negative log marginal likelihood (L 𝑀𝐿𝐿 ). Finally, the total loss function is the summation of all the aforementioned losses:EQUATION 
experiments 4.1 experimental setting: 
 Datasets. We evaluate our approach on the CIN dataset [18] , which consists of 1,000 test and 880 validation image-caption pairs. Following Goel et al. [17] , we use the annotations from the validation split of the CIN dataset as a small labeled set during training. To augment our data, we utilize the widely used text coreference resolution dataset English OntoNotes 5.0 [32] , which contains 3,493 documents. After carefully filtering the documents, we obtain 3,116 multimodal coreference examples as labeled data.Evaluation Metrics. In terms of evaluation metrics, we align with Goel et al. [17] and use CoNLL F1 for coreference resolution evaluation, which is the average F1 score across three metrics: MUC [39] , 𝐵 3 [3] , and 𝐶𝐸𝐴𝐹 𝑒 [30] . For narrative grounding, we consider a prediction to be correct if the Intersection over Union (IoU) between the predicted bounding box and the ground truth box is greater than 0.5 [18] .To validate the effectiveness of our model, we compare it against the following state-of-the-art baselines:• VL-BERT: Su et al. [37] developed VL-BERT with rich aggregation and alignment functions.• UNITER: Chen et al. [6] introduced UNITER, which uses Optimal Transport to align words to image regions.• VinVL: Zhang et al. [52] developed an object detection model to provide object-centric representations of images.• MAF: Wang et al. [41] leveraged fine-grained visual representations to model phrase-object correlation.• WS-MCR: Goel et al. [18] employed weak supervision to learn to identify coreference chains.• Semi-MCR: Goel et al. [17] proposed a semi-supervised approach and designed a multimodal fusion model for MCR.We comprehensively compare our ALUDA against text-based coreference and multimodal coreference methods on the CIN dataset. The results, presented in Table 1 , highlight the superiority of our method over the state-of-the-art (SoTA) baselines and reveals several key findings. Firstly, compared to traditional text-based coreference methods, multimodal methods consistently demonstrate superior performance by leveraging additional visual features. But without carefully incorporating visual information into the task, such as Vi-sualBERT, UNITER and VinVL models, they only achieve marginal improvements over text-based methods. MAF and WS-MCR are two weakly supervised methods, and both of them exhibit significant improvements in MUC scores compared to other single-modal and multimodal baselines. By employing the semi-supervised learning method and carefully tuning the model with a small amount of labeled data and a large amount of pseudo-labeled data, Semi-MCR achieves the current state-of-the-art (SoTA) results. Most notably, our model significantly surpasses the state-of-the-art by a substantial margin, with a CoNLL F1 improvement of 4.92%, an MUC F1 improvement of 9.57%. Moreover, compared to MAF and WS-MCR, our method demonstrates remarkable enhancements, with MUC F1 improvements of 25.78% and 22.24%, respectively.To evaluate the alignment between image regions and phrases in textual data, we set up a narrative grounding task and conduct a comprehensive comparison between our proposed method existing SoTA approaches, including MAF, WS-MCR, and Semi-MCR. The results in Table 2 demonstrate the superiority of our method. Compared to the current state-of-the-art approach (Semi-MCR), our method achieve a 4.26% and 3.69% improvement in noun phrase and pronoun coreference accuracy, respectively. It is worth noting that our proposed method outperforms the MAF, with a 15.24% and 13.83% increase in noun phrase and pronoun coreference accuracy, respectively. These results highlight the advantages of our method and emphasize the necessity of expanding labeled data and effectively leveraging unlabeled data in scenarios where large-scale annotated data is lacking.We perform ablation experiments to evaluate the contribution of each component in our model. As depicted in Table 3 , no variant matches the full model's performance, highlighting the indispensability of each component. Specifically, the absence of labeled data augmentation (LDA) results in the most significant performance drop. The CoNLL F1 and MUC F1 scores decreased by 3.12% and 4.51%, respectively, indicating the crucial role of expanding highquality labeled datasets in increasing data diversity and model performance. To validate the necessity of self-adaptive threshold (SAT), we remove it and set the threshold to 0.9. The sharp decrease in performance (2.37% drop in CoNLL F1) demonstrates the nonnegligible impact of self-adaptive threshold in the semi-supervised MCR task. Furthermore, removing the smoothing distance term (SDT) lead to a significant performance drop, indicating the importance of it in reducing noise in unlabeled data and helping the model learn more discriminative feature representations, thus improving the utilization of unlabeled data. To further investigate the effectiveness of our method, we conduct in-depth analyses to answer the following questions, with the aim to deeply mine the intuition and analyze implicit phenomena.Q1: What are the advantages of the self-adaptive threshold?We validate the effectiveness of our self-adaptive threshold by comparing it with different fixed thresholds in Figure 4 . The results indicate that the self-adaptive threshold consistently outperforms the fixed thresholds across all four evaluation metrics. This suggests that the self-adaptive threshold strikes a balance between data quality and quantity, enabling a more flexible selection of unlabeled data. Additionally, we observe that when no filtering of pseudo predictions is applied (i.e., setting the threshold to 0.0), the model performs poorly. This highlights the necessity of a threshold-based training strategy. When consistently using lower fixed threshold, such as 0.5, the model shows some improvement compared to including all pseudo predictions, emphasizing the importance of removing noisy samples to a certain extent. While setting higher fixed threshold, like 0.95, the model's performance does not improve compared to a threshold of 0.9, in fact, it even declines. This intriguing phenomenon suggests that excessively filtering out unlabeled data result in the loss of high-quality samples, thereby limiting the model's learning capacity.Q2: How to evaluate the quality of generated images? We conduct a quality evaluation of the images generated by the diffusion model to quantify and assess the quality of the generated images.To find images that better match the caption and evaluate without relying on real reference images and text, we choose two metrics, MUSIQ and CLIPScore, to evaluate the quality of our generated images. MUSIQ is used to assess the quality of the images themselves, while CLIPScore is used to evaluate whether the generated images can match and correlate with the given textual descriptions. The evaluation results, as shown in Figure 5 , reveal that the generated images (New) achieved high scores on both the MUSIQ and CLIP-Score metrics, approaching those of original images (Old). These results indicate that the generated images are of high quality and align well with the given textual descriptions.Q3: What are the advantages of the self-adaptive selection strategy? To validate the effectiveness of the self-adaptive selection strategy, we compare the results of it with different fixed ratio selection strategies, as shown in Table 4 . From the evaluation of image quality, we observe that the overall image quality is high, but there are still a few low-quality images. Thus, we sort the generated images based on their CLIPScore scores and select a portion of them to merge into our training set according to a certain ratio. We find that our self-adaptive selection strategy outperforms all the fixed ratio data selection strategies. We also observe that using all the generated image-caption pairs for the training set does not yield the best results. This might due to the presence of lower-quality images among the generated samples, necessitating the need for filtering based on certain criteria. Additionally, when using lower ratios of generated data, such as 25% and 50%, we notice a slight decrease in overall performance. This suggests that, apart from highquality samples, the quantity of samples in the training set is also crucial. In conclusion, the above analysis highlight the importance of quality filtering and emphasizes the need to find the optimal balance between high-quality samples and dataset size.Q4: What is the role of smooth distance term in performance improvement? We visualize the sample features using t-SNE to reveal the effectiveness of smooth distance term. In Figure 6 (a), when employing smooth distance term, the feature differences between positive and negative samples are highly evident. This indicates that the model is encouraged to learn distinct feature representations and achieves better discrimination between positive and negative samples. Conversely, in Figure 6 (b), when smooth distance term is removed, the feature distinctions between positive and negative samples are not significant. This limitation hampers the model's ability to learn discriminative feature representations. This emphasizes the importance of smooth distance term in addressing noise issues in unlabeled data and enhancing model performance. 
conclusion: 
 In this paper, we explore two major challenges in the semi-supervised MCR task: scarcity of labeled data and under-exploitation of unlabeled data, and propose a solution called SLUDA. SLUDA expands high-quality MCR training data through a self-adaptive expansion selection strategy, and makes full use of the unlabeled data through a self-adaptive thresholding strategy. Through experiment evaluation, we have found that all of our claimed innovative approaches and hypotheses have been demonstrated to be effective. One of the most interesting findings is that the performance of the model in the semi-supervised MCR task can be significantly improved by adaptively adjusting the threshold value or ratio according to the learning status of the model, both for unlabeled and labeled data. This finding provides insight into our understanding of the role of data quality and quantity in MCR, and offers great promise for advancing further development of this task. 
 
 Appendix:  


Decision: This paper introduces a self-adaptive fine-grained multi-modal data augmentation method (SLUDA) to improve semi-supervised multi-modal coreference resolution (MCR) by generating synthetic labeled data and leveraging both labeled and unlabeled data. The approach effectively tackles data scarcity and enhances model performance on the CIN dataset, showing significant improvements over existing baselines. The strengths of the paper include its innovative application of data augmentation techniques to multi-modal data, clear organization, and impressive performance gains. However, the evaluation is limited to a single dataset, and the paper would benefit from additional benchmarks to generalize its effectiveness. Furthermore, while the data augmentation approach is creative, it lacks detailed analysis and visualizations of the generated data. The self-adaptive threshold strategy and distance smoothing term are promising but require more rigorous validation and clearer formulation. Addressing these concerns in the camera-ready version by including more datasets, comprehensive analysis, ablation studies, and comparisons with recent large pre-trained multi-modal models would strengthen the paper's contributions.

Reviews: review: paper_summary: paper_summary: This paper focuses on multi-modal corresponding resolution (MCR) tasks. Due to the data scarcity issues, the authors propose a self-adaptive fine-grained multi-modal data augmentation method by  utilizing diffusion model to generate synthetic labeled data based on a few labeled data and exploiting MCR model trained on both original and synthetic labeled data to generate pseudo labels on unlabeled data. They evaluated their proposed method on CIN datasets and verified the effectiveness of this method.
strength_weakness: strength_weakness: 1. Although the basic idea is not very novel in data augmentation methods, its application on multi-modal data still needs more efforts. 
2. The paper is clearly written and well organized.
limitations: limitations: 1. There are only one dataset used in the experimental evaluation, which can not fully verify the effective of this method on other datasets. More benchmarks would be better.
2. Since the MCR task is relatively complex, a direct analysis (e.g., case study with both image and text, error analysis) is necessary to provide more insights for future researches.
paper_summary: paper_summary: 1. The authors propose SLUDA to address the Multimodal Coreference Resolution (MCR) task by enhancing training data from both labeled and unlabeled perspectives. The framework generates fine-grained image-caption pairs and selects high-quality data for labeled data augmentation, while a self-adaptive threshold strategy and distance smoothing term are used to utilize unlabeled data effectively.

2. The SLUDA framework employs two modality-specific Variational Autoencoders (VAEs) to learn latent representations for different modalities. Experimental results on the CIN dataset show that SLUDA achieves state-of-the-art performance, significantly outperforming existing baselines. It improves the MUC F1 score by at least 9.57% and the CoNLL F1 score by 4.92%, with a remarkable 22.24% enhancement against weakly-supervised baselines.
strength_weakness: strength_weakness: 1. The idea of using self-adaptive methods for both labeled and unlabeled data augmentation is really impressive. It’s like the framework is smart enough to know what data to use and how to use it, ensuring the quality and diversity of the training data (which is crucial for effective learning).

2. The performance improvements are quite remarkable. Achieving over a 22% boost in the MUC F1 score compared to weakly-supervised baselines is no small feat. This shows that the framework doesn't just look good on paper but actually delivers significant real-world benefits (which is what we all hope for in new methodologies).
limitations: limitations: 1. In section2.2, "labeled data may not always be sufficient due to the 224 cost of annotation", there may lack of related work to present in Section 2.2 , especially considering the task of " tapping into the unlabeled data" in the first stage of this paper:

[1] Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

[2] S-clip: Semi-supervised vision-language learning using few specialist captions

[3] Gpt self-supervision for a better data annotator

[4] OpenAnnotate2: Multi-Modal Auto-Annotating for Autonomous Driving

2. The framework relies heavily on synthetic data generation for labeled data augmentation. While this approach helps mitigate the issue of scarce labeled data, it might not fully capture the complexity and variability of real-world data. This could limit the model’s generalizability when applied to entirely new or diverse datasets.

3. The use of self-adaptive techniques and multiple VAEs adds a layer of complexity to the implementation. This could pose challenges for practitioners who might not have the resources or expertise to replicate the results.
paper_summary: paper_summary: This paper proposes a self-adaptive fine-grained multi-modal data augmentation framework called SLUDA for semi-supervised multi-modal coreference resolution (MCR). To address the scarcity of labeled data, the authors leverage text coreference resolution datasets and diffusion models to generate fine-grained text-to-image data with aligned text entities and bounding boxes. They introduce a self-adaptive selection strategy to curate the augmented data. To better utilize unlabeled data, they design a self-adaptive threshold strategy that dynamically adjusts the confidence threshold based on the model's learning status. Additionally, a distance smoothing term is incorporated to enhance the discriminative power of the model's feature representations. Experiments on the CIN dataset show that their framework significantly outperforms state-of-the-art baselines.
strength_weakness: strength_weakness: 1. The paper addresses important challenges in MCR, namely the scarcity of labeled data and under-exploitation of unlabeled data.

2. The proposed self-adaptive fine-grained data augmentation approach leveraging text datasets and diffusion models is novel and effective for expanding labeled MCR data.

3. The self-adaptive threshold strategy for utilizing unlabeled data based on the model's learning status is a thoughtful approach to balance data quality and quantity.
limitations: limitations: 1. The paper lacks a discussion and comparison with recent multi-modal large pretrained models (e.g., CLIP, ALIGN) which have shown strong zero-shot and few-shot capabilities on various multi-modal tasks. It would be valuable to see how the proposed approach compares to or could be combined with such models.

2. While the data augmentation approach is innovative, more details and analyses could be provided. For example, how is the quality of the generated data evaluated quantitatively? How much diversity does it add to the training set? Some visualizations of the generated samples would help readers better understand the approach.

3. The self-adaptive threshold strategy is interesting but lacks ablation studies to validate its effectiveness. It would be good to see how it compares to the fixed threshold baseline and analyze its behavior during training.

4. The distance smoothing term is not clearly formulated in the methodology. More mathematical details are needed to explain how it is incorporated into the loss function and how it affects the feature representations.