Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View
Year: 2023

Abstract:
We revisit the multimodal entity and relation extraction from a translation point of view. Special attention is paid on the misalignment issue in text-image datasets which may mislead the learning. We are motivated by the fact that the cross-modal misalignment is a similar problem of cross-lingual divergence issue in machine translation. The problem can then be transformed and existing solutions can be borrowed by treating a text and its paired image as the translation to each other. We implement a multimodal back-translation using diffusionbased generative models for pseudo-paralleled pairs and a divergence estimator by constructing a high-resource corpora as a bridge for low-resource learners. Fine-grained confidence scores are generated to indicate both types and degrees of alignments with which better representations are obtained. The method has been validated in the experiments by outperforming 14 state-of-the-art methods in both entity and relation extraction tasks. The source code is available at https://github.com/thecharm/TMR.

heading: 1 Introduction
text: Multimodal language understanding has received intensive attention recently for its advantage of mining semantics by collaborating the cross-modal inference (Yang et al., 2019a). Examples include methods for multimodal name entity recognition (MNER) (Zhang et al., 2018) and multimodal relation extraction (MRE) (Zheng et al., 2021a). Both benefit from the collaborative reasoning based on the alignment of textual and visual content. However, statistics on commonly adopted text-image relation benchmarks (e.g., TRC (Vempala and Preoţiuc-Pietro, 2019) and Twitter100k (Hu et al., 2017)) shows that the misalignment rate between images and texts is as high as 60%. Noise introduced by the misalignment can mislead the learning and degrade the performance of resulting models.
∗Corresponding author
As shown in Fig. 1, the misalignment can be categorized into partial and irrelevant alignment. In case of incomplete alignment, textual entities (e.g., NATO) might be mismatched to the visual evidence (e.g., person) which results in incorrect labels (e.g., PER). This further leads to underline relations between entities (e.g., <Trump, president of, USA), <USA, member of, NATO>) missing from the extractions. In case of irrelevant alignment, the textual entities might be randomly matched to visual evidence (e.g., MISC) resulting in dirty data for inference. While the misalignment with the ambiguity/distraction it brings to the learning has long been noticed, it has been rarely studied and addressed (Sun et al., 2021). The challenge is that it is nearly impossible to know the degree of misalignment prior to the inference. Otherwise, the inference may has already been done.
In this paper, we conduct a pilot study to address this problem. The motivation is that the misalignment of cross-modal pairs is a similar problem to the divergence of cross-lingual machine translations (Carpuat et al., 2017). The problem can thus be transformed by treating the text-image pairs in
6810
MNER/MRE as translations to each other. The divergence problem is more widely studied and existing solutions such as back-translation (Edunov et al., 2018) can be borrowed.
While this sounds appealing, it introduces new challenges as follows.
Modality Gap: The cross-lingual divergence is defined in a monomodal setting. The divergence can be measured explicitly by using features such as difference of sentence lengths, ratio of aligned words, and number of unaligned contiguous sequences (Carpuat et al., 2017). However, those features are not available in a cross-modal setting. We address it in an implicitly way in which disalignment of cross-lingual words (e.g., textual words and visual patches) is indicated by the divergence of their representations in the embedded space.
Parallelism: The detection/assessment of crosslingual divergence relies on large-scale parallel corpora, in which the sentences are aligned into wordlevel. The alignment is symmetric which makes high quality back-translation possible. However, in the cross-modal setting, MNER/MRE benchmark datasets are with a small scale due to the high cost of name entities labeling. The datasets are not well paralleled and there is no word-level alignment. We address those problems by taking advantage of the latest development of diffusion-based generative models (Saharia et al., 2022). Those models are trained on large-scale and better paralleled datasets, with which the back-translation can be conducted in a generate-to-translate way, in a sense that, for each text sentence, we can generate an image as its visual language “translation”. Visual grounding (Yang et al., 2019b) can then be employed to make the alignment into word-level. More details will be given in Section 3.3.
Low-Resource Benchmarks: The assessment of the divergence needs datasets on large-scale. This is not the case in MNER/MRE scenario. We borrow the idea of using high-resource corpora as a bridge to address the low-resource learning issue (Haddow et al., 2022; Gu et al., 2020). In this papaer, a new multimodal dataset is constructed for multimodal divergence estimation. An estimator is built which generates fine-grained confidence scores over 3 alignment categories of strengthen, weaken, and complement. It enables better argumentation for MNER/MRE than the simple similarity-based filtering schemes adopted previously. It also preserves the text-image pairs that are
not well-aligned but with complementary evidence. More details will be given in Section 3.4.

heading: 2 Related Work

heading: 2.1 Multimodal Entity and Relation Extraction
text: As the core components of knowledge graph construction, named entity recognition (NER) and relation extraction (RE) have received much more attention in the past few years. Previous studies (Zhang et al., 2018; Zheng et al., 2021b) revealed that incorporating visual information into text-based methods (Lample et al., 2016; Soares et al., 2019) can help improve the NER and RE performance, especially when sentences are short and ambiguous. These methods can be roughly divided into three categories: (1) encoding the features of the whole image and design effective attention mechanisms to capture the visual information related to texts (Lu et al., 2018; Moon et al., 2018). (2) incorporating object or region level visual features segmented from input image into textualbased methods with graph structure or transformers (Wu et al., 2020; Zheng et al., 2020; Zhang et al., 2021a; Zheng et al., 2021a). (3) hybrid fusion of multi-level visual features with textual expressions (Chen et al., 2022b,a). Despite the consistent improvement achieved by these attention-based methods, one major issue is that the texts and images are not always aligned well with each other. Recently, Sun et al. (2021) proposed RpBERT to address the above issue by learning a text-image similarity score to filter out the irrelevant visual representations. Zhao et al. (2022) explored inter-modal and intra-modal image-text relations by utilizing external matching from the dataset. However, some pairs not well aligned but with complementary will be neglected.

heading: 2.2 Vision-Language Pretraining Models
text: Large-scale pretrained models (PTMs) such as BERT (Kenton and Toutanova, 2019) and ViT (Dosovitskiy et al., 2020) have shown their strong abilities in representative learning and become a milestone in machine learning. Due to the success of PTMs in computer vision and natural language processing, many works are trying to adopt PTMs in multimodal domain (Han et al., 2021). Indeed, multimodal PTMs (Zhang et al., 2021b; Kim et al., 2021; Radford et al., 2021) can learn universal cross-modal representations and signifi-
cantly boost the downstream multimodal tasks (Antol et al., 2015; Datta et al., 2008). These methods differ mainly in the architecture for multimodal interactions. However, text-image misalignment has rarely been studied, although it is critical in real-world applications.

heading: 2.3 Generative Diffusion Models
text: Diffusion Models (Ho et al., 2020; Song et al., 2020) have emerged as a powerful toolkit in the family of generative models, with record-breaking results on many downstream applications such as image synthesis (Dhariwal and Nichol, 2021), video generation (Ho et al., 2022), and molecular generation (Jin et al., 2018). Recently, Imagen (Saharia et al., 2022) proposed a text-to-image diffusion model and a comprehensive benchmark for performance evaluation. Rombach et al. (2022) presented stable diffusion, a relatively lightweight text-to-image generator trained with large-scale web-crawled data and can synthesis photo-realistic images in few seconds. In this paper, we propose to transfer knowledge in generative diffusion models as back-translation for multimodal NER and RE tasks.

heading: 3 Translation Motivated Multimodal Representation Learning

heading: 3.1 Problem Formulation
text: Give a pair of a sentence t and an image v, our interest is the joint probability p(t, v), on the basis which the “translation” using either modality as the
source “language” can be obtained/evaluated (e.g., using p(t | v) or p(v | t)) (Carpuat et al., 2017). However, in the multimodal information extraction scenario, the translation is not a goal. We use it as a conceptual solution-seeking mindset. Specifically, our target is to build a function g(t, v) which learns the representations of p(t, v). We propose to make the learner aware of the modality misalignment (divergence) using
• Back-Translation: a generative diffusion model is employed as a predictor for p(v′ | t) which generates the back-translation of v. The divergence can be embedded by integrating the representations of v and v′;
• High-Resource Divergence Estimation: we learn a function d(t, v) to estimate the crossmodal divergence. The function is learned on a high-resource corpora independently and can be used to adjust p(t, v).
In this section, we introduce a general process for learning the representation first (i.e., g(t, v)), and then p(v′ | t) and d(t, v) can be implemented. Once the representation is obtained, multimodal information extraction tasks such as NER and MNRE can be conducted by learning the probability of p(l | g(t, v)) where l represents the label of name entities or relations depending on the task. The framework is shown in Fig. 2.

heading: 3.2 Multi-Grained Representation Learning
text: To ease the description, let us denote the resulting representation of a text-image pair as G = g(t, v). It can be implemented using a Transformer model (Kenton and Toutanova, 2019) as long as t and v can be tokenized (e.g., into words or patches) and embedded, so that the joint representation is learned regarding the cross-model correlation (ensured by the multi-head attention). Denote T and V as the tokenized embedding of t and v, respectively, the representation can be learned as
G = ∑ softmax
( WdV T
⊤ √ d
) T , (1)
where d is the dimension of textual embedding T and Wd is a cross-model attention matrix which is learned during the training.
However, granularity is a concern when the representation is cross-modal, because of the aforementioned Modality Gap and Parallelism challenges. We propose to build a multi-grained representation learning scheme, in which a 2-level of granularity is adopted so that a text is tokenized into words and phrases and an image is tokenized into patches and regions. We assume that the crossmodal representation can be generated on a fine scale based on word-patch correlations and the representation is coarse-grained when built on phraseregion correlations (Li et al., 2022).
Let us denote Tw and T p as the tokenized embedding of the text t at word and phrase level, respectively, in which the phrases is obtained using Stanford Parser following the method in Zhang et al. (2021a). The embedding are encoded using BERT (Kenton and Toutanova, 2019). Similarly, we denote V s and V r as the tokenized embedding of the image v at patch and region level, respectively, in which patches are obtained using fixed grid and regions are obtained using the visual grounding method toolkit (Yang et al., 2019b). We set the numbers of patches and regions as 49 and 3, respectively, by following the previous studies (Chen et al., 2022b,a). ResNet50 (He et al., 2016) is then employed to generate the visual embedding. The 2 levels of pairs (Tw,V s) and (T p,V r) are then be substituted into Eq. (1), resulting in the crossmodal representations Gf and Gc at fine and coarse level, respectively. A multi-grained representation G can then be generated as
G = Gf + Gc. (2)

heading: 3.3 Cross-Modal Back-Translation
text: We borrow the idea of back-translation from traditional machine translation methods (Edunov et al., 2018), in which the result in the target language is translated back to the source language to verify the quality or divergence. In our case, we treat the text t as a translation from an image v. A backtranslation v′ can then obtained by using
v′ = argmax p(v̂ | t), (3)
where v̂ is an image hypothesis. However, backtranslation usually requires parallel corpora to learn the probability of p(v̂ | t), which is not available in any NER/MNRE settings. We address this problem by taking advantage of recent advance in diffusion-based generative models (Saharia et al., 2022). Those models are trained using large-scale paralleled text-image pairs to learn the ability to generate an image contained on a give text prompt. The objective of those models is thus conceptually similar to Eq. (3). In our case, we use stable diffusion (Rombach et al., 2022), which is trained on a subset of LAION-5B (Schuhmann et al.) dataset. Upon back-translation, we feed the text t as a prompt to stable diffusion. The modal generates a v′ which can be used as an approximation of the back-translation from t.
To assess the divergence of translation, we cannot compare v′ to v like in text translation, because the cross-modal misalignment is at the semantic level and indicated by the correlation rather than the content. We thus compose a new pair (t, v′) and use the process introduced in Section3.2 to generate a back-translated cross-modal representation G′. Since v′ is generated directly from t, the alignment between them is better guaranteed than those sampled from user generated content on web or social media. It can be used a pseudoparalleled pair. Therefore, the original pair (t, v) is better aligned if G is similar to G′ or otherwise less aligned. There are different ways to use these two representations complementarily. Examples will be given in Section 3.5 under MNER/MRE scenario.

heading: 3.4 High-Resource Divergence Estimation
text: In this subsection, we implement an independent divergence estimator d(t, v). Existing methods address the issue by setting an attention mask on the reasoner trained on low-resource NER/MNRE benchmarks which simply filters out the less attended pairs (Zhang et al., 2018; Wu et al., 2020).
We argue that the training is easy to be biased by replying low-resource benchmarks which are neither sufficient on scale nor designed for divergence assessment purpose. More importantly, the filtering scheme also ignores pairs that are less aligned but with complementary evidence (e.g., Fig. 1). We construct a high-resource corpora which serves as a bridge to train the estimator independently. Furthermore, the estimator generates for each pair 3 confidence scores (αs, αc, αw) over the category set {strengthen, complement, weaken} for a more detailed divergence estimation. It can then be utilized as an augmenter (instead of a filter) for better representations of G and G′ as
  G∗ G′∗ 0   ⊤ =   αs αc αw   ⊤   Gf G′f Gc G′c 0 0   ,
w.r.t. αs + αc + αw = 1.
(4)
High-Resource Corpora Construction Different from Sun et al. (2021) using limited data crawled from social media (e.g., Twitter), we collect data from large-scale public image-text datasets to enhance the generalization of our estimator. We randomly select 100k data from MSCOCO (Lin et al., 2014) as the “Strengthen” samples, since the dataset contains fine-grained aligned imagetext pairs designed for tasks like Visual grounding and Scene graph generation. LAION-400M (Schuhmann et al., 2021) is chosen as the “Comple-
ment” dataset since it is built on web paired data and no strict rules are applied for the alignment between image contents and text tokens. Similar to MSCOCO, we select 100k image-text pairs from LAION-400M as training samples. We generate negative samples as the “Weaken” (unaligned) data by substituting the images in the “Strengthen” and “Complement” data with a different image randomly sampled from the two datasets. Finally, we accumulate 400k training samples, with 100k, 100k, 200k for “Strengthen”, “Complement” and “Weaken”, respectively. To verify the effectiveness and generalization, we further construct a in-domain test set of 10k data sampled from the two datasets and a out-of-domain test set of 1k data from the SBU dataset which contains both fine-grained and coarse-grained aligned text-image pairs. More supportive evidences and the generalization experiments are provided in Appendix B.3. Model Design We adopt the same structure as ViLT (Kim et al., 2021) that leverages a unified transformer to encode visual and textual contents. To be more specific, the input image v (or its backtranslation v′) is sliced into patches and flattened. Then a linear projection is applied to transfer the visual features to the same dimensions of token embeddings. The text and image embeddings are concatenated into a sequence Z and iteratively updated through D-dimensional Transformers. We get the pooled representations of the multimodal input sequence M as final output z. Details can be found in Figure 3 and Section 4.4.3. Supervised Contrastive Learning Conventional supervised methods use Cross-entropy Loss to distinguish samples with different classes. However, since our pretraining data are constructed on different datasets, simply applying cross-entropy loss will lead the model to learn a short-cut by utilizing the domain difference other than the semantic alignment. This results in poor generalization performance. To tackle this problem, we propose to use the supervised contrastive learning (Khosla et al., 2020) instead to push away the distance between anchors and negative samples generated from the positive classes “Strengthen” and “Complement”.
A self-supervised learning loss can be written
Lself = − ∑
i∈I log exp(zi · zj(i)/τ)∑ a∈A(i) exp(zi · za/τ) (5)
where z is the output of our estimator model, τ is a scalar temperature parameter. i, j, a denote
the anchor point, positive and negative samples, respectively. We can simply generalize the Eq. (5)to incorporate supervision as:
Lsup = ∑
i∈I
−1 |P (i)|
∑
p∈P (i) log exp(zi · zp/τ)∑ a∈A(i) exp(zi · za/τ)
(6) where P (i) is the set of indices of positives and |P (i)| denotes its cardinality.

heading: 3.5 Multimodal Information Extraction
text: We use the augmented representations G∗ and G′∗ for two tasks of NER and MNRE. Named Entity Recognition Following (Chen et al., 2022b; YU et al.), we adopt the CRF decoder to perform the NER task. We fuse the G∗ with its back-translation G′∗ using using multi-head extension (Kenton and Toutanova, 2019) and denoted the final representation for a pair (t, v) as
Ḡ = Multihead(G∗,G′∗) ∈ Rn×d (7)
which consists of the representation of n words from the text t. NER is then a task to predict probabilities of those words over a set of predefined entity labels (e.g., PER, ORG). Let us denote this label set as L = {l}. The probabilities are then denoted as Y = [y] ∈ Rn×|L| and calculated as
p ( y | Ḡ) = ∏n i=1 Fi(yi−1, yi, Ḡ)∑
lj∈L ∏n i=1 Fi(yi−1,j , yi,j , Ḡ) , (8)
where yi,j denotes the probability of the ith word over the jth label, and F represents potential functions in CRF. We use the maximum conditional likelihood estimation as the loss function
Lner = − n∑
i=1
log ( p(y|Ḡ) ) . (9)
Relation Extraction We merge the representations of textual entities, fine-grained and coarse-grained image-text pairs, as well as noun phrases to predict final relations. For a given pair of entities (ei, ej) corresponding to the ith and jth words from t, we generate its representation as
G̈i,j = Ti ⊕ Tj ⊕ p ⊕ h (10)
where Ti and Tj denote the embeddings of the two entities, respectively, ⊕ indicates the concatenation operation, p denote the summed features of noun phrases in the text t, and h denotes the summed
representation of the text-image pair and its backtranslation (i.e., h = G∗ + G′∗). We can then aggregate the likelihoods of this representation over a set of relation labels R = {r} as p(r | G̈i,j) = softmax(G̈i,j). Finally, we can calculate the RE loss with cross-entropy loss function
Lre = − n∑
i=1
log ( p(r | G̈i,j) ) . (11)

heading: 4 Experiment

heading: 4.1 Experimental Settings
text: Datasets and Metrics We adopt three publicly available datasets for evaluating our proposed method on MNER and MRE, including: 1) Twitter15 (Lu et al., 2018) and Twitter17 (Zhang et al., 2018) are two datasets for MNER, which include user posts on Twitter during 2014-2015 and 2016-2017, respectively. 2) MNRE (Zheng et al., 2021a) is a manually-annotated dataset for MRE task, where the texts and images are crawled from Twitter and a subset of Twitter15 and Twitter17. Statistics and experimental details are provided in Appendix. We use precision, recall and F1 value as the default evaluation metric and compare such results in the following sections. Baselines We compare our method with two groups of state-of-the-art (SOTA) methods as follows. Text-based Methods: CNN-BLSTM-CRF (Ma and Hovy, 2016), HBiLSTM-CRF (Lample et al., 2016), and BERT-CRF (Kenton and Toutanova, 2019) are classical sequence-labeling methods which show excellent prediction results on NER in newswire domain. PCNN (Zeng et al., 2015) is a distantlysupervised method for relation extraction, leveraging the knowledge from external knowledge base. MTB (Soares et al., 2019) is a SOTA method for many text-based RE tasks. Previous SOTA Multimodal Approaches: AdapCoAtt (Zhang et al., 2018) is the pioneer work that extracts named entities with co-attention mechanism. RpBERT (Sun et al., 2021) explicitly calculates image-text similarities by learning a classifier on Twitter data. OCSGA (Wu et al., 2020), UMT (YU et al.), UMGF (Zhang et al., 2021a), and MEGA (Zheng et al., 2021a) are the NER/RE methods that align fine-grained object features with textual representations with Transformers or Graph Neural Networks. VisualBERT (Li et al., 2019) is a vision-language pretraining model that can be
applied for MNER and MRE tasks. HVPNet (Chen et al., 2022b) and MKGFormer (Chen et al., 2022a), the latest SOTA for MNER and MRE, which develops a hierarchical structure to learn visual prefix from multiple views.

heading: 4.2 Comparison to SOTA
text: The results are shown in Table 1. It is easy to see our method outperforms other SOTA methods on on all datasets.
When compared to models relying on pure textual information, visual features contribute to the performance gain by 5% on MNER and 20% on MRE. Due to the short and ambiguous characteristics of texts in social media, it is difficult to identify entities and their relations in limited context.
Incorporating multi-grained visual and textual information performs better than relying on object or image level information solely. The SOTA method HVPNeT and our MTR gain better results (88.35% and 86.87% in Twitter-2017 dataset) than UMGF (85.51%) and UMT (85.31%) which align image and text in fine-grained object-level.
Our model outperforms HVPNet and MKGFormer which leverage hierarchical visual representations or powerful vision-language pretraining embeddings, in a relatively large margin (from 82% to 89%) on the MRE task. We observe a more obvious performance improvement on MRE datasets compared to that on MNRE. The difference comes from the different distributions of MRE and MNRE datasets. Our statistics show that the proportion
of complementary cases is significantly higher in MRE (51.5%) than in MNRE (15.7%). As mentioned in the paper, the proposed back-translation helps the two tasks by providing additional contextual information for inference. This benefits the complementary cases the most because it makes the identification of indirect relationships possible (otherwise, those cases will be considered as misalignments or used incorrectly like in the similaritybased methods).

heading: 4.3 Ablation Study
text: In this section, we conduct extensive experiments with the variants of our model to analyze the effectiveness of each component.
Back-translation: We ablate the procedure of generating back-translation images and the results in Table 1 show the component can boost model performance by 1-3% in MNER and MRE. Still, our ablated model gains comparable or superior performance against baselines which demonstrates the effectiveness of back-translation.
Multimodal Divergence Estimation: Compared with similarity-score based method RpBERT, our model shows stronger extraction and generalization performance with 3.18% improvement on Twitter2015 dataset. Also, our model achieves significant improvements (3% to 7%) over attention-based methods, revealing that TMR can improve conventional NER/RE methods by decomposing the divergence into fine-grained level.

heading: 4.4 Other Essentials of the Model

heading: 4.4.1 Low-resource Performance
text: We conduct experiments in low-resource scenarios following the setting of Chen et al. (2022b), by randomly sampling 5% to 50% from original training set. From the results in Figure 4, we can observe: 1) The methods utilizing multi-grained features (HVPNet and TMR) consistently outperform object-level models in MNER (UMGF) and MRE (MEGA). Multi-grained features can provide global and local views and help models infer entities and relations efficiently. 2) Moreover, our proposed TMR model performs better than HVPNet with external knowledge from generative diffusion models, which addresses the information lack problem in low-resource scenarios.

heading: 4.4.2 Improvements on Complementary Cases
text: To demonstrate the effectiveness of correlation decomposition, we further compare our method with SOTA method HVPNeT on complementary cases of MNRE test set. We argue that previous similarity-based methods ignore the cross-modal divergence, especially when texts and images are complementary. We export 832 cases with “complement score” higher than 0.5 from 1614 test samples. Our model achieves significant improvements against HVPNeT, especially on some categories (e.g., Present in, Locate at and Residence) that rely on deeper understanding of visual scenarios.

heading: 4.4.3 Generalization Performance of Multimodal Divergence Estimator
text: We extend conventional similarity score into finegrained level and weight the importance of incorporated visual information based on the pretrained divergence estimator. To verify the generalizations to data in other domain, we first construct test set collected with in-domain data (i.e., by sampling on MSCOCO and LAION400M). Then, We first request 2 annotators to label 1k test samples on out-of-domain data and then ask other 2 to review and rectify the test set. As shown in Table 3, we compare the estimator trained with different loss function. The results indicate that the model with cross-entropy loss suffers the generalization problem when transferred into out of domain data. The possible reason is that the model may learn a shortcut from the difference of image/text style on the data from the two datasets, other than taking the image-text correlation into consideration. We improve it by introducing negative sampling on indomain data to reduce the style bias and the F1 value on out-of-domain data increases from 61.8 to 80.01. We further apply the supervised contrastive learning to pull together the positive samples and push apart negative ones, resulting in better generalization performance.

heading: 4.4.4 Case Study
text: To validate the effectiveness and robustness of our method, we conduct case analysis for multimodal divergence estimation. Previous works simply calculate the image-text similarity with attention mechanism (HVPNeT) or pretrained classifier (RpBERT). As a result, visual information with low similarity score will be filtered out. We notice that our model and RpBERT can identify entities correctly when images are well-aligned with sentence in S1. However, RpBERT fails to extract the ORG entity “Foran” since it outputs a much lower similarity score. Our model successfully captures
the semantics of “team competition” and it can be used to complement the missing semantics, which helps extract “Foran” as a name of organization and the relation “member_of” between the two entities. Another case is that when the image is irrelevant to textual contents in S3, HVPNeT gives the wrong prediction due to the misleading of the image. Our method can address this problem by generating a back-translation image of “Taylor Swift” and the “awarding scene”, as shown in Figure 1.

heading: 5 Conclusion
text: We have revisited the misalignment issue in multimodal benchmarks. By borrowing the ideas from translation methods, we have implemented multimodal versions of back-translation and highresource bridging, which provide a multi-view to the misalignment between modalities. The method has been validated in the experiments and outperforms 14 SOTA methods.

heading: Acknowledgments
text: This work has been supported by the Hong Kong Research Grants Council under the Collaborative Research Fund (Project No. C1031-18G), the National Natural Science Foundation of China (62076100), Fundamental Research Funds for
the Central Universities, SCUT (x2rjD2220050), the Science and Technology Planning Project of Guangdong Province (2020B0101100002), CAAIHuawei MindSpore Open Fund, CCF-Zhipu AI Large Model Fund.
Limitations
The study is in its initial form. The efficiency is a major concern. This mainly results from the use of generative diffusion models, which are under heavy development. We believe this will be addressed soon in the near future. Further, the proposed framework is not end-to-end. It may introduce extra effort for training. We will deal with this issue in the future study.

heading: A Detailed Statistics of Dataset

heading: B Experimental Details
text: B.1 Multimodal Named Entity Recognition This section details the training procedures and hyperparameters for named entity recognition. We use the BERT-base-uncased model from hugging
face library . We follow UMGF (Zhang et al., 2021a) to revise some wrong annotations in the Twitter-2015 dataset. We utilize Pytorch to conduct experiments with 1 Nvidia 3090 GPUs. All optimizations are performed with the AdamW optimizer with a linear warmup of learning rate 3e-5 over the first 10% of gradient updates to a maximum value, then linear decay over the remainder of the training. And weight decay on all non-bias parameters is set to 0.01. We set the number of grounding regions and image patches to 3 and 49, respectively. Max length of noun phrases is set to 4 and max length for sentence is set to 80.
B.2 Multimodal Relation Extraction This section details the training procedures and hyperparameters for relation extraction. Similar to NER, we use the BERT-base-uncased model from hugging face library. We set the number of grounding regions and image patches to 3 and 49, respectively. Max length of noun phrases is set to 6 and max length for sentence is set to 128. We set the initialized learning rate to 1e-2.
B.3 Multimodal Divergence Estimation We adapt the main structure of ViLT (Kim et al., 2021) to decompose the image-text correlation, as shown in Figure 5. For all experiments, we use AdamW optimizer with base learning rate of 1e4 and weight decay of 1e-2. The learning rate was warmed up for 10% of the total training steps and was decayed linearly to zero for the rest of the training. We resize the shorter edge of input images to 384 and limit the longer edge to under 640 while preserving the aspect ratio. Patch projection of the model yields 12× 20 patches for an image with a resolution of 384× 640. We use the BERT-baseduncased tokenizer to tokenize text inputs. We pretrain the model for 100K steps on 8 NVIDIA V100 GPUs with a batch size of 32.
B.4 More Cases of Generative Back-translation
We provide more examples in Figure 6 to illustrate the power of generative back-translation. Compared to extract entities and their relations with only original images, the generated images provide a different view and help to align the image and text from a translation perspective.
Original Image Generated ImageSentence Facts
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe this will be addressed soon in the near future. Further, the proposed framework is not end-to-end. It may introduce extra effort for training. We will deal with this issue in the future study