SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiang Chen $^{1,2}$ , Ningyu Zhang $^{1,2*}$ , Lei Li $^{1,2}$ , Yunzhi Yao $^{1,2}$ , Shumin Deng $^{1,2}$ , Chuanqi Tan $^{3}$ , Fei Huang $^{3}$ , Luo Si $^{3}$ , Huajun Chen $^{1,2,*}$
<sup>1</sup>Zhejiang University & AZFT Joint Lab for Knowledge Engine, China
$^{2}$ Hangzhou Innovation Center, Zhejiang University, China
3Alibaba Group, China
{xiang_chen, zhangningyu, leili21, yyztodd, 231sm, huajunsir}@zju.edu.cn,
{chuanqi.tcq, f.huang, luo.si} @alibaba-inc.com
Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion Network (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. We further propose a dynamic gated aggregation strategy to achieve hierarchical multiscaled visual features as visual prefix for fusion. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance<sup>1</sup>.
Named entity recognition (NER) and relation extraction (RE) are important tasks in information extraction and knowledge base population, due to its research significance in natural language processing (NLP) and wide applications (Hosseini, 2019; Zhang et al., 2020; Qin et al., 2021; Zhang et al., 2021c). Currently, with the rapid development of multimodal learning, multimodal NER (MNER) and Multimodal RE (MRE) methods (Moon et al., 2018; Zheng et al., 2021) have been proposed to enhance linguistic representations with the aid of visual clues from images. It significantly extends the text-based models by taking images as additional inputs, since the visual contexts help to resolve ambiguous multi-sense words.
Input Text
#
Figure 1: Motivation for robust and effective hierarchical modality fusion.
[Destover O]: Destructi-vate malware has links to attacks on [South Korea LOC]2
BERT-CRF(Text): 1-O√,2-LOC √ UMGF:1-PER×,2-LOC √ Ours:1-O√,2-LOC √
Input Text
#
The madness outside of [LeBron PER]1 house now includes a car crash.
BERT-CRF(Text):1-OUMGF:1-OXOurs:1-PERV
The essence of MNER and MRE tasks is how to learn great visual features and how to incorporate it into textual representation for enhancing NER and RE. Early methods (Zhang et al., 2018; Moon et al., 2018) study how to incorporate the feature of whole image into the textual representation. Yu et al. (2020); Zhang et al. (2021a); Zheng et al. (2021) further validate that object-level visual fusion is more specific and important for MNER and MRE. Recently, RpBERT (Sun et al., 2021) propose to train a classifier of whether the "Image adds to the tweet meaning" before MNER tasks. However, they heavily rely on pre-training on large extra annotated corpus of image-text relevance and only focus on the whole image with ignoring the bias of relevant object-level visual fusion. In practice, irrelevant objects may directly exert negative effects on the text inference. Meanwhile, it is not trivial to acquire absolutely relevant object-level visual information to enhance the text. Thus, an effective method should be derived to learn better visual representation and alleviate error sensitivity of irrelevant object images for social media NER and RE tasks.
Considering images often appear before the text in a web document, we argue that images can be regarded as the prefix for their textual descriptions,
$^{1}$ Code is available in https://github.com/zjunlp/HVPNeT.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1607 - 1618
which is inspired by prompt learning (Gao et al., 2021; Li and Liang, 2021; Liang et al., 2022; Zhang et al., 2021d) in the language model. Specifically, given a image-text pair, we preprocess object-level image feature sequence of length  $V_{i}$  (visual prefix) to the text sequence at each self-attention layer of BERT (Devlin et al., 2019). Note that the visual prefix is a pluggable operation and don't require any annotation on relevance. Therefore, visual prefix can not only introduce object-level visual signals, but also further reduce the impact on the architecture representing text. Intuitively, visual prefix regarded as a prompt for text may helps alleviate the error sensitivity of irrelevant object images.
While Convolution Neural Networks (CNNs) contain the multi-scale information with pyramidal feature hierarchy (Ren et al., 2015) from low to high levels. And BERT encodes a rich hierarchy of linguistic information (Jawahar et al., 2019) from the bottom to the top. Inspired by Lin et al. (2017); Liu et al. (2018) that objects of different sizes can have appropriate feature representations at the corresponding scales, we propose to make each layer of BERT aware of hierarchical multiscale visual features to make a more enlightened and comprehensive forecasting decision.
To this end, we propose a novel Hierarchical Visual Prefix fusion Network (HVPNeT) for visual-enhanced entity and relation extraction. Specifically, inspired by SimVLM (Wang et al., 2021), we propose visual prefix-guided fusion mechanism involving concatenate object-level visual representation as the prefix of each self-attention layer in BERT, which is a more soft and robust attention module for visual enhanced NER and RE. We further design a dynamic gate for each layer to generate image-dependent paths, so that a variety of aggregated hierarchical multi-scaled visual features can be considered as visual prefix for enhancing NER and RE. Overall, we summarize the major contributions of our paper as follows:
- We present a hierarchical visual prefix fusion network towards MNER and MRE, incorporating hierarchical multi-scaled visual features through visual prefix-based attention mechanism at each self-attention layer of BERT to generate effective and robust textual representation for reducing error sensitivity.
- We utilize the exploitation of dynamic gates to fully leverage the hierarchical visual features.
Thus, textual representation of each layer in Transformer can be aware of corresponding hierarchical visual features adaptively. To the best of our knowledge, this paper is the first work to leverage hierarchical pyramidal visual features for multimodal learning.
- We evaluate our method on MNER and MRE tasks. Our experimental results on three benchmark datasets validate the effectiveness and superiority of our HVPNeT
Multimodal Entity and Relation Extraction As the crucial components of information extraction, named entity recognition (NER) and relation extraction (RE) have attracted much attention in the research community (Liu et al., 2019; Zhang et al., 2021b; Liu et al., 2021; Chen et al., 2021b,a). Previous studies typically focus on textual modality and standard text. As multimodal data become increasingly popular on social media platforms, early research focusing on textual modality and standard text is limited. Recently, several studies have focused on the MNER and MRE task, aiming to utilize the associate images to recognize the named entities and their relation better.
In the early stages, Zhang et al. (2018), Lu et al. (2018), (Moon et al., 2018) and Arshad et al. (2019) propose to encode the text through RNN and the whole image through CNN, then designing implicit interaction to model information between two modalities to explore multimodal NER tasks. Recently, Yu et al. (2020); Zhang et al. (2021a) propose to leverage regional image features to represent objects in the image to exploit fine-grained semantic correspondences based on Transformer and visual backbones.
While most of the current methods ignore the error sensitivity, one exception is that Sun et al. (2021), which proposes to learn a text-image relation classifier to enhance multimodal BERT to reduce the interference from irrelevant images while requiring extensive annotation for the irrelevance of image-text pairs.
Pre-trained Multimodal Representation The pre-trained multimodal BERT has recently achieved significant improvements in many multimodal tasks (e.g., visual question answering). We summarize and compare The existing visual-linguistic BERT models can be divided
into two aspects as follows: 1) Architecture. The single-stream structures consist of Unicoder-VL (Li et al., 2020), VisualBERT (Li et al., 2019), VL-BERT (Su et al., 2020), and UNITER (Chen et al., 2020b), where the text tokens and images are combined into a sequence and fed into BERT to learn contextual embeddings. The two-streams structures, LXMERT (Tan and Bansal, 2019) and ViLBERT (Lu et al., 2019), separately process the visual and language into two streams with interacting through cross-modality or co-attentional transformer layers. 2) Pretraining tasks. The pretraining tasks of multimodal visual-language model mainly consist of masked language modeling (MLM), masked region classification (MRC), and image-text matching (ITM). However, most of previous models are pre-trained on the datasets of image captioning (Sharma et al., 2018; Chen et al., 2015) or visual question answering where multimodal interactions are required. Applying current visual-language models to the MNER and MRE task may not result in a good performance, since MNER and MRE mainly focus on leveraging visual information to enhance the text rather than conducting prediction on the image side.
As illustrated in Figure 2, we present a novel hierarchical prefix fusion network for multi-modal entity and relation extraction. Note that our method can also be applied to other visual-enhanced tasks towards text.
3.1 Collection of Pyramidal Visual Feature
On the one hand, the image associated with a sentence maintains several visual objects related to the entities in the sentence, further providing more semantic knowledge to assist information extraction. On the other hand, the global image features may express abstract concepts, which play the role of a weak learning signal. Thus, we collect multiple visual clues for multimodal entity and relation extraction, which involves taking the regional image as the vital information and the global images as the supplement.
Given an image, we follow (Zhang et al., 2021a) to adopt the visual grounding toolkit (Yang et al., 2019) for extracting local visual objects with top  $m$  salience. Then, we rescale the global image and object image to  $224 \times 224$  pixels as the global image  $\mathcal{I}$  and visual objects  $\mathcal{O} = \{o_1, o_2, \dots, o_m\}$ .
In the area of CV, the feature fusion method that leveraging features from different blocks of pretrained models (Wang et al., 2019; Kim et al., 2018; Lin et al., 2017) is widely applied for improving model performance. Inspired by such practices, we take the first step to focus on the application of pyramid features in the area of multi-modality. We propose to fuse hierarchical image features into each Transformer layer; thus, leveraging a feature pyramid is essential. Typically, given an image, we encode it with a backbone model and generate a list of pyramidal feature maps  $\{F_1,F_2,F_3,\ldots ,F_c\}$  with different scales, then map them with  $M_{\theta}(\cdot)$  as follows:
$$
V _ {c} = \operatorname {C o n v} _ {1 \times 1} \left(F _ {c}\right), \tag {1}
$$
$$
V _ {i} = \operatorname {C o n v} _ {1 \times 1} \left(\operatorname {P o o l} \left(F _ {i}\right)\right), i = 1, 2, c - 1, \tag {2}
$$
where  $i$  denotes the  $i$ -th block of the backbone model,  $c$  denotes the number of blocks in the visual backbone model (here is 4 for ResNet),  $Pool$  represents the pooling operation, where the features are aggregated to the same spatial sizes. The  $1 \times 1$  convolutional layer is leveraged to map the pyramidal visual features to match the embedding size of the Transformer.
3.2 Dynamic Gated Aggregation
Although objects of different sizes can have appropriate feature representations at the corresponding scales, it is not trivial to decide which block in the visual backbone is assigned visual prefix for each layer in Transformer. To address this challenge, we propose constructing the densely connected routing space, where hierarchical multi-scaled visual features are connected with each transformer layer.
3.2.1 Dynamic Gate Module
We conduct routine processes through a dynamic gate module, which can be viewed as a procedure of path decision. The motivation of the dynamic gate is to predict a normalized vector, which represents how much to execute the visual feature of each block. In the dynamic gate,  $g_{i}^{(l)} \in [0,1]$  denotes the path probability from the  $i$ -th block of visual backbone to the  $l$ -th layer of Transformer. It is calculated as  $g^{(l)} = \mathbb{G}^{(l)}(V) \in \mathbb{R}^c$ , where  $\mathbb{G}^{(l)}(\cdot)$  denotes the gating function according to the  $l$ -th layer in Transformer,  $c$  represents the numbers of the block in backbone. We first produces the logits  $\alpha_{i}^{(l)}$  of the gate signals:
$$
\alpha^ {(l)} = f \left(W _ {l} \left(\frac {1}{c} \sum_ {i = 1} ^ {c} P \left(V _ {i}\right)\right)\right), \tag {3}
$$
Figure 2: The overall architecture of our hierarchical visual prefix for multimodal entity and relation extraction.
where  $f(\cdot)$  denotes the activate function Leaky_ReLU,  $P$  represents the global average pooling layer. We first squeeze the input features  $V_{i}$  with a shape of  $(d_i, h_i, w)$  from the  $i$ -th bloc by an average pooling operation. Then we add the features from multiple blocks to generate the average vectors. We further reduce the feature dimension by  $c$  with the MLP layer  $W_{l}$  and consider a soft gate via generating continuous values as path probabilities. Afterward, we generate the probability vector  $g^{(l)}$  for the  $l$ -th layer of Transformer as follows:
$$
g ^ {(l)} = \operatorname {S o f t m a x} \left(\alpha^ {(l)}\right) \tag {4}
$$
3.2.2 Aggregated Hierarchical Feature
Based on the above dynamic gate  $g^{(l)}$ , we can derive the final aggregated hierarchical visual feature  $V_{gated}$  to match the  $l$ -th layer in Transformer, as:
$$
V _ {g a t e d} ^ {(l)} = g ^ {(l)} V ^ {(l)}. \tag {5}
$$
Formally, the final visual features  $\tilde{V}_{gated}^{(l)}$  corresponding to the  $l$ -th layer of Transformer is obtained by the following concatenation operation,
$$
\tilde {V} _ {\text {g a t e d}} ^ {(l)} = \left[ V _ {\text {g a t e d}} ^ {(l, I)}; V _ {\text {g a t e d}} ^ {(l, o _ {1})}; \dots ; V _ {\text {g a t e d}} ^ {(l, o _ {m})} \right], \tag {6}
$$
which will be adopted to enhance layer-level representations of textual modality through visual prefix-based attention.
3.3 Visual Prefix-guided Fusion
We regard hierarchical multi-scaled image feature as visual prefix, and pretend the sequence of visual
prefix to the text sequence at each self-attention layer of BERT(Devlin et al., 2019) In particular, given an input sequence  $X = \{x_{1},x_{2},\dots,x_{n}\}$ , the contextual representations  $\pmb{H}^{l - 1}\in \mathbb{R}^{n\times d}$  is first projected into the query/key/value vector:
$$
\boldsymbol {Q} ^ {l} = \boldsymbol {H} ^ {l - 1} \boldsymbol {W} _ {l} ^ {Q}, \boldsymbol {K} ^ {l} = \boldsymbol {H} ^ {l - 1} \boldsymbol {W} _ {l} ^ {K}, \boldsymbol {V} ^ {l} = \boldsymbol {H} ^ {l - 1} \boldsymbol {W} _ {l} ^ {V}. \tag {7}
$$
As for aggregated hierarchical visual features  $\tilde{V}_{gated}^{(l)}$ , we use a set of linear transformations  $W_{l}^{\phi} \in \mathbb{R}^{d \times 2 \times d}$  for  $l$ -th layer to project them into the same embedding space<sup>2</sup> of textual representation in self-attention module. Besides, we define the operation of visual prompt  $\phi_{k}^{l}, \phi_{v}^{l} \in \mathbb{R}^{hw(m + 1) \times d}$  as:
$$
\left\{\phi_ {k} ^ {l}, \phi_ {v} ^ {l} \right\} = \tilde {V} _ {\text {g a t e d}} ^ {(l)} W _ {l} ^ {\phi}, \tag {8}
$$
where  $h w(m + 1)$  represents the length of the visual sequences,  $m$  denotes the number of visual objects detected by the object detection algorithm. Formally, the visual prefix-based attention are calculated as follows:
$$
P r e f i x \_ A t t e n t i o n ^ {l} = s o f t m a x \left(\frac {\boldsymbol {Q} ^ {l} \left[ \phi_ {k} ^ {l} ; \boldsymbol {K} ^ {l} \right] ^ {T}}{\sqrt {d}}\right) \left[ \phi_ {v} ^ {l}; \boldsymbol {V} ^ {l} \right]. \tag {9}
$$
Remark 1 We regard hierarchical multi-scaled visual features as visual prefix at each fusion layer and sequentially conduct multi-modal attention to update all textual states. In this way, the final textual states encode both the context and the cross-modal semantic information simultaneously. which
Remarkably, the key and value in the self-attention module contain the different information in two types of semantic space, here 2 means that we apply two sets of transformation parameters to project aggregated visual features to match the state update process, respectively.
is beneficial to reduce error sensitivity for irrelevant object elements.
3.4 Classifier
Based on above description, we get the final representation of BERT,  $H^{L} = U(X,\tilde{V}_{gated}^{(l)})$ , where  $U(\cdot)$  denotes the operation of visual prefix-based attention. Finally, we conduct different classifier layers for NER and RE, respectively.
Named Entity Recognition. Following (Moon et al., 2018; Yu et al., 2020), we also adopt the CRF decoder to perform the NER task. Formally, we feed the final hidden vectors  $\pmb{H}^{L} =$  of BERT to the CRF model. For a sequence of tags  $y = \{y_{1},\dots ,y_{n}\}$ , the probability of the label sequence  $y$  and the objective of NER are defined as follows (Lample et al., 2016a):
$$
\begin{array}{l} p \left(y \mid H ^ {L}\right) = \frac {\prod_ {i = 1} ^ {n} S _ {i} \left(y _ {i - 1} , y _ {i} , H ^ {L}\right)}{\sum_ {y ^ {\prime} \in Y} \prod_ {i = 1} ^ {n} S _ {i} \left(y _ {i - 1} ^ {\prime} , y _ {i} ^ {\prime} , H ^ {L}\right)}, \tag {10} \\ \mathcal {L} _ {n e r} = - \sum_ {i = 1} ^ {M} \log \left(p \left(y ^ {(i)} \mid U \left(X ^ {(i)}, \tilde {V} _ {\text {g a t e d}}\right)\right)\right). \\ \end{array}
$$
where  $Y$  represents the pre-defined label set with the BIO tagging schema, and  $S(\cdot)$  represents potential functions. Details can be referred in (Lample et al., 2016a).
Relation Extraction. An RE dataset can be denoted as  $\mathcal{D}_{re} = \{(X^{(i)},r^{(i)})\}_{i = 1}^{M}$ , the goal of RE is to predict the relation  $r\in \mathcal{V}$  between subject entity and object entity. Specifically, a [CLS] head is utilized to compute the probability distribution over the class set  $\mathcal{V}$  with the softmax function  $p(r|X) = \mathrm{Softmax}(\mathbf{W}\mathbf{H}^{\mathbf{L}}_{[\mathrm{CLS}]})$ , and the parameters of  $\mathcal{L}$  and  $\mathbf{W}$  are fine-tuned by minimizing the cross-entropy loss over  $p(r|X)$  on the entire  $\mathcal{X}$  as follows:
$$
\mathcal {L} _ {r e} = - \sum_ {i = 1} ^ {M} \log \left(p \left(r ^ {(i)} \mid U \left(X ^ {(i)}, \tilde {V} _ {\text {g a t e d}}\right)\right)\right). \tag {11}
$$
In the following section, we conduct experiments to evaluate our method on two multimodal information extraction tasks, MNER and MRE. Specifically, we adopt ResNet50 (He et al., 2016) as visual backbone and BERT-base (Devlin et al., 2019) as textual encoder. Results on three datasets demonstrate that our HVPNeT outperforms a number of unimodal and multimodal approaches.
We select three datasets for our experiments: Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Lu et al., 2018) for MNER, MNRE (Zheng et al., 2021) for MRE. Statistical details of datasets and experimental details are provided in Appendix A, B.
4.2 Compared Baselines
We compare our HVPNeT with several baseline models for a comprehensive comparison to demonstrate the superiority of our HVPNeT. Our comparison mainly focuses on three groups of models: the text-based models, previous SOTA MNER and MRE models, and the variants of our models.
Text-based models: we first consider a group of representative text-based models: 1) CNN-BiLSTM-CRF (Ma and Hovy, 2016), 2) HBiLSTM-CRF (Lample et al., 2016b) and 3) BERT-CRF for NER. The following models are specific for RE: 4) PCNN (Zeng et al., 2015); 5) MTB (Soares et al., 2019) is an RE-oriented pretraining model based on BERT.
Previous SOTA models: besides, we further consider another group of previous SOTA multi-modal approaches for MNER and MRE: 1) AdapCoAtt-BERT-CRF (Zhang et al., 2018); 2) OCSGA (Wu et al., 2020); 3) UMT (Yu et al., 2020); 4) UMGF (Zhang et al., 2021a), the newest SOTA for MNER, which proposes a unified multi-modal graph fusion approach for MNER. 5) BERT+SG is proposed in Zheng et al. (2021) for MRE, which concatenate the textual representation from BERT with visual features generated with scene graph (SG) tool (Tang et al., 2020). 6) MEGA (Zheng et al., 2021), the newest SOTA for MRE, which develops a dual graph for multi-modal alignment to capture this correlation between entities and objects for better performance. 7) VisualBERT(Li et al., 2019), different from the above SOTA methods mainly based on co-attention, VisualBERT is a single-stream structure, which is a strong baseline for comparison. And the results of VisualBERT listed in our paper are referred from Chen et al. (2020a)
Variants of Our Model: we set the ablation experiments to explore the effectiveness of our design. We conduct on the same parameter settings of HVP-NeT for each variant model for a fair comparison.
Modality	Methods	Twitter-2015	Twitter-2017	MNRE
Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
Text	CNN-BiLSTM-CRF	66.24	68.09	67.15	80.00	78.76	79.37	-	-	-
HBiLSTM-CRF	70.32	68.05	69.17	82.69	78.16	80.37	-	-	-
BERT-CRF	69.22	74.59	71.81	83.32	83.57	83.44	-	-	-
PCNN	-	-	-	-	-	-	62.85	49.69	55.49
MTB	-	-	-	-	-	-	64.46	57.81	60.86
Text+Image	AdapCoAtt-BERT-CRF	69.87	74.59	72.15	85.13	83.20	84.10	-	-	-
OCSGA	74.71	71.21	72.92	-	-	-	-	-	-
UMT	71.67	75.23	73.41	85.28	85.34	85.31	62.93	63.88	63.46
UMGF	74.49	75.21	74.85	86.54	84.50	85.51	64.38	66.23	65.29
BERT+SG	-	-	-	-	-	-	62.95	62.65	62.80
MEGA	70.35	74.58	72.35	84.03	84.75	84.39	64.51	68.44	66.41
VisualBERT	68.84	71.39	70.09	84.06	85.39	84.72	57.15	59.48	58.30
HVPNeT-Flat	73.76	75.32	74.54	84.43	86.42	85.41	79.32	78.20	78.75
HVPNeT-1T3	74.25	75.45	74.85	85.43	85.85	85.75	81.18	78.46	79.25
HVPNeT-OnlyObj	74.07	76.23	75.13	85.58	87.52	86.55	81.57	80.94	81.25
HVPNeT	73.87	76.82	75.32	85.84	87.93	86.87	83.64	80.78	81.85
Table 1: Performance comparison of different competitive baseline approaches for NER and RE. Since the original results of UMT, UMGF and MEGA only involve single extraction task, we reproduce their public code for more comprehensive comparison.
HVPNeT-Flat: This is another variant of our model without the pyramid structure. Here we assign the visual features with the output of the 4th block of ResNet and then map the visual features to each layer corresponding to BERT to conduct image-text fusion.
HVPNeT-1T3: As ResNet and BERT have four blocks and 12 layers, respectively thus, it is intuitive to directly map visual features in one block to the three layers in BERT. We denote this variant as HVPNeT-1T3 to compare with our final version with hierarchical visual features.
HVPNeT-OnlyObj: Visual objects are considered as fine-grained image representations. We conduct ablation by only adopting the object-level features in this model to validate the effect of the object features.
4.3 Overall Performance Comparison
4.3.1 Main Results
The experimental results of HVPNeT and all baselines on three testing sets are presented in Table 1. From the experimental results, we can observe that: