SlowGuess/NAACL_2022_OCR huggingface dataset:
Xinyu Wang $\diamond\ddagger$ , Min Gui $\diamond$ , Yong Jiang $\spadesuit$ , Zixia Jia $\diamond$ , Nguyen Bach $\clubsuit$ , Tao Wang, Zhongqiang Huang $\clubsuit$ , Fei Huang $\clubsuit$ , Kewei Tu $\diamond$ *
$\diamond$ School of Information Science and Technology, ShanghaiTech University
$\diamond$ Shanghai Engineering Research Center of Intelligent Vision and Imaging
$\spadesuit$ DAMO Academy, Alibaba Group
Shopee, Singapore
{wangxyl, jiazx, tukw}@shanghaiitech.edu.cn, minGui@shopee.com
{yongjiang.jy,z.huang,f.huang}@alibaba-inc.com
nguyenbach@microsoft.com
Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations obtained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text representations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their respective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose Image-text Alignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer-based pretrained textual embeddings can be better utilized. ITA first aligns the image into regional object tags, image-level captions and optical characters as visual contexts, concatenates them with the input texts as a new cross-modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction between the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practical in dealing with text-only inputs and robust to noises from images. In our experiments, we show that ITA models can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets, even without image information.<sup>1</sup>
Named Entity Recognition (NER) (Sundheim, 1995) has attracted increasing attention in natural language processing community. It has been applied to a lot of domains such as news (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003), E-commerce (Fetahu et al., 2021), social media (Strauss et al., 2016; Derczynski et al., 2017) and bio-medicine (Doğan et al., 2014; Li et al., 2016). Several recent studies focus on improving the accuracy of NER models through utilizing image information (MNER) in tweets (Zhang et al., 2018; Moon et al., 2018; Lu et al., 2018). Most approaches to MNER use the attention mechanism to model the interaction between image and text representations (Yu et al., 2020; Zhang et al., 2021a; Sun et al., 2021), in which image representations are from a pretrained feature extractor, i.e. ResNet (He et al., 2016), and text representations are extracted from pretrained textual embeddings, i.e. BERT (Devlin et al., 2019). Since these models are separately trained on datasets of different modalities and their feature representations are not aligned, it is difficult for the attention mechanism to model the interaction between the two modalities.
Recently, pretrained vision-language  $(\mathrm{V} + \mathrm{L})$  models such as LXMERT (Tan and Bansal, 2019), UNITER (Chen et al., 2020) and Oscar (Li et al., 2020b) have achieved significant improvement on several cross-modal tasks such as image captioning, VQA (Agrawal et al., 2015), NLVR (Young et al., 2014) and image-text retrieval (Suhr et al., 2019). Most pretrained  $\mathrm{V} + \mathrm{L}$  models are trained on image-text pairs and simply concatenate text features and image features as the input of pretraining. There are, however, two problems. First, texts in these datasets mainly contain common nouns
* Yong Jiang and Kewei Tu are the corresponding authors. ‡: This work was done when Xinyu Wang, Min Gui and Nguyen Bach were at Alibaba Group.
1Our code is publicly available at https://github. com/Alibaba-NLP/KB-NER/ITA.
Human Language Technologies, pages 3176 - 3189
instead of named entities<sup>2</sup> which leads to an inductive bias over common nouns and images. Second, despite its important role in pretraining V+L models, the image modality only plays an auxiliary role in MNER for disambiguation, and can sometimes even be discarded. These problems make pretrained V+L models perform weaker than pretrained language models for MNER.
Pretrained textual embeddings such as BERT, XLM-RoBERTa (Conneau et al., 2020) and LUKE (Yamada et al., 2020) have achieved state-of-the-art performance on various NER datasets through simple fine-tuning of pretrained textual embeddings. Since most of the transformer-based pretrained textual embeddings are trained over long texts, recent work (Akbik et al., 2019; Schweter and Akbik, 2020; Yamada et al., 2020; Wang et al., 2021) has shown that introducing document-level contexts can significantly improve the accuracy of a NER model. The attention mechanism in transformer-based pretrained textual embeddings can utilize contexts to improve the token representation of a sequence. Moreover, pretrained  $\mathrm{V + L}$  models such as Oscar and VinVL (Zhang et al., 2021b) can use object tags detected in images to significantly ease the alignments between text and image features. Therefore, the images in MNER can be converted to texts as well so that the image representations can be aligned to the space of text representations. As a result, the attention module of the pretrained textual embeddings have the capability to easily model the interactions between aligned image and text representations, without introducing a new attention module. In this paper, we propose ITA, a simple but effective framework for Image-Text Alignments. ITA converts an image into visual contexts in textual space by multi-level alignments. We concatenate the NER texts with the visual contexts as a new cross-modal input view and then feed it into a pretrained textual embedding model to improve the token representations of NER texts, which are fed into a linear-chain CRF (Lafferty et al., 2001) layer for prediction. In practice, a MNER model should be robust when there is only text information, as images may be unavailable or can introduce noises. Sometimes it is even undesirable to use images as image feature extraction can be inefficient in online serving. Therefore, we further propose to utilize the cross-modal input
view to improve the accuracy of textual input view, based on cross-view alignment that minimizes the KL divergence over the probability distributions of the two views.
ITA can be summarized in four aspects:
1. Object Tags as Local Alignment: ITA locally extracts object tags and its corresponding attributes of image regions from an object detector.
2. Image Captions as Global Alignment: ITA summarizes what the image is describing through predicting image captions from an image captioning model.
3. Optical Character Alignment: ITA extracts the texts presented in the image via optical character recognition (OCR).
4. Cross-View Alignment: we calculate the KL divergence between the output distributions of two input views.
We show in experiments that ITA can significantly improve the model accuracy on MNER datasets and achieve the state-of-the-art. The cross-view alignment module can significantly improve both the cross-modal and textual input views, and bridge the performance gap between the two views.
2 Approaches
We consider the NER task as a sequence labeling problem. Given a sentence  $\boldsymbol{w} = \{w_{1},\dots ,w_{n}\}$  with  $n$  tokens and its corresponding image  $I$ , an sequence labeling model aims to predict a label sequence  $\boldsymbol{y} = \{y_{1},\dots ,y_{n}\}$  at each position. In our framework, we focus on incorporating visual information to improve the representations of the input tokens by aligning visual and textual information effectively. We use a visual context generator to convert the image  $I$  into texts forming visual contexts  $\boldsymbol{w}' = \{w_1',\dots ,w_m'\}$  with  $m$  tokens. We then concatenate the input text and visual contexts as a cross-modal text+image  $(\mathbf{I} + \mathbf{T})$  input view instead of the text  $(\mathbf{T})$  input view. We feed the  $\mathbf{I} + \mathbf{T}$  input into a pretrained textual embeddings model to get stronger token representations of the input sentence. Then the token representations are fed into a linear-chain CRF layer to get the label sequence  $\boldsymbol{y}$ . To further improve the model accuracy of both input views, we use the cross-view alignment module to align the output distributions of  $\mathbf{I} + \mathbf{T}$  and  $\mathbf{T}$
2https://visualgenome.org/data_ analysis/statistics
Figure 1: The architecture of ITA. ITA aligns an image into object tags, image captions and texts from OCR. ITA takes them as visual contexts and then feeds them together with the input texts into the transformer-based embeddings. In the cross-view alignment module, ITA minimizes the distance between the output distribution of cross-modal inputs and textual inputs.
input views during training. The architecture of our framework is shown in Figure 1.
2.1 NER Model Architecture
We use a neural model with a linear-chain CRF layer, a widely used approach for the sequence labeling problem (Huang et al., 2015; Akbik et al., 2018; Devlin et al., 2019). The input is fed into a transformer-based pretrained textual embeddings model and the output token representations  $\{r_1,\dots ,r_n\}$  are fed into the CRF layer:
$$
p _ {\theta} (\boldsymbol {y} | \boldsymbol {w}) = \frac {\prod_ {i = 1} ^ {n} \psi \left(y _ {i - 1} , y _ {i} , \boldsymbol {r} _ {i}\right)}{\sum_ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y} (\boldsymbol {w})} \prod_ {i = 1} ^ {n} \psi \left(y _ {i - 1} ^ {\prime} , y _ {i} ^ {\prime} , \boldsymbol {r} _ {i}\right)}
$$
where  $\theta$  is the model parameters,  $\mathcal{V}(\boldsymbol{w})$  is the set of all possible label sequences given the input  $\boldsymbol{w}$ . Given the gold label sequence  $\hat{\boldsymbol{y}}$  in the training data, the objective function of the model for the T input view is:
$$
\mathcal {L} _ {\mathrm {T}} (\theta) = - \log p _ {\theta} (\hat {\boldsymbol {y}} | \boldsymbol {w}) \tag {1}
$$
The loss can be calculated using Forward algorithm.
2.2 Image-text Alignments
The transformer-based pretrained textual embeddings have strong representations over texts. Therefore, ITA converts the image information into textual space through generating texts from the image so that the learning of the self-attention in the
transformer-based model can be significantly eased compared with simply using image features from an object detector. We propose a local (LA), a global (GA) and an optical character alignment (OCA) approaches for alignments.
Object Tags as Local Alignment Given an image, the image information can be decomposed into a set of objects in local regions. The object tags of each region textually describe the local information in the image. To extract the objects, we use an object detector OD to identify and locate the objects in the image:
$$
\boldsymbol {a}, \boldsymbol {o} = \mathbf {O D} (I); \text {w h e r e}
$$
$$
\boldsymbol {a} = \left\{\boldsymbol {a} _ {1}, \boldsymbol {a} _ {2}, \dots , \boldsymbol {a} _ {l} \right\} \text {a n d} \boldsymbol {o} = \left\{o _ {1}, o _ {2}, \dots , o _ {l} \right\}
$$
The attribute predictions from the object detector contain multiple attribute tags  $\pmb{a}_i$  for each object  $o_i$ . We linearize and sort the objects in a descending order based on the confidence of the detection model. For each object, we heuristically keep 0 to 3 attributes with confidence scores above a threshold  $m$ . We linearize the attributes and put the attributes before the corresponding objects since the attributes are the adjectives describing the object tags. As a result, we take the predicted  $l$  object tags  $\pmb{o}$  and their attribute tags  $\pmb{a}$  from the object detector as the locally aligned visual contexts  $\pmb{w}^{\mathrm{LA}}$ :
$$
\boldsymbol {w} ^ {\mathrm {L A}} = \left\{\boldsymbol {a} _ {1}, o _ {1}, \boldsymbol {a} _ {2}, o _ {2}, \dots , \boldsymbol {a} _ {l}, o _ {l} \right\}
$$
Image Captions as Global Alignment Though the local alignment can localize the image into
objects, the objects cannot fully describe the of the whole image. Image captioning is a task that predicts the meaning of an image. Therefore, we align the image into  $k$  image captions by an image captioning model IC:
$$
\left\{\boldsymbol {w} ^ {1}, \boldsymbol {w} ^ {2}, \dots , \boldsymbol {w} ^ {k} \right\} = \mathbf {I C} (I)
$$
where  $\{\pmb{w}^1, \pmb{w}^2, \dots, \pmb{w}^k\}$  are captions generated from beam search with  $k$  beams. We concatenate the  $k$  captions together with a special separate token [X] to form the aligned global visual contexts  $\pmb{w}^{\mathrm{GA}}$ :
$$
\boldsymbol {w} ^ {\mathrm {G A}} = \left[ \boldsymbol {w} ^ {1}, [ \mathrm {X} ], \boldsymbol {w} ^ {2}, [ \mathrm {X} ], \dots , [ \mathrm {X} ], \boldsymbol {w} ^ {k} \right]
$$
The exact label (e.g. "[SEP]" in BERT) of the special [X] token depends on the selection of embeddings.
Optical Character Alignment Some image contain text when they are created to enrich the semantic information that the images want to convey. In order to better understand this type of image, we use an OCR model to identify and extract the texts in the image:
$$
\boldsymbol {w} ^ {\mathrm {O C A}} = \mathbf {O C R} (I)
$$
where  $\boldsymbol{w}^{\mathrm{OCA}}$  are the texts extracted by the OCR model. Note that  $\boldsymbol{w}^{\mathrm{OCA}}$  may be an empty text if there is no text in the image.
We concatenate the input sentence and our aligned visual contexts to form the  $\mathbf{I} + \mathbf{T}$  input view  $\hat{\boldsymbol{w}} = [\boldsymbol{w};\boldsymbol{w}^{\prime}]$ , where  $\boldsymbol{w}^{\prime}$  can be one of  $\boldsymbol{w}^{\mathrm{LA}}$ ,  $\boldsymbol{w}^{\mathrm{GA}}$ ,  $\boldsymbol{w}^{\mathrm{OCA}}$  or the concatenation of all (we denote it as All). The transformer-based embeddings are fed with the  $\mathbf{I} + \mathbf{T}$  input view and then output image-text fused token representations for each token  $\{r_1',\dots ,r_n'\}$ . The token representations are fed into the CRF layer to get the probability distribution  $p_{\theta}(\boldsymbol {y}|\hat{\boldsymbol{w}})$ . Similar to Eq. 1, the objective function of the model for the  $\mathbf{I} + \mathbf{T}$  input view is:
$$
\mathcal {L} _ {\mathrm {I} + \mathrm {T}} (\theta) = - \log p _ {\theta} (\hat {\boldsymbol {y}} | \hat {\boldsymbol {w}}) \tag {2}
$$
Cross-View Alignment There are several limitations in incorporating images into NER prediction: 1) the images may not available in testing; 2) aligning images to texts requires several pipelines in pre-processing instead of an end-to-end manner, which is so time-consuming that it is not applicable to some time-critical scenes such as online serving; 3) the noises in the image can mislead the MNER
model to make wrong predictions. To alleviate these issues, we propose Cross-View Alignment (CVA), which targets at reducing the gap between the  $\mathbf{I} + \mathbf{T}$  and  $\mathbf{T}$  input views over the output distributions so that the MNER model can better utilize the textual information in the input. During training, CVA minimizes the KL divergence over the probability distribution of  $\mathbf{I} + \mathbf{T}$  and  $\mathbf{T}$  input views:
$$
\mathcal {L} _ {\mathrm {C V A}} (\theta) = \mathrm {K L} \left(p _ {\theta} (\boldsymbol {y} | \boldsymbol {\hat {w}}) \| p _ {\theta} (\boldsymbol {y} | \boldsymbol {w})\right) \tag {3}
$$
Since the  $\mathbf{I} + \mathbf{T}$  input view has additional visual information in the input and we want the  $\mathbf{T}$  input view to match the accuracy of  $\mathbf{I} + \mathbf{T}$  input view, we only back-propagate through  $p_{\theta}(\boldsymbol{y}|\boldsymbol{w})$  in Eq. 3. Therefore, Eq. 3 is equivalent to calculating the cross-entropy loss over the two distributions:
$$
\mathcal {L} _ {\mathrm {C V A}} (\theta) = \sum_ {\boldsymbol {y} \in \mathcal {Y} (\boldsymbol {x})} p _ {\theta} (\boldsymbol {y} | \hat {\boldsymbol {w}}) \log p _ {\theta} (\boldsymbol {y} | \boldsymbol {w}) \tag {4}
$$
As the set of all possible label sequences  $\mathcal{V}(\pmb{x})$  is exponential in size, we calculate the posterior distributions of each position  $p_{\theta}(y_i|\pmb{w})$  and  $p_{\theta}(y_i|\hat{\pmb{w}})$  through forward-backward algorithm to approximate Eq. 4:
$$
\begin{array}{l} p _ {\theta} (y _ {k} | \ast) \propto \sum_ {\left\{y _ {0}, \dots , y _ {k - 1} \right\}} \prod_ {i = 1} ^ {k} \psi \left(y _ {i - 1}, y _ {i}, \boldsymbol {r} _ {i} ^ {*}\right) \\ \times \sum_ {\left\{y _ {k + 1}, \dots , y _ {n} \right\}} \prod_ {i = k + 1} ^ {n} \psi \left(y _ {i - 1}, y _ {i}, \boldsymbol {r} _ {i} ^ {*}\right) \\ \end{array}
$$
$$
\mathcal {L} _ {\mathrm {C V A}} (\theta) = \sum_ {i = 1} ^ {n} p _ {\theta} \left(y _ {i} \mid \hat {\boldsymbol {w}}\right) \log p _ {\theta} \left(y _ {i} \mid \boldsymbol {w}\right)) \tag {5}
$$
where  $\boldsymbol{r}_i^*$  represents either  $\boldsymbol{r}_i$  or  $\boldsymbol{r}_i'$ .
Training During training, we jointly train  $\mathbf{T}$  and  $\mathbf{I} + \mathbf{T}$  input views with the training objective in Eq. 1 and 2 together with the CVA alignment training objective in Eq. 5. As a result, the final training objective for ITA is:
$$
\mathcal {L} _ {\mathrm {I T A}} = \mathcal {L} _ {\mathrm {C V A}} + \mathcal {L} _ {\mathrm {T}} + \mathcal {L} _ {\mathrm {I} + \mathrm {T}}
$$
We conduct experiments on two MNER datasets. To show the effectiveness of our approaches, we use two embedding settings and compare our approaches with previous multi-modal approaches.
Datasets We show the effectiveness of our approaches on Twitter-15, Twitter-17 and SNAP Twitter datasets<sup>3</sup> containing 4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459 sentences in train/development/test split respectively. The Twitter-15 dataset is constructed by Zhang et al. (2018). The SNAP dataset is constructed by Lu et al. (2018) and the Twitter-17 dataset is a filtered version of SNAP constructed by Yu et al. (2020).
Model Configuration For token representations, we use BERT base model to fairly compare with most of the recent work (Yu et al., 2020; Zhang et al., 2021a; Sun et al., 2021). Recently, XLM-RoBERTa has achieved state-of-the-art accuracy on various NER datasets by feeding the input together with contexts to the model. To further utilize the visual contexts in transformer-based embeddings, we use XLM-RoBERTa large (XLMR) model as another embedding in our experiments. To extract object tags and image captions of the image, we use VinVL (Zhang et al., 2021b), which is a pretrained  $\mathrm{V + L}$  model based on a newly pretrained large-scale object detector based on the ResNeXt-152 C4 architecture. We use the object detection module of VinVL to predict object tags and their corresponding attributes. The number of object tags and attributes varies over the images and is no more than 100. We set the threshold  $m$  to be 0.1 for keeping the attributes of each object. For image captions, we use VinVL large model finetuned on MS-COCO (Lin et al., 2014) captions<sup>4</sup> with CIDEr optimization (Rennie et al., 2017). In our experiments, we use a beam size of 5 with at most 20 tokens for prediction and keep all the 5 captions as the visual contexts. For OCR, we use Tesseract OCR<sup>5</sup> (Smith, 2007), which is an open source OCR engine. We use the default configuration of the engine to extract texts in the image<sup>6</sup>.
Training Configuration During training, we finetune the pretrained textual embedding model by AdamW (Loshchilov and Hutter, 2018) optimizer. In experiments we use the grid search to find the learning rate for the embeddings within  $[1 \times 10^{-6}, 5 \times 10^{-4}]$ . For BERT embeddings, we finetune the embeddings with a learning rate of
Twitter-15	Twitter-17	SNAP
Train Modal	Approach	Eval Modal T I+T	Eval Modal T I+T	Eval Modal T I+T
BERT-CRF
T	BERT-CRF	74.79 -	85.18 -	85.98 -
I+T	ITA-LA	- 75.18	- 85.67	- 86.26
ITA-GA	- 75.17	- 85.75	- 86.72
ITA-OCA	- 75.01	- 85.64	- 86.52
ITA-All	- 75.15	- 85.78	- 86.79
ITA-LA+CVA	75.26 75.20	85.72 85.62	86.51 86.41
ITA-GA+CVA	75.45 75.52	85.96 85.85	86.42 86.39
ITA-OCA+CVA	75.26 75.30	85.73 85.79	86.64 86.59
ITA-All+CVA	75.67 75.60	85.98 85.72	86.83 86.75
XLMR-CRF
T	XLMR-CRF	77.37 -	88.73 -	89.39 -
I+T	ITA-LA	- 77.64	- 89.29	- 89.68
ITA-GA	- 77.78	- 89.32	- 89.78
ITA-OCA	- 77.94	- 89.31	- 89.64
ITA-All	- 77.81	- 89.62	- 90.10
ITA-LA+CVA	77.87 77.93	89.45 89.90	89.85 89.91
ITA-GA+CVA	78.03 78.02	89.41 89.62	89.85 90.09
ITA-OCA+CVA	77.57 77.59	89.32 89.55	89.90 89.84
ITA-All+CVA	78.25 78.03	89.47 89.75	90.02 90.15
Table 1: A comparison of ITA and our baseline.
Approach	Twitter-15	Twitter-17	SNAP
REPORTED F1 OF PREVIOUS APPROACHES
BERT-CRF†	71.81	83.44	-
OCSGA*	72.92	-	-
UMT†	73.41	85.31	-
RIVA‡	73.80	-	86.80
RpBERTbase	74.40	-	87.40
UMGFdiamond	74.85	85.51	-
OUR REPRODUCTIONS
BERT-CRF	74.79	85.18	85.98
UMT	72.83	84.88	-
UMGF	74.42	85.27	-
RpBERTbase	67.21	-	62.14
Ours: ITA-All+CVA	76.01	86.45	87.44
Table 2: A comparison of our approaches and state-of-the-art approaches.  $\clubsuit$ : Wu et al. (2020);  $\dagger$ : results are from Yu et al. (2020);  $\ddagger$ : Sun et al. (2020),  $\spadesuit$ : Sun et al. (2021), note that  $\mathbf{RpBERT}_{\mathrm{base}}$  uses the test set to select the best model;  $\diamond$ : results are from Zhang et al. (2021a).
$5 \times 10^{-5}$  with a batch size of 16. For XLMR embeddings, we use a learning rate of  $5 \times 10^{-6}$  and a batch size of 4 instead. For the learning rate of the CRF layer, we use a grid search over  $[0.05, 0.5]$  and  $[0.005, 0.05]$  for BERT and XLMR respectively. The MNER models are trained for 10 epochs and we report the average results from 5 runs with different random seeds for each setting.
In Table 1, we compare our approaches with our baselines with different training and evaluation modalities (T for the text-only input view and  $\mathbf{I} + \mathbf{T}$
3Twitter-15 and 17 datasets are available at https://github.com/jefferyYu/UMT.
4github.com/microsoft/Oscar
5github.com/tesseract-ocr/tesseract
$^6$ Please refer to Appendix A.2 for more statistics.
for the multi-modal input view). Results show that ITA models are significantly stronger than our BERT-CRF and XLMR-CRF baselines (Student's t-test with  $p < 0.05$ ). For the aligned visual contexts, LA, GA and OCA are competitive in most of the cases. To show the effectiveness of CVA, we report the evaluation results of both input views in evaluation. With CVA, the accuracy of both input views can be improved, especially the T input view. CVA can improve the T input view to be competitive with I+T input view. Moreover, the combination of all the alignments ITA-All+CVA can further improve the model accuracy in most of the cases. The accuracy of the MNER models can be significantly improved if we use XLMR embeddings, which shows the importance of the text modality in MNER. With XLMR embeddings, the model accuracy can be further improved with ITA. The relative improvements over the baseline models are sometimes higher with XLMR than with BERT, which shows that the visual contexts can be further utilized with stronger embeddings.