SlowGuess/NAACL_2022_OCR huggingface dataset:
Yongfei Liu<sup>1</sup> Chenfei Wu<sup>3</sup> Shao-yen Tseng<sup>2</sup> Vasudev Lal<sup>2</sup> Xuming He<sup>1</sup> Nan Duan<sup>3</sup>
$^{1}$  ShanghaiTech  $^{2}$  Intel Labs, Cognitive Computing Research  $^{3}$  Microsoft Research Asia
{liuyf3, hexm} @ shanghaiitech.edu.cn {chewu, nanduan} @ microsoft.com
{vasudev.lal, shao-yen.tseng}@intel.com
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multimodal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. More importantly, we propose to perform object knowledge distillation to facilitate learning cross-modal alignment at different semantic levels. To achieve that, we design two novel pretext tasks by taking object features and their semantic labels from external detectors as supervision: 1.) Object-guided masked vision modeling task focuses on enforcing object-aware representation learning in the multi-modal Transformer; 2.) Phrase-region alignment task aims to improve cross-modal alignment by utilizing the similarities between noun phrases and object labels in the linguistic space. Extensive experiments on a wide range of vision-language tasks demonstrate the efficacy of our proposed framework, and we achieve competitive or superior performances over the existing pretraining strategies.
With the success of BERT (Devlin et al., 2018) in language modeling, self-supervised Vision-and-Language Pretraining (VLP) has attracted much interest from AI community, which aims to learn generalizable multi-modal representations from large-scale image-text data. Combined with a pretrainthen-transfer strategy, it shows great potential in
tackling vision and language reasoning tasks, such as image-text retrieval, visual question answering (VQA) and visual entailment (Antol et al., 2015; Lee et al., 2018; Xie et al., 2019; Liu et al., 2021, 2020). A critical step in such representation learning is to jointly model linguistic entities and visual semantic concepts (e.g., attributes, objects, and relations), as well as their alignment. However, this is particularly challenging due to large discrepancy in visual and language representations (pixels vs words) and lack of entity-level cross-modal correspondence in supervision.
To tackle those challenges, most existing approaches (Li et al., 2021; Gan et al., 2020; Chen et al., 2020; Lu et al., 2019) adopt a two-step pretraining strategy that firstly utilizes off-the-shelf detectors to parse images into a set of object tokens, and then builds a multi-layer Transformer to learn visual and language embeddings jointly. In order to facilitate the multi-modal learning, those networks are typically trained via a set of carefully designed BERT-like objectives (e.g. Image-Text Matching). Despite its promising performance, the two-step strategy suffers from several limitations: 1) limited visual object concepts as the external detectors are trained on a predefined set of object categories; 2) lack of context cues outside of the object regions, which are crucial for complex reasoning tasks; 3) sub-optimal visual representation due to stage-wise training; and 4) computational inefficiency caused by additional detection modules. To overcome those limitations, recent works attempt to learn a joint visual-linguistic representations in an end-to-end manner (Huang et al., 2021, 2020; Xu et al., 2021; Kim et al., 2021). These methods directly take dense visual features from image grids as inputs to a multi-modal Transformer network, and hence do not rely on external object detectors in both pretraining and finetuning stages. Such model design significantly simplifies overall network architecture and allows deeper integration
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1589 - 1600 July 10-15, 2022 ©2022 Association for Computational Linguistics
between visual and language features. However, using grid-level features makes it difficult to capture object-level visual concepts, which often results in less expressive multi-modal representations and inferior performances in downstream tasks.
In this work, we propose a novel object-aware end-to-end (E2E) VLP approach that inherits the strengths of both types of pretraining strategies mentioned above. Our core idea, which we name KD-VLP, is to incorporate visual object concepts in the E2E multi-modal learning, which is instantiated by performing Knowledge Distillation from semantic objects (e.g., from the off-the-shelf detectors) during the pretraining stage. This allows the network to better capture object representations and hence facilitates learning the alignment of linguistic entities and visual concepts. To achieve this, we introduce two novel pretext tasks to perform object knowledge distillation based on a CNN+Transformer architecture: an object-based masked vision modeling task for enforcing object-aware feature embeddings, and a phrase-region alignment task for building correspondence between object regions and language entities.
Specifically, we adopt a typical CNN backbone+multi-modal Transformer model for the pretraining. Given an image-text pair, the visual backbone firstly computes a set of visual features on the image grid. Then a multi-layer Transformer takes the visual features and the corresponding text tokens as input to generate their multi-modal embeddings. Based on those embeddings, a set of task-specific heads compute the corresponding objectives to train the entire network in an end-to-end fashion. Here, in addition to the commonly-used image-text matching and masked language modeling objectives, we develop two object-aware pretext tasks. The first task, object-guided masked vision modeling (OMVM), aims to reconstruct the RoI features and semantic label of each object (from an external detector) using the surrounding visual context and text description. To facilitate cross-modal alignment, we also develop a knowledge-guided masking strategy, which samples object candidates for reconstruction according to the similarity scores between the noun phrases in the corresponding text and their semantic labels. The second task, phrase-region alignment (PRA), aims to further improve cross-modal alignment by matching the above-mentioned phrase-label similarity scores of
each phrase with the cross-modal similarity scores between the noun phrase embeddings and object region embeddings. After pretraining, we then transfer the learned multi-modal representations to different downstream vision-language tasks.
We perform pretraining on two widely-used indomain datasets: MSCOCO Caption (Lin et al., 2014) and Visual Genome (Krishna et al., 2016), and validate the learned multi-modal representations on five well-known visual-language tasks: Visual Question Answering (VQA), Image-text retrieval, Nature Language Visual Reasoning  $(\mathrm{NLVR}^2)$ , Visual Entailment (VE) and Visual Commonsense Reasoning (VCR). Empirical results show that our method outperforms the state-of-the-art end-to-end approaches by a sizeable margin. To better understand our method, we also provide a detailed ablation study and visualization.
The contributions of our work are three-fold:
- We propose a novel end-to-end pretraining strategy, capable of better encoding visual object concepts and facilitating multi-modal representation learning.
- We design an object-guided masked vision model task for distilling knowledge from external object detectors, and a phrase-region alignment task to facilitate learning better phrase-region correspondence.
- Compared with existing methods, we achieve competitive or superior performances without using external detection outputs during finetuning stage and model test.
The existing self-supervised VLP approaches can be largely categorized into two groups: the two-step pretraining and the end-to-end pretraining, depending on whether they rely on visual object embeddings as input for the Transformer.
Two-step Pretraining firstly employ an off-the-shelf object detector to convert an image into a set of object embeddings, and then feed them into a Transformer jointly with text embeddings to generate their multi-modal representations. Hence their visual feature networks are not optimized during both pretraining & finetuning stage. Most of these methods, such as LXMERT (Tan and Bansal, 2019), ViLBert (Lu et al., 2019), VL-Bert (Su et al., 2020), Unicoder-VL (Li et al., 2020a) and
UNITER (Chen et al., 2020), adopt BERT-like objectives to train their networks, which include Masked Language Modeling (MLM), Masked Vision Modeling (MVM) and Image-Text Matching (ITM). In addition, VILLA (Gan et al., 2020) develops an advanced adversarial pretraining and finetuning strategy to improve generalization ability. OSCAR (Li et al., 2020b) and VINVL (Zhang et al., 2021) introduce object labels to bridge different modalities and revisit the importance of visual features. Ernie-ViL (Yu et al., 2020) exploits structured knowledge in the text and constructs scene graph prediction tasks to learn joint representations. UNIMO (Li et al., 2021) proposes a unified model to leverage large-scale free text corpus, image collections, and image-text pairs simultaneously through a contrastive learning task. Despite their strong performances, those methods are limited by the object detector and neglect visual cues outside of object regions, often leading to mistakes in downstream tasks.
End-to-End (E2E) Pretraining directly feed dense features on image grids from a visual backbone network into a Transformer network along with text tokens. As such, both the visual and Transformer networks are optimized jointly in an end-to-end manner in the pretraining & finetuning stage. Pixel-Bert and SOHO (Huang et al., 2021, 2020) pioneer the use of the E2E pretraining architecture and propose a novel visual-dictionary masked vision modeling task. E2E-VLP (Xu et al., 2021) presents a pretraining framework supervised with additional object detection and image captioning tasks to enhance visual semantics learning. It is worth noting that their object detection pretext task requires millions of bounding boxes annotation, unable to generalize to large-scale image-text corpus. ViLT (Kim et al., 2021) is the first to unify vision and language with a pure Transformer network, which has a simpler structure and enjoys faster inference. However, compared to the two-step methods, they are typically less expressive in terms of object-level concepts and thus suffer from weaker performances on challenging visual reasoning tasks. Our method is in line with the E2E pretraining framework. The key difference is that we propose to facilitate learning object-aware multi-modal representations by performing object semantic knowledge distillation.
3.1 Problem Definition and Overview
The goal of self-supervised VLP is to learn a generic and transferable visual-linguistic representation from a large amount of image-text data, which can achieve strong generalization performances in downstream vision-language tasks. To this end, the pretraining framework typically develops a variety of carefully-designed cross-modal pretext tasks (e.g. MLM, ITM) to train a deep network that encodes the multi-modal representation. Formally, we denote the image-text corpus for training as  $\mathcal{X} = \{(I^i,D^i)\}_{i = 1}^{|\mathcal{X}|}$  where  $I$  represents the image and  $D$  is the corresponding language description. In general, we construct a pretraining network consisting of a representation network module  $\mathcal{M}_{\theta}$  and a set of task-specific network heads  $\{\Phi_{\theta_s}\}_{s = 1}^S$  where  $s$  indicates the pretext tasks. The overall pretraining objective is defined as follows,
$$
\min  _ {\theta , \theta_ {1}, \dots \theta_ {S}} \mathbb {E} _ {(I, D) \sim \mathcal {X}} \left[ \sum_ {s} L _ {s} \left(Y _ {s}, \Phi_ {\theta_ {s}} \circ \mathcal {M} _ {\theta} (I, D)\right) \right] \tag {1}
$$
where  $Y_{s}$  and  $L_{s}$  are task-specific ground-truth label and loss function respectively, and  $\circ$  is a network compound operator. After pretraining, we remove all the task-specific heads and apply the representation network  $\mathcal{M}_{\theta^{*}}$  with the learned parameters  $\theta^{*}$  to the downstream tasks, followed by task-specific fine-tuning.
In this work, we aim to design an E2E pretraining strategy for the VLP problem. To this end, we adopt a modular representation network, which takes image grid features from a CNN-based visual network and the corresponding text embeddings into a multi-modal Transformer (Huang et al., 2020, 2021). Our goal is to learn the visual network and the Transformer jointly, and yet to effectively encode object-level visual concepts in the multimodal representations. This enables us to capture rich cross-modal alignment between linguistic entities and visual semantic concepts for the downstream tasks, and meanwhile to enjoy the benefits of an efficient E2E network design without relying on detectors during fine-tuning and inference.
To achieve this, we propose a set of cross-modal pretext tasks that perform object knowledge distillation from external detectors in both semantic and feature space. Specifically, in addition to the image-text matching (ITM) and masked language modeling (MLM) tasks, we introduce two novel pretext tasks, Object-Guided Masked Vision Modeling
Figure 1: Overview: The model contains a Visual Backbone for preparing image embeddings and a Transformer for vision & language fusion. The entire framework is supervised by two novel proposed pretext tasks: Object-guided Masked Vision Modeling (OMVM), Phrase-Region Alignment (PRA) as well as two standard tasks: Masked Language Modeling (MLM) and Image-Text Matching (ITM).
(OMVM) and Phrase-Region Alignment (PRA), which take the object RoI feature embeddings and semantic labels from external detectors as supervision. The OMVM task masks out the object regions and forces the network to predict the corresponding external RoI feature embeddings and object labels while the PRA task exploits object labels to encourage the alignment between visual objects and language entities. Fig.1 illustrates an overview of our framework. Below we will first present the details of model architecture in Sec.3.2, followed by our design of pretext tasks in Sec.3.3.
Given an image-text pair, our model firstly computes the image embeddings and linguistic embeddings respectively, and then concatenates them into a sequence of tokens with two additional tokens [sep] and [cls] as inputs to a Transformer for generating multi-modal contextualized embeddings.
Visual Embedding We adopt a CNN backbone to extract image features  $\mathcal{V} = \{\mathbf{v}_i\}_{i=1}^L$  for each image  $I$  where  $L$  is the size of feature grids and  $\mathbf{v}_i \in \mathcal{R}^{d_v}$  is a feature vector with dimension  $d_v$ . In addition, each feature is further concatenated with its 2-D sine position embedding (Carion et al., 2020). Following SOHO, we use a ResNet-101(He et al., 2016) as the visual backbone, followed by additional 1x1 Conv and 2x2 strides Max-pooling to reduce the memory footprint.
Linguistic Embedding For the language  $D$ , we first tokenize the sentence into a sequence of word tokens using WordPiece (Wu et al., 2016), then encode them into word embeddings  $\mathcal{W} = \{\mathbf{w}_j\}_{j=1}^T$
where  $\mathbf{w}_j\in \mathcal{R}^{d_w}$  is the feature vector. Similarly, an index position (Devlin et al., 2018) embedding is supplemented to each word embedding.
Multi-modal Transformer After obtaining image and linguistic embeddings, we assemble them into a sequence of tokens  $\{\mathcal{V},[\mathbf{sep}],\mathcal{W},[\mathbf{cls}]\}$ , and adopt a multi-layer Transformer to compute their representations encoded by the final-layer states  $\{\mathcal{H}_{\mathcal{V}},\mathbf{h}_{\mathbf{sep}},\mathcal{H}_{\mathcal{W}},\mathbf{h}_{\mathbf{cls}}\}$  where  $\mathcal{H}_{\mathcal{V}} = \{\mathbf{h}_{\mathbf{v}_i}\}_{i = 1}^L$  and  $\mathcal{H}_{\mathcal{W}} = \{\mathbf{h}_{\mathbf{w}_j}\}_{j = 1}^T$  represent the states for visual and language part respectively. Finally, those representations are sent into each pretext task head to compute the supervision signals.
3.3 Pretext Tasks
We now describe our cross-modal pretext tasks for the E2E pretraining, aiming to learn more effective multi-modal representations. Below we will first introduce objects-aware pretext tasks that take external object features and semantic labels as supervision, followed by the standard MLM and ITM.
Specifically, for each image, we first generate a set of object proposals from an off-the-shelf detector, denoted as  $\{(o_n,c_n,\mathbf{f}_n)\}_{n = 1}^N$  where  $o_{n}\in \mathcal{R}^{4}$  is box location,  $c_{n}$  indicates object category, and  $\mathbf{f}_n\in \mathcal{R}^{d_o}$  is object RoI features with dimension  $\mathcal{R}^{d_o}$ . For ease of notation, we also introduce a binary mask on the feature map for each object  $o_{n}$  and denote its flattened version as  $\mathbf{m}_n\in \mathcal{R}^L$ . For the corresponding text, we extract a set of noun phrases  $\mathcal{P} = \{p_z\}_{z = 1}^{|\mathcal{P}|}$  with an external language tool and calculate the similarity  $\alpha_{z,n}$  between each
<sup>1</sup>We give an illustration in Suppl.
2https://spacy.io/
noun phrase  $p_{z}$  and the object category  $c_{n}$  in the linguistic space:
$$
\alpha_ {z, n} = \operatorname {C o s} \left(E _ {\text {e x t}} \left(p _ {z}\right), E _ {\text {e x t}} \left(c _ {n}\right)\right), \tag {2}
$$
where  $\operatorname{Cos}(\cdot, \cdot)$  indicates cosine distance function and  $E_{ext}$  represents an off-the-shelf language embedding (e.g. BERT). Using them as supervision, we design two novel pretext tasks to distill object-level knowledge below.
Object-guided Masked Vision Modeling (OMVM) The first task aims to learn more explicit object concepts in the E2E pretraining. Specifically, we sample an object each time and mask out its features in the Transformer input, and enforce the network to generate external object RoI features and semantic labels. To learn better cross-modal alignment, we propose a knowledge-guided masking strategy, which samples noun phrase-related object regions to mask based on the (normalized) similarity score  $\alpha_{z,n}$ . The selected object region is denoted with its binary mask, category and RoI features, as  $(\mathbf{m}^{*},c^{*},\mathbf{f}^{*})$ .
We design two learning objectives, Masked Region Classification (MRC) and Masked Region Feature Regression (MRFR) as below
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {O M V M}} = \mathbb {E} _ {(I, D) \sim \mathcal {X}} L _ {\mathrm {M R C}} \left(c ^ {*}, \mathcal {V} _ {\backslash \mathbf {m} ^ {*}}, \mathcal {W}\right) \tag {3} \\ + L _ {\mathrm {M R F R}} \left(\mathbf {f} ^ {*}, \mathcal {V} _ {\backslash \mathbf {m} ^ {*}}, \mathcal {W}\right) \\ \end{array}
$$
To calculate the losses  $L_{\mathrm{MRC}}$  and  $L_{\mathrm{MRFR}}$ , we first compute the object representation  $\mathbf{h}_{\mathbf{m}^*}$  for the masked region at the final layer, which is average-pooled over  $\mathcal{H}_{\mathcal{V}}$  based on its binary mask  $\mathbf{m}^*$ . For MRC, a multi-layer FC network  $\Phi_{\mathrm{MRC}}$  is adopted to predict its object category. Thus,  $L_{\mathrm{MRC}} = \mathrm{CE}(\Phi_{\mathrm{MRC}}(\mathbf{h}_{\mathbf{m}^*}), c^*)$  is the standard cross-entropy loss. In addition, we take another FC network  $\Phi_{\mathrm{MRFR}}$  to learn the object concept in feature space directly by minimizing the L2 distance,  $L_{\mathrm{MRFR}} = ||\Phi_{\mathrm{MRFR}}(\mathbf{h}_{\mathbf{m}^*}) - \mathbf{f}^*)||_2^2$ .
Phrase Region Alignment (PRA) The second task, PRA, mainly focuses on learning cross-modal alignment at object-level, which aims to pull positive phrase-region pairs closer and push negative pairs away. Here we utilize the similarity  $\alpha_{z,n}$  between the noun phrase and object category in the linguistic space as a guidance.
Concretely, we first compute the object representation  $\mathbf{h}_{\mathbf{m}_n}$  for each proposal and the phrase representation  $\mathbf{h}_{p_z}$ , both of which are obtained from
the final layer states of the Transformer. Specifically,  $\mathbf{h}_{\mathbf{m}_n}$  is average-pooled over  $\mathcal{H}_{\mathcal{V}}$  based on binary mask  $\mathbf{m}_n$  while  $\mathbf{h}_{p_z} = \frac{1}{|p_z|}\sum_{j\in p_z}\mathbf{h}_{\mathbf{w}_j}$  represents average states of word tokens within  $p_z$ . We define the cross-modal similarity as  $\hat{\alpha}_{z,n} = \mathrm{Cos}(\mathbf{h}_{p_z},\mathbf{h}_{\mathbf{m}_n})$
The task PRA minimizes the KL-divergence between the cross-modal similarities  $\hat{\alpha}_z = \{\mathrm{Softmax}(\hat{\alpha}_{z,n})\}_{n = 1}^N$  and the phrase-label similarities  $\alpha_{z} = \{\mathrm{Softmax}(\alpha_{z,n})\}_{n = 1}^{N}$  as below:
$$
L _ {\mathrm {P R A}} = \frac {1}{| \mathcal {P} |} \sum_ {z} D _ {K L} \left(\hat {\alpha} _ {z} \| \alpha_ {z}\right) \tag {4}
$$
Finally, denoting the mask set  $\mathcal{M} = \{\mathbf{m}_n\}_{n=1}^N$ , we have the overall PRA loss function as follows:
$$
\mathcal {L} _ {\mathrm {P R A}} = \mathbb {E} _ {(I, D) \sim \mathcal {X}} L _ {\mathrm {P R A}} \left(\left\{\alpha_ {z, n} \right\} _ {z, n = 1} ^ {| \mathcal {P} |, N}, \mathcal {M}, \mathcal {P}, \mathcal {V}, \mathcal {W}\right) \tag {5}
$$
Masked Language Modeling (MLM) We take the same masking strategy (15% prob. to mask) as in BERT (Devlin et al., 2018) to randomly mask out the input word tokens. Here, MLM aims to predict the original word index in vocabulary space for each masked token based on the whole image and its surrounding language context via the Transformer. Hence a cross-entropy loss is adopted:
$$
\mathcal {L} _ {\mathrm {M L M}} = - \mathbb {E} _ {(I, D) \sim \mathcal {X}} \log P (\mathbf {w} _ {j} | \mathcal {V}, \mathcal {W} _ {\backslash j}) \tag {6}
$$
Image-Text Matching (ITM) In ITM, the multi-layer Transformer is trained to distinguish whether the input image-text pairs are semantically matched based on the final layer [cls] token representation  $\mathbf{h}_{\mathrm{cls}}$ . To construct the training samples, we randomly replace the text for each image-text pair with another text from dataset with a probability of 0.5. Thus, the output label can be defined as  $y \in \{0,1\}$  where  $y = 1$  indicates matched pair. The training objective for the ITM task is to minimize binary cross-entropy loss:
$$
\mathcal {L} _ {\mathrm {I T M}} = - \mathbb {E} _ {(I, D) \sim \mathcal {X}} \log P (y | \mathcal {V}, \mathcal {W}) \tag {7}
$$
Pretraining Corpus: Following the E2E pretraining strategy (Huang et al., 2021, 2020; Xu et al., 2021), we take indomain datasets: MSCOCO (Lin et al., 2014) and VG (Krishna et al., 2016) as pretraining datasets since it is widely used in literature. In total, two datasets comprise about 200K images and 5.6M image-text pairs, where each image is associated with multiple captions.