SlowGuess/NAACL_2021_OCR huggingface dataset:
Liuian Harold Li†, Haoxuan You*°, Zhecan Wang*°, Alireza Zareian°, Shih-Fu Chang° & Kai-Wei Chang†
liunian.harold.li@cs.ucla.edu,
{hy2612, zw2627, az2407, sc250}@columbia.edu,
Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks. However, existing models require a large amount of parallel image-caption data for pre-training. Such data are costly to collect and require cumbersome curation. Inspired by unsupervised machine translation, we investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct "mask-and-predict" pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. We find that such a simple approach achieves performance close to a model pre-trained with aligned data, on four English V&L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V&L pre-training, while significantly reducing the amount of supervision needed for V&L models.
Pre-trained contextual vision-and-language (V&L) models (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Su et al., 2019; Chen et al., 2020c) have achieved high performance on various V&L tasks. However, different from contextual language models, such as BERT (Devlin et al., 2019a), which are trained on easily-accessible unannotated text corpora, existing V&L models are still a step away from self-supervision. They require a massive amount of aligned text-image pairs for "mask-and-predict" pre-training. Such aligned data are costly to collect and hard to scale up. For example, the widely used MS-COCO dataset (Chen et al., 2015) requires extensive
annotation from crowd workers.
In this paper, we explore unsupervised V&L pre-training with unaligned image and text corpora. This research direction aligns with the theme of unsupervised and self-supervised learning that moves from heavily-annotated data to unannotated data, e.g. unsupervised machine translation (Lample et al., 2018) and unsupervised image captioning (Feng et al., 2019). Unsupervised V&L pretraining is highly desirable as in many domains, aligned data is scarce (e.g. multimodal hate speech detection (Kiela et al., 2020) and the medical domain (Li et al., 2020c)) and it is easier to collect unaligned text and images. In addition to its practical implication, our endeavour challenges the widely held notion that image-caption corpora is indispensable for pre-training (Lu et al., 2019) and brings valuable insight into the role that aligned data play in V&L pre-training.
We are inspired by works on multi-lingual contextual language models (Pires et al., 2019). If we treat an image as a set of regions and each region as a visual token (Dosovitskiy et al., 2020), V&L models share a similar goal with multi-lingual models as they both learn shared representations across different domains. Although a multi-lingual language model pre-trained on non-parallel corpora such as mBERT (Devlin et al., 2019b) cannot align or translate languages out-of-the-box, its representation spaces for different languages can be easily aligned with a linear probe (Conneau et al., 2020). This property suggests the existence of universal latent symmetries in the unaligned contextual embedding spaces and is believed to contribute to
<sup>1</sup>Other datasets also require cumbersome curation. For example, while Conceptual Captions is crawled from the web, the authors report that from 5 billion images gathered over the Internet, only 3 million have paired high-quality captions after filtering (Sharma et al., 2018; Changpinyo et al., 2021).
$^{2}$ Following Lample et al. (2018) and Feng et al. (2019), we use the term "unsupervised" to refer to pre-training with unaligned data, while "supervised" refers to pre-training with aligned text and images.
*The two authors contributed equally.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5339-5350 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: An illustration of pre-training without aligned data. Given text, the model is trained to predict masked words; given an image, the model is trained to predict masked regions and detector tags. The semantic class "cake" appears in both the language modality and the visual modality and is linked through the detector tags. Note that we do not require a text segment with the word cake to appear together with the image. Rather, we assume that as long as the text corpora are general enough, the word cake will appear in the textual modality eventually. The model can thus learn V&L representations from such weak supervision signals.
mBERT's cross-lingual transfer ability. Thus we hypothesize that strong V&L representations can be similarly learned by "mask-and-predict" pretraining on unaligned language and vision data.
We propose unsupervised V&L pre-training with unaligned text and images (see an illustration in Figure 1). Specifically, we take VisualBERT (Li et al., 2019) as a running example and apply unsupervised pre-training, resulting in Unsupervised VisualBERT (U-VisualBERT). The model takes the form of a single Transformer that can accept inputs from both modalities. During each step of pre-training, unlike the existing models that observe a batch of text-image pairs, our model observes either a batch of text segments or a batch of images. When provided with text, part of the text is masked and the model is trained to predict the masked words; when provided with an image, part of the image regions are masked and the model is trained to predict properties of the masked regions.
To further encourage cross-modal fusion, we leverage the tags from an object detector as "anchor points" (Li et al., 2020b). For every object, we append its detected tag as a word to the visual input. The mask-and-predict objective is applied to the tags. For instance, for the image in Figure 1, the model can observe "cake" appears naturally as a word, a tag, and an image region. The direct typing of image regions and words can be learned and serves as a starting point for further alignment. The function of the detector tags resembles that of the "overlapping vocabulary" in multi-lingual language models, i.e., identical strings that appear in differ
ent languages with the same meanings (e.g., "DNA" appears in both English and French). As the "overlapping vocabulary" improves cross-lingual transfer (Wu and Dredze, 2019), we argue the detector tags can improve cross-modal grounding.
We first conduct controlled experiments by pretraining on an English image-caption corpus without providing the alignment, following unsupervised machine translation and image captioning (Gu et al., 2019). Results on four English V&L benchmarks (VQA (Goyal et al., 2017), NLVR² (Suhr et al., 2019), Flickr30K Image Retrieval (Plummer et al., 2015), and RefCOCO+ (Yu et al., 2016)) show that U-VisualBERT achieves comparable performance as models with access to text-image pairs (Section 4).
Additionally, our approach is effective in practical settings, 1) when using independently collected images and captions and 2) when using images and general-domain text (BookCorpus (Zhu et al., 2015)) without any captions (Section 5.1). Quantitative and qualitative analysis confirms the anchoring effect of the detector tags (Section 5.2). As a byproduct, we conduct preliminary experiments to show the promise of the approach in a semi-supervised setting, where a hybrid model pre-trained with both aligned and additional unaligned data surpasses a model pre-trained only on aligned data. (Section 6). The above experiments demonstrate the wide applicability of our method. We will open-source the project under https://github.com/uclanlp/visualbert.
Pre-trained V&L Transformers Various V&L models that are pre-trained with a "mask-and-predict" objective on aligned text-image data have been proposed (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Su et al., 2019; Chen et al., 2020c; Li et al., 2020a; Zhou et al., 2020; Huang et al., 2020; Yu et al., 2020; Gan et al., 2020). Two kinds of designs have been proposed. Two-stream models (Lu et al., 2019; Tan and Bansal, 2019; Yu et al., 2020) utilize separate Transformers (Vaswani et al., 2017) for each modality and a cross-modality module is adopted. Single-stream models (Li et al., 2019; Su et al., 2019; Chen et al., 2020c) directly input the text and visual embeddings into one single Transformer. They have been widely used by downstream tasks (Kiela et al., 2020). Probing tasks (Cao et al., 2020) confirm that they capture useful V&L information after pre-training.
Two studies also try to incorporate "tag" information during pre-training. Oscar (Li et al., 2020b) adds detected tags as additional signals when pretraining with aligned data. We, however, do so for pre-training with unaligned data and show that the tags serve a more important role in unsupervised pre-training (Section 5.2). VIVO (Hu et al., 2020) targets novel object captioning. They use manually annotated image-tag data for pre-training and image-caption data for fine-tuning. We do not use manually annotated data and the tags are noisily generated by a detector.
Self-supervised Representation Learning
Self-supervision involves creating supervision objectives from natural data, often by corrupting the input and training the model to reconstruct the input (Kolesnikov et al., 2019) or contrastive learning (Chen et al., 2020b). Self-supervised training on language (Peters et al., 2018; Devlin et al., 2019a) such as BERT has been proven useful for various NLP tasks (Liu et al., 2019), while self-supervised visual representation learning has been centered around learning low-level visual features, in hope of enhancing the backbone CNN (Doersch et al., 2015; Pathak et al., 2016; Noroozi and Favaro, 2016; Chen et al., 2020b). In this paper, we conduct V&L pre-training by optimizing a reconstructive objective on unlabeled language-only and image-only data. Thus, our proposed model could be regarded as "self-supervised". Notably, our contextual visual representation is
built on top of a pre-trained detector, operating at a level above local visual features.
Unsupervised Multi-lingual Language Model This work is inspired by multi-lingual representations trained without parallel corpora (Devlin et al., 2019b). They are effective for cross-lingual transfer, which involves learning a model in one language and applying it to another with no additional training. Studies (Wu and Dredze, 2019; Conneau et al., 2020) have confirmed several design choices that facilitate such transfer, e.g. shared parameters and overlapping vocabularies across languages, and we make similar design choices in U-VisualBERT (Section 3.2). We argue that multi-lingual representations bear resemblance to multi-modal representations as both seek to encode the alignment between two domains (Chen et al., 2020a).
Unsupervised Grounding Learning Prior works have explored learning grounding with weak or no supervision (Rohrbach et al., 2016; Xiao et al., 2017; Wang et al., 2020). Closest to this paper is unsupervised image captioning (Feng et al., 2019; Laina et al., 2019; Gu et al., 2019), which conducts image captioning with unpaired images and captions. Similar to this work, the detector tags serve as the anchor points for image captioning. However, unsupervised image captioning still requires captions, while our approach works with easy-to-collect general-domain text without any caption text (Section 5.1).
We first take Supervised VisualBERT (S-VisualBERT) as an example and illustrate how a typical V&L model is pre-trained with aligned data. Then we introduce unsupervised V&L pre-training, and the resulting model Unsupervised VisualBERT (U-VisualBERT).
As mentioned in Section 2, there are several V&L representation learning methods based on BERT. We take Supervised VisualBERT (S-VisualBERT) as an example, which will also be used as a baseline in the experiments. S-VisualBERT is modified from the original VisualBERT (Li et al., 2019) and augmented with the visual objectives from LXMERT (Tan and Bansal, 2019) and detector tags similar to Oscar (Li et al., 2020b) (discussed in detail in Section 3.2).
Every input to S-VisualBERT contains a text segment  $T$  and an image  $I$ . The text and the image are first mapped into embedding vectors respectively. Text embeddings  $T$  is a matrix in which each column vector represents the embedding of a subword in the text sequence, i.e.  $T = [w_{1:n}]$ . Following BERT, each subword embedding  $w_{i}$  is the sum of its token, position, and segment embedding. Image embeddings  $I$  include both the image region embeddings  $r_{1:m}$  and the detector tag embeddings  $d_{1:l}$  (see Section 3.2 for details). Each region embedding  $r_{i}$  is the sum of a visual feature vector from the detector and a spatial box coordinate embedding (Tan and Bansal, 2019). The text and visual embeddings are then passed through a Transformer to built contextual representations.
The model is pre-trained with a mask-and-predict objective. Given a text-image pair  $[T, I]$  from the aligned dataset  $D$ , we randomly mask out some words  $w_i$ , some regions  $r_j$ , and some tags  $d_k$  to obtain masked  $[\tilde{T}, \tilde{I}]$ . The model is trained to predict the masked words, the properties of the masked regions, and the masked tags given  $[\tilde{T}, \tilde{I}]$ . The pre-training objective can be summarized as:
$$
\min _ {\theta} \sum_ {[ T, I ] \in D} L _ {T + I + M} \left(f _ {\theta} ([ \tilde {T}, \tilde {I} ]), [ T, I ]\right).
$$
$f_{\theta}$  represents the embedding layer and the multilayer Transformer.  $L_{T + I + M}$  is the sum of 1) the masked language model loss  $L_{T}$ , 2) the image reconstruction loss  $L_{I}$ , and 3) an "text-image match" objective  $L_{M}$ . Specifically,  $L_{I}$  includes a tag reconstruction loss  $L_{I}^{tag}$  (more details in Section 3.2) and the two visual losses as in LXMERT (Tan and Bansal, 2019): the region feature regression loss  $L_{I}^{ref}$ , which forces the model to regress to the visual vector, and the noisy label classification loss  $L_{I}^{cls}$ , which predicts the detected labels of masked objects with the cross-entropy loss. With a probability of 0.5, we provide the model with a mismatched text-image pair instead of a matched pair, and  $L_{M}$  asks the model to predict whether the image matches the text. After the model is pre-trained, it can be fine-tuned for V&L tasks similar to how BERT is fine-tuned for NLP tasks.
3.2 Unsupervised Pre-training
We introduce the two core design choices of unsupervised pre-training: mask-and-predict pretraining with unaligned data and the detector tags.
Mask-and-Predict Pre-training with Unaligned Data We assume access to a text corpus  $D_{T}$  and an image corpus  $D_{I}$  for pre-training. During every pre-training step, we randomly sample either a batch of text from  $D_{T}$  or a batch of images from  $D_{I}$ . No alignment between text and images is provided to the model. When pre-training with a text segment  $T$ , the model is trained to reconstruct  $T$  given the masked  $\tilde{T}$ . When pre-training with an image  $I$ , the model is trained to reconstruct  $I$  given the masked  $\tilde{I}$ . A single Transformer is used throughout two modalities (i.e.  $\theta$  shared across modalities). The pre-training objective can be summarized as:
$$
\min _ {\theta} \sum_ {T \in D _ {T}} L _ {T} (f _ {\theta} (\tilde {T}), T) + \sum_ {I \in D _ {I}} L _ {I} (f _ {\theta} (\tilde {I}), I).
$$
After pre-training, the model is fine-tuned on downstream tasks just as its supervised counterpart, with the input being a text-image pair.
Detector Tags While mask-and-predict pretraining with unaligned data in itself achieves nontrivial performance (Section 5.2), we find it beneficial to provide noisy alignment signals in the form of the detector tags. When modeling an image  $I$ , for each region detected, we append the tag outputted by the object detector to the input. The detector (Ren et al., 2015) is pre-trained on a general object detection dataset (Krishna et al., 2017; Anderson et al., 2018) and the tags are essentially a bag of words that provide some noisy grounding signals to the model. During pre-training, we apply the mask-and-predict objective to the tags, which further encourages grounding.
We process the detector tags as a subword sequence  $d_{1:l}$  with spatial coordinates. Every tag subword is embedded as the sum of its token embedding and a spatial coordinate embedding. The token embedding is the same as the token embedding used in text modeling, while the spatial coordinate embedding is the same as the coordinate embedding of the corresponding region. The coordinate embedding allows the model to distinguish tags from different regions. With the de
3We adopt the next sentence prediction task in BERT when long documents are available.
Each tag corresponds to a region. A tag could be split into multiple subwords, so the total length of the tag subword sequence  $l$  is equal to or larger than the number of regions  $m$ .
5This design differs from that of Oscar (Li et al., 2020b). Oscar does not add the coordinate embeddings to tags to encourage the fusion of tag and visual representations.
tector tags added, the image  $I$  is embedded as a sequence of image region features  $\boldsymbol{r}_{1:m}$  followed by a sequence of detector tag embeddings  $d_{1:l}$ , i.e.  $\boldsymbol{I} = [r_{1:m}; d_{1:l}]$ . The tags are added during both pre-training and fine-tuning. Further, during pretraining, certain tag subwords are masked and the tag reconstruction loss  $L_I^{tag}$  supervises the model to predict the masked tags. The tags are predicted just as masked subwords are predicted in text modeling. The prediction softmax layer is shared between the tag and text subwords.
The parameters involved in modeling tags include the token embedding, the coordinate embedding, and the subword softmax embedding. These embedding parameters are shared across modalities and encourage the model to project text, visual, and tag representations into the same space (see Section 5.2 for an example). This resembles the design in multi-lingual language models, which use shared BPE embeddings and softmax weights across languages (Wu and Dredze, 2019).
As the domain and quality of data may affect the model performance, the conventional practice in unsupervised learning is to use aligned corpora without providing alignments, allowing for controlled comparison with a supervised model. For example, unsupervised machine translation creates unaligned corpora by splitting up parallel corpora (Lample et al., 2018) while unsupervised image captioning (Gu et al., 2019) create unaligned corpus by shuffling images and captions from MSCOCO (Chen et al., 2015). Following prior work, we first conduct experiments by using Conceptual Captions (CC) (Sharma et al., 2018) as the source of images and text for both the supervised and unsupervised model. Later in Section 5.1, we show that our method is effective when the images and captions are collected independently and when no caption text is used.
U-VisualBERT The model is pre-trained with shuffled captions and images. At each training step, we sample either a batch of images or a batch of text. Following VL-BERT (Su et al., 2019), we find it beneficial to include BookCorpus (Zhu et al., 2015), a general-domain text corpus, during pretraining. In sum, U-VisualBERT is trained on 3M images from CC, 3M captions from CC, and 2.5M
text segments from BookCorpus<sup>6</sup>.
S-VisualBERT We introduce a Supervised VisualBERT (S-VisualBERT) trained with aligned data as introduced in Section 3.1. S-VisualBERT is pretrained on 3M caption-image pairs from CC and 2.5M text segments from BookCorpus.
Compared Models Additionally, we list the performance of a Base VisualBERT that is initialized from BERT and does not undergo further pre-training. Previously reported supervised models that are trained on CC are also listed, including ViLBERT, VL-BERT, and UNITER. For UNITER, we include the version that is trained only on CC (UNITERcc) $^7$ . Although their network architectures differ from ours and cannot be directly compared, they jointly paint the picture of the performance we should expect by pre-training on CC. Models developed before BERT are listed as Pre-BERT (Gao et al. (2019) for VQA, Suhr et al. (2019) for NLVR $^2$ , Lee et al. (2018) for Flickr30K, and Yu et al. (2018) for RefCOCO+).
Setup For all the VisualBERT variants introduced in the paper, we initialize them from  $\mathrm{BERT}_{\text {base }}$  and pre-train for 10 epochs on their respective pre-training datasets with a batch size of 144. All models can be trained within 3 days on 4 V100s each with 16GB of memory. We use the Adam optimizer (Kingma and Ba, 2015) with a linear-decayed learning-rate schedule (Devlin et al., 2019a) and a peak learning rate at  $6 \times 10^{-5}$ . We conduct evaluations by fine-tuning on four downstream tasks: Visual Question Answering (VQA 2.0) (Goyal et al., 2017), Natural Language for Visual Reasoning (NLVR²) (Suhr et al., 2019), Image Retrieval (Flickr 30K) (Plummer et al., 2015), and Referring Expression (RefCOCO+) (Yu et al., 2016). We use a Faster R-CNN pre-trained on the Visual Genome dataset to extract region features (Anderson et al., 2018). For each task, we follow the recommended setting in previous works. For details, please refer to the appendix.
Results Table 1 summarizes the results. For each model, we list the type and amount of data used
<sup>6</sup>Our version of BookCorpus contains around 5M text segments with 64 words per segment. For computational reasons, we downsample the dataset such that during each epoch, the model observes only half of the text segments from BookCorpus. This downsampling is also done for the other VisualBERT variants.
The results are from Appendix A.6 of Chen et al. (2020c).
Model	Aligned	Unaligned
Image Text	VQA
Test-Dev	NLVR2	Flickr30K	RefCOCO+
Dev	Test-P	R@1	R@5	R@10	Dev	TestA	TestB
Pre-BERT	-	-	70.22	54.1	54.8	48.60	77.70	85.20	65.33	71.62	56.02
ViLBERT	3M	0	70.55	-	-	58.78	85.60	91.42	72.34	78.52	62.61
VL-BERT	3M	0	71.16	-	-	-	-	-	71.60	77.72	60.99
UNITERcc	3M	0	71.22	-	-	-	-	-	72.49	79.36	63.65
S-VisualBERT	3M	0	70.87±.02	73.44±.51	73.93±.51	61.19±.06	86.32±.12	91.90±.02	73.65±.11	79.48±.36	64.49±.22
Base	0	0	69.26	68.40	68.65	42.86	73.62	83.28	70.66	77.06	61.43
U-VisualBERT	0	3M	70.74±.06	71.74±.24	71.02±.47	55.37±.49	82.93±.07	89.84±.21	72.42±.06	79.11±.08	64.19±.54
Table 1: Evaluation results on four V&L benchmarks. Our unsupervised model trained with unaligned data (U-VisualBERT) achieves close performance with a supervised model trained with aligned data (S-VisualBERT). U-VisualBERT also rivals with several supervised models such as ViLBERT on most metrics.
Model	Text	VQA	NLVR2	Flickr30K	RefCOCO+
Caption	General	Test-Dev	Dev	Test-P	R@1	R@5	R@10	Dev	TestA	TestB
Base	-	-	69.26	68.40	68.65	42.86	73.62	83.28	70.66	77.06	61.43
U-VisualBERT	CC	BC	70.74	71.74	71.02	55.37	82.93	89.84	72.42	79.11	64.19
U-VisualBERTSBU	SBU	BC	70.70	71.97	72.11	56.12	82.82	90.12	73.05	79.48	64.19
U-VisualBERTNC	-	BC	70.47	71.47	71.19	54.36	82.22	89.24	72.96	79.30	64.25
Table 2: Unsupervised pre-training is applicable when images and captions are collected independently (U-VisualBERT $_{SBU}$ ) or when no caption text is provided (U-VisualBERT $_{NC}$ ).
during pre-training. $^{8}$  To control for randomness, we report the means and standard deviations of U-VisualBERT and S-VisualBERT across three runs.
U-VisualBERT outperforms the Base model on all benchmarks, while only lagging behind S-VisualBERT slightly on VQA, NLVR $^2$ , and RefCOCO+. U-VisualBERT even surpasses or rivals with some supervised models (e.g., ViLBERT on VQA and RefCOCO+, VL-BERT on RefCOCO+, and UNITER $_{\mathrm{cc}}$  on RefCOCO+). This shows that a model through unsupervised pre-training can perform comparably with supervised models.