SlowGuess/NAACL_2021_OCR huggingface dataset:
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the National Conference on Artificial Intelligence (AAAI).
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the International Conference on Computer Vision (ICCV).
A Fine-Tuning on Downstream Tasks
We describe the details of fine-tuning on the four downstream tasks: Visual Question Answering (VQA 2.0) (Goyal et al., 2017), Natural Language for Visual Reasoning  $(\mathrm{NLVR}^2)$  (Suhr et al., 2019), Image Retrieval (Flickr 30K) (Plummer et al., 2015), and Referring Expression (RefCOCO+) (Yu et al., 2016).
VQA Given an image and a question, the task is to correctly answer the question. We use the VQA 2.0 and use the Karpathy split for training and validation (Karpathy and Fei-Fei, 2015). We fine-tune with a binary cross-entropy loss. The model is trained with a batch size of 32 and a peak learning rate of  $5 \times 10^{-5}$  over 8 epochs.
$\mathbf{NLVR}^2$ $\mathbf{NLVR}^2$  involves determining whether a natural language caption is true about a pair of images. While more sophisticated fine-tuning strategy exists (Chen et al., 2020c), we follow LXMERT (Tan and Bansal, 2019) to pair the caption with each image, concatenate the “[CLS]” representation of the two pairs, and build a classifier on top. We find it beneficial to conduct a moderate amount of “task-specific pre-training” where we use the data from the dataset to conduct mask-and-predict pre-training as suggested by VisualBERT (Li et al., 2019). We conduct task-specific pre-training for at most 5 epochs and fine-tune from the epoch with the best validation LM loss. Fine-tuning is conducted for 8 epochs with a batch size of 32 and a peak learning rate of  $2 \times 10^{-5}$ .
Flickr30K The task of image retrieval involves finding the corresponding image from a collection of images given a caption. We follow the split of
Lee et al. (2018) and use 1,000 images for validation and test each and train on the rest of the dataset. During fine-tuning, we follow UNITER (Chen et al., 2020c) and sample two negative text-image pairs along with a positive sample. We train for 5K steps with a batch size of 8 and a peak learning rate of  $5 \times 10^{-5}$ .
RefCOCO+ The referring expression task involves locating an image region given a natural language phrase. We follow ViLBERT (Lu et al., 2019) and conduct evaluation on the RefCOCO+ dataset. We use the bounding box proposals provided by Yu et al. (2018). For each box proposal, the model is trained to classify if it matches the reference phrase or not. A proposal box is considered correct if it has an IoU with the gold box larger than 0.5. We train for 12 epochs with a batch size of 32 and a peak learning rate of  $5 \times 10^{-5}$ .
B Data Accessibility
The version of BookCorpus we used is downloaded from https://github.com/jackroos/VL-BERT/blob/master/data/PREPARE_DATA.md. The other datasets we used including Conceptual Captions, Open Images, VQA, NLVR², Flickr30K, and RefCOCO+ are publicly available.