SlowGuess/NAACL_2022_OCR huggingface dataset:
Natural Language Visual Reasoning  $(\mathbf{NLVR}^2)$ : NLVR² aims to determine whether a natural caption is true about a pair of photographs, which is full of semantic diversity, compositionality challenges. We follow UNITER to construct two imag-text pairs for each sample and concatenate their hcls features to infer true or false. All finetuning strategies are the same as before except for a batch size 12 per GPU.
Visual Question Answering (VQA): VQA requires a richer multi-modal understanding to solve the free-form and open-ended questions. VQA dataset contains 204K images from MSCOCO, 614K free-from nature language question and around 6M answers. It is typically formulated as a 3192-way classification problem and supervised by binary cross-entropy loss as in UNITER. The batch size here is 32 per GPU while other finetuning strategies are kept the same.
Visual Commonsense Reasoning (VCR): Given a question for an image, VCR needs to 1.) correctly answer  $(\mathrm{Q}\rightarrow \mathrm{A})$ ; 2.) provide a rationale justifying its answer  $(\mathrm{QA}\rightarrow \mathrm{R})$ ; 3.) reason both of them  $(\mathrm{Q}\rightarrow \mathrm{AR})$ , which requires higher-order cognition and commonsense reasoning about the world. Following UNITER, we introduce a second-stage pretraining over the VCR dataset due to severe difference in dataset distribution compared to indomain image-text corpus. In addition, we also utilize a similar person grounding (Park et al., 2020) pretext task to tightly align the person tags in text and their visual locations. During finetuning stage, we concatenate each question along with each possible answer to form four kinds of text inputs, and feed each of them into Transformer network with corresponding image embeddings. Finally, a binary cross-entropy loss is adopted to supervise each pair. Since VCR questions explicitly reference objects at specific locations, we implement coreferencing between text and image by replacing referenced entities in the questions with their corresponding box locations. In the second stage pretraining for VCR, we reduce the learning rate to a constant 5e-05 and trained for an additional 9K steps. Due to longer sequence lengths in the VCR dataset, a training batch-size of 224 is used. We also use a step size of 2 for gradient accumulation. After pretraining, we finetuned on the VCR task for 10K steps with a learning rate of 1e-04 for both the Transformer and the CNNs. Linear warmup of the learning rate is applied for 1000 steps, followed by a linear decay ending at a total of 10K steps.
B.5 Influence of image size
We adopt larger image size mainly for fair comparisons with most 2-step pretraining methods, PixelBert and E2E-VLP as all of them use the size (800, 1333). To investigate this, we pretrain our method with size (600,1000) and report the results
A woman sitting on a yellow surfboard on the beach.
a cat laying on a seat cushion holding a remote control.
(4)
the man is wearing a green suit and standing with an elephant.
(a) Knowledge-guided masking strategy. Red mask denotes the masked region in an image
(a) a train snakes through tracks next to trees on a clear day
(b) a woman dressed in leather poses on a motorcycle
(c) a man cleaning the parts of his bicycle
(d) a chocolate doughnut sitting on top of a table next to a cup of coffee
(b) Text-to-image attention maps. The bright region denotes higher attention scores between word tokens and image regions.
Table 7: Performance comparison of different image size
Models	Image Size	NVLR-dev	VQA-test dev
SOHO	(600,1000)	64.62	66.69
KD-VLP(ours)	(600,1000)	66.52	68.04
KD-VLP(ours)	(800,1333)	66.71	68.19
in Tab.7. We can see that our method has a mild performance drop, but still outperforms SOHO by a decent margin.
B.6 More Visualizations
As in Fig.5a, we observe that our knowledge-guided masking strategy masks out the image regions, which are highly related to the corresponding sentences. This design can force Transformer to infer object features and semantic labels based on the surrounding visual context and its language descriptions. On the contrary, SOHO randomly masks out either background regions (Fig.5a(1) & Fig.5a(2)) or local object parts (Fig.5a(3) & Fig.5a(4)), which are not related to the corresponding sentences with a high probability and result in inefficient multi-modal representation learning.
As shown in Fig.5b, it shows that our object-aware end-to-end multi-modal representations can accurately establish the correspondence between
word tokens and visual tokens, which demonstrates the superiority of our approach.