SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MLM utilizes vision features and text features of ID concepts and relations to predict the masked tokens in the caption sentence  $S$ . Masking tokens is a form of self-supervision, and is well known to improve performance [34]. Here, we randomly masked some tokens in  $S$  replacing as  $y^{msk}$  and predicted them with their visual and textual features. Since some tokens are replaced with " [mask]", the OOD score  $s(\cdot)$  becomes a function of the random masks. With this, we used  $s(\cdot)$  to calculate the predicted probability for a masked token as in Equation (3). Our final MLM loss can be written as,
$$
\mathcal {L} _ {\mathrm {m l m}} = \mathbb {E} _ {(v, \hat {l}) \sim D} \mathcal {H} \left(y _ {\text {m l m}}, p (v, \hat {l})\right) \tag {10}
$$
where  $\mathcal{H}$  denotes the cross-entropy,  $y_{mlm}$  is a one-hot vector where the ground truth tokens are with probabilities of  $1$ ,  $\hat{l}$  denotes the masked text.
In this section, we provide details on experimental setups that we used to conduct ablation studies, and more qualitative analysis of our proposed VK-OOD multimodal pipeline.
C.1 Implementation details
Datasets. Following standard practice in Vision, we used training strategies wherein we pre-trained on a fixed, large dataset and then fine-tuned on datasets specific to downstream tasks. We pre-trained on three datasets, including COCO [35], Visual Genome [28], and SBU Captions [47] with total of 1M images and 6.8M image-caption pairs, as approximate  $30\%$  less than the baseline(ViLT). Each caption is parsed to 1 - 3 triplets and augmented with 5 external knowledge triplets. For downstream datasets, we used Flickr30k [48] and COCO for image-text retrieval, VQAv2 [4] and OKVQA [41] for visual question answering (and ablation studies), and NLVR2 [56] for visual reasoning. We resized each image to the size of  $224 \times 224$  by center-cropping. In the merged attention module, each multimodal encoder layer consists of one multi-head self-attention block and one feedforward block, and total number of identical layers is equal to 12.
Encoder backbones. First, we retrieved explicit knowledge triplets as pre-processing, by using ConceptNet Numberbatch<sup>1</sup>. Next, for ViLT, we used RoBERTa [38] as text encoder and ViT-B/32 [49] as visual encoder. To scale with CLIP, we used CLIP-ViT-B/32 [49] as both backbones. Then, we followed the BLIP design with BERT-base as text encoder and ViT-B/16 as visual encoder.
Network training. We pre-trained the model for 10 epochs using AdamW optimizer [39] with learning rate of  $1e - 4$  and weight decay of  $1e - 2$ . We chose the warm-up phase of learning rate to be  $10\%$  of the total training steps, and the learning rate was decayed linearly to 0 afterwards. Then, we fine-tuned our model for 5 epochs with learning rate of  $2e - 4$  for all downstream tasks. In addition, we applied RandAugment [12] as augmentation strategy in fine-tuning steps. We pre-trained and fine-tuned both on 8 NVIDIA RTX 2080Ti GPUs, and for inference we used 1 NVIDIA RTX 2080Ti GPU.
1https://github.com/commonsense/conceptnet-numberbatch
Model	COCO	F30k
TR	IR	TR	IR
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
ViLT	61.8	86.2	92.6	41.3	72.0	82.5	81.4	95.6	97.6	61.9	86.8	92.8
UNITER	64.4	87.4	93.1	50.3	78.5	87.2	85.9	97.1	98.8	72.5	92.4	96.1
ALBEF	73.1	91.4	96.0	56.8	81.5	89.2	94.3	99.4	99.8	82.8	96.7	98.4
VinVL	74.6	92.6	96.3	58.1	83.2	90.1	-	-	-	-	-	-
BLIP	80.6	95.2	97.6	63.1	85.3	91.1	96.6	99.8	100	87.2	97.5	98.8
Ours(ViLT)	73.8	91.4	96	52.4	81.3	90.1	85.9	97.1	97.6	80.1	94.6	96.7
Ours(CLIP)	69.8	87.5	93.6	48.8	78.5	82.5	92.3	98.4	99.5	79.8	92.1	96.4
Ours(BLIP)	80.7	95.1	96.8	62.9	84.8	92.8	96.4	99.6	99.8	86.3	97.1	98.8
Table 6: Detailed results of image-text retrieval tasks on COCO and Flickr30k datasets. Our model with different backbones outperforms other models and achieve the best and second-best results.
Model	Objectives	VQA	Flickr30k
test-dev	TR@1	IR@1
ViLT	ITM	70.6	82.1	65.6
ViLT	MLM	72.8	-	-
ViLT	ITM+MLM	74.2	88.1	74.1
VK-OOD-s	ITM	72.1	84.5	69.8
VK-OOD-s	MLM	73.4	-	-
VK-OOD-s	ITM+MLM	74.8	89.0	77.2
Table 7: Ablation study experiment results of VK-OOD model. ViLT is our implementation without explicit knowledge and OOD detection layer. ITM is image-text matching, and MLM is masked language modeling. Results on VQA are on test-dev set. Both downstream results are in zero-shot settings. The bold values mean the best model in the table. Comparing with the baselines, our model with OOD detection layer outperforms on all objectives with two datasets. Training on combinations of objectives improves model performance.
C.2 Experimental Results
Details of image-text retrieval tasks. We now discuss detailed results on COCO and F30K datasets, as shown in Table 6. We can see that the model of OOD detection layer with ViLT has significant improvements in image retrieval and text retrieval tasks. Overall, we once again see that our model achieved the best and second-best results on both datasets comparing to other SOTA models.
Training Objectives with OOD detection Layer. To precisely characterize the performance benefits of using our OOD layer, we performed more ablations with the default training settings of the baseline and our model mentioned in Section 4.2. We considered different combinations of train objectives and evaluated them in zero-shot settings and observed our model performance on training objectives. Note that ViLT we compare to in Table 7 is our implementation with the same subset of training datasets. Our raw results are presented in Table 7. Here, we trained on pre-train datasets with  $\mathcal{L}_{\mathrm{itm}}$  in Equation (6),  $\mathcal{L}_{\mathrm{mlm}}$  in Equation (10), and the sum of  $\mathcal{L}_{\mathrm{itm}}$  and  $\mathcal{L}_{\mathrm{mlm}}$  losses as our training objectives. We see that training on image-text matching and masked language modeling is beneficial for both downstream tasks comparing to the baseline model. Specifically, there is improvements in image retrieval and text retrieval tasks. Thus, it is beneficial to train on both ITM and MLM for filtering outlier concepts and improved performance on downstream tasks.
C.3 Qualitative Analysis
As shown in Figure 2, the feature maps obtained using our VK-OOD pipeline can be clustered that are easily identified. This means that we can detect outliers and group images closest to the corresponding  $\mu_{k}$  with image and explicit knowledge triplets. Figure 6a and Figure 6b are examples with the nearest images in each cluster.
Figure 7 and Figure 8 show more qualitative examples of multimodal alignment results of our models. Here, we visualized the multimodal attention maps on images corresponding to concept triplets using Grad-cam [52]. Following our model architecture, the caption is parsed and integrated with knowledge triplets. The right bottom sub-figures of our model in Figure 7 and Figure 8 are the
(a) Example cluster 1
(b) Example cluster 2
Figure 6: Example images in the clusters on COCO val set.
Our model
Baseline
Figure 7: Visualization of the attention maps of image feature  $v_{q}$  and language features  $l_{j}$  of external knowledge alignment. The results are from our VK-OOD-s (ViLT) model. The original sample caption is "a giraffe and zebras mingle as cars drive out of an animal park". We highlight areas in the example image corresponding to different knowledge triplets. Comparing with the attention maps of the baseline model, our model learns object shapes such as zebras and localize those objects correctly.
multimodal alignment of original captions from MSCOCO dataset [35]. Other sub-figures show the alignments of extracted knowledge triplets on the image.
Interestingly, we found that VK-OOD model is able to capture concept "plug" as a part of "refrigerator" or "microwave" in Figure 7. The heatmap area of "plug" and "microwave" in Figure 7 clearly suggest that our model has the capability to exploit different relevance between visual and corresponding conceptual text features. In contrast, the baseline results did not show the relation between plug and microwave. Figure 8, shows that VK-OOD can detect three zebras comparing with baseline, but counting cars is not performing well as we expected - since the size (or scale) of cars is not sufficiently high, and moreover some parts of them are occluded.
microwave locate in room
refrigerator locate in room
plug is part of refrigerator
plug is part ofmicrowave
Our model
a refrigeratorfreezer next to amicrowave oven
plug is part ofmicrowave
Baseline
Figure 8: Visualization of the attention maps of image and knowledge concept triplets alignment. The results are from our VK-OOD-s (ViLT) model. The original sample caption is "a metallic refrigerator freezer next to a microwave oven". We highlight areas in the example image corresponding to different knowledge triplets. Comparing with the attention maps of the baseline model, our model learns the relations between the parts (i.e., plug) of the objects correctly.