SlowGuess/NAACL_2022_OCR huggingface dataset:
The PI input  $x1, y1, x2, y2, d$  received the best results for the Y and Z subsets. Although the improvements are small, it shows that our new depth feature can help solve the Z task. But also, the improvement on Y can be attributed to the depth input. Due to the graphical perspective, objects at the top correlate with the background and objects at the bottom with the foreground (see Figure 4b). Here, object depth can act as a top/down proxy.
For the downstream evaluation, we need to keep in mind that the underlining object detector is not perfect. Therefore, we face the issue that objects asked for in the questions are not always a part of LXMERT's visual input. Moreover, our contrastive evaluation scheme shows that LXMERT has difficulties to properly matching image and
text representation in a multimodal fashion. This can explain the small margin of improvements. The increase of top-1 accuracy is not reflected in the top-5 accuracy.
6 From Probing to Pre-training
In the previous section, we evaluated the role of PI in pre-trained LXMERT. In this section, we use the probing tasks as a part of model pre-training to improve weaknesses that we identified in the previous section. Alongside the established strategies, we add two tasks to learn mutual positions and fine-grained PI details in captions utilizing the CMM task. These strategies are elaborated below.
Positional Information Pre-training (PIP). Currently, all pre-training strategies rely on the visual features  $(f_{j},c_{j},a_{j})$  rather than on the PI. Only in a small fraction of the pre-training captions and questions positional keywords are present, as Table 2 shows. Hence, we add a new pre-training strategy which exclusively focuses on PI.
We take the PI head used in Mutual Position Evaluation and add it as a new classification task which is updated during pre-training. We weight PIP by 10, since the initial loss is noticeably lower than the losses of the other strategies. Until now only visual representation of the object features, labels and attributes was part of pre-training. Using PIP, we introduce an explicit unimodal connection between the PI input and the PI output, which was not previously available (see Figure 3).
Contrastive Learning using CMM (CL). During pre-training in classical CMM in  $50\%$  of all cases the caption is replaced with another random image description. This is similar to the main pretraining concept of CLIP. Yet, doing so, the model only learns to distinguish dissimilar text and images. There are no small differences in the captions that the model needs to be aware.
In line with Contrastive Evaluation on PI using CMM, we make CMM more complex. In  $50\%$  of all captions with PI keywords the word is replaced by its counterpart, so that it has to learn fine-grained PI differences during pre-training. Dissimilar to PIP, this pre-training strategy only affects a small portion of the pre-training samples, since PI keywords are rare. Yet, it operates on both modalities and hence is able to connect both data types. This idea can also be extended to other attributes (such as color, material, shape using VG's Scene Graph).
Results. Using both pre-training strategies, we train new models for all four PI input types. We assess the models using the same three evaluation schemes as the plain LXMERT model before.
Results of Mutual Position Evaluation are shown in Table 3 (bottom lines). We observe an increase in accuracy for all input types. The largest is for the empty input type with an accuracy of  $88.2\%$ , indicating the high correlation between feature  $f_{j}$  and position  $p_j$ . For the other versions improvements are smaller. In Table 9 in the Appendix, the accuracies for each of the nine classification tasks are displayed. The largest increase can be seen for the empty input type with up to 23.1 percent points for task (1) of the 9 mutual position classification tasks. For classifications based on depth, the best improvements are 9.7 percent points for task (7) and 8.0 percent point for task (9) utilizing  $x1, y1, x2, y2, d$ . This shows that the presence of depth is useful as expected.
In the original LXMERT version, the probe on Contrastive Evaluation on PI using CMM showed that the model is not able to solve this task successfully. Recall was steadily below 2 percent. Introducing the CL pre-training strategy increases matching accuracy to over 78 percent, as shown in Table 4 (bottom lines). In CMM, we are now able to perform matching between visual and textual representations regarding PI. As a consequence, we successfully force the model to connect both types in a multimodal manner.
Downstream Task Evaluation. The third evaluation is the downstream task and results are shown in Table 5 (bottom lines). In the two former probes, our extended pre-training helped the model to solve these tasks. However, interestingly, this is not the case for GQA evaluation. The best results for the top 1 and subset tasks are obtained by plain LXMERT. Only in the (not official) best 5 accuracy, evaluation our version achieves better results. One reason for this may be that our PIP weight is too high and needs to be tuned in further studies.
We found that PI has much less impact on downstream results as previously thought. Simple object centers are often sufficient. Bounding box data, which add object width, height and area, do not add the desired information that the models can utilize. Adding depth is marginally useful on the Z task, which suggests that this feature is useful.
PI Input	Top 1	X	Y	Z	P	Top 5
Probing	∅	58.1	65.7	62.0	46.4	58.0	85.0
x,y	59.4	69.6	62.0	49.6	60.2	85.0
x1,y1,x2,y2	59.0	66.2	61.8	49.4	58.9	85.3
x1,y1,x2,y2,d	58.6	66.0	62.4	50.0	58.4	85.1
Pre-training	∅	58.8	68.7	60.4	48.5	59.0	85.1
x1,y1	58.8	68.7	60.4	48.5	59.0	85.1
x1,y1,x2,y2,	58.7	67.6	61.5	48.3	58.6	85.4
x1,y1,x2,y2,d	58.7	67.8	62.0	49.1	59.0	85.8
Table 5: Evaluation on GQA testdev: Model comparison of plain LXMERT models (top lines) and our version with PIP and CL pre-training (bottom lines) for different PI Input types. Evaluation on Top 1 and Top 5 accuracy, moreover on subsets focusing on X, Y, and, Z keywords only and questions that focus on position (P) using zero-shot classification. Underlined numbers indicate the best overall model per column and bold numbers indicate the best model per block and column.
Current VL models make use of different PI inputs without evaluating their impact. In our work, we inspect the effect of such PI input types and investigate depth as a new input extension. In the original setting, the model is able to forward the positional information through the whole Transformer layer stack, but it cannot utilize it in the contrastive evaluation and only marginally in the downstream task. Overall, having any type of PI is helpful, though object-center values are often sufficient. However, object features  $f_{j}$  are already good proxies for where objects are located. Because this can be based on spurious correlations, we propose pre-training methods that should make the model rely on PI directly.
We introduced two new pre-training strategies. Firstly, Positional Information Pre-training to ensure that data is passed through the model properly and does not need to rely on feature correlations. This operates on visual component only and increases performance on the corresponding intrinsic evaluation task. Moreover, we introduce Contrastive Learning on PI using CMM. In doing so, we connect PI in the textual and visual modality. As a result, the model is now able to succeed in the contrastive evaluation task. However, these improvements do not affect the downstream performance on GQA.
It is not enough to add different features unchecked, trusting they are properly utilized by the Transformer. In line with BERTology (Rogers et al., 2020; Clark et al., 2019; Tenney et al., 2019), studies are important to understand better what a model is capable. The same is true for pre
training strategies. It is not sufficient to add new pre-training strategies, although they look promising. With our probing experiments, we tried to receive a better understanding of the inner workings of LXMERT. We see the importance to investigate differences between general concepts and impact on a downstream task.
We see two major issues for PI in VL models. Firstly, the pre-training data contains too little fraction of sentences with PI content. Hence, especially the CL pre-training strategy has not enough samples to learn from. Secondly, the used object detector is not very powerful (see predictions in Figure 5). Newer detection models like VinVL (Zhang et al., 2021) might help to have a better image representation, which consequently leverages performance regarding PI context.
The authors gratefully acknowledge the computing time granted by the Institute for Distributed Intelligent Systems and provided on the GPU cluster Monacum One at the Bundeswehr University Munich. The work at CUNI was funded by the Czech Science Foundation, grant no. 19-26934X.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick,
and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333-342.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR).
Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach. 2020. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9992-10002.
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1988-1997. IEEE Computer Society.
Guolin Ke, Di He, and Tie-Yan Liu. 2021. Rethinking Positional Encoding in Language Pre-training. In International Conference on Learning Representations.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dólar. 2019. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404-9413.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336-11344.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. ECCV 2020.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision - ECCV 2014, pages 740-755, Cham. Springer International Publishing.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy. Association for Computational Linguistics.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Yu-An Wang and Yun-Nung Chen. 2020. What do position embeddings learn? an empirical study of pre-trained language model positional encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6840-6849, Online. Association for Computational Linguistics.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208-3216.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinyl: Making visual representations matter in vision-language models. CVPR 2021.
A.1 PI Classification Head
The PI head is build up in the same manner as the other visual heads, i.e. Dense  $\rightarrow$  Activation  $\rightarrow$  Layer Normalization  $\rightarrow$  Dropout  $\rightarrow$  Dense.
A.2 PI Antonyms
For Contrastive Evaluation, we replace some PI keywords with its antonyms.
We substitute left with right, above with below, under with over, foreground with background, before with behind and vice versa.
A.3 PI Keywords
In Table 6 we list all PI keywords used in our evaluations.
Dim.	Keywords
X	left, right, beside, besides, alongside, side
Y	top, down, above, below, under, beneath, underneath, over, beyond, overhead
Z	behind, front, rear, back, ahead, before, foreground, background, before, forepart, far end, hindquarters
A.4 Pre-training Weights
In Table 7 we compare pre-training weights from LXMERT paper (Tan and Bansal, 2019) and the repository version (https://github.com/airsplay/lxmert/).
Table 6: Overview of positional keywords regarding dimension.
Version	MLM	CMM	ObjClassif	AttrClassif	FeatRegr	VQA
Paper	1	1	1	1	1	1
Repository	1	1	6.6	6.6	6.6	1
Table 7: Overview of pre-training weights in publication and GitHub version.
A.5 Text Examples
In Table 8 we provide examples from pre-training and downstream tasks with highlighted keywords.
Dataset	Example	Length
MS COCO	A very clean and well decorated empty bathroom	8
A panoramic view of a kitchen and all of its appliances.	11
Surfers waiting for the right wave to ride.	8
Two dogs are laying down next to each other.	9
A red stop sign with a Bush bumper sticker under the word stop.	13
VG	separate kitchen areas in a home	6
older red Volkswagen Beetle car	5
a woman walking down the sidewalk	6
A bag in the woman's left hand	7
stones under wood bench	4
GQA	Are there both a television and a chair in the picture?	11
That car is what color?	5
On which side of the picture is the lamp?	9
Is the table to the left or to the right of the appliance in the center?	16
Is there a bookcase behind the yellow flowers?	8
Table 8: Text examples from different datasets with word counts. Italic stands for PI keywords that are wrongly selected and bold words are correctly detected.
PI Input	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)
∅	65.0	84.1	82.1	89.9	95.6	72.3	77.7	75.3	78.4
x,y	95.1	95.6	96.2	96.1	95.8	74.1	83.3	75.7	84.4
x1,y1,x,2,y2	94.3	95.2	96.8	97.0	96.0	75.0	83.5	75.8	84.6
x1,y1,x,2,y2,d	94.0	95.0	96.6	96.8	96.0	74.9	88.7	76.3	89.1
∅	88.1	89.4	92.6	93.5	95.9	74.1	83.9	77.7	84.8
x,y	98.7	98.8	98.3	98.3	96.1	75.9	89.3	78.4	90.4
x1,y1,x,2,y2	98.8	98.9	98.7	99.5	96.3	77.0	89.7	78.9	90.9
x1,y1,x,2,y2,d	98.9	98.9	98.6	99.0	96.3	77.2	98.4	81.0	97.1
Table 9: Average accuracy per classification task (1-9) in Mutual Positional Evaluation for plain LXMERT (top lines) and our version (bottom lines).
A.6 Mutual Positional Evaluation Details
In Table 9 we provide detailed results for all 9 mutual PI tasks. Tasks (1)-(6) relate to X and Y coordinates and tasks (7)-(9) to Z coordinates. The numbering is explained in Section 3.