SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	COCO Test (5k images)	Flickr30K Test (1k images)
Text Retrieval	Image Retrieval	AR	Text Retrieval	Image Retrieval	AR
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
VSE++*	41.3	69.2	81.2	30.3	59.1	72.4	58.9	52.9	80.5	87.2	39.6	70.1	79.5	68.3
SCO*	42.8	72.3	83.0	33.1	62.9	75.5	61.6	55.5	82.0	89.3	41.1	70.5	81.1	69.9
GXN	42.0	-	84.7	31.7	-	74.6	-	56.8	-	89.6	41.5	-	80.0	-
SCAN-single	46.4	77.4	87.2	34.4	63.7	75.7	64.1	67.9	89.0	94.4	43.9	74.2	82.8	75.4
R-SCAN	45.4	77.9	87.9	36.2	65.6	76.7	65.0	66.3	90.6	96.0	51.4	77.8	84.9	77.8
CAMP	50.1	82.1	89.7	39.0	68.9	80.2	68.3	68.1	89.7	95.2	51.5	77.1	85.3	77.8
CAAN	52.5	83.3	90.9	41.2	70.3	82.9	70.2	70.1	91.6	97.2	52.8	79.0	87.9	79.8
ViLBERT	-	-	-	-	-	-	-	-	-	-	58.2	84.9	72.8	-
Unicoder-VL	62.3	87.1	92.8	46.7	76.0	85.3	75.0	86.2	86.3	99.0	71.5	90.9	94.9	88.1
UNITER-base	64.4	87.4	93.1	50.3	78.5	87.2	76.8	85.9	97.1	98.8	72.5	92.3	95.9	90.4
UNITER-large	65.7	88.6	93.8	52.9	79.9	88.0	78.1	86.9	98.1	99.2	75.5	94.0	96.6	91.7
OSCAR	73.5	92.2	96.0	57.5	82.8	89.8	82.0	-	-	-	-	-	-	-
LightningDOT*	60.1	85.1	91.8	45.8	74.6	83.8	73.5	83.9	97.2	98.6	69.9	91.1	95.2	89.3
+UNITERbase Re-Ranker	64.6	87.6	93.5	50.3	78.7	87.5	77.0	86.5	97.5	98.9	72.6	93.1	96.1	90.8
+UNITERlarge Re-Ranker	65.7	89.0	93.7	53.0	80.1	88.0	78.2	87.2	98.3	99.0	75.6	94.0	96.5	91.8
+OSCAR Re-Ranker	74.2	92.4	96.0	57.4	82.7	89.9	82.1	-	-	-	-	-	-	-
Table 1: Evaluation results on image-to-text and text-to-image retrieval over Flickr30k and COCO test sets. We compare the proposed method with both task-specific models: VSE++ (Faghri et al., 2017), GXN (Gu et al., 2018), SCO (Huang et al., 2018), SCAN (Lee et al., 2018), R-SCAN (Lee et al., 2019), CAMP (Wang et al., 2019) and CAAN (Zhang et al., 2020), and  $\mathrm{V} + \mathrm{L}$  pre-trained models: ViLBERT (Lu et al., 2019), Unicoder-VL (Li et al., 2020a), UNITER (Chen et al., 2020) and OSCAR (Li et al., 2020b). Models in bold* are embedding-based methods without cross-attention.
than the database (index) size. Next, we apply a stronger retrieval model (usually slower due to the use of cross-attention) to re-rank the retrieved top- $M$  pairs from the first stage. The final  $M$  similarity scores obtained from the second stage will be used to re-compute the desired top- $K$  retrieval  $(K \leq M)$  in Eqn. (7). Please refer to figure 2 for a more detailed visualization. Our experiments show that this two-stage approach can benefit from the best of both worlds: maintaining a constant fast speed per query while achieving state-of-the-art accuracy. Another advantage of this pipeline is that it can readily incorporate any advanced model as the re-ranker, thus future stronger image-text retrieval models can take advantage of Lightning-DOT for better efficiency.
This section discusses our experiments on pretraining and evaluating LightningDOT on downstream ITR benchmarks.
For pre-training, we use pre-processed data provided by Chen et al. (2020), including 4.2 million
images with 9.5 million associated captions from COCO (Chen et al., 2015), VG (Krishna et al., 2017), Conceptual Captions (Sharma et al., 2018), and SBU captions (Ordonez et al., 2011).
For evaluation, we use Flickr30k (Plummer et al., 2015) and COCO (Lin et al., 2014) datasets, which include  $31\mathrm{K} / 123\mathrm{K}$  images, respectively, each associated with 5 human-written captions. Following (Faghri et al., 2017), we split COCO into  $114\mathrm{K} / 5\mathrm{K} / 5\mathrm{K}$  and Flickr30K into  $29\mathrm{K} / 1\mathrm{k} / 1\mathrm{k}$  images for train, validation and test.
Downstream performance is measured by recall at  $K$  (R@K) for both image and text retrieval tasks. We also use an additional metric "AR", the average of R@K for all  $K$  across both image and sentence retrieval tasks.
4.2 Results on Flickr30K and COCO
We compare the proposed approach with state-of-the-art methods (with and without pre-training) and report the results in Table 1. Without cross-attention, our method outperforms non-pre-training approaches by large margins on all metrics. Specifically, our model improves over CAAN (Zhang et al., 2020) (SOTA method with cross-attention) by  $3.3\%$  (73.5 vs. 70.2) on COCO and  $9.5\%$  (89.3 vs. 79.8) on Flickr30K in terms of AR. When compared with methods without cross-attention (VSE++ (Faghri et al., 2017) and SCO (Huang et al., 2018)), LightningDOT achieves nearly
The computation time of LightningDOT is negligible compared to that of UNITER. Therefore, the empirical speed is proportional to the number of pairs UNITER has to rank: constant  $M$  for LightningDOT + UNITER vs. the whole database (index) size for UNITER only.
Model	COCO Full (123K Images)	Flickr30K Full (31K Images)
Text Retrieval	Image Retrieval	AR	Text Retrieval	Image Retrieval	AR
R@5	R@10	R@20	R@5	R@10	R@20	R@5	R@10	R@20	R@5	R@10	R@20
LightningDOT	40.1	51.0	62.0	28.2	37.4	47.8	44.4	69.6	78.9	86.1	51.8	62.3	72.3	70.2
+ Re-Ranker-base	47.9	58.5	67.8	35.7	45.2	55.2	51.7	74.2	81.7	88.2	56.9	66.7	75.6	73.9
+ Re-Ranker-large	48.0	59.0	68.9	37.3	46.8	56.4	52.7	75.1	83.9	90.5	60.1	69.5	78.3	76.2
Table 2: Results on the extreme retrieval setting of full Flickr30k and full COCO datasets.
Method	#images	SCAN	Ours	+Re-ranker
Flickr30K-test	1,000	1.8×	639×	46×
COCO-test	5,000	1.9×	1,927×	95×
Flickr30K-full	31,014	1.8×	6,591×	1,255×
COCO-full	123,287	1.9×	23,869×	2,235×
Table 3: Speedup w.r.t. UNITER-base. We compare LightningDOT (Ours) and +Re-Ranker, plus a lightweight cross-attention method SCAN (Lee et al., 2018). LightningDOT with/without UNITER-base re-ranker is significantly faster.
20-point gain on AR. Although LightningDOT achieves slightly lower AR than UNITER (pretraining method with cross-attention), with 3.5/1.1 points drop on Flickr30K/COCO, it is  $600 / 1900 \times$  faster than UNITER during inference time.
We further apply second-stage re-ranking, and use UNITER to score top-  $M$  retrieved image-text pairs from LightningDOT to obtain the final top $K$  ranked lists. With re-ranking, LightningDOT achieves an instant performance lift, surpassing UNITER on both benchmarks, while still 46-95 times faster than UNITER. With an even stronger re-ranker OSCAR, LightningDOT achieves similar results to the state-of-the-art performance on COCO.
4.3 Speed & Space Improvement
To demonstrate the efficiency of LightningDOT, we use UNITER-base as baseline to compare inference speed. We also compare with a more lightweight cross-attention method SCAN (Lee et al., 2018), which uses GRU (Chung et al., 2014) instead of a 12-layer Transformer. All methods are tested on a single Titan RTX GPU, with batch size of 400. As shown in Table 3, SCAN is  $\sim 1.9\times$  faster than UNITER-base across both benchmarks, as the computational cost of GRU is much cheaper than that of Transformer (performance drop is significant though). However, the speedup from SCAN is limited, as it computes cross-attention between each query and all images. On the other hand, LightningDOT is  $639\times$  faster than UNITER on Flickr30K. When tested with 5 times more im
ages in COCO, the speedup from LightningDOT is  $1927 \times$ . Even with re-ranking, LightningDOT is still much more efficient than UNITER-base ( $46 \times$  faster on Flickr30K and  $95 \times$  faster on COCO).
To mimic a real-life scenario for image retrieval, where the candidate pool contains hundreds of thousands of images, we combine all images from training, validation and test set to form a larger candidate pool. Note that models are still trained on the training set. Although the number of text queries remain the same, the number of candidate images scales up by  $>20\times$ , where cross-attention methods immediately become impractical. We refer this setting on both benchmarks as Flickr30k-full (31k) and COCO-full (123k). Our algorithm is  $6,591\times$  faster on Flickr30k-full and  $23,869\times$  faster on COCO-full, which clearly shows the advantage of LightningDOT and its potential in real-world applications. With re-ranking, LightningDOT is still more than  $1,000\times$  and  $2,000\times$  faster on Flickr30k-full and COCO-full, respectively. In general, for other re-rankers such as OSCAR, our algorithm can approximately speed up inference by  $N_{\mathrm{images}} / M$  times, where  $N_{\mathrm{images}}$  is the number of candidate images, and  $M$  is the number of re-ranked images from top- $M$  retrieved results by LightningDOT.
Similarly, we construct a full setting for text retrieval by combining all text queries from training, validation and test set. Results are summarized in Table 2. Considering the size of candidate pool has become more than  $20 \times$  larger, we adopt recall at top 5, 10, 50 as evaluation metrics. Our method achieves reasonably good performance, with AR of 44.4 on COCO and 70.2 on Flickr30K. Re-ranking further lifts AR to 56.4 and 76.2. Results from UNITER or SCAN are not included as the computation of pairwise scores is extremely expensive, given the excessive amount of retrieval candidates. While LightningDOT only takes minutes to evaluate, UNITER-base is estimated to take about 28 days $^{9}$  to evaluate under the full setting for both
9This estimation is based on the inference time taken by
Method	Text Retrieval	Image Retrieval
R@1	R@5	R@10	R@1	R@5	R@10	AR
R-CNN only	62.2	85.9	91.1	42.0	70.9	80.3	72.1
+Image Encoder	73.4	92.5	95.6	59.5	84.5	90.3	82.6
+PT†	83.5	96.4	98.7	68.6	90.5	94.8	88.8
LightningDOT	85.2	96.4	98.7	69.9	90.4	94.5	89.2
Table 4: Ablation studies on model design over Flickr30K validation set.  $\mathrm{PT}^{\dagger}$  indicates pre-training with MLM+MRM+CMR, while LightningDOT is pre-trained with VMLM+SMRM+CMR.
LightningDOT	Text Retrieval	Image Retrieval	AR
R@1	R@5	R@10	R@1	R@5	R@10
No PT	73.4	92.5	95.6	59.5	84.5	90.3	82.6
PT(CMR)	75.0	93.9	97.3	61.5	85.5	91.1	84.0
PT(All)	78.1	94.0	96.9	62.6	85.7	91.8	84.8
image retrieval and text retrieval.
In addition, We compare all models with the same setting: cache as much as possible for fastest speed, where our model outperforms others in both speed and space on image retrieval. The proposed algorithm maps each image to a 768-dimensional vector, which only consumes about  $300\mathrm{Mb}$  storage space for the whole COCO dataset. For cross-attention models such as SCAN, UNITER or OSCAR, they also need to cache image features, which typically requires to save a  $36 \times 2048$  dimensional vector per image, and it consumes about 28GB storage space for COCO dataset.
We conduct ablation studies on Flickr30K (Table 4) and compare LightningDOT (L4) against 3 ablated instances: (i) "R-CNN only" (L1): image representations are extracted from Faster R-CNN directly, with no image encoder applied; (ii) "+"Image Encoder" (L2): regional features are encoded with a 12-layer Transformer as the image encoder; (iii)  $^ { \text{一} + } \mathrm { P T } ^ { \dagger }$  " (L3): our model is pre-trained with MLM+MRM+CMR, then finetuned on Flickr30K. Note that the difference between MLM vs. VMLM and MRM vs. SMRM is whether the predictions of masked tokens (regions) rely on infused embeddings from the other modality.
Table 5: Ablation studies on pre-training tasks over Flickr30K validation set after finetuning on the corresponding training set. All pre-training experiments are conducted on COCO dataset only. PT is short for pre-training. PT(CMR) refers to pre-training using CMR task only, and PT(All) refers to pre-training with all of the three tasks.
Method	Multi30K	COCO	Meta-Ave
DE	FR	CS	ZH	JA
S-LIWE	72.1	63.4	59.4	73.6	70.0	67.7
MULE	64.1	62.3	57.7	75.9	75.6	67.1
SMALR	69.8	65.9	64.8	77.5	76.7	70.9
\( M^3P \)	82.0	73.5	70.2	81.8	86.8	78.9
UNITER	85.9	87.1	85.7	88.4	85.9	86.6
LightningDOT	83.3	83.7	82.2	87.2	82.3	83.7
+Re-Ranker	86.1	87.1	86.2	88.4	86.1	86.8
Table 6: Evaluation on multilingual image-text retrieval over Multi30K and COCO datasets. We compare with task-specific methods: S-LIWE (Wehrmann et al., 2019), MULE (Kim et al., 2020), SMALR (Burns et al., 2020), pre-trained method  $\mathbf{M}^3\mathbb{P}$  (Huang et al., 2020a) and UNITER with translate-test. Numbers in blue indicate the use of different dev/test splits of COCO compared to other methods. UNITER and Re-ranker are large model size.
Results show that "R-CNN only" is not sufficient in learning good image representations for ITR task, while image encoder with Transformer architecture can effectively learn contextualized image representations, hence achieving better performance. Pre-trained models (L3-4) generally achieve better performance, compared to non-pretrained models (L1-2). Comparing  $+PT^{\dagger}$  to the full instance of LightningDOT, dependency on the other modality in VMLM and SMRM brings universal performance lift across all metrics. This indicates that these cross-modal dependencies introduced by VMLM and SMRM are effective in learning the association between image and text inputs.
In addition, we investigate the effectiveness of each pre-training task in Table 5. Comparing to baseline without pre-training, pre-training with CMR alone lifts  $+1.4$  on AR. Pre-training with all three tasks achieves the best performance, indicating that the learning of contextualized word and region representations promotes better global alignment between image and text, and these three pre-training tasks work collaboratively to yield better visual-semantic embeddings.
4.5 Multilingual Image-Text Retrieval
We further report results on multilingual image-text retrieval tasks. Specially, we evaluate Lightning-DOT under the translate-test setting, which is to translate the test captions in other languages to English by leveraging Machine Translation (MT) tool. $^{10}$  Note that our method is only trained on English captions, without exploiting the original or translated captions from multilingual benchmarks.
UNITER-base on a smaller dataset.
<sup>10</sup>We use Microsoft Azure Translation API Service.
Figure 4: Retrieved top 10 images from the query "Sky view of a blue and yellow biplane flying near each other." The ground truth is in the red rectangle.
We consider two benchmarks: Multi30K (Elliott et al., 2016, 2017; Barrault et al., 2018) with captions in German, French and Czech; and COCO Japanese (Yoshikawa et al., 2017) and Chinese (Li et al., 2019b).
Average Recall (AR) is used as the evaluation metric. Meta-Ave, the average of AR over different languages across two benchmarks, is used as a global metric. More details on multilingual ITR benchmarks are included in Appendix.
We compare LightningDOT against 3 task-specific methods: S-LIWE (Wehrmann et al., 2019), MULE (Kim et al., 2020) and SMALR (Burns et al., 2020), which all exploit captions in different languages to learn multilingual or language-agnostic word embeddings. We also compare with a pre-trained model  $\mathbf{M}^3\mathbb{P}$  (Huang et al., 2020a), which is alternatively pre-trained with image-caption pairs labeled in English and cross-lingual corpus in 100 different languages. Note that all methods discussed above are trained/finetuned on captions in different languages. For fair comparison, we report performance of UNITER under the same translate-test setting, which is finetuned with English captions only and tested on translated captions.
Table 6 shows similar trends of performance improvements as on English benchmarks. Compared to both state-of-the-art task-specific methods and pre-trained models, LightningDOT under translate-test setting achieves new state of the art on most languages and establishes a strong baseline for future study on these multilingual benchmarks.
4.6 Qualitative Examples
We show an example of image retrieval results here at figure 4 for query as "Sky view of a blue and yellow biplane flying near each other". In addition to the ground truth image in the red rectangle, all the 10 images retrieved by our model are valid retrieval since multiple keywords ("sky", "blue", "yellow", "airplane", "near") are captured for each image. Please see the appendix A.4 for more examples.
In this paper, we propose a pre-training framework that learns joint visual-semantic embedding without any cross-attention between modalities. LightningDOT outperforms previous state of the art, while significantly speeding up inference time by  $600 - 2000 \times$  on Flickr30K and COCO image-text retrieval benchmarks. Future work includes extending the efficient training framework to other V+L tasks.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Find
ings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers. ACL.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A. Plummer. 2020. Learning to scale multilingual representations for vision-language tasks. In ECCV.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dólar, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Deep Learning and Representation Learning Workshop. NeurIPS.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation. ACL.
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language. ACL.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. In BMVC.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS.
Dan Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In CoNLL.
Jieuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang. 2018. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In CVPR.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652.
Haoyang Huang, Lin Su, Di Qi, Nan Duan, Edward Cui, Taroon Bharti, Lei Zhang, Lijuan Wang, Jianfeng Gao, Bei Liu, Jianlong Fu, Dongdong Zhang, Xin Liu, and Ming Zhou. 2020a. M3p: Learning universal representations via multitask multilingual multimodal pre-training. arXiv preprint arXiv:2006.02635.
Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. 2018. Learning semantic concepts and order for image and sentence matching. In CVPR.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020b. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Findings of EMNLP.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan A. Plummer. 2020. MULE: Multimodal Universal Language Embedding. In AAAI.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. In Deep Learning and Representation Learning Workshop. NeurIPS.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In ECCV.
Kuang-Huei Lee, Hamid Palangi, Xi Chen, Houdong Hu, and Jianfeng Gao. 2019. Learning visual relation priors for image-text matching and image captioning with neural scene graph generators. arXiv preprint arXiv:1909.09953.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. In AAAI.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019a. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. 2019b. Coco-cn for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. In NeurIPS.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In CVPR.