SlowGuess/NAACL_2022_OCR huggingface dataset:
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 696-707.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2020. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations (ICLR).
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017), pages 1-14.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 169-174.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 1597-1607.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670-680.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HIT), pages 4171-4186.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6894-6910.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770-778.
Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. 2018. Learning visually grounded sentence representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HIT), pages 408-418.
5964
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for BERT sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 2528-2540.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems (NeurIPS), pages 3294-3302.
Angeliki Lazaridou, Marco Baroni, et al. 2015. Combining language and vision with a multimodal skip-gram model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 153-163.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755.
Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1442-1459.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In International Conference on Language Resources and Evaluation (LREC), pages 216-223.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language
supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748-8763.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.
Hao Tan and Mohit Bansal. 2020. Vokenization: Improving language understanding via contextualized, visually-grounded supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2066-2080.
Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal. 2021. VidLanKD: Improving language understanding via video-distilled knowledge transfer. In Advances in Neural Information Processing Systems (NeurIPS), pages 24468-24481.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 9929-9939.
John Wieting, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A bilingual generative transformer for semantic sentence embedding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1581-1594.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 38-45.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5065-5075.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual
5965
denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), pages 67-78.
Eloi Zablocki, Benjamin Piwowarski, Laure Soulier, and Patrick Gallinari. 2018. Learning multi-modal word representation grounded in visual context. In Proceedings of the AAAI Conference on Artificial Intelligence.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601-1610.
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2019. Neural machine translation with universal visual representation. In International Conference on Learning Representations (ICLR).
Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4369-4379.
5966
Language Encoder Our implementation is based on the Hugging Face Transformers library $^6$  (Wolf et al., 2020). We start from the checkpoints of bert-base-uncased and roberta-base, and fine-tune the pre-trained models using a contrastive objective function. We use the 768-dimensional [CLS] token outputs before the MLP pooler layer as sentence embeddings for evaluation. Image Encoder We use ResNet-50 and extract 2048-dimensional feature vectors at the last layer. The image encoder is not fine-tuned. $^7$
Projection Heads We use distinct projection heads for different modalities and objectives. All of them are implemented by single-layer MLPs with Tanh activation. We map sentence embeddings to a 768-dimensional space before calculating the textual objective. We map both sentence embeddings and image feature vectors to a 256-dimensional shared space, and normalize them before calculating the multimodal objective.
Parameter Settings We explore 5 training settings in the paper: {wiki, wiki+flickr, wiki+coco, flickr, coco}. For wiki+flickr and wiki+coco, we sample mini-batches from either Wiki1M or the caption dataset in proportion to their data size. We adopt most of the parameter settings suggested by Gao et al. (2021). Moreover, temperature parameters  $\tau$  and  $\tau'$  are set to 0.05, and other hyperparameters are reported in Table 3. We use the dev set of STS-B to tune the trade-off parameter  $\lambda$  and ablation studies are shown in Table 4. We evaluate models every 125 training steps on STS-B dev set and keep the best checkpoint for final evaluation.
settings:	wiki	wiki+flickr	wiki+coco	flickr	coco
BERT
learning rate	3e-5
batch size	64
λ	-	0.01	0.01	0.05	0.05
epochs	3	3	3	6	3
RoBERTa
learning rate	1e-5
batch size	128
λ	-	0.01	0.01	0.01	0.01
epochs	3	3	3	6	3
Table 3: The hyperparameters used for different training settings and pre-trained encoders.
λ	0.001	0.01	0.05	0.1	0.5
MCSE-BERT	78.38	79.95	80.41	80.35	80.01
MCSE-RoBERTa	80.60	81.48	81.08	80.73	79.85
Table 4: STS-B performance of MCSE models trained on Flickr30k with different trade-off parameters.
B.1 Improvements on Different Subsets
To delve into the performance gap between MCSE-BERT and SimCSE-BERT, we calculate the Spearman's correlation for different subsets of each year's STS challenge separately. The improvements of MCSE over SimCSE are shown in Figure 4. In STS12, "MSRvid" subset achieves the largest improvement, which is a corpus of video descriptions. "Image" subsets in STS14 and STS15 also get considerable improvements. Meanwhile, the performance of "answers-students" subset in STS15 drops extensively, and none of the subsets in STS16 get noticeable improvement by MCSE. The results indicate that the subsets benefit to different degrees from the visually grounding because of domain discrepancy.
CLIP as Image Encoder We use CLIP (Radford et al., 2021) as an alternative image encoder. The implementation is based on the Sentence Transformer library $^{8}$  (Reimers and Gurevych, 2019) and we use the checkpoint clip-ViT-B-32 to extract 512-dimensional feature vectors. As shown in Table 7, different image encoders lead to very similar results, thus we use ResNet as the default image encoder.
Combining Wiki1M, Flickr30k and MS-COCO
We adopt the same parameter setting as wiki+flickr and wiki+coco, and train models on the combination of Wiki1M, Flickr30k, and MS-COCO. As shown in Table 5, MCSE models achieve 1.9 point and 2.6 point improvements when using BERT and RoBERTa, respectively.
B.3 Analysis
Sentence Retrieval We take BERT-based models trained on the Flickr30k train set (same seed) and conduct a sentence retrieval experiment on Flickr30k test set. Given an input sentence, the nearest neighbor will be retrieved based on cosine
$^{8}$ https://github.com/UKPLab/sentence-transformers
In our preliminary results, fine-tuning the image encoder does not have a significant impact on the STS performance.
5967
(a) training data: wiki+flickr
(b) training data: wiki+coco
Figure 4: The Spearman's correlation improvements over different subsets.
Model	Trained on wiki+flickr+coco
SimCSE-BERT	74.3±1.0
MCSE-BERT	76.2±0.3
SimCSE-RoBERTa	75.3±0.7
MCSE-RoBERTa	77.9±0.6
similarity. Some retrieval examples are shown in Table 8. We observe that (1) SimCSE is prone to retrieving sentences with similar syntax, while MCSE can retrieve sentences that vary in syntax and share semantics. Examples: Q1, Q3, Q6. (2) MCSE is better at recognizing similar event scenes and capturing the number of entities. Examples: Q2, Q4, Q5.
Cross-Modal Retrieval We take BERT-based models (same seed) and conduct cross-modal retrieval experiments. We use the metric Recall@K, which is calculated based on if the ground truth of
Table 5: Comparison of the average Spearman's correlation of 7 STS tasks. We report the means and standard deviations over 5 random seeds.
Model	image → text	text → image
R@1	R@5	R@1	R@5
MCSE-BERTwiki+flickr	16.7	43.5	22.5	50.4
MCSE-BERTflickr	20.4	50.2	23.8	52.5
MCSE-BERTwiki+coco	8.8	26.6	10.9	31.2
MCSE-BERTcoco	8.2	25.2	9.0	27.1
Table 6: Multimodal retrieval results on Flickr30k test set (1k) and MS-COCO minimal set (5k).
the query image or caption appears in the top-K retrieved captions or images. As results in Table 6 show, MCSE models also achieve a decent level of retrieval performance as a by-product of multimodal contrastive learning.
5968
Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.↑
flickr	SimCSE-BERT	62.1±0.5	73.8±0.9	64.2±0.6	74.2±0.8	74.8*±0.6	67.1±1.1	65.4±1.1	68.8±0.7
MCSE-ResNet-BERT	63.6*±0.7	74.0±0.9	65.5±1.1	75.5±0.2	71.6±0.4	74.0±0.4	69.8±0.3	70.6±0.5
MCSE-CLIP-BERT	63.1±0.7	73.9±1.0	65.8*±0.9	76.0*±0.7	70.7±0.3	74.9*±0.5	70.7*±0.3	70.7*±0.2
SimCSE-RoBERTa	66.6±0.5	78.3±0.5	69.7±0.6	77.7±0.5	76.3*±0.5	75.8±0.3	66.2±0.4	72.9±0.3
MCSE-ResNet-RoBERTa	67.6*±0.5	78.8±0.4	70.1±0.3	78.5±0.2	75.4±0.5	77.4*±0.3	68.6±0.3	73.8*±0.2
MCSE-CLIP-RoBERTa	67.0±0.5	78.6±0.4	69.8±0.5	78.7*±0.8	74.9±0.5	77.4*±0.4	69.5*±0.5	73.7±0.2
coco	SimCSE-BERT	59.3±0.9	73.0±1.2	62.7±0.6	74.7±0.7	74.4*±0.4	65.3±0.7	65.4±0.5	67.8±0.4
MCSE-ResNet-BERT	64.9*±0.5	74.8*±0.9	68.1*±0.6	76.8*±0.6	72.7±0.8	74.5*±0.4	69.7±0.4	71.6*±0.2
MCSE-CLIP-BERT	64.8±0.6	74.1±0.6	68.0±0.2	76.2±0.5	71.6±0.4	74.5*±0.3	70.3*±0.6	71.4±0.1
SimCSE-RoBERTa	64.7±0.6	79.2±0.4	70.2±0.4	79.0±0.6	78.2±0.5	73.8±0.5	64.6±0.3	72.8±0.3
MCSE-ResNet-RoBERTa	67.0*±0.8	79.4±0.4	70.9*±0.4	80.0*±0.4	77.8±0.5	76.9*±0.4	67.9±0.7	74.3*±0.3
MCSE-CLIP-RoBERTa	66.0±1.0	79.0±0.7	70.6±0.6	80.0*±0.8	77.6±0.5	76.5±0.4	68.4*±0.8	74.0±0.2
*: difference between SimCSE and MCSE (ResNet/CLIP) is significant at  $\alpha  = {0.05}$  according to an independent t-test.
Table 7: Performance comparison on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across 7 tasks. Models are trained with 5 random seeds and we report means and standard deviations.
Model	Result
Query 1: A young girl is washing her teddy bear in the kitchen sink.
SimCSE:	A middle-aged woman is vacuuming her kitchen floor with a canister vac.
MCSE:	A young girl, blond and wearing a polka-dot shirt, washes a stuffed animal in a vanity sink.
Query 2: Three chefs, wearing white hats and black aprons, are preparing food in a crowded kitchen.
SimCSE:	Numerous workers with blue shirts and white aprons are preparing fish for sale.
MCSE:	Three men are preparing food in a kitchen setting.
Query 3: A couple kisses in a shady walkway.
SimCSE:	A couple strolls down a path near benches and water.
MCSE:	Couple kissing outside on street.
Query 4: A man is standing on the streets taking photographs.
SimCSE:	People run a marathon on a city street with a crowd watching.
MCSE:	A guy wearing a white shirt is taking a picture.
Query 5: Two boys are playing in pool filled with sparkling blue water.
SimCSE:	A little girl is swimming under the crystal blue water.
MCSE:	Two children are swimming in a pool.
Query 6: An old man sitting on a bench staring at the ocean.
SimCSE:	A man sitting on a bench by the ocean.
MCSE:	An old man sits on a bench overlooking the water.
Table 8: Retrieved examples from Flickr30k test set.
5969