SlowGuess/NAACL_2022_OCR huggingface dataset:
Miaoran Zhang, Marius Mosbach, David Ifeoluwa Adelani, Michael A. Hedderich, and Dietrich Klakow
Spoken Language Systems (LSV)
Saarland Informatics Campus, Saarland University, Germany
{mzhang, mmosbach, didelani}@1sv.uni-saarland.de
{mhedderich, dklakow}@1sv.uni-saarland.de
Learning semantically meaningful sentence embeddings is an open problem in natural language processing. In this work, we propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. Through experiments on a variety of semantic textual similarity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multimodal data with a large text-only corpus, we improve the state-of-the-art average Spearman's correlation by  $1.7\%$ . By analyzing the properties of the textual embedding space, we show that our model excels in aligning semantically similar sentences, providing an explanation for its improved performance.
Sentence embedding learning, i.e., encoding sentences into fixed-length vectors that faithfully reflect the semantic relatedness among sentences, is a fundamental challenge in natural language processing (NLP). Despite the tremendous success of pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), it has been shown that the off-the-shelf sentence embeddings of PLMs without finetuning are even inferior to averaging Glove embeddings (Pennington et al., 2014) in terms of semantic similarity measure (Reimers and Gurevych, 2019). Hence, recent research (Li et al., 2020; Zhang et al., 2020; Su et al., 2021) focuses on adjusting the original sentence embeddings derived from PLMs in an unsupervised manner. In particular, there has been growing interest in adopting contrastive learning objectives to achieve this goal (Carlsson et al., 2020; Kim et al., 2021; Gao et al., 2021).
Although purely text-based models have led to impressive progress, it remains an open question to
what extent they capture the deeper notion of sentence meaning beyond the statistical distribution of texts, which lies outside of the text and is grounded in the real-world (Bender and Koller, 2020; Bisk et al., 2020). As a central part of the human perceptual experience, vision has been shown to be effective in grounding language models and improving performance on various NLP tasks (Zhang et al., 2019; Bordes et al., 2019; Zhao and Titov, 2020). We hypothesize that using vision as supplementary semantic information can further promote sentence representation learning.
In this work, we propose MCSE, an approach for multimodal contrastive learning of sentence embeddings. To exploit both visual and textual information, we adopt the state-of-the-art contrastive sentence embedding framework SimCSE (Gao et al., 2021) and extend it with a multimodal contrastive objective. In addition to the textual objective in SimCSE that maximizes agreement between positive sentence pairs, the multimodal objective maximizes agreement between sentences and corresponding images in a shared space. We conduct extensive experiments on standard Semantic Textual Similarity (STS) benchmarks and show the effectiveness of MCSE across various datasets and pre-trained encoders. We find that, using a small amount of multimodal data in addition to a text-only corpus yields significant improvements on STS tasks. By analyzing the alignment and uniformity properties of the embedding space (Wang and Isola, 2020), we show that MCSE better aligns the semantically similar sentences while maintaining uniformity, providing an explanation for its superior performance. $^{1}$
Sentence Representation Learning. Existing works for learning sentence embeddings can be
$^{1}$ Our code and pre-trained models are publicly available at https://github.com/uds-lsv/MCSE.
Human Language Technologies, pages 5959 - 5969
Different hidden dropout masks in two forward pas
image encoder
projection head
positive instance
negative instance
A group of people riding horses on the beach. People are gathering and relaxing in the park. A man is swimming in a competition.
text encoder
Figure 1: The overall architecture of MCSE. Compared to SimCSE, a new multimodal objective is calculated in the grounded space. For each input sentence, the positive instance is the paired image and the negative instances are all other in-batch images.
categorized into supervised (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019; Wieting et al., 2020) and unsupervised approaches (Li et al., 2020; Carlsson et al., 2020; Su et al., 2021; Kim et al., 2021; Gao et al., 2021; Liu et al., 2021; Yan et al., 2021). Supervised approaches mostly utilize supervision from annotated natural language inference data or parallel data. Unsupervised approaches are able to make use of the intrinsic semantic information embedded in the natural language text corpus by adjusting the training objective to STS tasks, thereby eliminating the need for a costly annotation process. In particular, contrastive learning objective (Carlsson et al., 2020; Kim et al., 2021; Gao et al., 2021; Liu et al., 2021; Yan et al., 2021) regularizes the embedding space by pulling positive (i.e., semantically similar) sentences closer and pushing apart negatives, showcasing great effectiveness in capturing the semantic similarity among sentences. Our approach adopts the contrastive learning framework and is built on top of the current state-of-the-art approach (Gao et al., 2021), further pushing the frontier of STS by leveraging multimodal semantic information.
Visually Grounded Representation Learning. There are various works showing that grounding NLP models to the visual world can improve textual representation learning. Lazaridou et al. (2015) and Zablocki et al. (2018) learn word embeddings by aligning words to the visual entity or visual context. Kiela et al. (2018) ground sentence embeddings by predicting both images and alternative captions related to the same image. Bordes et al. (2019) enhance the Skip-Thought model (Kiros et al., 2015) by learning a grounded space that preserves the structure of visual and textual spaces.
Recently, Tan and Bansal (2020) and Tang et al. (2021) train large scale language models with multimodal supervision from scratch with the goal of improving general language understanding. Different from the aforementioned works, we focus on learning visually grounded sentence embeddings by fine-tuning pre-trained models in a contrastive learning framework.
To exploit both visual and textual information, we adopt SimCSE (Gao et al., 2021) as the textual baseline and extend it with a multimodal contrastive learning objective.
3.1 Background: Unsupervised SimCSE
Data augmentation plays a critical role in contrastive self-supervised representation learning (Chen et al., 2020). The idea of unsupervised SimCSE is to use dropout noise as a simple yet effective data augmentation strategy. Given a collection of sentences  $\{x_{i}\}_{i = 1}^{m}$ , we construct a positive pair for each input  $x_{i}$  by encoding it twice using different dropout masks:  $h_i^z = g_\phi (f_\theta (x_i,z))$  and  $h_i^{z'} = g_\phi (f_\theta (x_i,z'))$ , where  $z$  and  $z^{\prime}$  denote different dropout masks,  $f_{\theta}(\cdot)$  is a pre-trained language encoder such as BERT, and  $g_{\phi}(\cdot)$  is a projection head on top of the [CLS] token. The training objective is:
$$
\ell_ {i} ^ {\mathrm {S}} = - \log \frac {e ^ {\operatorname {s i m} \left(\boldsymbol {h} _ {i} ^ {z _ {i}} , \boldsymbol {h} _ {i} ^ {z _ {i} ^ {\prime}}\right) / \tau}}{\sum_ {j = 1} ^ {N} e ^ {\operatorname {s i m} \left(\boldsymbol {h} _ {i} ^ {z _ {i}} , \boldsymbol {h} _ {j} ^ {z _ {j} ^ {\prime}}\right) / \tau}}, \tag {1}
$$
2The standard dropout masks in Transformers are used.
3There is a MLP pooler layer over [CLS] in BERT's implementation. Gao et al. (2021) use it with re-initialization.
Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.↑
BERT (first-last avg.)	39.7	59.4	49.7	66.0	66.2	53.9	62.1	56.7
RoBERTa (first-last avg.)	40.9	58.7	49.1	65.6	61.5	58.6	61.6	56.6
SimCSE-BERT◇	68.4	82.4	74.4	80.9	78.6	76.9	72.2	76.3
SimCSE-RoBERTa◇	70.2	81.8	73.2	81.4	80.7	80.2	68.6	76.6
SimCSE-BERT	67.8±1.6	80.0±2.1	72.5±1.7	80.1±0.8	77.6±0.8	76.5±0.8	70.1±0.9	74.9±1.1
SimCSE-RoBERTa	68.7±1.0	82.0±0.5	74.0±1.0	82.1±0.4	81.1±0.4	80.6±0.3	69.2±0.2	76.8±0.5
SimCSE-BERT	69.9±1.7	79.8±1.5	72.9±0.9	81.9±0.8	77.8±0.9	76.6±1.1	68.4±0.8	75.3±0.9
MCSE-BERT	71.4±0.9	81.8*±1.3	74.8*±0.9	83.6±0.9	77.5±0.8	79.5*±0.5	72.6*±1.4	77.3*±0.5
SimCSE-RoBERTa	69.5±0.9	81.6±0.5	74.1±0.6	82.4±0.3	80.9±0.5	79.9±0.3	67.3±0.5	76.5±0.4
MCSE-RoBERTa	71.7*±0.2	82.7*±0.4	75.9*±0.3	84.0*±0.4	81.3±0.3	82.3*±0.5	70.3*±1.3	78.3*±0.1
SimCSE-BERT	69.1±1.0	80.4±0.9	72.7±0.7	81.1±0.3	78.2±0.9	73.9±0.6	66.6±1.2	74.6±0.2
MCSE-BERT	71.2*±1.3	79.7±0.9	73.8±0.9	83.0*±0.4	77.8±0.9	78.5*±0.4	72.1*±1.4	76.6*±0.5
SimCSE-RoBERTa	66.4±0.9	80.7±0.7	72.7±1.1	81.3±0.9	80.2±0.8	76.8±0.6	65.7±0.7	74.8±0.5
MCSE-RoBERTa	70.2*±1.7	82.0*±0.7	75.5*±1.2	83.0*±0.6	81.5*±0.7	80.8*±1.0	69.9*±0.6	77.6*±0.8
*: difference between SimCSE and MCSE is significant at  $\alpha  = {0.05}$  according to an independent t-test.
Table 1: Performance comparison on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across 7 tasks.  $\diamond$ : single seed results from Gao et al. (2021). All other results are from our implementation. Models are trained with 5 random seeds and we report the means and standard deviations.
where  $N$  is the size of the mini-batch,  $\tau$  is a temperature parameter and  $\mathrm{sim}(h_1,h_2)$  is the cosine similarity  $\frac{h_1^T h_2}{\|h_1\| \cdot \|h_2\|}$ . After training, the [CLS] token outputs of the language encoder are taken as the sentence embeddings.
3.2 Multimodal Contrastive Learning
Beyond the textual objective in SimCSE, we introduce a multimodal objective within the contrastive learning framework. The overview of our MCSE model is shown in Figure 1. Given a collection of sentence-image pairs  $D = \{x_{i},y_{i}\}_{i = 1}^{m}$ , firstly we map sentence  $x_{i}$  and image  $y_{i}$  into a shared space:
$$
\pmb {s} _ {i} ^ {z} = g _ {\phi_ {1}} (f _ {\theta} (x _ {i}, z)), \pmb {v} _ {i} = g _ {\phi_ {2}} (f ^ {v} (y _ {i})), \quad (2)
$$
where  $f^v(\cdot)$  is a pre-trained image encoder such as ResNet (He et al., 2016), which is fixed during training.  $g_{\phi_1}(\cdot)$  and  $g_{\phi_2}(\cdot)$  are distinct projection heads for text and image modality respectively. To pull semantically close image-sentence pairs together and push away non-related pairs, we define the multimodal contrastive learning objective as:
$$
\ell_ {i} ^ {\mathrm {M}} = - \sum_ {z \in \{z _ {i}, z _ {i} ^ {\prime} \}} \log \frac {e ^ {\mathrm {s i m} (\pmb {s} _ {i} ^ {z} , \pmb {v} _ {i}) / \tau^ {\prime}}}{\sum_ {j = 1} ^ {N} e ^ {\mathrm {s i m} (\pmb {s} _ {i} ^ {z} , \pmb {v} _ {j}) / \tau^ {\prime}}}, (3)
$$
where  $\tau^{\prime}$  is a temperature parameter. Let  $\lambda$  denote the trade-off hyperparameter between two objectives, we formulate the final loss as:
$$
\ell_ {i} = \ell_ {i} ^ {S} + \lambda \ell_ {i} ^ {M}. \tag {4}
$$
Our method further regularizes the sentence representation in a way that aligns with the image representation in the grounded space.
Dataset We use Flickr30k (Young et al., 2014) and MS-COCO (Lin et al., 2014) as our multimodal datasets. Flickr30k contains 29,783 training images and MS-COCO contains 82,783 training images. Each image is annotated with multiple captions and we randomly sample only one caption to create image-sentence pairs. Following Gao et al. (2021), we use Wiki1M as the text-only corpus, which consists of  $10^{6}$  sentences randomly drawn from English Wikipedia.
Implementation Details We use  $\mathrm{BERT}_{\mathrm{base}}$  (Devlin et al., 2019) and  $\mathrm{RoBERTa}_{\mathrm{base}}$  (Liu et al., 2019) as language encoders and ResNet-50 (He et al., 2016) as the image encoder. Distinct single-layer MLPs are applied as projection heads. More details are provided in Appendix A.
Evaluation We evaluate the trained models on seven Semantic Textual Similarity (STS) tasks: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). Each of these datasets consists of a collection of sentence pairs and the goal is to predict a similarity score for each sentence pair. Following Gao et al. (2021), we report the Spearman's correlation  $(\times 100)$  between
5961
gold annotations and predicted scores in the "all" setting, i.e., for each task, we concatenate all the subsets and report the overall Spearman's correlation.
Augmenting text-only corpus with small scale multimodal data yields significant improvements. To fully utilize different types of data resources, we conduct experiments with a text-only corpus and multimodal data. SimCSE is trained on sentences and captions only, while MCSE additionally computes the multimodal objective for image_caption pairs. As shown in Table 1, averaging the off-the-shelf BERT and RoBERTa embeddings yields poor performance on STS tasks. SimCSE models significantly outperform the average embeddings. MCSE models, which have access to auxiliary visual information, further achieve noticeable improvements even if the amount of multimodal data is relatively small. When MCSE is applied to the combination of Wiki1M and Flickr30k, it improves the state-of-the-art result for BERT  $(76.3\rightarrow 77.3)$  and RoBERTa  $(76.6\rightarrow 78.3)$  by a decent margin. Looking at performance on the individual tasks, we find that MCSE models using BERT encoder perform worse on STS16. This can be attributed to the domain discrepancy, where some subsets that are close to the training distribution benefit more from visually grounding than others (see Appendix B.1).
To further investigate the impact of different datasets, we train models solely on multimodal data and report results in Table 2. We observe that, without the large text-only corpus, the performances decrease considerably compared to results in Table 1. Still, MCSE models consistently surpass SimCSE models (0.9 - 3.8 points improvement). Moreover, replacing the paired images with shuffled images before training MCSE leads to 0.8 - 5.0 points reduction in terms of average Spearman's correlation, further validating the efficacy of visual semantics. We also replace the ResNet encoder with CLIP (Radford et al., 2021) and our results show that different image encoders lead to similar results. Details are shown in Appendix B.2.
Grounding to the visual world improves alignment and maintains uniformity. To dissect the inner workings of MCSE, we use two quantifiable
Model	Trained on
flickr	coco
SimCSE-BERT	68.8±0.7	67.8±0.4
MCSE-BERT	70.6*±0.5	71.6*±0.2
w/shuffling	67.9±0.6↓	66.6±0.3↓
SimCSE-RoBERTa	72.9±0.3	72.8±0.3
MCSE-RoBERTa	73.8*±0.2	74.3*±0.3
w/shuffling	73.0±0.4↓	72.8±0.3↓
*: difference between SimCSE and MCSE is significant.
Table 2: Comparison of the average Spearman's correlation on 7 STS tasks (Avg. column in Table 1). We report the means and standard deviations over 5 seeds.
Figure 2: The alignment-uniformity plot of models when using BERT encoder. Colors of dots represent the average Spearman's correlation.
metrics proposed in Wang and Isola (2020): alignment and uniformity, as measurements of representation quality. Let  $p_{\mathrm{pos}}$  denote the positive pairs distribution and  $p_{\mathrm{data}}$  denote the data distribution. The alignment loss prefers encoders that assign similar features to semantically similar instances (assuming features have been normalized):
$$
\mathcal {L} _ {\text {a l i g n}} \triangleq \underset {(x, x ^ {+}) \sim p _ {\mathrm {p o s}}} {\mathbb {E}} \| f (x) - f \left(x ^ {+}\right) \| _ {2} ^ {2}. \tag {5}
$$
And the uniformity loss prefers a uniform distribution in the hypersphere:
$$
\mathcal {L} _ {\text {u n i f o r m}} \triangleq \log \underset {x, y \stackrel {i. i. d.} {\sim} p _ {\text {d a t a}}} {\mathbb {E}} e ^ {- 2 \| f (x) - f (y) \| _ {2} ^ {2}}. \tag {6}
$$
Gao et al. (2021) empirically showed that sentence embedding models with both lower alignment and uniformity achieve better performance in general. Similarly, we calculate the two losses on STS-B<sup>5</sup>
$^{4}$ Following (Gao et al., 2021), we take the average of the first and last layers, which is better than only using the last.
We take STS-B pairs with a score higher than 4.0 as  $p_{\mathrm{pos}}$  and the full STS-B as  $p_{\mathrm{data}}$ . Since Gao et al. (2021) did not
5962
Figure 3: Performances of different data scales. The full set indicates 30K and 87K samples for Flickr30k and MS-COCO respectively.
and results are presented in Figure 2. It shows that MCSE models achieve better alignment scores compared to SimCSE while maintaining uniformity. This analysis provides further support that visually grounding can enhance sentence representation learning by improving the alignment property of the textual embedding space.
For brevity, we take BERT-based models trained merely on caption datasets and investigate the impact of training data scales. More analysis results (sentence retrieval, cross-modal retrieval) are provided in Appendix B.3. We limit the number of training samples to 100, 500, 1000, 5000 and 10000, and compare their performance with the full set performance. In all of these settings, we optimize the models for same number of training steps as the full set setting. The results are shown in Figure 3. SimCSE achieves better performance than MCSE with limited samples, while MCSE starts to outperform SimCSE with the increasing data scale. We conjecture that this phenomenon can be ascribed to the progressive training of weights in multimodal projection heads.
Despite showing performance improvements on STS benchmarks, MCSE has its limitations as well. We take caption datasets as the source of multimodal information, while these datasets are collected and curated with non-negligible human efforts. It will have great practical value if we can properly leverage noisy image-sentence pairs or even get rid of the explicit alignments between im
release the code for calculating these two losses, the absolute values we obtained might be different from theirs. We make sure our calculation across different models is consistent.
ages and sentences. Furthermore, we find that only subsets from related domains can get significant improvements while others suffer from distribution shifts. It is critical to mitigate domain gaps for learning general-purpose sentence embeddings. In addition, the definition of "semantic similarity" is highly task-dependent. Besides STS benchmarks, it is worth exploring the performance gap between text-only models and multimodal models on other benchmarks that can also assess the quality of sentence representations.
In this paper, we propose MCSE, a novel approach for sentence embedding learning that applies a multimodal contrastive objective to align sentences and corresponding images in a grounded space. Experiments show that MCSE consistently improves the performance on STS tasks. We also highlight the superiority of our method by analyzing the alignment and uniformity properties of the embedding space. The multimodal objective is generic and can be potentially incorporated into other sentence embedding methods to boost their performance.
We thank Dingfan Chen, Fangzhou Zhai and Xiaoyu Shen for their helpful comments on the paper draft. We would also like to thank the reviewers for their valuable feedback. This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - project-id 232722074 - SFB 1102. This work was also partially funded by the EU Horizon 2020 project ROXANNE under grant number 833635 and COMPRISE grant number 3081705.
5963
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 252-263.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 81-91.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), pages 497-511.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385-393.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pages 32-43.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5185-5198.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735.
Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. 2019. Incorporating visual semantics into sentence representations within a grounded space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International