SlowGuess/NAACL_2022_OCR huggingface dataset:
Tables 2 and 3 show the results of our multilingual STS and STC experiments. Overall, EASE substantially outperforms the corresponding base models (i.e., mBERT and XLM-R) on both tasks. Similar to the results for the monolingual setting, the average performance of EASE exceeds that of SimCSE for multilingual STC tasks with an improvement of  $34.1\%$  to  $36.0\%$  for mBERT and  $31.8\%$  to  $35.4\%$  for XLM-R. This result suggests that even in a multilingual setting, EASE can encode high-level categorical semantic structures into sentence embeddings. Moreover, EASE significantly outperforms SimCSE in multilingual STS tasks. Specifically, the score of EASE-mBERT is better than that of SimCSE-mBERT (50.9 vs 57.2), and that of EASE-XLM-R is better than that of SimCSE-XLM-R (54.1 vs 57.1). This improvement is much more significant than the monolingual setting (Table 1), where the improvement is less than one point. This indicates the effectiveness
of language-independent entities as cross-lingual alignment supervision in learning multilingual sentence embeddings.
5.3 Cross-lingual Parallel Matching
We evaluate EASE on the Tatoeba dataset (Artetxe and Schwenk, 2019) to assess more directly its ability to capture cross-lingual semantics. This task is to retrieve the correct target sentence for each query sentence, given a set of parallel sentences. We perform the retrieval using the cosine similarity scores of the sentence embeddings. For each language-pair dataset, we compute the retrieval accuracy averaged over the forward and backward directions (English to the target language and vice-versa).
Table 4 shows the evaluation results for the languages in the CL training data. EASE significantly outperforms the corresponding base models and SimCSE for all languages. Notably, the mean performance of EASE-mBERT is better than that of vanilla mBERT by 13.5 percentage points. This indicates that EASE can capture cross-lingual semantics owing to the cross-lingual supervision of entity annotations, which aligns semantically similar sentences across languages. One interesting observation is that the performance of SimCSE-mBERT is worse than that of vanilla mBERT. We conjecture that this is because the SimCSE model is trained using only the positive sentence pairs within the same language, which sometimes leads to less language-neutral representations.
$^{6}$ https://en.wikipedia.org/wiki/Main_Page
To further explore the cross-lingual ability of EASE, we evaluate it on languages not included in the EASE training set (Table 5). The results show that EASE performs robustly on these languages as well, which suggests that, in EASE, the cross-lingual alignment effect propagates to other languages not used in additional training with EASE (Kvapilíková et al., 2020).
5.4 Cross-lingual Zero-shot Transfer
We further evaluate our sentence embeddings on a downstream task in which sentence embeddings are used as input features, especially in the cross-lingual zero-shot transfer setting. For evaluation in this setting, we use MLDoc (Schwenk and Li, 2018), a cross-lingual document classification dataset that classifies news articles in eight languages into four categories. We train a linear classifier using sentence embeddings as input features on the English training data, and evaluate the resulting classifier in the remaining languages. To directly evaluate the ability of the resulting sentence embeddings, we do not update the parameters of the sentence encoder but only train the linear classifier in this setting. The detailed settings are shown in Appendix D.
As shown in Table 6, our EASE models achieve the best average performance on both back-bones, suggesting that multilingual embeddings learned with the CL are also effective in the cross-lingual transfer setting.
6 Case Study: Fine-tuning Supervised Model with EASE
Existing multilingual sentence representation models trained on a large parallel corpus do not always perform well, especially for languages that are not included in the training data. In contrast, EASE requires only the Wikipedia text corpus, which is available in more than 300 languages.7 Thus, one possible use case for EASE would be to complement the performance of existing models in low-resource languages by exploiting the Wikipedia data in those languages.
To test this possibility, we fine-tune LaBSE (Feng et al., 2020), which is trained on both monolingual and bilingual data in 109 languages, with our EASE framework in five low-resource languages (kab, pam, cor, tr, mhr). These languages
Figure 2: Results of fine-tuning LaBSE with EASE framework on Tatoeba dataset.
are not present in the original training corpus, so the model performed particularly poorly on these languages. We fine-tune the model using 5,000 pairs each from English and the corresponding language data.
As shown in Figure 2, EASE improves the performance of LaBSE across all target languages, which is an intriguing result considering that LaBSE has already been trained on about six billion parallel corpora. These results suggest the potential benefit of combining EASE with other models using parallel corpora, especially for languages without or with only a few parallel corpora.
We conduct ablation experiments to better understand how each component of EASE contributes to its performance. We measure the performance of the models using monolingual STS in the monolingual setting and multilingual STS in the multilingual setting, without one of the following components: the self-supervised CL loss, hard negatives, and Wikipedia2Vec initialization (Table 7). As a result, we find all of the components to make an important contribution to the performance.
It is worth mentioning that entity CL alone (i.e., w/o self-supervised CL) also improves the baseline performance significantly. The performance contributions in the multilingual setting are particularly significant (49.3 for mBERT and 53.1 for XLM-R) and comparable to those for the SimCSE models. These results suggest that CL with entities by itself is effective in learning multilingual sentence embeddings.
$^{7}$ https://meta.wikimedia.org/wiki/List_of_Wikipediaias
Setting	EASE-BERTbase STS avg.	EASE-RoBERTa base STS avg.	EASE-mBERTbase mSTS avg.	EASE-XLM-Rbase mSTS avg.
Full model	76.9	76.8	57.2	57.1
w/o self-supervised CL	65.3	66.1	49.3	53.1
w/o hard negative	75.3	76.1	53.8	52.7
w/o Wikipedia2Vec	73.8	76.3	52.1	54.3
w/o all (vanilla model)	31.4	43.6	35.4	25.0
Table 7: Results of ablation study.
Figure 3:  $l_{\mathrm{align}} - l_{\mathrm{uniform}}$  plot of BERT-based (or mBERT-based) models in monolingual (left) and multilingual (right) settings.
7.2 Alignment and Uniformity
To further understand the source of the performance improvement with EASE, we evaluate two key properties to measure the quality of the representations obtained from contrastive learning (Wang and Isola, 2020): alignment measures the closeness of representations between positive pairs; uniformity measures how well the representations are uniformly distributed. We let  $f(x)$  denote the normalized representation of  $x$ , and compute the two measures using
$$
l _ {\text {a l i g n}} \triangleq \underset {(x, x ^ {+}) \sim p _ {\text {p o s}}} {\mathbb {E}} \| f (x) - f (x ^ {+}) \| ^ {2}, \tag {5}
$$
$$
l _ {\text {u n i f o r m}} \triangleq \log \underset {x, y \stackrel {i. i. d.} {\sim} p _ {\text {d a t a}}} {\mathbb {E}} e ^ {- 2 \| f (x) - f (y) \| ^ {2}}, \tag {6}
$$
where  $p_{\mathrm{pos}}$  denotes positive pairs and  $p_{\mathrm{data}}$  denotes the entire data distribution. We compute these metrics using BERT-based models on the STS-B development set data. For investigation in the multilingual setting, we compute them using mBERT-based models on the multilingual STS data used in the experiment Section 5. We compute the averages of alignment and uniformity for each language pair. For each setting, we take STS pairs with a score higher than 4 in the 0-to-5 scale as  $p_{\mathrm{pos}}$  and all STS sentences as  $p_{\mathrm{data}}$ .
As shown in Figure 3, the trends are similar in both settings: (1) both EASE and SimCSE significantly improve uniformity compared with that for the vanilla model; (2) EASE is inferior to SimCSE in terms of uniformity and superior in terms of alignment. This result suggests that entity CL does not have the effect of biasing embeddings towards a more uniform distribution. Instead, it has the effect of aligning semantically similar samples, which leads to the improved performance of the resultant sentence embeddings.
7.3 Qualitative analysis
To investigate how EASE improves the quality of sentence embeddings, we conduct qualitative analysis by comparing the results of EASE and SimCSE on STS Benchmark. Table 8a shows typical cases of how EASE improves sentence embeddings. We find that EASE embeddings are more robust to synonyms and grammatical differences since they are more aware of the topic similarity between sentences, resulting in more accurate score inference. On the other hand, as shown in the deterioration cases in Table 8b, EASE embeddings are sometimes overly sensitive to topical similarity, making it difficult to capture the correct meaning of the whole sentence.
Our experiments have demonstrated that entity supervision in EASE improves the quality of sentence embeddings both in the monolingual setting and, in particular, the multilingual setting. As recent studies have shown, entity annotations can be used as anchors to learn quality cross-lingual representations (Calixto et al., 2021; Nishikawa et al., 2021; Jian et al., 2022; Ri et al., 2022), and our work is another demonstration of their utility, particularly in sentence embeddings. One promising future direction is exploring how to better exploit the cross-lingual nature of entities.
Sentence1	Sentence2	Gold	EASE	SimCSE
i think you’re looking for mikey ( 1992) .	i think you’re looking for the movie	3.00	2.32	1.62
the new york senator’s new book , " living history , " appears a certain bestseller .	hillary clinton , the new york senator and former first lady , has a book out monday titled living history .	3.20	3.57	1.94
he was referring to john s. reed , the former citicorp chief executive who became interim chairman and chief executive of the exchange last sunday .	next week , john s. reed , the former citicorp chief executive who sunday became interim chairman and chief executive of the exchange , will take up his position .	4.00	3.52	2.73
(a) Improvement cases
Sentence1	Sentence2	Gold	EASE	SimCSE
it ’s not a good idea .	it ’s a good question .	0.00	2.88	1.33
suicide attack kills eight in baghdad	suicide attacks kill 24 people in bagh-dad	2.40	3.92	2.43
the nasdaq composite index rose 19.67 , or 1.3 percent , to 1523.71 , its high-est since june 18 .	the s and p 500 had climbed 16 per-cent since its march low and yesterday closed at its highest since dec. 2 .	0.80	3.25	2.04
(b) Deterioration cases
Table 8: Comparison of STS Benchmark results by monolingual EASE and SimCSE.
Our experiments also demonstrate the utility of Wikipedia as a multilingual database. As described in Section 6, Wikipedia entity annotations can compensate for the lack of parallel resources in learning cross-lingual representations. Wikipedia currently supports more than 300 languages, and around half of them have over 10,000 articles.[8] Moreover, Wikipedia is ever growing; it is expected to include more and more languages.[9] This will motivate researchers to develop methods for multilingual models including low-resource languages in the aid of entity annotations in Wikipedia.
However, the reliance on Wikipedia for training data may limit the application of the models to specific domains (e.g., general or encyclopedia domains). To apply EASE to other domains, one may need to annotate text from the domain either manually or automatically. Future work can investigate the effectiveness of the entity CL in other domains and possibly its the combination with an entity linking system.
Finally, we note that the supervision signal in EASE is inherently noisy. Different entities have
different characteristics as a topic indicator, and sentences that contain the same entity do not necessarily share meaning. Future work can address this by considering how an entity is used in a sentence to obtain more reliable supervision.
This work was partially supported by JSPS KAKENHI Grants JP16H06302, JP18H04120, JP21H04907, and JP21K18023, and by JST CREST Grants JPMJCR18A6 and JPMJCR20D3, Japan. JPMJCR18A6: JST CREST (FY2018-FY2023) PI Prof. Yamagishi JPMJCR20D3: JST CREST (FY2020-FY2025) PI Prof. Echizen JP16H06302: KAKENHI, Scientific Research (S) (FY2016-FY2020) PI Prof. Babaguchi, Co-PI Prof. Echizen JP18H04120: KAKENHI, Scientific Research (A) (FY2018-FY2020) PI Prof. Echizen, Co-PI Prof. Yamagishi
<Follows are newly accepted.> JP21H04907: KAKENHI, Scientific Research (A) (FY2021-FY2023) PI Prof. Echizen, Co-PI Prof. Yamagishi JP21K18023: KAKENHI, Early-Career Scientists (FY2021-FY2023) PI Dr. Le Trung-Nghia
https://meta.wikimedia.org/wiki/List_of_Wikipedias  
https://incubator.wikimedia.org/wiki/Incubator:Main_Page
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014).
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity.*
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
P. B. Baxendale. 1958. Machine-made index for technical literature: An experiment. IBM J. Res. Dev., 2(4):354-361.
Iacer Calixto, Alessandro Raganato, and Tommaso Pasini. 2021. Wikipedia entities as rendezvous across languages: Grounding multilingual language models by predicting Wikipedia hyperlinks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive
tension. In 9th International Conference on Learning Representations.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
H. P. Edmundson. 1969. New methods in automatic extracting. J. ACM, 16(2):264-285.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. CERT: Contrastive self-supervised learning for language understanding. ArXiv, arXiv/2005.12766.
Fangxiaoyu Feng, Yinfei Yang, Daniel Matthew Cer, N. Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. *ArXiv*, abs/2007.01852.
Evgeniy Gabrilovich and Shaul Markovitch. 2006. Overcoming the brittleness bottleneck using wikipedia: Enhancing text categorization with encyclopedic knowledge. In AAAI.
Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, IJCAI'07.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on
Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Koustava Goswami, Sourav Dutta, Haytham Assem, Theodorus Fransen, and John P. McCrae. 2021. Cross-lingual sentence embedding using multi-task learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119.
Xiaoze Jian, Yaobo Liang, Weizhu Chen, and Nan Duan. 2022. XLM-K: Improving cross-lingual language model pre-training with multilingual knowledge. In AAAI.
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for BERT sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, volume 28.
Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Ondřej Bojar. 2020. Unsupervised multilingual sentence embeddings for parallel corpus mining. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop.
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML'14.
Jeffrey Ling, Nicholas FitzGerald, Zifei Shan, Livio Baldini Soares, Thibault Févry, David Weiss, and Tom Kwiatkowski. 2020. Learning cross-context entity representations from text. ArXiv, arXiv/2001.03765.
Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. ArXiv, arXiv/1907.11692.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In 6th International Conference on Learning Representations.
J. B. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In Conference on Neural Information Processing Systems.
James R. Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1):32-38.
Sosuke Nishikawa, Ikuya Yamada, Yoshimasa Tsuruoka, and Isao Echizen. 2021. A multilingual bag-of-entities model for zero-shot cross-lingual text classification. ArXiv, arXiv/2110.07792.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).