SlowGuess/NAACL_2022_OCR huggingface dataset:
Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. 2022. mLUKE: The power of entity representations in multilingual pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.
Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. In 9th International Conference on Learning Representations.
Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. In Proceedings of the 2nd Workshop on Representation Learning for NLP.
Holger Schwenk and Xian Li. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In 8th International Conference on Learning Representations.
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2020. Wikipedia2Vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from Wikipedia. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017. Learning distributed representations of texts and entities from knowledge base. Transactions of the Association for Computational Linguistics, 5:397-411.
Ikuya Yamada, Hiroyuki Shindo, and Yoshiyasu Takefuji. 2018. Representation learning of entities and documents from knowledge base descriptions. In Proceedings of the 27th International Conference on Computational Linguistics.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2021. Universal sentence representation learning with conditional masked language model. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021. Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Using Comparable Corpora.
We implement our EASE model using transformers $^{10}$  libraries. For the monolingual settings, we use the STS-B development set as in (Gao et al., 2021). For multilingual settings, we use the STS-B and SICK-R development set. In this setting, we simply concatenate the entity-sentence paired data for all 18 languages and randomly sample from the concatenated data to construct batches. $^{11}$  In both settings, we train our model for one epoch, compute evaluation scores every 250 training steps on the development data, and keep the best model. We conduct a grid-search for batch size  $\in$  {64, 128, 256, 512} and learning rate  $\in$  {3e - 05, 5e - 05}. The chosen hyperparameters for each model is shown in Table 9.
Model	Batch size	Learning Rate
SimCSE-mBERTbase	128	3e-05
SimCSE-XLM-Rbase	128	3e-05
EASE-BERTbase	64	3e-05
EASE-RoBERTabase	128	5e-05
EASE-mBERTbase	256	5e-05
EASE-XLM-Rbase	64	3e-05
For the loss balancing term  $\lambda$  and softmax temperature  $\tau$  in the EASE models (section 3), we empirically find that  $\lambda = 0.01$ ,  $\tau = 100$  for the monolingual setting and  $\tau = 10$  for the multilingual setting work well.
Computing Infrastructure We run the experiments on a server with AMD EPYC 7302 16-Core CPU and a NVIDIA A100-PCIE-40GB GPU. The training of EASE takes approximately 1 hour.
B Pooling Methods for SimCSE and EASE
We compare several pooling methods on both SimCSE and EASE in the multilingual setting: [CLS] with MLP; [CLS] with MLP during training only; [CLS] without MLP; mean pooling. Table 10 shows the evaluation results based on the STS-B and SICK-R development set.
Table 9: Hyperparameters for experiment.
Pooler	SimCSE	EASE
[CLS] pooling
w/ MLP	63.0	65.0
w/ MLP (train)	72.0	73.3
w/o MLP	72.0	73.4
mean pooling	72.1	73.8
Table 10: Average Spearman's correlation for different pooling methods for SimCSE and EASE in multilingual setting on STS-B and SICK-R development set.
The mean pooling representation performs best on both models. We thus use mean pooling on both models in Section 5.
<sup>10</sup>https://huggingface.co/docs/transformers/index
In our preliminary experiments, we also tested a setting in which data in the same language were used within the same batch; we did not observe a consistent improvement in the performance of either the SimCSE or EASE models.
C Parallel Sentence Mining
We evaluate the multilingual sentence embeddings with the parallel sentence mining task using the BUCC 2018 shared task dataset (Zweigenbaum et al., 2018). The task is to find the parallel pairs given monolingual sentence pools in two languages, with  $2 - 3\%$  of the sentences being parallel, to find the parallel pairs.
Each model uses the raw embedding output and performance is evaluated without fine-tuning. We first encode all sentences into embeddings and compute the cosine similarity scores between all possible sentence pairs. We then retrieve the sentence pairs with above a fixed threshold and compute the F1 score using the ground-truth parallel pairs.
As the test set is not publicly available, we use the sample set to tune the threshold of the parallel sentence mining and the training set for evaluation, which is a common practice in similar studies (Hu et al., 2020; Feng et al., 2020).
The results are summarized in Table 11. Our EASE models outperform the SimCSE baselines across the languages, demonstrating that the entity contrastive objective improves the alignment of the multilingual sentence embeddings without a parallel corpora. However, performance is significantly poor than that of LaBSE, which is trained using massive amounts of parallel corpora, suggesting that we still need parallel resources to be competitive on this task.
en-de	en-fr	en-ru	en-zh
SimCSE-mBERTbase	13.2	19.2	7.9	11.5
EASE-mBERTbase	26.9	33.8	24.2	32.9
SimCSE-XLM-Rbase	31.8	32.3	28.9	19.9
EASE-XLM-Rbase	33.3	33.2	33.6	23.4
LaBSE	89.0	88.2	84.7	74.2
D Detailed Settings for MLDoc Experiment
We use the english.train.1000 and english.dev datasets for the training and validation data, respectively. We conduct a grid-search for batch size  $\in$  {32, 64, 128} and learning rate  $\in$  {0.1, 0.01, 0.001} using validation data 12. We run the experiment three times with different random seeds and record the average scores.
Table 11: The F1 scores on BUCC 2018 the training set. Retrieval is performed in forward search, i.e., English sentences as the targets and the other language as the queries.
Model	Batch size	Learning Rate
mBERTbase(avg.)	32	0.1
XLM-Rbase(avg.)	32	0.1
SimCSE-mBERTbase	32	0.1
SimCSE-XLM-Rbase	32	0.01
EASE-mBERTbase	32	0.01
EASE-XLM-Rbase	32	0.01
Table 12: Hyperparameters for MLDoc experiment
E Construction of MewsC-16 Dataset
To construct the MewsC-16 dataset, we collect sentences for each category in each language from the Wikinews dump. $^{12}$  We first select 13 topic categories in the English Wikinews $^{13}$  that are also defined in other languages (Science and technology, Politics and conflicts, Environment, Sports, Health, Crime and law, Obituaries, Disasters and accidents, Culture and entertainment, Economy and business, Weather, Education, Media). We then collect pages with topic categories for each language and remove the pages with two or more topic categories. We clean the text on each page with the Wikiextractor tool $^{14}$ , and split it into sentences using the polyglot sentence tokenizer. Finally, we use the first sentence assuming that it well represents the topic of the entire article (Baxendale, 1958; Edmundson, 1969). The corpus statistics for each language are shown in Table 13.
Language	# of sentences	# of label types	Language	# of sentences	# of label types
ar	2,243	11	fr	10,697	13
ca	3,310	11	ja	1,984	12
cs	1,534	9	ko	344	10
de	6,398	8	pl	7,247	11
en	12,892	13	pt	8,921	11
eo	227	8	ru	1,406	12
es	6,415	11	sv	584	7
fa	773	9	tr	459	7
total	65,425	13
Table 13: Corpus statistics for MewsC-16
$^{12}$ https://dumps.wikimedia.org/backup-index.html  
 $^{13}$ https://en.wikinews.org/wiki/Category:News_articles_by_section  
 $^{14}$ https://github.com/attardi/wikiextractor
F Baselines
For average GloVe embedding (Pennington et al., 2014), we use open-source GloVe vectors trained on Wikipedia and Gigaword with 300 dimensions. $^{15}$  We use the pretrained model from HuggingFace's Transformers $^{16}$  for vanilla pretrained language models, including BERT (bert-base-uncased) (Devlin et al., 2019), RoBERTa (roberta-base) (Liu et al., 2019), mBERT (bert-base-multilingual-cased) and XLM-R (xlm-roberta-base) (Conneau et al., 2020). We use the published checkpoints for unsupervised SimCSE (Gao et al., 2021) $^{17}$ , CT (Carlsson et al., 2021) $^{18}$ , and DeCLUTR (Giorgi et al., 2021). $^{19}$
G Monolingual STS and STC
Table 14 and 15 show the complete results for seven STS tasks and eight STC tasks. For STS, the average EASE performance is slightly better than that of SimCSE, although the advantage is not consistent across tasks. For most of the STC tasks, EASE consistently outperforms SimCSE. These results indicate that EASE stands out at capturing high-level categorical semantic structures and that its ability to measure sentence semantic similarity is comparable to or better than that of SimCSE.
Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.
GloVe embeddings (avg.)	55.1	70.7	59.7	68.3	63.7	58.0	53.8	61.3
BERTbase (avg.)	30.9	59.9	47.7	60.3	63.7	47.3	58.2	52.6
BERTbase-flow	58.4	67.1	60.9	75.2	71.2	68.7	64.5	66.6
BERTbase-whitening	57.8	66.9	60.9	75.1	71.3	68.2	63.7	66.3
IS-BERTbase	56.8	69.2	61.2	75.2	70.2	69.2	64.3	66.6
CT-BERTbase	61.6	76.8	68.5	77.5	76.5	74.3	69.2	72.1
SimCSE-BERTbase	68.4	82.4	74.4	80.9	78.6	76.9	72.2	76.3
EASE-BERTbase	72.8	81.8	73.7	82.3	79.5	78.9	69.7	77.0
RoBERTa(base)	32.1	56.3	45.2	61.3	62.0	55.4	62.0	53.5
RoBERTa(base) (first-last avg.)	40.9	58.7	49.1	65.6	61.5	58.6	61.6	56.6
DeCLUTR-RoBERTa base	52.4	75.2	65.5	77.1	78.6	72.4	68.6	70.0
SimCSE-RoBERTa base	68.7	82.6	73.6	81.5	80.8	80.5	67.9	76.5
EASE-RoBERTa base	70.9	81.5	73.5	82.6	80.5	80.0	68.4	76.8
Table 14: Spearman's correlation for monolingual semantic textual similarity tasks.
Model	AG	Bio	G-S	G-T	G-TS	SO	SS	Tweet	Avg.
GloVe embeddings (avg.)	83.2	30.7	59.0	58.3	67.4	29.9	70.4	52.1	56.4
BERTbase (avg.)	79.8	32.5	55.0	47.0	62.4	21.7	64.0	44.6	50.9
CT-BERTbase	79.2	38.7	65.5	60.7	69.8	67.9	55.5	55.2	61.6
SimCSE-BERTbase	74.4	34.3	59.5	57.8	64.4	49.6	64.3	52.1	57.1
EASE-BERTbase	85.8	36.2	60.5	60.4	67.0	68.1	71.7	54.8	63.1
RoBERTabase (avg.)	66.5	26.6	47.9	42.8	58.3	16.7	30.0	38.6	40.9
DeCLUTR-RoBERTabase	80.7	41.0	65.2	60.5	69.6	32.9	73.6	56.8	60.0
SimCSE-RoBERTabase	69.8	37.3	60.0	58.0	66.6	69.3	48.3	50.0	57.4
EASE-RoBERTabase	69.4	39.3	60.7	57.7	66.3	73.9	49.4	51.8	58.6
Table 15: Clustering accuracy for monolingual short text clustering tasks.
15 https://nlp.stanford.edu/projects/glove/  
16 https://github.com/huggingface/transformers  
17 https://github.com/princeton-nlp/SimCSE  
18 https://github.com/FreddeFrallan/Contrastive-Tension  
19 https://github.com/JohnGiorgi/DeCLUTR