SlowGuess/NAACL_2022_OCR huggingface dataset:
Sosuke Nishikawa†,‡*
sosuke-nishikawa@alumni.u-tokyo.ac.jp
Ryokan  $\mathbf{R}\mathbf{i}^{\dagger ,\ddagger}$
ryo0123@ousia.jp
Ikuya Yamada†,§
ikuya@ousia.jp
Yoshimasa Tsuruoka†
tsuruoka@logos.t.u-tokyo.ac.jp
Isao Echizen
iechizen@qq.com
†Studio Ousia ‡The University of Tokyo
$^{\S}$ RIKEN AIP National Institute of Informatics
We present EASE, a novel method for learning sentence embeddings via contrastive learning between sentences and their related entities. The advantage of using entity supervision is twofold: (1) entities have been shown to be a strong indicator of text semantics and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and thus offer useful cross-lingual alignment supervision. We evaluate EASE against other unsupervised models both in monolingual and multilingual settings. We show that EASE exhibits competitive or better performance in English semantic textual similarity (STS) and short text clustering (STC) tasks and it significantly outperforms baseline methods in multilingual settings on a variety of tasks. Our source code, pretrained models, and newly constructed multilingual STC dataset are available at https://github.com/studio-ousia/ease.
The current dominant approach to learning sentence embeddings is fine-tuning general-purpose pretrained language models, such as BERT (Devlin et al., 2019), with a particular training supervision. The type of supervision can be natural language inference data (Reimers and Gurevych, 2019), adjacent sentences (Yang et al., 2021), or a parallel corpus for multilingual models (Feng et al., 2020).
In this paper, we explore a type of supervision that has been under-explored in the literature: entity hyperlink annotations from Wikipedia. Their advantage is twofold: (1) entities have been shown to be a strong indicator of text semantics (Gabrilovich and Markovitch, 2007; Yamada et al., 2017, 2018; Ling et al., 2020) and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and
Figure 1: Illustration of the main concept behind EASE. Using a contrastive framework, sentences are embedded in the neighborhood of their hyperlink entity embeddings and kept apart from irrelevant entities. Here, we share the entity embeddings across languages for multilingual models to facilitate cross-lingual alignment of the representation.
thus offer a useful cross-lingual alignment supervision (Calixto et al., 2021; Nishikawa et al., 2021; Jian et al., 2022; Ri et al., 2022). The extensive multilingual support of Wikipedia alleviates the need for a parallel resource to train well-aligned multilingual sentence embeddings, especially for low-resource languages. To demonstrate the effectiveness of entity-based supervision, we present EASE (Entity-Aware contrastive learning of Sentence Embeddings), which produces high-quality sentence embeddings in both monolingual and multilingual settings.
EASE learns sentence embeddings with two types of objectives: (1) our novel entity contrastive learning (CL) loss between sentences and their related entities (Figure 1); (2) the self-supervised CL loss with dropout noise. The entity CL objective pulls the embeddings of sentences and their related entities close while keeping unrelated entities apart. The objective is expected to arrange the sentence embeddings in accordance with semantics captured
* Work done as an intern at Studio Ousia.
Human Language Technologies, pages 3870 - 3885
by the entities. To further exploit the knowledge in Wikipedia and improve the learned embeddings, we also introduce a method for mining hard negatives based on the entity type. The second objective, the self-supervised CL objective with dropout noise (Gao et al., 2021; Liu et al., 2021), is combined with the first one to enable sentence embeddings to capture fine-grained text semantics. We evaluate our model against other state-of-the-art unsupervised sentence embedding models, and show that EASE exhibits competitive or better performance on semantic textual similarity (STS) and short text clustering (STC) tasks.
We also apply EASE to multilingual settings. To facilitate the evaluation of the high-level semantics of multilingual sentence embeddings, we construct a multilingual text clustering dataset, MewsC-16 (Multilingual Short Text Clustering Dataset for News in 16 languages). Multilingual EASE is trained using the entity embeddings shared across languages. We show that, given the cross-lingual alignment supervision from the shared entities, multilingual EASE significantly outperforms the baselines in multilingual STS, STC, parallel sentence matching, and cross-lingual document classification tasks.
We further demonstrate the effectiveness of the multilingual entity CL in a more realistic scenario for low-resource languages. Using multilingual entity CL, we fine-tune a competitive multilingual sentence embedding model, LaBSE (Feng et al., 2020), and show that the tuning improves the performance of parallel sentence matching for low-resource languages under-supported by the model.
Finally, we analyze the EASE model by studying ablated models and the multilingual properties of the sentence embeddings to shed light on the source of the improvement in the model.
2.1 Sentence Embeddings
Sentence embeddings, which represent the meaning of sentences in the form of a dense vector, have been actively studied. One of the earliest methods is Paragraph Vector (Le and Mikolov, 2014) in which sentence embeddings are trained to predict words within the text. Subsequently, various kinds of training tasks have been explored including reconstructing or predicting adjacent sentences (Kiros et al., 2015; Logeswaran and Lee, 2018) and solving a natural language inference (NLI) task
(Conneau et al., 2017).
Recently, with the advent of general-purpose pretrained language models such as BERT (Devlin et al., 2019), it has become increasingly common to fine-tune pretrained models to produce high-quality sentence embeddings, revisiting the aforementioned supervision signals (Reimers and Gurevych, 2019; Yang et al., 2021), and using self-supervised objectives based on contrastive learning (CL). In this paper, we present a CL objective with entity-based supervision. We train our EASE model with entity CL together with self-supervised CL with dropout noise and show that the entity CL improves the quality of sentence embeddings.
Contrastive learning The basic idea of contrastive representation learning is to pull semantically similar samples close and keep dissimilar samples apart (Hadsell et al., 2006). CL for sentence embeddings can be classified by the type of positive pairs used. As representative examples, several methods use entailment pairs as positive pairs in NLI datasets (Gao et al., 2021; Zhang et al., 2021). To alleviate the need for an annotated dataset, self-supervised approaches are also being actively studied. Typical self-supervised methods involve generating positive pairs using data augmentation techniques, including discrete operations such as word deletion and shuffling (Yan et al., 2021; Meng et al., 2021), back-translation (Fang et al., 2020), intermediate BERT hidden representations (Kim et al., 2021), and dropout noise within transformer layers (Gao et al., 2021; Liu et al., 2021). Contrastive tension (CT)-BERT (Carlsson et al., 2021) regards as positive pairs the outputs of the same sentence from two individual encoders. DeCLUTR (Giorgi et al., 2021) uses different spans of the same document. In contrast to these methods that perform CL between sentences, our method performs CL between sentences and their associated entities.
Multilingual sentence embeddings Another yet closely related line of research is focused on learning multilingual sentence embeddings, which capture semantics across multiple languages. Early competitive methods typically utilize the sequence-to-sequence objective with parallel corpora to learn multilingual sentence embeddings (Schwenk and Douze, 2017; Artetxe and Schwenk, 2019); recently fine-tuned multilingual pretrained models have achieved state-of-the-art performance (Feng et al., 2020; Goswami et al., 2021). However, one
drawback of such approaches is that, to achieve strong results for a particular language pair, they need rich parallel or semantically related sentence pairs, which are not necessarily easy to obtain. In this work, we explore the utility of Wikipedia entity annotations, which are aligned across languages and already available in over 300 languages. We also show that the entity CL in a multilingual scenario effectively improves the alignment of sentence embeddings between English and low-resource languages not well supported in an existing multilingual model.
2.2 Learning Representations Using Entity-based Supervision
Entities have been conventionally used to model text semantics (Gabrilovich and Markovitch, 2007, 2006). Several recently proposed methods learn text representations based on entity-based supervision by predicting entities from their relevant text (Yamada et al., 2017) or entity-masked sentences (Ling et al., 2020). In the proposed EASE model, the existing self-supervised CL method based on BERT (Gao et al., 2021) is extended using entity-based supervision with carefully designed hard negatives. Moreover, it is applied to the multilingual setting by leveraging the language-agnostic nature of entities.
3 Model and Training Data
In this section, we describe the components of our learning method for sentence embeddings, EASE, which is trained using entity hyperlink annotations available in Wikipedia.
3.1 Contrastive Learning with Entities
Given pairs of a sentence and a semantically related entity (positive entity)  $\mathcal{D} = \{(s_i,e_i)\}_{i = 1}^m$ , we train our model to predict the entity embedding  $\mathbf{e_i}\in \mathbb{R}^{d_e}$  from the sentence embedding  $\mathbf{s_i}\in \mathbb{R}^{d_s}$ . Following the contrastive framework in Chen et al. (2020), the training loss for  $(s_i,e_i)$  with a minibatch of  $N$  pairs is:
$$
l _ {i} ^ {e} = - \log \frac {e ^ {\operatorname* {s i m} \left(\mathbf {s} _ {\mathrm {i}} , \mathbf {W e} _ {\mathrm {i}}\right) / \tau}}{\sum_ {j = 1} ^ {N} e ^ {\operatorname* {s i m} \left(\mathbf {s} _ {\mathrm {i}} , \mathbf {W e} _ {\mathrm {j}}\right) / \tau}}, \tag {1}
$$
where  $\mathbf{W} \in \mathbb{R}^{\mathrm{de} \times \mathrm{ds}}$  is a learnable matrix weight,  $\tau$  is a temperature hyperparameter, and  $\sin(\cdot)$  is the cosine similarity  $\frac{\mathbf{s}_1^\top \mathbf{s}_2}{\|\mathbf{s}_1\| \cdot \|\mathbf{s}_2\|}$ .
Data We construct the sentence-entity paired datasets from the January 2019 version of Wikipedia dump. We split text in the articles into sentences using polyglot. For each sentence, we extract the hyperlink entities as semantically related entities. Each entity forms a training instance  $(s_i, e_i)$  for the sentence. We restrict the entities to those that appear more than ten times as hyperlinks in the training corpus. They are converted into Wikidata entities, which are shared across languages, using inter-language links obtained from the March 2020 version of the Wikidata dump.
3.2 Hard Negative Entities
The introduction of hard negatives (data that are difficult to distinguish from an anchor point) has been reported to be effective in improving CL models (Gao et al., 2021; Robinson et al., 2021). We introduce a hard negative mining technique that finds negative entities similar to the positive entity but yet unrelated to the sentence.
Specifically, for each positive entity, we collect hard negative entity candidates that satisfy the following two conditions: (1) entities with the same type as the positive entity. Entity types are defined as the entities in the "instance of" relation on Wikipedia, following the work of Xiong et al. (2020). If there are more than one appropriate type, we randomly choose one; (2) entities that do not appear on the same Wikipedia page. Our assumption here is that entities on the same page are topically related to the positive entity and thus are not appropriate for negative data. Finally, we randomly choose one of the candidates to construct hard negative training data. For example, the "Studio Ghibli" entity has the type "animation studio" and one of the hard negative entity candidates is "Walt Disney Animation Studios".
Given datasets with hard negative entities  $\mathcal{D} = \{(s_i, e_i, e_i^-)\}_{i=1}^m$ , the loss function is
$$
l _ {i} ^ {e} = - \log \frac {e ^ {\operatorname* {s i m} \left(\mathbf {h} _ {\mathbf {i}} , \mathbf {W e} _ {\mathbf {i}}\right) / \tau}}{\sum_ {j = 1} ^ {N} \left(e ^ {\operatorname* {s i m} \left(\mathbf {h} _ {\mathbf {i}} , \mathbf {W e} _ {\mathbf {j}}\right) / \tau} + e ^ {\operatorname* {s i m} \left(\mathbf {h} _ {\mathbf {i}} , \mathbf {W e} _ {\mathbf {j}} ^ {-}\right) / \tau}\right)}. \tag {2}
$$
1https://polyglot.readthedocs.io/en/latest/Tokenization.html
In a preliminary experiment, we also tried constructing entity-sentence paired data from entities and the first sentence on their page, and found that the current approach performs better.
3https://en.wikipedia.org/wiki/Help: Interlanguagelinks
3.3 Pretrained Entity Embeddings
We initialize entity embeddings using English entity embeddings pretrained on Wikipedia. These embeddings are trained using the open-source Wikipedia2Vec tool (Yamada et al., 2020) and the January 2019 English Wikipedia dump. The vector dimension is set to 768, which is the same as those of the hidden representations of the base pretrained models, and the other hyperparameters to their default values. The parameters of the entity embedding matrix are updated during the training process.
3.4 Self-supervised Contrastive Learning with Dropout Noise
Self-supervised CL with dropout noise, which inputs a sentence and predicts itself using dropout as noise, is an effective method for learning sentence embeddings in an unsupervised way (Liu et al., 2021; Gao et al., 2021). We combine this method with our entity CL.
Given two embeddings with different dropout masks  $\mathbf{s_i},\mathbf{s_i}^+$  , the training loss of self-supervised CL  $l_{i}^{s}$  is defined by
$$
l _ {i} ^ {s} = - \log \frac {e ^ {\operatorname {s i m} \left(\mathbf {s} _ {\mathbf {i}} , \mathbf {s} _ {\mathbf {i}} ^ {+}\right) / \tau}}{\sum_ {j = 1} ^ {N} e ^ {\operatorname {s i m} \left(\mathbf {s} _ {\mathbf {i}} , \mathbf {s} _ {\mathbf {j}} ^ {+}\right) / \tau}}. \tag {3}
$$
In summary, our total loss is
$$
l _ {i} ^ {\text {e a s e}} = \lambda l _ {i} ^ {e} + l _ {i} ^ {s}, \tag {4}
$$
where  $l^e$  and  $l^s$  are defined in Equations (2) and (3) respectively, and  $\lambda$  denotes a hyperparameter that defines the balance between the entity CL and self-supervised CL with dropout noise. The details on the hyperparameters of the models can be found in Appendix A.
4 Experiment: Monolingual
We first evaluate EASE in monolingual settings. We fine-tune monolingual pre-trained language models using only English Wikipedia data.
We use one million pairs sampled from the English entity-sentence pairs described in Section 3 as training data. In this setting, we train sentence embedding models from pre-trained checkpoints of BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) and take the [CLS] representation as the sentence embedding. We add a linear layer after the
Model	7 STS avg.	8 STC avg.
GloVe embedding (avg.)	61.3†	56.4
BERT (avg.)	52.6	50.9
CT-BERTbase	72.1	61.6
SimCSE-BERTbase	76.3	57.1
EASE-BERTbase	77.0	63.1
RoBERTa (avg.)	53.5	40.9
DeCLUTR-RoBERTabase	70.0	60.0
SimCSE-RoBERTabase	76.6	57.4
EASE-RoBERTabase	76.8	58.6
Table 1: Sentence embedding performance on seven monolingual STS tasks (Spearman's correlation) and eight monolingual STC tasks (clustering accuracy). The highest values among the models with the same pretrained encoder are in bold.  $\dagger$ : results from Reimers and Gurevych (2019); all other results are reproduced or reevaluated by us using published checkpoints. The complete results are available in Appendix G.
output sentence embeddings only during training, as in Gao et al. (2021).
We compare our method with unsupervised sentence embedding methods including average GloVe embeddings (Pennington et al., 2014), average embeddings of vanilla BERT or RoBERTa, and previous state-of-the-art approaches such as SimCSE (Gao et al., 2021), CT (Carlsson et al., 2021), and DeCLUTR (Giorgi et al., 2021).
We evaluate sentence embeddings using two tasks: STS and STC. These tasks are supposed to measure the degree of sentence embeddings capturing fine-grained and broad semantic structures.
Semantic textual similarity STS is a measure of the capability of capturing graded similarity of sentences. We use seven monolingual STS tasks: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014). Following the settings of Reimers and Gurevych (2019), we calculate Spearman's rank correlation coefficient between the cosine similarity of the sentence embeddings and the ground truth similarity scores.
Short text clustering Another important aspect of sentence embeddings is the ability to capture categorical semantic structure, i.e., to map sentences from the same categories close together and those from different categories far apart (Zhang et al., 2021). We also evaluate sentence embeddings using eight benchmark datasets for STC (Zhang et al., 2021) to investigate how well our method can encode high-level categorical structures into
Model	EN-EN	AR-AR	ES-ES	EN-AR	EN-DE	EN-TR	EN-ES	EN-FR	EN-IT	EN-NL	Avg.
mBERTbase(avg.)	54.4	50.9	56.7	18.7	33.9	16.0	21.5	33.0	34.0	35.3	35.4
SimCSE-mBERTbase	78.3	62.5	76.7	26.2	55.6	23.8	37.9	48.1	49.6	50.3	50.9
EASE-mBERTbase	79.3	62.8	79.4	31.6	59.8	26.4	53.7	59.2	59.4	60.7	57.2
XLM-Rbase(avg.)	52.2	25.5	49.6	15.7	21.3	12.1	10.6	16.6	22.9	23.9	25.0
SimCSE-XLM-Rbase	77.9	63.4	80.6	36.3	56.2	28.9	38.9	51.8	52.6	54.2	54.1
EASE-XLM-Rbase	80.6	65.3	80.4	34.2	59.1	37.6	46.5	51.2	56.6	59.5	57.1
Table 2: Spearman's correlation for multilingual semantic textual similarity on extended version of STS 2017 dataset.
Model	ar	ca	cs	de	en	eo	es	fa	fr	ja	ko	pl	pt	ru	sv	tr	Avg.
mBERTbase(avg.)	27.0	27.2	44.3	36.2	37.9	25.6	41.1	35.0	25.9	44.2	31.0	35.0	30.1	23.4	28.9	34.9	33.0
SimCSE-mBERTbase	30.1	26.9	41.3	32.5	37.3	27.2	36.2	36.9	29.0	48.9	33.9	37.6	37.9	27.1	26.9	35.3	34.1
EASE-mBERTbase	31.9	29.6	38.8	38.5	30.2	34.5	37.2	36.7	30.4	49.3	36.2	40.0	41.0	27.0	30.5	44.7	36.0
XLM-Rbase(avg.)	26.0	24.7	28.2	29.4	23.0	23.5	22.1	36.6	23.6	38.8	22.0	24.2	32.8	18.0	33.2	26.0	27.0
SimCSE-XLM-Rbase	24.6	26.3	34.6	28.6	33.4	31.7	32.9	35.9	29.1	41.1	31.1	33.1	30.0	26.0	32.9	37.2	31.8
EASE-XLM-Rbase	25.3	26.7	43.2	37.0	34.9	34.2	37.2	42.4	32.0	46.0	32.8	41.6	33.4	31.3	27.2	41.8	35.4
Table 3: Clustering accuracy for multilingual short text clustering on MewsC-16 dataset.
sentence embeddings. These datasets contain short sentences, ranging from 6 to 28 average words in length, from a variety of domains such as news, biomedical, and social network service (Twitter). We cluster the sentence embeddings using K-Means (MacQueen, 1967) and compute the clustering accuracy using the Hungarian algorithm (Munkres, 1957) averaged over three independent runs.
Table 1 shows the evaluation results for the seven STS and eight STC tasks. Overall, our EASE methods significantly improve the performance of the base models (i.e., BERT and RoBERTa), and on average outperform the previous state-of-the-art methods on all tasks except STC with the RoBERTa backbone. The most significant improvement is observed for EASE-BERT, with an average improvement of  $61.6\%$  to  $63.1\%$  over the previous best result for STC tasks. These results suggest that EASE is able to measure the semantic similarity between sentences, and simultaneously excel at capturing high-level categorical semantic structure.
5 Experiment: Multilingual
To further explore the advantage of entity annotations as cross-lingual alignment supervision, we test EASE in multilingual settings: we fine-tune multilingual pre-trained language models using Wikipedia data in multiple languages.
We sample 50,000 pairs for each language and use them together as training data from the entity-sentence paired data in 18 languages. As our primary baseline model, we use a SimCSE model trained using the same multilingual data as EASE (i.e., sentences in entity-sentence paired data). In this setting, we start fine-tuning from pre-trained checkpoints of mBERT or XLM-R (Conneau et al., 2020) and take mean pooling to obtain sentence embeddings for both training and evaluation on both EASE and SimCSE. We also tested other pooling methods, but mean pooling was the best in this experiment for both models (Appendix B).
5.2 Multilingual STS and STC
We evaluate our method using the extended version of the STS 2017 dataset (Reimers and Gurevych, 2020), which contains annotated sentences for ten language pairs: EN-EN, AR-AR, ES-ES, EN-AR, EN-DE, EN-TR, EN-ES, EN-FR, EN-IT, and EN-NL. We compute Spearman's rank correlation as in Section 4.1. We also conduct experiments on our newly introduced multilingual STC dataset described as follows:
4We chose 18 languages (ar, ca, cs, de, en, eo, es, fa, fr, it, ja, ko, nl, pl, pt, ru, sv, tr) present in both the MewsC-16 dataset (see Section 5.2) and the extended version of STS 2017.
<sup>5</sup>We use the same entity embeddings trained on English Wikipedia as those of the monolingual settings (Section 3.3). Note that entities in all languages are converted to Wikidata entities that are shared across languages using inter-language links as described in Section 3.1.
Model	ar	ca	cs	de	eo	es	fr	it	ja	ko	nl	pl	pt	ru	sv	tr	Avg.
mBERTbase(avg.)	20.6	49.2	32.8	62.8	12.2	57.7	55.6	50.8	38.6	33.1	54.8	40.2	58.5	51.4	45.8	30.1	43.4
SimCSE-mBERTbase	16.4	51.5	30.7	57.0	18.2	54.8	54.5	49.9	39.6	28.1	52.7	37.9	53.6	46.8	45.5	25.0	41.4
EASE-mBERTbase	32.1	66.5	47.7	74.2	26.1	70.1	66.7	65.3	59.2	46.8	69.2	55.4	69.1	64.4	59.4	38.1	56.9
XLM-Rbase(avg.)	10.3	15.3	16.5	49.6	7.5	36.4	30.8	25.6	15.0	19.3	45.2	24.1	42.0	37.4	42.8	17.9	27.2
SimCSE-XLM-Rbase	38.4	57.6	55.7	80.6	46.0	68.9	70.4	66.4	60.0	54.1	73.1	65.3	75.1	71.1	76.7	56.4	63.5
EASE-XLM-Rbase	42.6	65.1	63.8	87.2	56.1	75.9	74.1	70.8	68.2	60.5	77.9	71.9	80.6	76.5	79.2	60.9	69.4
Table 4: Accuracy on Tatoeba dataset averaged over forward and backward directions (en to target language and vice-versa).
Model	Avg.
mBERTbase(avg.)	17.3
SimCSE-mBERTbase	16.8
EASE-mBERTbase	25.4
XLM-Rbase(avg.)	9.4
SimCSE-XLM-Rbase	28.5
EASE-XLM-Rbase	32.1
Table 5: Average accuracy for 94 languages not included in EASE training on Tatoeba.
Model	en (dev)	de	es	fr	it	ja	ru	zh	Avg.
mBERTbase(avg.)	89.5	68.0	68.1	70.6	62.7	61.2	61.5	69.6	65.9
SimCSE-mBERTbase	88.4	62.3	73.2	78.2	64.3	63.7	61.3	75.0	68.3
EASE-mBERTbase	89.0	69.9	69.2	80.1	66.8	62.8	64.4	73.2	69.5
XLM-Rbase(avg.)	90.9	82.7	79.8	72.1	72.5	71.1	69.6	71.4	74.2
SimCSE-XLM-Rbase	90.7	74.9	74.1	81.5	70.3	71.7	70.1	76.6	74.2
EASE-XLM-Rbase	90.6	77.9	75.6	83.9	72.6	72.8	71.1	81.6	76.5
Table 6: Classification accuracy for zero-shot cross-lingual text classification on MLDoc dataset.
MewsC-16 To evaluate the ability of sentence embeddings to encode high-level categorical concepts in a multilingual setting, we constructed MewsC-16 (Multilingual Short Text Clustering Dataset for News in 16 languages) from Wikinews. MewsC-16 contains topic sentences from Wikinews articles in 13 categories and 16 languages. More detailed information is available in Appendix E. We perform clustering and compute the accuracy for each language as in Section 4.1.