SlowGuess/NAACL_2022_OCR huggingface dataset:
Gábor Berend  
Institute of Informatics, University of Szeged  
berendg@inf.u-szeged.hu
In this paper, we advocate for using large pretrained monolingual language models in cross lingual zero-shot word sense disambiguation (WSD) coupled with a contextualized mapping mechanism. We also report rigorous experiments that illustrate the effectiveness of employing sparse contextualized word representations obtained via a dictionary learning procedure. Our experimental results demonstrate that the above modifications yield a significant improvement of nearly 6.5 points of increase in the average F-score (from 62.0 to 68.5) over a collection of 17 typologically diverse set of target languages. We release our source code for replicating our experiments at https://github.com/begab/sparsity_makes_sense.
Word sense disambiguation (WSD) is a longstanding and fundamental problem of Natural Language Processing, known to be affected by the knowledge acquisition bottleneck (Gale et al., 1992). Large pre-trained neural language models are known to effectively mitigate the problems related to the paucity of high quality, large-coverage sense annotated training data for WSD (Loureiro and Jorge, 2019; Loureiro et al., 2021b; inter alia).
Most recently, the knowledge acquisition bottleneck has been identified as an immense problem in the cross-lingual setting as well (Pasini, 2020). A straightforward solution for handling this problem is to apply large multilingual pre-trained language models in a zero-shot setting, however, this approach has a potential limitation owing to the curse of multilinguality (Conneau et al., 2020a), i.e., the inability of such models to handle the large number of languages involved during training such models to an equally good quality.
The research community replied to the limitations of large massively multilingual models by developing language-specific monolingual language
ISO	Huggingface model identifier
bg	DeepPavlov/bert-base-bg-cs-pl-ru-caused (Arkhipov et al., 2019)
ca	PlanTL-GOB-ES/roberta-base-ca (Armengol-Estapé et al., 2021)
da	Maltehb/danish-bert-botxo
de	bert-base-german-caused
es	dccuchile/bert-base-spanish-wwm-caused (Cañete et al., 2020)
et	EMBEDDIA/finest-BERT (Ulčar and Robnik-Sikonja, 2020)
eu	ixa-ehu/berteus-base-caused (Agerri et al., 2020)
fr	camembert-base (Martin et al., 2020)
gl	dvilares/bertinho-gl-base-caused (Vilares et al., 2021)
hr	EMBEDDIA/crosloengual-bert (Ulčar and Robnik-Sikonja, 2020)
hu	SZTAKI-HLT/hubert-base-cc (Nemeskey, 2021)
it	Musixmatch/umberto-commoncrawl-caused-v1
ja	cl-tohoku/bert-base-japanese-whole-word-masking
ko	snunlp/KR-BERT-charl6424
nl	GroNLP/bert-base-dutch-caused (de Vries et al., 2019)
sl	EMBEDDIA/sloberta
zh	bert-base-chinese
Table 1: Monolingual models from the transformers library (Wolf et al., 2020) covering all the (non-English) languages of the XL-WSD dataset (Pasini et al., 2021).
models. Table 1 provides a shortlist of recently published monolingual large pre-trained language models, related to the languages involved in the cross-lingual WSD test suit, XL-WSD (Pasini et al., 2021).
With the prevalence of large monolingual pretrained models, the important research question arises if their language-specific nature can be successfully exploited during zero-shot learning. Our research provides a thorough comparison of the application of large multilingual and monolingual pre-trained language models for zero-shot WSD.
Another crucial aspect that we carefully investigate in this paper is the integration of sparse contextualized word representations into cross-lingual zero-shot WSD. Sparse word representations have a demonstrated ability to align with word senses (Balogh et al., 2020; Yun et al., 2021). While the benefits of employing sparsity has been shown for WSD in English (Berend, 2020a), its viability in the cross-lingual setting has not yet been verified.
<sup>1</sup>With a slight abuse of notation, we also refer to models that support a handful of (related) languages (e.g. Slovenian and Croatian) as language-specific monolingual ones.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2459 - 2471 July 10-15, 2022 ©2022 Association for Computational Linguistics
In order to conduct such an analysis, we propose an algorithm for obtaining cross-lingual sparse contextualized word representations from independently trained monolingual language models.
The analysis and the investigation of the transfer capabilities of large pre-trained language models (such as mBERT or XLM) across languages has spurred significant research interest (Pires et al., 2019; Wu and Dredze, 2019, 2020; K et al., 2020). In contrast to the availability of multilingual neural language models, a series of recent papers have argued for the creation of dedicated neural language models for different languages (see e.g. Table 1). While monolingual neural language models can more accurately model the distinct languages, models that are trained in isolation of other languages cannot directly benefit from downstream application-specific annotated training data available in different languages.
Artetxe et al. (2020) proposed an approach for making monolingual models compatible with each other by first pre-training a masked language model on a source language, then freezing its parameters apart from its embedding layer that get replaced and trained for additional target languages using a standard masked language modeling objective. Note that this approach is complementary and strictly more resource intensive to ours, as it involves the pre-training of a (freezed) transformer model with respect to its embedding layer for a target language. In contrast, our approach can operate on monolingual language models fully pre-trained in total isolation from the source language encoder. Also, our approach learns substantially fewer parameters in the form of an alignment matrix between the hidden representations of the contextualized target and source language spaces.
Conneau et al. (2020b) analyzed the multilingual patterns emerging in large pre-trained language models. The authors found that "language universal representations emerge in pre-trained models without the requirement of any shared vocabulary or domain similarity". That work has demonstrated that monolingual BERT models can be effectively mapped for performing zero-shot cross-lingual named entity recognition and syntactic parsing. Similarly, Wang et al. (2019); Schuster et al. (2019) also illustrated the efficacy of linear transformations for using BERT-derived representations
in cross-lingual dependency parsing.
WSD has been a fundamental and challenging problem in NLP for many decades, dating back to (Weaver, 1949/1955). The utilization of contextualized word representations was first advocated by Peters et al. (2018), later popularized by (Loureiro and Jorge, 2019; Loureiro et al., 2021a). Bevilacqua et al. (2021) offers a survey of the recent approaches.
Most recently, Rezaee et al. (2021) have explored the usage of multilingual language models (XLM) in zero-shot WSD. While the experiments in (Rezaee et al., 2021) cover four related target languages (German, Spanish, French and Italian), our investigation involves a typologically diverse set of 17 target languages (beyond English) from (Pasini et al., 2021). Our work also extends that line of research in important aspects, as we show that the application of monolingual neural language models can vastly improve the performance of cross-lingual zero-shot WSD. Additionally, we also provide a careful evaluation of sparse contextualized word representations in zero-shot WSD.
Berend (2020a) introduced sparse contextualized word representations via the application of dictionary learning, and showed that sense representations that are obtained from the co-occurrence statistics of the sparsity structure of the contextualized word representations and their sense annotations can provide significant improvement in monolingual WSD. Our work relates to that line of research by providing a mapping-based procedure, which enables the usage of such sense representations created in some source language to be applied in other target languages as well. The kind of mapping we employ can be viewed as a generalization of the approach introduced in (Berend, 2020b) with the notable exception that in this work, we obtain sparse word representations for contextualized models as opposed to static word embeddings.
In order to allow for zero-shot transfer between monolingual language models pre-trained in isolation from each other, we need to determine a mapping between their hidden representations. We first introduce our methodology for doing so, then we integrate this to the creation of sparse contextualized word representations.
3.1 Mapping hidden representations
The alignment of word representations between independently constructed semantic spaces can be conveniently and efficiently performed via linear transformations. This has been a standard approach for non-contextualized word embeddings (Mikolov et al., 2013; Xing et al., 2015; Smith et al., 2017), but it has been shown to be useful in the contextualized case as well (Conneau et al., 2020b).
The standard approach is to obtain a collection of pairs of anchor points  $\{\pmb{x}_i,\pmb{y}_i\}_{i = 1}^n$  with  $\pmb{x}_i$  and  $\pmb{y}_i$  denoting the representation of semantically equivalent words in the target and source languages, respectively. The mapping  $W$  is then obtained as
$$
\min  _ {W} \sum_ {i = 1} ^ {n} \| W \boldsymbol {x} _ {i} - \boldsymbol {y} _ {i} \| _ {2} ^ {2}. \tag {1}
$$
As we deal with contextualized models, we can obtain various representations for a word even in the same context, by considering the hidden representations from different layers of the neural language models employed. Additionally, as constraining the mapping matrix to be an isometric one have proven to be a useful requirement, we define our learning task to be of the form
$$
\min  _ {W s. t. W ^ {\intercal} W = I} \sum_ {i = 1} ^ {n} \| W \boldsymbol {x} _ {i} ^ {(l _ {t})} - \boldsymbol {y} _ {i} ^ {(l _ {s})} \| _ {2} ^ {2}, \tag {2}
$$
with  $I$  denoting the identity matrix,  $\pmb{x}_i^{(l_t)}$  and  $\pmb{y}_i^{(l_s)}$  denoting the hidden representations obtained from the  $l_{t}^{\mathrm{th}}$  and  $l_{s}^{\mathrm{th}}$  layers of the target and source language neural language models, respectively.
Finding the optimal isometric  $W$  can be viewed as an instance of the orthogonal Procrustes problem (Schönemann, 1966) which can be solved by  $W_{\perp} = UV$ , with  $U$  and  $V$  originating from the singular value decomposition of the matrix product  $Y^{\top}X$ , where  $X$  and  $Y$  include the stacked target and source language contextual representations of pairs of semantically equivalent words.
As words of the input sequences to the neural language models can be split into multiple subtokens, we followed the common practice of obtaining word-level neural representations by performing mean pooling of the subword representations. Throughout our experiments, we also relied on the RCSLS criterion (Joulin et al., 2018), which offers a retrieval-based alternative of obtaining a mapping from the target to the source language representations.
3.2 Cross-lingual sparse contextualized word representations
Our approach extends the information theoretic algorithm introduced in (Berend, 2020a) for its application in the cross-lingual zero-shot WSD setting. In order to obtain sparse contextualized representations for the source language, we first populate  $Y \in \mathbb{R}^{d \times N}$  with  $d$ -dimensional contextualized representations of words determined for texts in the source language, and minimize the objective
$$
\min  _ {D \in \mathcal {C}, \boldsymbol {\alpha} _ {i} \in \mathbb {R} _ {\geq 0} ^ {k}} \sum_ {i = 1} ^ {N} \frac {1}{2} \| \boldsymbol {y} _ {i} - D \boldsymbol {\alpha} _ {i} \| _ {2} ^ {2} + \lambda \| \boldsymbol {\alpha} _ {i} \| _ {1}, \tag {3}
$$
where  $\mathcal{C}$  denotes the convex set of  $d\times k$  matrices with column norm at most 1,  $\lambda$  is a regularization coefficient and the sparse coefficients in  $\alpha$  are required to be non-negative. We used the SPAMS library (Mairal et al., 2009) for calculating  $D$  and  $\alpha$ .
Having obtained  $D$  for the source language, we determine a sparse contextualized word representation for a target language word with dense contextualized representation  $\pmb{x}_i$  as
$$
\min  _ {\boldsymbol {\alpha} _ {i} \in \mathbb {R} _ {\geq 0} ^ {k}} \frac {1}{2} \| W \boldsymbol {x} _ {i} - D \boldsymbol {\alpha} _ {i} \| _ {2} ^ {2} + \lambda \| \boldsymbol {\alpha} _ {i} \| _ {1}, \tag {4}
$$
where  $W$  is the alignment transformation as described earlier in Section 3.1. Eq. (4) reveals that the cross-lingual applicability of the sparse codes are assured by the mapping transformation  $W$  and the fact that the sparse target language representations are also using the same  $D$  that was determined for the source language, which also ensures the efficient calculation of sparse representations during inference time.
Apart from these crucial extensions we made for providing the use of contextualized sparse representations in the cross-lingual setting, the way we utilized them for the determination of sense representation and inference is identical to (Berend, 2020a). That is, for all sense-annotated words in the training corpus, we calculated a weighted cooccurrence statistics between a word pertaining to a specific semantic category and having non-zero coordinates along a specific dimension in their sparse contextualized word representations. These statistics are then transformed into pointwise mutual information (PMI) scores, resulting in a sense representation for all the senses in the training sense inventory.
Sense representations obtained that way measure the strength of the relation of the senses to the
different (sparse) coordinates. Inference for a word with sparse representation  $\alpha$  is simply taken as  $\arg \max_{s} \Phi \alpha^{\mathsf{T}}$ , where  $\Phi$  is the previously defined matrix of PMI values and  $s$  corresponds to the sense at which position the above matrix-vector products takes its largest value.
All the neural language models that we relied on during our experiments were obtained from the transformers library (Wolf et al., 2020). We used four NVIDIA Titan 2080 GPUs for our experiments.
As the multilingual language model, we used the 24-layer transformer architecture, XLM-RoBERTa (XLM-R for short) (Conneau et al., 2020a). We chose the cased BERT (Devlin et al., 2019) large model as the monolingual model for encoding English text. As for the rest of the monolingual language models involved in our experiments, we relied on the models listed in Table 1. These monolingual models have the same size as the BERT-base model, i.e., they consist of 12 transformer blocks and employ hidden representations of 768 dimensions.
For evaluation purposes, we used the extra-large cross-lingual evaluation benchmark XL-WSD, recently proposed in (Pasini et al., 2021). The database contains a high-quality sense annotated corpus for English as the concatenation of the SemCor dataset (Miller et al., 1994) and the sense definitions and example sentences from WordNet (Fellbaum, 1998). XL-WSD uses the unified cross-lingual sense inventory of BabelNet (Navigli and Ponzetto, 2012).
The dataset contains 17 additional typologically diverse languages besides English (that we listed in Table 1). The authors also released machine translated silver standard sense annotated training corpora for all the languages, which makes the language-specific fine-tuning of monolingual models possible, however, as shown in (Pasini et al., 2021), that approach resulted in inferior results compared to the application of multilingual models in the zero-shot setting.
Throughout the application of sparse contextualized representations, we employ the same set of hyperparameters that were used in (Berend, 2020a), i.e., we set the number of the regularization coefficient to  $\lambda = 0.05$  and the number of (sparse) coordinates to  $k = 3000$ . There made one optional
change, i.e., we decided whether to use the normalization of PMI values (Bouma, 2009) during the calculation of the sense representation matrix  $\Phi$  on a per language basis based on development set performances. An ablation study related to the (optional) normalization of PMI scores is reported in Table 5, Appendix B.
When we do not employ the sparsification of the contextualized word representations for determining the sense representations, we follow the approach introduced in (Loureiro and Jorge, 2019). That is, we take the centroid of word vectors belonging to a particular sense as the representation of that sense, and perform a nearest neighbor search during inference.
4.1 Alignment of contextualized representations
As the different layers of neural language models have been shown to provide different levels of utility towards different tasks, we experimented with mappings between different combinations of layers from the target and source language neural language models. Since the last few layers of the neural models are generally agreed to be the most useful for semantics-related tasks (Peters et al., 2018; Tenney et al., 2019; Reif et al., 2019), we decided to learn mappings between the hidden representations of any of the last four layers of the target and source language encoders.
We used BERT as the language specific encoder for the source language texts in English, but we also investigated the application of XLM-R, so that we can see the effects of replacing it by an encoder especially tailored for English. As for the target languages, we used the respective models for each language as listed in Table 1. Similar to the source language, we also investigated the case when target languages were encoded by the multilingual model.
In what follows, we label the different experimental settings according to the followings:
- multi  $\rightarrow$  multi means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language also obtained by the multilingual (XLM-R) encoder,
- multi  $\rightarrow$  mono, means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language obtained by the monolingual (English BERT) encoder,
- mono  $\rightarrow$  multi, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the multilingual (XLM-R) encoder,
- mono  $\rightarrow$  mono, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the monolingual (English BERT) encoder.
In order to obtain the cross-representational mappings, we accessed the Tatoeba corpus (Tiedemann, 2012) through the datasets library (Lhoest et al., 2021). The Tatoeba corpus contains translated sentence pairs for several hundreds of languages which we used for obtaining the pivot word mention pairs together with their contexts.
In addition to the Tatoeba corpus, we used the word2word library (Choe et al., 2020) containing word translation pairs between more than 3,500 language pairs. By denoting  $(S_{s_i}, S_{t_i})$  the  $i^{\text{th}}$  translated sentence pair from the Tatoeba corpus, we treated those  $(w_s \in S_{s_i}, w_t \in S_{t_i})$  word occurrences as being semantically equivalent, for which the  $w_t \in TranslationOf(w_s)$  and the  $w_s \in TranslationOf(w_t)$  relations simultaneously held according to the translation list provided by word2word.
As an example, given the German-English translation pair from Tatoeba, { 'de: ' 'Es steht ein Glas auf dem Tisch', 'en': 'There is a glass on the table }. underlined pairs of words with the same color would be treated as contextualized translation pairs of each other.
One benefit of our approach for determining contextual alignment of word pairs is that it does not require word level alignment of the parallel sentences, hence it suits such lower resource scenarios better, when only parallel sentences (without word level alignments) and a list of word translation pairs are provided. Naturally, different contextual alignment approaches could be integrated into our approach at this point, and this is something that we regard as potential future extension of our work.
We evaluated the quality of the mapping learned between the target and the source language representations by defining a contextualized translation retrieval task and evaluating it on its accuracy@1
Language	#sentences	Train	Test
bg	Bulgarian	17,797	14,212	3,554
ca	Catalan	1,663	3,912	979
da	Danish	30,089	20,000	5,000
de	German	299,769	20,000	5,000
es	Spanish	207,517	20,000	5,000
et	Estonian	2,428	2,365	592
eu	Basque	2,062	3,956	990
fr	French	262,078	20,000	5,000
gl	Galician	1,013	2,356	590
hr	Croatian	2,420	1,946	487
hu	Hungarian	107,133	20,000	5,000
it	Italian	482,948	20,000	5,000
ja	Japanese	204,893	20,000	5,000
ko	Korean	3,434	5,632	1,408
nl	Dutch	72,391	20,000	5,000
sl	Slovenian	3,210	1,285	322
zh	Chinese	46,114	20,000	5,000
Table 2: The number of sentence pairs included in the Tatoeba corpus between English and a target language and the number of contextualized translation pairs extracted for training and testing the mappings.
metric, i.e., for what fraction of the contextualized translation pairs – not seen during the determination of the mapping between the two representation spaces – are we able to rank the original translated context as the highest.
In the multi  $\rightarrow$  multi case, i.e., when both the target and source languages are encoded by the same multilingual model (XLM-R), it also makes sense to use the identity matrix as the mapping operator for mapping the target language contextual text representations to the semantic space of the source language (as long as the target and source language texts are obtained from the same layer of the multilingual encoder). We also evaluated the quality of this approach in our experiments that we refer to as the identity approach.
We list the statistics of the Tatoeba corpus and the size of the training and test contextualized translation pairs in Table 2. Our results on the top-1 contextualized translation retrieval accuracies along the different languages and combination of target and source encoder usage are reported in Figure 1. The quality of the combination which uses monolingual encoders for both the target and source languages (mono  $\rightarrow$  mono) performed the best.
(a) Mapping accuracies per languages.
(b) Mapping accuracies aggregated over languages.
Figure 1: The results of translation retrieval over the test sets of the different languages and different combinations of transformers used for the (English) source and the target languages.
Layer	BERT	XLM-R
Dense	Sparse	Dense	Sparse
21	74.39	77.45	69.29	74.51
22	74.87	77.60	67.87	74.50
23	74.45	77.86	67.48	74.26
24	73.58	76.21	64.50	70.06
Table 3: English results expressed in F-score.
4.2 Monolingual evaluation
We first conducted evaluations in the monolingual setting, i.e., we used the sense annotated training data to train and evaluate WSD models in English. The results of these experiments - depending on the encoder architecture used (BERT/XLM-R), the layer of the encoder utilized (\{21,...,24\}), and whether the sparsification of the contextualized representations took place (Dense/Sparse) - are included in Table 3.
Unsurprisingly, the application of the language-specific BERT model achieved better scores compared to that of XLM-R. An interesting observation though, is that the drop in performance is much more subtle for those cases when the contextualized representations are enhanced via sparsification, i.e., the typical loss in performance across the layers is only 3 points (apart from the final layer), opposed to the typical loss of 4-7 points in the dense case.
4.3 Cross-lingual zero-shot evaluation