Venue: NAACL 2022

Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations

Gabor Berend

Institute of Informatics, University of Szeged berendg@inf.u-szeged.hu

###### Abstract

In this paper, we advocate for using large pre-trained monolingual language models in cross lingual zero-shot word sense disambiguation (WSD) coupled with a contextualized mapping mechanism. We also report rigorous experiments that illustrate the effectiveness of employing sparse contextualized word representations obtained via a dictionary learning procedure. Our experimental results demonstrate that the above modifications yield a significant improvement of nearly 6.5 points of increase in the average F-score (from 62.0 to 68.5) over a collection of 17 typologically diverse set of target languages. We release our source code for replicating our experiments at [https://github.com/begab/sparsity_makes_sense](https://github.com/begab/sparsity_makes_sense).

## 1 Introduction

Word sense disambiguation (WSD) is a longstanding and fundamental problem of Natural Language Processing, known to be affected by the _knowledge acquisition bottleneck_[3]. Large pre-trained neural language models are known to effectively mitigate the problems related to the paucity of high quality, large-coverage sense annotated training data for WSD [9, 4]; _inter alia_.

Most recently, the knowledge acquisition bottleneck has been identified as an immense problem in the cross-lingual setting as well [26]. A straightforward solution for handling this problem is to apply large multilingual pre-trained language models in a zero-shot setting, however, this approach has a potential limitation owing to the _curse of multilingquality_[10], i.e., the inability of such models to handle the large number of languages involved during training such models to an equally good quality.

The research community replied to the limitations of large massively multilingual models by developing language-specific monolingual language models.1 Table 1 provides a shortlist of recently published monolingual large pre-trained language models, related to the languages involved in the cross-lingual WSD test suit, XL-WSD [26].

Footnote 1: With a slight abuse of notation, we also refer to models that support a handful of (related) languages (e.g. Slovenian and Croatian) as language-specific monolingual ones.

With the prevalence of large monolingual pretrained models, the important research question arises if their language-specific nature can be successfully exploited during zero-shot learning. Our research provides a thorough comparison of the application of large multilingual and monolingual pre-trained language models for zero-shot WSD.

Another crucial aspect that we carefully investigate in this paper is the integration of sparse contextualized word representations into cross-lingual zero-shot WSD. Sparse word representations have a demonstrated ability to align with word senses [1, 27]. While the benefits of employing sparsity has been shown for WSD in English [3], its viability in the cross-lingual setting has not yet been verified.

\begin{table}
\begin{tabular}{l l}  & **Go** \\ \hline
**bg** & DeepPavlov/bert-base-by-ca-p-i-ru-cased[1] \\
**ca** & PlaniT-COB-ES/roberta-baso-ca[1] \\
**da** & Raitteh/dnais-bert-botto \\
**de** & Bert-base-spann-cased \\
**es** & docoulic/bert-base-spanni-wom-cased[1] \\
**et** & HORING/filt-filter-BLER[1] \\
**en** & kxus-db/news-based[1] \\
**fa** & cannernet-base[1] \\
**ef** & divinares/herding-bi-bias-cased[1] \\
**fr** & EMBOSICM/ecisionguasi-bert[1] \\
**lr** & EMBOSICM/recisionguasi-bert[1] \\
**s** & SMITAN+IMU-ibert-baat-sco[1] \\
**rl** & Ruaitteh/ambort-common-cancel-vi \\
**jl** & ci-tcdn/bert-base-Japanese-whole-word-making \\
**j** & mnnip/K-BERT-classifi242 \\
**dl** & GrollNet/bert-base-match-cased[1] \\
**r** & Doshi/slaNTA \\
**hr** & bert-base-chinese \\ \end{tabular}
\end{table}
Table 1: Monolingual models from the transformers library [27] covering all the (non-English) languages of the XL-WSD dataset [26].

 

In order to conduct such an analysis, we propose an algorithm for obtaining cross-lingual sparse contextualized word representations from independently trained monolingual language models.

## 2 Related work

The analysis and the investigation of the transfer capabilities of large pre-trained language models (such as mBERT or XLM) across languages has spurred significant research interest Pires et al. (2019); Wu and Dredze (2019); Xu et al. (2020). In contrast to the availability of multilingual neural language models, a series of recent papers have argued for the creation of dedicated neural language models for different languages (see e.g. Table 1). While monolingual neural language models can more accurately model the distinct languages, models that are trained in isolation of other languages cannot directly benefit from downstream application-specific annotated training data available in different languages.

Artetxe et al. (2020) proposed an approach for making monolingual models compatible with each other by first pre-training a masked language model on a source language, then freezing its parameters apart from its embedding layer that get replaced and trained for additional target languages using a standard masked language modeling objective. Note that this approach is complementary and strictly more resource intensive to ours, as it involves the pre-training of a (freezed) transformer model with respect its embedding layer for a target language. In contrast, our approach can operate on monolingual language models fully pre-trained in total isolation from the source language encoder. Also, our approach learns substantially fewer parameters in the form of an alignment matrix between the hidden representations of the contextualized target and source language spaces.

Conneau et al. (2020) analyzed the multilingual patterns emerging in large pre-trained language models. The authors found that "_language universal representations emerge in pre-trained models without the requirement of any shared vocabulary or domain similarity_". That work have demonstrated that monolingual BERT models can be effectively mapped for performing zero-shot cross-lingual named entity recognition and syntactic parsing. Similarly, Wang et al. (2019); Schuster et al. (2019) also illustrated the efficacy of linear transformations for using BERT-derived representations in cross-lingual dependency parsing.

WSD has been a fundamental and challenging problem in NLP for many decades, dating back to Weaver (1949/1955). The utilization of contextualized word representations was first advocated by Peters et al. (2018), later popularized by Loureiro and Jorge (2019); Loureiro et al. (2021). Bevilacqua et al. (2021) offers a survey of the recent approaches.

Most recently, Rezaee et al. (2021) have explored the usage of multilingual language models (XLM) in zero-shot WSD. While the experiments in Rezaee et al. (2021) cover four related target languages German, Spanish, French and Italian), our investigation involves a typologically diverse set of 17 target languages (beyond English) from Pasini et al. (2021). Our work also extends that line of research in important aspects, as we show that the application of monolingual neural language models can vastly improve the performance of cross-lingual zero-shot WSD. Additionally, we also provide a careful evaluation of sparse contextualized word representations in zero-shot WSD.

Berend (2020) introduced sparse contextualized word representations via the application of dictionary learning, and showed that sense representations that are obtained from the co-occurrence statistics of the sparsity structure of the contextualized word representations and their sense annotations can provide significant improvement in monolingual WSD. Our work relates to that line of research by providing a mapping-based procedure, which enables the usage of such sense representations created in some source language to be applied in other target languages as well. The kind of mapping we employ can be viewed as a generalization of the approach introduced in Berend (2020) with the notable exception that in this work, we obtain sparse word representations for contextualized models as opposed to static word embeddings.

## 3 Methodology

In order to allow for zero-shot transfer between monolingual language models pre-trained in isolation from each other, we need to determine a mapping between their hidden representations. We first introduce our methodology for doing so, then we integrate this to the creation of sparse contextualized word representations 



### Mapping hidden representations

The alignment of word representations between independently constructed semantic spaces can be conveniently and efficiently performed via linear transformations. This has been a standard approach for non-contextualized word embeddings Mikolov et al. (2013); Xing et al. (2015); Smith et al. (2017), but it has been shown to be useful in the contextualized case as well Conneau et al. (2020b).

The standard approach is to obtain a collection of pairs of anchor points \(\{\mathbf{x}_{i},\mathbf{y}_{i}\}^{n}_{i=1}\) with \(\mathbf{x}_{i}\) and \(\mathbf{y}_{i}\) denoting the representation of semantically equivalent words in the target and source languages, respectively. The mapping \(W\) is then obtained as

\[\min_{W}\sum_{i=1}^{n}\lVert W\mathbf{x}_{i}-\mathbf{y}_{i}\rVert_{2}^{2}. \tag{1}\]

As we deal with contextualized models, we can obtain various representations for a word even in the same context, by considering the hidden representations from different layers of the neural language models employed. Additionally, as constraining the mapping matrix to be an isometric one have proven to be a useful requirement, we define our learning task to be of the form

\[\min_{W~{}s.t.~{}W^{\intercal}W=I}\sum_{i=1}^{n}\lVert W\mathbf{x}_{i}^{(l_{i})}- \mathbf{y}_{i}^{(l_{s})}\rVert_{2}^{2}, \tag{2}\]

with \(I\) denoting the identity matrix, \(\mathbf{x}_{i}^{(l_{i})}\) and \(\mathbf{y}_{i}^{(l_{s})}\) denoting the hidden representations obtained from the \(l_{t}\textsuperscript{th}\) and \(l_{s}\textsuperscript{th}\) layers of the target and source language neural language models, respectively.

Finding the optimal isometric \(W\) can be viewed as an instance of the orthogonal Procrustes problem Schonemann (1966) which can be solved by \(W_{\perp}=UV\), with \(U\) and \(V\) originating from the singular value decomposition of the matrix product \(Y^{\intercal}X\), where \(X\) and \(Y\) include the stacked target and source language contextual representations of pairs of semantically equivalent words.

As words of the input sequences to the neural language models can be split into multiple subtokens, we followed the common practice of obtaining word-level neural representations by performing mean pooling of the subword representations. Throughout our experiments, we also relied on the RCSLS criterion Joulin et al. (2018), which offers a retrieval-based alternative of obtaining a mapping from the target to the source language representations.

### Cross-lingual sparse contextualized word representations

Our approach extends the information theoretic algorithm introduced in Berend (2020a) for its application in the cross-lingual zero-shot WSD setting. In order to obtain sparse contextualized representations for the source language, we first populate \(Y\in\mathbb{R}^{d\times N}\) with \(d\)-dimensional contextualized representations of words determined for texts in the source language, and minimize the objective

\[\min_{D\in\mathcal{C},\mathbf{\alpha}_{i}\in\mathbb{R}^{k}_{\geq 0}}\sum_{ i=1}^{N}\frac{1}{2}\|\mathbf{y_{i}}-D\mathbf{\alpha_{i}}\|_{2}^{2}+\lambda\|\mathbf{ \alpha_{i}}\|_{1}, \tag{3}\]

where \(\mathcal{C}\) denotes the convex set of \(d\times k\) matrices with column norm at most 1, \(\lambda\) is a regularization coefficient and the sparse coefficients in \(\alpha\) are required to be non-negative. We used the SPAMS library Mairal et al. (2009) for calculating \(D\) and \(\alpha\).

Having obtained \(D\) for the source language, we determine a sparse contextualized word representation for a target language word with dense contextualized representation \(\mathbf{x_{i}}\) as

\[\min_{\mathbf{\alpha_{i}}\in\mathbb{R}^{k}_{\geq 0}}\frac{1}{2}\lVert W\mathbf{ x_{i}}-D\mathbf{\alpha_{i}}\rVert_{2}^{2}+\lambda\|\mathbf{\alpha_{i}}\|_{1}, \tag{4}\]

where \(W\) is the alignment transformation as described earlier in Section 3.1. Eq. (4) reveals that the cross-lingual applicability of the sparse codes are assured by the mapping transformation \(W\) and the fact that the sparse target language representations are also using the same \(D\) that was determined for the source language, which also ensures the efficient calculation of sparse representations during inference time.

Apart from these crucial extensions we made for providing the use of contextualized sparse representations in the cross-lingual setting, the way we utilized them for the determination of sense representation and inference is identical to Berend (2020a). That is, for all sense-annotated words in the training corpus, we calculated a weighted co-occurrence statistics between a word pertaining to a specific semantic category and having non-zero coordinates along a specific dimension in their sparse contextualied word representations. These statistics are then transformed into pointwise mutual information (PMI) scores, resulting in a sense representation for all the senses in the training sense inventory.

Sense representations obtained that way measure the strength of the relation of the senses to the

different (sparse) coordinates. Inference for a word with sparse representation \(\mathbf{\alpha}\) is simply taken as \(\arg\max_{s}\Phi\mathbf{\alpha}^{\intercal}\), where \(\Phi\) is the previously defined matrix of PMI values and \(s\) corresponds to the sense at which position the above matrix-vector products takes its largest value.

## 4 Experimental results

All the neural language models that we relied on during our experiments were obtained from the transformers library Wolf et al. (2020). We used four NVIDIA Titan 2080 GPUs for our experiments.

As the multilingual language model, we used the 24-layer transformer architecture, XLM-RoBERTa (XLM-R for short) Conneau et al. (2020). We chose the cased BERT Devlin et al. (2019) large model as the monolingual model for encoding English text. As for the rest of the monolingual language models involved in our experiments, we relied on the models listed in Table 1. These monolingual models have the same size as the BERT-base model, i.e., they consist of 12 transformer blocks and employ hidden representations of 768 dimensions.

For evaluation purposes, we used the extra-large cross-lingual evaluation benchmark XL-WSD, recently proposed in Pasini et al. (2021). The database contains a high-quality sense annotated corpus for English as the concatenation of the SemCor dataset Miller et al. (1994) and the sense definitions and example sentences from WordNet Fellbaum (1998). XL-WSD uses the unified cross-lingual sense inventory of BabelNet Navigli and Ponzetto (2012).

The dataset contains 17 additional typologically diverse languages besides English (that we listed in Table 1). The authors also released machine translated silver standard sense annotated training corpora for all the languages, which makes the language-specific fine-tuning of monolingual models possible, however, as shown in Pasini et al. (2021), that approach resulted in inferior results compared to the application of multilingual models in the zero-shot setting.

Throughout the application of sparse contextualized representations, we employ the same set of hyperparameters that were used in Berend (2020), i.e., we set the number of the regularization coefficient to \(\lambda=0.05\) and the number of (sparse) coordinates to \(k=3000\). There made one optional change, i.e., we decided whether to use the normalization of PMI values Bouma (2009) during the calculation of the sense representation matrix \(\Phi\) on a per language basis based on development set performances. An ablation study related to the (optional) normalization of PMI scores is reported in Table 5, Appendix B.

When we do not employ the sparsification of the contextualized word representations for determining the sense representations, we follow the approach introduced in Loureiro and Jorge (2019). That is, we take the centroid of word vectors belonging to a particular sense as the representation of that sense, and perform a nearest neighbor search during inference.

### Alignment of contextualized representations

As the different layers of neural language models have been shown to provide different levels of utility towards different tasks, we experimented with mappings between different combinations of layers from the target and source language neural language models. Since the last few layers of the neural models are generally agreed to be the most useful for semantics-related tasks Peters et al. (2018); Tenney et al. (2019); Reif et al. (2019), we decided to learn mappings between the hidden representations of any of the last four layers of the target and source language encoders.

We used BERT as the language specific encoder for the source language texts in English, but we also investigated the application of XLM-R, so that we can see the effects of replacing it by an encoder especially tailored for English. As for the target languages, we used the respective models for each language as listed in Table 1. Similar to the source language, we also investigated the case when target languages were encoded by the multilingual model.

In what follows, we label the different experimental settings according to the followings:

* multi\(\rightarrow\)multi means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language also obtained by the multilingual (XLM-R) encoder,
* multi\(\rightarrow\)mono, means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language obtained by the monolingual (English BERT) encoder 

* mono\(\rightarrow\)multi, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the multilingual (XLM-R) encoder,
* mono\(\rightarrow\)mono, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the monolingual (English BERT) encoder.

In order to obtain the cross-representational mappings, we accessed the Tatoeba corpus Tiedemann (2012) through the datasets library Lhoest et al. (2021). The Tatoeba corpus contains translated sentence pairs for several hundreds of languages which we used for obtaining the pivot word mention pairs together with their contexts.

In addition to the Tatoeba corpus, we used the word2word library Choe et al. (2020) containing word translation pairs between more than 3,500 language pairs. By denoting \((S_{s_{i}},S_{t_{i}})\) the \(i^{\text{th}}\) translated sentence pair from the Tatoeba corpus, we treated those \((w_{s}\in S_{s_{i}},w_{t}\in S_{t_{i}})\) word occurrences as being semantically equivalent, for which the \(w_{t}\in TranslationOf(w_{s})\) and the \(w_{s}\in TranslationOf(w_{t})\) relations simultaneously held according to the translation list provided by word2word.

As an example, given the German-English translation pair from Tatoeba, _'de:' Es steht ein Glas auf dem Tesch.', 'en': 'There is a glass on the table.)_, underlined pairs of words with the same color would be treated as contextualized translation pairs of each other.

One benefit of our approach for determining contextual alignment of word pairs is that it does not require word level alignment of the parallel sentences, hence it suits such lower resource scenarios better, when only parallel sentences (without word level alignments) and a list of word translation pairs are provided. Naturally, different contextual alignment approaches could be integrated into our approach at this point, and this is something that we regard as potential future extension of our work.

We evaluated the quality of the mapping learned between the target and the source language representations by defining a contextualized translation retrieval task and evaluating it on its accuracy@1 metric, i.e., for what fraction of the contextualized translation pairs - not seen during the determination of the mapping between the two representation spaces - are we able to rank the original translated context as the highest.

In the multi\(\rightarrow\)multi case, i.e., when both the target and source languages are encoded by the same multilingual model (XLM-R), it also makes sense to use the identity matrix as the mapping operator for mapping the target language contextual text representations to the semantic space of the source language (as long as the target and source language texts are obtained from the same layer of the multilingual encoder). We also evaluated the quality of this approach in our experiments that we refer to as the identity approach.

We list the statistics of the Tatoeba corpus and the size of the training and test contextualized translation pairs in Table 2. Our results on the top-1 contextualized translation retrieval accuracies along the different languages and combination of target and source encoder usage are reported in Figure 1. The quality of the combination which uses monolingual encoders for both the target and source languages (mono\(\rightarrow\)mono) performed the best.

\begin{table}
\begin{tabular}{l l r c c} \hline \hline Language & \multicolumn{2}{c}{\#sentences} & Train & Test \\ \hline bg & Bulgarian & 17,797 & 14,212 & 3,554 \\ ca & Catalan & 1,663 & 3,912 & 979 \\ da & Danish & 30,089 & 20,000 & 5,000 \\ de & German & 299,769 & 20,000 & 5,000 \\ es & Spanish & 207,517 & 20,000 & 5,000 \\ et & Estonian & 2,428 & 2,365 & 592 \\ eu & Basque & 2,062 & 3,956 & 990 \\ fr & French & 262,078 & 20,000 & 5,000 \\ gl & Galician & 1,013 & 2,356 & 590 \\ hr & Croatian & 2,420 & 1,946 & 487 \\ hu & Hungarian & 107,133 & 20,000 & 5,000 \\ it & Italian & 482,948 & 20,000 & 5,000 \\ ja & Japanese & 204,893 & 20,000 & 5,000 \\ ko & Korean & 3,434 & 5,632 & 1,408 \\ nl & Dutch & 72,391 & 20,000 & 5,000 \\ sl & Slovenian & 3,210 & 1,285 & 322 \\ zh & Chinese & 46,114 & 20,000 & 5,000 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The number of sentence pairs included in the Tateoba corpus between English and a target language and the number of contextualized translation pairs extracted for training and testing the mappings.

 



### Monolingual evaluation

We first conducted evaluations in the monolingual setting, i.e., we used the sense annotated training data to train and evaluate WSD models in English. The results of these experiments - depending on the encoder architecture used (BERT/XLM-R), the layer of the encoder utilized ({21,...,24}), and whether the sparsification of the contextualized representations took place (Dense/Sparse) - are included in Table 3.

Unsurprisingly, the application of the language-specific BERT model achieved better scores compared to that of XLM-R. An interesting observation though, is that the drop in performance is much more subtle for those cases when the contextualized representations are enhanced via sparsification, i.e., the typical loss in performance across the layers is only 3 points (apart from the final layer), opposed to the typical loss of 4-7 points in the dense case.

### Cross-lingual zero-shot evaluation

Table 4 includes the zero-shot cross-lingual WSD results for a collection of baseline approaches (Table 3(a)) from Pasini et al. (2021), followed by our models not utilizing the sparsification of the contextualized embeddings (Table 3(b)) and the ones that additionally benefit from sparsification as well (Table 3(c)). It is useful to note that the mono\(\rightarrow*\) approaches are strictly more resource efficient during inference as they are based on 12-layer encoders instead of the 24 layers of the multilingual XLM-R model.

At this point, we separate the multi\(\rightarrow\)multi results into two, i.e., 1) those obtained when relying on the hidden representations from the same layer of XLM-R without mapping (or equivalently, with the identity mapping from the target to source representations); and 2) those obtained when the target and source language contextual representations could originate from different layers of the XLM-R encoder, and a non-identity (either isometric or RCSLS) mapping was employed. We keep referring to the latter as multi\(\rightarrow\)multi, and denote the former type of experiments as multi (without the \(\rightarrow\)multi suffix as there were no real mappings performed in these cases). Inspecting the first two rows of Table 3(b) and Table 3(c) reveals that enhancing the multilingual encoder towards the treatment of a particular pair of languages by providing it a language pair specific mapping has a larger positive effect when using dense vectors. In fact, it increased the micro-averaged F-score over the 17 languages by 1.72 and 0.11 points for the dense and the sparse cases, respectively.

Overall, the micro-averaged F-score of our final approach managed to improve nearly 6.5 points (cf. the first row of Table 3(b) and the last row in Table 3(c)). A 5 point average improvement is due

Figure 1: The results of translation retrieval over the test sets of the different languages and different combinations of transformers used for the (English) source and the target languages.

\begin{table}
\begin{tabular}{r c c c c} \hline \hline  & \multicolumn{2}{c}{BERT} & \multicolumn{2}{c}{XLM-R} \\ Layer & Dense & Sparse & Dense & Sparse \\ \cline{2-5}  & 74.39 & 77.45 & 69.29 & 74.51 \\
22 & 74.87 & 77.60 & 67.87 & 74.50 \\
23 & 74.45 & 77.86 & 67.48 & 74.26 \\
24 & 73.58 & 76.21 & 64.50 & 70.06 \\ \hline \hline \end{tabular}
\end{table}
Table 3: English results expressed in F-score.



to the replacement of the XLM-R encoder for both the source language during training and target languages for inference (cf. the first and last row of Table 3(b)) and an additional 1.5 points of improvement was an effect of our sparsification in the cross-lingual setting. The inspection of the third and fourth rows in both Table 3(b) and Table 4(c) reveals that using a monolingual encoder during inference helps more compared to the application of a monolingual encoder for encoding the source language during training.

We conducted the McNemar test between our system outputs when a non-identity mapping was used between a pair of languages. Our investigation revealed that all such pairs of system outputs from Table 3(b) and Table 4(c) differ significantly from each other with \(p<0.0007\), with only four exceptions, i.e, 1) multi multi and multi mono from Table 3(b); 2) multi and multi Mono from Table 3(c); 3) mono multi from Table 3(d) and mono\(\rightarrow\)mono from Table 3(b); 4) multi\(\rightarrow\)mono from Table 3(c) and mono\(\rightarrow\)multi from Table 3(b).

Figure 2 summarizes the results of all the possible runs conducted. When using the multilingual encoder for both the target and source languages without a mapping step between the two (multi), we ran 4 different experiments per each language based on the hidden representations obtained from one of the last 4 layers of the multilingual encoder. For the remaining experiments relying on the dense and sparse representations, there were 32 and 64 experiments for each language, respectively. The 32 experiments were a result of choosing any of the 16 possible combination of the final four layers on the target and source language encoder, coupled with the type of mapping utilized (isometric/RCLS). For the experiments involving the sparse representations, there was an extra parameter, whether the normalization of the PMI scores for obtaining the sense representations to be performed, resulting

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c} \hline  & bg & ca & da & de & es & et eu & ft & gl lr & hv & hi & it ja & ko & nl & sl & zh & Avg. \\ \hline multi & 70.69 & 51.52 & **81.41** & 83.53 & 76.45 & 67.78 & 47.85 & 83.62 & 64.47 & 73.06 & 70.10 & 76.65 & 63.73 & 64.67 & 58.00 & 64.12 & 53.29 & 66.04 \\ multi–mult & 70.91 & 51.52 & 80.50 & 82.37 & 75.96 & 66.13 & 47.09 & 83.79 & 63.26 & 72.94 & 70.01 & 77.17 & 64.47 & 64.73 & 60.16 & 66.49 & 53.05 & 66.15 \\ multi–mono & 71.91 & 50.54 & 81.21 & 79.93 & **76.93** & 64.83 & 44.05 & 83.62 & 62.87 & 71.64 & 69.26 & 77.48 & 63.59 & 64.59 & 63.09 & 61.07 & 53.48 & 65.82 \\ mono–multi & 70.76 & 52.49 & 79.67 & 82.25 & 75.09 & 67.83 & 50.89 & 83.19 & 60.68 & 77.39 & 79.27 & 75.33 & 63.80 & 65.86 & 61.57 & 65.05 & 55.65 & 66.79 \\ mono–mono & **72.00** & **57.47** & 81.15 & 83.76 & 76.12 & **68.88** & **51.71** & 83.10 & 63.92 & **74.40** & **75.52** & 76.12 & **67.47** & **67.52** & **61.95** & 67.47 & **57.05** & **68.47** \\ \hline \end{tabular}
\end{table}
Table 4: Test set results on the XL-WSD benchmark. The hyperparameters of the individual approaches (e.g. which layer of the target language encoder to align with which layer of the source language encode) were determined based on the development set of each language.

Figure 2: Overall averaged results for all the experiments conducted for the different approaches.

 

in \(2\times 32\) experiments all together. Our ablation study in Table 5 illustrates that this extra factor of \(2\) for the sparse experiments did not provided us an unfair advantage, i.e., when fixing the value of normalization in any way, the overall results did not differ substantially.

The difference in the average performance of our approach transforming sparse contextualized representations obtained by monolingual models is significant (using unpaired t-test2, \(p<0.005\)) compared to any other configuration. This suggests that the mono\(\rightarrow\)mono approach has a robust advantage over alternative variants, and the improvements seen in Table 4 are _not_ an effect of careful hyperparameter selection, but they generalize over a wide range of choices.

Footnote 2: We used unpaired t-test as the number of experiments was not same in all cases, i.e., 4 experiments/language in the multi case, and either 32 or 64 experiments/language in the rest of the cases.

This effect is further corroborated in Figure 3, which offers a comparison between the two systems with the best average performance, i.e., mono\(\rightarrow\)mono that operates with the dense vectors (results are along the x-axis) and the same model but with the enhancement of sparsification (results are along the y-axis). Each data point corresponds to a setting with the same hyperparameter choices, and points above the diagonal line with slope one demonstrate the benefits of sparsification.

We have demonstrated the improved utility of mapping language-specific sparse contextualized representations for conducting zero-shot WSD, requiring large pre-trained language-specific text encoders for the target languages. While such models are available for all languages in XL-WSD, a variety of the existing languages lack their dedicated language-specific pre-trained language model.

As such, an important question emerges whether it is possible to enjoy the benefits of mapping sparse contextualized representations for zero-shot WSD in the absence of a large pre-trained language model dedicated to the target language. To this end, we shall inspect the results of our multi\(\rightarrow\)mono approach in Table 4, a series of mapping-based experiments in which we acted as if the monolingual language models (other than the one for English) did not exist. In these experiments, the sense embeddings were obtained with bert-large-cased (being specialized to English), and the mapping to the non-English target languages were performed towards their XLM-R representations during the evaluation. This way, we could simulate the effects of the absence of language-specific models.

The multi\(\rightarrow\)mono approach provided a substantially better average performance compared to the mere utilization of a multilingual encoder in the case of dense contextualized representations as it can be seen in Table 4b. The average results of multi\(\rightarrow\)mono are slightly inferior (albeit statistically insignificantly) to that of the multi approach for the application of sparse contextualized representations. However, when comparing the multi\(\rightarrow\)multi results with that of multi\(\rightarrow\)mono, we can see that by relying on a multilingual encoder alone, and allowing a mapping to be employed between its hidden representations pertaining to different languages, one can obtain the same (or even slightly better) performance as with the multi\(\rightarrow\)mono approach. This highlights the importance of monolingual encoders for the target language, which seems to be more important than having access to a monolingual encoder for the source language.

## 5 Conclusions

In this paper we provided a systematic investigation of the benefits of using large monolingual pretrained language models in place of multilingual language models, such as XLM-R. We have shown that since monolingual neural language models are specifically tailored for a single (or at most a few related) languages, they can effectively mitigate the _curse of mutilinguality_ typical of multilingual models, and their application can significantly improve the F-scores in zero-shot WSD. We additionally showed that the benefits of sparse con

Figure 3: Comparison of the two best performing systems when the same hyperparameters were employed.

 

textualized word representations, obtained via a dictionary learning procedure, also convey to the cross-lingual setting, and that it provides complementary improvements to the usage of monolingual neural language models.

## Acknowledgments

The research was supported by the Ministry of Innovation and Technology NRDI Office within the framework of the Artificial Intelligence National Laboratory Program. Additionally, we are thankful for the usage of ELKH Cloud ([https://sciencecloud.hu/](https://sciencecloud.hu/)) that helped us achieving the results published in this paper.

## References

* J. Acs (2019)Exploring BERT's Vocabulary. Blog Post. Cited by: SS3.4.
* R. Agerri, I. S. Vicente, J. A. Campos, A. Barrena, X. Saralegi, A. Soroa, and E. Agirre (2020)Give your text representation models some love: the case for basque. In Proceedings of the 12th Language Resources and Evaluation Conference, M. Arribov, M. Trofimova, Y. Kuratov, and A. Sorokin (Eds.), Vol. , Marseille, France, pp. 4781-4788. External Links: Link Cited by: SS5.
* M. Arkhipov, M. Trofimova, Y. Kuratov, and A. Sorokin (2019)Tuning multilingual transformers for language-specific named entity recognition. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, T. Burges, H. Larochelle, D. C. Manning, L. Bottou, F. Cortes, K. Cho, P. Dollar, and R. Garnett (Eds.), Vol. , NIPS'06, New York, NY, USA, pp. 89-93. External Links: ISBN 978-1-4503-3811-1, Document, Link Cited by: SS1, SS2.
* J. Armengol-Estape, C. P. Carrino, C. Rodriguez-Penagos, O. de Gibert Bonet, C. Armentano-Oller, A. Gonzalez-Agirre, M. Melero, and M. Villegas (2021)Are multilingual models the best choice for moderately undersourced languages? a comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, pp. 4933-4946. External Links: Link Cited by: SS1, SS2.
* M. Artetxe, S. Ruder, and D. Yogatama (2020)On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4623-4637. External Links: Link Cited by: SS1, SS2.
* V. Balogh, G. Berend, D. I. Diochnos, and G. Turan (2020)Understanding the semantic content of sparse word embeddings using a commonsense knowledge base. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7399-7406. External Links: Link Cited by: SS1, SS2.
* G. Berend (2020a)Sparsity makes sense: Word sense disambiguation using sparse contextualized word representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 8498-8508. External Links: Link Cited by: SS1, SS2.
* G. Berend (2020b)Massively multilingual sparse word representations. In International Conference on Learning Representations, M. Bevilacqua and R. Navigli (Eds.), Cited by: SS1, SS2.
* M. Bevilacqua, T. Pasini, A. Ranganato, and R. Navigli (2021)Recent trends in word sense disambiguation: a survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4330-4338. External Links: Link Cited by: SS1, SS2.
* G. Bouma (2009)Normalized (pointwise) mutual information in collocation extraction. In From Form to Meaning: Processing Texts Automatically, Proceedings of the Biennial GSCL Conference 2009, Vol. , Number, pages 31-40. External Links: Link Cited by: SS1, SS2.
* J. Canete, G. Chaperon, R. Fuentes, J. Ho, H. Kang, and J. Perez (2020)Spanish pre-trained bert model and evaluation data. In PMLDCAC at ICLR 2020, Cited by: SS1, SS2.
* Y. Joong Choe, Kyubyong Park, and Dongwoo Kim (2020)Word2word: a collection of bilingual lexicons for 3,564 language pairs. In Proceedings of the 12th Language Resources and Evaluation Conference, M. Dehghan, W. Li, and Z. Liu (Eds.), Vol. , Marseille, France, pp. 3036-3045. External Links: Link Cited by: SS1, SS2.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and Veselin Stoyanov (2020)Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8440-8451. External Links: Link Cited by: SS1, SS2.
* A. Conneau, S. Wu, H. Li, L. Zettlemoyer, and V. Stoyanov (2020)Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of 

the Association for Computational Linguistics_, pages 6022-6034, Online. Association for Computational Linguistics.
* de Vries et al. (2019) Wietes de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and M. Nissim. 2019. Bertje: A dutch bert model. _ArXiv_, abs/1912.09582.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fellbaum (1998) Christiane Fellbaum. 1998. _WordNet: An Electronic Lexical Database_. Bradford Books.
* Gale et al. (1992) William A. Gale, Kenneth W. Church, and David Yarowsky. 1992. A method for disambiguating word senses in a large corpus. _Computers and the Humanities_, 26(5):415-439.
* Joulin et al. (2018) Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
* Kraftikeyan et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In _International Conference on Learning Representations_.
* Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjian Chhablani, Bhavitya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clement Delangue, Theo Matussiere, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, Francois Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Loureiro and Jorge (2019) Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
* Loureiro et al. (2021a) Daniel Loureiro, Alipio Mario Jorge, and Jose Camacho-Collados. 2021a. LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond.
* Loureiro et al. (2021b) Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021b. Analysis and Evaluation of Language Models for Word Sense Disambiguation. _Computational Linguistics_, 47(2):387-443.
* Mairal et al. (2009) Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2009. Online dictionary learning for sparse coding. In _Proceedings of the 26th Annual International Conference on Machine Learning_, ICML '09, pages 689-696, New York, NY, USA. ACM.
* Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric de la Clergerie, Djame Seddah, and Benoit Sagot. 2020. CampBERT: a tasty French language model. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7203-7219, Online. Association for Computational Linguistics.
* Mikolov et al. (2013) Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. _CoRR_, abs/1309.4168.
* Miller et al. (1994) George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In _HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
* Navigli and Ponzeotto (2012) Roberto Navigli and Simone Paolo Ponzeotto. 2012. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artif. Intell._, 193:217-250.
* Nemeseky (2021) David Mark Nemeseky. 2021. Introducing hubERT. In _XVII. Magyar Szamitogepes Nyelveszeti Konferencia (MSZNY 2021)_, pages 3-14, Szeged.
* Pasini (2020) Tommaso Pasini. 2020. The knowledge acquisition bottleneck problem in multilingual word sense disambiguation. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 4936-4942. International Joint Conferences on Artificial Intelligence Organization. Survey track.
* Pasini et al. (2021) Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Xl-wsd: An extra-large and cross-lingual evaluation framework for word sense disambiguation. _Proceedings of the AAAI Conference on Artificial Intelligence_, 35(15):13648-13656.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for 

Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
* Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Rezaee et al. (2021) Kiamehr Rezaee, Daniel Loureiro, Jose Camacho-Collados, and Mohammad Taher Plehvar. 2021. On the cross-lingual transferability of contextualized sense embeddings. In _Proceedings of the 1st Workshop on Multilingual Representation Learning_, pages 107-115, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3118-3135, Online. Association for Computational Linguistics.
* Schonemann (1966) Peter H. Schonemann. 1966. A generalized solution of the orthogonal Procrustes problem. _Psychometrika_, 31(1).
* Schuster et al. (2019) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1599-1613, Minneapolis, Minnesota. Association for Computational Linguistics.
* Smith et al. (2017) Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In _Proceedings of the 5th International Conference on Learning Representations (ICLR 2017)_.
* Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)_, pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
* Ulcar and Robnik-Sikconja (2020) Matej Ulcar and Marko Robnik-Sikconja. 2020. Finestbert and crosloengual bert. In _Text, Speech, and Dialogue_, pages 104-111, Cham. Springer International Publishing.
* Vilares et al. (2021) David Vilares, Marcos Garcia, and Carlos Gomez-Rodriguez. 2021. Bertinho: Galician BERT representations. _Proces. del Lang. Natural_, 66:13-26.
* Wang et al. (2019) Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu.-. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5721-5727, Hong Kong, China. Association for Computational Linguistics.
* Weaver (1949/1955) Warren Weaver. 1949/1955. Translation. In William N. Locke and A. Donald Boothe, editors, _Machine Translation of Languages_, pages 15-23. MIT Press, Cambridge, MA. Reprinted from a memorandum written by Weaver in 1949.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
* Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 120-130, Online. Association for Computational Linguistics.
* Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics 

Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. 2021 Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In _Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 1-10, Online. Association for Computational Linguistics.

## Appendix A Analysis of the language models

We compare some of the basic properties of the pretrained language models that we employed in Figure 4 and Figure 5. This can be useful as the monolingual quality of the language models we used could influence and account for their utility when used in conjunction with our mapping-based algorithm.

Figure 4 includes quantitative scores over the different languages related to the subword tokenizers employed by the various language models. Fertility in Figure 3(a) refers to the average number of subtokens a single token gets separated into by the tokenizer of the given language model. Multi-token ratio (MTR) in Figure 3(b) indicates the fraction of tokens that gets split into more than one piece upon tokenization Acs et al. (2019); Rust et al. (2021). Smaller values of MTR mean a better adaptation of the tokenizer to the peculiarities of the given language. It can be seen that the monolingual models do a much better job compared to XLM-R, which can be part of the reason why mapping independently trained monolingual .

In Figure 4(a), we refer to the last four layers of the investigated models as {-4,-3,-2,-1} as the English BERT is a 24-layer model, whereas the rest of the monolingual models consist of 12 layers. This means that layer -1 refers to layer 24 for English and layer 12 for some non-English model. Even though Figure 4(a) shows pathological masked language modeling (MLM) losses for certain monolingual models (e.g. Bulgarian or Basque) when measured on the XL-WSD database, their mapping-based utilization in zero-shot WSD was still possible as indicated by our main results (see Table 4). A further interesting phenomenon is that the performance of XLM-R exceeds that of the bert-large-cased model in terms of MLM for English. These results suggest that the masked language modeling performance of pretrained language models and their utility in WSD are not strongly related with each other.

## Appendix B Analysis on using the normalization of PMI scores

Upon the calculation of the sense representation matrix \(\Phi\), involving the calculation of PMI scores between the various senses from the sense inventory and the coordinates of a sparse contextual representation being non-zero, Berend (2020) suggested the use of normalized PMI scores Bouma (2009). Our preliminary results suggested that the normalization of PMI scores can have a mixed effect over the different languages. Table 5 includes a detailed breakdown on this effect for the individual languages 



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \hline  & bg & ca & da & de & es & et & eu & ft & gl & hr & hu & it & ja & ko & nl & sl & zh & Avg. \\ \hline multi & 70.69 & 51.52 & 81.41 & 83.53 & 76.45 & 67.78 & 47.85 & 83.62 & 64.47 & 73.06 & 70.10 & 76.65 & 63.73 & 64.67 & 58.00 & 64.12 & 53.29 & 66.04 \\ multi\(\rightarrow\)multi & 70.91 & 51.52 & 80.50 & 82.37 & 75.96 & 66.13 & 47.09 & 83.79 & 63.26 & 72.94 & 70.01 & 77.71 & 64.47 & 64.73 & 60.16 & 66.49 & 53.05 & 66.15 \\ multi\(\rightarrow\)mono & 71.91 & 50.54 & 81.21 & 79.93 & 76.93 & 64.83 & 44.05 & 83.62 & 62.87 & 71.64 & 69.26 & 77.48 & 63.59 & 64.59 & 60.93 & 61.07 & 53.48 & 65.82 \\ mono\(\rightarrow\)multi & 70.76 & 52.49 & 79.67 & 82.25 & 75.09 & 67.83 & 50.89 & 83.19 & 60.68 & 73.99 & 72.97 & 73.92 & 63.80 & 65.86 & 61.57 & 65.07 & 55.65 & 66.79 \\ mono\(\rightarrow\)mono & 72.00 & 57.47 & 81.15 & 83.76 & 76.12 & 67.88 & 51.71 & 83.10 & 63.92 & 74.40 & 75.52 & 76.12 & 67.47 & 67.52 & 61.95 & 67.47 & 57.05 & 68.47 \\ \end{tabular}
\end{table}
Table 5: The effects of making the normalization of PMI scores (Bouma, 2009) (a) mandatory, (b) prohibited, (c) optional to use (based on development set results) during the creation of the sparse sense representations.

Figure 4: Comparison of the tokenizers of the multilingual (XLM-R) and the monolingual language models.

Figure 5: The comparison of the multilingual (XLM-R) and the monolingual neural language models in terms of subword tokenization and their MLM objective.