Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
Authors: Linlin Liu
Thien Hai Nguyen
Shafiq Joty
Lidong Bing
Luo Si
Venue:  (2021)

Abstract: Cross-lingual word embeddings (CLWE) have been proven useful in many cross-lingual tasks. However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic. In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate significant performance improvement for word sense disambiguation tasks. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spanish/Japanese/Chinese). Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment classification and XNLI tasks, respectively. We will release our code.

Under review as a conference paper at ICLR 2021
TOWARDS MULTI-SENSE CROSS-LINGUAL ALIGNMENT OF CONTEXTUAL EMBEDDINGS
Anonymous authors
Paper under double-blind review
ABSTRACT
Cross Cross-lingual word embeddings (CLWE) have been proven useful in many cross- lingual tasks. However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic. In this work, we propose a novel framework to align contextual embeddings at the sense level by leverag- ing cross-lingual signal from bilingual dictionaries only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate significant performance improve- ment for word sense disambiguation tasks. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spant Dpannis//apannss/Chhneeee Compared with the best baseline results, our cross-lingual models achieve 0.52%. 2.09% and 1.29% average performance improvements on zero-shot -rngsalingual NER, sentiment classification and XNLI tasks, respectively. We will release our code.
INTRODUCTION
Cross-lingual word embeddings (CLWE) provide shared representation space for knowledge transfer between languages. yielding state-of oftte--o-the-rrt performance in many cross-lingual natural language processing (NLP) tasks. Most of the previous works have focused on aligning static embeddings. To utilize the richer information captured by the pre-trained language model, more recent approaches attempt to extend previous methods to align contextual representations.
Aligning the dynamic and complex contextual spaces poses significant challenges, so most of the existing approaches only perform coarse-graings alignment. [Schuster et al. (2019) compute the average of contextual embeddings for each word as anchor, and then learn to align the static anchors using a bilingual dictionary. In another work, Aldarmaki Diab 2019) use parallel sentences in their approach, where they compute sentence representations by taking the average of contextual word embeddings, and then they learn a projection matrix to align sentence representations. They find that the learned projection matrix also works well for word-level NLP tasks. Besides, unsupervised multilingual language models (Devlin et al.) 2018 Artetxe & Schwenk 2019; Conneau et al.) 2019 Liu et al., 2020) pretrained on multilingual corpora have also demonstrated strong cross- lingull transfer performance. Cao et al. (2020) and Wang et al. (2020) show that unsupervised multilingual language model can be further aligned with parallel sentences.
Though contextual word embeddings are intended to provide different representations of the same word distinct contexts, [Schuster al. (2019) find that the contextual embeddings of different senses of one word are much closer compared with that of different words. This contributes to the anisomorphic embedding distribution of different languages and causes problems for rross-lingual alignment. For example, it will be difficult to align the English word bank and its Japanese translations #tTT and # that correspond to its two different senses, since the contextual embeddings of different and
senses of bank are close to each other while those of # and / are far. Recently, Zhang et al. (2019) propose two solutions to handle multi-sense words: 1) remove multi-sense words and then align anchors in the same way Schuster et al. (2019); 9) 2) generate cluster level average anchor for contextual embeddings of multi-sense words and then learn a projection matrix in an unsupervised way with MUSE (Conneau et al. 2017). They do not make good use of the bilingual dictionaries,

---

Under review as a conference paper at ICLR 2021
which are usually easy to obtain, even in low-resource scenarios. Moreover, their projection based approach still cannot handle the anisomorphic embedding distribution problem.
In this work, propose novel sense-aware plicitly, and then leverage a sense level translation task on top of it for cross-lingual model pretraining. The proposed sense level translation task enables models provide more isomorphic and better aligned cross-lingual embeddings. We only use the cooss-lingual signal from bilingual dictionaries for supervision. Our pretrained models demonstrate consistent performance improvements on zero-shot cross-lingual NER, sentiment classification and XNLI tasks. Though pretrained on less data, our model achieves the kate-of-the result on zero-shot cross- moos-lingual German NER task. To the best of our knowledge, we are the first to perform sense-level contextual embedding alignment with only bilingual dictionaries.
entropy loss to model multiple word senses ex-
2 BACKGROUND: PREDICTION TASKS OF LANGUAGE MODELS
Next token prediction and masked token prediction are two common tasks in neural language model pretraining. We take two well-know language models, ELMo (Peters et al. 2018) and BERT (Devlin al. 2018), as examples to illustrate these two tasks (architectures are shown in Appendix A).
Next token prediction ELMo uses next token prediction tasks in a bidirectional language model. Given sequence of N tokens (t1, .2. tty, it first prepares context independent representation for each token by using a convolutional neural network over the characters or by word embedding lookup (a.k.a. input embeddings)) These representations generate the contextual representations: h. for token t, at layer j. The model assigns a learnable output embedding for each token in the vocabulary, which has the same dimension h., Then, the forward language model predicts the token at position k with:
then fed into layers of LSTMs to
exp(hT_1.) 1-1 wp:)
p(tx 12,, 12.1,,,,....4. (1-1) softmax(h)
(1)
EK- exp(h]] 1,
where k' is the index of token la in the vocabulary, V is the size of the vocabulary, and (w)....; wv) the output embeddings for the tokens in the vocabulary. The backward language model is similar to the forward one. except that tokens are predicted in the reverse order. Since the forward and backward language models are very similar, we will only describe our proposed approach in the context of the forward language model in the subsequent sections.
Masked token prediction The Masked Language Model (MLM) in BERT is a typical example of masked token prediction. Given a sequence (f1.t2.... .tt.....,x,,, fx). this approach randomly masks a certain percentage (15%) of the tokens and generates masked sequence (m1, ....... my) ). where my [mask] if the token at position l is masked, otherwise m, = ty-- BERT first prepares the context independent representations (æ1, ( 22,..., N of the masked sequence via token embeddings. It is then fed into L layers of transformer encoder (Vaswani et al.] (017) to generate "bidirectional" contextual token representations. The final layer representations are then used to predict the masked token at position A as follows:
exp(hT, wK)
p(m: = 4 m............................... m,) = softmax (h], un))
(2)
EV [x
where K,, V, h and w are similarly defined as in Eq. I Unlike ELMo, BERT ties the input and output embeddings.
3 PROPOSED FRAMEWORK
We first describe our proposed sense-aware cross entropy loss to model multiple word senses explicitly in language model pretraining. Then, we present our joint training approach with sense alignment objective for cross-lingual mapping of contextual word embeddings. The proposed framework can be applied to most of the recent neural language models, such as ELMMo, BERT and their variants. See Table for a summary of the main notations used in this paper.
3.1
SENSE- AWARE CROSS ENTROPY LOSS
2

---

Under review as a conference paper at ICLR 2021
Table Summary of the main notations
Limitations of original training objectives The training tasks with Eq. I and maximmiz the normalized dot product of contextual represen- tations (1g- or hg.l)) with weight vector wy- The only difference is that 1. in Eq. encodes the information of previous tokens in the sequence, while fu,LL in Eq. 2 encodes the information of the masked sequence. Therefore, without loss of generality, we use hg-.L L de- note the contextual representation for predicting the next or masked token 4g-
Notation
Description
k-th token in sentence s-th sense of ft
tA. A
index of token ft in vocabulary number of L.STMTrransormer layers size of vocabulary
I.
V
S
maximum number of senses per token contextual representation token fa in layer j contextual representation used softmax function for predicting la i-th word in vocabulary s-th sense of output emhedding of wy
hAJ
he
14
Even though contextual language models like ELMo and BERT provide different token rep- resentation for each distinct context, the learned representations are not guaranteed to be sense separated. For example, |Schuster al. (2019) computed the average of ELMo embeddings for each word as an anchor, and found that the erage cosine distance between contextual em- beddings of multi-sense words and their corresponding anchors are much smaller than the average distance between anchors, which mean that the embeddings of different senses of one word are relatively near to each other comparing that of different words. We also observed the same with BERT embeddings. This finding suggests that sense clusters of a multi-sense word's s appearances are not well separated in the embedding space, and the current contextual language models still have room for improvement by considering finer-grained word sense disambiguation.
context-dependent output embedding (i.e. sense vector) of es,. sense cluster center of Ui.. sense cluster centers i dimension of contextual representations
Cia C d'
projection matrix dimension reduction
Notice that there only one weight vector uy for predicting the token l in the original training tasks. Ideally, we should treat the appearances of a multi-sense word in different contexts different tokens, and train the language models to predict different senses of the word. In the following, we propose novel sense-aware cross entropy loss explicitly model different senses of word in different contexts.
Sense-aware cross entropy loss Given a sequence (f1, .t..t.,,.ty)) tN), our proposed framework gen- erates contextual representations (hg, for token 44 layer E (1....: .......,) L)) the same way as the standard LMs. Different from existing methods, our approach maintains multiple context-dependen output embeddings (henceforth, sense vectors) for each token. Specifically, let be the maximum number of senses per token. Each word v; in the vocabulary contains S separate sense vectors (w;,1: W1 wr,s), where each w,,, corresponds interesting visualization examples). Following the notation in Section 2 use k' to denote the index of the output token t in the vocabulary. Therefore, the sense vectors of ty can be represented by (unk.) 1, Wk3.: 20 ... WK.s)). which randomly initialized and of the same dimension as he+ ,L. Note that we untie the input and output embeddings in our framework.
different sense (see Appendix for some
We propose word sense selection method shown in Algorithm to select the most likely sense vector when training with sense-level cross entropy loss. Tigure[[shove the architecture of our proposed models. Assuming sense s' is selected for token lg (which means sense vector Ww,s should be used), we have the following new prediction task:
exp(h W. .e'
p(tk,s/ context) = softmax(h ww
(3)
E exp TT
The sense-aware cross entropy loss for word sense prediction is defined as follows:
SSENSE = - log(p((x.><) |context))))
(4)
Word sense selection algorithm Word sense selection when training the language model can be handled non-stationary data stream clustering problem (Aggarwal et al., 2004 Khalilian & Mustapha] 2010 Abdullatif et al. 2018). The most intuitive way to select the corresponding sense
Since the hackward language model is similar the forward, only show the forward for simplicity.
3

---

Under review as a conference paper at ICLR 2021
aasic aciessas digoritan
C4Ep4l embeddings
ScEse selection siporiilan
orper
trys
ALC
AL-1.
Ne-i.
RALA
1STM
layers vaandormer
ispel arbedding
irpul eubbdddnng
Rp-1
smbedding
xeloxe
(noos]]
SC550 dusker
(a) Sense-aware next token prediction
(b) Sense-aware masked token prediction
Word sense selection
Our proposed framework for sense-aware next token and masked token prediction tasks. Figure (c) shows an example of word sense selection, where the two sense clusters of & (assume its vocabulary index is k/) are shifting in space. Center vectors C₁' and CK' are used locate cluster centers. Given f.. the algorithm performs dimension reduction on both hs. and center vectors, and then finds the most close cluster center Cv,2, so we know the output embedding corresponding to sense 2 (wxy should be used in the loss function. Cy 2 also makes small step towards ft,LL.
Figure 1:
vector for hx..L is to select the vector wy's with the maximum dot product value h 2.0k''. .x. or cosine similarity value cossim (ha- - J., w...). However, our experiments show that these methods do work well due curse of dimensionality, suboptimal learning rate and noisy h&* _L- We apply an online k k-means algorithm to cluster different senses of word in Algorithm [| For each sense vector w,.. we maintain cluster center C, which is of the same dimension as wi Therefore, each token in the vocabulary has S such cluster center vectors, denoted by C; (c,,1 (C,,1,C.,,.... Ca.s). When predicting token & in a given sequence, we apply Algorithm Io select the best sense vector based on ha.r (see Figure D. Notice that 1, is different from h for next token prediction (Figure [p) for which hu- 1,g-1, The cluster centers C, are not neural network parameters; instead, they are randomly initialized using normal distribution (0, a²) and updated through Algorithm| [ In addition, we also maintain projection matrix P for dimension reduction to facilitate effective sense clustering. P E dxd'' projects f2, and C₁ from dimension d to d', and is shared by all tokens in vocabulary. Similar to C, iss is also randomly initialized with normal distribution N (((0, 1), and then updated through Algorithm 2 Both Algorithm [ and 2 run in parallel, and are interrupted when the language model stops training.
C.......
Some rationales behind algorithm design are the following:
Algorithm Word sense selection
Algorithm 2 Projection matrix P update
1: Hyper-parameters:
number of senses S,
1: Hyper-parameters:
projection dimension d', up- date interval M, queue size Q
learning rate Initialize the set of all sense cluster centers C
2: Initialize P with (0, 1), queue H = 0, m
3: repeat
3: repeat
4: input: fu,,.. vocabulary index K' of the to-
4: input: İg.
ken predict
m = m +
5: Lookup sense cluster centers for k': Cw
6: Add fu. to queue H
(ou, 2..,2..... Cer.
7:
if size(H) ()) > then
6:
P = updated projection matrix from Alg.
8:
Pop the oldest element from queue H. end if
7:
if cosine similarity between car..r P and
h1 P is the largest among the vectors in Cy then
10:
if >= then
11:
P P= the first d' PCA components of H
8: C = -1 (1--c)
ahu.L 9: output: s' s'(wH', should be selected)
12:
13:
end if
10:
end if
output:
11:
until interrupted
15:
until interrupted
4

---

Under review as a conference paper at ICLR 2021
Directly computing cosine similarity between Ck',s and h suffers from the curse of dimension- ality. We maintain P for dimension reduction. Although many algorithms use random projection for dimension reduction, we find using PCA components can help improve clustering accuracy.
Since the neural model parameters keep being updated during training, the sense clusters become non-stationary, i.e., their locations keep changing. Experiments shows that when using P for dimension reduction, a slightly larger projection dimension d' will make the clustering algorithm less sensitive to cluster location change. We use d' 16 for ELMo, and d' 14 for BERT. We also notice that the sense clustering works well even P is updated sporadically. We can set a relatively large update interval in Algorithm 2 reduce computation cost.
A separate sense learning rate a should set for the clustering algorithm. A large a makes the algorithm less robust to noise, while small a leads to slow convergence.
essential to use the current token's 's contextual representation h; for sense selection even - I.I in the next token prediction task. If we use 1 for sense selection, experiments show that most of the variance comes from input embedding 2k-1. This introduces too much noise for word sense clustering.
though we use hag-
Dynamic pruning of redundant word senses To make the training more efficient, we keep track of relative sense selection frequency for each token in the vocabulary. Assume token o; has initial senses (Vi,1. U4.2 U,, , for which we compute the relative frequency p(U;,a) such that 0 I p(vt.s) I 1 and Σ. p(vr.s)
lower p(v;,s) means the sense less frequently selected compared with others. We check the relative frequencies after every E training steps, and if p(vi,s) < 3 (a threshold hyper-parameter), Vg is removed from the list of senses of 0/.
Remark on model size and parameters The sense cluster centers C' and the projection matrix P are only used to facilitate sense selection during model pretraining, which are not neural model parameters. The sense vectors w,,s will no longer be used after pretraining, which can also be discarded. Therefore, our models and the original models have exactly the same number of parameters when transferred to downstream tasks.
Remark model complexity The computational complexity of our algorithm is linear with respect to the size of data, so our method is scalable to train on very large datasets.
3.2 JOINT TRAINING WITH SENSE LEVEL TRANSLATION
Training language model with sense-aware cross entropy loss helps to learn contextual token repre- sentations that are sufficiently distinct for different senses (44.1) In this subsection, we extend it to cross-lingual settings and present novel approach to learn cross looss-lingual contextual word embeddings the sense level. Our approach uses bilingual seed dictionary? and can be applied both next and masked token prediction tasks.
For training the cross-lingual LM, we concatenate the (non-parallel) corpora of two languages, L1 and Lg, and construct joint vocabulary O t L Ulz, where Oh and 0 are the vocabularies of I, and L2, respectively. Algorithm is used to model the senses of tokens in the joint vocabulary. In addition to predicting the correct monolingual sense p(ta, (context) in Eq. 3 we also train the model to predict its sense level translation. Let vi be the translation of la and sense Vj.s* of Uj be the best sense level translation under the given context, we add the following sense-level translation prediction task to maximize probability of U1,
exp(t 1 W.j...))
p (context) softmax(h]] ,,,,'
(5)
1i=1 E Wi,,)
where Wj.s is the corresponding sense vector of Uj.a*
Similar to the previous subsection, we maintain sense cluster centers C₁ for each token v; O and the shared projection matrix P select the best translation sense. Assume lę has T' translations in dictionary, and each translation has S senses, then there are T x S possible sense level translations for to in the given context. If the cossim(hy,LPP c P.) value is the largest among the T x S sense cluster centers, then we select uj..* as the closest translation. An example is shown in Figure IIf not provided, it can he learned in unsupervised way, e.g., MUSE (Concaau al.
2017
5

---

Under review as a conference paper at ICLR 2021
(ongraiortoo)
R (uadder sead)
Seff
HR (ine: buak)
bark Ofier benk)
Figure 2: An example of English-Japanese sense-level joint training, which shows two possible Japanese translations (#RT and Hi)) of the English word bank. hz.2 is contextual representation of bank in finance context and Cg22 is the cluster center for this sense. C,,1, Ca,2; Cb,1, Cb,2 are different sense cluster centers of the two Japanese translations, among which Ca,2 is the closest to hg. after dimension reduction through PCA. Our sense level objective (Eq. (6) moves sense clusters for bank (organization) and (frforrgiiiatonnn closer to each other.
If token &R has at least one translation in the dictionary, the translation cross entropy loss can be computed as:
LTRAN = (og(p(e),** (contert))
(6)
If token t& has no translation in the seed dictionary, we use Eq. as the only loss. The joint training loss is defined as follows:
EsENSE +
l has translations
CIINTT
2
[SENSE,
otherwise
Further alignment (optional) Our sense-aware pretraining tries to move similar senses of two different languages close to each other as illustrated in Figure 2 This process makes the sense distributions of the two languages more isomorphic (some sense vector visualization examples are shown in Appendix [ Applying the linear projection approach proposed by Schuster et al. (2019) on top of the language model pretrained with our framework can further improve cross-lingual transfer on some tasks. See Appendix B for more details of our implementation.
EXPERIMENTS
4.1 EXPERIMENTS USING MONOLINGUAL MODELS
To verify the effectiveness of our proposed sense-aware cross entropy loss, we implement the monolingual models top of ELMo and BERT with the changes described in {3.1] which are named SaELMo (Sense-aware ELMo) and SaBERT (Sense-aware BERT) respectively. The algorithm for dynamic pruning of redundant word senses is optional, which is implemented on SaELMo only.
Pretraining settings We use the one billion word language modeling benchmark data (Chelba et al., 2013) to pretrain all the monolingual models. The corpus preprocessed with the provided scripts, and then converted to lowercase. We do not apply any subword tokenization. We use similar hyper-parameters as Peters et al. (2018) train the ELMo and SaELMo models, and similar hyper- parameters as Devlin et al. (2018) to train 4-yayer BERT- Tiny and SaBERT-Tine Tiny. Next sentence prediction task disabled in BERT-Tiny and SaBERT-Tiny, since this task irrelevant to our proposed changes. See Appendix D. for a complete list of hyper-parameters.
Word sense disambiguation (WSD) Since our context-aware cross entropy loss is designed to learn word senses better in the context, we first conduct experiments compare our monolingual model with the original models on the WSD task (Raganato et al. 2017). which is a task to associate words in context with the most suitable entry in a defined sense inventory. We use similar framework as Peters et al. (2018) to evaluate the monolingual models We use SemCor 3.0 (Miller al., 1993) as training data, and Senseval/SemFFee series (Edmonds & Cotton 2001 Moro & Navigli 2015 Navigli et al. 2013: Pradhan et al.) 2007 [Snyder & Palmer (2004) as test data. We use the pretrained models to compute the average of contextual representations for each sense in training
data, and then classify the senses of the target words in test sentences by finding the nearest neighbour.
We modified the script from: https:/githubbcoo/ttgggfffffiii Mo- http://deeetttttooooooeeeeeeeene Mr---W/Drrimmlmmmenttinnn...

---

Under review as a conference paper at ICLR 2021
WSD results are presented in Table 2 SaELMo shows significant performance improvements over the baseline ELMo model in all of the five test sets. SaBERT-Tiny also outperforms BERT-Timy except SE07, which is the smallest among the five test sets.
Table 2: Word sense disambiguation (F1 scores)
Model
SE2
SE3
SE07
SE13
SE15
ELMo
0.555
0.576
0.446
0.544
0.538
SaELMo (ours)
0.575
0.586
0.470
0.560
0.583
BERT-Tine Tiny
0.596
0,539
0.466
0.536
0.572
SaBERT-Tiny (ours)
0.611
0,546
0.446
0.550
0.579
4.2 EXPERIMENTS USING BILINGUAL MODELS
To verify the effectiveness of our cross-lingual framework, we implement the bilingual models on top of ELMo, named Bi-SaELMo that does not use linear projection for further alignment and Bi-SaEL that uses the linear projection. Sense vectors and cluster center vectors are not shared between the forward and backward language models. We use ELMo+Proo and Joint- ELMo+Proj our bascline models, where ELMo+Proj is proposed by /Schuster et al. (2019) and Joint-ELMo+Prooj is implemented following the framework recently proposed by Wang et al. (2020). Wang al. (2020) combine joint training and projection, and claim their framework is applicable to any projection method, so we implement the same projection method as Schuster et al. al.] (2019) did for Joint-ELMo++rooj. We also report results of ELMo and Joint-ELMo, which the counterparts of ELMo+Proj and MonntELMM+Proo without using linear projection.
Pretraining settings To pretrain language models, we sample a 500-million-token corpus for each language from the English, German, Spanish, Japanese and Chinese Wikipedia dump. The dictionaries used for pretraining models and learning the projection matrix were downloaded from the MUSE (Conneau et al., 2017) GitHub page We also add JMDict (Breen (204) the en-jp MUSE dictionary. Bilingual models were pretrained on en-de, en-es, en-jp and en-zh concatenated data with similar parameters as the monolingual models. ELMo and ELMo+Proj were pretrained on monolingual data, while the projection matrix of ELMo+Pro was learned using bilingual data. See Appendix D.2 for a complete list of paper-parameters.
Zero-shot cross-lingual NER A Bi- LSTM-CRF model implemented with the
Table 3: Zero-shot cross-lingual NER (F1)
Flair framework (Akbik et al. (0188) is used
for this task. For the CoNLLL-2002 (Tjong
Model
de
es
zh
Kim Sang 2002) and CoNLL-2003 (Sang & De Meulder 2003) datasets, the NER model was trained on English data, and evaluated on Spanish and German test data. For the OntoNotes 5.0 (Weischedel et al. 2013) dataset, the NER model was trained on all English data and evaluated on all Chi- nese data. We report the average Fl of 5 runs in Table 3 The results show that all of the models using linear projection out- perform their counterparts (not using linear projection). since minimizing token level distance is more important for cross-lingual NER tasks. Our sense-aware pretraining makes sense distributions of two languages more isomorphic, which further improves linear projection performance. Our model Si-SaELMo+PPro demonstrates consistent performance improvement in all the three languages. Moreover, our model outperforms finetuned XLM/XLM-R and Multilingual BERT on German data, and achieves state of the art even though it is pretrained on less data.
16.30
16.14
Joint-ELMo
56.49
58.91
53.47
EI MooMMj (Schuster et al. 2019
69.57
60.02
63.15
Joint-ELMo-Prry Wang 2020
71.59
65.19
59.08
Bi-SaELMo (ours)
63.83
60.65
55.83
Bi-SaELMo+ Proj (ours)
72.19
65.86
63.44
references, but not our buselines, since they are trained on much larger datasets and/or parallel sentences.
XLMM Finetune CCnneau & Lample
67.55
63.18
XLM-R FAMM-FFinctuie (Comeau al. 2019
71.40
78.64
BERT Finetune [Pires el al 2019
69.74
73.59
M-BERT Finetune
Dicuze 2019
69.56
74.96
M BERT Finetune+Ady Keinng el al 2019
71.90
74.30
M-BERT Feature-Proj
(Wang cl 2020
70.54
75.77
7

---

Under review as a conference paper at ICLR 2021
Zero-shot cross-lingual sentiment classification We use the multi- multi--ingual multi-domain Amazon review data (Prettenhoferr & Stein 2010) for evaluation on tross-lingual sentiment classification. The ratings in review data are converted into binary labels. The average of contextual word representations is used as the document/sentence representation for each review text/summary, which is then fed into a two-dense-layer model for sentiment classification. All the models are trained on English, and evaluated on German and Japanese test data in the same domain. We report the average accuracy of 5 runs in Table 4 Different from the NER task, the linear projection approach for cross-lingul alignment does not work for this task, since it may add noise to embedding features. Our model Bi- Bi-SaELMo demonstrates consistent improvements in all of the 6 evaluation tasks. The performance of Bi-SaELMo is significantly better than Joint-ELM which shows that our sense-level translation pretraining objective improves cross-lingual embedding alignment.
Table 4: Zero-shot sentiment classification accuracy
Table 5: Zero-shot XNLI accuracy
Model
de
Model
de
xh
books
music
dvvd
beokks
music
divi
LLMo
34.07
33.41
35.77
BLMo
Joios-ELMe
60:12
57.82
ho
75.22
66.25
Moooo Schuster 2010
55.51
53.17
ElMo-Proi Schreter 2019
50.22
49.94
50.65
Joins ELMorz Waig prry
63:33
58.34
Jom--F Mi-yrng
75.74
72:25
73.25
00
59.77
57.65
Bi touns)
60.98
BI Sak3 Mic (ours)
Mc-S.LMM-PPiiiio (ous)
77.46
70.84
76.33
66.25
74.97
63.99
62.17
68.16
69:08
55.01
69.08
61.57
Mo-Su (ours)
64.77
65.05
60.44
Zero-shot cross-lingual natural language inference (XNLI) We use XNLI (Conneau et al. 2018) 8)
and MultiNLI (Williams et al. 2018) data for evaluation on this task. The Bi-LSTM baseline mode was trained on MultiNLI English training data, and then evaluated on XNLI German, Spanish, Chinese test data. We report the average zero-shot XNLI accuracy of 2 runs in Table Our models show consistent improvements over the baselines on all of the three data sets. For zero-shot transfer to Chinese, both of our models outperform the best baseline by more than 2 points, which again demonstrates the effectiveness of our framework on distant language pairs.
RELATED WORK
Cross-lingual word embedding demonstrates strong performance in many cross-lingual transfer tasks. The projection-based approach has a long line of research on aligning static embeddings (Mikolov et al. 2013 Xing et al. 2013 [mmth et al., 2017: Joulin et al.) 2018). It assumes that the embedding spaces of different languages have an isomorphic structure, and fit an orthogonal matrix to project multiple monolingual embedding spaces to a shared space. Recent studies (Schuster et al.) 2019: Aldarmaki & Diab 2019) have extended this approach to contextual representation alignment. Besides, there are also many discussions on the limitations of the projection- -base approach, arguing that the isomorphic assumption is not true in general (Nakashole & Flauger 2018 Patra et al. 2018
J0u7i
Søgaard et al. 2018 Ormazabal et al.] (211)). Joint training is another line of research and early methods (Gouws al.] 20155 Luong al. 2015 Ammar et al. 2016) learn static word embeddings of multiple languages simultaneously. Extending joint training to cross- or multi-lingual language model pretraining has gained more attention recently. As discussed above, unsupervised multilingual language models (Devlin et al. 2018 Artetxe & Schwenk 2019 Conneau & Lample| 20019] Conneau Liu et al. (020)) also demonstrate strong cross- cross-lingual transfer performance.
ct al.
2019
There has been some work on sense-aware language models/embeddings (Rothe & Schüitze| 2015
Pilehvar & Collier 2016, Hedderich et al. 2019). and most of them require WordNet (Miller) 1998) or
other additional resource for supervision. Suster et al. (2016) utilize both monolingual and bilingual information from parallel corpora to learn multi-sense word embeddings. |Peters et al. (2019) embed WordNet knowledge into BERT with attention mechanism. [evine et al. (2019) pretrain BERS to predict both the masked words and their WordNet supersenses. Similar to our framework, there are also some unsupervised approaches, but most of them are used to learn static embeddings. |Huang et al. (2012) learn word representations with both local and global context, and then apply clustering algorithm to learn multi- protiiprroootpp vectors. Neelakantan et al. (2014) propose an extension to the Skip-gram model that leverage k-means clustering algorithm learns multiple embeddings per word type. Lee & Chen (2017) leverage reinforcement learning for modularized unsupervised sense level embedding learning. |oodd Graber et al. (2020) use Gumbel softmax for sense disambiguation when learning sense embeddings.
"haps//gtthuh.co///YUMM1 I
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSIONS
In this paper, we have introduced a novel sense-aware cross entropy loss model word senses explicitly, then we have further proposed a sense-level alignment objectíve for cross-lingual model pretraining using only bilingual dictionaries. The results of the experiments show the effectiveness of our monolingual and bilingual models on WSD, zero-shot cross-lingual NER, sentiment classification and XNLI tasks. In future work, we will study how to effectively extend our method multilingual models. In addition, using the sense cluster centers to learn the linear projection matrix would be another promising direction to further improve lingull alignment.
REFERENCES
Amr Abdullatif, Francesco Masulli, and Stefano Rovetta. Clustering of nonstationary data streams: A survey of fuzzy partitional methods. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, ((4):e1258. 2018.
Charu C Aggarwal, Jiawei Han, Jianyong Wang, and Philip Yu, A framework for projected clustering of high dimensional data streams. In Proceedings of the Thirtieth international conference Very large data bases- Volume 30, pp. 852-863, 2004.
Alan Akbik, Dunean Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pp- 1638-1649, 2018.
Hanan Aldarmaki and Mona Diab. Context-aware crosslingual mapping. arXiv preprint arXiv:1903.03343,, 2019.
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. Massively multilingual word embeddings. arXiv preprint arXiv11602.01925, 2016.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics. 7:597-610, Mar 2019. ISSN 2307-387X. doi: 10.1 162/tall 00288. URI. | //tx//xx/// doi
org/10. 10e//1..116/EEE. / Eac 00288
Jordan Boyd-Graber, Fenfei Guo, Leah Findlater, and Mohit lyyer. Which evaluations uncover sense representations that actually make sense? In Proceedings of the 12th Language Resources and Eval- uation Conference, pp. 1727-1738, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4 URL httph::tt////... htt.a///ww...ooo//tooooo//////// anthology/
org/
2020. 1020.1rec-1.211
Jim Breen. Jmdict: a japanese- dictionary. Proceedings the linguistic resources, pp. 65-72, 2004.
the workshop on multilingual
Steven Cao, Nikita Kitaev, and Dan Klein. Multilingual alignment of contextual word representations. arXiv preprini arXiv:2002.0355 2020.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2013.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, pp. 7057-7067, 2019.
Alexis Conneau, Guillaume Lample, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv: 2017.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018

---

Under review as a conference paper at ICLR 2021
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: 2019.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Philip Edmonds and Scott Cotton. Senseval-2: overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pp. 1-5, 2001.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representa- tions without word alignments. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
Michael A. Hedderich, Andrew Yates, Dietrich Klakow, and Gerard de Melo. Using multi-sense vector embeddings for reverse dictionaries. In Proceedings of the 13th International Conference Computational Semantics Long Papers, pp. 247-258, Gothenburg, Sweden, May 2019.
Association for Computational Linguistics. doi: 10. 10.18653/1/W11900221 URL httth::////// /www
aclweb. wewweb.org/aal hatp//annttooooyyl11oeoe -V
Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. Improving word representations
via global context and multiple word prototypes. In Proceedings of the 50th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 873-882, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https: www. ac web org/antho wttp:/hhhhooyyy111111111
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss transla- tion: Learning bilingual word mapping with retrieval criterion. arXiv preprint arXiv: / 07745, 2018.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. Adversarial learning with contextual embeddings for zero-resource cross-lingus lingual classification and ner. arXiv preprint arXiv: 00153, 2019.
Madjid Khalilian and Norwati Mustapha. Data stream clustering: Challenges and issues. arXiv preprint 1006v:1006.526. 2010.
Guang-He Lee and Yun-Nung Chen. MUSE: Modularizing unsupervised sense embeddings. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 327-337, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D1711100
URL https::////........ www. hwtp:wwwwwwweee. wwwweb.www.... wtt.://wwww...//tttttooooo//////
Yoav Levine. Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwarz,, Amnon Shashua, and Yoav Shoham. Sensebert: Driving some sense into bert. arXiv preprint arXiv: 1908:099000000 05646, 2019.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation, 2020.
Minh- Thang Luong, Hieu Pham, and Christopher D Manning. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 151-159, 2015.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pp. 55-60, 2014. URLhtttp pectp://wwz. www aclweb. htl.eb...rr//anttoooooo/////////
Tomas Mikolov, Quoe V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv: 4168, 2013.
George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
10

---

Under review as a conference paper at ICLR 2021
George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. A semantic concordance. In Proceedings of the workshop on Human Language Technology, pp. 303-308, Association for Computational Linguistics, 1993.
Andrea Moro and Roberto Navigli. Semeval-2015 task 13: Multilingual all-words sense disambigua- tion and entity linking. Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 288-288811 207,-997,7 2015.
Ndapa Nakashole and Raphael Flauger. Characterizing departures from linearity in word translation. arXiv preprint arXiv: 1806. 04508, 2018.
Roberto Navigli, David Jurgens, and Daniele Vannella. Semeval-200 2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pp. 222-231 2013.
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. Efficient non- parametric estimation of multiple embeddings per word vector space. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1059-1069, Doha, Qatar, October 2014. Association Computational Linguisties. doi: 10.3115/v1/D14-111
URL https: www. www.wb.wwwbbb...... htt.://ww......aaooooooo///----/
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. Analyzing the limitations of cross-lingual word embedding mappings. arXiv preprint 1906. 2019.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew Gormley, and Graham Neubig. Bliss in non-isometric embedding spaces. 2018.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. arXiv preprint arXiv: 1909.99994446 2019.
Mohammad Taher Pilehvar and Nigel Collier. De-conflated semantic representations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1690- Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10.18653/v1/ 117-1174. URL TtLp:tts:: www. ac
11174
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual multilingual bert? arXiv preprint arXiv: 50200, 2019.
Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. Semeval-2007 task- 17: English lexical sample, srl and all words. In Proceedings of the fourth international workshop on semantic evaluations (SemEval-2000) pp. 87-92, 2007.
Peter Prettenhofer and Benno Stein. Cross-language text classification using structural correspondence learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Lin- guistics, pp. 1118-1127, Uppsala. Sweden, July 2010. Association for Computational Linguistics. URL
www. a
Alessandro Raganato, Jose Camacho Camacho-Collados, and Roberto Navigli. Word sense disambiguation: unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 99- 99-110, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: www acc.a//w.......aaoooooo/////-1-.
Sascha Rothe and Hinrich Schütze. AutoExtend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1793 1803, Beijing, China, July 2015. Association for Computational Linguistics. doi: anthol StcholooyyP15-11733
1173. URL https:/ www wwwwwb. /rg/
11

---

Under review as a conference paper at ICLR 2021
Erik F Sang and Fien De Meulder. Introduction the conll-2000 shared task: Language Language-indeeendee independent named entity recognition. arXiv preprint cs/0306050, 2003.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Shori Papers), pp. 1599-1613, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1/NN191110
URL https: www. ac web. wwt.wwew.............e......a.aa wwtpg/anhhllogg///------
Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03839, 2017.
Benjamin Snyder and Martha Palmer. The english all-words task. In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pp. 41 41-43, 2004.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. On the limitations of unsupervised bilingual dictionary induction. arXiv preprint 1805.0300 2018
Simon Suster, Ivan Titov, and Gertjan Van Noord. Bilingual learning of multi-sense embeddings with discrete autoencoders. arXiv preprint arXiv: 1603.09/28 2016.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-22002 shared task: Language-inddeenddnt named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002
(CoNLL-2002), 2002. URL https: www awwweb. org/ http://www.wgg./ttooooo.oooo /w02 -
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime G. Carbonell. Cross-lingual alignment vs joint training: A comparative study and a simple unified framework. In International Conference on Learning Representations, 2020. URL
net/forum?idds3
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. Ontonotes release 5.0 1dc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23, 2013.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers). pp. 1112-1122, Association for Computational Linguisties, 2018. URL
org/ 1
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising lingsllingual effectiveness of bert. arXiv preprint arXiv: /90400994 2019.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal trans- form for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 1006-1011, 2015.
Zheng Zhang, Ruiqing Yin, Jun Zhu, and Pierre Zweigenbaum. Cross-lingual contextual word embeddings mapping with multi-sense words in mind. arXiv preprint
2019.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A PREDICTION TASKS OF LANGUAGE MODELS
Next token prediction and masked token prediction are two common tasks in neural language model (LM) pretraining. We take well-known language models, ELMo and BERT, as examples to illustrate these two tasks, which are shown in Figure
output emboddings (tied with input emhheddiags)
output embeddings
T2
softmace
!!
uty
softmaa
hLA
bg.
fig-1.A
stotes
LSTM
+
layer
L. layers transformee
Aty 1.t
input
input embeddings
emboddings
Rk-1
26
Fl-11
embbeding
emboddiing
sentence
fe-1
6
sentence
1
fe-1
[mosk]
(a) Next token prediction
(b) Masked token prediction
Figure 3: Next token and masked token prediction tasks of language models. For simplicity, we only show the forward language model in next token prediction.
B FURTHER ALIGNMENT (OPTIONAL)
Applying the linear projection approach proposed by |Schuster et al. (2019) on top our framework can further improve cross-lingual transfer on some tasks. After our cross-lingual model is finished training on the concatenated corpora of two languages, In and T.2, is used to generate contextual
token embeddings for the word pairs in the seed dictionary D = {(t T. )
1P) 7 Then, we
compute the average of all contextual embeddings for each token t denoted by a Finally, linear projection matrix W Rdxd is leamned minimize cross-lingual embedding distance:
D
W arg min W
Wa 2/211²
(8)
C VISUALIZATION OF SENSE VECTORS
We visualize the sense vectors of each model in dimensional PCA, and show some examples in Figures to 7 For our English monolingual model (SaELMo), the vectors close to two different sense vectors of the word may are shown (a) and (b) of Figure respectively. We observe that senses are well clustered in these two subfigures, where cluster (a) corresponds to "month", and cluster (b) corresponds to "auxiliary verb".
We do the same for the English-Japanese bilingual model (Bi-SaELMo, without projection), and show the vectors close two different sense vectors the English word bank in (c) and (d) of Figure 5 We can see both English and Japanese sense vectors (trade, M.T.. ae #5 etc.) in (c), most of which correspond to the sense "organization", though there are some noises. Similarly, most the sense vectors in (d) correspond to sense "river bank".
'If any token ta appears in both languages, we add that as an entry (tx, fx) to the dictionary as well. We use the tensorflow embedding projector bttps://poopccoortesssofooowoo for visualization.
13

---

Under review as a conference paper at ICLR 2021
Another two examples shown in Figures and Our framework exhibits good sense clustering and sense level cross- missull alignment behaviour in these examples. All sense vectors are dumped at training step 200,000, which is before pretraining complete.
january_5
emight
ecould.3
ejuty
m
eahould
edid)1
=february_3
octoberr_2
vaugust_5
eseptember_5
unavember_4
soughtS
#cannot
may_3
adoesn,
eafb.4
adde_2
atcoelved_
2
may
epriLA
+July_5
efebruary_2 2
ecould
ooocs
es11_1
char3
idccmmeer_2 2
smigh
almost
3
augustt1
wilL2
CUOMA
ebook
acoukE_2
enovember_2
3
m
may 2
mmight3
appiL
4L2
4that
1000_5
may_5
sapriL
february_5
anov_4
emarch_2
July_2
emay
d_OUd_2
<hod
1aug.1
1eb_1
owax_2
sept.5
chea.1
(a) "may" for month
(h) "may" as auxiliary verb
Figure 4: We visualize sense vectors of English monolingual model (SaELMo) in a two dimensional PCA, and show the vectors close to two different sense vectors of word may in (a) and (b).
ufter22
rig
darkk11
bank_3
af0611.11
oweet
dbark_2
<B
scouTh1
sbark
eporty_a
bank_2 2
+88,11
dbanko_3
dbark.1
aqoesmmentt.2
arodd_ 1
ichuch_2
(a) "bank" for organization
(b) "bank" for river bank
Figure 5: We visualize all sense vectors of en-jp bilingual model (Bi-SaELMo) in a two dimensional PCA, and show the vectors close to two different sense vectors of word bank.
14

---

Under review as a conference paper at ICLR 2021
60.3
ft.5
dbritsh_2
o
stwo.1
3
2
2high_2
scanacion_
obig.5
US_1 1
austraaloon
athern_4
#Tnanciel.
unthed_
uK
othwmn3
euk cuk_bunted_2
dranian.4
ROUT.4
ouur_5
aurited
shim
IU..3
uutted, 5
olloaal.
smesican
ochinese1
eourse
apoople
au..5
abttish_3
ehim1
nationaaional
4you.
625.4
igovermmntt
-US_2
efedensiS
(a) "us" country
(b) "us" as pronoun
Figure 6: We visualize sense vectors of English monolingual model (SaELMo) in two dimensional PCA, and show the vectors close to two different sense vectors of word us in (a) and (b).
idccomber_1 1
commueee 22
Wil mardn
captamberr
qUi.1
saugustux
Crapoo TPpao
decembore2
1spel1 2
epppt.2
jjnuar.2
Panou
2an 2
may_2
enovember.1
cootobr.1
eebbuary.
Hjuly22
Jann
mmsga_2
-may_T
lugustt1
mmdge_1
3
ddrf.3
amarch3
scoul122
maal
aaugust.3
cottober.1
1september
scecember
may_3
(a) "may" as auxiliary verb
(b) "may" month
Figure 7: We visualize sense vectors of en-de bilingual model (Bi-SaELMo) in a two dimensional PCA, and show the vectors close to two different sense vectors of word may.
15

---

Under review as a conference paper at ICLR 2021
D PRETRAINING DETAILS
D.1 MONOLINGUAL MODEL.
All the monolingual models were trained for one million steps. For better sense clustering perfor- mance, the maximum number of senses (S in word sense selection algorithm) was set to 1 for the first 20,000 steps to quickly get reasonable initial model, and then increased to 5 afterwards when pretraining Ma and SaBERT-Tiny, which is controlled by hyperparameter и context in our implementation. For SaELMo, token, but only use the first sense in the 20,000 steps, and then use the other 5 senses (the first sense will be disabled) afterwards. We implement this for SaBERT-Tiny in a slightly different way, where n moconeett can be set to 5 directly to achieve the same effect. We use two NVIDIA V100 GPUs to pretrain SaELMo, which takes about 15 days to complete training. We use one NVIDIA V100 GPU and for the hyperparameters used
context 6, so that the model initialize 6 senses for each
to pretrain SaBERT-Tiny. which takes about 5 days. See Tables and to pretrain SaELMo and SaBERT-Tiny respectively.
Table 6: Monolingual model hyperparameters: SaELMo
Hyperparameter
Value
max. word length batch_size
50
256
2
bidirectional
Truc
char cnn.embedkdingddin
16
char saar_enn:max, characterr. per Joken Joken
50
261
char.cmn:n highway
2
dropout
0.1
Istmocell cclip
3
Istmadim
Istmen lstmen_layers
2
Istm:proj__cip
Istm-projection.o dim
512
Istmuuse skip connections
True
all ail_clip.norm_vaa
10.0
n.epoccs
10
unroll.steps
16
D. aegative.samples batch
8192
conlext
cluster ppnnerrmmjj.im
16
pen_sample
20,000
remove less froseent.contexts contexts
0.1
learning .rate
0.2
sense learnin
0.01
Table 7: Monolingual model hyperparameters: SaBERT-Tiny
Hyperparameter
Value
attention probs dropout prob
directionality
bidi
hidden_act
gelu
badden_dropout.pee
0.1
hidden.size
512
initializerrang raanee intermediate. size
2048
max position embeddings
num sun_tteention beads num_hidden Jayers
fe.sze
512
pooler num attention heads
pooler_num_fc layes
pooler_size_per_hee
pooler_rype
Wor.Loken.trrasffo
type vocab size
2
27654
context
context.rep.lr
pea.dim
14
contextual warmup
20,000
16

---

Under review as a conference paper at ICLR 2021
D.2 BILINGUAL MODEL
As metioned in the paper, we use Wikipedia dump to pretrain the bilingual models. The Stanford CoreNLP tokenizer (Manning al. 2014) is used tokenize English, German, Spanish and Chinese data. And the spaCy tokenizer is used to tokenize Japanese data. All data are converted to lowercase. We convert Chinese data to simplified font to make it consistent with evaluation task datasets.
All the language models used in cross-lingual experiments were pretrained for 600,000 steps from scratch. Similar our monolingual models, maximum number senses (S in word sense selection algorithm) was set to 1 for the first 20,000 steps, and the increased to afterwards when pretraining Bi-SaELMo and Mo-SoELM We use two NVIDIA V100 GPUs to pretrain each Bi-SaELMo model, which takes about 10 days to complete the training. See Table 8 for the hyperparameters used to pretrain L. ProPooo
Table Bilingual model hyperparameters: MALLLMo/Bi-SaELMo-HP
Hyperparameter
Value
max word length
50
batch_size
256
n.gpus
2
bidirectional
True
char, con.cmbedddingdimm
16
char_enn:max.co www.:/caattatppr.uuccoe
50
char enn:n characters
261
char con.n highway
2
dropout
0.1
Istm:cell clip Istm:dim
4096
Istm:_Jlayer layers Istm:rroj clip
2
3
Istm:projettoon dim
$12
Istm:use_skip.co wannuueeskip.coomecciii een.ecoonsstso
True
clip norm. val epochs
10.0
uttosl
12
D n.negttive.sune www.reeeaassss.ec.. batch
8192
context
4
cluster.pre mnsster-rojj_imm
scaps
20,000
remove less freqent contexts
0.1
learning.rate
0.2
sense Jese.learning.rte
0.01
'Theoretically, in reasonable range, it is expected that larger S would be more helpful to capture the fine-grained senses. However, due limited computation power, only 3 here, and 5 for the monolingual models.
17


Reviews:
[{"review_id": "aODO-fHSdcX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis paper proposes the alignment of cross-lingual contextual embeddings not just at the word level, but at the sense level. It does this by relying purely on unaligned, unlabeled monolingual corpora used for pre-training, along with bilingual lexica. It does this by adapting the LM objective to be a sense-aware cross entropy loss, in which the sense is obtained by the use of a streaming k-means clustering algorithm combined with dimensionality reduction. If a bilingual lexicon is available, a sense-level translation objective can be added to encourage the model to predict the same sense in the other language (thereby encouraging identical senses of a word in the two languages to be closer together). \n\n**Positives:**\n* This is the first work (as far as I am aware) to perform sense-level alignment of contextual embeddings with only unlabeled, unaligned monolingual corpora and bilingual dictionaries.\n* The proposed method works for both MLM and the next token prediction objectives. It should thus be applicable to most contextual embedding methods.\n* The proposed method is extremely intuitive, but non-trivial. In particular, this work clearly presents its insights and well-justified tricks that seem to be crucial to the working of the proposed approach, for example: how simply selecting the vector $\\\\mathbf{w}\\_{k',s}$ with the maximum dot product $\\\\mathbf{h}^T\\_{k^*,s}\\\\mathbf{w}\\_{k',s}$ does not work; how/why dimension reduction is crucial; the need to use $\\\\mathbf{h}\\_{k,L}$ for sense selection even though $\\\\mathbf{h}\\_{k-1,L}$ is used for next token prediction.\n\n**Concerns:**\n* In the case of the results presented for WSD, the ELMo baseline seems to be substantially worse than that reported in literature [1, 2] (ref: Table 2).\n* While the paper talks about the applicability of the proposed method to both the next token prediction (in ELMo) and the MLM (in BERT) styles of language models, the experiments section seems very heavily focused on ELMo, not demonstrating the applicability of the method on BERT at all (aside from Table 2). Likewise, all the visualization in Appendix C focus solely on ELMo.\n\n**Suggestions**\n* In the section \"Limitations of original training objectives\", the authors state \"We also observed the same with BERT embeddings.\". This observation is central to the paper's argument for the need of the proposed approach in BERT, and elaborating on this with more analysis would have been nice.\n* While the motivation behind the various tricks adopted were very well explained (refer: Positives), ablations/experimental numbers showing the failure of the proposed approach without these tricks would have helped strengthen this quantitatively.\n\n**Questions:**\n* Does the proposed method have any impact on the monolingual task (for example, for en on CoNLL?)\n* The linear projection for cross-lingual alignment has been shown to not work on zero-shot sentiment classification. Do you have insight as to why this might be, why it adds noise to the embedding features?\n* Can the proposed method directly be used to improve Bilingual Lexicon Induction? Perhaps by using monolingual data to train models with the sense-aware cross entropy loss and dynamic pruning to obtain the different sense embeddings, and then aligning these sense embeddings (Eg: with MUSE). Or if a bilingual lexicon is available, performing joint training followed by a supervised BLI approach like RCSLS [3]. Would this help learn a many-many mapping for BLI (where normally 1:many/many:many are difficult to achieve precisely because senses aren't taken into account)?\n* Footnote 2 says that the dictionary can be learned in an unsupervised way (eg: using MUSE). Since the method here does not rely on a learned transformation, I'm assuming it was meant that we could use MUSE to generate the bilingual lexicon. How would this work with the proposed method, since unsupervised dictionaries generated with MUSE are usually 1:1/many:1 and tend to ignore multiple senses?\n\n**Minor details:**\n* Consider re-wording lines 7-10 in Algorithm 1 in terms of an argmax for conciseness, correctness and clarity\n\n[1] Peters, Matthew E., et al. \"Deep contextualized word representations.\" arXiv preprint arXiv:1802.05365 (2018).\n[2] Hadiwinoto, Christian, Hwee Tou Ng, and Wee Chung Gan. \"Improved word sense disambiguation using pre-trained contextualized word representations.\" arXiv preprint arXiv:1910.00194 (2019).\n[3] Joulin, Armand, et al. \"Loss in translation: Learning bilingual word mapping with a retrieval criterion.\" arXiv preprint arXiv:1804.07745 (2018).\n\n\n======================================================================\n\n**Update:**\n\nI would like to thank the authors for their response. The lack of adverse impact of the proposed approach on monolingual tasks and the described ablations certainly help strengthen things on the experimental side. \n\nHowever, my fundamental concerns still remain:\n* I'm unsure why the ELMo baseline is so much worse than that reported in literature.\n* With respect to the difficulty of training BERT, I can certainly empathize with the authors about the limited resources available in an academic setting. However, given that BERT was a key area of focus of the paper's methods section, showing the experimental results for BERT, even if BERT-tiny; note that BERT-tiny has just 2 layers and 128 hidden units, as opposed to 4 layers and 512 hidden units which this work uses (based on Section 4.1 and Table 7-- which corresponds to BERT-small, refer [here](https://github.com/google-research/bert#bert)), which should help reduce computational burden by quite a bit.\n\nOn account of this, I maintain my original rating of 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NLlXmEmVVoP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOverview\n===================\n\nThis is a really interesting paper, and it solves a big problem with cross-lingual representations: ignoring senses.  \n\nPros: Interesting model, relatively unexplored problem, the quantitative evaluations in the paper are reasonable\nCons: Lack of comparisons with unsupervised sense algorithms, no qualitative evaluation of senses\n\nStructure of algorithm\n===================\nThe structure of the approach reminds me of the Yarowsky algorithm, has an EM-like flavor for building up the sense distributions.  It would be useful and interesting to discuss why it has this structure.  It would be useful to discuss a little more about why this was chosen rather than an end-to-end model that does the clustering and sense selection together.\n\nIs this unsupervised?\n===================\n\nThe paper argues that it is unsupervised, \"Unlike these methods, our language models learn word senses in a fully self-supervised way\".  However, I don't think this tells the whole story.  It needs a *multilingual dictionary* to know how many senses a word has and if it has a translation.\n\nWhile MUSE (Conneau) provides unsupervised translations, the paper also adds  human-curated dictionary JMdict (including English, German, French, Russian and Dutch glosses), but the paper does not have a clear examination of the role of this additional supervision.\n\nGiven this level of supervision, it would be useful to have an explicit comparison with other supervised methods (e.g., have this model use WordNet or replace the other methods' dictionary with JMdict).\n\nWhat are comparable models?\n===================\n\nThere are, however, truly unsupervised sense induction models.  For instance, MUSE (Lee), GASI (Guo), and MSSG (Neelakantan).  Unfortuantely, this paper does not mention or compare with these models (they are, however, monolingual).\n\nWhat are the relevant tasks?\n===================\n\nWhile WSD is a relevant task, it would be useful to have sense in context (Huang)  and/or interpretability (Guo) as tasks.  These tasks better capture whether unsupervised senses can determine a word meaning (without supervised data from SemCor) or make sense to an end user.\n\nEither a formal or informal examination of what the sense clusters look like would help understand whether the algorithm is doing what it claims and to see if the output would be useful to humans (settings aside downstream tasks).\n\nWhile the NER and XNLI tasks are reasonable, I'd rather see sense-specific evaluations.\n\nRelated Work not Cited\n===================\n\nFenfei Guo, Jordan Boyd-Graber, Mohit Iyyer, and Leah Findlater. Which Evaluations Uncover Sense Representations that Actually Make Sense?. Linguistic Resources and Evaluation Conference, 2020.\n\nHuang, E. H., Socher, R., Manning, C. D., and Ng, A. Y.  (2012). Improving word representations via global context and multiple word prototypes. In Proceedings of the Association for Computational Linguistics.\n\nLee, G.-H. and Chen, Y.-N. (2017). MUSE: Modularizing unsupervised sense embeddings. In Proceedings of Empirical Methods in Natural Language Processing.\n\nNeelakantan, A., Shankar, J., Passos, A., and McCallum, A.  (2014). Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of Empirical Methods in Natural Language Processing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "59RfVDWzq9V", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\nResearch Problem: ELMo or BERT does not model word-sense explicitly, and it introduces challenges in representation learning, especially in the multilingual setting.\n\nThis paper proposes a novel approach to model word-sense explicitly for (masked) language models (refer as LM for simplicity purpose) in an unsupervised fashion. It maintains a set of sense vectors for each word and the sense vectors are updated with online clustering approach. The correct sense is selected with the contextual representation and the model is trained to predict the correct sense. Finally, a sense-level translation loss is proposed for bilingual models. This paper presents both English monolingual model and bilingual model with 4 language pairs. It outperforms models without proposed component on word sense disambiguation and cross-lingual transfer on three tasks.\n\n\n## Pros\n1. The proposed sense-aware LM learns to model word-sense without any supervision in monolingual LM.\n2. The proposed sense-aware LM and additional sense-level translation loss leads to improvement over regular bilingual models. In some cases, an extra projection step from previous work leads to further gain.\n\n## Cons\n1. This paper assumes the LM should be word-level, a relatively limited assumption when scaling to bigger corpus or more languages. It’s not straightforward to apply this approach to subword-level LM. It limits this paper to pretrain from scratch instead of fine-tune the existing BERT, multilingual BERT or XLM-R. It is not a problem for showcasing the proposed method in a controlled setting, but it limits the full potential of this paper, pushing state-of-the-art on top of the current best model. Additionally, this paper does not discuss the complexity of the proposed method.\n2. While the experiment presents some evidence that the proposed method learns to model word-sense in an unsupervised fashion, it does not include any result comparison with prior work on incorporating word-sense into LM. As a result, it’s unclear how well the model learns word-sense compared to models with supervision.\n3. It is unclear how much the sense-level translation loss contributes. This paper claims “... Bi-SaELMo is significantly better than Joint-ELMo, which shows that our sense-level translation pretraining objective improves cross-lingual embedding alignment”. However, the result presented in this paper does not support this claim. To support this claim, this paper should present **Bi-SaELMo without the sense-level translation loss**. Additionally, Bi-SaELMo assumes a bilingual dictionary compared to Joint-ELMo, yet the difference is not clearly discussed in the paper, especially in the result section.\n\n## Reasons for score\nOverall, I am leaning toward rejecting. While I find the sense-aware LM quite novel, the experiment presented does not support claim of contribution of sense-level translation loss. While the current result is promising, I find the limitation of the model not fully discussed and lack of comparison against other sense-aware LM in the monolingual setting. Hopefully the authors can clarify and address my concern in the rebuttal period. \n\n## After revision\nThank you for answering my question!\n\n> Our approach can be directly used to fine-tune pre-trained language models if word level tokenization is used.\n\nIt should be acknowledged that no multilingual word-based BERT exists as far as I know, partly due to the challenge with large vocabulary space and generalization. The sub-word approach you mentioned is a good proposal, but without evidence, I cannot assess whether it works. As a result, it's still a limitation of *this paper*, and should be clearly discussed in the paper.\n\n> The complexity of the proposed method.\n\nI am referring to the complexity w.r.t vocabulary size. The softmax normalization contains `nV` items, where `n` is the number of sense clusters and `V` is the vocabulary size. With the current limitation of word-based models, the vocabulary is already larger than sub-word models, the extra `n` factor cannot be disregarded.\n\n> We train bilingual language models without sense-level translation loss (and without projection), denoted by Bi-SaELMo-NT, for ablation study.\n\nThank you for the ablation study! However, I cannot assess it as support for the claim as it's not an apple-to-apple comparison, and I cannot evaluate this paper based on future projection. \n\nAs a result, I have to maintain my rating based on the revision.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0YS5yBhR9LJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to introduce multiple senses into pre-trained models. The proposed method selects senses dynamically while pretraining the model and applies a sense-aware cross-entropy loss for pretraining. This paper further proposes to jointly pre-train a sense-aware cross-lingual model with sense-level translation. The proposed model yields better performance than the baseline models under both monolingual and cross-lingual setting. \n\nStrength:\n\n1) The pipeline is designed very well and covers many aspects: taking care of the pruning while training; introduce projections to reduce parameters; from monolingual to cross-lingual setting. \n2) The paper is well written and easy to follow.\n3) Evaluation on three downstream tasks show significant improvements over baseline models, and for some case  (NER de), it performs even better than larger cross-lingual models.\n\nWeakness:\n\n1) Should cite more multi-sense papers (at least more papers before 2018). \n\nEspecially:\n\nhttps://www.aclweb.org/anthology/P12-1092.pdf \nhttps://www.aclweb.org/anthology/D14-1113/\n(both uses clustering methods to select senses)\n\nhttps://www.aclweb.org/anthology/N16-1160.pdf\nhttps://www.aclweb.org/anthology/D17-1034/\nhttps://www.aclweb.org/anthology/2020.lrec-1.214.pdf\n(all use similar softmax form to predict senses as the prediction task in your sense-aware cross-entropy loss)\n\n2) The proposed model outperforms the baseline models significantly.  However, the baseline models are pretty out-dated. And the scale is quite small. It is true that your model performs better on NER for DE than larger cross-lingual models. However, in most cases, it performs much worse than large-scale pre-trained cross-lingual models (https://arxiv.org/pdf/1901.07291.pdf). Have you tried to apply your methods to large-scale models? Is it possible that large-scale models have already captured the sense information within the context and the context provides enough information for disambiguations? Therefore introducing senses won't bring more capacity to the model?\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how to align word senses across languages. This has not been studied much as past work has primarily considered aligning word (embeddings) across languages. The paper is well written and well motivated. Unfortunately the empirical results are not very strong. The baselines are somewhat low and the gains are modest (the excuse that it is difficult to train BERT-sized models in academia is acknowledged). Overall, there is not enough support for acceptance at such a competitive venue as ICLR.

Decision: Reject