Massively Multilingual Sparse Word Representations
Authors: Gábor Berend
Venue:  (2020)
Keywords: sparse word representations
multilinguality
sparse coding

Abstract: In this paper, we introduce Mamus for constructing multilingual sparse word representations. Our algorithm operates by determining a shared set of semantic units which get reutilized across languages, providing it a competitive edge both in terms of speed and evaluation performance. We demonstrate that our proposed algorithm behaves competitively to strong baselines through a series of rigorous experiments performed towards downstream applications spanning over dependency parsing, document classification and natural language inference. Additionally, our experiments relying on the QVEC-CCA evaluation score suggests that the proposed sparse word representations convey an increased interpretability as opposed to alternative approaches. Finally, we are releasing our multilingual sparse word representations for the 27 typologically diverse set of languages that we conducted our various experiments on.

Published as a conference paper at ICLR 2020
MASSIVELY MULTILINGUAL SPARSE WORD REPRE- SENTATIONS
Gábor Berend
University of Szeged, Institute of Informatics Szeged, Hungary
2 MTA-SZTT RGAI, Szeged
Szeged, Hungary
Berendg@inf.o u- su-zeged
ABSTRACT
In this paper, we introduce MAMUS for constructing multilingual sparse word representations. Our algorithm operates by determining shared set of seman- tic units which get reutilized across languages, providing it a competitive edge both in terms of speed and evaluation performance. We demonstrate that proposed algorithm behaves competitively to strong baselines through a series rigorous experiments performed towards downstream applications spanning over dependency parsing, document classification and natural language inference. Ad- ditionally, our experiments relying on the QVEC-CCA evaluation score suggests that the proposed sparse word representations convey an increased interpretability as opposed to alternative approaches. Finally, we are releasing our multilingual sparse word representations for the 27 typologically diverse set of languages that we conducted our various experiments on.
INTRODUCTION
Cross-lingual transferability of natural language processing models is of paramount importance for low-resource languages which often lack sufficient amount of training data for various NLP tasks. A series of attempts have been made to remedy the shortage of labeled training data. Both part-of- speech tagging (Fang Cohn 2017 Zhang et 2016 Haghighi Haghiih & Klein 20066] Gouws & Spgaard, 2015 Kim et al. [015] Agié et al. 2015) and dependency parsing (Guo et al. [2155] Agií et al,
2016 have been investigated from that perspective. The mapping of distributed word representations of low-resource languages the embedding space of resource-rich language makes NLP models trained on the source language directly applicable for texts in resource-searce languages.
Overcomplete word representations aim expressing low dimensional distributed word representa- tions as a sparse linear combination of an overcomplete set of basis vectors. Using sparse word rep- resentations has only been reported to give better performance compared to dense representations for certain problems (Yogatama et al. 2015 Faruqui et al. 2015)
| Sun et al. 2016 Berend 2017), 7, but it is also argued provide increased interpretability (Murphy et al. 2012 Vyas & Carpuat 2016] Park et al. 2017 Subramanian et al. 2018). 1). Such bilingual sparse representations have straightfor- ward benents as any machine learning model that trained on the labeled training data using the sparse representation of some essurre-rrch source language can be used directly and interchange- ably for texts written in some target language. This way we can enjoy the benefits sparsity such as smaller and more interpretable models.
In this work, we propose a new algorithm for determining multilingual sparse word representations in which words with similar meaning across different languages described with similar sparse vectors. Our work has multiple advantageous properties compared to previous similar attempts (Vass& & Carpuat 2016 Upadhyay et al. 2018). Firstly, algorithm naturally fits multilingual setting, i.e. is not limited to pairs of languages, but capable of determining cross- -ongull sparse embeddings for multiple languages.
Our algorithm differs from previous approaches in that we obtain target language sparse word repre- sentations via series of convex optimization problems with substantially reduced parameter set,

---

Published as a conference paper at ICLR 2020
whereas previous solutions perform non-convex optimization for orders of more parameters. The model formulation we propose not only speeds up computation drastically, but our empirical results also suggest that the proposed sparse word vectors perform better when employed in various extrin- sie tasks, i.e. dependency parsing, document classification and natural language inference. We make sparse embeddings for 27 languages and the source code that used to obtain them publicly available at sttpp::/gittii com http///egabb/nnn mamus
MAMUS - THE PROPOSED ALGORITHM
In order facilitate the introduction of the MAssively Multilingual Sparse (dubbed as MAMUS) word representations, we first introduce our notation. In what follows, we denote dense word em- bedding matrices of the source and target languages as S E |= and T E R respectively, with V, and Vú indicating the vocabulary the source and target languages. The does not have to hold in general, however, it is going to be the case throughout the paper. We shall denote some symbolic word form as and its corresponding distributed vectorial representation in boldface, i.e. x.
condition
2.1 PREPROCESSING OF INPUT WORD EMBEDDINGS
Our algorithm takes as input pair of "traditionally" trained distributed dense word embeddings such as Glove (Pennington et al. 2014), word2vee (Mikolov et al.) (2013a) or fasttext (Bojanowski al. 2017). It important to note that we do not assume the existence of parallel text, meaning that we can train the input word embeddings in total isolation from each other.
We transform the dense input embedding matrices so that the word vectors comprising it have unit norm, as it is frequently met in the literature (Xing et al., 2015 Artetxe et al.) 2016 2017). This preprocessing step ensures that the dot product of word embeddings equals their cosine similarity. Unit normalization word emheddings also makes cross-lingual comparison more natural, since all embeddings have identical length, irrespective of the language they belong to.
2.2 MAPPING OF WORD EMBEDDINGS
In order align independently trained word representations, we learn linear mapping W. The mapping is expected to bring target language word vectors close to their semantically equivalent counterparts in the source embedding space.
As proposed by Mikolov et al. (2013b) such W can be simply defined by minimizing Σ |s: Wtall
with {(s,, 4)) being a set of word pairs which are cross-lingual equivalents of each other. If we construct matrices S,- = [si]/= and To [t]]/= by stacking the embeddings of translation pairs, we can express the solution as S,T,,, with T denoting the Moore-Penroee pseudoinverse of T.
be orthonormal can significantly improve the qual- ity of the mapping of word embeddings across languages (Smith et al.) 2017: Xing et al., 2015)) Artetxe et al., 2016 [Hamilton et al., 2016). Finding the optimal orthonormal W can viewed as an instance of the orthogonal Procrustes problem (Schönemann) 1966) which can be solved by W1 UV, with U and coming from the singular value decomposition of the matrix product STT, By applying isometric transformation, ensure that transformed embeddings preserve their norm.
Multiple studies have argued that ensuring
2.3 SPARSE CODING OF THE EMBEDDING SPACES
Sparse coding decomposes a X E R matrix signals (word embeddings in our case) into the product of dictionary matrix D mm and matrix of sparse coefficients Rx* where denotes the number of basis vectors to be employed. The columns D form an overcomplete set of basis vectors and the sparse nonzero coefficients in the ith column of indicate which column
2

---

Published as a conference paper at ICLR 2020
vectors from D should be incorporated in the reconstruction of x,. We minimize the objective
min 20
|| 2
Dall A/jai|1,,
where C. denotes the convex set of matrices with column norm at most and the sparse coefficients in a are required to be non-negative. Ensuring that all the coefficients in a are non-negative makes their cross-lingual comparison
natural, as the signs of sparse word coefficients cannot mismatch. Additionally, non-negativity has been reported to provide increased interpretability (Murphy et al.) 2012). We used the SPAMS library (Mairal al. 2009) for calculating D and a.
We can perform sparse coding as defined in for the independently created source and target emhedding matrices S and T, obtaining decompositions S DeO, and T z Da0 If we do so, however, sparse word representations extractable from matrices and a; are not comparable to any extent due to the fact that S and I' are decomposed independently of each other.
We propose to first solve single instance of the non-convex dictionary learning problem as de- fined in for the source language alone if goal was obtain monolingual sparse word representations. After determining D., we can apply this dictionary matrix to find sparse coefficient matrices for the isometrically transformed embeddings of all the target languages, hence solving much simpler complex optimization problem with reasonably fewer parameters in the case of target languages the form
min MKI
W - D,a + A||a||1
(2)
ETR
20
with W1 denoting the isometric transformation specific to a given source-target language pair. We summarize the working mechanisms of MAMUS in Algorithm |
Algorithm Pseudocode of MAMUS
Require: source and target embeddings R. Ti E Rmx|Vi TN E RRmx(Viy! (t)
semantically equivalent word pairs {(s)) 1/ (60 108,00 11))))
Ensure: Sparse representation matrices (ae, aty...., aix) procedure MAMUS(S, T)
S UNITNORMALIZE(S)
D 0 <- arg min||s Da:l + A||a/11
a.D
for k +- 1 to N do
Tg f- UNITNORMALIZE(TT
4
WT < arg min
Σ
|
Wt (1)
WTW=1 =1 =
042
f- arg min W WiTh - Diall + A||a||1
oER>o
end for return Oc Ot.. atN end procedure
2.4 DECISIVE DIFFERENCES TO BISPARSE
The goal of the BISPARSE (Vyas & Carpuat 2016) is to determine such sparse word representations that behave similarly for pair of languages. We subsequently juxtapose the conceptual differences between BISPARSE and MAMUS both from theoretical and practical perspectives.
BISPARSE (Vyas Carpuat 2016) determines sparse bilingual word representations by solving
M& M
min D....e 0.ER V.
2 (IX. 11 - +
Myyl(a
-a( -
+de aallaelll+++all + +Aollacallis
(3)
3

---

Published as a conference paper at ICLR 2020
where M is pairwise similarity matrix between the source and target language word pairs, A.. A and A denotes the regularization coefficients for the source, target language and the cross-lingual loss, respectively. Finally, a!!) and (f) refers to the ith and jth columns of matrices a, and ox. respectively.
As a first notable difference, BISPARSE operates by employing an explicit [V. x [V] pairwise similarity matrix, whereas MAMUS relies on list word translation pairs, which is used to rotate all the target language embeddings into the embedding space of the source language. BISPARSE hence requires an additional hyperparameter which controls for the strength of the cross-lingual regularization (ef. )²). According to experiments choosing this hyperparameter is crucial factor in employing BISPARSE.
The objective in (3) reveals that the original BISPARSE model optimizes the source and target word representations, jointly, i.e. a separate source language representation created whenever applied to a new target. This behavior BISPARSE limits its application for a single pair of languages, instead of the more general multilingual setting, when our goal is to represent more than two languages at time. As the default behavior of BISPARSE hampers truly multilingual application, we adopt similar to MAMUS, i.e. we first train sparse word representations for the source language by solving
1 I|,,, - a,a,,1i + >,11as111, 2
min D,ERmxA
E 20
kxIV,
with the source language being chosen English. Once the dictionary matrix of semantic atoms (D?) and the sparse coefficients (a²) are calculated for the source language, we next solve
IV.1 M ΣΣ =1
1 X, Draell?+ Aelloel1 +
Mi 10 at 1P11
min
A
(5)
4xI4 A D,ERuxk Ex
Identical to BISPARSE VVaa & Carpuat 2016), we solved the modified optimization problems in (4
and (5) relying the efficient Forward- forward-Backwand Backward Splitting Solver implementation of the FASTA framework (Goldstein et al. 2014 2015).
To summarize, the way BISPARSE works in its original formulation is that it creates sparse word representations for pairs of languages, both for the source and target language. One compelling characteristic of MAMUS, however, that it learns the D, and o, parameters independent from the target embedding space, thus the same decomposition of source language embeddings can be utilized for multiple target languages.
This makes our algorithm conveniently applicable to an arbitrary number of languages. Further- more, by using the dictionary matrix D; for all the target languages, MAMUS not only involves optimization over a substantially reduced set of parameters, it additionally enjoys the case of solv- ing a series a convex optimization problems. The model formulation of BISPARSE on the other hand is such, that it requires to solve a series of highly non-convex optimization problems for each of the target languages, since it requires the optimization over D; as well. This property is also true for the modified version of BISPARSE which is adapted for multilingual usage. We will refer to this adapted version of BISPARSE as MULTISPARSE for the rest of the paper.
3 EXPERIMENTS
Our primary source for evaluating our proposed representations is the massively multilingual eval- uation framework from (Ammar al. (0166), which also includes recommended corpora to be used for training word representations for more than 70 languages. All the embeddings used in our experiments were trained over these recommended resources, which a combination of the Leipzig Corpora Collection (Goldhahn et al. 2012) and Europarl (Koehn 2005).
For 11 languages (bg, cs, da. de, el. es, fi, fr, hu, it, sv) with ample parallel text to English bilingual dictionaries are also released part of the evaluation framework. For the remaining languages, Ammar et al. (2016b) released dictionaries that were obtained by translating the 20k most common English words with Google Translate. All the word representations involved in
4

---

Published as a conference paper at ICLR 2020
Table I: QVEC-CCA results on the English dev from
Ammar et al.
2016b as function of A.
X
0.1
0.2
0.3
0.4
0.5
English QVEC-CCA (development set)
0.811
0.806
0.804
0.803
0.799
Avg. nonzero coefficients per word form
22.9
9.1
5.2
3.4
2.3
experiments were trained over the previously introduced resources from (Ammar et al. 2016bb in order ensure the comparability of the different approaches.
We trained FasWtext-BBOW (Bojanowski et al. 2017) dense embeddings as inputs to our experi- ments. We simply used the default settings of fasttext for training, meaning that the original dense word representations were 100 dimensional. We conducted additional experiments with different input embeddings (word2vec and Glove) and also with different dimensionality (300), however, we omit these results for brevity as the general trends we observed were very similar. We set the number of semantic atoms in the dictionary matrix D consistently k 1200 throughout all our experiments.
3.1 MONOLINGUAL EXPERIMENTS
We first performed monolingual experiments in order to investigate the effects of choosing different regularization coefficients X, controlling the sparsity and the quality of sparse word representations.
Table [ reports the quality of our monolingual representations evaluated by the QVEC-CCA Tsvetkov al., 2016) correlation-based intrinsic evaluation metric, which was proposed as im- provement over the QVEC evaluation technique (Tsvetkov et al. 2015). The goal for both QVEC and QVEC-CCA is to quantify the extent to which embedding dimensions can be aligned human- interpretable concepts, such as word supersenses. The evaluation environment we utilize includes supersense tag annotations for Danish (Alonso et al., 2015, Martinez Alonso et al., 2016). English
Miller al. 1993) and Italian (Montemagni al. 2003).
During the monolingual experiments, we were solely focusing on the development set for English to set the hyperparameter controlling the sparsity of the representations. Table additionally contains the average number of nonzero coefficients yielded by the different regularization coefficients.
monolingual evaluation results from Table [ we decided to fix the regularization coefficient for MAMUS at 0.1 for all our upcoming mutlilingual experiments. As our choice for the value of À was based the performance achieved by MAMUS the development of single language and a single evaluation criterion, we chose this hyperparameter without the risk of overfitting to the test data regarding any of the multilingual evaluation scenarios.
Based
In the case of MULTISPARSE, we managed to obtain a similar performance and average sparse- ness (55.2 nonzero coefficient per word form) when utilizing X, 2. During our multilingual experiments, we set the target language regularization coefficient identically, i.e. A = 2. For the cross-lingual regularization term of MULTISPARSE, we chose Ag 5 based on our preliminary investigation based on the remaining two languages we had supersense tag annotations for, i.c. Dan- ish and Italian. The above set of hyperparameters resulted in 41.0 and 44.3 nonzero coefficients per word form on average, respectively.
3.1.1 MONOLINGUAL EVALUATION OF BISPARSE AND MULTISPARSE REPRESENTATIONS
We conducted an additional monolingual QVEC-CCA evaluation on Danish, English and Italian. In this experiment we compared the results of sparse word representations obtained by the vanilla BISPARSE formulation and its multilingual extension MULTISPARSE, This way effects of multilingual training as opposed to bilingual one.
assess the
The QVEC-CCA scores we obtained for the standard BISPARSE representations were 0.602, 0.789/0.786 and 0.585 for Danish, English and Italian, respectively. We report two English results (0.789/0.786) 786) as BISPARSE provided with two distinct representations for English depending the target language was jointly trained with. MULTISPARSE representations on the other hand obtained QVEC-CCA scores of 0.612, 0.808 and 0.596 for Danish, English and Italian, respectively.
5

---

Published as a conference paper at ICLR 2020
Our monolingual QVEC-CCA results suggest that the multilingual training objective performs com- parably (or even better) to the bilingual one employed in BISPARSE. A possible explanation for this that BISPARSE tries to optimize more difficult objective it involves the joint optimization of two non-convex problems with an increased number of parameters hence finding a good solution more difficult for BISPARSE.
3.2 MULTILINGUAL EXPERIMENTS
In the recent survey, Ruder et al. 2019) recommend to evaluate cross-lingual word embeddings on an intrinsic task and least one downstream task besides document classification. Following this suggestion, we measure the quality of word representations based on QVEC-CCA similar to our monolingual evaluation and also report their performance scores in cross- Pross-lingual document classification (CLDC), cross-lingual dependency parsing and cross-lingual natural language infer- ence (XNLI). Results from this point on are all obtained with the exact same hyperparameters as introduced before and on the test sets of the various tasks, unless stated otherwise.
3.2.1 BASELINE DENSE CROSS-LINGUAI. REPRESENTATIONS
Before delving into the multilingual experimental results, we introduce those additional dense dis- tributed cross-lingual representations that used in evaluation. These representations were trained over the same corpora and translation lists introduced earlier.
multiCluster (Ammar al. 601 builds graph from word forms
multiple languages with edges going between translated word pairs. Clusters are then formed based on the connected com- ponents of the graph. Word forms are finally replaced by the random identifier of the cluster they belong to and skip-gram model is trained for the corpus obtained this way.
multiCCA (Ammrr et al., 2016bb an extension of the approach introduced by Faruqui Dyer 22114. It seeks a linear operator which projects pre-trained word embeddings of some language to the embedding space of English such that the correlation between the representation of translated
word pairs is maximized.
Even though we trained our representations and further baselines on the same corpora and word translation lists as the multiCluster and multiCCA were built on, minor differences in the learned vocabularies exists (even between multiCluster and multiCCA) which are likely due to small dif- ferences in preprocessing. For this reason, pre-trained multiCluster and multiCCA allow for a less stringent comparison with the remaining representations, which were trained by ensuring to have identical vocabularies to the employed in approach. With this caution in mind, still think is useful to report evaluation results of these pre-trained embeddings as well.
MUSE (Lample et al., 2018) is capable of training dense looss-lnngull word embeddings in both unsupervised and supervised manner. We experiment with the supervised variant of MUSE which requires translated word pairs input similar to MAMUS. MUSE also incorporates method for it- eratively refining the list of word translation pairs that we relied on using its default parametrization.
3.2.2 MULTILINGUAL EXPERIMENTAL RESULTS
QVEC-CCA We next assess the quality of the different multilingual word representations accord- ing to the QVEC-CQRE CCC evaluation score. As mentioned earlier, the evaluation framework introduced by Ammar et al. (2016b) provides supersense tagging for Danish, English and Italian. In Table 2a we report the individual evaluation scores over the three subspaces and their combination, from which we can conclude that MAMUS has a clear advantage over all the alternative word representa- tion regarding this interpretability-onne oriented evaluation metric.
Downstream evaluations from Ammar et al. 2016b) We also performed 4-class cross-lingual document classification (CLDC) on newswire texts originating from the RCV corpus (Lewis et al.] 2004) over 7 languages (da, de, en, es, fr, it, sv). The model simultaneously tuned the training sections of the different languages as implemented in (Ammar et al. 20166). Table 2blletbbisst classification accuracies of the different methods.

---

Published as a conference paper at ICLR 2020
Table 2: Intrinsic (a) and extrinsic (b) evaluations from the test bed from (Ammar al. 2016b).
(a) Intrinsic QVEC-CCA evaluation scores of various cross- lingual word representations.
(b) Downstream evaluation results as average accu- racy for CLDC and UAS for dependency parsing.
da
en
it
ida,en.it)
CLDC
dependency parsing
multiCluster
0.475
0.539
0.468
0.372
90.79
61.39
multiCCA
0.501
0.635
0.510
0.415
92.18
62.82
MUSE
0.343
0.393
0.338
0.294
87.34
64.47
MULTISPARSE
0.612
0.808
0.596
0.480
86.45
59.20
MAMUS
0.632
0.819
0.620
0.503
91.84
63.53
Regarding our evaluation towards dependency parsing, we evaluated the transition-based stack- LSTM parser from (Ammar al.) (216a) over the Universal Dependencies v1.1 treebanks (Agič et al., 2015) covering 18 languages (bg, cs, da, de, el, en, es, eu, fa, fi, fr, ga, he, hr, hu, id, it, sv). Even though the parser is capable of incorporating additional features besides word embeddings, this capability of the parser was disabled, so that the effects of employing different word represen- tations can be assessed their own. The performance metric for parsing is reported in Table 2 unlabeled attachment score (UAS).
Table 2b illustrates that the performance obtained by dense representations varies largely between downstream tasks. The performance of MUSE, for instance, is the best when evaluated on depen- dency parsing, however, has the second lowest overall accuracy for CLDC. Evaluation scores for those models that are based on MAMUS representations rank second on both downstream evaluation tasks with minor performance gap to the best results obtained by different dense representations on the two tasks.
Natural Language Inference order to assess the capabilities the different representations towards natural language inference (NLI) in multilingual setting, also performed evaluation towards the XNLI dataset (Conneau et al. 2018). XNLI covers 15 languages and it can be viewed multilingual extension of the multiNLI (Williams et al.) 2018) dataset. The task in XNLI and multiNLI is to categorize sentence pairs comprising of a premise (p) and a hypothesis (h) sentence whether the relation between p and h is entailing, contradictory neutral.
We implemented a simple multilayer perceptron in Py ToTch v1.1 (Paszke et al. 2017) with two hidden layers employing ReLU nonlinearity. The MLP uses the categorical cross-entropy for loss function, which was optimized by Adam (Kingma & Ba, 2014). Based on the differently con- structed sparse and dense word representations, English multiNLI dataset and report average classification accuracies as the performance score. The five models only differed in the random initialization of the parameters account for the potential variability in model performances.
train five different NLI models based the
For pair of premise and hypothesis sentence representation pair (p. h),, the input to the MLP gets determined according to
[p: h: - h: 3 h). similar to WWilliams et al. 2018) with standing for vector concatenation and @ denoting element- wise multiplication. The vectorial representations for the individual sentences were also obtained in an identical way to the approach (dubbed as X X-CBOW) in (Conneau et al.) 2018), i.e. we took the mean of the vectorial representations of the words comprising sentence.
We report the average accuracy achieved by the different word representations for multiNLI (Ta- ble (al and the 15 languages of XNLI (Table 3b). Although the training corpora compiled by Ammar et al. (2016b) contains datasets for all the XNLI languages, the authors did not release any pre-trained multiCluster and multiCCA embeddings for Hindi (hi), Thai (th), Urdu (ur) and Vietnamese (vi). As result, we do not report multiCluster and multiCCA results for these four languages.
We can in Table 3a that according to the evaluation conducted on the development set of multiNLI - multiCCA performed the best (similar to CLDC), and the English subspace of the MA- MUS representations achieved the second best results. Table Sablee3h represenn the average performance

---

Published as a conference paper at ICLR 2020
Table 3: Averaged multiNLI and XNLI performances of 5 independent MLPs.
(a) Average multiNLI matched dev set accuracy of the five models used during XNLI evaluation.
multiCluster multiCCA
MUSE
MULTISPARSE
MAMUS
57.74
63.43
57.10
60.22
61.43
XNLI results obcained. Results bold the best given language. underlined indicate the second
hg
de
el
fr
-WW
hi
th
vi
multiCluster
moltiCCA
38.85
401.9
34.50
42.70
65.43
36.16
33.83
41.25
38.31
38.82
45.36
MUSE
42,83
45.64
41.60
46.24
58.39
44.53
43.64
44.70
37.53
42.21
44.57
38.98
33,50
38.57
35,92
MULTISPARSE
33.10
35.04
38.17 17
33.33
60.10
40.48
40.92
33,34
37.78
33.23
33,06
34.18
34.63
MAMUS
42.67
47.88
44.27
49.58
61.53
51.19
48.93
43.89
37.09
47.24
44.73
41.78
34.59
42,00
34.94
Mamuu
mISKRT
0.5
0.4
0.3
0.2
DW
20
hi th or VI
Figure 1: Comparison of the accuracies over the 15 languages of the XNLI datasets.
of five models that are purely trained on the English multiNLI training data relying on the various multilingual word representations upon evaluation. It can be seen in Table (3b that other than for Swahili the models trained on the English MAMUS representations perform either the best or the second best 14 out the 15 languages.
In order to account for the recent dominance of contextualized word representations, we additionally compared the application of MAMUS to that of multilingual BERT (Devlin et al. (2019) when uti- lized for XNLI. We employed multilingual BERT (mBERT) representations from Wolf et al. (2019) identically to how we used the con-contextualized word representations previously. That is, we took the average of the contextualized representations that derived from the last layer of mBERT and trained 5 MLPs with different weight initializations. Similar to our previous experiments, we con- ducted evaluation in the zero-shot setting for the 15 XNLI languages, i.e. by solely using multiNLI for training our models. The comparative results of the trained MLPs can be seen in Figure [
Figure [ reveals that the MLPs trained top of MAMUS representations are capable of achiev- ing more than 90% relative performance on average to those MLPs which rely on mBERT. Indeed, the average performance of the MLPs over all the languages and random initializations is 49.3 for mBERT, while MaMus-based MLPs had an average accuracy of 44.8. We thus think that MAMUS can be a viable alternative mBERT in multilingual settings when a small fraction of accuracy can be sacrificed for obtaining model in which the calculation of sentence representations is sub- stantially cheaper (ef. the computational requirements of averaging static word embeddings and contextual embeddings which require forward pass in transformer architecture).
Interestingly, when inspecting results for English in Figure we can see that MLPs taking MAMUS representations as input actually perform better than those utilizing mBERT by fair margin (61.5 versus 58.3). This could imply that our algorithm has the potential to transform static dense embed- dings in way which makes them useful contextualized mBERT representations for certain applications. Note that this hypothesis would require further investigation, that is left for future work.
8

---

Published as a conference paper at ICLR 2020
mocedd Vamus toporse
necled
nethol Viauu mufrasern
Vamus mupapore
70-
nomm
co-
Ky numm
10-
Prequency bin (3k word per bin)
Frequetic bin (3k word per bi))
Frequency sin (ilwoord per sir)
(a) da
(b)
(C)
Figure 2: The sparsity structure of the learned representations for the languages covered in the multilingual QVEC-CCA evaluation as a function of the frequency word forms. Plots also include the frequency binned average norms of the dense input embeddings. The orange curve indicates the norms of the embeddings per frequency bins, the blue are for the number of nonzero coefficients per word forms.
3.2.3 ANALYSIS OF THE SPARSITY STRUCTURE
As final assessment of the sparse word representations, we characterize their number of nonzero coefficients as function of the frequency of the words they correspond to.
From a human cognition point of view, we can argue that frequency can be a good proxy to the specialization in the meaning of word (Caraballo & Charniak 1999). Words with high frequency, e.g. car, dog and newspaper tend to refer to easily definable concepts, whereas less frequent words, such as gymnosperms or samizdat have more complex- hence more difficult to describe meaning.
In terms of sparse coding, this could be reflected by the fact that words with more complex meaning would rely on more semantic atoms from the dictionary matrix D. Encoding less frequent words with more bits can also be motivated from an information theoretic point of view.
In Figure we plot the average number of nonzero elements over the entire vocabulary of the Danish, English and Italian subspaces (grouped into bins of 1,000 words). We omit similar figures for the rest the languages for space considerations.
Figure 2 reveals that the sparsity structures induced by the different approaches differ substantially. MAMUS tends to behave more stable across the languages, i.c. tends to assign a similar amount of nonzero coefficients for words in different languages that belong the same frequency bin. The number of nonzero coefficients per a word form determined by MULTISPARSE, however behaves less predictably.
Figure 2 also illustrates that the sparsity structure of MULTISPARSE is highly influenced by the of the dense input embeddings. which are known be typically higher for more common word forms (Turian et al. 2010). Representations determined by MAMUS, however, tend to assign more nonzero coefficients to less frequent hence arguably more specialized word forms.
4 RELATED WORK
common technique employed overcome the absence of labeled training data apply cross-lingual projections to port the missing linguistic annotation for low-resource target languages (Yarowsky & Ngai 2001 Das & Petrov] 201 Tāckström et al., 2013) Agic et al.) [2015] Agié al.) 2016). Such projections often determined by word alignment algorithms which imply that these kind of approaches inherently require substantial amounts of parallel text.
There has been series of research conducted for handling low-resource languages. A dominant approach is to rely on some form of dictionary between a low-resource and a resource-rich language which can be used to perform canonical correlation analysis (Kim et al.) 2015), directly incorporated into the learning procedure of word embeddings (Gouws & Søgaard 2015) or use in post-hoc manner map independently trained word embeddings (Fang & Cohn 2017 Zhang et al. 2016).

---

Published as a conference paper at ICLR 2020
Model transfer techniques, in which initial model is trained on resource-rich language and adapted for a target language in a sumi-sspeee or unsupervised manner, are also popular (Fang Zhang et al. 2016).
& Cohn 2017:
There is massive research interest in transforming word representations such that they become com- parable across languages with little no supervision (Zhang et al. 201 Artetxe et al.) 2017 Smith et a 2017 Lample et al. 2018: Joulin et al. 2018). There has been a myriad of further
techniques introduced for determining cross-lingual distributed word representations (Klementiev
ct al. 2012: Hermann & Blunsom 2014 Faruqui & Dyer, 2014, Huang et al. 2013 Luong et al.
2015 Gouws et al. 2015 Vulić & Moens | Ammar et 2016b). inter alia. The proposed
approaches differ widely in the assumptions they make regarding the amount of available parallel
or comparable data for determining bilingual word embeddings. [Upadhyay et al. (2016) and Ruder
et al. (2199) provide extensive overviews on the available approaches.
Contrary to these methods, our algorithm relates to the line of research focusing on sparse word representations. Such sparse word representations have been shown to outperform dense word em- beddings in monolingual settings (Murphy et al. 2012 Yogatama et al. 2015: Faruqui et al. 2015
Berend 2017 Sun et al. 2016 Subramanian et al.) 2018).
Vyas & Carpuat 2016) proposed BISPARSE for obtaining comparable sparse word representations in bilingual setting. As mentioned earlier, BISPARSE needs to be modified in order to be applicable for obtaining multilingual representations. Perhaps even more importantly, even when applied the bilingual case, BISPARSE needs to perform highly non-convex optimization problem and for substantially larger number of parameters in contrast to MAMUS. [Upadhyay et al. (2018) extends (Vyas & Carpuai 2016) by additionally incorporating dependency relations into sparse coding.
CONCLUSIONS
In this paper we introduced MAMUS for determining cross-lingually comparable sparse word repre- sentations. Our model formulation allowed us to solve a series of convex optimization problems per each target language, which resulted more favorable overall training time hours for MAMUS as opposed to 300 hours when using MULTISPARSE) over the 27 languages we conducted our evalu- ations on. Finally, we make our multilingual sparse embeddings for 27 languages publicly available at httpt:::///// http::///gtttbb com com/bggbbbbbbbbbb............... haaegggabmmmmusss//// mamus
ACKNOWLEDGEMENTS
This research was partly funded by the project "Integrated program for training new generation of scientists in the fields of computer science", no EFOP-3663-VEKOP-16-20
supported by the EU and co-funded by the European Social Fund. This work was in part supported by the National Research, Development and Innovation Office of Hungary through the Artificial Intelligence Na- tional Excellence Program (grant no.: 118-1.2.1-NKP 201800000 8-00008). This research was supported by grant TUDFO/47138 1/2019-ITM of the Ministry for Innovation and Technology, Hungary.
REFERENCES
Zeljko Agié, Dirk Hovy, and Anders Søgaard. If all you have is a bit of the Bible: Learning POS taggers for truly low-resource languages. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 268-272, Beijing, China, July 2015. As- sociation for Computational Linguistics. URL http:/ www.
P15-2044
Željko Agić, Maria Jesus Aranzabe, Aitziber Atutxa, Cristina Bosco, Jinho Choi, Marie-Catherine de Marneffe, Timothy Dozat, Richard Farkas, Jennifer Foster, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Yoav Goldberg, Jan Hajič, Anders Trærup Johannsen, Jenna Kanerva, Juha Kuokkala, Veronika Laippala, Alessandro Lenci, Krister Lindén, Nikola Ljubešić. Teresa Lynn, Christo- pher Manning, Héctor Alonso Martínez, Ryan McDonald, Anna Missilä, Simonetta Monte- magni, Joakim Nivre, Hanna Nurmi, Petya Osenova, Slav Petrov, Jussi Piiullainen, Barbara Plank,
10

---

Published as a conference paper at ICLR 2020
Prokopis Prokopidis, Sampo Pyysalo, Wolfgang Seeker, Mojgan Seraji, Natalia Silveira, Maria Simi, Kiril Simov, Aaron Smith, Reut Tsarfaty, Veronika Vineze, and Daniel Zeman. Univer- sal dependencies 1.1, 2015. URL http:: h //hhl/hh...... www.dlanneeeeee..ae net wwt.enenn///ttt..... 1478 LIN- DAT/CLARIN digital library at the Institute of Formal and Applied Linguisties (ÜFAL), Faculty of Mathematics and Physics, Charles University.
Zeljko Agié, Anders Johannsen, Barbara Plank, Héctor Alonso Martínez, Natalie Schluter, and An- ders Søgaard. Multilingual projection for parsing truly low-resource languages. Transactions of the Association for Computational Linguistics, 4:301-312, 2016.
Héctor Martínez Alonso, Anders Johannsen, Sussi Olsen, Sanni Nimb, Nicolai Hartvig Sørensen, Anna Braasch, Anders Søgaard, and Bolette Sandford Pedersen. Supersense tagging for Danish. In Proceedings of the 20th Nordic Conference of Computational Linguistics, NODALIDA 2015, May 11-13, 2015, Institute of the Lithuanian Language, Vilnius, Lithuania, pp. 21-29, 2015. URL attp:://acweet wwtp:/tw...eeeeee..oo.....oooeee harh//nnhhllggyy /ttp:////111111.1110
Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. Many lan- guages, one parser. Transactions of the Association for Computational Linguistics, 4:431- 444, 2016a. ISSN 2307-387X, URL http::/wwwwttt.aaal.ccc wtt.:/aanaaaaa..ooo////////a/aaa oarpaa/a....o///o///mmmmeepp..pp
facl/article/view/S5
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. Massively multilingual word embeddings. CoRR, abs/ 602.01922 2016b.
Mikel Artetxe, Gorka Labaka, and Eneko Agiree. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289-2294, Association for Compu- tational Linguistics, 2016. doi: 10.18653/ URL http: /www///wwwww.../ acwwwelweee.. org/ anthology/DI6 1250
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 451-462. Association for Computational Linguistics,
2017. doi: 10. 10.18655/1/117--0022 URL [att:ttp://// //ac ac web. wwweb.orgg/athhooggg//eeee
Gábor Berend. Sparse coding of neural word embeddings for multilingual sequence labeling. Trans- actions of the Association for Computational Lingwistics, 5:247-261. 2017. ISSN 2307-387X.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association of Computational Linguistics, 5:135-146,
2017. URL URL http:: / //wp://wwwwwwww.. wwwp:w.wwwwls....... wwwwwb.wwwb.......... org/ anthology/O17--110
Sharon A. Caraballo and Eugene Charniak. Determining the specificity of nouns from text. In
Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP 1999, College Purk, MD, USA, June 21-22, 1999, 1999. URL https://
aclanthology wnt.:papprss////paomaaaaaa.aassa W99 0609/ w 0609
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk. and Veselin Stoyanov. XNLI: Evaluating cross- lingull sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2485-2485, Brussels, Belgium, actober-November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/D18-11 D18 1269
URL https::t/// www.wb.wwwwbbb.....m.m www.aeee....gg/ttttooooooo/////
Dipanjan Das and Slav Petrov. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Volume 1, HLT '11, pp. 600-609, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics. ISBN 978-1-932432-87-9 URL http:/ /d1. a cm. org/ ci tat on cfm?: 2tt.

---

Published as a conference paper at ICLR 2020
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Lng and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1883/v1//199111 1423.. URL https:
7/ 7/www. www.wa.wwwb........ org/ wttp://rr/.atooogo/////..3/ 1423
Meng Fang and Trevor Cohn. Model transfer for tagging low-resouree languages using a bilingual dictionary. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( VVolume 2: Short Papers), pp. 10p.587-592 Association for Computational Linguistics, 2017. doi: 10.18653/v 1/P17-2093,
URL htthttp: www. wwwwwb.wwwwb..... http://wwww.....aaoooooo/////e// http://wwww///.aaaaaa/aaaaaaaaar
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association
for Computational Linguistics, pp. 462-471, Gothenburg, Sweden, April 2014. Association for Computational Linguistics. htR.http::// wwwpwwww.../ ac web.org. peaorg/anthht ogy
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. Sparse overcom- plete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1491-1500, Beijing, China, July 2015. Association for Computational Linguistics. URL wttp::tpp///. / www.//ww....///////////./.../w// ac web. www.oogg//aaoooooo /P15 1144
Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. Building large monolingual dictionaries at the Leipzig Corpora collection: From 100 to 200 languages. In In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC' 12, 2012.
splitting Tom Goldstein, Christoph Studer, and Richard Baraniuk. A field guide to http://arxiv.
with FASTA implementation. arXiv eprint, abs/1411.3406, 2014. URL
org/abs/ 1aaabb//11.... 3406
Tom Goldstein, Christoph Studer, and Richard Baraniuk. FASTA: A generalized implementation of splitting, January 2015. http:///wwii.orr///////0001././/
Stephan Gouws and Anders Søgaard. Simple task specific bilingual word embeddings. In Proceed- ings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisticy: Human Language Technologies, pp. 1386-1390, Denver, Colorado, May-June 2015.
Association for Computational Linguistics. URL http: /ttp://wwwww...... www. aclweb. org/anthology/ 015--1577
Stephan Gouws, Yoshua Bengio, and Greg Corrado. BilBOWA: Fast bilingual distributed rep- resentations without word alignments. Proceedings of the 32nd International Conference
on Machine Learning, 2015. URL http://3mlr.on http://mml...rggppreeeeeee http://roceediicoo ngs/ patgs//pppr///3/
pdd
Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. Cross-lingual depen- dency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computationnal Linguistics and the 7th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pp. 1234-1244, Beijing, China, July 2015. Association for Computational Linguistics. UtL:ttE::/ wwww
www.://wwwwwwww...b..b.......... aclweb. org/anthology/
P15-1119
Aria Haghighi and Dan Klein. Prototype-driven learning for sequence models. In Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. HLT-NAACL '06, pp. 320-327, Stroudsburg, PA, USA, 2006. Association for Computational Linguistics. doi: 10.3115/122035, 1220876. URL https: http:////...........111111 0ttpr///...111//20333333333
1220876
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal sta- tistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1489- Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. URL http:/ / www. aclweb. org/ anthelogy/P16-112
12

---

Published as a conference paper at ICLR 2020
Karl Moritz Hermann and Phil Blunsom. Multilingual Models for Compositional Distributional Semantics. In Proceedings of ACL, June 2014. URL httt:://axxiv.org/abaa
4641
Kejun Huang, Matt Gardner, Evangelos Papalexakis, Christos Faloutsos, Nikos Sidiropoulos, Tom Mitchell, Partha P. Talukdar. and Xiao Fu. Translation invariant word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1084-1088
Lisbon, Portugal. September 2015, Association for Computational Linguistics. URL http:
7/ac 7/aclweb.. org/ wtargg/aahhooooyy/--111
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in
translation: Learning bilingual word mapping with retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2979-29844 As- sociation for Computational Linguistics, 2018. URL http:://alwwb.....// otte wrt.ortttggggyy/////
D18-1330
Young-Bum Kim, Benjamin Snyder, and Ruhi Sarikaya. Part-of-speech taggers for low-resource languages using CCA features. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 1302, Lisbon, Portugal, 2015. Association for Compu- tational Linguistics. URL http::/aclwe.. aalaaclwweb wttp:wewwwrg.eeeeeeooooeeooeeoee www.g//anoology/////oooo Detp://1111111111
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed represen- tations of words. In Proceedings of COLING 2012, pp. 1459- 1474. The COLING 2012 Organiz- ing Committee, 2012. URL httttttp::/// wwwp///www.../ ac wttp://wwwwwwwowowooggg.ooooo
Philipp Koehn. Europarl: Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pp. 79-86, Phuket, Thailand, 2005. AAMT,
AAMT. URL URL http:::/tttt/eteetee wtw.chhiii..iioo/onommmm 2tw.://00S--0000oo.....oo
Guillaume Lample, Alexis Conneau, Mare Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018.
96sainb
David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. Rev1: A new benchmark collection for text categorization research. Mach. Learn. Res., 5:361-397, December 2004. ISSN 1532-4435. URL http::ttp//ll.... acm..aaccm...mmmmmmmmmmmmmmmm..m org/ ci witatiiiiiicccocccc ion. on, cmm? i d= 005332. 1005345
Thang Luong, Hieu Pham, and Christopher D. Manning. Bilingual word representations with mono- lingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 151-159, Denver, Colorado, June 2015. Association for Computational Linguistics. URL http::tppp/. www. ac web.org..ooooo....... wwt.oorg//ttoloo///.o/ /155- 1521
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 689-696, New York, NY, USA, 2009. ACM. ISBN 978-1-605558 516-1. doi: 10. 145/ 1553374.1553463 Utthhtp:://do/ acm org / http://...111115111
1553463
Hector Martinez Alonso, Anders Trærup Johannsen, Sussi Olsen, Sanni Nimb, and Bolette Sandford Pedersen. An empirically grounded expansion of the supersense inventory. In Proceedings of the 8th Global Wordner Conference, pp. 199- 009-208,44 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word repre- sentations in vector space. CoRR, abs/1301.3781, 2013a. URL http:t//rrx/i..ooo org. abs/
13C1.3781
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. CoRR, abs/ 3309.4168, 2013b. URL httpht/p:///b/ uni- -riirrr de de/db/ db/ 309. html #Mi 1ol.v113
Journa 1 s/
13

---

Published as a conference paper at ICLR 2020
George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. A semantic concordance. In Proceedings of the Workshop on Human Language Technology, HLT '93. pP. 503-308, Strouds- burg. PA. USA, 1993. Association for Computational Linguisties. ISBN 1 1-55860-324-7. doi: 10.3115/1075671. 107544747777 1075742. URL https:///daao org.10..r011000 wtt.://////.1111100051 httpp//////o...or///.....1/////0 1077774
Simonetta Montemagni, Francesco Barsotti, Marco Battista, Nicoletta Calzolari, Omella Corazzari, Alessandro Lenci, Antonio Zampolli, Francesca Fanciulli, Maria Massetani, Remo Raffaelli, al. Building the italian syntactic-semante semantic treebank. In Treebanks, pp. 290- Springer Nether- lands, 2003.
Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-ne non-negative sparse embedding. In Proceedings of COLING 2012, pp. 1933-1950, Mumbai, India, December 2012. The COLING 2012 Organizing Committee. URL http://
www. ac web. www.oortggoooooooooooooo ottp://nnntooooy//1111113
Sungjoon Park, Jin Yeong Bak, and Alice Oh. Rotated word vector representations and their in-
terpretability. In Proceedings of the 2017 Conference Empirical Methods in Natural Lan- guage Processing, pp. 401-411, Association for Computational Linguistics, 2017. RRLhtt:
//www.aclweb.ok www.wb.wwebb.... org/ wttha/ooyo///o110000000000 htwp://cwwwb.orrrraaaaaooooo/aoa
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. ptt.ppp..15513311 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. URL http:: www. acl www.wwwwwww...w................. org/ /tt/aaatthooooy11-l11/a http:///wwww....ttoottooooorttrr
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. A survey of cross-lingual word embedding mod- els. J. Artif. Int. Res., 65(():5((9666666))))))))) 630, May 2019. ISSN 1076-9757. doi: 10. 1...a//.......... 1160.
1600
Peter H. Schönemann. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31(1), March 1966. doi: 10. 1000//////222 URL
com/
Lndex/10.100747 02289451
Samuel Smith, David H. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), 2017.
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg- Kirkpariikkkkkk and Eduard H. Hovy. SPINE: sparse interpretable neural embeddings. In Proceedings of the Thirty-Second AAAI Con- ference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelli- gence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 4921-4928, 2018. URL
https:/ / www. aaai. org/ ocs/ index.ndixc..o phppppppat httpa//////A///a/p/eeeeeee/////a /
Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng. Sparse word embeddings using & regularized online learning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 2911-2921, New York, New York, USA, 9-15 July 2016. AAAI Press / International Joint Conferences on Artificial Intelligence.
Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan T. McDonald, and Joakim Nivre. Token and type constraints for cross-lingual part-of-speech tagging. TACL, 1:1-12, 2013. URL https:
///ac12013.com co ccolumbia edu/ /acc//jjs/ii Index. http:ppacp/aattteeeee/eeeeaeae/e
Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. Evaluation of word vector representations by subspace alignment. In Proceedings of the 2015 Confer- Empirical Methods in Natural Language Processing, pp. 9049-205e Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 111.6653/1/DD5-... 1243. URL https: / www. c web. wwweb.org/antholooooooo htwpchhooooyyy/oo
14

---

Published as a conference paper at ICLR 2020
Yulia Tsvetkov, Manaal Faruqui, and Chris Dyer. Correlation- intrinsic evaluation of word vec- tor representations. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representa- tions for NLP, pp. 111-115, Berlin, Germany, August 2016. Association for Computational Lin- guistics. doi: 10. 10.18655/11/111-25552 URL https:: /wwww a acl web.. org/ anthool ogy/
W1 216--220)
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the As- sociation for Computational Linguistics, ACI. '10, pp. 384-394, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. URL hww.: /tte://ddl htl//dllaccc aac. wwwp//iaatitiinoooo/// cfm?
1858721
Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. Cross-lingual models of word em- beddings: An empirical comparison. In Proceedings of the 54th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 1661-1670, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http: / www. . acl web. org/ anthology/PI6- 1157
Shyam Upadhyay, Yogarshi Vyas, Marine Carpuat, and Dan Roth. Robust cross- Prossllnngual hypernymy detection using dependency context. Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 607-618. Association for Computational Linguisties, 2018. doi:
10.18653/v I/N18- 1056. URL http: / http://acllbb/// org anthologyyy/a/aaaoooooooolloooaa NN18 1056
Ivan Vulié and Maric-Francine Moens. Bilingual word embeddings from non non- parallel document- aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pp. 719-725. Association for Compu- tational Linguistics, 2015, doi: 10.3115/11/15--211.. URL http: /twp///wwww//.////////////ww awwweb. org/
Yogarshi Vyas and Marine Carpuat. Sparse bilingual word representations for cross-lingual lexi- cal entailment. In Proceedings of the 2016 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pp. 1187-1197.
San Diego, California, June 2016. Association for Computational Linguistics. URL http:
// www. aclweb. org/ anthologyyoooooooooooo /
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Pupers), pp. 1112-1122 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653//1/188-10000 URL https://////... www aclweb. Motthology/K181111
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: Sfate-of-the-ar natural language processing, 2019.
Chao Xing, Dong Wang. Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pp. 1006-1011, Denver, Colorado, May-June 2015. Association for Computational Linguisties. URL http: / http://cwwee..rrraaaaaoooooo////
-1104
David Yarowsky and Grace Ngai. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, NAACL '01, pp. 1-8. Stroudsburg, PA. USA, 2001. Association for Computational Linguistics. doi: 10. 3115/1073336.1 3115/1073336.1072223 10733333 URL https: /ww. 1ttp:/........11111111111110.... 3w1.11111111....1..1 10733662.1.77
Dani Yogatama, Manaal Faruqui, Chris Dyer, and Noah A. Smith. Learning word representations with hierarchical sparse coding. In Proceedings of ICMI., 2015.
15

---

Published as a conference paper at ICLR 2020
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Earth mover's distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empir- ical Methods in Natural Language Processing, pp. 1934-1945, Copenhagen, Denmark, Septem- ber 2017a. Association for Computational Linguistics. URL https: /www///wwww//w/////////////// ac web. org/ anthollogy/DI7-122
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial training for unsuper- vised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume Long Papers), pp. 1959-1970, Vancouver, Canada, July 2017b. Association for Computational Linguistics. URL /ttp::///wwwwbbb org/
anthology/P17-111
Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. Ten pairs to tag -ag--mulilinnn multiiininul POS tagging via coarse mapping between embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1307-1317, San Diego, California, June 2016. Association for Computational Linguisties. URL ht tttpp:/Ai www. ac web. org/ wwt./anthooogg//l....... 1156
16


Reviews:
[{"review_id": "rJe0V3HRKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to generate sparse multilingual embeddings. The key idea is to build only one set of source basis embeddings, and then represent all multilingual embeddings as a linear combination of these source embeddings. I felt the paper is a nice extension of the Vyas 2016 paper. Compared to existing approaches, their method will be faster to train and will need less data (particularly useful for low resource languages). Since I am less aware of work in this area, I cannot comment on whether the evaluation is complete. Particularly, I wonder if there is a qualitative way to show interpretability of the sparse vectors created by the method. We currently only have QVEC-CCA numbers to judge interpretability.  A few more suggestions:\n\n1. I got confused by the sentence \".. over a reduced number of parameters for each target language as it treats D_s as D_t .\". Things got clear from the equations, but will be good to fix.\n\n2. Figure 1 was very difficult to understand. Ideally your caption should be enough to understand the figure. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklpIWaPqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to build sparse multilingual word vectors that is designed to scale easily to many languages. The key idea is to pick one language to be the source language, and then to build word embeddings and then a sparse dictionary + sparse coefficients  for that source language monolingually. Other target languages then first align their embedding to the source using a seed list of translations and standard techniques, and then determine their sparse coefficients based on the fixed source sparse dictionary. This latter process is a convex optimization, which improves efficiency and stability. The method is tested with an established correlation-based intrinsic metric (QVEC-CCA) as well as by using the multilingual embeddings to project systems for cross-lingual document classification, dependency parsing and natural language inference.\n\nThe core idea of this paper is substantially simpler than the method it compares to (BiSparse), which jointly optimizes dictionaries, coefficients and cross-lingual coefficient alignment. So, the question that immediately comes to mind for me is, how much accuracy am I giving up for improved scalability to multiple languages? This could be easily tested for the two language case, where BiSparse could be directly compared without modification to their proposed method. Instead, BiSparse is modified to fit scale to the multilingual setting (becoming MultiSparse), but since it shares the constraint that the source dictionary and coefficients are fixed, it has already lost a lot of the power of joint optimization. I think the paper would be stronger with a two-language experiment where we would expect the proposed method to lose to BiSparse, but we could begin to understand what has been given up for scalability.\n\nI also wonder how relevant bilingual word embeddings are in a world of multilingual BERT and similar approaches. It would be interesting to know how cross-lingual embedding-in-context methods would do on the extrinsic evaluations in this paper, though I also acknowledge that this could be considered beyond the scope of the paper.\n\nOtherwise, this is a fine paper. It is well written and easy to follow. The experiments look sane, and the inclusion of both intrinsic and extrinsic tasks makes them fairly convincing. I have only a few remaining nitpicks:\n\n(1) As someone relatively unfamiliar with multilingual (as opposed to bilingual) word embedding research, it wasn’t clear to me how the experiments described here tested the multilinguality (as opposed to bilinguality) of the embeddings. It would be nice to provide an explanation for why (beyond the obvious efficiency gains) one couldn’t just do the necessary language pairs with bilingual methods for these experiments. And if one could perform the tests with bilingual methods, they should be included as baselines. \n\n(2) The discussion of MultiSpare hyper-parameter tuning appears in the Monolingual Experiments section, leading me to wonder what target languages were used for this tuning process.\n\n(3) In the second-last paragraph of 3.2.2, there is a sentence fragment that ends in “nonetheless their multiCluster and multiCCA embedding spaces contain no embeddings for”\n\n(4) The last paragraph before the Conclusion also feels like a fragment, or like two sentences have been spliced together: “We have detailed the differences to Upadhyay et al. (2018) extends the previous work by incorporating dependency relations into sparse coding.”\n\nThere are also several places where periods seem to have been left out (such as immediately before the above sentence).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1lY96o1cH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach for generating multilingual sparse representations. \nFor generating such representations, the proposed approach solves a series of convex optimization problems, serially for each language. Compared to previous work for generating sparse cross-lingual representations which is applicable to a pair of languages, the proposed approach is applicable to an arbitrary number of languages.\n\nThe paper argues that these sparse representations can lead to better performance for downstream tasks and interpretability. This is demonstrated using experiments on QVEC-CCA (for interpretability analysis), NLI, cross-lingual document classification, and dependency parsing (downstream tasks).\n\nOverall, the approach is well-motivated and performs well empirically. The experimental setup is also described in detail. \n\nMinor - I did not understand the benefit of MAMUS being \"stable\" across languages, as argued from Fig 1? The use of the word \"cognitively\" in the statement \"representations determined by MAMUS behave in a cognitively more plausible manner\" also seems a stretch to me.\n \nOne issue that the authors should discuss is whether such representations hold any extra value over contextual representations like multilingual Elmo, BERT etc. For instance, why would someone use MAMUS representations instead?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new method for creating word embeddings that can operate on corpora from more than one language.  The algorithm is simple, but rivals more complex approaches.  

The reviewers were happy with this paper.  They were also impressed that the authors ran the requested multi-lingual BERT experiments, even though they did not show positive results. One reviewer did think that non-contextual word embeddings were of less interest to the NLP community, but thought your arguments for the computational efficiency were convincing.
Decision: Accept (Poster)