Venue: EMNLP 2020

# Modeling the Music Genre Perception across Language-Bound Cultures

 Elena V. Epure\({}^{1}\), Guillaume Salha\({}^{1,2}\), Manuel Moussallam\({}^{1}\), Romain Hennequin\({}^{1}\)

\({}^{1}\) Deezer Research, Paris, France \({}^{2}\) LIX, Ecole Polytechnique, Palaiseau, France

research@deezer.com

###### Abstract

The music genre perception expressed through human annotations of artists or albums varies significantly across language-bound cultures. These variations cannot be modeled as mere translations since we also need to account for cultural differences in the music genre perception. In this work, we study the feasibility of obtaining relevant cross-lingual, culture-specific music genre annotations based only on language-specific semantic representations, namely distributed concept embeddings and ontologies. Our study, focused on six languages, shows that unsupervised cross-lingual music genre annotation is feasible with high accuracy, especially when combining both types of representations. This approach of studying music genres is the most extensive to date and has many implications in musicology and music information retrieval. Besides, we introduce a new, domain-dependent cross-lingual corpus to benchmark state of the art multilingual pre-trained embedding models.

+
Footnote †: [https://en.wikipedia.org](https://en.wikipedia.org)

+
Footnote †: [https://en.wikipedia.org](https://en.wikipedia.org)

## 1 Introduction

A prevalent approach to culturally study music genres starts with a common set of music items, _e.g._ artists, albums, tracks, and assumes that the same music genres would be associated with the items in all cultures Ferwerda and Schedl (2016); Skowron et al. (2017). However, music genres are subjective. Cultures themselves and individual musicological backgrounds influence the music genre perception, which can differ among individuals Sordo et al. (2008); Lee et al. (2013). For instance, a Westerner may relate _funk_ to _soul_ and _jazz_, while a Brazilian to _baile funk_ that is a type of _rap_Hennequin et al. (2018). Thus, accounting for cultural differences in music genres' perception could give a more grounded basis for such cultural studies. However, ensuring both a common set of music items and culture-sensitive annotations with broad coverage of music genres is strenuous Bogdanov et al. (2019).

To address this challenge, we study the feasibility of cross-culturally annotating music items with music genres, without relying on a parallel corpus. In this work, culture is related to a community speaking the same language Kramsch and Widdowson (1998). The specific research question we build upon is: assuming consistent patterns of music genres association with music items within cultures, can a mapping between these patterns be learned by relying on language-specific semantic representations? It is worth noting that, since music genres fall within the class of Culture-Specific Items Aixela (1996); Newmark (1988), cross-lingual annotation, in this case, cannot be framed as standard translation, as one also needs to model the dissimilar perception of music genres across cultures.

Our work focuses on four language families, Germanic (English-**en** and Dutch-**nl**), Romance (Spanish-**es** and French-**fr**), Japonic Japanese-**ja**), Slavic (Czech-**cs**), and on two types of language-specific semantic representations, ontologies and multi-word expression embeddings.

First, ontologies are often used to represent music genres, showing how they relate conceptually Schreiber (2016). We identify Wikipedia1, the online multilingual encyclopedia, to be particularly relevant to our study. It extensively documents worldwide music genres relating them through a coherent set of relation types across languages (_e.g. derivative genre_, _sub-genre_). Though the relations types are the same per language, the actual music genres and the way they are related can differ. Indeed, Pfeil et al. Pfeil et al. (2006) have shown that Wikipedia contributions expose cultural differences aligned with the ones in the physical world.

Footnote 1: [https://en.wikipedia.org](https://en.wikipedia.org)

Second, music genres can be represented from a distributional semantics perspective. Word vector

spaces are generated from large corpora following the distributional hypothesis, _i.e._ words with similar contexts have akin meanings. As languages are passed on culturally, we assume that the language-specific corpora used to create these spaces are sufficient to convey concepts' cultural specificity into their vector representations. In our study, we focus on multiple recent multilingual pre-trained models to generate word or sentence2 embeddings Arora et al. (2017); Grave et al. (2018); Artetxe and Schwenk (2019); Devlin et al. (2019); Lample and Conneau (2019), to account for variances in the used corpora or model designs.

Footnote 2: Music genres can be multi-word expressions. Previous work Shwartz and Dagan (2019) successfully embed phrases with sentence embedding models. In particular, contextualized language models result in more meaningful representations for diverse composition tasks.

Lastly, we combine the semantic representations by retrofitting distributed music genre embeddings to music genre ontologies. Retrofitting Faruqui et al. (2015) modifies each concept embedding such that the representation is still close to the distributed one, but also encodes ontology information. Initially, we retrofit music genres per language, using monolingual ontologies. Then, by partially aligning these ontologies, we apply retrofitting to learn multilingual embeddings from scratch.

The results show that we can model the cross-lingual music genre annotation with high accuracy by combining both types of language-specific semantic representations. When comparing the representations derived from multilingual pre-trained models, the smooth inverse frequency averaging Arora et al. (2017) of aligned word embeddings outperforms the state of the art approaches. To our knowledge, this simple method has been rarely used to embed multilingual sentences Vargas et al. (2019), and we hypothesize its potential as a strong baseline on other cross-lingual datasets and tasks too. Finally, embedding learning based on retrofitting leads to better multilingual music genre representations than when inferred with pretrained embedding models. This opens the possibility to learn embeddings for rare music genres or languages when aligned music genre ontologies are available.

Summing up, our contributions are: 1) a study on how effective language-specific semantic representations of music genres are for modeling cross-lingual annotation, without relying on a parallel music item corpus; 2) an extensive evaluation of multilingual pre-trained embedding models to derive representations for multi-word concepts in the music domain. Our study can enable complete musicological research, but also localized music information retrieval. This latter application is crucial for online music streaming platforms that leverage music genre annotations to provide worldwide, user-personalized music recommendations.

Our domain-specific study complements other works benchmarking general-language sentence representations Conneau et al. (2018). Finally, we provide an in-depth formal analysis of the retrofitting part of our method. We prove the strict convexity of retrofitting and show that the ontology concepts' final embeddings converge to the same values despite the order in which concepts are iteratively updated, on condition that we know a single initial node embedding in each connected component of the ontology.

## 2 Related Work

Music genres are conceptual representations encompassing a set of conventions between the music industry, artists, and listeners about individual music styles Lena (2012). From a cultural perspective, it has been shown that there are differences in how people listen to music genres Ferwerda and Schedl (2016); Skowron et al. (2017). Average listening habits in some countries span across many music genres and are less diverse in other countries Ferwerda and Schedl (2016). Also, cultural dimensions proved strong predictors for the popularity of specific music genres Skowron et al. (2017).

Despite the apparent agreement on the music style for which the music genres stand, conveyed in the earlier definition and implied in the related works too, music genres are subjective concepts Sordo et al. (2008); Lee et al. (2013). To address this subjectivity, Bogdanov et al. (2019) proposed a dataset of music items annotated with English music genres by different sources. In this line of work, we address the divergent perception of music genres. Still, we focus on multilingual, unsupervised music genre annotation without relying on content features, _i.e._ audio or lyrics. We also complement similar studies in other domains (art: Eleta and Golbeck, 2012) with another research method.

Then, in the literature, there are other works that benchmark pre-trained word and sentence embedding models. van der Heijden et al. (2019) compares multilingual contextual language models for 

named-entity recognition and part-of-speech tagging. Shwartz and Dagan (2019) use multiple static and contextual word embeddings to represent multi-word expressions and assess their capacity to capture meaning shift and implicit meaning in compositionality. Conneau et al. (2018) formulate a new task to evaluate cross-lingual sentence representation centered on natural language inference.

Compared to these works, our benchmark is aimed at cross-lingual annotation; we target a specific domain, music, for which we try concept embedding adaptation with retrofitting; and we also test a multilingual sentence representation obtained with smooth inverse frequency averaging of multilingual word embeddings. As discussed in a recent survey on cross-lingual word embedding models Ruder et al. (2019), there is a need to unlock domain-specific data to assess if general-language sentence representations are also accurate across domains. Our work builds towards this goal.

## 3 Cross-lingual Music Genre Annotation

Further, we formalize the cross-lingual annotation task and the strategy to evaluate it in Section 3.1. We describe the test corpus used in this work, together with its collection procedure in Section 3.2.

### Problem Formalization

The cross-lingual music genre annotation consists of inferring, for music items, tags in a target language \(L_{t}\), knowing tags in a source language \(L_{s}\). For instance, knowing the English music genres of _Fatboy Slim_ (_big beat_, _electronica_, _alternative rock_), the goal is to predict _rave_ and _rock alternative_ in Spanish. As shown in the example, but also Section 1, the problem goes beyond translation and instead targets a model able to map concepts, potentially dissimilar, across languages and cultures.

Formally, given \(S\) a set of tags in language \(L_{s}\), \(\mathcal{P}\) the partitions of \(S\) and \(T\) a set of tags in language \(L_{t}\), a mapping scoring function \(f:\mathcal{P}(S)\rightarrow\mathrm{I\!R}^{|T|}\) can attribute a prediction score to each target tag, relying on subsets of source tags drawn from \(S\)Hennequin et al. (2018); Epure et al. (2019, 2020). The produced score incorporates the degree of relatedness of each particular input source tag to the target tag. A common approach to compute relatedness in distributional semantics relies on cosine similarity. Thus, for \(\{s_{1},...,s_{K}\}\) source tags and any target tag \(t\), \(f\) can be defined as:

\[f_{t}(\{s_{1},s_{2},\ldots,s_{K}\})=\frac{1}{K}\sum_{k=1}^{K}\frac{\mathbf{s} _{k}{}^{T}\mathbf{t}}{||\mathbf{s}||_{2}||\mathbf{t}||_{2}}, \tag{1}\]

where \(||\cdot||_{2}\) is the Euclidean norm.

### Test Corpus

Wikipedia records worldwide music artists and their discographies, with a frequent mentioning of their music genres. By manually checking the Wikipedia pages of miscellaneous music items, we observed that their music genres vary significantly across languages. For instance, _Knights of Cyclonia_, a single by _Muse_, was annotated in Spanish as _progressive rock_, while in Dutch as _progressive metal_ and _alternative rock_. In Figure 1, we show another example of different annotations in English, Spanish, and Japanese from Wikipedia infoboxes. As Wikipedia writing is localized, contributors' culture can lead to differences in the multilingual content on the same topic Pfeil et al. (2006), particularly for subjective matters. Thus, Wikipedia was a suitable source for assembling the test corpus.

Using DBpedia Auer et al. (2007) as a proxy to Wikipedia, we collected music items such as artists and albums, annotated with music genres in at least two of the six languages (**en**, **nl**, **fr**, **es**, **cs** and **ja**). We targeted _MusicalWork_, _MusicalArtist_ and _Band_DBpedia resource types, and we only kept music items that were annotated with music genres which appeared at least \(15\) times in the corpus. Our final corpus includes 63246 music items. The number of annotations for each language pair is presented in Table 1. We also show in Table 2 the number of

\begin{table}
\begin{tabular}{l l l l l l} \hline
**Language** & **nl** & **fr** & **es** & **cs** & **ja** \\ \hline
**en** & 12604 & 28252 & 32891 & 4772 & 14752 \\
**nl** & & & 7139 & 7689 & 1885 & 3426 \\
**fr** & & & 15616 & 3046 & 8622 \\
**es** & & & & 3245 & 7644 \\
**cs** & & & & & 2065 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: Number of music items for language pair.

\begin{table}
\begin{tabular}{l|l|l|} \hline  & \multicolumn{3}{c}{**Number of unique music genres**} \\
**Language** & **Corpus** (_Avg. per item_) & **Ontology** \\ \hline
**en** & 558 _(2.12 \(\pm\) 1.34)_ & 10748 \\
**nl** & 204 _(1.71 \(\pm\) 1.06)_ & 1529 \\
**fr** & 364 _(1.75 \(\pm\) 1.06)_ & 2905 \\
**es** & 525 _(2.11 \(\pm\) 1.34)_ & 3988 \\
**cs** & 133 _(2.23 \(\pm\) 1.34)_ & 1418 \\
**ja** & 192 _(1.51 \(\pm\) 1.11)_ & 1609 \\ \hline \end{tabular}
\end{table}
Table 2: Number of unique music genres in the corpus (Section 3.2) and in the ontology (Section 4.1).

 

unique music genres per language in the corpus and the average number of tags for each music item.

The **en** and **es** languages use the most diverse tags. This can be because more annotations exist in these languages, in comparison to **cs**, which has the least annotations and least diverse tags. However, the mean number of tags per item appears relatively high for **cs**, while **ja** has the smallest mean number of tags per item.

## 4 Language-specific Semantic Representations for Music Genres

This work aims to assess the possibility of obtaining relevant cross-lingual music genre annotations, able to capture cultural differences too, by relying on language-specific semantic representations. Two types of semantic representations are investigated given their popularity: ontologies to represent music genre relations (presented in Section 4.1) and distributed embeddings to represent multiword expressions in general (presented in Section 4.2). In contrast to this unsupervised approach, mapping patterns of associating music genres with music items across cultures could have also been enabled with a parallel corpus. However, gathering a corpus that includes all music genres for each pair of languages is challenging.

### Music Genre Ontology

Conceptually, music genres are interconnected entities. For example, _rap west coast_ is a sub-genre of _hiphop_ or _musica electronica_ is the origin of _syntbunk_. Academic and practitioner communities often use ontologies or knowledge graphs to represent music genre relations and enrich the music genre definitions Schreiber (2016); Lisena et al. (2018). As mentioned in Section 1, we use in this study Wikipedia-based music genre ontologies because the multilingual Wikipedia contributions on the same topic can differ and these differences have been proven aligned with the ones in the physical world Pfeil et al. (2006).

We further describe how we crawl the Wikipediabased music genres ontologies for the six languages by relying on DBpedia. For each language, first, we constitute the seed list using two sources: the DBpedia resources of type _MusicGenre_ and their aliases linked through the _wikiPageRedirects_ relation; the music genres discovered when collecting the test corpus (introduced in Section 3.1) and their aliases. Then, music genres are fetched by visiting the DBpedia resources linked to the seeds through the relations _wikiPageRedirects_, _musicSubgnerne_, _stylisticOrigin_, _musicFusionGene_ and _derivative3_. The seed list is updated each time, allowing the crawling to continue until no new resource is found.

Footnote 3: We present these relations by their English names, which may be translated in the DBpedia versions in other languages.

In DBpedia, resources are sometimes linked to their equivalents in other languages through the relation _sameAs_. For most experiments, we rely on monolingual music genres ontologies. How 

ever, we also collect the cross-lingual links between music genres to include a translation baseline for cross-lingual annotation, _i.e._ for each music genre in a source language, we predict its equivalent in a target language using DBpedia. Besides, we try to learn aligned embeddings from scratch by relying on these partially aligned music genre ontologies, as will be discussed in Section 4.3.

The number of unique Wikipedia music genres discovered in each language is presented in Table 2. Let us note that the graph numbers are much larger than the test corpus numbers, emphasizing the challenge to constitute a parallel corpus that covers all language-specific music genres.

### Music Genre Distributed Representations

As music genres are multi-word expressions, we make use of existing sentence representation models. We also inquire into word vector spaces and obtain sentence embeddings by hypothesizing that music genres are generally compositional, _i.e._ the sense of a multi-word expression is conveyed by the sense of each composing word (_e.g. West Coast rap_, _jazz blues_; there are also non-compositional examples like _hard rock_). We set our investigation scope to multilingual pre-trained embedding models, and we consider both static and contextual word/sentence representations as described next.

Multilingual Static Word Embeddings.The classical word embeddings we study are the multilingual fastText word vectors trained on Wikipedia and Common Crawl Grave et al. (2018). The model is an extension of the Common Bag of Word Model CBOW Mikolov et al. (2013), which includes subword and word position information. The fastText word vectors are trained in distinct languages. Thus, we must ensure that the monolingual word vectors are projected in the same space for cross-lingual annotation. We perform the alignment with the method proposed by Joulin et al. (2018), which treats word translation as a retrieval task and introduces a new loss relying on a relaxed cross-domain similarity local scaling criterion.

Multilingual Contextual Word EmbeddingsContextual word embeddings Peters et al. (2017); Devlin et al. (2019), in contrast to the classical ones, are dynamically inferred based on the given context sentence. This type of embedding can address polysemy as the word sense is disambiguated through the surrounding text. In our work, we include two recent contextualized language models compatible with the multilingual scope: multilingual Bidirectional Encoder Representations from Transformers BERT, Devlin et al. (2019) and Cross-lingual Language Model XLM, Lample and Conneau (2019).

BERT Devlin et al. (2019) is trained to jointly predict a masked word in a sentence and whether sentences are successive text segments. Similar to fastText Grave et al. (2018), subword and word position information is also used. An input sentence is tokenized against a limited token vocabulary with a modified version of the byte pair encoding algorithm BPE, Sennrich et al. (2016). Multilingual BERT is trained as a single-language model, fed with 104 concatenated monolingual Wikipedias Pires et al. (2019); Wu and Dredze (2019).

XLM Lample and Conneau (2019) has a similar architecture to BERT. Also, it shares with BERT one training objective, the masked word prediction, and the tokenization using BPE, but applied on sentences differently sampled from each monolingual Common Crawl corpus. Compared to BERT, two other objectives are introduced, to predict a word from previous words and a masked word by leveraging two parallel sentences. Thus, to train XLM, several multilingual aligned corpora are used Lample and Conneau (2019).

Multilingual Sentence Embeddings.Contextualized language models can be exploited in multiple ways. First, as Lample and Conneau (2019) show, by training the transformers on multi-lingual data, cross-lingual word vectors are obtained in an unsupervised way. The word vectors can be accessed through the model lookup table. These embeddings are merely aligned but not contextual, thus directly comparable to fastText. For these three types of cross-lingual non-contextual word embeddings, fastText (**FT**), the multilingual BERT's lookup table (**mBERT**) and the XLM's lookup table (**XLM**), we compute the sentence embedding using the standard average (_avg_) or the smooth inverse frequency averaging (_sif_) introduced by Arora et al. (2017).

Formally, let \(c\) denote a music genre composed of multiple tokens \(\{t_{1},t_{2},\dots,t_{M}\}\), \(\mathbf{t}_{m}\) the embedding of each token \(t_{m}\) initialized from a given pretrained embedding model or to \(d\)-dimensional4 null vector \(\boldsymbol{\mathsf{0}}_{d}\) if \(t_{m}\) is absent from the model vocabulary, and \(\hat{\mathbf{q}}_{i}\in\mathds{I}^{d}\) the representation of \(c\) which we want to infer. The _avg_ strategy computes \(\hat{\mathbf{q}}_{i}\) as 

\(\frac{1}{M}\sum_{m=1}^{M}\mathbf{t}_{m}\). The _sif_ strategy computes \(\hat{\textbf{q}}_{i}\) as:

\[\overline{\textbf{q}}_{i}=\frac{1}{M}\sum_{m=1}^{M}\frac{a}{a+f_{t_{m}}}\textbf{ t}_{m} \tag{2}\]

\[\hat{\textbf{q}}_{i}=\overline{\textbf{q}}_{i}-\textbf{uu}^{T}\overline{ \textbf{q}}_{i} \tag{3}\]

where \(f_{t_{m}}\) is the frequency of \(t_{m}\), \(a\) is a hyperparameter usually fixed to \(10^{-3}\)(Arora et al., 2017) and **u** is the first singular vector obtained through the singular value decomposition Golub and Reinsch (1971) of \(\overline{\textbf{Q}}\), the embedding matrix computed with the Equation 2 for all music genres. Vocabulary tokens of pre-trained embedding models are usually sorted by decreasing frequency in the training corpus, _i.e._ the higher the rank, the more frequent the token. Thus, based on the Zipf's law Zipf (149), \(f_{w_{m}}\) can be approximated by \(1/z_{t_{m}}\), \(z_{t_{m}}\) being the rank of \(t_{m}\). The intuition of this simple sentence embedding method is that uncommon words are semantically more informative.

Second, contextualized language models can be used as feature extractors representing sentences from the contextual embeddings of the associated tokens. Multiple strategies exist to retrieve contextual token embeddings: to use the embeddings layer or the last hidden layer or to apply min or max pooling over time Devlin et al. (2019). To infer a fixed-length representation of a multi-word music genre, we try max and mean pooling over token embeddings Lample and Conneau (2019); Reiners and Gurevych (2019), obtained with the diverse strategies mentioned before. We denote these sentence embeddings \(\textbf{XLM}_{Ctxt}\) and \(\textbf{mBERT}_{Ctxt}\).

The contextualized language models can be further fine-tuned for particular downstream tasks, yielding better sentence representations Eisenschlos et al. (2019); Lample and Conneau (2019). Existing evaluations of cross-lingual sentence representations are centered on natural language inference (XNLI, Conneau et al., 2018) or classification Eisenschlos et al. (2019). The cross-lingual music genre annotation would be closer to the XNLI task; hence we could fine-tune the pre-trained models on a parallel corpus of music genres translations or music genre annotations. However, our research investigates language-specific semantic representations. Also, using translated music genres would not model their different perception across cultures while obtaining an exhaustive corpus of cross-lingual annotations is challenging.

Last, we explore **LASER**, a universal language-agnostic sentence embedding model Artetxe and Schwenk (2019). The model is based on a BiLSTM encoder trained on corpora in \(93\) languages to learn multilingual fixed-length sentence embeddings. As in other models, sentences are tokenized against a fixed vocabulary, obtained with BPE from the concatenated multilingual corpora. **LASER** appears highly effective without requiring task-specific fine-tuning Artetxe and Schwenk (2019).

### Retrofitting Music Genre Distributed Representations to Ontologies

Retrofitting Faruqui et al. (2015) is a method to refine vector space word representations by considering the relations between words as defined in semantic lexicons such as WordNet Miller (1995). The intuition is to modify the distributed embeddings to become closer to the representations of the concepts to which they are related. Ever since the original work, many uses of retrofitting have been explored to semantically specialize word embeddings in relations such as synonyms or antonyms Kiela et al. (2015); Kim et al. (2016), in other languages than a source one Ponti et al. (2019) or in specific domains Hangya et al. (2018).

Enhanced extensions of retrofitting exist, but they require supervision Lengerich et al. (2018). The original method Faruqui et al. (2015) is unsupervised and can simply yet effectively leverage distributed embeddings and ontologies for improved representations. Thus, we mainly rely on it, but we apply some changes as further described. Let \(\Omega=(C,E)\) be an ontology including the concepts \(C\) and the semantic relations between these concepts \(E\subseteq C\times C\). The retrofitting goal is to learn new concept embeddings, \(\textbf{Q}\in\mathrm{I\!R}^{n\times d}\) with \(n=|C|\) and \(d\) the embedding dimension. The learning starts with initializing each \(\textbf{q}_{i}\in\mathrm{I\!R}^{d}\), the new embedding for concept \(i\in C\), to \(\hat{\textbf{q}}_{i}\), the initial distributed embedding, and then iteratively updates \(\textbf{q}_{i}\) until convergence as follows:

\[\textbf{q}_{i}\leftarrow\frac{\sum_{j:(i,j)\in E}{(\beta_{ij}+\beta_{ji})} \textbf{q}_{j}+\alpha_{i}\hat{\textbf{q}}_{i}}{\sum_{j:(i,j)\in E}{(\beta_{ ij}+\beta_{ji})}+\alpha_{i}} \tag{4}\]

\(\alpha\) and \(\beta\) are positive scalars weighting the importance of the initial, respectively, the related concept embeddings in computation. The formula was reached through the optimization of the retrofitting objective using the Jacobi method Saad (2003).

Equation 4 is a corrected version of the original work, as for a concept \(i\), not only \(\beta_{ij}\) appears in it, but also \(\beta_{ji}\). That is to say that when computing the 

partial derivative of the retrofitting objective concerning \(i\), two non-zero terms are corresponding to the related concept \(j\): when \(i\) is the source and \(j\) is the target and vice-versa [16, 2]. The further modifications that we make regard the parameters \(\alpha\) and \(\beta\). For each \(i\in C\), Faruqui et al. (2015) fix \(\alpha_{i}\) to \(1\), and \(\beta_{ij}\) to \(\frac{1}{degree(i)}\) for \((i,j)\in E\) or \(0\) otherwise; \(degree(i)\) is the number of related concepts \(i\) has in \(\Omega\).

While many embedding models can handle unknown words nowadays, concepts may still have unknown initial distributed vectors, depending on the model's choice. For this case, expanded retrofitting [34] has been proposed, considering \(\alpha_{i}=0\), for each concept \(i\) with unknown initial distributed vector, and \(\alpha_{i}=1\) for the rest. Thus, **q\({}_{i}\)** is initialized to **0\({}_{d}\)** and updated by averaging the embeddings of its related concepts at each iteration. Let us notice that, through retrofitting, representations are not only modified but also learned from scratch for some concepts.

We adopt the same approach to initialize \(\alpha\). Moreover, we also adjust the parameters \(\beta\) to weight the importance of each related concept embedding depending on the relation semantics in our music genre ontology [8]. Specifically, we distinguish between equivalence and relatedness as follows:

\[\overline{\beta}_{ij}=\left\{\begin{array}{ll}1&:(i,j)\in E_{\epsilon}\subset E \\ \beta_{ij}&:(i,j)\in E-E_{\epsilon}\\ 0,&:(i,j)\not\in E\end{array}\right.\]

where \(E_{\epsilon}\) contains the equivalence relation types (_wikiPageRedirects_, _sameAs_); \(E-E_{\epsilon}\) contains the relatedness relation types (_stylisticOrigin_, _musicSubgenre_, _derivative_, _musicFusionGenre_). We label this modified version of retrofitting as **Rfit**.

Finally, we want to highlight a crucial aspect of retrofitting. Previous works [34, 37, 39] claim that, while the retrofitting updating procedure converges, the results depend on the order in which the updates are made. We prove in Appendix A that the retrofitting objective is strictly convex when at least one initial concept vector is known in each connected component. Hence, with this condition satisfied, retrofitting converges to the same solution always and independently of the updates' order.

\begin{table}
\begin{tabular}{|c|l l|l l l l|l l|} \hline
**Pair** & **GTrans** & **DbgSameAs** & **mBERT\({}_{avg}\)** & **FT\({}_{inf}\)** & **XLM\({}_{Cstat}\)** & **LASER** & **Rfit\({}_{all}\)P\({}_{sf}\)T\({}_{if}\)** \\ \hline

[MISSING_PAGE_POST]

s** & 60.3 \(\pm\) 0.2 & 72.2 \(\pm\) 0.4 & 85.5 \(\pm\) 0.5 & 85.6 \(\pm\) 0.6 & 85.9 \(\pm\) 0.5 & 82.7 \(\pm\) 0.5 & **88.0 \(\pm\) 0.

 

Experiments

Cross-lingual music genre annotation, as formalized in Section 3, is a typical multi-label prediction task. For evaluation, we use the Area Under the receiver operating characteristic Curve (AUC, Bradley, 1997), macro-averaged. We report the mean and standard deviations of the macro AUC scores using \(3\)-fold cross-validation. For each language, we apply an iterative split [25] of the test corpus that balances the number of samples and the tag distributions across the folds. We pre-process the music genres by either replacing special characters with space (_-/_) or removing them (_(__)':..!_$). For Japanese, we introduce spaces between tokens obtained with Mecab6. Embeddings are then computed from pre-processed tags.

Footnote 6: [https://taku910.github.io/mecab/](https://taku910.github.io/mecab/)

We test two translation baselines, one based on Google Translate7 **(GTrans)** and one on the DBpedia _SameAs_ relation (**DBpSamEs**). In this case, a source music genre is mapped on a single or no target music genre, its embedding being in the form {**0,1**}\({}^{|T|}\). For \(\textbf{XLM}_{Ctxt}\), we compute the sentence embedding by averaging the token embeddings obtained with mean pooling across all layers. For \(\textbf{mBERT}_{Ctxt}\), we apply the same strategy, but by max pooling the token embeddings instead. We chose these representations as they showed the best performance experimentally compared to the other strategies described in Section 4.2.

Footnote 7: [https://translate.google.com](https://translate.google.com)

When retrofitting language-specific music genre embeddings, we use the corresponding monolingual ontology (**Rfit\({}_{u\Omega}\)**). When we learn multilingual embeddings from scratch with retrofitting, by knowing only music genre embeddings in one language (_la_), we use the partially aligned DBpedia ontologies which contain the _SameAs_ relations (**Rfit\({}_{a\Omega}^{lo}\)**). For this case, we also propose a baseline representing a source concept embedding as a vector of geodesic distances in the partially aligned ontologies to each target concept (**DBp\({}_{a\Omega}\)NNDist**).

Results.Table 3 shows the cross-lingual annotation results. The standard translation, **GTrans**, leads to the lowest results being over-performed by a knowledge-based translation, more adapted to this domain (**DBpSamEs**). Also, these results show that translation methods fail to capture the dissimilar cross-cultural music genre perception.

The second part of Table 3 contains only the best8music genre embeddings computed with each word/sentence pre-trained model or method. When averaging static multilingual word embeddings, those from **mBERT** often yield the most relevant cross-lingual annotations, while when applying the _sif_ averaging, the aligned **FT** word vectors are the best choice. Between the two contextual word embedding models, \(\textbf{XLM}_{Ctxt}\) significantly outperforms \(\textbf{mBERT}_{Ctxt}\), thus we report only the former.

Footnote 8: This can be seen in Figure 3 for the example of the text embedding.

We can notice that all distributed representations of music genres can model quite well the varying music genre annotation across languages. \(\textbf{FT}_{sif}\) results in the most relevant cross-lingual annotations consistently for \(5\) out of \(6\) languages as a source. For **cs** though, the embeddings from \(\textbf{XLM}_{Ctxt}\) are sometimes slightly better. **LASER** under-performs for most languages but **ja**, for which the vectors obtained with \(\textbf{mBERT}_{avg}\) are less suitable.

The last column of Table 3 shows the results of cross-lingual annotation when using the \(\textbf{FT}_{sif}\) vectors retrofitted to monolingual music genre ontologies. The domain adaptation of concept embeddings, inferred with general-language pre-trained models, significantly improves music genre annotation modeling across all pairs of languages.

Table 4 shows the results when using retrofitting to learn music genre embeddings from scratch. Here, distributed vectors are known for one language (**en**, respectively **ja***) and the monolingual ontologies are partially aligned. Even though not necessarily all music genres are linked to their equivalents in the other language, the concept representations learned in this way are more relevant for cross-lingual annotation, for all pairs involving **en** as the source and for **ja-c**s and **ja-cn**. In fact, the



baseline (**DBp\({}_{\alpha\Omega}\)NNDist**) reveals that the aligned ontologies stand-alone can model the cross-lingual annotation quite well, in particular for **en**.

Discussion.The results show that using translation to produce cross-lingual annotations is limited as it does not consider the culturally divergent perception of music genres. Instead, monolingual semantic representations can model this phenomenon rather well. For instance, from _Milton Cardona_'s music genres in **es**, _salsa_ and _jazz_, it correctly predicts the Japanese equivalent of _fusion_ (? = -- >= >) in **ja**. Yet, while a thorough qualitative analysis requires more work, preliminary exploration suggests that larger gaps in perception might still be inadequately modeled. For instance, for _Santana_'s album _Welcome_ tagged with _ Jazz_ in **es**, it does not predict _pop_ in **fr**.

When comparing the distributed embeddings, a simple method that relies on a weighted average of multilingual aligned word vectors significantly outperforms the others. Although rarely used before, we question if we can notice such high performance with other multilingual data-sets. The cross-lingual annotations are further improved by retrofitting the distributed embeddings to monolingual ontologies. Interestingly, the vector alignment does not appear degraded by retrofitting to disjoint graphs. Or, the negative impact is limited and exceeded by introducing domain knowledge in representations. Further, as shown in Table 4, joining semantic representations in this way proves very suitable to learn music genre vectors from scratch.

Regarding the scores per language, we obtained the lowest ones for **ja** as the source. We could explain this by either a more challenging test corpus or still incompatible embeddings in **ja**, possibly because of the quality of the individual embedding models for this language and the completeness of the Japanese music genre ontology. Also, we did not notice any particular improvement for pairs of languages from the same language family, _e.g._**fr** and **es**. However, we would need a sufficiently sizeable parallel corpus exhaustively annotated in all languages to reliably compare the performance for pairs of languages from the same language family or different ones.

Finally, by closely analysing the results in Table 3, we noticed that given two languages \(L_{1}\) and \(L_{2}\), with more music genre embeddings in \(L_{1}\) than in \(L_{2}\) (from both ontology and corpus), the results of mapping annotations from \(L_{1}\) to \(L_{1}\) seems always better than the results from \(L_{2}\) to \(L_{1}\). This observation explains two trends in Table 3. First, the scores achieved for **en** or **es** as the source, the languages with the largest number of music genres, are the best. Second, the results for the same pair of languages could vary a lot, depending on the role each language plays, source, or target.

One possible explanation is that the prediction from languages with fewer music genre tags such as \(L_{2}\) towards languages with more music genre tags such as \(L_{1}\) is more challenging because the target language contains more specific or rare annotations. For instance, when checking the results per tag from **cs** to **en** we observed that among the tags with the lowest scores, we found _moombahnton_, _zeuhl_, or _candombe_. However, other common music genres, such as _latin music_ or _hard rock_, were also poorly predicted, showing that other causes exist too. Is the unbalanced number of music genres used in annotations a cultural consequence? Related work Ferwerda and Schedl (2016) seems to support this hypothesis. Then could we design a better mapping function that leverages the unbalanced numbers of music genres in cross-cultural annotations? We will dedicate a thorough investigation of these questions as future work.

## 6 Conclusion

We have presented an extensive investigation on cross-lingual modeling of music genre annotation, focused on six languages, and two common approaches to semantically represent concepts: ontologies and distributed embeddings9.

Footnote 9: [https://github.com/deezer/CrossCulturalMusGenrePerception](https://github.com/deezer/CrossCulturalMusGenrePerception)

Our work provides a methodological framework to study the annotation behavior across language-bound cultures in other domains too. Hence, the effectiveness of language-specific concept representations to model the culturally diverse perception could be further probed. Then, we combined the semantic representations only with retrofitting. However, inspired by paraphrastic sentence embedding learning, one can also consider the music genre relations as paraphrasing forms with different strengths Wieting et al. (2016). Finally, the models to generate cross-lingual annotations should be thoroughly evaluated in downstream music retrieval and recommendation tasks.

 



## References

* Aixela (1996) Javier Franco Aixela. 1996. Culture-specific items in translation. _Translation, power, subversion_, 8:52-78.
* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In _International Conference on Learning Representations_.
* Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. _Transactions of the Association for Computational Linguistics_, 7:597-610.
* Auer et al. (2007) Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In _The Semantic Web_, pages 722-735, Berlin, Heidelberg. Springer.
* Azimzadeh and Forsyth (2016) Parsiad Azimzadeh and Peter A. Forsyth. 2016. Weakly chained matrices, policy iteration, and impulse control. _SIAM Journal on Numerical Analysis_, 54(3):1341-1364.
* Bengio et al. (2006) Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. 2006. _Label Propagation and Quadratic Criterion_, semi-supervised learning edition, pages 193-216. MIT Press.
* Bogdanov et al. (2019) Dmitry Bogdanov, Alastair Porter, Hendrik Schreiber, Julian Urbano, and Sergio Oramas. 2019. The acousticbrainz genre dataset: Multi-source, multi-level, multi-label, and large-scale. In _Proceedings of the 20th Conference of the International Society for Music Information Retrieval (ISMIR 2019)_, pages 360-367, Delft, The Netherlands. International Society for Music Information Retrieval (ISMIR).
* Bradley (1997) Andrew P. Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. _Pattern Recognition_, 30(7):1145-1159.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Dattorro (2005) Jon Dattorro. 2005. _Convex Optimization & Euclidean Distance Geometry_. Meboo Publishing.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Eisenschlos et al. (2019) Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5702-5707, Hong Kong, China. Association for Computational Linguistics.
* Eleta and Golbeck (2012) Irene Eleta and Jennifer Golbeck. 2012. A study of multilingual social tagging of art images: Cultural bridges and diversity. In _Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work_, CSCW '12, pages 695-704, New York, NY, USA. Association for Computing Machinery.
* Epure et al. (2019) Elena V. Epure, Anis Khlif, and Romanin Hennequin. 2019. Leveraging knowledge bases and parallel annotations for music genre translation. In _Conference of the International Society of Music Information Retrieval_, ISMIR 2019.
* Epure et al. (2020) Elena V. Epure, Guillaume Salha, and Romain Hennequin. 2020. Multilingual music genre embeddings for effective cross-lingual music item annotation. In _Conference of the International Society of Music Information Retrieval_, ISMIR 2020.
* Fang et al. (2019) Lanting Fang, Yong Luo, Kaiyu Feng, Kaiqi Zhao, and Aiqun Hu. 2019. Knowledge-enhanced ensemble learning for word embeddings. In _The World Wide Web Conference_, WWW '19, page 427-437, New York, NY, USA. Association for Computing Machinery.
* Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1606-1615, Denver, Colorado. Association for Computational Linguistics.
* Ferwerda and Schedell (2016) Bruce Ferwerda and Markus Schedell. 2016. Investigating the relationship between diversity in music consumption behavior and cultural dimensions: A cross-country analysis. In _24th Conference on User Modeling, Adaptation, and Personalization (UMAP) Extended Proceedings: 1st Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP)_, Halifax, NS, Canada.
* Golub and Reinsch (1971) Gene H Golub and Christian Reinsch. 1971. Singular value decomposition and least squares solutions. In _Handbook for Automatic Computation: Volume II: Linear Algebra_, pages 134-151. Springer Berlin Heidelberg, Berlin, Heidelberg.
* Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_, Miyazaki,

Japan. European Languages Resources Association (ELRA).
* Hangya et al. (2018) Viktor Hangya, Fabienne Braune, Alexander Fraser, and Hinrich Schutze. 2018. Two methods for domain adaptation of bilingual tasks: Delightfully simple and broadly applicable. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 810-820, Melbourne, Australia. Association for Computational Linguistics.
* Hayes (2019) Dmetri Hayes. 2019. What just happened? Evaluating retrofitted distributional word vectors. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1062-1072, Minneapolis, Minnesota. Association for Computational Linguistics.
* van der Heijden et al. (2019) Niels van der Heijden, Samira Abnar, and Ekaterina Shutova. 2019. A comparison of architectures and pretraining methods for contextualized multilingual word embeddings. _Computing Research Repository_, arXiv:1912.10169.
* Hennequin et al. (2018) Romain Hennequin, Jimena Royo-Letelier, and Manuel Moussallam. 2018. Audio based disambiguation of music genre tags. In _Conference of the International Society of Music Information Retrieval_, ISMIR 2018.
* Joulin et al. (2018) Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
* Kiela et al. (2015) Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2044-2048, Lisbon, Portugal. Association for Computational Linguistics.
* Kim et al. (2016) Joo-Kyung Kim, Marie-Catherine de Marmeffe, and Eric Fosler-Lussier. 2016. Adjusting word embeddings with semantic intensity orders. In _Proceedings of the 1st Workshop on Representation Learning for NLP_, pages 62-69, Berlin, Germany. Association for Computational Linguistics.
* Kramsch and Widdowson (1998) Claire Kramsch and HG Widdowson. 1998. _Language and culture_. Oxford University Press.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In _33rd Conference on Neural Information Processing Systems (NeurIPS 2019)_, Vancouver, Canada.
* Lee et al. (2013) Jin Ha Lee, Kahyun Choi, Xiao Hu, and J. Stephen Downie. 2013. K-pop genres: A cross-cultural exploration. In _Conference of the International Society on Music Information Retrieval_, ISMIR 2013.
* Lena (2012) Jennifer C Lena. 2012. _Banding together: How communities create genres in popular music_. Princeton University Press.
* Lengerich et al. (2018) Ben Lengerich, Andrew Maas, and Christopher Potts. 2018. Retrofitting distributional embeddings to knowledge graphs with functional relations. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2423-2436, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Lisena et al. (2018) Pasquale Lisena, Konstantin Todorov, Cecile Cecconi, Francoise Lersche, Isabelle Canno, Frederic Puyrenner, Martine Voisin, Thierry Le Meur, and Raphael Troncy. 2018. Controlled vocabularies for music metadata. In _Conference of the International Society on Music Information Retrieval_, Paris, France.
* Volume 2_, NIPS'13, page 3111-3119, Red Hook, NY, USA. Curran Associates Inc.
* Miller (1995) George A. Miller. 1995. Wordnet: A lexical database for english. _Commun. ACM_, 38(11):39-41.
* Newmark (1988) Peter Newmark. 1988. _A textbook of translation_, volume 66. Prentice hall New York.
* Peters et al. (2017) Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
* Pfeil et al. (2006) Ulrikle Pfeil, Panayiotis Zaphiris, and Chee Siang Ang. 2006. Cultural Differences in Collaborative Authoring of Wikipedia. _Journal of Computer-Mediated Communication_, 12(1):88-113.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? _Computing Research Repository_, arXiv:1906.01502.
* Ponti et al. (2019) Edoardo Maria Ponti, Ivan Vulic, Goran Glavas, Roi Reichart, and Anna Korhonen. 2019. Cross-lingual semantic specialization via lexical relation induction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2206-2217, Hong Kong, China. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 

3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Ruder et al. (2019) Sebastian Ruder, Ivan Vulic, and Anders Sogaard. 2019. A survey of cross-lingual word embedding models. _Journal of Artificial Intelligence Research_, 65(1):569-630.
* Saad (2003) Yousef Saad. 2003. _Iterative methods for sparse linear systems_, 2nd edition. Society for Industrial and Applied Mathematics, USA.
* Saha et al. (2016) Tanay Kumar Saha, Shafiq Joty, Naeemul Hassan, and Mohammad Al Hasan. 2016. Dis-s2v: Discourse informed sen2vec. _Computing Research Repository_, arXiv:1610.08078.
* Schreiber (2016) Hendrik Schreiber. 2016. Genre ontology learning: Comparing curated with crowd-sourced ontologies. In _Proceedings of the International Conference on Music Information Retrieval (ISMIR)_, pages 400-406, New York, USA.
* Volume Part III_, ECML PKDD'11, page 145-158, Berlin, Heidelberg. Springer-Verlag.
* Senrrich et al. (2016) Rico Senrrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Shwartz and Dagan (2019a) Vered Shwartz and Ido Dagan. 2019a. Still a pain in the neck: Evaluating text representations on lexical composition. _Transactions of the Association for Computational Linguistics_, 7:403-419.
* Shwartz and Dagan (2019b) Vered Shwartz and Ido Dagan. 2019b. Still a pain in the neck: Evaluating text representations on lexical composition. _Transactions of the Association for Computational Linguistics_, 7:403-419.
* Skowron et al. (2017) Marcin Skowron, Florian Lemmerich, Bruce Ferwerda, and Markus Schedl. 2017. Predicting genre preferences from cultural and socio-economic factors for music retrieval. In _Advances in Information Retrieval_, pages 561-567, Cham. Springer International Publishing.
* Sordo et al. (2008) Mohamed Sordo, Oscar Celma, Martin Blech, and Enric Guaus. 2008. The Quest for Musical Genres: Do the Experts and the Wisdom of Crowds Agree? In _Conference of the International Society on Music Information Retrieval_.
* Speer and Chin (2016) Robyn Speer and Joshua Chin. 2016. An ensemble method to produce high-quality word embeddings. _Computing Research Repository_, arXiv:1604.01692.
* Vargas et al. (2019) Francisco Vargas, Kamen Brestnichi, Alex Papadopoulos Korifatis, and Nils Hammerla. 2019. Multilingual factor analysis. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1738-1750, Florence, Italy. Association for Computational Linguistics.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, beccas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
* Zipf (1949) George Kingsley Zipf. 1949. _Human behavior and the principle of least effort_. Cambridge, Addison-Wesley Press.

 



## Appendix A Strict Convexity of Retrofitting

**Theorem.** Let \(V\) be a finite vocabulary with \(|V|=n\). Let \(\Omega=(V,E)\) be an ontology represented as a directed graph which encodes semantic relationships between vocabulary words. Further, let \(\hat{V}\subseteq V\) be the subset of words which have non-zero initial distributed representations, \(\hat{\mathbf{q}}_{i}\). The goal of retrofitting is to learn the matrix \(\textbf{Q}\in\mathrm{I}\!\mathrm{R}^{d}\), stacking up the new embeddings \(\mathbf{q}_{i}\in\mathrm{I}\!\mathrm{R}^{d}\) for each \(i\in V\). The objective function to be minimized is:

\[\begin{array}{l}\Phi(\textbf{Q})=\sum_{i\in\hat{V}}\alpha_{i}||\mathbf{q}_{ i}-\hat{\mathbf{q}}_{i}||_{2}^{2}\\ +\sum_{i=1}^{n}\sum_{(i,j)\in E}\beta_{ij}||\mathbf{q}_{i}-\mathbf{q}_{j}||_{2 }^{2},\end{array}\]

where the \(\alpha_{i}\) and \(\beta_{ij}\) are positive scalars. Assuming that each connected component of \(\Omega\) includes at least one word from \(\hat{V}\), the objective function \(\Phi\) is strictly convex w.r.t. **Q**.

Proof.First of all, let \(\hat{\textbf{Q}}\) denote the \(n\times d\) matrix whose \(i\)-th row corresponds to \(\hat{\mathbf{q}}_{i}\) if \(i\in\hat{V}\), and to the \(d\)-dimensional null vector \(\textbf{0}_{d}\) otherwise. Let **A** denote the \(n\times n\) diagonal matrix verifying \(\textbf{A}_{ii}=\alpha_{i}\) if \(i\in\hat{V}\) and \(\textbf{A}_{ii}=0\) otherwise. Let **B** denote the \(n\times n\) symmetric matrix such as, for all \(i,j\in\{1,...,n\}\) with \(i\neq j\), \(\textbf{B}_{ij}=\textbf{B}_{ji}=-\frac{1}{2}(\beta_{ij}+\beta_{ji})\) and \(\textbf{B}_{ii}=\sum_{j=1,j\neq i}^{n}|\textbf{B}_{ij}|\). With these notations, and with \(Tr(\cdot)\) the trace operator for square matrices, we have:

\[\begin{array}{l}\sum_{i\in\hat{V}}\alpha_{i}||\mathbf{q}_{i}-\hat{\mathbf{ q}}_{i}||_{2}^{2}\\ = Tr\Big{(}(\textbf{Q}-\hat{\textbf{Q}})^{T}\textbf{A}(\textbf{Q}-\hat{\textbf{ Q}})\Big{)}\\ = Tr\Big{(}\textbf{Q}^{T}\textbf{A}\textbf{Q}-\hat{\textbf{Q}}^{T}\textbf{A} \textbf{Q}-\textbf{Q}^{T}\textbf{A}\hat{\textbf{Q}}+\hat{\textbf{Q}}^{T} \textbf{A}\hat{\textbf{Q}}\Big{)}.\end{array}\]

Also:

\[\sum_{i=1}^{n}\sum_{(i,j)\in E}\beta_{ij}||\mathbf{q}_{i}-\mathbf{q}_{j}||_{2 }^{2}=Tr\Big{(}\textbf{Q}^{T}\textbf{B}\textbf{Q}\Big{)}.\]

Therefore, as the trace is a linear mapping, we have:

\[\begin{array}{l}\Phi(\textbf{Q})=Tr\Big{(}\textbf{Q}^{T}(\textbf{A}+\textbf{ B})\textbf{Q}\Big{)}+\\ Tr\Big{(}\hat{\textbf{Q}}^{T}\textbf{A}\hat{\textbf{Q}}-\hat{\textbf{Q}}^{T} \textbf{A}\textbf{Q}-\textbf{Q}^{T}\textbf{A}\hat{\textbf{Q}}\Big{)}.\end{array}\]

Then, we note that **A** + **B** is a weakly diagonally dominant matrix (WDD) as, by construction, \(\forall i\in\{1,...,n\},|(\textbf{A}+\textbf{B})_{ii}|\geq\sum_{j\neq i}|( \textbf{A}+\textbf{B})_{ij}|\). Also, for all \(i\in\hat{V}\), the inequality is strict, as \(|(\textbf{A}+\textbf{B})_{ii}|=\alpha_{i}+\sum_{j\neq i}|\textbf{B}_{ij}|>\sum_ {j\neq i}|(\textbf{A}+\textbf{B})_{ij}|=\sum_{j\neq i}|\textbf{B}_{ij}|\), which means that, for all \(i\in\hat{V}\), row \(i\) of **A** + **B** is strictly diagonally dominant (SSD). Assuming that each connected component of graph \(G\) includes at least one node from \(\hat{V}\), we conclude that \(\textbf{A}+\textbf{B}\) is a weakly chained diagonally dominant matrix Azimzadeh and Forsyth (2016), _i.e._ that:

* \(\textbf{A}+\textbf{B}\) is WDD;
* for each \(i\in V\) such that row \(i\) is not SSD, there exists a _walk_ in the graph whose adjacency matrix is \(\textbf{A}+\textbf{B}\) (two nodes \(i\) and \(j\) are connected if \((\textbf{A}+\textbf{B})_{ij}=(\textbf{A}+\textbf{B})_{ji}\neq 0\)), starting from \(i\) and ending at a node associated to a SSD row.

Such matrices are nonsingular Azimzadeh and Forsyth (2016), which implies that \(\textbf{Q}\rightarrow\textbf{Q}^{T}(\textbf{A}+\textbf{B})\textbf{Q}\) is a positive-definite quadratic form. As \(\textbf{A}+\textbf{B}\) is a symmetric positive-definite matrix, there exists a matrix **M** such that \(\textbf{A}+\textbf{B}=\textbf{M}^{T}\textbf{M}\). Therefore, denoting \(||\cdot||_{F}^{2}\) the squared Frobenius matrix norm:

\[\begin{array}{l}Tr\Big{(}\textbf{Q}^{T}(\textbf{A}+\textbf{B})\textbf{Q}\Big{)} \\ =Tr\Big{(}\textbf{Q}^{T}\textbf{M}^{T}\textbf{MQ}\Big{)}=||\textbf{QM}||_{F}^{2} \end{array}\]

which is strictly convex w.r.t. **Q** due to the strict convexity of the squared Frobenius norm (see _e.g._ 3.1 in Dattorro (2005)). Since the sum of strictly convex functions of **Q** (first trace in \(\Phi(\textbf{Q})\)) and linear functions of **Q** (second trace in \(\Phi(\textbf{Q})\)) is still strictly convex w.r.t. **Q**, we conclude that the objective function \(\Phi\) is strictly convex w.r.t. **Q**.

**Corollary.** _The retrofitting update procedure is insensitive to the order in which nodes are updated._

The aforementioned updating procedure for **Q** Faruqui et al. (2015) is derived from Jacobi iteration procedure Saad (2003); Bengio et al. (2006) and converges for any initialization. Such a convergence result is discussed in Bengio et al. (2006). It can also be directly verified in our specific setting by checking that each irreducible element of \(\textbf{A}+\textbf{B}\), _i.e._ each connected component of the underlying graph constructed from this matrix, is irreducibly diagonally dominant (see 4.2.3 in Saad (2003)) and then by applying Theorem 4.9 from Saad (2003) on each of these components. Besides, due to its strict convexity w.r.t. **Q**, the objective function \(\Phi\) admits a unique global minimum. Consequently, the retrofitting update procedure will converge to the same embedding matrix regardless of the order in which nodes are updated.

## Appendix B Extended Results

The following Tables 5 and 6 provide more complete results from our experiments



\begin{table}
\begin{tabular}{|c|l l c|l c c|l c|} \hline
**Pair** & **FT\({}_{avg}\)** & **XLM\({}_{avg}\)** & **mBERT\({}_{avg}\)** & **FT\({}_{sif}\)** & **XLM\({}_{sif}\)** & **mBERT\({}_{sif}\)** & **XLM\({}_{ext}\)** & **mBERT\({}_{ext}\)** \\ \hline

[MISSING_PAGE_POST]

m** & 64.5 \(\pm\) 0.9 & 74.7 \(\pm\) 0.9 & 77.1 \(\pm\) 1.0 & 78.5 \(\pm\) 0.2 & 80.7 \(\pm\) 0.3 & 80.7 \(\pm\) 0.9 & 79.9 \(\pm\) 0



\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Pair** & **DBP\({}_{\alpha\Omega}\)NNDist** & **Rfit\({}_{\alpha\Omega}\)FT\({}_{ssf}\)** & **Rfit\({}^{\alpha\Omega}_{\alpha\Omega}\)FT\({}_{sif}\)** \\ \hline
**en-nl** & 83.7 ± 0.1 & 90.0 ± 0.1 & **90.7 ± 0.0** \\
**en-fr** & 82.7 ± 0.3 & 90.8 ± 0.2 & **91.7 ± 0.1** \\
**en-es** & 81.1 ± 0.3 & 89.9 ± 0.1 & **91.4 ± 0.2** \\
**en-cs** & 86.6 ± 0.3 & 90.4 ± 0.3 & **91.6 ± 0.4** \\
**en-ja** & 81.3 ± 0.1 & 86.7 ± 0.3 & **89.1 ± 0.2** \\ \hline
**Pair** & **DBP\({}_{\alpha\Omega}\)NNDist** & **Rfit\({}_{\alpha\Omega}\)FT\({}_{ssf}\)** & **Rfit\({}^{\alpha\Omega}_{\alpha\Omega}\)FT\({}_{sif}\)** \\ \hline

[MISSING_PAGE_POST]

n** & 70.4 ± 0.4 & **81.6 ± 0.8** & **82.3 ± 0.5** \\ \hline \end{tabular}
\end{table}
Table 6: Macro-AUC scores (in %) with vectors learned by leveraging aligned monolingual ontologies. The first column shows the results by relying on the aligned ontologies only. The second and third columns show the results obtained by retrofitting \(\textbf{FT}_{sif}\) embeddings to monolingual ontologies, respectively, aligned monolingual ontologies.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline
**Embedding Model** & **Dimension** \\ \hline
**FT** & 300 \\
**mBERT** & 768 \\
**XLM** & 2048 \\
**LASER** & 1024 \\ \hline \end{tabular}
\end{table}
Table 7: Embedding dimensions for pre-trained models used in our study, corresponding to the values provided and optimized by each model’s authors.