Venue: NAACL 2019

# Analyzing Bayesian Crosslingual Transfer in Topic Models

 Shudong Hao

Boulder, CO

shudonghao@gmail.com &Michael J. Paul

Information Science

University of Colorado

Boulder, CO

mpaul@colorado.edu

###### Abstract

We introduce a theoretical analysis of crosslingual transfer in probabilistic topic models. By formulating posterior inference through Gibbs sampling as a process of language transfer, we propose a new measure that quantifies the loss of knowledge across languages during this process. This measure enables us to derive a PAC-Bayesian bound that elucidates the factors affecting model quality, both during training and in downstream applications. We provide experimental validation of the analysis on a diverse set of five languages, and discuss best practices for data collection and model design based on our analysis.

## 1 Introduction

Crosslingual learning is an important area of natural language processing that has driven applications including text mining in multiple languages Ni et al. (2009); Smet and Moens (2009), cultural difference detection Gutierrez et al. (2016), and various linguistic studies Shutova et al. (2017); Barrett et al. (2016). Crosslingual learning methods generally extend monolingual algorithms by using various multilingual resources. In contrast to traditional high-dimensional vector space models, modern crosslingual models tend to rely on learning low-dimensional word representations that are more efficient and generalizable.

A popular approach to representation learning comes from the word embedding community, in which words are represented as vectors in an embedding space shared by multiple languages Ruder et al. (2018); Faruqui and Dyer (2014); Klementiev et al. (2012). Another direction is from the topic modeling community, where words are projected into a probabilistic topic space Ma and Nasukawa (2017); Jagarlamudi and III (2010). While formulated differently, both types of models apply the same principles--low-dimensional vectors exist in a shared crosslingual space, wherein vector representations of similar concepts across languages (_e.g._, "dog" and "hund") should be nearby in the shared space.

To enable crosslingual representation learning, knowledge is transferred from a source language to a target language, so that representations have similar values across languages. In this study, we will focus on probabilistic topic models, and "knowledge" refers to a word's probability distribution over topics. Little is known about the characteristics of crosslingual knowledge transfer in topic models, and thus this paper provides an analysis, both theoretical and empirical, of crosslingual transfer in multilingual topic models.

### Background and Contributions

Multilingual Topic ModelsGiven a multilingual corpus \(D^{(1,...,L)}\) in languages \(\ell=1,\ldots,L\) as inputs, a multilingual topic model learns \(K\) topics. Each multilingual topic \(k^{(1,...,L)}\) (\(k=1,\ldots,K\)), is defined as an \(L\)-dimensional tuple \(\left(\phi_{k}^{(1)},\ldots,\phi_{k}^{(L)}\right)\), where \(\phi_{k}^{(\ell)}\) is a multinomial distribution over the vocabulary \(V^{(\ell)}\) in language \(\ell\). From a human's perspective, a multilingual topic \(k^{(1,...,L)}\) can be interpreted by looking at the word types that have \(C\) highest probabilities in \(\phi_{k}^{(\ell)}\) for each language \(\ell\). \(C\) here is called _cardinality_ of the topic. Thus, a multilingual topic can loosely be thought of as a group of word lists where each language \(\ell\) has its own version of the topic.

Multilingual topic models are generally extended from Latent Dirichlet Allocation Blei et al. (2003), lda). Though many variations have been proposed, the underlying structures of multilingual topic models are similar. These models require either a parallel/comparable corpus in multiple languages, or word translations from a

dictionary. One of the most popular models is the polylingual topic model (Mimno et al., 2009, pltm), where comparable document pairs share distributions over topics \(\theta\), while each language \(\ell\) has its own distributions \(\{\phi_{k}^{(\ell)}\}_{k=1}^{K}\) over the vocabulary \(V^{(\ell)}\). By re-marginalizing the estimations \(\{\widehat{\phi}_{k}^{(\ell)}\}_{k=1}^{K}\), we obtain word representations \(\widehat{\varphi}^{(w)}\in\mathbb{R}^{K}\) for each word \(w\), where \(\widehat{\varphi}_{k}^{(w)}=\Pr(z_{w}=k|w)\), _i.e.,_ the probability of topic \(k\) given a word type \(w\).

Crosslingual TransferKnowledge transfer through crosslingual representations has been studied in prior work. Smet and Moens (2009) and Heyman et al. (2016) show empirically how document classification using topic models implements the ideas of crosslingual transfer, but to date there has been no theoretical framework to analyze this transfer process in detail.

In this paper, we describe two types of transfer--on-site and off-site--based on the nature of where and how the transfer takes place. We refer to transfer that happens while training topic models (_i.e.,_ during representation learning) as _onsite_. Once we obtain the low-dimensional representations, they can be used for downstream tasks. We refer to transfer in this phase as _off-site_, since the crosslinkingal tasks are usually detached from the process of representation learning.

ContributionsOur study provides a theoretical analysis of crosslingual transfer learning in topic models. Specifically, we first formulate on-site transfer as circular validation, and derive an upper bound based on PAC-Bayesian theories (Section 2). The upper bound explicitly shows the factors that can affect knowledge transfer. We then move on to off-site transfer, and focus on crosslingual document classification as a downstream task (Section 3). Finally, we show experimentally that the on-site transfer error can have impact on the performance of downstream tasks (Section 4).

## 2 On-Site Transfer

On-site transfer refers to the training procedure of multilingual topic models, which usually involves Bayesian inference techniques such as variational inference and Gibbs sampling. Our work focuses on the analysis of collapsed Gibbs sampling Griffiths and Steyvers (2004), showing how knowledge is transferred across languages and how a topic space is formed through the sampling process. To this end, we first describe a specific formulation of knowledge transfer in multilingual topic models as a starting point of our analysis (Section 2.1). We then formulate Gibbs sampling as circular validation and quantify a loss during this phase (Section 2.2). This formulation leads us to a PAC-Bayesian bound that explicitly shows the factors that affect the crosslinked training (Section 2.3). Lastly, we look further into different transfer mechanisms in more depth (Section 2.4).

### Transfer through Priors

Priors are an important component in Bayesian models like pltm. In the original generative process of pltm, each comparable document pair \((d_{S},d_{T})\) in the source and target languages \((S,T)\) is generated by the same multinomial \(\theta\sim\mathrm{Dir}(\alpha)\). Hao and Paul (2018) showed that knowledge transfer across languages happens through priors. Specifically, assume the source document is generated from \(\theta^{(d_{S})}\sim\mathrm{Dir}(\alpha)\), and has a sufficient statistics \(\mathbf{n}_{d_{S}}\in\mathbb{N}^{K}\) where each cell \(n_{k|d_{S}}\) is the count of topic \(k\) in document \(d_{S}\). When generating the corresponding comparable document \(d_{T}\), the Dirichlet prior of the distribution over topics \(\theta^{(d_{T})}\), instead of a symmetric \(\alpha\), is parameterized by \(\alpha+\mathbf{n}_{d_{S}}\). This formulation yields the same posterior estimation as the original joint model and is the foundation of our analysis in this section.

To see this transfer process more clearly, we look closer to the conditional distributions during sampling, and take pltm as an example. When sampling a token in target language \(x_{T}\), the Gibbs sampler calculates a conditional distribution \(\mathcal{P}_{x_{T}}\) over \(K\) topics, where a topic \(k\) is randomly drawn and assigned to \(x_{T}\) (denoted as \(z_{x_{T}}\)). Assume the token \(x_{T}\) is in document \(d_{T}\) whose comparable document in the source language is \(d_{S}\). The conditional distribution for \(x_{T}\) is

\[\mathcal{P}_{x,k} = \Pr(z_{x}=k;\mathbf{w}_{-},\mathbf{z}_{-})\] \[\propto \left(n_{k|d_{T}}+n_{k|d_{S}}+\alpha\right)\cdot\frac{n_{w_{T}| k}+\beta}{n_{:|k}+V^{(T)}\beta},\]

where the quantity \(n_{k|d_{S}}\) is added and thus transferred from the source document. Thus, the calculation of \(\mathcal{P}_{x}\) incorporates the knowledge transferred from the other language.

Now that we have identified the transfer process, we provide an alternative view of Gibbs sampling, _i.e.,_ circular validation, in the next section 



### Circular Validation

Circular validation (or reverse validation) was proposed by Zhong et al. (2010) and Bruzzone and Marconcini (2010) in transfer learning. Briefly, a learning algorithm \({\cal A}\) is trained on both source and target datasets (\(D_{S}\) and \(D_{T}\)), where the source is labeled and target is unlabeled. After predicting the labels for the target dataset using \({\cal A}\) (predictions denoted as \({\cal A}(D_{T})\)), circular validation trains another algorithm \({\cal A^{\prime}}\) in the reverse direction, _i.e.,_ uses \({\cal A}(D_{T})\) and \(D_{T}\) as the labeled dataset and \(D_{S}\) as the unlabeled dataset. The error is then evaluated on \({\cal A^{\prime}}(D_{S})\). This "train-predict-reverse-repeat" cycle has a similar flavor to the iterative manner of Gibbs sampling, which inspires us to look at the sampling process as circular validation.

Figure 1 illustrates this process. Suppose the Gibbs sampler is currently sampling \(x_{T}\) of word type \(w_{T}\) in target language \(T\). As discussed for Equation (1), the calculation of the conditional distribution \({\cal P}_{x_{T}}\) incorporates the knowledge transferred from the source language. We then treat the process of drawing a topic from \({\cal P}_{x_{T}}\) as a _classification_ of the token \(x_{T}\). Let \({\cal P}_{x_{T}}\) be a distribution over \(K\) unary _classifiers_, \(\left\{h_{k}\right\}_{k=1}^{K}\), and the \(k\)-th classifier labels the token as topic \(k\) with a probability of one:

\[h_{k}~{}\sim~{}{\cal P}_{x_{T}},\;\text{and}\;\Pr\left(z_{x_{T}}=k;h_{k} \right)~{}=~{}1. \tag{2}\]

This process is repeated between the two languages until the Markov chain converges.

The training of topic models is unsupervised, _i.e.,_ there is no ground truth for labeling a topic, which makes it difficult to analyze the effect of transfer learning. Thus, after calculating \({\cal P}_{x_{T}}\), we take an additional step called _reverse validation_, where we design and calculate a measure--circular validation loss--to quantify the transfer.

**Definition 1** (Circular validation loss, cvl).: _Let \({\cal S}_{w}\) be the set containing all the tokens of type \(w\) throughout the whole training corpus, and call it the sample of \(w\). Given a bilingual word pair \((w_{T},w_{S})\) where \(w_{T}\) is in target language \(T\) while \(w_{S}\) in source \(S\), let \({\cal S}_{w_{T}}\) and \({\cal S}_{w_{S}}\) be the samples for the two types respectively, and \(n_{w_{T}}\) and \(n_{w_{S}}\) the sizes of them. The empirical circular validation score (\(\widehat{\rm CVL}\)) is defined as_

\[\widehat{\rm CVL}(w_{T},w_{S}) = \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!

Proof.: See Appendix. 

### PAC-Bayes View

A question following the formulation of \(\widehat{\text{cvL}}\) is, what factors could lead to better transfer during this process, particularly for semantically related words? To answer this, we turn to theory that bounds the performance of classifiers and apply this theory to this formulation of topic sampling as classification.

The PAC-Bayes theorem was introduced by McAllester (1999) to bound the performance of Bayes classifiers. Given a hypothesis set \(\mathcal{H}\), the majority vote classifier (or Bayes classifier) uses every hypothesis \(h\in\mathcal{H}\) to perform binary classification on an example \(\mathbf{x}\), and uses the majority output as the final prediction. Since minimizing the error by Bayes classifier is NP-hard, an alternative way is to use a _Gibbs classifier_ as approximation. The Gibbs classifier first draws a hypothesis \(h\in\mathcal{H}\) according to a posterior distribution over \(\mathcal{H}\), and then uses this hypothesis to predict the label of an example \(\mathbf{x}\)(Germain et al., 2012). The generalization loss of this Gibbs classifier can be bounded as follows.

**Theorem 2** (PAC-Bayes theorem, McAllester (1999)).: _Let \(\mathcal{P}\) be a posterior distribution over all classifiers \(h\in\mathcal{H}\), and \(Q\) a prior distribution. With a probability at least \(1-\delta\), we have_

\[\mathcal{L}~{}\leq~{}\widehat{\mathcal{L}}+\sqrt{\frac{1}{2n}\left(\mathrm{ KL}\left(\mathcal{P}||Q\right)+\ln\frac{2\sqrt{n}}{\delta}\right)},\]

_where \(\mathcal{L}\) and \(\widehat{\mathcal{L}}\) are the general loss and the empirical loss on a sample of size \(n\)._

In our framework, a token \(x_{T}\) provides a posterior \(\mathcal{P}_{x_{T}}\) over \(K\) classifiers. The loss \(\widehat{\mathcal{L}}(x_{T},w_{S})\) is then calculated on a sample of \(\mathcal{S}_{w_{S}}\) in language \(S\). The following theorem shows that for a bilingual word pair \((w_{T},w_{S})\), the general cvl can be bounded with several quantities.

**Theorem 3**.: _Given a bilingual word pair \((w_{T},w_{S})\), with probability at least \(1-\delta\), the following bound holds:_

\[\textsc{cvl}(w_{T},w_{S})~{}\leq~{}\widehat{\textnormal{cvL}}( w_{T},w_{S})~{}+ \tag{3}\] \[\frac{1}{2}\sqrt{\frac{1}{n}\left(\mathrm{KL}_{w_{T}}+\mathrm{KL }_{w_{S}}+2\ln\frac{2}{\delta}\right)+\frac{\ln n^{*}}{n}},\] \[n=\min\big{\{}n_{w_{T}},n_{w_{S}}\big{\}},~{}~{}n^{*}=\max\big{\{} n_{w_{T}},n_{w_{S}}\big{\}}.\]

_For brevity we use \(\mathrm{KL}_{w}\) to denote \(\mathrm{KL}(\mathcal{P}_{x}||Q_{x})\), where \(\mathcal{P}_{x}\) is the conditional distribution from Gibbs sampling of token \(x\) with word type \(w\) that gives highest loss \(\widehat{\mathcal{L}}(x,w)\), and \(Q_{x}\) a prior._

Proof.: See Appendix. 

### Multilevel Transfer

Recall that knowledge transfer happens through priors in topic models (Section 2.1). Because the KL-divergence terms in Theorem 3 include this prior \(Q\), we can use this theorem to analyze the transfer mechanisms more concretely.

The conditional distribution for sampling a topic \(z_{x}\) for a token \(x\) during sampling can be factorized into document-topic and topic-word levels:

\[\begin{array}{rcl}\mathcal{P}_{x,k}&=&\Pr\left(z_{x}=k|w_{x}=w,\mathbf{w}_{ -},\mathbf{z}_{-}\right)\\ &=&\Pr\left(z_{x}=k|\mathbf{z}_{-}\right)\cdot\Pr\left(w_{x}=w|z_{x}=k, \mathbf{w}_{-},\mathbf{z}_{-}\right)\\ &\propto&\underbrace{\Pr\left(z_{x}=k|\mathbf{z}_{-}\right)}_{\textrm{document level}}\cdot \underbrace{\Pr\left(z_{x}=k|w_{x}=w,\mathbf{w}_{-}\right)}_{\textrm{word level}}\\ &\triangleq&\mathcal{P}_{\theta,x}\cdot\mathcal{P}_{\varphi,x,k},\\ &&\\ \mathcal{P}_{x}&\triangleq&\mathcal{P}_{\theta,x}\otimes\mathcal{P}_{\varphi, x},\end{array}\]

where \(\otimes\) is element-wise multiplication. Thus, we have the following inequality:

\[\mathrm{KL}\left(\mathcal{P}_{x}||Q_{x}\right)~{}=~{}\mathrm{KL} \left(\mathcal{P}_{\theta,x}\otimes\mathcal{P}_{\varphi,x}||Q_{\theta,x} \otimes Q_{\varphi,x}\right)\] \[\leq~{}\mathrm{KL}\left(\mathcal{P}_{\theta,x}||Q_{\theta,x} \right)+\mathrm{KL}\left(\mathcal{P}_{\varphi,x}||Q_{\varphi,x}\right),\]

and the KL-divergence term in Theorem 3 is simply the sum of the KL-divergences between the conditional and prior distributions on all levels.

Recall that pltm transfers knowledge at the document level, through \(Q_{\theta,x}\), by linking document translations together (Equation (1)). Assume the current token \(x\) is from a target document linked to a document \(d_{S}\) in the source language. Then the prior for \(\mathcal{P}_{\theta,x}\) is \(\widehat{\theta}^{(d_{S})}\), _i.e._, the normalized empirical distribution over topics of \(d_{S}\).

Since the words are generated _within_ each language under pltm, _i.e._, \(\phi_{k}^{(S)}\) is irrelevant to \(\phi_{k}^{(T)}\), no transfer happens at the word level. In this case, \(Q_{\varphi,x}\), the prior for \(\mathcal{P}_{\varphi,x}\), is simply a \(K\)-dimensional uniform distribution \(\mathcal{U}\). Then:

\[\mathrm{KL}_{w}~{}\leq~{}\mathrm{KL}\left(\mathcal{P}_{\theta,x}|| \widehat{\theta}^{(d_{S})}\right)+\mathrm{KL}\left(\mathcal{P}_{\varphi,x}|| \mathcal{U}\right)\] \[~{}~{}=~{}\underbrace{\mathrm{KL}\left(\mathcal{P}_{\theta,x}|| \widehat{\theta}^{(d_{S})}\right)}_{\textrm{crosslingual entropy}}+\log K -\underbrace{H(\mathcal{P}_{\varphi,x})}_{\textrm{monolingual entropy}}.\]

Thus, at levels where transfer happens (document- or word-level), a low crosslingual entropy is preferred, to offset the impact of monolingual entropy where no transfer happens.

 

Most multilingual topic models are generative admixture models in which the conditional probabilities can be factorized into different levels, thus KL-divergence term in Theorem 3 can be decomposed and analyzed in the same way as in this section for models that have transfer at other levels, such as Hao and Paul (2018), Heyman et al. (2016), and Hu et al. (2014). For example, if a model has word-level transfer, _i.e.,_ the model assumes that word translations share the same distributions, we have a KL-divergence term as,

\[{\rm KL}_{w} \leq~{}{\rm KL}\left({\cal P}_{\varphi,x}||\widehat{\varphi}^{( w)}\right)+{\rm KL}({\cal P}_{\theta,x}||{\cal U})\] \[=~{}{\rm KL}\left({\cal P}_{\varphi,x}||\widehat{\varphi}^{(ws)} \right)+\log K-H({\cal P}_{\theta,x}),\]

where \(w_{S}\) is the word translation to word \(w\).

## 3 Off-Site Transfer

Off-site transfer refers to language transfer that happens while applying trained topic models to downstream crosslingual tasks such as document classification. Because transfer happens using the trained representations, the performance of off-site transfer heavily depends on that of on-site transfer. To analyze this problem, we focus on the task of crosslingual document classification.

In crosslingual document classification, a _document classifier_, \(h\), is trained on documents from one language, and \(h\) is then applied to documents from another language. Specifically, after training bilingual topic models, we have \(K\) bilingual word distributions \(\{\widehat{\phi}_{k}^{(S)}\}_{k=1}^{K}\) and \(\{\widehat{\phi}_{k}^{(T)}\}_{k=1}^{K}\). These two distributions are used to infer document-topic distributions \(\widehat{\theta}\) on unseen documents in the test corpus, and each document is represented by the inferred distributions. A document classifier is then trained on the \(\widehat{\theta}\) vectors as features in source language \(S\) and tested on the target \(T\).

We aim to show how the generalization risk on target languages \(T\), denoted as \(R_{T}(h)\), is related to the training risk on source languages \(S\), \(\widehat{R_{S}}(h)\). To differentiate the loss and classifiers in this section from those in Section 2, we use the term "risk" here, and \(h\) refers to the document classifiers, not the topic labeling process by the sampler.

### Languages as Domains

Classic learning theory requires training and test sets to come from the same distribution \({\cal D}\), _i.e.,_\((\theta,y)\sim{\cal D}\), where \(\theta\) is the document representation (features) and \(y\) the document label (Valiant, 1984). In practice, however, corpora in \(S\) and \(T\) may be sampled from different distributions, _i.e.,_\(D^{(S)}=\{(\widehat{\theta}^{(d_{S})},y)\}\sim\widehat{\mathcal{D}}^{(S)}\) and \(D^{(T)}=\{(\widehat{\theta}^{(d_{T})},y)\}\sim\widehat{\mathcal{D}}^{(T)}\). We refer to these distributions as _document spaces_. To relate \(R_{T}(h)\) and \(\widehat{R_{S}}(h)\), therefore, we have to take their distribution bias into consideration. This is often formulated as a problem of domain adaptation, and here we can formulate this such that each language is treated as a "domain".

We follow the seminal work by Ben-David et al. (2006), and define \({\cal H}\)-distance as follows.

**Definition 2** (\({\cal H}\)-distance, Ben-David et al. (2006)).: _Let \({\cal H}\) be a symmetric hypothesis space, i.e., for every hypothesis \(h\in{\cal H}\) there exists its counterpart \(1-h\in{\cal H}\). We let \(m=\big{|}D^{(S)}\big{|}+\big{|}D^{(T)}\big{|}\), the total size of test corpus. The \({\cal H}\)-distance between \(\widehat{\mathcal{D}}^{(S)}\) and \(\widehat{\mathcal{D}}^{(T)}\) is defined as_

\[\frac{1}{2}\widehat{d}_{\cal H}\left(\widehat{\mathcal{D}}^{(S)}, \widehat{\mathcal{D}}^{(T)}\right)\] \[= \max_{h\in{\cal H}}\frac{1}{m}\sum_{\ell\in\{S,T\}}\sum_{{\bf x}_ {d}:h({\bf x}_{d})=\ell}\mathds{1}\left\{{\bf x}_{d}\in D^{(\ell)}\right\},\]

_where \({\bf x}_{d}\) is the representation for document \(d\), and \(h({\bf x}_{d})\) outputs the language of this document._

This distance measures how identifiable the languages are based on their representations. If source and target languages are from entirely different distributions, a classifier can easily identify language-specific features, which could affect performance of the document classifiers.

With \({\cal H}\)-distances, we have a measure of the "distance" between the two distributions \(\widehat{\mathcal{D}}^{(S)}\) and \(\widehat{\mathcal{D}}^{(T)}\). We state the following theorem from domain adaptation theory.

**Theorem 4** (Ben-David et al. (2006)).: _Let \(m\) be the corpus size of the source language, i.e., \(m=\big{|}D^{(S)}\big{|}\), \(c\) the VC-dimension of document classifiers \(h\in{\cal H}\), and \(\widehat{d}_{\cal H}\left(\widehat{\mathcal{D}}^{(S)},\widehat{\mathcal{D}}^{( T)}\right)\) the \({\cal H}\)-distance between two languages in the document space. With probability at least \(1-\delta\), we have the following bound,_

\[R_{T}(h) \leq~{}\widehat{R}_{S}(h)+\widehat{d}_{\cal H}\left(\widehat{ \mathcal{D}}^{(S)},\widehat{\mathcal{D}}^{(T)}\right)+\widehat{\lambda}+\] \[\sqrt{\frac{4}{m}\left(c\log\frac{2em}{c}+\log\frac{4}{\delta} \right)},\] (4) \[\widehat{\lambda} =~{}\min_{h\in{\cal H}}\widehat{R}_{S}(h)+\widehat{R}_{T}(h).\] (5) 

The term \(\widehat{\lambda}\) in Theorem 4 defines a _joint risk, i.e.,_ the training error on both source and target documents. This term usually cannot be estimated in practice since the labels for target documents are unavailable. However, we can still calculate this term for the purpose of analysis.

The theorem shows that the crosslingual classification risk is bounded by two critical components: the \(\mathcal{H}\)-distance, and the joint risk \(\widehat{\lambda}\). Interestingly, these two quantities are based on the same set of features with different labeling rules: for \(\mathcal{H}\)-distance, the label for each instance is its language, while \(\widehat{\lambda}\) uses the actual document label. Therefore, a better bound requires the consistency of features across languages, both in language and document labelings.

### From Words to Documents

Since consistency of features depends on the document representations \(\widehat{\theta}\), we need to trace back to the upstream training of topic models and show how the errors propagate to the formation of document representations. Thus, we first show the relations between \(\widetilde{\text{CVL}}\) and word representations \(\widehat{\varphi}\) in the following lemma.

**Lemma 1**.: _Given any bilingual word pair \((w_{T},w_{S})\), let \(\widehat{\varphi}^{(w)}\) denote the distribution over topics of word type \(w\). Then we have,_

\[1-\widehat{\varphi}^{(w_{T})\top}\cdot\widehat{\varphi}^{(w_{S})} \leq \widehat{\text{CVL}}(w_{T},w_{S}).\]

Proof.: See Appendix. 

We need to connect the word representations \(\widehat{\varphi}\), which are central to on-site transfer, to the document representations \(\widehat{\theta}\), which are central to off-site transfer. To do this, we make an assumption that the inferred distribution over topics \(\widehat{\theta}^{(d)}\) for each test document \(d\) is a weighted average over all word vectors, _i.e.,_\(\widehat{\theta}^{(d)}\propto\sum_{w}f_{w}^{d}\cdot\widehat{\varphi}^{(w)}\), where \(f_{w}^{d}\) is the normalized frequency of word \(w\) in document \(d\)(Arora et al., 2013). When this assumption holds, we can bound the similarity of document representations \(\widehat{\theta}^{(d_{S})}\) and \(\widehat{\theta}^{(d_{T})}\) in terms of word representations and hence their \(\widetilde{\text{CVL}}\).

**Theorem 5**.: _Let \(\widehat{\theta}^{(d_{S})}\) be the distribution over topics for document \(d_{S}\) (similarly for \(d_{T}\)), \(F(d_{S},d_{T})=\left(\sum_{w_{S}}{f_{w_{S}}^{d_{S}}}^{2}\cdot\sum_{w_{T}}{f_{ w_{T}}^{d_{T}}}^{2}\right)^{\frac{1}{2}}\) where \(f_{w}^{d}\) is the normalized frequency of word \(w\) in document \(d\), and \(K\) the number of topics. Then_

\[\widehat{\theta}^{(d_{S})\top}\cdot\widehat{\theta}^{(d_{T})}\] \[\leq F(d_{S},d_{T})\cdot\sqrt{K\cdot\sum_{w_{S},w_{T}}\big{(} \widehat{\text{CVL}}(w_{T},w_{S})-1\big{)}^{2}}.\]

Proof.: See Appendix. 

This provides a spatial connection between document pairs and word pairs they have. Many kernalized classifiers such as support vector machines (svm) explicitly use this inner product in the dual optimization objective (Platt, 1998). Since the inner product is directly related to the cosine similarity, Theorem 5 indicates that if two documents are spatially close, their inner product should be large, and thus the \(\widehat{\text{CVL}}\) of all word pairs they share should be small. In an extreme case, if \(\widehat{\text{CVL}}(w_{T},w_{S})=1\) for all the bilingual word pairs appearing in document pair \((d_{S},d_{T})\), then \(\widehat{\theta}^{(d_{S})\top}\cdot\widehat{\theta}^{(d_{T})}=0\), meaning the two documents are orthogonal and tend to be irrelevant topically.

With upstream training discussed in Section 2, we see that \(\widehat{\text{CVL}}\) has an impact on the consistency of features across languages. A low \(\widehat{\text{CVL}}\) indicates that the transfer from source to target is sufficient in two ways. First, languages share similar distributions, and therefore, it is harder to distinguish languages based on their distributions. Second, if there exists a latent mapping from a distribution to a label, it should produce similar labeling on both source and target data since they are similar. These two aspects correspond to the language \(\mathcal{H}\)-distance and joint risk \(\widehat{\lambda}\) in Theorem 4.

## 4 Experiments

We experiment with five languages: Arabic (ar, Semitic), German (de, Germanic), Spanish (es, Romance), Russian (ru, Slavic), and Chinese (zh, Sinitic). In the first two experiments, we pair each with English (en, Germanic) and train pltm on each language pair individually.

Training DataFor each language pair, we use a subsample of \(3,000\) Wikipedia comparable documents, _i.e.,_\(6,\!000\) documents in total. We set \(K=50\), and train pltm with default hyperparameters (McCallum, 2002). We run each experiment five times and average the results.

Test DataFor experiments with document classification, we use Global Voices (gv) in all five language pairs as test sets. Each document in this dataset has a "categories" attribute that can be used 

as the document label. In our classification experiments, we use _culture_, _technology_, and _education_ as the labels to perform multiclass classification.

EvaluationTo evaluate topic qualities, we use Crosslingual Normalized Pointwise Mutual Information Hao et al. (2018), cpnmi), an intrinsic metric of crosslingual topic coherence. For any bilingual word pair \((w_{T},w_{S})\),

\[\textsc{cnpmi}(w_{T},w_{S})~{}=~{}-\frac{\log\frac{\Pr(w_{T},w_{S})}{\Pr(w_{T}) \Pr(w_{S})}}{\log\Pr\left(w_{T},w_{S}\right)}, \tag{6}\]

where \(\Pr\left(w_{T},w_{S}\right)\) is the occurrence of \(w_{T}\) and \(w_{S}\) appearing in the same pair of comparable documents. We use \(10{,}000\) Wikipedia comparable document pairs outside pltm training data for each language pair to calculate cmpmi scores. All datasets are publicly available at [http://opus.nlpl.eu/](http://opus.nlpl.eu/) Tiedemann (2012). Additional details of our datasets and experiment setup can be found in the appendix.

### Sampling as Circular Validation

Our first experiment shows how \(\widehat{\mathrm{CVL}}\) changes over time during Gibbs sampling. According to the definition, the arguments of \(\widehat{\mathrm{CVL}}\) can include any bilingual word pairs; however, we suggest that it should be calculated specifically among word pairs that are expected to be related (and thus enable transfer). In our experiments, we select word pairs in the following way.

Recall that the output of a bilingual topic model is \(K\) topics, where each language has its own distribution. For each topic \(k\), we can calculate \(\widehat{\mathrm{CVL}}(w_{S},w_{T})\) such that \(w_{S}\) and \(w_{T}\) belong to the same topic (_i.e.,_ are in the top \(C\) most probable words in that topic), from the two languages, respectively. Using a cardinality \(C\) for each of the \(K\) topics, we have in total \(C^{2}\times K\) bilingual word pairs in the calculation of \(\widehat{\mathrm{CVL}}\).

At certain iterations, we collect the topic words as described above with cardinality \(C=5\), and calculate \(\widehat{\mathrm{CVL}}(w_{T},w_{S})\), \(\textsc{cnpmi}(w_{T},w_{S})\), and the error term (the \(\frac{1}{2}\sqrt{\cdots}\) term in Theorem 3) of all the bilingual word pairs. In the middle panel of Figure 2, \(\widehat{\mathrm{CVL}}\) over all word pairs from topic words is decreasing as sampling proceeds and becomes stable by the end of sampling. On the other hand, the correlations between cnpmi and \(\widehat{\mathrm{CVL}}\) are constantly decreasing. The negative correlations between \(\widehat{\mathrm{CVL}}\) and cnpmi implies that lower \(\widehat{\mathrm{CVL}}\) is associated with higher topic quality, since higherquality topic has higher cnpmi but lower \(\widehat{\mathrm{CVL}}\).

### What the PAC-Bayes Bound Shows

Theorem 3 provides insights into how knowledge is transferred during sampling and the factors that could affect this process. We analyze this bound from two aspects, the size of the training data (corresponding to \(\frac{\ln n^{*}}{n}\) term) and model assumptions (as in the crosslingual entropy terms).

#### 4.2.1 Training Data: Downsampling

One factor that could affect \(\widehat{\mathrm{CVL}}\), according to Theorem 3, is the balance of tokens of a word pair. In an extreme case, if a word type \(w_{S}\) has only one token, while another word type \(w_{T}\) has a large number of tokens, the transfer from \(w_{S}\) to \(w_{T}\) is negligible. In this experiment, we will test if increasing the ratio term \(\frac{\ln n^{*}}{n}\) in the corpus lowers the performance of crosslingual transfer learning.

To this end, we specify a _sample rate_\(\rho=0.2,0.4,0.6,0.8,\) and \(1.0\). For each word pair \((w_{T},w_{S})\), we calculate \(n\) as in the ratio term \(\frac{\ln n^{*}}{n}\), and remove \((1-\rho)\cdot n\) tokens from the corpus (rounded to the nearest integer). Smaller \(\rho\) removes more tokens from the corpus and thus yields a larger ratio term on average.

We use a dictionary from Wiktionary to collect word pairs, where each word pair \((w_{S},w_{T})\) is a translation pair. Figure 3 shows the results of downsampling using these two methods. Decreasing the sample rate \(\rho\) lowers the topic qualities. This implies that although pltm can process comparable corpora, which need not be exact translations, one still needs to be careful about the token balance between linked document pairs.

For many low-resource languages, the target language corpus is much smaller than the source corpus, so the effect of this imbalance is important to be aware of. This is an important issue when choosing comparable documents, and Wikipedia is an illustrative example. Although one can collect comparable documents via Wikipedia's interlanguage links, articles under the same title but in different languages can have very large variations on document length, causing the imbalance of samples \(\frac{\ln n^{*}}{n}\), and thus potentially suboptimal performance of crosslingual training.

#### 4.2.2 Model Assumptions

Recall that the crosslingual entropy term can be decomposed into different levels, _e.g.,_ document 

level and word level, and we prefer a model with low crosslingual entropy but high monolingual entropy. In this experiment, we show how these two quantities affect the topic qualities, using English-German (en-de) documents as an example.

Given pltm output in (en,de) and a cardinality \(C=5\), we collect \(C^{2}\times K\) bilingual word pairs as described in Section 4.1. For each word pair, we calculate three quantities: \(\widehat{\textsc{cvl}}\), cnpmi, and the inner product of the word representations. In Figure 4, each dot is a word pair \((w_{S},w_{T})\) colored by the values of these quantities. The word pair dots are positioned by their crosslinkingal and monolingual entropies.

We observe that \(\widehat{\textsc{cvl}}\) decreases with crosslingual entropy on document level. The larger the crosslinkal entropy, the harder it is to get a low \(\widehat{\textsc{cvl}}\) because it needs larger monolingual entropy to decrease the bound, as shown in Section 2.4. On the other hand, the inner product of word pairs shows an opposite pattern of \(\widehat{\textsc{cvl}}\), indicating a negative correlation (Lemma 1). In Figure 2 we see the correlation between cmpmi and \(\widehat{\textsc{cvl}}\) is around \(-0.4\) at the end of sampling, so there are fewer clear patterns for cnpmi in Figure 4. However, we also notice that the word pairs with higher cnpmi scores often appear at the bottom where crosslingual entropy is low while the monolingual entropy is high.

### Downstream Task

We move on to crosslingual document classification as a downstream task. At various iterations of Gibbs sampling, we infer topics on the test sets for another \(500\) iterations and calculate the quantities shown in the Figure 5 (averaged over all languages), including the \(\mathcal{H}\)-distances for both training and test sets, and the joint risk \(\widehat{\lambda}\).

We treat English as the source language and train support vector machines to obtain the best classifier \(h^{\star}\) that fits the English documents. This classifier is then used to calculate the source and target risks \(\widehat{R}_{S}(h^{\star})\) and \(\widehat{R}_{T}(h^{\star})\). We also include \(\frac{1}{2}\widehat{d}_{\mathcal{H}}\left(S,T\right)\), the \(\mathcal{H}\)-distance based on word rep

Figure 4: Each dot is a (EN,DE) word pair, and its color shows corresponding values of the indicated quantity. Best viewed in color.

Figure 3: Increasing \(\rho\) results in smaller values of \(\frac{\ln n^{*}}{n}\) for translation pairs. Topic quality, evaluated by cnpmi, increases as well.

Figure 2: As Gibbs sampling progresses, \(\widehat{\textsc{cvl}}\) of topic words drops, which leads to higher quality topics, and thus increases cynmi. The left panel shows this negative correlation, and we use shades to indicate standard deviations across five chains.

 

resentations \(\widehat{\varphi}\). As mentioned in Section 3.1, we train support vector machines to use languages as labels, and the accuracy score as the -distance.

The classification risks, such as , , are decreasing as expected (upper row in Figure 5), which shows very similar trends as in Figure 2. On the other hand, we notice that the -distances of training documents and vocabularies, , stabilize around to , meaning it is difficult to differentiate the languages based on their representations. Interestingly, the -distances of test documents are at a less ideal value, although they are slightly decreasing in most of the languages except Ar. However, recall that the target risk also depends on other factors than -distance (Theorem 4), and we use Figure 6 to illustrate this point.

We further explore the relationship between the predictability of languages vs document classes in Figure 6. We collect documents correctly classified for both document class and language labels, from which we randomly choose documents for each language, and use to plot t-SNE scatterplots. Note that the two plots are from the same set of documents, and so the spatial relations between any two points are fixed, but we color them with different labelings. Although the classifier can identify the languages (right panel), the features are still consistent, because on the left panel, the decision boundary changes its direction and also successfully classifies the documents based on actual label class. This illustrates why a single - distance does not necessarily mean inconsistent features across languages and high target risks.

## 5 Conclusions and Future Directions

This study gives new insights into crosslingual transfer learning in multilingual topic models. By formulating the inference process as a circular validation, we derive a PAC-Bayesian theorem to show the factors that affect the success of crosslingual learning. We also connect topic model learning with downstream crossingual tasks to show how errors propagate.

As the first step toward more theoretically justified crosslinkingal transfer learning, our study suggests considerations for constructing crosslinked transfer models in general. For example, an effective model should strengthen crosslingual transfer while minimizing non-transferred components, use a balanced dataset or specific optimization algorithms for low-resource languages, and support evaluation metrics that relate to cvl.

Figure 5: Gibbs sampling optimizes , which decreases the joint risk and -distances for test data.

 



## References

* Arora et al. (2013) Sanjeev Arora, Rong Ge, Yonatan Halpern, David M. Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. 2013. A Practical Algorithm for Topic Modeling with Provable Guarantees. In _Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013_, pages 280-288.
* Barrett et al. (2016) Maria Barrett, Frank Keller, and Anders Sogaard. 2016. Cross-lingual Transfer of Correlations between Parts of Speech and Gaze Features. In _COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan_, pages 1330-1339.
* Ben-David et al. (2006) Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Analysis of Representations for Domain Adaptation. In _Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006_, pages 137-144.
* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. _Journal of Machine Learning Research_, 3:993-1022.
* Bruzzone and Marconcini (2010) Lorenzo Bruzzone and Mattia Marconcini. 2010. Domain Adaptation Problems: A DASVM Classification Technique and a Circular Validation Strategy. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 32(5):770-787.
* Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014. Improving Vector Space Word Representations Using Multilingual Correlation. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden_, pages 462-471.
* Germain et al. (2012) Pascal Germain, Amaury Habrard, Francois Laviolette, and Emilie Morvant. 2012. PAC-Bayesian Learning and Domain Adaptation. _CoRR_, abs/1212.2340.
* Griffiths and Steyvers (2004) Thomas L Griffiths and Mark Steyvers. 2004. Finding Scientific Topics. _Proceedings of the National academy of Sciences_, 101(suppl 1):5228-5235.
* Gutierrez et al. (2016) E. Dario Gutierrez, Ekaterina Shutova, Patricia Lichtenstein, Gerard de Melo, and Luca Gilardi. 2016. Detecting Cross-cultural Differences Using a Multilingual Topic Model. _Transactions of the Association for Computational Linguistics_, 4:47-60.
* Hao et al. (2018) Shudong Hao, Jordan L. Boyd-Graber, and Michael J. Paul. 2018. Lessons from the Bible on Modern Topics: Low-Resource Multilingual Topic Model Evaluation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 1090-1100.
* Hao and Paul (2018) Shudong Hao and Michael J. Paul. 2018. Learning Multilingual Topics from Incomparable Corpora. In _Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018_, pages 2595-2609.
* Heyman et al. (2016) Geert Heyman, Ivan Vulic, and Marie-Francine Moens. 2016. C-BiLDA: Extracting Cross-lingual Topics from Non-parallel Texts by Distinguishing Shared from Unshared Content. _Data Mining and Knowledge Discovery_, 30(5):1299-1323.
* Hu et al. (2014) Yuening Hu, Jordan L. Boyd-Graber, Brianna Satinoff, and Alison Smith. 2014. Interactive Topic Modelling. _Machine Learning_, 95(3):423-469.
* Jagarlamudi and Daume III (2010) Jagadeesh Jagarlamudi and Hal Daume III. 2010. Extracting Multilingual Topics from Unaligned Comparable Corpora. In _Advances in Information Retrieval, 32nd European Conference on IR Research, ECIR 2010, Milton Keynes, UK, March 28-31, 2010. Proceedings_, pages 444-456.
* Klementiev et al. (2012) Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing Crosslingual Distributed Representations of Words. In _COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India_, pages 1459-1474.
* Ma and Nasukawa (2017) Tengfei Ma and Tetsuya Nasukawa. 2017. Inverted Bilingual Topic Models for Lexicon Extraction from Non-parallel Data. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_, pages 4075-4081.
* McAllester (1999) David A. McAllester. 1999. PAC-Bayesian Model Averaging. In _Proceedings of the Twelfth Annual Conference on Computational Learning Theory, COLT 1999, Santa Cruz, CA, USA, July 7-9, 1999_, pages 164-170.
* Kachites McCallum (2002) Andrew Kachites McCallum. 2002. MALLIET: A Machine Learning for Language Toolkit.
* Mimno et al. (2009) David M. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual Topic Models. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 880-889.
* Ni et al. (2009) Xiaochuan Ni, Jian-Tao Sun, Jian Hu, and Zheng Chen. 2009. Mining Multilingual Topics from Wikipedia. In _Proceedings of the 18th International Conference on World Wide Web, WWW 2009, Madrid, Spain, April 20-24, 2009_, pages 1155-1156.
* Platt (1998) John Platt. 1998. Sequential minimal optimization: A fast algorithm for training support vector machines. Technical report.



Sebastian Ruder, Ivan Vulic, and Anders Sogaard. 2018. A Survey of Cross-lingual Word Embedding Models. _Journal of Artificial Intelligence Research_, abs/1706.04902.
* Shutova et al. (2017) Ekaterina Shutova, Lin Sun, E. Dario Gutierrez, Patricia Lichtenstein, and Srini Narayanan. 2017. Multilingual Metaphor Processing: Experiments with Semi-Supervised and Unsupervised Learning. _Computational Linguistics_, 43(1):71-123.
* De Smet and Moens (2009) Wim De Smet and Marie-Francine Moens. 2009. Cross-language Linking of News Stories on the Web Using Interlingual Topic Modelling. In _Proceedings of the 2nd ACM Workshop on Social Web Search and Mining, CIKM-SWSM 2009, Hong Kong, China, November 2, 2009_, pages 57-64.
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23-25, 2012_, pages 2214-2218.
* Valiant (1984) Leslie G. Valiant. 1984. A Theory of the Learnable. _Communications of the ACM_, 27(11):1134-1142.
* Zhong et al. (2010) Erheng Zhong, Wei Fan, Qiang Yang, Olivier Verscheure, and Jiangtao Ren. 2010. Cross Validation Framework to Choose amongst Models and Datasets for Transfer Learning. In _Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part III_, pages 547-562.

## Appendix A Notation

See Table 1.

## Appendix B Proofs

**Theorem 1**.: _Let \({\widehat{\textsc{cv}}^{(t)}}(w_{T},w_{S})\) be the empirical circular validation loss of any bilingual word pair at iteration \(t\) of Gibbs sampling. Then \({\widehat{\textsc{cv}}^{(t)}}(w_{T},w_{S})\) converges as \(t\to\infty\)._

Proof.: We first notice the triangle inequality:

\[\left|{\widehat{\textsc{cv}}^{(t)}}(w_{T},w_{S})-{\widehat{ \textsc{cv}}^{(t-1)}}(w_{T},w_{S})\right|\] \[= \left|{\mathop{\mathrm{E}}_{x_{j},x_{T}}\left[\widehat{Z}^{(t)}( x_{T},w_{S})+\widehat{\mathcal{L}}^{(t)}(x_{S},w_{T})\right]}\right.\] \[\quad-{\mathop{\mathrm{E}}_{x_{S},x_{T}}\left[\widehat{Z}^{(t-1)} (x_{T},w_{S})+\widetilde{\mathcal{L}}^{(t-1)}(x_{S},w_{T})\right]}\right|\] \[= \left|{\mathop{\mathrm{E}}_{x_{T}\in\delta_{w_{T}}}\left[\widehat {\mathcal{L}}^{(t)}(x_{T},w_{S})\right]+{\mathop{\mathrm{E}}_{x_{S}\in\delta_ {w_{S}}}\left[\widehat{\mathcal{L}}^{(t)}(x_{S},w_{T})\right]}}\right.\] \[\quad-{\mathop{\mathrm{E}}_{x_{T}\in\delta_{w_{T}}}\left[\widehat {\mathcal{L}}^{(t-1)}(x_{S},w_{T})\right]}\Bigg{|}\] \[\leq \left|{\mathop{\mathrm{E}}_{x_{T}\in\delta_{w_{T}}}\left[\widehat {\mathcal{L}}^{(t)}(x_{T},w_{S})\right]-{\mathop{\mathrm{E}}_{x_{T}\in\delta_ {w_{T}}}\left[\widehat{\mathcal{L}}^{(t-1)}(x_{T},w_{S})\right]}}\right.\] \[\quad+{\mathop{\mathrm{E}}_{x_{S}\in\delta_{w_{S}}}\left[\widehat {\mathcal{L}}^{(t)}(x_{S},w_{T})\right]-{\mathop{\mathrm{E}}_{x_{S}\in\delta_ {w_{S}}}\left[\widehat{\mathcal{L}}^{(t-1)}(x_{S},w_{T})\right]}}\Bigg{|}\] \[\equiv \left|{\Delta\mathop{\mathrm{E}}_{x_{T}\in\delta_{w_{T}}}\left[ \widehat{\mathcal{L}}(x_{T},w_{S})\right]+\Delta\mathop{\mathrm{E}}_{x_{S} \in\delta_{w_{S}}}\left[\widehat{\mathcal{L}}(x_{S},w_{T})\right]}\right|\] \[\leq \left|{\Delta\mathop{\mathrm{E}}_{x_{T}\in\delta_{w_{T}}}\left[ \widehat{\mathcal{L}}(x_{T},w_{S})\right]}\right|+\left|{\Delta\mathop{ \mathrm{E}}_{x_{S}\in\delta_{w_{S}}}\left[\widehat{\mathcal{L}}(x_{S},w_{T}) \right]}\right|.\]

We look at the first term of the last equation, and the other term can be derived in the same way. We use \(\mathcal{P}_{x_{T}}\) to denote the invariant distribution of the conditional \(\mathcal{P}_{x_{T}}^{(t)}\) as \(t\to\infty\). Additionally, let \(\mathcal{P}_{x_{T},z_{B}}\) be the conditional probability for the to 

ken \(x_{T}\) being assigned to topic \(z_{x_{S}}\):

\[\mathcal{P}_{x_{T},z_{x_{S}}}=~{}\Pr\left(k=z_{x_{S}};w=w_{x_{T}},\mathbf{z}_{-}, \mathbf{w}_{-}\right).\]

Another assumption we made is once the source language is converged, we keep the states of it fixed. That is, \(z^{(t)}_{x_{S}}=z^{(t-1)}_{x_{S}}\), and only sample the target language. Taking the difference between the expectation at iterations \(t\) and \(t-1\), we have

\[\lim_{t\to\infty}\left|\Delta\mathop{\mathbb{E}}_{x_{T}\in S_{ w_{T}}}\left[\widehat{\cal L}(x_{T},w_{S})\right]\right|\] \[= \lim_{t\to\infty}\left|\mathop{\mathbb{E}}_{x_{T}\in S_{w_{T}}} \left[\widehat{\cal L}^{(t)}(x_{T},w_{S})\right]\right.\] \[-\mathop{\mathbb{E}}_{x_{T}\in S_{w_{T}}}\left[\widehat{\cal L}^ {(t-1)}(x_{T},w_{S})\right]\] \[= \lim_{t\to\infty}\left|\mathop{\mathbb{E}}_{x_{T}}\left[\frac{1} {n_{w_{S}}}\sum_{x_{S}}\mathop{\mathbb{E}}_{h\sim\mathcal{P}^{(t)}_{x_{T}}} \mathds{1}\left\{h(x_{S})\neq z^{(t)}_{x_{S}}\right\}\right]\right.\] \[-\mathop{\mathbb{E}}_{x_{T}}\left[\frac{1}{n_{w_{S}}}\sum_{x_{S}} \mathop{\mathbb{E}}_{h\sim\mathcal{P}^{(t)}_{x_{T}}}\mathds{1}\left\{h(x_{S}) \neq z^{(t-1)}_{x_{S}}\right\}\right]\right|\] \[= \lim_{t\to\infty}\frac{1}{n_{w_{S}}}\sum_{x_{S}}\mathop{\mathbb{ E}}_{x_{T}}\left[\Big{|}\mathop{\mathbb{E}}_{h\sim\mathcal{P}^{(t)}_{x_{T}}} \mathds{1}\left\{h(x_{S})\neq z_{x_{S}}\right\}\right.\] \[\left.-\mathop{\mathbb{E}}_{h\sim\mathcal{P}^{(t-1)}_{x_{T}}} \mathds{1}\left\{h(x_{S})\neq z_{x_{S}}\right\}\right]\right|\] \[= \lim_{t\to\infty}\frac{1}{n_{w_{S}}}\sum_{x_{S}\in S_{w_{S}}} \mathop{\mathbb{E}}_{x_{T}\in S_{w_{T}}}\left[\Big{|}\big{(}1-\mathcal{P}^{( t)}_{x_{T},z_{x_{S}}}\big{)}\right]\] \[-\left(1-\mathcal{P}^{(t-1)}_{x_{T},z_{s}}\big{)}\Big{|}\right]\] \[= \lim_{t\to\infty}\frac{1}{n_{w_{S}}}\sum_{x_{S}\in S_{w_{S}}} \mathop{\mathbb{E}}_{x_{T}\in S_{w_{T}}}\left[\big{|}\rho_{x_{T},z_{x_{S}}}- \mathcal{P}_{x_{T},z_{x_{S}}}\big{|}\right]\] \[= 0.\]

Therefore, we have

\[\lim_{t\to\infty}\left|\widetilde{\rm vL}^{(t)}(w_{T},w_{S})- \widetilde{\rm cVL}^{(t-1)}(w_{T},ws)\right|\] \[\leq \lim_{t\to\infty}\left|\Delta\mathop{\mathbb{E}}_{x_{T}\in S_{ w_{T}}}\left[\widehat{\cal L}(x_{T},w_{S})\right]\right|\] \[+\left|\Delta\mathop{\mathbb{E}}_{x_{S}\in S_{w_{S}}}\left[ \widehat{\cal L}(x_{S},w_{T})\right]\right|\] \[= 0.\qed\]

**Theorem 3**.: _Given a bilingual word pair \((w_{T},w_{S})\), with probability at least \(1-\delta\), the following bound holds:_

\[\text{cvl}(w_{T},w_{S})\;\leq\;\widehat{\rm cVL}(w_{T},w_{S})\;+\] \[\frac{1}{2}\sqrt{\frac{1}{n}\left(\mathrm{KL}_{w_{T}}+\mathrm{ KL}_{w_{S}}+2\ln\frac{2}{\delta}\right)+\frac{\ln n^{\star}}{n}},\] \[n=\min\big{\{}n_{w_{T}},n_{w_{S}}\big{\}},\quad n^{\star}=\max \big{\{}n_{w_{T}},n_{w_{S}}\big{\}}.\]

_For brevity we use \(\mathrm{KL}_{w}\) to denote \(\mathrm{KL}(\mathcal{P}_{x}||Q_{x})\), where \(\mathcal{P}_{x}\) is the conditional distribution from Gibbs sampling of token \(x\) with word type \(w\) that gives highest loss \(\widehat{\cal L}(x,w)\), and \(Q_{x}\) a prior._

Proof.: From Theorem 2, for target language, with probability at least \(1-\delta\),

\[\mathcal{L}(x_{T},w_{S})\] \[\leq \widehat{\cal L}(x_{T},w_{S})+\sqrt{\frac{\mathrm{KL}\left(\mathcal{ P}_{x_{T}}||Q_{x_{T}}\right)+\ln\frac{2\sqrt{n_{w_{S}}}}{\delta}}{2n_{w_{S}}}}\] \[= \widehat{\cal L}(x_{T},w_{S})+\sqrt{\frac{\mathrm{KL}\left(\mathcal{ P}_{x_{T}}||Q_{x_{T}}\right)+\ln\frac{2}{\delta}+\frac{\ln n_{w_{S}}}{2n_{w_{S}}}}{2}}\] \[\equiv \widehat{\cal L}(x_{T},w_{S})+\epsilon(x_{T},w_{S}).\]

For the source language, similarly, with probability at least \(1-\delta\),

\[\mathcal{L}(x_{S},w_{T})\] \[\leq \widehat{\cal L}(x_{S},w_{T})+\sqrt{\frac{\mathrm{KL}\left(\mathcal{ P}_{x_{S}}||Q_{x_{S}}\right)+\ln\frac{2}{\delta}+\frac{\ln n_{w_{T}}}{2}}{2n_{w_{ T}}}}\] \[\equiv \widehat{\cal L}(x_{S},w_{T})+\epsilon(x_{S},w_{T}).\]

Given a word type \(w_{T}\), we notice that only the KL-divergence term in \(\epsilon(x_{T},w_{S})\) varies among different tokens \(x_{T}\). Thus, we use \(\mathrm{KL}_{w_{S}}\) and \(\mathrm{KL}_{w_{T}}\) to denote the maximal values of KL-divergence over all the tokens,

\[\mathrm{KL}_{w_{S}} = \mathrm{KL}\left(\mathcal{P}_{x_{T}^{\star}}||Q_{x_{T}^{\star}} \right),\] \[x_{T}^{\star} = \underset{x_{T}\in S_{w_{T}}}{\arg\max}\;\epsilon(x_{T},w_{S});\] \[\mathrm{KL}_{w_{T}} = \mathrm{KL}\left(\mathcal{P}_{x_{S}^{\star}}||Q_{x_{T}^{\star}} \right),\] \[x_{S}^{\star} = \underset{x_{S}\in S_{w_{S}}}{\arg\max}\;\epsilon(x_{S},w_{T}).\]

Let \(n=\min\left\{n_{w_{T}},n_{w_{S}}\right\}\), and \(n^{\star}=\max\left\{n_{w_{T}},n_{w_{S}}\right\}\). Due to the fact that \(\sqrt{x}+\sqrt{y}\leq\frac{2}{\sqrt{x}+y}\) for \(x,y>0\), we hav 

\[+\frac{1}{2}\left(\sqrt{\frac{1}{2n_{w_{T}}}\left(\text{KL}_{w_ {S}}+\ln\frac{2}{\delta}+\frac{1}{2}\ln n_{w_{T}}\right)}\right.\] \[+\sqrt{\frac{1}{2n_{w_{S}}}\left(\text{KL}_{w_{S}}+\ln\frac{2}{ \delta}+\frac{1}{2}\ln n_{w_{S}}\right)}\right)\] \[\leq\;\widetilde{\mathrm{CVL}}(w_{r},w_{s})\]

which gives us the result. 

**Lemma 1**.: _Given any bilingual word pair \((w_{T},w_{S})\), let \(\widehat{\varphi}^{(w)}\) denote the distribution over topics of word type \(w\). Then we have,_

\[1-\widehat{\varphi}^{(w_{T})\top}\cdot\widehat{\varphi}^{(w_{S})}\;\leq\; \widetilde{\mathrm{CVL}}(w_{T},w_{S}).\]

Proof.: We expand the equation of \(\widehat{\mathrm{CVL}}\) as follows,

\[\begin{array}{l}\widehat{\mathrm{cvL}}(w_{T},w_{S})\\ =&\frac{1}{2}\mathop{\mathds{E}}_{x_{S},x_{T}}\left[\widehat{ \mathcal{L}}(x_{T},w_{S})+\widehat{\mathcal{L}}(x_{S},w_{T})\right]\\ =&\frac{1}{2}\left(\mathop{\mathds{E}}_{x_{T}}\left[\widehat{ \mathcal{L}}(x_{T},w_{S})\right]+\mathop{\mathds{E}}_{x_{S}}\left[\widehat{ \mathcal{L}}(x_{S},w_{T})\right]\right)\\ =&\frac{1}{2}\left(\sum_{x_{T}\in S_{w_{T}}}\sum_{x_{S}\in S_{ w_{S}}}\mathop{\mathds{E}}_{h\sim p_{x_{T}}}\llbracket\mathds{1} \left\{h(x_{S})\neq z_{x_{S}}\right\}\rrbracket\right)\\ =&\frac{1}{2}\left(\frac{\sum_{x_{T}\in S_{w_{T}}}\sum_{x_{S}\in S _{w_{S}}}\mathop{\mathds{E}}_{h\sim p_{x_{T}}}\llbracket\mathds{1}\left\{h( x_{T})\neq z_{x_{T}}\right\}}{n_{w_{S}}\cdot n_{w_{T}}}\right)\\ =&\frac{1}{2}\left(\frac{\sum_{x_{T}\in S_{w_{T}}}\sum_{x_{S} \in S_{w_{S}}}\left(1-\mathcal{P}_{x_{T},z_{x_{S}}}\right)}{n_{w_{T}}\cdot n_{ w_{S}}}\right.\\ &\quad+\frac{\sum_{x_{S}\in S_{w_{S}}}\sum_{x_{T}\in S_{w_{T}}}\left(1- \mathcal{P}_{x_{S},z_{x_{T}}}\right)}{n_{w_{S}}\cdot n_{w_{T}}}\right)\\ =&1-\frac{1}{2}\left(\frac{\sum_{x_{T}\in S_{w_{T}}}\sum_{x_{ S}\in S_{w_{S}}}\mathcal{P}_{x_{T},z_{x_{S}}}}{n_{w_{T}}\cdot n_{w_{S}}}\right.\\ &&\qquad\left.+\frac{\sum_{x_{S}\in S_{w_{S}}}\sum_{x_{ T}\in S_{w_{T}}}p_{x_{S},z_{x_{T}}}}{n_{w_{S}}\cdot n_{w_{T}}}\right)\\ =&1-\frac{1}{2}\sum_{k=1}^{K}\left(\widehat{\varphi}^{(w_{S})}_{ k}\cdot\frac{\sum_{x_{T}\in S_{w_{T}}}p_{x_{T},k}}{n_{w_{T}}}\right.\\ =&1-\frac{1}{2}\sum_{k=1}^{K}\left(\widehat{\varphi}^{(w_{S})}_{ k}\cdot\frac{\sum_{x_{T}\in S_{w_{T}}}p_{x_{T},k}}{n_{w_{T}}}\right.\\ &\quad+\widehat{\varphi}^{(w_{T})}_{k}\cdot\frac{\sum_{x_{S}\in S_{w_{S}}}p_{ x_{S},z_{x_{T}}}}{n_{w_{S}}}\right)\\ \geq&1-\frac{1}{2}\sum_{k=1}^{K}\left(\widehat{\varphi}^{(w_{S})}_{ k}\cdot\frac{n_{k|w_{T}}}{n_{w_{T}}}+\widehat{\varphi}^{(w_{T})}_{k}\cdot \frac{n_{k|w_{S}}}{n_{w_{S}}}\right)\\ =&1-\frac{1}{2}\sum_{k=1}^{K}\left(\widehat{\varphi}^{(w_{S})}_{ k}\cdot\widehat{\varphi}^{(w_{T})}_{k}+\widehat{\varphi}^{(w_{T})}_{k}\cdot \widehat{\varphi}^{(w_{S})}_{k}\right)\\ =&1-\widehat{\varphi}^{(w_{S})\top}\cdot\widehat{\varphi}^{(w_{T})}\end{array}\]

which concludes the proof. 

**Theorem 5**.: _Let \(\widehat{\theta}^{(d_{S})}\) be the distribution over topics for document \(d_{S}\) (similarly for \(d_{T}\)), \(F(d_{S},d_{T})=\left(\sum_{w_{S}}f^{d_{S}}_{w_{S}}\cdot\sum_{w_{T}}f^{d_{T}}_{ v_{T}}\cdot\widehat{\varphi}^{(w_{S})}_{k}\right)\) where \(f^{d}_{w}\) is the normalized frequency of word \(w\) in document \(d\), and \(K\) the number of topics. Then_

\[\begin{array}{rcl}\widehat{\theta}^{(d_{S})\top}\cdot\widehat{ \theta}^{(d_{T})}&\leq& F(d_{S},d_{T})\\ &\cdot&\sqrt{K\cdot\sum_{w_{S},w_{T}}\big{(}\widehat{\mathrm{ cvL}}(w_{T},w_{S})-1\big{)}^{2}}.\end{array}\]

Proof.: We first expand the inner product of \(\widehat{\theta}^{(d_{S})\top}\cdot\widehat{\theta}^{(d_{T})}\) as follows,

\[\begin{array}{rl}&\widehat{\theta}^{(d_{S})\top}\cdot\widehat{ \theta}^{(d_{T})}\\ =&\sum_{k=1}^{K}\widehat{\theta}^{(d_{S})}_{k}\cdot\widehat{\theta}^{( d_{T})}_{k}\\ =&\sum_{k=1}^{K}\left(\left(\sum_{w_{S}\in V^{(S)}}f^{d_{S}}_{ w_{S}}\cdot\widehat{\varphi}^{(ws)}_{k}\right)\right.\\ &\quad\cdot\left(\sum_{w_{T}\in V^{(T)}}f^{d_{T}}_{v_{T}}\cdot\widehat{\varphi}^{( w_{T})}_{k}\right)\right)\\ \leq&F(d_{S},d_{T})\cdot\sum_{k=1}^{K}\left(\left(\sum_{w_{S}\in V^{(S)}} \widehat{\varphi}^{(w_{S})^{2}}_{k}\right)^{\frac{1}{2}}.\end{array}\] 

where \(F(d_{S},d_{T})\) is a constant independent of topic \(k\), and the last inequality due to Holder's. We then focus on the topic-dependent part of the last inequality.

\[\sum_{k=1}^{K}\left(\left(\sum_{w_{g}\in V^{(S)}}\widetilde{\varphi }_{k}^{(w_{g})^{2}}\right)^{\frac{1}{2}}\cdot\left(\sum_{w_{T}\in V^{(T)}} \widehat{\varphi}_{k}^{(w_{T})^{2}}\right)^{\frac{1}{2}}\right)\] \[= \sum_{k=1}^{K}\left(\sum_{w_{g},w_{T}}\left(\widehat{\varphi}_{k} ^{(w_{g})}\cdot\widehat{\varphi}_{k}^{(w_{T})}\right)^{2}\right)^{\frac{1}{2}}\] \[\leq \sqrt{K}\cdot\left(\sum_{k=1}^{K}\sum_{w_{g},w_{T}}\left(\widehat {\varphi}_{k}^{(w_{g})}\cdot\widehat{\varphi}_{k}^{(w_{T})}\right)^{2}\right)^ {\frac{1}{2}}\] \[= \sqrt{K}\cdot\left(\sum_{w_{g},w_{T}}\sum_{k=1}^{K}\left( \widehat{\varphi}_{k}^{(w_{g})}\cdot\widehat{\varphi}_{k}^{(w_{T})}\right)^{2 }\right)^{\frac{1}{2}}\] \[\leq \sqrt{K}\cdot\left(\sum_{w_{g},w_{T}}\left(\sum_{k=1}^{K} \widehat{\varphi}_{k}^{(w_{g})}\cdot\widehat{\varphi}_{k}^{(w_{T})}\right)^{2 }\right)^{\frac{1}{2}}\] \[= \sqrt{K}\cdot\left(\sum_{w_{g},w_{T}}\left(\widehat{\varphi}^{( w_{T})\top}\cdot\widehat{\varphi}^{(w_{g})}\right)^{2}\right)^{\frac{1}{2}}.\]

Thus, we have the following inequality:

\[\widehat{\theta}^{(d_{S})\top}\cdot\widehat{\theta}^{(d_{T})} \;\leq\;F(d_{S},d_{T})\cdot\sqrt{K}\] \[\cdot\left(\sum_{w_{g},w_{T}}\left(\widehat{\varphi}^{(w_{T}) \top}\cdot\widehat{\varphi}^{(w_{g})}\right)^{2}\right)^{\frac{1}{2}}.\]

Plug in Lemma 1, we see that

\[\widehat{\theta}^{(d_{S})\top}\cdot\widehat{\theta}^{(d_{T})} \;\leq\;F(d_{S},d_{T})\cdot\sqrt{K}\] \[\cdot\left(\sum_{w_{g},w_{T}}\left(\overline{\mathrm{CVL}}(w_{T} ,w_{S})-1\right)^{2}\right)^{\frac{1}{2}}.\]

## Appendix C Dataset Details

### Pre-processing

For all the languages, we use existing stemmers to stem words in the corpora and the entries in Wiktionary. Since Chinese does not have stemmers, we loosely use "stem" to refer to "segment" Chinese sentences into words. We also use fixed stopword lists to filter out stop words. Table 2 lists the source of the stemmers and stopwords.

### Training Sets

Our training set is a comparable corpus from Wikipedia. For each Wikipedia article page, there exists an interlingual link to view the article in another language. This interlingual link provides the same article in different languages and is commonly used to create comparable corpora in multilingual studies. We show the statistics of this training corpus in Table 3. The numbers are calculated after stemming and lemmatization.

### Test Sets

#### c.3.1 Topic Coherence Evaluation Sets

Topic coherence evaluation for multilingual topic models was proposed by Hao et al. (2018), where a comparable corpus is used to calculate bilingual word pair co-occurrence and cnpmi scores. We use a Wikipedia corpus to calculate this score, and the statistics are shown in Table 4. This Wikipedia corpus does not overlap with the training set.

#### c.3.2 Unseen Document Inference

We use the Global Voices (GV) corpus to create test sets, which can be retrieved from the website [https://globalvoices.org](https://globalvoices.org) directly, or from the OPUS collection at [http://opus.nlpl.eu/GlobalVoices.php](http://opus.nlpl.eu/GlobalVoices.php). We show the statistics in Table 5. After the column showing number of documents, we also include the statistics of specific labels. The multiclass labels are mutual exclusive, and each document has only one label.

Note that although all the language pairs share the same set of English test documents, the document representations are inferred from different topic models trained specifically for that language pair. Thus, the document representations for the same English document are different across different language pairs.

Lastly, the number of word types is based on the training set and after stemming and lemmatization. When a word type in the test set does not appear in the training set, we ignore this type.

#### c.3.3 Wiktionary

In downsampling experiments (Section 4.2), we use English Wiktionary to create bilingual dictionaries, which can be downloaded at [https://dumps.wikimedia.org/enwiktionary/](https://dumps.wikimedia.org/enwiktionary/).

 



## Appendix D Topic Model Configurations

For each experiment, we run five chains of Gibbs sampling using the Polylingual Topic Model implemented in MALLET, 5 and take the average over all chains. Each chain has \(1{,}000\) iterations, and we do not set a burn-in period. We set the topic number \(K=50\). Other hyperparameters are \(\alpha=\frac{50}{K}=1\) and \(\beta=0.01\) which are the default settings. We do not enable hyperparameter optimization procedures.

Footnote 5: [http://mallet.cs.umass.edu/topics-polylingual.php](http://mallet.cs.umass.edu/topics-polylingual.php).