Year: 2019

Abstract:
Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity problem in short text or small collection of documents. However, no prior work has employed (pretrained latent) topics in transfer learning paradigm. In this paper, we propose a framework to perform transfer learning in neural topic modeling using (1) pretrained (latent) topics obtained from a large source corpus, and (2) pretrained word and topic embeddings jointly (i.e., multiview) in order to improve topic quality, better deal with polysemy and data sparsity issues in a target corpus. In doing so, we first accumulate topics and word representations from one or many source corpora to build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). Then, we identify one or multiple relevant source domain(s) and take advantage of corresponding topics and word features via the respective pools to guide meaningful learning in the sparse target domain. We quantify the quality of topic and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. We have demonstrated the state-ofthe-art results on topic modeling with the proposed transfer learning approaches.

heading: 1 INTRODUCTION
text: Probabilistic topic models, such as LDA (Blei et al., 2003), Replicated Softmax (RSM) (Salakhutdinov & Hinton, 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) (Larochelle & Lauly, 2012) are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks, such as information retrieval (IR). Though they have been shown to be powerful in modeling large text corpora, the topic modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient e.g., on short text or a corpus of few documents. To this end, several works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) have introduced external knowledge in traditional topic models via word embeddings Pennington et al. (2014). However, no prior work in topic modeling has employed topical embeddings (obtained from large document collection(s)), complementary to word embeddings.
Local vs Global Views: Though word embeddings (Pennington et al., 2014) and topics are complementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. Word embeddings have local context (view) in the sense that they are learned based on local collocation pattern in a text corpus, where the representation of each word either depends on a local context window (Mikolov et al., 2013) or is a function of its sentence(s) (Peters et al., 2018). Consequently, the word occurrences are modeled in a fine-granularity. On other hand, a topic (Blei et al., 2003; Gupta et al., 2019) has a global word context (view): TM infers topic distributions across documents in the corpus and assigns a topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. Therefore, it learns from word occurrences across documents and encodes a coarse-granularity description. Unlike topics, the word embeddings do not capture thematic structures (topical semantics) underlying in the document collection.
Consider the following topics (Z1-Z4), where (Z1-Z3) are respectively obtained from different (high-resource) source (S1-S3) domains whereas Z4 from the (low-resource) target domain T in the data-sparsity setting:
Z1 (S1): profit, growth, stocks, apple, fall, consumer, buy, billion, shares→ Trading Z2(S2): smartphone, ipad, apple, app, iphone, devices, phone, tablet→ Product Line Z3 (S3): microsoft, mac, linux, ibm, ios, apple, xp, windows→ Operating System/Company Z4 (T ): apple, talk, computers, shares, disease, driver, electronics, profit, ios→ ?
Usually, the top words associated with topics learned on a large corpus are semantically coherent and represent meaningful semantics, e.g., Trading, Product Line, etc. However in sparse-data setting, topics (e.g., Z4) are incoherent (noisy) and therefore, it is difficult to infer meaningful semantics. Additionally, notice that the word apple is topically/thematically contextualized (topic-word association) by different semantics in S1-S3 and referring to a Company. Unlike the topics, word embeddings encode syntactic and semantic relatedness in fine-granularity and therefore, do not capture thematic structures. For instance, the top-5 nearest neighbors (NN) of apple (below) in the embeddings (Mikolov et al., 2013) space suggest that it refers to a fruit; however, they do not express anything about its thematic context, e.g., Health.
apple NN==⇒ apples, pear, fruit, berry, pears, strawberry Motivation (1) Knowledge transfer using pretrained word and topic embeddings: Essentially, the application of TM aims to discover hidden thematic structures (i.e., topics) in text collection; however, it is challenging in data sparsity settings, e.g, in a short and/or small collection. This leads to suboptimal text representations and incoherent topics (e.g., topic Z4).
To alleviate the data sparsity issues, recent works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) have shown that TM can be improved by introducing external knowledge, where they leverage pretrained word embeddings (i.e., local view) only. However, the word embeddings ignore the thematically contextualized structures (i.e., document-level semantics), and can not deal with ambiguity. Given that the word and topic representations encode complementary information, no prior work has explored transfer learning in TM using pretrained topics obtained from a large corpus.
Motivation (2) Knowledge transfer from multiple sources of word and topic embeddings: Knowledge transfer via word embeddings is vulnerable to negative transfer (Cao et al., 2010) on the target domain when domains are shifted and not handled properly. For instance, consider a short-text document v: [apple gained its US market shares] in the target domain T . Here, the word apple refers to a company, and hence the word vector of apple (about fruit) is an irrelevant source of prior knowledge for both v and the topic Z4. In contrast, one can better model v and amend the noisy Z4 for coherence, given the meaningful word and topic embeddings.
Often, there are several topic-word associations in different domains, e.g., in topics Z1-Z3. Given a noisy topic Z4 in T and meaningful topics Z1-Z3 of S1-S3, we identify multiple relevant (source) domains and advantageously transfer their word and topic embeddings in order to facilitate meaningful and positive transfer learning in the sparse corpus, T . Contribution (1) To our knowledge, it is the first work in unsupervised topic modeling framework that introduces (external) knowledge transfer using (a) Global-view Transfer: Pretrained topic em-
beddings instead of using word embeddings exclusively, and (b) Multi-view Transfer: Pretrained word and topic embeddings jointly obtained from a large source corpus in order to deal with polysemy and alleviate data sparsity issues in a small target corpus.
Contribution (2) Multi-source Transfer: Moreover, we first learn word and topic representations on multiple source domains to build WordPool and TopicPool, respectively and then perform multi-view and multi-source transfer learning within neural topic modeling by jointly using the complementary representations. In doing so, we guide the (unsupervised) generative process of learning hidden topics of the target domain by embeddings in WordPool and TopicPool such that the hidden topics become more meaningful and representative in explaining the target corpus.
We evaluate the effectiveness of our transfer learning approaches in neural topic modeling using 7 (5 low-resource and 2 high-resource) target and 5 (high-resource) source corpora from news and medical domains, consisting of short-text, long-text, small and large document collections. Particularly, we quantify the quality of text representations via generalization (perplexity), interpretability (topic coherence) and text retrieval. The code is provided with the supplementary.

heading: 2 KNOWLEDGE TRANSFER IN NEURAL TOPIC MODELING
text: Consider a sparse target domain T and a set of |S| source domains S, we first prepare two knowledge bases (KBs) of representations (or embeddings) from each of the sources: (1) WordPool: Pretrained word embeddings matrices {E1, ...,E|S|}, where Ek ∈ RE×K , and (2) TopicPool: Pretrained latent topic embeddings {Z1, ...,Z|S|}, where Zk ∈ RH×K encodes a distribution over a vocabulary of K words. E and H are word embedding and latent topic dimensions, respectively. While topic modeling on T , we introduce the two types of knowledge transfers from one or many sources: Local (LVT) and Global (GVT) View Transfer using the two KBs of pretrained word (i.e., WordPool) and topic (i.e., TopicPool) embeddings, respectively. Specially, we employ a neural autoregressive topic model (i.e., DocNADE (Larochelle & Lauly, 2012)) to build the WordPool and TopicPool.
Notice that a superscript indicates a source. See Table 1 for the notations used in this work.

heading: 2.1 NEURAL AUTOREGRESSIVE TOPIC MODELS
text: DocNADE (Larochelle & Lauly, 2012) is an unsupervised neural-network based topic model that is inspired by the benefits of NADE (Larochelle & Murray, 2011) and RSM (Salakhutdinov & Hinton, 2009) architectures. RSM has difficulties due to intractability leading to approximate gradients of the negative log-likelihood, while NADE does not require such approximations. On other hand, RSM is a generative model of word count, while NADE is limited to binary data. Specifically, Doc-
NADE factorizes the joint probability distribution of words in a document as a product of conditional distributions and efficiently models each conditional via a feed-forward neural network.
Algorithm 1 Computation of log p(v) and Loss L(v) Input: A target training document v, |S| source domains Input: WordPool: KB of pretrained word embedding matrices {E1, ...,E|S|} Input: TopicPool: KB of pretrained latent topics {Z1, ...,Z|S|} Parameters: Θ = {b, c,W,U,A1, ...,A|S|} Hyper-parameters: θ = {λ1, ..., λ|S|, γ1, ..., γ|S|, H} Initialize: a← c and p(v)← 1 for i from 1 to D do
hi(v<i)← g(a), where g = {sigmoid, tanh} p(vi = w|v<i)← exp(bw+Uw,:hi(v<i))∑
w′ exp(bw′+Uw′,:hi(v<i))
p(v)← p(v)p(vi|v<i) compute pre-activation at step, i: a← a + W:,vi if LVT then
get word embedding for vi from source domain(s) a← a + ∑|S| k=1 λ
k Ek:,vi L(v)← − log p(v) if GVT then L(v)← L(v) + ∑|S| k=1 γ k ∑H j=1 ||Akj,:W − Zkj,:||22
DocNADE Formulation: For a document v = (v1, ..., vD) of size D, each word index vi takes value in {1, ...,K} of vocabulary size K. DocNADE learns topics in a language modeling fashion (Bengio et al., 2003) and decomposes the joint distribution p(v)= ∏D i=1 p(vi|v<i) such that each autoregressive conditional p(vi|v<i) is modeled by a feed-forward neural network using preceding words v<i in the sequence:
hi(v<i) = g(c + ∑ q<i W:,vq ) and p(vi = w|v<i) = exp(bw + Uw,:hi(v<i))∑ w′ exp(bw′ + Uw′,:hi(v<i))
for i ∈ {1, ...D}, where v<i is the subvector consisting of all vq such that q < i i.e., v<i ∈ {v1, ..., vi−1}, g(·) is a non-linear activation function, W ∈ RH×K and U ∈ RK×H are weight matrices, c ∈ RH and b ∈ RK are bias parameter vectors. H is the number of hidden units (topics). Figure 1 (left) (without WordPool) provides an illustration of the ith autoregressive step of the DocNADE architecture, where the parameter W is shared in the feed-forward networks and hi encodes topic-proportion embedding. Importantly, the topic-word matrix W has a property that the column vector W:,vi corresponds to embedding of the word vi, whereas the row vector Wj,: encodes the jth topic. We leverage this property to introduce external knowledge via word and topic embeddings.
Additionally, DocNADE has shown to outperform traditional models such as LDA (Blei et al., 2003) and RSM (Salakhutdinov & Hinton, 2009) in terms of both the log-probability on unseen documents and retrieval accuracy. Recently, Gupta et al. (2019) has improved topic modeling on short texts by introducing word embeddings (Pennington et al., 2014) in DocNADE architecture. Thus, we adopt DocNADE to perform transfer learning within the neural topic modeling framework.
Algorithm 1 (for DocNADE, set LVT and GVT to False) demonstrates the computation of log p(v) and negative log-likelihood L(v) that is minimized using gradient descent. Moreover, computing hi is efficient (linear complexity) due to NADE architecture that leverages the pre-activation ai−1 of (i− 1)th step in computing the pre-activation ai. See Larochelle & Lauly (2012) for further details.

heading: 2.2 MULTI-VIEW (MVT) AND MULTI-SOURCE TRANSFERS (MST) IN TOPIC MODELING
text: Here, we describe a transfer learning framework in topic modeling that jointly exploits the complementary knowledge using the WordPool and TopicPool, the KBs of pretrained word and (latent) topic embeddings, respectively obtained from large document collections (DCs) from several sources. In
T 1 T 2 T 3 T 4 T 5 T 6
S1 I I R D D D S2 D D D I D D S3 R R I D D D S4 R R R D D D S5 D D D D - -
Table 3: Domain overlap in sourcetarget corpora. I: Identical, R: Related and D: Distant domains.
doing so, we first apply the DocNADE to generate a topic-word matrix for each of the DCs, where its column-vector and row-vector generate Ek and Zk, respectively for the kth source.
LVT+MST Formulation for Multi-source Word Embedding Transfer: As illustrated in Figure 1 (left) and Algorithm 1 with LVT=True, we perform transfer learning on a target T using the WordPool of pretrained word embeddings {E1, ...,E|S|} from several sources S (i.e., multi-source):
hi(v<i) = g(c + ∑ q<i W:,vq + ∑ q<i |S|∑ k=1 λk Ek:,vq )
Here, k refers to the kth source and λk is a weight for Ek that controls the amount of knowledge transferred in T , based on domain overlap between target and source(s). Recently, DocNADEe (Gupta et al., 2019) has incorporated word embeddings (Pennington et al., 2014) in extending DocNADE; however, it is based on a single source.
GVT+MST Formulation for Multi-source Topic Embedding Transfer: Next, we perform knowledge transfer exclusively using the TopicPool of pretrained topic embeddings (e.g., Zk) from one or several sources, S. In doing so, we add a regularization term to the loss function L(v) and require DocNADE to minimize the overall loss in a way that the (latent) topic features in W simultaneously inherit relevant topical features from each of the source domains S, and generate meaningful representations for the target T . The overall loss L(v) due to GVT+MST in DocNADE is given by:
L(v) = − log p(v) + |S|∑ k=1 γk H∑ j=1 ||Akj,:W − Zkj,:||22
Here, Ak∈RH×H aligns latent topics in the target T and kth source, and γk governs the degree of imitation of topic features Zk by W in T . Consequently, the generative process of learning meaningful topics in W of T is guided by relevant features in {Z}|S|1 to address data-sparsity. Algorithm 1 describes the computation of the loss, when GVT = True and LVT = False.
Moreover, Figure 1 (right) illustrates the need for topic alignments between target and source(s). Here, j indicates the topic (i.e., row) index in a topic matrix, e.g., Zk. Observe that the first topic (gray curve), i.e., Z1j=1 ∈ Z1 of the first source aligns with the first row-vector (i.e., topic) of W (of target). However, the other two topics Z1j=2, Z 1 j=3 ∈ Z1 need alignment with the target topics.
MVT+MST Formulation for Multi-source Word and Topic Embeddings Transfer: When LVT and GVT are True (Algorithm 1) for many sources, the two complementary representations are jointly used in transfer learning using WordPool and TopicPool, and therefore, the name multi-view and multi-source transfers.

heading: 3 EVALUATION AND ANALYSIS
text: Datasets: Table 2 describes the datasets used in high-resource source and low-and high-resource target domains for our experiments. The target domain T consists of four short-text corpora (20NSshort, TMNtitle, R21578title and Ohsumedtitle), one small corpus (20NSsmall) and
two large corpora (TMN and Ohsumed). However in source S , we use five large corpora (20NS, R21578, TMN, AGnews and PubMed) in different label spaces (i.e, domains). Here, the corpora (T 5, T 6 and S5) belong to medical and others to news. Additionally, Table 3 suggests domain overlap (in terms of label match) in the target and source corpora, where we define three types of overlap: I (identical) if all labels match,R (related) if some labels match, and D (distant) if a very few or no labels match. Note, our modeling approaches are completely unsupervised and do not use the data labels. See the data labels in appendices.
Reproducibility: For evaluations in the following sections, we follow the experimental setup similar to DocNADE (Larochelle & Lauly, 2012) and DocNADEe (Gupta et al., 2019), where the number of topics (H) is set to 200. While DocNADEe requires the dimension (i.e., E) of word embeddings be the same as the latent topic (i.e., H), we first apply a projection on the concatenation of the pretrained word embeddings obtained from several sources and then, introduce the prior knowledge in each of the autoregressive step following DocNADEe. We apply it in configurations where Glove and/or FastText (E=300) (Bojanowski et al., 2017) are employed. See appendices for the experimental setup, hyperparameters1 and optimal values of λk ∈ [0.1, 0.5, 1.0] and γk ∈ [0.1, 0.01, 0.001] (determined using development set) in different source-target configurations. (code provided)
Baselines: As summarized in Table 4, we consider several baselines including (1) LDA-based and neural network-based topic models that use the target data, (2) topic models using pretrained word embeddings (i.e., LVT) from Pennington et al. (2014) (Glove), (3) unsupervised document representation, where we employ doc2vec (Le & Mikolov, 2014) and EmbSum (to represent a document by summing the embedding vectors of its words using Glove) in order to quantify the quality of document representations, (4) zero-shot topic modeling, where we use all source corpora and no target corpus, and (5) data-augmentation, where we use all source corpora along with a target corpus for TM on T . Using DocNADE, we first prepare the two KBs: WordPool and TopicPool from each of the source corpora and then use them in knowledge transfer to T . Tables 5 and 6 show the comparison of our proposed transfer learning approaches (i.e., LVT using WordPool, GVT using TopicPool, MVT and MST) with the baselines TMs that (1) do not, and (2) do employ pretrained word embeddings (e.g., DocNADE and DocNADEe, respectively).

heading: 3.1 GENERALIZATION: PERPLEXITY (PPL)
text: To evaluate generative performance of TM, we estimate the log-probabilities for the test documents and compute the average held-out perplexity per word as, PPL = exp ( − 1
N ∑N t=1 1 |vt| log p(vt) ) ,
where N and |vt| are the number of documents and words in a document vt, respectively.
1selected with grid search; suboptimal results (see appendices) by learning λ and γ with backpropagation
Tables 5 and 6 quantitatively show PPL scores on the five target corpora (four short-text and one long-text) by the baselines and proposed transfer learning approaches (i.e., GVT, MVT and MST) using one or four sources. In Table 5 using TMN (as a single source) for LVT, GVT and MVT on TMNtitle, we see improved (reduced) PPL scores: (655 vs 706), (689 vs 706) and (663 vs 706) respectively in comparison to DocNADE. We also observe gains due to MST+LVT, MST+GVT and MST+MVT configurations on TMNtitle. Similarly in MST+LVT for R21578title, we observe a gain of 5.2% (182 vs 192), suggesting that transfer learning using pretrained word and topic embeddings (jointly) from one or many sources helps due to positive knowledge transfer, and it also verifies domain relatedness (e.g., in TMN-TMNtitle and AGnews-TMN). Similarly, Table 6 shows gains in PPL (e.g., on TMNtitle, R21578title, etc.) compared to DocNADEe.
T S Model Topic-words (Top 5)
2 0 N S s h o r t
20NS DNE shipping, sale, prices, expensive, price -GVT sale, price, monitor, site, setup +GVT shipping, sale, price, expensive, subscribe
AGnews DNE microsoft, software, ibm, linux, computer -GVT apple, modem, side, baud, perform +GVT microsoft, software, desktop, computer, apple
T M N t i t l e AGnews DNE miners, earthquake, explosion, stormed, quake TMN DNE tsunami, quake, japan, earthquake, radiation -GVT strike, jackson, kill, earthquake, injures +GVT earthquake, radiation, explosion, wildfire
Table 8: Source S and target T topics before (-) and after (+) topic transfer(s) (GVT) from one or more sources. DNE: DocNADE
chip source corpora target corpus
20NS R21578 AGnews 20NSshort
-GVT +GVT key chips chips virus chips
encrypted semiconductor chipmaker intel technology encryption miti processors gosh intel
clipper makers semiconductor crash encryption keys semiconductors intel chips clipper
Table 9: Five nearest neighbors of the word chip in source and target semantic spaces before (-) and after (+) knowledge transfer (MST+GVT)
In Table 7, we show PPL scores on two medical target corpora: Ohsumtitle and Ohsumed using two sources: AGnews (news corpus) and PubMed (medical abstracts) to perform cross-domain and in-domain knowledge transfers. We see that using PubMed for LVT on both the target corpora improves generalization. Overall, we report a gain of 17.3% (1268 vs 1534) on Ohsumtitle and 8.55% (1497 vs 1637) on Ohsumtitle, compared to DocNADEe. Additionally, MST+GVT and MST+MVT boost generalization performance compared to DocNADE(e).

heading: 3.2 INTERPRETABILTY: TOPIC COHERENCE (COH)
text: While PPL is used for model selection, adjusting parameters (e.g. H) and quantitative comparisons, Chang et al. (2009) showed in some cases humans preferred TMs (based on the semantic quality of topics) with higher (worse) PPLs. Thus beyond perplexity, we compute topic coherence to estimate the meaningfulness of words in each of the topics captured. In doing so, we choose the coherence measure proposed by Röder et al. (2015) that identifies context features for each topic word using a sliding window over the reference corpus. We follow Gupta et al. (2019) and compute COH with the top 10 words in each topic. Essentially, the higher scores imply the more coherent topics.
Tables 5 and 6 (under COH column) demonstrate that our proposed approaches (GVT, MVT and MST) of transfer learning in TMs show noticeable gains in COH and thus, improve topic quality. For instance in Table 5, when AGnews is used as a single source for 20NSsmall datatset, we observe a gain in COH due to GVT (.563 vs .462) and MVT (.566 vs .462). Additionally, noticeable gains are reported due to MST+LVT (.542 vs .462), MST+GVT (.585 vs .462) and MST+MVT (.637 vs .462), compared to DocNADE. Importantly, we find a trend MVT>GVT>LVT in COH scores for both the single-source and multi-source transfers. Similarly, Table 6 show noticeable gains (e.g., 39.3%, 9.95%, 7.08%, etc.) in COH due to MST and MVT with Glove and FastText word embeddings. Moreover, Table 7 shows gains in COH due to GVT on Ohsumedtitle and Ohsumed, using pretrained knowledge from PubMed. Overall, the GVT, MVT and MST boost COH for all the five target corpora compared to the baseline TMs (i.e., DocNADE and DocNADEe). This suggests that there is a need for the two complementary (pretrained word and topics) representations and multi-source transfer learning in order to guide meaningful topic learning in T . The results on both the low- and high-resource targets across domains conclude that the proposed modeling scales.

heading: 3.3 APPLICABILITY: INFORMATION RETRIEVAL (IR)
text: For a greater impact of TMs, we further evaluate the quality of document representations and perform a document retrieval task on the target datasets, using their label information only to compute precision. We follow the experimental setup similar to Lauly et al. (2017), where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity between their document vectors. To compute retrieval precision for each fraction (e.g., 0.02), we average the number of retrieved training documents with the same label as the query.
Tables 5 and 6 depict precision scores at retrieval fraction 0.02 (similar to Gupta et al. (2019)), where the configuration MST+MVT outperforms both the DocNADE and DocNADEe, respectively in retrieval performance on the four target (short-text) datasets. A gain in IR performance is noticeable for highly overlapping domains, e.g., TMN-TMNtitle (.555 vs .521 in Table 5 and .576 vs .540 in Table 6) than the related, e.g., AGnews-TMNtitle (.534 vs .521 in Table 5 and .565 vs .540 in Table 6). We observe large gains in precision at retrieval fraction 0.02: (a) Table 5: 20.7% (.326 vs .270) on 20NSsmall, 9.21% (.569 vs .521) on TMNtitle and 8.28% (.314 vs .290) on 20NSshort, (b) Table 6: 8.84% (.320 vs .294) on 20NSshort and 9.21% (.578 vs .540) on TMNtitle, and (c) Table 7: 4.91% (.192 vs .183) on Ohsumed and 4.0% (.182 vs .175) on Ohsumedtitle.
Additionally, Figures 2a, 2b, 2c and 2d illustrate precision on 20NSshort, 20NSsmall, TMNtitle and R21578title, respectively, where our approaches (MST+GVT and MST+MVT) consistently outperform the baselines at all fractions. Moreover, we split the training data of TMNtitle into several sets: 20%, 40%, 60%, 80% of the training set and then retrain DocNADE, DocNADEe and DocNADE+MST+MVT. We demonstrate the impact of transfer learning in sparse-data settings using WordPool and TopicPool jointly on IR task. Figure 2e plots precision at retrieval (recall) fraction 0.02 and demonstrates that the proposed modeling consistently outperform DocNADE(e).

heading: 3.4 ZERO-SHOT AND DATA-AUGMENTATION EVALUATIONS
text: Figures 2a, 2b, 2c and 2d show precision in the zero-shot (source-only training) and dataaugmentation (source+target training) configurations. Observe that the latter helps in learning
meaningful representations and performs better than zero-shot; however, it is outperformed by MST+MVT, suggesting that a naive (data space) augmentation does not add sufficient prior or relevant information to the sparse target. Thus, we find that it is beneficial to augment training data in feature space (e.g., LVT, GVT and MVT) especially for unsupervised TMs using WordPool and TopicPool. Beyond IR, we further investigate computing topic coherence (COH) for zero-shot and data-augmentation baselines, where the COH scores (Figure 2f) suggest that MST+MVT outperforms DocNADEe, zero-shot and data-augmentation.

heading: 3.5 QUALITATIVE ANALYSIS: TOPICS AND NEAREST NEIGHBORS (NN)
text: For topic level inspection, we first extract topics using the rows of W of source and target corpora. Table 8 shows the topics (top-5 words) from source and target domains. Observe that the target topics become more coherent after transfer learning (i.e., +GVT) from one or more sources. The blue color signifies that a target topic has imitated certain topic words from the source. Observe that we also show topics from source domain(s) that align with the topics from target.
For word level inspection, we extract word representations using the columns of W. Table 9 shows nearest neighbors (NNs) of the word chip in 20NSshort (target) corpus, before and after GVT using three knowledge sources. Observe that the NNs in the target become more meaningful.

heading: 4 CONCLUSION
text: Within neural topic modeling, we have introduced transfer learning approaches using complementary representations: pretrained word (local semantics) and topic (global semantics) embeddings exclusively or jointly from one or many sources (i.e., multi-view and multi-source). We have shown that the proposed approaches better deal with data-sparsity issues, especially in a short-text and/or small document collection. We have demonstrated learning meaningful topics and quality document representations on 7 (low- and high-resource) target corpora from news and medical domains.

heading: A DATA DESCRIPTION
text: In order to evaluate knowledge transfer within unsupervised neural topic modeling, we use the following seven datasets in the target domain T following the similar experimental setup as in DocNADEe:
1. 20NSshort: We take documents from 20NewsGroups data, with document size less (in terms of number of words) than 20.
2. 20NSsmall: We sample 20 document (each having more than 200 words) for training from each class of the 20NS dataset. For validation and test, 10 document for each class. Therefore, it is a corpus of few (long) documents.
3. TMNtitle: Titles of the Tag My News (TMN) news dataset. 4. R21578title: Reuters corpus, a collection of new stories from nltk.corpus. We
take titles of the documents. 5. Ohsumedtitle: Titles of Ohsumed abstracts. Source: disi.unitn.it/
moschitti/corpora.htm. 6. Ohsumed: Ohsumed dataset, collection of medical abstracts. Source: disi.unitn.
it/moschitti/corpora.htm. 7. TMN: The Tag My News (TMN) news dataset.
To prepare knowledge base of word embedings (local semantics) and latent topics (global semantics) features, we use the following six datasets in the source S:
1. 20NS: 20NewsGroups corpus, a collection of news stories from nltk.corpus. 2. TMN: The Tag My News (TMN) news dataset. 3. R21578: Reuters corpus, a collection of new stories from nltk.corpus. 4. AGnews: AGnews data sellection. 5. PubMed: Medical abstracts of randomized controlled trials. Source: https://
github.com/Franck-Dernoncourt/pubmed-rct.

heading: B GETTING WORD AND LATENT TOPIC REPRESENTATIONS FROM SOURCE(S)
text: Since in DocNADE, the column of W:,vi gives a word vector of the word vi, therefore the dimension of word embeddings in each of the Ek is same (i.e., H = 200). Thus, we prepare the knowledge base of word representations Ek from kth source using DocNADE, where each word vector is of H = 200 dimension.
Since the row vector of Wj,: in DocNADE encodes jth topic feature, therefore each latent topic (i.e., row) in feature matrix W is a vector of K dimension, corresponding the definition of topics that it is a distribution over vocabulary. H is the number of latent topics and K is the vocabulary size, where K varies across corpora. Thus, we train DocNADE to learn a feature matrix specific to each of the source corpora, e.g. Wk ∈ RH×K of kth source.
For a target corpus of vocabulary size K ′ , the DocNADE learns a feature matrix WT ∈ RH×K′ . Similarly, Wk ∈ RH×K for kth source of vocabulary sizeK. Since in the sparse-data setting for the
target, K ′ << K due to additional word in the source. In order to perform GVT, we need the same topic feature dimensions in the target and source, i.e., K ′ of the target. Therefore, we remove those column vectors from Wk ∈ RH×K of the kth source for which there is no corresponding word in the vocabulary of the target domain. As a result, we obtain Zk as a latent topic feature matrix to be used in knowledge transfer to the target domain. Following the similar steps, we prepare a KB of Zs such that each latent topic feature matrix from a source domain gets the same topic feature dimension as the target.

heading: C EXPERIMENTAL SETUP
text: For DocNADE and DocNADEe in different knowledge transfer configurations, we follow the same experimental setup as in DocNADE and DocNADEe. We rerun DocNADE and DocNADEe using the code released for DocNADEe.
C.1 EXPERIMENTAL SETUP FOR GENERALIZATION
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with sigmoid hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in PPL for H = 200 topics than using 50, therefore we use H = 200 in our experiments.
See Table 11 for hyperparameters used in generalization task, i.e., computing PPL.
See section C.4 to reproduce scores of Table 1.
C.2 EXPERIMENTAL SETUP FOR IR TASK
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with tanh hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in
precision for the retrieval task for H = 200 topics than using 50, therefore we use H = 200 in our experiments. We follow the similar experimental setup as in DocNADEe. For model selection, we used the validation set as the query set and used the average precision at 0.02 retrieved documents as the performance measure. Note that the labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same (Table 2) train/development/test split of documents for all the datasets during learning.
Given DocNADE, the representation of a document of size D can be computed by taking the last hidden vector hD at the autoregressive stepD. Since, the RSM and DocNADE strictly outperformed LDA, therefore we only compare DocNADE and its recent extension DocNADEe. We use the same number of topic dimensions (H = 200) across all the source domains and the target in training with DocNADE.
See Table 12 for the hyperparameters in the document retrieval task, where λk and γk are weights for kth source. We use the same grid-search for all the source domains. We set γk smaller than λk to control the degree of imitation of the source domain(s) by the target domain. We use the development set of the target corpus to find the optimal setting in different configurations of knowledge transfers from several sources.
See section C.4 to reproduce scores of Table 5.
C.3 {λ, γ} AS PARAMETER VS HYPERPARAMETERS
Here, we treat λ and γ as parameters of the model, instead of hyperparameters and learn them with backpropagation. We initialize each λk = 0.5 and γk = 0.01 for each of the sources. We perform experiments on short-text datasets in MST+LVT, MST+GVT and MST+MVT configurations. We evaluate the topic modeling using PPL, topic coherence and retrieval accuracy.
Table 13 reports the scores, when λ and γ are (1) learned with backpropagation, and (2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Therefore in the work, we have reported scores considering {λ, γ} as hyperparameters.
C.4 REPRODUCIBILITY: OPTIMAL CONFIGURATIONS OF λ AND γ
As mentioned in Tables 11 and 12, the hyper-parameter λk takes on values in [1.0, 0.5, 0.1] for each of the word embeddings matrix Ek and γk in [0.1, 0.01, 0.001] for each of the latent topic features Zk, respectively for the kth source domain. To determine an optimal configuration, we perform grid-search over the values and use the scores on the development set to determine the best setting. We have a common model for PPL and COH scores due to generalization criteria.
To reproduce scores (best in Table 5), we mentioned the best settings of (λk, γk) in MST+MVT configuration for each of the target and source combinations:
1. Generalization (PPL and COH) in MST+MVT when target is 20NSshort: (λ20NS = 1.0, γ20NS = 0.001, λTMN = 0.1, γTMN = 0.001, λR21578 = 0.5, γR21578 = 0.001, λAGnews = 0.1, γAGnews = 0.001
2. Generalization (PPL and COH) in MST+MVT when target is TMNtitle: (λ20NS = 0.1, γ20NS = 0.001, λTMN = 1.0, γTMN = 0.001, λR21578 = 0.5, γR21578 = 0.001, λAGnews = 1.0, γAGnews = 0.001
3. Generalization (PPL and COH) in MST+MVT when target is R21578title: (λ20NS = 0.1, γ20NS = 0.001, λTMN = 0.5, γTMN = 0.001, λR21578 = 1.0, γR21578 = 0.001, λAGnews = 1.0, γAGnews = 0.001
4. Generalization (PPL and COH) in MST+MVT when target is 20NSsmall: (λ20NS = 0.5, γ20NS = 0.001, λTMN = 0.1, γTMN = 0.001, λR21578 = 0.1, γR21578 = 0.001, λAGnews = 0.1, γAGnews = 0.001
5. Generalization (PPL and COH) in MST+MVT when target is Ohsumedtitle: (λAGnews = 0.1, γAGnews = 0.001, λPubMed = 1.0, γPubMed = 0.001
6. Generalization (PPL and COH) in MST+MVT when target is Ohsumed: (λAGnews = 0.1, γAGnews = 0.001, λPubMed = 1.0, γPubMed = 0.001
7. IR in MST+MVT when target is 20NSshort: (λ20NS = 1.0, γ20NS = 0.1, λTMN = 0.5, γTMN = 0.01, λR21578 = 0.1, γR21578 = 0.001, λAGnews = 1.0, γAGnews = 0.01
8. IR in MST+MVT when target is TMNtitle: (λ20NS = 0.1, γ20NS = 0.01, λTMN = 1.0, γTMN = 0.01, λR21578 = 0.1, γR21578 = 0.01, λAGnews = 0.5, γAGnews = 0.001
9. IR in MST+MVT when target is R21578title: (λ20NS = 0.1, γ20NS = 0.01, λTMN = 1.0, γTMN = 0.01, λR21578 = 1.0, γR21578 = 0.01, λAGnews = 0.5, γAGnews = 0.001
10. IR in MST+GVT when target is 20NSsmall: (γ20NS = 0.01, γTMN = 0.01, γR21578 = 0.1, γAGnews = 0.01
11. IR in MST+MVT when target is Ohsumedtitle: (λAGnews = 0.1, γAGnews = 0.001, λPubMed = 1.0, γPubMed = 0.1
12. IR in MST+MVT when target is Ohsumed: (λAGnews = 0.1, γAGnews = 0.001, λPubMed = 0.5, γPubMed = 0.1
The hyper-parameters mentioned above also applies to a single source transfer configuration.
Additionally, we have also provided the code.
While DocNADEe requires the dimension (i.e., E) of word embeddings be the same as the latent topic (i.e., H), we first apply a projection on the concatenation of the pretrained word embeddings obtained from several sources and then, introduce the prior knowledge in each of the autoregressive step following DocNADEe. We apply it in configurations where Glove and/or FastText (E=300) are employed. IN these settings, we use a single mixture weight λ ∈ [1.0, 0.5, 0.1] over the projected vector and then, introduced in TM following DocNADEe.
C.5 EXPERIMENTAL SETUP FOR NVDM AND PRODLDA
For NVDM, we run the code availale at github.com/ysmiao/nvdm and train for 200 topics.
For ProdLDA, we run the code availale at github.com/akashgit/autoencoding_vi_ for_topic_models and train for 200 topics.
C.6 EXPERIMENTAL SETUP FOR GLOVE-DMM
We used LFTM (https://github.com/datquocnguyen/LFTM) to train glove-DMM model. It is trained for 200 iterations with 2000 initial iterations using 200 topics. For short texts, we set the hyperparameter beta to 0.1, for long texts to 0.01; the mixture parameter lambda was set to 0.6 for all datasets. IR task was performed using relative topic proportions as input, where we inferred the topic distribution of the training and test documents and used the relative distribution as input in computing similarities in documents based on the inferred relative topic distribution.
C.7 EXPERIMENTAL SETUP FOR DOC2VEC
We used gensim (https://github.com/RaRe-Technologies/gensim) to train Doc2Vec models. Models were trained with distributed bag of words, for 1000 iterations using a window size of 5 and a vector size of 500.

Review summaries:
On the basis of existing topic modelling approaches, the authors apply a transfer learning approach to incorporate additional knowledge to topic models, using both word embeddings and topic models. The underlying idea is that topic models contain a global view that differs on a thematic level, while word embeddings contain a local, immediate contextual view. The combination of both local and global view transfer to enhance a topic model is the main contribution of this paper, especially when using multiple sources (therefore the title: multi-source multi-view transfer).

The paper proposes a multi-source and multi-view transfer learning for neural topic modelling with the pre-trained topic and word embedding. The method is based on NEURAL AUTOREGRESSIVE TOPIC MODELs --- DocNADE (Larochelle&Lauly,2012). DocNADE learns topics using language modelling framework. DocNADEe (Gupta et al., 2019) extended DocNADE by incorporating word embeddings, the approach the authors described as a single source extension of the existing method.