Title: MULTI-SOURCE MULTI-VIEW TRANSFER LEARNING IN NEURAL TOPIC MODELING WITH PRETRAINED TOPIC AND WORD EMBEDDINGS

Abstract: Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity problem in short text or small collection of documents. However, no prior work has employed (pretrained latent) topics in transfer learning paradigm. In this paper, we propose a framework to perform transfer learning in neural topic modeling using (1) pretrained (latent) topics obtained from a large source corpus, and (2) pretrained word and topic embeddings jointly (i.e., multiview) in order to improve topic quality, better deal with polysemy and data sparsity issues in a target corpus. In doing so, we first accumulate topics and word representations from one or many source corpora to build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). Then, we identify one or multiple relevant source domain(s) and take advantage of corresponding topics and word features via the respective pools to guide meaningful learning in the sparse target domain. We quantify the quality of topic and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. We have demonstrated the state-ofthe-art results on topic modeling with the proposed transfer learning approaches.
Often, there are several topic-word associations in different domains, e.g., in topics Z 1 -Z 3 . Given a noisy topic Z 4 in T and meaningful topics Z 1 -Z 3 of S 1 -S 3 , we identify multiple relevant (source) domains and advantageously transfer their word and topic embeddings in order to facilitate meaningful and positive transfer learning in the sparse corpus, T .
Contribution (1) To our knowledge, it is the first work in unsupervised topic modeling framework that introduces (external) knowledge transfer using (a) Global-view Transfer: Pretrained topic em-Gain% (vs DocNADE) 4.01 3.43 13.8 12.3 4.08 4.35 Gain% (vs DocNADEe) 17.3 2.03 4.00 8.55 2.22 4.91 Table 7: PPL, COH, IR at retrieval fraction 0.02.

1. INTRODUCTION:

Probabilistic topic models, such as LDA (Blei et al., 2003) , Replicated Softmax (RSM) (Salakhutdinov & Hinton, 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) (Larochelle & Lauly, 2012) are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks, such as information retrieval (IR). Though they have been shown to be powerful in modeling large text corpora, the topic modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient e.g., on short text or a corpus of few documents. To this end, several works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) have introduced external knowledge in traditional topic models via word embeddings Pennington et al. (2014) . However, no prior work in topic modeling has employed topical embeddings (obtained from large document collection(s)), complementary to word embeddings.

Local vs Global Views: Though word embeddings (Pennington et al., 2014) and topics are complementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. Word embeddings have local context (view) in the sense that they are learned based on local collocation pattern in a text corpus, where the representation of each word either depends on a local context window (Mikolov et al., 2013) or is a function of its sentence(s) (Peters et al., 2018) . Consequently, the word occurrences are modeled in a fine-granularity. On other hand, a topic (Blei et al., 2003; Gupta et al., 2019) has a global word context (view): TM infers topic distributions across documents in the corpus and assigns a topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. Therefore, it learns from word occurrences across documents and encodes a coarse-granularity description. Unlike topics, the word embeddings do not capture thematic structures (topical semantics) underlying in the document collection. Usually, the top words associated with topics learned on a large corpus are semantically coherent and represent meaningful semantics, e.g., Trading, Product Line, etc. However in sparse-data setting, topics (e.g., Z 4 ) are incoherent (noisy) and therefore, it is difficult to infer meaningful semantics. Additionally, notice that the word apple is topically/thematically contextualized (topic-word association) by different semantics in S 1 -S 3 and referring to a Company.

Unlike the topics, word embeddings encode syntactic and semantic relatedness in fine-granularity and therefore, do not capture thematic structures. For instance, the top-5 nearest neighbors (NN) of apple (below) in the embeddings (Mikolov et al., 2013) space suggest that it refers to a fruit; however, they do not express anything about its thematic context, e.g., Health.

apple:

N N = = ⇒ apples, pear, fruit, berry, pears, strawberry

Motivation (1) Knowledge transfer using pretrained word and topic embeddings: Essentially, the application of TM aims to discover hidden thematic structures (i.e., topics) in text collection; however, it is challenging in data sparsity settings, e.g, in a short and/or small collection. This leads to suboptimal text representations and incoherent topics (e.g., topic Z 4 ).

To alleviate the data sparsity issues, recent works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) have shown that TM can be improved by introducing external knowledge, where they leverage pretrained word embeddings (i.e., local view) only. However, the word embeddings ignore the thematically contextualized structures (i.e., document-level semantics), and can not deal with ambiguity. Given that the word and topic representations encode complementary information, no prior work has explored transfer learning in TM using pretrained topics obtained from a large corpus.

Motivation (2) Knowledge transfer from multiple sources of word and topic embeddings: Knowledge transfer via word embeddings is vulnerable to negative transfer (Cao et al., 2010) on the target domain when domains are shifted and not handled properly. For instance, consider a short-text document v: [apple gained its US market shares] in the target domain T .

Here, the word apple refers to a company, and hence the word vector of apple (about fruit) is an irrelevant source of prior knowledge for both v and the topic Z 4 . In contrast, one can better model v and amend the noisy Z 4 for coherence, given the meaningful word and topic embeddings. beddings instead of using word embeddings exclusively, and (b) Multi-view Transfer: Pretrained word and topic embeddings jointly obtained from a large source corpus in order to deal with polysemy and alleviate data sparsity issues in a small target corpus.

v 1 v i-1 v i v i E |s| ... Embedding lookups v 1 v i-1 v K E 1 λ 1 λ |s| . . . v D ... visible units, v ϵ {1, ..., K} D ith autoregressive conditional, p(v i | v <i ) h i (v <i ) U ϵ ℝ K×H hidden-bias, c ϵ ℝ H W ϵ ℝ H×K

Contribution (2) Multi-source Transfer: Moreover, we first learn word and topic representations on multiple source domains to build WordPool and TopicPool, respectively and then perform multi-view and multi-source transfer learning within neural topic modeling by jointly using the complementary representations. In doing so, we guide the (unsupervised) generative process of learning hidden topics of the target domain by embeddings in WordPool and TopicPool such that the hidden topics become more meaningful and representative in explaining the target corpus.

We evaluate the effectiveness of our transfer learning approaches in neural topic modeling using 7 (5 low-resource and 2 high-resource) target and 5 (high-resource) source corpora from news and medical domains, consisting of short-text, long-text, small and large document collections. Particularly, we quantify the quality of text representations via generalization (perplexity), interpretability (topic coherence) and text retrieval. The code is provided with the supplementary. Notice that a superscript indicates a source. See Table 1 for the notations used in this work.

2.1 NEURAL AUTOREGRESSIVE TOPIC MODELS DocNADE (Larochelle & Lauly, 2012 ) is an unsupervised neural-network based topic model that is inspired by the benefits of NADE (Larochelle & Murray, 2011) and RSM (Salakhutdinov & Hinton, 2009) architectures. RSM has difficulties due to intractability leading to approximate gradients of the negative log-likelihood, while NADE does not require such approximations. On other hand, RSM is a generative model of word count, while NADE is limited to binary data. Specifically, Doc-NADE factorizes the joint probability distribution of words in a document as a product of conditional distributions and efficiently models each conditional via a feed-forward neural network.

Algorithm 1 Computation of log p(v) and Loss L(v)

Input: A target training document v, |S| source domains Input: WordPool: KB of pretrained word embedding matrices {E 1 , ..., E |S| } Input: TopicPool: KB of pretrained latent topics {Z 1 , ..., Z |S| } Parameters: Θ = {b, c, W, U, A 1 , ..., A |S| } Hyper-parameters: θ = {λ 1 , ..., λ |S| , γ 1 , ..., γ |S| , H} Initialize: a ← c and p(v) ← 1 for i from 1 to D do h i (v <i ) ← g(a), where g = {sigmoid, tanh} p(v i = w|v <i ) ← exp(bw+Uw,:hi(v<i)) w exp(b w +U w ,: hi(v<i)) p(v) ← p(v)p(v i |v <i ) compute pre-activation at step, i: a ← a + W :,vi if LVT then get word embedding for v i from source domain(s) a ← a + |S| k=1 λ k E k :,vi L(v) ← -log p(v) if GVT then L(v) ← L(v) + |S| k=1 γ k H j=1 ||A k j,: W -Z k j,: || 2 2 DocNADE Formulation: For a document v = (v 1 , ..., v D ) of size D, each word index v i takes value in {1, .

.., K} of vocabulary size K. DocNADE learns topics in a language modeling fashion (Bengio et al., 2003) and decomposes the joint distribution p(v)= D i=1 p(v i |v <i ) such that each autoregressive conditional p(v i |v <i ) is modeled by a feed-forward neural network using preceding words v <i in the sequence:

h i (v <i ) = g(c + q<i W :,vq ) and p(v i = w|v <i ) = exp(b w + U w,: h i (v <i )) w exp(b w + U w ,: h i (v <i ))

for i ∈ {1, ...D}, where v <i is the subvector consisting of all v q such that q < i i.e.,

v <i ∈ {v 1 , ..., v i-1 }, g(•) is a non-linear activation function, W ∈ R H×K and U ∈ R K×H are weight matrices, c ∈ R H and b ∈ R K are bias parameter vectors.

H is the number of hidden units (topics).

Figure 1 (left) (without WordPool) provides an illustration of the ith autoregressive step of the Doc-NADE architecture, where the parameter W is shared in the feed-forward networks and h i encodes topic-proportion embedding. Importantly, the topic-word matrix W has a property that the column vector W :,vi corresponds to embedding of the word v i , whereas the row vector W j,: encodes the jth topic. We leverage this property to introduce external knowledge via word and topic embeddings.

Additionally, DocNADE has shown to outperform traditional models such as LDA (Blei et al., 2003) and RSM (Salakhutdinov & Hinton, 2009) in terms of both the log-probability on unseen documents and retrieval accuracy. Recently, Gupta et al. (2019) has improved topic modeling on short texts by introducing word embeddings (Pennington et al., 2014) in DocNADE architecture. Thus, we adopt DocNADE to perform transfer learning within the neural topic modeling framework.

Algorithm 1 (for DocNADE, set LVT and GVT to False) demonstrates the computation of log p(v) and negative log-likelihood L(v) that is minimized using gradient descent. Moreover, computing h i is efficient (linear complexity) due to NADE architecture that leverages the pre-activation a i-1 of (i -1)th step in computing the pre-activation a i . See Larochelle & Lauly (2012) for further details.

2.2. MULTI-VIEW (MVT) AND MULTI-SOURCE TRANSFERS (MST) IN TOPIC MODELING:

Here, we describe a transfer learning framework in topic modeling that jointly exploits the complementary knowledge using the WordPool and TopicPool, the KBs of pretrained word and (latent) topic embeddings, respectively obtained from large document collections (DCs) from several sources. In is also used in target domain. '-': unlabeled data. doing so, we first apply the DocNADE to generate a topic-word matrix for each of the DCs, where its column-vector and row-vector generate E k and Z k , respectively for the kth source.

T 1 T 2 T 3 T 4 T 5 T 6 S 1 I I R D D D S 2 D D D I D D S 3 R R I D D D S 4 R R R D D D S 5 D D D D --

LVT+MST Formulation for Multi-source Word Embedding Transfer: As illustrated in Figure 1 (left) and Algorithm 1 with LVT=True, we perform transfer learning on a target T using the WordPool of pretrained word embeddings {E 1 , ..., E |S| } from several sources S (i.e., multi-source):

h i (v <i ) = g(c + q<i W :,vq + q<i |S| k=1 λ k E k :,vq )

Here, k refers to the kth source and λ k is a weight for E k that controls the amount of knowledge transferred in T , based on domain overlap between target and source(s). Recently, DocNADEe (Gupta et al., 2019) has incorporated word embeddings (Pennington et al., 2014) in extending Doc-NADE; however, it is based on a single source.

GVT+MST Formulation for Multi-source Topic Embedding Transfer: Next, we perform knowledge transfer exclusively using the TopicPool of pretrained topic embeddings (e.g., Z k ) from one or several sources, S. In doing so, we add a regularization term to the loss function L(v) and require DocNADE to minimize the overall loss in a way that the (latent) topic features in W simultaneously inherit relevant topical features from each of the source domains S, and generate meaningful representations for the target T . The overall loss L(v) due to GVT+MST in DocNADE is given by:

L(v) = -log p(v) + |S| k=1 γ k H j=1 ||A k j,: W -Z k j,: || 2 2

Here, A k ∈R H×H aligns latent topics in the target T and kth source, and γ k governs the degree of imitation of topic features Z k by W in T . Consequently, the generative process of learning meaningful topics in W of T is guided by relevant features in {Z} |S| 1

to address data-sparsity. Algorithm 1 describes the computation of the loss, when GVT = True and LVT = False.

Moreover, Figure 1 (right) illustrates the need for topic alignments between target and source(s). Here, j indicates the topic (i.e., row) index in a topic matrix, e.g., Z k . Observe that the first topic (gray curve), i.e., Z 1 j=1 ∈ Z 1 of the first source aligns with the first row-vector (i.e., topic) of W (of target). However, the other two topics Z 1 j=2 , Z 1 j=3 ∈ Z 1 need alignment with the target topics. MVT+MST Formulation for Multi-source Word and Topic Embeddings Transfer: When LVT and GVT are True (Algorithm 1) for many sources, the two complementary representations are jointly used in transfer learning using WordPool and TopicPool, and therefore, the name multi-view and multi-source transfers.

3. EVALUATION AND ANALYSIS:

Datasets: Table 2 describes the datasets used in high-resource source and low-and high-resource target domains for our experiments. The target domain T consists of four short-text corpora (20NSshort, TMNtitle, R21578title and Ohsumedtitle), one small corpus (20NSsmall) and Baselines (Related Works) Features NTM AuR LVT GVT|MVT|MST LDA (Blei et al., 2003) RSM (Salakhutdinov & Hinton, 2009 ) DocNADE (Larochelle & Lauly, 2012) NVDM (Miao et al., 2016 ) ProdLDA (Srivastava & Sutton, 2017) Gauss-LDA (Das et al., 2015) glove-DMM (Nguyen et al., 2015) DocNADEe (Gupta et al., 2019 ) EmbSum doc2vec (Le & Mikolov, 2014) this work Here, the corpora (T 5 , T 6 and S 5 ) belong to medical and others to news.

Additionally, Table 3 suggests domain overlap (in terms of label match) in the target and source corpora, where we define three types of overlap: I (identical) if all labels match, R (related) if some labels match, and D (distant) if a very few or no labels match. Note, our modeling approaches are completely unsupervised and do not use the data labels. See the data labels in appendices.

Reproducibility: For evaluations in the following sections, we follow the experimental setup similar to DocNADE (Larochelle & Lauly, 2012) and DocNADEe (Gupta et al., 2019) , where the number of topics (H) is set to 200. While DocNADEe requires the dimension (i.e., E) of word embeddings be the same as the latent topic (i.e., H), we first apply a projection on the concatenation of the pretrained word embeddings obtained from several sources and then, introduce the prior knowledge in each of the autoregressive step following DocNADEe. We apply it in configurations where Glove and/or FastText (E=300) (Bojanowski et al., 2017) are employed. See appendices for the experimental setup, hyperparameters 1 and optimal values of λ k ∈ [0.1, 0.5, 1.0] and γ k ∈ [0.1, 0.01, 0.001] (determined using development set) in different source-target configurations. (code provided)

Baselines: As summarized in Table 4 , we consider several baselines including (1) LDA-based and neural network-based topic models that use the target data, (2) topic models using pretrained word embeddings (i.e., LVT) from Pennington et al. (2014) (Glove), ( 3) unsupervised document representation, where we employ doc2vec (Le & Mikolov, 2014) and EmbSum (to represent a document by summing the embedding vectors of its words using Glove) in order to quantify the quality of document representations, (4) zero-shot topic modeling, where we use all source corpora and no target corpus, and ( 5) data-augmentation, where we use all source corpora along with a target corpus for TM on T . Using DocNADE, we first prepare the two KBs: WordPool and TopicPool from each of the source corpora and then use them in knowledge transfer to T .

Tables 5 and 6 show the comparison of our proposed transfer learning approaches (i.e., LVT using WordPool, GVT using TopicPool, MVT and MST) with the baselines TMs that (1) do not, and (2) do employ pretrained word embeddings (e.g., DocNADE and DocNADEe, respectively).

3.1. GENERALIZATION: PERPLEXITY (PPL):

To evaluate generative performance of TM, we estimate the log-probabilities for the test documents and compute the average held-out perplexity per word as, vt) , where N and |v t | are the number of documents and words in a document v t , respectively. 5 ), DocNADEe: DocNADE+Glove and Gain%: Bold vs DocNADEe. For all the configurations, we apply a projection on word embeddings concatenated from several sources.

P P L = exp -1 N N t=1 1 |v t | log p(

Tables 5 and 6 quantitatively show PPL scores on the five target corpora (four short-text and one long-text) by the baselines and proposed transfer learning approaches (i.e., GVT, MVT and MST) using one or four sources. In Table 5 using TMN (as a single source) for LVT, GVT and MVT on TMNtitle, we see improved (reduced) PPL scores: (655 vs 706), (689 vs 706) and (663 vs 706) respectively in comparison to DocNADE. We also observe gains due to MST+LVT, MST+GVT and MST+MVT configurations on TMNtitle. Similarly in MST+LVT for R21578title, we observe a gain of 5.2% (182 vs 192) , suggesting that transfer learning using pretrained word and topic embeddings (jointly) from one or many sources helps due to positive knowledge transfer, and it also verifies domain relatedness (e.g., in TMN-TMNtitle and AGnews-TMN). Similarly, Table 6 shows gains in PPL (e.g., on TMNtitle, R21578title, etc.) compared to DocNADEe. In Table 7 , we show PPL scores on two medical target corpora: Ohsumtitle and Ohsumed using two sources: AGnews (news corpus) and PubMed (medical abstracts) to perform cross-domain and in-domain knowledge transfers. We see that using PubMed for LVT on both the target corpora improves generalization. Overall, we report a gain of 17.3% (1268 vs 1534) on Ohsumtitle and 8.55% (1497 vs 1637) on Ohsumtitle, compared to DocNADEe. Additionally, MST+GVT and MST+MVT boost generalization performance compared to DocNADE(e).

3.2. INTERPRETABILTY: TOPIC COHERENCE (COH):

While PPL is used for model selection, adjusting parameters (e.g. H) and quantitative comparisons, Chang et al. (2009) showed in some cases humans preferred TMs (based on the semantic quality of topics) with higher (worse) PPLs. Thus beyond perplexity, we compute topic coherence to estimate the meaningfulness of words in each of the topics captured. In doing so, we choose the coherence measure proposed by Röder et al. (2015) that identifies context features for each topic word using a sliding window over the reference corpus. We follow Gupta et al. (2019) and compute COH with the top 10 words in each topic. Essentially, the higher scores imply the more coherent topics.

Tables 5 and 6 (under COH column) demonstrate that our proposed approaches (GVT, MVT and MST) of transfer learning in TMs show noticeable gains in COH and thus, improve topic quality. For instance in Table 5 , when AGnews is used as a single source for 20NSsmall datatset, we observe a gain in COH due to GVT (.563 vs .462) and MVT (.566 vs .462). Additionally, noticeable gains are reported due to MST+LVT (.542 vs .462), MST+GVT (.585 vs .462) and MST+MVT (.637 vs .462), compared to DocNADE. Importantly, we find a trend MVT>GVT>LVT in COH scores for both the single-source and multi-source transfers. Similarly, Table 6 show noticeable gains (e.g., 39.3%, 9.95%, 7.08%, etc.) in COH due to MST and MVT with Glove and FastText word embeddings. Moreover, Table 7 shows gains in COH due to GVT on Ohsumedtitle and Ohsumed, using pretrained knowledge from PubMed. Overall, the GVT, MVT and MST boost COH for all the five target corpora compared to the baseline TMs (i.e., DocNADE and DocNADEe). This suggests that there is a need for the two complementary (pretrained word and topics) representations and multi-source transfer learning in order to guide meaningful topic learning in T . The results on both the low-and high-resource targets across domains conclude that the proposed modeling scales. 

3.3. APPLICABILITY: INFORMATION RETRIEVAL (IR):

For a greater impact of TMs, we further evaluate the quality of document representations and perform a document retrieval task on the target datasets, using their label information only to compute precision. We follow the experimental setup similar to Lauly et al. (2017) , where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity between their document vectors. To compute retrieval precision for each fraction (e.g., 0.02), we average the number of retrieved training documents with the same label as the query.

Tables 5 and 6 depict precision scores at retrieval fraction 0.02 (similar to Gupta et al. (2019)) , where the configuration MST+MVT outperforms both the DocNADE and DocNADEe, respectively in retrieval performance on the four target (short-text) datasets. A gain in IR performance is noticeable for highly overlapping domains, e.g., TMN-TMNtitle (.555 vs .521 in Table 5 and .576 vs .540 in Table 6 ) than the related, e.g., AGnews-TMNtitle (.534 vs .521 in Table 5 and .565 vs .540 in Table 6 ). We observe large gains in precision at retrieval fraction 0.02: (a) Observe that the latter helps in learning meaningful representations and performs better than zero-shot; however, it is outperformed by MST+MVT, suggesting that a naive (data space) augmentation does not add sufficient prior or relevant information to the sparse target. Thus, we find that it is beneficial to augment training data in feature space (e.g., LVT, GVT and MVT) especially for unsupervised TMs using WordPool and TopicPool. Beyond IR, we further investigate computing topic coherence (COH) for zero-shot and data-augmentation baselines, where the COH scores (Figure 2f ) suggest that MST+MVT outperforms DocNADEe, zero-shot and data-augmentation.

3.5 QUALITATIVE ANALYSIS: TOPICS AND NEAREST NEIGHBORS (NN)

For topic level inspection, we first extract topics using the rows of W of source and target corpora. Table 8 shows the topics (top-5 words) from source and target domains. Observe that the target topics become more coherent after transfer learning (i.e., +GVT) from one or more sources. The blue color signifies that a target topic has imitated certain topic words from the source. Observe that we also show topics from source domain(s) that align with the topics from target.

For word level inspection, we extract word representations using the columns of W. Table 9 shows nearest neighbors (NNs) of the word chip in 20NSshort (target) corpus, before and after GVT using three knowledge sources. Observe that the NNs in the target become more meaningful.

4. CONCLUSION:

Within neural topic modeling, we have introduced transfer learning approaches using complementary representations: pretrained word (local semantics) and topic (global semantics) embeddings exclusively or jointly from one or many sources (i.e., multi-view and multi-source). We have shown that the proposed approaches better deal with data-sparsity issues, especially in a short-text and/or small document collection. We have demonstrated learning meaningful topics and quality document representations on 7 (low-and high-resource) target corpora from news and medical domains.

B GETTING WORD AND LATENT TOPIC REPRESENTATIONS FROM SOURCE(S):

Since in DocNADE, the column of W :,vi gives a word vector of the word v i , therefore the dimension of word embeddings in each of the E k is same (i.e., H = 200). Thus, we prepare the knowledge base of word representations E k from kth source using DocNADE, where each word vector is of H = 200 dimension.

Since the row vector of W j,: in DocNADE encodes jth topic feature, therefore each latent topic (i.e., row) in feature matrix W is a vector of K dimension, corresponding the definition of topics that it is a distribution over vocabulary. H is the number of latent topics and K is the vocabulary size, where K varies across corpora. Thus, we train DocNADE to learn a feature matrix specific to each of the source corpora, e.g. W k ∈ R H×K of kth source.

For a target corpus of vocabulary size K , the DocNADE learns a feature matrix W T ∈ R H×K . Similarly, W k ∈ R H×K for kth source of vocabulary size K. Since in the sparse-data setting for the misc.forsale, comp.graphics, rec.autos, comp.windows.x, rec.sport.baseball, sci.space, rec.sport.hockey, soc.religion.christian, rec.motorcycles, comp.sys.mac.hardware, talk.religion.misc, sci.electronics, comp.os.ms-windows.misc, sci.med, comp.sys.ibm.pc.hardware, talk.politics.mideast, talk.politics.guns, talk.politics.misc, alt.atheism, sci.crypt R21578title and R21578 trade, grain, crude, nat-gas, corn, rice, rubber, sugar, tin, palm-oil, veg-oil, ship, coffee, lumber, wheat, gold, acq, interest, money-fx, copper, ipi, carcass, livestock, oilseed, soybean, earn, bop, gas, lead, jobs, zinc, cpi, gnp, soy-oil, dlr, yen, nickel, groundnut, heat, sorghum, sunseed, pet-chem, cocoa, rapeseed, cotton, money-supply, iron-steel, l-cattle, alum, palladium, platinum, strategic-metal, reserves, groundnut-oil, lin-oil, meal-feed, rape-oil, sun-meal, sun-oil, hog, barley, potato, orange, retail, soy-meal, cotton-oil, oat, fuel, silver, income, wpi, tea, lei, coconut, coconut-oil, copra-cake, dfl, dmk, naphtha, propane, instal-debt, nzdlr, housing, nkr, rye, castor-oil, jet, palmkernel, cpu, rand target, K << K due to additional word in the source. In order to perform GVT, we need the same topic feature dimensions in the target and source, i.e., K of the target. Therefore, we remove those column vectors from W k ∈ R H×K of the kth source for which there is no corresponding word in the vocabulary of the target domain. As a result, we obtain Z k as a latent topic feature matrix to be used in knowledge transfer to the target domain. Following the similar steps, we prepare a KB of Zs such that each latent topic feature matrix from a source domain gets the same topic feature dimension as the target.

C EXPERIMENTAL SETUP:

For DocNADE and DocNADEe in different knowledge transfer configurations, we follow the same experimental setup as in DocNADE and DocNADEe. We rerun DocNADE and DocNADEe using the code released for DocNADEe.

C.1 EXPERIMENTAL SETUP FOR GENERALIZATION:

We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with sigmoid hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in PPL for H = 200 topics than using 50, therefore we use H = 200 in our experiments.

See Table 11 for hyperparameters used in generalization task, i.e., computing PPL.

See section C.4 to reproduce scores of Table 1 .

C.2 EXPERIMENTAL SETUP FOR IR TASK:

We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with tanh hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in Table 13 reports the scores, when λ and γ are (1) learned with backpropagation, and ( 2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Therefore in the work, we have reported scores considering {λ, γ} as hyperparameters.

C.4 REPRODUCIBILITY: OPTIMAL CONFIGURATIONS OF λ AND γ

As mentioned in Tables 11 and 12 , the hyper-parameter λ k takes on values in [1.0, 0.5, 0.1] for each of the word embeddings matrix E k and γ k in [0.1, 0.01, 0.001] for each of the latent topic features Z k , respectively for the k th source domain. To determine an optimal configuration, we perform grid-search over the values and use the scores on the development set to determine the best setting.

We have a common model for PPL and COH scores due to generalization criteria.

To reproduce scores (best in Table 5 ), we mentioned the best settings of (λ k , γ k ) in MST+MVT configuration for each of the target and source combinations:

:

selected with grid search; suboptimal results (see appendices) by learning λ and γ with backpropagation

Figures and Tables:

Figure 1: (Left) DocNADE (LVT+MST): Multi-source transfer learning in TM by introducing pretrained word embeddings from a WordPool at each autoregressive step i. Double circle → multinomial (softmax) unit. (Right) Multi-source transfer learning in TM by introducing pretrained (latent) topic embeddings from a TopicPool, illustrating topic alignments between source and target corpora in GVT+MST configuration. Each outgoing row from Z k signify a topic embedding of the corresponding source corpus, DC k . Here, TM refers to a DocNADE topic model.

Figure 2: (a, b, c, d) Retrieval performance (precision) on four datasets. (e) Precision at recall fraction 0.02, each for a fraction (20%, 40%, 60%, 80%, 100%) of the training set of TMNtitle. (f) Zero-shot and data-augmentation (DA) experiments for topic coherence on TMNtitle and Ohsumed.

Degree of relevance of E k in T b ∈ R K , c ∈ R H Visible-bias, hidden-bias γ k Degree of imitation of Z k by W v, k, L An input document, kth source, loss E k ∈ R E×K , Word embeddings of kth source W ∈ R H×K Encoding matrix of DocNADE in T Z k ∈ R H×K Topic embeddings of kth source U ∈ R K×HDecoding matrix of DocNADE Description of the notations used in this workConsider the following topics (Z 1 -Z 4 ), where (Z 1 -Z 3 ) are respectively obtained from different (high-resource) source (S 1 -S 3 ) domains whereas Z 4 from the (low-resource) target domain T in the data-sparsity setting:Z 1 (S 1 ): profit, growth, stocks, apple, fall, consumer, buy, billion, shares → Trading Z

Table content: <table><tr><td>Notation</td><td>Description</td><td>Notation</td><td>Description</td></tr><tr><td>LVT, GVT</td><td>Local-view Transfer, Global-view Transfer</td><td>A k ∈ R H×H</td><td>Topic-alignment in T and Z k</td></tr><tr><td>MVT, MST</td><td>Multi-view Transfer, Multi-source Transfer</td><td>K, D</td><td>Vocabulary size, document size</td></tr><tr><td>T , S</td><td>A target domain, a set of source domains</td><td>E, H</td><td>Word embedding dimension, #topics</td></tr><tr><td>λ k</td><td/><td/><td/></tr></table>

Pretrained word embeddings matrices {E 1 , ..., E |S| }, where E k ∈ R E×K , and (2) TopicPool: Pretrained latent topic embeddings {Z 1 , ..., Z |S| }, where Z k ∈ R H×K encodes a distribution over a vocabulary of K words. E and H are word embedding and latent topic dimensions, respectively. While topic modeling on T , we introduce the two types of knowledge transfers from one or many sources:

Table content: <table><tr><td>2 KNOWLEDGE TRANSFER IN NEURAL TOPIC MODELING</td></tr><tr><td>Consider a sparse target domain T and a set of |S| source domains S, we first prepare two knowledge</td></tr><tr><td>bases (KBs) of representations (or embeddings) from each of the sources: (1) WordPool: Local (LVT) and Global (GVT) View Transfer using the two KBs of pretrained word (i.e., WordPool)</td></tr><tr><td>and topic (i.e., TopicPool) embeddings, respectively. Specially, we employ a neural autoregressive</td></tr><tr><td>topic model (i.e., DocNADE (Larochelle &amp; Lauly, 2012)) to build the WordPool and TopicPool.</td></tr></table>

Data statistics: Short/long texts and/or small/large corpora in target and source domains. Symbols-K: vocabulary size, L: average text length (#words), C: number of classes and k: thousand. For short-text, L<15. S 3

Table content: <table><tr><td/><td colspan="2">Target Domain Corpora</td><td/><td>Source Domain Corpora</td></tr><tr><td>ID</td><td>Data</td><td>Train Val Test K</td><td colspan="2">L C ID Data Train Val Test K L C</td></tr><tr><td>T 1</td><td colspan="3">20NSshort 1.3k 0.1k 0.5k 1.4k 13.5 20 S 1</td><td>20NS 7.9k 1.6k 5.2k 2k 107.5 20</td></tr><tr><td colspan="5">T 2 20NSsmall 0.4k 0.2k 0.2k 2k 187.5 20 S 2 R21578 7.3k 0.5k 3.0k 2k 128 90</td></tr><tr><td>T 3</td><td colspan="3">TMNtitle 22.8k 2.0k 7.8k 2k 4.9 7 S 3</td><td>TMN 22.8k 2.0k 7.8k 2k</td><td>19 7</td></tr><tr><td colspan="5">T 4 R21578title 7.3k 0.5k 3.0k 2k 7.3 90 S 4 AGNews 118k 2.0k 7.6k 5k</td><td>38 4</td></tr><tr><td colspan="5">T 5 Ohsumedtitle 8.3k 2.1k 12.7k 2k 11.9 23 S 5 PubMed 15.0k 2.5k 2.5k 3k 254.8 -</td></tr><tr><td>T 6</td><td colspan="3">Ohsumed 8.3k 2.1k 12.7k 3k 159.1 23</td></tr></table>



Table content: <table><tr><td>:</td><td colspan="2">Domain</td></tr><tr><td colspan="3">overlap in source-</td></tr><tr><td colspan="2">target corpora.</td><td>I:</td></tr><tr><td colspan="3">Identical, R: Re-</td></tr><tr><td colspan="3">lated and D: Distant</td></tr><tr><td>domains.</td><td/></tr></table>

Baselines (related works) vs this work. Here, NTM and AuR refer to neural network-based TM and autoregressive assumption, respectively. DocNADEe → DocNADE+Glove embeddings. two large corpora (TMN and Ohsumed). However in source S, we use five large corpora (20NS, R21578, TMN, AGnews and PubMed) in different label spaces (i.e, domains).

Table content: <table/>



Table content: <table><tr><td>KBs from</td><td>Model/</td><td/><td/><td colspan="10">Scores on Target Corpus (in sparse-data and sufficient-data settings)</td><td/></tr><tr><td colspan="2">Source Transfer</td><td/><td colspan="2">20NSshort</td><td/><td colspan="2">TMNtitle</td><td colspan="3">R21578title</td><td colspan="3">20NSsmall</td><td/><td>TMN</td></tr><tr><td>Corpus</td><td colspan="15">Type P P L COH IR P P L COH IR P P L COH IR P P L COH IR P P L COH</td></tr><tr><td>Baseline TM</td><td colspan="15">NVDM 1047 .736 .076 973 .740 .190 372 .735 .271 957 .515 .090 833 .673</td></tr><tr><td colspan="16">without Word-ProdLDA .689 .062 1527 .744 .170 480 .742 .200 1181 .394 .062 1519 .577</td></tr><tr><td colspan="16">Embeddings DocNADE .667 .290 706 .709 .521 192 .713 .657 594 .462 .270 584 .636</td></tr><tr><td/><td colspan="15">LVT .673 .298 705 .709 .523 194 .708 .656 594 .455 .288 582 .649</td></tr><tr><td>20NS</td><td colspan="15">GVT .690 .303 718 .720 .527 184 .698 .660 594 .500 .310 590 .652</td></tr><tr><td/><td colspan="15">MVT .690 .314 714 .718 .528 188 .715 .655 600 .499 .311 588 .650</td></tr><tr><td/><td colspan="13">LVT .668 .296 655 .731 .548 187 .703 .659 593 .460 .273</td><td>-</td><td>-</td></tr><tr><td>TMN</td><td colspan="13">GVT .692 .294 689 .728 .555 191 .709 .660 596 .521 .276</td><td>-</td><td>-</td></tr><tr><td/><td colspan="13">MVT .687 .297 663 .747 .553 195 .720 .660 599 .507 .292</td><td>-</td><td>-</td></tr><tr><td/><td colspan="15">LVT .667 .292 704 .715 .522 186 .715 .676 593 .458 .267 581 .636</td></tr><tr><td>R21578</td><td colspan="15">GVT .672 .293 716 .719 .526 194 .706 .672 595 .485 .279 591 .646</td></tr><tr><td/><td colspan="15">MVT .670 .296 716 .720 .528 194 .724 .676 599 .490 .280 589 .650</td></tr><tr><td/><td colspan="15">LVT .677 .297 682 .723 .533 185 .710 .659 592 .458 .260 564 .668</td></tr><tr><td>AGnews</td><td colspan="15">GVT .695 .300 728 .735 .534 190 .717 .663 598 .563 .282 601 .684</td></tr><tr><td/><td colspan="15">MVT .696 .290 718 .740 .533 189 .727 .659 599 .566 .279 592 .686</td></tr><tr><td/><td colspan="15">LVT .678 .308 663 .732 .547 182 .739 .673 594 .542 .277 568 .674</td></tr><tr><td>MST</td><td colspan="15">GVT .705 .305 704 .746 .550 192 .727 .673 599 .585 .326 602 .680</td></tr><tr><td/><td colspan="15">MVT .740 .314 680 .752 .569 188 .745 .685 600 .637 .285 600 .690</td></tr><tr><td colspan="16">Gain% (vs DocNADE) 1.23 10.9 8.28 7.22 6.06 9.21 5.20 4.49 4.26 0.34 37.9 20.7 3.42 8.50</td></tr><tr><td>KBs from</td><td>Model/</td><td/><td/><td colspan="10">Scores on Target Corpus (in sparse-data and sufficient-data settings)</td><td/></tr><tr><td>Source</td><td>Transfer</td><td colspan="3">20NSshort</td><td/><td colspan="2">TMNtitle</td><td colspan="3">R21578title</td><td colspan="3">20NSsmall</td><td/><td>TMN</td></tr><tr><td>Corpus</td><td colspan="15">Type P P L COH IR P P L COH IR P P L COH IR P P L COH IR P P L COH</td></tr><tr><td/><td>doc2vec</td><td>-</td><td>-</td><td>.090</td><td>-</td><td>-</td><td>.190</td><td>-</td><td>-</td><td>.518</td><td>-</td><td>-</td><td>.200</td><td>-</td><td>-</td></tr><tr><td/><td>EmbSum</td><td>-</td><td>-</td><td>.236</td><td>-</td><td>-</td><td>.513</td><td>-</td><td>-</td><td>.587</td><td>-</td><td>-</td><td>.214</td><td>-</td><td>-</td></tr><tr><td colspan="2">Baseline TM Gauss-LDA</td><td>-</td><td>-</td><td>.080</td><td>-</td><td>-</td><td>.408</td><td>-</td><td>-</td><td>.367</td><td>-</td><td>-</td><td>.090</td><td>-</td><td>-</td></tr><tr><td colspan="2">with Word-glove-DMM</td><td>-</td><td colspan="2">.512 .183</td><td>-</td><td colspan="2">.633 .445</td><td>-</td><td colspan="2">.364 .273</td><td>-</td><td colspan="2">.578 .090</td><td>-</td><td>.705</td></tr><tr><td colspan="16">Embeddings DocNADEe .674 .294 680 .719 .540 187 .721 .663 590 .455 .274 572 .664</td></tr><tr><td colspan="16">20NS MVT+Glove .721 .320 688 .741 .565 183 .724 .667 597 .561 .306 570 .693</td></tr><tr><td colspan="14">TMN MVT+Glove .731 .295 673 .750 .576 184 .716 .672 599 .594 .261</td><td>-</td><td>-</td></tr><tr><td colspan="9">R21578 MVT+Glove .705 .295 689 .738 .540 185</td><td colspan="7">.737 .691 595 .485 .255 577 .697</td></tr><tr><td colspan="9">AGnews MVT+Glove .734 .302 706 .748 .565 190</td><td colspan="7">.734 .675 598 .573 .284 585 .703</td></tr><tr><td colspan="16">MST MVT+Glove .739 .304 673 .752 .570 183 .742 .684 598 .631 .282 582 .710</td></tr><tr><td/><td colspan="15">+ FastText .741 .313 673 .751 .578 183 .744 .684 599 .634 .254 582 .711</td></tr><tr><td colspan="2">Gain% (vs DocNADEe)</td><td>-</td><td colspan="8">9.95 8.84 1.03 4.60 7.04 2.14 3.20 4.22</td><td>-</td><td colspan="3">39.3 2.92 .35</td><td>7.08</td></tr></table>

State-of-the-art comparisons with TMs using word embeddings: PPL, COH and IR at retrieval fraction 0.02. Scores are reported on each of the target, given KBs. Here, MVT: LVT+GVT (Table

Table content: <table/>

Source S and target T topics before(-)

Table content: <table><tr><td colspan="5">and after (+) topic transfer(s) (GVT) from one or</td></tr><tr><td colspan="3">more sources. DNE: DocNADE</td><td/><td/></tr><tr><td/><td/><td>chip</td><td/><td/></tr><tr><td/><td colspan="2">source corpora</td><td colspan="2">target corpus</td></tr><tr><td>20NS</td><td>R21578</td><td>AGnews</td><td colspan="2">20NSshort -GVT +GVT</td></tr><tr><td>key</td><td>chips</td><td>chips</td><td>virus</td><td>chips</td></tr><tr><td colspan="2">encrypted semiconductor</td><td>chipmaker</td><td colspan="2">intel technology</td></tr><tr><td>encryption</td><td>miti</td><td>processors</td><td>gosh</td><td>intel</td></tr><tr><td>clipper</td><td>makers</td><td colspan="3">semiconductor crash encryption</td></tr><tr><td>keys</td><td>semiconductors</td><td>intel</td><td>chips</td><td>clipper</td></tr></table>



Table content: <table/>

20.7% (.326 vs .270) on 20NSsmall, 9.21% (.569 vs .521) on TMNtitle and 8.28% (.314 vs .290) on 20NSshort, (b) Table 6: 8.84% (.320 vs .294) on 20NSshort and 9.21% (.578 vs .540) on TMNtitle, and (c) Table 7: 4.91% (.192 vs .183) on Ohsumed and 4.0% (.182 vs .175) on Ohsumedtitle.

Table content: <table><tr><td>Additionally, Figures 2a, 2b, 2c and 2d illustrate precision on 20NSshort, 20NSsmall, TMNtitle</td></tr><tr><td>and R21578title, respectively, where our approaches (MST+GVT and MST+MVT) consistently</td></tr><tr><td>outperform the baselines at all fractions. Moreover, we split the training data of TMNtitle into</td></tr><tr><td>several sets: 20%, 40%, 60%, 80% of the training set and then retrain DocNADE, DocNADEe</td></tr><tr><td>and DocNADE+MST+MVT. We demonstrate the impact of transfer learning in sparse-data settings</td></tr><tr><td>using WordPool and TopicPool jointly on IR task. Figure 2e plots precision at retrieval (recall)</td></tr><tr><td>fraction 0.02 and demonstrates that the proposed modeling consistently outperform DocNADE(e).</td></tr><tr><td>3.4 ZERO-SHOT AND DATA-AUGMENTATION EVALUATIONS</td></tr><tr><td>Figures 2a, 2b, 2c and 2d show precision in the zero-shot (source-only training) and data-</td></tr><tr><td>augmentation (source+target training) configurations.</td></tr></table>

Label space of the corpora used

Table content: <table><tr><td>Hyperparameter</td><td>Search Space</td></tr><tr><td>retrieval fraction</td><td>[0.02]</td></tr><tr><td>learning rate</td><td>[0.001]</td></tr><tr><td>hidden units, H</td><td>[200]</td></tr><tr><td>activation function (g)</td><td>sigmoid</td></tr><tr><td>iterations</td><td>[100]</td></tr><tr><td>λ k</td><td>[1.0, 0.5, 0.1]</td></tr><tr><td>γ k</td><td>[0.1, 0.01, 0.001]</td></tr></table>

Hyperparameters in Generalization in DocNADE, DocNADEe, LVT, GVT and MVT configurations for 200 topics

Table content: <table/>

Peer review (iclr2020):
This is an emergency review. This work proposes a novel method to use pre-trained topic embeddings and pre-trained word embeddings obtained from various corpora in the transfer learning framework. Their model architecture is based on DocNADE, unsupervised neural-network based topic model, and the authors propose two strategies to use pre-trained topic embeddings and pre-trained word vectors. 1) Addition of a weighted sum of pre-trained word embeddings and the hidden vector of DocNADE. 2) L2-Regularization term between topic embedding of DocNADE and pre-trained topic embeddings. They propose to align these two embeddings by multiplying align matrix *A* to the topic embedding of DocNADE. They show the transfer learning performance of their model on various source/target domain datasets, including medical target corpora, and verify that their model outperforms on a short text and small document collection. Strengths. 1. Comparison with the data augmentation baseline shows the performance gain is not only from bigger training data. Even though comparison with the naive baseline (data augmentation) seems too obvious, I think the results clearly show their claim about the importance of using transfer learning in neural topic modeling domain. 2. As the first approach that introduces a novel transfer learning framework with pre-trained topic embeddings, they show tons of experimental results with various datasets and metrics to show the specification of their method. Their experimental setting is well designed. Weaknesses and comments: Their method to combine pre-trained word embeddings and pre-trained topic embeddings is too simple. Since this is the first approach to use topic embedding in the transfer learning field, the simplicity of the proposed method is somewhat necessary. However, a weighted sum of pre-trained topic/word vectors seems not enough to transfer multisource knowledge. For instance, word vectors obtained from individual training processes do not share embedding vector space. As you apply the alignment method to topic embeddings from various sources, you should align word embeddings too.