Multi-source Multi-view Transfer Learning in Neural Topic Modeling with Pretrained Topic and Word Embeddings
Authors: Pankaj Gupta
Yatin Chaudhary
Hinrich Schütze
Venue:  (2020)
Keywords: Neural Topic Modeling
Transfer Learning
Unsupervised learning
Natural Language Processing

Abstract: Though word embeddings and topics are complementary representations, several
past works have only used pretrained word embeddings in (neural) topic modeling
to address data sparsity problem in short text or small collection of documents.
However, no prior work has employed (pretrained latent) topics in transfer learning
paradigm. In this paper, we propose a framework to perform transfer learning
in neural topic modeling using (1) pretrained (latent) topics obtained from a large
source corpus, and (2) pretrained word and topic embeddings jointly (i.e., multiview)
in order to improve topic quality, better deal with polysemy and data sparsity
issues in a target corpus. In doing so, we first accumulate topics and word representations
from one or many source corpora to build respective pools of pretrained
topic (i.e., TopicPool) and word embeddings (i.e., WordPool). Then, we identify
one or multiple relevant source domain(s) and take advantage of corresponding
topics and word features via the respective pools to guide meaningful learning
in the sparse target domain. We quantify the quality of topic and document representations
via generalization (perplexity), interpretability (topic coherence) and
information retrieval (IR) using short-text, long-text, small and large document
collections from news and medical domains. We have demonstrated the state-ofthe-
art results on topic modeling with the proposed transfer learning approaches.

Under review as a conference paper at ICLR 2020
MULTI-SOURCE MULTI-VIEW TRANSFER LEARNING IN NEURAL TOPIC MODELING WITH PRETRAINED TOPIC AND WORD EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic model- ing to address data sparsity problem in short text or small collection of documents, However, no prior work has employed (pretrained latent) topics in transfer learn- ing paradigm. In this paper, propose framework to perform transfer learning in neural topic modeling using (1) pretrained (latent) topics obtained from a large source corpus, and (2) pretrained word and topic embeddings jointly (i.e., multi- view) in order to improve topic quality, better deal with polysemy and data sparsity issues in a target corpus. In doing so, we first accumulate topics and word repre- sentations from one or many source corpora to build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). Then, we identify one or multiple relevant source domain(s) and take advantage of corresponding topics and word features via the respective pools guide meaningful learning in the sparse target domain. We quantify the quality of topic and document rep- resentations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. We have demonstrated the state-of- the-art results on topic modeling with the proposed transfer learning approaches.
INTRODUCTION
Probabilistic topic models, such as LDA (Blei al. 2003). Replicated Softmax (RSM) (Salakhut- dinov & Hinton 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) (Larochelle & Lauly] 2012) are often used to extract topies from text collections and learn latent document representations to perform natural language processing tasks, such as information re- trieval (IR). Though they have been shown to powerful modeling large text corpora, the topic modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient c.g., on short text or a corpus of few docu- ments. To this end, several works (Das et al., 2015); Nguyen al.) 2015 Gupta et al.) (019) have introduced external knowledge in traditional topic models via word embeddings |Pennington et al.) (2014). However, no prior work in topic modeling has employed topical embeddings (obtained from large document collection(s)), complementary to word embeddings.
Local vs Global Views: Though word embeddings (Pennington et al., 2014 and topics are com- plementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. Word embeddings have local context (view) in the sense that they are learned based on local collocation pattern in text corpus, where the representation of each word either depends on a local context window (Mikolov et al., (003) or is function of its sen- tence(s) (Peters et al.) 2018). Consequently, the word occurrences are modeled in a fine-granularit. On other hand, a topic (Blii et al. 2003 Gupta et al. 2019) has a global word context (view): TM in- fers topic distributions across documents in the corpus and assigns topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. Therefore, it learns from word occurrences across documents and encodes coarse coarse-grantn -aanss--grllaaity de- scription. Unlike topics, the word embeddings do not capture thematic structures (topical semantics) underlying in the document collection.

---

Under review as a conference paper at ICLR 2020
Notation
Deseription
Notation A E R K,D
Description
LVT. GVT
Local-view Transfer, Global-view Transfer Multi-view Transfer, Multi-source Transfer A target domain, set of source domains Degree of relevance of E* in T Degrec of imitation of Z by W
Topic-alignment in T and Z Vocabulary document size Word embedding dimension, #topics Visible-bias hidden-bias
MVT, MST
T.S
X
b C RR bCR..cCR*
v,
input document, kth source, loss Encoding matrix of DocNADE in Decoding matrix of DocNADE
E
CR
Word embeddings of kth source Topic embeddings of kth source
W C
IR " Z
UE R
Table 1: Description of the notations used in this work
Consider the following topics (Z1-2z), where (Z)-Zz) are respectively obtained from different (high-resource) source (S"-S3) domains whereas Z1 from the (low-resource) target domain in the data-sparsity setting:
21 ( profit, growth, stocks, apple.fall consumer, buy, billion, shares + Trading
(y(Say: smartphone, ipad, apple, app, iphone, devices, phone, tablet - Product Line Z3 (S*): microsoft, mac, linux, ihm, ios, apple, xp, windows Operating System/Company
74 (T): apple, talk, computers, shares, disease, driver, electronics, profit, ios >
Usually, the top words associated with topies learned on a large corpus are semantically coherent and represent meaningful semantics, e.g., Trading, Product Line, etc. However in sparse-data setting, topics (e.g.. Zạ) incoherent (noisy) and therefore, it is difficult to infer meaningful semantics. Additionally, notice that the word apple is oppicallythemmaically contextualized (topic-word associ- ation) by different semantics S' S and referring a Company.
Unlike the topics, word embeddings encode syntactic and semantic relatedness in fine- ume-granularity and therefore, do not capture thematic structures. For instance, the top-5 nearest neighbors (NN) of apple (below) in the embeddings (Mikolov et al. (203)) space suggest that it refers to a fruit; however, they do not express anything about its thematic context, e.g., Health.
apple apples, pear, fruit, berry, pears, strawberry
Motivation (1) Knowledge transfer using pretrained word and topic embeddings: Essentially, the application of TM aims to discover hidden thematic structures (i.c., topics) in text collection; however, it challenging in data sparsity settings, e.g, in a short and/or small collection. This leads to suboptimal text representations and incoherent topics (e.g., topic Z4).
To alleviate the data sparsity issues, recent works (Das et al. 2015 Nguyen et al., 2013 Gupta al., 2019) have shown that TM can be improved by introducing external knowledge, where they leverage pretrained word embeddings (i.e., local view) only. However, the word embeddings ignore the thematically contextualized structures (i.e., document-level semantics), and can not deal with ambiguity. Given that the word and topic representations encode complementary information, no prior work has explored transfer learning TM using pretrained topics obtained from large corpus.
Motivation (2) Knowledge transfer from multiple sources of word and topic embeddings:
Knowledge transfer via word embeddings vulnerable to negative transfer (Cao et al., 2010) on the target domain when domains are shifted and not handled properly. For instance, consider short-text document v: [apple gained its US market shares] in the target domain T. Here, the word apple refers to company, and hence the word vector of apple (about fruit) is an irrelevant source of prior knowledge for both v and the topic Z₁. In contrast, one can better model and amend the noisy Z1 for coherence, given the meaningful word and topic embeddings.
Often, there are several topic-word associations in different domains, c.g., in topics Z1-Z3. Given a noisy topic Zu in T and meaningful topics Z Zı-Za of S identify multiple relevant (source) domains and advantageously transfer their word and topic embeddings in order to facilitate mean- ingful and positive transfer learning in the sparse corpus, T.
Contribution (1) To our knowledge, it is the first work in unsupervised topic modeling framework that introduces (external) knowledge transfer using (a) Global-view Transfer: Pretrained topic em-
2

---

Under review as a conference paper at ICLR 2020
autoregressive
visible-bias, b RK
conditional, plj (uy)
WordPool Word Embedblings
Source
Target
Topic
RKHH
Proportonn
E11
Embedding
(V
Topicatt Tepicaz
Topicas
hidden-bias,
X1
Z
M la
A2
W
RM
HoK
Targer embanhingg
V1
V11
Embeddiing knokupas
DC*
TM
2
calumn::
FOMB: Topic embesklnng
Word embestanus
visible units, VE (1, .... KJ
joft
Document
ToplcPoof: KB pretrained toples
Source-target topic alignments
Collections (DC)
projection space
Multi-source transfer learning TM by introducing pre- trained word embeddings from a WordPool at each autoregressive step i. Double circle + multino- mial (softmax) unit. (Right) Multi-source transfer learning in TM by introducing pretrained (latent) topic embeddings from a TopicPool, illustrating topic aligmments between source and target cor- pora GVT+MST configuration. Each outgoing row from z signify a topic embedding of the corresponding source corpus, DC* Here, T'M refers to DocNADE topic model.
Figure I: (Left) DocNADE (LVT+MST):
beddings instead of using word embeddings exclusively, and (b) Multi-view Transfer: Pretrained word and topic embeddings jointly obtained from large source corpus in order to deal with poly- semy and alleviate data sparsity issues in small target corpus.
Contribution (2) Multi-source Transfer: Moreover, we first learn word and topic representations on multiple source domains to build WordPool and TopicPool, respectively and then perform multi-view and multi- source transfer learning within neural topic modeling by jointly using the complementary representations. In doing so, guide the (unsupervised) generative process of learning hidden topics the target domain by embeddings in WordPool and TopicPool such that the hidden topics become more meaningful and representative explaining the target corpus.
We evaluate the effectiveness of our transfer learning approaches in neural topic modeling using 7 (5 low-resource and 2 high-resource) target and 5 (high-resource) source corpora from news and med- ical domains, consisting of short-text, long-text, small and large document collections. Particularly, quantify the quality text representations via generalization (perplexity), interpretability (topic coherence) and text retrieval. The code is provided with the supplementary.
2 KNOWLEDGE TRANSFER IN NEURAL TOPIC MODELING
Consider sparse target domain T and a set of SS| source domains S, we first prepare two knowledge bases (KBs) of representations (or embeddings) from each of the sources: (1) WordPool: Pretrained word embeddings matrices (E' .... E where E* E Ex and (2) TopicPool: Pretrained latent topic embeddings {Z) .... ž where ² E IR H encodes a distribution over a vocabulary of K words. E and H word embedding and latent topic dimensions, respectively. While topic modeling on T, we introduce the two types of knowledge transfers from one or many sources: Local (LVT) and Global (GVT) View Transfer using the two KBs of pretrained word (i.e., WordPool) and topic (i.c., TopicPool) embeddings, respectively. Specially, we employ neural autoregressive topic model (i.e., DocNADE (I (Larochelle & Lauly 2012) to build the WordPool and TopicPool.
Notice that a superscript indicates source. See Table for the notations used in this work.
2.1 NEURAL AUTOREGRESSSIV TOPIC MODELS
DocNADE (Larochelle & Lauly 2012) is an unsupervised neural-network based topic model that is inspired by the benefits of NADE (Larochelle & Murray] (011]] and RSM (Salakhutdinov & Hinton 2009) architectures. RSM has difficulties due intractability leading approximate gradients of the negative liceliiod,oodd, while NADE does not require such approximations. On other hand, RSM is generative model of word count, while NADE is limited to binary data. Specifically, Doc-
3

---

Under review as a conference paper at ICLR 2020
NADE factorizes the joint probability distribution of words in a document as product of conditional
distributions and efficiently models each conditional via a fced-forward neural network.
Algorithm 1 Computation of ((v) and Loss L(v)
Input: A target training document v, |S] source domains
Input: WordPool: KB of pretrained word embedding matrices (E', ..... E
Input: TopicPool: KB of pretrained latent topics (Z), (Z,,..., Z
Hyper-parameters: 0 = {X ..... Als ..... A
Parameters: Q {b, W, U, A
.7 ..... H}
Initialize: + c and p(v) f-1
for i from 1 to D do
hv(v,s) + g(a), where g (sigmoid, tanh] exp(bu Un. hv(ve))) exp(b + hv(ss))
p(u: w/ve)) V
p(v) + compute pre-activation step, a a- W Wes if LVT then
get word embedding for 0 from source domain(s) a a a - Σk x E.C
[(v) 6- p(v)
if GVT then
[(v) + [(v) + Els the E W- 1
DocNADE Formulation: For document v (UI...., UD) of size D, each word index u takes value in {1..... K] of vocabulary size K. DocNADE learns topics in a language modeling fashion (Bengio et al. 2003) and decomposes the joint distribution p(v)=II== p(us/ve;) such that each autoregressive conditional p(v; Vei) is modeled by a feed-forward neural network using preceding words Viz in the sequence:
exp(b + + h. Σ W W.n))) and p(u: = w/ves) En exp(bw + Uw Www,h, (vvvs))) 9
(v(v = g(c+ +
for i E (I,...D. where Vei is the subvector consisting of all vy such that q < i ie., Vei E (Ui,..., 1i-1 J. gf-) is a non-linear activation function, W E H and U E R are weight matrices, e E IR and b E RK are bias parameter vectors. H is the number of hidden units (topics).
T (without WordPool) provides an illustration of the ith autoregressive step of the Doc- NADE architecture, where the parameter W is shared in the ferd-forward networks and h, encodes topic-proportion embedding. Importantly, the topic-word matrix W has a property that the column vector W ,.,; corresponds to embedding of the word v, whereas the row vector W,,- encodes the jth topic. We We leverage this property to introduce external knowledge via word and topic embeddings.
Additionally, DocNADE has shown to outperform traditional models such as LDA (Blei al. 2003) and RSM (Salakhutdinov & Hinton 2009) in terms of both the log-probability on unseen documents and retrieval accuracy. Recently, Gupta et al. (2019) has improved topic modeling short texts by introducing word embeddings (Pennington et al. 2014) in DocNADE architecture. Thus, we adopt DocNADE to perform transfer learning within the neural topic modeling framework.
Algorithm (for DocNADE, set LVT and GVT to False) demonstrates the computation of log (v) and negative log-likelihood L(v) that is minimized using gradient descent. Moreover, computing h; is efficient (linear complexity) due to NADE architecture that leverages the pre-activation a;-1 of (i 1) step in computing the pre-activation a, See Larochell Lauly (2012) for further details.
2.2 MULTI-VIEN (MVT) AND MULTI- TRANSFERS (MST) IN TOPIC MODELING
Here, describe transfer learning framework in topic modeling that jointly exploits the comple- mentary knowledge using the WordPool and TopicPool, the KBs of pretrained word and (latent) topic embeddings, respectively obtained from large document collections (DCs) from several sources. In
4

---

Under review as a conference paper at ICLR 2020
Target
Domain
Corpora
Source Domain Corpora.
ID
Data
Train
Val
Test
Data
Train
Val
K
8
1
R
D
D
T
20NSshort
1.3k 0.1k
0.5k
1.4k
13.5
20
g1
20NS
7.9k
1.6k
5.2k
2k
107.5 20
S
D
D
T
20NSsmall
0.4k 0.2k
0.2k
2k
187.5
20
82
R21578
7.3k
0.5k
3.0k
2k
128 90
S
R
R
7
D
D
TMNiille
22.8k 2.0k
7.8k
89
TMN
22.8k
2.0k
7.8k
S
R
R
D
R21578ude
0.5k
3.0k
S4
AGNews
P
Obsumedtile
8.3k 2.1k
12.7k
2k
11.9
PubMed
15.0k
2.5k
2.5k 3k 254.8
Table 3:
Domain
Ohsumed
8.3k
2.1k
12.7k
3k
159.1
23
overlap in source- target corpora. I: Identical, R: Re- lated and D: Distant domains.
Table Data statisties: Short/long texts and/or small/large corpora in target and source domains. Symbols- K: vocabulary size, L: average text length (#words), C: number of classes and k: thousand. For short-text, L<15. 8 is also used in target domain. --"; unlabeled data.
doing so, we first apply the DocNADE to generate topic-word matrix for each of the DCs, where its column-vector and row-vector generate Ex and Z respectively for the kth source.
LVT+MST Formulation for Multi-source Word Embedding Transfer: As illustrated in Figure | (left) and Algorithm | with LVT=True, we perform transfer learning on target T using the WordPool of pretrained word embeddings (E'.
E!.] from several sources S (i.e., multi-source)::
hv(v<))= g
Ew...
W 93
1.0% ΣΣ
A E 3,09
Here, & refers to the kth source and X is weight for Ex that controls the amount of knowledge transferred in T. based on domain overlap between target and source(s). Recently, DocNADEe (Gupta et al. 2019) has incorporated word embeddings (Pennington et al. 2014) in extending Doc- NADE; however, it is based on a single source.
GVT+MST Formulation for Multi-source Topic Embedding Transfer: Next. perform knowl- edge transfer exclusively using the TopicPool of pretrained topic embeddings (e.g.. Z*) from one or several sources, S. In doing so, add regularization term the loss function L(v) and require DocNADE to minimize the overall loss in a way that the (latent) topic features W simultane- ously inherit relevant topical features from each of the source domains S, and generate meaningful representations for the target T. The overall loss [(v) due to GVT+MST in DocNADE is given by:
[(v) (vv
Σ
EIIA W 1
Here, H aligns latent topics in the target T and kth source, and 7h governs the degree of imitation of topic features = by W in T. Consequently, the generative process of learning meaningful topics in W of T is guided by relevant features in {2]|s] to address data-sparsity. Algorithm describes the computation of the loss, when GVT True and LVT False.
Moreover, Figure D (right) illustrates the need for topic alignments between target and source(s). Here, j indicates the topic (i.e.. row) index in topic matrix, e.g.. z Observe that the first topic (gray curve), i.e., Zj_1 E Z of the first source aligns with the first row-vector (i.e., topic) of W (of target). However, the other two topics 2 -,=3 E Z' need alignment with the target topics.
MVT+MST Formulation for Multi-source Word and Topic Embeddings Transfer: When LVT and GVT are True (Algorithm[]) for many sources, the two complementary representations jointly used in transfer learning using WordPool and TopicPool and therefore, the name multi-view and multi-source transfers.
EVALUATION AND ANALYSIS
Datasets:
Table describes the datasets used in high-resource source and low-and high- high-resource target domains for our experiments. The target domain T consists of four short-text corpora (20Nsshort, il,title, k21578tit11 tle and it small corpus (20NSsmal) and
5

---

Under review as a conference paper at ICLR 2020
Features
Baselines (Related Works)
NTM
AuR
LVT
GVTJMVT|MST
LDA Blei et al.
2003
RSM Salakhutdinov & Hinton
2009
DocNADE [Lrochelle & Lauly.
2012
NVDM Miio et al.
2016
DA Srivastava & Sutton.
2017
Gauss (Das al.
2015
glove-DMM (gguyen al.
2015
DocNADEe (Gupta al.
2019
EmbSum
doc2vee (Le & Mikolov 2014
this work
Table 4:
Baselines (related works) vs this work. Here, NTM and AuR refer neural network-based TM and autoregressive assumption, respectively. DocNADEe + DocNADE+Glove embeddings.
two large corpora (TMN and Ohsumed). However in source S, we use five large corpora (2UNS, R21578, TMN, AGnews and PubMed) in different label spaces (i.e, domains). Here, the corpora (T", T and 8*) belong to medical and others to news.
Additionally, Table 3 suggests domain overlap (in terms of label match) in the target and source corpora, where we define three types of overlap: I (identical) if all labels match, R (related) if some labels match, and D (distant) if very few or no labels match. Note, our modeling approaches completely unsupervised and do not use the data labels. See the data labels in appendices.
For evaluations in the following sections, follow the experimental setup similar to DocNADE (Larochelle & Lauly] 2012) and DocNADEe (Gupta et al. 2019). where the number of topics (H)) is set to 200. While DocNADEe requires the dimension (i.c.. E') of word embeddings be the same as the latent topic (i.e., H), we first apply a projection on the concatenation of the pre- trained word embeddings obtained from several sources and then, introduce the prior knowledge in each of the autoregressive step following DocNADEe. We apply it in configurations where Glove and/or FastText (E=300) (Bojanowski et al. (017) are employed. See appendices for the experimen- tal setup, hyperparameterg and optimal values of \ [0.1, 0.5, 1.0) and 7. E [0.1, 0.01, 0.001] (determined using development set) different source-target configurations. (code provided)
Reproducibility:
Baselines: As summarized in Table 4 we consider several baselines including (1) LDD--bseed and neural network-based topic models that use the target data, (2) topic models using pretrained word embeddings (i.e., LVT) from Pennington al. (2014) (Glove), (3) unsupervised document represen- tation, where we employ doc2vec (Le & Mikolov 2014) and EmbSum (to represent a document by summing the embedding vectors of its words using Glove) in order quantify the quality of doc- ument representations,
(4) zero-shot topic modeling, where we use all source corpora and no target corpus, and (5) data-angmentatiin, where 'M on TT Using DocNADE, we first prepare the two KBs: WordPool and TopicPool from each of the source corpora and then use them in knowledge transfer to
all source corpora along with target corpus for
Tables 5 and 6 show the comparison of our proposed transfer learning approaches (i.e., LVT using WordPool, GVT using TopicPool, MVT and MST) with the baselines TMs that (1) do not, and (2) do employ pretrained word embeddings (e.g., DocNADE and DocNADEe, respectively).
3.1 GENERALIZATION: PERPLEXITY (PPL)
To evaluate generative performance of TM, we estimate the log- hog-probabilities for the test documents and compute the average held-out perplexity per word as, PPL exp - N Σ M log lo))(,
where N and v, are the number of documents and words in document respectively.
selected with grid search; suhoptimal results (see appendices) by learning À and 7 with backpropagation

---

Under review as a conference paper at ICLR 2020
KBs from
Model/
Scores on Target Corpus (in sparse-data and sufficient-data settings)
Source
Transfer
2hNr
TM
R21578tit
20NS
TMN
Corpus
Type
PPL
COH
IR
PPL
COH
1R
PPL
COH
IR
PPL
COH
JR
PPL
COH
Baseline TM
NVDM
1047
.736
.76
973
.740
.190
372
.735
.271
957
.515
.000
833
.673
without Word-
DrodLDA
923
.689
.022
1527
.744
.170
480
.742
.200
1181
.394
.062
1519
.577
Embeddings
DocNADE
646
.667
.290
706
.709
.521
192
.713
.657
594
.462
.270
584
.66
I.T
630
.673
.298
705
.709
.523
194
.708
.656
594
.455
.288
582
.649
20NS
GVT
646
.600
.303
.720
184
.698
594
.310
590
.652
MVT
6,38
.690
.314
714
.718
.528
188
.715
.655
600
.499
.311
588
.650
LVT
649
.668
.296
655
.731
.548
187
.703
593
.460
.273
TMN
GVT
661
.692
.294
689
.728
.555
191
.709
.660
596
.521
.276
MVT
658
.687
.297
663
.747
.553
195
.720
599
.507
.292
LVT
656
.667
.292
704
.715
.522
186
.715
.676
593
.458
.267
581
.636
R21578
GVT
654
.672
.293
716
.719
.526
194
.706
.672
595
.485
.279
591
.646
MVT
650
.670
.296
716
.720
.528
194
.724
.676
S99
.490
.280
589
.650
LVT
650
.677
.297
682
.723
.533
185
.710
.659
592
.458
.260
564
.668
AGnews
GVT
667
.695
.300
728
.735
.534
190
.717
.663
598
.563
.282
601
.684
MVT
659
.696
.290
718
.740
.533
189
.727
.659
599
.566
.279
592
.686
LVT
640
.678
.308
663
.732
.547
182
.739
594
.542
.277
568
.674
MST
GVT
658
.705
.305
704
.746
.550
192
.727
.673
599
.585
.326
602
.680
MVT
656
.740
.314
.752
.569
188
.745
600
.637
.285
600
.690
Gain% (vx DecNADE)
1,23
10.9
8.28
6.06
5.20
4.49
4.26
0.34
37.9
20.7
3.42
8.50
Table 5: State-of-the-ard comparisons with TMs: Perplexity (PPL., topic coherence (COH) and precision (IR) at retrieval fraction 0.02. Scores are reported on each of the target, given KBs from one or several sources. Please read cosemn-wise. Bold: best in column. Gain%: Bold vs DocNADE.
KBs from
Model/
Scores Target Corpus sparse-data and sufficient-data settings)
Source
Transfer
20X5short
ItMtitle
R21578title
20NSsmall
TMN
Corpus
Type
PPL COH
IR
PPL COH
IR
PPL COH
IR
PPL COII
IR
PPL COM
doc2vec
.090
.190
.188
200
EmbSum
.236
.133
.57
.244
Baseline TM
Gauss-LDA
.080
408
367
090
with Word-
gluve-DMM
.512
.183
.633
.45
.344
.273
.578
.090
.705
Embeddings
DocNADEe
629
.674
.294
680
.719
540
187
.721
.663
590
.455
.244
572
.664
20NS MVT4Glove
630
.721
.320
688
.741
.65
183
.724
.667
597
.561
.306
570
.693
TMN MVT+Glove
640
.731
.295
673
770
.576
184
.716
.672
599
.594
.261
R21578
MVT+Glove
633
.705
.295
689
.738
.40
185
.737
.691
595
.485
.255
577
.697
AGnews
MVT+Glove
642
.734
.302
706
.748
.665
190
.734
.675
598
.573
.284
585
.703
MST
MVT+Glove
644
.739
.304
673
.752
.570
183
.742
.684
598
.631
.282
582
.710
FastText
654
.741
.313
673
.751
.578
183
.744
.684
599
.634
.254
582
.711
Gain % (vs DocNADDe)
9.95
8.84
1.03
4.60
7.04
2.14
3.20
4.22
39.3
2.92
.35
7.08
Table 6: State-of-the-art comparisons with TMs using word embeddings: PPL. COH and IR re- trieval fraction 0.02. Scores are reported on each of the target, given KBs. Here, MVT: LVT+GVT (Table (5). DocNADEe: DocNADE+Glove and Gain%: Bold vs DocNADEe. For all the configura- tions, we apply projection on word embeddings concatenated from several sources.
Tables 5 and 6 quantitatively show PPL scores on the five target corpora (four short-text and one long-text) by the basclines and proposed transfer learning approaches (i.c.. GVT, MVT and MST) using one or four sources. In Table 5 using TMN (as single source) for LVT, GVT and MVT on TMNtitle, we see improved (reduced) PPL scores: (655 vs 706), (689 vs 706) and (663 vs 706) respectively in comparison to DocNADE. We also observe gains due to MST+LVT, MST+GVT and MST+MVT configurations on TMNE itle. Similarly in MST+LVT for R21578titlc. we observe a gain of 5.2% (182 vs 192), suggesting that transfer learning using pretrained word and topic em- beddings (jointly) from one or many sources helps due to positive knowledge transfer, and it also verifies domain relatedness (c.g., in ONN-MMNtiC. it and AGnews-TMN)) Similarly, Table 6 shows gains in PPI. (e.g., TMNtitle, e, R21 #21578tits itle, etc.) compared to DocNADFe.
7

---

Under review as a conference paper at ICLR 2020
KBs from
Model/
Scores on Target Corpus
S
Model
Topic-words (Top
20NS
DNE
shipping, sale, prices, expensive, price sale, price, monitor, sile, selup
Source
Transfer
Chaumedtitlo
Ohsumed
20NGshort
-GVT
Corpus
Type
PPL
CON
1R
PPL COH
IR
+GVT
shipping, sale, price, expensive, subseribe
ProdLDA
1121
.334
080
1677
.646
.080
AGreww
DNE
microsofi, software, ibm, linux, computer apple, modein, side, band, perform
DocNADB
1321
.788
.160
1706
.62
.184
baselines
EmbSum
.150
.148
+GVT
microsoft, software, desktop, computer, apple
DocNADLi
1534
.738
.175
1637
.183
AGnews
DNE
miners, earthquake, explosion, stormed, quake
IVT
1587
.732
.160
1717
.67
.184
THN
DNE
tsunami, quake, jagan, earthquake, radiation strkke jackson, kill, earthquake, injures
GVT
1529
.732
.160
1594
GGVT
AGnews
MVT
1528
.734
.160
1598
.666
.184
+GVT
eartiquake, radiation, explosion, wildlire
BioEmm
1488
.747
.176
1595
.681
.187
LVT
1268
.732
.172
1535
.669
.190
Table 8: Source S and target T topics before (+)
GVT
1392
.740
.173
1718
.671
.192
and after (+) topic transfer(s) (GVT) from one or
PubMed
MVT
1408
.743
.178
1514
.674
.191
more sources. DNE: DocNADE
BloEEm
1364
.753
.182
1633
.689
.191
chip
LVT
1268
.733
.172
1536
.668
.190
GVT
1391
.740
.172
1504
.666
.192
source corpora
target corpus
MST
MVT
1399
.744
.177
1607
20NS
R21578
AGnews
20NSshort
BioEmb
1375
.751
.180
1497
.693
.190
.GVT
+GVT
FatTxt
1350
.753
.178
1641
.688
.187
key
chips serniconductor miti
chips
virus
chips
encrypted
chipmaker
intel
technology
Gain % (vs N
4.01
3.43
13.8
12.3
4.08
4.35
Gain% (vs DocNADFe)
17.3
2.03
4.00
8.55
2.22
4.91
encryption
procesorr
gosh
intel
clipper
makers
semiconductor
crash
encryption
Table 7: PPL., COH, at retrieval fraction 0.02. BioEmb and BioFastText: 200-dimensional word vectors from large biomedical corpus (Moen & Ananiadou 2013). BioEmb: MVT+BioEmb.
keys
semiconductors
intel
chips
clipper
Table 9: Five nearest neighbors of the word chip in source and target semantic spaces before (-) and after (+) knowledge transfer (MST+GVT)
In Table 7 we show PPL scores on two medical target corpora: Ohsumt itle and Ohsumed using two sources: AGnews (news corpus) and PubMed (medical abstracts) to perform cross-domain and in-domain knowledge transfers. We see that using PubMed for LVT on both the target corpora improves generalization. Overall, we report gain of 17.3% (1268 vs 1534) on Onsuntitie and 8.55% (1497 vs 1637) on Ohsumt te, compared to DocNADEe. Additionally, MST+GVT and MST+MVT boost generalization performance compared to DocNADE(e).
3.2 INTERPRETABILTY. TOPIC COHERENCE (COH)
While PPL is used for model selection, adjusting parameters (e.g. H) and quantitative comparisons, Chang et al. (2009) showed in some cases humans preferred TMs (based on the semantic quality of topics) with higher (worse) PPLs. Thus beyond perplexity, we compute topic coherence to estimate the meaningfulness of words in each of the topics captured. In doing so, we choose the coherence measure proposed by Röder et al. (2015) that identifies context features for each topic word using sliding window over the reference corpus. We follow Gupta et al. (019)) and compute COH with the top 10 words in each topic. Essentially, the higher scores imply the more coherent topics.
Tables 5 and 6 (under COH column) demonstrate that our proposed approaches (GVT, MVT and MST) of transfer learning in TMs show noticeable gains in COH and thus, improve topic quality. For instance in Table[5] when AGnews is used a single source for 2ONSsmall datatset, we observe a gain in COH due to GVT (.563 vs .462) and MVT (.566 vs ,462)). Additionally, noticeable gains are reported due MST+LVT (.542 vs .462), MST+GVT (.585 vs .462) and MST+MVT (.637 vs .622). compared to DocNADE. Importantly, we find a trend MVT>>VT>>VT in COH scores for both the single-source and multi-source transfers. Similarly, Table 6 show noticeable gains (e.g., 39.3%, 9.95%, 7.08% etc.) in COH due to MST and MVT with Glove and FastText word embeddings. Moreover. Sable/show, gains in COH due to GVT on Obsumedt itlc and Oh umed, using pretrained knowledge from P PubMed. Overall, the GVT, MVT and MST boost COH for all the five target corpora compared to the baseline TMs (i.e., DocNADE and DocNADEe). This suggests that there is a need for the two complementary (pretrained word and topics) representations and multi-source transfer learning in order to guide meaningful topic learning in T. The results on both the low- and high-resource targets across domains conclude that the proposed modeling scales.
8

---

Under review as a conference paper at ICLR 2020
0.5
0.55
MST-MIV DocNADEe
MST+GiVT
MST-MVT
0.40
DocNADEe
0.6
DocNADLe
0.42
EmniSun
EmbbSum
EiniSum
Precisen
0.38
sugment
033
data augrient
Prediden
0.50
data sugment
ZECO tho
0.5
6.8
0.26
0.20
0.22
0.45
0:22
6.15
0.4
0.18
6:14
0.002
0.01
0.35
0,002
0,005
01.11
0.02
0.05
0.1
0.2
Fraction Retriered Documentss (Recall)
Pruction Retrieved Decuments (Recall)
Fraction Retrieved Ducuments (Rucall)
(a) IR: 20NSshort
(b) IR: 20NSsmall
(c) IR: TMNtitle
MST-MVT
0.56
DocNADEe
Prectoinn at Recall Fracdin
0.75
LmnSSun
0.72
0.7
Preetsten
sugient
0.59
0.7
0.63
0.6
0.07
0.64
0.0
Toplcc
zeno-shor
Inddtt--nggmeen Dn MST+MVT
0.00
DocNADIFe
0.6
Bi DiDNNADFe
DecNADE
0.5
0.59
0.01
100%
0.50
Fraction Retriered Documennss (Recall)
Fraction of training
TMNNile
Obsumed
(d) IR: R21578title
(e) IR: TMNtille
(I) COH: Zero-shot & DA
Figure 2: (a, b, c, d) Retrieval performance (precision) on four datasets.
(c) Precision at recall
fraction 0.02, each for fraction (20%, 40%, 60%, 80%, 100%) of the training set of TMNtille. (f) Zero-shot and data-augmentation (DA) experiments for topic coherence on TMNtitle and Ohsumed.
3.3 APPLICABILITY: INFORMATION RETRIEVAL (IR)
For greater impact of TMs, we further evaluate the quality of document representations and per- form a document retrieval task on the target datasets, using their label information only to compute precision. We follow the experimental setup similar to Lauly al. (2017), where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity between their document vectors. To compute retrieval precision for each fraction (e.g., 0.02), we average the number of retrieved training documents with the same label as the query.
Tables5] depdii precision scores at retrieval fraction 0.02 (similar to Gupta et al. (2019)), where the configuration MST+MVT outperforms both the DocNADE and DocNADEe, respectively in retrieval performance on the four target (short-text) datasets. A gain in IR performance is noticeable for highly overlapping domains, e.g., TMN-TMNtitle (.555 vs .521 in Table 5] .576 vs .540 in Table than the related, e.g., AGnews- AGrewe-TMNtitlee TMNtitle (.534 vs .521 in Table and .565 vs .540 in Table 6). We observe large gains in precision at retrieval fraction 0.02: (a) Table 20.7% (.326 .270)
on 20NSsmal1 1, 9.21 % (.569 vs .521) on rMNt tl and 8.28% (.314 vs .200) on 20Nsshort, (b) Table 6 8.84% (.320 vs .294) on 20NSshort and 9.21 21% (.578 vs .540) on iMNtitle, and (c) Table 7 4.91% (.192 vs. 183) on Ohsumed and 4.0% (.182 vs .155 on Ohsumedtitle
Additionally, Figures 2a 2b 2c and 2d illustrate precision on 20NSshort, 20Nssmal1, TMNt and R21578t ilee. respectively, where our approaches (MST+GVT and MST+MVT) consistently outperform the baselines at all fractions. Moreover, we split the training data of TMN:i=le into several sets: 20%, 40%, 60%, 80% of the training set and then retrain DocNADE, DocNADEe and DoeNADE-MST-MVT We demonstrate the impact of transfer learning in sparse-data settings using WordPool and TopicPool jointly on IR task. Figure [2]] plots precision at retrieval (recall) fraction 0.02 and demonstrates that the proposed modeling consistently outperform DocNADE(c).
3.4 ZERO-SHOP AND DATA- AUGMENTATION EVALUATIONS
Figures 2a
2d show precision in the zero-shot (source-only training) and data- augmentation (tagee training) configurations. Observe that the latter helps in learning
2b
2
and

---

Under review as a conference paper at ICLR 2020
meaningful representations and performs better than zero-shot; however, it is outperformed by MST+MVT, suggesting that a naive (data space) augmentation does not add sufficient prior or rel- evant information to the sparse target. Thus, we find that it is beneficial to augment training data in feature space (e.g., LVT, GVT and MVT) especially for unsupervised TMs using WordPool and TopicPool. Beyond IR, we further investigate computing topic coherence (COH) for zero-shot and data-augmentation baselines, where the COH scores (Figure 2 suggest that MST+MVT outper- forms DocNADEe, zero-shot and data-augmentation.
3.5
QUALITATIVE ANALYSIS: TOPICS AND NEAREST NEIGHBORS (NN)
For topic level inspection, we first extract topics using the rows of W of source and target corpora. Table [8) shows the topics (top-5 words) from source and target domains. Observe that the target topics become more coherent after transfer learning (i.e., +GVT) from one or more sources. The blue color signifies that a target topic has imitated certain topic words from the source. Observe that we also show topics from source domain(s) that align with the topics from target.
For word level inspection, we extract word representations using the columns of W. Table 9 shows nearest neighbors (NNs) of the word chip in 20Nrshort (target) corpus, before and after GVT using three knowledge sources. Observe that the NNs in the target become more meaningful.
4 CONCLUSION
Within neural topic modeling, have introduced transfer learning approaches using complemen- tary representations: pretrained word (local semantics) and topic (global semantics) embeddings exclusively or jointly from one or many sources (i.e., multi-view and multi-source). We have shown that the proposed approaches better deal with data-sparsity issues, especially in a short-text and/or small document collection. We have demonstrated learning meaningful topies and quality document representations on 7 (low- and high-resource) target corpora from news and medical domains.
REFERENCES
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003. URL http:
www. wtt..oo//appp.//////eeoeeeem
html
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. Latent dirichlet allocation. Journal of Ma- chine Learning Research, 3:923-1022, 2003. http:httt:///eet jmlr. otg.papprrs/3//
blei03a. html
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5:135-146. 2017.
Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. Adaptive transfer learning. In Proceedings of the Twenty- Twenty-Fourth. AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press, 2010. URL http:/ www. aaai org/ocs/
index. php/ AAA htt//AAAAO////p/ /tta://AAAA.O//pppp//eeee///////
Jonathan Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceed- ings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada., pp. 288-296, 2009.
Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian Ida for topic models with word embeddings. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 795 804, Association for Computational Linguistics, 2015. doi: 10.3115/v1/P15-0077 URL
http: //acl web. arg/antho 1ogg/
10

---

Under review as a conference paper at ICLR 2020
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the Thirty- AAAI Conference Artificial Intelligence, 2019. WRLLttp://axxiv.orr
abs 180918888880 709
Hugo Larochelle and Stanislas Lauly. neural autoregressive topic model. In Peter L. Bartlett, Fer- nando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems, pp. 2717-2725, 2012. URL
paper/4613 neural aut topic- model
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Geoffrey J. Gordon. David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth Interna- tional Conference on Artificial Intelligence and Statistics, AISTATS, volume 15 of JMLR Pro- ceedings. pp. 29-37. JMLR.org, 2011.
Stanislas Lauly, Yin Zheng, Alexandre Allauzen, and Hugo Larochelle. Document neural autore- gressive distribution estimation. Journal of Machine Learning Research, 18:11::1-11::4, 2017. URL
017 .tmll
Quoe V. Le and Tomas Mikolov. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning, ICML, volume 32 of JMLR Workshop and Conference Proceedings, pp. 1188-1198 JMLR.org, 2014. URLhttp:
w ///mmr..rr//proceeiiiggg paapeerrrsnpppppppespeeeeaa Ctttppers/////11e v32/1e 1e1 4. html
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Pro- ceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1727-1736. JMLR.org. 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Christopher C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Informa- fion Processing Systems, pp. 3111-3119, 2013. URL http:://apppr:: nips cc/paper/ words and-aaraspssssssseeee cechbraass-add--teeeeee and compositoooooiiiionn
SPFGH Moen and Tapio Salakoski2 Sophia Ananiadou. Distributional semantics resources for
biomedical text processing. Proceedings of LBM, pp. 39 44, 2013.
Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. TACL, 3:299-3133 2015. URL https::/taccl2213.coo
columbia edu/ wttp:ed////ojjsooeeeee
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 1532-1543, Association for Computational Linguistics, 2014. doi: 10.3115/ URL http://aclwebh /wwwwwww..beeeeeeee. tttebbooggaaaie /ttp:///aaaoooooo//111111111
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pp. 2227-22377 Association for Computational Linguistics, 2018. doi: 10. (0.18653/v1/N11 8- 1202. URL http:://eewweb..g///mtttoooooll N18 N18-1202
Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pp. 399-408. ACM, 2015. URL
https://des 000 1145/21848828 0115222665533 2685324
Ruslan Salakhutdinov and Geoffrey E. Hinton. Replicated softmax: an undirected topic model. In Yoshua Bengio, Dale Schuurmans, John D. Lafferty, Christopher K. I.

---

Under review as a conference paper at ICLR 2020
Williams, and Aron Culotta (eds.), Advances in Neural Information Processing Sys- tems 22: 23rd Annual Conference on Neural Information Processing Systems, pp. 1607-1614. Curran Associates, Inc., 2009. URL http://papez::
3856-res cated- so ftwatmtaaaaaaaammmmmmaaa an- -e rectee cttp:://aaipp..cpapppppeepp cc/pape/
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In 5th International Conference Learning Representations, ICLR, 2017. URL https://arxiv:
pxc/paE/1703. 0144/
A
DATA DESCRIPTION
In order to evaluate knowledge transfer within unsupervised neural topic modeling, we use the fol- lowing seven datasets in the target domain T following the similar experimental setup as in Doc- NADEe:
1. 20NSshort: We take documents from 20NewsGroups data, with document size less (in terms of number of words) than 20.
2. 20NSsmal1: We sample 20 document (each having more than 200 words) for training from each class of the 20NS dataset. For validation and test, 10 document for each class. Therefore, it is a corpus of few (long) documents.
3. TMNtitle: Titles of the Tag My News (TMN) news dataset.
4. R21578title: Reuters corpus, collection of new stories from nltk. corpus
We
take titles of the documents.
5. Ohsumedtitle: Titles of Ohsumed abstracts. mosshhtti/corpooa..co
Source:
www.unitnn..m
6. Ohsumed: Ohsumed dataset, collection of medical abstracts. Source: disi unitn.
it stmmocchhtt//ooooorrae
7. TMN: The Tag My News (TMN) news dataset.
To prepare knowledge base of word embedings (local semanties) and latent topics (global semantics) features, we use the following six datasets in the source S:
1. 20NS: 20NewsGroups corpus, a collection of news stories from altk.corpu
2. TMN: The Tag My News (TMN) news dataset.
3.
R21578: Reuters corpus, a collection of new stories from nl corpus
4. AGnews: AGnews data sellection.
5. PubMed: Medical abstracts of randomized controlled trials.
Source:
https://
B GETTING WORD AND LATENT TOPIC REPRESENTATIONS FROM SOURCE(S)
Since in DocNADE, the column of W ..;; gives word vector of the word therefore the dimension of word embeddings in each of the E is same (i.e., H = 200). Thus, we prepare the knowledge base word representations E from kth source using DocNADE, where each word vector of H 200 dimension.
Since the row vector of W,, j.: DocNADE encodes jth topic feature, therefore each latent topic (i.e.. row) in feature matrix W is vector of K dimension, corresponding the definition of topics that it is a distribution over vocabulary. H is the number latent topics and is the vocabulary size, where K varies across corpora. Thus, train DocNADE leam feature matrix specific to each the source corpora, e.g. W E T H of keth source.
For target corpus of vocabulary size K', the DocNADE learns a feature matrix WT HxK"" Similarly, W E H for kth source of vocabulary size K. Since in the sparse-data setting for the
12

---

Under review as a conference paper at ICLR 2020
data TMNiile and TMN AGnews
labels / classes
world, us, sport, business, sei sci_lech, entertainment, health
business, sci_lech, sports, world
fosc.forsale, comp-graphics rec.autos,
windowsx,
rec rec.sport, bascbull, bassbpl... sci. sci.space, rec.sport.hockey soc. woc.religion.criiae chriiiiinniiiiin rec. motoreycllss.. comp.sys stspp.sssmma... sys.mac.hardwarc religion.misee electroccs, comp.or os wet.widdows.iis
20NSshort, 20NSsmall, 20NS
sci.med, comp.sys sys.ibm.pe pe.dardaare, talk politics mideast. talk polkkpoiiics. guns, walk,politie..iise
atheism, sei.erypt
trade, grain, crude, nat-gas, corn, rice, rubber, sugar, tin, palm-oil, veg-oil, ship. coffee. lumber, wheat, gold, acq, interest, money-fx, copper. ipi, carcass, livestock. oilseed, soybean, carn, bop, gas, lead, jobs, zinc, opi. gap, soy-oil, dlr, yen, nickel, groundnut, heat, sorghum, sunseed, pet-chem, cocoa, rapeseed, cotton, moncy-supply, iron-stcel, l-cattle, alum, palladium, platinum, strategic-metal. reserves, groundnut-oil, lin-oil, meal-feed, rape-oil, sun-meal, sun-oil, hog. barley. potato, orange, retail, soy-mcal, cotton-oil. cat, fuel, silver, income, wpi, tea, lci, coconut, coconut-oil, copra-cake, dfl, dmk,, naphtha, propane, instal-debt, nzdlr, housing. nkr, rye. castor-oil, jet. palmkemel, epu, rand
R21578titic and R21578
Table 10: Label space of the corpora used
Hyperparameter
Search Space
retrieval fraction learning rate hidden units,
[0.02]
[0.001]
[200]
activation function (g)
sigmoid
iterations
[100]
X
[1.0, 0.5, 0.1]
[0.1, 0.01, 0.001]
Table 11: Hyperparameters in Generalization in DocNADE, DocNADEe, LVT, GVT and MVT
configurations for 200 topics
target, K' << K due additional word in the source. In order perform GVT, need the same topic feature dimensions in the target and source, i.e., K' of the target. Therefore, we remove those column vectors from W E R of the kth source for which there is no corresponding word in the vocabulary of the target domain. As a result, we obtain Z as a latent topic feature matrix to be used knowledge transfer to the target domain. Following the similar steps, we prepare a KB of Zs such that each latent topic feature matrix from source domain gets the same topic feature dimension as the target.
C EXPERIMENTAL SETUP
For DocNADE and DocNADEe in different knowledge transfer configurations, follow the same experimental setup as in DocNADE and DocNADEe. We rerun DocNADE and DocNADEe using the code released for DocNADEe.
C.I EXPERIMENTAL SETUP FOR GENERALIZATION
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with sigmoid hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in PPL. for H 200 topics than using 50, therefore we use 200 in our experiments.
See Table II for hyperparameters used in generalization task, i.e., computing PPL.
See section to reproduce scores Table 1.
C.2 EXPERIMENTAL SETUP FOR IR TASK
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with tank hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in
13

---

Under review as a conference paper at ICLR 2020
Hyperparameter
Search Space
retrieval fraction
[0.02]
learning rate
[0.001]
hidden units, H
[200]
activation function (g)
tanh
iterations
[100]
X
[1.0, 0.5, 0.1]
10.1, 0.01, 0.001]
Table 12: Hyperparameters search in the Information (text) Retrieval task, where A and of are weights for kth source. We use the same grid-search for all the source domains. Hyperparameters in IR task in DocNADE, DocNADEe, LVT, GVT and MVT configurations for 200 topics
KBs from
Model/
Scores Target Corpus sparse-data setting)
setting
Source
Transfer
20M5short
IMNtitle
321578ti:le
20N5 small
Corpus
Type
PPL COH
PPL
COH
IR
PPL COH
IR
PPL
COH
IR
.730
.335
440
.286
651
.712
.523
460
.273
parameterized
MST
MVT
667
.660
667
.730
.535
183
.714
.661
A41
.23
+ Glove
662
672
.731
.540
.716
.662
634
A12
207
LVT
640
.678
.308
663
.732
-547
182
.739
.673
594
.542
.277
GVT
658
.705
.305
704
.766
.550
.727
599
.585
.326
hyper parameterized
MST
MVT
.740
680
.752
.569
.745
600
.637
.285
Table 13: {X, 7 as Parameter vs Hyperparameters: Perplexity (PPL), topic coherence (COH) and precision (IR) retrieval fraction 0.02, when and y are (1) learned with backpropagation, and (2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Therefore, in the paper we have reported scores considering {), 7} hyperpa- rameters. + Glove: MVT+Glove embeddings, Please read column-wise. Bold: best in column.
precision for the retrieval task for H 200 topics than using 50, therefore we use H 200 in our experiments. We follow the similar experimental setup as in DocNADEe. For model selection, we used the validation set as the query set and used the average precision 0.02 retrieved documents as the performance measure. Note that the labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same (Table 2 train/development/es split of documents for all the datasets during learning.
Given DocNADE, the representation of document of size D can be computed by taking the last hidden vector hp at the autoregressive step D. Since, the RSM and DocNADE strictly outperformed LDA, therefore we only compare DocNADE and its recent extension DocNADFe. We use the same number of topic dimensions (H 200) across all the source domains and the target in training with DoCNADE.
See Table 12 for the hyperparameters in the document retrieval task, where A and y are weights for kth source. We use the same grid-search for all the source domains. We set smaller than x to control the degree of imitation of the source dsmain(s) by the target domain. We use the development set of the target corpus to find the optimal setting in different configurations of knowledge transfers from several sources.
See section C.4 reproduce scores Table
C.3 {) AS PARAMETER vs HYPERPARAMETERS
Here, we treat and y as parameters of the model, instead of hyperparameters and learn them with backpropagation. We initialize each A 0.5 and 7e = 0.01 for each of the sources. We perform experiments on short-text datasets in MST+LVT, MST+GVT and MST+MVT configurations. We evaluate the topic modeling using PPL., topic coherence and retrieval accuracy.
14

---

Under review as a conference paper at ICLR 2020
Table 13 reports the scores, when À and y are (1) learned with backpropagation, and (2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Therefore in the work, have reported scores considering {}, 7 as hyperparameters.
C.4 REPRODUCTRILIIYY OPTIMAL CONFIGURATIONS OF À AND
X takes on values in [1.0, 0.5, 0.1] for cach of the word embeddings matrix E. and 7thh in 0.01, 0.001] for each of the latent topic features z", respectively for the 6d source domain. To determine an optimal configuration, we perform determine the best setting. We have a common model for PPL and COH scores due to generalization criteria.
As mentioned in Tables 11 and 12 the
grid-search the values and use the scores the development
To reproduce scores (best in Table 5), mentioned the best settings of (A*, 74) in MST+MVT configuration for each of the target and source combinations:
1. Generalization (PPL. and COH) in MST+MVT when target is 20NSshort,, (X20NS = 1.0, 22ONS 0.001, XTMN 0.1, TTMN 0.001, A R21578
0.5, y R21578 0.001,
A AGrews = 0.1, 7 AGnews = 0.001
2. Generalization (PPL and COH) in MST+MVT when target is TMNt Mtleiitle: (A29NS 0.1.
7 20N 0.001, XTMN
1.0, TTMN = 0.001, 121578 0.5, 7 R22378 0.001.
X AGnews = 1.0, y AGneus 0.001
3. Generalization (PPL and COH) in MST+MVT when target is R2157ttitle: (A (A2NNS_
0.1, 7,2NN = 0.001, ATMN = 0.5, 7 TMN 0.001, R21576 = 1.0, 7,0,77'm0 722578 0.001,
X AGhews 1.0, 7 ACneus 0.001
4. Generalization (PPL and COH) MST+MVT when target is 20NSsma 11: (2 20NS =
0.5, 222NS 0.001, XTMN 0.1, TMN 0.001, A R2157 0.1, 7 R21578 0.001,
A A.Gnws = 0.1, 7 AGnews 0.001
5. Generalization (PPL and COH) in MST+MVT when target is Ohsumedtitle: (AAGnews = 0.1, 7 AGnews = -0.001, PabMed 1.0, 7 PubMed 0.001
6. Generalization (PPI. and COH) in MST+MVT when target is Ohsumed: (AAGneus 0.1, AGrous 0.001, A PubMed
1.0, Pubbeed 0.001
7. IR in MST+MVT when target is 20NSshort: (A 2055 = 1.0, 7 20NS 0.1, ATMN
0.5, TMN 0.01, X 122158 0.1, 122157 0.001, X AGneus
1.0,
AGnews 0.01
8. in MST+MVT when target TMNtitle: (X20NS 0.1, 20NN 0.01, XTMN
1.0, 7 TMN 0.01, X R21578 = 0.1, 7 R21578 = 0.01, X AGnews = 0.3, AGnews 0.001
9. IR in MST+MVT when target is R21578title: (220NS
0.1, 20NS 0.01,
A TMN = 1.0, 7 TMN = 0.01, A R21578 = 1.0, 7 R21578 = 0.01, X AGnrass = 0.5.
AGnews = 0.001
in MST+GVT when target 20NSsma 11: (7 20NS 0,01, TMN 0.01, R21378
0.1, 7 AGnews 0.01
11. IR in MST+MVT when target is iisumedditte. (AAGnews 0.1, AGnews 0.001,
A Pubbled = 1.0, 7 PubMed = 0.1
12. IR in MST+MVT when target is Ohsumed: ().Geeess = 0.1, AGnews 0.001.
A PublMed = 0.5, 7 PubMed = 0.1
The Additionally, we have also provided the code.
mentioned above also applies to a single source transfer configuration.
While DocNADEe requires the dimension (i.c., E') word embeddings the same as the latent topic (i.e., H)), we first apply projection on the concatenation of the pretrained word embeddings obtained from several sources and then, introduce the prior knowledge in each of the autoregressive step following DocNADEe. We apply it in configurations where Glove and/or FastText (E=300) are employed. IN these settings, we use a single mixture weight E [1.0, 0.5, 0.1] over the projected vector and then, introduced in TM following DocNADEe.
15

---

Under review as a conference paper at ICLR 2020
C.5 EXPERIMENTAL SETUP FOR NVDM AND PRODLDA
For NVDM, we run the code availale at gi github.. wwt.gg....ooyysaaeeee aatp.///..ooyymlla////mmmmnaaadd train for 200 topics.
For ProdLDA, the code availale ggtthibb.com/kkaaaaggtttttttttnn
for for_topic_met c_model s and train for 200 topies.
C.6 EXPERIMENTAL SETUP FOR GLOVE-DMM
com da tpwaatuuuunggungnncT to train glove-DMM model. It is trained for 200 iterations with 2000 initial iterations using 200 topics. For short texts, we set the hyperparameter beta to 0.1, for long texts to 0.01; the mixture parameter lambda was set to 0.6 for all datasets. IR task was performed using relative topic proportions input, where inferred the topic distribution of the training and test documents and used the relative distribution as input in computing similarities in documents based on the inferred relative topic distribution.
We used LFTM (tttps:/
C.7 EXPERIMENTAL SETUP FOR DOC2VEC
We used gensim https://gitt::
com/RaRe- www.:ooogies//enee.......... to train Doc2Vec models. Models were trained with distributed bag of words, for 1000 iterations using window size of 5 and vector size of 500.
16


Reviews:
[{"review_id": "SygTrGoa9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an emergency review.\n\nThis work proposes a novel method to use pre-trained topic embeddings and pre-trained word embeddings obtained from various corpora in the transfer learning framework. \n\nTheir model architecture is based on DocNADE, unsupervised neural-network based topic model, and the authors propose two strategies to use pre-trained topic embeddings and pre-trained word vectors.\n1) Addition of a weighted sum of pre-trained word embeddings and the hidden vector of DocNADE.\n2) L2-Regularization term between topic embedding of DocNADE and pre-trained topic embeddings. They propose to align these two embeddings by multiplying align matrix \"A\" to the topic embedding of DocNADE.\n\nThey show the transfer learning performance of their model on various source/target domain datasets, including medical target corpora, and verify that their model outperforms on a short text and small document collection.\n\nStrengths.\n1. Comparison with the data augmentation baseline shows the performance gain is not only from bigger training data. Even though comparison with the naive baseline (data augmentation) seems too obvious, I think the results clearly show their claim about the importance of using transfer learning in neural topic modeling domain.\n2. As the first approach that introduces a novel transfer learning framework with pre-trained topic embeddings, they show tons of experimental results with various datasets and metrics to show the specification of their method. Their experimental setting is well designed.\n\nWeaknesses and comments:\nTheir method to combine pre-trained word embeddings and pre-trained topic embeddings is too simple. Since this is the first approach to use topic embedding in the transfer learning field, the simplicity of the proposed method is somewhat necessary. However, a weighted sum of pre-trained topic/word vectors seems not enough to transfer multisource knowledge. For instance, word vectors obtained from individual training processes do not share embedding vector space. As you apply the alignment method to topic embeddings from various sources, you should align word embeddings too.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkgKHY_aKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "On the basis of existing topic modelling approaches, the authors apply a transfer learning approach to incorporate additional knowledge to topic models, using both word embeddings and topic models. The underlying idea is that topic models contain a global view that differs on a thematic level, while word embeddings contain a local, immediate contextual view. The combination of both local and global view transfer to enhance a topic model is the main contribution of this paper, especially when using multiple sources (therefore the title: multi-source multi-view transfer).\nGiven a document collection, DocNADE is used to generate the topic-word matrix. In the local view transfer step, the pre-trained WordPool is used, from which knowledge is transferred on the target document. The global view transfer is done by transferring knowledge from the pre-trained TopicPool to the target. As described in Algorithm 1 in the paper, both Word- and TopicPool are jointly used in the transfer learning process. \nFor evaluation, three different measures are taken into account: Perplexity, Topic Coherence and Precision (Information Retrieval). In comparison to a DocNADE only approach, all values are better in the settings that use the transfer learning approach. Compared to DocNADE + word embeddings, the results are competitive as well. In both experiments, the multi-source setting evaluates best overall.\n\nIn conclusion, the paper shows that exploiting multiple sources and views in transfer learning leads to an overall improvement in the given tasks. The main contribution is the usage topic models in a transfer learning framework. Additionally the use of multi-source word embeddings is novel too, especially in the joint setting with the topic model transfer. The paper shows how the DocNADE approach is enhanced to make use of both local and global view transfer and how this enhancement leads to improved performance on various related tasks. \nStill, the overall contribution is mostly in combining existing methods and can be judged as rather incremental.\n\nMinor note: A small mistake has been found in Table 5. The best perplexity value in the first column is not the bold 638, but the 630 in the local-view transfer setting.\n\nEdit after rebuttal: In my review I did not value the contribution of the transfer learning approach enough. So, when also considering the extensive evaluation I am now leaning towards accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeGN5C85B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a multi-source and multi-view transfer learning for neural topic modelling with the pre-trained topic and word embedding. The method is based on NEURAL AUTOREGRESSIVE TOPIC MODELs --- DocNADE (Larochelle&Lauly,2012). DocNADE learns topics using language modelling framework. DocNADEe (Gupta et al., 2019) extended DocNADE by incorporating word embeddings, the approach the authors described as a single source extension of the existing method.\n\nIn this paper, the proposed method adds a regularizer term to the DocNADE loss function to minimize the overall loss whereas keeping the existing single-source extension. The authors claimed that incorporating the regularizer will facilitate learning the (latent) topic features in the trainable parameters simultaneously and inherit relevant topical features from each of the source domains and generate meaningful representations for the target domain. The analysis and evaluation were presented to show the effectiveness of the proposed method. However, the results are not significantly improved than the based line model DocNADE. \n\nOverall, the paper is written well. However, it is not clear to me that the improved results are resulted due to multi-source multi-view transfer learning or for the better leaning of the single-source model due to the incorporation of the regularizer. \n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a transfer learning framework in neural topic modeling. Authors claim and reviewers agree that this view of transfer learning in the realm of topic modeling is novel.

However, after much deliberation and discussion among the reviewers, we conclude that this paper does not contribute sufficient novelty in terms of the method. Also, reviewers find the experiments and results not sufficiently convincing.

I sincerely thank the authors for submitting to ICLR and hope to see a revised paper in a future venue.
Decision: Reject