Venue: NAACL 2021

# Multi-source Neural Topic Modeling in Multi-view Embedding Spaces

*Pankaj Gupta1, *Yatin Chaudhary1,2, Hinrich Schutze2

1DRIMCo GmbH Munich, Germany

2CIS, University of Munich (LMU) Munich, Germany info@drimco.net lyatin.chaudhary@drimco.net

###### Abstract

Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity in short-text or small collection of documents. This work presents a novel neural topic modeling framework using multi-view embedding spaces: (1) pretrained topic-embeddings, and (2) pretrained word-embeddings (context-insensitive from Glove and context-sensitive from BERT models) _jointly_ from one or _many sources_ to improve topic quality and better deal with polysemy. In doing so, we first build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). We then identify one or more relevant source domain(s) and transfer knowledge to guide meaningful learning in the sparse target domain. Within neural topic modeling, we quantify the quality of topics and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. Introducing the multi-source multi-view embedding spaces, we have shown state-of-the-art neural topic modeling using 6 source (high-resource) and 5 target (low-resource) corpora.

+
Footnote †: equal contribution

_Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4205–4217 June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Probabilistic topic models, such as LDA Blei et al. (2003), Replicated Softmax (RSM) Salakhutdinov and Hinton (2009) and Document Neural Autoregressive Distribution Estimator DocNADE Larochelle and Lauly (2012) are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks, such as information retrieval IR. Though they have been shown to be powerful in modeling large text corpora, the topic modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient, e.g., on short-text or a corpus of few documents. It leads to a poor quality of topics and representations.

To address data sparsity issues, several works Das et al. (2015); Nguyen et al. (2015); Gupta et al. (2019, 2020) have introduced external knowledge in traditional topic models, e.g., incorporating word embeddings obtained from Glove Pennington et al. (2014) or word2vec Mikolov et al. (2013a). However, no prior work in topic modeling has employed multi-view embedding spaces: (1) _pretrained topics_, i.e., topical embeddings obtained from large document collections, and (2) _pretrained contextualized word embeddings_ from large-scale language models like BERT Devlin et al. (2019).

Though topics and word embeddings are complementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. A topic model Blei et al. (2003) is a statistical tool to infers topic distributions across a collection of documents and assigns a topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. Therefore, a topic has a _global view_ representing semantic structures hidden in document collection. On other hand, word embeddings have primarily _local view_ in the

\begin{table}
\begin{tabular}{c|p{85.4pt}|p{85.4pt}} \hline \hline
**Topic** & **Topic Words** & **Topic Label** \\ \hline \multirow{2}{*}{\(Z_{1}\left(S^{1}\right)\)} & profit, growth, stocks, **apple**, fall, consumer, buy, billion, shares & _Trailing_ \\ \cline{2-3}  & smartphone, ipad, **apple**, app, phone, devices, phone, tablet & _Product Line_ \\ \hline \multirow{2}{*}{\(Z_{3}\left(S^{4}\right)\)} & microsoft, mac, hnu, bin, nos, applex, p windows, software & _Operating System_ \\ \hline \multirow{2}{*}{\(Z_{4}\left(\mathcal{T}\right)\)} & **apple**, talk, computers, shares, disease, driver, electronics, profit, los & ? \\ \hline \hline \end{tabular}
\end{table}
Table 1: Coherent (\(Z_{1}\)-\(Z_{3}\)) vs Incoherent (\(Z_{4}\)) topics from high-resource (\(S^{1}\)-\(S^{3}\)) and low-resource (\(\mathcal{T}\)) texts

sense that they are learned based on local collocation pattern in a text corpus, where the representation of each word often depends on a local context window Mikolov et al. (2013) or is a function of its sentence(s) Peters et al. (2018). Consequently, they are not aware of the thematic structures underlying the document collection. Additionally, recent studies Peters et al. (2018); Devlin et al. (2019); Liu et al. (2019) have shown a reasonable success in several NLP applications by employing pretrained contextualized word embeddings, where the representation of a word is different in different contexts (i.e., context-sensitive). In context of this work, the representations due to global and local (context-sensitive or context-insensitive) views together are referred as _multi-view_ embeddings.

For example in Table 1, consider four topics (\(Z_{1}\)-\(Z_{4}\)) of different domains where the topics (\(Z_{1}\)-\(Z_{3}\)) are respectively obtained from three different high-resource source (\(\mathcal{S}^{1}\)-\(\mathcal{S}^{3}\)) domains whereas \(Z_{4}\) from a low-resource target domain \(\mathcal{T}\) (especially in the data-sparsity settings). Observe that the topics about _Trading_ (\(Z_{1}\)), _Product Line_ (\(Z_{2}\)) and _Operating System_ (\(Z_{3}\)) are coherent and and represent meaningful semantics at document-level via lists of topic words. However in sparse-data settings, the topic \(Z_{4}\) discovered is incoherent (noisy) and it is difficult to infer meaningful document semantics.

Unlike the topics, word embeddings (context-insensitive) encode syntactic and semantic relatedness in fine-granularity and therefore, do not capture thematic structures. For instance, the top-5 nearest neighbors (NN) of _apple_ (below) in word embedding Mikolov et al. (2013) space suggest that it refers to a _fruit_ and do not express any topical information (e.g., _Trading_, _Product Line_ or _Health_) in the corpora. Similarly given the NN of the word _fall_, it is difficult to infer its association with document-level semantics, e.g., _Trading_ as expressed by \(Z_{1}\) in topic-embedding space.

```
applye=\(\xrightarrow[]{\text{NN}}\) apples, pear, fruit, berry, pears, strawberry fall\(\xrightarrow[]{\text{NN}}\) falling, falls, drop, tumble, rise, plummet
```

Therefore, topic and word embedding spaces encode complementary semantics. Different to context-insensitive word embeddings, the word _apple_ is referring to an organization and contextualized by different topical semantics respectively in the three sources \(\mathcal{S}^{1}\)-\(\mathcal{S}^{3}\). Thus, it arises the need for context-sensitive embeddings in topic modeling.

**Contribution (1) Multi-view Neural Topic Modeling using pretrained word and topic embeddings**: To alleviate the data sparsity issues, it is the **first work** in unsupervised neural topic modeling (NTM) within transfer learning paradigm that employs _multi-view embedding spaces_ via: (a) _Global-view Transfer_ (_GVT_): Pretrained topic embeddings instead of using word embeddings exclusively, and (b) _Multi-view Transfer_ (_MVT_): Pretrained topic and word embeddings (context-insensitive from Glove Pennington et al. (2014) and context-sensitive from large-scale language models such as BERT Devlin et al. (2019) jointly to address data sparsity and polysemy issues.

**Contribution (2) Multi-source Multi-view Neural Topic Modeling**: A single source of prior knowledge is often insufficient due to incomplete and non-overlapping domain information required by a target domain. Therefore, there is a need to leverage multiple sources of prior knowledge, dealing with domain-shifts Cao et al. (2010) among the target and sources. In doing so, we first learn word and topic representations on multiple source domains to build WordPool and TopicPool, respectively and then perform _multi-view_ and _multi-source_ transfer learning in neural topic modeling by jointly using the complementary representations.

We evaluate the effectiveness of multi-source neural topic modeling in multi-view embedding spaces using 7 (5 low-resource and 2 high-resource) target and 5 (high-resource) source corpora from news and medical domains, consisting of short-text, long-text, small and large document collections. We have shown state-of-the-art results with significant gains quantified by generalization (perplexity), interpretability (topic coherence) and text retrieval. The code is available at [https://github.com/YatinChaudhary/Multi-view-Multi-source-Topic-Modeling](https://github.com/YatinChaudhary/Multi-view-Multi-source-Topic-Modeling).

\begin{table}
\begin{tabular}{c|l} \hline
**Notation** & **Description** \\ \hline LYT, GVT & Local-view Transfer, Global-view Transfer \\ MVT, MST & Multi-view Transfer, Multi-source Transfer \\ \(\mathcal{T},\mathcal{S}\) & A target domain, a set of source domains \\ \(\mathbf{v}\), \(k\), \(\mathcal{L}\) & An input document, \(k\)th source, loss \\ \(K,D\) & Vocabulary size, document size \\ \(E,H\) & Word embedding dimension, Hotpics \\ \(\mathbf{W}\in\mathbb{R}^{H\times K}\) & Encoding matrix of DocNADE in \(\mathcal{T}\) \\ \(\mathbf{U}\in\mathbb{R}^{K\times H}\) & Decoding matrix of DocNADE \\ \(\lambda^{k}\) & Degree of relevance of \(\mathbf{E}^{k}\) in \(\mathcal{T}\) \\ \(\lambda^{k}\) & Degree of imitation of \(\mathbf{Z}^{k}\) by \(\mathbf{V}\) \\ \(\mathbf{E}^{k}\in\mathbb{R}^{B\times K}\) & Word embeddings of \(kh\) source \\ \(\mathbf{Z}^{k}\in\mathbb{R}^{H\times K}\) & Topic embeddings of \(kh\) source \\ \(\mathbf{A}^{k}\in\mathbb{R}^{H\times H}\) & Topic-alignment in \(\mathcal{T}\) and \(\mathbf{Z}^{k}\) \\ \(\mathbf{b}\in\mathbb{R}^{K},\epsilon\in\mathbb{R}^{H}\) & Visible-bias, hidden-bias \\ _DC_ & Document Collection \\ \end{tabular}
\end{table}
Table 2: Description of the notations used in this work 



## 2 Knowledge-Aware Topic Modeling

Consider a sparse target domain \({\cal T}\) and a set of source domains \({\cal S}\), we first prepare two knowledge bases (KBs) of representations (or embeddings) from document collections of each of the \(|{\cal S}|\) sources: (1) WordPool: a KB of pretrained word embeddings matrices \(\{{\bf E}^{1},...,{\bf E}^{|{\cal S}|}\}\), where \({\bf E}^{k}\in\mathbb{R}^{E\times K}\), and (2) TopicPool: a KB of pretrained latent topic embeddings \(\{{\bf Z}^{1},...,{\bf Z}^{|{\cal S}|}\}\), where \({\bf Z}^{k}\in\mathbb{R}^{H\times K}\) encodes a distribution over a vocabulary of \(K\) words. Here, \(k\in[1,...,|{\cal S}|]\) in superscript indicates knowledge of \(k\)th source, and \(E\) and \(H\) are word embedding and latent topic dimensions, respectively. While topic modeling on \({\cal T}\), we introduce the two types of knowledge transfers from one or many sources: _Local_ (LVT) and _Global_ (GVT) _View Transfer_ using the two KBs of pretrained word (i.e., WordPool) and topic (i.e., TopicPool) embeddings, respectively. Specially, we employ a neural autoregressive topic model, i.e., DocNADE as backbone in building the pools and realizing the multi-source multi-view framework.

Table 2 describes the notations used. _Notice_ that the superscript used in notations indicates a source.

### Neural Autoregressive Topic Models

DocNADE Larochelle and Lauly (2012) is an unsupervised neural-network based generative topic model that is inspired by the benefits of NADE Larochelle and Murray (2011) and RSM Salakhutdinov and Hinton (2009) architectures. Specifically, DocNADE factorizes the joint probability distribution of words in a document as a product of conditional distributions and efficiently models each conditional via a feed-forward neural network (ffnet), following reconstruction mechanism.

**DocNADE Formulation**: For a document \({\bf v}=(v_{1},...,v_{D})\) of size \(D\), each word index \(v_{i}\) takes value in \(\{1,...,K\}\) of vocabulary size \(K\). DocNADE learns topics in a language modeling fashion Bengio et al. (2003) and decomposes the joint distribution \(p({\bf v})\)=\(\prod_{i=1}^{D}p(v_{i}|{\bf v}_{<i})\) such that each autoregressive conditional \(p(v_{i}|{\bf v}_{<i})\) is modeled by a ff-net using preceding words \({\bf v}_{<i}\) in the sequence:

for each word \(i\in\{1,...,D\}\) where \({\bf v}_{<i}\) is the subvector consisting of all \(v_{q}\) such that \(q<i\) i.e., \({\bf v}_{<i}\in\{v_{1},...,v_{i-1}\}\), \(g(\cdot)\) is a non-linear activation function, \({\bf W}\in\mathbb{R}^{H\times K}\) and \({\bf U}\in\mathbb{R}^{K\times H}\) are weight matrices, \({\bf c}\in\mathbb{R}^{H}\) and \({\bf b}\in\mathbb{R}^{K}\) are bias parameter vectors. \(H\) is the number of hidden units (the number of topics to be discovered).

Figure 1 (left) (exceptWordPool) describes the DocNADE architecture for the \(i\)th autoregressive step, where the parameter \({\bf W}\) is shared in the feedforward networks and \({\bf h}_{i}\) encodes latent documenttopic proportion. The value of each unit \(j\) in the hidden vector signifies contribution of the \(j\)th topic in the proportion. Importantly, the topic-word matrix \({\bf W}\) has a property that the column vector \({\bf W}_{:,v_{i}}\) corresponds to embedding of the word \(v_{i}\), whereas the row vector \({\bf W}_{j:}\) encodes latent features for the \(j\)th topic (i.e., topic-word distribution). We leverage this property to introduce external knowledge via word and topic embeddings.

Figure 1: (Left) DocNADE (LVT+MST): Multi-source transfer learning in NTM for a document \({\bf v}\) by introducing pretrained word embeddings from a WordPool at each autoregressive step \(i\). Double circle \(\rightarrow\) multinomial (softmax) unit Larochelle and Lauly (2012). (Right) DocNADE (GVT+MST): Multi-source transfer learning in NTM by introducing pretrained (latent) topic embeddings from a TopicPool, illustrating topic alignments between source and target corpora. Each outgoing row from \({\bf Z}^{k}\)\(\in\)\(\mathbb{R}^{H\times K}\) signify a topic embedding of corresponding \(k\)th source corpus, \(DC^{k}\). Here, NTM refers to a DocNADE Larochelle and Lauly (2012) based Neural Topic Model.



Algorithm 1 (for DocNADE, set both LVT and GVT to _False_) demonstrates the computation of \(\log p({\bf v})\) and loss (i.e., negative log-likelihood) \(\mathcal{L}({\bf v})\) that is minimized using stochastic gradient descent. Moreover, computing each \({\bf h}_{i}\) is efficient (linear complexity) due to NADE architecture that leverages the pre-activation \({\bf a}_{i-1}\) of \((i-1)\)th step in computing \({\bf a}_{i}\) for the \(i\)th step (line #6). See Larochelle and Lauly (2012) for further details.

_Why DocNADE backbone_: It has shown outperforming traditional models such as LDA and RSM. Additionally, Gupta et al. (2019a,b) have extended DocNADE on short texts by introducing context-insensitive word embeddings; however, based on a single-source transfer. Thus, we adopt DocNADE.

### MVT and MST in Neural Topic Modeling

We describe our transfer learning framework in topic modeling that jointly exploits the complementary prior knowledge accumulated in (WordPool, TopicPool), obtained from large document collections (DCs) from several sources. In doing so, we first apply the DocNADE to generate a topic-word matrix for each of the DCs, where its column-vector and row-vector generate \({\mathbf{E}^{k}}\) and \({\mathbf{Z}^{k}}\), respectively for the \(k\)th source. See _appendix_ for the mechanics of extracting word and topic embeddings from the topic-word matrix of a source.

**LVT+MST Formulation for Multi-source Word Embedding Transfer**: As illustrated in Figure 1 (left) and Algorithm 1 (with LVT being _True_, line #7), we perform transfer learning on a target \(\mathcal{T}\) using the WordPool of pretrained word embeddings \(\{{\mathbf{E}^{1}},...,{\mathbf{E}^{|\mathcal{S}|}}\}\) from several sources \(\mathcal{S}\) (i.e., multi-source) under the two schemes:

_scheme_ **(i)**: Using a domain-relevance factor \(\lambda\) for every source in the WordPool such that the hidden vector \({\bf h}_{i}\) encodes document-topic distribution, augmented with prior knowledge in form of pretrained word embeddings from several sources:

\[{\bf h}_{i}({\bf v}_{<i})=g({\bf c}+\sum_{q<i}{\bf W}_{:,v_{q}}+\sum_{q<i} \sum_{k=1}^{|\mathcal{S}|}\lambda^{k}~{}{\mathbf{E}^{k}_{:,v_{q}}})\]

Here, \(k\) refers to the \(k\)th source and \(\lambda^{k}\) is a weight for \({\mathbf{E}^{k}}\) that controls the amount of knowledge transferred in \(\mathcal{T}\), based on cross-domain overlap.

_scheme_ **(ii)**: Using a projection matrix \({\mathbf{P}}\in\mathbb{R}^{H\times P}\) with \(P=E\times|\mathcal{S}|\) in order to align word-embedding spaces of the target and all source domains for all \(D\) words in the document \({\bf v}\) such that:

For \(q\in\{i,...D\}:{\hat{\bf e}}_{q}=\text{concat}({\mathbf{E}^{1}}_{:,v_{q}},.. ,{\mathbf{E}^{k}_{:,v_{q}}})\)

\[{\bf h}_{i}({\bf v}_{<i})=g({\bf c}+\sum_{q<i}{\bf W}_{:,v_{q}}+\sum_{q<i}{\bf P }\cdot{\hat{\bf e}}_{q})\]

Unlike scheme (i), the second schema allows us to automatically determine shifts in the target and source domains, identify and transfer relevant prior knowledge from many sources without configuring \(\lambda\) for every source. To better guide TM, we also introduce pre-trained contextualized word embedding from BERT, concatenating with \({\hat{\bf e}}_{q}\).

**GVT+MST Formulation for Multi-source Topic Embedding Transfer**: Next, we perform knowledge transfer exclusively using the TopicPool of pretrained topic embeddings (e.g., \({\mathbf{Z}^{k}}\)) from one or several sources, \(\mathcal{S}\). In doing so, we add a regularization term to the loss function \(\mathcal{L}({\bf v})\) and require DocNADE to minimize the overall loss in a way that the (latent) topic features in \({\bf W}\) simultaneously inherit relevant topical features from each of the source domains \(\mathcal{S}\), and thus, it generates meaningful representations for the target \(\mathcal{T}\) in order to address data-sparsity. The overall loss \(\mathcal{L}({\bf v})\) due to GVT+MST configuration in DocNADE is: 

Here, \({\bf A}^{k}{\in}\mathbb{R}^{H\times H}\) aligns latent topics in the target \(\mathcal{T}\) and \(k\)th source, and \(\gamma^{k}\) governs the degree of imitation of topic features \({\bf Z}^{k}\) by \({\bf W}\) in \(\mathcal{T}\). Consequently, the generative process of learning meaningful topics in \({\bf W}\) of the target domain \(\mathcal{T}\) is guided by relevant topic features \(\{{\bf Z}_{1}\}_{1}^{|S|}\in\texttt{ TopicPool}\). Algorithm 1 (line #11) describes the computation of the loss, when GVT = _True_ and LVT = _False_.

Moreover, Figure 1 (right) illustrates the need for topic alignments between target and source(s). Here, \(j\) indicates the topic (i.e., row) index in a topic matrix, e.g., \({\bf Z}^{k}\). Observe that the first topic (gray curve), i.e., \(Z_{j=1}^{1}\in{\bf Z}^{1}\) of the first source aligns with the first row-vector (i.e., topic) of \({\bf W}\) (of target). However, the other two topics \(Z_{j=2}^{1},Z_{j=3}^{1}\in{\bf Z}^{1}\) need alignment with the target.

**MVT+MST Formulation for Multi-source Word and Topic Embeddings Transfer**: When LVT and GVT are _True_ (Algorithm 1) for many sources, the two complementary representations are jointly used in transfer learning using WordPool and TopicPool, and therefore, the name _multi-view_ and _multi-source_ transfers.

**Computational complexity of NTM**: For DocNADE, the complexity of computing all hidden layers \({\bf h}_{i}({\bf v}_{<i})\) is in \(O(DH)\) and all \(p({\bf v}|{\bf v}_{<i})\) in \(O(KDH)\). Thus, the overall complexity of DocNADE is in \(O(DH+KDH)\).

Within the proposed transfer learning framework, the complexity of computing all hidden layers (LVT+MST in _scheme_ (i)) and topic-embedding transfer term (GVT+MST) is in \(O(DH+|\mathcal{S}|DH)\) and \(O(|\mathcal{S}|KH)\), respectively. Since \(|\mathcal{S}|<<\!\!H\), thus the overall complexity of DocNADE with MVT+MST is in \(O(DH+KDH+KH)\).

## 3 Evaluation and Analysis

**Datasets**: Table 3 describes the datasets used in high-resource source and low-and high-resource target domains for our experiments. The target domain \(\mathcal{T}\) consists of four short-text corpora (20NSshort, TMNtitle, R21578title and Ohsumedtitle), one small corpus (20NSsmall) and two large corpora (TMN and Ohsumed). However in source \(\mathcal{S}\), we use five large corpora (20NS, R21578, TMN, AGnews and PubMed) in different label spaces (i.e, domains). Here, the corpora (\(\mathcal{T}^{5}\), \(\mathcal{T}^{6}\) and \(\mathcal{S}^{5}\)) belong to _medical_ and others to _news_.

Additionally, Table 4 suggests domain overlap (label match) in the target and source corpora, where we define 3 types of overlap: \(\mathcal{I}\) (identical) if all labels match, \(\mathcal{R}\) (related) if some labels match, and \(\mathcal{D}\) (distant) if a very few or no labels match. Note, our approaches are completely unsupervised and do not use the data labels (_appendix_).

**Reproducibility**: We follow the experimental setup similar to DocNADE Larochelle and Lauly (2012) and DocNADEe Gupta et al. (2019a), where the number of topics (\(H\)) is set to \(200\). While DocNADEe requires the dimension (i.e., \(E\)) of word embeddings be the same as the latent topic (i.e., \(H\)), we follow _scheme (ii)_ (Algorithm 1) to introduce 

pre-trained word embeddings from Glove, FastText (\(E\)=300) Bojanowski et al. (2017) and BERT-base (\(E\)=768) models. See _appendix_ for the experimental setup, hyperparameters and optimal values of \(\lambda^{k}\in[0.1,0.5,1.0]\) and \(\gamma^{k}\in[0.1,0.01,0.001]\).

**Baselines (Related Works)**: (1) _Topic Models without Transfer Learning_ that learn topics in isolation using the given target corpus only. We employ LDA-based variant, i.e., ProdLDA Srivastava and Sutton (2017) and neural network-based variants, i.e., DocNADE (autoregressive) and NVDM non-autoregressive Miao et al. (2016).

(2) _Topic Models with Transfer Learning_ that leverages pre-trained context-insensitive word embeddings Pennington et al. (2014). We consider topic models based on both LDA, i.e., GaussLDA Das et al. (2015) and glove-GMM Nguyen et al. (2015), and neural networks, i.e., DocNADEe Gupta et al. (2019a). They do not leverage pretrained topic-embeddings (i.e., _GVT_), contextualized word-embedding and MST-MVT techniques.

(3) _Unsupervised Document Representation_ to quantify the quality of document representations. We use 3 strategies: doc2vec Le and Mikolov (2014), EmbSum-Glove and EmbSum-BERT (represent a document by summing the pre-trained embeddings of it's words from Glove and BERT).

(4) _Zero-shot Topic Modeling_ to demonstrate transfer learning capabilities of the proposed framework, where we build (train) a TM using all source corpora and evaluate on the target corpus \(\mathcal{T}\), and

(5) _Data-augmentation_ that first augments the target corpus with all the source corpora and then builds a TM to evaluate transfer learning on \(\mathcal{T}\).

Table 5 summarizes the comparison of this work with the aforementioned baselines. Tables 6 and 7 employ baseline TMs without and with transfer learning, respectively.

### Generalization: Perplexity (PPL)

To evaluate generative performance of DocNADE-based NTM, we compute average held-out perplexity per word: \(\textit{PPL}=\exp\big{(}-\frac{1}{N}\sum_{t=1}^{N}\frac{1}{|\mathbf{v}_{t}|} \log p(\mathbf{v}_{t})\big{)}\), where \(N\) and \(|\mathbf{v}_{t}|\) are the number of documents and words in a document \(\mathbf{v}_{t}\), respectively.

Tables 6 and 7 quantitatively show PPL scores on the five target corpora using one or four sources. In Table 6 using TMN (as a single source) for LVT, GVT and MVT transfer types on the target TMNtitle, we see improved (reduced) PPL scores: (\(655\) vs \(706\)), (\(689\) vs \(706\)) and (\(663\) vs \(706\)) respectively in comparison to DocNADE. We also observe gains due to MST+LVT, MST+GVT and MST+MVT configurations on TMNtitle. Similarly in MST+LVT for R21578title, we observe a gain of 5.2% (182 vs 192), suggesting that multi-source transfer learning using pretrained

\begin{table}
\begin{tabular}{c c r|r||c c c c c|c c c c|c c c} \hline  & \multicolumn{3}{c|}{**KBs from Model**} & \multicolumn{3}{c|}{**Model**} & \multicolumn{6}{c}{**Scores on Target Corpus (in sparse-data and sufficient-data settings)**} \\ \cline{3-13}  & \multicolumn{3}{c|}{**Source**} & \multicolumn{3}{c|}{**or Transfer**} & \multicolumn{3}{c|}{**ZONShort**} & \multicolumn{3}{c|}{**TDUnittle**} & \multicolumn{3}{c|}{**R21578title**} & \multicolumn{3}{c|}{**ZONSmall**} & \multicolumn{3}{c}{**Thm**} \\ \multirow{-2}{*}{**VERT-1**} & \multirow{-2}{*}{**Expens**} & \multirow{-2}{*}{**Type**} & \multicolumn{3}{c|}{_PPL COH_} & \multicolumn{3}{c|}{_IR_} & \multicolumn{3}{c|}{_PPL COH_} & \multicolumn{3}{c|}{_IR_} & \multicolumn{3}{c|}{_PPL COH_} & \multicolumn{3}{c|}{_IR_} & \multicolumn{3}{c|}{_PPL COH_} & \multicolumn{3}{c|}{_IR_} & \multicolumn{3}{c}{_PPL COH_} \\ \hline \multirow{3}{*}{**BSTcontrol**} & **Baseline TM** & NVDM & 1047 & .736 & .076 & 973 & .740 & .190 & 372 & .735 & .271 & 957 & .515 & .090 & 833 & .673 \\  & \multirow{-2}{*}{**without Word-**} & \multirow{-2}{*}{**Webmding**} & \multirow{-2}{*}{**PoolAL**} & 923 & .689 & .062 & 1527 & .744 & .170 & 480 & .742 & .200 & 1181 & .394 & .062 & 1519 & .577 \\  & & & DoNeMB & 646 & .667 & .290 & 706 & .521 & 192 & .713 & .657 & .594 & .466 & .220 & .584 & .636 \\ \hline \hline \multirow{7}{*}{**CIFAR**} & \multirow{-2}{*}{**20NS**} & \multirow{-2}{*}{**SVT**} & **630** & .673 & .298 & .705 & .709 & .523 & 194 & .708 & .656 & .594 & .455 & .288 & .582 & .649 \\  & & & GVT & 646 & .690 & .303 & .718 & .720 & .527 & 184 & .698 & .660 & .594 & .500 & .310 & .590 & .652 \\  & & & MYT & 638 & .690 & **.314** & .714 & .718 & .528 & 188 & .715 & .655 & 600 & .499 & .311 & .588 & .650 \\ \cline{2-13}  & \multirow{-2}{*}{**TMN**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} \\  & & & NYT & 649 & .668 & .294 & .689 & .728 & .555 & 191 & .709 & .660 & .596 & .521 & .276 & - \\  & & & MYT & 658 & .687 & .297 & .663 & .747 & .553 & 195 & .720 & .660 & .599 & .507 & .292 & - \\ \cline{2-13}  & \multirow{-2}{*}{**R21578**} & \multirow{-2}{*}{**RVT**} & \multirow{-2}{*}{**WMT**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} \\  & & & MYT & 656 & .667 & .292 & .704 & .715 & .522 & 186 & .715 & .676 & .593 & .458 & .267 & .581 & .636 \\ \cline{2-13}  & \multirow{-2}{*}{**R21578**} & \multirow{-2}{*}{**RVT**} & \multirow{-2}{*}{**GUT**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} & \multirow{-2}{*}{**FULL**} \\  & & & GGT & 65 

word and topic embeddings (jointly) helps improving TM, and it also verifies domain relatedness (e.g., in TMN-TMNtitle and AGnews-TMN). Similarly, Table 7 reports gains in PPL (e.g., on TNMtitle, R21578title, etc.) compared to the baseline DocNADEe. PPL scores due to BERT can be not computed since its embeddings are aware of both preceding and following contexts.

In Table 8, we show PPL scores on 2 medical target corpora: Ohsumtitle and Ohsumed using 2 sources: AGNews (_news_) and PubMed (_medical_) to perform cross-domain and in-domain transfers. We see that using PubMed for LVT on both the targets improves generalization. Overall, we report a gain of 17.3% (\(1268\) vs \(1534\)) on Ohsumedtitle and 8.55% (\(1497\) vs \(1637\)) on Ohsumed datasets, compared to DocNADEe.

### Interpretabilty: Topic Coherence (COH)

While PPL is used for model selection, Chang et al. (2009) showed in some cases humans preferred TMs (based on the semantic quality of topics) with higher (worse) perplexities. Therefore, we also estimate the quality of topics. We follow Roder et al. (2015) and Gupta et al. (2019a) to compute COH of the top 10 words in each topic. Essentially, the higher scores imply the coherent topics.

Tables 6 and 7 (under COH column) demonstrate that our approaches (GVT, MVT and MST) show noticeable gains and thus improve topic quality. For instance in Table 6, when AGnews is used as a single source for 20NSsmall datatset, we observe a gain in COH due to GVT (.563 vs .462) and MVT (.566 vs .462). Additionally, noticeable gains are reported due to MST+LVT (.542 vs .462), MST+GVT (.585 vs .462) and MST+MVT (.637 vs .462), compared to DocNADE. Importantly, we find a trend MVT\(>\)GVT\(>\)LVT in COH scores for both the single-source and multi-source transfers. Similarly, Table 7 show noticeable gains (e.g., 40.7%, 10.4%, 7.08%, etc.) in COH due to MST+MVT+Glove +FastText+BERT setting. Moreover, Table 8 shows gains in COH due to GVT on Ohsumedtitle and Ohsumed, using pretrained knowledge from PubMed. Overall, the GVT, MVT and MST boost COH for all the five target corpora compared to the baseline TMs (i.e., DocNADE and DocNADEe). The improvements suggest that the approaches scale across domains.

### Applicability: Information Retrieval (IR)

We further evaluate the quality of document representations and perform an IR task using the label information only to compute precision. We follow the experimental setup similar to Gupta et al. (2019a). See the details in _appendix_.

Tables 6 and 7 report precision scores at retrieval fraction \(0.02\) where the configuration MST+MVT outperforms both the DocNADE and DocNADEe for all 4 targets. We observe _large gains_ in precision: (a) Table 6: 20.7% (.326 vs .270) on 20NSsmall, 9.21% (.569 vs .521) on

\begin{table}
\begin{tabular}{c c|r r|r r|r r|r r|r r|r r|r r} \hline \multicolumn{1}{c}{} & \multirow{2}{*}{**KBs from Source Corpus**} & \multicolumn{3}{l|}{**Model**} & \multicolumn{6}{c|}{**Scores on Target Corpus** (in sparse-data and sufficient-data settings)} \\ \cline{4-13}  & & \multirow{2}{*}{**or Transfer**} & \multicolumn{3}{l|}{20NSshort} & \multicolumn{3}{l|}{TDultitle} & \multicolumn{3}{l|}{R21578title} & \multicolumn{3}{l|}{20NSsmall} & \multicolumn{3}{l}{TDMul} \\ \cline{4-13}  & & & & \multicolumn{3}{l|}{TPU} & \multicolumn{1}{c}{_COLH_} & \multicolumn{1}{c}{_IR_} & \multicolumn{1}{c}{_PPL COH_} & \multicolumn{1}{c}{_IR_} & \multicolumn{1}{c}{_PPL COH_} & \multicolumn{1}{c}{_IR_} & \multicolumn{1}{c}{_PPL COH_} & \multicolumn{1}{c}{_IR_} & \multicolumn{1}{c}{_PPL COH_} & \multicolumn{1}{c}{_IR_} & \multicolumn{1}{c}{_PPL COH_} & \multicolumn{1}{c}{_OH_} \\ \cline{3-13}  & & \multicolumn{3}{l|}{docDev} & - & - & .090 & - & - & .190 & - & - & .518 & - & - & .200 & - & - \\  & & EmusMun-Glove & - & - & - & .236 & - & - & .513 & - & - & .587 & - & - & .214 & - & - \\  & & EmbSum-BERT & - & - & .261 & - & - & .499 & - & - & .594 & - & - & .262 & - & - \\  & \multirow{2}{*}{**Baseline**} & TM & Gauss-LDA & - & - & .080 & - & - & .408 & - & - & .367 & - & .090 & - & - \\  & & glove-DMM & - & .512 & .183 & - & .633 & .445 & - & .364 & 273 & - & .578 & .090 & - & .705 \\  & \multirow{2}{*}{_Embeddings_} & \multicolumn{1}{c}{\(\rightarrow\) DocNADEe} & **629** & .674 & .294 & .680 & - & .719 & .540 & 187 & .721 & .663 & **590** & .455 & .274 & 572 & .664 \\ \hline \multirow{11}{*}{**Baselines**} & 20NS & MVT+Glowe & G30 & .721 & .308 & .688 & .741 & .565 & **183** & .724 & .667 & 597 & .561 & **.306** & **570** & **.693** \\  & \multirow{2}{*}{_TRN_} & MTV+GTove & GT040 & .731 & .295 & **673** & .750 & .576 & **184** & .716 & .672 & 599 & .594 & .261 & - & - \\  & \multirow{2}{*}{_R21578_} & MWT+Glove & G33 & .705 & .295 & 689 & .738 & .540 & 185 & .737 & **.691** & 595 & .485 & .255 & .577 & .697 \\  & \multirow{2}{*}{_AGners_} & MVT+Glove & G42 & .734 & .302 & .706 & .748 & .565 & 190 & .734 & .675 & 598 & .573 & .284 & .585 & .703 \\ \cline{1-1} \cline{3-13}  & \multirow{2}{*}{_MSST_} & MVT+Glove & G44 & .739 & .304 & **673** & **.752** & .570 & **183** & .742 & .684 & 598 & .631 & .282 & .582 & .710 \\  & & + FastText & Gest & .741 & .313 & **673** & .751 & .578 & **183** & .744 & .684 & .599 & .634 & .254 & **.711** \\  & \multirow{2}{*}{-} & BERT & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – & – \\ \hline \hline \multicolumn{1}{c}{} & **Gain\% (vs DecNADEe)** & 00.16 & 170.4 & 19.5 & 11.03 & 44.60 & 11.19 & 13.33 & 13.20 & 14.22 & 10.85 & 40.70 & 12.92 & 1.35 & 17.08 \\ \hline \end{tabular}
\end{table}
Table 7: State-of-the-art comparisons against baseline TMs using context-insensitive word embeddings: PPL, COH and IR at retrieval fraction \(0.02\). Scores are reported on each of the target, given the KBs. Here, MVT \(\rightarrow\) LVT+GVT, DocNADEe \(\rightarrow\) DocNADE+Glove, Bold \(\rightarrow\) Best score (in column), Underline \(\rightarrow\) Second best score (in column) and Gain% \(\rightarrow\) Bold vs DocNADEe. For all the configurations, we apply a projection on ([non-]contextualized) word embeddings from several sources, i.e., _scheme (ii)_.

 

TMNtitle, etc., (b) Table 7: 11.9% (.604 vs .540) on TMNtitle and 9.5% (.322 vs .294) on 20NSshort, etc., (c) Table 8: 14.4% (.183 vs .160) on Ohsumedtitle. Additionally, Figures 1(a), 1(b), 1(c) and 1(d) illustrate precision-recall curves on 20NSshort, 20NSsmall, TMNtitle and R21578title respectively, where MST+MVT and MST+GVT consistently outperform the baselines at all fractions.

### Zero/Few-shot and Data-augmentation

Figures 1(a), 1(b), 1(c) and 1(d) show precision in the _zero-shot_ (source-only training) and _data-augmentation_ (source+target training) configurations. Observe that the latter helps in learning meaningful representations and performs better than zero-shot; however, it is outperformed by MST+MVT, suggesting that a naive (data space) augmentation does not add sufficient prior or relevant information to the sparse target. Thus, we find that it is beneficial to augment training data in feature space (e.g., LVT, GVT and MVT) especially for unsupervised topic models using WordPool and TopicPool.

Moreover in the _few-shot_ setting, we first split the training data of TMNtitle into several sets: 20%, 40%, 60%, 80% of the training set and then retrain DocNADE, DocNADEe and DocNADE+MST+MVT on each as a sparse target. We demonstrate transfer learning in such sparse-data settings using the KBs: WordPool and TopicPool jointly. Figure 1(e) plots precision at retrieval fraction \(0.02\) and validates that the proposed modeling consistently outperforms both the baselines: DocNADE and DocNADEe.

Beyond IR, we further investigate computing topic coherence (COH) for the zero-shot and data-augmentation baselines, where the COH scores in Figure 1(f) suggest that MST+MVT outperforms DocNADEe, zero-shot and data-augmentation.

### Topics and Nearest Neighbors (NN)

For topic level inspection, we first extract topics using the rows of **W** of source and target corpora. Table 9 shows the topics (top-5 words) from source and target domains. Observe that the target topics become more coherent after transfer learning (i.e., +GVT) from one or more sources. The blue color signifies that a target topic has imitated certain topic words from the source. We also show a topic (the last) improved due to multi-source transfer.

For word level inspection, we extract word representations using the columns of **W**. Table 10

Figure 2: (a, b, c, d) Retrieval performance (precision@recall) on 4 datasets: 20NSshort, 20NSsmall, TMNtitle and R21578title. (e) Precision at recall fraction \(0.02\), each for a fraction (20%, 40%, 60%, 80%, 100%) of the training set of TMNtitle. (f) Zero-shot and data-augmentation (DA) for COH on TMNtitle and Ohsumed.

 

shows nearest neighbors (NNs) of the word _chip_ in 20NSshort (target) corpus, before (-) and after (+) topic knowledge transfer via GVT using three sources (i.e., MST+GVT). Observe that the NNs in the target become more meaningful by gaining knowledge mainly from 20NS source.

## 4 Conclusion

We have presented a state-of-the-art neural topic modeling framework using multi-view embedding spaces: pretrained topic-embeddings and word-embeddings (context-sensitive and context-insensitive) from one or many sources to improve quality of topics and document representations.

## Acknowledgments

This research was supported by the Federal Ministry for Economic Affairs and Energy (Bundeswritschaftsministerium: bmwi.de), grant 01MD19003E (PLASS: Platform for Analytical Supply Chain Management Services, plass.io) at Siemens AG (Technology- Machine Intelligence), Munich Germany.

## References

* Iz Beltagy et al. [2019] Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JICNLP)_, Hong Kong, China. Association for Computational Linguistics.
* Bengio et al. [2003] Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. _Journal of Machine Learning Research_, 3:1137-1155.
* Blei et al. [2003] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. _Journal of Machine Learning Research_, 3:993-1022.
* Bojanowski et al. [2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _TACL_, 5:135-146.
* Cao et al. [2010] Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. 2010. Adaptive transfer learning. In _Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010_. AAAI Press.
* Chang et al. [2009] Jonathan Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In _Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009, Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada._, pages 288-296.

\begin{table}
\begin{tabular}{c c c|c|c}  & \multicolumn{2}{c|}{source corpora} & \multicolumn{2}{c}{target coppus} \\ \hline
20NS & R21578 & AGnews & \multicolumn{2}{c}{20NSshort} \\  & & & -GVVT & **+GVT** \\ \hline key & chips & chips & virus & chips \\ encrypted & semiconductor & chipmaker & intel & technology \\ encryption & miti & processors & gpsh & intel \\ clipper & makers & semiconductor & crash & encryption \\ keys & semiconductors & intel & chips & clipper \\ \end{tabular}
\end{table}
Table 10: Five nearest neighbors of the word _chip_ in a target and three source semantic spaces before (-) and after (+) transfer via MST+GVT configuration

\begin{table}
\begin{tabular}{c|l l r r r r}
**KBs from** & \multirow{2}{*}{**Model**} & \multicolumn{5}{c}{**Scores on Target Corpus**} \\ \cline{3-7}
**Course** & & \multicolumn{2}{c||}{**or Transfer**} & \multicolumn{2}{c}{Ohsmedtitle} & \multicolumn{2}{c}{Ohsmedtitle} & \multicolumn{2}{c}{Ohsmedtitle} \\

- & .150 &
- & .148 \\  & EmkSum-SciBERT &
- & .160 &
- & .165 \\  & DocNADE & 1534 & .738 & .175 & 1637 & .674 & .183 \\ \hline \hline \multirow{5}{*}{_AGnews_} & LVT & 1587 & .732 & .160 & 1171 & .657 & .184 \\  & GVT & 1529 & .732 & .160 & 1594 & .665 & .185 \\  & MVT & 1528 & .734 & .160 & 1598 & .666 & .184 \\  & + BioEnb & 1488 & .747 & .176 & 1595 & .681 & .187 \\ \hline \multirow{5}{*}{_PubMed Med_} & LVT & **1268** & .732 & .172 & 1535 & .669 & .190 \\  & GVT & 1329 & .740 & .173 & .178 & .671 & **.192** \\  & MVT & 1408 & .743 & .178 & .1514 & .674 & .191 \\  & + BioEnb & 1364 & **.753** & .182 & 1633 & **.689** & .191 \\ \hline \multirow{5}{*}{_MST_} & LVT & **1268** & .733 & .172 & 1536 & .668 & .190 \\  & GVT & 1391 & .740 & .172 & 1504 & .666 & **.192** \\  & MVT & 1399 & .744 & .177 & 1607 & .679 & .191 \\  & Bioch & 1375 & .751 & **1049** & .693 & .190 \\  & + BioFastText & 1350 & **.753** & .178 & 1641 & .688 & .187 \\  & + SciBERT &
- & **.753** & **.183** & . – & 682 & .182 \\ \hline \multirow{5}{*}{_MSI_} & **Gaño (vs DeNADE)** & 4.01 & .343 & .144 & .123 & .148 & .08 & .74 & .35 \\  & **Gaño (vs DeNADE)** & 17.3 & .203 & .146 & .785 & .122 & .24 & .491 \\ \end{tabular}
\end{table}
Table 8: PPL, COH, and IR at fraction 0.02. BioEmb and BioFastText [13]: 200-dimension; SciBERT: Pretrained BERT-variant [14]. + BioEnb: MVT+BioEmb

\begin{table}
\begin{tabular}{c|c|c} \(\mathcal{T}\) & \(\mathcal{S}\) & **Model** & **Topic-words (Top 5)** \\ \hline \multirow{5}{*}{**Baselines**} & 20NS & DecNADE & shipping, sale, prices, expensive, price \\  & -GVT & sale, price, monitor, site, setup \\  & -GVT & shipping, sale, price, expensive, subscriberie \\  & Adnews & DecNADE & microsoft, software, ibn, ima, tinux, computer \\  & -GVT & apple, modem, side, band, perform \\  & +GVT & microsoft, software, desktop, computer, apple \\ \hline \multirow{5}{*}{**Baselines**} & 20NS & DecNADE & miners, earthquake, explosion, stormnet, quake \\  & Titel & DocNADE & tsunami, guate, japan, earthquake, radiation \\  & -GVT & strike, jackson, kill, earthquake, injares \\  & +GVT & earthquake, radiaton, explosion, wildfire \\ \end{tabular}
\end{table}
Table 9: Source \(\mathcal{S}\) and target \(\mathcal{T}\) topics before (-) and after (+) topic transfer (GVT) from one/more source(s) 

Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 795-804. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Gupta et al. (2019a) Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schutze. 2019a. Document informed neural autoregressive topic models with distributional prior. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_.
* Gupta et al. (2019b) Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schutze. 2019b. textTOvec: Deep contextualized neural autoregressive topic models of language with distributed compositional prior. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Gupta et al. (2020) Pankaj Gupta, Yatin Chaudhary, Thomas A. Runkler, and Hinrich Schutze. 2020. Neural topic modeling with continual lifelong learning. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 3907-3917. PMLR.
* Larochelle and Lauly (2012) Hugo Larochelle and Stanislas Lauly. 2012. A neural autoregressive topic model. In _Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems_, pages 2717-2725.
* Larochelle and Murray (2011) Hugo Larochelle and Iain Murray. 2011. The neural autoregressive distribution estimator. In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS_, volume 15 of _JMLR Proceedings_, pages 29-37. JMLR.org.
* Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proceedings of the 31th International Conference on Machine Learning, ICML_, volume 32 of _JMLR Workshop and Conference Proceedings_, pages 1188-1196. JMLR.org.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Namam Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_, volume 48 of _JMLR Workshop and Conference Proceedings_, pages 1727-1736. JMLR.org.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
* Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems_, pages 3111-3119.
* Moen and Sanakoski (2013) SPFGH Moen and Tapio Salakoski2 Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. _Proceedings of LBM_, pages 39-44.
* Nguyen et al. (2015) Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2015. Improving topic models with latent feature word representations. _TACL_, 3:299-313.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543. Association for Computational Linguistics.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Ruder et al. (2015) Michael Ruder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015_, pages 399-408. ACM.
* Salakhutdinov and Hinton (2009) Ruslan Salakhutdinov and Geoffrey E. Hinton. 2009. Replicated softmax: an undirected topic model. In _Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems_, pages 1607-1614. Curran Associates, Inc.
* Srivastava and Sutton (2017) Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In _5th International Conference on Learning Representations, ICLR_ 

Data Description

In order to evaluate knowledge transfer within unsupervised neural topic modeling, we use the following seven datasets in the target domain \({\cal T}\) following the similar experimental setup as in DocNADEe: (1) 20NSshort: We take documents from 20NewsGroups data, with document size (number of words) less than 20. (2) 20NSsmall: We sample 20 document (each having more than \(200\) words) for training from each class of the 20NS dataset. For validation and test, 10 document for each class. Therefore, it is a corpus of few (long) documents. (3) TMNtitle: Titles of the Tag My News (TMN) news dataset. (4) R21578title: Reuters corpus, a collection of new stories from nltk.corpus. We take titles of the documents. (5) Ohsumedtitle: Titles of Ohsumed abstracts. Source: disi.unitn.it/moschitti/corpora.htm. (6) Ohsumed: Ohsumed dataset, collection of medical abstracts. Source: disi.unitn.it/moschitti/corpora.htm. (7) TMN: The Tag My News (TMN) news dataset.

To **prepare knowledge base** of word embeddings (local semantics) and latent topics (global semantics) features, we use the following six datasets in the source \({\cal S}\): (1) 20NS: 20NewsGroups corpus, a collection of news stories from nltk.corpus. (2) TMN: The Tag My News (TMN) news dataset. (3) R21578: Reuters corpus, a collection of new stories from nltk.corpus. (4) AGnews: AGnews data selection. PubMed: Medical abstracts of randomized controlled trials. Source: [https://github.com/Franck-Dernoncourt/pubmed-rct](https://github.com/Franck-Dernoncourt/pubmed-rct).

See Table 3 (in paper content) describes each of the datasets, where a short-text refers to a text document having less than 15 words. Notice that each of the datasets in the target and source domains, we see overlap in their label spaces. See Table 4 for the label information for each of the source and target corpora. Additionally in supplementary, we have also provided the code and pre-processed datasets used in our experiments.

## Appendix B Getting Word and Latent Topic Representations from Source(s)

Since in DocNADE, the column of \({\bf W}_{;v_{i}}\) gives a word vector of the word \(v_{i}\), therefore the dimension of word embeddings in each of the \({\bf E}^{k}\) is same (i.e., \(H=200\)). Thus, we prepare the knowledge base of word representations \({\bf E}^{k}\) from \(k\)th source using DocNADE, where each word vector is of \(H=200\) dimension.

Since the row vector of \({\bf W}_{j,:}\) in DocNADE encodes \(j\)th topic feature, therefore each latent topic (i.e., row) in feature matrix \({\bf W}\) is a vector of \(K\) dimension, corresponding the definition of topics that it is a distribution over vocabulary. \(H\) is the number of latent topics and \(K\) is the vocabulary size, where \(K\) varies across corpora. Thus, we train DocNADE to learn a feature matrix specific to each of the source corpora, e.g. \({\bf W}^{k}\in\mathbb{R}^{H\times K}\) of \(k\)th source.

For a target corpus of vocabulary size \(K^{{}^{\prime}}\), the DocNADE learns a feature matrix \({\bf W}^{{\cal T}}\in\mathbb{R}^{H\times K^{\prime}}\). Similarly, \({\bf W}^{k}\in\mathbb{R}^{H\times K}\) for \(k\)th source of vocabulary size \(K\). Since in the sparse-data setting for the target, \(K^{\prime}<<K\) due to additional word in the source. To perform GVT, we need the same topic feature dimensions in the target and source, i.e., \(K^{\prime}\) of the target. Therefore, we remove those column vectors from \({\bf W}^{k}\in\mathbb{R}^{H\times K}\) of the \(k\)th source for which there is no corresponding word in the vocabulary of the target domain. As a result, we obtain \({\bf Z}^{k}\) as a latent topic feature matrix to be used in knowledge transfer to the target domain. Following the similar steps, we prepare a KB of \({\bf Z}\)s such that each latent topic feature matrix from a source domain gets the same topic feature dimension as the target.

## Appendix C Experimental Setup

For DocNADE and DocNADEe in different knowledge transfer configurations, we follow the same experimental setup as in DocNADE and DocNADEe. We rerun DocNADE and DocNADEe using the code released for DocNADEe. For all the hyperpa

\begin{table}
\begin{tabular}{c|l} \hline \multicolumn{1}{p{19.5pt}}{**data**} \\ \hline  & world, us, sport, business, sct, jcbnt, entertainment, health \\ \hline Adnews & business, sct, jcbnt, sports, world \\ \hline \multirow{3}{*}{20NSShort: 20NSSmall,} & misc:forual, comp graphics, rec, matuons, comp windows, x, recs. spot, baseball, sci space, recs. spot hockey, socre religiousite, mcnotropics, comp sys.mac hardware, tuk relx.region mixse, sct electronics, comp on www-windows.mixse, sct.med, compys.psm, bnd rharware, talk politics, multicast, mld, poliotis, smas, split conflicts, mk, alnk politics, snippet \\ \cline{2-2}  & trade, gain, curke, cross, tricks, wibber, squat, pull-off, vrgc-ql, align, coffee, wheat gold, acq, intercept, money-fc, cares, liveness, abiced, ojsement, gen, pops, lead, nice, gwep, soy-old, pllr, nickel, pornphant, hard, sorghum, unsured, coco, arpesged, cotton, memory-supply, intoc-stell, palladium, platinum, simple, untignet, renciess, pygmatout, llf-oil, incel-feed, san-meal, sun-dl, bag, barley, potato, orange, soybean-mel, conton-oil, fuel, silver, income, uplv, lex, loccount, coconut, coom-oil, copya-clea, propane, instad-dbnl, hombie, nizry, crato-oi, palramolecular, tin, copper, cyti pet-chem, rage-oil, out napbha, cpu, read, alarm \\ \hline \end{tabular}
\end{table}
Table 11: Label space of the corpora. TMN*TMN or TMNtitle 

rameters, optimal values are selected based on the performance on development set.

### Experimental Setup for Generalization

We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with _sigmoid_ hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in PPL for \(H=200\) topics than using \(50\), therefore we use \(H=200\) in our experiments. See Table 12 for hyperparameters used in generalization task, i.e., computing PPL.

### Experimental Setup for IR Task

We treat all test documents as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity between their document vectors. To compute retrieval precision for each fraction (e.g., \(0.02\)), we average the number of retrieved training documents with the same label as the query.

We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with _tanh_ hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in precision for the retrieval task for \(H=200\) topics than using \(50\), therefore we use \(H=200\) in our experiments. We follow the similar experimental setup as in DocNADEe. For model selection, we used the validation set as the query set and used the average precision at 0.02 retrieved documents as the performance measure. Note that the labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same train/development/test split of documents as for PPL setup.

Given DocNADE, the representation of a document of size \(D\) can be computed by taking the last hidden vector \({\bf h}_{D}\) at the autoregressive step \(D\). Since, the RSM and DocNADE strictly outperformed LDA, therefore we only compare DocNADE and its recent extension DocNADEe. We use the same number of topic dimensions (\(H=200\)) across all the source and target in training DocNADE.

See Table 13 for the hyperparameters in the document retrieval task, where \(\lambda^{k}\) and \(\gamma^{k}\) are weights for \(k\)th source. We use the same grid-search for all the source domains. We set \(\gamma^{k}\) smaller than \(\lambda^{k}\) to control the degree of imitation of the source domain(s) by the target domain. We use the development set of the target corpus to find the optimal setting in different configurations of knowledge transfers from several sources.

### **{\(\boldsymbol{\lambda}\), \(\boldsymbol{\gamma}\)****as Parameter vs Hyperparameters**

Here, we treat \(\lambda\) and \(\gamma\) as parameters of the model, instead of hyperparameters and learn them with backpropagation. We initialize each \(\lambda^{k}\) = \(0.5\) and \(\gamma^{k}\) = \(0.01\) for each of the sources. We perform experiments on short-text datasets in MST+LVT, MST+GVT and MST+MVT configurations. We evaluate the topic modeling using PPL, topic coherence and retrieval accuracy. Table 14 reports

\begin{table}
\begin{tabular}{c|c} \hline
**Hyperparameter** & **Search Space** \\ \hline retrieval fraction & [0.02] \\ learning rate & [0.001] \\ hidden units, \(H\) & [200] \\ activation function (\(g\)) & sigmoid \\ iterations & [100] \\ \(\lambda^{k}\) & [1.0, 0.5, 0.1] \\ \(\gamma^{k}\) & [0.1, 0.01, 0.001] \\ \hline \end{tabular}
\end{table}
Table 13: Hyperparameters search in the IR task, where \(\lambda^{k}\) and \(\gamma^{k}\) are weights for \(k\)th source.

 

the scores, when \(\lambda\) and \(\gamma\) are (1) learned with back-propagation, and (2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Thus, we have reported scores considering {\(\lambda\), \(\gamma\)} as hyper-parameters.

### Reproducibility: Optimal Configurations of \(\lambda\) and \(\gamma\)

As mentioned in Tables 12 and 13, the hyperparameter \(\lambda^{k}\) takes on values in [1.0, 0.5, 0.1] for each of the word embeddings matrix \(\mathbf{E}^{k}\) and \(\gamma^{k}\) in [0.1, 0.01, 0.001] for each of the latent topic features \(\mathbf{Z}^{k}\), respectively for the \(k^{th}\) source domain. To determine an optimal configuration, we perform grid-search over the values and use the scores on the development set to determine the best setting. We have a common model for PPL and COH scores due to generalization.

To **reproduce** scores (best/bold in Table 5, we mentioned the best settings of (\(\lambda^{k}\), \(\gamma^{k}\)) in MST+MVT configuration for each of the target and source combinations:

1. Generalization (PPL and COH) in MST+MVT when **target** is 20NSshort: (\(\lambda^{20NS}=1.0\), \(\gamma^{20NS}=0.001\), \(\lambda^{TMN}=0.1\), \(\gamma^{TMN}=0.001\), \(\lambda^{R21578}=0.5\), \(\gamma^{R21578}=0.001\), \(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\)
2. Generalization (PPL and COH) in MST+MVT when **target** is 621578title: (\(\lambda^{20NS}=0.1\), \(\gamma^{20NS}=0.001\), \(\lambda^{TMN}=1.0\), \(\gamma^{RMN}=0.001\), \(\lambda^{R21578}=1.0\), \(\gamma^{R21578}=0.001\), \(\lambda^{AGnews}=1.0\), \(\gamma^{AGnews}=0.001\)
3. Generalization (PPL and COH) in MST+MVT when **target** is 621578title: (\(\lambda^{20NS}=0.1\), \(\gamma^{20NS}=0.001\), \(\lambda^{TMN}=0.5\), \(\gamma^{TMN}=0.001\), \(\lambda^{R21578}=1.0\), \(\gamma^{R21578}=0.001\), \(\lambda^{AGnews}=1.0\), \(\gamma^{AGnews}=0.001\)
4. Generalization (PPL and COH) in MST+MVT when **target** is 20NSsmall: (\(\lambda^{20NS}=0.5\), \(\gamma^{20NS}=0.001\), \(\lambda^{TMN}=0.1\), \(\gamma^{TMN}=0.001\), \(\lambda^{R21578}=0.1\), \(\gamma^{R21578}=0.001\), \(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\)
5. Generalization (PPL and COH) in MST+MVT when **target** is 20NSsmall: (\(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\), \(\lambda^{PubMeded}=1.0\), \(\gamma^{PubMeded}=0.001\)
6. Generalization (PPL and COH) in MST+MVT when **target** is Ohsumed: (\(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\), \(\lambda^{PubMeded}=1.0\), \(\gamma^{PubMeded}=0.001\)
7. IR in MST+MVT when **target** is 20NSshort: (\(\lambda^{20NS}=1.0\), \(\gamma^{20NS}=0.1\), \(\lambda^{TMN}=0.5\), \(\gamma^{TMN}=0.01\), \(\lambda^{R21578}=0.1\), \(\gamma^{R21578}=0.01\), \(\gamma^{R21578}=0.01\), \(\lambda^{AGnews}=0.5\), \(\gamma^{AGnews}=0.001\)
9. IR in MST+MVT when **target** is R21578title: (\(\lambda^{20NS}=0.1\), \(\gamma^{20NS}=0.01\), \(\lambda^{TMN}=1.0\), \(\gamma^{TMN}=0.01\), \(\lambda^{RI21578}=1.0\), \(\gamma^{R21578}=0.01\), \(\lambda^{AGnews}=0.5\), \(\gamma^{AGnews}=0.001\)
10. IR in MST+GVT when **target** is 20NSsmall: (\(\gamma^{20NS}=0.01\), \(\gamma^{TMN}=0.01\), \(\gamma^{RQ21578}=0.1\), \(\gamma^{AGnews}=0.01\)
11. IR in MST+MVT when **target** is 20NSsmall: (\(\gamma^{20NS}=0.01\), \(\gamma^{TMN}=0.01\), \(\gamma^{R21578}=0.1\), \(\gamma^{AGnews}=0.01\)
11. IR in MST+MVT when **target** is 20NSsmall: (\(\lambda^{20NS}=0.01\), \(\gamma^{TMN}=0.01\), \(\gamma^{R21578}=0.1\), \(\gamma^{AGnews}=0.01\)
12. IR in MST+MVT when **target** is (\(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\), \(\lambda^{PubMeded}=1.0\), \(\gamma^{ PubMed}=0.1\)
12. IR in MST+MVT when **target** is Ohsumed: (\(\lambda^{AGnews}=0.1\), \(\gamma^{AGnews}=0.001\), \(\lambda^{PubMeded}=0.5\), \(\gamma^{PubMed}=0.1\)

The hyper-parameters mentioned above also applies to a single source transfer configuration