SlowGuess/NAACL_2021_OCR huggingface dataset:
KBs from Source Corpus	Model or Transfer Type	Scores on Target Corpus (in sparse-data and sufficient-data settings)
20NSshort	TMNtitle	R21578title	20NSsmall	TMN
PPL	COH	IR	PPL	COH	IR	PPL	COH	IR	PPL	COH	IR	PPL	COH
Baselines	Baseline TM without Word-Embeddings	NVDM	1047	.736	.076	973	.740	.190	372	.735	.271	957	.515	.090	833	.673
ProldLA	923	.689	.062	1527	.744	.170	480	.742	.200	1181	.394	.062	1519	.577
DocNADE	646	.667	.290	706	.709	.521	192	.713	.657	594	.462	.270	584	.636
Proposed	20NS	LVT	630	.673	.298	705	.709	.523	194	.708	.656	594	.455	.288	582	.649
GVT	646	.690	.303	718	.720	.527	184	.698	.660	594	.500	.310	590	.652
MVT	638	.690	.314	714	.718	.528	188	.715	.655	600	.499	.311	588	.650
TMN	LVT	649	.668	.296	655	.731	.548	187	.703	.659	593	.460	.273	-	-
GVT	661	.692	.294	689	.728	.555	191	.709	.660	596	.521	.276	-	-
MVT	658	.687	.297	663	.747	.553	195	.720	.660	599	.507	.292	-	-
R21578	LVT	656	.667	.292	704	.715	.522	186	.715	.676	593	.458	.267	581	.636
GVT	654	.672	.293	716	.719	.526	194	.706	.672	595	.485	.279	591	.646
MVT	650	.670	.296	716	.720	.528	194	.724	.676	599	.490	.280	589	.650
AGnews	LVT	650	.677	.297	682	.723	.533	185	.710	.659	592	.458	.260	564	.668
GVT	667	.695	.300	728	.735	.534	190	.717	.663	598	.563	.282	601	.684
MVT	659	.696	.290	718	.740	.533	189	.727	.659	599	.566	.279	592	.686
MST	LVT	640	.678	.308	663	.732	.547	182	.739	.673	594	.542	.277	568	.674
GVT	658	.705	.305	704	.746	.550	192	.727	.673	599	.585	.326	602	.680
MVT	656	.740	.314	680	.752	.569	188	.745	.685	600	.637	.285	600	.690
Gain% (vs DocNADE)	↑2.48	↑10.9	↑8.28	↑7.22	↑6.06	↑9.21	↑5.20	↑4.49	↑4.26	↑0.34	↑37.9	↑20.7	↑3.42	↑8.50
Table 6: State-of-the-art comparisons with TMs: Perplexity (PPL), topic coherence (COH) and precision@recall (IR) at retrieval fraction 0.02. Scores reported on each of the target, given KBs from several sources. LVT and GVT employ WordPool and TopicPool, respectively. MVT employs both. LVT+MST scores using scheme (i). Here, Bold  $\rightarrow$  Best score (in column) and Gain%  $\rightarrow$  Bold vs DocNADE.
pre-trained word embeddings from Glove, FastText  $(E = 300)$  (Bojanowski et al., 2017) and BERT-base  $(E = 768)$  models. See appendix for the experimental setup, hyperparameters and optimal values of  $\lambda^k\in [0.1,0.5,1.0]$  and  $\gamma^k\in [0.1,0.01,0.001]$ .
Baselines (Related Works): (1) Topic Models without Transfer Learning that learn topics in isolation using the given target corpus only. We employ LDA-based variant, i.e., ProdLDA (Srivastava and Sutton, 2017) and neural network-based variants, i.e., DocNADE (autoregressive) and NVDM (non-autoregressive) (Miao et al., 2016).
(2) Topic Models with Transfer Learning that leverages pre-trained context-insensitive word embeddings (Pennington et al., 2014). We consider topic models based on both LDA, i.e., GaussLDA (Das et al., 2015) and glove-GMM (Nguyen et al., 2015), and neural networks, i.e., DocNADEe (Gupta et al., 2019a). They do not leverage pretrained topic-embeddings (i.e., GVT), contextualized word-embedding and MST-MVT techniques.
(3) Unsupervised Document Representation to quantify the quality of document representations. We use 3 strategies: doc2vec (Le and Mikolov, 2014), EmbSum-Glove and EmbSum-BERT (represent a document by summing the pre-trained embeddings of it's words from Glove and BERT).
(4) Zero-shot Topic Modeling to demonstrate
transfer learning capabilities of the proposed framework, where we build (train) a TM using all source corpora and evaluate on the target corpus  $\mathcal{T}$ , and
(5) Data-augmentation that first augments the target corpus with all the source corpora and then builds a TM to evaluate transfer learning on  $\mathcal{T}$ .
Table 5 summarizes the comparison of this work with the aforementioned baselines. Tables 6 and 7 employ baseline TMs without and with transfer learning, respectively.
3.1 Generalization: Perplexity (PPL)
To evaluate generative performance of DocNADE-based NTM, we compute average held-out perplexity per word:  $PPL = \exp \left(-\frac{1}{N}\sum_{t=1}^{N}\frac{1}{|\mathbf{v}_t|}\log p(\mathbf{v}_t)\right)$ , where  $N$  and  $|\mathbf{v}_t|$  are the number of documents and words in a document  $\mathbf{v}_t$ , respectively.
Tables 6 and 7 quantitatively show PPL scores on the five target corpora using one or four sources. In Table 6 using TMN (as a single source) for LVT, GVT and MVT transfer types on the target TMN title, we see improved (reduced) PPL scores: (655 vs 706), (689 vs 706) and (663 vs 706) respectively in comparison to DocNADE. We also observe gains due to MST+LVT, MST+GVT and MST+MVT configurations on TMN title. Similarly in MST+LVT for R21578 title, we observe a gain of  $5.2\%$  (182 vs 192), suggesting that multi-source transfer learning using pretrained
KBs from Source Corpus	Model or Transfer Type	Scores on Target Corpus (in sparse-data and sufficient-data settings)
20NSshort	TMNtitle	R21578title	20NSsmall	TMN
PPL	COH	IR	PPL	COH	IR	PPL	COH	IR	PPL	COH	IR	PPL COH
Baselines	doc2vec	-	-	.090	-	-	.190	-	-	.518	-	-	.200	-	-
EmbSum-Glove	-	-	.236	-	-	.513	-	-	.587	-	-	.214	-	-
EmbSum-BERT	-	-	.261	-	-	.499	-	-	.594	-	-	.262	-	-
Baseline TM with Word-Embeddings	Gauss-LDA	-	-	.080	-	-	.408	-	-	.367	-	-	.090	-	-
glove-DDM	-	.512	.183	-	.633	.445	-	.364	.273	-	.578	.090	-	.705
→ DocNADEe	629	.674	.294	680	.719	.540	187	.721	.663	590	.455	.274	572	.664
Proposed	20NS	MVT+Glove	630	.721	.320	688	.741	.565	183	.724	.667	597	.561	.306	570	.693
TMN	MVT+Glove	640	.731	.295	673	.750	.576	184	.716	.672	599	.594	.261	-	-
R21578	MVT+Glove	633	.705	.295	689	.738	.540	185	.737	.691	595	.485	.255	577	.697
AGnews	MVT+Glove	642	.734	.302	706	.748	.565	190	.734	.675	598	.573	.284	585	.703
MST	MVT+Glove	644	.739	.304	673	.752	.570	183	.742	.684	598	.631	.282	582	.710
+ FastText	654	.741	.313	673	.751	.578	183	.744	.684	599	.634	.254	582	.711
+ BERT	-	.744	.322	-	.752	.604	-	.745	.680	-	.640	.282	-	.709
Gain% (vs DocNADEe)	↓0.16	↑10.4	↑9.5	↑1.03	↑4.60	↑11.9	↑3.33	↑3.20	↑4.22	↓0.85	↑40.7	↑2.92	↑3.5	↑7.08
Table 7: State-of-the-art comparisons against baseline TMs using context-insensitive word embeddings: PPL, COH and IR at retrieval fraction 0.02. Scores are reported on each of the target, given the KBs. Here, MVT  $\rightarrow$  LVT+GVT, DocNADEe  $\rightarrow$  DocNADE+Glove, Bold  $\rightarrow$  Best score (in column), Underline  $\rightarrow$  Second best score (in column) and Gain%  $\rightarrow$  Bold vs DocNADEe. For all the configurations, we apply a projection on ([non]-contextualized) word embeddings from several sources, i.e., scheme (ii).
word and topic embeddings (jointly) helps improving TM, and it also verifies domain relatedness (e.g., in TMN-TMNTitle and AGnews-TMN). Similarly, Table 7 reports gains in PPL (e.g., on TMNTitle, R21578title, etc.) compared to the baseline DocNADEe. PPL scores due to BERT can be not computed since its embeddings are aware of both preceding and following contexts.
In Table 8, we show PPL scores on 2 medical target corpora: Ohsumtitle and Ohsumed using 2 sources: AGnews (news) and PubMed (medical) to perform cross-domain and in-domain transfers. We see that using PubMed for LVT on both the targets improves generalization. Overall, we report a gain of  $17.3\%$  (1268 vs 1534) on Ohsumed title and  $8.55\%$  (1497 vs 1637) on Ohsumed datasets, compared to DocNADEe.
3.2 Interpretability: Topic Coherence (COH)
While PPL is used for model selection, Chang et al. (2009) showed in some cases humans preferred TMs (based on the semantic quality of topics) with higher (worse) perplexities. Therefore, we also estimate the quality of topics. We follow Röder et al. (2015) and Gupta et al. (2019a) to compute COH of the top 10 words in each topic. Essentially, the higher scores imply the coherent topics.
Tables 6 and 7 (under COH column) demonstrate that our approaches (GVT, MVT and MST) show noticeable gains and thus improve topic quality. For instance in Table 6, when AGnews is used as
a single source for 20NSsmall dataset, we observe a gain in COH due to GVT (.563 vs .462) and MVT (.566 vs .462). Additionally, noticeable gains are reported due to MST+LVT (.542 vs .462), MST+GVT (.585 vs .462) and MST+MVT (.637 vs .462), compared to DocNADE. Importantly, we find a trend MVT>GVT>LVT in COH scores for both the single-source and multi-source transfers. Similarly, Table 7 show noticeable gains (e.g.,  $40.7\%$ ,  $10.4\%$ ,  $7.08\%$ , etc.) in COH due to MST+MVT+Glove +FastText+BERT setting. Moreover, Table 8 shows gains in COH due to GVT on Ohsumedtitle and Ohsumed, using pretrained knowledge from PubMed. Overall, the GVT, MVT and MST boost COH for all the five target corpora compared to the baseline TMs (i.e., DocNADE and DocNADEe). The improvements suggest that the approaches scale across domains.
3.3 Applicability: Information Retrieval (IR)
We further evaluate the quality of document representations and perform an IR task using the label information only to compute precision. We follow the experimental setup similar to Gupta et al. (2019a). See the details in appendix.
Tables 6 and 7 report precision scores at retrieval fraction 0.02 where the configuration MST+MVT outperforms both the DocNADE and DocNADEe for all 4 targets. We observe large gains in precision: (a) Table 6:  $20.7\%$  (.326 vs .270) on 20NSsmall,  $9.21\%$  (.569 vs .521) on
(b) IR: 20NSsmall
(a) IR: 20NSshort
(d) IR: R21578title
Figure 2: (a, b, c, d) Retrieval performance (precision@recall) on 4 datasets: 20NSshort, 20NSsmall, TMNtitle and R21578title. (e) Precision at recall fraction 0.02, each for a fraction  $(20\%, 40\%, 60\%, 80\%, 100\%)$  of the training set of TMNtitle. (f) Zero-shot and data-augmentation (DA) for COH on TMNtitle and Ohsumed.
(e) IR: TMNtitle
(c) IR: TMNtitle
(f) COH: Zero-shot and DA
TMNtitle, etc., (b) Table 7:  $11.9\%$  (.604 vs .540) on TMNtitle and  $9.5\%$  (.322 vs .294) on 20NSshort, etc., (c) Table 8:  $14.4\%$  (.183 vs .160) on Ohsumedtitle. Additionally, Figures 2a, 2b, 2c and 2d illustrate precision-recall curves on 20NSshort, 20NSsmall, TMNtitle and R21578title respectively, where MST+MVT and MST+GVT consistently outperform the baselines at all fractions.
3.4 Zero/Few-shot and Data-augmentation
Figures 2a, 2b, 2c and 2d show precision in the zero-shot (source-only training) and data-augmentation (source+target training) configurations. Observe that the latter helps in learning meaningful representations and performs better than zero-shot; however, it is outperformed by MST+MVT, suggesting that a naive (data space) augmentation does not add sufficient prior or relevant information to the sparse target. Thus, we find that it is beneficial to augment training data in feature space (e.g., LVT, GVT and MVT) especially for unsupervised topic models using WordPool and TopicPool.
Moreover in the few-shot setting, we first split the training data of TMN title into several sets:  $20\%$ ,  $40\%$ ,  $60\%$ ,  $80\%$  of the training set and
then retrain DocNADE, DocNADEe and Doc-NADE+MST+MVT on each as a sparse target. We demonstrate transfer learning in such sparsedata settings using the KBs:WordPool and TopicPool jointly. Figure 2e plots precision at retrieval fraction 0.02 and validates that the proposed modeling consistently outperforms both the baselines:DocNADE and DocNADEe.
Beyond IR, we further investigate computing topic coherence (COH) for the zero-shot and data-augmentation baselines, where the COH scores in Figure 2f suggest that MST+MVT outperforms DocNADEe, zero-shot and data-augmentation.
3.5 Topics and Nearest Neighbors (NN)
For topic level inspection, we first extract topics using the rows of  $\mathbf{W}$  of source and target corpora. Table 9 shows the topics (top-5 words) from source and target domains. Observe that the target topics become more coherent after transfer learning (i.e., +GVT) from one or more sources. The blue color signifies that a target topic has imitated certain topic words from the source. We also show a topic (the last) improved due to multi-source transfer.
For word level inspection, we extract word representations using the columns of  $\mathbf{W}$ . Table 10
KBs from Source Corpus	Model or Transfer Type	Scores on Target Corpus
Ohsumeditedtitle	Ohsumed
PPL	COH	IR	PPL	COH	IR
benchlines	ProdLDA	1121	.734	.080	1677	.646	.080
DocNADE	1321	.728	.160	1706	.662	.184
EmbSum-BioEmb	-	-	.150	-	-	.148
EmbSum-SciBERT	-	-	.160	-	-	.165
DocNADEe	1534	.738	.175	1637	.674	.183
AGnews	LVT	1587	.732	.160	1717	.657	.184
GVT	1529	.732	.160	1594	.665	.185
MVT	1528	.734	.160	1598	.666	.184
+ BioEmb	1488	.747	.176	1595	.681	.187
PubMed	LVT	1268	.732	.172	1535	.669	.190
GVT	1392	.740	.173	1718	.671	.192
MVT	1408	.743	.178	1514	.674	.191
+ BioEmb	1364	.753	.182	1633	.689	.191
MST	LVT	1268	.733	.172	1536	.668	.190
GVT	1391	.740	.172	1504	.666	.192
MVT	1399	.744	.177	1607	.679	.191
+ BioEmb	1375	.751	.180	1497	.693	.190
+ BioFastText	1350	.753	.178	1641	.688	.187
+ SciBERT	-	.753	.183	-	.682	.182
Gain% (vs DocNADE)	↑4.01	↑3.43	↑14.4	↑12.3	↑4.08	↑4.35
Gain% (vs DocNADEe)	↑17.3	↑2.03	↑4.60	↑8.55	↑2.22	↑4.91
Table 8: PPL, COH, and IR at fraction 0.02. BioEmb and BioFastText (Moen and Ananiadou, 2013): 200-dimension; SciBERT: Pretrained BERT-variant (Beltagy et al., 2019). + BioEmb: MVT+BioEmb
T	S	Model	Topic-words (Top 5)
20NShort	20NS	DocNADE	shipping, sale, prices, expensive, price
-GVT	sale, price, monitor, site, setup
+GVT	shipping, sale, price, expensive, subscribe
AGnews	DocNADE	microsoft, software, ibm, linux, computer
-GVT	apple, modem, side, baud, perform
+GVT	microsoft, software, desktop, computer, apple
TMNtitle	AGnews	DocNADE	miners, earthquake, explosion, stormed, quake
TMN	DocNADE	tsunami, quake, japan, earthquake, radiation
-GVT	strike, jackson, kill, earthquake, injures
+GVT	earthquake, radiation, explosion, wildfire
shows nearest neighbors (NNs) of the word chip in 20NSshort (target) corpus, before (-) and after (+) topic knowledge transfer via GVT using three sources (i.e., MST+GVT). Observe that the NNs in the target become more meaningful by gaining knowledge mainly from 20NS source.
We have presented a state-of-the-art neural topic modeling framework using multi-view embedding spaces: pretrained topic-embeddings and word-embeddings (context-sensitive and context-insensitive) from one or many sources to improve quality of topics and document representations.
Table 9: Source  $\mathcal{S}$  and target  $\mathcal{T}$  topics before (-) and after (+) topic transfer (GVT) from one/more source(s)
source corpora	target corpus
20NS	R21578	AGnews	20NSshort
-GVT	+GVT
key	chips	chips	virus	chips
encrypted	semiconductor	chipmaker	intel	technology
encryption	miti	processors	gosh	intel
clipper	makers	semiconductor	crash	encryption
keys	semiconductors	intel	chips	clipper
Table 10: Five nearest neighbors of the word chip in a target and three source semantic spaces before (-) and after (+) transfer via MST+GVT configuration
This research was supported by the Federal Ministry for Economic Affairs and Energy (Bundeswirtschaftsministerium: bmwi.de), grant 01MD19003E (PLASS: Platform for Analytical Supply Chain Mangement Services, plass.io) at Siemens AG (Technology- Machine Intelligence), Munich Germany.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. 2010. Adaptive transfer learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press.
Jonathan Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada., pages 288-296.
Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 795-804. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019a. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019b. textTOvec: Deep contextualized neural autoregressive topic models of language with distributed compositional prior. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Pankaj Gupta, Yatin Chaudhary, Thomas A. Runkler, and Hinrich Schütze. 2020. Neural topic modeling with continual lifelong learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3907-3917. PMLR.
Hugo Larochelle and Stanislas Lauly. 2012. A neural autoregressive topic model. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems, pages 2717-2725.
Hugo Larochelle and Iain Murray. 2011. The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS, volume 15 of JMLR Proceedings, pages 29-37. JMLR.org.
Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning, ICML, volume 32 of JMLR Workshop and Conference Proceedings, pages 1188-1196. JMLR.org.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1727-1736. JMLR.org.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems, pages 3111-3119.
SPFGH Moen and Tapio Salakoski2 Sophia Aniadou. 2013. Distributional semantics resources for biomedical text processing. Proceedings of LBM, pages 39-44.
Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2015. Improving topic models with latent feature word representations. TACL, 3:299-313.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237. Association for Computational Linguistics.
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pages 399-408. ACM.
Ruslan Salakhutdinov and Geoffrey E. Hinton. 2009. Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems, pages 1607-1614. Curran Associates, Inc.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, ICLR.
A Data Description
In order to evaluate knowledge transfer within unsupervised neural topic modeling, we use the following seven datasets in the target domain  $\mathcal{T}$  following the similar experimental setup as in DocNADEe: (1) 20NSshort: We take documents from 20NewsGroups data, with document size (number of words) less than 20. (2) 20NSsmall: We sample 20 document (each having more than 200 words) for training from each class of the 20NS dataset. For validation and test, 10 document for each class. Therefore, it is a corpus of few (long) documents. (3) TMNtitle: Titles of the Tag My News (TMN) news dataset. (4) R21578title: Reuters corpus, a collection of new stories from nltk.corpus. We take titles of the documents. (5) Ohsumedtitle: Titles of Ohsumed abstracts. Source: disi-unitn.it/moschitti/corpora.htm. (6) Ohsumed: Ohsumed dataset, collection of medical abstracts. Source: disi-unitn.it/moschitti/corpora.htm. (7) TMN: The Tag My News (TMN) news dataset.
To prepare knowledge base of word embeddings (local semantics) and latent topics (global semantics) features, we use the following six datasets in the source  $S$ : (1) 20NS: 20News-Groups corpus, a collection of news stories from nltk.corpus. (2) TMN: The Tag My News (TMN) news dataset. (3) R21578: Reuters corpus, a collection of new stories from nltk.corpus. (4) AGnews: AGnews data Sellection. PubMed: Medical abstracts of randomized controlled trials. Source: https://github.com/Franck-Dernoncourt/pubmed-rct.
See Table 3 (in paper content) describes each of the datasets, where a short-text refers to a text document having less than 15 words. Notice that each of the datasets in the target and source domains, we see overlap in their label spaces. See Table 4 for the label information for each of the source and target corpora. Additionally in supplementary, we have also provided the code and pre-processed datasets used in our experiments.
B Getting Word and Latent Topic Representations from Source(s)
Since in DocNADE, the column of  $\mathbf{W}_{:,v_i}$  gives a word vector of the word  $v_{i}$ , therefore the dimension of word embeddings in each of the  $\mathbf{E}^k$  is same (i.e.,  $H = 200$ ). Thus, we prepare the knowledge base of word representations  $\mathbf{E}^k$  from  $k$ th source using