SlowGuess/NAACL_2021_OCR huggingface dataset:
*Pankaj Gupta<sup>1</sup>, *Yatin Chaudhary<sup>1,2</sup>, Hinrich Schütze<sup>2</sup>
$^{1}$ DRIMCo GmbH Munich, Germany
$^{2}$ CIS, University of Munich (LMU) Munich, Germany
info@drimco.net|lyatin.chaudhary@drimco.net
Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity in short-text or small collection of documents. This work presents a novel neural topic modeling framework using multi-view embedding spaces: (1) pretrained topic-embeddings, and (2) pretrained word-embeddings (context-insensitive from Glove and context-sensitive from BERT models) jointly from one or many sources to improve topic quality and better deal with polysemy. In doing so, we first build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). We then identify one or more relevant source domain(s) and transfer knowledge to guide meaningful learning in the sparse target domain. Within neural topic modeling, we quantify the quality of topics and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. Introducing the multi-source multi-view embedding spaces, we have shown state-of-the-art neural topic modeling using 6 source (high-resource) and 5 target (low-resource) corpora.
Probabilistic topic models, such as LDA (Blei et al., 2003), Replicated Softmax (RSM) (Salakhutdinov and Hinton, 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) (Larochelle and Lauly, 2012) are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks, such as information retrieval (IR). Though they have been shown to be powerful in modeling large text corpora, the topic
Topic	Topic Words	Topic Label
Z1(S1)	profit, growth, stocks, apple, fall, consumer, buy, billion, shares	Trading
Z2(S2)	smartphone, ipad, apple, app, iphone, devices, phone, tablet	Product Line
Z3(S3)	microsoft, mac, linux, ibm, ios, apple, xp, windows, software	Operating System
Z4(T)	apple, talk, computers, shares, disease, driver, electronics, profit, ios	?
Table 1: Coherent  $(Z_{1} - Z_{3})$  vs Incoherent  $(Z_{4})$  topics from high-resource  $(S^{1} - S^{3})$  and low-resource  $(\mathcal{T})$  texts
modeling (TM) still remains challenging especially in the sparse-data setting, especially for the cases where word co-occurrence data is insufficient, e.g., on short-text or a corpus of few documents. It leads to a poor quality of topics and representations.
To address data sparsity issues, several works (Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019a, 2020) have introduced external knowledge in traditional topic models, e.g., incorporating word embeddings obtained from Glove (Pennington et al., 2014) or word2vec (Mikolov et al., 2013a). However, no prior work in topic modeling has employed multi-view embedding spaces: (1) pretrained topics, i.e., topical embeddings obtained from large document collections, and (2) pretrained contextualized word embeddings from large-scale language models like BERT (Devlin et al., 2019).
Though topics and word embeddings are complementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. A topic model (Blei et al., 2003) is a statistical tool to infers topic distributions across a collection of documents and assigns a topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. Therefore, a topic has a global view representing semantic structures hidden in document collection. On other hand, word embeddings have primarily local view in the
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4205-4217 June 6-11, 2021. ©2021 Association for Computational Linguistics
sense that they are learned based on local collocation pattern in a text corpus, where the representation of each word often depends on a local context window (Mikolov et al., 2013b) or is a function of its sentence(s) (Peters et al., 2018). Consequently, they are not aware of the thematic structures underlying the document collection. Additionally, recent studies (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019) have shown a reasonable success in several NLP applications by employing pretrained contextualized word embeddings, where the representation of a word is different in different contexts (i.e., context-sensitive). In context of this work, the representations due to global and local (context-sensitive or context-insensitive) views together are referred as multi-view embeddings.
For example in Table 1, consider four topics  $(Z_{1} - Z_{4})$  of different domains where the topics  $(Z_{1} - Z_{3})$  are respectively obtained from three different high-resource source  $(S^{1} - S^{3})$  domains whereas  $Z_{4}$  from a low-resource target domain  $\mathcal{T}$  (especially in the data-sparsity settings). Observe that the topics about Trading  $(Z_{1})$ , Product Line  $(Z_{2})$  and Operating System  $(Z_{3})$  are coherent and and represent meaningful semantics at document-level via lists of topic words. However in sparse-data settings, the topic  $Z_{4}$  discovered is incoherent (noisy) and it is difficult to infer meaningful document semantics.
Unlike the topics, word embeddings (context-insensitive) encode syntactic and semantic relatedness in fine-granularity and therefore, do not capture thematic structures. For instance, the top-5 nearest neighbors (NN) of apple (below) in word embedding (Mikolov et al., 2013b) space suggest that it refers to a fruit and do not express any topical information (e.g., Trading, Product Line or Health) in the corpora. Similarly given the NN of the word fall, it is difficult to infer its association with document-level semantics, e.g., Trading as expressed by  $Z_{1}$  in topic-embedding space.
apple  $\stackrel{\mathrm{NN}}{\Longrightarrow}$  apples, pear, fruit, berry, pears, strawberry
fall  $\stackrel{\mathrm{NN}}{\Longrightarrow}$  falling, falls, drop, tumble, rise, plummet
Therefore, topic and word embedding spaces encode complementary semantics. Different to context-insensitive word embeddings, the word apple is referring to an organization and contextualized by different topical semantics respectively in the three sources  $S^1 - S^3$ . Thus, it arises the need for context-sensitive embeddings in topic modeling.
Contribution (1) Multi-view Neural Topic Modeling using pretrained word and topic em
Notation	Description
LVT, GVT	Local-view Transfer, Global-view Transfer
MVT, MST	Multi-view Transfer, Multi-source Transfer
T, S	A target domain, a set of source domains
v, k, L	An input document, kth source, loss
K, D	Vocabulary size, document size
E, H	Word embedding dimension, #topics
W ∈ RH×K	Encoding matrix of DocNADE in T
U ∈ RK×H	Decoding matrix of DocNADE
λk	Degree of relevance of Ek in T
γk	Degree of imitation of Zk by W
Ek ∈ RE×K	Word embeddings of kth source
Zk ∈ RHT×K	Topic embeddings of kth source
Ak ∈ RHT×H	Topic-alignment in T and Zk
b ∈ RK, c ∈ RH	Visible-bias, hidden-bias
DC	Document Collection
Table 2: Description of the notations used in this work
beddings: To alleviate the data sparsity issues, it is the first work in unsupervised neural topic modeling (NTM) within transfer learning paradigm that employs multi-view embedding spaces via: (a) Global-view Transfer (GVT): Pretrained topic embeddings instead of using word embeddings exclusively, and (b) Multi-view Transfer (MVT): Pretrained topic and word embeddings (context-insensitive from Glove (Pennington et al., 2014) and context-sensitive from large-scale language models such as BERT (Devlin et al., 2019) jointly to address data sparsity and polysemy issues.
Contribution (2) Multi-source Multi-view Neural Topic Modeling: A single source of prior knowledge is often insufficient due to incomplete and non-overlapping domain information required by a target domain. Therefore, there is a need to leverage multiple sources of prior knowledge, dealing with domain-shifts (Cao et al., 2010) among the target and sources. In doing so, we first learn word and topic representations on multiple source domains to build WordPool and TopicPool, respectively and then perform multi-view and multi-source transfer learning in neural topic modeling by jointly using the complementary representations.
We evaluate the effectiveness of multi-source neural topic modeling in multi-view embedding spaces using 7 (5 low-resource and 2 high-resource) target and 5 (high-resource) source corpora from news and medical domains, consisting of short-text, long-text, small and large document collections. We have shown state-of-the-art results with significant gains quantified by generalization (perplexity), interpretability (topic coherence) and text retrieval. The code is available at https://github.com/YatinChaudhary/Multi-view-Multi-source-Topic-Modeling.
Figure 1: (Left) DocNADE (LVT+MST): Multi-source transfer learning in NTM for a document  $\mathbf{v}$  by introducing pretrained word embeddings from a WordPool at each autoregressive step  $i$ . Double circle  $\rightarrow$  multinomial (soft-max) unit (Larochelle and Lauly, 2012). (Right) DocNADE (GVT+MST): Multi-source transfer learning in NTM by introducing pretrained (latent) topic embeddings from a TopicPool, illustrating topic alignments between source and target corpora. Each outgoing row from  $\mathbf{Z}^k \in \mathbb{R}^{H \times K}$  signify a topic embedding of corresponding  $k$ th source corpus,  $DC^k$ . Here, NTM refers to a DocNADE (Larochelle and Lauly, 2012) based Neural Topic Model.
2 Knowledge-Aware Topic Modeling
Consider a sparse target domain  $\mathcal{T}$  and a set of source domains  $S$ , we first prepare two knowledge bases (KBs) of representations (or embeddings) from document collections of each of the  $|\mathcal{S}|$  sources: (1) WordPool: a KB of pretrained word embeddings matrices  $\{\mathbf{E}^1, \dots, \mathbf{E}^{|S|}\}$ , where  $\mathbf{E}^k \in \mathbb{R}^{E \times K}$ , and (2) TopicPool: a KB of pretrained latent topic embeddings  $\{\mathbf{Z}^1, \dots, \mathbf{Z}^{|S|}\}$ , where  $\mathbf{Z}^k \in \mathbb{R}^{H \times K}$  encodes a distribution over a vocabulary of  $K$  words. Here,  $k \in [1, \dots, |S|]$  in superscript indicates knowledge of  $k$ th source, and  $E$  and  $H$  are word embedding and latent topic dimensions, respectively. While topic modeling on  $\mathcal{T}$ , we introduce the two types of knowledge transfers from one or many sources: Local (LVT) and Global (GVT) View Transfer using the two KBs of pretrained word (i.e., WordPool) and topic (i.e., TopicPool) embeddings, respectively. Specially, we employ a neural autoregressive topic model, i.e., DocNADE as backbone in building the pools and realizing the multi-source multi-view framework.
Table 2 describes the notations used. Notice that the superscript used in notations indicates a source.
2.1 Neural Autoregressive Topic Models
DocNADE (Larochelle and Lauly, 2012) is an unsupervised neural-network based generative topic model that is inspired by the benefits of NADE (Larochelle and Murray, 2011) and RSM (Salakhutdinov and Hinton, 2009) architectures. Specifically, DocNADE factorizes the joint probability distribution of words in a document as a product of conditional distributions and efficiently models each
conditional via a feed-forward neural network (ff-net), following reconstruction mechanism.
DocNADE Formulation: For a document  $\mathbf{v} = (v_{1},\dots,v_{D})$  of size  $D$ , each word index  $v_{i}$  takes value in  $\{1,\dots,K\}$  of vocabulary size  $K$ . DocNADE learns topics in a language modeling fashion (Bengio et al., 2003) and decomposes the joint distribution  $p(\mathbf{v}) = \prod_{i=1}^{D} p(v_{i}|\mathbf{v}_{<i})$  such that each autoregressive conditional  $p(v_{i}|\mathbf{v}_{<i})$  is modeled by a ff-net using preceding words  $\mathbf{v}_{<i}$  in the sequence:
$$
\mathbf {h} _ {i} (\mathbf {v} _ {<   i}) = g (\mathbf {c} + \sum_ {q <   i} \mathbf {W} _ {:, v _ {q}}) \text {a n d} g = \{\text {s i g m o i d , t a n h} \}
$$
$$
p (v _ {i} = w | \mathbf {v} _ {<   i}) = \frac {\exp \left(b _ {w} + \mathbf {U} _ {w ^ {\prime}} ; \mathbf {h} _ {i} (\mathbf {v} _ {<   i})\right)}{\sum_ {w ^ {\prime}} \exp \left(b _ {w ^ {\prime}} + \mathbf {U} _ {w ^ {\prime} , \cdot} ; \mathbf {h} _ {i} (\mathbf {v} _ {<   i})\right)}
$$
for each word  $i\in \{1,\dots,D\}$  where  $\mathbf{v}_{< i}$  is the subvector consisting of all  $v_{q}$  such that  $q < i$  i.e.,  $\mathbf{v}_{< i}\in \{v_1,\ldots ,v_{i - 1}\}$ ,  $g(\cdot)$  is a non-linear activation function,  $\mathbf{W}\in \mathbb{R}^{H\times K}$  and  $\mathbf{U}\in \mathbb{R}^{K\times H}$  are weight matrices,  $\mathbf{c}\in \mathbb{R}^H$  and  $\mathbf{b}\in \mathbb{R}^{K}$  are bias parameter vectors.  $H$  is the number of hidden units (the number of topics to be discovered).
Figure 1 (left) (except WordPool) describes the DocNADE architecture for the  $i$ th autoregressive step, where the parameter  $\mathbf{W}$  is shared in the feedforward networks and  $\mathbf{h}_i$  encodes latent document-topic proportion. The value of each unit  $j$  in the hidden vector signifies contribution of the  $j$ th topic in the proportion. Importantly, the topic-word matrix  $\mathbf{W}$  has a property that the column vector  $\mathbf{W}_{:,v_i}$  corresponds to embedding of the word  $v_i$ , whereas the row vector  $\mathbf{W}_{j,:}$  encodes latent features for the  $j$ th topic (i.e., topic-word distribution). We leverage this property to introduce external knowledge via word and topic embeddings.
Algorithm 1 (for DocNADE, set both LVT and GVT to False) demonstrates the computation of  $\log p(\mathbf{v})$  and loss (i.e., negative log-likelihood)  $\mathcal{L}(\mathbf{v})$  that is minimized using stochastic gradient descent. Moreover, computing each  $\mathbf{h}_i$  is efficient (linear complexity) due to NADE architecture that leverages the pre-activation  $\mathbf{a}_{i - 1}$  of  $(i - 1)\mathrm{th}$  step in computing  $\mathbf{a}_i$  for the  $i$ th step (line #6). See Larochelle and Lauly (2012) for further details.
Why DocNADE backbone: It has shown outperforming traditional models such as LDA and RSM. Additionally, Gupta et al. (2019a,b) have extended DocNADE on short texts by introducing context-insensitive word embeddings; however, based on a single-source transfer. Thus, we adopt DocNADE.
2.2 MVT and MST in Neural Topic Modeling
We describe our transfer learning framework in topic modeling that jointly exploits the complementary prior knowledge accumulated in (WordPool, TopicPool), obtained from large document collections (DCs) from several sources. In doing so, we first apply the DocNADE to generate a topic-word matrix for each of the DCs, where its column-vector and row-vector generate  $\mathbf{E}^k$  and  $\mathbf{Z}^k$ , respec
tively for the  $k$ th source. See appendix for the mechanics of extracting word and topic embeddings from the topic-word matrix of a source.
LVT+MST Formulation for Multi-source Word Embedding Transfer: As illustrated in Figure 1 (left) and Algorithm 1 (with LVT being True, line #7), we perform transfer learning on a target  $\mathcal{T}$  using the WordPool of pretrained word embeddings  $\{\mathbf{E}^1,\dots,\mathbf{E}^{|S|}\}$  from several sources  $S$  (i.e., multi-source) under the two schemes:
scheme (i): Using a domain-relevance factor  $\lambda$  for every source in the WordPool such that the hidden vector  $\mathbf{h}_i$  encodes document-topic distribution, augmented with prior knowledge in form of pretrained word embeddings from several sources:
$$
\mathbf {h} _ {i} (\mathbf {v} _ {<   i}) = g (\mathbf {c} + \sum_ {q <   i} \mathbf {W} _ {:, v _ {q}} + \sum_ {q <   i} \sum_ {k = 1} ^ {| S |} \lambda^ {k} \mathbf {E} _ {:, v _ {q}} ^ {k})
$$
Here,  $k$  refers to the  $k$ th source and  $\lambda^k$  is a weight for  $\mathbf{E}^k$  that controls the amount of knowledge transferred in  $\mathcal{T}$ , based on cross-domain overlap.
scheme (ii): Using a projection matrix  $\mathbf{P} \in \mathbb{R}^{H \times P}$  with  $P = E \times |\mathcal{S}|$  in order to align word-embedding spaces of the target and all source domains for all  $D$  words in the document  $\mathbf{v}$  such that:
$$
\text {F o r} q \in \{i, \dots D \}: \hat {\mathbf {e}} _ {q} = \operatorname {c o n c a t} \left(\mathbf {E} _ {i: v _ {q}} ^ {1}, \dots , \mathbf {E} _ {i: v _ {q}} ^ {k}\right)
$$
$$
\mathbf {h} _ {i} \left(\mathbf {v} _ {<   i}\right) = g \left(\mathbf {c} + \sum_ {q <   i} \mathbf {W} _ {:, v _ {q}} + \sum_ {q <   i} \mathbf {P} \cdot \hat {\mathbf {e}} _ {q}\right)
$$
Unlike scheme (i), the second schema allows us to automatically determine shifts in the target and source domains, identify and transfer relevant prior knowledge from many sources without configuring  $\lambda$  for every source. To better guide TM, we also introduce pre-trained contextualized word embedding from BERT, concatenating with  $\hat{\mathbf{e}}_q$ .
GVT+MST Formulation for Multi-source Topic Embedding Transfer: Next, we perform knowledge transfer exclusively using the TopicPool of pretrained topic embeddings (e.g.,  $\mathbf{Z}^k$ ) from one or several sources,  $S$ . In doing so, we add a regularization term to the loss function  $\mathcal{L}(\mathbf{v})$  and require DocNADE to minimize the overall loss in a way that the (latent) topic features in  $\mathbf{W}$  simultaneously inherit relevant topical features from each of the source domains  $S$ , and thus, it generates meaningful representations for the target  $\mathcal{T}$  in order to address data-sparsity. The overall loss  $\mathcal{L}(\mathbf{v})$  due to GVT+MST configuration in DocNADE is:
$$
\mathcal {L} (\mathbf {v}) = - \log p (\mathbf {v}) + \sum_ {k = 1} ^ {| S |} \gamma^ {k} \sum_ {j = 1} ^ {H} \left| \left| \mathbf {A} _ {j,:} ^ {k} \mathbf {W} - \mathbf {Z} _ {j,:} ^ {k} \right| \right| _ {2} ^ {2}
$$
ID	Target Domain Corpora	ID	Source Domain Corpora
Data	Train	Val	Test	K	L	C	Data	Train	Val	Test	K	L	C
T1	20NSshort	1.3k	0.1k	0.5k	1.4k	13.5	20	S1	20NS	7.9k	1.6k	5.2k	2k	107.5	20
T2	20NSsmall	0.4k	0.2k	0.2k	2k	187.5	20	S2	R21578	7.3k	0.5k	3.0k	2k	128	90
T3	TMNtitle	22.8k	2.0k	7.8k	2k	4.9	7	S3	TMN	22.8k	2.0k	7.8k	2k	19	7
T4	R21578title	7.3k	0.5k	3.0k	2k	7.3	90	S4	AGNews	118k	2.0k	7.6k	5k	38	4
T5	Ohsumedtitle	8.3k	2.1k	12.7k	2k	11.9	23	S5	PubMed	15.0k	2.5k	2.5k	3k	254.8	-
T6	Ohsumed	8.3k	2.1k	12.7k	3k	159.1	23
Table 3: Data statistics: Short/long texts and/or small/large corpora in target and source domains. Symbols-  $K$  : vocabulary size,  $L$  : average text length (#words),  $C$  : #classes and  $k$  : thousand. For short-text,  $L < {15}$  .  ${\mathcal{S}}^{3}$  is also used in target. '-': unlabeled data.
T1	T2	T3	T4	T5	T6
S1	I	I	R	D	D	D
S2	D	D	D	I	D	D
S3	R	R	I	D	D	D
S4	R	R	R	D	D	D
S5	D	D	D	D	-	-
Here,  $\mathbf{A}^k\in \mathbb{R}^{H\times H}$  aligns latent topics in the target  $\mathcal{T}$  and  $k$ th source, and  $\gamma^k$  governs the degree of imitation of topic features  $\mathbf{Z}^k$  by  $\mathbf{W}$  in  $\mathcal{T}$ . Consequently, the generative process of learning meaningful topics in  $\mathbf{W}$  of the target domain  $\mathcal{T}$  is guided by relevant topic features  $\{\mathbf{Z}\}_{1}^{|S|}\in \mathrm{TopicPool}$ . Algorithm 1 (line #11) describes the computation of the loss, when GVT = True and LVT = False.
Moreover, Figure 1 (right) illustrates the need for topic alignments between target and source(s). Here,  $j$  indicates the topic (i.e., row) index in a topic matrix, e.g.,  $\mathbf{Z}^k$ . Observe that the first topic (gray curve), i.e.,  $Z_{j=1}^{1} \in \mathbf{Z}^{1}$  of the first source aligns with the first row-vector (i.e., topic) of  $\mathbf{W}$  (of target). However, the other two topics  $Z_{j=2}^{1}, Z_{j=3}^{1} \in \mathbf{Z}^{1}$  need alignment with the target.
MVT+MST Formulation for Multi-source Word and Topic Embeddings Transfer: When LVT and GVT are True (Algorithm 1) for many sources, the two complementary representations are jointly used in transfer learning using WordPool and TopicPool, and therefore, the name multi-view and multi-source transfers.
Computational complexity of NTM: For Doc-NADE, the complexity of computing all hidden layers  $\mathbf{h}_i(\mathbf{v}_{< i})$  is in  $O(DH)$  and all  $p(\mathbf{v}|\mathbf{v}_{< i})$  in  $O(KDH)$ . Thus, the overall complexity of Doc-NADE is in  $O(DH + KDH)$ .
Within the proposed transfer learning framework, the complexity of computing all hidden layers (LVT+MST in scheme (i)) and topic-embedding transfer term (GVT+MST) is in  $O(DH + |S|DH)$  and  $O(|S|KH)$ , respectively. Since  $|S| \ll H$ , thus the overall complexity of DocNADE with MVT+MST is in  $O(DH + KDH + KH)$ .
3 Evaluation and Analysis
Datasets: Table 3 describes the datasets used in high-resource source and low-and high-resource target domains for our experi
Table 4: Domain overlap in source-target corpora.  $\mathcal{I}$  : Identical,  $\mathcal{R}$  : Related and  $\mathcal{D}$  : Distant domains.
Baselines
(Related Works)	Features
NTM	AuR	LVT	GVT	MVT
LDA
RSM	✓
DocNADE	✓	✓
NVDM	✓
ProdLDA
Gauss-LDA	✓
glove-DMM	✓
DocNADEe	✓	✓	✓
EmbSum-Glove, EmbSum-BERT
doc2vec
this work	✓	✓	✓	✓	✓
Table 5: Baselines (related works) vs this work. Here, NTM and AuR refer to neural network-based TM and autoregressive assumption, respectively. DocNADEe  $\rightarrow$  DocNADE+Glove embeddings.
ments. The target domain  $\mathcal{T}$  consists of four short-text corpora (20NSshort, TMNtitle, R21578title and Ohsumeditedtitle), one small corpus (20NSsmall) and two large corpora (TMN and Ohsumed). However in source  $S$ , we use five large corpora (20NS, R21578, TMN, AGnews and PubMed) in different label spaces (i.e., domains). Here, the corpora  $(\mathcal{T}^5,\mathcal{T}^6$  and  $S^5)$  belong to medical and others to news.
Additionally, Table 4 suggests domain overlap (label match) in the target and source corpora, where we define 3 types of overlap:  $\mathcal{I}$  (identical) if all labels match,  $\mathcal{R}$  (related) if some labels match, and  $\mathcal{D}$  (distant) if a very few or no labels match. Note, our approaches are completely unsupervised and do not use the data labels (appendix).
Reproducibility: We follow the experimental setup similar to DocNADE (Larochelle and Lauly, 2012) and DocNADEe (Gupta et al., 2019a), where the number of topics  $(H)$  is set to 200. While DocNADEe requires the dimension (i.e.,  $E$ ) of word embeddings be the same as the latent topic (i.e.,  $H$ ), we follow scheme (ii) (Algorithm 1) to introduce