SlowGuess/ACL_2020_OCR huggingface dataset:
tion (Rezende et al., 2014; Kingma and Welling, 2014). Based on the central limit theorem,  $\mathcal{N}(\mu = \lambda, \sigma^2 = \lambda)$  can approximate Poisson  $(\lambda)$ . Thus, we sample from the Poisson distribution directly to avoid the issue of discretization and use the Gaussian distribution as an approximation when calculating the Poisson distribution's gradient. Particularly, the reparameterization of a Gaussian distribution  $x \sim \mathcal{N}(\mu, \sigma^2)$  is  $x = \mu + \epsilon \cdot \sigma$ ,  $\epsilon \sim \mathcal{N}(0,1)$ .
We employ the following three datasets to evaluate the effectiveness of our models: Reuters<sup>1</sup>, 20News, and MXM song lyrics (Miao et al., 2017). The Reuters dataset contains 7,758 training documents and 3,005 testing documents. The 20News corpus consists of 18,773 news articles under 20 categories. These news articles are divided into 11,268 training documents and 7,505 testing documents. The 20 categories include sports, electronics, automotive, and so forth, and the number of documents under each category is almost the same. MXM is the official lyrics collection of the Million Song Dataset, which contains 210,519 training documents and 27,143 testing documents, respectively. By following (Miao et al., 2017), we use the originally provided vocabulary with 5,000 words for MXM, while for Reuters and 20News, we use stemming, stop words filtering, and the 2,000 most frequently occurred words as vocabularies. The statistics of these datasets are presented in Table 1.
Dataset	Reuters	20News	MXM
Train.Size	7,758	11,268	210,519
Test.Size	3,005	7,505	27,143
Label number	90	20	-
Vocabulary size	2,000	2,000	5,000
The following models are adopted as baselines: HDP (Teh et al., 2004), NVDM (Miao et al., 2016), NVLDA and ProdLDA (Srivastava and Sutton, 2017), GSM (Miao et al., 2017), and iDocNADE (Gupta et al., 2019). Among these baselines, HDP is a classical mixture modeling method followed
<https://www.nltek.org/book/ch02.html
6164
the equivalence with the normalized GNB process (Zhou and Carin, 2012). In HDP, the model parameters are estimated by Monte Carlo sampling. NVDM, NVLDA, ProdLDA, and GSM are all neural topic models based on NVI. Considering that word embeddings have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in natural language processing tasks, we also present the result of a neural autoregressive topic model that leverages word embeddings (i.e., iDocNADE). Particularly, the publicly available codes of  $\mathrm{HDP}^2$ ,  $\mathrm{NVDM}^3$ , NVLDA and ProdLDA<sup>4</sup>, and iDocNADE<sup>5</sup> are directly used. As an extended model of NVDM, the baseline of GSM is implemented by us based on the code of NVDM. To ensure fair comparisons on various NVI-based methods, unless explicitly specified, we set the number of topics to 50, the hidden dimension of MLP to 256, and use one sample for NVI by following (Miao et al., 2017). For the batch size, the learning rate, and other model parameters, grid search is carried out on the training set to determine their optimal values and achieve the held-out performance.
To evaluate the quality of topics generated by different models, we use perplexity and topic coherence as evaluation criteria. The perplexity of each model on a testing set  $\widetilde{D}$  is: perplexity  $(\widetilde{D}) = \exp \left(-\frac{1}{|\widetilde{D}|} \sum_{\widetilde{d}} \frac{1}{N_{\widetilde{d}}} \log p(\widetilde{d})\right)$ , where  $\log p(\widetilde{d})$  represents the log-likelihood of the model on document  $\widetilde{d}$ , and  $N_{\widetilde{d}}$  is the number of words in  $\widetilde{d}$ . The lower the perplexity is, the more likely for a model to generate  $\widetilde{D}$ . Therefore, if a model obtains a lower perplexity than others in the testing set, it can be considered as the better one. For all NVI-based topic models, the variational lower bound, which is proven to be the upper bound of perplexity (Mnih and Gregor, 2014), is used to calculate the perplexity by following (Miao et al., 2016, 2017). When calculating the topic coherence, we use the normalised pointwise mutual information (NPMI) which measures the relationship between word  $w_i$  and other  $T - 1$  top words (Lau et al., 2014) as follows: NPMI  $(w_i) = \sum_{j=1}^{T-1} \left[ \log \frac{P(w_i, w_j)}{P(w_i) P(w_j)} / -\log P(w_i, w_j) \right]$ . The higher the value of topic coherence, the more explainable the topic is.
5.3 Performance Comparison
Table 2 shows the perplexity and topic coherence of different models on the test datasets. We can observe that NB-NTM outperforms most baselines, and GNB-NTM performs the best in all cases. The results validate that the NB distribution can model over-dispersed documents well. Furthermore, the latent semantics of these corpora may be hierarchically dependent. In other words, the topics at the corpus level and those of each document are not independent but correlated with one another.
Model	Perplexity	Topic coherence
Reuters	20News	MXM	Reuters	20News	MXM
HDP	302.3	730.7	319.5	0.305	0.223	0.356
NVDM	224.9	855.0	252.3	0.133	0.138	0.109
NVLDA	578.6	1252.2	668.7	0.253	0.240	0.216
prodLDA	648.1	1267.2	852.9	0.332	0.329	0.313
GSM	266.2	963.5	330.5	0.192	0.211	0.177
iDocNDAE	202.8	844.6	294.7	0.130	0.151	0.222
NB-NTM	181.0	740.6	247.5	0.341	0.343	0.340
GNB-NTM	146.7	602.8	216.8	0.377	0.375	0.427
Table 2: Perplexity and topic coherence results, where the latter is an average of three coherence scores by calculating 5, 10, and 15 top words for each topic.
In terms of the model efficiency, neural topic models can be trained much faster than HDP on a large corpus by GPU acceleration. Take the large-scaled MXM dataset as an example, the training time of both NB-NTM and GNB-NTM is around one hour using a GeForce GTX 960 GPU, while HDP needs more than three hours to converge using an AMD R5 3600 CPU. Under the same environment, the training time of all NVI-based topic models is close. In general, NVLDA, prodLDA, and NVDM run slightly faster than NB-NTM because the Gaussian reparameterization approach is simpler than the Gamma one. GSM and GNB-NTM are slightly slower than others because the former introduces more parameters to model the topic-word distribution, while the latter introduces more sampling operations.
As an illustration, we also qualitatively evaluate the semantic information learned by different models on the 20News training set. The baselines of HDP, NVLDA, and prodLDA, which achieve competitive topic coherence scores, are selected for comparison. Table 3 presents 5 of the most representative topics with the corresponding top 10 words, from which we can observe that although all these models can identify the chosen topics reasonably, our NB-NTM and GNB-NTM perform better than the other baselines in most cases.
$^{2}$ https://github.com/soberqian/  
TopicModel4J  
 $^{3}$ https://github.com/ysmiao/nvdm  
 $^{4}$ https://github.com/akashgit/  
autoencoding_vi_for_topi_c Models  
 $^{5}$ https://github.com/pgcool/iDocNADEe
6165
Topic	HDP	NVLDA	prodLDA	NB-NTM	GNB-NTM
Religion	god●who people atheism●believe●religion●does atheists●his evidence	heaven●christinterpretation scripture●christian●church●truth●lord●believe●christianity●	shall worship christians●religious●belief●bible●atheists●heaven●acts●religions●	belief●religion●athos●moral●scripture●jesus●church●christian●christianity●god●	athos●beliefs●morale●truth●church●christian●jesus●christianity●belief●
Encryption	key●unitkeyboardkeys●cablelock●fitcrossbackwomen	keys●bradchipcryptophoneencryption●cryptography●agencies●security●	encryption●courtclipper semiescrow●encrypted●drugsguncriminalcriminals	rsa●agencies●encrypted●cryptography●security●scheme●government escrow●nsa●secure●	cryptography●cryptopen ●security●keys●nsa●secure●key●government agencies●
Sport	game●fighting●fourgames●almostlevelpolicecokilleffective	cup●torontoplayed●patrickwingsplayers●rangersleafsteams●baseball●	players●team●teams●winning●icehimnhl●season●hockey●leafs	player●win●play●bostondetroitcup●playoffs●players●season●games●	hockey●sport●game●games●baseball●fans●season●teams●wingsleafs
Space	itsearth●organizationsfirsthighmission●shell●theirsuchprogram	shuttle●jpl●developmentphysicsorbit●rocket●costenergyearth●space●	commercial cryptographymission●imagelunar●processingestablishedrocket●remotesoviet	lunar●historontoorbit●years●mission●dcyear●nationalspace●	mission●algorithmnasa●chiporbiodevelopmentsolar●space●technologysatellite●
Hardware	card●bitmac●mbmemory●mhzram●monitor●speedbus●	cable●floppy●dxcontroller●mbpin●shippingbranddrivemotherboard●	mouse●floppy●card●libsimmsbutton●printricemegram●motherboard●	floppy●bus●ram●printer●memory●card●controller●motherboard●ide monitor●	cache●vga●display●printer●interface●pc●dxprocessor●motherboard●ram●
5.4 Impact of the Number of Topics
In this part, we test the impact of the number of topics on the performance of our models. Figure 3 shows the convergence process of NB-NTM and GNB-NTM on the 20News training set with  $K = 20, 50, 100, 200$  in terms of the perplexity. We can observe that as  $K$  increases, the perplexity values of both models decrease under each epoch. This is because the NVI framework is essentially an encoder-decoder, and the increase of the topic number enables the models to encode and reconstruct documents better. We also notice that with the continuous growth of  $K$ , the improvement of perplexity is getting lower. Table 4 presents the results of our models on the 20News testing set under the above conditions, in which a similar trend can be observed as aforementioned.
5.5 Evaluation on Learning Dispersed Topics
Compared to the existing neural topic models, another feature of our models is that the generated
(a) NB-NTM
(b) GNB-NTM
Figure 3: The convergence behavior of our models with different numbers of topics on the 20News training set.
Table 3: Top 10 words of 5 topics learned by different models on 20News, where  $\bullet$  means the word is related to the corresponding topic by checking manually.
Number of topics	Perplexity	Topic coherence
NB-NTM	GNB-NTM	NB-NTM	GNB-NTM
20	800.8	717.3	0.307	0.351
50	740.6	602.8	0.343	0.375
100	654.3	501.2	0.331	0.360
200	572.4	424.4	0.330	0.351
Table 4: Perplexity and topic coherence of our models on the 20News testing set with different topic numbers.
topics are dispersed, and thus, the intermediate variables can be more explainable. To validate the effectiveness of our models on learning dispersed topics, we first count the total number of words under each manually labeled category (i.e., topic) as the topic-word number distribution shown in Figure 4 (a). Then we run our NB-NTM and GNB-NTM on the entire 20News testing set to get the corresponding values of  $r$ . After normalization, the proportion of different topics obtained by NB-NTM and GNB-NTM at the corpus level is presented in Figure 4 (b) and Figure 4 (c), respectively. For the convenience of the result presentation, we set the number of topics to 20 for both models. Note that the 20 topics do not need to correspond to the 20 categories, because we here focus on testing whether the topic proportions generated by our two models are in accordance with their model structures/characteristics. From these results, we can observe that the proportion of topics obtained by NB-NTM is close to the topic-word number distribution. On the other hand, GNB-NTM obtains more dispersed proportions of topics than NB-NTM. These results suggest that GNB-NTM tends to allocate less but more important topics to the corpus, i.e., the topics generated by GNB-NTM are more discriminative. Since the document-topic distribution is not directly modeled and the Gaussian distribution samples are not non-negative, the previous neural methods except GSM cannot obtain explainable intermediate variables. For the baseline of GSM, Miao et al. (2017) had demonstrated that the topics with higher probabilities were evenly
6166
distributed on the same 20News dataset, which indicates that our models outperform GSM on learning dispersed document topics.
We also study the dispersion of intermediate variables (i.e., topics) at the document level. By randomly select a document as an example, we get the normalized document topic weight  $\lambda$  from NB-NTM and GNB-NTM to explore whether the topic distributions of the document generated by our models are reasonable. As shown in Figure 5, the document is about a standard computer, and the most related topics with large topic distributions are all related to computers, which validates the practical meaning of intermediate variables of both NB-NTM and GNB-NTM at the document level. From the keywords in the most related topics, we further observe that GNB-NTM can identify more computer-related words than NB-NTM. When compared to the whole semantic space as shown in Figure 4, both NB-NTM and GNB-NTM generate more dispersed proportions of topics at the document level. This phenomenon is consistent with the over-dispersed feature (i.e., the variance is larger than the mean) of documents.
In this paper, we present two neural mixed counting models named NB-NTM and GNB-NTM. Different from the current time consuming Bayesian methods, our models apply to large-scale datasets through the efficient back-propagation algorithm and GPU acceleration. When compared to the existing neural topic models, both NB-NTM and GNB-NTM can well model the random variables with
Figure 4: Qualitative analysis on the model interpretability at the corpus level, where (a) is the topic-word number distribution generated by the label information, (b) is the proportion of 20 topics obtained by NB-NTM, and (c) is the proportion of 20 topics obtained by GNB-NTM. All results are generated from the 20News testing set.
Figure 5: The proportion and key words of 20 topics obtained by our models on a document instance.
over-dispersed and hierarchically dependent characteristics. Extensive experiments on real-world datasets validate the effectiveness of our models in terms of perplexity, topic coherence, and producing explainable intermediate variables by generating dispersed proportions of document topics. The results also indicate that NB distribution families can characterize text data aptly, which is essentially due to their conformity with the over-dispersed and sparse properties of natural language.
We are grateful to the reviewers for their constructive comments and suggestions on this study. This work has been supported by the National Natural Science Foundation of China (61972426), Guangdong Basic and Applied Basic Research Foundation (2020A1515010536), HKIBS Research Seed Fund 2019/20 (190-009), the Research Seed Fund (102367), and LEO Dr David P. Chan Institute of Data Science of Lingnan University, Hong Kong. This work has also been supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/FDS16/E01/19), Hong Kong Research Grants Council through a General Research Fund (project no. PolyU 1121417), and by the Hong Kong Polytechnic University through a start-up fund (project no. 980V).
6167
Ayan Acharya, Joydeep Ghosh, and Mingyuan Zhou. 2015. Nonparametric bayesian factor analysis for dynamic count matrices. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics.
David M. Blei and John D. Lafferty. 2005. Correlated topic models. In Proceedings of the 19th Annual Conference on Neural Information Processing Systems, pages 147-154.
David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, pages 113-120.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022.
John Frank Charles Kingman. 1967. Completely random measures. Pacific Journal of Mathematics, 21(1):59-78.
Dehua Cheng and Yan Liu. 2014. Parallel gibbs sampling for hierarchical dirichlet processes via gamma processes equivalence. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 562-571.
Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, pages 795-804.
Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391-407.
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 6505-6512.
Thomas Hofmann. 1999. Probabilistic latent semantic analysis. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, pages 289-296.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations.
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530-539.
John Reidar Mathiassen, Amund Skavhaug, and Ketil Bø. 2002. Texture similarity measure using kullback-leibler divergence between gamma distributions. In Proceedings of the 7th European Conference on Computer Vision, pages 133-147.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning, pages 2410-2419.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, pages 1727-1736.
Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31th International Conference on Machine Learning, pages 1791-1799.
Christian A. Naesseth, Francisco J. R. Ruiz, Scott W. Linderman, and David M. Blei. 2017. Reparameterization gradients through acceptance-rejection sampling algorithms. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 489-498.
Eric T. Nalisnick and Padhraic Smyth. 2017. Stick-breaking variational autoencoders. In Proceedings of the 5th International Conference on Learning Representations.
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 6345-6381.
Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. 2009. Labeled LDA: A supervised topic model for credit attribution in multilabeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 248-256.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, pages 1278-1286.
Akash Srivastava and Charles A. Sutton. 2017. Autoencoding variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations.
6168
Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2004. Sharing clusters among related groups: Hierarchical dirichlet processes. In Proceedings of the 18th Annual Conference on Neural Information Processing Systems, pages 1385-1392.
Romain Thibaux and Michael I. Jordan. 2007. Hierarchical beta processes and the indian buffet process. In Proceedings of the 11th International Conference on Artificial Intelligence and Statistics, pages 564-571.
Michalis K. Titsias. 2007. The infinite gamma-poisson feature model. In Proceedings of the 21st Annual Conference on Neural Information Processing Systems, pages 1513-1520.
Chong Wang, John W. Paisley, and David M. Blei. 2011. Online variational inference for the hierarchical dirichlet process. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pages 752-760.
Xuerui Wang and Andrew McCallum. 2006. Topics over time: a non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 424-433.
Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013. A biterm topic model for short texts. In Proceedings of the 22nd International World Wide Web Conference, pages 1445-1456.
Limin Yao, David M. Mimno, and Andrew McCallum. 2009. Efficient methods for topic model inference on streaming document collections. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 937-946.
He Zhao, Piyush Rai, Lan Du, Wray L. Buntine, and Mingyuan Zhou. 2019. Variational autoencoders for sparse and overdispersed discrete data. arXiv preprint arXiv:1905.00616, abs/1905.00616.
Mingyuan Zhou. 2018. Nonparametric bayesian negative binomial factor analysis. Bayesian Analysis, 13(4):1061-1089.
Mingyuan Zhou and Lawrence Carin. 2012. Augment-and-conquer negative binomial processes. In Proceedings of the 26th Annual Conference on Neural Information Processing Systems, pages 2555-2563.
Mingyuan Zhou and Lawrence Carin. 2015. Negative binomial process count and mixture modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):307-320.
Mingyuan Zhou, Oscar Hernan Madrid Padilla, and James G Scott. 2016. Priors for random count matrices derived from a family of negative binomial processes. Journal of the American Statistical Association, 111(515):1144-1156.
6169