SlowGuess/ACL_2020_OCR huggingface dataset:
Jiemin Wu $^{1,*}$ , Yanghui Rao $^{1,\dagger}$ , Zusheng Zhang $^{1}$ , Haoran Xie $^{2}$ , Qing Li $^{3}$ , Fu Lee Wang $^{4}$ , Ziye Chen $^{1}$
$^{1}$ School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China
$^{2}$ Department of Computing and Decision Sciences, Lingnan University, Hong Kong
$^{3}$ Department of Computing, The Hong Kong Polytechnic University, Hong Kong
$^{4}$ School of Science and Technology, The Open University of Hong Kong, Hong Kong
wujm29@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn,
hrxie2@gmail.com, csqli@comp.polyu.edu.hk,
pwang@ouhk.edu.hk, {zhangzsh3, chenzy35}@mail2.sysu.edu.cn
Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics. However, the existing parameter inference method like Monte Carlo sampling is quite time-consuming. In this paper, we propose two efficient neural mixed counting models, i.e., the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are developed to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Experiments on real-world datasets indicate that our models outperform state-of-the-art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generating dispersed proportions of document topics.
Mixture modeling is an essential topic in statistics and machine learning areas, owing to generating the random probability measure of data samples belonging to multiple clusters. In unsupervised learning tasks such as topic discovery, mixture modeling has gained increasing attention from researchers (Wang et al., 2011; Zhou and Carin, 2012, 2015; Zhou, 2018; Zhao et al., 2019). Specifically, mixture modeling over document words devotes to assign these words to different topics via random probability measures. Hierarchical Dirichlet
Process (HDP) (Teh et al., 2004) is one of the representative methods in mixture modeling, which can characterize the two-level dependency of random probability measures. Although we can use Monte Carlo sampling or variational inference to estimate the parameters in HDP, it requires the help of indirect construction of random variables such as the Chinese Restaurant Franchise (Teh et al., 2004) or the Stick-Breaking construction (Wang et al., 2011) due to the lack of conjugation between the two-tier Dirichlet processes. This makes the inference of HDP mostly complicated (Zhou et al., 2016).
The mixed counting models represented by the Negative Binomial (NB) process (Titsias, 2007) and the Gamma Negative Binomial (GNB) process (Zhou and Carin, 2012) have solved this problem to a certain extent, in which, the normalized GNB process has been proven to be equivalent to HDP (Zhou and Carin, 2012). Because both NB and GNB processes satisfy the properties of completely random measures (Charles Kingman, 1967), the generative process of random probability measures among various mixed components is independent and becomes straightforward. Moreover, they naturally introduce non-negative constraints and have been proven as able to model over-dispersed data. In the case of mining latent topics of documents, the over-dispersed property indicates that the variance is larger than the mean for document-topic distributions. When compared to the NB process, the GNB process has an extra feature of describing more flexible stochastic phenomena with hierarchical dependencies. Despite the above advantages, with the increase of data size and observable information, the aforementioned parameter inference method like Monte Carlo sampling or variational inference has gradually become an important factor limiting the usage scenarios of mixed counting models (Miao et al., 2016). The reason is that Monte Carlo sampling has a high computational
* The first two authors contributed equally to this work which was finished when Jiemin Wu was an undergraduate student of his final year.
† The corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6159-6169 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
cost, and variational inference becomes intractable when applied to models with complex variable dependencies (Acharya et al., 2015).
Neural variational inference (NVI) is a flexible and fast parameter inference framework based on neural networks (Mnih and Gregor, 2014). It can be regarded as a generalization of variational autoencoder applicable to natural language processing tasks. Based on NVI, several neural topic models had been proposed and achieved encouraging performance in document modeling (Miao et al., 2016; Srivastava and Sutton, 2017; Miao et al., 2017). These models used the neural network to learn the distribution relationship between input documents and latent topics due to its excellent function fitting ability and scalability. Particularly, the neural network parameters can be trained by back-propagation through the reparameterization of a continuous distribution (Naesseth et al., 2017) or using variance reduction techniques for a discrete distribution (Mnih and Gregor, 2014). However, the hidden variables in the above neural topic models lack good interpretability, and it is also impossible to model over-dispersed and hierarchically dependent document sets for these methods.
In this paper, we propose two novel neural mixed counting models dubbed the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) based on NB and GNB processes, respectively. The general motivation is to combine the advantages of NVI and mixed counting models. On the one hand, NVI-based models are fast and easy to estimate but hard to interpret. On the other hand, document modeling via mixed counting models is easy to interpret but difficult to infer. In our NB-NTM and GNB-NTM, we develop NVI algorithms to infer parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Extensive experiments on real-world datasets validate the effectiveness of our proposed models in perplexity, topic coherence, and dispersed topic learning. Furthermore, the proposed models can describe the hierarchical dependence of random probability measures and introduce non-negative constraints, which renders the intermediate variables generated by our methods to have good interpretability.
The remainder of this article is organized as follows. In Section 2, we summarize the related studies on topic discovery. In Section 3, we introduce
the definitions and properties of background methods. The proposed models are described in Section 4, the experimental evaluations are shown in Section 5, and we draw the conclusions in Section 6.
Topic discovery aims to use the statistical information of word occurrences to obtain the abstract semantic structure embedded in a document set. From Bayesian methods represented by latent semantic analysis (LSA) (Deerwester et al., 1990), probabilistic latent semantic analysis (PLSA) (Hofmann, 1999), latent Dirichlet allocation (LDA) (Blei et al., 2003), and Hierarchical Dirichlet Process (HDP) (Teh et al., 2004), topic discovery had been widely researched in natural language processing and applied to many scenarios. For instance, the above models were extended to capture topic relevance (Blei and Lafferty, 2005) and topic evolution over time (Wang and McCallum, 2006; Blei and Lafferty, 2006). Algorithms for short text (Yan et al., 2013), tagged data (Ramage et al., 2009), and stream data (Yao et al., 2009) were also proposed. Considering the importance of prior distributions in LDA-based models, some research efforts tried to use beta and Gaussian distributions instead of the Dirichlet distribution as the prior of probabilistic graphical models (Thibaux and Jordan, 2007; Das et al., 2015). Although the Bayesian method is a natural way to represent the latent structure of a document set in topic discovery, as the structure of such a model becomes deeper and more complex, pure Bayesian inference becomes intractable due to the high dimensional integrals required (Miao et al., 2016). To address this issue, Cheng and Liu (2014) proposed a parallel Monte Carlo sampling method for HDP based on multi-threading. Unfortunately, it needs to traverse every word of all topics (i.e., threads) in the whole corpus when updating the topic-word distribution, rendering a large time cost for thread communication.
With the development of deep learning, especially the introduction of NVI, there is a new direction to discover topics based on neural networks. For example, Miao et al. (2016) assumed that word distributions in each document could be represented by hidden variables sampled from multiple Gaussian distributions, and they used the variational lower bound as the objective function of their model named NVDM. Srivastava and Sutton (2017) employed the logical Gaussian distribution to approxi
mate the Dirichlet distribution, which improved the variational auto-encoder and LDA simultaneously. Miao et al. (2017) proposed a method named GSM to model the document-topic distribution explicitly. In their study, the topic-word distribution was introduced into the decoder. Besides the above NVi-based methods, Nalisnick and Smyth (2017) developed a stick-breaking variational auto-encoder for image generation. Nan et al. (2019) proposed a model named W-LDA in the Wasserstein autoencoder framework. They employed the Maximum Mean Discrepancy (MMD) in W-LDA to match the proposed distribution and the prior distribution. However, the accuracy of MMD relied heavily on the number of samples for each distribution, and the kernel function in MMD had a significant influence on the performance. By leveraging word embeddings, Gupta et al. (2019) proposed a neural autoregressive topic model dubbed iDocNADE to enrich the context of short text. Experiments indicate that iDocNADE outperformed state-of-the-art generative topic models.
The recent relevant work to ours is the method proposed in (Zhao et al., 2019), which regarded the NB distribution as the prior in modeling the over-dispersed discrete data. However, the parameters of this method were still derived from the latent variables that obey the Gaussian distribution. Thus, these latent variables do not satisfy the non-negative constraint and lack good interpretability. Furthermore, the above method did not model topics explicitly, making it hard to generate document-topic and topic-word distributions.
3.1 Negative Binomial Process
Let  $X \sim \mathrm{NBP}(G_0, p)$  denote a NB process defined on the product space  $\mathbb{R}_+ \times \Omega$ , where  $G_0$  is a finite continuous basic measure on a completely separable measure space  $\Omega$ , and  $p$  is a scale parameter. For each Borel set  $A \subset \Omega$ , we use  $X(A)$  to denote a count random variable describing the number of observations that reside within  $A$ . Then,  $X(A)$  obeys the NB distribution  $\mathrm{NB}(G_0(A), p)$ . Given the  $k^{th}$  component  $\pi_k$  and its weight  $r_k$  on  $\Omega$ , if  $G_0$  is expressed as  $G_0 = \sum_{k=1}^{\infty} r_k \delta_{\pi_k}$ , where  $\delta$  is the Dirac delta function, then  $X \sim \mathrm{NBP}(G_0, p)$  can be expressed by  $X = \sum_{k=1}^{\infty} n_k \delta_{\pi_k}$ , where  $n_k \sim \mathrm{NB}(r_k, p)$ .
The NB distribution  $m\sim \mathrm{NB}(r,p)$  has a probability density function  $f_{M}(m) = \frac{\Gamma(r + m)}{m!\Gamma(r)} (1 -$
$p)^r p^m$ , where  $\Gamma(\cdot)$  denotes the gamma function. For the above probability density function, the mean and the variance are  $\mu = r / (1 - p)$  and  $\sigma^2 = rp / (1 - p)^2 = \mu + r^{-1}\mu^2$ , respectively. Because the mean is smaller than the variance, i.e., the variance-to-mean ratio is greater than 1, NB distributions have shown great advantages in over-dispersed data modeling (Zhou and Carin, 2012). Moreover, since the NB distribution  $m \sim \mathrm{NB}(r,p)$  can be extended to a Gamma distribution and a Poisson distribution, i.e.,  $m \sim \mathrm{Poisson}(\lambda)$  and  $\lambda \sim \mathrm{Gamma}(r,p / (1 - p))$ , the NB process mentioned earlier can be extended to a Gamma-Poisson process (Zhou and Carin, 2015) as follows:  $X \sim \mathrm{PP}(\Lambda)$ , and  $\Lambda \sim \mathrm{GaP}(G_0, (1 - p) / p)$ , where  $\mathrm{PP}(\cdot)$  and  $\mathrm{GaP}(\cdot)$  denote the Poisson process and the Gamma process, respectively. The random probability measure corresponding to each mixed component in the NB process can be directly sampled from the NB distribution without resorting to the Chinese Restaurant Franchise, the Stick-Breaking, or other construction methods, because each random measure is independent of the others, i.e., the NB process is completely random.
3.2 Gamma Negative Binomial Process
In the NB process, each Poisson process shares the same Gamma process prior with a fixed mean. Based on the NB process, the GNB process assigns another Gamma process as a prior to its mean, making it easier to model over-dispersed data (Zhou et al., 2016). Particularly, the generative process of random variables for the GNB process is as follows:  $G \sim \mathrm{GaP}(G_0, \eta)$ ,  $\Lambda_j \sim \mathrm{GaP}(G, (1 - p_j) / p_j)$ , and  $X_j \sim \mathrm{PP}(\Lambda_j)$ , where  $j$  is the subset index,  $\eta$  is the scale parameter of the first-level Gamma process, and the basic measure  $G_0$  in the NB process is replaced by another random measure  $G$ . It has been shown that HDP is a normalized form of the GNB process in (Zhou and Carin, 2012). However, unlike HDP, the GNB process explicitly introduces the parameter  $p_j$  to control the dispersion degree of instantaneous measurement, making the latter model more flexible.
3.3 Neural Variational Inference
NVI is often used as an efficient parameter inference framework for complex and deep-seated structural models. Inspired by the variational autoencoder, NVI assumes that the observed data  $\pmb{d}$  is subject to a certain probability distribution determined by a hidden variable  $h$ . In contrast to
variational auto-encoders on handling the case of continuous latent variables (Kingma and Welling, 2014), NVI can deal with both discrete and continuous latent variables. Specifically, a neural network is used to infer the proposed distribution  $q(h|d)$ . As stated in (Miao et al., 2017), Monte Carlo estimates of the gradient must be employed for models with discrete latent variables. In the case of  $q(h|d)$  being continuous, the hidden variable  $h$  is firstly obtained by sampling from  $q(h|d)$  through the corresponding reparameterization approach. Then, the likelihood  $p(d|h)$  is used to reproduce the observed data from hidden variables, and the objective is to minimize the Kullback-Leibler (KL) divergence of the proposed distribution and the actual posterior distribution. Finally, the variational lower bound is obtained by  $\mathcal{L} = \mathbb{E}_{q(h|d)}\log p(d|h) - D_{\mathrm{KL}}[q(h|d)\| p(h)]$ , where the first term is the expectation of the log-likelihood, and the second one is the KL divergence between the inferred distribution and a predefined prior. To sum up, NVI first uses a neural network to infer the proposed distribution  $q(h|d)$ , and then maximizes the variational lower bound by backpropagation to fit the actual posterior distribution  $p(h|d)$ . Such a framework learns the distribution of input data well, enabling it to combine with the traditional probability graphical models (e.g., LDA) and infer model parameters quickly (Srivastava and Sutton, 2017). However, how to effectively integrate the distributed dependencies in mixed counting models into the framework of variational inference is still quite a challenging problem.
4 Proposed Models
In this section, we respectively detail our NB-NTM and GNB-NTM for dispersed topic discovery.
4.1 Negative Binomial-Neural Topic Model
With a NB process prior, we propose the NB-NTM to model the counting of document words. Furthermore, a novel NVI framework is developed for parameter inference. Let  $D = \{d^{1},\dots,d^{|D|}\}$  be the input with  $|D|$  documents and each document  $d\in \mathbb{R}^V$  be a bag-of-words representation, where  $V$  is the vocabulary size. Since it is impossible to draw all the countably infinite atoms of a Gamma process, we first employ the finite truncation strategy, in which, a number of topics  $K$  (i.e., the truncated level) is set manually (Nalisnick and Smyth, 2017; Zhou, 2018). Note that although  $K$  is fixed,
if  $K$  is set to be large enough, not necessarily all topics would be used and hence a truncated model still preserves its nonparametric ability; whereas if  $K$  is set to be small, asymmetric priors on the topic weights are also maintained (Zhou, 2018). Then we can express the generative process of NB-NTM for document  $\pmb{d}$  as follows:
$$
\boldsymbol {r} = f _ {1} (\boldsymbol {d}), \quad \boldsymbol {p} = f _ {2} (\boldsymbol {d}), \tag {1}
$$
$$
\boldsymbol {\lambda} \sim \mathrm {G a m m a} \left(\boldsymbol {r}, \boldsymbol {p} / \left(\mathbf {1} - \boldsymbol {p}\right)\right), \qquad (2)
$$
$$
\boldsymbol {n} \sim \text {P o i s s o n} (\boldsymbol {\lambda}), \tag {3}
$$
where  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$  are two multilayer perceptrons (MLPs) applying to generate the variational parameters  $\pmb{r}$  and  $\pmb{p}$ . Specifically,  $\pmb{r}$  is the component weight of  $G$ , i.e., the topic measure at the corpus level, and  $G = \sum_{k=1}^{K} r_{k} \delta_{\pi_{k}}$ .  $\lambda$  represents the weights of topics at the document level, which can be used to estimate the topic measure on  $\pmb{d}$  by  $\Lambda = \sum_{k=1}^{K} \lambda_{k} \delta_{\pi_{k}}$ . In the above,  $\lambda_{k}$  denotes the  $k^{th}$  component of  $\lambda$ . Finally,  $\pmb{n}$  is the component weight of  $\Pi$  that represents a Poisson process at the word level, and  $\Pi = \sum_{k=1}^{K} n_{k} \delta_{\pi_{k}}$ . The framework of NB-NTM is shown in Figure 1, and the parameter inference process is described as follows.
Figure 1: Framework of NB-NTM.
For the logarithmic likelihood of each document  $\pmb{d}$ , we can derive the variational lower bound by  $\mathcal{L} = -D_{KL}(q(\pmb{\lambda}|\pmb{d})||p(\pmb{\lambda})) + \mathbb{E}_{q(\pmb{\lambda}|\pmb{d})}\left[\sum_{i=1}^{N_d}\log p(\pmb{\omega}_i|\pmb{\lambda})\right]$ . In the above,  $q(\pmb{\lambda}|\pmb{d})$  is the encoder's inference of posterior probability, i.e., Gamma  $(\pmb{r},\pmb{p})$ ,  $\pmb{\omega}_i \in \mathbb{R}^V$  is the one-hot representation of the word at the  $i^{th}$  position,  $N_d$  is the number of words in document  $\pmb{d}$ , and  $p(\pmb{\lambda})$  is the Gamma prior for  $\pmb{\lambda}$ , i.e., Gamma  $(\pmb{\xi},\pmb{c})$ . The KL divergence between  $q(\pmb{\lambda}|\pmb{d})$  and  $p(\pmb{\lambda})$ , i.e., Gamma  $(\pmb{r},\pmb{p})$  and Gamma  $(\pmb{\xi},\pmb{c})$ , is calculated by following (Mathiassen et al., 2002):  $D_{KL}(q(\pmb{\lambda}|\pmb{d})||p(\pmb{\lambda})) = \sum_{k=1}^{K}[(r_k - 1)\Psi(r_k) -$
6162
$\log p_k - r_k - \log \Gamma (r_k) - (\pmb {\xi} - 1)(\Psi (r_k) + \log p_k) +$ $\log \Gamma (\pmb {\xi}) + \pmb {\xi}\log \pmb {c} + \frac{r_kp_k}{\pmb{c}} ],$  where  $\Psi (\cdot)$  is the Digam.   
ma function. The conditional probability over each   
word  $p(\omega_i|\lambda)$  is modeled by softmax function,   
as follows:  $p(\omega_i|\lambda) = \frac{\exp\{\sigma(\pmb{n}^T R\pmb{\omega}_i + \pmb{b}_i)\}}{\sum_{j = 1}^{V}\exp\{\sigma(\pmb{n}^T R\pmb{\omega}_j + \pmb{b}_j)\}},$    
where  $R$  and  $\textit{\textbf{b}}$  denote the weight matrix and the   
bias term, respectively. We present the parameter   
inference process of NB-NTM in Algorithm 1, in   
which, the variational lower bound  $\mathcal{L}$  is used to cal  
calculate gradients and model parameters are updated   
by Adam (Kingma and Ba, 2015).
Output: Document-topic distribution  $\theta$ , topic-word distribution  $\phi$ .
4.2 Gamma Negative Binomial-Neural Topic Model
Based on the NB-NTM, we further propose the GNB-NTM by assigning another Gamma process as a prior to the NB process. As shown in Figure 2, the generative process of GNB-NTM for document
$\pmb{d}$  is given below:
$$
\gamma = f _ {1} (\boldsymbol {d}), \quad \eta = f _ {2} (\boldsymbol {d}), \tag {4}
$$
$$
\boldsymbol {r} \sim \operatorname {G a m m a} (\gamma , \eta), \tag {5}
$$
$$
\boldsymbol {\lambda} \sim \operatorname {G a m m a} (\boldsymbol {r}, p / (1 - p)), \tag {6}
$$
$$
p = f _ {3} (\boldsymbol {d}), \quad \boldsymbol {n} \sim \text {P o i s s o n} (\boldsymbol {\lambda}). \tag {7}
$$
In the above,  $\gamma$  and  $\eta$  are the parameters of the first-level Gamma process, and  $p$  is the scale parameter of the second-level Gamma process. The differences between GNB-NTM and NB-NTM are three-fold. Firstly, another Gamma process  $G_{0}$  is introduced over the existing Gamma process  $G$  as a prior of its shape parameter, so as to characterize the multi-level dependencies of random variables. In particular,  $G_{0} = \sum_{k=1}^{K} \gamma_{k} \delta_{\pi_{k}}$ . Secondly, a scale parameter  $p$  is introduced for each document to describe the dispersion degree of all words in the document. Thirdly, the GNB-NTM employs  $\boldsymbol{n} + \boldsymbol{r}$  as the input of the decoder by following the production rule of the observed variable in (Zhou and Carin, 2012). Using  $\boldsymbol{n} + \boldsymbol{r}$  as the input also helps to incorporate the global topic information into the decoder's inference of posterior probability  $q(\boldsymbol{r}|\boldsymbol{d})$ . Thus, the conditional probability over each word  $p(\omega_{i}|\boldsymbol{r})$  is modeled as follows:
$$
p (\boldsymbol {\omega} _ {i} | \boldsymbol {r}) = \frac {e x p \{\sigma ((\boldsymbol {n} + \boldsymbol {r}) ^ {T} R \boldsymbol {\omega} _ {i} + \boldsymbol {b} _ {i}) \}}{\sum_ {j = 1} ^ {V} e x p \{\sigma ((\boldsymbol {n} + \boldsymbol {r}) ^ {T} R \boldsymbol {\omega} _ {j} + \boldsymbol {b} _ {j}) \}}.
$$
Figure 2: Framework of GNB-NTM, where both  $r$  and  $n$  are used as input in the decoder.
Similar to NB-NTM, the variational lower bound is derived by:  $\mathcal{L} = \mathbb{E}_{q(\boldsymbol{r}|\boldsymbol{d})}\left[\sum_{i=1}^{N_d}\log p(\boldsymbol{\omega}_i|\boldsymbol{r})\right] - D_{KL}(q(\boldsymbol{r}|\boldsymbol{d})||p(\boldsymbol{r}))$ , where  $p(\boldsymbol{r})$  is the Gamma prior for  $\boldsymbol{r}$ , i.e.,  $\mathrm{Gamma}(\xi, c)$ . The parameter inference for GNB-NTM is presented in Algorithm 2. We use the variational lower bound to calculate gradients and apply Adam to update parameters of GNB-NTM, which are the same as NB-NTM.
4.3 Reparameterization Approach
The Gamma and Poisson sampling operation cannot be differentiated, making it intractable to up-
6163
date model parameters through back-propagation. Here, we describe the reparameterization approach for smoothing gradients. For the Gamma distribution  $x \sim \mathrm{Gamma}(\alpha, \beta)$  with  $\alpha > 1$ , the reparameterization can be obtained by the reject-sampling method (Naesseth et al., 2017), i.e.,  $x = \frac{1}{\beta}\left(\alpha - \frac{1}{3}\right)\left(1 + \frac{\varepsilon}{\sqrt{9\alpha - 3}}\right)^3$ ,  $\epsilon \sim \mathcal{N}(0,1)$ . Besides, the shape augmentation method (Naesseth et al., 2017) is applied to convert  $\alpha \leq 1$  to  $\alpha > 1$  to increase the accept rate of each rejection sampler. For the Poisson distribution which is discrete, we use the Gaussian distribution as an approxima