Venue: NAACL 2021

# Inductive Topic Variational Graph Auto-Encoder for Text Classification

Qianqian Xie

Department of Computer Science,

University of Manchester

qianqian.xie@manchester.ac.uk &Jimin Huang

School of Computer Science,

Wuhan University

huangjimin@whu.edu.cn Pan Du

Department of Computer Science and Operations Research, University of Montreal

pandu@iro.umontreal.ca &Min Peng

School of Computer Science,

Wuhan University

pengm@whu.edu.cn &Jian-Yun Nie

Department of Computer Science and Operations Research,

University of Montreal

nie@iro.umontreal.ca

###### Abstract

Graph convolutional networks (GCNs) have been applied recently to text classification and produced an excellent performance. However, existing GCN-based methods do not assume an explicit latent semantic structure of documents, making learned representations less effective and difficult to interpret. They are also transductive in nature, thus cannot handle out-of-graph documents. To address these issues, we propose a novel model named inductive Topic Variational Graph Auto-Encoder (T-VGAE), which incorporates a topic model into variational graph-auto-encoder (VGAE) to capture the hidden semantic information between documents and words. T-VGAE inherits the interpretability of the topic model and the efficient information propagation mechanism of VGAE. It learns probabilistic representations of words and documents by jointly encoding and reconstructing the global word-level graph and bipartite graphs of documents, where each document is considered individually and decoupled from the global correlation graph so as to enable inductive learning. Our experiments on several benchmark datasets show that our method outperforms the existing competitive models on supervised and semi-supervised text classification, as well as unsupervised text representation learning. In addition, it has higher interpretability and is able to deal with unseen documents.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4218–4227

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Recently, graph convolutional networks (GCNs)Kipf and Welling (2017); Velickovic et al. (2018) have been successfully applied to text classification tasks Peng et al. (2018a); Yao et al. (2019); Liu et al. (2020); Wang et al. (2020). In addition to the local information captured by CNN or RNN, GCNs learn word and document representations by taking into account the global correlation information embedded in the corpus-level graph, where words and documents are nodes connected by indexing or citation relations.

However, the **hidden semantic structures**, such as latent topics in documents Blei et al. (2003); Yan et al. (2013); Peng et al. (2018b), is still ignored by most of these methods Yao et al. (2019); Huang et al. (2019); Liu et al. (2020); Zhang et al. (2020), which can improve the text representation and provide extra **interpretability** (in which the probabilistic generative process and topics make more sense to humans compared to neural networks, i.e. topics can be visually represented by top-10 or 20 most probable word clusters). Although few studies such as Wang et al. (2020) have proposed incorporating a topic structure into GCNs, the topics are extracted in advance from the set of documents, independently from the graph and information propagation among documents and words. We believe that the topics should be determined in accordance with the connections in the graph. For example, the fact that two words are connected provides extra information that these words are on a similar topic(s). Moreover, existing GCN-based methods are limited by their transductive learning nature, i.e. a document can be classified only if it is already seen in the training phase Wang et al. (2020); Yao et al. (2019); Liu et al. (2020). The lack of **inductive learning** ability for unseen documents is a critical issue in practical text classification applications, where we have to deal with new documents. It is

intuitive to decouple documents with the global graph and treat each document as an independent graph Huang et al. (2019); Zhang et al. (2020); Ding et al. (2020); Zhang and Zhang (2020); Xie et al. (2021). However, no attempt has been made to address both aforementioned issues.

To address these issues, we incorporate the topic model into variational graph auto-encoder (VGAE), and propose a novel framework named inductive Topic Variational Graph Auto-Encoder (T-VGAE). T-VGAE first learns to represent the words in a latent topic space by embedding and reconstructing the word correlation graph with the GCN probabilistic encoder and probabilistic decoder. Take the learned word representations as input, a GCN-based message passing probabilistic encoder is adopted to generate document representations via information propagation between words and documents in the bipartite graph. We compare our model with existing related work in Table 1. Different from previous approaches, our method unifies topic mining and graph embedding learning with VGAE, thus can fully embed the relations between documents and words into dynamic topics and provide interpretable topic structures into representations. Besides, our model builds a document-independent word correlation graph and a word-document bipartite graph for each document instead of a corpus-level graph to enable inductive learning.

The main contributions of our work are as follows:

1. We propose a novel model T-VGAE based on topic models and VGAE, which incorporates latent topic structures for inductively document and word representation learning. This makes the model more effective and interpretable.
2. we propose to utilize the auto-encoding variational Bayes (AEVB) method to make efficient black-box inference of our model.
3. Experimental results on benchmark datasets demonstrate that our method outperforms the existing competitive GCN-based methods on supervised and semi-supervised text classification tasks. It also outperforms topic models on unsupervised text representation learning.

## 2 Related Work

### Graph based Text Classification

Recently, GCNs have been applied to various NLP tasks Zhang et al. (2018); Vashishth et al. (2019). For example, TextGCN Yao et al. (2019) was proposed for text classification, which enriches the corpus-level graph with the global semantic information to learn word and document embeddings. Inspired by it, Liu et al. Liu et al. (2020) further considered syntactic and sequential contextual information and proposed TensorGCN. However, none of them utilized the latent semantic structures in the documents to enhance text classification. To address the issue, Wang et al. (2020) proposed dynamic HTG (DHTG), in an attempt to integrate the topic model into graph construction. DHTG learned latent topics from the document-word correlation information (similar to traditional topic models), which will be used for GCN based document embedding. However, the topics in DHTG were learned independently from the word relation graph and the information propagation process in the graph, in which word relations are ignored.

Moreover, the existing GCN-based methods also require a pre-defined graph with all the documents and cannot handle out-of-graph documents, thus limiting their practical applicability.

To deal with the inductive learning problem, Huang et al. (2019); Zhang et al. (2020); Ding et al. (2020); Zhang and Zhang (2020) proposed to consider each document as an independent graph for text classification. However, the latent semantic structure and interpretability are still ignored in these methods. Different from previous approaches, we aim to deal with both issues of dynamic topic structure and inductive learning. We propose to combine the topic model and graph based information propagation in a unified framework with VGAE to learn interpretable representations for words and documents.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline Model & Explainability & Learning & Topics \\ \hline TextGCN Xiao et al. (2019) & - & transductive & - \\ TenroGCN Liu et al. (2020) & - & transductive & - \\ DIFTS Wang et al. (2020) & ✓ & transductive & static \\ T-FGN Chung et al. (2019) & - & inductive & - \\ TG-Trans Zhang and Zhang (2020) & - & inductive & - \\ TextMJ Chang et al. (2020) & - & inductive & - \\ HyperGAP Cheng et al. (2020) & - & inductive & - \\ Our model & \(\surd\) & inductive & dynamic \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison with related work. We compare the manner of model learning, whether incorporate the latent topic structure and the manner of topic learning of these models.

 



### Graph Enhanced Topic Models

There are also studies trying to enhance topic models with efficient message passing in the graph data structure of GCNs. GraphBTM (Zhu et al., 2018) proposed to enrich the biterm topic model (BTM) with the word co-occurrence graph encoded with GCNs. To deal with data streams, (Van Linh et al., 2020) proposed graph convolutional topic model (GCTM), which introduces a knowledge graph modeled with GCNs to the topic model. (Yang et al., 2020) presented Graph Attention TOpic Network (GATON) for correlated topic modeling. It tackles the overfitting issue in topic modeling with a generative stochastic block model (SBM) and GCNs. In contrast with these studies, we focus on integrating the topic model into GCN-based VGAE for supervised learning tasks and derive word-topic and document-topic distributions simultaneously.

### Variational Graph Auto-encoders

Variational Graph Auto-encoders (VGAEs) have been widely used in graph representation learning and graph generation. The earliest study (Kipf and Welling, 2016) proposed VGAE method, which extended variational auto-encoder (VAE) on graph structure data for learning graph embedding. Based on VGAE, (Pan et al., 2018) introduced an adversarial training to regularize the latent variables and further proposed adversarially regularized variational graph autoencoder (ARVGA). (Hasanzadeh et al., 2019) incorporated semi-implicit hierarchical variational distribution into VGAE (SIG-VAE) to improve the representation power of node embeddings. (Grover et al., 2019) proposed Graphite model that integrated an iterative graph refinement strategy into VGAE, inspired by low-rank approximations. However, to the best of our knowledge, our model is the first effort to apply VGAE to unify the topic learning and graph embedding for text classification, thus can provide better interpretability and overall performance.

## 3 Method

### Graph Construction

Formally, we denote a corpus as \(C\), which contains \(D\) documents and the ground truth labels \(Y\in c=\{1,...,M\}\) of documents, where \(M\) is the total number of classes in the corpus. Each document \(t\in C\) is represented by a sequence of words \(t=\{w_{1},...,w_{n_{t}}\}(w_{i}\in v)\), where \(n_{t}\) is the number of words in document \(t\) and \(v\) is the vocabulary of size \(V\).

From the whole corpus, we build a word correlation graph \(G=(v,e)\) containing word nodes \(v\) and edges \(e\), to capture the word co-occurrence information. Similar to previous work (Yao et al., 2019), we utilize the positive point mutual information (PPMI) to calculate the correlation between two word nodes. Formally, for two words \((w_{i},w_{j})\), we have

\[PMI(w_{i},w_{j})=\max(log\frac{p(w_{i},w_{j})}{p(w_{i})p(w_{j})},0) \tag{1}\]

where \(p(w_{i},w_{j})\) is the probability that \((w_{i},w_{j})\) co-occur in the sliding window and \(p(w_{i}),p(w_{j})\) are the probabilities of words \(w_{i}\) and \(w_{j}\) in the sliding window. They can be empirically estimated as \(P(w_{i},w_{j})=\frac{n(w_{i},w_{j})}{n}\) and \(P(w_{i})=\frac{n(w_{i})}{n}\), where \(n(w_{i},w_{j})\) is the number of co-occurrences of \((w_{i},w_{j})\) in the sliding windows, \(n(w_{i})\) is the number of occurrences of \(w_{i}\) in the sliding windows and \(n\) the total number of sliding windows. For two word nodes \((w_{i},w_{j})\), the weight of the edge between them can be defined as:

\[A^{v}_{i,j}=\begin{cases}PPMI(w_{i},w_{j}),&i\neq j\\ 1,&i=j\end{cases} \tag{2}\]

where \(A^{v}\in\mathbb{R}^{V*V}\) is the adjacency matrix which represents the word correlation graph structure \(G\).

Different from the existing studies (Yao et al., 2019; Liu et al., 2020; Wang et al., 2020) that consider all documents and words in a heterogeneous graph, we propose to build a separate graph for each document to enable inductive learning. Typically, documents can be represented by the document-word matrix \(A^{d}\in\mathbb{R}^{D\times V}\), in which the row \(A^{d}_{i}=\{x_{i1},...,x_{iv}\}\in\mathbb{R}^{1\times V}\) represents the document \(i\), and \(x_{ij}\) is the TF-IDF weight of the word \(j\) in document \(i\). The decoupling of documents from a global pre-defined graph enables our method to handle new documents.

### Topic Variational Graph Auto-encoder

Based on \(A^{v}\) and \(A^{d}\), we propose the T-VGAE model, as shown in Figure 1. It is a deep generative model with structured latent variables based on GCNs.

#### 3.2.1 Generative Modeling

We consider that the word co-occurrence graph \(A^{v}\) and the bipartite graph \(A^{d}_{t}\) of each document \(t\) are generated from the random process with two latent

variables \(z^{v}\in\mathbb{R}^{V\times K}\) and \(z_{t}^{d}\in\mathbb{R}^{1\times K}\), where \(K\) denotes the number of latent topics. The generating process for \(A^{v}\), \(A^{d}\) and \(Y\) are as follows (see Figure 2(a)):

1. For each word \(i\) in vocabulary \(v\), draw the latent variable \(z_{i}^{v}\) from the prior \(p_{\theta}(z_{i}^{v})\)
2. For each observed edge \(A_{i,j}^{v}\) between words \(i\) and \(j\), draw \(A_{i,j}^{v}\) from conditional distribution \(p_{\theta}(A_{i,j}^{v}|z_{i}^{v},z_{j}^{v})\)
3. For each document \(t\) in corpus \(C\): 1. Draw the latent variable \(z_{t}^{d}\) from the prior \(p_{\theta}(z_{t}^{d})\) 2. Draw \(A_{t}^{d}\) from the conditional distribution \(p_{\theta}(A_{t}^{d}|z_{t}^{d},z^{v})\) 3. Draw \(Y_{t}\) from the conditional distribution \(p_{\theta}(Y_{t}|z_{t}^{d})\)

where \(\theta\) is the set of parameters for all prior distributions. Here, we consider the centered isotropic multivariate Gaussian priors \(p(z^{v})=\prod_{i=1}^{V}p(z_{i}^{v})=\prod_{i=1}^{V}\mathcal{N}(z_{i}^{v}|0,I)\) and \(p(z^{d})=\prod_{t=1}^{D}p(z_{t}^{d})=\prod_{t=1}^{D}\mathcal{N}(z_{t}^{d}|0,I)\).

Notice that the priors \(p(z^{v})\) and \(p(z^{d})\) are parameter free in this case. According to the above generative process, we can maximize the marginal likelihood of observed graph \(A^{v}\), \(A^{d}\) and \(Y\) to learn parameters \(\theta\) and latent variables as follows:

\[\begin{split}& p(A^{v},A^{d},Y|Z^{v},Z^{d},X^{v})=\\ &\prod_{t=1}^{D}p_{\theta}(Y_{t}|z_{t}^{d})p_{\theta}(A_{t}^{d}| z_{t}^{d},z^{v})p_{\theta}(z_{t}^{d})\\ &\prod_{i=1}^{V}\prod_{j=1}^{V}p_{\theta}(A_{i,j}^{v}|z_{i}^{v}( z_{j}^{v})^{\mathrm{T}})p_{\theta}(z^{v})\end{split} \tag{3}\]

Because the inference of true posterior of latent variable \(z^{v}\) and \(z^{d}\) is intractable, we further introduce the variational posterior distribution \(q_{\phi}(z^{v},z^{d}|A^{d},A^{v},X^{v})\) with parameters \(\phi\) to approximate the true posterior \(p_{\theta}(z^{v},z^{d})=p_{\theta}(z^{v})p_{\theta}(z^{d})\). We make the structured mean-field (SMF) assumption \(q_{\phi}(z^{v},z^{d}|A^{d},A^{v},X^{v})=q_{\phi}(z^{v}|A^{v},X^{v})q_{\phi}(z^ {d}|A^{d},z^{v})\), where \(X^{v}\in\mathbb{R}^{V\times M}\) are the feature vectors of words and \(M\) is the dimension of the feature vectors (see Figure 2(b)). We can yield the following tractable stochastic evidence lower bound (ELBO):

\[\begin{split}&\mathcal{L}(\theta,\phi;A^{v},A^{d},X^{v})\\ &\quad=\mathbb{E}_{q_{\phi}(z^{v}|A^{v},X^{v})}[\log p_{\theta}( A^{v}|z^{v})]\\ &+\mathbb{E}_{q_{\phi}(z^{d}|A^{d},z^{v})}[\log p_{\theta}(A^{ d}|z^{d},z^{v})]\\ &+\mathbb{E}_{q_{\phi}(z^{d}|A^{d},z^{v})}[\log p_{\theta}(Y|z^{ d})]\\ &-KL[q_{\phi}(z^{v}|A^{v},X^{v})||p_{\theta}(z^{v})]\\ &-KL[q_{\phi}(z^{d}|A^{d},z^{v})||p_{\theta}(z^{d})]\end{split} \tag{4}\]

where the first three terms are the reconstruction terms, and the latter two terms are the Kullback-Leibler (KL) divergences of variational posterior

Figure 1: The architecture of T-VGAE.As shown in the Figure, for a new test document \(i\), its latent representation \(z_{i}^{d}\) is generated by the UAMP probabilistic encoder based on its document-word vector \(A_{i}^{d}\) and learned word topic distribution matrix \(z^{v}\). Then, \(z_{i}^{d}\) is fed into the trained MLP classifier \(f_{y}\) to predict the output label. Therefore, new test documents can be classified do not need to be included in the training process, thus enabling inductive learning of our model.

Figure 2: The generative and inference processes.

 

distributions and true posterior distributions. Using auto-encoding variational Bayes (AVB) approach Kingma and Welling (2013), we are able to parametrize the variational posteriors \(q_{\phi}\) and true posteriors \(p_{\theta}\) with the GCN-based probabilistic encoder and decoder, to conduct neural variational inference (NVI).

#### 3.2.2 Graph Convolutional Probabilistic Encoder

For the latent variable \(z^{v}\), we make the mean-field approximation that: \(q_{\phi}(z^{v}|A^{v},X^{v})=\prod_{i=1}^{V}q_{\phi}(z^{v}_{i}|A^{v},X^{v})\). For simplify the model inference, we consider the multivariate normal variational posterior with a diagonal covariance matrix as previous neural topic models Miao et al. (2016); Bai et al. (2018)that: \(q_{\phi}(z^{v}_{i}|A^{v},X^{v})=\mathcal{N}(z^{v}_{i}|\mu^{v}_{i},diag((\sigma ^{v}_{i})^{2}))\), where \(\mu^{v}_{i},(\sigma^{v}_{i})^{2}\) are the mean and diagonal covariance of the multivariate Gaussian distribution.

We use the graph convolutional neural network to parametrize the above posterior and inference \(z^{v}\) with the input graph \(A^{v}\) and feature vectors \(X^{v}\):

\[(H^{v})^{l+1} =\rho(\hat{A}^{v}(H^{v})^{l}(W^{v})^{l}) \tag{5}\] \[\mu^{v} =\rho(\hat{A}^{v}(H^{v})^{l+1}(W^{v}_{\mu})^{l+1})\] \[logg^{\circ} =\rho(\hat{A}^{v}(H^{v})^{l+1}(W^{v}_{\sigma})^{l+1})\]

where \(\mu^{v},\sigma^{v}\) are matrices of \(\mu^{v}_{i},\sigma^{v}_{i}\), \(l\) is the number of GCN layers, we use one layer in our experiments, \(\{W^{v}_{\mu},W^{v}_{\sigma}\}\in\phi\) are weight matrices, \(\rho\) is the ReLU, \(\hat{A}^{v}=(D^{v})^{-\frac{1}{2}}A^{v}(D^{v})^{-\frac{1}{2}}\) is the symmetrically normalized adjacent matrix of the word graph, and \(D^{v}\) denotes the corresponding degree matrix. The input of GCN is the feature vectors \(X_{v}\) which is initialized as the identity matrix \(I\), i.e., \((H^{v})^{0}=X^{v}=I\), same as in Yao et al. (2019). Then, \(z^{v}\) can be naturally sampled as follows according to the reparameterization trick Kingma and Welling (2013): \(z^{v}=\mu^{v}+\sigma^{v}\odot\epsilon\), where \(\odot\) is the element-wise product, and \(\epsilon\sim\mathcal{N}(0,I)\) is the noise variable. Through the message propagation of the GCN layer, words that co-occur frequently tend to achieve similar representations in the latent topic space.

Similar to \(z^{v}\), we also have:

\[q_{\phi}(z^{d}|A^{d},z^{v}) =\prod_{t=1}^{D}q_{\phi}(z^{d}_{t}|A^{d}_{t},z^{v}) \tag{6}\] \[q_{\phi}(z^{d}_{t}|A^{d}_{t},z^{v}) =\mathcal{N}(z^{d}_{t}|\mu^{d}_{t},diag((\sigma^{d}_{t})^{2}))\]

where \(\mu^{d}_{t},(\sigma^{d}_{t})^{2}\) are the mean and diagonal covariance of the multivariate Gaussian distribution. Although there are two types of nodes - word and document - in the bipartite graph \(A^{d}\), we mainly focus on learning representations of document nodes based on the representations of word nodes learned from \(A^{v}\) in this step. Therefore, we propose the unidirectional message passing (UDMP) process on \(A^{d}\), which propagates the information from word nodes to documents: \(H^{d}_{t}=\rho(\sum_{i=1}^{V}A^{d}_{ti}z^{v}_{i}W^{d})\) where \(\rho\) is the Relu activation function, \(W^{d}\) is the weight matrix.

Then, we parametrize the posterior and inference \(z^{d}\) based on UDMP:

\[\begin{split}\mu^{d}&=UMDP(A^{d},z^{v},W^{d}_{\mu})\\ logg^{d}&=UDMP(A^{d},z^{v},W^{d}_{\sigma})\end{split} \tag{7}\]

where \(\mu^{d},\sigma^{d}\) are matrices of \(\mu^{d}_{t},(\sigma^{d}_{t})^{2}\), \(UDMP\) is the message passing as in Equation 4, \(W^{d}_{\mu},W^{d}_{\sigma}\) are weight matrices. Similarly, we sample \(z^{d}\) as follows \(z^{d}=\mu^{d}+\sigma^{d}\odot\varepsilon\), where \(\varepsilon\sim\mathcal{N}(0,I)\) is the noise variable. Through the propagation mechanism of UDMP, documents which share similar words tend to yield similar representations in the latent topic space.

Although T-VGAE can learn topics \(z^{v}\) and document-topic representations \(z^{d}\) as in traditional topic models, we do not focus on proposing a novel topic model, but aim to combine the topic model with VGAE, to improve word and document representations with latent topic semantic and provide probabilistic interpretability. Moreover, rather than learning topics and document-topic representations from the document-word feature \(A^{d}\) as LDA topic models Blei et al. (2003), we propose to learn word-topic representations \(z^{v}\) from word co-occurrence matrix \(A^{v}\), and then infer document-topic representations \(z^{d}\) based on the document-word feature \(A^{d}\) and word-topic representations \(z^{v}\), which is similar to the Biterm topic model Yan et al. (2013).

#### 3.2.3 Probabilistic Decoder

With the learned \(z^{v}\) and \(z^{d}\), ideally, the observed graph \(A^{v}\) and \(A^{d}\) can be reconstructed through a decoding process. For \(A^{v}\), we assume \(P_{\theta}(A^{v}|z^{v})\) conforms to a multivariate Gaussian distribution, whose mean parameters are generated from the inner product of the latent variable \(z^{v}\):

\[\begin{split} P_{b}(A^{v}|z^{v})&=\prod_{i=1}^{V} p_{\theta}(A^{v}_{i}|z^{v})\\ p_{\theta}(A^{v}_{i}|z^{v})&=\prod_{i=1}^{V} \mathcal{N}(A^{v}_{i}|\rho(z^{v}_{i}(z^{v})^{\mathrm{T}}),I)\end{split}\] (8) 

where \(\rho\) is the nonlinear activation function.

Similarly, the inner product between \(z^{v}\) and \(z^{d}\) is used to generate \(A^{d}\), which is sampled from the multivariate Gaussian distribution:

\[P_{\theta}(A^{d}|z^{d},z^{v})= \prod_{i=1}^{D}p_{\theta}(A_{i}^{d}|z_{i}^{d},z^{v}) \tag{9}\] \[P_{\theta}(A_{i}^{d}|z_{i}^{d},z^{v})= \prod_{i=1}^{D}\mathcal{N}(A_{i}^{d}|\rho(z_{i}^{d}(z^{v})^{ \mathsf{T}}),I)\]

For categorical labels \(Y\), we assume \(p_{\theta}(Y|z^{d})\) follows a multinomial distribution \(P_{\theta}(Y|z^{d})=Mul(Y|f_{y}(z^{d}))\), whose label probability vectors are generated from \(z^{d}\), where \(f_{y}\) is the multi-layer neural network. For each document \(t\), the prediction is given by \(\hat{y}_{t}=argmax\limits_{y\in c}P_{\theta}(y|f_{y}(z_{t}^{d}))\).

#### 3.2.4 Optimization

We can rewrite Equation 4 to yield the final variational objective function:

\[\begin{split}\mathcal{L}(\theta,\phi)&\approx \sum_{i=1}^{V}\sum_{j=1}^{V}\log p_{\theta}(A_{i,j}^{v}|z_{i}^{v},z_{j}^{v})\\ &+\sum_{t=1}^{D}\left(\log p_{\theta}(A_{t}^{d}|z_{t}^{d},z^{v}) +\log p_{\theta}(Y_{i}|z_{t}^{d})\right)\\ &-KL[q_{\phi(z^{v})}||p_{\theta}(z^{v})]\\ &-KL[q_{\phi(z^{d})}||p_{\theta}(z^{d})]\end{split} \tag{10}\]

with following reconstruction terms and KL divergences:

\[\begin{split}\log p_{\theta}(A_{i}^{v}|z^{v})& \approx||A_{i}^{v}-\rho(z_{i}^{v}(z^{v})^{\mathsf{T}})||^{2}\\ \log p_{\theta}(A_{t}^{d}|z_{t}^{d},z^{v})&\approx|| A_{t}^{d}-\rho(z_{t}^{d}(z^{v})^{\mathsf{T}})||^{2}\\ \log p_{\theta}(Y_{t}|z_{t}^{d})&\approx Y_{t}\log \hat{y}_{t}+(1-Y_{t})\log(1-\hat{y}_{t})\\ KL[q_{\phi}(z_{i}^{u})||p_{\theta}(z_{i}^{v})]\\ \approx\frac{1}{2}\sum_{j=1}^{V}((\mu_{ij}^{v})^{2}+& (\sigma_{ij}^{v})^{2}-(1+\log(\sigma_{ij}^{v})^{2}))\\ KL[q_{\phi}(z_{t}^{d})||p_{\theta}(z_{t}^{d})]\\ \approx\frac{1}{2}\sum_{j=1}^{V}((\mu_{ij}^{d})^{2}+& (\sigma_{ij}^{d})^{2}-(1+\log(\sigma_{ij}^{d})^{2}))\end{split} \tag{11}\]

Through maximizing the objective with stochastic gradient descent, we jointly learn the latent word and document representations, which can efficiently reconstruct observed graphs and predict ground truth labels.

## 4 Experiment

In this section, to evaluate the effectiveness of our proposed T-VGAE, experiments are conducted on both supervised and semi-supervised text classification tasks, as well as unsupervised topic modeling tasks.

### Datasets and settings

#### 4.1.1 Datasets

We conduct experiments on five commonly used text classification datasets: 20NewsGroups, Ohsumed, R52 and R8, and MR. We use the same data preprocessing as in Yao et al. (2019). The overview of the five datasets is depicted in Table 2.

#### 4.1.2 Baselines

We compare our method with the following two categories of baselines:

**text classification**: 1)TF-IDF+LR: the classical logistic regression method based on TF-IDF features. 2) CNN Kim (Kim, 2014): the convolutional neural network based method with pre-trained word embeddings. 3) LSTM Liu et al. (2016): the LSTM based method with pre-trained word embeddings. 4) SWEM Shen et al. (2018): the word embedding model with pooling strategies. 5) fastText Joulin et al. (2016): the averages word embeddings for text classification. 6) Graph-CNN Peng et al. (2018a): a graph CNN model based on word embedding similarity graphs 7) LEAM Wang et al. (2018): the label-embedding attentive models with document embeddings based on word and label descriptions. 8) TextGCN Yao et al. (2019): a GCN model with a corpus-level graph to learn word and document embeddings. 9) DHTG Wang et al. (2020): a GCN model with a dynamic hierarchical topic graph based on the topic model.

**topic modeling**: 1) LDA Blei et al. (2003): a classical probabilistic topic model. 2) NVDM

Figure 3: The augmentation of test accuracy with our model under different topic size \(k\).

 

(Miao et al., 2016): a deep neural variational document topic model. 3) AVITM (Srivastava and Sutton, 2017): an autoencoding variational Bayes (AEVB) topic model based on LDA. 4) GraphBTM (Zhu et al., 2018): an enriched biterm topic model (BTM) with the word co-occurrence graph encoded by GCN.

#### 4.1.3 Settings

Following Yao et al. (2019), we set the hidden size \(K\) of latent variables and other neural network layers as 200 and set the window size in PPMI as 20. The dropout is only utilized in the classifier, and is set to \(0.85\). We train our model for a maximum of 1000 epochs with Adam (Kingma and Ba, 2015) under learning rate \(0.05\). \(10\%\) of the data set is randomly sampled and spared as the validation set for model selection. The parameter settings of all baselines are the same as their original papers or implementations.

### Performance

#### 4.2.1 Supervised Classification

We present the test performances of models in text classification among five datasets in Table 3. We can see that our model consistently outperforms all the baselines on each dataset, which proves the effectiveness of our proposed methods. Compared with TextGCN, our method yields better performance in both datasets. It demonstrates the importance of integrating the latent semantic structures in text classification. It is also observed from the superior performance of DHTG when compared with TextGCN. However, DHTG only learns from the document-word correlation while our method fully exploits both word-word and document-word correlation information, resulting in a significant improvement over DHTG. This proves the effectiveness of unified topic modeling and graph representation learning in text classification. Moreover, there are no test documents involved during the training of our method, which shows the inductive learning ability of our method, different from TextGCN and DHTG which requires a global graph including all documents and words.

#### 4.2.2 Effects of Correlation Information of Different Order

In Table 4, we further present the test accuracy of our method using different layers of GCN en

Figure 4: Test accuracy of different models under varying training data proportions.

\begin{table}
\begin{tabular}{c|c|c} \hline
**Model** & **R52** & **R8** \\ \hline \(l=0\) & 0.9143 \(\pm\) 0.0015 & 0.9495 \(\pm\) 0.0011 \\ \(l=1\) & **0.9505**\(\leq\) 0.0010 & **0.9768**\(\leq\) 0.0014 \\ \(l=2\) & 0.8942 \(\pm\) 0.0012 & 0.9667 \(\pm\) 0.0014 \\ \(l=3\) & 0.7326 \(\pm\) 0.0012 & 0.8795 \(\pm\) 0.0010 \\ \hline \end{tabular}
\end{table}
Table 4: Test Accuracy on document classification task averaged on 10 times using different layers of GCN encoder, i.e. \(l\in(0,1,2,3)\).

 

coder, to demonstrate the impact of a different order of word-word correlation information in \(A^{v}\). On datasets R52 and R8, our method achieves the best performance when the layer number is \(1\). This is different from TextGCN and DHTG, which generally have the best performance with \(2\) layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph \(A^{d}\). If the layer number is set to \(1\) when encoding \(A^{v}\), it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG. In Table 4, we further present the test accuracy of our method using different layers of GCN encoder, to demonstrate the impact of different orders of word-word correlation information in \(A^{v}\). On datasets R52 and R8, our method achieves the best performance when the layer number is \(1\). This is different from TextGCN and DHTG, which generally have the best performance with \(2\) layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph \(A^{d}\). If the layer number is set to \(1\) when encoding \(A^{v}\), it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG.

#### 4.2.3 Effects of Number of Topics

Figure 3 shows the changes of the test accuracy along with different numbers of topics on five datasets. We can see that the test accuracy on five datasets generally improves with the increase of the number of topics and reaches the peak when the topic number is around \(200\). The number of topics shows more impact on the Oshumed dataset than on the other four datasets. This does not seem to be related to the number of classes in the dataset. We suspect it has to do with the nature of the text (medical domain vs. other domains).

#### 4.2.4 Semi-Supervised Classification

In Figure 4, we further present the semi-supervised classification test accuracy on datasets 20NG and R8 where different proportions (\(1\%\), \(5\%\),\(10\%\) and \(20\%\)) of the original training set are used. We can see that, in cases where labeled samples are limited, our model still consistently outperforms all the baselines. Compared with other methods, TextGCN and our model can preserve good performance with few labeled samples (\(1\%\), \(5\%\)). This illustrates the effect of label propagation in GCN for semi-supervised learning. When compared with TextGCN, our model yields better performance because of its inductive learning capability and the incorporation of the latent topic semantics.

#### 4.2.5 Document Topic Modelling

We further evaluate the performance of models on unsupervised topic modeling tasks. We generally assume that the more topics are coherent, the more they are interpretable. Following Srivastava and Sutton (2017), We use the average pairwise PMI of the top 10 words in each topic and the perplexity with the ELBO as quality measures of topics. We show in Table 6 the measures under different topic numbers in the 20NG dataset. We remove the supervised loss of our method and the result of GraphBTM is not presented for unable to learn document topic representation for each document.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline
**Metrics** & \multicolumn{2}{c|}{Topic coherence} & \multicolumn{2}{c}{Perplexity} \\ \cline{2-5}

- \\ T-VGAE & **0.37** & **0.59** & **615** & **665** \\ \hline \end{tabular}
\end{table}
Table 6: The average topic coherence (higher is better) and perplexity (lower is better) with different topic numbers.

Figure 5: The t-SNE visualization of test document embeddings of 20NG by different models.

\begin{table}
\begin{tabular}{l|l} \hline
**Category** & **Topic** \\ \hline  & \multirow{2}{*}{**Type1: item season hockey game nhl players win**} \\
**Sport** & & play baseball chilp. 18902 \\
**Total: clipper hockey season team encryption key** \\  & nul tool baschball gt. 1.1985 \\ \hline  & [t]: Ic:: LS+IS? event x 6128 cars entertopo imitbloir \\
**Acts** & car swimming 1.1931 \\  & T62:: l115 x car cars lex encryption daubenspeck \\  & stem clipper sport 0.8977 \\ \hline  & [t]: I2: name cstring episode ataxia iis apple e650 \\
**Elce** & tomj geb powerbook 1.1603 \\  & T71: mac dod quada cartis apple bile isi \\  & encyclonic leix U. 9/789 \\ \hline \end{tabular}
\end{table}
Table 5: The top-10 words and coherence score of topics in 20NG dataset from \(z^{v}\).

 

In the table, we can see that our model outperforms the others in terms of topic coherence, which could be attributed to the combination of word co-occurrence graph and message passing in GCN. The message passing leads to similar representations of words that co-occur frequently in the latent topic space, thus improves the semantic coherence of learned topics, as shown in Table 5 that related words tend to belong to the same topic. Our method also benefits from document-word correlation, and yield better performance when compared with GraphBTM which encode bi-term graph via GCN.

#### 4.2.6 Document Representations

We utilize t-SNE to visualize the latent test document representations of the 20NG dataset learned by our model, DHTG and TextGCN in Figure 5, in which each dot represents a document and each color represents a category. Our method yields the best clustering results compared with the others, which means the topics are more consistent with pre-defined classes. It shows the superior interpretability of our method for modeling the latent topics along with both word co-occurrence graph and document-word graph when compared with DHTG.

## 5 Conclusion

In this paper, we proposed a novel deep latent variable model T-VGAE via combining the topic model with VGAE. It can learn more interpretable representations and leverage the latent topic semantic to improve the classification performance. T-VGAE inherits advantages from the topic model and VGAE: probabilistic interpretability and efficient label propagation mechanism. Experimental results demonstrate the effectiveness of our method along with inductive learning. As future work, it would be interesting to explore better-suited prior distribution in the generative process. It is also possible to extend our model to other tasks, such as information recommendation and link prediction.

## Acknowledgments

This research is supported by the CSC Scholarship offered by China Scholarship Council. We would like to thank the anonymous reviewers for their constructive comments. We thank MindSpore for the partial support of this work, which is a new deep learning computing framework1.

Footnote 1: [https://www.mindspore.cn/](https://www.mindspore.cn/)

## References

* Bai et al. (2018) Haoli Bai, Zhuangbin Chen, Michael R Lyu, Irwin King, and Zenglin Xu. 2018. Neural relational topic models for scientific article analysis. In _CIKM_, pages 27-36.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _JMLR_, (Jan):993-1022.
* Ding et al. (2020) Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu. 2020. Be more with less: Hypergraph attention networks for inductive text classification. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4927-4936.
* Grover et al. (2019) Aditya Grover, Aaron Zweig, and Stefano Ermon. 2019. Graphite: Iterative generative modeling of graphs. In _ICML_, pages 2434-2444.
* Hasanzadeh et al. (2019) Arman Hasanzadeh, Ehshan Hajiramezanali, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019. Semi-implicit graph variational auto-encoders. In _NIPs_.
* Huang et al. (2019) Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019. Text level graph neural network for text classification. _arXiv preprint arXiv:1910.02356_.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. _arXiv:1607.01759_.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _EMNLP_, pages 1746-1751.
* Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. _arXiv preprint arXiv:1312.6114_.
* Kingma and Ba (2015) DP Kingma and JL Ba. 2015. Adam: A method for stochastic optimization. In _ICLR_.
* Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. _arXiv:1611.07308_.
* Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In _ICLR_.
* Liu et al. (2016) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In _IJCAI_, pages 2873-2879. AAAI Press.
* Liu et al. (2020) Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. _AAAI_.

 

Yishu Miao, Lei Yu, and Phil Blunsom. 2016Neural variational inference for text processing. In ICML, pp. 1727-1736. Cited by: SS4.
* S. Pan, R. Hu, G. Long, J. Jiang, L. Yao, and C. Zhang (2018)Adversarially regularized graph autoencoder for graph embedding. In IJCAI, pp. 2609-2615. Cited by: SS1, SS2.
* H. Peng, J. Li, Y. He, X. Liu, M. Bao, L. Wang, Y. Song, and Q. Yang (2018a)Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In WWW, pp. 1063-1072. Cited by: SS1, SS2.
* M. Peng, Q. Xie, Y. Zhang, H. Wang, X. Jenny Zhang, J. Huang, and G. Tian (2018b)Neural sparse topical coding. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2332-2340. Cited by: SS1, SS2.
* D. Shen, G. Wang, W. Wang, M. Renqiang Min, Q. Su, Y. Zhang, C. Li, R. Henao, and L. Carin (2018)Baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms. arXiv:1805.09843. External Links: Link Cited by: SS1, SS2.
* A. Srivastava and C. Sutton (2017)Autoencoding variational inference for topic models. arXiv:1703.01488. Cited by: SS1, SS2.
* N. Van Linh, T. X. Bach, and K. Than (2020)Graph convolutional topic model for data streams. arXiv, pp. arXiv-2003. Cited by: SS1, SS2.
* Sh. Vashishtha, M. Bhandari, P. Yadav, P. Rai, C. Bhattacharyya, and P. Talukdar (2019)Incorporating syntactic and semantic information in word embeddings using graph convolutional networks. In ACL, pp. 3308-3318. Cited by: SS1, SS2.
* P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2018)Graph attention networks. In ICLR, Cited by: SS1, SS2.
* Guoyin Wang, Chunyuan Li, W. Wang, Y. Zhang, D. Shen, X. Zhang, R. Henao, and L. Carin (2018)Joint embedding of words and labels for text classification. arXiv:1805.04174. Cited by: SS1, SS2.
* Z. Wang, C. Wang, H. Zhang, Z. Duan, M. Zhou, and B. Chen (2020)Learning dynamic hierarchical topic graph with graph convolutional network for document classification. Cited by: SS1, SS2.
* Qianqian Xie, J. Huang, P. Du, M. Peng, and J. Nie (2021)Graph topic neural network for document representation. In Proceedings of The Web Conference 2021, Cited by: SS1, SS2.
* X. Yan, J. Guo, Y. Lan, and X. Cheng (2013)A biterm topic model for short texts. In Proceedings of the 22nd international conference on World Wide Web, pp. 1445-1456. Cited by: SS1, SS2.
* Liang Yang, F. Wu, J. Gu, C. Wang, X. Cao, D. Jin, and Y. Guo (2020)Graph attention topic modeling network. In WWW, pp. 144-154. Cited by: SS1, SS2.
* L. Yao, C. Mao, and Y. Luo (2019)Graph convolutional networks for text classification. In AAAI, Cited by: SS1, SS2.
* Haopeng Zhang and Jiawei Zhang (2020)Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8322-8327. Cited by: SS1, SS2.
* Yufang Zhang, X. Yu, Z. Cui, S. Wu, Z. Wen, and L. Wang (2020)Every document owns its structure: inductive text classification via graph neural networks. arXiv preprint arXiv:2004.13826. Cited by: SS1, SS2.
* Yuhao Zhang, P. Qi, and C. D. Manning (2018)Graph convolution over pruned dependency trees improves relation extraction. In EMNLP, pp. 2205-2215. Cited by: SS1, SS2.
* Qile Zhu, Z. Feng, and X. Li (2018)Graphbtm: graph enhanced autoencoded variational inference for btierm topic model. In EMNLP, pp. 4663-4672. Cited by: SS1, SS2.