SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\log p _ {\theta} \left(A _ {t} ^ {d} \mid z _ {t} ^ {d}, z ^ {v}\right) \approx \left| \left| A _ {t} ^ {d} - \rho \left(z _ {t} ^ {d} \left(z ^ {v}\right) ^ {\top}\right) \right| \right| ^ {2}
$$
$$
\log p _ {\theta} \left(Y _ {t} \mid z _ {t} ^ {d}\right) \approx Y _ {t} \log \hat {y} _ {t} + \left(1 - Y _ {t}\right) \log \left(1 - \hat {y} _ {t}\right)
$$
$$
\begin{array}{l} K L \left[ q _ {\phi} \left(z _ {i} ^ {v}\right) | | p _ {\theta} \left(z _ {i} ^ {v}\right) \right] \\ \approx \frac {1}{2} \sum_ {j = 1} ^ {V} \left(\left(\mu_ {i j} ^ {v}\right) ^ {2} + \left(\sigma_ {i j} ^ {v}\right) ^ {2} - \left(1 + \log \left(\sigma_ {i j} ^ {v}\right) ^ {2}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} K L \left[ q _ {\phi} \left(z _ {t} ^ {d}\right) | | p _ {\theta} \left(z _ {t} ^ {d}\right) \right] \\ \approx \frac {1}{2} \sum_ {j = 1} ^ {V} \left(\left(\mu_ {t j} ^ {d}\right) ^ {2} + \left(\sigma_ {t j} ^ {d}\right) ^ {2} - \left(1 + \log \left(\sigma_ {t j} ^ {d}\right) ^ {2}\right)\right) \tag {11} \\ \end{array}
$$
Through maximizing the objective with stochastic gradient descent, we jointly learn the latent word and document representations, which can efficiently reconstruct observed graphs and predict ground truth labels.
In this section, to evaluate the effectiveness of our proposed T-VGAE, experiments are conducted on both supervised and semi-supervised text classification tasks, as well as unsupervised topic modeling tasks.
Table 2: Summary statistics of five datasets (Yao et al., 2019)
Dataset	Doc	Train	Test	Word	Node	Class	Average
Len
20NG	18,846	11,314	7,532	42,757	42,757	20	221.26
R8	7,674	5,485	2,189	7,688	7,688	8	65.72
R52	9,100	6,532	2,568	8,892	8,892	52	69.82
Ohsumed	7,400	3,357	4,043	14,157	14,157	23	135.82
MR	10,662	7,108	3,554	18,764	18,764	2	20.39
Figure 3: The augmentation of test accuracy with our model under different topic size  $k$ .
4.1 Datasets and settings
We conduct experiments on five commonly used text classification datasets: 20NewsGroups, Ohsumed, R52 and R8, and MR. We use the same data preprocessing as in (Yao et al., 2019). The overview of the five datasets is depicted in Table 2.
We compare our method with the following two categories of baselines:
text classification: 1) TF-IDF+LR: the classical logistic regression method based on TF-IDF features. 2) CNN (Kim, 2014): the convolutional neural network based method with pre-trained word embeddings. 3) LSTM (Liu et al., 2016): the LSTM based method with pre-trained word embeddings. 4) SWEM (Shen et al., 2018): the word embedding model with pooling strategies. 5) fast-Text (Joulin et al., 2016): the averages word embeddings for text classification. 6) Graph-CNN (Peng et al., 2018a): a graph CNN model based on word embedding similarity graphs 7) LEAM (Wang et al., 2018): the label-embedding attentive models with document embeddings based on word and label descriptions. 8) TextGCN (Yao et al., 2019): a GCN model with a corpus-level graph to learn word and document embeddings. 9) DHTG (Wang et al., 2020): a GCN model with a dynamic hierarchical topic graph based on the topic model.
topic modeling: 1) LDA (Blei et al., 2003): a classical probabilistic topic model. 2) NVDM
<sup>1</sup Its code is not released yet, therefore we only report the test micro precision here.
Table 3: Micro precision, recall and F1-Score on document classification task. We report mean ± standard deviation averaged on 10 times following previous methods (Yao et al., 2019).
Model	20NG	MR	Ohsumed
Measure	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall
TF-IDF+LR	0.8212 ± 0.0000	0.8301± 0.0000	0.8300± 0.0000	0.7452 ± 0.0000	0.7432 ± 0.0000	0.7431 ± 0.0000	0.5454 ± 0.0000	0.5454 ± 0.0000
CNN	0.8213 ± 0.0052	0.7844± 0.0022	0.7880± 0.0020	0.7769 ± 0.0007	0.7366 ± 0.0026	0.7390 ± 0.0018	0.5842 ± 0.0106	0.4429 ± 0.0057
LSTM	0.7321 ± 0.0185	0.7025 ± 0.0046	0.7016 ± 0.0050	0.7769 ± 0.0086	0.7526 ± 0.0062	0.7432 ± 0.0024	0.4925 ± 0.0107	0.4852 ± 0.0046
SWEM	0.8518 ± 0.0029	0.8324 ± 0.0016	0.8273 ± 0.0021	0.7668 ± 0.0063	0.7481 ± 0.0026	0.7428 ± 0.0023	0.6313 ± 0.0055	0.6280 ± 0.0041
LEAM	0.8190 ± 0.0024	0.8026 ± 0.0014	0.8132 ± 0.0021	0.7693 ± 0.0045	0.7438 ± 0.0036	0.7562 ± 0.0023	0.5859 ± 0.0079	0.5832 ± 0.0026
fastText	0.7937 ± 0.0030	0.7726 ± 0.0046	0.7730 ± 0.0028	0.7512 ± 0.0020	0.7411 ± 0.0013	0.7406 ± 0.0025	0.5769 ± 0.0049	0.5594 ± 0.0012
Graph-CNN	0.8139 ± 0.0032	0.8106 ± 0.0056	0.8099 ± 0.0042	0.7721 ± 0.0027	0.7643 ± 0.0034	0.7667 ± 0.0029	0.6390 ± 0.0053	0.6345 ± 0.0032
TextGCN	0.8634 ± 0.0009	0.8627 ± 0.0006	0.8627 ± 0.0011	0.7673 ± 0.0020	0.7640 ± 0.0010	0.7636 ± 0.0010	0.6834 ± 0.0056	0.6820 ± 0.0014
DHTG1	0.8713 ± 0.0007	-	-	0.7721 ± 0.0011	-	-	0.6880 ± 0.0033	-
T-VGAE	0.8808 ± 0.0006	0.8804 ± 0.0010	0.8802 ± 0.0009	0.7803 ± 0.0011	0.7805 ± 0.0011	0.7805 ± 0.0011	0.7002 ± 0.0014	0.7008 ± 0.0010
Model	Ohsumed	R52	R8
Measure	F1	Precision	Recall	F1	Precision	Recall	F1
TF-IDF+LR	0.5453 ± 0.0000	0.8693 ± 0.0000	0.8670 ± 0.0000	0.8687 ± 0.0000	0.9375 ± 0.0000	0.9366 ± 0.0000	0.9344 ± 0.0000
CNN	0.4295 ± 0.0018	0.8760 ± 0.0048	0.8711 ± 0.0012	0.8431 ± 0.0015	0.9572± 0.0052	0.9534± 0.0014	0.9519± 0.0017
LSTM	0.4864 ± 0.0060	0.9053 ± 0.0091	0.8932 ± 0.0022	0.8910 ± 0.0018	0.9634 ± 0.0033	0.9612 ± 0.0025	0.9608 ± 0.0031
SWEM	0.6252 ± 0.0032	0.9295 ± 0.0024	0.9236 ± 0.0022	0.9180 ± 0.0022	0.9531 ± 0.0026	0.9487 ± 0.0024	0.9462 ± 0.0018
LEAM	0.5824 ± 0.0022	0.9183 ± 0.0023	0.9041 ± 0.0017	0.9002 ± 0.0030	0.9330 ± 0.0024	0.9211 ± 0.0012	0.9207 ± 0.0014
fastText	0.5587 ± 0.0026	0.9282 ± 0.0009	0.9146 ± 0.0012	0.9112 ± 0.0026	0.9611 ± 0.0021	0.9467 ± 0.0018	0.9501 ± 0.0022
Graph-CNN	0.6278 ± 0.0023	0.9274 ± 0.0023	0.9106 ± 0.0030	0.9098 ± 0.0028	0.9697 ± 0.0012	0.9387 ± 0.0018	0.9403 ± 0.0014
TextGCN	0.6820 ± 0.0012	0.9354 ± 0.0018	0.9340 ± 0.0012	0.9339 ± 0.0010	0.9704 ± 0.0010	0.9703 ± 0.0009	0.9700 ± 0.0012
DHTG	-	0.9393 ± 0.0010	-	-	0.9733 ± 0.0006	-	-
T-VGAE	0.7004± 0.0010	0.9505 ± 0.0010	0.9500 ± 0.0012	0.9500 ± 0.0010	0.9768 ± 0.0014	0.9766 ± 0.0009	0.9765 ± 0.0009
Table 4: Test Accuracy on document classification task averaged on 10 times using different layers of GCN encoder, i.e.  $l \in  \left( {0,1,2,3}\right)$  .
Model	R52	R8
l = 0	0.9143 ± 0.0015	0.9495 ± 0.0011
l = 1	0.9505 ± 0.0010	0.9768 ± 0.0014
l = 2	0.8942 ± 0.0012	0.9667 ± 0.0014
l = 3	0.7326 ± 0.0012	0.8795 ± 0.0010
(Miao et al., 2016): a deep neural variational document topic model. 3) AVITM (Srivastava and Sutton, 2017): an autoencoding variational Bayes (AEVB) topic model based on LDA. 4) GraphBTM (Zhu et al., 2018): an enriched biterm topic model (BTM) with the word co-occurrence graph encoded by GCN.
4.1.3 Settings
Following (Yao et al., 2019), we set the hidden size  $K$  of latent variables and other neural network layers as 200 and set the window size in PPMI as 20. The dropout is only utilized in the classifier, and is set to 0.85. We train our model for a maximum of 1000 epochs with Adam (Kingma and Ba, 2015) under learning rate 0.05.  $10\%$  of the data set is randomly sampled and spared as the validation set for model selection. The parameter settings of all baselines are the same as their original papers or implementations.
4.2.1 Supervised Classification
We present the test performances of models in text classification among five datasets in Table 3. We can see that our model consistently outperforms
Figure 4: Test accuracy of different models under varying training data proportions.
all the baselines on each dataset, which proves the effectiveness of our proposed methods. Compared with TextGCN, our method yields better performance in both datasets. It demonstrates the importance of integrating the latent semantic structures in text classification. It is also observed from the superior performance of DHTG when compared with TextGCN. However, DHTG only learns from the document-word correlation while our method fully exploits both word-word and document-word correlation information, resulting in a significant improvement over DHTG. This proves the effectiveness of unified topic modeling and graph representation learning in text classification. Moreover, there are no test documents involved during the training of our method, which shows the inductive learning ability of our method, different from TextGCN and DHTG which requires a global graph including all documents and words.
4.2.2 Effects of Correlation Information of Different Order
In Table 4, we further present the test accuracy of our method using different layers of GCN en
coder, to demonstrate the impact of a different order of word-word correlation information in  $A^v$ . On datasets R52 and R8, our method achieves the best performance when the layer number is 1. This is different from TextGCN and DHTG, which generally have the best performance with 2 layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph  $A^d$ . If the layer number is set to 1 when encoding  $A^v$ , it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG. In Table 4, we further present the test accuracy of our method using different layers of GCN encoder, to demonstrate the impact of different orders of word-word correlation information in  $A^v$ . On datasets R52 and R8, our method achieves the best performance when the layer number is 1. This is different from TextGCN and DHTG, which generally have the best performance with 2 layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph  $A^d$ . If the layer number is set to 1 when encoding  ${A}^{v}$  , it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG.
4.2.3 Effects of Number of Topics
Figure 3 shows the changes of the test accuracy along with different numbers of topics on five datasets. We can see that the test accuracy on five datasets generally improves with the increase of the number of topics and reaches the peak when the topic number is around 200. The number of topics shows more impact on the Oshumed dataset than on the other four datasets. This does not seem to be related to the number of classes in the dataset. We suspect it has to do with the nature of the text (medical domain vs. other domains).
4.2.4 Semi-Supervised Classification
In Figure 4, we further present the semi-supervised classification test accuracy on datasets 20NG and R8 where different proportions (1%, 5%, 10% and 20%) of the original training set are used. We can see that, in cases where labeled samples are limited, our model still consistently outperforms all the baselines. Compared with other methods, TextGCN and our model can preserve good performance with few labeled samples (1%, 5%). This illustrates the effect of label propagation in GCN
for semi-supervised learning. When compared with TextGCN, our model yields better performance because of its inductive learning capability and the incorporation of the latent topic semantics.
4.2.5 Document Topic Modelling
Table 5: The top-10 words and coherence score of topics in 20NG dataset from  ${z}^{v}$  .
Category	Topic
Sport	T57: team season hockey game nhl players win play baseball chip 1.8902
T64: clipper hockey season team encryption key nhl toal baseball gt 1.1985
Autos	T61: lcs x11r5 xpert x 6128 cars enterpoop lintlibdir car xwininfo 1.1931
T62: x11r5 x car cars lcs encryption daubenspeck xterm clipper xpert 0.8977
Elec	T12: mac centris graphics quadra iisi apple c650 tomj geb powerbook 1.1603
T71: mac dod quadra centris apple bike iisi encryption lciii lc 0.9789
Table 6: The average topic coherence (higher is better) and perplexity (lower is better) with different topic numbers.
Metrics	Topic coherence	Perplexity
Model	K=50	K=200	K=50	K=200
LDA	0.17	0.14	728	688
NVDM	0.08	0.06	837	884
AVITM	0.24	0.19	1059	1128
GraphBTM	0.28	0.26	-	-
T-VGAE	0.37	0.59	615	665
(a) T-VGAE
(b) DHTG
(c) TextGCN
Figure 5: The t-SNE visualization of test document embeddings of 20NG by different models.
We further evaluate the performance of models on unsupervised topic modeling tasks. We generally assume that the more topics are coherent, the more they are interpretable. Following (Srivastava and Sutton, 2017), We use the average pairwise PMI of the top 10 words in each topic and the perplexity with the ELBO as quality measures of topics. We show in Table 6 the measures under different topic numbers in the 20NG dataset. We remove the supervised loss of our method and the result of GraphBTM is not presented for unable to learn document topic representation for each document.
In the table, we can see that our model outperforms the others in terms of topic coherence, which could be attributed to the combination of word co-occurrence graph and message passing in GCN. The message passing leads to similar representations of words that co-occur frequently in the latent topic space, thus improves the semantic coherence of learned topics, as shown in Table 5 that related words tend to belong to the same topic. Our method also benefits from document-word correlation, and yield better performance when compared with GraphBTM which encode bi-term graph via GCN.
4.2.6 Document Representations
We utilize t-SNE to visualize the latent test document representations of the 20NG dataset learned by our model, DHTG and TextGCN in Figure 5, in which each dot represents a document and each color represents a category. Our method yields the best clustering results compared with the others, which means the topics are more consistent with pre-defined classes. It shows the superior interpretability of our method for modeling the latent topics along with both word co-occurrence graph and document-word graph when compared with DHTG.
In this paper, we proposed a novel deep latent variable model T-VGAE via combining the topic model with VGAE. It can learn more interpretable representations and leverage the latent topic semantic to improve the classification performance. T-VGAE inherits advantages from the topic model and VGAE: probabilistic interpretability and efficient label propagation mechanism. Experimental results demonstrate the effectiveness of our method along with inductive learning. As future work, it would be interesting to explore better-suited prior distribution in the generative process. It is also possible to extend our model to other tasks, such as information recommendation and link prediction.
This research is supported by the CSC Scholarship offered by China Scholarship Council. We would like to thank the anonymous reviewers for their constructive comments. We thank MindSpore for the partial support of this work, which is a new
deep learning computing framework<sup>1</sup>.
Haoli Bai, Zhuangbin Chen, Michael R Lyu, Irwin King, and Zenglin Xu. 2018. Neural relational topic models for scientific article analysis. In CIKM, pages 27-36.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. JMLR, (Jan):993-1022.
Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu. 2020. Be more with less: Hypergraph attention networks for inductive text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4927-4936.
Aditya Grover, Aaron Zweig, and Stefano Ermon. 2019. Graphite: Iterative generative modeling of graphs. In ICML, pages 2434-2444.
Arman Hasanzadeh, Ehsan Hajiramezanali, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019. Semi-implicit graph variational auto-encoders. In NIPs.
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019. Text level graph neural network for text classification. arXiv preprint arXiv:1910.02356.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv:1607.01759.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
DP Kingma and JL Ba. 2015. Adam: A method for stochastic optimization. In *ICLR*.
Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv:1611.07308.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In *IJCAI*, pages 2873-2879. AAAI Press.
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. AAAI.
<sup>1</sup>https://www.mindspore.cn/
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In ICML, pages 1727-1736.
Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. 2018. Adversarily regularized graph autoencoder for graph embedding. In *IJCAI*, pages 2609–2615.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018a. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In WWW, pages 1063-1072.
Min Peng, Qianqian Xie, Yanchun Zhang, Hua Wang, Xiuzhen Jenny Zhang, Jimin Huang, and Gang Tian. 2018b. Neural sparse topical coding. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2332-2340.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. arXiv:1805.09843.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. arXiv:1703.01488.
Ngo Van Linh, Tran Xuan Bach, and Khoat Than. 2020. Graph convolutional topic model for data streams. arXiv, pages arXiv-2003.
Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. 2019. Incorporating syntactic and semantic information in word embeddings using graph convolutional networks. In ACL, pages 3308-3318.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In ICLR.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. arXiv:1805.04174.
Zhengjue Wang, Chaojie Wang, Hao Zhang, Zhibin Duan, Mingyuan Zhou, and Bo Chen. 2020. Learning dynamic hierarchical topic graph with graph convolutional network for document classification.
Qianqian Xie, Jimin Huang, Pan Du, Min Peng, and Jian-Yun Nie. 2021. Graph topic neural network for document representation. In Proceedings of The Web Conference 2021.
Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013. A biterm topic model for short texts. In Proceedings of the 22nd international conference on World Wide Web, pages 1445-1456.
Liang Yang, Fan Wu, Junhua Gu, Chuan Wang, Xiaochun Cao, Di Jin, and Yuanfang Guo. 2020. Graph attention topic modeling network. In WWW, pages 144-154.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In AAAI.
Haopeng Zhang and Jiawei Zhang. 2020. Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8322-8327.
Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang. 2020. Every document owns its structure: Inductive text classification via graph neural networks. arXiv preprint arXiv:2004.13826.
Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In EMNLP, pages 2205-2215.
Qile Zhu, Zheng Feng, and Xiaolin Li. 2018. Graphbtm: Graph enhanced autoencoded variational inference for bitterm topic model. In EMNLP, pages 4663-4672.