SlowGuess/ACL_2020_OCR huggingface dataset:
We choose the following models as baselines: LDA (Blei et al., 2003) extracts topics based on word co-occurrence patterns from documents. We implement LDA following the parameter setting suggested in (Griffiths and Steyvers, 2004).
NVDM (Miao et al., 2016) is an unsupervised text modeling approach based on VAE. We use the original implementation of the paper<sup>4</sup>.
<sup>1</sup>http://qwone.com/ jason/20Newsgroups/
2https://cs.nyu.edu/\~roweis/data/
<sup>3</sup>http://archive.ics.uci.edu/ml/datasets/Bag+of+Words
<sup>4</sup>https://github.com/ysmiao/nvdm
Figure 4: The comparison of average topic coherence vs. different topic proportion on three datasets.
GSM(Miao et al., 2017) is an enhanced topic model based on NVDM, we use the original implementation in our experiments<sup>5</sup>.
NVLDA (Srivastava and Sutton, 2017), also built on VAE but with the logistic-normal prior. We use the implementation provided by the author<sup>6</sup>.
ProdLDA (Srivastava and Sutton, 2017), is a variant of NVLDA, in which the distribution over individual words is a product of experts. The original implementation is used.
ATM (Wang et al., 2019a), is a neural topic modeling approach based on adversarial training, we implement the ATM following the parameter setting suggested in the original paper.
4.2 Topic Coherence Evaluation
Topic models are typically evaluated with the likelihood of held-out documents and topic coherence. However, Chang et al. (2009) showed that a higher likelihood of held-out documents does not correspond to human judgment of topic coherence. Thus, we follow (Röder et al., 2015) and employ four topic coherence metrics (C_P, C_A, NPMI and UCI) to evaluate the topics generated by various models. In all experiments, each topic is represented by the top 10 words according to the topic-word probabilities, and all the topic coherence values are calculated using the Palmetto library<sup>7</sup>.
We firstly make a comparison of topic coherence vs. different topic proportions. Experiments are
Dataset	Model	C_P	C_A	NPMI	UCI
20Newsgroups	NVDM	-0.2558	0.1286	-0.0984	-2.9496
GSM	-0.2318	0.1067	-0.0400	-1.6083
NVLDA	0.1205	0.1763	-0.0207	-1.3466
ProdLDA	0.1858	0.2155	-0.0083	-1.5044
LDA	0.2361	0.1769	0.0523	0.3399
ATM	0.1914	0.1720	0.0207	-0.3871
BAT	0.2597	0.1976	0.0472	0.0969
Gaussian-BAT	0.3758	0.2251	0.0819	0.5925
Grolier	NVDM	-0.1877	0.1456	-0.0619	-2.1149
GSM	0.1974	0.1966	0.0491	-0.0410
NVLDA	-0.2205	0.1504	-0.0653	-2.4797
ProdLDA	-0.0374	0.1733	-0.0193	-1.6398
LDA	0.1908	0.2009	0.0497	-0.0503
ATM	0.2105	0.2188	0.0582	0.1051
BAT	0.2312	0.2108	0.0608	0.1709
Gaussian-BAT	0.2606	0.2142	0.0724	0.2836
NYtimes	NVDM	-0.4130	0.1341	-0.1437	-4.3072
GSM	0.3426	0.2232	0.0848	0.6224
NVLDA	-0.1575	0.1482	-0.0614	-2.4208
ProdLDA	-0.0034	0.1963	-0.0282	-1.9173
LDA	0.3083	0.2127	0.0772	0.5165
ATM	0.3568	0.2375	0.0899	0.6582
BAT	0.3749	0.2355	0.0951	0.7073
Gaussian-BAT	0.4163	0.2479	0.1079	0.9215
Table 2: Average topic coherence on three datasets with five topic settings [20, 30, 50, 75, 100].
conducted on the datasets with five topic number settings [20, 30, 50, 75, 100]. We calculate the average topic coherence values among topics whose coherence values are ranked at the top  $50\%$ ,  $70\%$ ,  $90\%$ ,  $100\%$  positions. For example, to calculate the average C_P value of BAT @90%, we first compute the average C_P coherence with the selected topics whose C_P values are ranked at the top  $90\%$  for each topic number setting, and then average the five coherence values with each corresponding to a particular topic number setting.
The detailed comparison is shown in Figure 4. It can be observed that BAT outperforms the baselines on all the coherence metrics for NYTimes datasets. For Grolier dataset, BAT outperforms all the baselines on C.P, NPMI and UCI metrics, but
<sup>5</sup>https://github.com/link strife/NVDM-GSM
$^{6}$ https://github.com/akashgit/autoencoding vi for topic models
<sup>7</sup>https://github.com/dice-group/Palmetto
Figure 5: The comparison of average topic coherence vs. different topic number on 20Newsgroups, Grolier and NYTimes.
Model	Topics
Gaussian-BAT	voter campaign poll candidates democratic election republican vote presidential democrat
song album music band rock pop sound singer jazz guitar
film movie actor character movies director series actress young scenes
flight airline passenger airlines aircraft shuttle airport pilot carrier planes
BAT	vote president voter campaign election democratic governor republican black candidates
album band music rock song jazz guitar pop musician record
film actor play acting role playing character father movie actress
flight airline delay airlines plane pilot airport passenger carrier attendant
LDA	voter vote poll election campaign primary candidates republican race party
music song band sound record artist album show musical rock
film movie character play actor director movies minutes theater cast
flight plane ship crew air pilot hour boat passenger airport
ATM	voter vote poll republican race primary percent election campaign democratic
music song musical album jazz band record recording mp3 composer
film movie actor director award movies character theater production play
jet flight airline hour plane passenger trip plan travel pilot
Table 3: Topic examples extracted by models, italics means out-of-topic words. These topics correspond to 'election', 'music', 'film' and 'airline' respectively, and topic examples of other models are omitted due to poor quality.
gives slightly worse results compared to ATM on C_A. For 20Newsgroups dataset, BAT performs the best on C_P and NPMI, but gives slightly worse results compared to ProdLDA on C_A, and LDA on UCI. By incorporating word embeddings through trainable Gaussian distribution, Gaussian-BAT outperforms all the baselines and BAT on four coherence metrics, often by a large margin, across all the three datasets except for Grolier dataset on C_A when considering  $100\%$  topics. This may be attribute to the following factors: (1) The Dirichlet prior employed in BAT and Gaussian-BAT could exhibit a multi-modal distribution in latent space and is more suitable for discovering semantic patterns from text; (2) ATM does not consider the relationship between topic distribution and word distribution since it only carry out adversarial training in word distribution space; (3) The incorpora
tion of word embeddings in Gaussian-BAT helps generating more coherent topics.
We also compare the average topic coherence values (all topics taken into account) numerically to show the effectiveness of proposed BAT and Gaussian-BAT. The results of numerical topic coherence comparison are listed in Table 2 and each value is calculated by averaging the average topic coherences over five topic number settings. The best coherence value on each metric is highlighted in bold. It can be observed that Gaussian-BAT gives the best overall results across all metrics and on all the datasets except for Grolier dataset on C_A. To make the comparison of topics more intuitive, we provide four topic examples extracted by models in Table 3. It can be observed that the proposed BAT and Gaussian-BAT can generate more coherent topics.
Moreover, to explore how topic coherence varies with different topic numbers, we also provide the comparison of average topic coherence vs. different topic number on 20newsgroups, Grolier and NYTimes (all topics taken into account). The detailed comparison is shown in Figure 5. It could be observed that Gaussian-BAT outperforms the baselines with 20, 30, 50 and 75 topics except for Grolier dataset on C_A metric. However, when the topic number is set to 100, Gaussian-BAT performs slightly worse than LDA (e.g., UCI for 20Newsgroups and C_A for NYTimes). This may be caused by the increased model complexity due to the larger topic number settings. Likewise, BAT can achieve at least the second-best results among all the approaches in most cases for NYTimes dataset. For Grolier, BAT also performs the second-best except on C_A metric. However, for 20newsgroups, the results obtained by BAT are worse than ProdLDA (C_A) and LDA (UCI) due to the limited training documents in the dataset, though it still largely outperforms other baselines.
4.3 Text Clustering
We further compare our proposed models with baselines on text clustering. Due to the lack of document label information in Grolier and NYTimes, we only use 20Newsgroups dataset in our experiments. The topic number is set to 20 (ground-truth categories) and the performance is evaluated by accuracy  $(ACC)$ :
$$
A C C = \max  _ {\text {m a p}} \frac {\sum_ {i = 1} ^ {N _ {t}} \operatorname {i n d} \left(l _ {i} = \operatorname {m a p} \left(c _ {i}\right)\right)}{N _ {t}} \tag {14}
$$
where  $N_{t}$  is the number of documents in the test set,  $\operatorname{ind}(\cdot)$  is the indicator function,  $l_{i}$  is the ground-truth label of  $i$ -th document,  $c_{i}$  is the category assignment, and map ranges over all possible one-to-one mappings between labels and clusters. The optimal map function can be obtained by the Kuhn-Munkres algorithm (Kuhn, 1955). A larger accuracy value indicates a better text clustering results.
Dataset	NVLDA	ProdLDA	LDA	BAT	G-BAT
20NG	33.31%	33.82%	35.36%	35.66%	41.25%
Table 4: Text clustering accuracy on 20Newsgroups (20NG). 'G-BAT' refers to 'Gaussian-BAT'. The best result is highlighted in bold.
The comparison of text clustering results on 20Newsgroups is shown in Table 4. Due to the
poor performance of NVDM in topic coherence evaluation, its result is excluded here. Not surprisingly, NVLDA and ProdLDA perform worse than BAT and Gaussian-BAT that model topics with the Dirichlet prior. This might be caused by the fact that Logistic-Normal prior does not exhibit multiple peaks at the vertices of the simplex, as depicted in Figure 1. Compared with LDA, BAT achieves a comparable result in accuracy since both models have the same Dirichlet prior assumption over topics and only employ the word co-occurrence information. Gaussian-BAT outperforms the second best model, BAT, by nearly  $6\%$  in accuracy. This shows that the incorporation of word embeddings is important to improve the semantic coherence of topics and thus results in better consistency between cluster assignments and ground-truth labels.
In this paper, we have explored the use of bidirectional adversarial training in neural topic models and proposed two novel approaches: the Bidirectional Adversarial Topic (BAT) model and the Bidirectional Adversarial Topic model with Gaussian (Gaussian-BAT). BAT models topics with the Dirichlet prior and builds a two-way transformation between document-topic distribution and document-word distribution via bidirectional adversarial training. Gaussian-BAT extends from BAT by incorporating word embeddings into the modeling process, thereby naturally considers the word relatedness information captured in word embeddings. The experimental comparison on three widely used benchmark text corpus with the existing neural topic models shows that BAT and Gaussian-BAT achieve improved topic coherence results. In the future, we would like to devise a nonparametric neural topic model based on adversarial training. Besides, developing correlated topic models is another promising direction.
We would like to thank anonymous reviewers for their valuable comments and helpful suggestions. This work was funded by the National Key Research and Development Program of China(2017YFB1002801) and the National Natural Science Foundation of China (61772132). And YH is partially supported by EPSRC (grant no. EP/T017112/1).
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223.
Ben Athiwaratkun, Andrew Wilson, and Anima Anandkumar. 2018. Probabilistic fasttext for multi-sense word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1-11.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pages 288-296.
Xueqi Cheng, Xiaohui Yan, Yanyan Lan, and Jiafeng Guo. 2014. Btm: Topic modeling over short texts. IEEE Transactions on Knowledge and Data Engineering, 26(12):2928-2941.
Jeff Donahue, Philipp Krahenbuhl, and Trevor Darrell. 2016. Adversarial feature learning. arXiv preprint arXiv:1605.09782.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. 2016. Adversarily learned inference. arXiv preprint arXiv:1606.00704.
William Fedus, Ian Goodfellow, and Andrew M Dai. 2018. Maskgan: better text generation via filling in the... arXiv preprint arXiv:1801.07736.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl 1):5228-5235.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of Wasserstein gans. In Advances in neural information processing systems, pages 5767-5777.
Geoffrey E Hinton and Ruslan R Salakhutdinov. 2009. Replicated softmax: an undirected topic model. In Advances in neural information processing systems, pages 1607-1614.
Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017. Bag of tricks for efficient text classification. EACL 2017, page 427.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2:83-97.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331-339.
Zeyang Lei, Yujiu Yang, and Min Yang. 2018. Saan: A sentiment-aware attention network for sentiment analysis. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1197-1200. ACM.
Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 375-384. ACM.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, pages 3155-3165.
Qiao Liu, Haibin Zhang, Yifu Zeng, Ziqi Huang, and Zufeng Wu. 2018. Content attention model for aspect based sentiment analysis. In Proceedings of the 2018 World Wide Web Conference, pages 1023-1032. International World Wide Web Conferences Steering Committee.
Yishu Miao, Edward Grefenstette, and Phil Blun-som. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2410-2419. JMLR.org.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In International conference on machine learning, pages 1727-1736.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pages 399-408. ACM.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488.
Hanna M Wallach, David M Mimno, and Andrew McCallum. 2009. Rethinking lda: Why priors matter. In Advances in neural information processing systems, pages 1973-1981.
Rui Wang, Deyu Zhou, and Yulan He. 2019a. Atm: Adversarial-neural topic model. Information Processing & Management, 56(6):102098.
Rui Wang, Deyu Zhou, and Yulan He. 2019b. Open event extraction from online text using a generative adversarial network. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 282-291, Hong Kong, China.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence.
Deyu Zhou, Liangyu Chen, and Yulan He. 2014. A simple bayesian modelling approach to event extraction from twitter. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 700-705.