SlowGuess/NAACL_2022_OCR huggingface dataset:
Maarten Grootendorst. 2020. Bertopic: Leveraging bert and c-tfidf to create easily interpretable topics.
Yuan Jin, He Zhao, Ming Liu, Lan Du, and Wray Buntine. 2021. Neural attention-aware hierarchical topic model. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1042-1052, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Kar Wai Lim and Wray Buntine. 2015. Bibliographic analysis with the citation network topic model. In Proceedings of the Sixth Asian Conference on Machine Learning, volume 39 of Proceedings of Machine Learning Research, pages 142-158, Nha Trang City, Vietnam. PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Leland McInnes and John Healy. 2017. Accelerated hierarchical density based clustering. In Data Mining Workshops (ICDMW), 2017 IEEE International Conference on, pages 33-42. IEEE.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1727-1736. JMLR.org.
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6345-6381, Florence, Italy. Association for Computational Linguistics.
David Newman, Joy Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 100-108, Los Angeles, California. Association for Computational Linguistics.
Juan Ramos et al. 2003. Using tfidf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 29-48. Citeseer.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pages 399-408. ACM.
Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728-1736, Online. Association for Computational Linguistics.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021. OCTIS: Comparing and optimizing topic models is simple! In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 263-270, Online. Association for Computational Linguistics.
Laure Thompson and David Mimno. 2020. Topic modeling with contextualized word representation clusters. arXiv preprint arXiv:2010.12626.
Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020. Which *BERT? A survey organizing contextualized encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7516-7533, Online. Association for Computational Linguistics.
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021. Topic modelling meets deep neural networks: A survey. arXiv preprint arXiv:2103.00498.
A Configuration Details
We implement LDA and NTMs based on OCTIS (Terragni et al., 2021)  $^{3}$  and use their default settings. Specifically, ProdLDA, CombinedTM, and ZeroShotTM share the same configurations, i.e. one hidden layer with 100 neurons, ADAM optimizer and Momentum as 0.99; we randomly dropout  $20\%$  hidden units; we run 100 epochs of each model, and the batch size is 64. For BERT+KM, we follow Sia et al. (2020) by reducing embedding dimension to 50 using Principal Component Analysis (PCA) and adopting TF to select words. For BERT+UMAP+HDBSCAN, we follow BERTopic Grootendorst (2020) and allows it to reduce the topic numbers. For our methods, we implement clustering-based experiments based on BERTopic (Grootendorst, 2020)  $^{4}$ . We reduce embedding dimension to 5 using UMAP. We use BERT, RoBERTa, and SBERT embeddings provided by HuggingFace  $^{5}$ , and SimCSE embeddings provided from its official Github  $^{6}$ .
B Comparison of Topic Words
We run Sup-SimCSE(RoBERTa<sub>base</sub>)+UMAP on 20Newsgroup and show the differences of topic diversities produced by distinct word selecting methods in Table 5. It is clear that TFIDFi and  $\mathbf{TF_i}$  tend to choose common words across multiple topics.
Topic	Weighting Method	Topic Words
Topic 1	TFIDF × IDFiTFIDFiTFi	car bike ride engine brake tire drive mile road front car bike good brake drive make ride time engine tire car bike good drive make time engine ride back year
Topic 2	TFIDF × IDFiTFIDFiTFi	armenian turkish people kill israeli genocide village jewish war government armenian people turkish genocide government make israeli kill time village people armenian turkish make kill government time year state child
Table 5: Comparison of topic words generated using different weighting methods when  $K = 30$ . Repeated words across topics are marked with an underline. Incoherent words are in bold.
<sup>3</sup>https://github.com/MIND-Lab/OCTIS
<sup>4</sup>https://github.com/MaartenGr/BERTopic
<sup>5</sup>https://huggingface.co/models
<sup>6</sup>https://github.com/princeton-nlp/SimCSE