SlowGuess/NAACL_2021_OCR huggingface dataset:
on any task is better than continuing to train the encoder on the exact data later used for the CTM. This means that we may not attribute the effectiveness of topic classification solely to the model's seeing in-domain data before topic modeling; rather, some property of fine-tuning itself is better at inducing representations conducive to topic modeling.
Conversely, the TCCTM approach using non-fine-tuned embeddings produces more coherent topics than all fine-tuning tasks except topic classification on in-domain Wikipedia data. This means that the topic classification task itself is also responsible for the high topic coherences observed, and not just the fine-tuned sentence embeddings. Nonetheless, topic classification is more effective when used to fine-tune sentence embeddings, rather than as a part of the CTM objective—further cementing the importance of embeddings to topic quality. There seems to be interference—or perhaps overfitting—when combining TCCTM with embeddings fine-tuned on other tasks. Indeed, fine-tuning on document classification and NLI results in slightly less coherent topics than simply using TCCTM on non-fine-tuned sentence embeddings. Perhaps this could be mitigated with task-specific
Model	French	German	Portuguese	Dutch	MEAN
Match	KL	Match	KL	Match	KL	Match	KL	Match	KL
CTM (No FT)	20.11	0.71	41.68	0.46	24.85	0.67	46.74	0.40	33.30	0.56
CTM+FT (NLI)	53.68	0.39	56.29	0.33	54.38	0.36	56.98	0.31	55.33	0.35
CTM+FT (DC)	35.53	0.61	42.09	0.49	38.12	0.53	49.70	0.40	41.36	0.51
CTM+FT (TC, COVID)	41.09	0.54	46.39	0.47	43.56	0.48	51.11	0.40	45.54	0.47
CTM+FT (TC, Wiki)	45.02	0.50	51.11	0.40	42.58	0.49	50.68	0.40	47.17	0.44
CPT (No FT)	23.62	0.68	40.75	0.45	22.89	0.65	45.13	0.42	33.10	0.55
CPT+FT (NLI)	43.43	0.45	48.09	0.38	43.04	0.46	49.53	0.38	46.02	0.42
CPT+FT (TC, COVID)	41.70	0.53	43.67	0.44	39.91	0.60	47.44	0.43	43.18	0.50
CPT+FT (TC, Wiki)	47.02	0.45	51.53	0.36	45.83	0.44	52.54	0.34	49.23	0.40
TCCTM (No FT)	18.81	0.71	41.18	0.46	19.21	0.72	45.49	0.42	31.17	0.58
TCCTM+FT (NLI)	53.30	0.38	55.52	0.33	53.75	0.37	56.40	0.30	54.74	0.34
TCCTM+FT (DC)	41.83	0.51	48.72	0.42	38.80	0.53	49.73	0.39	44.77	0.46
Random	0.92	1.48	1.22	1.39	1.24	1.48	1.09	1.44	1.12	1.44
Table 2: Percentage of held-out documents assigned the same topic in English and other languages (Match, higher is better) and the mean KL divergence between the English and target language topic distributions per-document (KL, lower is better). For the random baseline, we compare randomly sampled English articles rather than using aligned articles.
fine-tuning over  $\lambda$ ; we leave this to future work.
5.2 Polylingual Topic Modeling
Table 2 presents results for zero-shot cross-lingual topic transfer. All models, including without fine-tuning, are far better than random chance on both metrics. This indicates that multilingual encoders contain enough cross-lingual alignment as is to induce cross-lingual topic alignment. Nonetheless, we also find that fine-tuning the embeddings on any task produces better multilingual topic alignments than not fine-tuning; NLI consistently shows the best cross-lingual transfer. Document classification is generally a worse fine-tuning task than topic classification for cross-lingual transfer, despite achieving similar monolingual performance.
When performing continued pre-training without fine-tuning, we find that results tend to be comparable to the CTM without fine-tuning, though slightly better. When performing both continued pre-training and fine-tuning, we achieve only slightly higher results compared to simply fine-tuning; thus, in both monolingual and multilingual settings, the fine-tuning task is more important for topic transfer than seeing in-domain data or having a better in-domain language model.
The TCCTM objective alone produces fairly poor multilingual topic alignments, despite its posi
Figure 2: Performance on the Semantic Textual Similarity (STS) benchmark (Spearman correlation between cosine similarity of sentence representations and gold labels for STS tasks) versus mean Match and KL per-language for sentence embedding models fine-tuned on various tasks (all with XLM-R-based sentence embeddings.) The outlier is the XLM-R model fine-tuned on NLI, as it was explicitly designed and tuned for STS (Reimers and Gurevych, 2020). We do not include TC-CTM as it does not modify sentence embeddings.
tive effect in monolingual contexts; however, it consistently performs effective cross-lingual transfer when paired with sentence embeddings fine-tuned on document classification. When paired with embeddings fine-tuned on NLI, TCCTM achieves almost the same scores as the CTM model using the same embeddings. Thus, the fine-tuning task
Figure 3: Row-normalized confusion matrices comparing topic assignments from the contextualized topic model in English and French on aligned documents, both without fine-tuned sentence embeddings (left) and with embeddings fine-tuned on NLI (right). Both are based on XLM-R.
used for the sentence embeddings is the most important factor for cross-lingual transfer.
Correlation with Existing Benchmarks To further investigate the role of fine-tuning in inducing better transfer, we employ the Semantic Textual Similarity (STS) benchmark (Cer et al., 2017);[6] this has been used to evaluate the quality of sentence embeddings more broadly in previous works (Reimers and Gurevych, 2019, 2020). Performance is evaluated by measuring the Spearman correlation between the cosine similarity of sentence representations and gold labels for the sentence similarity tasks contained in STS. Here, we try correlating this metric with measures of topic quality, as well as with topic transfer (Figure 2). While STS does not correlate strongly with NPMI ( $\rho = 0.46$ ,  $P > 0.1$ ), it correlates very well with both Match and KL ( $\rho = 0.93$  and  $\rho = 0.96$ , respectively, and  $P < .005$  for both). This implies that well-tuned sentence embeddings are not necessarily the most important factor in producing good topics, but they are quite important for cross-lingual transfer. However, cross-lingual transfer performance saturates quickly at STS Spearman coefficients over 55, such that an increase of over  $50\%$  in STS results in only an  $8\%$  increase in Match and  $4\%$  reduction in KL. Thus, one could perhaps trade off STS for better cross-lingual transfer at scores above this threshold. We leave this to future work.
We find further evidence for STS' weak correlation with NPMI and STS' strong correlation with Match and KL when observing the performance of TCCTM: it does not modify the sentence embeddings, so one would expect that TCCTM would perform similarly to the regular CTM if sentence embeddings are of primary importance. This is not the case for NPMI, as TCCTM seems to greatly improve topic quality when using a non-fine-tuned model and have a slightly negative effect when using a fine-tuned model. However, cross-lingual TCCTM performance is consistently comparable to CTM performance with respect to Match and KL when the fine-tuning datasets are the same.
Why is fine-tuning important for cross-lingual transfer? Figure 3 displays confusion matrices comparing the topics obtained in English versus those obtained in French for the same documents using both the CTM (not fine-tuned) and CTM+FT (NLI) model. We present confusion matrices for all target languages in Appendix A. When the embeddings are not fine-tuned, we see that a typical pattern of error is the CTM assigning foreign documents topics from a small subset of the 100 available topics, regardless of the actual content of the document; this is indicated by the frequency of vertical striping in the confusion matrix. After fine-tuning, errors look more evenly distributed across topics and less frequent in general, though there is still slight striping at topic 81. This striping also occurs after fine-tuning at topic 81 for Portuguese and (to a smaller extent) Dutch, but not German. Thus,
This consists of combined English STS data from SemEval shared tasks from 2012-2017. The exact data we use may be downloaded here: https://sbert.net/datasets/stsbenchmark.tsv.gz
Lang	Sample Document	Topic
en	Niccolò Zucchi was an Italian Jesuit, astronomer, and physicist...	12: star, constellation, sky, cluster, galaxy
fr	Niccolò Zucchi...était un prêtre jésuite italien, astronomer et physique...	12: star, constellation, sky, cluster, galaxy
pt	Niccolò Zucchi fosti um jesùta, astrónomo e=fisico italiano...	12: star, constellation, sky, cluster, galaxy
de	Niccolò Zucchius, auch Niccolò Zucchi, war ein italienischer Astronom und Physiker...	12: star, constellation, sky, cluster, galaxy
n1	Niccolò Zucchi was een Italiaans astronomom...	12: star, constellation, sky, cluster, galaxy
en	Chambilly is a commune in the Saône-et-Loire département...	81: relocated, traveling, transformed, completion, gaining
fr	Chambilly est une commune française, située dans le département de Saône-et-Loire...	51: tributary, border, flows, passes, alps
pt	Chambilly é una comuna francesa...no département de Saône-et-Loire...	89: dubbed, estimate, forty, moment, onwards
de	Chambilly ist eine Französische Gemeinde...im Département Saône-et-Loire...	51: tributary, border, flows, passes, alps
n1	Chambilly is een gemeente in het Franse département Saône-et-Loire...	21: quebec, nord, maritime, seine, calais
Table 3: Sample documents for the topics with highest (top) and lowest (bottom) cross-lingual precision.
CTMs trained on monolingual data are prone to assigning foreign documents topics from a small subset of the available topics, but this can be heavily mitigated with well-tuned sentence embeddings.
What kinds of topics have high cross-lingual precision, and which have lower precision? We calculate the mean precision per-topic of cross-lingual topic transfer from English to all other target languages using the CTM+FT (NLI) model,7 finding that topics which are more qualitatively coherent tend to have higher cross-lingual precision. Topics that are less semantically clear or which compete with similar topics tend to exhibit more cross-lingual variance. Examples of the highest- and lowest-precision topics may be found in Table 3.
We sometimes observe competing topics which semantically overlap. In our dataset, this typically occurs for short articles which describe small towns and obscure places, such as in the bottom example of Table 3; topics 51 and 21 appear most frequently for these articles. Many instances of topics 81 and 89 (the lowest-precision topics in our dataset) also occur in short articles about small towns or obscure places; we hypothesize that this is often due to the probability mass of more relevant topics being split, thus allowing these topics which contain generally higher-probability tokens to be assigned.
In monolingual settings, the best topics are achieved through contextualized topic modeling using sentence embeddings fine-tuned on the topic classification task. This holds whether the topic classification objective is used during fine-tuning or integrated into the CTM itself. However, in zero-shot polylingual settings, it is far more important to
fine-tune sentence embeddings on any task than to have seen in-domain data during pre-training or to use the topic classification objective. As the topic classification task can be performed on any corpus which has enough documents for topic modeling, supervision for this task is always available; this supervision bootstrapping can therefore serve as a simple way to increase topic quality and transfer for contextualized topic models in the absence of any other data, regardless of domain.
There exists a weak but positive correlation between sentence embedding quality (as measured by the STS benchmark) and topic coherence, but a strong correlation between sentence embedding quality and cross-lingual topic transfer performance. Nonetheless, these preliminary findings also suggest that transfer saturates quickly at quite low STS scores and that STS does not correlate well with topic quality, so we do not necessarily recommend directly optimizing over STS for neural topic modeling.
Future work should investigate fine-tuning on multilingual datasets, as well as explicitly inducing cross-lingual topic alignments. Because the CTM currently generates topics in one language and then transfers into other languages, it would also be beneficial to investigate methods of generating topics in parallel across languages during topic modeling.
This material is based on work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 1746891. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
We wish to thank Shuoyang Ding, Chu-Cheng Lin, and the reviewers for their helpful feedback on earlier drafts of this work.
7Recall (and therefore  $F_{1}$  ) is dominated by topics which are consistently incorrectly assigned to foreign documentsthe same topics which cause vertical striping in Figure 3.
Loulwah AlSumait, Daniel Barbara, James Gentle, and Carlotta Domeniconi. 2009. Topic significance ranking of LDA generative models. In Machine Learning and Knowledge Discovery in Databases, pages 67-82, Berlin, Heidelberg. Springer Berlin Heidelberg.
Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2020a. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv preprint arXiv:2004.03974.
Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. 2020b. Cross-lingual contextualized topic models with zero-shot learning. arXiv preprint arXiv:2004.07737.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Jordan Boyd-Graber, Yuening Hu, and David Mimno. 2017. Applications of topic models. 11(2-3):1-154.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems, volume 22, pages 288-296. Curran Associates, Inc.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian LDA for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 795-804, Beijing, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453.
Ran Ding, Ramesh Nallapati, and Bing Xiang. 2018. Coherence-aware neural topic modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 830-836, Brussels, Belgium. Association for Computational Linguistics.
Jingfei Du, Edouard Grave, Belize Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. 2020. Self-training improves pre-training for natural language understanding. arXiv preprint arXiv:2010.02194.
Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 6505-6512.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Shudong Hao and Michael Paul. 2018. Learning multilingual topics from incomparable corpora. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2595-2609.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Jagadeesh Jagarlamudi and Hal Daumé. 2010. Extracting multilingual topics from unaligned comparable corpora. In European Conference on Information Retrieval, pages 444-456. Springer.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR).
Rahul Krishnan, Dawen Liang, and Matthew Hoffman. 2018. On the challenges of learning with inference networks on sparse, high-dimensional data. In International Conference on Artificial Intelligence and Statistics, pages 143-151. PMLR.
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530-539.
Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. 2016. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, page 165-174, New York, NY, USA. Association for Computing Machinery.
Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical word embeddings. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, pages 2418-2424.
Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pages 2410-2419.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pages 1727-1736.
David Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 880-889, Singapore. Association for Computational Linguistics.
Nanyun Peng, Yiming Wang, and Mark Dredze. 2014. Learning polylingual topic models from code-switched social media documents. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 674-679, Baltimore, Maryland. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813.
Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019a. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15-18, Minneapolis, Minnesota. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019b. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569-631.
Holger Schwenk and Xian Li. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Suzanna Sia, Ayush Dalmia, and Sabrina J Mielke. 2020. Tired of topic models? Clusters of pretrained word embeddings make for fast and good topics too! arXiv preprint arXiv:2004.14914.
Akash Srivastava and Charles Sutton. 2017. Neural variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations (ICLR).
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844.
Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020. Which BERT? A survey organizing contextualized encoders. In EMNLP.
Michelle Yuan, Benjamin Van Durme, and Jordan L Ying. 2018. Multilingual anchoring: Interactive topic modeling and alignment across languages. In Advances in Neural Information Processing Systems, pages 8653-8663.
Bing Zhao and Eric P. Xing. 2006. BiTAM: Bilingual topic AdMixture models for word alignment. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 969-976, Sydney, Australia. Association for Computational Linguistics.
Shengjia Zhao, Jiaming Song, and S. Ermon. 2017. InfoVAE: Information maximizing variational autoencoders. arXiv preprint arXiv:1706/02262.
A Confusion Matrices for All Target Languages
Figures 4, 5, 6, and 7 present row-normalized confusion matrices comparing topic assignments for aligned documents in English and all other target languages. We present figures for CTMs based on non-fine-tuned embeddings (left) as well as embeddings fine-tuned on NLI (right). All embeddings are based on XLM-R.
B English Topic Counts
As the provided confusion matrices are row-normalized, they do not present the relative frequency of various topics in English. Thus, we present counts of the most probable topics for the English test documents according to a CTM based on non-fine-tuned embeddings and a CTM based on embeddings fine-tuned on NLI (Figure 8).
C More Example Topics
Table 4 presents more sample documents for various high-precision topics. The lowest-precision topics all contain similar top tokens and patterns of error as Table 3 (as topics 81 and 89 do), so we focus on displaying other types of topics which transfer well across languages.