Topic Modelling with Word Embeddings
Authors: Fabrizio Esposito, Anna Corazza, Francesco Cutugno (Università di Napoli Federico II, 2016)
Topic: Distributional Semantics

Abstract
This work aims at evaluating and comparing two different frameworks for the unsupervised topic modelling of the CompWHoB Corpus, namely our political-linguistic dataset. The first approach is represented by the application of the latent DirichLet Allocation (henceforth LDA), defining the evaluation of this model as baseline of comparison. The second framework employs Word2Vec technique to learn the word vector representations to be later used to topic-model our data. Compared to the previously defined LDA baseline, results show that the use of Word2Vec word embeddings significantly improves topic modelling performance but only when an accurate and task-oriented linguistic pre-processing step is carried out.

Introduction
Over recent years, the development of political corpora AUTHOR has represented one of the major trends in the fields of corpus and computational linguistics. Being carriers of specific content features, these textual resources have met the interest of researchers and practitioners in the study of topic detection. Unfortunately, not only has this task turned out to be hard and challenging even for human evaluators but it must be borne in mind that manual annotation often comes with a price. Hence, the aid provided by unsupervised machine learning techniques proves to be fundamental in addressing the topic detection issue.\newline Topic models are a family of algorithms that allow to analyse unlabelled large collections of documents in order to discover and identify hidden topic patterns in the form of cluster of words. While LDA AUTHOR has become the most influential topic model AUTHOR, different extensions have been proposed so far: Rosen-Zvi et al. AUTHOR developed an author-topic generative model to include also authorship information; Chang et al. AUTHOR presented a probabilist topic model to infer descriptions of entities from corpora identifying also the relationships between them; Yi Yang et al. AUTHOR proposed a factor graph framework for incorporating prior knowledge into LDA.\newline In the present paper we aim at topic modelling the CompWHoB Corpus AUTHOR, a political corpus collecting the transcripts of the White House Press Briefings. The main characteristic of our dataset is represented by its dialogical structure: since the briefing consists of a question-answer sequence between the US press secretary and the news media, the topic under discussion may change from one answer to the following question, and vice versa. Our purpose was to address this main feature of the CompWHoB Corpus associating at each answer/question only one topic. In order to reach our goal, we propose an evaluative comparison of two different frameworks: in the first one, we employed the LDA approach by extracting from each answer/question document only the topic with the highest probability; in the second framework, we applied the word embeddings generated from the Word2Vec model AUTHOR to our data in order to test how dense high-quality vectors represent our data, finally comparing this approach with the previously defined LDA baseline. The evaluation was performed using a set of gold-standard annotations developed by human experts in political science and linguistics. In Section 2 we present the dataset used in this work. In Section 3, the linguistic pre-processing is detailed. Section 4 shows the methodology employed to topic-model our data. In Section 5 we present the results of our work.

Conclusion
In this contribution we have presented a comparative evaluation of two unsupervised learning approaches to topic modelling. Two experiments were carried out: in the first one, we applied a classical LDA model to our dataset; in the second one, we trained our model using Word2Vec so as to generate the word embeddings for topic-modelling our test set. After clustering the output of the two approaches, we evaluated them using the external criterion of purity. Results show that the use of word embeddings outperforms the LDA approach but only if a linguistic task-oriented pre-processing stage is carried out. As at the moment no comprehensive explanation can be provided, we can only suggest that the main reason for these results may lie in the fluctuating length of each document in our dataset. In fact, we hypothesise that the use of word embeddings may prove to be the boosting factor of Word2Vec topic model since encoding information about the close context of the target term. As part of future work, we aim to further investigate this aspect and design a topic model framework that could take into account the main structural and linguistic features of the CompWHoB Corpus. The authors would like to thank Antonio Origlia for the useful and thoughtful discussions and insights.