Improving Topic Modeling for Textual Content with Knowledge Graph Embeddings

Topic modeling techniques has been applied in many scenarios in recent years, spanning textual content, as well as many different data sources. The existing researches in this field continuously try to improve the accuracy and coherence of the results. Some recent works propose new methods that capture the semantic relations between words into the topic modeling process, by employing vector embeddings over knowledge bases. In this paper we study various dimensions of how knowledge graph embeddings affect topic modeling performance on textual content. In particular, the objective of the work is to determine which aspects of knowledge graph embedding have a significant and positive impact on the accuracy of the extracted topics. In order to obtain a good understanding of the impact, all steps of the process are examined and various parameterization of the techniques are explored. Based on the findings, we improve the state of the art with the use of more advanced embedding approaches and parameterizations that produce higher quality topics. The work also include a set of experiments with 2 variations of the knowledge base, 7 embedding methods, and 2 methods for incorporation of the embeddings into the topic modeling framework, also considering a set of variations of topic number and embedding dimensionality. Introduction In the current age of information, larger and larger amounts of data are generated and collected every second around the world. A significant portion of this data is in the form of textual content.The need for understanding this vast amount of textual content keeps increasing as everything in the world becomes more data-driven but mostly because of the fact that it’s impossible for us to do it manually. The fields of Natural Language Processing and Machine Learning offer automated to understand large amounts of textual data. Vector representations of words (Mikolov et al. 2013) (Řehůřek and Sojka 2010) (Pennington, Socher, and Manning 2014) (Joulin et al. 2016) have been used for many Natural Language Processing tasks such as syntactic parsing (Socher et al. 2013a) and sentiment analysis (Socher et Copyright held by the author(s). In A. Martin, K. Hinkelmann, A. Gerber, D. Lenat, F. van Harmelen, P. Clark (Eds.), Proceedings of the AAAI 2019 Spring Symposium on Combining Machine Learning with Knowledge Engineering (AAAI-MAKE 2019). Stanford University, Palo Alto, California, USA, March 25-27, 2019. al. 2013b), and it also is being used in the Topic Modeling field (Hinton and Salakhutdinov 2009)(Srivastava, Salakhutdinov, and Hinton 2013)(Cao et al. 2015)(Nguyen et al. 2015)(Yao et al. 2017). One of these papers with a method called KGE-LDA (Yao et al. 2017) aims to improve the performance of topic modeling by obtaining the vector representations of words from external knowledge bases such as WordNet (Miller 1995) and FreeBase (Bollacker et al. 2008) instead of learning them from documents. According to their reported results, this approach is successful and improves the topic coherence by 9.5% to 44% and document classification accuracy by 1.6% to 5.4% compared to LDA (Blei, Ng, and Jordan 2003). Their approach improves the results with one specific method to obtain the word representations, but it’s not clear whether vectors obtained through other methods that can capture better semantics of networks are able to boost the accuracy of topic modeling. The vector embedding methods that have proven to be more successful in other fields such as Link Prediction can possibly capture the semantics of the external knowledge base more accurately. Another question that remains to be answered in this context is whether a larger knowledge base in terms of entities or a denser knowledge base in terms of relations between entities can also contribute to better representations of words. The primary motive to this question lies in the fact that the knowledge graphs do not have the complete semantic representation of the real world, and can be improved with different relations between entities. This paper presents two approaches to improve Topic Modeling. The first approach applies various Multirelational Network Embedding Methods by computing the vectors on the same network, and incorporating the results into the topic modeling framework that has been taken as the base method of this work. The mentioned embedding methods all follow a translation-based approach to vectors with incremental improvements over the original work which is TransE (Bordes et al. 2013). Since knowledge embeddings are increasingly used for topic modeling, there is lack of a comprehensive study that discovers the effects of knowledge encoded by various methods. Therefore, the primary motive of this work is to push the state of the art in this field forward by the application of more advanced methods and knowledge bases for obtaining better knowledge graph embeddings in order to improve topic modeling. The second approach modifies the network of the knowledge graph itself, and manages to significantly increase the density of the network by adding syntactic dependency relations between words in a sentence that are computed from the same text corpus used for the topic modeling. This combination is performed by computing the dependency trees of the sentences in the text corpus, and adding each relation to the knowledge graph between the corresponding entities, thus updating and enlarging the network. It studies the knowledge encoded by this denser network in terms of relations between entities, and how it affects the overall performance of embeddings, and consecutively topic modeling. The paper is organized as follows: Section 2 presents the related works. Section 3 contains the details of the employed methods that are used in this paper. Section 4 describes the source codes and implementations of the used methods. Section 5 presents the results of the experiments and discusses these outcomes. Section 6 concludes and draws some possible future work.