# Recurrent Embedded Topic Model
## 1 Introduction
An important weakness that LDA tends to demonstrate is based on the size of the vocabulary, where the bigger the vocabulary, the lower the accuracy (measured by the Coherence-Normalized Perplexity metric); thus, one of the biggest strengths of the ETM is making more accurate predictions where the vocabulary is bigger, as shown by Figure 1. It is important to note that even though when the size of the vocabulary is small, the ETM still outperforms LDA. Although the ETM works great as a combination of LDA and CBOW, it is unable to analyze the order of the words in the text. If the topic models were able to use the sequence of the words, they would outperform current methodologies for calculating topic proportions. To use the order of the words in the text, this issue can be tackled with the recurrency of a Recurrent Neural Networks (RNN). To be specific, it can be tackled by the use of Long Short-Term Memory neural networks (LSTMs) [18], which have been experimented on, with great success, in Natural Language Processing (NLP) problems [19]. In this paper, we propose a new modification to the current ETM (Recurrent Embedded Topic Modelling, RETM) by implementing LSTMs into the NN and making adjustments to the overall architecture, where, measured by the perplexity metric, it severely outperforms the ETM and LDA. The intuition behind using an LSTM is to remove the possibility of having the vanishing/exploding gradient problem that usually hurts recurrent neural networks (given that the recurrency for this problem is based on the order of the words). Some of the experiments that were used to validate the declaration of the RETM outperforming LDA and ETM are based on four different datasets, containing text-based data from emails,books, news and movies which were utilized for calculating the topic distributions. In all four datasets RETM proved to have an improvement of at least 15 times better in the training dataset and at least 10% improvement in test dataset. In a nutshell, the reasoning behind the modification was to use the word order of the documents and reuse the embedding layer from the CBOW model to generate the values of each word inside the calculation of the topic-document matrix (theta) from the ETM model and thus improve the topic assignment for each text. The contribution of this work is based on the perplexity metric for the topic models: when the ETM is fitted with an LSTM using the word order and the embedding layer is used as the recurrency for the RNN while calculating the topic-document distribution, an improvement of at least 15 times on average in the training dataset for all models tested (Combined Topic Model, ZeroShot Cross-lingual Topic Model, LDA, ETM) and an improvement of at least 10% over test data were achieved, ensuring higher-quality topics and making the models more successful for topic modelling-related tasks. To the best of our knowledge, this is the first time recurrence has been implemented in a topic model, as we propose with RETM. The rest of the paper is organized as follows: Section 2 tackles the related work; later, in Section 3, the mathematical support of the model can be found alongside the inference and estimation and the pseudo code on how the model runs; this is followed by Section 4, which contains the code and experimentation description; next, in Section 5, the results are shown in detail and a discussion takes place; and finally, Section 6 will close with the general conclusions of the model and some recommendations for future work.