# Recurrent Embedded Topic Model
## 1 Introduction  
In recent years, understanding what is stored inside big data has proved to be a challenge. According to the estimations of [PERSON] in Statista [1], in 2023, there should be around 120 zettabytes of data created, captured, consumed and copied worldwide. This trend keeps getting higher every single year (Table 1). Experts from Intersystems [2] estimate that 85% of all data that exist are unstructured data, most of which is held in text-based documents such as emails, memos, documents, social media feeds, etc. Topic modelling is an area of data analytics that analyzes the words of texts to discover the themes that run through them and help with the organization of unstructured, text-based data at a scale that would be humanly impossible [3]. \begin{table}
\begin{tabular}{c c} \hline \hline
**Year** & **Number of Zettabytes Estimated to Be Stored Worldwide** \\ \hline
2020 & 64.2 \\
2021 & 79 \\
2022 & 97 \\
2023 & 120 \\
2024 & 147 \\
2025 & 181 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Table shows an estimation of how many zettabytes are currently stored worldwide. Data from Petroc Taylor in Statista [1]. Several topic models exist but there has been a clear evolution in the quality of the topic models over the years. Some of the common ones are based on latent semantic analysis but were outperformed by Latent Dirichlet Allocation (LDA), which is also currently one of the most popular models [4; 5]. LDA has been used in a variety of fields and for a variety of use cases. Some of the highlights are in political science (where 50 years' worth of data was fitted into LDA to descriptively understand the solutions and strategies that have been adopted to reduce crime) [6], software engineering (where LDA was fitted on source code from 1555 projects and managed to extract the main concepts from the source code) [7] and social media (where a modification of LDA was used to align events and Twitter feedback) [8]. Although LDA performs a good job at topic modelling, when using specific metrics to measure quality such as perplexity, it still has a long way to go. Perplexity is a commonly used metric for measuring the performance of language models [9]. When LDA is measured by perplexity against modifications of the model, it proves to be outperformed every time [10; 11; 12], demonstrating that there is still room for improvement in modifying LDA. LDA assumes that each document can be represented by a mixture of topics. The aim of the model is to create a distribution with the proportions of how much every topic has been mentioned, throughout the document, and a distribution with the importance of each word on representing each topic. When LDA is fitted to a database of documents, it will provide a low-dimensional representation for each document and word representations for each topic [5]. On the other hand, there have been realizations in other unsupervised learning problems, such as transforming raw words into vectors. The embedding layer was first proposed by [PERSON] et al. [13] and was later explored by [PERSON] et al. [14; 15], where they noticed a high level of accuracy for words that have similar syntactic and semantic relationships [14] and can also capture synonyms, antonyms and spelling variations [16], until finally arriving at the Continuous Bag of Words (CBOW) embedding layer [15]. Both models (LDA and CBOW) have been used together by [PERSON] et al. [17] in their embedded topic modelling (ETM). The ETM can be seen as the best of its two parts: from one side, as a topic model, it can provide a low-dimensional vector with the proportions of the topics inside the text trying to generate all the syntactic and semantic relationships; on the other side, it acts as an embedding model, where it can represent the vocabulary as more robust and standardized vectors and exploit the relationships between words, making the ETM a superior topic model.