# Recurrent Embedded Topic Model
## 4 Code and Experimentation  
As previously stated, the current model and code act as a modification to the current ETM model that was originally proposed by [PERSON] et al. in their paper Topic Modelling in Embedding spaces [PERSON] et al. (2018). This paper also includes an original version of the code that can be found on Github Github (2018), but although all the core code is inside the Github, in this paper we preferred to use a modification of the code that was created by [PERSON], which provided some adaptations and an overall better structure Github (2018). To be specific, some of the changes inside the code for this paper were based on the architecture of the model. The perplexity calculation inside the training dataset was captured by reusing the existing calculations' (Reconstruction loss) that were happening on each epoch. Another of the big changes has to do with the creation of a new pipeline for loading and transforming (capitalization, stopwords removal and punctuation/special characters removal) the datasets. After the data are ready to be used, the models were fitted, evaluated and compared to each other (LDA, ETM, Zeroshot-CTM, CTM and RETM). The data pipeline, model fitting and evaluation can be found in main.py inside the repository. Requirements.txt lives inside the repository for help on easy installation. Github is available for this code Github (2018). Finally, all the code was implemented on Pytorch, where the stochastic optimization was used with [PERSON] (2018) for the learning rate and using the ELBO as the metric to be optimized on the NN (RETM). Among the models that are used for comparison to RETM, we can find CTM and Zeroshot-CTM, which were both taken from Github (2018). Although there is no direct implementation of the perplexity metrics for CTM and Zeroshot-CTM, the perplexity metric was calculated by reusing the reconstruction loss, in the same way that the ETMand RETM calculate theirs. In the scenario of LDA, the model was used from the Github page of ScikitLearn (2018). For the datasets used in this paper, we compare the performance of the five models in different applications of what unstructured text data might look like. The way we compared the performance was by fitting and predicting the models on different datasets and extracting their perplexity and topic coherence metrics (same data and same process for all five models). The datasets range between emails (2018), genre classification of movies from IMDb (2018), book summaries (2018) and news (2018). All these datasets vary drastically between each other, from the words and sentence structure to the number of words in each document (rows) and the probability of occurrence of each of the words inside the text, as can be seen in Table 2. Although the datasets provide thousands of rows to use, given computational limitations the amount rows that were used was lowered to 30,000, where in experimental fashion, there was small differences comparing to full dataset for each file were used (the shuffle was conducted with random seed to ensure repeatability). The computer that was used to create and run all models has the following specs: System Manufacturer = Gigabyte, System Model = Aorus 15P XC, BIOS = FB07, Operating System = Windows 11 Home, Python 3.9 (plus all the dependencies that can be found on the requirements.txt of the repository), RAM = 32 GB, CPU = Intel Core i7-10870H and GPU = Nvidia GeForce RTX 3070 Laptop. After the dataset is loaded into memory, a small preparation of the text was conducted prior to using it on the model. The text transformation pipeline was as follows:  
1. Apply lower capitalization. 2. Remove stop words using nltk.corpus. 3. Remove all numbers and punctations (including special characters). 4. Randomize all rows with a random seed = 42. 5. Extract the top 30,000 rows. The metric that was used to compare the performance of the model was the perplexity, which represents one of the most important metrics for topic modelling.