## 3 Fundamentals
### Recurrent Embedded Topic Modelling  
In this paper, the Recurrent Embedded Topic Model (RETM) is proposed to enhance the quality of the topics for data analysis and data organization purposes by reusing the embedding layer into a recurrent neural network (LSTM). The rationale behind is based on trying to extract the meaning of the word currently analyzed by each of the values it has on the \(L\) dimensional space of the CBOW. To simplify, since it has been proven that the CBOW model contains the \"meaning\" of the word in an \(L\) dimensional space [34], then using the recurrency of the order of the words in the text can help the model to provide \"context\" when determining the topic distributions. Thus, given the recurrency of the \(L\) dimensional space inside an LSTM, the neural network can further predict \(\theta_{d}\), which corresponds to the topic-document distribution of the model. As it can be seen in the pseudo code 2 after obtaining the normalized bag of words for each document on the minibatch, the embedded layer is used to generate the third dimension of the tensor (the recurrency layer of the LSTM). This means that the shape of the tensor is [\(w_{dn}\), \(\mathbf{d}\), \(L\)], and it is important to note that \(\mathbf{d}\) is defined in the pseudo code as the mini batch of documents and \(w_{n}\) is referring to maximum number of words \(w\) that exist in \(D\). After the tensor is created, the LSTM is computed with the variational parameters and then a dropout is executed to prevent overfitting. Next, the computations of \(\mu_{d}\) and \(\Sigma_{d}\) are calculated to sample using a logistic normal distribution, \(\theta_{d}\). Prior to the calculation of the ELBO, the tensor must be returned to a two-dimensional space to be able to multiply against \(\beta\), this reduction of the dimensionality is made with an aggregation based on max (the aggregation is further explain after Algorithm 2), until finally arriving at the calculation of the ELBO and the gradient. Coming back to the training and inference/estimation of the model, it remains the same as what the ETM has already proposed. Algorithm 2 shows how the model works step by step. Some recommendations to ensure that the RETM works properly are to use only a text column (discard everything else) and apply some cleaning techniques to the raw text prior to using the RETM. After the LSTM calculations (in step number 15), where \(\theta_{d}\) is a third-dimensional tensor) and prior to computing the ELBO, \(\theta_{d}\) must be transformed to a two-dimensional tensor. To achieve this two-dimensional tensor, an aggregation was applied by using max. Max was used to prioritize the most salient words for each topic creating clear cut topics. It is important to know that this strategy can create bias to extreme values. If using another aggregation, min can create more general topics with the downfall of hitting the topic coherence metrics where the words will be more dispersed between them. Average can provide a balanced representation of the topics leading to more comprehensive topics. In the end, max was used since it provided better perplexity scores, but the door is not closed for more experimentation on the reduction of the recurrent dimension. ```
1:Generate train and test datasets plus the vocabulary vector
2:Calculate the CBOW or use the standard one
3:Initialize model and variational parameters
4:for epoch do
5: Compute \(\beta_{k}=\text{softmax}\big{(}\rho^{\top}\alpha_{k}\big{)}\) for each topic \(k\)
6: Choose a minibatch d of documents
7:for each document \(d\) in d do
8: Get normalized bag-of-word.