# Enhancing Sequence Movie Recommendation System Using Deep Learning and KMeans
## 3 Methodology of Movie Recommendation Systems
### Model Architecture
#### 3.2.1 Process ONE
The input is passed through three separate linear layers to produce the Q, K, and V matrices, as presented below:  
\[\text{A}_{i}{=}\text{MultiHead}\big{(}\hat{E}\big{)}=\text{Concat}(\text{ head}_{1},\dots,\text{head}_{h})\text{W}_{0} \tag{1}\]  
\[\text{head}_{i}=\text{Attention}\Big{(}\hat{E}\text{W}^{\text{Q}},\hat{E} \text{W}^{\text{K}},\hat{E}\text{W}^{\text{V}}\Big{)} \tag{2}\]  
where \(\text{W}^{\text{Q}}\), \(\text{W}^{\text{k}}\), and \(\text{W}^{\text{V}}\in\mathbb{R}^{\text{dxd}}\) are the projection matrices that makes the model more flexible, \(\hat{E}\) is the embedding matrix of all movies, \(W_{0}\) is a learnable weight matrix that can form the final representation of the movie at \(i\), and \(h\) is the number of heads. In our transformer layer, a point-wise feed-forward network (FFN) enabled the model to adapt, providing the flexibility needed to understand complex movie interactions within the sequence. These networks had hidden layers and non-linear activation functions, which allowed the model to learn complex patterns in the movie ID sequence. FFN consisted of two fully connected layers. At the same time, we utilized dropout layer normalization to optimize the model, while avoiding overfitting and speeding convergence with LeakyReLU in FFN. These were applied to each position, separately and identically, during training. The overall output of the transformer layer between the multi-head attention and stacking FFN layers, which represents the candidate movie that contains characteristics of the users' behavior sequence, can be defined as follows:  
\[\mathrm{N}_{i}=LayerNormalization\big{(}\big{(}\hat{E}\big{)}+Dropout(\mathrm{ A}_{i})\big{)} \tag{3}\]  
\[F_{i}=FFN(N_{i}) \tag{4}\]  
\[T_{i}=LayerNormalization\big{(}N_{i}+Dropout(\mathrm{LeakyReLU}(F_{i}\mathrm{W }_{1}+\mathrm{b}_{1})\mathrm{W}_{2}+\mathrm{b}_{2})\big{)} \tag{5}\]  
where \(\mathrm{W}_{1},\mathrm{W}_{2}\in\mathbb{R}^{\mathrm{dwd}},\mathrm{b}_{1}\), and \(\mathrm{b}_{2}\in\mathbb{R}^{\mathrm{d}}\) refer to the learnable weight's matrix and bias parameters. Next, we concatenated the user demographic embedding layer \(u_{j}\) and the output of the transformer, with target item as \(T_{i}\), for the representation operation of the two vectors and passed them together through a multilayer perceptron (MLP) network. The output layer computed the predicted rating of the next target movie using a dense layer. The dense layer allowed for a linear transformation of all flattened and concatenated features, as expressed below:  
\[x_{i,j}=\mathrm{Concate}\big{(}T_{i},u_{j}\big{)} \tag{6}\]  
\[\hat{y}_{i,j}=\sigma\big{(}\mathrm{W}\!\cdot\!x_{i,j}+\mathrm{B}\big{)} \tag{7}\]  
where \(\hat{y}_{i,j}\) is the predicted rating for user \(j\) on movie \(i\)-th, \(\mathrm{W}\) is the weight matrix for the output layer, (.) is the dot product between the weight matrix \(\mathrm{W}\) and the input vector \(x_{i,j}\)B is the bias vector for the output layer, and \(\sigma\) is the sigmoid activation function.