# Enhancing Sequence Movie Recommendation System Using Deep Learning and KMeans
## 3 Methodology of Movie Recommendation Systems
### Model Architecture
#### 3.2.1 Process ONE  
Predicting User Ratings for Movies  
This section introduces the first process used for model rating prediction. The primary purpose of our work was to predict how a particular user might rate a given movie. Figure 1: Movie recommendation system architecture. The process includes various necessary elements suitable for our predictive model. The model took in features such as user ID, movie ID, user's demographic data, and historical movie ratings (sequence experiences of the target user), and output predicted ratings. User-Movie Embedding Layer  
We divided learning data into two parts: user demographic and movie sequence embedding. User demographic data consisted of data such as user ID, gender, age, and occupation. Each category of input features first passed through the embedding layer to become a dense feature vector. An embedding matrix \(u_{j}\in\mathbb{R}^{u\times d}\) was created to transform the integer index into a dense vector of fixed size, where \(u\) represented the vocabulary size (the total number of unique categorical elements), and \(d\) was the embedding dimension. These dense vectors could typically reduce the dimensionality of the input features, capturing the underlying relationships and characteristics of each categorical feature to feed them directly into the multi-layer perceptron (MLP). Movie features contained the movie IDs in sequence, and movie genre. Firstly, the movie IDs, in sequence, were transformed into dense feature vectors through the embedding layer. The input movies' genre features were transformed into feature vectors through muti-hot encoding. We also obtained embeddings for each movie ID in the behavior sequence, including the target movie, which was essential for the transformer model to understand the sequence's temporal dynamics. Furthermore, we incorporated a learnable position encoding matrix to enhance the input order sequence to address the lack of inherent positional information in the transformer architecture. For each movie, we then concatenated the movie ID sequence with the positional encoding matrix and created an embedding matrix \(\hat{E}_{i}\in\mathbb{R}^{\text{nxd}}\), which represents the embedding vector of the \(i\)-th movie in each behavior sequence of user u after adding the position vector. In our process, we fed our sequence of movie embeddings into a single transformer layer, before concatenating the output with the user features. Model Training  
The transformer model was a type of deep learning model that was primarily used in the processing of sequential data such as natural language. It could capture the user dynamic nature influences on users' recent activities. In our study, we analyzed the structure of the transformer layer, which enhanced the model's ability to capture the long-range dependencies and relationships between movies in a sequence, allowing it to make more accurate and personalized recommendations. The transformer layer comprised a multi-head attention layer and a Position-Wise Feed-Forward Neural Network (FFN). The multi-head attention mechanism allowed for the model to focus on different sections of the input sequence in different ways (e.g., longer-term dependencies versus shorter-term dependencies) when making prediction ratings for each target movie. The multi-head attention module can be used in algorithms. The module is expected to be process information multiple times in parallel via an attention mechanism. This mechanism allows the model to jointly attend to information from different representation subspaces at different positions, providing a more complex understanding of the input sequence. After that, the output of the distinct attention was connected and linearly transformed into the aspect that was predicted. In our case, the multi-attention operation takes the embedding \(\hat{E}_{i}\) as input.