## 3 Materials and Methods
### Transformer Layer  
The transformer layer leverages a self-attention mechanism, enabling the model to dynamically assign weights to individual input tokens within sequences derived from skip-gram embeddings. This mechanism facilitates the capture of long-range dependencies and relationships between tokens across the entire input sequence. We use the transformer which is presented by [PERSON] et al. [27]. The transformer comprises a sequence of sub-layers, including multi-head self-attention and position-wise feedforward networks. The multi-head self-attention mechanism projects the input sequence into multiple parallel representations through concurrent applications of attention. In this case, it enables the model to attend to diverse aspects of the input concurrently. Representations are subsequently concatenated and linearly transformed to generate the final output. The position-wise feedforward network introduces non-linearity, further augmenting the model's capacity to capture intricate patterns within the data. Overall, the transformer layer revolutionizedrecommendation engines by providing a scalable and efficient architecture for processing sequential data. Inherent to the transformer architecture is the absence of a native mechanism for capturing sequential order within its input. Positional encodings are incorporated to address this limitation. Encodings are introduced by summation with the input embeddings prior to their presentation to the transformer layers. This summation process empowers the model to distinguish between tokens based on their relative positions within the sequence. A prevalent approach for generating positional encodings leverages trigonometric functions, such as sine and cosine. The calculation of these encoding vectors considers both the position of a token and the dimensionality of the embedding space. The position encoding is calculated with the mathematical Equation (6):  
\[PE_{(pos,2i)}=\sin\Bigl{(}\frac{pos}{10000^{2i/d_{model}}}\Bigr{)} \tag{6}\]  
\[PE_{(pos,2i+1)}=\cos\Bigl{(}\frac{pos}{10000^{2i/d_{model}}}\Bigr{)} \tag{7}\]  
where _pos_ is the position of the item in the sequence, \(i\) is the dimension index of the positional encoding, \(d_{model}\) is the dimensionality of the model's embeddings. These positional encodings are co-trained with the remaining model parameters during the training process. This approach furnishes the model with explicit positional information without introducing the burden of supplementary parameters. The combined application of positional encodings with skip-gram embeddings empowers the model to effectively exploit both the context information gleaned from the skip-gram embeddings and the positional context conveyed by the encodings. The mathematical model of the transformer layer can be described as follows:  
As depicted in Equation (8), the self-attention mechanism is mathematically defined as:  
\[Attention(Q,K,V)=softmax\Bigl{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Bigr{)}V \tag{8}\]  
where \(Q=XW^{Q},K=XW^{K},V=XW^{V}\), and \(d_{k}\) is the dimension of the key vectors. As depicted in Equation (9), multi-head attention is mathematically defined as:  
\[MultiHead(X)=Concat(head_{1},\dots,head_{h})W^{O} \tag{9}\]  
where \(head_{i}=Attention\Bigl{(}XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}\Bigr{)}\), and \(W^{O}\) is the output projection matrix. As depicted in Equation (10), the position-wise feedforward network is mathematically defined as:  
\[FFN(X)=ReLU(XW_{1}+b_{1})W_{2}+b_{2} \tag{10}\]  
where \(W_{1}\) and \(W_{2}\) are weight matrices, and \(b_{1}\) and \(b_{2}\) are bias vectors.