## 3 Methodology and Materials
### Applying with Attention Mechanisms
#### 3.2.1 Applying Self-Attention with a Positional Layer  
In models such as Transformers, the self-attention mechanism includes multiple attention heads within each self-attention sublayer. For an input sequence, each attention head processes an input sequence, \(\mathbf{x}=(\mathbf{x}_{1},...,\mathbf{x}_{n})\), consisting of n elements, where \(x_{i}\in\ \mathbb{R}^{d_{x}}\), and computes a new sequence \(\mathbf{z}=(\mathbf{z}_{1},...,\mathbf{z}_{n})\) of the same length, where \(z_{i}\in\ \mathbb{R}^{d_{z}}\). Each output element, \(z_{i}\), is determined by a weighted sum of the input elements that have been linearly transformed (Equation (1)). \[z_{i}=\sum_{j=1}^{n}\alpha_{ij}\Big{(}x_{j}W^{V}\Big{)} \tag{1}\]  
The weight coefficient \(\alpha_{ij}\) for the contribution of input element \(x_{i}\) to output element \(z_{i}\) is computed using a SoftMax function (Equation (2)). \[\alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^{n}\exp e_{ik}} \tag{2}\]  
The SoftMax function (Equation (2)) is applied to a set of scores that measure the compatibility or similarity between \(x_{i}\) and \(x_{j}\). These scores are typically computed using a trainable linear transformation of the input elements, involving dot products among the query, key, and value vectors derived from the input elements. Finally, the output of each attention head, represented by different \(z\) sequences, is concatenated and linearly transformed through a fully connected layer. Algorithm 1 outlines the function Self_Attention, which takes three input matrices: \(Q\) (_queries_), \(K\) (_keys_), and \(V\) (_values_). The function initializes three empty lists: _attention_scores_, _attention_weights_, and _output_. For each query in \(Q\), the dot product is computed with each key in \(K\) to generate attention scores. These scores are normalized using the SoftMax function to obtain the attention weights. Each value in \(V\) is then multiplied by its corresponding attention weight, and the results are aggregated into the output list. In an additional approach, [PERSON] et al. [38] introduced positional encodings before self-attention, allowing the model to capture the order of the sequence. Incorporating positional layer self-attention into data preprocessing primarily aims to leverage its capability to effectively understand and encode the positional relationships within sequential data. Traditional methods of sequence data processing often depend on manual feature engineering or simplistic assumptions about sequences, which can result in a significant loss of context and information. In contrast, positional layer self-attention automates the recognition of complex dependencies, thus avoiding the pitfalls of traditional approaches. In positional layer self-attention, data preprocessing enhances the performance of self-attention mechanisms by ensuring that the temporal or sequential relationships inherent in the data are not only preserved but are also highlighted and actively utilizedfor analysis or predictions. ```
1 Input: Q, K, V;
2 Output: output list S
3 Initialized empty lists: attention_scores, attention_weights, output
4 for each query in Qdo
5 for each key in K do
6 Compute attention score as dot product of query and key
7 Add attention score to attention_scores list
8 endfor
9 for each score in attention_scores do
10 Compute attention weight as SoftMax of score
11 Add attention weight to attention_weights list
12 endfor
13 for each value in V do
14 Multiply value by corresponding attention weight
15 Add result to output list
16 endfor
17 endfor
``` ```  
**Algorithm 1**Self_Attention  
Moving to the method by which a positional self-attention system addresses experimental challenges by incorporating positional information, consider that the input representation \(x_{i}\in\mathbb{R}^{n\times d}\) contains the d-dimensional embeddings for \(n\) tokens of a sequence. The positional encoding outputs \(X+P\) using a positional embedding matrix \(P\in\mathbb{R}^{n\times d}\) of the same shape.