# Weight Adjustment Framework for Self-Attention Sequential Recommendation
## 3 Methodology
### Self-Attention Layer  
The self-attention layer is the core of the sequential recommendation system, calculating the relationship weights between elements within the sequence through the self-attention mechanism. This layer enables the model to highlight the most critical interaction features and suppress less important information, optimizing the distribution of attention weights. This is particularly crucial for capturing the temporal dependencies in user behavior. This section will introduce the computational methods of the self-attention layer. Using the filtered matrix \(Z^{l}\), calculate \(Q\) (Query), \(K\) (Key), and \(V\) (Value), respectively. As with the previous transformer, here, \(Q\), \(K\), and \(V\) are all obtained through linear transformations:  
\[\left\{ \begin{aligned} & Q=Z^{l}W^{Q}\\ & K=Z^{l}W^{K}\\ & V=Z^{l}W^{V}\end{aligned} \right. \tag{6}\]  
where \(W^{Q}\), \(W^{K}\), and \(W^{V}\in\mathbb{R}^{d\times d}\) are the learnable matrices in the linear transformations and \(d\) represents the dimension of the vector for each item. To address the issue of suboptimal position encoding potentially leading to the inaccurate allocation of attention weights, we adopted a spatial calibrator scheme [15] and abandoned traditional positional encoding techniques. Specifically, first calculate the order and logarithmic distance between item pairs relative to their positions in the input sequence. We can define the actual order \(o_{ij}\) and the actual logarithmic distance \(d_{ij}\) between positions \(i\) and \(j\) in the input sequence as follows:  
\[o_{ij}=\mathbb{I}(i<j)=\left\{ \begin{aligned} & 1,& i<j\\ & 0,&\text{otherwise}\end{aligned} \right. \tag{7}\]  
\[d_{ij}=\text{ln}(1+|i-j|) \tag{8}\]  
Then, in each self-attention layer, use the query \(\mathbf{q}_{i}^{l}\) and key \(\mathbf{k}_{j}^{l}\) to predict the order and distance that items should have. The predicted order \(\widehat{o}_{ij}\) and distance \(\widehat{d}_{ij}\) can be represented as follows:  
\[\widehat{o}_{ij}=\text{sigmoid}\Big{(}\textit{affine}^{(o)}\left(\left[q_{i} ^{l};k_{j}^{l}\right]\right)\Big{)} \tag{9}\]  
\[\widehat{d}_{ij}=\text{affine}^{(d)}\Big{(}\left[q_{i}^{l};k_{j}^{l}\right] \Big{)} \tag{10}\]  
The difference between the predictions and the true values is measured using sigmoid cross-entropy and \(L_{2}\) loss, after which the original attention weights are modified:  
\[s_{ij}^{(o)}=o_{ij}\text{ln}\big{(}\widehat{o}_{ij}\big{)}+\big{(}1-o_{ij} \big{)}\big{(}1-\text{ln}\big{(}\widehat{o}_{ij}\big{)}\big{)} \tag{11}\]  
\[s_{ij}^{(d)}=-\frac{\theta^{2}\Big{(}d_{ij}-\widehat{d}_{ij}\Big{)}^{2}}{2} \tag{12}\]  
\[\text{A}_{s}=\text{softmax}\Bigg{(}\frac{QK^{T}}{\sqrt{d}}+\mathbf{s}^{(o)}+ \mathbf{s}^{(d)}\Bigg{)} \tag{13}\]  
where \(\text{A}_{s}\) represents the attention weights and \(\theta\) is a learnable scalar. In the self-attention layer, the attention weights can be affected by data noise. To simulate the perturbation process, in the l-th attention layer, we incorporate a uniform distribution \(\mu\) into the attention weights \(\text{A}_{s}^{l}\) through a mask, \(\text{M}^{l}\):  
\[\text{A}_{p}^{l}=\text{M}^{l}\odot\text{A}_{s}^{l}+\left(1-\text{M}^{l}\right) \odot\mu \tag{14}\]  
\[\text{M}^{l}=\text{sigmoid}\left(\frac{Q^{l}W_{\mathbb{Q}_{p}}^{l}\left(K^{l }W_{K_{p}}^{l}\right)^{T}}{\sqrt{d}}\right) \tag{15}\]  
where \(\odot\) denotes element-wise multiplication, \(\text{Q}^{l}\) and \(\text{K}^{l}\) are the query and key at the \(l\)-th layer, and \(\text{W}_{\mathbb{Q}_{p}}^{l}\in\mathbb{R}^{d\times d}\) and \(\text{W}_{K_{p}}^{l}\in\mathbb{R}^{d\times d}\) are learnable matrices.