# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 4 Experiments
### Visualization Results
#### 4.5.2 Attention Scores in Sequence Pooling Layer  
Figures 7 and 8 represent the attention scores in the sequence pooling layer on both datasets. The results are averaged on the test samples. We do interpolation smoothing on the attention score heatmaps. According to the heatmaps of the two datasets, we can obtain the following conclusions:  
* More recent user behaviors obtain higher attention weights, which shows that more recent user behaviors tend to be more important for the target item prediction task.
* Compared with target pooling, the attention pooling in the heatmaps attends to both the current user behavior and a small portion of historical behaviors informative for the prediction task. Therefore, the visualization results can demonstrate the superiority of attention pooling to target pooling in the sequence pooling layer.  
quence model to provide high-order features for the Transformer. Besides, positional embeddings and linear time interval embeddings are introduced to the Transformer layer to model the positional and time interval information of user behaviors. We also novelly pool the Transformer layer output with scaled bilinear attention that explicitly learns the relevance between the user's historical behaviors and the target item. Extensive experimental results on both public and private datasets show that our proposed model outperforms SOTA baselines on the CTR prediction task. Ablation studies confirm that the interacting layer can extract high-order information from both sequential and non-sequential features before the Transformer model. Besides, we observe the effectiveness of embedding positional and timestamp information and maintaining causality in the Transformer layer. The experiments further reveal the importance of attention pooling after the Transformer layer. Among the various attention mechanisms investigated, scaled bilinear attention is superior in modeling user-item relevance. Finally, our visualization results regarding feature interaction weights and attention-pooling weights provide good model explainability. In the future, we plan to replace the self-attention interacting layer with more recent SOTA feature interaction modules to enhance the model performance.  
Conceptualization, S.G.; methodology, Q.Y. and M.Z.; resources, M.Z.; software, Q.Y. and S.G.; writing--original draft preparation, Q.Y. and H.L.; writing--review and editing, H.L. and Y.L.; funding acquisition, Q.Y.; supervision, Y.L. and M.Z.; visualization, H.L. All authors have read and agreed to the published version of the manuscript.  
This work is supported by the National Natural Science Foundation of China (Grant No. 62071189).  
Not applicable.  
Our data and source code are available at [[https://github.com/liuhaozhe6788/FESeq](https://github.com/liuhaozhe6788/FESeq)]([https://github.com/liuhaozhe6788/FESeq](https://github.com/liuhaozhe6788/FESeq)), accessed on 21 March 2024. The dataset Bundle is available from the corresponding author upon reasonable request, this data is not publicly available due to privacy.  
The author wishes to extend sincere appreciation to [PERSON] for providing access to the Bundle dataset, which was crucial for this study. Gratitude is also owed to [PERSON] for his expert assistance with data visualization, which greatly enhanced the understanding and presentation of the research findings.  
The authors declare no conflicts of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} MC & Markov chains \\ FM & Factorization machine \\ PNN & Product-based neural networks \\ GRU & Gated Recurrent Unit \\ AUGRU & GRU with an attentional update gate \\ AUC & Area under the curve \\ CNN & Convolutional Neural Network \\ CTR & Click-through rate \\ DIN & Deep interest network \\ DNN & Deep neural network \\ FESeq & Feature-Interaction-Enhanced Sequence Model \\ MLP & Multilayer perceptron \\ RNN & Recurrent Neural Network \\ SOTA & State-of-the-art \\ \end{tabular}