# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 2 Related Work
### Sequential Recommendation  
Sequential recommenders model the sequential user behaviors and predict the next item based on the historical information from the user's action.  
Conventional methods include pattern mining and MC. [PERSON] [32] proposed a personalized sequence pattern mining approach to discover the frequent sequence pattern from the user interaction sequence, but it neglects more fine-grained information in the sequence. Fossil [17] is a high-order MC model that learns local dynamics in the user sequence.  
Previous deep learning methods include RNNs and CNNs. RNN-based sequence models can capture long-term information from the past sequence and predict future items. [PERSON]. [18] applies the Gated Recurrent Unit (GRU) to sequential recommendations and uses ranking loss to train GRUs. [PERSON]. [33] uses Long Short-Term Memory to model the temporal dynamics of user behaviors and item features. [PERSON]. [34] proposed a Hierarchical RNN that learns short-term information about each user session through a session-level RNN and long-term dependencies of all sessions from a user through a user-level RNN. CNNs can model the sequential patterns by using convolutional filters. [PERSON] [19] transforms the user action sequence into an \"image\" and processes the image data by using horizontal and vertical convolutional filters to model point-level and union-level patterns, respectively. [PERSON]. [35] applies a stack of 1D dilated CNNs to enlarge the receptive field and capture long-range item dependencies directly from the sequence data.  
Recent progress in sequential recommendations can be attributed to enhancements in model architectures and pretraining techniques from the Natural Language Processing (NLP) field [36], such as the attention network and Transformer. The idea of attention network architecture [37] is to attend to the historical user interactions most relevant to the target item. DIN directly measures the attention between the interacted items and the target item and aggregates the representations of the interacted items with the attention scores. DIEN improves DIN by using a GRU to extract the user's interest and a GRU with an attentional update gate (AUGRU) to learn the evolution of the user's interest. DMR [24] uses attention pooling to learn the representation of the user's historical interactions.  
The breakthrough in Transformer architectures in NLP also fuels novel Transformer-based methods in sequential recommendations. For example, AtRank [38] leverages a multi-head self-attention mechanism to learn the latent representation of heterogeneous user behaviors and the attention network to combine multiple user behavior representations. SASRec [21] utilizes a Transformer decoder to predict the next item based on the user's historical actions. BST [22] captures long-term dependencies from the user's historical behaviors by using a Transformer encoder and represents the position information as the linear time intervals between the user's historical interaction and the current interaction. BERT4 Rec [39] adopts the vanilla BERT model and a masked item prediction training task to predict the next item. SSE-PT [40] novelly introduces user embeddings to the vanilla Transformer and adds a stochastic shared embedding regularization term to avoid overfitting. TiSARec [41] proposes the relative time interval embedding and designs a self-attention mechanism that combines the absolute position and relative time interval information. DMIN [42] refines the user's sequence behaviors through Transformer layers and models the user's multiple interests through multi-head self-attention, where each head learns a dimension of the user's latent interest.  
Since the Transformer backbone has become mainstream in current CTR prediction models that deal with sequential user data, we adopt the Transformer to model the sequential user behaviors.