# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 1 Introduction  
The click-through rate (CTR) prediction task is a binary classification problem that measures the likelihood of a user clicking an item. A higher CTR indicates that a higher percentage of the recommended items are clicked by the users, leading to more revenues for the advertising platforms. Therefore, research in CTR prediction has gained attention from academia and industry [1, 2]. Currently, two mainstream methods exist to solve the CTR prediction task in recommender systems: feature interaction and sequential recommendation. While both methods aim to provide personalized recommendations by mining useful information from the user-item interactions, they process the feature fields differently. Feature interaction methods combine features that describe the current interaction and generate high-order feature combinations relevant to the prediction task. As a specific research field in non-sequential recommendation systems, feature interaction has evolved tremendously. Previous works on feature interaction methods [3, 4] adopted the inner product operation of individual features, which managed to extract simple low-order feature crosses. Wide and Deep [5] first proposed a two-stream structure, whichcomplements a linear model with a deep neural network (DNN) to observe both low-order and high-order feature combinations. Since then, the two-stream architecture has become the paradigm of many current state-of-the-art (SOTA) models, such as DeepFM [6], DCN [7], and FinalMLP [8]. Recent studies achieve feature interaction through the attention network [9; 10] and self-attention mechanism [11]. The multi-stream structure is a plausible solution to further improve the performance of the feature interaction model, which integrates multiple low-order models and a high-order model in a parallel manner [12]. However, feature interaction methods cannot take advantage of the user's historical behavior sequences and thus fail to extract the user's temporal interests and behavioral patterns. Sequential recommendation methods can model the sequential dependencies of the user's historical behaviors. By capturing the underlying sequential patterns along the timeline in the user's actions, such as clicks, purchases, and ratings, sequential recommenders achieve better performance than feature interaction methods in real-world scenarios, such as e-commerce [13], media platforms [14], and news recommendations [15]. Previous sequential recommenders focused mainly on traditional methods, such as Markov chains (MC) and hidden Markov models [16; 17], which calculate the user state transition probability based on the previous N-order interactions. More sophisticated deep learning methods, such as Recurrent Neural Networks (RNNs) [18] and Convolutional Neural Networks (CNNs) [19], then emerged and have been vigorously applied in this field. A multitude of sequential recommendation research nowadays focuses on the Transformer architecture [20]. The self-attention mechanism in the Transformer calculates the relevance of different user behavior pairs to capture the long-term dependencies of the user's sequence data [21; 22], making self-attention-based sequence recommenders perform much better than MC-based and previous deep-learning-based methods. Sequence recommenders focus on mining the dynamics of the user's historical interactions but ignore the high-order features in the user's sequential interactions. In short, either a pure feature interaction model or a pure sequential recommender has its disadvantages in processing non-sequential and sequential feature fields. Therefore, endeavors have been made to integrate feature interaction and sequential recommendation in CTR prediction. For example, JointCTR [23] manages to combine feature interaction models and a sequential recommendation model in a parallel manner. Yet, the parallel architecture treats feature interaction and sequential recommendations separately, making it suboptimal.