# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 1 Introduction
Actually, two limitations can be found in the parallel architecture:  
* First, the feature interactions of sequential features are ignored. For example, the performance of sequential recommendation in the gaming scenario is constrained without feature interactions between the player's current number of coin stocks and the current number of virtual gadget stocks in each historical behavior. * Second, it cannot learn the feature combinations of non-sequential and sequential features, such as a combination between the user's gender and the genre sequence of the interacted movies in a movie recommendation. To overcome the limitations of the parallel architecture, we aim to integrate feature interaction and sequential recommendation in a cascading manner. We obtain high-order features from non-sequential and sequential feature fields, combine them with the original features, and feed them into the sequence recommendation model for user behavior representation learning. In this paper, we propose a Feature-Interaction-Enhanced Sequence model (FESeq) that considers feature interaction as an incipient feature engineering step for the Transformer model. The high-order features learned from the feature interaction model can further enhance the expressive power of the sequential recommender. Besides, given that linear time intervals and positions in a user sequence are useful for learning the user's temporal interest, we introduce a linear time interval embedding layer and a learned positional embedding layer to the Transformer. Inspired by the explicit user-to-item relevance modeling in DMR [24] and the attention-based sequence pooling in DIN [13], we design a novel attention-based sequence pooling layer that aggregates the Transformer output sequence through attention. Using scaled bilinear attention [25], the attention score in the sequence pooling layer explicitly measures the similarity between the user's historical interactions and the target item. We believe that attention-based sequence pooling is better than simple target pooling and average pooling because it can adaptively learn the relevance of the user's historical behaviors to the target item and assign higher weights to more relevant behaviors. We formulate the recommendation ranking problem as a click-through-rate prediction task. The experimental results reveal that our model is superior to the SOTA CTR prediction methods on public and industrial datasets. Our contributions with this paper are as follows:  
* We propose a novel sequential model named FESeq that makes feature interaction an automatic feature engineering step for the sequential model. The feature interaction in FESeq captures high-order feature combinations and further enhances the representation learning of sequential user interactions. * We introduce a learned positional embedding and a linear time interval embedding in the Transformer to model the timeline interest from the user's historical interactions. * We design a new sequence pooling method based on scaled bilinear attention that explicitly learns the similarities between each user interaction and the target item. * Our extensive experiments validate the effectiveness of the proposed FESeq model. After integrating feature interaction into sequence feature learning in a cascading manner, FESeq beats all SOTA baselines, including a JointCTR framework that jointly trains feature interaction and sequence feature learning modules in parallel (the AUC increases by +1.46% on Ele.me and +0.30% on Bundle). The following sections of the paper are organized as follows: Related works are reviewed in Section 2, and our proposed architecture is presented in Section 3. Experimental results and analyses are included in Section 4, and finally, we conclude our work in Section 5.