# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 3 Proposed Model
### Feature Interaction as an Automatic Feature Engineering Step for Sequential Recommendation  
We propose to stack the feature interaction layers and the sequential model to enhance the expressiveness of the sequential model. We introduce an interacting layer to extract high-order features from the original ones. Besides, we find that the input feature fields of the interacting layer should include both sequential features and non-sequential features because a feature interaction model such as AutoInt can capture the relevance between two heterogeneous feature fields. For example, in [11], AutoInt assigns significant influences between the genre of the movie, a sequential feature field in the sequential recommendation setting, and the gender of the user, a non-sequential feature field. To combine non-sequential and sequential feature fields, the non-sequential fields are transformed into sequential fields. Specifically, we broadcast non-sequential feature embeddings \(e_{ns}\) to sequences \(E_{ns}=(e_{ns},e_{ns},...,e_{ns})\), which have the same length as sequential features, and concatenate \(E_{ns}\) with sequential feature embeddings \(E_{s}\) along the feature field dimension to generate embeddings from all feature fields \(E_{in}\), which are then the input of the interacting layer. We choose self-attention layers in AutoInt [11] as the core part of the interacting layer. The \(i\)-th AutoInt layer based on self-attention requires the following operations:  
\[Q=E_{in}W^{Q},K=E_{in}W^{K},V=E_{in}W^{V} \tag{1}\]  
\[\hat{E}_{in}=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d}}\bigg{)}V \tag{2}\]  
\[E_{in}^{i}=ReLU(\hat{E}_{in}^{i-1}+E_{in}^{i-1}) \tag{3}\]  
where \(W^{Q},W^{K},W^{V}\in R^{d\times d}\), and \(d\) are the dimensions of query, key, and value. According to Figure 1, for the interacting layer with n AutoInt layers, the output of the last AutoInt layer \(E_{in}^{n}\) becomes the input of the Transformer layer. Therefore, FESeq unifies the feature interaction and the sequential model by stacking two types of layers together. The interacting layer can be considered as an automatic feature engineering step before the sequential modeling. Due to the residual connection in each AutoInt layer, the output of the last AutoInt layer \(E_{in}^{n}\) maintains the original features and generates as high as \(2^{n}\)-order feature combinations. The AutoInt layers engineer new high-order features from the provided feature fields. This feature engineering process can enhance the relevance of different original features and provide informative feature combinations for the sequential modeling of user behaviors. The feature engineering step is automatic because, after the joint training of the interacting layer and the sequential model, the interacting layer can automatically learn high-order features useful to the predictive sequential model. Here are four details about the interacting layer not shown in Equation (3) for simplicity. * In general, feature fields at different positions of a user behavior sequence tend to share similar high-order feature combinations. Thus, we introduce parameter sharing so that the input fields at each position of the user behavior sequence share the same parameters in the interacting layer. * Since the interacting layer preserves the dimension, we can further stack multiple layers to extract higher-order feature interaction. * The AutoInt model can achieve base-2 exponential degree growth. Therefore, stacking a few layers (no more than three layers) is sufficient to extract high-order feature interactions in real-world settings. * Input feature interaction layers and output item feature interaction layers are designed to extract feature interactions from input feature embeddings and output item embeddings, respectively.