# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 3 Proposed Model
### A Position and Time-Aware Sequence-Refiner Layer Based on Transformer  
The sequence-refiner layer learns a deep-level representation of the user behaviors by capturing long-range dependencies of the user behavior sequence. The core of this module is a position and time-aware Transformer layer. To model the temporal interest of the user's behavior sequence, both a positional embedding and a linear time interval embedding are concatenated with \(E_{in}^{M}\) and become the input to the Transformer layer, which is shown in Figure 2.  
**Positional embedding**: Since self-attention models do not contain any position-aware structures, such as the recurrent structure in RNNs and convolution filter operations in CNNs, we need to add a learned positional embedding \(p\in R^{d}\) to the feature embedding \(E_{in}^{M}\) in Section 3.2, where \(M\) represents the number of input feature interaction layers. The positional embedding \(p\) is obtained from a look-up table \(P\in R^{n\times d}\) that encodes the position ids to a fixed-length embedding. As more recent behaviors have more influence on the target item prediction, we assign position ids to the user behaviors in a reverse time order. **Linear time interval embedding**: As is shown in [41], positional embedding fails to represent the time interval information between neighboring behaviors. However, user behaviors closer to the current timestamp tend to have greater impacts on the current prediction. Therefore, we propose to insert linear time interval embedding \(ti\in R^{d}\) into the input feature embeddings \(E_{in}^{M}\). Linear time interval embedding embeds the time interval between the timestamp of the historical user behavior \(t_{-i}\) and the current timestamp \(t_{0}\), \(\Delta t_{i}=t_{0}-t_{-i}\), through time embedding \(Ti\in R^{m\times d}\). Finally, the input embeddings of the Transformer layer in the sequence-refiner layer are represented as  
Figure 2: Illustration of the position and time-aware sequence-refiner layer, where both positional and linear time embeddings are concatenated with the user behavior sequence. \[\hat{E}= \begin{bmatrix}E_{in_{1}}^{M}+p_{n}+ti_{n}\\ E_{in_{2}}^{M}+p_{n-1}+ti_{n-1}\\ \ldots\\ E_{in_{n}}^{M}+p_{1}+ti_{1}\end{bmatrix}  \tag{4}\]  
The Transformer layer is causal and contains a multi-head self-attention layer and point-wise feed-forward networks. **Multi-head self-attention layer**: The calculations in the multi-head self-attention layer are almost the same as Equation (2), except that we use multiple self-attention heads. The process is then described as  
\[S=MH(\hat{E})=\text{concat}(head_{1},head_{2},...,head_{h})W^{H} \tag{5}\]  
\[head_{i}=\text{Attention}(\hat{E}W^{Q},\hat{E}W^{K},\hat{E}W^{V}) \tag{6}\]  
where \(W^{Q},W^{K},W^{V}\in R^{d\times d}\), and \(W^{H}\in R^{hd\times d}\) are projection matrices. \(h\) is the number of attention heads. Apart from more efficient parallel computations, multi-head self-attention has a stronger expressive power of learning the user behavior representation than single-head self-attention. As the input sequence is projected to representations in multiple latent subspaces, each subspace can effectively learn a type of the user's heterogeneous behavior or the user's latent interest. **Causality**: The causal Transformer has been applied in tasks such as neural machine translation and next-item recommendation. The \(i+1\)-th element is predicted based on the \(i\)-th element in the sequence. Each item \(i\) only attends to the previous item \(j\) (\(i>j\)) and cannot see the future items, which effectively eliminates the information leakage problem. While our proposed model predicts the target item based on all historical user behaviors, making causality not a requirement, our experiment reveals that a causal Transformer in a sequence-refiner layer leads to a better CTR prediction performance than a non-causal Transformer. A possible explanation is that the sequence pooling layer in Section 3.4 calculates the relevance of all user behaviors and the target item.