# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 4 Experiments
### Ablation Studies
#### 4.3.6 Influence of Attention Score Calculation Methods  
Apart from scaled bilinear attention, we also implement two methods to calculate attention scores in the sequence pooling layer, which are the scaled-dot product and DIN attention:  
* **Scaled-dot product** In [20], the scaled-dot product calculates the similarity between the query \(q\) and key \(K_{t}\): \[g(q,K_{t})=\frac{q^{T}K_{t}}{\sqrt{d_{h}}}\] (16) where the scale factor \(\sqrt{d_{h}}\) can avoid excessively large dot product results. The scaled-dot product can simply and directly represent the user-item similarity.
* **DIN attention** In the DIN model [13], attention scores are calculated via MLP networks with Dice activation. In our model, the inputs of MLP are the query \(q\), key \(K_{t}\), and the outer-product of the query and key. The MLP networks then reduce the input data to attention scores. The DIN attention is calculated as \[g(q,K_{t})=\text{Dropout(Dice(Concat($q,K_{t},q\otimes K_{t}$)W^{(1)}+b^{(1)}))}+W ^{(2)}+b^{(2)}\] (17) Compared with the scaled-dot product and scaled bilinear attention, the drawback of MLP-based DIN attention is that it cannot explicitly represent the similarity between the user behavior and the target item. The scaled-dot product is a simple method to calculate the attention score and model the user-item similarity, but it fails to learn more complex user-item relations without additional learning parameters. Compared with the scaled-dot product, scaled bilinear attention can capture more complex information about the relations between the user behavior and the target item.  
As is shown in Table 9, the proposed model uses scaled bilinear attention by default. The model \"Scaled Dot Attention\" uses scaled-dot attention to calculate the attention scores, and the model \"DIN Attention\" uses DIN attention for the attention score calculation. The results verify that using scaled bilinear attention for attention score calculations in the sequence pooling layer achieves the best performance. An explanation for the results is that MLP-based DIN attention cannot explicitly represent the similarity between the user behavior and the target item and that the scaled-dot product fails to learn complex user-item relations without additional learning parameters.  
\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Datasets** & **Evaluation** & **FESeq** & **Target Pooling** & **Mean Pooling** \\ \hline \multirow{2}{*}{Ele.me} & AUC & **0.6127** & 0.6084 & 0.6016 \\  & Logloss & **0.0918** & 0.0923 & 0.0922 \\ \multirow{2}{*}{Bundle} & AUC & **0.8912** & 0.8820 & 0.8838 \\  & Logloss & 0.0117 & **0.0108** & 0.0109 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Ablation studies of models with different sequence pooling methods in sequence pooling layer. The bold number denotes the best value in the column.  
\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Datasets** & **Evaluation** & **FESeq** & **Scaled Dot Attention** & **DIN Attention** \\ \hline \multirow{2}{*}{Ele.me} & AUC & **0.6127** & 0.6124 & 0.5987 \\  & Logloss & **0.0918** & 0.0920 & 0.0939 \\ \multirow{2}{*}{Bundle} & AUC & **0.8912** & 0.8868 & 0.8821 \\  & Logloss & 0.0117 & **0.0109** & 0.0112 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation studies of models with different attention score calculation methods in sequence pooling layer. The bold number denotes the best value in the column.