# Feature-Interaction-Enhanced Sequential Transformer for Click-Through Rate Prediction
## 3 Proposed Model
### A Sequence Pooling Layer Based on Scaled Bilinear Attention and User-Item Similarity  
Since the sequence-refiner layer preserves the dimension of the input and output data, the output of the sequence-refiner layer is still sequence data. Therefore, pooling is needed to aggregate the sequence data for downstream MLP network prediction. Current popular pooling methods include _concat pooling_ (a concatenation of all behaviors in a sequence [22]), _target pooling_ (the target behavior as the pooling result), _mean pooling_ (the mean of all behaviors in a sequence), and attention-based weighted-sum pooling [24]. While being simple, the first three pooling methods have disadvantages. Concatenation can preserve all sequence behavior representations, but the output of the concat pooling layer is an extremely high-dimensional vector, thus increasing the number of training parameters. Our experiments also demonstrate that concat pooling makes the network training unstable. Worse still, the model cannot even learn from training data from scratch. Target pooling ensures that the pooling output is a low-dimensional vector, which not only preserves the target behavior representation but also includes the connections of all historical behaviors to the target behavior. However, target pooling assumes that only the target behavior has a strong influence on target item prediction and eliminates all direct influences from historical behaviors. Mean pooling assumes that all user behaviors in a sequence have equal influences on the target item prediction. Either assumption is suboptimal for modeling the influence of each user behavior on the target item: although the target behavior tends to have the most importance, the influences of recent historical behaviors are not negligible. Attention-based weighted sum pooling utilizes an attention network to adaptively learn the influences of all user behaviors on the target item and combine all user behaviors in a sequence by using the attention scores. The attention-based weighted sum pooling method has the following advantages:  
* Attention pooling has weaker inductive biases compared with target pooling and mean pooling. Therefore, the model is not constrained by fixed and sometimes flawed prior assumptions. * The attention weights in the attention-pooling method can explicitly represent the similarity between the user behavior and the target item, which is the core idea of collaborative filtering. Therefore, attention pooling is suitable for the CTR prediction task. Next, we will introduce the attention-based weighted sum pooling method. As is shown in Figure 3, the attention network requires the query, key, and value. The key and value can be obtained from two linear projections of the sequence-refiner-layer output \(F\). The query is obtained from a different linear projection of the target item embedding \(e_{out}^{N}\) in Section 3.2. The projections to the query, key, and value are represented as  
\[q=W^{q}e_{out}^{N},K=FW^{K},V=FW^{V} \tag{9}\]  
where \(W^{q},W^{K},W^{V}\in R^{d\times d_{h}}\) are three linear projection matrices. The linear projections unify the dimensions of the user behavior and the target item into the same hidden dimension \(d_{h}\) and improve the flexibility of the attention network. We pass \(q\), \(K\), and \(V\) through a dropout layer and a PReLU activation layer to avoid overfitting. Then, the attention-pooling process is described as follows:  
\[u=\text{PReLU}(\sum_{t=1}^{n}{(\alpha_{t}V_{t})W+b}) \tag{10}\]  
\[\alpha_{t}=g(q,K_{t}) \tag{11}\]  
where the function \(g(\cdot,\cdot)\) represents the equation of the attention score \(\alpha_{t}\). \(K_{t}\) and \(V_{t}\) represent the key and value of the \(t\)-th user behavior, respectively. \(W\) and \(b\) are learnable training parameters that reduce the hidden dimension \(d_{h}\) to the input dimension \(d\). Notably, the attention score \(\alpha_{t}\) reflects the similarity between the \(t\)-th user behavior and the target item.