## 3 The ALIEN Architecture
### Model Description
#### 3.4.1 Attributes-Driven Latent Information Extraction Layer (USER \(\times\) CANDIDATE)
\(H_{l-1}\) denotes the number of feature interactions in the \(l\)-th layer:  
\[H_{l-1}=\dfrac{(m+n)!}{(m+n-l)!\;l!}, \tag{4}\]  
where \(1\leq l\leq O\), and \(O\) denotes the highest order of feature interactions to be modeled in this paper. Equation (3) implies that the order of interactions increases with the growth of the layer depth of CIN. Take as an example the generation of 3-order feature interactions. From Equation (3), we find that the 3-order feature interactions are the output vectors of the second layer in CIN:  
\[\mathbf{\mathcal{F}}_{(3)}^{u,v_{c}}=[\mathbf{X}_{1}^{2},\mathbf{X}_{2}^{2},\cdots,\mathbf{X}_ {H_{2}}^{2}]. \tag{5}\]Similarly, the higher-order feature interactions can be obtained:  
\[\mathbf{\mathcal{F}}_{(i)}^{u,v_{c}}=[\mathbf{X}_{1}^{i-1},\mathbf{X}_{2}^{i-1},\cdots,\mathbf{X} _{H_{i-1}}^{i-1}], \tag{6}\]  
where \(1\leq i\leq O\). We define \(\ell_{\mathbf{\mathcal{F}}_{(i)}^{u,v_{c}}}\) as the number of \(i\)-order feature interactions:  
\[\ell_{\mathbf{\mathcal{F}}_{(i)}^{u,v_{c}}}=\left\{ \begin{array}{ll}m+n,&i=1\\ \frac{(m+n)(m+n-1)}{2},&i=2\\ H_{i-1},&i\geq 3\end{array} \right. \tag{7}\]  
The complete representation vector of low and high-order feature interactions is:  
\[\mathbf{\mathcal{F}}^{u,v_{c}}=[\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}},\mathbf{\mathcal{F}} _{(2)}^{u,v_{c}},\mathbf{\mathcal{F}}_{(3)}^{u,v_{c}},...,\mathbf{\mathcal{F}}_{(O)}^ {u,v_{c}}]. \tag{8}\]  
Thus, the number of low-order and high-order feature interactions \(\ell_{\mathbf{\mathcal{F}}^{u,v_{c}}}\) is:  
\[\ell_{\mathbf{\mathcal{F}}^{u,v_{c}}}=\sum_{i=1}^{O}\ell_{\mathbf{\mathcal{F}}_{(i)}^ {u,v_{c}}}. \tag{9}\]  
From Equations (7) and (9), it is obvious that \(\mathbf{\mathcal{F}}_{(i)}^{u,v_{c}}\in\mathbb{R}^{\ell_{\mathbf{\mathcal{F}}_{(i)}^ {u,v_{c}}}\times k}\) and \(\mathbf{\mathcal{F}}^{u,v_{c}}\in\mathbb{R}^{\ell_{\mathbf{\mathcal{F}}^{u,v_{c}}} \times k}\). _The synthesizer_. Synthesizer is an extension of the self-attention mechanism. The advantage of the synthesizer mechanism is that it replaces the inner product \(\mathbf{Q}\mathbf{K}^{\top}\) in the vanilla self-attention mechanism with the synthesizing function \(\varphi(\cdot)\), which results in reduced computational complexity; parameter costs which are approximately 10% lower than those for the vanilla self-attention mechanism; and competitive performance with the vanilla one [37]. Among its several variants, we utilized a mixed version of synthesizer, i.e., synthesizer (random + vanilla), which is the mixture of synthesizer random and vanilla self-attention models. The architecture of synthesizer (random + vanilla) is illustrated in Figure 2. The objective of synthesizer in the attributes-driven latent information extraction layer (USER \(\times\) CANDIDATE) is to learn the different influences of each feature interaction on the candidate item. Therefore, we achieve it by applying \(\mathbf{\alpha}_{syn}=\varphi(\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}})\in\mathbb{R}^{ \ell_{\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}}}\times\ell_{\mathbf{\mathcal{F}}_{(1)}^{u, v_{c}}}}\) as the feature interaction-level attention score matrix for \(\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}}\). We discuss the influences of feature interactions in Section 5.3.1. \(\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}}\) is used as the input, and the output vector \(\widehat{\mathbf{\mathcal{F}}}^{u,v_{c}}\) of synthesizer is defined as:  
\[\widehat{\mathbf{\mathcal{F}}}^{u,v_{c}}=\mathbf{\alpha}_{syn}\psi_{value}(\mathbf{ \mathcal{F}}_{(1)}^{u,v_{c}})=\varphi(\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}})\psi_{ value}(\mathbf{\mathcal{F}}_{(1)}^{u,v_{c}}), \tag{10}\]  
where  
\[\varphi(\cdot)=\left\{ \begin{array}{ll}\mathbf{W}_{syn},&\text{Random Mode}\\ \text{Softmax}_{syn}(\psi_{query}(\cdot)\psi_{key}(\cdot)^{\top}),&\text{ Vanilla Mode}\\ \text{Softmax}_{syn}(\mathbf{W}_{syn}+\psi_{query}(\cdot)\psi_{key}(\cdot)^{\top}),& \text{Random}+\text{Vanilla Mode}\end{array}