DeepEnFM: Deep neural networks with Encoder enhanced Factorization Machine
Authors: Qiang Sun
Zhinan Cheng
Yanwei Fu
Wenxuan Wang
Yu-Gang Jiang
Xiangyang Xue
Venue:  (2020)
Keywords: CTR
Attention
Transformer
Encoder

Abstract: Click Through Rate (CTR) prediction is a critical task in industrial applications, especially for online social and commerce applications. It is challenging to find a proper way to automatically discover the effective cross features in CTR tasks. We propose a novel model for CTR tasks, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). Instead of learning the cross features directly, DeepEnFM adopts the Transformer encoder as a backbone to align the feature embeddings with the clues of other fields. The embeddings generated from encoder are beneficial for the further feature interactions. Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs. Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing information among different attention heads. Our model is validated on the Criteo and Avazu datasets, and achieves state-of-art performance.

Under review as a conference paper at ICLR 2020
DEEPENFM: DEEP NEURAL NETWORKS WITH EN- CODER ENHANCED FACTORIZATION MACHINE
Anonymous authors Paper under double-blind review
ABSTRACT
Click Through Rate (CTR) prediction is critical task in industrial applications, especially for online social and commerce applications. It is challenging to find proper way to automatically discover the effective cross features in CTR tasks. We propose a novel framework for CTR task, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). Instead of learning the cross features directly. DeepEnFM adopts the Transformer encoder as a backbone align the feature embeddings by the clues of other fields. The embeddings gen- erated from the encoder are beneficial for further feature interactions. Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs. Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing informa- tion among different attention heads. Our model is validated on the Criteo and Avazu datasets, and achieves the state-of-art performance.
INTRODUCTION
This paper studies the problem of predicting the Click Through Rate (CTR), which is an essential task in industrial applications, such as online advertising, and e-commerce. To be exact, the adver- tisements of cost-per-click (CPC) advertising system are normally ranked by the eCPM (effective cost per mille). which is computed as the prodcut of bid price and CTR (click-through) rate). To pre- dict CTR precisely, feature representation an important step in extracting the good, interpretable patterns from training data. For example, the co-occurrence of "Valentine's 's Day", "chocolate" and "male" can be viewed as one meaningful indicator/feature for the recommendation. Such hand- crafted feature type is predominant in CTR prediction (Lee et al. 2012). until the renaissance of Deep Neural Networks (DNNs).
Recently, a more effective manner, i.e., representation learning has been investigated in CTR predic- tion with some works (Guo et al. 2017 Qu et al., 2016 Wang et al. 2017 Lian et al., 2018 Song ett al. 2018). which implicitly explicitly learn the embeddings of high-order feature extractions among neurons or input elements by the expressive power of DNNs or FM. Despite their noticeable performance improvement, DNNs and explicit high order feature-based methods (Wang al. 2017
Zhang et al. 2016; Guo et al. 2017 Lian et al. 2018) seek better feature interactions merely based on the naive feature embeddings.
Few efforts have been made in addressing the task of holistically understanding and learning rep- resentations of inputs. This leads many practical problems, such "polysemy" in the learned feature embeddings existed in previous works. For example, the input feature 'chocolate' is much closer to the 'snack' than 'gift' in normal cases, while we believe 'chocolate' should be better paired with "gift' if given the occurrence input "Valentine's Day". This is one common polysemy prob- lem in CTR prediction.
Towards fully understanding the inputs, we re-introduce to CTR, the idea of Transformer en- coder (Vaswani et al.] 2017), which is oriented in Natural Language Processing (NLP). Such an encoder can efficiently accumulate and extract patterns from contextual word embeddings in NLP, and thus potentially would be very useful in bolistically representation learning in CTR. Critically, the Transformer encoder has seldom been applied to CTR prediction with the only exception arxiv paper AutoInt (Song et al.) 2018), which, however, simply implements the multi-head self- attention (MHSA) mechanism of encoders, to directly extract high-order feature interactions. We

---

Under review as a conference paper at ICLR 2020
argue that the output of MHSA/encoder should be still considered as first-order embedding influ- enced by the other fields, rather than a high-order interaction feature.
To this end, our main idea is to apply the encoder to learn context-aware feature embedding, which contains the clues from the content of other features. Thus the "polysemy" problem can be solved naturally, and the second-order interaction of such features
represent more meaning. Contrast
to AutoInt(Song et al. 2018), which feeds the output of encoder directly to the prediction layer or a DNN, our work not only improves the encoder be more suitable for CTR task, but also feeds the encoder output to FM, since both our encoder and FM are based on vector-wise learning mechanism. And we adopt DNN to learn the bit-wise high-order feature interactions in a parallel way, which avoids interweaving the vector-wise and bit-wise interactions in stacked way.
Formally. we propose novel framework Deep neural networks with Encoder enhanced Factoriza- tion Machine (DeepEnFM). DeepEnFM focuses on generating better contextual aligned vectors for FM and uses DNN as a bit-wise information supplement. The architecture adopting both Deep and FM part is inspired by DeepFM (Guo et al.) (017). The encoder is endowed with bilinear attention and max-pooling power. First, we observed that unlike the random order of words in a sentence, the features in a transaction are in fixed order of fields. For example, the fields of features are arranged in an order of [Gender, Age, Price ...}. When the features are embedded in dense vec- tors, the first and second vectors in transaction always represent the field "Gender" and "Age". To make use of this advantage, we add a bilinear mechanism to the Transformer encoder. We use bilinear functions different field pairs is modeled with different functions. The embedding size in CTR tasks is usu- ally around 10, which allows the application of bilinear functions without unbearable computing complexity. Second, the original multi-head outputs are merged by concatenation, which considers the outputs are complementary to each other. We argue that there are also suppressing information between different heads. We apply max- poollini merge mechanism to extract both complementary and suppressing information from the multi-head outputs. Experimental results on Criteo and Avazu datasets have demonstrated the efficacy of our proposed model.
replace the simple dot product in attention. In this way, feature similarity of
To summarize, our contributions are three-fold. (1) We propose a novel model DeepEnFM, which combines both DNN and encoder enhanced FM. (2) The state-of-the-art results on Criteo and Avazu datasets have shown the efficacy of our proposed model. (3) Extensive ablation study has shown the contribution of each component, and revealed the relations between DNN, encoder, and FM by developing different architectures.
RELATED WORK
CTR prediction is critical task in Recommendation System (Adomavicius and Tuzhilin, 2005 Batmaz et al. 2019), which aims to predict the probability of a user click behavior based on the given item. The key challenge of the CTR is automatically learn combinatory/cross features (e.g., a 2-way feature: "Valentine's Day" and "chocolate", a 3-way feature: "Valentine's Day", "chocolate" and "male") from the inputs.
Traditional learning methods, such as Logistic Regression (Le) et al. 2012), learn weights of processed features in supervised way for prediction. To gain a better performance, the manually designed combinatory features are fed into LR model, which is laborious and unable to cover the numerous combination of features.
To learn the combinatory features automatically in a explicit way, Rendle proposes Factorization Machine (Rendle 2010) model. which treats the weight of feature pair as the dot of two latent vectors. Moreover, HOFM (Blondel et al., 2016) and CIN (Lian et al. 2018) are proposed to capture the interaction of arbitrary order explicit manner. However, the primary/fist-order field em- beddings are utilized without considering the contextual information, which leads to the 'polysemy' problem in field embedding unsolved.
Deep Neural Networks (DNN) (Krizhessky et al. 2012 have made remarkable progresses in CV
and NLP tasks by its extraordinary representation learning power Several DNN based CTR models developed on the basis of primary field embeddings 4Zhang et al. 2016 Cheng et al. 2016 Guo et al. 2017; Lian et al.] 20188) Wang et al.] (0077) or product layers (Qu et al. 2016) with a plain DNN. One of the major drawbacks of the plain DNN is that it tackles the field embeddings as large
2

---

Under review as a conference paper at ICLR 2020
Last Normstiztion
FM
DNN
Position-wisc Feed Forward
1000
00oo
Encoder
0000 OOCO 0000
x0.3
Multi-Head Attention
0.3
pooa
oooo
pooo
Field x,
Field xs
Fields,
Fieldx
Field
Umbeddinge
Limbedding
Umbedding
(a) Framework
(b) The embedding layer
Structure of the encoder layer
Figure 1: (a) The framework is mainly composed of encoder, DNN and FM. (b) The embedding layer maps both categorical fields and numerical fields into dense vectors. (c) The encoder layer composed of multi-head self-attention module (MHSA) and position- feed forward module with the layer normalization and residual connection.
flat layer, which means there are only interactions between neurons. Thus, DNN can be utilized as a bit-wise interaction feature learner to assist the vector-wise learner.
Recently, Attention mechanism, which has been widely used in Machine Translation (Sutskever et al. 2014 and Image Caption (Xu et al., 2015) tasks, has been adopted in CTR models (Xiao al. 2017 Song al. 2018). Specially. Transformer (Vaswani al., 2017) encoder, empowered with multi-head self-attention, is demonstrated to be capable of generating context-aware word em- beddings in the work BERT (Devlin et al., 2018). Therefore, it seems promising to use Transformer encoder to solve the 'polysemy' problem in CTR. Thus. propose model DeepEnFM to em- ploy the encoder for context- -ontex--aware field embedding. A related work to our model is AutoInt (Song et al. 2018). However, AutoInt only utilizes the multi-head self-attention of encoder for high-order feature extraction rather than for field embedding.
3 METHODS
Task Formulation. The CTR task is to predict the click behavior from record of multiple fields. Assume we have training record as {x, y), where x represents the fields of instance and y {0. 1} represents the click behavior. The instance fields may contain both categorical fields and numerical fields. We denote y when the click behavior 'click'; otherwise, y = Given unseen test (xx) using all available training
record z², the goal is then to learn a mapping function information and predict the probability user click behavior y", where the y e [0. 1] represents the probability of click behavior.
Algorithm Algorithm of DeepEnFM
1: E Embedding(xx
2: for <- 1, T. do
3: E()) Encoder (E--1)
4: end for
5: YDNN DNN (FNt(Featte
6: yrm (M(E44 4)
7: 0 Prediction(y DNN.N
Overview. Our framework DeepEnFM is illustrated in Fig. aa), which is composed of five compo- nents: the embedding layer, encoder, FM, DNN and prediction layer. The workflow is depicted in Alg. Firstly, the embedding layer projects the input fields into low dimensional space. Then the encoder adaptively updates the embedding results with respect to the clues from other fields through the multi-head self-attention module (MHSA). Next, FM calculates score which integrates the
3

---

Under review as a conference paper at ICLR 2020
explicit interactions between encoder outputs, Simultancously, the original results from the embed- ding layer are fed into DNN to learn implicit feature interactions at wis- level. Finally, we apply the prediction layer to map the intermediate outputs of the FM and DNN into the final score with sigmoid function.
3.1 THE COMPONENTS OF DEEPENFM
Embedding. The embedding layer converts the instance input from a high dimensional space a low dimensional, dense vector. As shown in Fig. [b), we denote the record input fields as x = [x1, x2,...,, .M]], where x; can be either a one-hot vector (categorical field) field). We assume that the field embedding size is d, and the output of embedding layer can be arranged as a matrix E [er; :2i..... ear] For categorical field, the field value is used look up the corresponding embedding vector as the token index in word embedding. For numerical field, the field index is used to look up the related embedding vector, and the field value impacts the embedding as factor.
scalar (numerical
The encoder stack of I. encoder layers with residual connection, which is designed to align the field embedding results accumulatively according to the other field contents, The output of L-th layer can be denoted as E(L), As shown in Fig. c), the encoder layer contains two modules: multi-head self self-attention module and position- wise geed-forward module. At the end of each module, there is a residual connection followed by the layer normalization. We will further explain these two modules Sec. 3.2
Encoder.
FM. FM is leveraged to explicitly capture the first order and second order feature interactions as routine in practice. In our model, FM calculates the score based on the encoder outputs E(L) which are the context-aware field embeddings.
W Σ
Σ
(T) e
УFM
DNN. The DNN aims to capture the implicit high-order interactions between different fields in the bit-wise level. The output of embedding layer flattened to large vector. Then the large vector is fed into multiple-layer feed-forward neural network to extract features.
YDNN DNN(Flatten(E
We do not use the output encoder as the input of DNN, because both the encoder and FM learn at vector-wise level, while DNN aims at bit-wise level. Sharing encoder input may jeopardize the performance of encoder.
Prediction. The prediction layer is full connection layer, using a simple logistic regression func- tion based on the outputs of FM and DNN to predict the final result.
= a(yYM: YONN/W, + bp)
(3)
3.2 THE DETAILS OF ENCODER
Multi-head Self-attention Module (MHSA). The procedure of MHSA (Vaswani al. (017) is
described in Alg. 2 and Fig. P(a). Suppose the number of heads h for each head of MHSA, has the following steps.
(1) The input vector set E is mapping into Query Q, Key K and Value V R á. (2) The similarity matrix is calculated by each pair of query q and key k vectors. (3) The output is calculated as a weighted sum of value vectors v according to S. Finally, the outputs of different attention heads are merged to the final output.
As shown in Alg. 2 the similarity and merge functions are the key parts of MHSA. The design principle of similarity and merge function mainly depends on the generalization, efficiency and architecture requirements. We will discuss the different implements between MHSA in Transformer and DeepEnFM encoder. As for MHSA in Transformer encoder, we firstly introduce the similarity function in Transformer encoder shown in Eq. 4 The similarity function is scaled dot function, which computes the similarity between different feature pairs equally. It merges the output vectors by concatenation and linear transformation with Wo Rdd
4

---

Under review as a conference paper at ICLR 2020
4
k
69
V
0
:
S
9M
MM
KM
VM
(a) One head in multi-head self-attention module
(b) Bilinear similarity
Figure 2: (a) The procedure for one head in MHSA includes vector mapping, similarity computing and output vector generation. (b) The bilinear function used in MHSA of DeepEnFM encoder.
Algorithm Algorithm of multi- head self-attention
1: for h {I,.... H} do
2:
(k))
K(h, V
EW!()
(h)
EW
(k) K EW
3:
for j.j
{1, M)
(h)
4:
q
5:
Similarity (q)
)
6:
aij
E,,, (xp(s
O (h)
7:
8:
end for
9:
end for
10:
O Merge (1)
(9 (b) T (K)
Similarity sdot (q. (h) (h))
(4)
vd
Merge cat (0 (1)
(H)) = concat (o0 0(2) ((,,.,,..,, 0(H) wo
(5)
MHSA in DeepEnFM Encoder. To be clear, we argue that the similarity function between different field pairs could be personalized since the field order is fixed and the field embedding size is very low in CTR. Thus customize the similarity function for each field pair query /²ah) and key k. by
performing bilinear function with specified weight matrix (h(h) (K5)* as shown in Fig. b). Second, we believe that different heads extract features from different viewpoints, so the results contain both complementary and suppressing information to each other. Instead of concatenating all of the head outputs, we keep E mm with the same dimensions of E, and use max-pooling to get the salient output O. Further, we take the output O of MHSA as the residual shown in Fig. (c).
(9/ (")TTW" (
Similarity billnear (q.
(h)) (
(h)
(6.))
(6)
Vdl.
Merge
Mergemaapoool maxpool (1)
(H))
Maxpool (Oa) (2)) O(m)
Position-wise Feed-forward Module. The cosition-wise/fielld wise ct al. 2017) is a mme-hddeen-layer neural network that firstly maps the field embedding to higher dimensional embedding space, then back to the original dimensional space.
module (Vaswani
3.3 THE ARCHITECTURE OF DEEPENFM
The encoder, FM and DNN in our DeepEnFM can be combined in parallel or stacked way, we explore different architectures for ablation study. We consider that the encoder serve
5

---

Under review as a conference paper at ICLR 2020
FM
DNN
FM
DNN
FM
Encoder
DNN
0O00
0OOO
100o 0ooo
0000
Encader
Encoder
0000
0000 COOO
0000 0000
0000
0000
O
0000
Fillx
Fkdi
(a) Encoder for DNN
(b) Encoder for Both
(c) Encoder in parallel
Figure 3: Encoder serves in different positions. (a) Encoder serves the feature extractor for DNN. (b) Encoder serves as the feature extractor for both FM and DNN. (c) Encoder serves as a parallel branch for prediction.
bedding layer for the downstream task, as the BERT in NLP tasks (Devlin et al. 2018). Specially, we stack the encoder and FM together as the left branch as shown in Fig. [[), since both of them learn at vector-wiss level. Three different architectures are described in Fig. [3] which will be further discussed in ablation study.
4 EXPERIMENTS
We conduct experiments
datasets.
(1) Criteo dataset. It contains 45 million click records. Fach record includes 13 dense features and 26 categorical ones. For fair comparison with the other methods, we follow the standard split here: 80%, 10% and 10% of the data as the train, validation and test data respectively. (2) Avazu dataset. contains 40 million click records, 23 fields totally. We also randomly select 80%, 10% and 10% data for train, validation and test respectively. The categorical features with the occurrence more than 10 and 5 times are preserved for Criteo and Avazu respectively. others are set to 'Unknown' as the data processing in AutoInt (Song et al. 2018).
Evaluation. The validation data used to search for hyper- parameter setting. When the hyper- parameters determined, both the the train and validation data are used for training, and the perfor- mance is evaluated on the test data. The experiment result is evaluated on AUC (Area Under ROC) and Logloss (Cross Entropy).
Setting. We implement our approach by TensorFlow. We choose Adam as optimizer and the learn- ing rate is 0.001. We set the L2 regularization À as 0,0001 and dropout rate as 0.5. The batch size, embedding size, and DNN layer size set to 2048, 10 and 400 respectively. We use Xavier initialization for dense layers and random normal initialization for bilinear attention weights. The DeepEnFM model is trained on V100 for 1 hours per training epoch.
Competitors. We compare our model against several state-of saiiee--fthheee-ee -thth-rrt methods: (1) LR (Lee al.
2012). LR simply learns weights of first-order features. (2) FM (Rendle) 2010). FM learns a latent
vector for each feature and models feature interactions by dot products of these vectors. This model learns both first and second-order features. (3) DNN (Zhang et al. 2016). DNN takes embedded dense vectors as its input and learns high-order features in an implicit way. (4) IPNN (Qu al. 2016). IPNN feeds the dense embeddings to inner product layer firstly. Then the output is fed to DNN. (5) OPNN (Qu et al., 2016). OPNN replaces IPNN's inner product layer with an outer product layer. This replacement reduce the complexity of the model. (6)PNN* (Qu et al.) 2016). PNN* use the product layer to concatenate the inner and outer product. (7) DeepFM (Guo et al.) 2017). DeepFM introduces embedding layer to get dense latent vectors for FM. Then these dense (8) AutoInt (Song et al.) 2018). AutoInt feeds the output of its embedding layer to interacting layer. Then MHSA based interacting layer learn high-order features, and a sigmoid output layer or DNN is applied after the interacting layer. (9) DeepEnFM. Our model employs encoder-enhanced FM with DNN.
vectors are also fed to DNN in parallel.

---

Under review as a conference paper at ICLR 2020
Table 1: The results of different models Criteo and Avazu dataset. DNN depth represents the
layers of neural networks. ATT depth represents the layers of MSHA or encoder.
Model
Criteo
Avazu
Depth
AUC (f)
Logloss (L)
AUC (t)
Logloss (4)
DNN ATT
LR
0.7839
0.4655
0.7688
0.3880
FM
0.7860
0.4672
0.7838
0.3779
DNN
0.7942
0.4560
0.7731
0.3841
IPNN
0.8001
0.4662
0.7850
0.3857
2
OPNN
0.6630
0.5210
0.7840
0.3852
2
PNN*
0.7991
0.4699
0.7845
0.3869
DeepFM 0.7928
0.4584
0.7840
0.3777
2
AutoInt 0.8021
0.4511
0.7836
0.3778
2
DeepEnFM
0.8077
0.4444
0.7847
0.3700
2
2
Table 2: The ablation study Criteo dataset. In the first part, we study the contribution of each module enocder. In the second part, we evaluate the major component. In the third part, we evaluate the variant architectures.
Model
AUC
Logloss (L)
DeepEnFM w/o Maxpool
0.8069
0.4448
DeepEnFM w/o Bilinear
0.8025
0.4496
DeepEnFM w/o MHSA
0.8016
0.4511
DeepEnFM w/o FF
0.8057
0.4459
DeepEnFM w/o LN
0.8075
0.4453
DeepEnFM w/o Maxpool & Bilinear
0.8029
0.4496
Encoder Deep
0.8059
0.4456
Encoder FM
0.8037
0.4485
Encoder for Deep
0.7965
0,4622
Encoder for Both
0,8064
0.4461
Encoder in Parallel
0.8075
0.4440
DeepEnFM (Full model)
0.8077
0.4444
4.1 RESULTS
We evaluate the CTR with the standard settings Criteo and Avazu dataset, shown in Tab. We highlight the following observations. (1) Our model achieves the best performance of both AUC and LogLoss metries Criteo. Specifically, Criteo DeepEnFM outperforms the AutoInt by 0.56% DeepFM by 1.49% on AUC respectively. And DeepEnFM also reduces 0.0067 and 0.0140 on the Logloss comparing AutoInt and DeepPM respectively. These strongly demonstrate the efficacy of our bilinear encoder. On Avazu dataset, our model achieves the second place in AUC and the first place in Logloss. While, DeepEnFM has still outperformed AutoInt and DeepFM on AUC and Logloss, which indicates the efficacy of our bilinear attention and encoder. (2) The improvement of DeepFM over FM shows the contribution implicit high-order information DNN part. (3) The gap between FM and LR reflects that the explicit second order interaction information is important for the CTR task. summary, all of the implicit high-order information, explicit second order interaction information, and attentional information are critical for the CTR task.
4.2 ABLATION STUDY
We conduct extensive ablation study of DeepEnFM as follows: (1) the contribution of each module in the encoder of DeepEnFM; (2) the impact of encoder position; (3) the importance of the depth of encoder layers; (4) the influence of number of heads in multi-head self-attention module.
Modules in DeepEnFM encoder. To study the contribution of each modules, propose several variants of our DeepEnFM. (1) 'DeepEnFM w/o Maxpool': the outputs in MHSA are merged by concatenation instead of max max-pooling. (2) 'DeepEnFM w/o Bilinear': the similarity score is calcu- lated by scaled dot product instead bilinear function. (3) 'DeepEnFM w/o. MHSA': the encoder implemented without multi-head self-attention module. (4) 'DeepEnFM w/o PF: the encoder is

---

Under review as a conference paper at ICLR 2020
implemented without point-wise implemented without layer normalization. (6) 'DeepEnFM w/o Maxppol Bilinear': the encoder is implemented in a standard way.
module. (5) 'DeepEnFM w/ LN:: the encoder is
As shown in the first part of Tab. 2 the performance of 'DeepEnFM w/o. MHSA' is the worst, and "DeepEnFM w/o. Bilinear' is the second worst, which shows that the our bilinear attention module critical in DeepEnFM for CTR. Our full model beats all of the variants, which demonstrates the efficacy of each module in encoder.
Components in DeepEnFM. To evaluate the effects of each component, we propose two reduced version of our DeepEnFM. (1) 'Enocoder Deep': the FM component is removed. (2) 'Encoder FM': the DNN is removed.
The results in the second part of Tab. 2 shows that 'Encoder + Deep' is better than 'Encoder + FM'. The gap to the full model demonstrates the effect of each component.
To evaluate the influence of encoder position, we build several variants with the encoder in different position as shown in Fig. (1) 'Encoder for DNN": we feed the encoder output DNN, instead to FM: (2) 'Encoder for Both': we feed the encoder output to both the DNN and FM; (3) 'Encoder in Parallel': the original embedding output is fed into DNN, FM and encoder simultaneously. (4) Our DeepEnFM can be viewed as 'Encoder for FM', which uses the encoder to align the embedding for better explicit feature interaction learning.
Encoder Position.
As shown in the third part Tab. 2 the results show that our DeepEnFM model 'Encoder for FM' achieves the best results, which indicates that the encoder is more compatible with FM, since both encoder and FM extract features at vector-wise level. The 'Encoder Parallel' model achieves the second place, since they can learn in independent way. The 'Encoder for Both' model is less effective than DeepEnFM, which indicates that serving simultaneously for both the vector wise and bit-wise learning modules may harm the power of encoder.
Table 3: The performance of models with different number of layers on Criteo dataset. Setting/Model
DeePEnFM
AutoInt
AUC (t)
Logloss (L)
AUC (t)
Logloss (4)
L=0
0.7928
0.4584
0.7934
0.4568
1=1
0.8049
0.4473
0,8001
0.4510
L=2
0.8077
0.4444
0,8021
0.4511
Depth of encoder layers. We conduct experiments to evaluate the impact of the encoder layers in DeepEnFM, compared with the depth of MHSA layers in AutoInt. Table [3] shows that the perfor- mance of DeepEnFM has been improved as the depth of layers grows, and the gain of the first encoder layer is larger than the second layer. Particularly, we can see the improvement of an encoder layer is larger than a MHSA layer.
Table 4: The performance of models with different number of heads on Criteo dataset. Setting/Model
DeepEnFM
AutoInt
AUC (t)
Logloss (L)
AUC (t)
Logloss (4)
H=1
0.8078
0.4441
0,8010
0.4529
H
0.8077
0.4444
0.8021
0.4511
H=5
0.8054
0.4466
0.8021
0.4991
Number of Heads. We implement DeepEnFM with the head number of 1, 2, 5. Table 4 shows that the model achieves the best results when the number of heads is for DeepEnFM. This is reasonable, the query and key vector dimension reaches the maximum when the number of heads is 1, since the query vector dimension in MHSA is calculated as $. With more query and key dimension, the bilinear function can gain more generalization ability at the cost of more weights and computation. The result achieves the best balance between the performance and cost when the number of heads is 2 for DecpEnFM. The results of AutoInt show that the influence of head number is smaller than DeepEnFM, because it uses scale dot product function, which is less sensitive to the dimension of query and key vectors.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS
In this paper, we propose a novel framework named Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM)), which aims to learn a better aligned vector embedding through the encoder. The encoder combines the bilinear attention and max-pooll max-pooling method to gather both the complementary and suppressing information from the content of other fields. The extensive experiments demonstrate that our approach achieves state-of-art performance on Criteo and Avazu dataset.
REFERENCES
Gediminas Adomavicius and Alexander Tuzhilin. Toward the next generation of recommender systems: A survey the state-of-the-art and possible extensions. ITKDE, 2005. 2
Zeynep Batmaz, Ali Yurekli, Alper Bilge, and Cihan Kalcli. A review on deep learning for recom- mender systems: challenges and remedies. Artificial Intelligence Review, 2019. -
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization
machines. In NIPS, 2016. 2
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for recom- mender systems. In Pruceedings of the Ist workshop on deep learning for recommender systems. ACM, 2016.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
3.3
Huifeng Guo, Ruiming Tang. Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: factorization- machine based neural network for ctr prediction. arXiv preprint arXiv:1703.0424/ 2017.
Alex Krizhevsky, Ila Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012. 2
Kuang-chih Lee, Burkay Orten, Ali Dasdan, and Wentong Li. Estimating conversion rate in display advertising from past erformance data. In KDD. ACM, 2012. D
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In KDD. ACM, 2018.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. In ICDM. IEEE, 2016.
Steffen Rendle. Factorization machines. In ICDM. IEEE, 2010. 2
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. arXiv preprint 18X00 /921, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017. 2 3.2 3.2
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD'17 17. ACM, 2017.
Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional fac- torization machines: Learning the weight of feature interactions via attention networks. arXiv preprint arXiv: /tt.//008.111111 2017.

---

Under review as a conference paper at ICLR 2020
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. 2
Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data. In European conference on information retrieval. Springer. 2016.
10


Reviews:
[{"review_id": "ryxkuCyxFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a model for Click-Through Rate Prediction using a model consisting of an embedding layer, a Transformer stack, a Factorization Machine, and a DNN. \n\nI have several major concerns about the submission:\n2. Relevance: This work is extremely application specific, the application is not relevant to this community.\n1. Clarity and writing: The contributions which are relevant to the ICLR community are not explained well and the paper needs copy-editing for English grammar\n4. Novelty: While seemingly showing good results on some benchmarks, the model is a mix of many components and it's not clear which components actually improve performance and would be worth further study. \n\n\nMinor comments:\n\nApplying the DNN directly on top of the embeddings, and having a parallel stack of Encoder-FM, is not well explained. What does it mean that \"DNN aims at bit-wise level\" if the DNN receives the same embedding features as the encoder, which supposedly \"learn[s] at vector wise level\"? \n\nReferences to datasets are missing\n\nAblation study is limited, and has surprising results. E.g. even completely removing self-attention barely makes a dent in how well the method compares to other published work, moving it from rank 1 to rank 2. Otherwise only small tweaks with even more minor effects are made. What about removing e.g. the FM, other major components?\n\nThe biggest architectural innovations here are the bi-linear attention mechanism and max-pooling self attention. They are hard to interpret in this context. It's not clear how they would perform in a simpler architecture (e.g. vanilla BERT or Transformer) and in the context of a more standard benchmark. That study would have a lot more relevance to this community than the present one. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "SJeEerv-5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper applies Multi-Head Self-Attention (MHSA) to a CTR prediction model with some small changes. The empirical results on two public datasets show it improves performance over some baselines.\n\nFirst of all, the novelty of the proposed algorithm is limited in that it mainly applies existing mulit-head self-attention. The paper does include some small modifications to MHSA and achieves better performance, such as bi-linear similarity and max-pooling. However, the nature of these changes seems more incremental.\n\nThe experiment section is very detailed and the paper conducts several ablation studies to understand which components contribute the most, which is nice. However, the paper is missing several important baselines, for example, Deep & Cross [1], which makes the results less convincing.\n\nAnother issue with the paper is that it does not control the model capacity when comparing performance. It is usually the case that increasing model capacity leads to better performance. Given that MHSA and bi-linear similarity have increased a lot of model parameters, it is more fair to compare performance across models with similar capacity. In fact, in [1], they show the logloss on Criteo dataset can be as low as 0.4423 when using large enough parameters.\n\nMinor: in the ablation study, it shows head = 1 has the best performance. In this case, why max-pooling is needed? \n\nReference:\n[1] Wang, R., Fu, B., Fu, G. and Wang, M., 2017, August. Deep & cross network for ad click predictions. In Proceedings of the ADKDD'17 (p. 12). ACM.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1ltPsiCFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers proposes DeepEnFM approach for CTR prediction task. In detail, Transformer encoder is applied on top of embeddings to generate new projected embeddings. Such transformer encoder is composed of self-attention with bilinear (to replace dot) and multi-head, which is followed by a mx pooling layer and then a FC layer. Position encoding is utilized then. Besides, some resnet-style trick in placed in the middle. Such encoder output is fed into FM and raw embeddings are feed into DNN part. These two parts are then used for final prediction. Some experimental results show the improvement of the proposed method over other methods.\n\nThe major questions are:\n\n*  The assumption of “The field embedding size is very low in CTR” is not reasonable. Do we have any study to verify this hypothesis?\n* Regarding to above hypothesis, i think it doesn’t hold for all the CTR prediction tasks. Computation cost will be dramatically increased when embedding size increases because of bilinear between key and query and the FC on top of self-attention.\n* The novelty of the proposed method needs to justified to reach the bar of ICLR. The major reason is that 1) the proposed method just replaces MHSA with two changes, i.e., bilinear + max pooling, 2) other tricks such as resnet-style connection, layer norm and position encoding have been adopted everywhere.\n* The gain of proposed method is not so clear though the author test to remove each component from the architecture. As the change of encoder part is on top of MHSA, but there is no experiment to show the gain compared to using original MHSA instead of newly proposed bilinear + max pooling. I suggest to do this for better understanding the gain of changes.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the problem of CTR prediction by using a Transformer based encoder to capture interactions between features. They suggest simple modifications to the basic Multiple Head Self Attention (MSHA) mechanism and show that they get the best performance on two publicly available datasets. 

While the reviewers agreed that this work is of practical importance, they had a few objections which I have summarised below: 
1) Lack of novelty: The reviewers felt that the adoption of MSHA for the CTR task was straightforward. The suggested modifications in the form of Bilinear similarity and max-pooling were viewed as incremental contributions. 
2) Lack of comparison with existing work: The reviewers suggested some additional baselines (Deep and Cross) which need to be added (the authors have responded that they will do so later).
3) Need to strengthen experiments: The reviewers appreciated the ablation studies done by the authors but requested for more studies to convincingly demonstrate the effect of some components. One reviewer also pointed that the authors should control form model complexity to ensure an apples-to-apples comparison (I agree that many papers in the past have not done this but going froward I have a hunch that many reviewers will start asking for this) . 

IMO, the above comments are important and the authors should try to address them in subsequent submissions.

Based on the reviewer comments and lack of any response from the authors, I recommend that the paper in it current form cannot be accepted. 
Decision: Reject