DeepEnFM: Deep neural networks with Encoder enhanced Factorization Machine

1 INTRODUCTION . This paper studies the problem of predicting the Click Through Rate ( CTR ) , which is an essential task in industrial applications , such as online advertising , and e-commerce . To be exact , the advertisements of cost-per-click ( CPC ) advertising system are normally ranked by the eCPM ( effective cost per mille ) , which is computed as the prodcut of bid price and CTR ( click-through rate ) . To predict CTR precisely , feature representation is an important step in extracting the good , interpretable patterns from training data . For example , the co-occurrence of “ Valentine ’ s Day ” , “ chocolate ” and “ male ” can be viewed as one meaningful indicator/feature for the recommendation . Such handcrafted feature type is predominant in CTR prediction ( Lee et al. , 2012 ) , until the renaissance of Deep Neural Networks ( DNNs ) . Recently , a more effective manner , i.e. , representation learning has been investigated in CTR prediction with some works ( Guo et al. , 2017 ; Qu et al. , 2016 ; Wang et al. , 2017 ; Lian et al. , 2018 ; Song et al. , 2018 ) , which implicitly or explicitly learn the embeddings of high-order feature extractions among neurons or input elements by the expressive power of DNNs or FM . Despite their noticeable performance improvement , DNNs and explicit high order feature-based methods ( Wang et al. , 2017 ; Zhang et al. , 2016 ; Guo et al. , 2017 ; Lian et al. , 2018 ) seek better feature interactions merely based on the naive feature embeddings . Few efforts have been made in addressing the task of holistically understanding and learning representations of inputs . This leads to many practical problems , such as “ polysemy ” in the learned feature embeddings existed in previous works . For example , the input feature ‘ chocolate ’ is much closer to the ‘ snack ’ than ‘ gift ’ in normal cases , while we believe ‘ chocolate ’ should be better paired with ‘ gift ’ if given the occurrence input as “ Valentine ’ s Day ” . This is one common polysemy problem in CTR prediction . Towards fully understanding the inputs , we re-introduce to CTR , the idea of Transformer encoder ( Vaswani et al. , 2017 ) , which is oriented in Natural Language Processing ( NLP ) . Such an encoder can efficiently accumulate and extract patterns from contextual word embeddings in NLP , and thus potentially would be very useful in holistically representation learning in CTR . Critically , the Transformer encoder has seldom been applied to CTR prediction with the only one exception arxiv paper AutoInt ( Song et al. , 2018 ) , which , however , simply implements the multi-head selfattention ( MHSA ) mechanism of encoders , to directly extract high-order feature interactions . We argue that the output of MHSA/encoder should be still considered as first-order embedding influenced by the other fields , rather than a high-order interaction feature . To this end , our main idea is to apply the encoder to learn a context-aware feature embedding , which contains the clues from the content of other features . Thus the “ polysemy ” problem can be solved naturally , and the second-order interaction of such features can represent more meaning . Contrast to AutoInt ( Song et al. , 2018 ) , which feeds the output of encoder directly to the prediction layer or a DNN , our work not only improves the encoder to be more suitable for CTR task , but also feeds the encoder output to FM , since both our encoder and FM are based on vector-wise learning mechanism . And we adopt DNN to learn the bit-wise high-order feature interactions in a parallel way , which avoids interweaving the vector-wise and bit-wise interactions in a stacked way . Formally , we propose a novel framework – Deep neural networks with Encoder enhanced Factorization Machine ( DeepEnFM ) . DeepEnFM focuses on generating better contextual aligned vectors for FM and uses DNN as a bit-wise information supplement . The architecture adopting both Deep and FM part is inspired by DeepFM ( Guo et al. , 2017 ) . The encoder is endowed with bilinear attention and max-pooling power . First , we observed that unlike the random order of words in a sentence , the features in a transaction are in a fixed order of fields . For example , the fields of features are arranged in an order of { Gender , Age , Price ... } . When the features are embedded in dense vectors , the first and second vectors in a transaction always represent the field “ Gender ” and “ Age ” . To make use of this advantage , we add a bilinear mechanism to the Transformer encoder . We use bilinear functions to replace the simple dot product in attention . In this way , feature similarity of different field pairs is modeled with different functions . The embedding size in CTR tasks is usually around 10 , which allows the application of bilinear functions without unbearable computing complexity . Second , the original multi-head outputs are merged by concatenation , which considers the outputs are complementary to each other . We argue that there are also suppressing information between different heads . We apply a max-pooling merge mechanism to extract both complementary and suppressing information from the multi-head outputs . Experimental results on Criteo and Avazu datasets have demonstrated the efficacy of our proposed model . To summarize , our contributions are three-fold . ( 1 ) We propose a novel model DeepEnFM , which combines both DNN and encoder enhanced FM . ( 2 ) The state-of-the-art results on Criteo and Avazu datasets have shown the efficacy of our proposed model . ( 3 ) Extensive ablation study has shown the contribution of each component , and revealed the relations between DNN , encoder , and FM by developing different architectures . 2 RELATED WORK . CTR prediction is a critical task in Recommendation System ( Adomavicius and Tuzhilin , 2005 ; Batmaz et al. , 2019 ) , which aims to predict the probability of a user click behavior based on the given item . The key challenge of the CTR is to automatically learn combinatory/cross features ( e.g. , a 2-way feature : “ Valentine ’ s Day ” and “ chocolate ” , a 3-way feature : “ Valentine ’ s Day ” , “ chocolate ” and “ male ” ) from the inputs . Traditional learning methods , such as Logistic Regression ( LR ) ( Lee et al. , 2012 ) , learn weights of processed features in supervised way for prediction . To gain a better performance , the manually designed combinatory features are fed into LR model , which is laborious and unable to cover the numerous combination of features . To learn the combinatory features automatically in a explicit way , Rendle proposes Factorization Machine ( Rendle , 2010 ) model , which treats the weight of a feature pair as the dot of two latent vectors . Moreover , HOFM ( Blondel et al. , 2016 ) and CIN ( Lian et al. , 2018 ) are proposed to capture the interaction of arbitrary order in a explicit manner . However , the primary/first-order field embeddings are utilized without considering the contextual information , which leads to the ‘ polysemy ’ problem in field embedding unsolved . Deep Neural Networks ( DNN ) ( Krizhevsky et al. , 2012 ) have made remarkable progresses in CV and NLP tasks by its extraordinary representation learning power . Several DNN based CTR models are developed on the basis of primary field embeddings ( Zhang et al. , 2016 ; Cheng et al. , 2016 ; Guo et al. , 2017 ; Lian et al. , 2018 ; Wang et al. , 2017 ) or product layers ( Qu et al. , 2016 ) with a plain DNN . One of the major drawbacks of the plain DNN is that it tackles the field embeddings as a large flat layer , which means there are only interactions between neurons . Thus , DNN can be utilized as a bit-wise interaction feature learner to assist the vector-wise learner . Recently , Attention mechanism , which has been widely used in Machine Translation ( Sutskever et al. , 2014 ) and Image Caption ( Xu et al. , 2015 ) tasks , has been adopted in CTR models ( Xiao et al. , 2017 ; Song et al. , 2018 ) . Specially , Transformer ( Vaswani et al. , 2017 ) encoder , empowered with multi-head self-attention , is demonstrated to be capable of generating context-aware word embeddings in the work BERT ( Devlin et al. , 2018 ) . Therefore , it seems promising to use Transformer encoder to solve the ‘ polysemy ’ problem in CTR . Thus , we propose our model DeepEnFM to employ the encoder for context-aware field embedding . A related work to our model is AutoInt ( Song et al. , 2018 ) . However , AutoInt only utilizes the multi-head self-attention of encoder for high-order feature extraction rather than for field embedding . 3 METHODS . Task Formulation . The CTR task is to predict the click behavior from a record of multiple fields . Assume we have a training record as { x , y } , where x represents the fields of instance and y ∈ { 0 , 1 } represents the click behavior . The instance fields xmay contain both categorical fields and numerical fields . We denote y = 1 when the click behavior is ‘ click ’ ; otherwise , y = 0 . Given a unseen test record x∗ , the goal is then to learn a mapping function y∗ = Ψ ( x∗ ) using all available training information and predict the probability of user click behavior y∗ , where the y∗ ∈ [ 0 , 1 ] represents the probability of click behavior . Algorithm 1 Algorithm of DeepEnFM 1 : E ( 0 ) = Embedding ( x ) 2 : for l← 1 , L do 3 : E ( l ) = Encoder ( E ( l−1 ) ) 4 : end for 5 : yDNN = DNN ( Flatten ( E ( 0 ) ) ) 6 : yFM = FM ( E ( L ) ) 7 : ŷ = Prediction ( yDNN , yFM ) Overview . Our framework DeepEnFM is illustrated in Fig . 1 ( a ) , which is composed of five components : the embedding layer , encoder , FM , DNN and prediction layer . The workflow is depicted in Alg . 1 . Firstly , the embedding layer projects the input fields into a low dimensional space . Then the encoder adaptively updates the embedding results with respect to the clues from other fields through the multi-head self-attention module ( MHSA ) . Next , FM calculates a score which integrates the explicit interactions between encoder outputs . Simultaneously , the original results from the embedding layer are fed into DNN to learn implicit feature interactions at bit-wise level . Finally , we apply the prediction layer to map the intermediate outputs of the FM and DNN into the final score with sigmoid function . 3.1 THE COMPONENTS OF DEEPENFM . Embedding . The embedding layer converts the instance input x from a high dimensional space to a low dimensional , dense vector . As shown in Fig . 1 ( b ) , we denote the record input fields as x = [ x1 , x2 , . . . , xM ] , where xi can be either a one-hot vector ( categorical field ) or a scalar ( numerical field ) . We assume that the field embedding size is d , and the output of embedding layer can be arranged as a matrix E = [ e1 ; e2 ; . . . ; eM ] . For a categorical field , the field value is used to look up the corresponding embedding vector as the token index in word embedding . For a numerical field , the field index is used to look up the related embedding vector , and the field value impacts the embedding as a factor . Encoder . The encoder is a stack of L encoder layers with residual connection , which is designed to align the field embedding results accumulatively according to the other field contents . The output of L-th layer can be denoted as E ( L ) . As shown in Fig . 1 ( c ) , the encoder layer contains two modules : a multi-head self-attention module and a position-wise feed-forward module . At the end of each module , there is a residual connection followed by the layer normalization . We will further explain these two modules in Sec . 3.2 . FM . FM is leveraged to explicitly capture the first order and second order feature interactions as a routine in practice . In our model , FM calculates the score based on the encoder outputs E ( L ) , which are the context-aware field embeddings . yFM = M∑ i=1 wixi + M∑ i=1 M∑ j=i+1 e ( L ) i · e ( L ) j ( 1 ) DNN . The DNN aims to capture the implicit high-order interactions between different fields in the bit-wise level . The output of embedding layer is flattened to a large vector . Then the large vector is fed into a multiple-layer feed-forward neural network to extract features . yDNN = DNN ( Flatten ( E ( 0 ) ) ) ( 2 ) We do not use the output of encoder as the input of DNN , because both the encoder and FM learn at vector-wise level , while DNN aims at bit-wise level . Sharing encoder input may jeopardize the performance of encoder . Prediction . The prediction layer is a full connection layer , using a simple logistic regression function based on the outputs of FM and DNN to predict the final result . y = σ ( [ yFM , yDNN ] wp + bp ) ( 3 )

Summary: This papers proposes DeepEnFM approach for CTR prediction task. In detail, Transformer encoder is applied on top of embeddings to generate new projected embeddings. Such transformer encoder is composed of self-attention with bilinear (to replace dot) and multi-head, which is followed by a mx pooling layer and then a FC layer. Position encoding is utilized then. Besides, some resnet-style trick in placed in the middle. Such encoder output is fed into FM and raw embeddings are feed into DNN part. These two parts are then used for final prediction. Some experimental results show the improvement of the proposed method over other methods.