Deep Relational Factorization Machines

1 INTRODUCTION . Many supervised learning tasks need to model data with numerous categorical features , which is usually converted into a set of binary features using one-hot encoding . However , when the original categorical features have high cardinalities , such data becomes high-dimensional and sparse . The difficulty in modeling such data is that , most machine learning techniques rely on co-occurrence of features to model their interactions , while in sparse data such co-occurrences are relatively rare compared to the number of possible feature combinations , and hence over-fitting occurs . This is particularly common in web applications , which may involve high-cardinality categorical features such as user IDs , item IDs , and ZipCodes , etc . Factorization Machines ( FMs ) was introduced byRendle ( 2010 ) to model such high-dimensional sparse data . The key idea of FMs is learning a latent vector of each one-hot encoded feature , and capture an arbitrary pairwise ( order-2 ) interaction by inner product of respective latent vectors . The success of FMs has been evidenced by applications such as click-through rate prediction ( Rendle , 2012 ; Ta , 2015 ) and recommendation ( Rendle et al. , 2011 ; Wu et al. , 2017 ) . To further improve the performance of FMs , numerous variants have been proposed ( Blondel et al. , 2016 ; Chen et al. , 2019 ; Guo et al. , 2017 ; He & Chua , 2017 ; Juan et al. , 2016 ; Lian et al. , 2018 ; Qu et al. , 2016 ; Wang et al. , 2017 ; Xiao et al. , 2017 ) . For instance , Field-aware Factorization Machine ( FFM ) ( Juan et al. , 2016 ) is proposed to conduct fine-grained feature interaction . With the development of deep neural networks in recent years , some deep variants have been proposed . For instance , DeepFM ( Guo et al. , 2017 ) combines FM and deep neural networks ( DNN ) to do both the second-order and high-order feature interaction . Deep & Cross Network ( DCN ) ( Wang et al. , 2017 ) stacks multiple interaction layer to learn the high-order feature interaction . Attentional Factorization Machines ( Xiao et al. , 2017 ) employs a neural attention network to additionally weight each interaction term in FMs . Order-aware Embedding Neural Network ( Guo et al. , 2019 ) learns an order-specific latent vector for each binary feature in Higher-order Factorization Machines . However , all the aforementioned FMs variants only focus on the feature interaction . In real-world applications , there also exists sample interaction . For instance , when predicting CTR in a social network , two users in the same basketball group have a large probability to click the same advertisement of basketball shoes since they are supposed to have similar hobbies for basketball . Thus , it is necessary and helpful to incorporate the sample interaction when conducting prediction . Graph Convolutional Networks ( GCN ) and the ilk ( Kipf & Welling , 2016 ) use a graph convolution opera- tion to capture the correlation between nodes in a graph . Specifically , this operation aggregates all neighbors ’ information when making prediction for each node ( sample ) in a graph . As a result , the prediction encodes the interaction between nodes . Recently , GCN has been widely used in numerous applications to capture the interaction in the sample level , such as recommendation ( Wang et al. , 2019 ; Ying et al. , 2018 ) . However , although GCN can incorporate the sample interaction , yet it is difficult for GCN to deal with the sparse categorical features well . Summarily , FMs and GCNs are two general classes of techniques that are typically used for different applications . Both have been shown to be the state-of-the-art in their own respective areas . Meanwhile , they also suffer their intrinsic drawbacks . To overcome the disadvantages of using either approach independently while inheriting the complementary advantages of both approaches , a straightforward solution is to combine these two technologies together . However , how to seamlessly unify these two independent models together to capture both the feature interaction and sample interaction is challenging . To address this issue , we propose a novel Deep Relational Factorization Machine ( DRFM ) . In particular , to model the relationship between different features , we tackle it from the perspective of graph . More specifically , the interaction in different orders between features is reformulated by the path in a feature concurrence graph , with which our method can easily capture the feature interaction in different order by using graph convolutional operation . As far as we know , this is the first work dealing with the feature interaction from the graph view . Moreover , to model the interaction between different samples , we proposed a novel sample interaction component which can capture the high-order sample interaction both linearly and exponentially . Extensive experimental results confirmed the effectiveness of our proposed method . At last , we summarize the contribution of this work as follows : • We disclosed the relationship between the feature interaction and the graph , and proposed a novel graph-based method to deal with the feature interaction . This opens a new avenue to deal with high-dimensional categorical features . • We proposed a general framework that fuses FMs and GCNs into a single unified learning approach , called DRFM . It overcomes the disadvantages of using either approach independently while inheriting the complementary advantages of both approaches . • We demonstrated the effectiveness of DRFM for both link prediction and regression tasks . 2 RELATED WORK . Factorization Machines ( FM ) was first proposed by Rendle ( 2010 ) . With a factorized interaction term , FM is good at dealing with the data with sparse categorical features . However , the standard FM can only capture the second-order feature interaction . To utilize the high-order feature interaction , Blondel et al . ( 2016 ) proposed the high-order factorization machine ( HOFM ) which explicitly incorporates the high-order feature combinations . In addition , to conduct fine-grained feature interaction , Juan et al . ( 2016 ) proposed the Field-aware Factorization Machine ( FwFM ) which assigns multiple latent representation to each feature in terms of feature groups . Similarly , Chen et al . ( 2019 ) also proposed to represent each feature with multiple latent representations according to the frequency of feature occurrences . Recently , deep neural networks ( DNN ) have shown promising performance on a wide variety of tasks , such as computer vision , natural language processing . Inspired by this , some researchers proposed to combine DNN with FM to fully utilize their advantages . For instance , Factorizationmachine supported Neural Networks ( FNN ) first pre-trains the factorization machine to get the latent representation of features and then feeds these representations to DNN to learn high-order feature interaction implicitly ( Zhang et al. , 2016 ) . To train FM and DNN in an end-to-end way , Productbased Neural Network ( PNN ) proposed by Qu et al . ( 2016 ) introduced a product layer to connect the feature embedding layer and DNN layers . However , both of these two models only focus on the high-order feature interaction , ignoring the low-order interaction . To address this issue , Guo et al . ( 2017 ) proposed DeepFM which models FM and DNN in two branches and trains them simultaneously . Wang et al . ( 2017 ) proposed Deep & Cross Network ( DCN ) to explicitly capture the feature interaction with different orders . Similarly , xDeepFM ( Lian et al. , 2018 ) also aims at capturing feature interactions with different orders , but it uses the inner product rather than outer product like DCN . As we discussed earlier , FM and its variants aim at capturing the feature interaction . In some realworld applications , it is necessary to capture the sample interaction . To do that , Kipf & Welling ( 2016 ) proposed graph convolutional networks ( GCN ) . Specifically , GCN employs the graph convolutional operation to capture the interaction between samples and their neighbors . Recently , GCN has been applied to various tasks to capture the sample interaction . For instance , Ying et al . ( 2018 ) proposed PinSage to explore the item-item interaction in the recommender system . Wang et al . ( 2019 ) proposed the Neural Graph Collaborative Filtering ( NGCF ) to utilize the user-item interaction for recommendation . 3 PRELIMINARIES . In this section , we are going to present some preliminaries about factorization machine and graph convolutional neural networks . Throughout this paper , a graph is represented by G = ( V , E ) where V = { vi } represents the node set and E = { ( vi , vj ) } represents the edge set . In this paper , we focus on the attributed graph and the node feature matrix is represented by X = [ x1 , x2 , · · · , xn ] ∈ Rd×n . In this work , we focus on the high-dimensional categorical features , which are very common in real-world applications , such as recommendation and CTR prediction . Specifically , we assume the feature of each node is represented as xi = [ 0 , 1 , 0 , · · · , 1 , 1 , 0 , 0 ] T ∈ Rd whose features is categorical and the number of non-zero values is much less than d. In addition , if a node has the ground-truth ( e.g . the regression task in our experiment ) , it is denoted by Y = [ yi ] ∈ Rn . Note that we will use samples and nodes interchangeably throughout this paper . Based on aforementioned terminologies , Factorization Machine ( FM ) is defined as follows : ŷi = b+w Txi + ∑ p < q 〈vp , vq〉xi , pxi , q ( 1 ) where ŷi ∈ R denotes the prediction of node vi , xi = [ xi , p ] ∈ Rd represents node features where xi , p is the p-th feature of node vi , vp ∈ Rk stands for the embedding of the p-th feature . Compared the regular linear model , FM can capture the interaction between different features . Specifically , in the non-linear term , the dot product 〈vp , vq〉 computes the interaction between feature xi , p and xi , q . However , FM can only capture the interaction inside each node , ignoring interaction between different nodes . Convolution is an effective operator to capture the local correlation . The regular convolutional operator is used to extract features by exploring the feature correlation . On the contrary , the graph convolutional operator is proposed to explore the sample correlation . Specifically , the graph convolutional operation in the l-th hidden layer of the Graph Convolutional Neural Network ( GCN ) is defined as follows : zl+1i = 1√ |N ( i ) | ∑ i′∈N ( i ) 1√ |N ( i′ ) | Wl+1hli′ hl+1i = f ( z l+1 i ) ( 2 ) where hli ∈ Rdl denotes the hidden representation of node vi in the l-th layer , Wl+1 ∈ Rdl×dl+1 represents the model parameter , N ( i ) indicates the neighbors of node vi , and f ( · ) stands for the non-linear activation function . It can be found that the representation zl+1i of the i-th sample is constructed by aggregating its neighbors . In this way , GCN can capture the sample interaction . However , although GCN can explore the interaction between different samples , yet it is not good at exploring the feature interaction . In the following section , we will propose a new model to address these two issues of FM and GCN .

Summary: This paper proposes to combine the graph neural networks and factorization machines. First, the authors propose a relational feature interaction component (RFO) tp deal with the categorical features. This component first uses the factorization machine to project the features to h^FI(x), then it uses an aggregation operation to get the prediction y^RFI. To explore high-order correlations, the authors further propose to calculate a concurrence graph, on which RFI propagates the embedding vectors to get relational high-order correlations. To further model high-order sample interactions, this work then presents a special graph convolutional operation that considers the element-wise products of the encoded features.