Deep Relational Factorization Machines

1 INTRODUCTION . Many supervised learning tasks need to model data with numerous categorical features , which is usually converted into a set of binary features using one-hot encoding . However , when the original categorical features have high cardinalities , such data becomes high-dimensional and sparse . The difficulty in modeling such data is that , most machine learning techniques rely on co-occurrence of features to model their interactions , while in sparse data such co-occurrences are relatively rare compared to the number of possible feature combinations , and hence over-fitting occurs . This is particularly common in web applications , which may involve high-cardinality categorical features such as user IDs , item IDs , and ZipCodes , etc . Factorization Machines ( FMs ) was introduced byRendle ( 2010 ) to model such high-dimensional sparse data . The key idea of FMs is learning a latent vector of each one-hot encoded feature , and capture an arbitrary pairwise ( order-2 ) interaction by inner product of respective latent vectors . The success of FMs has been evidenced by applications such as click-through rate prediction ( Rendle , 2012 ; Ta , 2015 ) and recommendation ( Rendle et al. , 2011 ; Wu et al. , 2017 ) . To further improve the performance of FMs , numerous variants have been proposed ( Blondel et al. , 2016 ; Chen et al. , 2019 ; Guo et al. , 2017 ; He & Chua , 2017 ; Juan et al. , 2016 ; Lian et al. , 2018 ; Qu et al. , 2016 ; Wang et al. , 2017 ; Xiao et al. , 2017 ) . For instance , Field-aware Factorization Machine ( FFM ) ( Juan et al. , 2016 ) is proposed to conduct fine-grained feature interaction . With the development of deep neural networks in recent years , some deep variants have been proposed . For instance , DeepFM ( Guo et al. , 2017 ) combines FM and deep neural networks ( DNN ) to do both the second-order and high-order feature interaction . Deep & Cross Network ( DCN ) ( Wang et al. , 2017 ) stacks multiple interaction layer to learn the high-order feature interaction . Attentional Factorization Machines ( Xiao et al. , 2017 ) employs a neural attention network to additionally weight each interaction term in FMs . Order-aware Embedding Neural Network ( Guo et al. , 2019 ) learns an order-specific latent vector for each binary feature in Higher-order Factorization Machines . However , all the aforementioned FMs variants only focus on the feature interaction . In real-world applications , there also exists sample interaction . For instance , when predicting CTR in a social network , two users in the same basketball group have a large probability to click the same advertisement of basketball shoes since they are supposed to have similar hobbies for basketball . Thus , it is necessary and helpful to incorporate the sample interaction when conducting prediction . Graph Convolutional Networks ( GCN ) and the ilk ( Kipf & Welling , 2016 ) use a graph convolution opera- tion to capture the correlation between nodes in a graph . Specifically , this operation aggregates all neighbors ’ information when making prediction for each node ( sample ) in a graph . As a result , the prediction encodes the interaction between nodes . Recently , GCN has been widely used in numerous applications to capture the interaction in the sample level , such as recommendation ( Wang et al. , 2019 ; Ying et al. , 2018 ) . However , although GCN can incorporate the sample interaction , yet it is difficult for GCN to deal with the sparse categorical features well . Summarily , FMs and GCNs are two general classes of techniques that are typically used for different applications . Both have been shown to be the state-of-the-art in their own respective areas . Meanwhile , they also suffer their intrinsic drawbacks . To overcome the disadvantages of using either approach independently while inheriting the complementary advantages of both approaches , a straightforward solution is to combine these two technologies together . However , how to seamlessly unify these two independent models together to capture both the feature interaction and sample interaction is challenging . To address this issue , we propose a novel Deep Relational Factorization Machine ( DRFM ) . In particular , to model the relationship between different features , we tackle it from the perspective of graph . More specifically , the interaction in different orders between features is reformulated by the path in a feature concurrence graph , with which our method can easily capture the feature interaction in different order by using graph convolutional operation . As far as we know , this is the first work dealing with the feature interaction from the graph view . Moreover , to model the interaction between different samples , we proposed a novel sample interaction component which can capture the high-order sample interaction both linearly and exponentially . Extensive experimental results confirmed the effectiveness of our proposed method . At last , we summarize the contribution of this work as follows : • We disclosed the relationship between the feature interaction and the graph , and proposed a novel graph-based method to deal with the feature interaction . This opens a new avenue to deal with high-dimensional categorical features . • We proposed a general framework that fuses FMs and GCNs into a single unified learning approach , called DRFM . It overcomes the disadvantages of using either approach independently while inheriting the complementary advantages of both approaches . • We demonstrated the effectiveness of DRFM for both link prediction and regression tasks . 2 RELATED WORK . Factorization Machines ( FM ) was first proposed by Rendle ( 2010 ) . With a factorized interaction term , FM is good at dealing with the data with sparse categorical features . However , the standard FM can only capture the second-order feature interaction . To utilize the high-order feature interaction , Blondel et al . ( 2016 ) proposed the high-order factorization machine ( HOFM ) which explicitly incorporates the high-order feature combinations . In addition , to conduct fine-grained feature interaction , Juan et al . ( 2016 ) proposed the Field-aware Factorization Machine ( FwFM ) which assigns multiple latent representation to each feature in terms of feature groups . Similarly , Chen et al . ( 2019 ) also proposed to represent each feature with multiple latent representations according to the frequency of feature occurrences . Recently , deep neural networks ( DNN ) have shown promising performance on a wide variety of tasks , such as computer vision , natural language processing . Inspired by this , some researchers proposed to combine DNN with FM to fully utilize their advantages . For instance , Factorizationmachine supported Neural Networks ( FNN ) first pre-trains the factorization machine to get the latent representation of features and then feeds these representations to DNN to learn high-order feature interaction implicitly ( Zhang et al. , 2016 ) . To train FM and DNN in an end-to-end way , Productbased Neural Network ( PNN ) proposed by Qu et al . ( 2016 ) introduced a product layer to connect the feature embedding layer and DNN layers . However , both of these two models only focus on the high-order feature interaction , ignoring the low-order interaction . To address this issue , Guo et al . ( 2017 ) proposed DeepFM which models FM and DNN in two branches and trains them simultaneously . Wang et al . ( 2017 ) proposed Deep & Cross Network ( DCN ) to explicitly capture the feature interaction with different orders . Similarly , xDeepFM ( Lian et al. , 2018 ) also aims at capturing feature interactions with different orders , but it uses the inner product rather than outer product like DCN . As we discussed earlier , FM and its variants aim at capturing the feature interaction . In some realworld applications , it is necessary to capture the sample interaction . To do that , Kipf & Welling ( 2016 ) proposed graph convolutional networks ( GCN ) . Specifically , GCN employs the graph convolutional operation to capture the interaction between samples and their neighbors . Recently , GCN has been applied to various tasks to capture the sample interaction . For instance , Ying et al . ( 2018 ) proposed PinSage to explore the item-item interaction in the recommender system . Wang et al . ( 2019 ) proposed the Neural Graph Collaborative Filtering ( NGCF ) to utilize the user-item interaction for recommendation . 3 PRELIMINARIES . In this section , we are going to present some preliminaries about factorization machine and graph convolutional neural networks . Throughout this paper , a graph is represented by G = ( V , E ) where V = { vi } represents the node set and E = { ( vi , vj ) } represents the edge set . In this paper , we focus on the attributed graph and the node feature matrix is represented by X = [ x1 , x2 , · · · , xn ] ∈ Rd×n . In this work , we focus on the high-dimensional categorical features , which are very common in real-world applications , such as recommendation and CTR prediction . Specifically , we assume the feature of each node is represented as xi = [ 0 , 1 , 0 , · · · , 1 , 1 , 0 , 0 ] T ∈ Rd whose features is categorical and the number of non-zero values is much less than d. In addition , if a node has the ground-truth ( e.g . the regression task in our experiment ) , it is denoted by Y = [ yi ] ∈ Rn . Note that we will use samples and nodes interchangeably throughout this paper . Based on aforementioned terminologies , Factorization Machine ( FM ) is defined as follows : ŷi = b+w Txi + ∑ p < q 〈vp , vq〉xi , pxi , q ( 1 ) where ŷi ∈ R denotes the prediction of node vi , xi = [ xi , p ] ∈ Rd represents node features where xi , p is the p-th feature of node vi , vp ∈ Rk stands for the embedding of the p-th feature . Compared the regular linear model , FM can capture the interaction between different features . Specifically , in the non-linear term , the dot product 〈vp , vq〉 computes the interaction between feature xi , p and xi , q . However , FM can only capture the interaction inside each node , ignoring interaction between different nodes . Convolution is an effective operator to capture the local correlation . The regular convolutional operator is used to extract features by exploring the feature correlation . On the contrary , the graph convolutional operator is proposed to explore the sample correlation . Specifically , the graph convolutional operation in the l-th hidden layer of the Graph Convolutional Neural Network ( GCN ) is defined as follows : zl+1i = 1√ |N ( i ) | ∑ i′∈N ( i ) 1√ |N ( i′ ) | Wl+1hli′ hl+1i = f ( z l+1 i ) ( 2 ) where hli ∈ Rdl denotes the hidden representation of node vi in the l-th layer , Wl+1 ∈ Rdl×dl+1 represents the model parameter , N ( i ) indicates the neighbors of node vi , and f ( · ) stands for the non-linear activation function . It can be found that the representation zl+1i of the i-th sample is constructed by aggregating its neighbors . In this way , GCN can capture the sample interaction . However , although GCN can explore the interaction between different samples , yet it is not good at exploring the feature interaction . In the following section , we will propose a new model to address these two issues of FM and GCN .

Summary: In this paper, the authors propose generalize the FM to consider both interaction between features and interaction between samples. For the interaction between features, the authors propose to use graph convolution to capture high-order feature interactions. Moreover, the authors construct a graph on the instances based on similarity. Then a GCN is applied to the sample graph where the feature embedding is shared between the two components. Experiments are carried out on four datasets with tasks of link prediction and regression. Comparison to several baselines demonstrate the superior performance of the proposed method.