Deep Relational Factorization Machines
Authors: Hongchang Gao
Gang Wu
Ryan Rossi
Viswanathan Swaminathan
Heng Huang
Venue:  (2020)

Abstract: Factorization Machines (FMs) is an important supervised learning approach due to its unique ability to capture feature interactions when dealing with high-dimensional sparse data. However, FMs assume each sample is independently observed and hence incapable of exploiting the interactions among samples. On the contrary, Graph Neural Networks (GNNs) has become increasingly popular due to its strength at capturing the dependencies among samples. But unfortunately, it cannot efficiently handle high-dimensional sparse data, which is quite common in modern machine learning tasks.  In this work, to leverage their complementary advantages and yet overcome their issues,  we proposed a novel approach, namely Deep Relational Factorization Machines,  which can capture both the feature interaction and the sample interaction. In particular, we disclosed the relationship between the feature interaction and the graph, which opens a brand new avenue to deal with high-dimensional features. Finally, we demonstrate the effectiveness of the proposed approach with experiments on several real-world datasets.

Under review as a conference paper at ICLR 2020
DEEP RELATIONAL FACTORIZATION MACHINES
Anonymous authors
Paper under double-blind review
ABSTRACT
Factorization Machines (FMs) is an important supervised learning approach due to its unique ability to capture feature interactions when dealing with high- dimensional sparse data. However, FMS assume each sample is independently ob- served and hence incapable of exploiting the interactions among samples. On the contrary, Graph Neural Networks (GNNs) has become increasingly popular due to its strength at capturing the dependencies among samples. But unfortunately, it cannot efficiently handle high-dimensional sparse data, which is quite common in modern machine learning tasks. In this work, to leverage their complementary advantages and yet overcome their issues, we proposed novel approach, namely Deep Relational Factorization Machines, which can capture both the feature in- teraction and the sample interaction. In particular, we disclosed the relationship between the feature interaction and the graph, which opens a brand new avenue deal with high-dimensional features. Finally, we demonstrate the effectiveness the proposed approach with experiments several real-world datasets.
INTRODUCTION
Many supervised learning tasks need to model data with numerous categorical features, which is usually converted into set of binary features using one-hot encoding. However, when the original categorical features have high cardinalities, such data becomes high-dimensional and sparse. The difficulty in modeling such data is that, most machine learning techniques rely on co-occurrence of features to model their interactions, while in sparse data such co-occurrences are relatively rare compared to the number of possible feature combinations, and hence over-fitting occurs. This par- ticularly common in web applications, which may involve high-cardinality categorical features such as user IDs, item IDs, and ZipCodes, etc. Factorization Machines (FMs) was introduced byRendle (2010) to model such high-dimensional sparse data. The key idea of FMs is learning a latent vector of each one-hot encoded feature, and capture an arbitrary pairwise (order-2) interaction by inner product of respective latent vectors. The success of FMs has been evidenced by applications such as thick-through rate prediction (Rendle 2012
Ta 2015) and recommendation (Rendle et al. 2011
Wu et al. 2017).
To further improve the performance of FMs, numerous variants have been proposed (Blondel et al.
2016 Chen et al., 2019 Guo et al. 2017, He & Chua 2017: Juan et al. 2016) Lian et al. 2018
Qu et al.] 2016 Wang et al., 2017
Xiao et al.) (017). For instance, Field-aware Factorization is proposed to conduct fine-grained feature interaction. With the development of deep neural networks in recent years, some deep variants have been proposed.
Machine (FFM) (Juan et 2016)
For instance, DeepFM (Guo al. 2017) combines FM and deep neural networks (DNN) do both the second-order and high-order feature interaction. Deep & Cross Network (DCN) (Wang al. 2017) stacks multiple interaction layer to learn the high-order feature interaction. Attentional Factorization Machines (Xiao et al., (017) employs neural attention network to additionally weight each interaction term FMs. Order-aware Embedding Neural Network (Guo et al.) 2019) learns an order-specific latent vector for each binary feature in Higher-order Factorization Machines.
However, all the aforementioned FMs variants only focus on the feature interaction. In real-world applications, there also exists sample interaction. For instance, when predicting CTR in a social network, two users in the same basketball group have a large probability to click the same adver- tisement of basketball shoes since they are supposed to have similar hobbies for basketball. Thus, it is necessary and helpful to incorporate the sample interaction when conducting prediction. Graph Convolutional Networks (GCN) and the ilk (Kipf & Welling 2016) graph convolution opera-

---

Under review as a conference paper at ICLR 2020
tion to capture the correlation between nodes in a graph. Specifically, this operation aggregates all neighbors' information when making prediction for each node (sample) in a graph. As a result, the prediction encodes the interaction between nodes. Recently, GCN has been widely used in numer- ous applications to capture the interaction in the sample level, such as recommendation (Wang et al.) 2019 Ying et al., 2018). However, although GCN can incorporate the sample interaction, yet it is difficult for GCN to deal with the sparse categorical features well.
Summarily, FMs and GCNs are two general classes of techniques that are typically used for dif- ferent applications. Both have been shown to be the state-of-the-art in their own respective areas. Meanwhile, they also suffer their intrinsic drawbacks. To overcome the disadvantages of using ei- ther approach independently while inheriting the complementary advantages of both approaches, straightforward solution is to combine these two technologies together. However, how seamlessly unify these two independent models together to capture both the feature interaction and sample in- teraction is challenging. To address this issue, we propose novel Deep Relational Factorization Machine (DRFM). In particular, to model the relationship between different features, we tackle it from the perspective of graph. More specifically, the interaction in different orders between features is reformulated by the path in feature concurrence graph, with which our method can easily cap- ture the feature interaction in different order by using graph convolutional operation. As far as we know, this is the first work dealing with the feature interaction from the graph view. Moreover, to model the interaction between different samples, we proposed a novel sample interaction compo- nent which can capture the high-order sample interaction both linearly and exponentially. Extensive experimental results confirmed the effectiveness of our proposed method. At last, we summarize the contribution of this work follows:
We disclosed the relationship between the feature interaction and the graph, and proposed a novel graph- based method to deal with the feature interaction. This opens new avenue to deal with high-dimensional categorical features.
We proposed general framework that fuses FMs and GCNs into a single unified learning approach, called DRFM. overcomes the disadvantages of using either approach indepen- dently while inheriting the complementary advantages of both approaches.
We demonstrated the effectiveness of DRFM for both link prediction and regression tasks.
2 RELATED WORK
Factorization Machines (FM) was first proposed by Rendle (010)) With a factorized interaction term, FM is good at dealing with the data with sparse categorical features. However, the standard FM can only capture the second-order feature interaction. To utilize the high-order feature interac- tion, (Blodell et al. (2016) proposed the high-order factorization machine (HOFM) which explicitly addition, to conduct fine-grained feature inter-
incorporates the high-order feature combinations. action, Juan et al. (2016) proposed the Field-aware Factorization Machine (FwFM) which assigns multiple latent representation to each feature in terms of feature groups. Similarly, Chen et al. (2019) also proposed to represent each feature with multiple latent representations according the frequency of feature occurrences.
Recently, deep neural networks (DNN) have shown promising performance on wide variety of tasks, such as computer vision, natural language processing. Inspired by this, some researchers proposed combine DNN with FM fully utilize their advantages. For instance, Factorization- machine supported Neural Networks (FNN) first pre-trains the factorization machine to get the latent representation of features and then feeds these representations to DNN to learn high-order feature interaction implicitly (Zhang et al.) 2016). To train FM and DNN in an end-to-end way, Product- based Neural Network (PNN) proposed by Qu al. (2016) introduced a product layer to connect the feature embedding layer and DNN layers. However, both of these two models only focus on the high-order feature interaction, ignoring the low-order interaction. To address this issue, Guo et al. 2017) proposed MeepI'M which models FM and DNN in two branches and trains them simultane- ously. Waag et al. (2017) proposed Deep & Cross Network (DCN) to explicitly capture the feature interaction with different orders Similarly, xDeepFM (Lian al. 2018) ) also aims at capturing fea- ture interactions with different orders, but it uses the inner product rather than outer product like DCN.
2

---

Under review as a conference paper at ICLR 2020
As we discussed earlier, FM and its variants aim capturing the feature interaction. In some real- world applications, is necessary to capture the sample interaction. To do that, Kipf & Welling (2016) proposed graph convolutional networks (GCN). Specifically, GCN employs the graph con- volutional operation to capture the interaction between samples and their neighbors. Recently, GCN has been applied to various tasks to capture the sample interaction. For instance, Ying et al. (2018) proposed PinSage to explore the item-item interaction in the recommender system. Waang et al. (219)) proposed the Neural Graph Collaborative Filtering (NGCF) utilize the user-item interac- tion for recommendation.
3 PRELIMINARIES
In this section, we are going to present some preliminaries about factorization machine and graph convolutional neural networks.
Throughout this paper, a graph is represented by 9 = (V,E) where V = {vi) represents the node set and & = {(v ug)} represents the edge set. In this paper, focus the attributed graph and the node feature matrix is represented by X [x1, X2, x,.] Rddx In this work, focus on the high-dimensional categorical features, which are very common in real-world applications. such recommendation and CTR prediction. Specifically, represented as x; = [0. 1,0, ... 1, 1, 0, 0J* E R whose features is categorical and the number of non-zero values is much less than d. In addition, if node has the ground-trun truth (e.g. the regression task in our experiment), it is denoted by Y [v]] E R". Note that we will use samples and nodes interchangeably throughout this paper.
assume the feature of each node is
Based on aforementioned terminologies, Factorization Machine (FM) is defined as follows:
i = b b + E(vp Vo)
p-oq
where 0 E IR denotes the prediction of node U; x; [xt.pl Rd represents node features where Tip is the p-th feature of node us Vp E E stands for the embedding of the p-th feature. Compared the regular linear model, FM can capture the interaction between different features. Specifically, in the non-linear term, the dot product (Vp, Vg) computes the interaction between feature Ti.p and Fi.g. However, FM can only capture the interaction inside each node, ignoring interaction between different nodes.
Convolution is an effective operator to capture the local correlation. The regular convolutional op- erator is used to extract features by exploring the feature correlation. the contrary, the graph convolutional operator is proposed to explore the sample correlation. Specifically, the graph volutional operation in the 1-th hidden layer of the Graph Convolutional Neural Network (GCN)is defined as follows:
1
Σ
w w
VWO PEN() VIN(M))
h. 1+1 = f()
where b/ E RRd denotes the hidden representation of node in the 1-th layer, W1+ E Rdixd4 represents the model parameter, (i) indicates the neighbors of node u,, and f()) stands for the non-linear activation function. It can be found that the representation (+1 of the i-th sample is constructed by aggregating its neighbors. In this way. GCN can capture the sample interaction. However, although GCN can explore the interaction between different samples, yet it is not good at exploring the feature interaction. In the following section, we will propose new model address these two issues of FM and GCN.
4 DEEP RELATIONAL FACTORIZATION MACHINE
As shown in Eq. (T). the regular FM (Rendle 2010) only considers the interaction between different features, ignoring the interaction between different samples. In many real-world applications, the sample interaction might be important for prediction. For instance, in a recommender system, the interaction between users and items is important make accurate recommendation. If ignoring this
3

---

Under review as a conference paper at ICLR 2020
kind of interaction, it will be difficult to find the potential items to do recommendation. Another good example is the campaign performance prediction in online advertising. Specifically, an advertiser may only change small part of its original campaign, such targeting countries, and launch campaign. As a result, these two campaigns share a lot of common information. If we can capture the correlation between these campaigns, it will be beneficial to get better prediction result.
Based on the aforementioned intuitionn a natural question is that: how to capture the feature interac- tion and sample interaction simultaneously? A straightforward method combine them together directly as follows:
D1 yfu PM + Vi
CCN
UM w"xx x;
Σ py Va) p<o
vi GON g(f(
Σ PENTO VIV(F)
xxxx)))
VN(1)
where g() denotes the prediction function based on node features. Although this straightforward method can explore the feature interaction and sample interaction simultaneously, yet it can be seen that GCN and FM are almost independent. Specifically, the prediction from FM does not use the sample interaction and that from GCN does not involve the feature interaction too.
To address this issue, we propose the Deep Relational Factorization Machine (DRFM). In detail, our proposed DRFM also has two components: the sample interaction component and the relational fea- ture interaction component. As for the sample interaction component, we proposed a novel sample interaction layer which acts on the sample graph. As for the relational feature interaction compo- nent, we proposed to capture both the high-order feature interaction based on the feature gruph and the sample interaction based the sample graph.
4.1 RELATIONAL FEATURE INTERACTION
The relational feature interaction (RFI) component aims at dealing with the categorical features to capture the feature interaction. At the same time, it should capture the sample interaction. Based on these goals, the prototype of our relational feature interaction component is defined as follows:
h (x,) = T X; Σ(ver Vapai
P<o
(4)
RFI Vi
Σ
(xx))
VW(1)
(EN() VI(V)
where F. (x²) denotes the tures to capture the feature interaction, JRF represents the prediction for node v; from the RFI component which considers both the feature interaction and sample interaction. Compared with the naive method, we can find that our proposed RFI can capture the sample interaction when dealing with categorical features, while the naive method cannot.
kemel which is used to deal with categorical fea-
However, Eq. (4) can only capture the second-order interaction between different features, ignoring the high-order Interaction. As we know, second order may be not enough due to the complexity of real-world datasets. Thus, it is necessary to capture the high-order feature interaction. tion, the relationship between different features might be highly non-linear. Thus, it is important to explore the non-linearity between different features. To address these issues, we further propose novel high-order feature-interaction kemel. Specifically, we deal with this issue from totally new perspective. In particular, given a sample with categorical features, we can construct a con- current feature graph in terms of the concurrence between different features. For instance, given x = [0, 1, 0, 1, 1, 0, 0] where the first, fourth, and fifth feature appear simultaneously, then there should be link between them in the concurrence graph due their concurrence, which is shown as
addi-
4

---

Under review as a conference paper at ICLR 2020
follows:
0
0
0
0
1
0
1
0
1
G
0
1
0
1
(5)
0
1
0 0
0
0
For the concurrence graph, each feature is viewed as a node in the graph. As a result, a path in this graph indicates the concurrence of the features in this graph. Then, long path corresponds to a high-order feature interaction, while a short path corresponds to a low-order feature interac- tion. Consequently, lot of operations on the graph can be used to deal with the high-dimensional categorical features. In particular, inspired by the graph convolutional operation, we propose the following model to capture the high-order interaction between different features layer by layer:
1+1 graph_convly vgg
5 vW
1+1 (WW
(6)
h ht+1
p
p=Ptp==1
where v denotes the embedding of the p-th feature in the /-th layer. ví encodes the high-order interaction in layers. Here, the graph conv to capture the interaction between different features. Unlike the standard graph convolutional operation, we propose the following interaction kernel:
graph.convl vp voy) = Σ
g:Gpo=
where denotes the element-wise product. It can be seen that we always use the embedding in the input layer vp to interact with that in the high layer. In this way. the first layer will capture the second-order interaction. In the second layer, it will capture the third-order interaction, and in the high layers. high-order interaction will be captured. In other words, our method can capture the high-order interaction linearly. On the contrary, if use vś, do the product, the order will increase in an exponential way which might be too aggressive. Moreover, unlike existing high-order methods (Wang al. 2017, Lian et al., 2018), we conduct the non-linear transformation for the feature embedding in each layer, which is shown in the second and third formulation in Eq. (6). to handle the highly non-linear relationship. At last, to capture the feature interaction in different order, concatenate h in all layers as follows:
h [(h)) (B2)),....(B/))7 (1/
h RFI
Σ
VI((1) VEN() VI((F)
In summary, we proposed a novel feature interaction component based on the feature concurrence graph, which can capture the high-order interaction and explore the non-linearity. As far as we know, this is the first work trying to deal with feature interaction from the graph perspective. This will open a new avenue to tackle high-dimensional categorical features by using graph operations.
4.2 SAMPLE INTERACTION
With our proposed RFI component, our model can capture the high-order feature interaction. How- ever, the RFI component cannot capture high-order sample interaction. To address this issue, we develop a novel sample-interaction (SI) component, which is used to further explore the interaction between different samples. Moreover, the sample-interaction component should benefit from the RFI component. Therefore, enforce the SI component shares the same feature embedding with RFI component. this way, the feature embedding will be updated by both components. On the contrary, in the naive method, two components only share the raw input features. The high-order
5

---

Under review as a conference paper at ICLR 2020
sample interaction information cannot be used update the feature embedding. As result, in our method, the two components can benefit each other.
Specifically, the SI component is defined as follows:
E b = h! + VW(I PENO) VI((F)
h h
(9)
h (
where hς E -1 Vp. Compared with the regular graph convolutional operation, our method conducts explicit sample interaction by h h². In addition, by using residual connection, method can capture the sample interaction both linearly and exponentially.
Similar with the feature interaction component, to get different orders of sample interaction, we keep all the intermediate h.. Then, we concatenate all of them as the node representation:
h
(10)
At last, after obtaining the representation from these two components, we concatenate them together for prediction as follows:
ý = [(h RT. (h:")
(11)
With this prediction, we can use our model for both classification and regression tasks.
5
EXPERIMENTS
In this section, we design experiments to verify the performance of the proposed approach.
5.1 DATASETS
Modcloth is dataset where users rate the clothes they bought. Each user has five attributes:
[user id, bra size, cup size, hips, height Each cloth has three attributes: [item id, size, category]. There are 47,185 users and 1,364 clothes.
Renttherunway is dataset where 88,178 users rate 5,795 clothes they rented. Here, each user has six attributes: [user_id, weight, body_type, age, bust_size, height] while each cloth has three attributes: [item_id size, category).
Book-crossing contains the historical rating information for books by users. Users have three attributes: [user_id, location, age]. Books also have three attributes fisbn, yearofpub- lication, publisher). In addition, the number of users is 278,858 and the number of books is 271,360.
Company- X-CTR data an advertiser-level CTR data which includes the winning rate of each campaign bidding for exposure. Each campaign has 63 categorical attributes.
In our experiments, all attributes of these datasets are transformed categorical features. Then, we use the one-hot encoding to represent the categorical feature. Moreover, the first three datasets construct bipartite graph respectively. for the last dataset, campaigns construct a regular graph. At last, summarize the statistics of these datasets in Table
5.2 EXPERIMENTAL SETTINGS
Throughout our experiments, we evaluate our method on two tasks: link prediction (Section 5.3 and regression (Section 5.4). For the link prediction task, we use the first four datasets. In particular, all the existing links in graph are viewed as positive links while non-existing links are treated negative ones. We randomly select 10% positive links for the training set and 10% positive links for the validation set. The rest positive links used for the testing set. In addition, negative links for these three sets where the amount of negative links is same as that of positive links in each set. As for the regression task, use the Company-X X-CTTR data, campaigns with the same
randomly select the

---

Under review as a conference paper at ICLR 2020
Table 1: Real-world data used for link prediction and regression. Here, the features of first three datasets include those of users and items.
Dataset
#Nodes
#Edges
#Features
Modcloth
48,549
40,607
47,222+1,399
Renttherunway
93,973
82,415
88,382+5,
Book-crossing
550,218
515,568
336,2834288,268
Company-X-CTR
144,876
72,438
48,474
Table 2: Link prediction results (AUC).
Methods
Modcloth
Renttherunway
Book-crossing
DeepFM
0.8839 ++0.0230)
0.7518 (+0.0236)
0.6807 (+0.0651)
DCN
0.8454 (+000640)
0.6892 (=0.0467)
0.6720 (+0.0321)
PNN
0,9079 (#0.0083)
0.7565 (#0.0221)
0.7793 (#0.0344)
XDeepFM
0,8919 (1:00366)
0.7314
(#:0.049)
0.6917
(#:0.066)
GCN
0.8985 (=0.0007)
0.7428
(=0.0005)
0.9075(#0.0004)
DRFM
0.9223 (400.0033
0.7930 (1:00015)
0.9115(+0.0005)
%Gain
1.6%
4.9%
0.4%
placement id are connected to construct the graph. The winning rate of each campaign is normalized to [0, 1].
To evaluate the performance of our proposed method, we compare with baseline methods, which are described as follows:
DeepFM (Guo et al. 2017) combines the standard FM and multi-layer perceptron neu- ral (MLP) network together where the input of MLP is the feature embedding from FM. DeepFM can capture the second-order feature interaction explicitly by FM and high-order feature interaction implicitly by MI.P
DCN (Wang et al. 2017) is proposed to capture the high-order feature interaction explicitly by stacking multiple interaction layers together.
PNN (Qu et al. 2016) uses product layer to capture the feature interaction explicitly and then stacks MLP over the product layer to capture the high-order interaction implicitly.
XDeepFM (Lian et al. 2018) also aims at capturing the high-order feature interaction ex- plicitly layer by layer like DCN. But uses a different feature interaction layer from DCN.
GCN (iipf Welling 2016) graph convolutional neural network whose goal to capture the correlation among samples for prediction.
Throughout our experiments, we set the embedding size of each feature to 10 for all methods. As for the sample interaction component of method, the dimension is set to [16, 16). To make fair comparison, GCN and the baseline methods with an MLP component are also set in the same way. As for the feature interaction component of our method, the dimension is set to [10, 10, 10). Similarly, baseline methods with the high-order feature interaction component are also set to the same dimension. Moreover, the batch size to 1024 and the learning rate set to 0.0001.
5.3 LINK PREDICTION
In Table 2 we show the results for link prediction across number of graphs from different domains. In all cases, DRFM outperforms the baseline methods across all the different real-world data sets. Compared to the best baseline method, DRFM achieves gain in AUC of 1.6% for Modcloth, 4.9% for Renttherunway, 0.4% for Book-crossing. From Table 2 we also observe that by combining FMs and GCNs, the standard deviation in AUC is much smaller than other FM-based methods and
7

---

Under review as a conference paper at ICLR 2020
0.00
04225
6942
0.300
0010
0.00
0.166
OPN
Deep
aDeeptM
Figure 1: The link prediction result of DRFM and DRFM-second.
Figure 2: Regression results for Company-X-CTR
typically less than GCN as well. These results indicate utility of combining FMs and GCNs for learning more accurate and robust model for prediction.
To further verify the effectiveness of our proposed high-order feature interaction component, we compare DRFM with its variant which replaces the hihg-order feature interaction component with the regular FM component. Here, we call it DRFM-second since it can only capture the second- order feature interaction. Due to the space limitation, we only report the result of Modecloth in Figure [ It can be seen that DRFM with our high-order feature interaction component has much better performance than that with only the regular FM component, which confirms the effectiveness of our high-order feature interaction component.
5.4 REGRESSION
To further verify the performance of our proposed method, we use DRFM to predict the winning rate of a campaign bidding for exposure. Here, each campaign configured with different attributes, such as targeting countries, targeting device types, user segment rules, etc. Advertisers might change only one or two attributes and launch a another campaign. This new campaign and its original campaign share lot of common information so that they are highly correlated. Thus, it will be beneficial to capture the relationship between different campaigns when making prediction. To this end, construct the graph for campaigns. Specifically, in this dataset, this kind of new campaigns and their original campaigns share the same ID. Thus, we can construct the graph in terms of the shared ID. More specifically, two campaign connected if they share same ID.
The result is shown in Figure Here, to measure the performance of different methods, we use Root Mean Squared Error (RMSE) the metric. A smaller value indicates better result. It can be that the regular GCN performs worse than all the other baseline methods. The possible reason is that GCN does not utilize the feature interaction for prediction, while feature interaction confirmed to be powerful technique in the high- sigg--dmensiinal CTR prediction. Moreover, our method DRFM outperforms all state-of of the-art FMs, which confirms the effectiveness of incorporating the relational information for prediction.
CONCLUSION
In this work, we described a new class of models that combine Factorization Machines (FMs) and Graph Neural Network (GNNs) into a unified learning approach. By seamlessly combining FMs and GNNs, we obtain the unique advantages offered by each while overcoming the issues that arise when either is used independently. Using real-world data from different domains, we demonstrated the effectiveness of combining GNNs and FMs for both link prediction and regression tasks. While this work demonstrated the utility combining FMs and GNNs in single unified learning framework, there remains many open research problems to investigate in future work. One important future direction is to explore other FM and GNN variants (besides the vanilla ones used in this work), and systematically investigate the utility and effectiveness of these different combinations.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization machines. In Advances in Neural Information Processing Systems, pp. 3351-3359, 2016.
Xiaoshuang Chen, Yin Zheng, Jiaxing Wang, Wenye Ma, and Junzhou Huang. Rafm: Rank-aware factorization machines. arXiv preprint arXiv: 1905,07570, 2019.
Huifeng Guo, Ruiming Tang. Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim: factorization- machine based neural network for ctr prediction. arXiv preprint
2017.
Wei Guo, Ruiming Tang, Huifeng Guo, Jianhua Han, Wen Yang, and Yuzhou Zhang. Order-aware embedding neural network for ctr prediction. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1121-1124. ACM, 2019.
Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference Research and Development in Information Retrieval, pp. 355-364. ACM, 2017.
Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization machines for ctr prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, pp- 43-50, ACM, 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 160%, (1606000900) 2016.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang. Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm:: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1754-1763. ACM, 2018.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1149-1154. IEEE, 2016.
Steffen Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining,
pp. 995-1000. IEEE, 2010.
Steffen Rendle. Social network and click-through prediction with factorization machines. In KDD- Cup Workshop, pp. 113, 2012.
Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-Thiemee Fast context- aware recommendations with factorization machines. In Proceedings the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 635-644.
ACM, 2011.
Anh-Phuong Ta. Factorization machines with follow-the bolloo--te-rrgullrizzeeeeeed for etr prediction in display advertising. In 2015 IEEE International Conference Big Data (Big Data), pp. 2889- 2891. IEEE, 2015.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pp. 12. ACM, 2017.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering, arXiv preprint arXiv:1905.00 08108, 2019.
Gang Wu, Viswanathan Swaminathan, Saayan Mitra, and Ratnesh Kumar. Context-aware video recommendation based on session progress prediction. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 1428-1433. IEFF, 2017.
Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional fac- torization machines: Learning the weight of feature interactions via attention networks. arXiv preprint arXiv: /708. 100.177087..7777 2017.

---

Under review as a conference paper at ICLR 2020
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983. ACM, 2018.
Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data. In European conference on information retrieval, pp. 45-57. Springer, 2016.
10


Reviews:
[{"review_id": "r1x93hOWtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose generalize the FM to consider both interaction between features and interaction between samples. For the interaction between features, the authors propose to use graph convolution to capture high-order feature interactions. Moreover, the authors construct a graph on the instances based on similarity. Then a GCN is applied to the sample graph where the feature embedding is shared between the two components. Experiments are carried out on four datasets with tasks of link prediction and regression. Comparison to several baselines demonstrate the superior performance of the proposed method.\n\nStrength:\n1. The idea of utilizing GCN on the feature co-occurrence graph is interesting and innovative. The idea could possibly be combined with other variants of Deep FM models.\n2. It is an interesting idea to combine sample similarity together with feature co-occurrence for better prediction accuracy.\n\nWeakness:\n1. Many descriptions in the paper are not very clear. First, the authors only mention how prediction is carried out with trained parameters. However, there is no description of the training process like what is the target used for the two components. What is the training procedure? Are the two components trained jointly? Second, the authors provide little description on how the sample similarity graph is constructed excepts for the Ad campaign dataset. Third, it is not clear how is the link prediction evaluation carried out. From the size of the graph, the authors seem to include both user and item in the graph. However, the user and item has disjointed feature set. It is not clear how the GCN is computed for the heterogenous nodes in the graph. Moreover, how is link prediction carried out, by taking inner product (cosine similarity) of the final representation.\n2. For equation (8) in section 4.1, why we need to compute h_i^{RFI}. This should be the feature representation of sample i. However, the average is computed without include sample i itself. Also, are the neighbors defined in the sample similarity graph? Should we use the sample interaction in section 4.2 to capture that?\n3. Though it is interesting idea to use graph convolution on the feature occurrence graph, it would be much better if the authors could provide more intuition on the output of the GCN. It would be helpful to study a few simple cases like without non-linearity. Is it a generalization to high-order FM without non-linearity? Also, it would be interesting to see experiments results using the graph convoluted feature representation directly for final representation. Also, some visualization of the learned feature embedding also helps.\n4. The authors should carry out ablation study for different components of the model. Moreover, it would be much better if the authors can carry out experiments on some widely used recommendation datasets and use standard evaluation metrics for ranking.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1eyeua6tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to combine the graph neural networks and factorization machines. First, the authors propose a relational feature interaction component (RFO) tp deal with the categorical features. This component first uses the factorization machine to project the features to h^FI(x), then it uses an aggregation operation to get the prediction y^RFI. To explore high-order correlations, the authors further propose to calculate a concurrence graph, on which RFI propagates the embedding vectors to get relational high-order correlations. To further model high-order sample interactions, this work then presents a special graph convolutional operation that considers the element-wise products of the encoded features. \n\nMy comments are as follows:\n\n- The idea of integrating the GNN and FMs is interesting and intuitive. However, the proposed method is simple.\n\n- As the work proposes a new model architecture, a graphical illustration and the pseudo-code is necessary for the audiences.\n\n- Some parts are useless for the whole paper. For example, 'a straightforward method is to combine...' (Eq. 3). The discussion of the simple RFI-component is also needless since the paper mainly proposes the high-order version. These paragraphs should be simplified or removed. \n\n- The graph convolution operation in Eq. (7) first considers all the element-wise products of the embedding vectors, which is the same as the original FM. Then the authors use G, the concurrence graph, to propagates the embedding vectors. One concern is that the original FMs also consider the graphical information in G, i.e. the concurrence relation. Will the GNN technique improve the usage of this topological information of the features? \n\n- An ablation study is necessary to show the contribution of the proposals. For example, comparing the naive RFI and high-order RFI; the performance with and without RFI/SI components.\n\n- I would be grateful if the authors provide the running time comparison of the proposed method.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skev-GNHqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to combine FMs and GNNs to capture both sample and feature interactions. First, feed the feature from FMs to GNNs. Second, build high-order interactions.\n\nStrength:\n[1] This paper tries to solve an interesting question and the idea is simple and intuitive\n[2] Experiments show that the proposed approach outperforms a number of baselines\n\nMy comments:\n[1] main concern: lack of ablation study. It would be great to analyze the effects of different components and illustrate/visualize the learned features to see what's the difference and why/how such difference help\n[2] another concern is complexity. It would be great to see learning curve and computational time analysis", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to combine FMs and GNNs. All reviewers voted reject, as the paper lacks experiments (eg ablation studies) and novelty. Writing can be significant improved - some information is missing. Authors did not respond to reviewers questions and concerns. For this reason, I recommend reject. 

Decision: Reject