ColdExpand: Semi-Supervised Graph Learning in Cold Start

1 INTRODUCTION . Graph-based semi-supervised learning has attracted much attention thanks to its applicability to real-world problems . For example , a social network is graph-structured data in which people in the network are considered to be nodes and relationships between people are considered to be edges : two people are friends or sharing posts , etc . With this structural information , we can infer some unknown attributes of a person ( node ) based on the information of people he is connected to ( i.e. , semi-supervised node classification ) . In the case of retail applications , customers and products can be viewed as heterogeneous nodes and edges between customers and products can represent relationships between the customers and the purchased products . Such a graph can be used to represent spending habits of each customer and we can recommend a product to a user by inferring the likelihood of connection between the user and the product ( i.e. , semi-supervised link prediction ) . Recent progress on Graph Neural Networks ( GNN ) ( Bruna et al. , 2013 ; Kipf & Welling , 2016a ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ; Jia et al. , 2019 ) allows us to effectively utilize the expressive power of the graph-structured data and to solve various graph related tasks . Early GNN methods tackled semi-supervised node classification task , a task to label all nodes within the graph when only a small subset of nodes is labeled , achieving a satisfactory performance ( Zhou et al. , 2004 ) . Link prediction is another graph-related task that was covered comparatively less than other tasks in the field of GNNs . In the link prediction task , the goal is to estimate the likelihood of connection between two nodes given node feature data and topological structure data . Link prediction can be used in recommendation tasks ( Chen et al. , 2005 ; Sarwar et al. , 2001 ; Lika et al. , 2014 ; Li & Chen , 2013 ; Berg et al. , 2017 ) or graph completion tasks ( Kazemi & Poole , 2018 ; Zhang & Chen , 2018 ) . Most of the work on semi-supervised graph learning and link prediction assumes a static graph , that is , the structural information is at least “ partially ” observable in terms of nodes . In the real world , however , new users or items can be added ( as nodes ) without any topological information ( Gope & Jain , 2017 ) . This is also referred as the cold start problem when a new node is presented to an existing graph without a single edge . In contrast to the warm start case , in which at least some topological information is provided , the cold start problem is an extreme case where there isn ’ t any topological information to refer . In this setting , previous semi-supervised learning algorithms can not propagate information to the cold nodes . Even though the cold start problem is an extreme setting , it is an inevitable problem that occurs often in the real world data . In this paper , we postulates the cold start problem as a fundamental issue in graph learning and propose a new learning setting , expanded semi-supervised learning . In expanded semi-supervised learning we extend the original semi-supervised learning setting even to new cold nodes that are disconnected from the graph as shown in Figure 1 . To this end , we suggest the ColdExpand method , the method that uses multi-task learning strategy to alleviate the cold start problem that typical graph learning methods face . We experimentally prove that by adding an additional goal to the existing link prediction method , performance on the link prediction task is enhanced in every benchmark dataset and at most by 24 % . We also prove that our method can expand semi-supervised node classification even to unseen , cold nodes . To the best of our knowledge , this is the first study to expand semisupervised learning methods to unseen nodes . In the next section , we briefly introduce related works . In Section 3 , we define our problem definition . Finally , in Section 4 we propose our ColdExpand model followed by our experimental environments along with corresponding results . 2 RELATED WORK . 2.1 SEMI-SUPERVISED NODE CLASSIFICATION . Semi-supervised learning is a combination of unsupervised learning and supervised learning where true labels are only partially given . Semi-supervised learning algorithms aim to improve the model ’ s performance by using unlabeled data points on top of labeled ones to approximate the underlying marginal data distribution ( Van Engelen & Hoos , 2020 ) . One of the most popular fields of semi-supervised learning is graph based semi-supervised node classification where all nodes should be classified when only a few node labels are given . In order to smooth the given subset of labels throughout the graph , various methods have been studied to effectively represent nodes . Deep Walk ( Perozzi et al. , 2014 ) , LINE ( Tang et al. , 2015 ) , node2vec ( Grover & Leskovec , 2016 ) were early deep-learning-based methods targeting the node classification task by learning latent representations from truncated random walks . However , these models fail to share parameters between nodes causing learning to be inefficient . Kipf & Welling ( 2016a ) introduced Graph Convolutional Networks ( GCN ) which use an efficient layer-wise propagation rule by approximating the first-order of spectral graph convolution . By limiting the spectral convolution to the first-order , GCNs not only lightened computation cost of the operation but also alleviated the over-fitting problem that previous spectral methods had . Commonly , GCN model is formed from stacked convolutional layers ; the number of which decides how many hops of neighbors to consider for the convolution operation . Gilmer et al . ( 2017 ) presented Message Passing Neural Networks ( MPNN ) as a general form of spatial convolution operation and treated GCN as specific kind of a message passing process . In MPNNs , information between nodes is delivered directly by edges without visiting any spectral domains . 2.2 LINK PREDICTION . Links in graph-structured data appeal important interactions between nodes that may not be represented solely by node attributes . To this end , link prediction is an important task in order to effectively utilize graph data . In the link prediction task , we aim to predict whether an edge exists between two nodes from the same graph . Predicting the existence of links between nodes is useful for many applications including recommendation systems and graph completion tasks . On a high level , link prediction methods can be separated into similarity-based and learning-based methods . Similarity-based link prediction is the simplest way to predict the link by assigning a similarity score Sxy between two nodes x , y . Common Neighbors ( Marchette & Priebe , 2008 ) , Salton Index , Jaccard Index ( Leydesdorff , 2008 ) , Sorensen Index , Hub Promoted Index , Hub Depressed Index ( Zhou et al. , 2009 ) , Leicht-Holme-Newman Index ( LHN1 ) ( Leicht et al. , 2006 ) each define similarity scores by using the overlapping set of neighboring nodes between two nodes . While the former neighborhood-based methods focus on somewhat local similarity , other similarity-based methods consider global similarity by taking into account higher order links . Katz Index ( Katz , 1953 ) considers the ensemble of all paths between two nodes summed with quadratic weight according to the distance of each path . SimRank ( Jeh & Widom , 2002 ) also uses random walk process but measures how soon two random walkers from the two different nodes meet . However , when a cold node is introduced , the required structural information needed for previous similarity-based approaches is missing , thus making them impossible to be applied in cold start situations ( Wang et al. , 2016 ) . On the other hand , learning-based methods formulate link prediction as a binary classification problem given feature embedding vectors of two nodes ; if an edge exists between the two nodes a positive label is given and vice versa . Kipf & Welling ( 2016b ) introduced the graph auto-encoder ( GAE ) , an unsupervised learning technique , which aims to reconstruct the original adjacency matrix . As a variant of GAE , variational graph auto-encoders ( VGAE ) additionally train a separate GCN module resulting in the logarithm of the variant matrix , log ( σ2 ) . Additional Kullback-Leibler divergence loss between the distribution of encoder and decoder is added to the original reconstruction loss . Pan et al . ( 2019 ) further improved upon GAEs by applying adversarial training to regularize the latent node embedding matrix Z to match a prior distribution of the original node feature matrix X . A simple MLP-based discriminator D was used for adversarial learning between the real distribution of G = { X , A } and the latent node matrix Z . Unlike similarity-based methods , learning-based methods can be applied to cold start problems since they make use of node feature information along with topological structure information . 2.3 COLD START PROBLEM . The cold start problem is a common problem in real-world graph-structured data where new nodes may be added to an existing graph . Most work tackling the cold start problem focuses on solving the recommendation problem in a cold start setting . Recommendation systems are widely used in the real world applications such as friend recommendation , movie recommendation and purchase recommendation . In the recommendation problem , the task is to predict recommendation links from a heterogeneous graph containing two types of graphs ; a user-user social graph and user-item rating graph . Users are involved in both types of graphs to connect the two types of graphs into a single heterogeneous graph . The collaborative filtering ( CF ) method is one of the traditional methods where a rating is predicted based on similarities between users or items . In short , the CF method makes recommendations based on ratings given by other users in the social graph . Sarwar et al . ( 2001 ) introduced an item-based CF method , which computes item-item similarities , and proved it outperforms other user-based collaborative filtering recommendation methods . Efforts to make use of CF for recommendation systems in the cold start scenario have been made . Lika et al . ( 2014 ) approached the cold user problem by exploiting CF followed by hard-clustering of users based on their attributes . Once the cluster of the new user has been decided , CF is only applied to members of the same cluster . Berg et al . ( 2017 ) formulated the matrix completion task of recommendation systems as a link prediction problem by viewing the user-item graph as a bipartite interaction graph . However , they instead used a warm start setting where at least one rating for a user is given which is not precisely a cold start assumption . 3 PROBLEM DEFINITION . A graph G is denoted as a pair ( V , E ) with V = { v1 , · · · , vN } the set of nodes ( vertices ) , and E ∈ V × V the set of edges . Each node vi is associated with a feature vector xi ∈ RF . To make notation more compact , the set of node feature vectors of graph G is denoted as a matrix X = [ x1 , x2 , · · · , xN ] > ∈ RN×F . Additionally , a graph has an N -by-N adjacency matrix A where Ai , j represents the existence of an edge between vi and vj and a degree matrix D , a diagonal matrix which contains information about the degree of each node . In the cold start setting , we will call those nodes that are newly introduced without any link from the existing graph cold nodes and let them be denoted by vci . In the same manner , existing nodes in the graph are denoted as vej . Then , Xe only contains feature vectors of existing nodes xe0 , x e 1 , x e 2 , . . . , x e Ne and Xc contains feature vectors of cold nodes xc0 , x c 1 , x c 2 , . . . , x c Nc , where Nc and Ne are the number of existing nodes and cold nodes , respectively . In our problem setting , we assume that new nodes are added to the existing graph and need to find possible links between new nodes and existing nodes followed by a node classification to figure out the hidden attributes of new nodes . Our problem definition is as follows : Given existing graph , G = ( Ve , A ) , and cold nodes Xc , find out all possible links between cold nodes Xc and existing nodes Xe , then predict the class of cold nodes Xc .

Summary: This paper is about the cold-start problem for representation learning on dynamics graphs. More specifically, the proposed method (ColdExpand) uses convolutional networks and multi-task learning (node classification loss and link prediction loss) to learn embeddings for new unseen nodes in the graph, i.e., nodes that we only know their features and not their connectivity to the existing graph. Such a problem is useful for applications of the link prediction problem and more specifically, recommendation tasks or graph completion tasks. The authors test their technique using three benchmark datasets (Cora, CiteSeer and PubMed) and by using building blocks of existing works they create some baselines to compare. They claim to be the first study for semi-supervised learning to unseen nodes. However, there are a couple of works in this field already, as for example the GraphSAGE model that they cite in this paper (see below for more).