Label Propagation Networks

1 INTRODUCTION . We study the problem of graph-based semi-supervised learning ( SSL ) , where the goal is to correctly label all nodes of a graph , of which only a few are labeled . Methods for this problem are often based on assumptions regarding the relation between the graph and the predicted labels . One such assumption is smoothness , which states that adjacent nodes are likely to have similar labels . Smoothness can be encouraged by optimizing an objective where a loss term L over the labeled nodes is augmented with a quadratic penalty over edges : min f L ( yS , fS ) + λ ∑ ( i , j ) ∈E wij‖fi − fj‖22 ( 1 ) Here , y are the true labels , f are “ soft ” label predictions , S is the set of labeled nodes , and w are non-negative edge weights . The quadratic term in Eq . ( 1 ) is often referred to as Laplacian Regularization since ( for directed graphs ) it can equivalently be expressed using the graph Laplacian ( Belkin et al. , 2006 ) . Many early methods for SSL have adopted the general form of Eq . ( 1 ) ( Zhu et al. , 2003 ; Zhou et al. , 2004 ; Belkin et al. , 2004 ; Bengio et al. , 2006 ; Baluja et al. , 2008 ; Talukdar & Crammer , 2009 ; Weston et al. , 2012 ) . Algorithms such as the seminal Label Propagation ( Zhu et al. , 2003 ) are simple , efficient , and theoretically grounded but are limited in two important ways . First , predictions are parameterized either naïvely or not at all . Second , edge weights are assumed to be given as input , and in practice are often set heuristically . Recent deep learning methods address the first point by offering intricate predictive models that are trained discriminatively ( Weston et al. , 2012 ; Perozzi et al. , 2014 ; Yang et al. , 2016 ; Kipf & Welling , 2016 ; Grover & Leskovec , 2016 ; Hamilton et al. , 2017 ; Monti et al. , 2017 ) . Nonetheless , many of them still require w as input , which may be surprising given the large body of work highlighting the importance of good weights ( Zhu et al. , 2003 ; Kapoor et al. , 2006 ; Wang & Zhang , 2008 ; Belkin et al. , 2004 ; Karasuyama & Mamitsuka , 2013 ) . While some methods consider some form of weight learning ( Velickovic et al. , 2017 ; Monti et al. , 2018 ) , to some extent they have drifted away from the original quadratic criterion . Other works address the second point by proposing disciplined ways for learning w. However , these either assume specific simple parameterizations ( Zhang & Lee , 2007 ; Karasuyama & Mamitsuka , 2013 ) , or altogether consider weights disjointly from predictions ( Wang & Zhang , 2008 ; Liu et al. , 2010 ) . Our goal in this paper is to simultaneously addresses both issues . We propose a framework that , given a graph , jointly learns both a parametric predictive model and the edge weights . To do this , we begin by revisiting the Label Propagation ( LP ) , and casting it as a differentiable neural network . Each layer in the network corresponds to a single iterative update , making a forward pass equivalent to a full run of the algorithm . Since the network is differentiable , we can then optimize the weights of the LP solution using gradient descent . As we show , this can be done efficiently with a suitable loss function . The key modeling point in our work is that labeled information is used as input to both the loss and the network . In contrast to most current methods , our network ’ s hidden layers directly propagate labeling information , rather than node or feature representations . Each layer is therefore a self-map over the probability simplex ; special care is therefore needed when introducing non-linearities . To this end , we introduce two novel architectural components that are explicitly designed to operate on distributions . The first is an information-gated attention mechanism , where attention is directed based on the informativeness and similarity of neighboring nodes ’ states . The second is a novel “ bifurcation ” operator that dynamically controls label convergence , and acts as a balancing factor to the model ’ s depth . Our main guideline in designing our model was to tailor it to the semi-supervised setting . The result is a slim model having relatively few parameters and only one model-specific hyper-parameter ( depth ) , making it suitable for tasks where only few labeled nodes are available . The final network provides a powerful generalization of the original propagation algorithm that can be trained efficiently . Experiments on benchmark datasets in two distinct learning settings show that our model compares favorably against strong baselines . 1.1 RELATED WORK . Many SSL methods are based on Eq . ( 1 ) or on similar quadratic forms . These differ in their assumed input , the optimization objective , and the parametric form of predictions . Classic methods such as LP ( Zhu et al. , 2003 ) assume no parametric form for predictions , and require edge weights as inputs . When node features are available , weights are often set heuristically based on some similarity measure ( e.g. , wij = exp ( ‖xi − xj‖22/σ2 ) ) . LP constrains predictions on S to agree with their true labels . Other propagation methods relax this assumption ( Belkin et al. , 2004 ; Bengio et al. , 2006 ) , add regularization terms ( Baluja et al. , 2008 ) , or use other Laplacian forms ( Zhou et al. , 2004 ) . Some methods aim to learn edge weights , but do not directly optimize for accuracy . Instead , they either model the relations between the graph and features ( Wang & Zhang , 2008 ; Liu et al. , 2010 ) or simply require f as input ( Daitch et al. , 2009 ; Kalofolias , 2016 ; Dong et al. , 2016 ) . Methods that focus on accuracy are often constrained to specific parameterizations or assumptions ( Karasuyama & Mamitsuka , 2013 ) . Zhang & Lee ( 2007 ) optimize the leave-one-out loss ( as we do ) , but require a series of costly matrix inversions . Several recent works in deep learning have been focused on graph inputs in general ( Battaglia et al. , 2018 ) and specifically for inductive SSL . The main idea behind these methods is to utilize a weighted graph to create meaningful vector representations of nodes , which are then fed into a classifier . Methods are typically designed for one of two settings : when the input includes only a graph , and when node features are available . When the input includes only a graph ( and no features ) , node representations are generated using embedding techniques . Perozzi et al . ( 2014 ) use a SkipGram model over random walks on the graph , which are used to define context . Grover & Leskovec ( 2016 ) further this idea by introducing expressive parameterized random walks , while Tang et al . ( 2015 ) focus on optimizing similarities between pairs of node embeddings . Various methods have been proposed to utilize node features , when available . Spectral methods , stemming from a CNN formulation for graphs ( Bruna et al. , 2014 ) , include different approximations of spectral graph convolutions ( Defferrard et al. , 2016 ; Kipf & Welling , 2016 ) adaptive convolution filters ( Monti et al. , 2017 ) , or attention mechanisms ( Velickovic et al. , 2017 ) Embedding approaches have been suggesting for handling bag-of-words representations ( Yang et al. , 2016 ) and general node attributes such as text or continuous features ( Duran & Niepert , 2017 ) . Many of the above methods can be thought of as propagating features over the graph in various forms . Our method , in contrast , propagates labels . The main advantage label propagation is that labeled information is used not only to penalize predictions ( in the loss ) , but also to generate predictions . 1.2 PRELIMINARIES . We begin by describing the learning setup and introducing notation . The input includes a ( possibly directed ) graph G = ( V , E ) , for which a subset of nodes S ⊂ V are labeled by yS = { yi } i∈S with yi ∈ { 1 , . . . , C } . We refer to S as the “ seed ” set , and denote the unlabeled nodes by U = V \ S , and the set of i ’ s ( incoming ) neighbors by Ni = { j : ( j , i ) ∈ E } . We use n = |V | , m = |E| , ` = |S| , and u = |U | so that n = ` + u . In a typical task , we expect ` to be much smaller than n. We focus on the transductive setting where the goal is to predict the labels of all i ∈ U . Most methods ( as well as ours ) output “ soft ” labels fi ∈ ∆C , where ∆C is the C-dimensional probability simplex . For convenience we treat “ hard ” labels yi as one-hot vectors in ∆C . All predictions are encoded as a matrix f with entries fic = P [ yi = c ] . For any matrix M , we will use MA to denote the sub-matrix with rows corresponding to A . Under this notation , given G , S , yS , and possibly x , our goal is to predict soft labels fU that match yU . In some cases , the input may also include features for all nodes x = { xi } i∈V . Importantly , however , we do not assume the input includes edge weights w = { we } e∈E , nor do we construct these from x . We denote by W the weighted adjacency matrix of w , and use W̃ and w̃ for the respective ( row ) -normalized weights . 2 UNROLLING LABEL PROPAGATION . Many semi-supervised methods are based on the notion that predictions should be smooth across edges . A popular way to encourage such smoothness is to optimize a ( weighted ) quadratic objective . Intuitively , the objective encourages the predictions of all adjacent nodes to be similar . There are many variations on this idea ; here we adopt the formulation of ( Zhu et al. , 2003 ) where predictions are set to minimize a quadratic term subject to an agreement constraint on the labeled nodes : f∗ ( w ; S ) = argmin f : fS=yS ∑ ( i , j ) ∈E wij‖fi − fj‖22 ( 2 ) In typical applications , w is assumed to be given as input . In contrast , our goal here is to learn them in a discriminative manner . A naïve approach would be to directly minimize the empirical loss . For a loss function L , regularization term R , and regularization constant λ , the objective would be : min w 1 ` ∑ i∈S L ( yi , f ∗ i ( w ; S ) ) + λR ( w ) ( 3 ) While appealing , this approach introduces two main difficulties . First , f∗ is in itself the solution to an optimization problem ( Eq . ( 2 ) ) , and so optimizing Eq . ( 3 ) is not straightforward . Second , the constraints in Eq . ( 2 ) ensure that f∗i = yi for every i ∈ S , making the loss term in Eq . ( 3 ) redundant . While some methods solve this by replacing these with weak constraints , a third issue is that it is still not clear why optimizing edge weights for yS should generalize well to the unlabeled nodes . In what follows , we describe how to overcome these issues . We begin by showing that a simple algorithm for approximating f∗ can be cast as a deep neural network . Under this view , the weights ( as well as the algorithm itself ) can be parametrized and optimized using gradient descent . We then propose a loss function suited to SSL , and show how the above network can be trained efficiently with it .

Summary: This paper presents an interesting idea for the following task: given a graph and a subset of labelled nodes, infer the labels on the remaining nodes. Here the authors will make prediction for absent labels based on local averages on the graph of the neighbouring soft labels. The main originality is that the local average is weighted and the weights are learnt.