Graph Neural Networks for Soft Semi-Supervised Learning on Hypergraphs

1 INTRODUCTION . In the last decade , deep learning models have been successfully embraced in many different fields and proved to achieve unprecedented performance on a vast range of applications Krizhevsky et al . ( 2012 ) ; Goodfellow et al . ( 2014 ) ; Bahdanau et al . ( 2015 ) ; LeCun et al . ( 2015 ) . Graph Convolutional Network ( GCN ) Kipf & Welling ( 2017 ) was recently proposed as an adaptation of a particular deep learning model ( i.e. , convolutional neural networks Lecun et al . ( 1998 ) ) to enable handling graph-structured data . GCN was shown to be , in particular , effective in semi-supervised learning on attributed graphs . GCNs have inspired the current state-of-the art models for graph-based SSL Wu et al . ( 2019a ) ; Veličković et al . ( 2018 ) ; Vashishth et al . ( 2019 ) . GCNs inherently assume that the labels of interest are numerical or categorical variables . However , in many real-world applications such as co-authorship networks , recommendation networks , etc. , vertex labels can be naturally represented by probability distributions or histograms . Moreover , these real-world network datasets have complex relationships going beyond pairwise associations . Such relationships can be modelled naturally and flexibly by hypergraphs . Moreover , hypergraphs can encode additional relationships with directions as illustrated in Figure 1 and these hypergraphs are directed hypergraphs Gallo et al . ( 1993 ) . Inspired by a prior work Solomon et al . ( 2014 ) that generalised label propagation to graph-based soft SSL setting and motivated by the fact that GNNs have inspired state-of-the-art models for traditional graph-based SSL , we make the following contributions . • We explore GNNs for soft SSL in which vertex labels are probability distributions . Motivated by real-world applications , we propose DHN ( Directed Hypergraph Network ) , a novel method for directed hypergraphs . DHN can be applied for soft-SSL using existing tools from optimal transportation ( Section 3 ) . • We provide generalisation error bounds for a one-layer GNN within the framework of algorithmic stability . We establish that such models , which use filters with bounded eigenvalues independent of graph size , can satisfy the strong notion of uniform stability and thus is generalisable . In particular , the algorithmic stability of a one-layer GNN depends on the largest absolute eigenvalue of the graph convolution filter ( Section 4 ) . • We demonstrate DHN ’ s effectiveness through detailed experimentation on real-world data . In particular , we demonstrate superiority over state-of-the-art hypergraph-based neural networks . We provide new empirical benchmarks for soft-SSL on directed hypergraphs and make the code available to foster reproducible research ( Section 5 ) . 2 RELATED WORK . Graph-based deep learning : Geometric deep learning Bronstein et al . ( 2017 ) is an umbrella phrase for emerging techniques attempting to generalise ( structured ) deep neural networks to nonEuclidean domains such as graphs and manifolds . GCN Kipf & Welling ( 2017 ) and their various extensions are the current state-of-the art for graph-based SSL Wu et al . ( 2019a ) ; Veličković et al . ( 2018 ) ; Monti et al . ( 2018 ) ; Vashishth et al . ( 2019 ) ; Ma et al . ( 2019 ) ; Qu et al . ( 2019 ) and graphbased unsupervised learning Hamilton et al . ( 2017a ) ; Veličković et al . ( 2019 ) problems . GCNs have also been applied for semi-supervised graph classification Li et al . ( 2019 ) . The reader is referred to a comprehensive literature review Bronstein et al . ( 2017 ) and extensive surveys Hamilton et al . ( 2017b ) ; Battaglia et al . ( 2018 ) ; Zhang et al . ( 2018 ) ; Wu et al . ( 2019b ) ; Sun et al . ( 2018 ) ; Zhou et al . ( 2018 ) on this topic . Recently , graph-based deep models ( also messagepassing neural networks Gilmer et al . ( 2017 ) ) have been analysed theoretically Dehmamy et al . ( 2019 ) ; Ying et al . ( 2019 ) ; Knyazev et al . ( 2019 ) ; Chen et al . ( 2019b ) ; Maron et al . ( 2019 ) ; Morris et al . ( 2019 ) ; Xu et al . ( 2019b ) ; Chen et al . ( 2019a ) ; Kawamoto ( 2018 ) ; Xu et al . ( 2018 ) ; Chen et al . ( 2018 ) . We note a work on stability and generalisation bounds of GCNs Verma & Zhang ( 2019 ) . Learning on hypergraphs : Hypergraph is a combinatorial structure consisting of vertices and hyperedges , where each hyperedge is allowed to connect any number of vertices , thus generalizing graphs . This additional flexibility facilitates the capture of higher order interactions among objects ; applications have been found in many fields such as computer vision Govindu ( 2005 ) , network clustering Demir et al . ( 2008 ) , folksonomies Ghoshal et al . ( 2009 ) , cellular networks Klamt et al . ( 2009 ) , and community detection Chien et al . ( 2018 ) . The seminal work on hypergraphs Zhou et al . ( 2007 ) introduced the popular Agarwal et al . ( 2006 ) ; Feng et al . ( 2018 ; 2019 ) clique expansion of a hypergraph . Hypergraph neural networks ( HGNN ) Feng et al . ( 2019 ) use the clique expansion while HyperGCN Yadati et al . ( 2019 ) uses the mediator- based Laplacian to extend GCNs to hypergraphs . Another line of work uses the mathematically appealing tensor methods Shashua et al . ( 2006 ) ; Bulò & Pelillo ( 2009 ) ; Kolda & Bader ( 2009 ) but they are limited to uniform hypergraphs . Recent developments work for arbitrary hypergraphs and fully exploit the hypergraph structure Hein et al . ( 2013 ) ; Zhang et al . ( 2017 ) ; Chan & Liang ( 2018 ) ; Wendler et al . ( 2019 ) . These developments are motivated from the spectral theory of hypergraphs which is an active area of research Li & Milenkovic ( 2017 ) ; Chien et al . ( 2019 ) ; Chitra & Raphael ( 2019 ) ; Li & Milenkovic ( 2018b ) ; Li et al . ( 2018a ) ; Li & Milenkovic ( 2018a ) Graph-based soft SSL : Researchers have shown that using unlabelled data during training can improve label prediction significantly Chapelle et al . ( 2010 ) ; Zhu et al . ( 2009 ) ; Subramanya & Talukdar ( 2014 ) ; Yang et al . ( 2016 ) . While most methods assume that labels of interest are numerical or categorical variables , other works “ soften ” this assumption and handle “ soft labels ” such as histograms Corduneanu & Jaakkola ( 2005 ) ; Tsuda ( 2005 ) . One way of propagating histograms is to minimise the Kullback-Leibler ( KL ) divergence Subramanya & Bilmes ( 2011 ) . Recent studies have replaced the metric-agnostic KL divergence with metric-aware Wasserstein distance ( interactions between histogram bins ) for graphs Solomon et al . ( 2014 ) and hypergraphs Gao et al . ( 2019 ) . Embeddings in Wasserstein space : There exist at least a couple of recent works that embed Gaussian distributions in the Wasserstein space Muzellec & Cuturi ( 2018 ) ; Zhu et al . ( 2018 ) . Inspired by a recent work Frogner et al . ( 2019 ) , in this work , we focus on embedding input data as a discrete probability distrirbution on a fixed support set . The Wasserstein distance and its gradient require the solution of a linear program Villani ( 2008 ) and are costly to compute Peyré & Cuturi ( 2019 ) . A popular efficient approximation is the Sinkhorn divergence Cuturi ( 2013 ) in which the underlying problem is regularised and is computed efficiently by a fixed-point iteration . Recent works have shown that it is suitable for gradient-based optimisation through automatic differentiation Frogner et al . ( 2015 ) ; Genevay et al . ( 2018 ) ; Frogner et al . ( 2019 ) . A couple of recent works compute Wasserstein distance between graph pairs Xu et al . ( 2019a ) ; Vayer et al . ( 2019 ) . 3 METHOD . In this section , we first describe soft SSL on directed hypergraphs and then propose DHN ( Directed Hypergraph Network ) for the problem . 3.1 DIRECTED HYPERGRAPH . A directed hypergraph Gallo et al . ( 1993 ) is an ordered pairH = ( V , Ed ) where V = { v1 , · · · , vn } is a set of n vertices and Ed = { ( t1 , h1 ) , · · · , ( tm , hm ) } ⊆ 2V × 2V is a set of m directed hyperedges . Each element in Ed is an ordered pair ( t , h ) where t ⊆ V is the tail and h ⊆ V is the head with t 6= ∅ , h 6= ∅ . Denote the set of all undirected hyperedges by E. E = ⋃ ( t , h ) ∈Ed ( t ∪ h ) . Denote I ∈ { 0 , 1 } |V |×|E| to be the incidence matrix of E i.e . I ( v , e ) = 1 if v ∈ e and 0 otherwise . 3.2 SOFT SSL ON DIRECTED HYPERGRAPHS . We consider the problem of predicting probability distributions for the vertices inH = ( V , Ed ) given a typically small subset Vk ⊆ V of vertices with known distributions . In this work , we are concerned with discrete distributions modelled on a metric space i.e . an ordered pair ( M , C ) in which M is a set and C is the cost function ( metric ) associated with the set . Furthermore , we assume that we are provided with a feature matrix , XV ∈ Rn×DV , in which each vertex v ∈ V is represented by a DV -dimensional feature vector xv ( here n = |V | ) . We are also provided with a hyperedge feature matrix XE ∈ Rm×DE with xe , e ∈ E as DE-dimensional feature representations ( here m = |Ed| ) . Our objective is to learn a labelling function Z = h ( H , XV , XE ) that maps each vertex to a probability distribution in the space of discrete probability distributions PF ( M ) on F atoms ( F is number of histogram bins ) defined on the metric space ( M , C ) . The cost function C can be represented by a non-negative symmetric matrix of size F × F . Note that each row of Z ∈ [ 0 , 1 ] n×F maps each vertex v ∈ V to a probability distribution Zv ∈ [ 0 , 1 ] F . The function h is going to be trained on a supervised loss , L , w.r.t to the vertices in Vk so that the trained h can be used to predict distributions of all the vertices in V \ Vk . We now give an example application and then the details of the labelling function h followed by the supervised loss L. Example application : Predicting topic distributions of authors in co-authorship networks can be posed as a soft SSL problem on directed hypergraphs . V represents the set of authors , E the set of all collaborations ( documents ) , Ed the citation relationships among the documents , F the number of possible research interests of authors ( Machine Learning , Theoretical Computer Science , etc . ) , XV and XE any available features on the authors and documents respectively ( e.g . text attributes ) .

Summary: In this paper, the authors propose a soft semi-supervised learning approach on a hypergraph. On the one hand, the vertex labels should be not only numerical or categorical variables but also probability distributions. On the other hand, hypergraphs provide a much more flexible mean to encode the real-world complicated relationship compared to the essential pairwise association. Specifically, the authors carefully obtain the generalization error bounds for a one-layer graph neural network. The experiments support the theory and provide the empirical verification of the proposed method. The appendix offers plenty of details that are helpful for the reader to understand both the theoretical and practical perspectives of this paper. Also, the code looks good to me. The structure of the provided codebase is clear and well documented.