HyperSAGE: Generalizing Inductive Representation Learning on Hypergraphs

1 INTRODUCTION . Graphs are considered the most prevalent structures for discovering useful information within a network , especially because of their capability to combine object-level information with the underlying inter-object relations ( Wu et al. , 2020 ) . However , most structures encountered in practical applications form groups and relations that can not be properly represented using pairwise connections alone , hence a graph may fail to capture the collective flow of information across objects . In addition , the underlying data structure might be evolving and only partially observed . Such dynamic higher-order relations occur in various domains , such as social networks ( Tan et al. , 2011 ) , computational chemistry ( Gu et al. , 2020 ) , neuroscience ( Gu et al. , 2017 ) and visual arts ( Arya et al. , 2019 ) , among others . These relations can be readily represented with hypergraphs , where an edge can connect an arbitrary number of vertices as opposed to just two vertices in graphs . Hypergraphs thus provide a more flexible and natural framework to represent such multi-way relations ( Wolf et al. , 2016 ) , however , this requires a representation learning technique that exploits the full expressive power of hypergraphs and can generalize on unseen nodes from a partially observed hypergraph . Recent work in the field of geometric deep learning have presented formulations on graph structured data for the tasks of node classification ( Kipf & Welling , 2016 ) , link prediction ( Zhang & Chen , 2018 ) , or the classification of graphs ( Zhang et al. , 2018b ) . Subsequently , for data containing higher-order relations , a few recent papers have presented hypergraph-based learning approaches on similar tasks ( Yadati et al. , 2019 ; Feng et al. , 2019 ) . A common implicit premise in these papers is that a hypergraph can be viewed as a specific type of regular graph . Therefore , reduction of hypergraph learning problem to that of a graph should suffice . Strategies to reduce a hypergraph to a graph include transforming the hyperedges into multiple edges using clique expansion ( Feng et al. , 2019 ; Jiang et al. , 2019 ; Zhang et al. , 2018a ) , converting to a heterogeneous graph using star expansion ( Agarwal et al. , 2006 ) , and replacing every hyperedge with an edge created using a certain predefined metric ( Yadati et al. , 2019 ) . Yet these methods are based on the wrong premise , motivated chiefly by a larger availability of graph-based approaches . By reducing a hypergraph to regular graph , these approaches make existing graph learning algorithms applicable to hypergraphs . However , hypergraphs are not a special case of regular graphs . The opposite is true , regular graphs are simply a specific type of hypergraph ( Berge & Minieka , 1976 ) . Therefore , reducing the hypergraph problem to that of a graph can not fully utilize the information available in hypergraph . Two schematic examples outlining this issue are shown in Fig.1 . To address tasks based on complex structured data , a hypergraph-based formulation is needed that complies with the properties of a hypergraph . A major limitation of the existing hypergraph learning frameworks is their inherently transductive nature . This implies that these methods can only predict characteristics of nodes that were present in the hypergraph at training time , and fail to infer on previously unseen nodes . The transductive nature of existing hypegraph approaches makes them inapplicable in , for example , finding the most promising target audience for a marketing campaign or making movie recommendations with new movies appearing all the time . An inductive solution would pave the way to solve such problems using hypergraphs . The inductive learning framework must be able to identify both the node ’ s local role in the hypergraph , as well as its global position ( Hamilton et al. , 2017 ) . This is important for generalizing the learned node embeddings that the algorithm has optimized on to a newly observed hypergraph comprising previously unseen nodes , thus , making inductive learning a far more complex problem compared to the transductive learning methods . In this paper , we address the above mentioned limitations of the existing hypergraph learning methods . We propose a simple yet effective inductive learning framework for hypergraphs that is readily applicable to graphs as well . Our approach relies on neural message passing techniques due to which it can be used on hypergraphs of any degree of cardinality without the need for reduction to graphs . The points below highlight the contributions of this paper : • We address the challenging problem of representation learning on hypergraphs by proposing HyperSAGE , comprising a message passing scheme which is capable of jointly capturing the intra-relations ( within a hyperedge ) as well as inter-relations ( across hyperedges ) . • The proposed hypergraph learning framework is inductive , i.e . it can perform predictions on previously unseen nodes , and can thus be used to model evolving hypergraphs . • HyperSAGE facilitates neighborhood sampling and provides the flexibility in choosing different ways to aggregate information from the neighborhood . • HyperSAGE is more stable than state-of-the-art methods , thus provides more accurate results on node classification tasks on hypergraphs with reduced variance in the output . 2 RELATED WORK . Learning node representations using graph neural networks has been a popular research topic in the field of geometric deep learning ( Bronstein et al. , 2017 ) . Graph neural networks can be broadly classified into spatial ( message passing ) and spectral networks . We focus on a family of spatial message passing graph neural networks that take a graph with some labeled nodes as input and learn embeddings for each node by aggregating information from its neighbors ( Xu et al. , 2019 ) . Message passing operations in a graph simply propagate information along the edge connecting two nodes . Many variants of such message passing neural networks have been proposed , with some popular ones including Gori et al . ( 2005 ) ; Li et al . ( 2015 ) ; Kipf & Welling ( 2016 ) ; Gilmer et al . ( 2017 ) ; Hamilton et al . ( 2017 ) . Zhou et al . ( 2007 ) introduced learning on hypergraphs to model high-order relations for semisupervised classification and clustering of nodes . Emulating a graph-based message passing framework for hypergraphs is not straightforward since a hyperedge involves more than two nodes which makes the interactions inside each hyperedge more complex . Representing a hypergraph with a matrix makes it rigid in describing the structures of higher order relations ( Li et al. , 2013 ) . On the other hand , formulating message passing on a higher dimensional representation of hypergraph using tensors makes it computationally expensive and restricts it to only small datasets ( Zhang et al. , 2019 ) . Several tensor based methods do perform learning on hypergraphs ( Shashua et al. , 2006 ; Arya et al. , 2019 ) , however they are limited to uniform hypergraphs only . To resolve the above issues , Feng et al . ( 2019 ) and Bai et al . ( 2020 ) reduce a hypergraph to graph using clique expansion and perform graph convolutions on them . These approaches can not utilize complete structural information in the hypergraph and lead to unreliable learning performance for e.g . classification , clustering and active learning ( Li & Milenkovic , 2017 ; Chien et al. , 2019 ) . Another approach by Yadati et al . ( 2019 ) , named HyperGCN , replaces a hyperedge with pair-wise weighted edges between vertices ( called mediators ) . With the use of mediators , HyperGCN can be interpreted as an improved approach of clique expansion , and to the best of our knowledge , is also the state-of-the-art method for hypergraph representation learning . However , for many cases such as Fano plane where each hyperedge contains at most three nodes , HyperGCN becomes equivalent to the clique expansion ( Dong et al. , 2020 ) . In spectral theory of hypergraphs , methods have been proposed that fully exploit the hypergraph structure using non-linear Laplacian operators ( Chan et al. , 2018 ; Hein et al. , 2013 ) . In this work , we focus on message passing frameworks . Drawing inspiration from GraphSAGE ( Hamilton et al. , 2017 ) , we propose to eliminate matrix ( or tensor ) based formulations in our neural message passing frameworks , which not only facilitates utilization of all the available information in a hypergraph , but also makes the entire framework inductive in nature . 3 PROPOSED MODEL : HYPERSAGE . The core concept behind our approach is to aggregate feature information from the neighborhood of a node spanning across multiple hyperedges , where the edges can have varying cardinality . Below , we first define some preliminary terms , and then describe our generic aggregation framework . This framework performs message passing at two-levels for a hypergraph . Further , for any graphstructured data , our framework emulates the one-level aggregation similar to GraphSAGE ( Hamilton et al. , 2017 ) . Our approach inherently allows inductive learning , which makes it also applicable on hypergraphs with unseen nodes . 3.1 PRELIMINARIES . Definition 1 ( Hypergraph ) . A general hypergraph H can be represented as H = ( V , E , X ) , where V = { v1 , v2 , ... , vN } denotes a set of N nodes ( vertices ) and E = { e1 , e2 , ... , eK } denotes a set of hyperedges , with each hyperedge comprising a non-empty subset from V. X ∈ RN×d denote the feature matrix , such that xi ∈ X is the feature vector characterizing node vi ∈ V. The maximum cardinality of the hyperedges in H is denoted as M = max e∈E |e| . Unlike in a graph , the hyperedges of H can contain different number of nodes and M denotes the largest number . From the definition above , we see that graphs are a special case of hypergraphs with M=2 . Thus , compared to graphs , hypergraphs are designed to model higher-order relations between nodes . Further , we define three types of neighborhoods in a hypergraph : Definition 2 ( Intra-edge neighborhood ) . The intra-edge neighborhood of a node vi ∈ V for any hyperedge e ∈ E is defined as the set of nodes vj belonging to e and is denoted by N ( vi , e ) Further , let E ( vi ) = { e ∈ E | vi ∈ e } be the sets of hyperedges that contain node vi . Definition 3 ( Inter-edge neighborhood ) . The inter-edge neighborhood of a node vi ∈ V also referred as its global neighborhood , is defined as the neighborhood of vi spanning across the set of hyperedges E ( vi ) and is represented by N ( vi ) = ⋃ e∈E ( vi ) N ( vi , e ) . Definition 4 ( Condensed neighborhood ) . The condensed neighborhood of any node vi ∈ V is a sampled set of α ≤ |e| nodes from a hyperedge e ∈ E ( vi ) denoted by N ( vi , e ; α ) ⊂ N ( vi , e ) . 3.2 GENERALIZED MESSAGE PASSING FRAMEWORK . We propose to interpret the propagation of information in a given hypergraph as a two-level aggregation problem , where the neighborhood of any node is divided into intra-edge neighbors and inter-edge neighbors . For message aggregation , we define aggregation function F ( · ) as a permutation invariant set function on a hypergraph H = ( V , E , X ) that takes as input a countable unordered message set and outputs a reduced or aggregated message . Further , for two-level aggregation , let F1 ( · ) and F2 ( · ) denote the intra-edge and inter-edge aggregation functions , respectively . Schematic representation of the two aggregation functions is provided in Fig.2 . Similar to X we also define Z as the encoded feature matrix built using the outputs zi of aggregation functions . Message passing at node vi for aggregation of information at the lth layer can then be stated as x ( e ) i , l ← F1 ( { xj , l−1 | vj ∈ N ( vi , e ; α ) } ) , ( 1 ) xi , l ← xi , l−1 + F2 ( { x ( e ) i , l |vi ∈ E ( vi ) } ) , ( 2 ) where , x ( e ) i , l refers to the aggregated feature set at vi obtained with intra-edge aggregation for edge e. The combined two-level message passing is achieved using nested aggregation function F = F2 . To ensure that the expressive power of a hypergraph is preserved or at least the loss is minimized , the choice of aggregation function should comply with certain properties . Firstly , the aggregation function should be able to capture the features of neighborhood vertices in a manner that is invariant to the permutation of the nodes and hyperedges . Many graph representation learning methods use permutation invariant aggregation functions , such as mean , sum and max functions ( Xu et al. , 2019 ) . These aggregations have proven to be successful for node classification problems . For the existing hypergraph frameworks , reduction to simple graphs along with a matrix-based message passing framework limits the possibilities of using different types of feature aggregation functions , and hence curtails the potential to explore unique node representations . Secondly , the aggregation function should also preserve the global neighborhood invariance at the ‘ dominant nodes ’ of the graph . Here , dominant nodes refer to nodes that contain important features , thereby , impacting the learning process relatively more than their neighbors . The aggregation function should ideally be insensitive to the input , whether the provided hypergraph contains a few large hyperedges , or a larger number of smaller ones obtained from splitting them . Generally , a hyperedge would be split in a manner that the dominant nodes are shared across the resulting hyperedges . In such cases , global neighborhood invariance would imply that the aggregated output at these nodes before and after the splitting of any associated hyperedge stays the same . Otherwise , the learned representation of a node will change significantly with each hyperedge split . Based on these considerations , we define the following properties for a generic message aggregation function that should hold for accurate propagation of information through the hypergraphs . Property 1 ( Hypergraph Isomorphic Equivariance ) . A message aggregation function F ( · ) is equivariant to hypergraph isomorphism , if for two isomorphic hypergraphs H = ( V , E , X ) and H∗ = ( V∗ , E∗ , X∗ ) , given that H∗ = σ •H , and Z and Z∗ represent the encoded feature matrices obtained using F ( · ) on H and H∗ , the condition Z∗ = σ • Z holds . Here , σ denotes a permutation operator on hypergraphs . Property 2 ( Global Neighborhood Invariance ) . A message aggregation scheme F ( · ) satisfies global neighborhood invariance at any node vi ∈ V for a given hypergraph H = ( V , E , X ) if for any operation Γ ( · ) , such that H∗ = Γ ( H ) , and zi and z∗i denote the encoded feature vectors obtained using F ( · ) at node vi on H and H∗ , the condition z∗i = zi holds . Here Γ ( H ) could refer to operations such as hyperedge contraction or expansion . The flexibility of our message passing framework allows us to go beyond the simple aggregation functions on hypergraphs without violating Property 1 . We introduce a series of power mean functions as aggregators , which have recently been shown to generalize well on graphs ( Li et al. , 2020 ) . We perform message aggregation in hypergraphs using these generalized means , denoted byMp and provide in section 4.2 , a study on their performances . We also show that with appropriate combinations of the intra-edge and inter-edge aggregations Property 2 is also satisfied . This property ensures that the representation of a node after message passing is invariant to the cardinality of the hyperedge , i.e. , the aggregation scheme should not be sensitive to hyperedge contraction or expansion , as long as the global neighborhood of a node remains the same in the hypergraph . Aggregation Functions . One major advantage of our strategy is that the message passing module is decoupled from the choice of the aggregation itself . This allows our approach to be used with a broad set of aggregation functions . We discuss below a few such possible choices . Generalized means . Also referred to as power means , this class of functions are very commonly used for getting an aggregated measure over a given set of samples . Mathematically , generalized means can be expressed as Mp = ( 1 n ∑n i=1 x p i ) 1 p , where n refers to the number of samples in the aggregation , and p denotes its power . The choice of p allows providing different interpretations to the aggregation function . For example , p = 1 denotes arithmetic mean aggregation , p = 2 refers to mean squared estimate and a large value of p corresponds to max pooling from the group . Similarly , Mp can be used for geometric and harmonic means with p→ 0 and p = −1 , respectively . Similar to the recent work of Li et al . ( 2020 ) , we use generalized means for intra-edge as well as inter-edge aggregation . The two functions F1 ( · ) and F2 ( · ) for aggregation at node vi is defined as F ( i ) 1 ( s ) = 1|N ( vi , e ) ||N ( vi ) | ∑ vj∈N ( vi , e ) |E ( vi ) |∑ m=1 1 |N ( vi , em ) | −1 xpj 1 p ( 3 ) F ( i ) 2 ( s ) = 1 |E ( vi ) | ∑ e∈E ( vi ) ( F1 ( s ) ) p 1p ( 4 ) where we use ‘ s ’ for concise representation of the unordered set of input as shown in Eq.1 . Here and henceforth in this paper , we remove the superscript index ‘ ( i ) ’ for the sake of clarity and further occurrences of the two aggregation functions shall be interpreted in terms of node vi . Note that in Eq . 3 and Eq . 4 , we have chosen the power term p to be same for F1 and F2 so as to satisfy the global neighborhood invariance as stated in Property 2 . Note , the scaling term added to F1 is added to balance the bias in the weighting introduced in intra-edge aggregation due to varying cardinality across the hyperedges . These restrictions ensure that the joint aggregation F2 ( · ) satisfies the property of global neighborhood invariance at all times . Proof of the two aggregations satisfying Property 2 is stated in Appendix B. Sampling-based Aggregation . Our neural message passing scheme provides the flexibility to adapt the message aggregation module to fit the desired computational budget through aggregating information from only a subset N ( vi , e ; α ) of the full neighborhood N ( vi , e ) , if needed . We propose to apply sub-sampling only on the nodes from the training set , and use information from the full neighborhood for the test set . The advantages of this are twofold . First , reduced number of samples per aggregation at training time reduces the relative computational burden . Second , similar to dropout ( Srivastava et al. , 2014 ) , it serves to add regularization to the optimization process . Using the full neighborhood on test data avoids randomness in the test predictions , and generates consistent output .

Summary: In this paper, the authors study the problem of learning node embeddings for hypergraphs. While most of the existing studies consider reducing hyper-graphs into graphs, this paper studies learning embeddings directly on the hypergraphs using two stages of aggregations / sampling. The efficacy of the proposed method is illustrated in a semi-supervised as well as inductive setting, where the method achieves better performances than the baselines.