DyHCN: Dynamic Hypergraph Convolutional Networks

1 INTRODUCTION . Graph Convolutional Network ( GCN ) Scarselli et al . ( 2008 ) extends deep neural networks to process graph data , which encodes the relations between nodes via propagating node features over the graph structure . GCN has become a promising solution in a wide spectral of graph analytic tasks , such as relation detection Schlichtkrull et al . ( 2018 ) and recommendation Ying et al . ( 2018 ) . An emergent direction of GCN research is extending the graph covolution operations to hypergraphs , i.e. , hypergraph convolutional networks Zhu et al . ( 2017 ) ; Zhou et al . ( 2007 ) ; Zhang et al . ( 2017 ) ; Feng et al . ( 2019b ) ; Yadati et al . ( 2019 ) , where high-order node relations are represented as hyperedges ( one hyperedge can connect multiple nodes ) . For instance , in a hypergraph of stocks , an financial event relevant to several stocks is represented as a hyperedge . While a surge of attention paid on hypergraph convolutional networks , most of them discard the dynamic property of hypergraphs in real-world applications , e.g. , new hyperedges ( i.e. , events ) emerge in the hypergraph of stocks ( see Fig . 1 ) , where the evolution of the hypergraph is crucial for the analytic tasks ( e.g. , stock price prediction ) . Aiming to bridge the gap , this work explore the central theme of dynamic hypergraph and the corresponding GCN . Formally , a hypergraph with n nodes and m hyperedges is represented as G = ( V , E , A , H , X ) where V and E denote the set of nodes and hyperedges respectively ; A ∈ Rn×m is an incidence matrix with binary value indicating the connectedness of nodes ; H ∈ Rm×c and X ∈ Rn×d are features represent the hyperedges and nodes respectively . In order to account for the evolution , we first extend the concept of static hypergraph to dynamic hypergraph , which has two different formulations when treating the time as continuous value or discrete value . 1 ) Discrete-time formulation . A straightforward solution is to treat a time window with length of T ( e.g. , T days ) as a sequence of time-steps and get a snapshot at each time-step . In this way , a dynamic hypergraph is defined as GD = [ G1 , · · · , Gt , · · · , GT ] T where Gt is a hypergraph dumped at time-step t. 2 ) Continuous formulation . By treating time as a continuous variable , the dynamic hypergraph can be defined as GC = ( G0 , U ) where G0 is the initial status ( a hypergraph ) and U = { ( pt , v t , at ) |t < = T } is a streaming of updates . pt denotes the target variable ( e.g. , a row of X ) changed at time t ; v t denotes the latest value of the target variable , at denotes the action of change , including add , delete , update . It should be noted that both formulations have pros and cons , e.g. , the discrete-time formulation is more friendly to existing analytic techniques on static hypergraph such as HCN while the continuous-time formulation records the accurate time of changes . This work focuses on the discrete-time formulation and makes the first attempt to extend HCN to dynamic hypergraph . A big challenge of capturing spatial-temporal dependency in a dynamic hypergraph is that it is tough to extract the features of those changing nodes or hyperedges in a unified manner for the sake of varied scales of nodes and hyperedges . Besides , how to absorb their dynamic properties is very important for various application tasks . Towards this end , we need to design the proper convolution operations on dynamic hypergraph . There are two challenging toughs : 1 ) at each time step , since there are various relations between hyperedges and nodes , it is important to update the node features by considering various relations in the hyperedges ; 2 ) due to dynamically changes of the node features , modeling the temporal dependency needs to extract the corresponding temporal features . In this work , we propose a framework of Dynamic Hypergraph Convolutional Networks ( DyHCN ) to tackle the challenges , which has two modules : Hypergraph Convolution ( HC ) module and Temporal Evolution ( TE ) module . In a dynamic hypergraph , the set of hyperedges at each time step includes different hyperedge embeddings and each hyperedge contains different numbers of nodes . We exploit three submodules to update an node ’ s embeddings in HC : inner attention , outer attention , and embeddings update . Firstly , inner attention transform node features along with its hyperedge into the node-hyperedge feature ; and then outer attention utilizes attention mechanism to estimate the importance of each hyperedge and output the importance weights ; and then we update the node ’ s embeddings by aggregating node-hyperedge , hyperedge and node features with the weight of each hyperedge . Getting the nodes embeddings , we extract temporal features of nodes ’ embeddings and make a prediction by the TE module . Extensive experimental results on two real-world datasets validate the superior performance of DyHCN over the existing baselines which proves the effectiveness of DyHCN on dynamically hypergraphs . The rest of the paper is organized as follows . Section 2 introduces the preliminary knowledge about GCN and the hypergraph convolutional network . Section 3 explains the proposed DyHCN method . Section 4 introduces related work about GCN on the graph and hyperedge . Applications and experimental results are presented in Section 5 . Finally , we conclude this work in Section 6 . 2 PRELIMINARY . Graph Convolutional Network Given a graph G = ( V , E ) with N nodes vi ∈ V , edges ( vi , vj ) ∈ E , an adjacency matrix A ∈ RN×N and a degree matrix Dii = ∑ j Aij . With the input signal x , Kipf & Welling ( 2016 ) considers spectral convolutions on graphs with a filter gθ = diag ( θ ) in the Fourier domain , gθ ? x = UgθUT x , where U is the matrix of eigenvectors of the normalized graph Laplacian L = IN − D−1/2AD−1/2 = UΛUT , with a diagonal matrix of eigenvalues Λ and the graph Fourier transform UTx . In order to reduce the computation complexity , gθ is approximated with Chebyshev polynomials Tk ( x ) = 2xTk−1 ( x ) −Tk−2 ( x ) Defferrard et al . ( 2016 ) , which can be formulated as : gθ ≈ ∑K k=0 θkTk ( Λ̂ ) , where Λ̂ = 2 λmax Λ − I , λmax denotes the largest eigenvalue of Laplacian matrix , θk denotes the Chebyshev coefficients . Kipf & Welling ( 2016 ) proved that the GCN can be simplified to K=1 and λmax ≈ 2 , which is the state-of-the-art of GCN . Hypergraph Convolutional Network A hypergraph can be formulated as G = ( V , E , W ) , where V is a set of vertes , E is a set of hyperedges and W is a diagonal matrix which denotes the weight of each hyperedge . The adjacency matrix of hypergraph G can be denoted by H ∈ R|V|×|E| . The degree of node is d ( v ) = ∑ e∈E w ( e ) h ( v , e ) and the degree of edge δ ( e ) = ∑ v∈V h ( v , e ) . De and Dv denotes the matrices of edge degrees and node degrees . The spectral convolution of x and filter g can be formulated as g ? x = Φ ( ( ΦT g ) ( ΦT x ) ) = Φg ( Λ ) ΦT x , where denotes the element-wise multiplication and g ( Λ ) is a function of Fourier coefficients Feng et al . ( 2019b ) . As simplified in GCN , the convolution operation can be simplified to g ? x ≈ θD−1/2v HWDe −1HTD−1/2v x . 3 DYNAMIC HYPERGRAPH CONVOLUTIONAL NETWORKS . 3.1 FORMULATION OF DYNAMIC HYPERGRAPH . Dynamic hypergraph can be formulated into two categories : discrete-time and continuous-time dynamic hypergraph . The discrete-time approach views dynamic hypergraph as a collection of static graph snapshots over time , while the continuous counterpart extracts fine-grained temporal information on nodes and hyperedges which characterize the dynamic evolution of hypergraph . Discrete-time Dynamic Hypergraph Discrete-time dynamic hypergraph can be formulated as GD = ( Vt , Et , At , Ht , Xt ) , where Xt = [ xt1 , xt2 , · · · , xtn ] T ∈ Rn×d , Ht = [ ht1 , ht2 , · · · , htm ] T ∈ Rm×c , xti ( i = 1 , 2 , · · · , n ) denotes the feature of the i -th node and htj ( j = 1 , 2 , · · · , m ) denotes the feature of the j-th hyperedge , and m , n is the number of hyperedges and nodes on hypergraph Gt ( hypergraph on time step t ) . At ∈ Rn×m is an incidence matrix with binary value indicating the connectedness of nodes on hypergraph Gt . Vt is the set of nodes , Et is the set of hyperedges . C te = [ u t 1 , u t 2 , · · · , u tkte ] T ∈ Rkte×d and D tu = [ et1 , et2 , · · · , etktu ] T ∈ Rktu×c are used to denote the node set contained in hyperedge e and the hyperedge set containing the node u at time setp t respectively . Note that k te and k t u are the number of nodes in hyperedge e and the number of hyperedges containing node u on time t , respectively . As the representation evolve over time , we capture the spatial dependency by hypergraph convolutional networks and use CNNs to model the temporal dependency . Continuous-time Dynamic Hypergraph Continuous-time dynamic hypergraph can be defined as GC = ( G0 , U ) where G0 is the initial status ( a hypergraph ) and U = { ( pt , v t , at ) |t < = T } is a streaming of updates . pt denotes the target variable ( e.g. , a row of X ) changed at time t ; v t denotes the latest value of the target variable , at denotes the action of change , including add , delete , update . Due to a static hypergraph model can be extended to dynamic hypergraphs by applying it on each snapshots and then aggregating the results of the model , and the distinction between an evolving and a temporal network is less important Skarding et al . ( 2020 ) , we adapt discrete-time dynamic hypergraph to build the DyHCN model in our work . DyHCN DyHCN is composed of two modules : hypergraph convolution ( HC ) and temporal evolution ( TE ) . The HC module is designed to aggregate features among nodes and hyperedges with attention mechanisms and update the embeddings of centroid nodes . The TE module is used for capturing dynamic changes in temporal features . The framework of DyHCN is illustrated in Fig.2 , 3.2 HYPERGRAPH CONVOLUTION . Hypergraph convolution consists of three submodules : inner attention , outer attention , and embeddings update . In particular , inner attention aggregates nodes ’ features to hyperedge , outer attention uses attention mechanisms to determine the importance of each hyperedge , and embeddings update submodule aggregates node-hyperedge features , hyperedge features and the node features to update centroid node embeddings with the weight of each hyperedge . Inner attention The inner attention is shown on the left plane of Fig . 3 which aggregates node embeddings to node-hyperedge features by using a self-attention mechanism . With a multi-layer perceptron ( MLP ) we can get the weight score of each node . For a specific node xti on time step t , the input of inner attention is C te = [ u t 1 , u t 2 , · · · , u tkte ] T ∈ Rkte×d and the output of node-hyperedge embedding d t is the weighted sum of node features , which is formulated as : ωt = softmax ( C tewe + be ) , ( 1 ) dt = kte∑ j=0 ωtju t j , ( 2 ) where we ∈ Rd×1 and be ∈ R k t e×1 are trainable parameters , ωt ∈ R kte×1 is the weight of nodes in hyperedge , dt ∈ R1×d denotes the node-hyperedge features , and k te denotes the number of nodes in hyperedge , d is node feature dimension . Outer attention Due to multiple hyperedges related to center node , and the importance of each hyperedge is different , we propose an outer attention submodule to determine the weight of each hyperedge . The right plane of Fig . 3 shows the outer attention submodule which calculates the weight of each hyperedge based on hyperedge features . For specific node xti , the input of outer attention is D tu = [ e t 1 , e t 2 , · · · , etktu ] T ∈ Rktu×c , a hyperedge set containing vertex xti , and the output is ωth , the weight of each hyperedge on time step t. rtu = sigmoid ( D t uwu + bu ) , ( 3 ) ωth = softmax ( r t u ) , ( 4 ) where wu ∈ Rc×1 , bu ∈ Rk t u ×1 are trainable parameters and ωth ∈ Rk t u ×1 is the weight vector of each hyperedge , k tu is the number of hyperedges containing vertex u at time step t , and c is the hyperedge feature dimension . Embeddings Update With the output of inner attention and out attention , we update the centroid node embeddings sti by aggregating node ’ s input features x t i , node-hyperedge features dt and hyperedge features hti with the weight of hyperedges ω t h. We explore three aggregation methods , 1 ) Concatenated features We concatenate the node-hyperedge features and hyperedge features directly with the activation funciton of tanh , qt = tanh [ dt : hti ] ∈ R1× ( d+c ) . 2 ) Dot-product features We multiply the node-hyperedge features with hyperedge features with the element-wise operation to model the interaction of two kinds features , tanh , qt = tanh [ dt hti ] ∈ R1×d ( by setting d=c ) , where denotes element-wise product operation . 3 ) MLP features We concatenate the node-hyperedge features with hyperedge features with an MLP process to aggregate the features , qt = tanh ( [ dt : hti ] Wc + bc ) ∈ R1×d , where Wc ∈ R ( d+c ) ×d , bc ∈ R1×d are trainable parameters . Note that , htc only stands for the concatenated features for one hyperedge , so for k t u hyperedges , we can get a concatenated features matrix Qti = [ qt0 , qt1 , · · · , qtktu ] T which denotes the influence from nodes and each hyperedge . Considering the weight of each hyperedge ωth , we first calculate the weighted sum of concatenated features Qti to measure the influence from all hyperedges and related nodes . And then update the specific node embedding sti with the input feature x t i and the influence embeddings . zti = sum ( ω t h ·Q t i ) , ( 5 ) sti = tanh ( [ x t i : z t i ] Wh + bh ) , ( 6 ) where zti ∈ R1×d is the weighted aggregated features , Wh ∈ R2d×d and bh ∈ R1×d are trainable parameters .

Summary: This paper proposes a method called DyHCN for learning dynamic hypergraph convolutional networks where the hypergraph structure is allowed to evolve over time. The interactions within each hyper edge, that between nodes, as well as related are used to learn the hypergaph embedding. The evolution of the centroid nodes is then modelled using LSTM. DyHCN gives better modelling accuracy as compared to some existing ones.