DyHCN: Dynamic Hypergraph Convolutional Networks

1 INTRODUCTION . Graph Convolutional Network ( GCN ) Scarselli et al . ( 2008 ) extends deep neural networks to process graph data , which encodes the relations between nodes via propagating node features over the graph structure . GCN has become a promising solution in a wide spectral of graph analytic tasks , such as relation detection Schlichtkrull et al . ( 2018 ) and recommendation Ying et al . ( 2018 ) . An emergent direction of GCN research is extending the graph covolution operations to hypergraphs , i.e. , hypergraph convolutional networks Zhu et al . ( 2017 ) ; Zhou et al . ( 2007 ) ; Zhang et al . ( 2017 ) ; Feng et al . ( 2019b ) ; Yadati et al . ( 2019 ) , where high-order node relations are represented as hyperedges ( one hyperedge can connect multiple nodes ) . For instance , in a hypergraph of stocks , an financial event relevant to several stocks is represented as a hyperedge . While a surge of attention paid on hypergraph convolutional networks , most of them discard the dynamic property of hypergraphs in real-world applications , e.g. , new hyperedges ( i.e. , events ) emerge in the hypergraph of stocks ( see Fig . 1 ) , where the evolution of the hypergraph is crucial for the analytic tasks ( e.g. , stock price prediction ) . Aiming to bridge the gap , this work explore the central theme of dynamic hypergraph and the corresponding GCN . Formally , a hypergraph with n nodes and m hyperedges is represented as G = ( V , E , A , H , X ) where V and E denote the set of nodes and hyperedges respectively ; A ∈ Rn×m is an incidence matrix with binary value indicating the connectedness of nodes ; H ∈ Rm×c and X ∈ Rn×d are features represent the hyperedges and nodes respectively . In order to account for the evolution , we first extend the concept of static hypergraph to dynamic hypergraph , which has two different formulations when treating the time as continuous value or discrete value . 1 ) Discrete-time formulation . A straightforward solution is to treat a time window with length of T ( e.g. , T days ) as a sequence of time-steps and get a snapshot at each time-step . In this way , a dynamic hypergraph is defined as GD = [ G1 , · · · , Gt , · · · , GT ] T where Gt is a hypergraph dumped at time-step t. 2 ) Continuous formulation . By treating time as a continuous variable , the dynamic hypergraph can be defined as GC = ( G0 , U ) where G0 is the initial status ( a hypergraph ) and U = { ( pt , v t , at ) |t < = T } is a streaming of updates . pt denotes the target variable ( e.g. , a row of X ) changed at time t ; v t denotes the latest value of the target variable , at denotes the action of change , including add , delete , update . It should be noted that both formulations have pros and cons , e.g. , the discrete-time formulation is more friendly to existing analytic techniques on static hypergraph such as HCN while the continuous-time formulation records the accurate time of changes . This work focuses on the discrete-time formulation and makes the first attempt to extend HCN to dynamic hypergraph . A big challenge of capturing spatial-temporal dependency in a dynamic hypergraph is that it is tough to extract the features of those changing nodes or hyperedges in a unified manner for the sake of varied scales of nodes and hyperedges . Besides , how to absorb their dynamic properties is very important for various application tasks . Towards this end , we need to design the proper convolution operations on dynamic hypergraph . There are two challenging toughs : 1 ) at each time step , since there are various relations between hyperedges and nodes , it is important to update the node features by considering various relations in the hyperedges ; 2 ) due to dynamically changes of the node features , modeling the temporal dependency needs to extract the corresponding temporal features . In this work , we propose a framework of Dynamic Hypergraph Convolutional Networks ( DyHCN ) to tackle the challenges , which has two modules : Hypergraph Convolution ( HC ) module and Temporal Evolution ( TE ) module . In a dynamic hypergraph , the set of hyperedges at each time step includes different hyperedge embeddings and each hyperedge contains different numbers of nodes . We exploit three submodules to update an node ’ s embeddings in HC : inner attention , outer attention , and embeddings update . Firstly , inner attention transform node features along with its hyperedge into the node-hyperedge feature ; and then outer attention utilizes attention mechanism to estimate the importance of each hyperedge and output the importance weights ; and then we update the node ’ s embeddings by aggregating node-hyperedge , hyperedge and node features with the weight of each hyperedge . Getting the nodes embeddings , we extract temporal features of nodes ’ embeddings and make a prediction by the TE module . Extensive experimental results on two real-world datasets validate the superior performance of DyHCN over the existing baselines which proves the effectiveness of DyHCN on dynamically hypergraphs . The rest of the paper is organized as follows . Section 2 introduces the preliminary knowledge about GCN and the hypergraph convolutional network . Section 3 explains the proposed DyHCN method . Section 4 introduces related work about GCN on the graph and hyperedge . Applications and experimental results are presented in Section 5 . Finally , we conclude this work in Section 6 . 2 PRELIMINARY . Graph Convolutional Network Given a graph G = ( V , E ) with N nodes vi ∈ V , edges ( vi , vj ) ∈ E , an adjacency matrix A ∈ RN×N and a degree matrix Dii = ∑ j Aij . With the input signal x , Kipf & Welling ( 2016 ) considers spectral convolutions on graphs with a filter gθ = diag ( θ ) in the Fourier domain , gθ ? x = UgθUT x , where U is the matrix of eigenvectors of the normalized graph Laplacian L = IN − D−1/2AD−1/2 = UΛUT , with a diagonal matrix of eigenvalues Λ and the graph Fourier transform UTx . In order to reduce the computation complexity , gθ is approximated with Chebyshev polynomials Tk ( x ) = 2xTk−1 ( x ) −Tk−2 ( x ) Defferrard et al . ( 2016 ) , which can be formulated as : gθ ≈ ∑K k=0 θkTk ( Λ̂ ) , where Λ̂ = 2 λmax Λ − I , λmax denotes the largest eigenvalue of Laplacian matrix , θk denotes the Chebyshev coefficients . Kipf & Welling ( 2016 ) proved that the GCN can be simplified to K=1 and λmax ≈ 2 , which is the state-of-the-art of GCN . Hypergraph Convolutional Network A hypergraph can be formulated as G = ( V , E , W ) , where V is a set of vertes , E is a set of hyperedges and W is a diagonal matrix which denotes the weight of each hyperedge . The adjacency matrix of hypergraph G can be denoted by H ∈ R|V|×|E| . The degree of node is d ( v ) = ∑ e∈E w ( e ) h ( v , e ) and the degree of edge δ ( e ) = ∑ v∈V h ( v , e ) . De and Dv denotes the matrices of edge degrees and node degrees . The spectral convolution of x and filter g can be formulated as g ? x = Φ ( ( ΦT g ) ( ΦT x ) ) = Φg ( Λ ) ΦT x , where denotes the element-wise multiplication and g ( Λ ) is a function of Fourier coefficients Feng et al . ( 2019b ) . As simplified in GCN , the convolution operation can be simplified to g ? x ≈ θD−1/2v HWDe −1HTD−1/2v x . 3 DYNAMIC HYPERGRAPH CONVOLUTIONAL NETWORKS . 3.1 FORMULATION OF DYNAMIC HYPERGRAPH . Dynamic hypergraph can be formulated into two categories : discrete-time and continuous-time dynamic hypergraph . The discrete-time approach views dynamic hypergraph as a collection of static graph snapshots over time , while the continuous counterpart extracts fine-grained temporal information on nodes and hyperedges which characterize the dynamic evolution of hypergraph . Discrete-time Dynamic Hypergraph Discrete-time dynamic hypergraph can be formulated as GD = ( Vt , Et , At , Ht , Xt ) , where Xt = [ xt1 , xt2 , · · · , xtn ] T ∈ Rn×d , Ht = [ ht1 , ht2 , · · · , htm ] T ∈ Rm×c , xti ( i = 1 , 2 , · · · , n ) denotes the feature of the i -th node and htj ( j = 1 , 2 , · · · , m ) denotes the feature of the j-th hyperedge , and m , n is the number of hyperedges and nodes on hypergraph Gt ( hypergraph on time step t ) . At ∈ Rn×m is an incidence matrix with binary value indicating the connectedness of nodes on hypergraph Gt . Vt is the set of nodes , Et is the set of hyperedges . C te = [ u t 1 , u t 2 , · · · , u tkte ] T ∈ Rkte×d and D tu = [ et1 , et2 , · · · , etktu ] T ∈ Rktu×c are used to denote the node set contained in hyperedge e and the hyperedge set containing the node u at time setp t respectively . Note that k te and k t u are the number of nodes in hyperedge e and the number of hyperedges containing node u on time t , respectively . As the representation evolve over time , we capture the spatial dependency by hypergraph convolutional networks and use CNNs to model the temporal dependency . Continuous-time Dynamic Hypergraph Continuous-time dynamic hypergraph can be defined as GC = ( G0 , U ) where G0 is the initial status ( a hypergraph ) and U = { ( pt , v t , at ) |t < = T } is a streaming of updates . pt denotes the target variable ( e.g. , a row of X ) changed at time t ; v t denotes the latest value of the target variable , at denotes the action of change , including add , delete , update . Due to a static hypergraph model can be extended to dynamic hypergraphs by applying it on each snapshots and then aggregating the results of the model , and the distinction between an evolving and a temporal network is less important Skarding et al . ( 2020 ) , we adapt discrete-time dynamic hypergraph to build the DyHCN model in our work . DyHCN DyHCN is composed of two modules : hypergraph convolution ( HC ) and temporal evolution ( TE ) . The HC module is designed to aggregate features among nodes and hyperedges with attention mechanisms and update the embeddings of centroid nodes . The TE module is used for capturing dynamic changes in temporal features . The framework of DyHCN is illustrated in Fig.2 , 3.2 HYPERGRAPH CONVOLUTION . Hypergraph convolution consists of three submodules : inner attention , outer attention , and embeddings update . In particular , inner attention aggregates nodes ’ features to hyperedge , outer attention uses attention mechanisms to determine the importance of each hyperedge , and embeddings update submodule aggregates node-hyperedge features , hyperedge features and the node features to update centroid node embeddings with the weight of each hyperedge . Inner attention The inner attention is shown on the left plane of Fig . 3 which aggregates node embeddings to node-hyperedge features by using a self-attention mechanism . With a multi-layer perceptron ( MLP ) we can get the weight score of each node . For a specific node xti on time step t , the input of inner attention is C te = [ u t 1 , u t 2 , · · · , u tkte ] T ∈ Rkte×d and the output of node-hyperedge embedding d t is the weighted sum of node features , which is formulated as : ωt = softmax ( C tewe + be ) , ( 1 ) dt = kte∑ j=0 ωtju t j , ( 2 ) where we ∈ Rd×1 and be ∈ R k t e×1 are trainable parameters , ωt ∈ R kte×1 is the weight of nodes in hyperedge , dt ∈ R1×d denotes the node-hyperedge features , and k te denotes the number of nodes in hyperedge , d is node feature dimension . Outer attention Due to multiple hyperedges related to center node , and the importance of each hyperedge is different , we propose an outer attention submodule to determine the weight of each hyperedge . The right plane of Fig . 3 shows the outer attention submodule which calculates the weight of each hyperedge based on hyperedge features . For specific node xti , the input of outer attention is D tu = [ e t 1 , e t 2 , · · · , etktu ] T ∈ Rktu×c , a hyperedge set containing vertex xti , and the output is ωth , the weight of each hyperedge on time step t. rtu = sigmoid ( D t uwu + bu ) , ( 3 ) ωth = softmax ( r t u ) , ( 4 ) where wu ∈ Rc×1 , bu ∈ Rk t u ×1 are trainable parameters and ωth ∈ Rk t u ×1 is the weight vector of each hyperedge , k tu is the number of hyperedges containing vertex u at time step t , and c is the hyperedge feature dimension . Embeddings Update With the output of inner attention and out attention , we update the centroid node embeddings sti by aggregating node ’ s input features x t i , node-hyperedge features dt and hyperedge features hti with the weight of hyperedges ω t h. We explore three aggregation methods , 1 ) Concatenated features We concatenate the node-hyperedge features and hyperedge features directly with the activation funciton of tanh , qt = tanh [ dt : hti ] ∈ R1× ( d+c ) . 2 ) Dot-product features We multiply the node-hyperedge features with hyperedge features with the element-wise operation to model the interaction of two kinds features , tanh , qt = tanh [ dt hti ] ∈ R1×d ( by setting d=c ) , where denotes element-wise product operation . 3 ) MLP features We concatenate the node-hyperedge features with hyperedge features with an MLP process to aggregate the features , qt = tanh ( [ dt : hti ] Wc + bc ) ∈ R1×d , where Wc ∈ R ( d+c ) ×d , bc ∈ R1×d are trainable parameters . Note that , htc only stands for the concatenated features for one hyperedge , so for k t u hyperedges , we can get a concatenated features matrix Qti = [ qt0 , qt1 , · · · , qtktu ] T which denotes the influence from nodes and each hyperedge . Considering the weight of each hyperedge ωth , we first calculate the weighted sum of concatenated features Qti to measure the influence from all hyperedges and related nodes . And then update the specific node embedding sti with the input feature x t i and the influence embeddings . zti = sum ( ω t h ·Q t i ) , ( 5 ) sti = tanh ( [ x t i : z t i ] Wh + bh ) , ( 6 ) where zti ∈ R1×d is the weighted aggregated features , Wh ∈ R2d×d and bh ∈ R1×d are trainable parameters .

Summary: The paper extends over hypergraph convolutional networks (HCN) by adding a temporal evolution module in order to solve prediction tasks in a dynamic environment. The main part of the paper is the description of the proposed system. It is composed of a HCN for computing node embeddings at each time step and a LSTM as the temporal module. Experimental results are provided for dynamic prediction tasks over stock datasets.