DyHCN: Dynamic Hypergraph Convolutional Networks
Authors: Nan Yin
zhigang luo
wenjie wang
Fuli Feng
Xiang Zhang
Venue:  (2021)

Abstract: Hypergraph Convolutional Network (HCN) has become a default choice for capturing high-order relations among nodes, \emph{i.e., } encoding the structure of a hypergraph. However, existing HCN models ignore the dynamic evolution of hypergraphs in the real-world scenarios, \emph{i.e., } nodes and hyperedges in a hypergraph change dynamically over time. To capture the evolution of high-order relations and facilitate relevant analytic tasks, we formulate dynamic hypergraph and devise the Dynamic Hypergraph Convolution Networks (DyHCN). In general, DyHCN consists of a Hypergraph Convolution (HC) to encode the hypergraph structure at a time point and a Temporal Evolution module (TE) to capture the varying of the relations. The HC is delicately designed by equipping inner attention and outer attention, which adaptively aggregate nodes' features to hyperedge and estimate the importance of each hyperedge connected to the centroid node, respectively. Extensive experiments on the Tiigo and Stocktwits datasets show that DyHCN achieves superior performance over existing methods, which implies the effectiveness of capturing the property of dynamic hypergraphs by HC and TE modules.

Under review as a conference paper at ICLR 2021
DYHCN: DYNAMIC HYPERGRAPH CONVOLUTIONAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Hypergraph Convolutional Network (HCN) has become a default choice for cap- turing high-order relations among nodes, i.e. encoding the structure of a hyper- graph. However, existing HCN models ignore the dynamic evolution of hyper- graphs in the real-world scenarios, ie., nodes and hyperedges in a hypergraph change dynamically over time. To capture the evolution of high-order relations and facilitate relevant analytic tasks, we formulate dynamic hypergraph and devise the Dynamic Hypergraph Convolutional Networks (DyHCN). In general, DyHCN consists of a Hypergraph Convolution (HC) to encode the hypergraph structure at a time point and Temporal Evolution module (TE) to capture the varying of the relations. The HC is delicately designed by equipping inner attention and outer at- tention, which adaptively aggregate nodes' features to hyperedge and estimate the importance of each hyperedge connected to the centroid node, respectively. Exten- sive experiments on the Tigo and Stocktwits datasets show that DyHCN achieves superior performance over existing methods, which implies the effectiveness of capturing the property of dynamic hypergraphs by HC and TE modules.
INTRODUCTION
Graph Convolutional Network (GCN) graph data, which encodes the relations between nodes via propagating node features over the graph structure. GCN has become a promising solution in wide spectral of graph analytic tasks, such as relation detection (Schlichtkrull et al. (2018) and recommendation Ying et al. (2018). An emer- gent direction of GCN research is extending the graph covolution operations to hypergraphs, i.e. hypergraph convolutional networks Zhu et al. (2017); Zhou et al. (2007): Zhang et al. (2017): Feng et al. (2019b); Yadati et al. (2019). where high-order node relations are represented as hyperedges (one hyperedge connect multiple nodes). For instance, in hypergraph of stocks, event relevant to several stocks is represented as a hyperedge. While a surge of attention paid on hypergraph convolutional networks, most of them discard the dynamic property of hypergraphs in real-world applications, egg., new hyperedges (i.e., events) emerge in the hypergraph of stocks (see Fig. DD. where the evolution of the hypergraph is crucial for the analytic tasks (e.g., stock price prediction). Aiming to bridge the gap, this work explore the central theme of dynamic hypergraph and the corresponding GCN.
(GCN)Scarselli Scarselli al. (2008) extends deep neural networks to process
financial
Formally, a hypergraph with n nodes and m hyperedges is represented as G EV, E Roxm is an incidence where V and E denote the set of nodes and hyperedges respectively; m and X E Raxd are matrix with binary value indicating the connectedness of nodes; H E features represent the hyperedges and nodes respectively. In order to account for the evolution, we first extend the concept of static hypergraph to dynamic hypergraph, which has two different formu- lations when treating the time as continuous value or discrete value. 1) Discrete-time formulation. A straightforward solution is to treat time window with length of T (e.g., T days) as a sequence of time-steps and get a snapshot at each time-step. In this way, a dynamic hypergraph is defined as gD = =g' ... , ....... TT where G a hypergraph dumped at time-step t. 2) Continuous formulation. By treating time continuous variable, the dynamic hypergraph can be defined as g -g U)) where g" is the initial status (a hypergraph) and U [(p', v', a²) a')]t <= T} is streaming of updates. p² denotes the target variable (e.g., a row of X) changed at time t; 8² de- notes the latest value of the target variable, a' denotes the action of change, including add, delete,

---

Under review as a conference paper at ICLR 2021
16
Suppperr S)3
4.2
Hypergraph
4/
1111
4/
evolution
C5
6
predict
(M))
(.()))
Figure I: The evolution of dynamic hypergraph.
update. It should be noted that both formulations have pros and cons, egg. the discrete- time for- mulation is more friendly to existing analytic techniques on static hypergraph such as HCN while the continuous-time formulation records the accurate time changes. This work focuses on the discrete-time formulation and makes the first attempt to extend HCN to dynamic hypergraph.
big challenge of capturing spatial-temporal dependency in dynamic hypergraph is that it is tough to extract the features of those changing nodes or hyperedges in unified manner for the sake of varied scales of nodes and hyperedges. Besides, how to absorb their dynamic properties is very important for various application tasks. Towards this end, need to design the proper convolution operations on dynamic hypergraph. There are two challenging toughs: 1) at each time step, since there various relations between hyperedges and nodes, it is important to update the node features by considering various relations in the hyperedges; 2) due to dynamically changes of the node features, modeling the temporal dependency needs to extract the corresponding temporal features.
In this work, we propose a framework of Dynamic Hypergraph Convolutional Networks (DyHCN) to tackle the challenges, which has two modules: Hypergraph Convolution (HC) module and Tem- poral Evolution (TE) module. In a dynamic hypergraph, the set of hyperedges at each time step includes different hyperedge embeddings and each hyperedge contains different numbers of nodes. We exploit three submodules to update an node's 's embeddings in HC: inner attention, outer attention, and embeddings update. Firstly, inner attention transform node features along with its hyperedge into the node-hyperedge feature; and then outer attention utilizes attention mechanism to estimate the importance of each hyperedge and output the importance weights; and then update the node's s embeddings by aggregating oyde-hypreddge, hyperedge and node features with the weight of cach hyperedge. Getting the nodes embeddings, make a prediction by the TE module. Extensive experimental results on two real-world datasets val- idate the superior performance of DyHCN over the existing baselines which proves the effectiveness of DyHCN on dynamically hypergraphs.
extract temporal features of nodes' embeddings and
The rest of the paper is organized as follows. Section 2 introduces the preliminary knowledge about GCN and the hypergraph convolutional network. Section explains the proposed DyHCN method. Section 4 introduces related work about GCN on the graph and hyperedge. Applications and experimental results are presented in Section Finally, we conclude this work in Section
2 PRELIMINARY
Graph Convolutional Network Given graph 9 (V, E) with N nodes i, E V. edges (u, 1y) E, an adjacency matrix A N*N and degree matrix D.; Σ, Aij- With the input signal x, Kipf & Welling (0066) considers spectral convolutions on graphs with filter ge diag(!) in the Fourier domain, ge x Ug UT x, where
the matrix of eigenvectors of the normalized graph 1/2 1/D -1/2 UAUT. with a diagonal matrix of eigenvalues and the graph Fourier transform U xr. order to reduce the computation complexity, go approximated with Chebyshev polynomials Tş (x) = 2x:T-11(e (x)) -T (2) Defferrard
Laplacian I. = In D-11/2
Deffferrrrf et al. (2016), which can be
formulated as: go Σk= Tx (e where A
2 A Xmaa
Amar denotes the largest eigenvalue
2

---

Under review as a conference paper at ICLR 2021
of Laplacian matrix, R7 denotes the Chebyshev coefficients, Kipf & Welling (0166) proved that the GCN can be simplified to K=1 and Amaz RJ 2, which is the state-of-the-arr of GCN.
hypergraph can be formulated as G (V, E, W), where V is a set of vertes, & is a set of hyperedges and W is a diagonal matrix which denotes the weight of each hyperedge. The adjacency matrix of hypergraph g can be denoted by H R |I The degree of node is d(u)) E Beeg w(e)h((, e) and the degree of edge 5(e) Eeey h(u,e). b(,yh( D. and D, denotes the matrices of edge degrees and node degrees. The spectral convolution x and filter can be formulated g* "(( g) @ x)) Lg(A)Φ²x, where @ denotes the element-wise multiplication and (A) is function of Fourier coefficients Feng et al. (2019b). As simplified in
Hypergraph Convolutional Network
GCN, the convolution operation can be simplified to ~ 0D, H/W. -,
3 DYNAMIC HYPERGRAPH CONVOLUTIONAL NETWORKS
3.1 FORMULATION OF DYNAMIC HYPERGRAPH
Dynamic hypergraph can be formulated into two categories: discrete-time and continuous-tin time dy- namic hypergraph. The discrete-time approach views dynamic hypergraph as collection of static graph snapshots over time, while the continuous counterpart extracts fine-grained temporal informa- tion on nodes and hyperedges which characterize the dynamic evolution of hypergraph.
Discrete-time Dynamic Hypergraph Discrete-time dynamic hypergraph can be formulated as gd (V* E. A', H², )'),, where X = [af æ/, ... , E gnxd H² = [hj h/ h!''"" E Rmxe, (i 1,2, ni) denotes the feature of the i-th node and h = 1,2, 2.... m)) denotes the feature of the j-th hyperedge, and m, n is the number of hyperedges and nodes on hypergraph g' (hypergraph
time step A' Raxm is an incidence matrix with binary value indicating the connectedness of nodes on hypergraph g". V is the set of nodes, Ef is the set of hyperedges. c |ui, u......, u E R&xdd and D = [[ei ....... e I" E RAxe are used to denote the node set contained in hyperedge and the hyperedge set containing the node at time setp I respectively. Note that le and hy are the number of nodes hyperedge and the number of hyperedges containing node u time t, respectively. As the representation evolve over time, capture the spatial dependency by hypergraph convolutional networks and use CNNs to model the temporal dependency.
Continuous-time Dynamic Hypergraph Continuous dynamic hypergraph can be defined as g = - where g" is the initial status (a hypergraph) and U = [(p', v², a²) d')lt <= T} is a of X) changed at time (; v denotes
streaming of updates. p' denotes the target variable (e.g., the latest value of the target variable, a' denotes the action of change, including add, delete, update. Due to a static hypergraph model can be extended to dynamic hypergraphs by applying on each snapshots and then aggregating the results of the model, and the distinction between an evolving and a temporal network is less important [Skarding al. (020), we adapt discrete-time dynamic hypergraph to build the DyHCN model in our work.
DyHCN DyHCN composed of two modules: hypergraph convolution (HC) and temporal evo- lution (TE). The HC module is designed to aggregate features among nodes and hyperedges with attention mechanisms and update the embeddings of centroid nodes. The TE module used for capturing dynamic changes in temporal features. The framework of DyHCN is illustrated in Fig 2
3.2 HYPERGRAPH CONVOLUTION
Hypergraph convolution consists of three submodules: inner attention, outer attention, and embed- dings update. In particular, inner attention aggregates nodes' features to hyperedge, outer attention uses attention mechanisms to determine the importance of each hyperedge, and embeddings update submodule aggregates mode-hyperedg features, hyperedge features and the node features to update centroid node embeddings with the weight of each hyperedge.
3

---

Under review as a conference paper at ICLR 2021
Hypergrap
Outer Atten Inner Atten
Outer Inner Atten
Outer Atten
Inner Atten
HC
Apenadge Weiight
Mode-odge Fecture
Mpereége Weight
Kode-edge Feature
Medo Fextre
Embedddings update
Embeddings update
Embeddings update
Centrod Sertin Feature
Conto aid Vertox
Centrsid Ventex
Festure
TE
1STM
Figure 2: The framework of DyHCN: HC module consists of inner attention, outer attention and em- beddings update submodules, which aggregates various features to centroid vertex, and TE module extracts temporal features for prediction.
The inner attention is shown the left plane of Fig. which aggregates node embeddings to node-hyperedge features by using a self-attention mechanism. With a multi-layer perceptron (MLP) we can get the weight score of each node. For a specific node xf on time step t, the input of inner attention is C = [u], u u IT E x and the output of hypeeedee embedding d' is the weighted sum of node features, which is formulated as:
Inner attention
w/ softmar Co,wa b.).
d'
u
where we E Rddx and b R ki are trainable parameters, wi E IR M.x1 is the weight of nodes in hyperedge, d' Ixx denotes the hypereyee features, and k, denotes the number of nodes in hyperedge, d is node feature dimension.
Outer attention Due to multiple hyperedges related to center node, and the importance of each hyperedge is different, we propose hyperedge. The right plane of Fig. 3 shows the outer attention submodule which calculates the weight of each hyperedge based on hyperedge features. For specific node x; the input of outer attention is D = [ef, e², ... e E R& a hyperedge set containing vertex s; and the output w/,, the weight of each hyperedge time step
attention submodule determine the weight of each
r sigmoid( D', w., + bw).
(3)
w/f = saftmax(11 t)
(4)
where w.. E Rex1 by E R x1 are trainable parameters and w E TR4 x1 the weight vector of each hyperedge, K is the number of hyperedges containing vertex time step and is the hyperedge feature dimension.
Embeddings Update With the output inner attention and out attention, we update the cen- troid node embeddings sf by aggregating node's input features æ!. node-hyperedge features d' and hyperedge features h; with the weight of hyperedges w/,- We explore three aggrega- tion methods, 1) Concatenated features We concatenate the node-hyperedge features and hyper- edge features directly with the activation funciton of tanh, q' tonh dt h:] IR' (Rx((=e)) 2) Dot-product features We multiply the hode-hy features with hyperedge features with the element- operation model the interaction of two kinds features, lanh, q¹ tanh dd' @ h.]

---

Under review as a conference paper at ICLR 2021
ci
<5
r
rw
a
Figure Inner attention on left and outer attention on the right.
Rixd (by setting d=c), where C denotes dlement-wise product operation. 3) MLP features We con- catenate the node-hype features with hyperedge features with an MLP process aggregate the features, q' ttnn((d' h b²) Rixd where W (d++))d b E RR' !xldd are trainable parameters. Note that, h: only stands for the concatenated features for one hyperedge, so for k hyperedges, we can get a concatenated features matrix Q = [46 q ... .6e 1 which denotes the influence from nodes and each hyperedge.
Considering the weight of each hyperedge wh. first calculate the weighted of concatenated features Q measure the influence from all hyperedges and related nodes. And then update the specific node embedding of with the input feature æf and the influence embeddings.
z sum Q
(5)
$f tanh((x)) (mr] 2 W + ba):
(6)
where zद Rixd is the weighted aggregated features, Wa Rdddd and b, E Rxdd trainable parameters.
3.3 TEMPORAL EVOLUTION
The centroid node embeddings extracted by HC are independent different time steps, will embeddings for each centroid node along with time, i.e., S; = [s , s We adopt temporal evolution module process temporal information extraction. The TE module utilize the LSTM model to extract temporal features which can be used for classification or regression tasks.
O, (S.). 8 (tanh((.)) W. + b.)) + by:
(8)
where O E IR Xxdim the temporal features extracted by LSTM, dim is the hidden dimension of LSTM. W, E R dimnx b, R RxxR W,, E xxl, by E R xl are trainable parameters, k is the hidden dimension size for MIP and the final output size which determined by detail task.
RELATED WORK
GCN regular graphs Existing graph-based learning solutions are divided into directions: spectral methods and spatial methods. Spectral graph convolution transform features to the spectral domain by using graph Laplacian eigenvectors and then conclude node or graph features by spectral convolution. However, the computation cost in Laplacian factorization is expensive, Defferrard et al. (2016) introduced Chebyshev polynomials to approximate Laplacian eigenvectors, and further, Kipf & Welling (2016) simplified the process by localized first-order approximation of spectral graph convolutions. On the other side, spatial methods generate node embeddings by aggregating neigh- borhoods features. GraphSAGE: generates embeddings by sampling and aggregating features from a node's local neighborhood Hamilton et al. (2017). GAT leverage self-attentional mechanism and weighting different nodes in neighborhood to generate node embeddings Veličković et al. (2017). The graph-based learning limits the relationships into pairwise, however, in many applications, the relations between objects are in higher-order that cannot be formulated by graph structure.
GCN on hypergraph To evaluate the higher-ordes relations between nodes, Zhou et al. (2007) introduced the first hypergraph learning, where multiple nodes share the same hyperedge. On the
5

---

Under review as a conference paper at ICLR 2021
Table 1: Details of Tingo and Stockwits datasets
Dataset
Nodes
Avg hyperedges
Training days
Validation days
Testing days
Tiggo
91
691
529
151
76
Stocktwits 91
177.1
712
203
103
direction spectral methods, considering different subsets of nodes in the same hyperedge may have different structural importance, & Milenkovic (2017) proposed inhomogeneous hypergraph partitioning model to assign different costs to different hyperedge cuts. [i Milenkovic (2018)
defined the notion of p p-Laplacians which constitute the basis of new spectral hypergraph clustering methods. Chan et al. (2018) considered stochastic diffusion process and introduces a new hyper-
graph Laplacian operator generalizing the Laplacian matrix of graphs. Yadati et (2019) simplified hypergedges into simple edges with mediators and demonstrate the effectiveness through detailed experiments. The other way, spatial methods, Gilmer et al.) (2017) raised Message Passing Neural Networks (MPNN) framework which learns a message-passing algorithm and aggregate features for node representation. Feng al. (2019b) introduced the first hypergraph deep learning method hy- pergraph neural network (HGNN). However, most of the existing works focus on static hypergraph structure which has little effort on optimizing the hypergraph structure during the learning process. Zhan. et al. (2018) is the first dynamic hypergraph structure learning method that optimizes the label projection matrix and the hypergraph structure itself simultaneously. But DHSL. fails ex- ploit high-order relations among features Jiang et al. (2019). DHGCN Jiang et al. (2019) proposed a stacked layers framework to evaluate the dynamic hypergraph with KNN to build the dynamic hyperedges. However, the input of DHGCN is fixed, which means the relations among nodes fixed and the hypergraph structure just update on each layer. But in the real world, the relations of nodes may be connected temporarily, and existing models cannot process temporary connections or change connections among different nodes.
(Gimmr
EXPERIMENTS
The DyHCN model can be applied to various tasks that can be formulated as dynamic hypergraph. In our work, we adopt DyHCN with news and social comment datasets for stock price prediction.
5.1 EXPERIMENTAL SETTING
Tüingo dataset The dataset covers the news content, stocks contained in the news, and the release time of news. On a specific trading day, there are varity news and each news many contains different numbers of stocks, we construct hypergraph with news as hyperedges and stocks as nodes. We construct the dynamic hypergraph based on crawled news from June 22, 2016 to June 23, 2019, total of 756 trading days with one hypergraph on each trading day. Inspired by [Cen et al.) 2019), we adapt Finance Event Dictionary (TFED) to extract fine-graines events, and pick out the most activate 91 stocks on market for price prediction.
Stockwits dataset The Stocktwits dataset is stock comment dataset which can be crawled from the web of stockwits The dataset covers the stock comment content, stocks mentioned in the com- ment, and the comment time. On a specific trading day, we construct hypergraph with comments as hyperedges and stocks as nodes. We pick out 91 stocks with the highest market value in different industries on S&P 500 for price prediction, and collect the data from Aug. 7, 2014 to Aug, 20, 2018, a total of 1015 trading days with one hypergraph on each trading day. The details of datasets are shown on Table [
With the construction of dynamic hypergraph, we assign the nodes featuress with the hidden em- bedding of price and volume extracted by LSTM, and hyperedges featrues with the embedding represented by SiooeePeennngtoo et al. (2014). The feature dimension of hyperedges and nodes are set to 50. The training, validation, and testing sets separated in Table To measure the result
'htpss/wwww.ting..com hhtps:/stocktwiis.coom

---

Under review as a conference paper at ICLR 2021
Table 2: Performance Tingo and Stocktwits dataset with hidden size 128 of TE
Dataset
Tingo
Stocktwits
Models
MAE
MAPE
MSE
Promote
MAE
MAPE
MSE
Promote
DA-RNN
0.1869
0.4786
0.0467
55.37%
0.1855
1.0950
0.0384
71.55%
HAN
0.1466
0.3017
0.0548
42.43%
0.1272
0.5517
0.0414
61.29%
RSR
0.0946
0.2596
0.0183
7.57%
0.0893
0.1870
0.0158
18. 19%
DHGCN
0.0984
0.2754
0.0216
15.08%
0.0845
0.1782
0.0156
14.86%
DyHCN 0.0873
0.2533
0.0160
0.0732
0.1660
0.0118
of different models for prediction, we use three evaluation metrics, the mean squared error (MSE), mean absolute error (MAE) and mean absolute percentage error (MAPE).
Baselines To evaluate the result of our proposed DyHCN model, we compare the experiment result with the traditional time series, NLP-based, graph-based and hypergraph-basce model: 1) DA -NNN Hsu et al. (2009) One of the state-of-thes thet--rrt models for time series prediction. 2) HAN Hu et al. (2018) The representations of the NLP model for stock price prediction. 3) RSR Feng et al. (2019a) The otate-of-the-art graph-based model for price prediction. 4) DHGCN Jiang et al. (2019) The hypergraph-baasee model for prediction. Because the model RSR and DHGCN are designed for static graph/hypergraph, we present the RSR and DHGCN for daily price prediction. To compare with baseline models, we use DyHCN with stacked layer HC and TE module for price prediction, and add dropout layer with the dropout rate of 0.5 before TE module. We set the learning rate 0.005 and training epoch 1000.
5.2 RESULTS AND ANALYSIS
We report the performance of all methods in Table Table and Fig. From Table we have the following observations:
1) Compared with DA-RNN, the MAE and MAPE scores of HAN decreases by 21 1.66% and 36.96% from O. 1869, 0.4786 to 0.1466, 0.3017, while the MSE increases by 17.34% from 0.0467 to 0.0548, indicating that the extra features such as Tiinoo or Stocktwits comment data are useful for stock price prediction.
2) The MAE, MAPE and MSE results of RSR outperform HAN, and decrease by 35.47%, 13.95% and 66.61% respectively, indicating that the consideration of relations between different stocks would improve the performance of the prediction result.
3) Comparing the graph-based model RSR with hypergraph-basse model DHGCN, there is no sig- nificant difference in the performance of the model. However, in the Tiingo dataset, the results of RSR decrease by 3.86%, 5.74%, 15.28% comparied with DHGCN respectively, while in the Stocktwits comment dataset, the model shows the opposite result, the three metrics of RSR increase by 5.68%, 4.94%, 1.28% comparied with DHGCN. This shows that the performance of RSR and DHGCN models are not stable on different datasets.
4) The results of DyHCN outperform RSR and DHGCN. On Tringo dataset, the MAE decreases by 7.72%. 11.28%. the MAPE decreases by 2.43%, 8.02%, and the MSE decreases 12.57%. 25, 25.93% compared with RSR and DHGCN respectively. On Stocktwits comment dataset, the MAE decreases by 18.03%, 13.37%, the MAPE decreases by 1 11.23% 6.85%, and the MSE decreases by 25.32%, 24.36% compared with RSR and DHGCN respectively. The result shows that in both Tiingo and Stocktwits comment datasets, the performance of DyHCN would keep stable, and with the consider- ation of dynamic information, the performance is better than static graph/hypergraph based model.
5) Comparing DyHCN with DA-RNN, HAN. RSR and DHGCN, the average loss of MAE, MAPE and MSE decrease by 55.37%, 42.43%. 7.57% and 15.08% on Tiingo dataset respectively, and 71.55% 61.29%, 18.19 19%, 14.86% Stocktwits comment dataset respectively.
To test the stability and the scalability of the model, we evaluate different feature aggregation meth- ods. Fig, shows the performance of different feature aggregation methods with hidden size of 16, 32, 64, and 128 on TE module. Comparing the results, the MLP feature concatenate method reminds stable and outperforms the cat and nulti-feature aggregate methods. Besides, with the comparison
7

---

Under review as a conference paper at ICLR 2021
Tiingo
Tilingo
cco
Tiingo
MAE
MSE
300
10
32
120
Hidden size
Hidden size
Hidden size
3200
Stocktwits
Stocktwits
Stocktwits
MAE
MAPE
MSE
Hidden size
Hicden
Hidden
Figure 4: Performance of different features aggregate methods on Tiingo and Stocktwits social comment datasets with the hidden size of 16, 32, 64 and 128 TE.
Table 3; Performance on Tiigoo and Stocktwits dataset with hidden size 128 TE
Dataset
Tingo
Stocktwits
Models
MAE
MAPE
MSE
MAE
MAPE
MSE
DyHCN(no inner)
0.1303
0.3988
0.0347
0.0745
0.1667
0.0122
DyHCN(no outer)
0.0887
0.2613
0.0165
0.0732
0.1695
0.0120
DyHCN(no TE)
0.0982
0.2754
0.0218
0.0842
0.1784
0.0160
HGCN(with TE et al. (2019b)
0.2943
0.8025
0.1771
0.2943
0,8025
0.1771
DyHCN
0.0873
0.2533
0.0160
0.0732
0.1660
0.0118
of results with different hidden size, the performance has no significant difference, indicating that the hidden size of LSTM is not the major factor for model prediction.
In addition to the comparison with baselines above, we also evaluate the effectiveness of submodules including inner attention, outer atteniton, HC and TE. We evaluate the effectiveness of each module which shown in Table 3, we use DyHCN without inner attention, DyHCN without outer attention and DyHCN without TE to evaluate the effectiveness of inner, outer attention and TE module. Also, we use HGCN model Feng et al. (2019b) which aggregate node features on static hypergrap to replace HC module to evaluate the effectiveness of HC module. The result shows that, both datasets, the performance of DyHCN are better than DyHCN without inner, outer and TE module. The inner and outer attention module determines the importance of each message, and passes the corresponding information to the centroid which would be used for prediction more acurrate. The TE module considers the impact of previous information and extracts the temporal features from series embeddings, which would be better than from individual time step. In addition, the performance of HGCN Feng al. (2019b) with TE model is also worse than DyHCN, while HGCN works well on static hypergraph, indicating that DyHCN is more suitble for dynamic hypergraph tasks.
6 CONCLUSION
In this paper, we proposed a framework of dynamic hypergraph convolutional networks (DyHCN), which consists of hypergraph convolution (HC) and temporal evolution (TE) module. The HC is delicately designed by equipping inner attention and outer at-tention, which adaptively aggregate nodes' features to hyperedge and estimate theimportance of each hyperedge connected to the cen- troid node, respectively. And then update the centroid node embeddings by aggregating various related features. The TE captures the long-range temporal information of dynamic hypergraph fea- tures. Based on the two modules, the DyHCN get the dynamic relations between different nodes with the dynamic weight of hyperedges on different time steps. DyHCN can be used for various dynamic hypergraph, and extensive experiments the newly
tasks that can be formulated structed Tiingo and Stocktwits comment datasets show that our proposed DyHCN outperforms the state-of-fe of-the-artttttt model for the modeling of dynamic hyper-order relations.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
T-H Hubert Chan, Anand Louis, Zhihao Gavin Tang, and Chenzi Zhang. Spectral properties of hypergraph laplacian and approximation algorithms. Journal of the ACM (JACM), 65(3):1-48, 2018.
Deli Chen, Yanyan Zou, Keiko Harimoto, Ruihan Bao, Xuancheng Ren, and Xu Sun. Incorporating fine-grained events in stock movement prediction. arXiv preprint -XXi:1910.05072 2019.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. In Advances in neural information processing systems,
pp. 3844 3852, 2016.
Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo. Yiqun Liu, and Tat-Seng Chua. Temporal relational ranking for stock prediction. ACM Transactions Information Systems (TOIS), 37(2):
1-30, 2019a.
Yifan Feng, Haoxuan You, Zizhao Zhang. Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3558-3565, 2019b.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Vole Volume 70, pp. 1263-1272. JMLR. org, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances neural information processing systems, pp. 1024-1034, 2017.
Sheng-Hsee Hsun Hsu, JJ Po-An Hsich, Ting-Chin Chih Chih, and Kuci-Chu Hsu. A two-stage architecture for stock price forecasting by integrating self-organizing map and support vector regression. Expert Systems with Applications, 36(4):7947-795 2009.
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie- Yan Liu. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In Proceedings of the eleventh ACM international conference web search and data mining, pp. 261-269, 2018.
Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, and Yue Gao. Dynamic hypergraph neu- ral networks. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence (ICCAI), pp. 2635-2641, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: (000009990777 2016.
Pan Li and Olgica Milenkovic. Inhomogeneous hypergraph clustering with applications. In Ad- vances in Neural Information Processing Systems, pp. 2308-2318, 2017.
Pan Li and Olgica Milenkovic. Submodular hypergraphs: p-laplacians, cheeger inequalities and spectral clustering, arXiv preprint urXiv:1803.03833 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove:
Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions Neural Networks, 20(1):61-80, 2008
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 593-607. Springer, 2018.
Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modelling of dynamic networks using dynamic graph neural networks: A survey. arXiv preprint MXXi:2005.07499 2020.

---

Under review as a conference paper at ICLR 2021
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1090. 2017.
Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar. Hypergen: A new method for training graph convolutional networks on hypergraphs. In Advances in Neural Information Processing Systems, pp. 1511-1522, 2019.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983, 2018.
Zhihong Zhang. Lu Bai, Yuanheng Liang, and Edwin Hancock. Joint hypergraph learning and sparse regression for feature selection. Pattern Recognition, 63:291-309, 2017. 2017..
Zizhao Zhang, Haojie Lin, Yue Gao, and KLISS BNRist. Dynamic hypergraph structure learning. In IJCAI, pp. 3162-3169, 2018.
Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clustering, classification, and embedding. In Advances in neural information processing systems, pp. 1601 1608, 2007.
Xiaofeng Zhu, Yonghua Zhu, Shichao Zhang, Rongyao Hu, and Wei He. Adaptive hypergraph learning for unsupervised feature selection. In ICAAI, pp. 3581-3587, 2017.
10


Reviews:
[{"review_id": "jtLx_XFoQQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method called DyHCN for learning dynamic hypergraph convolutional networks where the hypergraph structure is allowed to evolve over time. The interactions within each hyper edge, that between nodes, as well as related are used to learn the hypergaph embedding. The evolution of the centroid nodes is then modelled using LSTM. DyHCN gives better modelling accuracy as compared to some existing ones.\n\nPros:\n- The formulation is clearly presented.\n- The inner and outer attention models adopted are reasonable\n- Empirical experiments have been carried out to verify its effectiveness.\n\nCons:\n- Only discrete-time dynamic hypergraph is considered.\n- The attention model and the modelling of the evolution of the centroid nodes are not particularly novel.\n- The performance improvement as compared with the SOTA method is incremental.\n- Only one particular prediction is adopted for the performance evaluation.\n\nSpecific comments\n- In Inner attention section, x_i^t is referred to as a nod but it should be th feature of node I\n\nQn:\n- Other than stock price prediction, what are the other possible prediction tasks? Can the performance comparison be carried out based on the additional predication tasks?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YAOrTmNzKD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Minor: Multiply typos and grammatical errors: (esp. section 5.1 onwards).\nFor example, see here: \"With the construction of dynamic hypergraph, we assign the nodes featuress with the hidden embedding of price and volume extracted by LSTM, and hyperedges featrues with the embedding represented by GloVe Pennington et al. (2014)\"\n\nClarity:  \n(1) Clear description of the problem statement and model overall. Images are well drawn\n(2) Reason for sigmoid in eqn(3) is not clear\n(3) How the initial node/hyperedge features are generated is not very clear . This is important given that the newly constructed Tiingo and Stocktwits in the paper are proposed as benchmarks for comparison with baseline models. And that there are no additional datasets apart from these 2.\n\nNovelty:\n(1) Similar problem statement addressed in https://dl.acm.org/doi/pdf/10.1145/3394486.3403389 - Needs to be added as a baseline.\n(2) The paper proposes a static hypergraph encoding technique and feed the encoding from snapshots at different timestamps into an LSTM for time -series  prediction. \n(3) The static hypergraph encoding technique in itself is not very novel. The paper uses attention mechanism for aggregation of node features into hyperedges (inner attention) and yet another attention mechanism (outer attention) for aggregating influences of hyperedges onto nodes .Finally an aggregation layer is used to update node embeddings. This setup is again similar to the paper mentioned above.\n\nExperiments: (1) The paper proposes a mechanism for encoding hypergraphs which evolve over time. However, the authors run experiments on only 2 graphs and compared with 4 baseines. \n(2) There are only two datasets both related to stock prediction. The procedure for node/hyperedge feature vector generation needs more explanation. Also, it’ll be good to run the model+baselines on additional graphs from other domains (pointer can be seen at paper linked above)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wu2tq8LPIRI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends over hypergraph convolutional networks (HCN) by adding a temporal evolution module in order to solve prediction tasks in a dynamic environment. The main part of the paper is the description of the proposed system. It is composed of a HCN for computing node embeddings at each time step and a LSTM as the temporal module. Experimental results are provided for dynamic prediction tasks over stock datasets.\n\nThe proposed system seems to be a reasonable choice for solving prediction tasks on dynamic hypergraphs. But, in my opinion, some choices could have been justified and variations of the system could have been compared. The only considered system variation is the aggregation method and the best one is the only one using trainable parameters. Thus I think that there is room for improvement. Also the current version is difficult to read due to many typos, mispellings, repetitions, ... Therefore, in my opinion, the paper is not ready for publication.\n\nDetailed comments.\n\n* Too many typos, mispellings, ... to be given in this review\n\n* Introduction. It could be improved. Formal definitions are not necessary. I am not convinced that the two formulations for the evolution are useful and they are repeated in Section 3.1. The presentation of the system could be improved: the description of the \"two challenging toughs\" is not convincing enough; the description of the DyHCN system is unclear (for instance what is a node-hyperedge ?).\n\n* Section 2. §GCN. There is a huge recent literature on GNNs. Please give useful links and please explain why you choose to only present some GCNs. The definition is not easy to read.\n\n* Section 2. §HCN. Also many HNNs models have been proposed. Please justify your choice. The definition is not easy to read. Also definitions of hypergraphs should be coherent (definition in Section 2 wrt definition in section 3.1).\n\n* Section 3.1. It is already presented in the introduction. I am not convinced that the continuous-time presentation is useful because it is not used.\n\n* Section 3. Please introduce what is the meaning of a \"centroid node\"\n\n* Section 4. §GCN should be updated with recent references.\n\n* Section 5.1. I would like to know why it is useful to model the task with hypergraphs. Why is it important to know for a stock event in which news or which comment it appears?\n\n* Section 5. It is not clear to me how the authors model the task when using time series prediction. Also for RSR and DHGCN.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7CZtgRy6Uw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a pipeline for dynamic hypergraph convolutional networks, with a two-fold component that handles both the hypergraph convolution and the temporal evolution. I think the paper is a nice contribution to the less-well studied area of time dependent networks, and to some extent, also that of hypergraph embedding. The application to finance and the data sets used are interesting, with hypergraphs/edges encoding a proxy for co-occurence behaviour of stocks.\n\nI think the experimental section is less strong, with only two data sets considered (albeit indeed the setting is less usual and less such data sets exist), and the authors only compared against a single other methodology. Regarding the performance metrics, the 3 ones currently used are less effective/appropriate when dealing with financial/stock price prediction data, where more relevant measures would be PnL and Sharpe Ratio, especially when weights are taken into account that relate to the liquidity of each of the instruments considered in the portfolio. The number of testing days is also very small, in order to be able to draw meaningful results, especially for a portfolio of 91 instruments.  \n\nThe paper is for the most part fairly straightforward to follow, though it is full of typos: a few examples being\nan financial, this work explore, an node, Spectral graph convolution transform features, outer at-tention, to name a few.\nThe paper should be proofread very carefully.\n\nSome of the notation could be better explained. I don’t see any mention of what y represents? (future price/returns)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper builds upon hypergraph convolutional networks (HCN), extending them to time-varying hypergraphs in dynamical settings.  However, as some of the reviewers pointed out, it would be useful to explore other system variations to better justify the choices in this particular approach; perhaps an evaluation on a wider set of datasets would also strengthen the contribution of the paper,  as well as adding evaluation metrics that can be more appropriate for the application considered (stock market prediction). Also, concerns were raised by several reviewers regarding the somewhat incremental  improvement over the state of art, and the degree of novelty in the proposed approach. Overall, while the problem considered is important and the approach is promising, the paper in its current shape  is somewhat borderline and may require a bit of additional work to be ready for publication.

Decision: Reject