Title: AN ATTENTION-BASED MODEL FOR LEARNING DYNAMIC INTERACTION NETWORKS Anonymous authors Paper under double-blind review

Abstract: While machine learning models achieve human-comparable performance on sequential data, exploiting structured knowledge is still a challenging problem. Spatio-temporal graphs have been proved to be a useful tool to abstract interaction graphs and previous works exploits carefully designed feed-forward architecture to preserve such structure. We argue to scale such network design to real-world problem, a model needs to automatically learn a meaningful representation of the possible relations. Learning such interaction structure is not trivial: on the one hand, a model has to discover the hidden relations between different problem factors in an unsupervised way; on the other hand, the mined relations have to be interpretable. In this paper, we propose an attention module able to project a graph sub-structure in a fixed size embedding, preserving the influence that the neighbors exert on a given vertex. On a comprehensive evaluation done on real-world as well as toy task, we found our model competitive against strong baselines.

1. INTRODUCTION:

In nature, many events are the consequence of the cooperation among different components of a whole system. Similarly, the modern society is characterised by the interaction between different economic and political forces not completely understood. Being able to define such forces is an essential step towards the understanding of the modern world as well as the creation of general intelligence, but it requires the capability of reason at two different level: first, a single individual; second, as a group of interactive units. Moreover, such problem setting is challenging for different reasons: first, a meaningful global structure has to be preserved; then, in large interactive networks some links could be irrelevant, thus introducing noise in the learned embedding.

On the one hand, recurrent neural networks (RNNs) has been proven to successfully solve longterm end-to-end learning tasks Sutskever et al. (2014) ; Graves & Jaitly (2014) ; Fragkiadaki et al. (2015) ; Mikolov et al. (2010) . Nonetheless, they cannot condition their distribution based on spatial relations. Recently, the Sequence-to-Sequence model Sutskever et al. (2014) (Seq2Seq) enriched RNNs' expressive power to solve complex sequential tasks. However, one of the most significant limitations of such systems lies in its capability of handling only array data structure, while many different problems are better expressed as a graph structure. On the other hand, traditional graph embedding techniques Perozzi et al. (2014) ; Cavallari et al. (2017) ; Tang et al. (2015) presuppose the existence of a well-defined graph structure, while usually these models are not suited to preserve the evolution of the node's attributes Dong et al. (2017) ; Wei et al. (2017) or they are not able to automatically determine the influence that the neighbours exert on a target node Ying et al. (2018) ; Battaglia et al. (2016) or do not leverage the supervised information during learning Perozzi et al. (2014) ; Tang et al. (2015) ; Grover & Leskovec (2016) .

Jointly learn an interactive graph structure and predicting the future behaviour of the system is a non-trivial task. A straightforward solution is to compute a pairwise influence between each pair of nodes in the dataset. Such computed scores can be used as weights of a fully connected graph. Then any inference algorithm can be used to perform regression or classification task at the node level. However, such an approach is unscalable and lack a global view of the problem since each pair of nodes is evaluated independently. Nevertheless, such problem setting is a typical scenario in many real-world applications. For example, consider that it is possible to represent the traffic flow of a given geographic area through a network structure of interconnected traffic stations where the node's attributes are proportional to the car flow across such stations (Fig. 1 ). The inference model then has to reason over a multitude of connected stations in order to predict the correct flow, since it is not a single car that generates traffic jam, but it is a multitude of vehicles at close locations that cause traffic congestion. Having such an intelligent system would be valuable to most of the transportation authorities and urban planner willing to predict and manage the road's bottleneck and improve the urban design.

During the last years, researchers found useful to represent such interactive structure with a graph, more in detail, spatio-temporal graphs (st-graphs) Li & Nevatia (2008) ; Lezama et al. (2011) ; George & Shekhar (2008) ; Liu et al. (2011) became a powerful tool to represent and study such systems. In Jain et al. (2016) , the authors propose a pioneering framework, named Structural-RNN, where the temporal modeling ability of RNNs is combined with st-graphs generating a mixture of RNNs able to combine the best of both worlds. Recently, interaction networks Battaglia et al. (2016) ; van Steenkiste et al. (2018) ; Sukhbaatar et al. (2016) became a popular framework to solve physical and structured problems. Exploiting a similar design to well-known GNNs Scarselli et al. (2009) , where messages are passed over connected graphs, these networks can efficiently separate the problem's objects from their relations; thus adapting at the changes expected in such dynamic environment. However, most of these models adopt a pairwise approach which is not able to properly preserve the graph structure Santoro et al. (2017) ; van Steenkiste et al. (2018) or they are not able to capture the evolution of the entire systemKipf et al. (2018) ; Hoshen (2017) .

In this work, we propose a novel attention-based interaction network able to leverage a dynamic interactive structure representative of the underlying problem. Similarly to Veličković et al. (2017) , our model employs a self-attention mechanism to evaluate the influence that all neighbours have w.r.t. a given vertex. However, we are seeking for a fixed size embedding representative of the evolving interactive structure as well as of the node attributes to perform a regressive or auto-regressive task at the node level. Note that we are also different from Lee et al. (2018) since we propose a supervised learning algorithm trained end-to-end to perform a node-level task, instead of a reinforcement learning formulation for graph-level classification. We extensively evaluate such an approach on three real-world datasets as well as on three synthetic datasets. The main contributions of this paper are: 1. we propose a model able to automatically infer the interactive structure of a problem; 2. to better preserve the problem's semantic, we propose a different parametrisation of the node and neighbours; 3. we show that our attention architecture is comparable or outperforms structured and unstructured algorithms.

2. DYNAMIC INTERACTION ATTENTION NETWORK:

At each instant a node is influenced by its internal state as well as by the dynamic interaction with its neighbours. To this end, the objective of this work is to propose an architecture able to automatically learn a dynamic graph structure G = (V, E, W t ), where V is the nodes' set, E is the edges' set and W t is the set of the edge's weight at time t (w t j,i ) representative of the influence exerted by node v j on node v i . On the one hand, to grasp such dynamic graph structure, it is necessary to learn appropriate edge's weights based on: 1. the node's attributes; 2. the neighbourhood behaviours; 3. and the system performance at previous time-steps. For example, consider the traffic prediction problem: in order to obtain an accurate prediction of the car's flow through a target station, it is Attention < l a t e x i t s h a 1 _ b a s e 6 4 = " w a P M 5 +

V V h g A F u H s c k Y V p B 7 3 R L t I = " > A A A C H 3 i c b V B N S 8 N A E J 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b x 4 r G C s Y U 2 l M 1 2 0 y 7 d f L A 7 K a 2 h / 8 S T o L / F k 3 j t T / H m p s 3 B t j 4 Y e L w 3 w 8 w 8 P x F c o W 1 P j b X 1 j c 2 t 7 d J O e X d v / + D Q P D p + V n E q K X N p L G L Z 9 I l i g k f M R Y 6 C N R P J S O g L 1 v A H 9 7 n f G D K p e B w 9 4 T h h X k h 6 E Q 8 4 J a i l j m m 2 k Y 0 w u 0 N k U a 5 M O m b F r t o z W K v E K U g F C t Q 7 5 k + 7 G 9 M 0 1 P N U E K V a j p 2 g l x G J n A o 2 K b d T x R J C B 6 T H W p p G J G T K y 2 a X T 6 x z r X S t I J a 6 I r R m 6 t + J j I R K j U N f d 4 Y E + 2 r Z y 8 X / v F a K w Y 2 X 8 S h J 9 W N 0 v i h I h Y W x l c d g d b l k F M V Y E 0 I l 1 7 d a t E 8 k o a j D W t i C f P C y 8 E X W k y T p c z r K s 3 K W k 1 k l 7 m X 1 t m o / X l V q d h F a C U 7 h D C 7 A g W u o w Q P U w Q U K Q 3 i F d / g w 3 o x P 4 8 v 4 n r e u G c X M C S z A m P 4 C + P i k w Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w a P M 5 + V V h g A F u H s c k Y V p B 7 3 R L t I = " > A A A C H 3 i c b V B N S 8 N A E J 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b x 4 r G C s Y U 2 l M 1 2 0 y 7 d f L A 7 K a 2 h / 8 S T o L / F k 3 j t T / H m p s 3 B t j 4 Y e L w 3 w 8 w 8 P x F c o W 1 P j b X 1 j c 2 t 7 d J O e X d v / + D Q P D p + V n E q K X N p L G L Z 9 I l i g k f M R Y 6 C N R P J S O g L 1 v A H 9 7 n f G D K p e B w 9 4 T h h X k h 6 E Q 8 4 J a i l j m m 2 k Y 0 w u 0 N k U a 5 M O m b F r t o z W K v E K U g F C t Q 7 5 k + 7 G 9 M 0 1 P N U E K V a j p 2 g l x G J n A o 2 K b d T x R J C B 6 T H W p p G J G T K y 2 a X T 6 x z r X S t I J a 6 I r R m 6 t + J j I R K j U N f d 4 Y E + 2 r Z y 8 X / v F a K w Y 2 X 8 S h J 9 W N 0 v i h I h Y W x l c d g d b l k F M V Y E 0 I l 1 7 d a t E 8 k o a j D W t i C f P C y 8 E X W k y T p c z r K s 3 K W k 1 k l 7 m X 1 t m o / X l V q d h F a C U 7 h D C 7 A g W u o w Q P U w Q U K Q 3 i F d / g w 3 o x P 4 8 v 4 n r e u G c X M C S z A m P 4 C + P i k w Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w a P M 5 + V V h g A F u H s c k Y V p B 7 3 R L t I = " > A A A C H 3 i c b V B N S 8 N A E J 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b x 4 r G C s Y U 2 l M 1 2 0 y 7 d f L A 7 K a 2 h / 8 S T o L / F k 3 j t T / H m p s 3 B t j 4 Y e L w 3 w 8 w 8 P x F c o W 1 P j b X 1 j c 2 t 7 d J O e X d v / + D Q P D p + V n E q K X N p L G L Z 9 I l i g k f M R Y 6 C N R P J S O g L 1 v A H 9 7 n f G D K p e B w 9 4 T h h X k h 6 E Q 8 4 J a i l j m m 2 k Y 0 w u 0 N k U a 5 M O m b F r t o z W K v E K U g F C t Q 7 5 k + 7 G 9 M 0 1 P N U E K V a j p 2 g l x G J n A o 2 K b d T x R J C B 6 T H W p p G J G T K y 2 a X T 6 x z r X S t I J a 6 I r R m 6 t + J j I R K j U N f d 4 Y E + 2 r Z y 8 X / v F a K w Y 2 X 8 S h J 9 W N 0 v i h I h Y W x l c d g d b l k F M V Y E 0 I l 1 7 d a t E 8 k o a j D W t i C f P C y 8 E X W k y T p c z r K s 3 K W k 1 k l 7 m X 1 t m o / X l V q d h F a C U 7 h D C 7 A g W u o w Q P U w Q U K Q 3 i F d / g w 3 o x P 4 8 v 4 n r e u G c X M C S z A m P 4 C + P i k w Q = = < / l a t e x i t > x1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p y P c t / h V f Z M Z U / W g c o 2 X D O H B I Z 0 = " > A A A B + n i c b V B P S 8 M w H P 3 V v 3 P + q / P o J T g E T 6 M V Q b 0 N v H i c Y N 1 g K y V N s y 0 s b U q S y k b p V / H i Q c W r n 8 S b 3 8 Z 0 6 0 E 3 H 4 Q 8 3 v v 9 y M s L U 8 6 U d p x v a 2 1 9 Y 3 N r u 7 Z T 3 9 3 b P z i 0 j x q P S m S S U I 8 I L m Q v x I p y l l B P M 8 1 p L 5 U U x y G n 3 X B y W / r d J y o V E 8 m D n q X U j / E o Y U N G s D Z S Y D c G o e C R m s X m y q d F k L t F Y D e d l j M H W i V u R Z p Q o R P Y X 4 N I k C y m i S Y c K 9 V 3 n V T 7 O Z a a E U 6 L + i B T N M V k g k e 0 b 2 i C Y 6 r 8 f J 6 9 Q G d G i d B Q S H M S j e b q 7 4 0 c x 6 q M Z y Z j r M d q 2 S v F / 7 x + p o f X f s 6 S N N M 0 I Y u H h h l H W q C y C B Q x S Y n m M 0 M w k c x k R W S M J S b a 1 F U 3 J b j L X 1 4 l 3 k X r p u X c X z b b T t V G D U 7 g F M 7 B h S t o w x 1 0 w A M C U 3 i G V 3 i z C u v F e r c + F q N r V r V z D H 9 g f f 4 A L W a U q w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p y P c t / h V f Z M Z U / W g c o 2 X D O H B I Z 0 = " > A A A B + n i c b V B P S 8 M w H P 3 V v 3 P + q / P o J T g E T 6 M V Q b 0 N v H i c Y N 1 g K y V N s y 0 s b U q S y k b p V / H i Q c W r n 8 S b 3 8 Z 0 6 0 E 3 H 4 Q 8 3 v v 9 y M s L U 8 6 U d p x v a 2 1 9 Y 3 N r u 7 Z T 3 9 3 b P z i 0 j x q P S m S S U I 8 I L m Q v x I p y l l B P M 8 1 p L 5 U U x y G n 3 X B y W / r d J y o V E 8 m D n q X U j / E o Y U N G s D Z S Y D c G o e C R m s X m y q d F k L t F Y D e d l j M H W i V u R Z p Q o R P Y X 4 N I k C y m i S Y c K 9 V 3 n V T 7 O Z a a E U 6 L + i B T N M V k g k e 0 b 2 i C Y 6 r 8 f J 6 9 Q G d G i d B Q S H M S j e b q 7 4 0 c x 6 q M Z y Z j r M d q 2 S v F / 7 x + p o f X f s 6 S N N M 0 I Y u H h h l H W q C y C B Q x S Y n m M 0 M w k c x k R W S M J S b a 1 F U 3 J b j L X 1 4 l 3 k X r p u X c X z b b T t V G D U 7 g F M 7 B h S t o w x 1 0 w A M C U 3 i G V 3 i z C u v F e r c + F q N r V r V z D H 9 g f f 4 A L W a U q w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p y P c t / h V f Z M Z U / W g c o 2 X D O H B I Z 0 = " > A A A B + n i c b V B P S 8 M w H P 3 V v 3 P + q / P o J T g E T 6 M V Q b 0 N v H i c Y N 1 g K y V N s y 0 s b U q S y k b p V / H i Q c W r n 8 S b 3 8 Z 0 6 0 E 3 H 4 Q 8 3 v v 9 y M s L U 8 6 U d p x v a 2 1 9 Y 3 N r u 7 Z T 3 9 3 b P z i 0 j x q P S m S S U I 8 I L m Q v x I p y l l B P M 8 1 p L 5 U U x y G n 3 X B y W / r d J y o V E 8 m D n q X U j / E o Y U N G s D Z S Y D c G o e C R m s X m y q d F k L t F Y D e d l j M H W i V u R Z p Q o R P Y X 4 N I k C y m i S Y c K 9 V 3 n V T 7 O Z a a E U 6 L + i B T N M V k g k e 0 b 2 i C Y 6 r 8 f J 6 9 Q G d G i d B Q S H M S j e b q 7 4 0 c x 6 q M Z y Z j r M d q 2 S v F / 7 x + p o f X f s 6 S N N M 0 I Y u H h h l H W q C y C B Q x S Y n m M 0 M w k c x k R W S M J S b a 1 F U 3 J b j L X 1 4 l 3 k X r p u X c X z b b T t V G D U 7 g F M 7 B h S t o w x 1 0 w A M C U 3 i G V 3 i z C u v F e r c + F q N r V r V z D H 9 g f f 4 A L W a U q w = = < / l a t e x i t > 2 4

x2, x3, x4 

0 C Z N X Q 1 a L 0 G c 8 S I F h V w y Y / p d P 8 f Q + q L g E q x z Y S B n / I E N o W + h Y i m Y s J z k V N E 9 y 8 Q 0 y b R d C u m E / d p R s t T U Y 9 u T K c O R + a n V 5 F 9 a v 8 D k N C y F y g s E x T 8 v S g p J M a N 1 6 D Q W G j j K s Q W M a 2 F n p X z E N O N o v 6 Y O o f v

0 C Z N X Q 1 a L 0 G c 8 S I F h V w y Y / p d P 8 f Q + q L g E q x z Y S B n / I E N o W + h Y i m Y s J z k V N E 9 y 8 Q 0 y b R d C u m E / d p R s t T U Y 9 u T K c O R + a n V 5 F 9 a v 8 D k N C y F y g s E x T 8 v S g p J M a N 1 6 D Q W G j j K s Q W M a 2 F n p X z E N O N o v 6 Y O o f v

0 C Z N X Q 1 a L 0 G c 8 S I F h V w y Y / p d P 8 f Q + q L g E q x z Y S B n / I E N o W + h Y i m Y s J z k V N E 9 y 8 Q 0 y b R d C u m E / d p R s t T U Y 9 u T K c O R + a n V 5 F 9 a v 8 D k N C y F y g s E x T 8 v S g p J M a N 1 6 D Q W G j j K s Q W M a 2 F n p X z E N O N o v 6 Y O o f v

E K O 0 h g z h S i U s u v Y i f I y K B R B F O e 1 X i p x A t E Q 9 n F X Q w Z j L L 1 s 4 j 6 3 j j U T W h E X + j F l T d i / E x m M Z W F Q d 8 Z Q D e S 8 V p D / a d 1 U R R d e R l i S K s z Q d F G U U k t x q 4 j C C o n A S N G R B h A J o r 1 a a A A F R E o H N r N F k e F 4 5 o q s L 2 A y I O i p y M q Z T 2 Y R u K e N y 4 Z 9 e 1 Z v 2 m V o V X A I j s A J c M A 5 a I J r 0 A I u Q G A M n s E r e D N e j H f j w / i c t l a M c m Y f z J T x / Q s C c q Z R

< / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 q 

X h M 6 D G G 9 T + 8 k B D C c k y Q + p M u 9 U = " > A A A C I 3 i c b V D L S s N A F J 3 U V 6 2 v + N i 5 C R b B V U l E U H c F N 6 6 k g r G F N o T J Z N I O n c y E m Y n a h v y L K 0 G / x Z W 4 c e G H

E K O 0 h g z h S i U s u v Y i f I y K B R B F O e 1 X i p x A t E Q 9 n F X Q w Z j L L 1 s 4 j 6 3 j j U T W h E X + j F l T d i / E x m M Z W F Q d 8 Z Q D e S 8 V p D / a d 1 U R R d e R l i S K s z Q d F G U U k t x q 4 j C C o n A S N G R B h A J o r 1 a a A A F R E o H N r N F k e F 4 5 o q s L 2 A y I O i p y M q Z T 2 Y R u K e N y 4 Z 9 e 1 Z v 2 m V o V X A I j s A J c M A 5 a I J r 0 A I u Q G A M n s E r e D N e j H f j w / i c t l a M c m Y f z J T x / Q s C c q Z R

< / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 q necessary to also consider the status of the neighbour stations. On the other hand, with respect to Lee et al. ( 2018) our objective is to perform a regression task at the node level. To better preserve the graph structure, we partition the nodes based on the problem's aspects they represent, formally we distinguish between:

X h M 6 D G G 9 T + 8 k B D C c k y Q + p M u 9 U = " > A A A C I 3 i c b V D L S s N A F J 3 U V 6 2 v + N i 5 C R b B V U l E U H c F N 6 6 k g r G F N o T J Z N I O n c y E m Y n a h v y L K 0 G / x Z W 4 c e G H

E K O 0 h g z h S i U s u v Y i f I y K B R B F O e 1 X i p x A t E Q 9 n F X Q w Z j L L 1 s 4 j 6 3 j j U T W h E X + j F l T d i / E x m M Z W F Q d 8 Z Q D e S 8 V p D / a d 1 U R R d e R l i S K s z Q d F G U U k t x q 4 j C C o n A S N G R B h A J o r 1 a a A A F R E o H N r N F k e F 4 5 o q s L 2 A y I O i p y M q Z T 2 Y R u K e N y 4 Z 9 e 1 Z v 2 m V o V X A I j s A J c M A 5 a I J r 0 A I u Q G A M n s E r e D N e j H f j w / i c t l a M c m Y f z J T x / Q s C c q Z R < / l a t e x i

P V = the target nodes → V1 the neighbours nodes → V2

Such partition P V aims to better preserve the difference between node's features w.r.t. the neighbours' attributes. Let us define a vertex i as v i , similarly let us define the attributes time series of

v i as x i = [x 1 i , ..., x T i ]

where x t i ∈ R d represent the feature vector at time t. Then, Fig. 2 shows a graphical representation of the node partitioning. Specifically, Fig. 2a report a target node v 1 with three neighbours [v 2 , v 3 , v 4 ]. According to our goal, at time t, we aim to learn the target node feature status (x t+1 1 ) subject to the neighbours' influence and attribute represented respectively by the edge weight w t 2,1 , w t 3,1 , w t 4,1 and x 2 , x 3 , x 4 . Thus, in Fig. 2b we obtain two distinct parametrisation: the first one for the node features (RNN V1 (x 1 )), and the second representative of the neighbours attributes (RNN V2 (

EQUATION

)). Note that in Fig. 2b •,

• stand for the concurrent processing of different neighbours in the same batch, also note that two distinct RNN are used to encode the different time series. Such partitioning enables us to reduce the number of parameters to learn since all the neighbours share the same parametrisation while it better preserve the problem's semantic since the nodes and the respective neighbours have different encoding procedure. We define N i as the set of the neighborhood of node v i ; thus the propose attention mechanism aims to learn a meaningful edge weights

w N1 = [w 1 2,1 , w 1 3,1 , w 1 4,1 , ..., w T 2,1 , w T 3,1 , w T 4,1

] for all the time-step and for all the neighbours of the target vertex v 1 .

To this end, the proposed Dynamic Interaction Attention Network (DIAN) is technically different from Hoshen (2017); Veličković et al. (2017) since it uses a different attention formulation that not only span over the graph structure but also over time; moreover, it is based on a different edge representation.

Note that the proposed DIAN model is different from the Structural-RNN architecture because it does not provide a prior interaction structure. Instead, it automatically learns the influence w t j,i that each edge exert on a given incident node. Such automatic mechanism allows the DIAN architecture to better generalise on new tasks as it does not require the manual crafting of each relation. As a side effect, due to the attention module used, the interaction structure learned becomes explicit and straightforward to interpret. Our model is also different from NRI Kipf et al. (2018) since we do not process each edge independently through a message passing process, but we obtain a global view of the problem through the attention mechanism. Also note that partitioning the edges in semantically different group reduce the number of parameters to learn w.r.t. the NRI and the Structural-RNN architecture, since DIAN do not have a different network for each edge type, but only different encoders for the a given node and its neighbours. 2018) or process each problem aspects with different RNNs organised in a feedforward structure Jain et al. (2016) . However, the first approach is not able to accurately differentiate between the node attributes as well as the neighbours' attribute, while the second approach is not able to automatically learn the influence of each neighbour.

To preserve the best of both worlds, we propose to obtain different parametrisation for the node and the neighbours' aspects. Since the nodes' state change over time, we have to ensure that such dynamism is preserved. RNNs are a deep learning model that has been proven capable to preserving the smooth transition between subsequent time-step of a given sequence Graves & Jaitly (2014); Fragkiadaki et al. (2015) ; Mikolov et al. (2010) , thus are an excellent architecture to capture such dynamics. An RNN hidden state, relative to the input vertex i, can be expressed as:

h t i = f rnn (x t i , h t-1 i )

where f rnn (•, •) is a non-linear function that combines two source of information. Note that in this work we used the well-known GRU model Cho et al. (2014) as recurrent function f rnn (•, •).

As shown in Fig. 3 , given a target node v i , we obtain a different parametrisation of the node and neighbours attributes using different RNNs. Specifically, we define

h t i = f rnn V 1 (x t i , h t-1 i

) as the target node representation at time t and

h t Ni = f rnn V 2 ([x t j ] j∈Ni , h t-1

Ni ) as the neighbours representation at time t, where in this case [•] j∈Ni , as show in Fig. 2b , stand for a batch composed by all the j neighbours present in the set N i . Consequently we define

h i = [h 1 i , ..., h T i ] as the encoded time series of vertex i processed by f rnn V 1 and h Ni = [h 1 Ni , ..., h T Ni ]

as the time series of v i 's neighbours processed by f rnn V 2 . We are aware that in this setting f rnn V 2 is trained more times than f rnn V 1 due to the different batch size: approximately D times more where D is the average degree of the graph G. Although this raises some theoretical concern, empirically we have not found it to be an issue.

In order to obtain the dynamic weight representative of the influence of each neighbour, a selfattention mechanism is employed. Specifically, we propose to use the scaled dot product attention Vaswani et al. (2017) to learn the edges weight:

EQUATION

where w Ni = [w j,i ] j∈Ni , and W Q , W K are linear transformation to ensure that h i and h Ni belong to the same semantic space. Respect to Veličković et al. (2017) we are different in two aspects: first, we jointly attend both over time and overall the neighbours; secondly, we do not attend directly on the edges representation as in Veličković et al. (2017) , but rather on the nodes their-self.

The weights are then used to obtain a relevant fixed-size embedding of the meaningful neighbours:

h Ni = w Ni • W V h Ni (3)

where W V is an other linear transformation. Such formulation enables the DIAN architecture to focus only on the relevant sub-graph, reducing the noise present in the structure of complex systems. Note, at time t the influence of a neighbour j over a node i is defined as :

h t j = 1≤t≤t w t j,i • W V h t j (4)

thus, we not only focus on the relevant neighbours, but we also align the time series of each neighbours w.r.t. the target node.

To increase the expressive power of our model K parallel attention heads are executed as suggested in Veličković et al. (2017) ; Vaswani et al. (2017) , but instead of using an average polling function, we found more useful to linearly combine (trough W H ) each attention head to find the final representation:

EQUATION

where h k Ni stand for the neighbours representation of the k-th attention head, σ(•) as an activation function (we used ReLU) and [•] K k=1 stand to the concatenation of all the K attention output. Note that we also have found empirically useful to apply a layer normalisation Ba et al. (2016b) on the final neighbours embedding.

The final sub-structure embedding relative to a target vertex v i is then obtained as the concatenation [h i , h Ni ] and it is used to perform the regression task unifying the node's information with the relevant neighbours. Also, we have found useful to add recurrent connection from the final output to the target node input, such connection have been proven effective when modeling volatile timeseries. It should be noted that:

1. Since we deal with dynamic influence, at time t, it is important to mask all the t + 1, ..., T time-steps in order to maintain the time causality of the problem. 2. Unlike GAT Veličković et al. (2017) , our network architecture has to jointly attend on different neighbours as well as on different time-steps. Moreover, we present a different attention mechanism as well as a different node representation able to better preserve the difference between the node's attributes and neighbours' attributes. 3. As stated in Veličković et al. (2017) , such attention architectures are not able to scale on interaction graphs with a high average degree. One possible solution is to develop a linear attention formulation Ba et al. (2016a) or to follow the recent trend in GCN Chen et al. (2018) ; Hamilton et al. (2017) ; Ying et al. (2018) where only a sample of the total neighbours is processed at each batch; however this is left as future works.

3. EXPERIMENTS:

To evaluate the insight of Sec. 1 and 2, we have designed a series of real-world as well as toy tasks. The designed toy tasks are synthetically generated dataset created with the specific purpose of studying the convergence of the DIAN model w.r.t. some specific data characteristics. Instead, the real-world datasets assess the performance of the proposed architecture on existing problems.

Note that during all the experiments we used the mean squared error (MSE) as loss function and the rooted mean squared error (RMSE) as evaluation metrics.

3.1. DATASETS:

Synthetic Datasets. The synthetic datasets have been designed to replicate the same problem setting of the real-world regression task while challenging three main abilities required by the proposed architecture. Moreover, each synthetic dataset is composed of 10K examples for training, 2K for evaluating and 2K for testing.

• The Dynamic Noise dataset aims to test the dynamic filtering ability of our neighbours attention mechanism. Specifically, a monotonically increasing sequence is assigned to each vertex. Each target node is then connected with a random number of neighbours ranging between one and four. While an irregular behaviour characterises one neighbour, the other neighbours have a time series correlated to the one of the target node. At each time-step, the goal is to compute the sum between the values of the target node and the relevant neighbours. Whit this dataset, we aim to test the attention representation ability while having a variable number of relevant neighbours.

• The Time-shifted dataset aims to test the aligning time series ability of our model w.r.t. similar architecture such as GAT and structural-RNN. Specifically, each target node is associated with a triangular wave-formed time series, while each neighbour has a relevant wave-formed series shifted ahead in time and amplitude. The objective is to predict the one-step-ahead value of the target node.

In order to solve this task, a given model has to align the node time series with its neighbours to detect the shift point of the triangular wave-form sequences correctly.

• The Noise Time-shifted dataset combines the two previous task to investigate which network architecture is able to align and detect a variable number of relevant neighbours jointly. To this purpose, this synthetic dataset presents a target nodes characterised by a triangular wave-form time series, up to two irrelevant neighbours and at least two relevant neighbours having the same node's triangular time series but shifted in time and in amplitude by a random offset. Note also that in this task the final objective is to predict the node's next step value.

An example for each handcrafted datasets is reported in Appx. 4.1.

Real-World Datasets. Three real-world datasets are used to evaluate the proposed architecture. Please note that while in Sec. 1 we use the traffic flow prediction problem as a motivational example, here we also test our model on different tasks such as energy load forecast and credit risk. We argue that all those tasks have the same problem structure. For example, buildings closely geo-located are subject to the same climatic factors; thus they share a common consumption pattern. Similarly, in the credit risk context, the company owner and the company manager risk is related to the company credit status.

• The EnerNOC GreenButton1 dataset contains anonymized geo-located energy usage data from 100 commercial/industrial sites over the entire 2012. The energy consumption is re-sampled at 3hour intervals, anomalies readings are eliminated, and the energy consumption is divided by the building's square footage to have a more comparable metric. Note that, all the time series are normalized through the softplus function (sof tplus(x) = log(1 + e x )) and categorical features as the building type, the time of day and the day of the week are encoded in a one-hot format. We assume that buildings closely geo-located are subject to the same climatic factors, so they share a common consumption pattern. To this end, top-4 geographically closest sites are used as neighbours of a given building.

• The PeMS Bay-Area traffic2 dataset, as shown in Fig. 1a , consists of traffic data crawled from 3665 traffic-control stations around the San Francisco Bay area. Traffic data consist of car flow (car/10 min.) and average speed (mph) through each station for the weekdays ranging from May 2017 to July 2017. Each training example is formed by one-day data for a given station, and the objective is to predict the one hour ahead traffic congestion. We found useful to normalize the time series with a scaled softlog function (sof tlog(x) = log(1 + x 10 )). Our intuition is that geographically closely stations suffer from similar traffic condition, so for each traffic-control terminal we compute top-6 closest stations as potential neighbours.

• The Bank risk dataset consists of a private dataset of bank customers. Since Basel 2 agreement, banks are required to wisely execute a credit risk program to better handle loans and premiums discounts. Thus, for each customer, the internal credit status, the internal risk, the balance sheet and socio-demographic information are monthly provided. Moreover, a special section also identifies the links that bind customers with each other: for example, a family account can be the guarantee of a son's account, or the company owner and the company manager are related to the company credit 

3.2. BASELINES:

We designed our baselines to specifically evaluate how different neural networks can learn and exploit structured information in the data.

• Structural- RNN Jain et al. (2016) : directly model a graph structure given in input thanks to its feed-forward architecture, but is not explicitly able to dynamically adapt to the changes is such graph structure.

• Simple- RNN Cho et al. (2014) : to investigate the impact that the neighbours' information has on the final performance, we use a single GRU to only model the node sequences.

• Structural-FF: to evaluate the importance of modeling the smooth transition present in our time series, we propose a baseline that uses the same feed-forward architecture of the S-RNN architecture, but it employees multilayer perceptrons (MLPs) instead of the RNNs.

• Simple-FF: to understand how difficult it is to solve the proposed tasks, a simple MLP that only process the node information is used in the evaluation. Note that such network architecture, can neither model the smooth time transition, neither the structured information.

• GAT Veličković et al. (2017) : to evaluate the impact of our edge formulation and the proposed node partitioning, we also use GAT as a baseline. While being an architecture similar to the proposed one, the GAT is not able to preserve the smooth time translation since process each time-step independently.

• RNNGAT: since the used datasets are all time dependent; we decided to enhance the GAT using an RNN instead of the MLP to encode the edges and preserve the time transition better. Note that also in this architecture the well-known GRU design is used as the RNN implementation.

Moreover, we designed degenerated versions of our model to evaluate the impact of our proposed edge encoding, the recurrent connection and the attention module:

• DIAN rec : is a degenerated version the model shown in Fig. 3 that do not present the recurrent connection.

• DIAN trans : extend the findings of the work done in Vaswani et al. (2017) to the graph embedding setting. In this architecture, each edge is encoded using MLPs instead of RNNs. Then the attention architecture is expected to jointly learn the graph architecture as well as the time dependencies.

3.3. EXPERIMENTS SETUP:

We report a short summary of the parameters used during the experiments in Tab. 2. To obtain a fair comparison among all the baselines, we set the same number of hidden units u in all the evaluated models. In all the synthetic datasets, in particular, we set u = 5 while in the real-world forecasting problems we set u = 128. We have used 4 attention heads (h = 4) in all the experiments since the results shown in Fig. 4a demonstrate that 4-heads give the best trade-off between the network 2011) is used as optimizer with 0.01 as learning rate (lr = 0.01).

Moreover, it is worth noting that for all the experiments a ReLU activation function is applied after the encoding functions f rnn (), while in the real-world dataset, an ELU activation function is also applied after the final linear projection layer. It is also worth mentioning that, for the PeMS dataset, a temporal window (tw) is used to reducing the number of time-steps considered during back-propagation and speed up the learning process of the different models. In particular, on the PeMS dataset, we use a window size of 20 time-steps (tw = 20). Finally, note that all the models are trained for 100 epochs while the evaluation is performed every ten epochs. The parameter setting that better perform on the eval set is then used for testing. Remark: The reported results are the average of four independent execution.

3.4. RESULTS ON SYNTHETIC DATASETS:

Tab. 3 report the test results on the synthetic datasets for all the architecture mentioned in Sec. 3.2. As it is possible to see, the proposed DIAN model can outperform all the baselines obtaining a relative improvement between 4.3% and 76.5% in two out of three tasks, but it is outperformed by the DIAN rec architecture by 6.2%. However, such experiments demonstrate the advantages of the proposed node's parametrisation and attention mechanism over a feed-forward architecture or a GAT network. In details, w.r.t. the Structural-RNN our model can automatically focus only on the relevant neighbours; while w.r.t. the GAT model, the proposed architecture can better model the smooth temporal evolution of the environment. Instead, the RNNGAT results validate the idea of having a different parametrisation for the node and the neighbours, since the RNNGAT model is outperformed by the proposed DIAN architecture, especially on the Dynamic Noise and the Noise Time-shifted datasets. On the one hand, adequately encoding the edges is an important factor, since the performances of the DIAN trans and GAT model are significantly lower that others models. On the other hand, DIAN rec is a valuable baseline, providing comparable results to the DIAN model. However, the Dynamic Noise results demonstrate that the recurrent connection effectively helps to model volatile time series. Additional explanations are reported in Appx. 4.1.

3.5. RESULTS ON REAL-WORLD DATASETS:

In Tab. 4, we report the results of the experiments conducted on the three real-world datasets.

As shown, the proposed DIAN architecture outperforms all the baselines obtaining relative improvements that range form 0.76% (Structural-RNN and DIAN rec in the bank dataset) to 803.79% (DIAN trans in the PeMS dataset). On the one hand, it is interesting to observe how the EnerNOC dataset presents a valuable interactive structure since the Structural-RNN method performs comparably to the DIAN rec . On the other hand, the DIAN model is able to outperform all the others methods significantly. This suggests that the recurrent connection also provide valuable information when learning the interactive structure.

Thus, the DIAN model can obtain a better graph representation.

Similarly, the PeMS dataset presents a strong interactive structure since the DIAN rec can outperform baseline such as the Simple-RNN by a relative 5.2%. Moreover, the proposed DIAN model is able to better align and learn a valuable interactive structure since outperform both the Structural-RNN and the DIAN rec architectures. Notice that, neither the DIAN trans neither the GAT neither the RNNGAT can learn a meaningful problem representation. This could be a consequence of their design which it is not optimised to learn a smooth transition of the dynamic interactive structure; recall that the attention mechanism of the RNNGAT does not span over subsequent time-steps, so producing suboptimal results. Finally, note that such results validate on real-world problems the hypothesis of having a different parametrisation for each problem aspect to better preserve the graph structure since the RNNGAT performances are significantly worsted that the DIAN and the Structural-RNN approach on EnerNOC and PeMS.

Instead, the Bank dataset seems to present an unclear interactive structure, since complex models, such as the DIAN or the Structural-RNN are comparable to a Simple-RNN (2.66% of relative improvement).

Lastly, as for the toy datasets, we report additional information in Appx. 4.2.

3.6. SENSITIVITY:

To better evaluate the proposed model, we tested the impact of adding additional attention heads in Fig. 4a . Such experiments validate the idea that adding additional attention heads can focus on multiple aspects of the problem and create a richer embedding. However, on the toy task, four heads provide the best trade-off between the model complexity and the performances. For example, h = 4 is not suffering from over-fitting in the Dynamic Noise dataset and present comparable performances on the Noise Time-shifted dataset.

Finally, Fig. 4b shows the impact that the hidden dimension (parameter u) has on the real-world performances. As expected, a bigger hidden layer provides better performances. However, u = 128 provide significant better performances w.r.t. other settings.

4. CONCLUSION:

In this paper, we proposed a novel dynamic graph attention network. Unlike previous works, our model: 1. can jointly learn the underlying graph and align the nodes' time series; 2. gives a human-readable interpretation of the actual underlying graph structure; 3. can easily adapt to graph with different neighbours number, but cannot handle interaction graph with high degree. While this is not a problem for the tested tasks, further work is required in order to relax this assumption and, hence, develop linear attention. Note that a linear attention formulation would be highly valuable 

v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > Time-steps < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t >

(a) Dynamic Noise dataset with 1 irrelevant neighbour and 3 relevant 

Time-series < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > Time-steps < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z

v p 0 = " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9

V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0

= " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9

V 6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w v p 0 

= " > A A A B 8 X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h B s D H d p V L A I 2 F h G y J l A c o S 9 z V y y Z D + O 3 T 0 h H P k Z N h Y q t v 4 b O / + N m + Q K T X w w 8 H h v h p l 5 c c q Z s b 7 / 7 a 2 t b 2 x u b Z d 2 y r t 7 + w e H l a P j R 6 M y T S G k i i v d i Y k B z i S E l l k O n V Q D E T G H d j y + m / n t J 9 C G K d m y k x Q i Q Y a S J Y w S 6 6 R u i w m 4 N K A Z m H 6 l 6 t f 8 O f A q C Q p S R Q W a / c p X b 6 B o J k B a y o k x 3 c B P b Z Q T b R n l M C 3 3 M g M p o W M y h K 6 j k g g w U T 4 / e Y r P n T L A i d K u p M V z 9 f d E T o Q x E x G 7 T k H s y C x 7 M / E / r 5 v Z 5 D r K m U w z C 5 I u F i U Z x 1 b h 2 f 9 4 w D R Q y y e O E K q Z u x X T E d G E W p d S 2 Y U Q L L + 8 S s J 6 7 a b m P 9 S r j d s i j R I 6 R W f o A g X o C j X Q P W q i E F G k 0 D N 6 R W + e 9 V

2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C

x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n

W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C

x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n

W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3 2 I H R T q C W f L m c 9 7 n i + k P L W / E = " > A A A B 8 H i c b V A 9 S w N B E J 2 L X z F + R S 1 t D o N g Y 7 h L o 4 J F w M Y y Q m K C y R H 2 N n P J k t 2 9 Y 3 d P C E f + h Y 2 F i q 0 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p h w p o 3 n f T u F t f W N z a 3 i d m l n d 2 / / o H x 4 9 K D j V F F s 0 Z j H q h M S j Z x J b B l m O H Y S h U S E H N v h + H b m t 5 9 Q a R b L p p k k G A g y l C

x i l B g r P T a Z w A t t M N H 9 c s W r e n O 4 q 8 T P S Q V y N P r l r 9 4 g p q l A a S g n Neighbors 2

W n d 9 L z F B R p R h l O O 0 1 E s 1 J o S O y R C 7 l k o i U A f Z / O K p e 2 a V g R v F y p Y 0 7 l z 9 P Z E R o f V E h L Z T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0 g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < / l a t

< l a t e x i t s h a 1 _ b a s e 6 4 = " h r 7 T T x u 5 q Z w Y 

N 8 + Z Y D y M K 7 E f 2 n k = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L a S i b 7 b R d u s m G 3 Y l Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m R a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T R 6 M y z c H n S i r d j Z g B K R L w U a C E b q q B x Z G E T j S 5 m f u d J 9 B G q O Q B p y m E M R s l Y i g 4 Q y s F d y B G 4 0 h p Q 5 v 9 a s 1 t u A v Q d e I V p E Y K t P v V

w Y N 8 + Z Y D y M K 7 E f 2 n k = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L a S i b 7 b R d u s m G 3 Y l Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m R a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T R 6 M y z c H n S i r d j Z g B K R L w U a C E b q q B x Z G E T j S 5 m f u d J 9 B G q O Q B p y m E M R s l Y i g 4 Q y s F d y B G 4 0 h p Q 5 v 9 a s 1 t u A v Q d e I V p E Y K t P v V

w Y N 8 + Z Y D y M K 7 E f 2 n k = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L a S i b 7 b R d u s m G 3 Y l Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m R a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T R 6 M y z c H n S i r d j Z g B K R L w U a C E b q q B x Z G E T j S 5 m f u d J 9 B G q O Q B p y m E M R s l Y i g 4 Q y s F d y B G 4 0 h p Q 5 v 9 a s 1 t u A v Q d e I V p E Y K t P v V

A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b 7 a R d u s m G 3 Y 1 Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m h a n g 2 r j u t 1 P a 2 N z a 3 i n v V v b 2 D w 6 P q s c n j 1 p m i q H P p J C q G 1 K N g i f o G 2 4 E d l O F N A 4 F d s L J z d z v P K H S X C Y P Z p p i E N N R w i P O q L F S 7 w 7 5 a B x K p Y k 3 q N b c h r s A W S d e Q W p Q o D 2 o f v W H k m U x J o Y J q n X P c 1 M T 5 F Q Z z g T O K v 1 M Y 0 r Z h I 6 w Z 2 l C Y 9 R B v j h 5 R i 6 s M i S R V L Y S Q x b q 7 4 m c x l p P 4 9 B 2 x t S M 9 a o 3 F / / z e p m J r o K c J 2 l m M G H L R V E m i J F k / j 8 Z c o X M i K k l l C l u b y V s T B V l x q Z U s S F 4 q y + v E 7 / Z u G 6 4 9 8 1 a q 1 6 k U Y Y z O I c 6 e H A J L b i F N v j A Q M I z v M K b Y 5 w X 5 9 3 5 W L a W n G L m F P 7 A + f w B F A 6 Q k Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D O Z X B g X V o a L L k H L h y N r X D N 0 z e e I = " > A A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b 7 a R d u s m G 3 Y 1 Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m h a n g 2 r j u t 1 P a 2 N z a 3 i n v V v b 2 D w 6 P q s c n j 1 p m i q H P p J C q G 1 K N g i f o G 2 4 E d l O F N A 4 F d s L J z d z v P K H S X C Y P Z p p i E N N R w i P O q L F S 7 w 7 5 a B x K p Y k 3 q N b c h r s A W S d e Q W p Q o D 2 o f v W H k m U x J o Y J q n X P c 1 M T 5 F Q Z z g T O K v 1 M Y 0 r Z h I 6 w Z 2 l C Y 9 R B v j h 5 R i 6 s M i S R V L Y S Q x b q 7 4 m c x l p P 4 9 B 2 x t S M 9 a o 3 F / / z e p m J r o K c J 2 l m M G H L R V E m i J F k / j 8 Z c o X M i K k l l C l u b y V s T B V l x q Z U s S F 4 q y + v E 7 / Z u G 6 4 9 8 1 a q 1 6 k U Y Y z O I c 6 e H A J L b i F N v j A Q M I z v M K b Y 5 w X 5 9 3 5 W L a W n G L m F P 7 A + f w B F A 6 Q k Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D O Z X B g X V o a L L k H L h y N r X D N 0 z e e I = " > A A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b 7 a R d u s m G 3 Y 1 Q Q n + G F w 8 q X v 0 3 3 v w 3 b t s c t P X B w O O 9 G W b m h a n g 2 r j u t 1 P a 2 N z a 3 i n v V v b 2 D w 6 P q s c n j 1 p m i q H P p J C q G 1 K N g i f o G 2 4 E d l O F N A 4 F d s L J z d z v P K H S X C Y P Z p p i E N N R w i P O q L F S 7 w 7 5 a B x K p Y k 3 q N b c h r s A W S d e Q W p Q o D 2 o f v W H k m U x J o Y J q n X P c 1 M T 5 F Q Z z g T O K v 1 M Y 0 r Z h I 6 w Z 2 l C Y 9 R B v j h 5 R i 6 s M i S R V L Y S Q

:

https://open-enernoc-data.s3.amazonaws.com/anon/index.html

https://bit.ly/2sB27Pa

Whit respect to the comparable GAT architecture, the DIAN model is able to preserve the graph structure better, thanks to the node partitioning; compared to the GCN, the proposed model can give different importance score to the neighbours, thus better model the interactive structure.

Figures and Tables:

Figure 1: As motivational example Fig. 1b shows the latent interaction graph formed by the different traffic monitor stations around the San Francisco bay area located according to Fig. 1a.

t e x i t s h a 1 _ b a s e 6 4 = " 4 O A 7 a 2U R F 1 O f X s v f J o r G S p f G m 1 g = " > A A A C Z n i c d V H L S s Q w F E 3 r u 7 7 G F y 7 c B A f B h Q w d F d S d 4 M a l g q P C t A x p e j s T T N O S 3 I p D q R / p z r 0 b / 8 J 0 7 M L n h Z D D O b k n N y d R L o V B 3 3 9 1 3 K n p m d m 5 + Q V v c W l 5 Z b W 1 t n 5 r s k J z 6 P F M Z v o + Y g a k U N B D g R L u c w 0 s j S T c R Q 8 X t X 7 3 C N q I T N 3 g O I c w Z U M l E s E Z W m r Q e g 4 k J N j 3 K A 0 i G A p V M q 3 Z u C p l Z a m a z G R s x q n d y q d q U B5 W B z Q I / p S O / p e O a 7 M A V N y 4 e 4 E W w x G G n j d o t f 2 O P y n 6 G 3 Q b

z y b 9 B 7 7 B z 1 v G v j 9 v n f p P G P N k h u 2 S f d M k J O S e X 5 I r 0 C C d v j u d s O J v O u 7 v q b r n b n 0 d d p + n Z I N / K p R 9 R p L Y d < / l a t e x i t > < l a t e x i t s h a 1 _ b as e 6 4 = " 4 O A 7 a 2U R F 1 O f X s v f J o r G S p f G m 1 g = " > A A A C Z n i c d V H L S s Q w F E 3 r u 7 7 G F y 7 c B A f B h Q w d F d Sd 4 M a l g q P C t A x p e j s T T N O S 3 I p D q R / p z r 0 b / 8 J 0 7 M L n h Z D D O b k n N y d R L o V B 3 3 9 1 3 K n p m d m 5 + Q V v c W l 5 Z b W 1 t n 5 r s k J z 6 P F M Z v o + Y g a k U N B D g R L u c w 0 s j S T c R Q 8 X t X 7 3 C N q I T N 3 g O I c w Z U M l E s E Z W m r Q e g 4 k J N j 3 K A 0 i G A p V M q 3 Z u C p l Z a m a z G R s x q n d y q d q U B 5 W B z Q I / p S O / p e O a 7 M A V N y 4 e 4 E W w x G G n j d o t f 2 O P y n 6 G 3 Q b

z y b 9 B 7 7 B z 1 v G v j 9 v n f p P G P N k h u 2 S f d M k J O S e X 5 I r 0 C C d v j u d s O J v O u 7 v q b r n b n 0 d d p + n Z I N / K p R 9 R p L Y d < / l a t e x i t > < l a t e x i t s h a 1 _ b as e 6 4 = " 4 O A 7 a 2U R F 1 O f X s v f J o r G S p f G m 1 g = " > A A A C Z n i c d V H L S s Q w F E 3 r u 7 7 G F y 7 c B A f B h Q w d F d Sd 4 M a l g q P C t A x p e j s T T N O S 3 I p D q R / p z r 0 b / 8 J 0 7 M L n h Z D D O b k n N y d R L o V B 3 3 9 1 3 K n p m d m 5 + Q V v c W l 5 Z b W 1 t n 5 r s k J z 6 P F M Z v o + Y g a k U N B D g R L u c w 0 s j S T c R Q 8 X t X 7 3 C N q I T N 3 g O I c w Z U M l E s E Z W m r Q e g 4 k J N j 3 K A 0 i G A p V M q 3 Z u C p l Z a m a z G R s x q n d y q d q U B 5 W B z Q I / p S O / p e O a 7 M A V N y 4 e 4 E W w x G G n j d o t f 2 O P y n 6 G 3 Q b

z y b 9 B 7 7 B z 1 v G v j 9 v n f p P G P N k h u 2 S f d M k J O S e X 5 I r 0 C C d v j u d s O J v O u 7 v q b r n b n 0 d d p + n Z I N / K p R 9 R p L Y d < / l a t e x i t > wN 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 q X h M 6 D G G 9 T + 8 k B D C c k y Q + p M u 9 U = " > A A A C I 3 i c b V D L S s N A F J 3 U V 6 2 v + N i 5 C R b B V U l E U H c F N 6 6 k g r G F N o T J Z N I O n c y E m Y n a h v y L K 0 G / x Z W 4 c e G H u H P S Z m F b L w x z O O d e 7 r k n S C i R y r a / j M r S 8 s r q W n W 9 t r G 5 t b 1 j 7 u 7 d S 5 4 K h F 3 E K R e d A E p M C c O u I o r i T i I w j A O K 2 8 H w q t D b D 1 h I w t m d G i X Y i 2 G f k Y g g q D T l m w e 9 g N N Q j m L 9 Z Y + 5 n 9 3 4 T u 6 b d b t h T 8 p a B E 4 J 6 q C s l m / + 9

u H P S Z m F b L w x z O O d e 7 r k n S C i R y r a / j M r S 8 s r q W n W 9 t r G 5 t b 1 j 7 u 7 d S 5 4 K h F 3 E K R e d A E p M C c O u I o r i T i I w j A O K 2 8 H w q t D b D 1 h I w t m d G i X Y i 2 G f k Y g g q D T l m w e 9 g N N Q j m L 9 Z Y + 5 n 9 3 4 T u 6 b d b t h T 8 p a B E 4 J 6 q C s l m / + 9

u H P S Z m F b L w x z O O d e 7 r k n S C i R y r a / j M r S 8 s r q W n W 9 t r G 5 t b 1 j 7 u 7 d S 5 4 K h F 3 E K R e d A E p M C c O u I o r i T i I w j A O K 2 8 H w q t D b D 1 h I w t m d G i X Y i 2 G f k Y g g q D T l m w e 9 g N N Q j m L 9 Z Y + 5 n 9 3 4 T u 6 b d b t h T 8 p a B E 4 J 6 q C s l m / + 9

Figure 2: Graphical representation of our interaction graph, where semantically similar relations share the same parameters and the attention model is used to learn the factor interactions.

Figure 3: Overall model architecture. 2.1 MODEL ARCHITECTURE Most of the related work firstly encode the graph's edges as concatenation of the nodes features and then apply a graph attention network (GAT) to learn a proper network structure Duan et al. (2017); Hoshen (2017); Veličković et al. (2017); van Steenkiste et al. (2018) or process each problem aspects with different RNNs organised in a feedforward structureJain et al. (2016). However, the first approach is not able to accurately differentiate between the node attributes as well as the neighbours' attribute, while the second approach is not able to automatically learn the influence of each neighbour.

Fig.5report an example of the three synthetic dataset generated to validate the proposed DIAN architecture. Please note that the target time series of the Dynamic Noise dataset (Fig.5a) has a significantly bigger dynamic; thus require the model to adapt to sudden changes. < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w

T E D P S y 9 5 M / M / r p i a 6 C j I m k 9 S g p I t F U c p d E 7 u z 9 9 0 B U 0g N n 1 h C q G L 2 V p e O i C L U 2 J B K N g R / + e V V 0 q p V r 6 v e f a 1 S v 8 n T K M I J n M I 5 + H A J d b i D B r S A g o R n e I U 3 R z s v z r v z s W g t O P n M M f y B 8 / k D 5 i 2 Q l A = = < /l a t e x i t > Time-shifted dataset Time-series < l a t e x i t s h a 1 _ b a s e 6 4 = " j F D r l R t M s B j J s N g u s H u w l / 1 w

6 8 d + 9 j 0 b r m F T M n 6 A + 8 z x + Z / 5 D 6 < / l a t e x i t > Time-steps < l a t e x i t s h a 1 _ b a s e 6 4 = " A Z 4 3

Figure 5: A training example for each synthetic dataset. Note that the neighbors are represented by a thinner line w.r.t to the node and label time series. Moreover, noise neighbors have the thinnest line.

r 9 5 A 8 S y G B L l k x g S e m 2 K Y M 4 2 C S 5 h V e p m B l PE J G 0 F g a c J i M G G + O H l G L 6 w y o E O l b S V I F + r v i Z z F x k z j y H b G D M d m 1 Z u L / 3 l B h s O r M B d J m i E k f L l o m E m K i s 7 / p w O h g a O c W s K 4 F v Z W y s d M M 4 4 2 p Y o Nw V t 9 e Z 3 4 z c Z 1 w 7 1 v 1 l r 1 I o 0 y O S P n p E 4 8 c k l a 5 J a 0 i U 8 4 U e S Z v J I 3 B 5 0 X 5 9 3 5 W L a W n G L m l P y B 8 / k D F Z G Q k g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h r 7 T T x u 5 q Z

r 9 5 A 8 S y G B L l k x g S e m 2 K Y M 4 2 C S 5 h V e p m B l PE J G 0 F g a c J i M G G + O H l G L 6 w y o E O l b S V I F + r v i Z z F x k z j y H b G D M d m 1 Z u L / 3 l B h s O r M B d J m i E k f L l o m E m K i s 7 / p w O h g a O c W s K 4 F v Z W y s d M M 4 4 2 p Y o Nw V t 9 e Z 3 4 z c Z 1 w 7 1 v 1 l r 1 I o 0 y O S P n p E 4 8 c k l a 5 J a 0 i U 8 4 U e S Z v J I 3 B 5 0 X 5 9 3 5 W L a W n G L m l P y B 8 / k D F Z G Q k g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h r 7 T T x u 5 q Z

r 9 5 A 8 S y G B L l k x g S e m 2 K Y M 4 2 C S 5 h V e p m B l PE J G 0 F g a c J i M G G + O H l G L 6 w y o E O l b S V I F + r v i Z z F x k z j y H b G D M d m 1 Z u L / 3 l B h s O r M B d J m i E k f L l o m E m K i s 7 / p w O h g a O c W s K 4 F v Z Wy s d M M 4 4 2 p Y o N w V t 9 e Z 3 4 z c Z 1 w 7 1 v 1 l r 1 I o 0 y O S P n p E 4 8 c k l a 5 J a 0 i U 8 4 U e S Z v J I 3 B 5 0 X 5 9 3 5 W L a W n G L m l P y B 8 / k D F Z G Q k g = = < / l a t e x i t > Neighbors 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " D O Z X B g X V o a L L k H L h y N r X D N 0 z e e I = " > A

x b q 7 4 m c x l p P 4 9 B 2 x t S M 9 a o 3 F / / z e p m J r o K c J 2 l m M G H L R V E m i J F k / j 8 Z c o X M i K k l l C l u b y V s T B V l x q Z U s S F 4 q y + v E 7 / Z u G 6 4 9 8 1 a q 1 6 k U Y Y z O I c 6 e H A J L b i F N v j A Q M I z v M K b Y 5 w X 5 9 3 5 W L a W n G L m F P 7 A + f w B F A 6 Q k Q = = < / l a t e x i t > Neighbors 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " S X H a g 8 e X + G p b j e F Z M y c k F J Y H b 0 o = " > A A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L U g 3 o r e P E k F Y w W 0 l A 2 2 0 m 7 d J M N u x u h l P 4 M L x 5 U v P p v v P l v 3 L Y 5 a O u D g c d 7 M 8 z M i z L B t X H d b 6 e 0 t r 6 x u V X e r u z s 7 u 0f V A + P H r T M F U O f S S F V J 6 I a B U / R N 9 w I 7 G Q K a R I J f I x G 1 z P / 8 Q m V 5 j K 9 N + M M w 4 Q O U h 5 z R o 2 V g l v k g 2 E k l S b n v W r N b b h z k F X i F a Q G B d q 9 6 l e 3 L 1 m e Y G q Y o F o H n p u Z c E K V 4 U z g t N L N N W a U j e g A A 0 t T m q A O J / O T p + T M K n 0 S S 2 U r N W S u / p 6 Y 0 E T r c R L Z z o S a o V 7 2 Z u J / X p C b + D K c 8 D T L D a Z s s S j O B T G S z P 4 n f a 6 Q G T G 2 h D L F 7 a 2 E D a m i z N i U K j Y E b / n l V e I 3 G 1 c N 9 6 5 Z a 9 W L N M p w A q d Q B w 8 u o A U 3 0 A Y f G E h 4 h l d 4 c 4 z z 4 r w 7 H 4 v W k l P M H M M f O J 8 / F x S Q k w = = < /l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S X H a g 8 e X + G p b j e F Z M y c k F J Y H b 0 o = " > A A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L U g 3 o r e P E k F Y w W 0 l A 2 2 0 m 7 d J M N u x u h l P 4 M L x 5 U v P p v v P l v 3 L Y 5 a O u D g c d 7 M 8 z M i z L B t X H d b 6 e 0 t r 6 x u V X e r u z s 7 u 0f V A + P H r T M F U O f S S F V J 6 I a B U / R N 9 w I 7 G Q K a R I J f I x G 1 z P / 8 Q m V 5 j K 9 N + M M w 4 Q O U h 5 z R o 2 V g l v k g 2 E k l S b n v W r N b b h z k F X i F a Q G B d q 9 6 l e 3 L 1 m e Y G q Y o F o H n p u Z c E K V 4 U z g t N L N N W a U j e g A A 0 t T m q A O J / O T p + T M K n 0 S S 2 U r N W S u / p 6 Y 0 E T r c R L Z z o S a o V 7 2 Z u J / X p C b + D K c 8 D T L D a Z s s S j O B T G S z P 4 n f a 6 Q G T G 2 h D L F 7 a 2 E D a m i z N i U K j Y E b / n l V e I 3 G 1 c N 9 6 5 Z a 9 W L N M p w A q d Q B w 8 u o A U 3 0 A Y f G E h 4 h l d 4 c 4 z z 4 r w 7 H 4 v W k l P M H M M f O J 8 / F x S Q k w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S X H a g 8 e X + G p b j e F Z M y c k F J Y H b 0 o = " > A A A B 8 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L U g 3 o r e P E k F Y w W 0 l A 2 2 0 m 7 d J M N u x u h l P 4 M L x 5 U v P p v v P l v 3 L Y 5 a O u D g c d 7 M 8 z M i z L B t X H d b 6 e 0 t r 6 x u V X e r u z s 7 u 0 f V A + P H r T M F U O f S S F V J 6 I a B U / R N 9 w I 7 G Q K a R I J f I x G 1 z P / 8 Q m V 5 j K 9 N + M M w 4 Q O U h 5 z R o 2 V g l v k g 2 E k l S b n v W r N b b h z k F X i F a Q G B d q 9 6 l e 3 L 1 m e Y G q Y o F o H n p u Z c E K V 4 U z g t N L N N W a U j e g A A 0 t T m q A O J / O T p + T M K n 0 S S 2 U r N W S u / p 6 Y 0 E T r c R L Z z o S a o V 7 2 Z u J / X p C b + D K c 8 D T L D a Z s s S j O B T G S z P 4 n f a 6 Q G T G 2 h D L F 7 a 2 E D a m i z N i U K j Y E b / n l V e I 3 G 1 c N 9 6 5 Z a 9 W L N M p w A q d Q B w 8 u o A U 3 0 A Y f G E h 4 h l d 4 c 4 z z 4 r w 7 H 4 v W k l P M H M M f O J 8 / F x S Q k w = = < / l a t e x i t >1.0 2.01 2.02 3.03 3.04 3.05 6.06 7.07 8.08 9.09 1.1 4.11 3.12 5.13 2.14 1.15 3.16 2.17 9.18 0.19 2.2 3.21 3.22 4.23 4.24 4.25 7.26 8.27 9.28 10.t e x i t s h a 1 _ b a s e 6 4 = " V 6 S B u i l T t b 4 T D 6 L y b S M 8 0 s j f 4 g Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b z a Z d u h 9 h d y O U 0 L / g x Y O K V 3 + R N / + N S Z u D t j 4 Y e L w 3 w 8 y 8 M O H M W N f 9 d i o b m 1 v b O 9 X d 2 t 7 + w e F R / f j k 0 a h U E + o T x Z X u h 9 h Q z i T 1 L b O c 9 h N N s Q g 5 7 Y X T m 8 L v P V F t m J I P d p b Q Q O C x Z D E j 2 B b S n Y r o q N 5 w W + 4 C a J 1 4 J W l A i e 6 o / j W M F E k F l Z Z w b M z A c x M b Z F h b R j i d 1 4 a p o Q k m U z y m g 5 x K L K g J s s W t c 3 S R K x G K l c 5 L W r R Q f 0 9 k W B g z E 2 H e K b C d m F W v E P / z B q m N r 4 K M y S S 1 V J L l o j j l y C p U P I 4 i p i m x f J Y T T D T L b 0 V k g j U m N o + n l o f g r b 6 8 T v x 2 6 7 r l 3 r c b n W a Z R h X O 4 B y a 4 M E l d O A W u u A D g Q k 8 w y u 8 O c J 5 c d 6 d j 2 V r x S l n T u E P n M 8 f V P 6 N 3 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V 6 S B u i l T t b 4 T D 6 L y b S M 8 0 s j f 4 g Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b z a Z d u h 9 h d y O U 0 L / g x Y O K V 3 + R N / + N S Z u D t j 4 Y e L w 3 w 8 y 8 M O H M W N f 9 d i o b m 1 v b O 9 X d 2 t 7 + w e F R / f j k 0 a h U E + o T x Z X u h 9 h Q z i T 1 L b O c 9 h N N s Q g 5 7 Y X T m 8 L v P V F t m J I P d p b Q Q O C x Z D E j 2 B b S n Y r o q N 5 w W + 4 C a J 1 4 J W l A i e 6 o / j W M F E k F l Z Z w b M z A c x M b Z F h b R j i d 1 4 a p o Q k m U z y m g 5 x K L K g J s s W t c 3 S R K x G K l c 5 L W r R Q f 0 9 k W B g z E 2 H e K b C d m F W v E P / z B q m N r 4 K M y S S 1 V J L l o j j l y C p U P I 4 i p i m x f J Y T T D T L b 0 V k g j U m N o + n l o f g r b 6 8 T v x 2 6 7 r l 3 r c b n W a Z R h X O 4 B y a 4 M E l d O A W u u A D g Q k 8 w y u 8 O c J 5 c d 6 d j 2 V r x S l n T u E P n M 8 f V P 6 N 3 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V 6 S B u i l T t b 4 T D 6 L y b S M 8 0 s j f 4 g Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 L 0 o t 4 K X j x J B W M L b S i b z a Z d u h 9 h d y O U 0 L / g x Y O K V 3 + R N / + N S Z u D t j 4 Y e L w 3 w 8 y 8 M O H M W N f 9 d i o b m 1 v b O 9 X d 2 t 7 + w e F R / f j k 0 a h U E + o T x Z X u h 9 h Q z i T 1 L b O c 9 h N N s Q g 5 7 Y X T m 8 L v P V F t m J I P d p b Q Q O C x Z D E j 2 B b S n Y r o q N 5 w W + 4 C a J 1 4 J W l A i e 6 o / j W M F E k F l Z Z w b M z A c x M b Z F h b R j i d 1 4 a p o Q k m U z y m g 5 x K L K g J s s W t c 3 S R K x G K l c 5 L W r R Q f 0 9 k W B g z E 2 H e K b C d m F W v E P / z B q m N r 4 K M y S S 1 V J L l o j j l y C p U P I 4 i p i m x f J Y T T D T L b 0 V k g j U m N o + n l o f g r b 6 8 T v x 2 6 7 r l 3 r c b n W a Z R h X O 4 B y a 4 M E l d O A W u u A D g Q k 8 w y u 8 O c J 5 c d 6 d j 2 V r x S l n T u E P n M 8 f V P 6 N 3 w = = < / l a t e x i t >

Datasets summary. Note that for all the datasets, all the examples related to a given node and associated neighbors are only present in the training, evaluation or test set, thus we are in an inductive learning setting. Our intuition is that exploiting such relations could enhance the evaluation of the future risk of each customer. The goal of this dataset is to predict the risk score one time-step ahead. The overall datasets characteristics are summarized in Tab. 1.

Table content: <table><tr><td/><td colspan="4">EnerNOC PeMS Bank Synthetic</td></tr><tr><td>Node number</td><td>100</td><td colspan="2">3665 13953</td><td>14000</td></tr><tr><td>Neighbors for each node</td><td>4</td><td>6</td><td>4</td><td>4</td></tr><tr><td>Sequence lenght</td><td>16</td><td>138</td><td>17</td><td>10</td></tr><tr><td>Training examples</td><td>9570</td><td colspan="2">50000 9676</td><td>10000</td></tr><tr><td>Eval examples</td><td>4176</td><td colspan="2">12000 2000</td><td>2000</td></tr><tr><td>Test examples</td><td>4002</td><td colspan="2">11300 2000</td><td>2000</td></tr><tr><td>status.</td><td/><td/><td/><td/></tr></table>

Parameters setting used during the experiments.

Table content: <table><tr><td/><td>Symbol</td><td>Synthetic dataset</td><td>Real-world datasets</td></tr><tr><td>Hidden units</td><td>u</td><td>5</td><td>128</td></tr><tr><td>Attention heads</td><td>h</td><td>4</td><td>4</td></tr><tr><td>Learning rate</td><td>lr</td><td>0.01</td><td>0.01</td></tr></table>

Results on synthetic datasets.

Table content: <table><tr><td>Model</td><td colspan="3">Dynamic Noise Time-shifted Noise Time-shifted</td></tr><tr><td/><td>RMSE</td><td>RMSE</td><td>RMSE</td></tr><tr><td>Structural-RNN</td><td>1.56</td><td>0.20</td><td>0.41</td></tr><tr><td>Simple-RNN</td><td>5.39</td><td>0.53</td><td>0.54</td></tr><tr><td>Structural-FF</td><td>2.21</td><td>0.63</td><td>1.19</td></tr><tr><td>Simple-FF</td><td>5.64</td><td>1.46</td><td>1.44</td></tr><tr><td>DIAN</td><td>0.51</td><td>0.17</td><td>0.23</td></tr><tr><td>DIAN rec</td><td>0.90</td><td>0.16</td><td>0.24</td></tr><tr><td>DIAN trans</td><td>5.39</td><td>0.29</td><td>0.35</td></tr><tr><td>GAT</td><td>4.36</td><td>0.60</td><td>1.23</td></tr><tr><td>RNNGAT</td><td>1.21</td><td>0.25</td><td>0.4</td></tr><tr><td colspan="3">complexity and the performance improvements. Adagrad Duchi et al. (</td><td/></tr></table>

Real-world datasets' results.

Table content: <table><tr><td>Model</td><td colspan="3">EnerNOC PeMS Bank</td></tr><tr><td/><td>RMSE</td><td colspan="2">RMSE RMSE</td></tr><tr><td>Structural-RNN</td><td>0.045</td><td>55.85</td><td>1.33</td></tr><tr><td>Simple-RNN</td><td>0.049</td><td>58.75</td><td>1.36</td></tr><tr><td>Structural-FF</td><td>0.067</td><td>65.91</td><td>1.44</td></tr><tr><td>Simple-FF</td><td>0.063</td><td>71.4</td><td>1.45</td></tr><tr><td>DIAN</td><td>0.028</td><td>38.86</td><td>1.32</td></tr><tr><td>DIAN rec</td><td>0.046</td><td>55.65</td><td>1.33</td></tr><tr><td>DIAN trans</td><td>0.055</td><td>347.87</td><td>4.43</td></tr><tr><td>GAT</td><td>0.061</td><td>251.51</td><td>1.37</td></tr><tr><td>RNNGAT</td><td>0.055</td><td>245.95</td><td>1.36</td></tr></table>

Peer review (iclr2019):
 This paper proposes to apply attention mechanism to process message passing in a graph over time. It suffers from the following limitations: - The problem definition is unclear. What is the objective of the proposed model? Predict the attributes of a specific node or all the nodes? Forecast the types or weights of edges? Dynamically change the graph architecture? It is vague in the elaboration. The authors should prove precise and concise mathematical definitions of the task. - The background discussion is inadequate. As the proposed method is a follow-up research of GNN, the authors should present more discussions on previous GNN, GCN, and message passing designs. - The technique contribution is limited. In my opinion, the proposed framework is a straightforward simplification of a general GNN design. The authors highlight their contributions on sub-graph (neighbors) operation over time. Nevertheless, this is nature consideration or built-in design in a GNN with message passing mechanism. - The proposed applying attention mechanism in graph operation is nothing new without a comprehensive design. For instance, does the proposed method can be scaled up for a high-degree graph? There is no discussion or analysis but only empirical descriptions. - What does it by claiming "DIAN do not have a different network for each edge type, but only different encoders for a given node and its neighbors." If "edge type" means different modalities or embeddings of different dimensions, the proposed method can not handle different edge type by simply using an RNN encoder either. - Equation (1) should be removed or it is necessary to explain what Q and K represent. - Typos. "... the a given node..." and etc.