Flow Neural Network for Traffic Flow Modelling in IP Networks
Authors: Xiangle Cheng
Yuchen He
Feifei Long
Shihan Xiao
Fenglin Li
Venue:  (2021)
Keywords: Flow neural network
contrastive induction learning
representation learning
spatio-temporal induction

Abstract: This paper presents and investigates a novel and timely application domain for deep learning: sub-second traffic flow modelling in IP networks. Traffic flows are the most fundamental components in an IP based networking system. The accurate modelling of the generative patterns of these flows is crucial for many practical network applications. However, the high nonlinearity and dynamics of both the traffic and network conditions make this task challenging, particularly at the time granularity of sub-second. In this paper, we cast this problem as a representation learning task to model the intricate patterns in data traffic according to the IP network structure and working mechanism. Accordingly, we propose a customized Flow Neural Network, which works in a self-supervised way to extract the domain-specific data correlations. We report the state-of-the-art performances on both synthetic and realistic traffic patterns on multiple practical network applications, which provides a good testament to the strength of our approach.

Under review as a conference paper at ICLR 2021
FLOW NEURAL NETWORK FOR TRAFFIC FLOW MOD- ELLING IN IP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents and investigates novel and timely application domain for deep learning: sub-second traffic flow modelling in IP networks. Traffic flows are the most fundamental components in an IP based networking system. The accurate modelling of the generative patterns of these flows is crucial for many practical network applications. However, the high nonlinearity and dynamics of both the traffic and network conditions make this task challenging, particularly at the time granularity of sub-second. this paper, we cast this problem repre- sentation learning task to model the intricate patterns in data traffic according to the IP network structure and working mechanism. Accordingly, we propose a cus- tomized Flow Neural Network, which works in a self-supervised way to extract the domain-specific data correlations. We report the state-of-the-ar performances both synthetic and realistic traffic patterns multiple practical network appli- cations, which provides a good testament to the strength our approach.
INTRODUCTION
Deep Learning (DL) has gained substantial popularity in light of its applicability to real-world tasks across computer vision, natural language processing (Goodfellow et al. 2016). protein structure prediction (Senior al. 2020 and challenging games such as Go (Silver al. Typically, the data for these learning tasks takes the form of either grids, sequences, graphs or their combinations. The tremendous efforts on customizing neural network structures (Krizhevsky al.] 2012 Kiros
et al. 2015 Hochreiter & Schmidhuber 1997) and learning strategies (Sermanet et al. 2018 8 Oord al. 2019) to explore the data-specifie properties underpin the success of modern DI. these
domains. Following the same design philosophy, we wish to capitalize these advancements to develop a customized neural network and self-supervised learning strategy to tackle the crucial and timely challenge of traffic flow modelling in IP networks.
1.1 TRAFFIC FL.W MODELLING IN IP NETWORKS
An IP network is a communication network that uses Internet Protocol (IP) to send and receive messages between or more devices such computers, mobile phones. The messages could be general application data such as video, emails or control signals of any connected devices. When sending the messages from a source of encoded messages into set of IP packets, The packets then travel through communications links and routers switches in given routing path sequentially, thus forming the traffic flows in an IP network (Hunt) 1992). As one of the most commonly used global networks, the IP network provides the majority of such data transmission services to support today's Internet applications such as video streaming, voice-over-IP, and Internet of Things. Therefore, a good understanding of the behaviorial patterns of the underlying traffic flows plays a crucial role in network planning, traffic management, well optimizing Quality of Service (QoS, e.g., transmission rate, delay). This challenge is termed as traffic flow modelling and is fundamental to IP networking research and practice. However, the high nonlinearity, randomness and complicated self similarity (Leland) et al. 1994) of these traffic thwart extensive traditional analytical and learning models, particularly at fine- -naainddddd time scales, such as traffic flow modelling at sub-second level.
destination, the source device encapsulates the bit chunks
Consider the illustrative example in Fig. [ which depicts multiple packet flows with shared for- warding nodes and links in their routing paths. The sender of each flows streams data packets the

---

Under review as a conference paper at ICLR 2021
Sender
10 Cépp lnk
Recelver
Packets
Buffer
Sender 2
R3
Sender
Receiver
R2
R6
Figure 1: Traffic flows in IP networks.
receiver at a dynamic sending rate, which is determined according to many factors such as its rate demand, existing traffic loads, available link bandwidth, and etc. The packets usually experience various delays on the journey due to actions such as forwarding processing, link transmission, pack- et queueing. For example, when the sum rate of Sender 2 and 3 exceeds 10 Gbps. the router R2 R2-R4 will hold off and cache the arriving packets in their buffers until the links from R2 to Receiver become free, causing what is known as the queueing delay. The extent of these delays depends multiple factors, including the amount of traffic going on, the capacity of the router's 's output queue, link bandwidth etc. The random establishment, interaction and termination of massive flow con- nections give rise to network dynamics. This illustrates the complexity of traffic flow modelling in IP network even for the simple example. This challenge exacerbated when the traffic loads are running over 100 Gbps and in network with significantly larger size in practice.
1.2 MOTIVATING FLOWNN BASED TRAFFIC FLOW MODELLING
flow pattern can be defined as anything that follows a trend and exhibits some kind of regularity, e.g., distribution, periodicity etc. The modelling of traffic flow patterns can be done mathematically or by the use of data-driven learning algorithms. We argue that developing a customized FlowNN in the context of IP traffic flow modelling is important in two aspects: 1) improving the performances of supported network applications from the accurate modelling towards the behavioral patterns of traffic flows in IP network, particularly at the time scale of sub-second level; 2) providing an exciting new "playground" and neural network model for the DL community to solve real- world-motivated research challenges by deeply combining its structure and working mechanisms. Next, make the following two clarifications.
Why not using traditional mathematical models. The past decades have seen numerous traffic models proposed to mathematically model the traffic characteristics of networks (Gebali) 2015). For example, extensive studies use the Poisson model characterize the traffic by assuming the arrival pattern between two successive packets follows Poisson process. Considering the heavy tailed distri- bution and burstiness of the data-center traffic, recent work [Benson et al. (2010) models the traffic arrival pattern as a log-normal process. To capture the temporal patterns and make predictions ac- cordingly, Seasonal Autoregressive Integrated Moving Average (SARIMA) is exploited in (Ergenc) & Ertan 2019) to model the traffic time series. These analytical models may generate outputs that are easier to interpret, but are bonded to the specific working circumstance and assumptions. More importantly. these statistical models function at coarse time scales of hours and assume relatively moother traffic patterns. However, as reported in many practical traffic measurements in c.g. [enssn] et al. (2010; 2011);; Greenberg et al. (200), most flows last less than minute. This implicates tasks requiring trattic models at finer-grained time scales are beyond the capability of these traditional models.
Fig. plots the traffic traces we collected from a practical backbone network network-WIDE WIDE which shows the realistic traffic patterns when the packet flows sampled by different time scales. The long time-scale plot in Fig. shlssows clear a "tide-effect" associated with daily human activities. By contrast, the traffic traces Fig. [alget more noisy and difficult to recognize obvious patterns when they are counted by millisecond.
http::/mant wide. ad. http.aaa..p/aaa.iiiim.mma htm1
2

---

Under review as a conference paper at ICLR 2021
Data rate in Mbps
100
2020103/01 02:00 2020/10/01 06:00 UTC (remolation: Ohour)
Maps
60
200
2000
4000
6000
8000
10000
CO/D4
03/97
00/10
09: 13
09:16
09:19
09
09:22
09:25
09/28
Time in millisecond
00:00
00:00
(a) Timescale: millisecond
(b) Timescale: hour
Figure 2: Time series of practical traffic traces from WIDE backbone network binned by two differ- ent time scales.
Why not using existing neural network models. When put in the context of data-driven learning, the traffic flow modelling problem can be reduced to the representation learning task. If treating
the traffic flows as the general spatio-temporal data, extensive existing neural networks fit such task, including Convolutional Neural Net (CNN, (Mozo et al.) 2018), Graph Neural Net (GNN, (Rusek) jet al. 2019)), Recurrent Neural Net (RNN) as well as their variants and combinations (e.g.. STHGC- N ((Kalanderet et al. 2020), STGCN YYu et al. 2018), and Xiao al. (2018); Polson & Sokolov (2017) 7); Cui et al. 2020); Guo et al. 2019): Lin et al. (2019). ) The customized designs to the data specific properties make the success of these existing models, such the convolutional operation to capture the spatially local correlations in CNN and the aggregation operation to extract the adjacent link cor- relations in GNN, and on. As human-engineered industrial system with clear system structure and working mechanism, the IP network creates domain-speci spatio-temporal data correlations, which are difficult to capture for the incumbent spatio-temporal models if without any modification. One of the most important difference is that the spatial data in IP networks is not only correlated to other spatial data the same point in time, but also able to directly influence the future realizations of correlated locations with strict order (i.e., the Spatio-Tena Temporal Induction effect as we will disclose later). Moreover, these existing studies only target at a coarse- graiineg timescale above minutes or even hours. Models at a sub-second granularity, as FlowNN functions, require deeply combining the spatio-temporal data trends, as well as the system structural knowledge and working mechanism.
1.3 OUR CONTRIBUTIONS
We claim two critical contributions: 1) we formulate the crucial traffic flow modelling problem in IP networks as the representation learning task in deep learning, and develop a customized neu- ral network-FlowwNV and the associated Induction Operation to extract the domain-specife spatio- temporal data correlations in IP traffic flows. To the best of our knowledge, this is the first work to design customized neural network and learning strategy by deeply combining the IP network struc- ture and working mechanism. The Induction Operation also makes it the first time for data-driven learning model able to infer the data features at a millisecond granularity, which is usually treated as the 'noise' region by existing coarse- grainner models; 2) we report the state-of- ohe-the-art perfor- mance over the baselines in different type of practical network applications, which provides a good testament of our model.
SPATIO-TEMPORAL INDUCTION
By stacking the networking feature timeseries sampled at all the nodes that a flow passes through, the IP traffic flow data can be organized in the form of high-dimensional tensor time series, as shown in Fig. 3al The feature (denoted as x²/,.) could be the average flow rate (i.e., the amount of packet bits received in each unit measurement time) at each node or the average per-hop packet delay etc. The routing path herein constitutes the most significant attribute for the generative process of each IP traffic flow. This creates many peculiar properties in such flow data. For example, for a flow with routing path [1-+-+122] 12] in Fig, Baj the current data at node 4 was originated from the history data at its predecessor node 1, but delayed by at leas the link delay A. These data will also flow to its successor node after certain delays. This shows that the flow state at a node is physically
Packet processing and queueing will also impose extra delay.
3

---

Under review as a conference paper at ICLR 2021
Data Mbpss
80
1042
70
Link dolyy
Spatall nodils altnng rosting
Flow Wensor A
50
40
differenze
Fastuud (..g feos
posies
0.trud
each
adday
530
$35
540
545
Time in millssecond
(b)
Figure 3: (a) IP traffic data in the form of flow tensor time series. The S-shaped curve characterizes the data correlationship along the spatio-temporal dimensions; (b) A sample of data rates of flow and the rate difference recorded at two neighboring nodes.
driver by the past flow state at its predecessor node. Therefore, such time-resolved flow data not only tells us who related whom but also when and in which order relations occur. This forms the S-shaped data correlation pattern, as exemplified in Fig. [3a] An accurate modelling of the flow data requires attention such domain-specific data properties, which are absent in the majority of existing learning models if not all of them.
Fig. 3b plots sample of flow traces at two neighboring path nodes from the WIDE dataset. We can observe that an exceeding of data rate at node I over node 2 for some duration (e.g.. T1) will always induce a subsequent duration T2 when the data rate at node 2 is greater than that at node 1. Moreover, the cumulative amount of data bits the two nodes forward in such two durations are almost same, as indicated by the rate difference between the two nodes at the bottom of Fig. 3b This illustrates that there is stronger correlation among the data in these two durations, and the future realizations at T2 are subject to the constraint of the states at Ti by local flow conservation between the two nodes.
In analogy to the concept of Electromagnetic Induction in Physics, in what follows, we introduce the Cratio-Temprrd Induction (STT) effect in IP traffic flows. Accordingly, learning model is proposed to model the network traffic flows at a fine-grained timescale.
Definition 1. Temporll Induction is the production of the temporal evolutions of a flow from the history spatial patterns at its correlated locations.
The STI builds physical relationship between the spatial and temporal patterns of IP flow data. This provides a more accurate interpretation to the underlying data generating process than the trending information manifested by the data itself. Such induction effect is created by the IP network structure and working mechanism, which is preserved when the flow data is sampled at any timescale in practice.
Next, we develop an Induction Operation to concretely capture the S-shaped data correlation in IP flow data and propose what call FlowNN to generate the desired traffic model for IP networks.
FLOW NEURAL NETWORK
As aforementioned, after the packet flows leave the source devices, they may experience delays from many factors (e.g., link transmission, packet processing, queueing etc) during the journey throughout the routing path. Accordingly, each node in the path will record a transformed time-series view of the same source flow due the transmission and forwarding actions from the network. This
'hhe exotic competition from other co-located flows also drives the flow evolution in part. This can be included by appending the aggregated coflows as an auxiliary feature.
'Flow conservation is the network property that except the source and sink, the total incoming flow of node is same as outgoing flow or incoming flow of successor node the routing path no packet is lost. "Electoomagnetic induction is the production of electric currents across a conductor in changing magnetic field, which determines relationship between electricity and magnetism.
4

---

Under review as a conference paper at ICLR 2021
Flow
Flow N/
Node
Node n
Support downstream
upplications
Initial embodding Inyer
Path-level Coding
Flow
Node-level Induction
Rendmn Layer
classification OJ
x
Induction layer
Flow control!
prediction
QoS
f=1,2//
(=0,
Induction layer
inference
Input
FlowNN
Output
Figure 4: Framework of FlowNN. With the raw feature data æy.,, as inputs, FlowNN extracts both the path-level and the induced node-level feature patterns as learned feature vectors, which can then be used for different application tasks by changing the Readout layer.
implicates the existence of similarity among these time series since they are from the same source flow. Such path- wide similarity reflects the shared forwarding policies all path nodes follow, as well as the original behaviors of the source flows at the sending side, such as the dynamics of user demands. On the other hand, the flow patterns recorded at each node may end up being different from the patterns at the sending side, depending the experienced networking actions. With all these in mind, we attend to such similarities and differences by two key blocks blocks-Path-levee Coding and Node-level Induction, and build the framework of FlowNN Fig.
Path-level Coding: This block constructs an encoding network to extract the patterns that can repre- sent the state of the whole path. Such path-level patterns capture the similar part of behaviors within the time series received from all path nodes and will be the base to derive the evolved patterns at each path node. Technically, we tested both the very recent Siamese et al. 2019)) and GNN based encoders (Rusek et al. 2019) to extract the path-level patterns. Sur- prisingly, they are not outperforming direct encoding by the Gated Recurrent Unit (GRU) with all raw node data as inputs. This stands to the reason that the raw data from the correlated locations conveys the direct guidance for the near future realization at node. The operations in above model- e.g., SumPooling or node aggregation, actually destroy such data correlations and introduce more 'noise'. More details are provided later experiments. Consequently, in the following, we directly apply a GRU to encode all the raw inputs.
(Maron et al. 2020 Ye
Node-level Induction: This block encodes the individual patterns at different nodes conditioned on the path-level patterns from the source flows. Looking into how flows induced in Fig. 3b| can observe an explicit data correlations subject to local flow conservation among the correlated data sequences. Next, we propose the Induction Operation and the associated Contrastive Induction Learning to extract such data correlations.
As illustrated in Fig. 3bb the induction process operates any two consecutive durations like Ti and Tg. which can be Identified through the rate differences between two neighboring nodes. The induction performed separately for each path node, as shown Fig. For ease of exposition, we denote the data sequences at all path nodes in Ti and the data sequence at the induced node in T2 as and T. respectively. Then, the Induction Operation performs the conditional induction function f(T))) to capture the correlations between source sequence S and target sequence T. The function S(T)S) can be approximated by specialized Contrastive Induction Learning self-supprvised manner as follows.
Contrastive Induction Learning: discussed in Path-level Coding, we first take the outputs of path-level feature patterns, [ heen, as the initial coding states of source sequence S. A recur- rent encoder (c.g., G (Chung et al. 20144) is then applied to generate the context of S with (1'ss hen as inputs. Conditioned on the context e, a GRU decoder can be then used to produce the
'Not that the encoder is not limited to GRU, but any type of recurrent models can be used. More recent recurrent models stacked with modules e.g., attention blocks could help improve results further.
5

---

Under review as a conference paper at ICLR 2021
state codes, {h4 J1E22 for the target sequence T. With the state codes of both S and T at hand, the key of the induction function S) is to force the learned state codes of S and T to express the data correlations subject to local flow conservation. Inspired by Contrastive Predictive Coding (CPC) (Oord et al. 2019), we learn the mappings from S to T in a way that maximally preserves the mutual information of the source and target sequences defined follows:
((T;c)= = ET( cog log P(T]c)
-000)-E(77 T,0
((T)
where p(-) is the probability function.
In analogy to CPC, we rely on a contrastive loss function to train the model. As the practice of contrastive learning (Chen al. 2020, Oord al. 2019), first construct training batch of N₂ sequence samples, Φ = {Ti" 72, 7N, F. 7 is the positive sample from the conditional distribution (((cc) for the target sequence T and other No sequences are the negative samples from the original distribution p(T). Accordingly, we define the following contrastive induction loss C to optimize the mutual information in Equation T Tas follows:
,(T
L1 = EE log ΣT,EO [(
(2)
where f(-) is a score function proportional to the ratio p ((T(c)/ (T)pp())...)))))r())) ). As the way done in CPC, this can be technically implemented by using simple log- bilinear model:
f(T,, c) = Σ exp(trans(hh)) kt7z)
(3)
TE2>
where trans[hy] is the transpose of the state coding h from the initial embedding layer in Fig.
The induction loss Cy defines categorical cross-entropy of classifying the positive sample correctly. Optimizing L, will result in high similarity between the true embedding codes h4; and the induced
state codes 7 The above induction pattern is learned by directly comparing the target coding with the codings of randomly sampled negative samples in latent space in a self-supervised manner. Such process requires no direct labels the output space, which makes easier generalize the induced knowledge diverse applications in IP networks.
4 EXPERIMENTS
We validate the efficiencies of FlowNN on two crucial networking application tasks in IP networks- flow rate prediction and end-to-end delay inference.
Datasets. We conduct experiments on two publicly available flow datasets WIDE and NumFabrid WIDE are the realistic traffic traces collected from real-world network environment, and NumFabric are the synthetic traffic widely applied in recent networking researches (Nagaraj et al., 2016: Lei cet al. 2019). The network topology includes core nodes, leaf nodes and x 16 hosts organized in accordance with Fae- network architecture (Al-Fares et al.) 2008). We sampled the flow rates of 50 flows as well as the associated aggregated flow rates at each nodes at the time scale of 1ms for total length 18432ms. Each flow traverses routing path with 5 nodes. We found that the flow tensor time series of such length are enough to learn a stable flow pattern at a sub-second granularity.
Baselines. We compare the proposed FlowNN with the following baselines: 1) Seasonal Auto- Regression Integrated Moving Average (SARIMA) (Hyndman & Khandakar] 2008 Wang et al.) :006;; 2) GRU (Chung al. 2014); 3) multiple GRUS (multiGRU); STHGCN (Kalander al. 2020 and 5) STGCN (Yu et al. 2018). Particularly. GRU encodes and predicts each time-series sequence stand-alone without any reference to the information from other spatial nodes. In con- trast, multiGRU uses separate GRUs to encode each time-series sequence but predicts with Fully- connected Neural Network (FNN) by jointly concatenating the codings of sequences from all spatial nodes. STHGCN is the recent work for networking traffic analysis, which uses graph convolutional
TThe data can he generated by running the simulation code released in htttpa:/knagara: bittbicket..or//nnagraajja.e

---

Under review as a conference paper at ICLR 2021
0.3
Norallzedd flow rate
RlokNN
Validation MSELosS
0.4
Naive
1.5
Ground truth
SARIMA
GRU
multiGAU
0.6
STHGCN
STGCN
FlowNN
-0.5
-1.
0.1
Validatioo time point millisecond
20
40
60
100
epoch
(b)
(a)
Figure (a) Validation MSELoss; (b) Normalized flow rate predicted by different models.
network (Kipf & Welling 2016) and GRU to predict the spatial states of all nodes each time. Finally, STGCN is a widely applied model for transportation traffic analysis, which is build upon the spatio-temporal graph convolution.
Implementation Details. We use Adam (Kingma & Ba 2014) with a learning rate of le-t. All hidden dimensions set to 128 and the layer size for GRUs is 3. We train sampled time series window of length 256. A batch of N, 10 is used to draw the samples for the contrastive induction loss in Equation 2 The initial embedding layer in FlowNN performed for each node data by separated FNNs. We first pre-train the FlowNN with the contrastive induction loss until convergence and then fine-tune the model with MSE loss (Equation 4 for different application tasks.
Evaluation metrics. The following metrics are adopted for evaluating quality of prediction ý with the ground truth y.
Mean squared error:
MSE-
0.)² (ys
(4)
1=1
y: 91
Relative absolute error:
RAE
(5)
2/
1=1
Di=1 (y): 2) (0: 1)
Correlation coefficient: Corr
(6)
2i=1 (y:
72
)2
Σ
(0: 5)
2i=1
where ÿ and i are the mean values of y and ÿ.
4.1 PATH-WIDE VIEWS MATTER FOR IP TRAFFIC FLOW MODELLING!
We first use the one-step-ahead prediction task to demonstrate that path-wide views of FlowNN are more informative than the widely used single-node view or graph-level views in the literature.
Fig. shows the validation MSE loss across training epochs for the one-step-ahead prediction task. Naive solution directly uses the previous observation as the next-step prediction. The results of Naive and SARIMA provide a good testament to the prediction difficulty for traditional non- sewtiiscooongggnnuaaate
networking prediction task. We can see that the loss of SARIMA is still high as 80% of the Naive solution. However, the losses of all DL based solutions are less than 20% of Naive. This shows the powerful advantages of the deep learning principle in difficult tasks. Table I shows the test performances of different methods. Specifically, GRU holds a single node view and learns the time series of each node separately and the achieved loss is only 22.5% of SARIMA. By contrast, the other DL solutions harvest the information from all path nodes to predict (i.c., the path-wide views) and the loss are remarkably reduced further against GRU. As discussed in Path-level Coding, STGCN and STHGCN inherit the graph aggregation operation,
DL solutions

---

Under review as a conference paper at ICLR 2021
Table I: Test performance comparison of different models on one-step-ahead prediction task
NumFabric RAE
WIDE RAE
Model
MSE
Corr
MSE
Corr
Naive
0.5074
1.6818
0.7462
1.0017
5.7229
0.2619
SARIMA
0.4043
1.5122
0.7924
0.7164
4.4304
0.2979
GRU
0.0911
1.3441
0.9514
0.4634
3.8070
0.7038
multiGRU
0.0409
0.9406
0.9784
0.3972
3.7637
0.7532
STGCN
0.0738
0.8510
0.9608
0,4351
2.9718
0.6791
STHGCN
0.0716
0.8930
0.9587
0.4341
3.2210
0.5620
FlowNN
0.0297
0.7960
0.9843
0.3523
3.6223
0.7703
Tonki
TanRN
0.91
STG2
STGCH
0.35
0.33
6401
20
DTU
Teet Coop
0.02
6.90
28,
0.001
1.0
2.5
1.0
3.0
1.0
1.0
15
2.0
(a) Test MSE
(b) Test RAE.
(c) Test Corr.
Figure 6: Test performances of different DL models on multi-step--aead predictions task.
their performances better than GRU, but inferior to multiGRU. Such findings discourage the spatial aggregation operation in GCN as the data correlation exploited by GCN deviates a lot from the truly S-shaped correlation in IP flow data. Finally, the proposed FlowNN outperforms all the baselines.
Fig. 5b compares the predictions by different models against the ground truth. We observe that FlowNN and multiGRU capture the trend of ground truth accurately. Benefiting from the Induction Operation, FlowNN avoids over-reliance on the far distant history as the vanilla GRU and its variants function and shows the fastest response to the dynamic changes among the traffic flows.
4.2 WIDE APPLICABILITY ON DIFFERENT APPLICATION TASKS
Multi-step-ahead predictions: We first exam the effectiveness of FlowNN on the multi-step-ahead prediction task. In this task, we first pretrain FlowNN based on the contrastive induction loss in Equation until convergence. Following the practice of the recent work for the short-term (second- level granularity) traffic forecasting in (Mozo et al. 2018). we then take the output of pre- pra-trdined FlowNN and finetune FNN based Readout layer Fig. to infer the mean of multi-step- predictions. The mean realizations the multi-step ahead are crucial to many short-term networking planning and control tasks in IP networks. Fig. 6 compares the test performances of different DL models on the NumFabric dataset. The test results shows that FlowNN outperforms all these recent baselines all metrics.
In today's IP network, it is important to guarantee that the provided QoS meets the subseribed service agreement, such end-to-end transmission delay less than 50ms for online gaming. However, it is difficult to build an accurate delay model for human experts (Xiao et al. 2018, Rusek al.] 2019) practical networks since many factors may influence the delay, including the dynamic traffic demands, packet processing and queueing etc. In this task, we apply the learned traffic flow model from FlowNN to perform the traffic-driven delay inference. Specifically, we apply the same FlowNN model pretrained in multi-step-aheal prediction task. and finetune a new FNN based Readout layer to infer the next-step delay. Table 2 shows the test performances of different models. Although pretrained on the dataset of traffic flow rates, FlowNN still achieves the best results on the task of inferring the data with different physical meaning. This shows the robustness of FlowNN across different tasks.
End-to-end delay inference:
8

---

Under review as a conference paper at ICLR 2021
Table 2: Test performance comparison of different models on end-to-end delay inference
Model
MSE
RAE
Corr
GRU
0.1364
1.8333
0,9300
multiGRU
0.1011
1.4008
0.9466
STGCN
0.1263
2.0247
0,9412
STHGCN
0.0922
1 1.4683
0,9533
FlowNN
0.0724
1.3570
0.9617
Table 3: One-step-ahead prediction performances of different models in Out-Of-Distribution test. For Same dataset lest, both the model training and test are conducted on WIDE. By contrast, in Cross dataset test, models are first trained on NumFabric, but then test on WIDE by finetuning their Readout layers.
Same datasel test
Cross dataset test
Model
MSE
RAE
Corr
MSE
RAE
Corт
multiGRU
0.3972
3.7637
0.7532
0,4272
3.7717
0.7126
STHGCN
0.4341
3.2210
0.5620
0.5236
3.3330
0.4715
FlowNN
0.3523
3.6223
0.7703
0.4173
3.9001
0.7213
4.3 GENERALITY TEST ON OUT-OF-DISTRIBUTION DATASET
In this subsection, test the model generality when FlowNN model, pretrained with the con- trastive induction loss on one dataset, say NumFabric, used to work an environment that is different from the trained one. This is performed by testing with the one-step-ahead prediction task on
dataset (i.e., cross dataset test). Specifically, we take the FlowNN model pretrained the NumFabric dataset and finetune the FNN based Readout layer performances on the WIDE dataset. As comparison, we also finetune the Readout layers of multi- GRU and STHGCN already trained NumFabric to their performances on WIDE, From the test results shown in Table we can observe that FlowNN achieves the best performances except that RAE is slightly higher. Moreover, the performances of FlowNN achieved in cross dataset test even better than the results of other baselines achieved in same dataset test (except multiGRU). This shows the good generality and robustness of FlowNN.
its prediction
Discussions: We observe that multiGRU, to certain extent, also works well in above experiments, although is inferior to FlowNN. This can interpreted by the shortcut learning, as disclosed in Geirhos al. 2020). As illustrated by the data correlation in Section the near future realization of a node is highly correlated to the path-wide values at last step. This makes a shortcut for multiGRU that a prediction by directly weighting the last-step wath value will capture such kind of data correlation, although this can not further extract the effect of local flow conservation along the routing path FlowNN functions.
CONCLUSION
In this paper. we formulated the crucial traffic flow modelling problem in IP networks, and de- velop a customized neural network-FlowNN and the associated Induction Operation to extract the domain-specific spatio-temporal data correlations in IP traffic flows. This study makes the pioneer- ing work to design customized neural network and learning strategy by deeply combining the IP network structure and working mechanism. We reported the state-of-the-arr performances for mul- tiple practical networking application tasks, which demonstrates the strength of our approach. As 'playground' for both networking and deep learning communities. the research of network intelligence still in its infancy. We hope this work will inspire more innovations in this field in future.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. A scalable, commodity data center network architecture. ACM SIGCOMM computer communication review, 38(4):63-74, 2008
Theophilus Benson, Aditya Akella, and David A. Maltz. Network traffic characteristics data centers in the wild. IMC, pp. 1-14, 2010.
Theophilus Benson, Ashok Anand, Aditya Akella, and Ming Zhang. engineering for data centers. IMACM CONEXTC, pp. 1-12, 2011.
Microte: fine grained traffic
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv:2002.05709w 2020.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS Deep Learning Workshop, 2014.
Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recur- rent neural network: deep learning framework for network-scale traffic learning and forecast- ing. IEEE Transactions on Intelligent Transportation Systems, 21(11) ((111))48834440 48944, 2020.
Doganalp Ergenc and Onur Ertan. On network traffic forecasting using autoregressive models. arXiv preprint arXiv: 1912.12220v1, 2019.
Fayez Gebali. Modeling Network Traffic, pp. 445-492 Springer International Publishing, Cham, 2015. ISBN 978-3-399-15657--6 doi: 000007/9778-339-1555777 Ttg/10.1007/9//
URL https:
319 319-1557---1111111
Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.00 NXX1:20044.0778003 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Albert Greenberg, James R. Hamilton, and Navendu Jain. V12: A scalable and flexible data center
network. SIGCOMM, pp. 1-12, 2009.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. AAAI, 2019.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
C. Hunt. TCP/IP Network Administration. OReilly and Associates: Sebastopol, CA., 1992.
Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: The forecast package for r. Journal of Statistical Software, 27(3):1 1-,,,,3133311 2008
Marcus Kalander, Min Zhou, Chengzhi Zhang, Hanling Yi, and Lujia Pan. Spatio-tempora hy- brid graph convolutional network for traffic forecasting in telecommunication networks. arX- 1v:2009.09849911, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional network- s. NIPS, 2016.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta- sun, and Sanja Fidler. Skip-thought vectors. NIPS, pp. 1-9, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. NIPS, pp. 1-9, 2012.
10

---

Under review as a conference paper at ICLR 2021
Kai Lei, Meng Qin, Bo Bai, Gong Zhang, and Min Yang. Gen-gan: A non-linear temporal link prediction model for weighted dynamic networks. IEEE INFOCOMM, 2019.
Will E. Leland, Murad S. Taqqu, Walter Willinger, and Daniel V. Wilson. On the self-similar nature of ethernet traffic. IEEE/ACM Transactions on Networking, 2(1):1-15, 1994.
Ziqian Lin, Jie Feng. Ziyang Lu, Yong Li, and Depeng Jin. Deepstn+: Context-aware spatial- temporal neural network for crowd flow prediction in metropolis. AAAI, 2019.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. ICML, 2020.
Alberto Mozo, Bruno Ordozgoiti, and Sandra Gomez-Canaval. Forecasting short-term data center network traffic load with convolutional neural networks. PLOS One, 13(2), 2018.
Kanthi Nagaraj. Dinesh Bharadia, Hongzi Mao, Sandeep Chinchali, Mohammad Alizadeh, and Sachin Katti. Numfabric: Fast and flexible bandwidth allocation in datacenters, ACM SIGCOMM, pp. 188-201, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 177.v:1807.0374455 2019.
Nicholas G. Polson and Vadim O. Sokolov. Deep learning for short-term traffic flow prediction. Transportation Research Part C: Emerging Technologies, 79:1-17, 2017.
Krzysziof Rusck. Jos Suarez- Varela, Paul Almasan, Pere Barlet-Ros, and Albert Cabellos-Aparicio. Routenet: Leveraging graph neural networks for network modeling and optimization in sdn. arXiv preprint arXiv:1910.01508, pp. 1-12, 2019.
Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, and Green Tim. Improved protein structure prediction using potentials from deep learning. Nature, pp. 706-710, 2020.
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine Time-contrastive networks: Self- supervised learning from multi-view observation. CVPR, 2018.
David Silver, Julian Schrittwiseser, Karen Simonyan, Antonoglou loannis, and Aja Huang. Master- ing the game go without human knowledge. Nature, pp. 354-359, 2017.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint ar Xiv: 19090009 2019.
Xiaozhe Wang, Kate Smith-Miles, and Rob Hyndman. Characteristic-basst clustering for time series data. Data Mining and Knowledge Discovery, 13(3):335 -344,,66,,, 2006.
Shihan Xiao, Dongdong He, and Zhibo Gong. Deep-q: Traffic-driven qos inference using deep generative network. Proceedings of the 2018 Workshop on Network Meets Al ML. pp. 67-73, 2018.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. CVPR, 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. IJCAI, 2018.
11


Reviews:
[{"review_id": "l4dt9vS_0oi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a new Flow Neural Network (FlowNN) to model the flow-structured data. The main idea is to employ the unique properties of the network traffic flows, i.e., flowing invariance and variance. \n\nOverall this seems like a nice attempt to model the flow-structured data. However, the paper should be improved in the following aspects.\n\n- Presentation: My first concern is the writing of this paper. \n1) After reading the first two sections, I was really confused about what the problem is. According to the experiment part, this paper seems to study the flow prediction task. What is the difference between this problem and spatio-temporal prediction? I strongly suggest that the authors write a section for introducing the problem statement. \n2) What is flow data? There seems no mathematical formulation of the network flow data (seemingly a tensor). \n3) More related works (e.g., flow-structure data mining, spatio-temporal GCNs, spatio-temporal forecasting) should be included to help the audiences better understand the context.\n\n- Experiments: \n1) The baselines are weak. To the best of my knowledge, in the recent 2 years, there are many papers [1, 2, 3] published at top-tier conferences (e.g., KDD, IJCAI, AAAI) that achieve much better results than the baselines included in this study. Why not comparing FlowNN with them?\n2) The proposed method is only evaluated on a single dataset, which limits its universality. If possible, more datasets should be considered.\n3) How about the robustness and stability of the proposed method? The variance of the performance should be present.\n4) In real-world applications such as traffic forecasting, multi-step ahead forecasting is more practical than 1-step prediction. It would be good to evaluate the effectiveness when predicting more future horizons.\n\n- Minor issues:\n1) The reference of GCN-GRU is wrong. As I know, Yu et al. 2018 combined temporal convolutional networks with GCN, instead of using RNN for capturing temporal dependencies.\n2) what is the meaning of the S-shaped road in Fig. 1?\n\nReference:\n[1] Wu et al., Graph WaveNet for Deep Spatial-Temporal Graph Modeling, IJCAI 2019\n[2] Guo et al., Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting, AAAI 2019\n[3] Pan et al., Urban Traffic Prediction from Spatio-Temporal Data Using Deep Meta Learning, KDD 2019\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ikClxc9IUX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes a new way to analyse flow structured data using what they call flow neural networks, which supposedly better exploits correlations between different connected nodes at different time-points. The approach is tested on the public dataset NumFabric with better results than the compared to approaches.\n\nAlthough I believe I get the rough outline of what the authors are proposing, there is simply too many unclear aspects of the work for it to be published at ICLR. Prior work, especially the benchmarked methods, also seems to be mostly tangentially related, e.g. covering general methods for time-series or graph analysis and not specific methods for traffic analysis.\n\nQuality\nA single dataset is used for comparison and it is unclear how relevant the baselines are, as there appears to be a lot of prior work on deep learning for traffic analysis, some of this is cited, but not benchmarked against. See also the additional references [1, 2], and the large number of works citing them. It would be good if the authors could explain why their chosen set of baselines are relevant.\n\nClarity\nThe work is hard to understand. An incomplete list of things follows below.\n\n- Crucially I don't clearly understand the STI effect as explained in definition 1.\n- Why do you specifically mention self-driving Tesla cars, wouldn't this be relevant for other self-driving cars as well?\n- What do you mean by more than 1 million flows, that is, how do you count flows?\n- \"As the advent of many advanced machine learning (particular deep learning), intelligent flow analysis tools are gaining more momentum to proceed\", sentence does not make sense.\n- I do not understand the paragraph starting with \"Additionally, in contrast to the learning analysis of formation-agnostic natural objects, ...\" What is a formation-agnostic natural object? How can a network system enjoy rich domain expertise? How can network traffic experience something? Etc.\n- temproal -> temporal\n- was originated -> originated\n- \"This essentially discriminates\", does not really make sense to me in this context.\n- Sentence starting with \"In the path-constrained propagation process as doing in above IP traffic flows,..\", does not make sense to me.\n- What is APP?\n- It would be good to use the same number of decimal points for the result of each method in Table 1.\n- \"However, all deep learning based solutions achieve several times to tens times better accuracy.\", I do not see this in the figure.\n- Please use consistent naming schemes, see \"uniGRU\" and \"univarGRU\".\n- In summarizing the results (4.1), I think the word improves is a bit unclear when talking about reduction in loss. E.g. the loss is only xx percent of yy would be more clear.\n- \"This is benefited from the STI effect\" -> \"This is benefiting from the STI effect\"\n\nOriginality\nThe proposed work may very well be original and novel. I am just not convinced.\n\nSignificance\nIn its present form, I doubt it will have much influence.\n\n[1] Polson, Nicholas G., and Vadim O. Sokolov. \"Deep learning for short-term traffic flow prediction.\" Transportation Research Part C: Emerging Technologies 79 (2017): 1-17.\n[2] Cui, Zhiyong, et al. \"Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting.\" IEEE Transactions on Intelligent Transportation Systems (2019).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8xJC38u5rOo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n- The paper claims to discover a universal “spatio-temporal induction (STI) effect” in network traffic flows, and developed a model FlowNN to learn representations of flow-structure data. However, the STI effect was not clearly explained and the problem is not well formulated, making it hard for readers to understand the value of this work.\n\nStrong points: \n- Modeling flow-structure data and internet traffic flows is an interesting problem.\n- The paper motivates the problem well.\n\nWeak points: \n- The writing of this paper is very confusing and unclear. \n- The problem was not clearly formulated or articulated.\n- The descriptions of the model and STI effect are confusing.\n\nRecommendation: \n- I strongly recommend a reject. While the topic is interesting, the paper is hardly understandable in its current form. The contributions cannot be well understood and evaluated. Improving clarity will make it a much stronger submission in the future.\n\nComments & questions:\n- What exactly is the task? It was never clearly stated in the paper. Only mentioning “Learning the representations” or “regression tasks” is unclear and confusing.\n- What is the training and test data split? How does the approach generalize to unseen networks or flows?\n- The definition of STI is unclear. In Definition 1, the paper writes “Spatio-Temporal Induction defines the synchronized map process to produce the future/spatial evolution between neighboring nodes.” This does not define the STI effect. What exactly is the “synchronized map process”? Different network flows process packets at different paces, so how are they synchronized?\n- How can the representation (Fig 3a) scale with the number of nodes in the network? You would need one such 3D matrix for a flow. The paper evaluates with 28 nodes in total (Sec. 4), but the real internet is orders of magnitude larger (billions and millions of flows).\n- The meaning of Fig 4. is unclear. “a spike at a upstream node in Fig. 4a spurts the flow at downstream nodes in certain subsequent time windows.” However, it’s very hard to see that from the figure. It seems like all flows are fluctuating at different time steps.\n- Many claims seem to be wrong or not supported. A few examples:\n  - “a encoding net with shared weights is learnt to extract the common patterns shared in all received inputs.” Share weights imply extracting “similar” patterns across flows, how can the design extract “common” patterns?\n  - “This reconstructs the evolution patterns of the APP source flows that drive the evolution processes at all nodes along the path.” -> How exactly?\n- The meaning of many sentences are unclear. A few examples:\n  - “the flow rates be- tween two neighbouring or any pair of nodes in the routing path will become larger interchangeably than the other so that the flux is conserved among nodes.”\n  - “This not only experimentally confirms the S-shaped spatio-temporal correlation present in the flow-structured data, but also further confines the correlation with the explicit (partial) flow conservation.”\n  - “Induction is operated on … by the paired induction operator.”\n- Many grammar issues. A few examples:\n  - “Data can be organized compact and complete”\n  - “propagation process as doing in above IP traffic flows”\n  - “While any encoder and decoder can be used so long as we can backpropagate through it.”\n\nSuggestions\n- Writing-wise, it will be helpful to simplify the wording and be direct and specific. \n- It’ll be helpful to ask people not familiar with the project to read the paper and get feedback.\n\n==== Updates after the response ====\n\nI appreciate the authors’ effort in updating the manuscript. The new manuscript has significant changes, but still many questions are left unanswered. Thus, I’m keeping my rating and recommendation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4LPO5nWkRng", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe goal of this study is the 1-step prediction of flow rate in flow networks. They first define a “spatial-temporal induction effect (STI)” and claim it to be the universal property of flow networks. Their main contribution is their proposed “flow neural network” which is based on the STI effect and a combination of GCN and GRU architectures.  According to authors, the novelty of their work lies in the fact that they consider the spatiotemporal features of the flow network simultaneously, whereas the previous works only consider them separately. \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \n\nStrengths of the paper: \n\nThe raised concern over the fact that considering the time series of the node alone does not capture the full complexity of the flow dynamics is valid and an interesting direction to pursue. \n\nTheir proposed model achieves a better validation loss than the benchmarks \n\nThe paper does a decent job in pointing out the flaws of the previous models. \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \n\nDetailed Review: \n\nMajor concerns: \n\nFor a paper that heavily relies on empirical results (as mentioned throughout the paper), there is a serious lack of various experimentations to prove the followings:\n\n- The claimed universality of spatiotemporal induction (STI) effect, \n\n- Considering the spatial-temporal features separately (i.e., spatial-spatial and temporal-temporal) indeed results in considerably poorer performance in different settings, \n\n- The practical benefit of the proposed architecture tested on different real-world networks from different domains (to comply with the title and the claim in the paper that the proposed method is extensible to other flow networks besides IP networks, as well). \n\nThe authors claim the inefficiency of previous studies (some of which are based on strong mathematical theories) on the same domain (e.g., Harchol-Balter, 2013; Ciucu & Schmitt, 2012; Ciucu et al., 2019) without proof and do not consider these works in their baselines. In fact, many of the selected baseline belong to relatively old studies that are not necessarily SOTA for the flow prediction task right now. There are numerous recent studies on network flow prediction (e.g., traffic flow, crowd flow, etc.) combing the spatial-temporal features of the system through (e.g., these two papers to name a few:  \n\nhttps://www.sciencedirect.com/science/article/pii/S0968090X19301330?casa_token=_HkqrohsGwEAAAAA:_qI_xcFnpiFkXGzqxhXNTfEx4jJA_EoAN673pBimQezdlIFCw_79EWTcB9XIhpSiCGwhJ9W_Pyx9 \n\nhttps://www.aaai.org/ojs/index.php/AAAI/article/view/3892) \n\nI suggest authors develop their related work section further and use more SOTA baselines in their experiments. Also, choose more than one dataset from different domains for validating the proposed model. \n\n%%%%%%%%%%%%% \n\nThe definition of “spatial temporal induction” is rather vague and does not clarify what STI effect means, both theoretically and intuitively. There are no proof on the “universality” of this effect for all flow networks and the two properties listed in page 4 are neither proven nor connected to the rest of the paper. What is the significance of STI and why do we care about its two properties (if they indeed hold). \n\n%%%%%%%%%%%%% \n\nThe paper is rather tough to read. There are instances of using abbreviations before they are defined and mathematical notations with variables that are not properly defined (ref. Page 5). \n\n%%%%%%%%%%%%% \n\nHow are the flow timeseries defined as (in-flow? Out-flow?). On the same note, how does the ordering of the subtraction affect figure 4.b (e.g. changing 130-136 to 136-130 while keeping the rest of subtractions the same). \n\n%%%%%%%%%%%%% \n\nThe figures in general are not well connected to the text. Majority of them include details that are not explained in the text (or the caption) and a number of them are hard to interpret. For example, \n\nFigure 1.b: what is the red box for? \n\nFigure 3.a: what is x_ijt in terms of flow rate (I can only attribute it to in-flow and out-flow which does not seem to be used in this study)? And what does X_iit imply? How is the blue curve (called S curve) obtained in the figure? \n\nFigure 3.b: what is to be inferred from this figure? \n\nFigure 4.a and 4.b: Are these based on real data or they are only toy examples? If it is the latter, then it cannot be used to prove the existence of STI effect. If it is the former, what is the data? It is hard to attribute meaning to the graphs without knowing the context of the data. \n\n Moreover, in figure 4.a, shouldn’t the spike in the source node 41 reach to neighboring node 130 with some time delay (the same for other nodes as well)? \n\nFigure 5: Again, what is X_ij?  What is the direct output from “flowing invariant coding” used for? \n\nFigure 6: I find it hard to interpret the caption from the figure. \n\nFigure 8: what is to be inferred from the red boxes? \n\nI suggest authors increase the font and quality of their figures and enrich their captions to convey the intended message with more clarity. \n\n%%%%%%%%%%%%% \n\nThe problem statement and novelty of the study is not clear from the introduction section.  I suggest authors clearly define the problem and point out the novelty of their work in comparison with current studies from the beginning. \n\n%%%%%%%%%%%%% \n\nPlease elaborate on how the flow structured data is a “radically new data type” (ref. Section 1) \n\n%%%%%%%%%%%%% \n\nThe results in figure 7 are called “validation” loss. Does that imply these nodes have been included in the training or hyperparameter tuning of the model? If yes, it is more interesting to know how model performs for flow prediction of unseen nodes. \n\n%%%%%%%%%%%%% \n\nIt is interesting to know how the flow prediction task for K-step ahead will be.  \n\n%%%%%%%%%%%%% %%%%%%%%%%%%% \n\nMinor Concerns: \n\nThe paper has to be self-sufficient, so for equation 1 and 2, more explanation is needed instead of referring readers to the previous work.  \n\n%%%%%%%%%%%%% \n\nSome typos and grammatical errors that need to be fixed. Some examples: \n\nSection 1: “tens to tens thousand of nodes”, “particular deep learning” \n\nFigure 3 caption: repeated “the” \n\nSection 2: “any pair of nodes in the routing path will become larger interchangeably than the other so that the flux is conserved among nodes” \n\nSection 3: “a encoding”, “a RNN”, “While any encoder and decoder can be used so long as we can backpropagate through it.”, “Analogy to CPC, we define the following induction loss...” \n\nPage 6: the two lines before section 4 do not seem to connect correctly with the previous page. \n\n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \n\nQuestions: \n\nPlease answer the questions raised in “detailed review” above. \n\n \n\n \n\n \n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The revised paper is a solid improvement.  However, all reviewers and I find that there are still a number of issues that prevent the paper from being acceptable at the current stage.  For example, some important parts are still unclear, especially the definition of STI effect.  The observation of STI effect requires more theoretical or empirical investigation, in addition to a toy example.
Decision: Reject