Functional Relation Field: A Model-Agnostic Framework for Multivariate Time Series Forecasting
Authors: Bing Yu
Ting Li
Jianguo Li
Bin Dong
Zhanxing Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Functional Relation Field
Spatio-Temporal Forecasting
Constraint Optimization
Multivariate Time Series

Abstract: In multivariate time series forecasting, the most popular strategy for modeling the relationship between multiple time series is the construction of graph, where each time series is represented as a node and related nodes are connected by edges, i.e. spatial-temporal graph neural networks. The graph structure is either given apriori or learned based the similarity between nodes. However, the relationship between multiple time series is typically complicated, for instance, the sum of outflows from upstream nodes may be equal to the inflows of downstream nodes. Such relations widely exist in many real-world multivariate time series forecasting scenarios, yet are far from well studied. In these cases, graph might only be a crude description on the dependency between nodes. To this end, we explore a new framework to model the inter-node relationship in a more precise way based our proposed inductive bias for graphs, Functional Relation Field, where a group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series. These learned functions are versatile: they can then be used to discover the underlying graph structure by identifying the most relevant neighbors of the target node; and on the other hand, the learned functions will form a “field” where the nodes in the backbone prediction networks are enforced to satisfy the constraints defined by these functions. The experiment is conducted on one toy dataset to show our approach can well recover the true constraint relationship between nodes. And two real-world MiniApp calling traffic and road network datasets are also considered with various different backbone networks. Results show that the prediction error can be reduced remarkably with the aid of the proposed functional relation field framework.

Under review as a conference paper at ICLR 2023
FUNCTIONAL RELATION FIELD: A MODEL- FRAMEWORK FOR MULTIVARIATE TIME SERIES FORE- CASTING
Anonymous authors Paper under double-blind review
ABSTRACT
multivariate time series forecasting, the most popular strategy for modeling the relationship between multiple time series is the construction of graph, where each time series is represented as a node and related nodes are connected by edges, i.c. spatial-temppral graph neural networks. The graph structure is either given apriori or learned based the similarity between nodes. However, the relationship between multiple time series is typically complicated, for instance, the sum of outflows from upstream nodes may be equal to the inflows of downstream nodes. Such relations widely exist in many real-world multivariate time series forecasting scenarios, yet are far from well studied. In these cases, graph might only be a crude description on the dependency between nodes. To this end, we explore a new framework to model the inter-node relationship in more precise way based our proposed inductive bias for graphs, Functional Relation Field, where group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series. These learned functions are versatile: they can then be used to discover the underlying graph structure by identifying the most relevant neighbors of the target node; and on the other hand, the learned functions will form a "field" where the nodes in the backbone prediction networks are enforced to satisfy the constraints defined by these functions. The experiment is conducted on one toy dataset to show our approach can well recover the true constraint relationship between nodes. And two real-world MiniApp calling traffic and road networks datasets are also considered with various different backbone networks. Results show that the prediction eтror can be reduced remarkably with the aid of the proposed functional relation field framework.
INTRODUCTION
Multivariate time series forecasting has surged recently due to its strong expressiveness of the spatio-temporal dependence among the data and its enormous popularity in vast application ar- eas, such as the prediction of urban traffic, com- puter network flow, cloud micro-services calling flow, and rigid body motion, name a few (Li et al. 2018 Yu et al. 2018 8 Bai et al. 2020 Yan 2018 Liu et al. 2020). The most popular and straightiorward strategy for modeling the relationship between multiple time series the introduction of graph, where each time series represented as a node and related nodes are connected by edges. This particular inductive bias for multivariate time series prediction results in the so called spatial- graph neural networks (Yu al.) 2018). The graph structure is either given apriori (e.g. in traffic flow prediction, each road as a node has connected roads forming the graph.) or learned based the similarity between nodes Yu et al. 2019 Bai al. 2020 [Shang et al. 2021). However, in practice, the relationship
Torget
Node
Oa1 Tire-saiis sinilaity
OU Nods constaiats
Comparison between traditional graph- based modeling and our approach. (a) Graph struc- ture learning with time-series similarity; (b) Func- tional relation field, modeling inter-node functional relationship in a linear form, where is the target node and (A, B, C) represent dependent nodes.
Figure 1:
al.

---

Under review as a conference paper at ICLR 2023
between multiple time series is typically complicated. For instance, there often exist constraints among the nodes, ranging from the equality between the inflow and the outflow for a node in a traffic network to the geometric constraints of the rigid body motion. Such relations widely exist in many real-world multivariate time series forecasting scenarios, yet are far from well studied. In these cases, graph might not be sufficient for characterizing the dependency between nodes.
As a remedy, in this work, we explore a new framework to model the inter-node relationship in more precise manner than graph, Functional Relation Field (FRF), where group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series explicitly. These learned functions are versatile: first they can then be used to discover the underlying graph structure by identifying the most relevant neighbors of the target node; and on the other hand, the learned functions will form a "field" where the nodes in the backbone prediction networks are further enforced to satisfy the constraints defined by these functions. As illustrated in [ the left panel shows the traditional graph neural networks assuming similar time series have edge connections, while our framework on the right panel models the dependency between nodes through a functional relationship. e.g. a linear form to enforce the constraints between the flows of target and dependent nodes. In our framework, we mainly solve the following two issues: (i) How to learn the functional field? We need to select the dependent nodes that have a relationship with the target node, and express the constraint in functional form; (ii) How to guarantee the constraints satisfaction? The (functional) constraints relationship should be maintained in the predicted output in both training and test process.
To address these issues, we propose a two-stage approach that can discover the functional relations (i.e. constraints) from data and further integrate the constraints seamlessly when forecasting the multivariate time series. Specifically, we first train a neural network with a selected target node as its output and all the other nodes as dependent variables (i.e. the imput of this neural network), and identify the most relevant dependent nodes based on this trained network. We then re-train it to learn the relationship among the target and the discovered relevant nodes. Next, we incorporate these functional constraints into the network backbones by imposing them to the predicted output during both training and test process. More precisely, the output of the network could be guaranteed to satisfy the constraints by utilizing the constraint-satiffee transformation and loss minimization. We compare the proposed approach with SVM, fully connected networks, fully connected LSTM, and five backbone models (i.e., STGCN (Yu et al. 2018), AGCRN (Bai et al., 2020), Autoformer (Wul et al. 202II. FEDformer (Zhou et al. 2022). SCINet (Liu et al. 2022)). Experimental results show that our approach significantly improves the performance over the original network backbones and other baseline models.
RELATED WORK
Univariate time series forecasting. Recently, much research focuses on time series forecasting with deep learning models due to their powerful representational capability and prediction performance, including feed-forward neural network, RNN (Rumelhart) 1986) and its variants LSTM (Hochreiter
& Schmidhuber 1997) and GRU (Cho et al. 2014). The transformer architecture and its variants
( (Vaswani al. 2017 Simm al. 2020 Zhou al., 2021 Child et al. 2019 Lim et al., 2020 Li
jet 20.9 Wu et al. 2021 Zhou et al.) 2022 also made much progress on univariate time-series
forecasting on learning long-range dependence. In order model the trend and seasonality of time series in an interpretable way, N-beats (Oreshkin et al.] 2020) network that stacked very deep full- connection network based on backward and forward residual links has improved the multi-horizon prediction accuracy significantly. Moreover, DeepAR (Salinas et al. 2020) and Deep State-Space Model (DSSM) (Rangapuram et al. 2018) stack lulti-layer LSTM network to generate parameters of one-step-ahead Gaussian predictive distributions for multi-horize prediction.
Multivariate time series forecasting. Spatio-temporal graph neural networks (Yu et al. 2018 Chen et al., 2019 Pan et al., [2021] Li et (0220) have been proposed to model the spatial correlation and temporal dependency in multivariate time-series. Apart from capturing the temporal dependence, these methods further model the spatial dependence among all time series via graph neural networks, leveraging the information from the neighboring time series to help forecasting the target one. It is well known that an informative graph structure is important to the graph time series forecasting. Therefore, many algorithms (Bai et al., 22000) (Soo et al. 2016 Shang et al., 202T) were proposed to discovery the underlying graph structure. AGCRN Bai al. 2020 assumed the graph structure
2

---

Under review as a conference paper at ICLR 2023
unknown and adopted adaptive approach to learn the embedding vectors for all nodes, and then replaced the adjacency matrix in graph convolutions with function of the node embeddings. However, the similarity graph calculated with the learned node embedding is dense and continuous graph instead of sparse and discrete graph. Therefore, GTS (Shang et al. 2021) formulated the graph structure learning problem probabilistic graph model to learn the discrete graph through optimizing the mean performance over the graph distribution.
Different from the existing multivariate time series prediction methods, AGCRN Bai et al. 2020) (with a fully connected graph) and STGCN (Yu et al. 2018) (with a given graph), we consider more precise way, i.e. functional relations as constrants, learn the connection between time series. The new inductive bias expressed by these functional relations can be applied to different backbone networks help recover the graph structure and act as regularization in both training and test process.
2 METHODOLOGY: FUNCTIONAL RELATION FIELD
Multivariate time series forecasting. Suppose we have N time series {= with length T, written compactly as X E RNxT Each time series can be denoted as a node, where Ii, E R for each node and time step t. 2, E RR is the time slice of X at the f-th time step. The multi-step forecasting problem of multivariate time series can be formulated as predicting the future M frames of the multivariates given the last H time slices:
11..1..... ae = argmax P(I Mr - #+ (.. zxt),
(1)
where [1t+1, ... a5 and (ūt-1.-.. ((+M) represent the true and predicted values the future time steps, M is the number of future steps. Note that here we use y denote the output so to differentiate it from the input x.
Forecasting with functional relations. In many real-world scenarios, the relationship between multiple time series is typically complicate, graph might not be sufficient for modelling their depen- dency, particularly for the cases values of multivariate time series at each time step are subject to some intrinsic constraints. Existing methods have not incorporated these constraints into their models. In this work, we intend to show that models with the account of constraints (expressed with functional relationship) are superior to those without constraints in terms of prediction performance. As an example, suppose that the flow in a computer network satisfies the homogeneous linear constraints, at each time step 4, the following linear constraints hold for slice Zy:
(2) where A E M is a matrix that is constant across time. In other more complex cases, the intertemporal. Here, concentrate
Az; = 0, Vt,
constraints can be non-homogeneous, non-linear, time-invariant constraints that is not intertemporal. As such, the constraints can be described by set of functions ff with size m, i.c functional relation field,
f= f2, Sm). fi(x.) = 0, Vi, VL.
Based on the constraints defined above, we consider the following constrained multivariate time series prediction problem,
(ū++) 1, 1..,, 0++M] arg max P([ We+ M n -, x.)) s.t. f((1+t)) = O, 1 SSTS M, 1sis m.
(4)
However, most real-world scenarios, neither the functional form F nor the specific weights variables involved in the constraints are given, and one of our objectives is to extract such information from the data and solve the problem (4). We now elaborate the functional relation field for multivariate times series prediction in the following.
The schematic diagram the proposed framework is depicted in Figure including two parts. The first part displayed Figure 2 a) shows how we learn the functional relations, i.c. the constraints between nodes. Assuming that the constraints are unknown, we aim to find the constrained nodes and the specific functional form for these constraints. The constraint function in this paper is
3

---

Under review as a conference paper at ICLR 2023
(3) Functional Relation Field
Thii comatraine networt
Betnining the finctiona rehiiim netwwoks
Independent rodie
Consrrmt nodes set and colovate todes M.
Louned Sanction celasio
(b) (b)Applynng Fanctional Relation
Eygr
8
Learned function
Input
Training Phoser Crrttrin-SSriffcttiio loss
relation: TOO =
Fig(10)
Bockione
Profect catgue
Function: 10 -
Network
Tolling Phane: Corstraint-atiisnaai transformationn Eq(15)
Outpat Layer
Profict oupp
Pres
Constraint sutiflil ouput
Perr
Figure 2: The schematic diagram of functional relation field framework. The two subfigures denote the two stages: (a) The training data is employed to discover the nodes in each constraint function and these functions are expressed by constraint network; (b) The learned constraints are incorporated in the backhone models (cf. Section 2.2 in three complementary ways so as to improve the forecasting performance.
approximated by a neural network, named as functional relation network or constraint network. After training the functional relation network, we can identify the most relevant neighbors and produce a more informative graph structure. Then we can proceed to integrate the learned constraints into the backbone graph neural networks for multivariate time series prediction, as shown in Figure Figure[(b). We enforce these constraints to the output of spatio-temporal graph neural networks during both training and test phases. For the outputs of the networks, we add a constraint-satisfin transformation layer during the inference process such that the outputs strictly satisfy the constraints. Altogether, we refer to the proposed framework as functional relation field-enhanced spatio-temport graph networks (FRF-STG). It is following, we will describe the two stages including learning functional relation network and how to apply the constraints induced by the functional relation between nodes in more details.
and can be applied to different backbone graph networks. In the
2.1 LEARNING THE FUNCTIONAL RELATION NETWORK
We start with discussing the first question: how to learn the unknown constraints (i.e. the functional relations) from the multivariate time series data? As demonstrated in Figure [[a), we assume that there exists a constraint for each node. We first discover the relevant nodes involved in these constraints and then express the constraint functions via neural networks.
Identifying constrained nodes and their relevant nodes. Here we consider a simplified case where the functional relation between nodes can be formulated as:
Ft 9 9 ()), Vt.
(5)
i.e. for each target node i, we use a constraint network gi to approximate the function relation taking all the remaining (N - 1) nodes input. We then train the constraint network to predict the value of
the i-th node with the loss function
Cpred,()) = | - 1l
where 2..5 and ,t,; represent the estimated and observed values of node i at time step t. Second, a threshold cerr is set, and treat Ty as a constrained node if both the training and validation error smaller than corr. Otherwise, x4 is unpredictable with the other nodes, indicating it has weak dependency with other nodes. Then, to identify the most relevant nodes set A,, for target node i, we introduce the sensitivity of input change to the output for the trained constraint network, measured by the absolute value of the partial derivative:
0g
j *
Zzi,j
We calculate the average gradients over the training and the validation set for node j. Then, we specify another threshold Cgrad here and consider the node j as the most relevant node of target i j.g larger than Cyrad- Besides, if the cardinality of NG is larger than the scale threshold J. we further shrink N; by only keeping the top-J nodes with the largest di.j-
Retraining the functional relation network. Since we filter out the irrelevant nodes for the discov- ered constrained node F4 it is necessary to re-train the constraint network using the relevant nodes in N2 as inputs, denoted as Tr,,G = {xu,l) E Ng).
(8)
4

---

Under review as a conference paper at ICLR 2023
Regarding the architecture of the functional relation network 9t, adopt simple attention based structure for each node i, described as follows.
(9) where C4 is the attention weight vector generated from the relevant nodes 2t,V, and 2t,5 is the reconstructed input with the constraint nodes. Others alternatives for designing the functional relation network is also possible.
3 = CofftazzMMLR (FLL.) ft, a..=a,,o Tt.No
2.2 APPLYING THE CONSTRAINTS
The constraints learned by the functional relation network are versatile. A naive usage construct meaningful graph structure by drawing edges between the identified target and its dependent nodes. Secondly, we propose to incorporate the learned constraints into the backbone prediction network in both training and test process through constraint-sale sattuiinttstsssaation loss minimization and constraint- satisfaction transformation, respectively. Both of them are used to guarantee that the constraints are maintained in the outputs of the backbone network.
Constraint satisfaction in training phase. We expect the output the backbone network, ý (M++1, 0t++2..., Dt++ J. satisfy the learned constraints that could reveal the underlying structure of the multivariate time series. A straightforward yet effective way of implementing the constraint satisfaction is loss minimization over the functional relation network based the output of the backbone prediction network,
N M
(FFF
-
(10)
(=1==1
Therefore, the overall loss function for training the backbone prediction network include two terms, Liotal L(j.)) 3) (CFRF())
(11)
where À is tradeoff coefficient for balancing the supervised term and constraint satisfaction.
Constraint satisfaction in testing phase. Furthermore, although the constraints are fully utilized during training, there is guarantee that the constraints hold for the outputs during the inference process. Therefore, it is necessary to perform construint-satisacccii transformation on outputs of the prediction networks.
Let us first consider the linear constraint Ax, = O, Vt. Suppose that ÿ {ū++1 9t+2... (++M) } and y (V++1. 9t+2: .... 3t++M 3 denote the predicted output of the backbone network and the ground truth, respectively. To make the output Dt+r to satisfy the linear constraint, we can project the predicted output onto the hyperplane Ar; 0 as Hi- with closed-fore solution,
T D++ AT (AAT)) ABJ+T.
(12)
On the other hand, for non-linear constraint set (yy = (J), (.m.).....(())
(n)) = 0, where each constraint fe(y) represents y; - 9. (V.,N; 0, there are analytical solutions, but can solve an optimization problem with nonlinear equality constraints, i.c. finding the nearest projection point on the plane fyy 0 given the reference point üztr for T I.
min 10++7
oc+r 112, s.t. f( = 0.
(13)
simple approximate method for solving this constrieed quadratic programming is to 0f as the Jacobian matrix. Assuming ut+r ss Vetr. conduct iterative projections. Denote äg closed to the surface f(x) = 0. We derive the first-order Taylor expansion of f(x:) jt+r
f(x) ~ f(i(+)) + J.. (x - Ô(t7).
(14)
Equating f(x) to zero with x = Detr yields
Ue+r 9 J(JT J) -
(15)
Then we can repeat the above transformation several times (e.g. number of projections K 10 times used in our experiments) until the constraints are well satisfied by evaluating whether F(z) = En [x]) is small enough.
5

---

Under review as a conference paper at ICLR 2023
2.3 FUNCTIONAL RELATION FIELD-ENHANCED
GRAPH NETWORKS
In this part, we integrate the proposed functional relation field framework into five representative backbone models, STGCN (Yu et al.] 2018), AGCRN (Bai et al. 2020), Autoformer (Wu al.) 2021). FEDformer (Zhou et al., 2022) and SCINet (Liu et al. 2022) to boost their prediction performance, referred as FRF-STGCN. FRE-AGCRN, FRF-Autoformer, FRF-FeDformer and FRF- SCINet, respectively. In the first stage, we learn the functional relation network, based on which the most relevant nodes can be identified. And the resultant graph structure could be used for the five backbone networks. In the second stage, we enforce the learned constraints in the training and inference process, as described in Figure 2
Since different backbone networks has their own specific design, we need adapt FRF to these backbones. For the constraint satisfaction of output, in AGCRN and SCINet, the networks produce all the prediction results at multiple time steps in one batch, and therefore, the constraint-satisfeed transformation is applied to the prediction at each time step respectively for times as described in Eq. [5). For STGCN, we apply the above transformation sequentially to each future time step, obtain the transformed predictions, and then feed the predictions to STGCN to produce the predictions at the next time step. We repeat this procedure until finish the multi-step forecasting task.
Algorithm 1: Training and inference of functional relation field
Trained function relation networks f. hyper-parameters À and K. Output: constraint-sate satisfied output B
Input:
// Training Phase;
repeat
Forward on backbone network to get ÿt
b on training dataset;
Backk-propagate with the loss Ltotal Eq. and run Adam.
constraint-sutiisaction loss
until stopping criteria is met;
// Inference 2hase;
Forward on the trained backbone network to obtain de+r
P on dataset;
for k in K do
Calculate g0++ by Eq. 15
constraint- transformation:
end
3 EXPERIMENT
In this section, we conduct experiments on five datasets including one synthetic graph dataset, two real-word MiniApp calling flow datasets and two traffic flow datasets to demonstrate the effectiveness of FRF on learning the underlying relationship between nodes and boosting prediction performance of these backbone networks. The code for reproducibility is attached the Supplementary Materials.
The baseline models. We first compare our framework with two traditional forecasting models including Historical Average (HA) and Support Vector Regression (SVR). Then, we also conduct experiments on two classical univariate time series prediction models, including Feed-Forward Neural Network (FNN) and Full-Connected LSTM (FC-LSTM (Sutskever et al. 2014)). We select the widely used graph time series model STGCN YYu et al. 2018), AGCRN (Bai al. 2020), and the univariate time series forecasting models based on transformer architectures Autoformer (Wu et al. 2021). FEDformer (Zhou et al. 2022) and another state-of-the-art univariate prediction model SCINet (Liu) et al. 2022 as our backbone networks. We refer the readers to the supplementary materials for the detailed experimental settings.
3.1 DATASETS AND SETTINGS
Binary tree dataset. We first generate an artificial graph time series dataset. The graph structure for this dataset is complete binary tree with 255 nodes. For each leaf node i, its value is noisy sinusoidal wave across time, 2i, = A2 sin 2xx + +),, where n4,e ~ (0.955, 1.05). We sort all leaf T.
nodes from left to right an increasing order of their periods. For a non-leaf node p. we denote its left and right child as l and r. We further set the value of node to be the geometric mean of its two children and r, Pp. = /Zl, Ty We sample one point every 5 minutes, so there are 288 points per day. We generate the data for 40 days, including 30 days for training (i.e.. 30 288 8640 time points), 5 days for validation, and 5 days for testing. We intentionally design this dataset since has true graph structure between different time series and the constraints between nodes are explicit,

---

Under review as a conference paper at ICLR 2023
and thus it is a suitable testbed to compare the superiority of FRF over those without FRF. In the experiments, for the backbone with FRF, we assume the constraints are unknown and learn them using the proposed method in Section 2.1
MiniApp calling flow dataset and 2. These two datasets are real-word flow data from two popular online payment MiniApps, attached in the Supplementary Materials. For the two MiniApps, there are N = 30, 23 filtered pages linking to each other in the calling process, which produces visiting request flow from one page another. constituting graph with N 30, 23 nodes. We aggregate the flow with averaged value every 5 minutes for each node, so there are 288 points per day. For the first Mpn we collect 21 days of data, including 15 days for training. days for validation, and days for test. For the second one, 24 days of data are collected, including 18 days for training, 3 days for validation, and 3 days for testing.
PEMSD4 and PEMSD8 traffic datasets. This benchmark dataset is popular for multi-variate time series prediction, describing the traffic speed in San Francisco Bay Area with 307 sensors on 29 roads hhttps://papers: theode. www coom//atastt///eeeeett
The other one consists of 170 detectors on roads in San Bernardino area (bttps::/paperrwwtthcttt htt.://aattteteeeeeeaaeeee
Settings of constraint network and hyper-parameters, For the architectures of the constraint network, we compare two a 4-layer MI.P and a self-attention network, and the results show the latter more effective. We measure the constraint relationship with MAPE, where the large MAPE indicates the time-invariate constraint is weak. Specifically, the MAPEs for BinaryTree, MiniAPPI, MiniApp2, PEMSD4, PEMSD8 datasets are 0.10, 0.008, 0.01, 0.02, 0.07 respectively. The larger MAPE means the weaker constraint relationship. therefore the proposed FRF model is applicable to backbone network only when the MAPE of constraint network small. In addition, we only tune the parameters of FRF while keeping the other hyper-parameters setting the same as backbone networks.
3.2 RESULTS
Overall performance Table summmarzess the performance of all the compared models on the five datasets, including the proposed FRF approach coupled with STGCN, AGCRN, Autoformer, FED- former and SCINet, denoted as FRF-STGCN and FRF-AGCRN, FRF-Autoformer, FRF-FEDformer and FRF-SCINet, respectively.
For the binary tree dataset, we predict the future 12 time steps and evaluate the performance in terms of three metrics (MAE, RMSE, MAPE). Since the underlying true constraints are known, we report the experimental results of our models with both true and learned constraints, denoted as "T" and "L". We can observe that deep learning-basee models typically outperform the traditional ones, as expected. Furthermore, the proposed functional relation field can further improve the performance of the original backbone models. Regardless of the differences between the two backbone networks, FRF can consistently improve the prediction accuracy for both of the backbones. indicating that the FRF framework could be potentially applied wide variety of backbones.
For the two MiniApp datasets, we omit the metric MAPE since the scale of data changes dramatically across time such that MAPE fails characterize the performance of different models. Due to the еггог accumulation problem for multi-step prediction in STGCN, the performance of this model pales in comparison with its non-iterative counterpart. As a result, we only report the results of the non-iterative version of STGCN. Since the underlying true constraint relationship between nodes are not available, we only report the FRF with learned constraints. We can easily observe that augmentation of the proposed FRF can consistently boost the performance of the five backbone networks. Specifically, FRF improves STGCN by 36.3% and 6.9% on the two datasets, also improves AGCRN by 14.6% and 7.0%, respectively.
For traffic datasets PEMSD4 and PEMSD8, one particular reason we choose SCINet as the baseline is that the reported results can achieve state-of-the-art prediction performance on this task. We can observe that even relying on such a strong baseline, FRF framework can still improve its performance of with a margin 0.6% and 0.3% on both datasets, respectively. For other backbones, again see that FRF further improves the prediction performance, showing the effectiveness of FRF as a monel-agnostic framework.
Learning the relationship between nodes. We further test whether FRF could discover the under- lying true constraints between nodes. First, investigate whether we can reliably estimate the
7

---

Under review as a conference paper at ICLR 2023
Table I: Model performance on BinaryTres and MiniApp datasets. "(T)" and "(L)" represent the models with true and learned constraints, respectively. Bold font is used to show the advantage over
backbones. "-" represents that the ground truth of the functional relationship is not available.
Binary tree
MipiApp
MiniApp 2
Methods
MAE
RMSE
MAPE
MAE
RMSE
MAE
RMSE
HA
12.64
19.19
22.16%
3.97
9.77
11.02
35.23
SVR
8.71
14.00
15.85%
2.56
7.06
6.83
21.68
FNN
5.77
10.04
9.73%
2.09
6.26
5.43
16.84
FC-LSTM
17.08
22.83
32.40%
2.05
4.08
8.14
19.64
STGCN
2.65
5.82
4.36%
1.90
5.26
4.50
14.
FRE-STGCN (T)
2.40
5.68
3,94%
FRF-STGON (L)
2.50
5.71
4.12%
1.21
3.32
4.19
11.11
AGCRN
2.56
5.77
4.25%
0.41
1.17
1.43
3.79
FRE-AGCRN (T)
2.30
5.54
3.87%
FRF-AGCRN (L)
2.37
5.57
4.00%
0.35
0.92
1.33
3.39
Autoformer
8.54
13.16
15.41%
1.03
2.79
2.69
6.85
FRF-Autoformer (T)
8.34
12.79
14.75% %
FRF-Autoforer (L)
8.34
12.83
14.38%
0.77
2.18
2.46
5,70
FEDformer
8.54
13.24
15.45%
0.60
1.80
2.08
5.13
FRF-FEDformer (T)
8.10
12.80
14.40 %
FRF-FEDforner (L)
8.29
12.99
14.79%
0.58
1.76
2.03
4.98
SCTNet
5.43
9.37
9.46%
0.52
1.51
1.78
3.88
FRF-SCINet (T)
5.36
9.34
9.53%
FRF-SCINet (L.
5.37
9,27
9.43%
0.47
1.34
1.71
3.65
Table 2: Model performance on two traffic datasets, PeMSD4 and PeMSD8. "_" represents the value is too large and thus ignored.
PEMSD4
PEMSD8 RMSE
Model Type
Methods
MAE
RMSE
MАPE
МAE
MAPE
STGCN Yu et al. FRGCT
2018
21.61
35.25
13.84%
17.28
27.19
11.14%
20.70
33.90
13.46%
16.46
26.05
10.68 %
Multivariate
AGCRN Bai et al.
2020
19.81
32.58
13.18%
16.52
26.12
10.53%
FRF-AGCRN Autoformer FRF-Autoformer
19.59
31.85
13.08 %
16.04
25.28
10.30%
21.42
34.09
18.49
28.78
21.24
33.93
18.23
28.66
FEDFormer
21.59
34.23
18.52
29.23
Univariate
FRF-FEDFormer
21.29
33.82
18.15
LC et al. 2022
19.27
31.27
11.91%
15.71
24.60
10%
FRI-SCINet
19.15
31.09
11.80 %
15.67
24.57
10 %
3.2
3.2
3.2
3.0
AGCRN
3.0
AGCRN
3.0
AGCRN
2.8
FRF-AGCRN (L)
2.8
FRF-AGORN (L)
2.8
FRF-AGCRN (L)
2.6
2.6
FRE-AGCRI (T).
2.6
FRE-AGCRI (T)
2.4
2.4
2.4
2.2
2.2
2.2
2.0
2.0
2.0
0.1%
0.5%
1.0%
5.0%
0.001
10
Influence of Eur on BinaryTree
Influence of on BinaryTree
Influence of K on BinaryTrre
Figure 3:
Performance comparison of three kinds of hyper-parameters including eerr, AR and K binary tree dataset with the SOTA backbone AGCRN.
target node given the values of constraint nodes. To be exact, we compute क.,; (((x)N))) and compare ε with Tz,i in terms of MAPE, For the test data of the synthetic binary tree, the resulting MAPE 0.399%. Note that the MAPE of AGCRN or STGCN reported in Table D around 4% without considering the constraints. Therefore, using the learned constraints can well regularize the predictions given by the original network backbones as well further improve the forecasting performance. On the other hand, we compare the performance of the proposed algorithm when using the true and estimated constraints, showing the results Table I We can observe that the performance based on both the true and estimated constraints is almost the same, indicating that the constraints are accurately learned. Additionally, we visualize the learned constraints by connecting each constrained node with their most relevant neighbors graph, shown in Figure 4 The structure of the binary tree well recovered. although some extra edges are involved.
Hyperparameters Sensitivity. FRF enhanced model introduces additional three kinds of hyper- parameters including validation error threshold Earr. the loss tradeoff coefficient and the number of output transformation K. Therefore, conduct paramees sensitivity experiments on binary
8

---

Under review as a conference paper at ICLR 2023
The learned constraints of the Binary Tree Dataset by connecting each constrained node with their most related nodes. We J = 2,33, 3, 4 for every node to plot this figure, so there lack of connections when J = 2 and some redundant connections when J 4.
Figure 4:
Table 3: Ablation study on explicit graph, constraint graph learned from constraint network and constraint satisfaction components using STGCN as the backbone network.
Explicit Constraint Graph
Constraint
PeMSD4
PeMSD8
MiniAppl
Graph
Satisfaction
MAE
RMSE
MAE
RMSE
MAE
RMSE
1
21.61
35.25
17.28
27.19
1.90
5.26
2
21.26
35.13
16.79
26.61
1.48
3.87
3
20.70
33.90
16.46
26.05
1.21
3.32
Table 4: Ablation study on constraint-satiisaction loss minimization and constraint-satiifaction transformation. The backbone AGCRN is used for Binary Tree, MiniAppl and MiniApp2; and
SCINet for PEMSD4.
FRF Training
FRF Inference
Binary Tree
MiniAppl
MiniApp2
PEMSD4
#
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
1
X
X
2.56
5.77
0.41
1.17
1.43
3.79
19.27
31.27
X
2.51
5.68
0,40
1.19
1.38
3,66
19.20
31.16
3
X
V
2.33
5.58
0.40
1.14
1.45
3.74
19.22
31.15
4
V
2.30
5.54
0.35
0.92
1.33
3.29
19.15
31.09
tree dataset using backbone AGCRN as shown in Fig 3 We observe that the performance slightly improves when the cerr increases due to more constraints are discovered, while the performance decreases with large Eery because of the introduced noise. Even more, the FRF enhanced model performs worse than backbone network when ferr 5.0. Consistently, FRF enhanced model performs better when À 0.1 and worse than backbone with large A. For the K, the larger K improves the backbone more significantly than smaller & because iterating more times makes the non-linear constraint optimization problem more accurate.
Ablation Study. We first conduct an ablation study on the constraint graph learned from constraint network using the STGCN as backbone network in Table We can observe that the constraint graph performs better than explicit graph extracted from prior knowledge on both traffic and MiniApp datases. In addition, for backbone networks without explicit graph structure such as AGCRN and SCINet, we investigate the effectiveness of constraint-satisface satisfaction transformation as shown in Table 4 finding that both the two components contribute to the forecasting performance. Specifically, for the backbone network AGCRN which achieves the state-of-the-art performance on binary tree dataset, FRF enhances the backbone by 1.95% in training phase and by 9.0% in inference phase, while the combination of two components improves the performance by 16. in total.
loss minimization and constraint-
CONCLUSION
In this paper, we have proposed to enhance the multivariate time series forecasting with new inductive bias, function relation fieild (FRF), which is model-agnostic. FRF can discover the intrinsic graph structure, as well improve flow forecasting performance by applying constraint function relationship to the output in training and testing phases. The constraints learned by FRF can be incorporated into existing backbone networks, consistently improving the prediction performance. Experimental results show that the proposed FRF framework can reliably learn the constraints from the time-series data and restore the graph structure. Moreover, these constraints turn help improve the prediction accuracy by notable margin, regardless of the diversity of the network architecture in different backbone models. We expect that this FRF inductive bias could be potentially employed in other multivariate settings beyond times series scenarios.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In In proceedings of 34th Conference Neural Information Processing Systems, 2020.
Cen Chen, Kenli Li, Sin G. Teo, Xiaofeng Zou, Kang Wang, Jie Wang, and Zeng Zeng. Gated residual recurrent graph neural networks for traffic prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):485 Jul. 2019.
Rewon Child, Scott Gray, Alec Radford, and Hya Sutskever. Generating long sequences with sparse transformers. 2019.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN mucoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1724-1733 Doha, Qatar, October 2014, Association for Computational Linguistics. doi: 10.3115/v1/D14-11799
orq/D14-1179
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 973):1335- 1997.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04689 2018
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-X Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time se- ries forecasting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL 67755055c00224422222222222388883
http://prrooeddnn...e...eeire wttprr/pp..ccpppee////////////// pdf
Ting Li, Junbo Zhang, Kainan Bao, Yuxuan Liang. Yexin Li, and Yu Zheng. Autost: Efficient neural architecture search for spatio-temporal prediction. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '20, pp. 794-802, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379998 doi: (0.1144533394448.3403122 URL //cttps://di...oo 000 110.0..111111333441183 33115/3333344866633222 3403122
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu, Diffusion convolutional recurrent neural network:
Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
net URL httpps://pppeeeeeeeeeee httpl//wwwwe///ounnteeeeeee forum? SJ.HHUWJJGW...
Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable
time series forecasting. 2020.
H. Liu, J. Lafferty, and L. Wasserman. The nonparanormal:
Semiparametric estimation of high dimensional undirected graphs. Journal of Machine Learning Research, 10(10), 2009.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022, 2022.
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In CVPR, 2020.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
Zheyi Pan, Songyu Ke, Xiaodu Yang, Yuxuan Liang, Yong Yu, Junbo Zhang, and Yu Zheng. Autostg: Neural architecture search for predictions of spatio-temporal graph. In Proceedings of the Web Conference 2021, wWW '21, pp. 1846-1855, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/344238 3.449816. URL
htttps://do..oggg http://6....rr/r101..1///322222. (4486
10

---

Under review as a conference paper at ICLR 2023
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL htttpps://prcceeeinngg/ nwwrips wttpcccppppp./01..1.. 2 8/118//flle 0 04g39556667700222200366666622222 Paper pdf
D. E. Rumelhart. Learning representations by error propagation. 1986.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- Journal
casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 181- 1191, 2020. ISSN 0169-2070. doi: http:///bboooo///....o/fffirooot 0011. URL https / htt.://eeeeacccc..oooooooeeeeeee
com/ ott.a//c......cccee.eeraarooo//r
Y. Seo, Michal Defferrard, P. Vandergheynst, and X. Bresson. Structured sequence modeling with graph convolutional recurrent networks. 2016.
Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In International Conference on Learning Representations, 2021. URL https
//openreview
Jaak Simm, Adam Arany, Edward De Brouwer, and Yves Moreau. Expressive graph informer networks. 2020.
I. Sutskever, O. Vinyals, and Q V. Le. Sequence sequence learning with neural networks. In In proceedings of 34th Conference on Neural Information Processing Systems, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz. Kaiser, and Illia Polosukhin. Attention all you need.
Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL http:ss://ppoceeeeinsneeeeee wttparpppp.ccpappe////////////// (ew.ee/44...eeeeeooooooooooeeeed
pdf
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting, In Advances in Neural Information Processing Systems, 2021.
S. Yan, Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton- action recognition. In AAAI, 2018.
B. Yu, M. Li, J. Zhang, and Z. Zhu. 3d graph convolutional networks with temporal graphs: A spatial information free framework for traffic forecasting. 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI, 2018.
H. Yu, S. Wu, L. Xin, and J. Dauwels. Fast bayesian inference of sparse networks with automatic sparsity determination. Journal of Machine Learning Research, 21(114):1-54 2020.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting, In The Thirty- Fifth AAAI Conference on Artificial Intellience, AAA/ 2021, Virtual Conference, volume 35, pp. 11106-1115, AAAI Press, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022.
11

---

Under review as a conference paper at ICLR 2023
A PERFORMANCES ON MORE BACKBONES
GTS Shang et al. 2021) The discrete graph structure learning model learns graph structure among multiple time series and forecasts them simultaneously with a GNN. There are two differences between GTS and our proposed FRF. On one hand, GTS performs prediction under GNN paradigm which is model-specific while FRF model-agnostic applying the function field to forecasting loss optimization. On the other hand, existing studies including AGCRN and GTS construct the graph based on the time-series similarity, while the FRF is the first proposed to exploiting the the constraint function relation to enhance the multi-variate time-series forecasting. We conduct experiments on Binary tree, Miniapp1 and Miniapp2 datasets using the opensource code thttps://ge thub. chub chaoshanges/ CTS. git) shown in table demonstrating that FRF can also improve the forecasting performance on GTS. The code of FRF-GTS and the running log is released in the supplementary material.
NRI Kipf et al. 2018). The neural relational inference (NRI) model is an unsupervised model that learns to infer interactions and forecasting with Istm. We conduct experiments Bi- nary tree, Miniappl and Miniapp2 dataset using the opensource code (https: /gi thub. c sthhnfetaya /NRI ggtt). The results NRT network in table showing that there is large margin from the SOTA backbone AGCRN Bai et al. (2020).
Table 5: Performance comparison of FRF enhanced GTS and NRI Networks on Binary Tree and MiniApp datasets. FRF-AGCRN is the SOTA model.
Binary tree
MiniApp
MiniApp
Methods
MAE
RMSE
MAPE
MAE
RMSE
MAE
RMSE
NRI Kipf et al.
2018
22.77
30.15
39.58%
2.50
6.89
8.04
16.92
GTS Shang et al.
(2021
5,85
9.19
8,87%
1.92
2.32
3.88
7.26
FRI-GIS
5.67
8.21
7.85%
FRF-GTS (L.
8.33
8.08%
1.77
1.84
2.70
5.14
FRF-AGCRN ( SOTA))
2.30
5.54
3.87%
0.35
0.92
1.33
3,39
B EXPERIMENTAL SETTINGS
The error threshold. For the binary tree dataset and MiniApp calling flow datasets which have strong constraint relationships, we set Eery = 0.01 to filter the constaint nodes. However, for traffic dataset PEMSD4 and PEMSD8 with relative weak constraints, set carr 0.025 achieve the best performance. The sepee-pprameters sensitivity experiments of cerr on PEMSD4 and PEMSD8 datasets shown in Fig
20.2
18.0
AGCRN
AGCRN FRF-AGCRN
17.5
20.0
FRF-AGCRN
17.0
MAE
19.8
MAE
16.5
19.6
16.0
19.4
15.5
19.2
15.0
0%
2.5%
5%
7.5% 10% 12.5%
0%
5.0% 7.5% 10% 12.5% Influence of Eerr on PEMSD8
2.5%
Influence of Eerr on PEMSD4
Figure 5: Performance comparison of cerr on PEMSD4 and PEMSD8 dataest on backbone AGCRN.
The function relation graph. Note that for the real datasets, the graph structure is not given in advance. In order to use STGCN, we adopt Gaussian copula graphical models Liu et al. (2009); Yu et al. (2020) to learn the graph structure from the data, and take the learned graph benchmark graph. For the FRF enhanced backbone network STGCN Yu et al. (2018), we replace the fixed graph structure with the learned constraint graph then achieve better performance. As results shown in table
12

---

Under review as a conference paper at ICLR 2023
can observe that constraint graph performs better than graph learned with copula graphical model.
Besides, for uni variaae backbones SCINet, Autoformer and FEDformer taking time-series rela- tionship into considcration, As well as graph model AGCRN, which have optimized with learned node embedding dynamically ignoring the origin graph, don't exploit constraint relation at graph construction stage. The function relation applied in training stage and output constraints.
The setting of J. For binary tree dataset, set / = 4 to recover the function relation shown in Fig We set J 6 for two MiniApp flow calling datasets. For traffic dataset PEMSD4 with 307 nodes and PEMSI8 with 170 nodes, achieve best performance when J 30.
4
The detailed settings at À and k. the training stage, we only tune the trade off coefficient À and iteration times K while keep all other parameters the same SOTA settings in benchmark. The detailed settings are shown in
Table 6: Detailed hyper-parameter settings of all graph time-series and univariate backbone networks on five datasets.
Binary tree
MiniApp
MiniApp 2
PEMSD4
PEMSD8
Methods
A
K
À
K
À
K
À
K
À
K
FRE-STGCN
0.1
0.01
0.01
0.001
FRF-AGCRN
0.1
0.01
5
0.01
10
0.01
5
0.1
10
ORF-Autoformec
0.01
20
0.001
5
0.001
0,001
5
0.001
10
0.01
20
0.001
5
0.001
0.001
5
0.001
10
FRF-SCINet
0.01
0.001
0.001
0.0001
0.0001
5
C VISUALIZATION OF LEARNED FUNCTION RELATION
The flow visualization of different relations. We show the comparison of learned function relation and origin relation on MiniAppl dataset in table 6 Note that, the origin relation of MiniApp is learned by Gaussian copula graphical models Liu al. (2009); Yu al. (2020). We can observe that the flows of the target node has the same pattern and scale with relevant node on learned function, while has different scale on origin graph. The results demonstrating that learned function more effective to capture the flow relationship.
target node relevant node 20 relevant node
target node
relevant node
relevant node
Flow
Flow
3003
2000
2002
4020
5002
6000
1200
2000
3000
4000
5000
8000
FRF Learned relation of target node
Origin relation target node 16
12
target node 17
700
target node 17
relevant node 25
relevant node
500
relevant node 20
relevant node
25
500
Flow
relevant node 10
300
12
200
100
FRF Learned relation of target node
Origin relation target node 17
Figure 6: Flow visualization of learned function and origin relation MiniAppl dataset.
13

---

Under review as a conference paper at ICLR 2023
D DISCUSSION ON HYPERPARAMETERS AND COMPUTATIONAL COMPLEXITY
Hypee--arameters. There are three newly-introduced hyperr-aaaaeeers including error threshold cerr, trade-off coefficient À and number of iterations K. The Eerr and À can be easily chosen based on the validation loss. And a larger K could be used to obtain more accurate optimization and achieve better performance. So, there balance between performance gain and computation. We typically set it as K = 10 which could work well for all the tasks we have considered.
Computational complexity. On one hand, the computational complexity increases in the forecasting network training caused by the K iterations of output constraint satisfaction. The K' is usually setted as small number 5 or 10, which is computationally easy. And the main time-consuming operations come from forward and back propagation of backbones rather than the output constraint. On the other hand, we need to train the constraint network for all time-series. Fortunately, the constraint network is simple two-layer attention network, which only has small number of parameters but effective enough to capture the complex function relation. For example, in MiniAppl task, each constraint network only has around 3,000 parameters, the training time is in the scale of seconds. Thus, helieve training a constraint network is very fast and does not require much computational resources. The small size of the constraint networks is amenable large-scale multi-variate time series.
14


Reviews:
[{"review_id": "zOhDXL0M25", "reviewer": "Reviewer_RLjD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of time series forecasting with constraints.  The proposed functional relation field framework is aimed at learning constraints from multi-variate time series data. Then, the authors develop the training and inference method incorporating the learned constraints. The proposed method is evaluated on both synthetic and real datasets. ", "summary_of_the_review": "The paper focuses on the problem of learning hidden constraints in multivariate time series and making use of the learned constraints in the training and inference of forecasting models. \n\nThe problem is not new. The technical contribution is incremental, given that some design choices are standard, and some seem problematic and need more clarification. \n\nSome experiment results look interesting, however, given the aforementioned weakness, it needs improvements to be more solid and convincing. ", "strength_and_weaknesses": "Strength:\n\nThis paper explores the idea of capturing hidden inter-variate constraints/relations in multivariate time series and imposing the discovered constraints on the training and inference process. \n\nIn the experiments, the proposed framework is applied to baselines to compare the performance difference. This is a valid idea. \n\nWeakness:\n\nThe problem of identifying inter-variate constraints/relations and applying them in forecasting is not new, and several works have studied it, e.g., [1, 2]. The technical contribution in this paper looks marginal, given that the technique used in this paper is standard and no news insights seem uncovered. Meanwhile, the presentation, some design choices, and evaluation have the following issues. \n\n(a) Eq.(1) seems to formulate the time series forecasting from the perspective of the probability model, while the rest of the paper follows the standard point forecast paradigm. In the probability model, the forecast is not necessarily the mode and could be mean, quantile, or interval. Eq.(1) seems disconnected from the problem of this paper. \n\n(b) Eq.(4)-(6) present the \"constraint\", which is not rigorous w.r.t. the concept of constraint. The presented constraint is essentially closer to the concept of correlation or relation, since it is simply derived from how well the other variables fit the target variable.      This is highly data or observation-dependent. Moreover, it needs a threshold to determine the set of relevant variables. For multi-variables in different value domains or distributions, finding proper thresholds seems nontrivial and would affect the overall performance.      This way of identifying  \"constraint\"  seems ad-hoc and arbitrary.\n\n(c) From Eq.(10), the constraint discovered from X seems to be applied to Y. It is a bit confusing to present this way. \n\n(d) Eq.13 - Eq.15 seems problematic. In Eq.(13), the minimization and the constraint are mutually exclusive, i.e., the minimization problem is for relaxing the constraint, and if the constraint is to behold, the minimization is unnecessary. Meanwhile, Eq.(13) is simply a  least-squares problem w.r.t. $\\tilde{y}$ and the iterative process seems redundant.\n\n(e) In the experiment, only the newly introduced hyperparameter is compared. It would be good to also show the hyperparameters in training, since they affect the end performance significantly in many cases. \n\n[1] Wu, Zonghan, et al. \"Connecting the dots: Multivariate time series forecasting with graph neural networks.\" Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020.\n\n[2] Li, Zhuoling, et al. \"Dynamic Graph Learning-Neural Network for Multivariate Time Series Modeling.\" arXiv preprint arXiv:2112.03273 (2021).\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ySYBo1YV6Dq", "reviewer": "Reviewer_dWww", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about multivariate time series forecasting with structure learning. Existing works usually assume the graph structure of the multiple time series is given or learned by \nthe node similarity. However, in some applications, the relationship between time series can be much more complicated and the graph structure is not enough. This paper proposes to\nuse functional relation field to model the inter-node relationship. Experiments on one synthetic and two real-world datasets show that the proposed method can enhance the existing \nspatial-temporal forecasting model.", "summary_of_the_review": "The paper proposes a novel method to learn the complicated structure for multi-variate time series forecasting task. However, it misses some important baseline.", "strength_and_weaknesses": "Strength:\n\nThe proposed method is novel. There are no existing papers that consider the complicated constraints relationship among nodes.\n\nThe proposed method is technically sound\n\nEmpirical performance is promising\n\nWeakness:\n\nThe paper did not compare with some baselines that perform joint forecasting and structure learning, e.g., GTS [1] and NIR [2].\n\n[1] DISCRETE GRAPH STRUCTURE LEARNING FOR FORECASTING MULTIPLE TIME SERIES, ICLR 2021\n[2] Neural Relational Inference for Interacting Systems. ICML 2018\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "doqQByxCvWl", "reviewer": "Reviewer_3BB6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method to automatically learn constraints over nodes in a graph, and apply the constraints for spatial-temporal forecasting. The major contribution is the proposed constraint learning framework.", "summary_of_the_review": "The motivation is strong, the proposed method is novel. But the writing is hard to follow, some concepts are not well-explained.", "strength_and_weaknesses": "Strength:\n1. The motivation for exploring relationships between nodes in terms of constraints is insightful.\n\nWeaknesses:\n1. The writing is hard to follow.\n2. Some parts are not clear:\n(1) what is the physical meaning of linear constraints (equ. 2)?\n(2) In equ. 3, why set f_i(x_t)=0\n(3) In section 2.2, is the target-dependent relationship bidiractional?\n(4) It seems the proposed model needs to go over all the nodes to check the target and dependent, what is the complexity of the method?\n3. \\epsilon_{grad} is a hyperparameter. But the experiment does not evaluate the effects of \\epsilon_{grad}.\n4. It seems too many hyperparameters to control the constraint learning. It may expect much more effort for hyperparameter tuning, which may jeopardize the applicability of the method.\n5. Typo: the second paragraph on page 2, \"...while our framework on the left panel models the...\" it should be\" the right panel\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CqPpgPlzYR", "reviewer": "Reviewer_n4W8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new model for multivariate forecasting with structured relationship between time series.  They propose to learn a static functional relationship between each node and every other node (where the target node/series value is equal to a function of the values of the other nodes/series), and then incorporate these functional relationships into any existing forecast model - via adding a regularization term (penalizing forecasts that deviate from the relationships) to the regular forecast loss (weighted by a hyper parameter lambda), and also via subsequently projecting the final forecasts onto the (learned) constraints.\n\nThe authors validate their approach by comparing results on several datasets for several recent forecast methods with and without the proposed functional relationship component and demonstrate consistently improved forecast error metrics using the proposed approach.  They also perform a variety of hyper parameter sensitivity and ablation analysis, and some analysis around learned functional relationships.  ", "summary_of_the_review": "Overall I found the approach to be interesting and novel and could be seen as a generalization / enhancement of two classes of approaches: hierarchical forecasting and graph-based forecasting.  For the former it can be seen as extending the functional relationship to more general kinds (rather than just hierarchical aggregates) and for the latter, enabling encoding more information about the relationships between series. \n\nPotential weaknesses include not learning the relationships end to end - so they can not be influenced by the downstream forecast modeling, and the functional relationships being too restricting and non-dynamic (as they are fixed over time).  However the latter can be controlled to some extent with hyper parameters (such as lambda), and both may be beneficial to avoid overfitting.\n\nThe major weaknesses that prevent me from raising my score higher mainly arise because of the large number of hyper parameters introduced (not just the ones mentioned but even the hyper parameters associated with the relationship network architecture and training) along with lack of clear, principled hyper parameter selection and no repeated randomized experiments to provide confidence intervals / std. dev., and the limited scalability of the method.  The latter of which is not discussed (and no run times reported).  In particular it's hard to imagine this method could be used well in practice with so many hyper parameters that must be tuned, and it would really only be tractable for a small number of time series as for n time series, 2n networks have to be trained just for the relationship learning part of the model.  \n\n  ", "strength_and_weaknesses": "Strengths:\n\n1) I find the approach to be interesting and novel - I personally have not seen this kind of approach taken before, and I feel it can be seen as generalizing a couple common classes of approaches: hierarchical forecasting - which does incorporate functional relationships but of a fixed form (hierarchical aggregates), and graph-based forecasting - which does not specify the particular relationship between nodes as is done here.  This enhances the hierarchical approach by generalizing the functional form encoded to be not just hierarchical or linear, and also generalizes the graph approach to encode more specific relationships between variables explicitly.  \n\n2) The paper was well-written and organized so it was easy to follow and understand.  \n\n3) The method is logical and sound.\n\n4) The number of datasets and different models compared to and enhanced with the proposed method lend good credence to the proposed approach, along with the additional analyses - ablation study, hyperparameter sensitivity, and analysis of the relationships learned.\n\n\nWeaknesses:\n\n1) A graph learning / graph based approach (e.g., with graph neural nets) could theoretically learn arbitrary relationships between nodes as well (as for example this would be encoded in multiple layers of graph neural networks) - the authors never really say what is the advantage of the proposed approach compared to this idea.\n\n2) The learned relationships are static / stationary - that is for the same series values x, the relationship will be the same regardless of time.  This seems a bit over-restricting and not realistic for time series which often have non-stationary relationships as well.  \nIt seems like this approach might be over-constraining to enforce some relationship learned across all time points - i.e., in many cases it may be more realistic if the functional relationship changes over time or context (which could also be captured by encoding context in additional exogenous series).  \n\n3) Additionally, some prior work on learning the graph structure for forecasting allows the graph structure learned to be influenced by the forecast modeling as well - which is a disadvantage of this approach as it is learned separately. \n\n4) It would have been useful to also see a comparison with prior work on reconciliation (at least given a known structure) - as typical approaches can encode arbitrary linear relationships between time series as well \n\n5) As the constraints are learned (in a completely uncontrolled fashion with flexible neural nets) - how can you guarantee the constraints don't contradict each other at all?  If they do, how can the constraint projection work and what do you do in those cases?  This could easily happen for example if the model learns some (incorrect) relationship like: x_1 = 2 * x_2 and x_2 = 2 * x_1 for 2 variables / time series x_1 and x_2 (as a simplified case to illustrate the point).  \n\n6) A major weakness is the large number of hyper parameters introduced by the approach - beyond the hyper parameters used for each forecast model itself as well.  There are even more than really tested and pointed out because the relationship network architecture and training introduces its own set of hyper parameters as well (and this could even be done for the two different types of networks for learning the relationship as well).  Several hyper parameters are chosen seemingly without the typical validation process (seemingly choosing whatever gave the best results) - along with the fact of being so many that need to be precisely tuned this limits practical usefulness of the method and confidence it will work well on other datasets for real use (where we have to do our best to select hyper parameters for everything).\n\nApplying this method, the number of hyper parameters that need to be tuned is quite daunting - tuning the hyper parameters (architecture and learning) for the relationship networks.  Tuning the various relationship thresholds.  Tuning lambda and K in the learning objective.  And finally tuning the multiple hyper parameters of the backbone forecast model as well.  It would be helpful for the authors to add some discussion on how to address this complexity and for clear description of how all hyper parameters were chosen (some hyper parameter numbers are just reported in tables and mentioned they worked better, but the process for choosing them is not clearly explained).  Discussing this issue and how it could be addressed, and further study if certain fixed values or procedures would work sufficiently, could strengthen this work.\n\n\n7) Another major weakness of the proposed methods is the limited scalability - and lack of discussion around this weakness along with lack of analysis of computational complexity / reported run times as a function of the number of time series.    Adding these could help strengthen this work.\nIn particular the computational complexity of the method seems daunting, as we have to train a neural network with the roughly the same amount of data as for the forecast model itself, for every single time series, twice, just for learning the functional relationships.  So for thousands of time series this amounts to training thousands of neural networks, which can be further multiplied for hyper parameter tuning / optimization.  \nSome discussion should really be added around this, and also if there are any ways to address it, and would strengthen the paper.\n\n\n8) Also repeated experiments are not performed - and std. dev. of metric scores / confidence intervals are not reported so it's hard to determine the significance of differences in metric scores, and robustness of the results.  Ideally metric scores would be averaged over multiple random runs and multiple test time series windows.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject