Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes using graph neural net (GNN) operations to combine per-series embeddings, to enable multivariate forecasting.

Specifically, N individual series are separately encoded for a given time window to get representations per series.  These representations are then updated with a GNN - either assuming a fully connected graph (with edge weights computed as part of the model), or using a bipartite graph (using a smaller set of K << N auxilliary nodes).  I.e., after multiple layers of the GNN, the representations are updated with information from the other series' representations.  Finally, the final representations are passed through per-series decoders.

The latent bipartite graph formulation enables more efficient operation of the GNN component as instead of computing O(N^2) messages only O(NK) need to be computed in a given pass.

The authors compare the proposed approach with other Graph based forecasting approaches (including ablated versions of the proposed approach) on 6 datasets - 2 where there is a given graph structure (so past work requiring the graph structure to be known can be used) and 4 where there is not.  They demonstrate competitive performance of the proposed approach (including the bipartite graph approach) and significant speed up of forward passes using the bipartite formulation especially for larger N.  They also examine the inferred adjacency matrices for different methods on some synthetic data.  Additionally, they show hyper parameter sensitivity results for varying K (number of latent nodes in the bipartite graph) for one dataset.
main_review: **Strengths:**
1. The approach is interesting and has some novel elements, especially using a latent bipartite graph as a step towards better scalability. 

Enabling multivariate forecasting with unknown relationships among series while also addressing the scalability challenge is an important problem in forecasting, and this is a promising direction.

Additionally I really liked how the authors posed the idea as a simple add-on to univariate methods, to be able to tack on multivariate modeling to a given univariate method - though it may have been interesting to actually test this idea out more in the experiments (as in, take some other or recent state of the art global univariate models and see if using this approach adding the graph neural net to it would help - for the given datasets and especially if there are some that really depend on multivariate models).


2. A large collection of datasets and experiments were performed, and an extensive set of graph neural net based forecasting methods compared with.


3. Additional useful ablation study, hyper parameter sensitivity study, and adjacency matrix result analyses was also provided.  This provides more interesting information and understanding of the proposed approaches.


**Weaknesses:**
1.  The complete method details and procedure is not clear from the description, and code is also not provided, so as it is the work does not seem fully reproducible.

a.) For the auxilliary nodes, U, it says they are initialized with random noise, but nothing else is said about them - i.e., how they are used in training and inference.  In particular, is this initialization done once at the beginning of training / initializing the entire model, and the values of U are fixed from then on to those values (for the rest of training and subsequent test prediction)?
Or are the embedding values updated as part of training as well?  Or are different random values used in each time window?  This is not clear.

b.) The description is a little confusing in a few places.  In particular, in Equation 3 and later references back to it - it would be helpful to provide the actual definitions of the various phi functions (at least in the appendix) and dimensions / example dimensions of  various inputs and outputs such as the embeddings.

c.) It would be helpful to understand the actual training process, even if in the appendix or said briefly - i.e., presumably sliding windows were used across the full input time series to generate each sample, presumably for every step size of 1?  Presumably 1 batch is just different time windows, containing all time series, so inputs are BxTxN where B is batch size, T is time window size, and N is the number of time series?

d.) Earlier when describing past work, context c is mentioned.  How is context c incorporated into the proposed model, as it seems absent from the method description?  What about exogenous series (i.e., context or features that vary with time but are not part of the series we are predicting)?  


2. Experiment details are missing (e.g., based on the description I assume proper hyper parameter selection was not done) and results seem not too convincing, aside from speedup using the bipartite graph formulation.

a.) Various missing experiment details:
i. When describing the data, what is "Enc. Length" and "Dec. Length" in Table 2?  Is it "Enc. Length" the history window size used in each case, and "Dec. Length" the prediction window size?   If so that means for the first two datasets the next 12 values are being predicted, how then are different horizons evaluated / what is the meaning of the different horizons in the results table?  For the second set of datasets, why not evaluate predicting multiple time points (such as next 48 hours) as is commonly done in other past work, as opposed to single time point prediction?

ii. It would also be best to report the periodicity of the time series - i.e., is it daily, hourly, etc...

iii. Also details about the evaluation procedure are missing - in particular, was a sliding test window used to evaluate the performance, how many test windows were used, was re-training done before each, etc.?  Some of these may be able to be found in the pointed to prior work (saying it is the same setup) but really these details should be included in the appendix, and what was specifically done here for all particulars cannot be known from just the setup (such as if retraining is done or not).

iv. What context (c) or exogenous series were used in the experiments?

b.) As far as I can tell, since specific and different hyper parameters are reported in the appendix for each data set, it seems like the hyper parameters were used that gave the best results on the test set, which if so can be misleading and not reflect the actual performance.  Hyper parameters should be selected from a grid (or using HPO) based on validation data only.  I saw no mention of hyper parameter selection being performed in this manner, and I did not see reported the set of hyper parameters searched over for each method, so I have to assume it was not done.  If it was done, please report the set of hyper parameters searched over for each method, data set, and horizon.  Code would have also been useful to verify this here.

c.) Although 5 different runs were performed and averages across those were reported, no standard deviations are reported. The std. dev. should be reported as well, especially since in many cases there is only a tiny difference in scores compared to past methods.

d.) It is hard to read through the details of the results since there are so many, but from what I can tell it does not seem like the proposed method significantly improves over past methods - results seem practically the same or only tiny improvements.  It might help to present overall average results (for example mean and std. scores across all datasets as a final column, and plotting the different average metric scores with error bars vs. dataset size / number of series or horizon), percent improvements, and especially significance analyses (e.g., a critical difference diagram I think could be most useful here given the number of different methods and dataset+horizons - e.g., see https://mirkobunse.github.io/CriticalDifferenceDiagrams.jl/dev/ or https://github.com/hfawaz/cd-diagram).

e.) Comparisons to other classes of models would be nice to include as well - in particular good, recent global univariate and non-graph multivariate methods.
It would be nice to have representatives from the state of the art of global univariate methods (such as the recent transformer approaches such as Li et al. 2019 or more recent works that improved on this) in the comparison to see how they stack up, and validate the hypothesis that the multivariate modeling is even necessary in all these datasets.
Similarly it would be nice to see the comparison with recent non-graph neural net approaches, as they are just as relevant here, and there are recent approaches with state of the art results, some mentioned here, like, Salinas et al. 2019 and other more recent ones and ones focusing on scalable multivariate forecasting (mentioned below).  This would validate the hypothesis that using graphs is necessary / could help over these other approaches.


3. Novelty is somewhat limited.

As pointed out there are several works learning graph structure as part of forecasting (and some works on hierarchical structure, though not applied to forecasting - e.g., Ying, Rex, et al. "Hierarchical graph representation learning with differentiable pooling." NeurIPS 2018).  One additional missed work for learning graph structure as part of multivariate forecasting is the following:
* Cao, Defu, et al. "Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting." NeurIPS 2020.

Arguably the main novelty comes from introducing the latent bipartite graph in order to improve the scalability, however, the are other past works that also focus on improving scalability for multivariate forecasting, and that can learn the relationships scalably.
While I feel the approach proposed here is novel given the usage of graphs, I also feel this idea is in line with other prior work that also consider somewhat similar ways of enabling scalable multivariate forecasting (e.g., using a smaller set of latent node = series), and it may be worth mentioning this connection and approaches targeted at scalable multivariate forecasting.  

One is mentioned - Salinas et al. 2019 - that similarly uses per-series model components to embed each series and combine them with a low-rank Gaussian copula, which enables more scalable modeling and training (even enabling sub-sampling among the series).  
Another closely related line of work targeted at scalability is to use a smaller set of latent global series (similar in concept to the latent nodes introduced here since a node corresponds to a time series) - and modeling learns these latent global series (or how to derive them from the inputs) and the input series relationship to these latent global series to get predictions for individual time series.  These similarly enable improving the complexity of the forecast modeling part from O(N^2) to O(NK^2) using the K latent series.  E.g.:
* Sen et al. "Think globally, act locally: a deep neural network approach to high-dimensional time series forecasting." NeurIPS 2019.
* Nguyen et al. "Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting." AAAI 2021.

It would be nice to compare and contrast against these as well, at least in text to call out the related line of work, and pros and cons.  Ideally it would also be nice to see a comparison in the experiments to some of these or else another recent state of the art multivariate forecasting method that is not considering graph structure / graph neural nets.

summary_of_the_review: Overall I think the idea is interesting and in particular the direction of a latent bipartite graph as a graph neural net way of making multivariate forecasting more scalable is novel, however it could be useful to point out and contrast to related alternate approaches for scalable multivariate forecasting, which are similar but not using graph neural nets.

However there are several questions about the method and questions and concerns about the experiments that I feel need to be addressed before I can recommend the paper for acceptance - i.e., specifics of the method so it could be reproduced and understood by the readers, and details of the experiments to determine how hyper parameters were chosen and if the results show significant improvements.

Without additional clarifications and details around the experiments and the method I am leaning on the side of rejection.  However, if these can be addressed I could change my recommendation.

 

***Update after responses and discussions:***

I appreciate the responses, and read the author responses, updates, and other reviews and comments.  I tend to agree with the other reviewers' points and don't feel all concerns are addressed with the updates. 

As pointed out the key novelty is really using the bipartite graph to increase computational efficiency, and as other reviewers have argued little analyses around this is done, larger data sets should be included, and training time scaling reported.  Despite the authors claim that larger data sets do not exist, and aside from simulated data, some larger ones exist and have been used in several other papers. E.g., the large wiki dataset has > 100,000 time series (and used in one of the scalable time series forecasting papers I cited and other past work it cites), and the recent M5 dataset and competition also has 30,490 inter-related time series at the lowest level, and there are several forecasting papers published about it as well.  Additionally more can be found with some searching / reading. The authors would have at least seen the wiki dataset if they looked at other prior papers including what I mentioned in my review. 

In particular, as the main benefit they are claiming from their method is enabling scaling for multivariate forecasting to large amounts of time series, they ideally should consider related work on scaling forecasting for large collections of time series and at least discuss this area of work and contrast to their work, which they did not, despite the recommendation. They also should compare to the state-of-the-art multivariate forecasting methods (including without considering graphs, to demonstrate the graph approach is helpful), and especially need to compare to those designed to scale to many time series, which also was not done.

I also agree with the other reviewer - to only compare and report forward evaluation run times leaves uncertainty about practical performance, as training is the biggest issue in most cases.  Further, there seems little reason not to report it and explain what it reveals unless the benefits are not as much in this case - which would also be interesting and fine just needs to be reported and some analyses provided. Ideally the analyses around training and forward time should be performed for varying number of time series across multiple datasets, e.g., plotting on x-axis the number of time series, and on y-axis the training and forward time (and nice to also see the error metrics as well per number of time series).  It could even be combined in the same tables or plots for forward time by showing average training time per step and average forward time per step, for example, on dual y-axis if space is a concern) vs. number of time series.   This could potentially also be generated from larger datasets if needed by sub-sampling series, or just using the natural number of time series in different datasets.

The fact training time isn't reported makes one wonder if there could be some reasons it's not providing as much scaling benefits for training, e.g., maybe the train time is actually not much if any better, due to the extra propagation steps having more of an impact at training time as perhaps slowing down the back propagation updates (which we don't see from forward time alone), and also from the fact that all time series still must be present at once so there could be some dominating factor on the data side that doesn't change.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to combine information of multivariate time series by extending univariate architectures. The technique uses a graph representation to represent interactions by assuming a bipartite structure, which allows the technique to scale the representation to reduce the complexity from $O(N^2)$ to $O(N K)$ using K additional nodes. The architecture represents each time series as one of N nodes and associate K embeddings (nodes) and thus there are connections between the N nodes and the K nodes but not within each group of nodes. The architecture encodes the confounders, co-linearities, and other information among the time series to improve forecasting. The experiments show the performance (and time efficiency) of the technique against several baselines on METR-LA and PEMS-BAY datasets. The experiments also show the performance on single-step forecasting on four publicly available datasets. Finally, synthetic datasets were used to evaluate the adjacency matrices in control scenarios. 

main_review: Among the strengths we can think of the method, and the representation. Namely: 
Methods that account for relations due to confounders or other relations are important to consider and directly incorporating them in time series analysis  could avoid using regularization during model fitting. 
The latent graph inference idea could be useful to automatically identify the relations existent in a multi-variate time series model more broadly; for instance, in partially observed settings. 
With respect to the weaknesses, while the paper is relatively well organized, there are many details that are not clear. In particular, the architecture is presented and its components listed, but the connection of these components and the overall goal of the paper is not discussed in enough detail as to justify the choices. For instance how are colinearities identified by the architecture? How are non-linear dependencies discovered and summarized by the architecture? What other confounding can the architecture detect?
summary_of_the_review: The paper has some strengths that could be useful in applications of multivariate time series forecasting. The graph representation could be useful not only for prediction but also for analysis and interpretability of results. However, I feel that more can be said about the architecture and how it affects the type of relations that it is capable of represent.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to infer a relational latent space in order to do multi-variate time series forecasting. The main contribution is reducing the computational complexity of the inferred graph by converting the fully connected graph learning problem to a bipartite graph learning method in which the nodes are connected to K auxiliary nodes. 
main_review: The paper is fairly well-written and its motivation is clear. It is also technically correct. The authors tried their proposed method on a few small datasets.

The FC-GNN version proposed in the submitted paper is neither novel nor significant. The main argument of the paper is reducing the computational complexity of the relational inference, which I think is a valid motivation for this paper, but I would have rather expected to try a more challenging dataset with a large number of nodes for achieving the same goal, i.e. see if the reduction in computational complexity in this way is helpful in a practical manner or not. Inferring relations between a small number of features, i.e. 8 to 800, is not a challenging problem, and other papers such as FNP (the functional neural processes, NeurIPS 2019) and BayReL (Bayesian Relational Learning, NeurIPS 2020) inferred a much larger relational graph (structured latent space) with a simple method $O(N^2)$, however, it is really challenging to learn a graph with 20,000 nodes.

Even the BP-GNN model proposed in this paper is learning a Gaussian mixture in my opinion, as it will learn the connection weight of each node with Gaussian auxiliary nodes, leading to having a Gaussian mixture latent space. I believe the authors at least need to show that adding such a computational learning $O(NK)$ is improving the performance compared to a mixture of Gaussian latent space as a baseline. 

Apart from that, the authors are learning one graph for a period of time. In my opinion, the inferred graph only captures high-level information, it might miss the short-term dependencies. An ablation study on this might be very helpful similar to those that have been done on NLP.

I also believe the authors still are able to compare their model with NRI. In addition, for those methods "Multivariate with a known graph", the authors can construct ad-hoc graphs such as identity and/or fully connected graphs to compare the performance.  

I don’t find the plots in Figure 2 very insightful and it is quite hard to see any informative insights regarding BP-GNN in my opinion. I encourage the authors to create an ablation study based on the BP-GNN and see if they can reconstruct the bipartite graph. Even for the fully connected graph, I prefer to see a real example that you have the graph and see if the method can learn the graph. The authors should be able to find some data in traffic applications for example. 

The computational complexity needs to be discussed in the main paper as it is the main argument of the paper. 

Comparisons with GRNN and RNN methods are needed in which you have the true graph or you do not have it to see if learning the graph dependency is helpful or not. 

The standard deviation of the performance should be reported, considering the results are very close to each other. 
summary_of_the_review: Overall, I think that this paper would need to be extremely solid on the experimental side with potentially further experiments, e.g. using a larger number of features, robustness to missing data, etc, in order to make up for the somewhat limited contributions in terms of novelty.
review_title: null

Meta-review: This paper studies multivariate time series forecasting by making relational inference in a latent space. It attempts to address the important issue of reducing the computational complexity of the inferred graph. This motivation is well articulated.

Despite its merits, concerns have been raised regarding the relatively weak evaluation without using datasets involving more many nodes to demonstrate the scalability of the proposed method, which is a major selling point of the paper. As such, while the motivation of the work is clear, its experimental evaluation is not thorough enough to demonstrate the scalability of the proposed method.

The authors made the remark in their response that they are not aware of any public time series dataset of this size (which is not agreed by another reviewer who pointed out that some much larger datasets were used in other papers). Note that it is not uncommon in other work to use synthetic datasets to evaluate the scalability as well as other properties of the proposed methods.

Moreover, clarity of the presentation also has room for improvement.

The paper has potential for publication in a top venue if the comments and suggestions are incorporated to revise the paper.
Decision: Reject