Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is about multivariate time series forecasting with structure learning. Existing works usually assume the graph structure of the multiple time series is given or learned by 
the node similarity. However, in some applications, the relationship between time series can be much more complicated and the graph structure is not enough. This paper proposes to
use functional relation field to model the inter-node relationship. Experiments on one synthetic and two real-world datasets show that the proposed method can enhance the existing 
spatial-temporal forecasting model.
strength_and_weaknesses: Strength:

The proposed method is novel. There are no existing papers that consider the complicated constraints relationship among nodes.

The proposed method is technically sound

Empirical performance is promising

Weakness:

The paper did not compare with some baselines that perform joint forecasting and structure learning, e.g., GTS [1] and NIR [2].

[1] DISCRETE GRAPH STRUCTURE LEARNING FOR FORECASTING MULTIPLE TIME SERIES, ICLR 2021
[2] Neural Relational Inference for Interacting Systems. ICML 2018

clarity,_quality,_novelty_and_reproducibility: The paper is written well and provides source code in the supplementary for reproducibility.
summary_of_the_review: The paper proposes a novel method to learn the complicated structure for multi-variate time series forecasting task. However, it misses some important baseline.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on the problem of time series forecasting with constraints.  The proposed functional relation field framework is aimed at learning constraints from multi-variate time series data. Then, the authors develop the training and inference method incorporating the learned constraints. The proposed method is evaluated on both synthetic and real datasets. 
strength_and_weaknesses: Strength:

This paper explores the idea of capturing hidden inter-variate constraints/relations in multivariate time series and imposing the discovered constraints on the training and inference process. 

In the experiments, the proposed framework is applied to baselines to compare the performance difference. This is a valid idea. 

Weakness:

The problem of identifying inter-variate constraints/relations and applying them in forecasting is not new, and several works have studied it, e.g., [1, 2]. The technical contribution in this paper looks marginal, given that the technique used in this paper is standard and no news insights seem uncovered. Meanwhile, the presentation, some design choices, and evaluation have the following issues. 

(a) Eq.(1) seems to formulate the time series forecasting from the perspective of the probability model, while the rest of the paper follows the standard point forecast paradigm. In the probability model, the forecast is not necessarily the mode and could be mean, quantile, or interval. Eq.(1) seems disconnected from the problem of this paper. 

(b) Eq.(4)-(6) present the "constraint", which is not rigorous w.r.t. the concept of constraint. The presented constraint is essentially closer to the concept of correlation or relation, since it is simply derived from how well the other variables fit the target variable.      This is highly data or observation-dependent. Moreover, it needs a threshold to determine the set of relevant variables. For multi-variables in different value domains or distributions, finding proper thresholds seems nontrivial and would affect the overall performance.      This way of identifying  "constraint"  seems ad-hoc and arbitrary.

(c) From Eq.(10), the constraint discovered from X seems to be applied to Y. It is a bit confusing to present this way. 

(d) Eq.13 - Eq.15 seems problematic. In Eq.(13), the minimization and the constraint are mutually exclusive, i.e., the minimization problem is for relaxing the constraint, and if the constraint is to behold, the minimization is unnecessary. Meanwhile, Eq.(13) is simply a  least-squares problem w.r.t. $\tilde{y}$ and the iterative process seems redundant.

(e) In the experiment, only the newly introduced hyperparameter is compared. It would be good to also show the hyperparameters in training, since they affect the end performance significantly in many cases. 

[1] Wu, Zonghan, et al. "Connecting the dots: Multivariate time series forecasting with graph neural networks." Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020.

[2] Li, Zhuoling, et al. "Dynamic Graph Learning-Neural Network for Multivariate Time Series Modeling." arXiv preprint arXiv:2112.03273 (2021).




clarity,_quality,_novelty_and_reproducibility: 
In this paper, some formulation is not precise and clear enough, and the presentation needs improvements as listed above.

The problem studied in this paper is not new. The technical contribution seems marginal. 


summary_of_the_review: The paper focuses on the problem of learning hidden constraints in multivariate time series and making use of the learned constraints in the training and inference of forecasting models. 

The problem is not new. The technical contribution is incremental, given that some design choices are standard, and some seem problematic and need more clarification. 

Some experiment results look interesting, however, given the aforementioned weakness, it needs improvements to be more solid and convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new model for multivariate forecasting with structured relationship between time series.  They propose to learn a static functional relationship between each node and every other node (where the target node/series value is equal to a function of the values of the other nodes/series), and then incorporate these functional relationships into any existing forecast model - via adding a regularization term (penalizing forecasts that deviate from the relationships) to the regular forecast loss (weighted by a hyper parameter lambda), and also via subsequently projecting the final forecasts onto the (learned) constraints.

The authors validate their approach by comparing results on several datasets for several recent forecast methods with and without the proposed functional relationship component and demonstrate consistently improved forecast error metrics using the proposed approach.  They also perform a variety of hyper parameter sensitivity and ablation analysis, and some analysis around learned functional relationships.  
strength_and_weaknesses: Strengths:

1) I find the approach to be interesting and novel - I personally have not seen this kind of approach taken before, and I feel it can be seen as generalizing a couple common classes of approaches: hierarchical forecasting - which does incorporate functional relationships but of a fixed form (hierarchical aggregates), and graph-based forecasting - which does not specify the particular relationship between nodes as is done here.  This enhances the hierarchical approach by generalizing the functional form encoded to be not just hierarchical or linear, and also generalizes the graph approach to encode more specific relationships between variables explicitly.  

2) The paper was well-written and organized so it was easy to follow and understand.  

3) The method is logical and sound.

4) The number of datasets and different models compared to and enhanced with the proposed method lend good credence to the proposed approach, along with the additional analyses - ablation study, hyperparameter sensitivity, and analysis of the relationships learned.


Weaknesses:

1) A graph learning / graph based approach (e.g., with graph neural nets) could theoretically learn arbitrary relationships between nodes as well (as for example this would be encoded in multiple layers of graph neural networks) - the authors never really say what is the advantage of the proposed approach compared to this idea.

2) The learned relationships are static / stationary - that is for the same series values x, the relationship will be the same regardless of time.  This seems a bit over-restricting and not realistic for time series which often have non-stationary relationships as well.  
It seems like this approach might be over-constraining to enforce some relationship learned across all time points - i.e., in many cases it may be more realistic if the functional relationship changes over time or context (which could also be captured by encoding context in additional exogenous series).  

3) Additionally, some prior work on learning the graph structure for forecasting allows the graph structure learned to be influenced by the forecast modeling as well - which is a disadvantage of this approach as it is learned separately. 

4) It would have been useful to also see a comparison with prior work on reconciliation (at least given a known structure) - as typical approaches can encode arbitrary linear relationships between time series as well 

5) As the constraints are learned (in a completely uncontrolled fashion with flexible neural nets) - how can you guarantee the constraints don't contradict each other at all?  If they do, how can the constraint projection work and what do you do in those cases?  This could easily happen for example if the model learns some (incorrect) relationship like: x_1 = 2 * x_2 and x_2 = 2 * x_1 for 2 variables / time series x_1 and x_2 (as a simplified case to illustrate the point).  

6) A major weakness is the large number of hyper parameters introduced by the approach - beyond the hyper parameters used for each forecast model itself as well.  There are even more than really tested and pointed out because the relationship network architecture and training introduces its own set of hyper parameters as well (and this could even be done for the two different types of networks for learning the relationship as well).  Several hyper parameters are chosen seemingly without the typical validation process (seemingly choosing whatever gave the best results) - along with the fact of being so many that need to be precisely tuned this limits practical usefulness of the method and confidence it will work well on other datasets for real use (where we have to do our best to select hyper parameters for everything).

Applying this method, the number of hyper parameters that need to be tuned is quite daunting - tuning the hyper parameters (architecture and learning) for the relationship networks.  Tuning the various relationship thresholds.  Tuning lambda and K in the learning objective.  And finally tuning the multiple hyper parameters of the backbone forecast model as well.  It would be helpful for the authors to add some discussion on how to address this complexity and for clear description of how all hyper parameters were chosen (some hyper parameter numbers are just reported in tables and mentioned they worked better, but the process for choosing them is not clearly explained).  Discussing this issue and how it could be addressed, and further study if certain fixed values or procedures would work sufficiently, could strengthen this work.


7) Another major weakness of the proposed methods is the limited scalability - and lack of discussion around this weakness along with lack of analysis of computational complexity / reported run times as a function of the number of time series.    Adding these could help strengthen this work.
In particular the computational complexity of the method seems daunting, as we have to train a neural network with the roughly the same amount of data as for the forecast model itself, for every single time series, twice, just for learning the functional relationships.  So for thousands of time series this amounts to training thousands of neural networks, which can be further multiplied for hyper parameter tuning / optimization.  
Some discussion should really be added around this, and also if there are any ways to address it, and would strengthen the paper.


8) Also repeated experiments are not performed - and std. dev. of metric scores / confidence intervals are not reported so it's hard to determine the significance of differences in metric scores, and robustness of the results.  Ideally metric scores would be averaged over multiple random runs and multiple test time series windows.
clarity,_quality,_novelty_and_reproducibility: Clearly and well written, organized and explained - with some grammar and typos that should be fixed (see below).  I find the approach to be highly novel as mentioned, and reproducibility is good as code is provided and details around experiments, datasets, and hyper parameters are reported.  I do feel the complete experiment process / how all hyper parameters and architectures are chosen is not fully explained.


Grammar issues and typos throughout hurt readability - e.g., in intro: "...learned based the similarity..." instead of "...learned based on the similarity...",
"...a more precise manner than graph...", "...the introduction of graph...", "...were proposed to discovery the...", "...relationship between multiple time series is typically complicate...", "Others alternatives ... is also possible", (typo) "function relation fieild"


Incorrect statement: "finding the nearest projection point on the plane f(y) = 0" if f is nonlinear as described, this may not define a plane

summary_of_the_review: Overall I found the approach to be interesting and novel and could be seen as a generalization / enhancement of two classes of approaches: hierarchical forecasting and graph-based forecasting.  For the former it can be seen as extending the functional relationship to more general kinds (rather than just hierarchical aggregates) and for the latter, enabling encoding more information about the relationships between series. 

Potential weaknesses include not learning the relationships end to end - so they can not be influenced by the downstream forecast modeling, and the functional relationships being too restricting and non-dynamic (as they are fixed over time).  However the latter can be controlled to some extent with hyper parameters (such as lambda), and both may be beneficial to avoid overfitting.

The major weaknesses that prevent me from raising my score higher mainly arise because of the large number of hyper parameters introduced (not just the ones mentioned but even the hyper parameters associated with the relationship network architecture and training) along with lack of clear, principled hyper parameter selection and no repeated randomized experiments to provide confidence intervals / std. dev., and the limited scalability of the method.  The latter of which is not discussed (and no run times reported).  In particular it's hard to imagine this method could be used well in practice with so many hyper parameters that must be tuned, and it would really only be tractable for a small number of time series as for n time series, 2n networks have to be trained just for the relationship learning part of the model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method to automatically learn constraints over nodes in a graph, and apply the constraints for spatial-temporal forecasting. The major contribution is the proposed constraint learning framework.
strength_and_weaknesses: Strength:
1. The motivation for exploring relationships between nodes in terms of constraints is insightful.

Weaknesses:
1. The writing is hard to follow.
2. Some parts are not clear:
(1) what is the physical meaning of linear constraints (equ. 2)?
(2) In equ. 3, why set f_i(x_t)=0
(3) In section 2.2, is the target-dependent relationship bidiractional?
(4) It seems the proposed model needs to go over all the nodes to check the target and dependent, what is the complexity of the method?
3. \epsilon_{grad} is a hyperparameter. But the experiment does not evaluate the effects of \epsilon_{grad}.
4. It seems too many hyperparameters to control the constraint learning. It may expect much more effort for hyperparameter tuning, which may jeopardize the applicability of the method.
5. Typo: the second paragraph on page 2, "...while our framework on the left panel models the..." it should be" the right panel"?
clarity,_quality,_novelty_and_reproducibility: The motivation is strong, the proposed method is novel. But the writing is hard to follow, some concepts are not well-explained.
summary_of_the_review: The motivation is strong, the proposed method is novel. But the writing is hard to follow, some concepts are not well-explained.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to model inter-variate constraints in multivariate time series and imposing these constraints on the predicted output during training and inference. The approach is demonstrated on a toy problem and two real-world dataset.

Strength: The paper consider an important problem, namely leveraging the relationship among time series to improve multivariate forecasting. The exposition is clear and the two-stage approach interesting

Weaknesses: Unfortunately, several concerns remain about the present submission, and we strongly encourage the authors to pursue their work to address them in depth. 
- In particular, the proposed process that identifies the relevant node for each target node individually might be suboptimal compared to joint identification for all nodes (similarly to graphical lasso vs neighborhood selection in gaussian graphical modeling ). Also selection of relevant nodes based on gradients is questionable and prone the identification of spurious dependencies. Important dependency might also me missed due to the use of a single threshold across nodes.  A two-staged process seems suboptimal compared to a single stage process where sparsity in imposed, e.g. using sparse attention networks. 
- Regularizing the prediction loss with the constrain satisfaction squared loss is quite ad-hoc as argued by Reviewer dWww. 
- Also it seems suboptimal that the learning of the functional relational field should be done once and for all, and then this fixed field should be applied. 
- From a formulation perspective, step (a) in Figure 2 of the paper ignores what can be predicted using past values of all the time series, while it might be argued that one might be better off if one considers that the functional relationship might be akin to "instantaneous dependencies" and should be estimated after subtracting what can be accounted for / predicted  by the past values of the times series. 
- As pointed out by Reviewer n4W8 , the number of parameters is quite daunting. Also heir tuning and sensitivity should be studied more systematically. 

justification_for_why_not_higher_score: The paper has many unresolved issues as pointed out in the reviews and in the above meta review. Also the formulation and approaches used in each step are questionable.
Decision: Reject