Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on unsupervised anomaly detection in multivariate time series. This is a important problem because anomalies are rare and labeling anomalies for supervised learning is labor intensive. To make progress in this challenging domain, they  use learned graph structure and normalizing flows, which seem well suited for the task. They evaluate their methods on multiple datasets, and compare against multiple modern deep networks. 
main_review: I liked this paper and read it with great interest. The strength of this paper rests in a) an excellent problem setup, and b) well chosen methods that address real challenges in the field and provide useful insights into problems. The paper does not make unrealistic assumptions. The evaluation datasets are well chosen and the baselines do not seem like "strawman" comparisons. 

I am particularly impressed with Figure 4. Finding the temporal shifts for anomalies, in particular anomalies that develop slowly is a valuable contribution, and being able to quantify that with with a Bayesian graph is a novel contribution that deserves to be called out. Overall, I would be interested to hear more about the explainable side of this work - Bayesian networks are often useful for counterfactual reasoning which makes me optimistic that these methods are not only useful for detecting anomalies, but also potentially explainable. 

There are some places where I think the evaluation could be improved. ROC-AUC is useful real-valued metric, but in practice, there are always questions about the choice of threshold and inclusion of of some ROC curves, along with uncertianty bounds on those curves would make this work even stronger.

This paper also suffers from complexity and I somewhat question its reproducability. I don't doubt the numerical results, but with a complex system such as this one, there would need to be a significant investment of effort if anyone else wanted to use these methods and I don't think sufficient details are provided in . Providing a code repository would address this. 

If I have a major complaint about this paper it is that their main hypothesis: "Anomalies lie on low density regions of the data distribution" remains untested. 
summary_of_the_review: This is a strong paper and I recommend it for acceptance. It addresses a real problem in a manner that is both novel and pragmatic. I hope the authors will not take my constructive criticism too harshly, the paper is strong in its current form and my suggested improvements are only suggestions for ways I think it could be made even stronger. I would love to see this work see application in other areas and my comments reflect ways to speed that process.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the task of detecting anomalies in multiple time-series, for the setting where multiple instances of a fixed time-window are available for learning. The authors propose a method that involves using a directed acyclic graph (DAG) to model the (contemporaneous) dependencies between the variables in the time-series. An RNN is applied to form a representative state for each time-series at each point in time. A mapping is applied using the DAG to process the states from the previous time-step and the ancestor states from the current time-step to construct a “dependency representation” vector for each time series. A normalizing flow is then applied to map to a base distribution so that the log-density of the instance (the observed multiple time-series) can be evaluated. The conditional densities for each time series can also be evaluated. The paper includes an evaluation of the performance of the approach on a proprietary dataset, with a comparison to state-of-the-art baselines. Some qualitative results are also included for the task of density estimation for a public traffic dataset. 
main_review: Strengths of the paper:
(1) The paper is clearly written and describes a novel technique. The combination of the DAG and the normalizing flow is a very interesting approach, and the application to the multiple time-series anomaly detection problem is a novel contribution. 

(2) For the analyzed dataset, the proposed method appears to significantly outperform the baselines. 

(3) The authors provide an ablation study to demonstrate that all of the components of the algorithm contribute to the improved performance.
 
Weaknesses of the paper:
(1) The quantitative results are only derived for one dataset (which is unfortunately proprietary). The labels for this dataset are noisy, making the significance of the results more difficult to assess. The experiments on the traffic dataset seem incomplete and it is difficult to conclude much from them. 

(2) There are no measures of variability or confidence intervals for the provided results. There are no tests to determine if the performance differences are statistically significant. The authors calculate AUC-ROCs, but do not display the ROCs, meaning that the comparison to baselines boils down to a single number. 

(3) The absence of synthetic data or well-understood data means that it is difficult to determine whether the derived DAGs are meaningful.  

MAIN REVIEW

The paper addresses a very interesting and challenging problem and provides an elegant and intriguing solution. The combination of the RNN, the DAG to capture the dependency structure, and the normalizing flow appears to be a promising way to represent multiple time-series. The experimental results suggest that the joint learning, with enforcement of an acyclic graph, can lead to better performance. 

Methodologically, I think the paper presents a novel, technically sound, and well-motivated approach. There have been some recent approaches in the multivariate time-series forecasting literature to combine latent representations and normalizing flow, but I am unaware of any work that incorporates a DAG in a similar way. 

The experiments are the weaker portion of the paper.

(1) I think the paper would have benefited significantly from the inclusion of an analysis of synthetic data. With control over the ground truth, the performance of the algorithm could be explored and better understood. In particular, one could assess how well the DAGs are being estimated. 

(2) In the absence of synthetic dataset, the experiments would be considerably more compelling if more datasets were analyzed (preferably with at least one public dataset). 

(3) It is highly desirable that some form of confidence interval (or alternative measure of variability) is provided for the obtained performance. Statistical significance tests should also be conducted. 

(4) Compressing the performance metrics to a single number is undesirable. I would have been very interested to see a comparison of the actual estimated ROCs. Often the AUC can give a misleading picture because one is more interested in relative performance in the low false-alarm regime. Figures such as 2(b) are very useful, but the paper does not provide the equivalent figures for any baseline algorithms, making comparison difficult.

(5) More details about hyperparameter tuning process would be welcome. The statement “The hyperparameters are all tuned based on the log-density on the validation set” is not sufficient to reproduce the results unless it is clear which hyperparameters were tuned, how they were tuned and over what ranges. For example, it is not clear if all architectural choices are included (hidden dimensions, number of flow blocks) 

(6) The experiments for the traffic data seem incomplete, and it is difficult to draw much of a conclusion from them. 

Questions:

(1) Is it possible to provide some form of measure of the variability in the results, e.g. bootstrapped confidence intervals?

(2) Some of the forecasting architectures come closer to having an A matrix that is applied to previous states, i.e. AH_{t-1} W_2. With this form there is no need to enforce a DAG structure, but predictive relationships are learned. There may also be sparsity encouragement during the training process. I thought this form might be one of the ablation studies to demonstrate the importance of the modeling of contemporaneous dependencies, but my understanding was that the two ablation cases investigating alternative dependencies were different from this approach. Could you comment – do you have insight into the importance of learning a contemporaneous as opposed to a predictive A? Please clarify if my understanding of one of the ablation settings is wrong. 

summary_of_the_review: ** After the authors' response and the improvements to the paper, I have changed the score to a "6"; marginally above the acceptance threshold. 

-----------------------------------------------------
I have recommended “marginally below the acceptance threshold”. 


I like the methodology that is presented in the paper, but I think there is a need for more extensive experimentation. Synthetic data would provided a much more controlled testing environment. An examination over more datasets would provide more compelling evidence that the proposed method is advantageous, and perhaps highlight the types of problems where the approach provides a major benefit, as opposed to those where it offers a less significant improvement (or is perhaps even detrimental due to its flexibility). More details concerning the experimental methodology are required. Results that provide a more detailed comparison between the proposed methods and baselines would be helpful (some figures rather than just a single number). Measures of variability in the results and statistical significance tests would be beneficial in understanding how dramatic an improvement has been made.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a model for anomaly detection on multivariate, high-dimensional time series data, where there are statistical dependencies across the different time series (referred to as "constituents" in the paper). The authors model statistical dependence using a Bayesian network and temporal dependence using an RNN. One key feature of the proposed model is that it simultaneously learns the structure of the Bayesian network AND a representation of the temporal state. The authors combine these two representations, which then gets used as input to a normalized flow technique, which makes computation of the likelihood of the data mathematically and computationally feasible.

The authors show that the proposed model has improved performance for the anomaly detection task compared to recently-proposed neural-network anomaly detection methods.
main_review: Strengths:
- Good solution to a problem that has applications in many diverse areas
- Authors model correlations between time series, which is often ignored in multivariate time series work
- Authors effectively integrate mature components from a variety of different subfields in statistics, representation learning, and neural networks
- Despite all the different moving parts, the model is clearly explained
- Authors evaluate their methods in relevant datasets

Weaknesses:
- It is not clear how to quantitatively select the threshold for flagging something as an anomaly
- The chosen evaluation metric is difficult to interpret
- No discussion on scalability

Extended comments:
I found the presented model to be quite interested, and I believe that it addresses some of the weaknesses of existing anomaly detection methods in novel ways. Particularly, I believe that combining a Bayesian network with and RNN to jointly learn dependencies across time and constituencies adds great value to multivariate time series modeling. The fact that the model is able to learn the structure of the Bayesian network as a continuous optimization problem is quite advantageous as well.

I was also quite pleased with the presentation of the paper. Despite the fact that the authors are combining components from various ML and statistics subfields, the description is easy to follow.

Where I think the paper could improve is in the Experiments section. I understand that there is noise in ground truth labeling in anomaly detection datasets; however, the proposed evaluation metric to address this source of noise is difficult to interpret and frankly seems to be an afterthought. I would prefer to see a "hard" label; the authors could declare an anomaly as "successfully detected" if it falls within some time window of the prediction, where the size of that time window is application-specific.

Another weakness of the paper is on selecting the threshold to flag something as an anomaly. This is not a weakness specific to the paper, and potential solutions have been studied before. I think it would strengthen the paper if the authors propose heuristics to numerically choose a threshold, without having to pick and choose from a histogram (which is very subjective). Since there is a Gaussian distribution at the end of the model, would it be possible to say that an observation is anomalous if the probability of seeing such extreme values is less than 0.05 (or some other threshold)?

Finally, I would say that the ablation study is inconclusive. There is a marginal increase in performance; however, couldn't this be explained merely by the fact that GANF has more parameters (and hence more expressibility) than the models where you take out one of the components?
summary_of_the_review: Given my comments above, I think that there are enough strong points in the paper to put it above the acceptance threshold. I am open to being more optimistic about it after the author response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine learning of the DAG structure between variables and learning of distribution using normalizing flows for unsupervised anomaly detection from multivariate time series. Once the distribution is learned, one can detect anomalies as low density regions. The effectiveness of the proposed method is empirically evaluated on real-world datasets.
main_review: ## Strength

The problem of anomaly detection from multivariate time series is relevant. In particular, anomaly detection in a fully unsupervised manner is technically challenging and an important research topic. Many existing methods (implicitly) assume availability of partial supervised information, such as normal profile, even if they are said to be unsupervised, while we cannot often assume such a situation in practice.

## Weakness

This paper is overall not well-written, and there are several concerns regarding presentation, quality and evaluation.

- First, I cannot understand the problem setting. The authors say that a give dataset is $(\mathbf{X}^1, ..., \mathbf{X}^n)$, and each $\mathbf{X}^i \in \mathbb{R}^{T \times D}$. In the typical setting of multivariate time series, there are $D$ each time series, and the length (the number of time stamps) of each time series is $T$. Then a given dataset can be treated as a single matrix $X \in \mathbb{R}^{T \times D}$. Therefore it is not clear why there are $n$ such matrices in the problem setting introduced in this paper? What is each $i$ here? Moreover, it is said that there are $|\mathcal{D}|$ such $(\mathbf{X}^1, ..., \mathbf{X}^n)$ as a dataset. Since repeated sampling is fundamentally impossible in time series analysis, it is also not clear what does such a dataset mean. In addition, the authors assume a DAG structure over such indices from 1 to $n$ ($n$ variables?), while its interpretation is also not clear. Please note that it makes sense if a DAG structure exists over $D$ time series or $T$ time stamps, while the setting in this paper is different.

- I am also confused with the anomaly measure. It is defined for each $\mathbf{X}^i$ as explained in Sec.5.2. However, the goal of anomaly detection for time series is usually to find anomalous regions of time stamps, and it seems that this task is actually considered in experiments (in Fig. 2b). Since the proposed measure cannot directly achieve the task, some clarification is required.

- There are also concerns regarding the experimental protocol. The authors say that a dataset is divided into training/validation/test sets. However, since the problem is unsupervised, such separation does not make sense, and one can just use the entire dataset excluding the ground truth labels for training and evaluate the prediction performance using the ground truth.

- Parameter sensitivity is not examined in experiments. Since parameter tuning is fundamentally difficult in unsupervised learning (as one cannot use CV), at least the sensitivity should be examined, otherwise it is not clear how the obtained results are useful and robust w.r.t. parameter changes. 

- Figure 4 has almost no information in its current state. Illustration should be improved.

summary_of_the_review: Although this paper studies a relevant problem, the quality, presentation, and evaluation of this paper are not convincing.
review_title: null

Meta-review: The paper tackles the problem of detecting anomalies in multiple time-series. All the reviewers agreed that the methodology is novel, sound and very interesting. Initially, there were some concerns regarding the experimental evaluation, however, the rebuttal and subsequent discussion cleared up these concerns to some extent and all reviewers are eventually supporting or strongly supporting acceptance.
Decision: Accept (Spotlight)