Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors developed an auto-encoder-based anomaly detection model called DyAD (Dynamic system Anomaly Detection) for time series anomaly detection. They primarily dealt with the high-dimensionality issue caused via viewing time series anomaly detection as hypothesis testing on dynamical systems. They demonstrated the performance of their model on both public dataset and on a newly prepared EV dataset.
strength_and_weaknesses: Strengths:
- The study demonstrates that a well-formulated probabilistic model can significantly increase sample efficiency and result in better model design.
- The dataset released by the authors could be useful for the community.
- The proposed model performed significantly better than the other candidate models.

Weaknesses:
- Properly formulating the probabilistic model underlying the data is not a brand-new concept. This is more prevalent in literature.
- Experiments are somewhat limited

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The authors make clear and concise statements about their motivation of the study. In my opinion, however, this is not a fresh idea that a well-formulated probabilistic model can make sample efficiency much higher and lead to better model design. The authors released the source code and an EV dataset. I believe the study is reproducible.
summary_of_the_review: This is an interesting study for time series anomaly detection. I believe the research community may find value from this study. Even though the authors released the source code, it would be helpful if they could briefly discuss their implementation details in the manuscript (maybe in the appendix). Please add a comma (,) after e.g. (page 6).

The study demonstrated state-of-the-art results in several datasets, including an EV dataset and two spacecraft datasets. Although the authors claim that they validated the efficacy of their model using these datasets, I have a little reservation about the diversity of the datasets and the generalizability of the proposed model. I would like the authors to extend their experiments on one or more general purpose tasks, such as energy systems, stock analysis and meteorological data to evaluate the robustness of the model. We cannot draw conclusions about the robustness of a model based on limited experiments.
 
Overall, I am bit concerned about the generalizability of the model since the experiments are not conducted on a variety of datasets. I therefore do not recommend accepting the paper in the current form considering a high bar to ICLR.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper suggests a specific method for anomaly detection in time series data. The approach is based on hypothesis testing. Given the distribution of the normal data and the empirical one from test data, the decision of normal vs. abnormal is made. The authors suggest as an application area the evaluation of the battery health of e-cars and provide a new data set for future benchmarks in anomaly detection. In the experiments, they evaluate their method on such data. In addition, two public benchmarks are used to compare with existing methods. The experimental results seem to support the claims of the authors
strength_and_weaknesses: Time series anomaly detection is an important area nowadays to monitor dynamic systems (for example, weather) and to gain insights into unknown processes by analysing detected anomalies. This usually comes with a continuous time series in which intervals that show anomalous behaviour/relationship of the dynamic process variables involved have to be detected. However, this paper seems to treat anomaly detection as an outlier detection method in static data built from a time series model, i.e. the whole time series is input to the system to determine whether it shows anomalous behaviour. This observation is a major criticism since the paper title is - at least for me - misleading. However, such preconditions/assumptions might be of importance in some applications, like the introduced battery health data set. But even in this case, the paper lacks originality and contribution to the area. Another criticism is the strong assumption of the Gaussianity of the involved variables. I wonder whether Kalman-Filter with control would do a similar job with a proper evaluation function of the model match. 
clarity,_quality,_novelty_and_reproducibility: The paper lacks novelty and originality. I am also not satisfied with the presentation of the work. I have difficulties identifying those parts of the work the authors treat as their original contributions. The data set's description is unclear to me since I am not an expert in this area. Also, the presentation of the results on the benchmarks in section 6 is insufficient to evaluate and reproduce the performance. 
summary_of_the_review: If the contribution by the authors is the solution for the battery system anomaly detection (Sec. 5), then it boils down to Algorithm 1 (basically an averaging operation) and a variance test. This is not sufficient for me as the contribution, even if there might be more complex problems around that might be tackled by their method. Furthermore, the theoretical part of their paper assumes Gaussianity for the time series process variables (page 6, enumeration 1-3). I cannot see any novelty in this since the authors do not relate such dynamic systems to the Kalman filter.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a novel anomaly detection scheme based on hypothesis testing, that is directed towards multivariate problems where higher dimensionality can prove to be challenging for statistical testing. The authors leverage an autoencoder architecture for modelling the underlying dynamical system, and come up with a robust approach for determining whether a time series is anomalous or not. The contributions of the paper are framed around a new dataset on the battery power of electric vehicles that was curated for this paper (and made publicly available), but the authors also feature additional experiments on more widely-used datasets such as SMAP. Notable performance improvements are observed across all settings. 
strength_and_weaknesses: - I appreciate that the authors have introduced a new complex dataset for anomaly detection, whose properties may overlap with time series encountered in other real-world domains. I believe that moving beyond the standard datasets used for evaluation is a particularly effective means of identifying new research problems and directions. 
- However, the paper treads a fine line between tackling an application-specific problem and proposing a more general solution to anomaly detection in multivariate time series datasets, which can be confusing at times. I believe the Battery System problem itself could be better explained early on in the paper via more illustrative examples of what a fault looks like. One other thing I was unsure of while reading the paper is whether the problem set-up of detecting anomalous vehicles (here described by several collected time series), would be better posed as a classification problem overall rather than time series anomaly detection. Given that the detection appears to be carried out offline, it is unclear whether there may be other more obvious means of detecting issues or degradations in this case.
- I found that there was a bit of a disconnect between sections 3 and 4, where it wasn’t always easy to understand how certain aspects introduced in Section 3 are then manifested in the architecture of DyAD.
- The use of auto-encoders for time series anomaly detection is fairly common, and I would have liked the novel aspects of this formulation to be more clearly expressed. The takeaways on why DyAD is more appealing than competing techniques was not clear to me by the end of reading the paper beyond the performance improvements obtained in the experiments.

clarity,_quality,_novelty_and_reproducibility: The assumptions and overall steps included in the constructoin of DyAD appear to be correct. As highlighted earlier, I think the paper’s overall writing could be improved in order for certain concepts to be more easily understood, especially the links between hypothesis testing and the overall DyAD architecture. Given how central the Battery Safety dataset is to the paper, I would also advise the authors to consider including more samples or illustrations that convey why the model architecture advised for in this work is especially suitable.
summary_of_the_review: This paper frames its contributions around a novel dataset that poses different challenges to other anomaly detection datasets that are usually considered in the literature. The authors also develop a new model (DyAD) and robust scoring procedure that outperforms several other competing techniques. Nevertheless, I believe that some of the concepts put forward in the paper are not developed clearly enough, and the novelty of the modelling contributions were particularly unclear while reading the paper. Overall, I would encourage the authors to revise how the paper is currently structured in order to reduce confusion in regards to these aspects, and highlight the contributions more clearly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an anomaly detection method in time series data, referred to as ‘DyAD’ that is based on the hypothesis testing on dynamical systems. The method leverages an autoencoder architecture for modeling the underlying dynamical system. All of reviewers agree that a new dataset for battery power of electric vehicles is valuable for community and the timely topic is tackled by a well-formulated probabilistic model. However, there are critical concerns that should be carefully considered for future submissions. The presentation should be improved. It is not clear which part is a new contribution since the approach in this paper is not a fresh idea. Sections 3 and 4 are not well connected. In some place, the notation is confusing. Since the use of an autoencoder is popular in anomaly detection, I would like to suggest the authors to improve Figure 1 to highlight how different the overall scheme proposed in this paper is from existing ones. Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments informative and can improve their paper by addressing these carefully in future submissions.

justification_for_why_not_higher_score: First of all, the presentation is not clear enough to highlight the main contribution.
Decision: Reject