Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors propose a 2 level meta-learning approach for recommendation of algorithm configurations in multivariate time series anomaly detection. At the first level, the decision is if a new time series is similar to the ones used for training. Depending on the result, a different meta-model is used to make the recommendation.

The evaluation separates two scenarios: new time series from known domain and new domain. Both scenarios make sense but the second is not usually considered.

The results show that the approach competes with the state-of-the-art and that the two-level approach is better than the two base-level recommendation models.
overall_review: The idea of separately handling similar and different time series in multivariate scenarios, both in modelling and evaluation, is quite interesting and should be followed.

However, even though the method is interesting, the motivation for the solutions incorporated into it are not entirely clear to me.

On the other hand, the evaluation methodology is quite interesting and should be followed in empirical studies concerning recommendation of algorithms multivariate time series.

More detailed comments are provided above.

Other comments:
-couldn't UReg optimize for ranking rather than for error?
-fig 2 not references in text

review_summary: Although I'm not entirely convinced by the method, I find the general approach interesting, particularly in what concerns evaluation.

actions_required_to_increase_overall_recommendation: Explain and motivate the method more clearly.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors propose Hydra, a meta-learning recommendation framework for novelty detection. Hydra uses two inner recommender systems to deal with: known and novelty data. The tool has an inner mechanism that automatically actives which recommender should be used.
overall_review: The paper proposes a new meta-recommender and presents a nice comparison to some related works in the specific area (meta-learning recommenders for novelty detection and time series). The methodology is also well described, and results are organized according to the research questions and present statistical evaluations. Results also present good implications for discussing Hydra's performance against simpler baselines and MetaOD.

On the other hand, it is missing comparisons versus cited baselines (LOTUS, ELECT). The authors of these studies could be contacted by email and asked about their tools.
review_summary: Based on the previous comments, I think the paper presents a good study, still in its initial, but with many questions that can contribute to the AutoML area in time series. Thus, my recommendation is to accept the paper.
actions_required_to_increase_overall_recommendation: ...
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes Hydra, a meta-learning-based model recommender for anomaly detection in multivariate time series data. The authors highlight the real-world restrictions that make existing methods unusable, and address the need for generalization to previously known or unseen data sources. The proposed approach outperforms the current state of the art, achieving a higher performance even in difficult scenarios where data similarity is minimal. The experiments are conducted on a large number of public datasets from different data sources, and the ablation study confirms the efficacy of Hydra in recommending models in the top 10% for known sources of data and the top 13% for unseen sources. This work is an important contribution to the field of unsupervised model recommendation for anomaly detection and provides a promising direction for future research in this area.
overall_review: Positive aspects:

The paper proposes Hydra, the first meta-recommender for anomaly detection in multivariate time series data, which is a novel contribution to the field of unsupervised model recommendation.
The paper addresses real-world restrictions, such as limited time to issue a recommendation and the need for generalization to previously unseen data sources, which are important considerations in practical applications.
The paper conducts a thorough experimental evaluation using a large number of public datasets from different data sources, and provides extensive analysis of the results.
The paper clearly presents the proposed approach, including the use of meta-learning and the design of Hydra, making it easy for readers to understand the methodology.

Negative aspects:

The paper could benefit from providing a more detailed discussion of the limitations and challenges of existing methods for unsupervised model recommendation for anomaly detection in multivariate time series data, to help readers better understand the motivation for the proposed approach.
The paper could provide more information on the specific design of Hydra, such as how the meta-learning framework is applied to the problem of model recommendation for anomaly detection, and how the proposed approach addresses the real-world restrictions and need for generalization to previously unseen data sources.
The paper could benefit from providing details of dataset split (train/val/test)
review_summary: Based on my assessment, I recommend accepting the paper with minor revisions. The paper proposes Hydra, a meta-recommender for anomaly detection in multivariate time series data, and conducts a thorough experimental evaluation to demonstrate its effectiveness. The proposed approach addresses real-world restrictions and provides a novel contribution to the field of unsupervised model recommendation. While there are some areas where the clarity could be improved, including a more detailed discussion of existing limitations and challenges, providing more information on the specific design of Hydra, and more detailed analysis of the experimental results, these issues can be addressed with minor revisions. Overall, the paper provides a valuable contribution to the field and warrants acceptance with minor revisions.
actions_required_to_increase_overall_recommendation: since the work claim that their work can be applied for real-world time series data, they should evaluate Hydra in a larger dataset and more complicated data distributions
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors presenta meta-learning based approach for AutoML on unsupervised anomaly detection in multivariate time series. The system `Hydra` is performs favorably copmpared to SoTA approaches, especially for novel or unseen data sources.
The architecture consists of (i) novelty detector (ii) model recommenders based on novelty in the meta-feature space. The recommender for novel  data relies on SVD and Random Forest to rank models. 
A large number of datasets are utilized for the meta-training phase, and catch22 is used for computing meta-features. A wide set of anomaly detection approaches are covered.Performance is demonstrated to be better than the current SoTA. The authors present a fairly comprehensive evaluation and ablation study as well.
overall_review: The paper has many strong points.
1. The core idea is robust, based on meta learning. The whole architecture with the novelty detection (of data) component and the recommender heads are well founded and make intuitive sense.
2. A wide ranger of anomaly detection algorithms are covered in the meta learning process. Also, many datasets are used, and hyperparameter tuning is done.
3. Empirical evaluation and analysis is performed adequately. The ablation study (Sec 5.3.1, Sec 5.3.2) is useful in providing the user understanding. 

Negative aspects:
1. Some statements are not very clear. For instance Sec 5.3.1,Line 274-Line 276.
2. Sec 5.4 provides computational requirements in an anecdotal fashion. A clear figure or performance numbers would be effective in this regard. 
3. Some design choices could be better explained. e.g. use of catch22. 
4. How are hyperparameters tuned for the recommended algorithm is not clear.
5. Meta learning based approaches (such as AutoSklearn 1, 2) have been successful. The authors concede that their system requires more data for better performance. How does that limit the utility of the current iteration is not clear. The concern that there is inadequate data is not addressed.
review_summary: The paper presents an effective solution for a interesting and practical problem. The paper presents some good insights , and is well written and the formulation is sound. However a few points that need to be addressed remain as noted.
actions_required_to_increase_overall_recommendation: Provide computational overhead figures in a more comprehensive manner.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The authors provide a "recommender system" called hydra to predict which anomaly detector to use on a new dataset, given the past performance data of the set of detectors on other datasets. 
Their main contributions lie in 
a. applying an anomaly-specific set of meta-features
b. identifying clusters in the (projected) meta-feature space and selecting a recommender that best suits this cluster.
c. identifying if a new dataset is not in the vicinity of the training distribution. In that case 


overall_review: Strengths:
1. The approach can distinguish in the meta-feature space whether it has encountered similar datasets (based on the distance in the projected meta-feature space). This way, the model can exploit more specialized recommenders for particular sets of the meta-feature space. 

Weaknesses
1. I would like to start by pointing out that the name of the method Hydra is not well motivated and that it easily is confused with a widely used python package [hydra.cc](hydra.cc) which is used for configurable experimentation. I recommend changing the name.

2. The definition in L.44 of what meta-learning is appears to be wrong. Instead, this seems to be a description of algorithm selection.
This is also one of my concerns regarding how the paper is situated; while technically correct, that a recommendation is made and similarities exist in this line of thought, the entire outline rather strikes me as an algorithm selection problem. This phrasing is more easily referable in this research area. Crucial here is that the configuration (algorithm and its hyperparameters) is a discrete set for which we want to predict a performance given some meta-experience (performance data) with that set. Also, the section headers and paragraphs 3., 3.1, 3.2 simply describe an algorithm selector. This description also facilitates the unfortunately complex title "Unsupervised Model Recommendation through Meta-Learning"

3. As with most algorithm selectors, the dependency on meta-features eventually limits the performance of a selector by the quality of these features and how well they are "aligned"/indicative of the performance. The time-series aspect of this paper can mostly be reduced to a selection of time-series targeted meta-features. In effect, this opens up a lot of baselines from the AS literature to compare against. 

4. I acknowledge, that in AS, datasets are hard to come by, but I wouldn't go as far as claiming that 4 benchmarks are an unseen extent.

5. A potentially important hyperparameter regarding the complexity in both functional expressivity and cost of this method lies in the number of singular vectors. Another set of important hyperparameters are those regarding a) the UMAP projection and b) the decision on how to cluster the meta-feature. Effectively, this will change the model's performance.
review_summary: I would think that this paper should be placed in the algorithm selection setting and may require considerable rewriting (that using this perspective will simplify the paper) and also require more baselines from the AS field. Any potential empirical gain should be demonstrated against those baselines.
actions_required_to_increase_overall_recommendation: Look at the weakness section.
review_title: null

Decision: Accept(Main Track)