Swift Hydra:  Self-Reinforcing Generative Framework for Anomaly Detection with Multiple Mamba Models

Authors: Nguyen Hoang Khoi Do, Truc Nguyen, Malik Hassanaly, raed alharbi, Jung Taek Seo, My T. Thai
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Anomaly detection, Reinforcement learning, Generative AI

Abstract:
Despite a plethora of anomaly detection models developed over the years, their ability to generalize to unseen anomalies remains an issue, particularly in critical systems. This paper aims to address this challenge by introducing Swift Hydra, a new framework for training an anomaly detection method based on generative AI and reinforcement learning (RL). Through featuring an RL policy that operates on the latent variables of a generative model, the framework synthesizes novel and diverse anomaly samples that are capable of bypassing a detection model. These generated synthetic samples are, in turn, used to augment the detection model, further improving its ability to handle challenging anomalies. Swift Hydra also incorporates Mamba models structured as a Mixture of Experts (MoE) to enable scalable adaptation of the number of Mamba experts based on data complexity, effectively capturing diverse feature distributions without increasing the model’s inference time. Empirical evaluations on ADBench benchmark demonstrate that Swift Hydra  outperforms other state-of-the-art anomaly detection models while maintaining a relatively short inference time. From these results, our research highlights a new and auspicious paradigm of integrating RL and generative AI for advancing anomaly detection.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729748378583

thread_type:
official_review

<thread_object>
creator: reviewer_uiuh

<content>
summary:
This paper proposes Swift Hydra, a novel anomaly detection framework based on the conditional variational autoencoder (CVAE) and reinforcement learning.
The proposed method generates high-quality anomalies using CVAE and reinforcement learning, and performs anomaly detection using a mixture of expert models based on Mamba (MoME). 
Experiments on various datasets demonstrated improved anomaly detection performance and reduced inference time compared to existing methods.

strengths:
- This paper introduces several strategies to achieve high-accuracy anomaly detection. For example, the reward function in reinforcement learning incorporates the entropy of the union of the training data and the generated data, which encourages the generation of diverse anomalies. Additionally, in the MMoE model, experts are selected probabilistically to avoid the winner-take-all problem.
- The visualization of the generated anomalies is interesting. It clearly shows a clean separation between normal data and anomalies, and we can also observe the generation of a wide variety of anomalies.

weaknesses:
- Although the proposed method works well, there are a few parts that are unclear to me. I will write more details in the Questions section, so please refer to it.

questions:
1. Why does the proposed method use CVAE? Since VAE generally has lower data generation quality compared to other generative models, I don't think it's very suitable for data augmentation purposes in this case. For example, would the proposed method work well with a diffusion model, which can be considered a special case of VAE?
2. The proposed method performs anomaly detection in the data space, but wouldn’t it be more beneficial to take advantage of CVAE’s strengths by performing anomaly detection in the lower-dimensional latent space? What is the reason for performing anomaly detection in the data space?
3. I understand that MMoE performs binary classification between normal and anomaly data. In binary classification, a decision boundary is set between the normal and anomaly data included in the training set, which makes it difficult to handle unseen anomalies that are far from the known anomalies in the training data. Can the proposed method address this issue?
4. If the proposed method can handle unseen anomalies, I believe it is because CVAE can generate such unseen anomalies. Why is the proposed method able to generate unseen anomalies? Even though the entropy is incorporated into the reward of the reinforcement learning, I think generative models can only generate data similar to the training data.
5. Is it possible to visualize the generated samples and decision boundaries of the proposed method using two-dimensional toy data? For example, if there is normal data shaped like a sine curve, and the generated anomalies are arranged surrounding that curve, I believe the method would be able to handle both seen and unseen anomalies.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730193142497

thread_type:
official_review

<thread_object>
creator: reviewer_8b4d

<content>
summary:
This method focus on generalization to unseen anomalies. To do that, this method generates new anomaly samples using reinforcement learning and conditional VAE. The conditional VAE generate samples conditioned on the class anomaly. The reinforcement learning enforces the generated samples to be diverse and difficult to detect by the detection model. For the detection model, in order to keep the complexity low while can capture diverse feature distribution, Mixture of Experts consisting of multiple Mamba experts are utilized.

strengths:
Adding new anomalous synthetic data during training to increase generalization using reinforcement learning seems to be novel.

weaknesses:
See questions.

questions:
1. Which part of the experiments that prove the generalization to unseen anomalies?

2. It is unclear whether this work use both real and fake data or only real data. I can't find the information in Line 94-105. My guess from other part of the paper is using both real and fake, but I am not 100\% sure. 

3. Why is this method called reinforcement learning method if the action is (afaik) backpropagatable directly from the reward/loss? 

4. How is the balanced dataset updated? I can't find any information in Algorithm 1 in the appendix.

5. Why not also generating normal data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730201822246

thread_type:
official_review

<thread_object>
creator: reviewer_9izt

<content>
summary:
This paper proposes a novel tabular anomaly detection method by designing a RL-based anomaly data augmentation technique and a Mixture of Experts (MOE) based anomaly detection module. The authors verified the effectiveness of their proposed method with 3 baselines on ADBench and also provided visualized proofs of the effectiveness of their reward function and anomalous data augmentation technique.

strengths:
1. Overall well written. The paper is easy to follow and organized in a clear orchestration. 
2. Novel idea. This paper proposes a novel idea that using RL to make anomalous data augmentation. Besides, the authors also prove the effectiveness of their proposed reward function by showing the consistency between AUC-ROC and the reward.
3. It is proved that the RL-based anomaly data generation method can generate anomalies of high diversities, as show in Fig 2.

weaknesses:
1. Weak soundness of experiment. The experiments are not persuasive enough. The baselines do not cover enough SOTA tabular anomaly detection methods. The authors only compares the proposed methods with 3 baselines. Besides, there is no ablation study to verify the effectiveness of RL-based anomaly data generation technique and the MOE anomaly detection module separately.
2. Unrealistic assumption in mathematical proof. In the mathematical proof of theorem 2 and theorem 3, it is actually assumed that single anomaly detector can not distinguish anomalies and normality in overlap region. However, this assumption is only available for linear classification methods (e.g. support vector machine). The deep neural networks are non-linear classification methods and can deal with interlaced anomalies and normality in  overlap region well. In this situation, theorem 2 and 3 can not be deduced.
3. The illustration of the process of "gating network" and "Tackling winner-take-all" is somewhat ambiguous. What confused me is that given both of these two mechanism actually are used to assign input to different expert, how could we use them together?

questions:
1. In theorem 1, do you mean $\hat{x}_i=\mathcal{M}_\phi(\hat{z}_i,y_i=1)$?
2. Could you please add more SOTA tabular anomaly detection methods?
3. Could you please add ablation experiments?
4. In Table 1, why Swift Hydra (MOME) has less inference time than Swift Hydra (Single)?
5. Could you please further explain how to use "gating network" and "Tackling winner-take-all" together?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628832287

thread_type:
official_review

<thread_object>
creator: reviewer_rkhh

<content>
summary:
This paper presents a novel framework, Swift Hydra, for training an anomaly detection method that leverages generative AI and reinforcement learning. It introduces Mamba models structured as a Mixture of Experts (MoE), allowing for scalable adaptation of the number of Mamba experts based on data complexity. Experiments conducted on the ADBench benchmark demonstrate that Swift Hydra outperforms other state-of-the-art anomaly detection models while maintaining a relatively short inference time.

strengths:
1.The writing in this paper is clear and well-structured.

2.The incorporation of reinforcement learning, conditional VAE, and Mixture of Experts (MoE) is novel and well-justified.

3.The paper conducts thorough experiments on ADBench and demonstrates strong performance.

weaknesses:
1.Typically, experiments relying on only one dataset are insufficient. It would be beneficial to include additional analysis and comparisons using other popular datasets.

2.It appears that Swift Hydra is not specifically designed for tabular anomaly detection. Have you explored the generalizability of this method to other domains, such as image data or time-series data?

3.The paper introduces several complex components, such as reinforcement learning, which may complicate implementation. Additionally, could you provide an analysis of the computational cost associated with training, including metrics such as the number of parameters and training time per batch?

4.How does the performance of Swift Hydra fare in multi-class or cross-dataset scenarios?

questions:
See weakness above. I will consider raising my score if my concerns are addressed well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732847915162

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers and Area Chair,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

In this work, we present a new framework, Swift Hydra, for training an anomaly detection method that leverages Reinforcement Learning to guide Generative AI to generate diverse and challenging data. This approach enables the detector to better generalize to unseen anomalies. Additionally, we introduce Mamba models, structured as a Mixture of Mamba Experts (MoME). This design allows for scalable adaptation of the number of Mamba experts based on the complexity of the data.

As highlighted by the reviewers, our method is both novel and well-justified (RKHh, 9izT, 8B4d), providing interesting insights (uiUH), and our paper is well-written and well-structured (RKHh, 9izT). We are grateful for your constructive feedback, which has been instrumental in improving our manuscript. In response, we have incorporated our discussion into the manuscript to further strengthen it as follows:
________________________________________
**Clarifications**

- **Dataset Diversity (Reviewer RKHh)**. Reviewer RKHh expressed concern that Swift Hydra was only tested on a single dataset, ADBench. We clarified in the main paper that ADBench actually consists of 57 datasets spanning diverse domains, including Image, Healthcare, and Natural Language Processing (lines 340-343). Additionally, we included a detailed table in Appendix C.6 that summarizes the number of data points, the percentage of anomalies, and the domain of each dataset. 

- **Model Size and Training Cost (Reviewer RKHh)**. Reviewer RKHh requested a discussion on model size and training cost. In response, we provided a detailed analysis in Appendix C.1.2, outlining the number of parameters and training time for each module, including the CVAE, Gating Network, Experts, Single Detector, and others.

- **Winner-Take-All Behavior (Reviewer 9izT)**. We clarified the occurrence and mechanism of the winner-take-all behavior in the MoE and its gating network (lines 297–302). Furthermore, we detailed how Swift Hydra addresses this phenomenon. Additionally, we emphasized when the gating network is deactivated (lines 303–304) and when it is activated (lines 335–336).

- **Typographical Corrections (Reviewer 9izT)**. We fixed the typo in line 242, as pointed out by reviewer 9izT.

- **Backpropagation on the Reward Function (Reviewer 8B4d)**. To address reviewer 8B4d's concern, we clarified the conditions under which direct backpropagation on the reward function can be used to identify feasible actions (i.e., early training episodes) in lines 251–255.

- **Algorithm Updates (Reviewer 8B4d)**. Following reviewer 8B4d's suggestion, we updated the pseudocode in Algorithm 1 (Appendix A.1) to specify how the balanced dataset is updated in lines 851-854.

- **Trim Function (Reviewer 8B4d)**. We explained the circumstances under which the Trim function (a helper function) might trim out generated data and detailed the strategies for handling such cases (lines 877-884).

- **Normal Data Generation (Reviewer 8B4d)**: We added a point that Swift Hydra can be utilized for generating and synthesizing data in other application contexts where collecting real data is expensive and scarce (lines 534–535), as discussed with Reviewer 8B4d.
________________________________________
**Experiments**

- **Comparisons with SOTA Methods (Reviewer 9izT)**. Following reviewer 9izT's suggestion, we conducted additional comparisons of Swift Hydra with more state-of-the-art (SOTA) detection models on the ADBench benchmark. A detailed analysis of these comparisons is included in Appendix C.2.1.

- **Ablation Studies (Reviewer 9izT)**. Based on reviewer 9izT's feedback, we reorganized the ablation study section and conducted additional experiments (Appendix C.4) to evaluate:
  - The impact of the Self-Reinforcing Module (Appendix C.4.1).
  - The effectiveness of probabilistic cluster assignments (Equation 6) (Appendix C.4.2).
  - The influence of the KL term and reconstruction term in Equation 2 on the AUC-ROC of Swift Hydra (Appendix C.4.3).

- **Toy Data Experiment (Reviewer uiUH)**. As suggested by reviewer uiUH, we added a two-dimensional toy data experiment to illustrate the generalization ability of Swift Hydra in Appendix C.3. This experiment demonstrates how the generated unseen anomalies tighten the decision boundary of MoME around normal data.

In the revised manuscript, these updates are temporarily highlighted in blue for your convenience. Once again, we sincerely appreciate your thoughtful feedback and the opportunity to improve our manuscript.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Overview.** The work addresses a weakly-supervised anomaly detection problem where incomplete set of anomaly examples are given for training detectors in a way that can generalize to unseen anomalies. It introduces a method that utilizes reinforcement learning to guide generative models (conditional VAE) to generate diverse synthetic anomaly samples, and then uses the generated samples to train a MoE + Mamba-based anomaly detector. The method is verified on ADBench.

**Pros.** 
- Well-written paper [RKHh, 9izT]
- The method effectively leverages RF, conditional VAE, MoE, and Mamba for the studied AD setting, and its effectiveness is empirically and theoretically justified [RKHh, 9izT, 8B4d, uiUH]
- The method shows promising results on tabular data benchmark ADBench [RKHh]
- The visualization results justify the quality of generating diverse anomalies [9izT, uiUH]

**Cons.**
- Only one dataset is used in the experiments [RKHh]
- The method seems data-type-agnostic, but its performance on handling complex structure/dependency in non-tabular data such as image/time series data is unclear [RKHh]
- Lack of computational complexity analysis [RKHh]
- Missing state-of-the-art competing methods on tabular data [9izT]
- No ablation study to verify the effectiveness of RL and MoE [9izT]
- Unrealistic assumption in theorem 2 and theorem 3 [9izT]
- The role and the advantages of the proposed modules against existing similar methods are unclear [uiUH]