Title: PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Anomaly  Detection

AI Agent

Agentic AI Engineer

Multimodal Large Language Model
Abstract: Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios. Previous methods typically rely on domain-specific training data and manual adjustments when applying to new scenarios and unseen anomaly types, suffering from high labor costs and limited generalization. Therefore, we aim to achieve generalist VAD, \ie, automatically handle any scene and any anomaly types without training data or human involvement. In this work, we propose PANDA, an agentic AI engineer based on MLLMs. Specifically, we achieve PANDA by comprehensively devising four key capabilities: (1) self-adaptive scene-aware strategy planning, (2) goal-driven heuristic reasoning, (3) tool-augmented self-reflection, and (4) self-improving chain-of-memory. Concretely, we develop a self-adaptive scene-aware RAG mechanism, enabling PANDA to retrieve anomaly-specific knowledge for anomaly detection strategy planning. Next, we introduce a latent anomaly-guided heuristic prompt strategy to enhance reasoning precision. Furthermore, PANDA employs a progressive reflection mechanism alongside a suite of context-aware tools to iteratively refine decision-making in complex scenarios. Finally, a chain-of-memory mechanism enables PANDA to leverage historical experiences for continual performance improvement. Extensive experiments demonstrate that PANDA achieves state-of-the-art performance in multi-scenario, open-set, and complex scenario settings without training and manual involvement, validating its generalizable and robust anomaly detection capability. Code is released at https://github.com/showlab/PANDA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a training free model for VAD called PANDA. An LLM acts as the core reasoner. A four stage pipeline has been introduced to first pull scene-specific rule snippets from text, then a language-vision model evaluates videos with those rules. After this, there is a reflection module to use external tools such as super resolution if the confidence is low and then re running the evaluation. Finally a memory system is implemented to guide the later decisions.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The framework design, combining planning, reasoning, utilization of external tools and memory for generalist VAD is compelling idea, which is extending from LLM agents to the video domain.\n\n- There are comprehensive empirical study with four datasets and ablations on design choices.\n\n- The paper is very clear and easy to follow.\n\nWeakness:\n\n- My main concern is the real-time performance of the model. In VAD we inherently rely on real-time systems. Catching an anomaly in a reasonable time frame is an important aspect of VAD. The authors report average inference speeds of 0.53–0.86 FPS across four datasets. In PANDA the raw video stream is first down-sampled to one frame per second and inference is carried out on 5-frame clips drawn from that 1 FPS stream. The authors (and your supplemental table) report effective throughputs of 0.53 – 0.86 analysed frames per second. To my understanding for example at 0.86 FPS the agent processes each sampled frame in about 1.16s, so the backlog grows slowly. In practice the display or decision stream will drift behind real time by roughly one second per minute of video. At the lower end, 0.53 FPS approximately doubles that drift. For time-critical response scenarios the agent would require further optimization. If the authors could clarify this or provide comparison with other models mentioned in the paper it would strengthen the papers promise.\n\n- Another concern is that the model still depends on a manually curated anomaly-rule knowledge base. Authors need to clarify on the effectiveness and the cost of building and maintaining this base, and its effect on generalisation to unseen anomaly categories. I understand that the domain seems to be anomalous \"actions\" and not dealing with unseen or novel anomalies, thus I need clarification with this point.\n\n- There are some minor typos such as \"planing\" or \"Kowleage\" in table 3. It would be great if authors can go over the paper for a final clean up.\n\n- One last minor concern is that, the model relies mostly on already developed models and the contribution is limited to the design of the system.\n\nQUESTIONS:\nPlease refer to the weaknesses part, but here is the main questions:\n\n1- The first question I have is regarding the real-time performance of the model. Please refer to the first weakness.\n\n2- I would appreciate it if authors can clarify on the unseen anomalies and also the effect of varying the number of rules. Evaluate PANDA on anomaly categories absent from the knowledge base to reveal generalisation behaviour is very helpful.\n\n3- Please estimate annotation time per anomaly category or provide an automated rule-generation procedure, and discuss how the knowledge base scales to hundreds of categories. Clear evidence that the rule set can be created with minimal manual effort, or that it can be learned automatically would help the decision.\n\n4- Please polish the presentation of the paper and fix spelling issues etc.\n\nLIMITATIONS:\nThe societal-impact section acknowledges surveillance use but does not analyse privacy risks or potential misuse. A brief discussion would help.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a solution to make multimodal large language models (MLLMs) behave like a detective with 4 modules: (1) a RAG to make them have strategy planning ability, (2) a heuristic prompt strategy to make them have goal-driven heuristic reasoning, (3) a progressive reflection mechanism to make them have self-reflection, and (4) a chain-of-memory mechanism for self-improving ability. The paper tests the methods on representative datasets including UCF-Crime, XD-Violence, UBnormal, and CSAD.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-motivated to make AIs behave like detectives in video anomaly detection. The paper includes additional modules to equip the model with the desired ability, and the selection of them is reasonable.\n2. The paper tests the proposed method in various datasets and presents a comprehensive result for study.\n\nWeaknesses:\nThe main concern for this paper is that it doesn't illustrate the necessity of why we must have those modules, and in fact these modules actually could be redundant. \n1. First, the paper doesn't have concrete reasons to explain why we must use those techniques to model the four abilities in detectives. Whether they are optimal design is not discussed. The current solution is more like a combination of hot techniques without illustrating why they are essential in design.\n2. Second, these modules may not be necessary because an important baseline is missing. This paper is categorized into training-free methods but it doesn't compare with another method not fine-tuning LLMs published in CVPR'25. (VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models). The proposed method uses many additional modules to achieve VAD, however, based on the performance on XD-Violence and UCF-Crime, it doesn't outperform VERA which simply uses good prompts obtained from verbalized learning between LLMs. These results could indicate that actually all the introduced modules in this paper is not necessary and a good prompt could outperform them. The paper needs a more thoughtful discussion on models not using fine-tuning with more works such as VERA.\n3. Third, the additional computation overhead for including these modules are not discussed in this paper. This is an obvious trade-off for including those modules and need detailed discussion.\n\nQUESTIONS:\n1. Why we must use those modules to model the desired ability? Are they optimal? What are the other solutions?\n2. Are these modules really necessary? VERA has an AUC~86% in UCF-Crime and just need good prompts for MLLMs. \n3. What's the additional computation overhead will be brought by these modules?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to develop a generalisable video anomaly detection method which should handle any scene and any anomaly types without training data or human involvement.  Self-adaption and scene-aware strategy planning,  goal-driven heuristic reasoning,  tool-augmented self-reflection, and  self-improving chain-of-memory are the four main components of the proposed method.  Extensive experiments demonstrate that the proposed method (dubbed as PANDA) achieves state-of-the-art performance in multi-scenario, open-set, and complex scenario settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis work proposes PANDA, training-free VAD that can automatically handle any scene and any anomaly types without training data or human involvement.  PANDA is evaluated on four VAD benchmarks representing three distinct setting multi-scenario, open-set, and complex scenario settings.\n\nWeekness:\n1. In the implementation details, it is mentioned that “the maximum number of reflection rounds r is set to 3.” The paper suggests that PANDA is able to assign a normal or anomalous score to all previously insufficient samples within these rounds. However, intuitively, there may be hard cases where even after three reflection rounds, PANDA might still fail to confidently assign a label. Could the authors clarify how such unresolved or ambiguous cases are handled, if any remain after the maximum number of reflection rounds?\n2. All the methods compared in Table 1 perform anomaly detection in an offline setting. As stated in the supplementary material (L19–L20), \"If future information is accessed, the method is considered offline; otherwise, it falls under the online setting.\" However, PANDA online setting still outperforms offline, training-free methods. This seems counterintuitive. Could you clarify why PANDA achieves such strong performance even in the online setting?\n3. An analysis of how frequently the reflection mechanism is invoked i.e., the number of clips requiring reflection rounds would be valuable. Additionally, it would be insightful to report how many clips transition from \"insufficient\" to either \"normal\" or \"anomalous\" after each reflection round. Finally, please clarify how many clips, if any, remain in the \"insufficient\" category even after the maximum of three reflection rounds.\n4. Figure 2 lacks clarity and does not accurately reflect the proposed method. According to the methodology, when the VLM reasoning status is marked as “insufficient,” the MLLM reflection module is invoked. However, this is not clearly depicted in the figure. Instead, the arrow from the VLM reasoning block appears to lead directly into another instance of VLM reasoning, which is misleading. To improve clarity, the figure should explicitly illustrate the transition from VLM reasoning to MLLM reflection in the case of an “insufficient” outcome. Additionally, incorporating step numbers into the diagram would help convey the sequential flow of the pipeline more effectively. As it stands, the figure may confuse readers rather than aiding in their understanding of the method.\n5. Since LongCoM evolves over time, it initially lacks any prior information about insufficient cases. How does the Experience-Driven Reflection module handle such scenarios where no historical context is available?\n6. L241-242 is ambiguous “In offline reasoning mode, the perception phase is sampling M = 300 frames uniformly for the whole video, while only the initial M = 10 frames are sampled in offline mode.” please clarify this statement.\n7. PANDA incorporates several tools to enhance video content analysis. Could you elaborate on how these specific tools were selected, and what criteria guided the choice of each component?\n8. Is the anomaly knowledge subjective to each dataset?\n9. In Supplementary Figure 1, which visualizes the detailed PANDA pipeline, the Short-CoM component appears to be missing. For completeness and clarity, it would be helpful to include Short-CoM in the diagram as well.\n10. Supp. Figure 5, where does the “{formatted_enhancement_prompt} come from?\n11. How the proposed method compares to unsupervised VAD such as: \"Generative cooperative learning for unsupervised video anomaly detection\" in CVPR 2023?\n\nQUESTIONS:\nPlease respond to the weaknesses mentioned above.\n\nLIMITATIONS:\nFailure cases are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PANDA, a new paradigm for video anomaly detection (VAD) targeting general scenarios and open anomaly types. PANDA leverages multimodal large models (MLLMs) and simulates a detective-like “perceive-reason-reflect-learn” decision process. Through adaptive scene perception, goal-driven reasoning, tool-augmented reflection, and self-improving memory chains, PANDA achieves anomaly detection that requires no training or manual adjustments, adapting to different scenes and anomaly types. Extensive experiments on multiple datasets (UCF-Crime, XD-Violence, UBnormal, CSAD) covering diverse, open, and complex scenarios show that PANDA achieves SOTA-level performance without any training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces a “detective agent” paradigm, incorporating RAG (retrieval-augmented generation), chain-of-thought self-reflection, tool-chain support, and memory chain mechanisms into VAD, achieving anomaly detection without training or manual prompt engineering.\n2. The approach breaks free from the heavy dependence of traditional VAD methods on specific scenes, event types, rules, or models.\n\nWeaknesses & Questions:\n\n1. The extensive use of RAG, repeated reasoning, and tool-chain calls in PANDA incurs significant inference latency and computational cost—preliminary estimates suggest FPS is even less than 0.1. The authors should provide detailed inference timings and discuss this issue, as high computational cost could significantly limit practical use.\n2. The novelty is limited; the core idea is to replace the prompts in existing MLLM-based VAD methods (such as SUVAD, LAVAD, VERA, etc.) with RAG and chain-of-thought techniques.\n3. During the lengthy reasoning process, model hallucinations may be amplified at each step. The authors should conduct case studies and thoroughly explain whether severe hallucinations occur and how much they affect result correctness.\n4. Technical details are lacking; the authors should supplement detailed implementation of RAG and tool invocations.\n\nQUESTIONS:\nPlease respond to each item of the weaknesses mentioned above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}