HAWK: Learning to Understand Open-World Video Anomalies
Authors: Jiaqi Tang
Hao LU
RUIZHENG WU
Xiaogang Xu
Ke Ma
Cheng Fang
Bin Guo
Jiangbo Lu
Qifeng Chen
Ying-Cong Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Anomalies Understanding
Primary area: machine_vision
TL;DR: A large vision-language model for understanding open-world video anomalies.

Abstract: Video Anomaly Detection (VAD) systems can autonomously monitor and identify disturbances, reducing the need for manual labor and associated costs. However, current VAD systems are often limited by their superficial semantic understanding of scenes and minimal user interaction. Additionally, the prevalent data scarcity in existing datasets restricts their applicability in open-world scenarios.
In this paper, we introduce HAWK, a novel framework that leverages interactive large Visual Language Models (VLM) to interpret video anomalies precisely. Recognizing the difference in motion information between abnormal and normal videos, HAWK explicitly integrates motion modality to enhance anomaly identification. To reinforce motion attention, we construct an auxiliary consistency loss within the motion and video space, guiding the video branch to focus on the motion modality. Moreover, to improve the interpretation of motion-to-language, we establish a clear supervisory relationship between motion and its linguistic representation. Furthermore, we have annotated over 8,000 anomaly videos with language descriptions, enabling effective training across diverse open-world scenarios, and also created 8,000 question-answering pairs for users' open-world questions. The final results demonstrate that HAWK achieves SOTA performance, surpassing existing baselines in both video description generation and question-answering. Our codes/dataset/demo will be released at https://github.com/jqtangust/hawk.

HAWK: Learning to Understand Open- Video Anomalies
Jiaqi Tang Hao Lu Ruizheng Wu Xiaogang Xu
Ke Ma
Cheng Fang" Bin Guo" Jiangbo Lu Qifeng Chen² Ying-Cong Chen 1.2.3
The Hong Kong University of Science and Technology (Guangzhou)
The Hong Kong University of Science and Technology *HKUST(GZ) SmartMore Joint Lab
'SmartMore Corporation 'The Chinese University of Hong Kong *Zeejanng University
Northwestern Polytechnical University
{jtang092, hlu585) www.sett..uutttu..o edw.g..edd.co
(ruizheng wu, siangbo)@smartmorr com xiaogangxu00ggmal..
(2544552413, sura)@mail. edu. cqfoust.hi yingcongchennhkuust--gg edu.
guobenupu. edu. en
Abstract
Video Anomaly Detection (VAD) systems can autonomously monitor and identify disturbances, reducing the need for manual labor and associated costs. However, current VAD systems are often limited by their superficial semantic understanding of scenes and minimal user interaction. Additionally, the prevalent data scarcity in existing datasets restricts their applicability in open-world scenarios. In this paper, we introduce HAWK. K, a novel framework that leverages interactive large Visual Language Models (VLM) to interpret video anomalies precisely. Recognizing the difference in motion information between abnormal and normal videos, HAWK ex- plicitly integrates motion modality to enhance anomaly identification. To reinforce motion attention, we construct an auxiliary consistency loss within the motion and video space, guiding the video branch to focus on the motion modality. Moreover, to improve the interpretation motion-to- languugg,, we establish a clear supervi- sory relationship between motion and its linguistic representation. Furthermore, we have annotated over 8,000 anomaly videos with language descriptions, enabling effective training across diverse open-world scenarios, and also created 8,000 question- question-answering answering pairs for users' open-world questions. The final results demon- strate that HAWK achieves SOTA performance, surpassing existing baselines in both video description generation and ineestion-answrring Our codes/dataset/demoo will be released at https: //g github.... com/ jttaggstttaaae
Introduction
"Have eyes like HAWK!" Longman Dictionary
In recent years, the deployment of Video Anomaly Detection (VAD) systems has seen significant uptick across a diverse array domains, including but not limited to, autonomous driving [45, 25], surveillance [6, 23], and crime scene analysis [33]. The inherent capability of these systems to autonomously monitor and identify disturbances within a scene has markedly diminished the reliance manual labor, thereby streamlining operational efficiency and reducing associated costs.
"Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A) Traditional Video nomaly Detoction
Provious Video Understanding
Resparee
Unclear Zone
Model
This video
True OR False
with cars driving on a snowy road. The scenery is quite dark. with and log, and the occesional variety shots
Two- Categocies Classileer
"Car Collision
Histerdeding
driving on the snowy mad in different situations.
Video
Resporese:
Class I: Collision Storting
Koui
the curmett vehice falls to brice time road conditioss. The divee is unable to stop befom colldinng cousing minor accident that cuuld potentillly to the vehicle and a traffic stop for the velicles on the road...
Class 2: Collision at W Waiting
slippery
Class [Detected]
Faa Tuekng Roj
Obstacle at Roadway
Anemale
Class N: Out-of-Conter
Multi Categories Classiiiee
A nomaly Zone (yy Motion)
Armoaa Labes
in damage
Vidao
(B) Video A nommayy Detection & Classification
(D) Video Understamding for A Anomaly (Ours)
Figure 1: Different framework in video anomaly detection. (A) shows traditional video anomaly detection methods, which use binary classifiers detect anomalies. (B), following (A), introduces multi-class classifier for integrating semantic information, allowing users to obtain different types of anomaly information. Neither (A) nor (B) can interact with users. (C) is previous video understanding framework that can interactively provide richer semantic information for users, but cannot specifically locate video anomalies. Our framework (D) enhances the anomaly understanding capability and provides annotated labels with rich semantic information.
Despite the extensive focus on anomaly detection in most existing VAD systems [23, 44, 33, 31, 9, 13, 19, 34, 40, 48, 52] (as shown in Fig. I (A)), there is often lack of deeper semantic understanding of the scenes and insufficient interaction with users. While et al. [31] and Wu et al. [42] incorporated semantic information for video anomaly detection, their frameworks are limited as multiple-class classifiers (as displayed in Fig. 1 (B)). Consequently, the functionality of these systems is confined to the detection of anomalous frames, necessitating further manual analysis by users to analyze the detected anomalies comprehensively. Although Lv et al. [27] has pioncered the development of a large language model for the video anomaly explanation, their approach primarily relies pseudo labels for training. The lack of robust training data severely constrains its practical applicability. Besides, such method focuses more acquiring long-range context information rather than anomaly-related features on anomaly understanding (as exhibited in Fig. (C)).
To solve the above challenges, we propose an interactive large visual-language model [21, 18, 29], HAWK, for precisely understanding video anomalies (as illustrated in Fig. 1 (D)). Considering that the motion in normal and abnormal videos is significantly different [44, 52], explicitly integrate motion modality by a dual- bual-branch framework in HAWK to enhance the understanding of anomalies (Section 4.1). Besides, to reinforce motion attention, we construct auxiliary consistency loss based on the mutual information between the original video (appearance feature) and its motion in tight space (Section 4.2), to implicitly guide the video branch to focus on motion-related features. However, the interpretation of motion to the corresponding language remains unclear. Therefore, extract the motion-related language (verbs and their entities) from the original description to directly supervise the visual and linguistic representations of motion, for accurately enhancing the interpretation of video anomaly in HAWK (Section 4.3).
Furthermore, we also collect seven video anomaly datasets from various scenarios and generate language descriptions for each video. Besides, to address the open-ended questions raised by users, we utilize language descriptions of the videos to generate potential question-answer pairs for training. Since these datasets cover range of scenarios (Section 3), including crime (UCF-Crime [33]), campus environments (ShanghaiTechh Tech [22] and CUHK Avenue [23]). pedestrian walkways (UCSD Pedi [6] and Ped2 [37]), traffic situations (Do'AA [45]), and human behavior (UBnormal [2]). and finally, the model tends to generalize to open-world scenarios.
To train our framework, we initially pre-train it on WebVid [3] to equip it with the capability to understand general videos. Then, we fine-tuned on our proposed video anomaly dataset to enhance its understanding of video anomalies across multiple scenarios. Compared to other baselines, our

---

model achieves SOTA performance in both Text-Level and GPT-Guided Metrics. Our contributions are summarized as follows:
We propose a novel video-language framework, HAWK, aiming at understanding video anomalies, which incorporates motion modality enhance its capability.
We generate rich language descriptions for seven different video anomaly datasets. Mean- while, considering the diversity of open-world problems, we also generate question-answer pairs to tackle potential user inquiries.
Compared to other large video models, our framework demonstrates SOTA performance for video anomaly understanding and question- -anoonn-nnweinngg across multiple scenarios, which will help open- woold anomaly understanding in the future.
2
Related Work
Video Anomaly Detection Video Anomaly Detection (VAD) usually focuses identifying unex- pected events from the video and has been widely applied in various fields, including autonomous driving [45], public surveillance [6, 23], and crime scene analysis [33] etc. Previous VAD meth- ods [27, 33, 23, 44, 9, 13, 19, 34, 40, 48, 52] are designed in numerous pathways. Lu et al. [23] designed to learn video features only from normal videos, and hand-craft features or deep-learning- based features are leveraged. Sultani et al. [33] proposed multiple instance learning (MIL), which is the main paradigm for many seakkly-uupervied learning methods. Recently, Lv et al. [27] first proposed video-based large language models in the framework of VAD.
However, these methods lack sufficient semantic comprehension of scenes and offer inadequate user interaction. Several approaches [31, 42] have introduced multi-class classifiers to integrate semantic information with various types of anomaly information. Nevertheless, their output is still limited. In contrast, our framework not only integrates more comprehensive semantic information as a general video understanding system but also provides advanced interaction capabilities for users.
Large Model in Video Understanding Recent studies have demonstrated the reliable capabilities of large models in video understanding. Beyond powerful vision-language models [16, 51, 21, 24], recent research has increasingly explored more modalities [27, 18, 28, 46, 26]. Bain et [l.[3] introduced large-scale dataset with general video content descriptions. Several LLM-based works[18, 28, 46, 26] aim to comprehend visual content. Additionally, Vidco-LLaMa [49] ex- tends comprehension both auditory and visual information, while Su et al.[32] utilize modal encoders to understand across six modalities. Recently, Lv et al.[ proposed video-based large language models for VAD tasks in weakly supervised framework. In this paper, we introduce the motion modality in our proposed vision- languagl model, which enhances the model's 's ability to locate anomalies by prioritizing relevant video content.
3 Data Engineering
Previous datasets are inadequate for addressing our problem. Most existing VAD datasets, such as UBnormal [2] and DoTA [45], only contain simple video category labels and lack detailed language descriptions. This results in video understanding models lacking accurate and comprehensive supervision, creating a significant obstacle to identifying anomalies in videos. Recently, Lv et al. [27] attempted to create pseudo language descriptions for anomaly videos. However, these descriptions are naive combinations of labels and fixed text, relying on a rigid format that offers only limited information. Other datasets, like Wed) include only general descriptions of video content, which may not direct the model's focus on anomalies.
Our Principle To tackle the above problems, we annotate detailed language descriptions specifically for anomaly scenes in seven different existing < <VIDEO> datasets. These seven datasets include variety of anomalous scenarios such as crime (UCF-Cirme [33]), campus (ShanghaiTech [22] and CUHK Avenue [23]), pedestrian walkways (UCSD Ped1 [6] and Ped2 [37]), traffic (DoTA [45]), and human behavior (UBnormal [2]. With the support of these visual scenarios, we can perform comprchensive fine-tuning for various abnormal scenarios, being closer open- scenarios.
3

---

Video Clips
Dense Caption
Perception Tools
00:00-00:01: Caption
GPT
Manual Checking
Anomaly Video Description
00:01-00:02: Caption 2
4.0
00:02-00:33: Caption 3
InteraVideo TageTTot GRIT
......
00: 00:50-01:00: Caption - 4
Example of Deuse Caption:
GPT Prompt:
Example of Destription:
with lights driving anewy road at night in winter 00: 01-00 02: car and a jeep daving en the slippery conditions.
02:
video Dense Capton, Pleese help us to fully describe the details of this video. especially for the aromaly in the
deuse caption
Whule drving on sauwe
vehicle may have lost trutton duc to the slippery conditions and slid anto cousing dumage its front bumper si0 p0st Answering
Olt 58-00:33 our with red lights have lost traction in winber 0it 39-01: driving swoyy highway the windssheed.
the
Open Tpee-QQeee Questions
5W2H What Who, Where, W How, Why
GPT 4.0
Manual Checking
Question 1: What...?
GPT 4.0
nsswrr 1: is...
Question 2: Who... ?
Answer- 2: He/Shel...
Q Question 3: Where...?
A nsswer 3: at.......
Anomaly-Relatee Related Prompt
Question 100: How...?
Answer 100: To...
GPT Prompt:: Our visual language model elfectively detects anomalias videos,
Example of pen-Q uestion: What actions were taken to abdress the the video?
GPT Prompt:
Example of A Answering:
analyser hhs will give your anomaly video description and question about the cummnt video. please answer question based the given description.
explicilly mentioned what actions were takmn address tosus. Howerer, I con infer and sugnest common stops that are sypically taken following such incident....
usars potentall questions. Tharefore, gloase help to ensto: 100 user prompts representngg possible vidco content questions, please consicering two principles in generation: SW2H and Anomaly- Relatod.
Figure 2: Generation pipeline of our dataset. In the first line, we first segment videos into clips and generate dense captions for each segment, including a comprehensive description of the video content. Then, we use GPT-4 to guide the generation of corresponding anomalous video descriptions based on these descriptions, which are then manually checked reduce mistakes. In the second line, to generate user-centered QA pairs, we first use GPT-4 to generate open-ended questions based on the proposed two principles. Then, the questions and video descriptions are jointly input into GPT-4 to provide possible answers.
Moreover, to better account for real-world user situations, we believe that language descriptions should not only include descriptions of the video anomalies themselves, but also address open questions asked by users. Therefore, we construct open-ended question-answer
pairs for each scenario to further enhance model's practical ability to answer users' varying questions. The procedure for answering users' questions is shown in Fig. 2. The data format of can be described by the Eq. (1),
<VIDEO>: (DIs: <DESCRIPTION> QA: <QUESTIONS + <ANSWERING>).
(1)
Anomaly Video Description Generation To construct natural language descriptions
<DEsCRIPTION> for anomalous video datasets, we refer to previous research such as LLaVa [21] and VideoChat [18], and employ GPT-4 [1] as an assistant. We first split the video into dense clips to ensure key information is captured. Following VideoChat [18], we use perception tools (Intern Video [38]. Tag2Text [14], or GRiT [39]) to automatically generate captions for each key clip, obtaining a dense representation of the videos (except for the UCF-Crime dataset, which already has dense representation built in [47]). Next, use GPT-4 [1] generate moomlly-relatee descriptions based on the captions for each video. Unlike other general video understanding datasets [21, 18), we provide prompts for GPT-4 generate specific descriptions closely related to video anomalies. Finally, due to varying quality of dense captions, some videos may have incorrect annotations. Thus, manually recheck the final generated video anomaly descriptions ensure label accuracy.
HlmannCenttrc Qusstion-Answering Pairs Generation So far, we have obtained nearly accurate descriptions of anomaly videos. However, our framework may still face challenges with more open- ended questions from users. Therefore, anomaly-related question-answering is a significant practical requirement. Given the diversity of open- world scenes, users may ask questions involving various pronouns. Thus, mainly consider these two principles: ① Anomaly-related, be strongly related to the anomaly in the video. ② 5W2H, we introduce seven different question pronouns (What, Who, Where, When, How, How much and Why) to simulate various question formats that users may employ. This enables us address wide range of open questions related to
questions should

---

video anomalies. We input these two principles into GPT-4 [1] to generate open questions for anomaly videos. We then manually review and select the 100 most suitable questions. which are randomly assigned to each video. Finally, GPT-4 [1] will generate <ANSWERS> to these <QUEESTONSS>
Our data is more practical compared to previous ones: it not only understands multiple anomalies in videos but also supports question-answering in open scenarios (More details in Appendix D).
Methodology
To construct a practical framework for understanding video anomalies, our goal is to accurately
interpret these anomalies into natural language. However, most previous studies [18, 49, 29, 20, 27] focus on enhancing general video understanding capabilities while neglecting video anomalies. This oversight results in equal attention being given to all parts of the video, such the background and human appearances, often at the expense of key anomaly features, as shown in Fig. 1 (C). Consequently, these approaches are not effective in accurately focusing on anomaly-related features.
Overview of Solution The core of our solution is guiding visual instruction to focus on anomalies. Previous studies in video anomaly detection [44, 52] have demonstrated that motion-related feature help identify multiple anomalies. Therefore, Section 4.1, we first explicitly integrate motion modality into proposed framework to target moomaly-reatee features. Subsequently, in Sec- tion 4.2, we maintain mutual information consistency between the appearance and motion modalities within a tight feature space. implicitly guiding the appearance branch to reinforee motion attention. Finally, in Section 4.3, to improve the interpretation of motion-to-lannuugge we extract motion-related language descriptions directly match the motion and its corresponding motion-related language.
4.1 Explicit Motion Modality Integration
To enhance the capability of interpreting anomalies, we build a framework, HAWK, to explicitly integrate motion modality. HAWK has a dual-branee architecture, with fo the original video understanding network and fm for motion understanding. Inspired by Video-LLaMA [49], fo and fm share the same architecture but separate parameters in Fig. 3. Eq. (2) denotes our framework as,
P... Y = LLAMA (L. (fm. @ f((T)),
(2)
where Xy E RTxCXH* represents the < <VIDEO> input for extracting appearance feature, and T denotes the temporal dimension. Xm = M (X,). ). with (-) being the motion extractor.
fo(.) and fm()) are the frozen pre-trained video encoders from BLIP-2 [17], which consist of one EVA-CLIP [10] and one pre-trained Video Q-Former to output embeddings. Then, the output embed- dings from f.).) and f.) are passed through learnable projection networks for video and motion, P.(.) and (.) respectively. These networks aim to project visual (video and motion) embedding into the language feature space for interpreting. ft(-) is the frozen text token to embedding projection, that makes textual information can be inputted into LLaMA-2 [35]. is for combining our input prompt, we define our prompt as: "Here is the impul video embedding: different frames. please help me to <DE- SCRIBE VIDEO> <QUESTION>." " <DESCRIBE VIDEO> and <QUESTION> are the question classes for video description generation and video question answering respectively (Details see Appendix D). By combining the visual token embedding with the textual embedding, (T). LLaMA- [35], is employed to generate the final language response, Y. This framework explicitly integrates the motion modality during visual instruction tuning, significantly targeting anomaly-related features.
and motion embedding <MOTION EMBEDDING>
4.2 Implicitly Motion Attention Reinforcement
Although we integrate the motion modality to facilitate fine-tuning of HAWK, motion and video branches operate independently. Therefore, we cannot expect the original video branch to extract appearance features that focus on the region where the anomaly occurred (i.e.. motion). To help HAWK focus more on these regions, we observed the containment relationship in mutual information between motion and the original video. We use this relationship construct an auxiliary consistency loss function, implicitly reinforcing the motion attention (Fig. (2).
5

---

Video X
Encoder * fo
Video Projection
Video
Conceterry in Light
Embeedinng
Similarity
MLD Pryedee
Xm
CLIP
Papesss
LLaMA *
Video Nittee
Encoder 2% fm
Motion Projection
Mobon Embbediing
Vádeo anguage
cur
Ferter
VLP
NLP ProjesooaP
Motion Xm
Motion
Language
Tokenization
Text Projection
Text Embedding
ng. Coorld you provide &escriipian of the unomaly In this video?
bern Teletorm
Figure 3: Overview of Hawk. During training (Black and Gray path), we aim to optimize for video- language matching loss, along with Video-Motion Consistency and Motion-Languue Matching, During inference (only Gray path), we generate language descriptions using video, motion, and text.
Extract Motion Specifically, to obtain motion, we employ a motion describer M(.), which gener- ates motion between two successive frames shown in Eq. (3),
(()) Motion
M
(X x(--1),,
(3)
where (-) is the motion describer at the time step t, we currently use Gunnar Farneback's algorithm [11], and x x((-1) E RIxCxHxW denote the video frames at time steps t and 1.
(()
Motion E 2x includes two channels motion vector in X (horizontal) and Y (vertical) directions. We use the optical flow magnitude from these channels Mask, normalized to [0. 1] and multiplied with the original video appearance, to hide other non-motion regions, as Eq. (4),
(
NORM( (t) Motion ())² (x(+)
Motion (Y))) x (())
(4)
Mask
where x is the operator of pixel-wise multiplication. x X RXCCXHX donate the original video and our input motion information at time step t, respectively. We usually extract T frames as motion input Xm E R xCxxCxxx same as X
Build Cy Loss Then, we consider that Xm only con- tains key information for anomaly and it is contained in
X., and feature space from X. is more sparse. Therefore, we compact features from Xm and X, into tight space. At this space, aim to maintain the mutual information between Xm and X, consistency, and this way, the appearance feature can be focused on the motion region. Therefore, we construct an auxiliary loss to promote X, motion attention, in Eq. (5),
Video
Original Deseription
ancualy chactiis accieuuly oiile oil lncker moothy crang Der
Mation Fotmeter
Depondmny Pander
Search Nodien
Nerry
Yerhe Fettles
X
X
Motion
Deseription Motion
LMV = -- COS SIM x =
(5)
X X I
where Cf and X Cm (mm) de- note the tightly compressed representations of Xy and Xm. respectively, by the compression functions C, and Cm. C. and Cm share some initial shallow layer param- eters with P. and Pm, (as Fig. 3). Then, following a sub- sequent tight projection compresses both X, and X.m into a more compacted space.
Figure 4: Visualization of HAWK's loss. the original video-to-language loss. is the cosine similarity loss for mo- tion modality adaptation. motion-to- lo-lauge loss.
3 the
Finally, with this auxiliary loss, we can reinforce the mo- tion attention in the appearance feature, and HAWk's feature space will focus on more abnormal related features, which will promote the understanding of anomalies in the whole framework.

---

4.3 Interpreting Motion-to-Language
Although HAWK has already accommodated the motion modality in visual input, the corresponding motion from language is still unclear. This limitation hinders HAWK's interpretation in motion modality. Hence, to augment this relationship. we aim reinforce the correspondence between motion and their linguistic representation.
Extract Motion-related Language Previous studies [5, 36, 43, 15] have proved that the repre- sentation of motion in the language is predominantly from verbs and their corresponding entities. Therefore, extract linguistic representation, the first step to do dependency parsing for the original sentences, as Eq. (6),
(6) where D(-) is the dependency parsing and gt is the ground truth. Ggt represents the graph of the dependeney structure, which symbolizes the syntactic relationships among the words in sentence.
Ggt D(Y
Based on this graph, we can extract predicates (verbs) V, and also entities closely related these predicates, such as subjects S, objects O, indirect subjects Sj, and indirect objects O, These elements are then combined form short phrases representing motion, as in Eq. (7),
ygt (V,S, 0.) GG)
where Mi(-) is the language motion extraction operator, and Y is the motion-related language.
Build LM Loss After obtaining motion-related language,
establish strong supervision
between motion both vision and linguistic representation (as Fig. 4 3),, significantly enhancing the ability to interpret motion to language in HAWK. Consequently, we design motion-language matching as an auxiliary loss, as Eq. (8),
(ML m, Y = Σ
mi gt log YYm')
(8)
S.L. Y LLAMA (Pm (Im))))))
where C- is the cross-entropy loss, which contains N words.
Optimization Goal Finally, our total loss L shows as, L= to x Lv + +t x LMv t x LML where LV is original video language loss (as Fig. 4 4)) and to. and t2 is the haper-parameter.
5 Experiments
In this section, we will provide a comprehensive introduction to the experiments, including the processes of training and festing, the establishment of baselines, the methods of evaluations, and the detailed examination of ablation experiments pertaining to HAWK.
Training & Testing To enhance our framework's anomaly understanding capabilities, we' ve structured our training and testing process into three stages, as Fig. 5, Stage involves pre-training on the WebVid dataset [3] to acquire a general understanding of video content. In Stage 2, we finetune the model's 's focus towards video anomaly understanding by em- ploying specially curated dataset described in Section 1. con- sisting of over 8, 000 videos. We use 90% of these videos for training and allocate the remaining 10% for testing purposes. We jointly train on two tasks: video <DESCRIPTION> gener- ation and video SOOESSIION==<NNSEERRRR we evaluate these two tasks independently in the testing set to ensure our model's effectiveness.
Stage 1
WeiVid
2.5M
General Video U U'ddertanddin
Anomaly Video Description
A Asomnaly Video
Stage Video
Question-c nswering
Anomaly
D
4
Undesstanding
Stage
10%
10%
Testing
Figure 5: Training & Testing.
In Stage 3,
To evaluate the anomaly understanding performance of our proposed framework, we conduct comparisons with SOTA video understanding baselines. We select five baselines: Video-
Baselines
ChatGPT [29], VideoChat [18), Video-LLLMA [49], L.LaMA-dappter [50], and Video-ILLaV [20]. Our comparison aims to determine whether these baselines can fully understand and interpret video
7

---

Table 1:
Quantitative performance of (A) anomaly video description generation and (B) video
crestion-answering. Red indicates the best performance, while blue denotes the second best.
(A) Anomaly Vidco Description Generation
Teve-Level (1) [30]
GPT- Guided (*) [21]
Method
Backbones
BLEU-1
BLEU-2
BLEU-3
BLEU-4
Reasonability
Detail
Consistency
Video-ChatGPT [29]
ILaMA 7B
0.107
0.046
0.017
0.008
0.084
0.108
0.055
VideoChat [18]
Vicuna 7B
0.053
0.023
0.008
0.003
0.107
0.205
0.054
Video-LLaMA [49]
LLaMA-2 7B
0.062
0.025
0.009
0.004
0.120
0.217
0.066
LLaMA-AAapter [50]
LLaMA-2 7B
0.132
0.052
0.018
0.008
0.060
0.091
0.038
Video-LLLaV [20]
Vicuna 7B
0.071
0.030
0.012
0.005
0.077
0.115
0.038
Ours
LLaMA-2 7B
0,270
0.139
0.074
0.043
0.283
0.320
0.218
(B) Anomaly Video Question-Answerin
Text-Level (1) [30]
GPT- (*) [21]
Method
Backbones
BLEU-I
BLEU-2
BLEU-3
BLEU-4
Reasonability
Detail
Consistency
Video-ChaIGPT [29]
LJaMA 7B
0.177
0.096
0.058
0.038
0.508
0.430
0.421
VideoChat [18]
Vicuna 7B
0.261
0.133
0.074
0.043
0.699
0.631
0.598
Video-LLaMA [49]
L
0.156
0.081
0.045
0.586
0.485
0.497
I.LaMA-Adapter [50]
LLaMA-2
0.199
0.109
0.067
0.043
0.646
0.559
0.549
Video-LLaVA [20]
Vicuna 7B
0.094
0.054
0.034
0,023
0.393
0.274
0.316
Ours
LLaMA-2 7B
0.319
0.179
0.112
0.073
0.840
0.794
0.753
anomalies. To ensure the fairness of experiments, employed the baselines with the same size (7B parameters) as the backbone.
Evaluation Metrics To accurately evaluate model's performance in understanding video anoma- lies, we firstly adopt four Texe-Level metrics, from BLEU (Bilingual Evaluation Understudy) [30]-1 to BLEU-4 to measure word overlap between the model-generated text and the ground truth. This approach enables us to objectively assess the similarity and also take into account various levels of granularity the text-level, thus providing a clear indicator of how well the model understands and describes anomalies.
Besides, expand our evaluation framework by incorporating insights from recent research in LLaVa [21] or Video-ChatGPT [29], utilizing GPT-Guided [1] methods to assess the quality of the generated text. GPT [1] serves critical evaluator, generating scores for three key aspects of the language produced, with each aspect scored on scale from 0 to 1. These three aspects are as,
Reasonability: evaluates the logical reasoning and coherence of the generated language.
Detail: assesses the level of detail and specificity of the generated language.
Consistency: evaluates the coherence and consistency of the generated language.
By leveraging GPT [1] an evaluative tool, aim to provide a nuanced understanding of the text's quality, focusing on aspects that traditional metrics may overlook.
Quantitative Evaluation Table 1 (A) and (B) demonstrate the effectiveness of our model to describe abnormal phenomena. Our proposed model sigmificantly outperforms the previous baselines, achieving SOTA performance in every metric for both Text-level and GPT-guided metrics, thus can generate text that more closely aligns with actual scenarios.
Qualitative Evaluation Table 2 (A) and (B) demonstrate that our proposed framework achieves optimal qualitative performance in video description generation and ceesstion-nsweriig respectively. Compared with other baselines, HAWK can accurately understand and focus on video anomalies. For example, in Table 2 (A) Video-LLaMa [49], it pays more attention to the clothing information from the people (wearing blue and red jacket), while ignoring the motion-related anomaly (slipping). In Table 2 (B) -
it may produce hallucinations (Iwo people... who were hit by the car), which differ from the original video anomaly (car suddenly braking). In contrast, HAWK generates descriptions that are close to the real semanties (driver losing control).
8

---

Table 2: Qualitative performance (A) anomaly video description generation, and (B) question- answering. Red texts indicate key semantic inconsistencies, whereas Green texts signify that the generated results are closely aligned with the Ground Truth. [YELLOW) indicates the text problem.
(A) Anomaly Video Description Generation
Prompt
Please help fully describe the dotails of this video, especially for anomaly the video.
00:02
00:06
00:14
Video-ChatGPT [29]]
[NOT FOCUS ON ANOMALY] The most interesting part of the video is person walking the snow covered sidewalk, which is not common sight on a snowy street.
VideoChat [18|:
The video captures pedestrian crossing in snowy towa at night. The crossing features white and are some lights people are walking on the sidewalk, while others are crossnng the street.
crussing sign,
[NOT FOCUS ON ANOMALY]
Vidco-LLaMA [49]]
First, the video shows city with parked both sides. There also people walking the sidewalk. Next, the view changes to IFOCUS ON PEOPLE'S CLOTHING) man and woman stamding crosswalk, wearing
juckets respectively.
L.aaA-Adappter [80]:
There an anomaly the video: a person appears he walking on the street, hut they are not visible the blurred image. This person which is unusual as most people would be staying indoors during such weather conditions.
be walking midst of
[HALLUCINATIO]]
Video-LLaVA 120]:
The video shows
street while waman lying on the ground is not moving, and unclear what happened to her.
walking
lying the ground. LNCLEAR] The
Ours:
In this video, the man and woman crossing the street crosswalk might be impacted by the anomaly. The woman could slip on soowy sidewalk and be injured, which would anomaly vidco should mindful of their surroundings.
Ground Truth:
In this anomaly situation, probably due to a medical emergency, possibly suffering an injury. This could also lead to chain reaction, other podestrians trying the person and potentially slipping themselves.
people walking across the snowy crosswalk suddenly falls down
(B) Anomaly Question-Answering
Question
Why the vehicle the video moving ematically?
00:01
00:02
00:04
Video-ChatGIT [29]:
The other people affected by the anomaly in the video are the HAALLUCINATION two people in the and the is driving away.
who hit the They lying the
VideoChat
people or vehicles that is shown in the video the black car and the street and trees on either side of it.
|WRONG DESCRIPTION
video. only thing
Video-I.LaMA [49]:
Since the video not providing any additional context, it is possible determine if there other people affocted by the anomaly. It could be INO IDENTIFY CENTTIIIIGGGIII YING) a minor problem that only affects the drivees or pedestrians
car, or could be something more significant
LLaMA-Adapter [50]:
|WRONG] The other people affected by the amomaly are the driver of the car and the motorcyclist.
Video-LLaVA [20]:
HAALLUCINATION The driver of the car is seen driving with the hood open.
Ours:
The vehicle the video is moving erraticllly likely due the driver losing control, The speed and steering the vehicle may have been affected by the uneven road conditions, cansing the cur to swerve unpredictably and almost hitting the white The abrupt maneuvers might be indicative of the driver's attempts to regain control or avoid collisions with other vehicles or obstacles on the road.
Ground Truth:
Based the description you's provided, he consequence of the white car suddenly braking on an empty highway. This abrupt action could have led to the driver the vehicle behind reacting panicked ahrupt to avoid collision, resulting in erratic movements such swerving or harsh braking.
the erratic movement of the vehicle the video may
Ablation Study We conducted ablation experiments on three key structures proposed in this paper and analyzed their impact on the overall performance in Table 3 (A) and (B).
Effectiveness of Motion Information: We ablate all the motion components, including fms Pm and the motion input Xm for proving the effectiveness of introducing motion modality.

---

Table 3: Ablation study of (A) anomaly video description generation and (B) video question-
answering. Red indicates the best performance, while blue denotes the second best.
(A) Anomaly Vidco Description Generation
Tevel (1) [30]
GPT- Guidee (1) [21]
BLBU-1
BLEU-2
BLEU-3
BLEU-4
Reasonability
Detail
Consistency
w/o Motion Information
0.249
0.121
0.062
0.034
0.253
0.306
0.189
w/o Video- Motion Consistency
0.249
0.123
0.064
0,036
0.261
0.295
0.194
w/o Motion-Language Matching Loss
0.270
0.135
0.073
0.041
0.276
0.320
0.212
Ours
0.270
0.139
0.074
0.043
0.283
0.320
0.218
(B) Anomaly Video Question-Answertt
Text-Level (f) [30]
GPT-Guided (1) [21]
BLEU-1
BL.UU-2
BL.UU-3
BLEU-4
Reasonability
Detail
Consistency
w/o Motion Information
0.309
0.171
0.105
0.065
0.837
0.790
0.743
w/o Video-Motion Consistency
0.313
0.172
0.105
0,066
0.833
0.784
0.742
w/o Motion- Laaiuu- Matching Loss
0.316
0.176
0.109
0.069
0.836
0.788
0.752
Ours
0.319
0.179
0.112
0.073
0.840
0.794
0.753
Table 4: Qualitative evaluation of ablation study. Red/Green texts indicate language semantic inconsistencyyconsiiteee with the Ground Truth respectively. [YELLOW] indicates the text problem.
w/o Motion Information:
While driving down the [UNRELATED INFORMATION] snowy street, the vehicle suddenly skids a patch of loe causing it to veer into another cur that was [ALLUCINATION] parked beside a tree on the side of the road.
Video-Motion Consistency:
fails to stop time the snowy conditions. As result, skids and sideswipes the white car parked on the side of the road, causing minor damage to both vehioles.
viden,
[WRONG CONDITION] slippery road caused by
w/o Motion-Langunnn Matching:
As the white truck drives down the street, it suddenly skids on the wet road surface, losing control, and sideswiping parked white car. The impact results in visible damage both vehicles. [UNCLEAR CONDITIÓN] with smoke emilting from the truck's 's side and the car's 's mirrors shattering.
Ours:
While driving down scrapes against a parked car, causing minor damage to both vehicles.
street with parked both sides, the current vehicle's 's fromt right side
Ground Truth:
While driving down the street, the silver car suddenly swerves to avoid a parked car, but clips its rear humper, causing damage to vehicles.
When explicit motion information is lacking, the model's ability to describe the motions-
related anomaly diminishes, leading to inaccurate descriptions even hallucinations (Table 4 w/o Motion Information), then impedes the overall performance (Table 3).
Effectiveness of Video-Motion Consistency: The absence of rideo-motion consistency constraints reduces the generative model's 's ability to adapt to the motion modality, causing difficulties in accurately understanding motion scenes (Table w/o Video-Motion Consis- tency), then impedes the overall performance (Table 3)-
Effectiveness of Motion-Language Matching: Without motion-language matching loss, the correlation between motion and language becomes unclear. This ambiguity leads to the generation of language that includes unspecified motion information (Table 4 w/o Motion-Langurn Matching), subsequently degrading the overall performance (Table 3).
6 Conclusion
In conclusion, we have developed novel video-language framework for understanding video anomalies across various scenarios. By incorporating motion features and constructing rich linguistic descriptions, our model demonstrates SOTA performance in the open world. has the potential to benefit practical applications in diverse domains and paves the way for improving the model's 's interactivity with users, enabling more efficient and effective communication in addressing user- specific inquiries related to vidco anomalies.
10

---

Acknowledgments and Disclosure of Funding
This paper is supported by Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. NNo.2023BI1 1212010077) the Innovation and Technology Fund of HKSAR under grant number GHX/054/2IGD, the Natural Science Foundation of Zhejiang Province, China, under No. LD24F020002, and National Science Fund for Distinguished Young Scholars (62025205).
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 4, 5,
[2] Acsintoae, A., Florescu, A.. Georgescu, M.I., Mare, T., Sumedrea, P., Ionescu, R.T., Khan, F.S.. Shah, M.: Ubnormal: New benchmark for supervised open-set video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
20143-20153 (2022) 2, 3, 16, 26
[3] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: IEEE International Conference on Computer Vision (2021)
3, 7,
[4] Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE transactions on pattern analysis and machine intelligence 33(3), 500-513 (2010) 19
[5] Cadiot, P., Lebas, F., Visetti, Y.M.: The semantics of the motion verbs. Space Languages: Linguistic Systems and Cognitive Categories 66, 175 (2006) 7
[6] Chan, A.B., Vasconcelos, N.: Modeling, clustering, and segmenting video with mixtures of dynamic textures. IEEE transactions on pattern analysis and machine intelligence 30(5), 909-926 (2008) 2, 3, 16, 27
[7] Dosovitskiy, A., Fischer, P., Ilg. E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T.: Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2758-2787 (2015) 19
[8] Du, H., Zhang, S., Xie, B., Nan, G., Zhang. J., Xu, J., Liu, H., Leng. Liu, J., Fan, H., Huang. D., Feng, J., Chen, L., Zhang, C., Li, X., Zhang, H., Chen, J., Cui, Q., Tao, X.: Uncovering what, why and how: comprehensive benchmark for causation understanding of video anomaly. In: IEEE Conference Computer Vision and Pattern Recognition (CVPR) (2024) 21
[9] Dubey, S., Boragule, A., Jeon, M.: 3d resnet with ranking loss function for abnormal activity detection in videos. In: 2019 International Conference Control, Automation and Information Sciences (ICCAIS). pp. 1-6. IEEE (2019) 2,
[10] Fang, Y., Wang, W., Xie, B., Sun, Q.S., Wu, L.Y., Wang, X., Huang, T., Wang, X., Cao, Y.: Eva: Exploring the limits of masked visual representation learning scale. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 19359 (2022) 5
[11] Farneback, G.: Fast and accurate motion estimation using orientation tensors and parametric motion models. In: Proceedings 15th International Conference on Pattern Recognition. ICPR- 2000. vol. 1, pp. 135-139. IEEE (2000)
[12] Farnebäck, G.: Two-frame motion estimation based polynomial expansion. In: Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings 13, pp. 363-370. Springer (2003) 19
[13] He, C., Shao, J., Sun, J.: An anomaly-introduced learning method for abnormal event detection. Multimedia Tools and Applications 77, 39573-29588 (2018) 2,

---

[14] Huang, X., Zhang. Y., Ma, J., Tian, W., Feng. R., Zhang, Y., Li, Y., Guo, Y., Zhang, L.: Tag2text: Guiding vingunelanguage model via image tagging. arXiv preprint arXiv:2303.05657 (2023)
[15] Langacker, R.W.: Nouns and verbs. Language pp. 53 -4 (1987)
[16] Li, Li, D., Savarese. S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023) 3
[17] J., Li, D., Savarese, S., Hoi, S.C.H.: 2 Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning (2023) 5
[18] Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y: Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0635, (2023) 2, 3, 4, 7, 8, 22, 23, 24, 25, 26, 27
[19] Li, S., Liu, F., Jiao, L.: Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1395-1400 -903 (2022) 2, 3
[20] Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection, arXiv preprint arXiv:2311.10122 (2023) 5, 7, 22, 23, 24, 25, 26, 27
[21] Liu. H., Li, C.,, Wu, Q., Lee. YJ.:: Visual instruction tuning. Advances in neural information processing systems 36 (2024) 2, 4, 10
[22] Liu, W., W. Luo, D.L.,, Gao, S.: Future frame prediction for anomaly detection - new baseline. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 2, 3, 15,
[23] Lu, C., Shi, J., Jia, J.: Abnormal event detection 150 fps matlab. In: Proceedings of the IEEE international conference on computer vision. pp. 2720-2721 (2013) 1,2, 15, 23
[24] Lu, H., Niu, X., Wang. J., Wang, Y., Hu, Q. Tang, J. Zhang, Y., Yuan, K., Huang, B., Yu, 7.,, al.: Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshop (2024) 3
[25] Lu, H., Tang, J., Xu, X., Cao, X., Zhang. Y., Wang, G., Du, D., Chen, H., Chen, Y.: Scaling multi-camera 3d object detection through weak-to-strong eliciting. arXiv (2024)
[26] Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., Wei, Z.: Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07200 (2023)
[27] Lv, H., Sun, Q.: Video anomaly detection and explanation via large language models. arXiv preprint arXiv:2401. .05002 (2024) 3.5
[28] Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards detailed video un- derstanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)
3
[29] Muhammad Maaz, Hanoona Rasheed, S.K., Khan, F: Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306.05425 05424 (2023) 2, 7, 8, 9, 22, 23, 24, 25, 26, 27
[30] Papineni, K., Roukos, S., Ward, T.. Zhu, WJJ.: Bleu: method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting the Association for Computational Linguistics. pp. 311-318 (2002) 8, 10
[31] Pu, Y., Wu, X., Wang, S.: Learning prompt-enhanced context features for weakly-supervised video anomaly detection. arXiv preprint arXiv:2306. 14451 (2023) 2,
[32] Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., Cai, D.: Pandagpt: One model to instruction-follow them all. arXiv preprint 2iii:2305.1636 (2023) 3
12

---

[33] Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6479-6488
(2018) 1, 2, 3, 15, 24
[34] Tian, Y., Pang, G., Chen, Y., Singh, R., Verjans, J.W. Carneiro, G.: Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4975-4986 (2021) 2,
[35] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S.. Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922
(2023) 5
[36] Vo, N.P.A., Manotas, I., Sheinin, V., Popeseu, O.: Identifying motion entities natural language and a case study for named entity recognition. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 5250-5258 (2020)
[37] Wang. S., Miao, Z.: Anomaly detection in crowd scene. In: IEEE 10th International Conference on Signal Processing Proceedings, pp. 1223. IEEE (2010) 2, 3, 16
(38] Wang, Y., Li, K., Li, Y., He, Y., Huang. B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang. Z.,, et al.: Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.0319 (2022) 4
[39] Wu, J., Wang. J., Yang. Z., Gan, Z., Liu, Z., Yuan, J., Wang. L.: Grit: generative region-to-text transformer for object understanding, arXiv preprint urXiv:2212.0022 (2022) 4
[40] Wu, P., Liu, J.: Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Transactions on Image Processing 30, 3513-3527 (2021) 2,
[41] Wu, P., Liu, j., Shi, Y., Sun, Y., Shao, F., Wu, Z., Yang, Z.: Not only look, but also listen: Learning multimodal violence detection under weak supervision. In: European Conference Computer Vision (ECCV) (2020) 20
[42] Wu, P., Zhou, X., Pang, G., Sun, Y., Liu, J., Wang, P., Zhang, Y.: Open-vocabulary video anomaly detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024) 2,
[43] Wunderlich, D.: Cause and the structure of verbs. Linguistic inquiry pp. 27-68 (1997) 7
[44] Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553 (2015) 2, 3,5
[45] Yao, Y., Wang, X., Xu, M., Pu, Z., Wang, Y., Atkins, E., Crandall, D.J.: Dota: unsupervised detection of traffic anomaly in driving videos. IEEE transactions on pattern analysis and machine intelligence 45(1), 444-459 (2022) 1, 2, 3, 16, 22
[46] Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al.: owl: Modularization empowers large language models with multimodality. arXiv preprint MrXiv:2304.14177 (2023) 3
[47] Yuan, T., Zhang, X., Liu, K., Liu, B., Chen, C., Jin, J., Jiao, Z. Towards surveillance video-and- language understanding: New dataset, baselines, and challenges (2023) 4
[48] Zaheer, M.Z., Mahmood, A., Astrid, M., Lee, S.I.: Claws: Clustering assisted weakly supervised learning with normaley suppression for anomalous event detection. In: Computer Vision-ECCO 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16. pp. 358 336.376.55 Springer (2020) 2,
[49] Zhang. H., Li. X., Bing, L.: caamo- An instruction-tuner audio visuall language model for video understanding. arXiv preprint arXiv:2306.0285. (2023) 3, 5, 7, 8, 9, 22, 23, 24, 25, 26, 27
[50] Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Qiao, Y.: Llama-adapter: Efficient fine-tuning language models with zero-init attention. arXiv preprint 1Xi1:23033.16199 (2023) 7, 8, 9, 22, 23, 24, 25, 26, 27
13

---

[51] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In: The Twelfth International Conference on Learning Representations (2024) 3
[52] Zhu, Y., Newsam, S.: Motion-aware feature for improved video anomaly detection. arXiv preprint 1rXiv:1907.10211 (2019) 2,
14

---

A Summary of Appendix
This appendix provides supplementary information that was not included in the main paper. Firstly, we address the security statement of our study, ensuring the confidentiality and integrity of the data used. Additionally, we provide detailed explanations of the training and testing resources utilized, including information the hardware and software configurations. We also present statistics and distribution of the training data, along with the costs associated with human resources involved in the study. Furthermore, we describe the evaluation metrics employed to assess the performance of our method. Moreover, we present additional qualitative results comparisons, showcasing the effectiveness of our approach. Additionally, we provide an open- world demo, demonstrating the real-world applicability of our method. Finally, we discuss the existing limitations of our paper and propose potential avenues for future research.
B Security Statement
To prevent any potential misuse and ensure responsible use, we have strictly limited the application scope of our proposed method, HAWK. Unless authorized, HAWK is only permitted for use in research domains.
Additionally, access to the proposed dataset is restricted to qualified institutions and organizations, who must provide a clear purpose for its use. We explicitly prohibit the application of the dataset in situations that may cause potential danger or have a significant social impact.
These measures are in place to ensure the ethical and responsible use of our research.
C Details in Training and Testing
Computational Resource During the pre-training phase, we utilized four Nvidia RTX A6000 GPUs' train on the WebVid dataset [3] for approximately 120 hours. In the fine-tuning phase, we employed two Nvidia RTX A6000 GPUs to fine-tune on our proposed dataset for about 80 hours.
Efffccieccy During testing, the average model response time for each round of conversation with HAWK is approximately 2ms. Additionally, considering the available graphics memory, the model can handle video clips of up to 32 frames. Therefore, it is necessary to extract different frames from longer videos.
Hyper-parameters In the loss function, to is set to for our main task, video-to-languaat and t and t2 to 0.1, as auxiliary tasks for balancing different loss values.
D Details in Dataset
Dataset Introduction and Statistics Our study utilizes video anomaly datasets, each encom- passing different scenes. The detailed statistics and introduction of these datasets are as follows:
UCF-Cirme [33]: The UCF-Crime dataset comprises an extensive collection of 128 hours of video. It consists of 1,900 long and untrimmed real-world surveillance videos, featuring 13 distinct classes of realistic anomalies. These anomalies are carefully chosen due to their notable implications for public safety.
ShanghaiTech [22]: The ShanghaiTech Campus dataset comprises 13 scenes characterized by complex light conditions and varied camera angles. It encompasses 130 instances of abnormal events and encompasses over 270,000 training frames. Notably, this dataset includes annotations for both frame-level and pixel-level ground truth of abnormal events, providing comprehensive insight into anomaly detection and localization tasks.
CUHK Avenue [23]: The CUHK Avenue Dataset comprises 16 training and 21 testing video clips designed for abnormal event detection. Captured within the CUHK campus avenue,
"https://www nvidia. com/ aam/en--ee http:///ww/...ggaa//aaaaaaaaaaaa
15

---

these videos encompass total of 30,652 frames, divided into 15,328 frames for training and 15,324 frames for testing. The training videos capture normal situations, while the testing videos include both normal and abnormal events.
UCSD Dataset [6, 37]: The UCSD Anomaly Detection Dataset was captured using a stationary camera positioned walkways. The crowd density within these walkways exhibits variability, spanning from sparsely populated areas to densely crowded environments. It is split into 2 subsets, each corresponding to a different scene. Pedl [6] includes a total 34 training video samples and 36 testing video samples, while Ped2 [37] consists of 16 training video samples and 12 testing video samples.
elevation, providing an overhead view of pedestrian
DoTA [45]: The Detection of Traffic Anomaly (DOTA) Dataset introduces the When- When-Where- What pipeline with temporal, spatial, and categorical annotations. It contains 4677 videos, all with resolution of 1280 720 pixels. Notably, the original videos were extracted at frame rate of 10 fps in this dataset.
UBnormal [2]: The UBnormal dataset supervised open-set benchmark designed explic- itly for video anomaly detection, comprising diverse virtual scenes. It introduces abnormal events annotated at the pixel level during training, which enables the utilization of fully- supervised learning techniques for abnormal event detection.
In our study, we extend upon these existing datasets by implementing our data engineering pipeline. This pipeline generates comprehensive descriptions of video anomalies and formulates open questions derived from these anomalies.
Data Distribution To demonstrate the applicability of our data in an open-world scenario, we conducted statistical analysis of the data distribution. Figure illustrates the data distribution of all the datasets we utilized, indicating that our method can effectively support various open- open-world datasets. Besides, we acknowledge the need to expand our dataset further enhance the model's applicability in this task.
Vialin Plot of PCA Results
20000
10000
10000
-20000
Do'A
UCSD Ped1
UCSD Pedz Podghhanghattoch UBNormal
Crime CUNK Avenue
Figure 6: Violin plot of data distribution. We use PCA dimensional reduction measure the feature distribution of different datasets, where there are significant differences in the feature distribution.
Manual Checking Before conducting the experiments, we manually checked the textual descrip- tions generated for the videos. Specifically, consider the following aspects:
16

---

1. Error Correction: We removed text descriptions that contained obvious errors about the video content and supplemented the correct object, behavior, and scene information. (For instance. GPT tends to misidentify dogs in videos, describe running pedestrians as skateboards and motorcycles, and mistake scenes containing water as rainy days.)
2. Detail Enhancement: We provided more detailed textual descriptions of anomalies in the video (such as pedestrians lingering or jumping in the middle of the road).
3. Human Resource Cost:
We formed a team of five annotators to conduct Manual Checking all the videos. Since most of the videos already had automatically generated annota- tions, each annotator invested approximately 30 hours of work during the labeling process, processing about 1700 videos.
Table 5 below provides example of before and after manual checking.
Table 5: An Example of Manual Checking.
Before Manual Checking (GPT-4)
After Manual Checking
The video depicts different scenes of people walking and sitting in front of building and a train station. There are also shots of woman with a red bag sitting on the ground, a man walking by with a shopping bag, and a young boy in a red jacket standing outside a building. There are also several shots of people standing at bus stop, and sign with glowing red hand.
The video shows scenes in front of the subway entrance, with students and pedestrians coming and going or entering and exiting the subway, In addition, two school security guards patrol through, and at the subway entrance, two stu- dents are chatting and discussing.
In this example, the caption generated by GPT-4 included hallucinations (such "a man walking by with shopping bag" or "a young boy in red jacket") which were corrected after Manual Checking. Additionally, GPT-4's description was inaccurate (for instance, "bus stop" should have been more accurately described as "subway entrance").
<DESCRIBE, VIDEO> and Generated Open-World World <QUESTION> We set 20 problems for <DESCRIBE VIDEO>, and during each iteration in training, we randomly select one of them.
you describe anonsly
video?
2. How sould you detail the anonaly found in the video? video?
anomaly can you identify
Could you explein the anonaly observed in the video?
Can you point out the anomaly in the video?
What's the anonaly depicted in the video? could you specify the anonaly present in the video? you perceive Can you highlight the anonaly xithin the video?
7.
ancualy in video?
9.
10.
What anonaly
11.
the video? Can you detail the specific ananaly encountered in the video? sould you describa the particular anonaly the video? What detaila can you provide about the anonaly in the video? you elucidate the ancaaly
Could you characterize the anemaly
12.
13.
14.
15.
video? Can you illustrate the nature of the anemaly in the video? What
16.
17.
the anonaly could you outline the anomaly observed the video? 19. How doen the anomaly in the video manifest? anonaly
video can you describa?
18.
you clarify the aspecta
videa?
We have also generated 100 <QUESTIONS> for open-world world anomalies. To mimic user behavior, some these questions are closely related to the video scene, while others are less closely related. However, all of these questions are potential inquiries in an open-world scenario.
Ubo causing the disturbance in the video? the
activity bappening 3. Uben did the anomaly occur in the video?
video?
is strango event taking place 5. why the object in the video behaving abnormally?
video?
How the anonsly in the video affecting the surroundings?
How much danage caused by the incident in the video?
Ubo is the main person involved in the unusual event?
the
sudden change
video?
17

---

10.
When does the suspicious activity Whare can I find nore information about the incident in the vidao?
the video?
11.
12.
Why the people the video reacting in say?
13.
How can I identify the source of the prohlen in the vidao?
14.
much time does the abnormal evant Nho are the other people affected the anonaly the video?
videc?
15.
16.
that actions vere taken to address the issue in the video?
17.
When the rideo recorded, and
recent event?
18.
Whare alse can I find similar incidants in other videos? Why the vebicle the video noving erratically?
19.
20.
How can I prevent such anomalies from occurring in the future?
21.
How much impact does the abnormal event have the overall situation?
22.
Who should contact if I notice similar snonaly in another video? what steps can I take to investigate the issue further?
23.
24.
When is the best time to report Where can I find resourcea to help ma underatand the anonaly batter?
video?
25.
Why did the equipaent
video malfunction?
27.
How can I differentiata batueen normal and abnornal bahavior in a video?
28.
How much does it cost to implement a systen that detects anonalies in videos? Who provide expert edvice handling video anonalies?
29.
30.
What is the nost common type of anomaly found in videos?
31.
Whan should Where can I find a 1ist of known video anonalies and their descriptions7
concerned about anomaly
video?
32
important How can I improve ability spot anomallea in videos?
Nhy
detect
analyze anonalies in videos?
34.
35.
How much training is required to become proficient in detecting video anomalies?
36
Who I collaberate vith to better understand video anonalies?
37.
What are the potential consequences of ignoring an anonaly in video?
When did the trand analyzing anomalies in videoa begin?
39.
Wheze can I find exemples of eucosesfully resolved video anomaly caees?
40.
Why sone anomalies in videos go unnoticed?
41.
How can I report video anomaly to the appropriate authorities?
42.
How much time is needed to thoroughly analyze a video anomaly?
43.
Who respensible for nonitoring and addressing video ancmalies?
44.
What are the best toola to use for dotecting anomalies in videos?
When
necessary to escalate video anonaly for further investigation?
46.
Where can I find guidelines on hov to handle video anotalies?
47.
Why sone video anonalies How can I ensure the accuracy of video anomaly detection system?
serious consequences?
48.
49.
How much effort is needed to maintain a video anomaly detection system?
50.
Who should be informed when videa anonaly detected?
61.
what are the signa that indicate a potantial anonaly in a video?
When should perform follow-up analysis Where can I find support for deal with video anomalies? crucial How can I improve the officiency of ny video anomaly datection proceas?
detected video anomaly?
53.
Why
quickly
video anomaly detected?
65.
56.
How much data is needed to accurstely detect ancnalies in videos?
87
Who help fine-tune ny video anonaly datection system?
58.
what are the key factors to conaider when analyzing video anonalies?
When should update ny Where can I find the latest research on video anomaly detection techniques?
anonsly detection eysteu?
60.
61.
Why necessary to
anonaly detection system place?
62.
How can I nininize false alarma in video anonaly datection system?
63.
Hox much does it cost to maintain a video anomaly dotection system?
64.
encounter difficultiea with ny What are the best practices for dealing with video anoualieg?
consult
anonaly detection systen?
65.
When appropriste involve lav enforcement Where can I find commuitt of profossionals who specialize in video anomaly dotection?
video anously case?
67.
68.
Why sone videa anomalies require imadiate attantion?
69.
How can I enhence the perfornance of ny video anonaly detection systen?
70.
How much should invent in a video anomaly detection systan?
71-
provide training bow detect and analyze video anonaliea?
72.
What are the noet effective nethode for detecting ancualies in videos?
73.
When should seek external help for video anemaly case?
74.
Where can I find comprehensive database of video anoralies?
75.
continuesaly nonitor videoa Bow can I validate the results of video anonaly detection system?
Why it inportant
anonallea?
76.
77.
How much influence da external factors have on video anomalies? for aseistance What are the main challenges in detecting and anslyzing video anomalies?
complex video anonaly case?
79.
80.
When necessary to involve
stakcholders in
anomaly case?
81.
Where can I find case studies on suscesaful video anomaly datection projecta?
82.
Why is it essential to have aystematic approach video anonaly detection? video anonaly detection øyeten for different scenarios?
83.
Hew I optinize
84.
How much storage is needed to archive video anomalies for future analysis? undetected video snomalies?
should be beld sccountable
86.
What are the nost ceunon reasens for video anomalies to occur?
87.
When should reavaluate ny video anonaly detection systen?
88.
Where can I find information the latest video anonaly dataction technologies?
89.
Why is it beneficial to collaborate with others in the field of video anoualy detection? Hew I ensure the confidentiality of video anomaly cases?
90.
18

---

91.
Hew much should rely automated systens for video anonaly detection?
92
Who can contact for technical mupport with ny video anonaly datection system?
What
ethical considerations when dealing When should I notify the public about video anomaly case?
video anonaliea?
94.
95.
sources information video anomalies?
96.
it inportant to have backup plan for desling with video anonalies?
97.
How can I customize my video anonaly detection system for specific use cases?
98
Hew much should allocate for analyzing video anomalies?
99
Who can I turn to for guidance on handling sensitive video anonaly cases? critical
consider choosing video anonaly detection systen?
Demonstrating that Our model outperforms GPT-4-based Data Generation. To demonstrate that our model has better detection capabilities than GPT-4, compared our model's 's results with the unchecked labels generated by GPT-4 on the same testset, as shown in the following Table
Table 6: Comparison with GPT-based Data Engineering Pipeline.
Methods
Backbones
LLM Size
BLEU--
BLEU-2
BLEU-3
BLEU-
Reasonability
Detail
Consistency
Data Engineering Pipeline Ours
GPT-4 [1]
0.188
0,098
0.056
0.034
0,189
0.313
0.158
LLaMA 2
7B
0.270
0.139
0.074
0.043
0,283
0.320
0.218
Clearly, the results indicate that our model can better assist in understanding video anomalies compared to GPT-4, achieving late-of-theeart (SOTA) performance in both text-level and GPT. guided evaluations.
E Efficiency of Gunnar Farneback's Algorithm for Motion Modality Extraction
Firstly, Gunnar Farneback's s algorithm demonstrates remarkable efficiency in generating video optical flows-even even on CPU platforms. For each frame, the efficiency of this algorithm surpasses that of other widely deployed methods, as illustrated in the table below:
Table 7: Performance Comparison of Different Optical Flow Methods.
Methods
Gunnar Farneback [12]
LDOF [4]
FlownetS [7] FlownetC [7]
65 (CPU)
Seconds Per Frame
0.02 (CPU)
0.08 (GPU)
0.15 (GPU)
2.5 (GPU)
Secondly, for processing one video, the motion of multiple rounds of dialogue necessitates just single iteration of video motion extraction. Consequently, the response time for processing one video (about 0.72 seconds) is significantly shorter than the time required to generate single round of dialogue (about 1.5 seconds). This cost deemed acceptable for practical anomaly detection system for users.
Certainly, concur that future research into efficient methods for optical flow extraction, including end-to-end optical flow extraction strategies, will likely further augment the efficiency of system.
F Details in GPT-Guided Metrics
In the GPT-Guided metrics, employ GPT-4 auxiliary tool to evaluate the generated re- sponse HAWk. Our evaluation focuses on three primary dimensions: Reasonability, Detail, and Consistency.
We first set the system prompt as follows: Initially, we establish the system prompt as shown below:
("role": "system", "content":
"You are an intelligent chatbot designed for evaluating the generative
outputs for video-based pairs. you will be given two answers, one reference ground truth and one our generated, but this does not mean that the reference GT is the only answer. Your task is to give the score of the predicted answers.")
19

---

Our system prompt is designed to compare the degree of matching between image pairs. However, this does imply fine-grained matching at the text level. Instead, emphasizes the semantic information-related aspects.
To assess particular dimension of the metric, employ the following prompt:
("role": "user", "content": "### Video Description Generation
Please evaluate the following video-based video description pair: Reference:
<DESCRIPTION Ours: <DESCRIPIOON_Dur>>>
### Video Question-Anewering
Please evaluate the following video-based video question-answer pair:
Question: <QUESTION>
Reference: <ANSWER_GT>
Ours: <ANSWER_Ourss
Provide your evaluation only as CaaasonabilityDetaaal Consistency> score where the <Reasonability Dettabilyyyyetttlllllllllaaallll | Consistency> score is a FLOAT value between 0 and 1, with 1 indicating the highest level of iiasonabilityy/ettailll [ettyy//etaal//osssseeneee form of Python dictionary string with key 'score', where its value is the <Reasonability /
Please generate the response in the
score in FLOAT, not STRING. DO NOT PROVIDE ANY OTHER QUTPUT TEXT OR EXPLANATION Only provide the Python dictionary string. For example, your response should look like this: {'score': 0. 0.675)."
We have developed distinct prompts for two tasks: Video Description Generation and Video Question- Answering. The primary difference is the addition of the <QUESTION> field in Video Question- Answering, which indicates what kind question the model should answer. <DESCRIPTION _ and <DESCRIPTION Ours> represent the Ground Truth and generated video description, respectively. Similarly, <ANSWER_ DEESER__TTTTE and <ANSWER_OurSS signify the Ground Truth and our generated video answers, respectively. < Reasonability Detail Consistency represents the three dimensions we aim to evaluate. Lastly, besides the essential reminders, we have constrained GPT's output format to sscoree: 0.675).
G More Results
Table (A), (B), (C), (D), (E), and (F) below present additional qualitative results from different datasets. the tables, red texts indicate key semantic inconsistencies with the Ground Truth, while green texts signify that the generated results closely align with the Ground Truth.
H Open-World Video Anomaly Understanding Demo
We present a demo showcasing the use of HAWK in an open-world scenario, using XD-Violence [41] (which is not included in our dataset). The practical capability of the system in unknown scenario the open world is depicted in Fig.7 and Fig.8. Furthermore, HAWK can provide accurate answers users' questions and engage in long dialogues in the open world.
I Limitations
Hallucination Although most of the hallucinations can be decreased through motion, some error motion may still also cause hallucinations. Future work may need to consider connection between the hallucination and the abnormal region more precisely.
20

---

Background Information Leveraging background information can provide a robust prior for understanding video anomalies. We will try integrate information related scenes, backgrounds, and objects in a large-model-basee video anomaly understanding model.
Video-level v.s. Streaming Data The goal of this paper video-level video anomaly understanding. However, for a video anomaly detection system, anomaly detection in streaming is essential, so to increase the practical application ability, we need to design more practical system for streaming data.
Data Limitations While our dataset includes multiple anomaly scenarios and our framework is designed for an open-world setting, the limitations of our data make it difficult to fully support open-world scenarios. This significant drawback of our study. To address this limitation, we recommend building larger and more diverse open datasets.
Keyframes First, our current methodology for dataset construction involves sampling the video at consistent one-second intervals. This technique is strategically chosen to ensure that all possible anomalies within the videos are comprehensively captured (Some anomalies only happened in 1-2 seconds), thereby significantly reducing the likelihood of missing critical accidents. While realize that this may lead to degree of caption redundancy, we still prioritize the facilitation of thorough annotation to ensure that all anomalies are detected.
In addition, we have leveraged the capabilittess of GPT-4 for generating captions, especially for anomalous events. Due to GPT-4's advanced text generation and summarization abilities, it serves as an effective tool in minimizing redundancy, ensuring that the extracted captions are both high-quality and succinet.
After the initial processing with GPT-4, we also undertake manual checking process. This step is crucial for further reducing any residual redundancy and correcting possible errors within the captions, thereby ensuring the quality and accuracy of our dataset.
Certainly, we agree that the utility of keyframes is an effective strategy, especially for much longer videos, and believe its potential to significantly enhance data annotation efficiency. This will be the future work.
J Future Work
In this section, we discuss potential avenues for future research to build upon this paper.
Expanding Applicability One possible direction for future work is to expand the applicability of the model to wider range of scenarios (like most recent research by Du al. [8]). While our current dataset includes multiple anomaly scenarios, there are still many diverse scenarios the model may handle. We could explore ways incorporate additional data sources or create new datasets that cover a broader range of scenarios. Additionally, we could investigate ways to deploy the model in cloud, edge, and hybrid computing environments support more diverse deployment scenarios.
Enriching Task Capabilities While our current model is able to detect anomalies in videos, it does not provide location information where the anomaly occurs. This information could be valuable for certain applications, such as security monitoring or surveillance. Therefore, a valuable direction for future work would be investigate ways to improve the model's ability localize anomalies. By improving the model's ability to locate the scene of an anomaly, we could further enhance its usability and expand its range of applications.
21

---

(A) Anomaly Video Deseription Generation in DoTA [45].
Prompt Please help fully descrbbe the details of this video, especially for the unomally in the video.
00:00
00:01
00:02
Video-ChatGPTT [29]:
The video shows car driving down a busy street, with a person sitting in the passenger seat. The car is surrounded by other cars, and the driver is looking the camera. The sitting the passenger a person walking in street, and truck driving by.
turning corner, and the person looking the camera. The vidco also shows
VideoChat [18]:
video a car driving on road with camera mounted inside car. It is a clear day with some clouds in the sky. There are cars and other vehicles on the road, with one vehicle parked on the side of the road. The road is lined with trees both sides. The car is driving along the road, passing other vehicles, and crossing roads. The anomaly in vidco occurs when the car suddenly stops front of bus, blocking the road.
Vidco-LLaMA [49]:
car driving down street, and there are buildings and cars parked along the side of the road. Finally, the car is seen driving down the street at night. The camera is shown outside the car, and there are white lights on the road. There trees and buildings in the distance, and the cars the road are all different colors. The video ends with the turning and driving down the street night.
LLaMA-Adapter [50]:
video shows a car driving down road, with a person inside vehicle. Suddenly, the car appears to be hit by large object, causing it to flip over and crash. The person inside the car is thrown out, and the car is left in wreeked state the road. The scene is captured from dash cam perspective, providing clear view of the incident.
Video-LLaVA [20]:
The video starts with driving down street, and the camera captures the view from inside car. The car is seen driving past blue truck and red car. The camera then pans to the side of the road, where man seen walking down the sidewalk. The camera then pans back to the street, where blue truck is seen driving down the road.
Ours:
While driving down the street, the current vehicle is involved in an accident when sudden stop minor damage to both vehicles a startled reaction from the occupants.
rear-end the white car front of resulting
Ground Truth:
As the continues down the busy tree-lined street, the traffic light ahead suddenly turns red. The white car in front the vehicle abruptly comes to a stop, causing the driver recording the vidco to brake hard but ultimately colliding into the rear of the white car with loud impact.
22

---

(B) Anomaly Video Description Generation in CUHK Avenue [23].
Prompt Please help me fully descrbbe the details of this video, especially for the unomally in the video.
00:00
00:02
00:04
Video-ChatGPT [29]:
The video shows a man sitting on bench a busy city street, surrounded by many people. Suddenly. a dog appears and starts walking towards the man. The dog is wearing red collar and is accompanied by person. The dog walking towards the man, who sitting on the bench, and the man watching the dog as approaches. The dog is walking towards man, and it appears to be a unique and unexpected event in the busy city street.
VideoChat [18]:
The video shows a group of students walking outside the entrance to an office building located in park. There are two white benches in front of the entrance, a large blue sign above it that reads Science and Engineering. People are standing and walking around the entrance while the students are walking towards the building.
the left side and
the right side. The entrance has
Video-LLaMA [49]:
First, the video shows a busy street with many people walking around. There is sign for bus stop one corner. Next, the video focuses line of people standing front of blue building. with man black shirt and black jacket walking towards them. Finally, the video shows crowd of people at bus stop, with people walking to and from bus. There a green park bench and a building visible behind the people. The street is also filled with trees and a black car is visible in the background.
LLaMA-Adapter [50]:
The video shows group of people walking down hallway. with some of them carrying backpacks. There at Icast 12 people in the scene, with some walking pairs or small groups. The hallway long has few benches along the way. One the people the group appears to be wearing a tie, which stands out as an anomaly in the otherwise casual attire of the other individuals.
Video-LLaVA [20]:
In the video, man is seen standing in hallway while talking on his cell phone. He is wearing red shirt and is standing in front of group of people who walking by. The man's red shirt stands out against the crowd, making him easily noticeable. The man's 's hehavior quite unusual, as he is not paying attention to his surroundings while talking on the phone. is unclear what the man is doing or why he is not paying attention to the people around him.
Ours:
The vidco is set subway entrance near campus, with pedestrians entering and exiting the area from different directions. crowd of students, teachers, and tourists is standing the subway entrance, watching a performance by a street musician. They are throwing coins and applauding. Other students and teachers are walking and talking near the subway entrance.
Ground Truth:
The vidco is set at subway entrance where some pedestrians are hurriedly entering and exiting the subway passing through nearby pathways. An- other group of people captivated by performance and stands the subway entrance watch.
23

---

(C) Anomaly Video Deseription Generation in UCF-Crime [33]
Prompt Please help fully descrbbe the details of this video, especially for the unomally in the video.
01:06
01:15
01:36
Video-ChatGPI [29]:
The video shows group of people riding motorcycles and a car in city street. The video also shows man walking in the street. and man riding motorcycle. The video is taken at night, and the street is dimly lit. The anomaly in the video person riding motorcyele who is wearing helmet. This is safety concern as it is illegal can lead to severe injuries in case of an accident.
VideoChat [18]:
This video appears to be a surveillance video of a residential street at night in India. The video shows street with some motorcycles parked along the side of the road and few people walking around. However, the two anomalies of man riding the motorcycle and the man walking down the street make it stand out.
Vidco-LLaMA [49]:
First, the video shows a street an Indian city at night. There are few buildings on the street, and people are walking around. Next, there is a close-up of street at night with cars parked on it. The street is narrow, and there many buildings both sides. Then, the video shows street night with motorcycle parked it. There also bicycle parked the sidewalk and a white car parked next to bike.
LLaMA-Adapter [50]:
video shows man walking down a street night, nearby. Suddenly, the scene shifts to a different location, where a woman is walking down the street. The man and the woman appear to be walking in opposite dircctions, and the man is carrying backpack. The vidco ends with the man walking down the street, still carrying the backpack.
car parked
Video-LLaVA [20]:
The video shows person riding motorcycle down narrow alley night. person is wearing white shirt black pants. The person rides motorcycle quickly, and the camera follows them until they reach the end of the alley. However, there is an anomaly in the video. The person on the motoreycle not visible in the video, and the camera only shows the motorcycle and the surrounding environment.
Ours:
The video shows what appears he hit-and-run incident attempted theft. man blue pants and shorts is shown walking along the side of the road, where he is subsequently struck by a motorcycle traveling from behind. The man, possibly injured, then falls to the ground, which could be indicative of either the collision itself or previous event that has left him unconscious but necessarily injured.
Ground Truth:
The video portrays routine urban activity that people and vehicles moving along a street. The abnormal part unfolds when motorcycle stops next to a pedestrian, leading to a confrontation. The altercation becomes physical when the person who dismounted from the motorcycle starts arguing with the passerby, eventually escalating the passerby is pushed to the ground. Following the altercation, the individuals involved in the aggressive hastily escape the scene on motorcycle, which adds a hit-and-run element to the incident, indicating a consciousness of guilt or an attempt to evade law enforcement.
24

---

(D) Anomaly Video Question-Answering in ShanghaiTess Tech [22]
Question Who people aftected the anomaly video?
00:00
0&CP2
00:01
Video-ChatGPT [29]:
The other people affected by the anomaly in the video are the people walking on the sidewalk. They are walking in line, and suddenly, person starts running in the opposite direction. The person running is wearing red shirt.
VideoChat [18]:
It is possible that there may be other people in the video who cross the yellow line, although they not visible the frame that point in the video. The pedestrian who crosses traffic line appears to walking away from the camera and appears to be aware of the traffic line, which suggests that they are not necessarily unaware of the rules of the road.
Video-LLaMA [49]:
It only talks about a group of people walking down the street. It does not mention any specific people being affected by the anomaly.
LLaMA-Adapter [50]:
The other people affected by the anomaly in the video are the people walking the sidewalk, who unaware of the man's 's disappearance.
Video-LLaVA [20]:
The video shows that the other people are also affected by the anomaly. running and jumping over the obstacle in the video.
They
Ours:
The anomaly in the video that I described involves pedestrian walkway while other people are nearby. which could potentially cause a collision or accident. Additionally, other pedestrians or passersby in the vicinity may also be affected by the situation, potentially being startled distracted by the incident. Additionally, other pedestrians passersby in the vicinity may also affected by the situation, potentially being startled distracted by the incident.
walking
Ground Truth:
The other people affected by the anomaly in the video would include: The pedestrian who was collided with by the running man, as they might sustain injuries from the impact. 2. Bystanders or other individuals in the park who may witness the incident and could potentially be involved in the ensuing chaos need provide assistance. 3. Any friends family members individuals directly involved present or affected indirectly by the consequences of the incident.
collision who might be
25

---

(E) Anomaly Video Question-Answeringg in UBNormal [2].
Question How much abnormal event have on the overall situation?
00:11
01122
00:01
Video-ChatGPT [29]:
The abnormal event of a person falling on the ground the middle of the train station has a significant impact on the overall situation. It disrupts the flow of people walking around the station, they forced stop and look the person lying the ground. This creates and concern among the people, as they may not know what to do how to help the person.
of confusion
VideoChat [18]:
Based on the information provided. it's unclear if there is any abnormal event occurring in the video. cannot assess the overall situation without further context information.
Vidco-LLaMA [49]:
The abnormal event of a person lying on the ground could disrupt the normal functioning of the metro station. It may delays disruptions in the train service, potentially affecting passengers' plans travel experiences. Additionally, first responders may have to intervene to ensure the person's 's safety.
LLaMA-Adapter [50]:
The abnormal event of a train passing through a tunnel with no passengers train cars significantly impacts the overall situation. In the image, the tunnel empty, which unusual as trains typically filled with pas- sengers or cargo. This lack of activity could due various reasons, such as maintenance work, a temporary suspension of services, or even an emergency situation.
Video-LLaVA [20]:
The abnormal event of the man falling down has a significant impact on the overall situation. It disrupts the normal flow of the game and adds an element of surprise and unpredictability. The players' reactions the event, such as the man getting up and continuing to play, show that the game continues despite the unexpected interruption.
Ours:
Based on the description provided, the abnormal event in the video, which is the man in blue lying on the ground, could potentially have a significant impact on the overall situation. This is because the man's unexpected appearance the ground could cause distraction confusion among the other passengers, which could turn affect smooth operation of the train and the evacuation process if necessary. Additionally, if the man is injured. it could also impact the response time and resources needed to address the situation.
Ground Truth:
Based on the description provided, the abnormal event of person col- lapsing or someone appearing intoxicated and losing their balance has significant impact on overall situation. Such an event can lead serious consequences, including severe injury or fatality, it occurs near train tracks and coincides with the arrival of a train. The risk of a train not being able to stop in time and the individual's inability to move out of harm's way due to their condition magnifies the potential danger and impact of the event. Therefore, the impact is considered critical to the safety of the individuals involved and any potential hystanders.
26

---

(F) Anomaly Video QnsweinnA in UCSD Ped1 [6].
Question
causing the disturbancein video?
00:00
00:11
00:22
Video-ChatGPT [29]:
The disturbance in the video is caused by group of people walking down a sidewalk.
VideoChat [18]:
There is no clear indication of any disturbance in the video. The students are simply walking down the sidewalk, and there that appear be causing any disruption. The video focuses the students they walk, with some them stopping to stand or talk with each other. There are no sounds or other indications of any disturbance in the video.
objects or actions
Video-LLaMA [49]:
There is no disturbance the vidco. The vidco shows people walking on the sidewalk and street in a peaceful environment.
LLaMA-Adapter [50]:
The disturbance in the video is caused by a group of people walking and talking, creating crowd.
Video-LLaVA [20]:
The video does not provide any information about the cause of the distur- bance. However, is clear that the people in the video trying to avoid the disturbance and continue their way.
Ours:
Based the description provided. the cyclist causing the disturbance in the video. The eyclist riding on a path that is designated for pedestrians only, which is the anomaly that could lead to an accident and disturbance. The cyclist is not adhering to the proper traffic rules, which could result in collision with pedestrian.
Ground Truth:
Based on the description provided, it suggests that either the cyclist or the pedestrian, both, could be causing the disturbance by being collision course. Without specific details actions, right-of-way, or the context Icading up to the potential collision, cannot conclusively determine who at fault.
27

---

User
HAWK
Can you help me understand this video?
Of course, happy help you understand a video! Can you please provide more information about the video, such as a description, the context, or specific part you're trying to understand?
Additionally, please let me know what kind of help you need is it analysis of the content, breakdown of the video's meaning, something else?
What anomaly can you identify in the video?
Based on the description you provided, there are several anomalies that can be identified in the video: Car fire: The most obvious anomaly is the fire that occurs near the end of the video. A catching fire without any prior indication malfunction or mechanical issue is unusual and potentially dangerous.
2. Lights flickering: The lights on the car and the building nearby flicker, which could be indicative of electrical issue malfunction in the camera equipment.
How can we solve this problem?
To solve the problem of the fire and the sudden stop the middle of the road, it's important to gather more information and context. Here are some steps that can taken:
Check for any mechanical issues:
2. Look for any external factors:
3. Consult with experts:
4. Conduct a thorough investigation:
Figure 7: Demo on Open-World Video Anomaly Understanding. (Part 1)
28

---

User
HAWK
Where did the fire start?
Based on the information provided in the video, it is difficult to determine with certainty where the fire started. However, there are few possibilities:
1. Engine compartment:...
Battery: ... Electrical system: External sources:
When did the fire break out? What's the impact?
Based on the information provided the video, the fire broke out suddenly and without warning in the middle of the night. The exact time of the fire is not specified, but it is clear that it occurred while the car was parked in secure area.
The impact of the fire significant, it caused significant damage sudden stop in the middle in the area ran towards the car fire, which suggests that the fire was sudden and unexpected. The fire also caused the car to be engulfed in flames, which could have potentially spread to other nearby vehicles structures.
the car and resulted the road. The people
I'm fireman. How do solve this problem?
As fireman, you have the training and expertise to handle emergency situations like the depicted in the video. Here some steps you can take to solve the problem:
Assess situation: When you arrive at the scene, take a moment to assess the situation and determine the best course of action. Check if there are any hazards present, such as flames or smoke, and if anyone is in danger.
2. Call for backup: If the situation is too intense dangerous for you handle alone, call for backup from other firefighters. This will ensure that you have enough personnel handle the situation effectively.
3. Use firefighting equipment:...
4. Evacuate people:...
5. Contain the fire:..
Figure 8: Demo on Open-World Video Anomaly Understanding. (Part 2)
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and title accurately reflect the content of the article.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the paper's 's limitations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
Our paper does not require mathematical theory proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We will release our code, data, and demo and most of the key information is included in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: We will make the data and code publicly available, but to prevent misuse, our data is subject requests from eligible research institutions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ccips.cc/public//uu guwdee/dss///// CCodeSubeesiaceecc ssisnsolssiiiiissse
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present most of the key parameters in the paper, and the remaining details will be open-sourced.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our paper does not include the results of significance statistics, we used given random seed maintain a random initialization, and the results are fixed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //terr///......./
Answer: [Yes]
Justification: Details in Appendix.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Details Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Details in Appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We strictly follow the principle of open data.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details in Appendix.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
These are not included in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
These are not included in our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "KnFAQEs5XF", "reviewer": "Reviewer_b4dX", "summary": "This paper presents HAWK, a framework that uses large Visual Language Models (VLMs) to accurately interpret video anomalies. HAWK integrates motion and video modalities through a dual-branch framework, enhanced by an auxiliary consistency loss to focus on motion-related features. The authors annotated over 8,000 anomaly videos with language descriptions and created 8,000 question-answer pairs to train the model across diverse scenarios. HAWK demonstrates state-of-the-art performance.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The framework benefits from extensive annotations and question-answer pairs, improving training quality. \n2. HAWK achieves superior results in video description generation and question-answering tasks, outperforming existing baselines.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. In Table 1, it would be better to indicate the LLM backbone of the methods for a fair comparison.\n2. From the ablation study in Table 3, it appears that even without the motion modality, the baseline model achieves comparable performance to other methods without motion modality. This may be because the high-quality dataset and strong LLM backbone contribute more to the performance, which weakens the perceived technical contribution of the motion modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous and thoughtful review of our work. We greatly appreciate your affirmation of our work's framework, experimental results, and writing quality.\n\nWe are also thankful for the profound questions you raised regarding our paper and will provide detailed explanations and clarifications in the following sections.\n\n---\n**Response to Q1: What is the LLM backbone of the baselines?**\n\nTo ensure the fairness of our experiments, we employed large language models (LLMs) of the same size (7B parameters) as the backbone for the baselines, as illustrated in the table below:\n\n| Methods | Backbones | LLM Size |\n| :--- | ---: | ---: |\n| Video-ChatGPT [1] | LLaMA 7B | 7B |\n| VideoChat [2] | Vicuna 7B | 7B | \n| Video-LLaMA [3] | LLaMA-2 7B |7B | \n| LLaMA-Adapter [4] | LLaMA-2 7B | 7B |\n| Video-LLaVA [5] | Vicuna 7B | 7B |\n| Ours | LLaMA-2 7B | 7B |\n\nThis information will be incorporated into the experimental section of our work.\n\n\n>[1] Video-ChatGPT: Towards detailed video understanding via large vision and language models. ACL 2024.\n\n>[2] VideoChat: Chat-Centric Video Understanding. CVPR 2024.\n\n>[3] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. EMNLP 2023 Demo.\n\n>[4] LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. ICLR 2024.\n\n>[5] Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. Axriv 2024.\n\n---\n**Response to Q2: Technical contribution of integrating motion modality.**\n\nTo further substantiate the contribution of motion information to the network architecture, we conducted tests using our provided data on baseline models that do not incorporate motion information.\n\n**NOTICE: The results are depicted in Tables (A) and (B) in the global rebuttal mentioned ABOVE.**\n\nGiven the absence of motion modality information in these baselines, the experimental results can affirm that integrating motion information into our framework significantly will help to enhance model performance.\n\nMoreover, in other studies, such as VideoChatGPT [1] and VideoChat [2], the construction of datasets is regarded as a significant contribution, whereas the model architecture is not highly emphasized. Thus, the dataset construction aspect of our work also constitutes a core contribution of this paper.\n\nCertainly, we also agree that developing better frameworks remains a promising research direction. Our future research can delve deeper to explore and unlock more possibilities.\n\n>[1] Video-ChatGPT: Towards detailed video understanding via large vision and language models. ACL 2024.\n\n>[2] VideoChat: Chat-Centric Video Understanding. CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Czu2KP4GTr", "reviewer": "Reviewer_K7jv", "summary": "The paper proposes a new variant of the video anomaly detection. Prior methods were vanilla classification methods, and this paper proposes more descriptive anomaly description and also QA along with that. The paper first gives the dataset creation strategy and then introduces the motion model and the video architecture to provide a text description of the anomaly. The authors also have baselines to compare against recent methods.", "questions": "Please answer the question regarding the experimental details. Overall, I am positive about the paper and hence, will increase the rating once the clarification on the experiments is provided.\n\nMinor comment\n\nGuieded -> Guided in Tab 4", "limitations": "Yes, limitations are discussed and they correctly reflect the shortcomings that I see.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The problem setup is novel and fills the gap in current VAD methods.\n\n2. The dataset creation strategy is very intuitive and the quality looks good from the examples provided.\n\n3. The evaluation protocol is correctly explained.", "weaknesses": "My main concern is regarding the experimentation and the lack of some crucial baselines.\n\n1. The proposed method benefits from the training data, whereas all the baselines are zero-shot. It is difficult to now evaluate the contribution of your motion model and the training data. It is essential to show how the baseline methods like VideoChat will behave when they are fine-tuned with the training data the authors propose. Without these baselines, it is difficult to evaluate the effectiveness.\n\n2. The GPT-4 output shown in Fig. 2 (a car with red lights has lost traction....) is already pretty good. Due to this, the authors must show that the trained model is able to perform better than the basic dataset creation strategy. It is understood that the dataset is created using off the shelf models and expected to perform worse than the trained video and motion model, but a baseline that uses the pseudo-dataset creation strategy is important. (Or an appropriate justification of why it cannot be added)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable time you have dedicated to providing critical reviews and also are grateful for your recognition of the problem setup, dataset, and experimental evaluation in our work.\n\nIn response to your issues, we offer the following more clarifications as follows.\n\n---\n**Response to Q1:  Demonstrating the effectiveness of our motion model through fine-tuning other baselines with our proposed dataset**\n\nTo demonstrate the significance of the motion model, we have conducted ablation studies, the results are presented in Table 3 (A) and (B) of the main paper. Compared with \"w/o motion\", the results demonstrate that the motion modality, video-motion consistency, and motion-language matching, can significantly enhance the overall performance of the framework. Additionally, this experiment represents a fair comparison due to the same training and testing procedure.\n\nTo further demonstrate the effectiveness of the motion model, we fine-tuned other frameworks using the same dataset we introduced. \n\n**NOTICE: The results are depicted in Tables (A) and (B) in the global rebuttal mentioned ABOVE.**\n\nBased on the data presented in Tables (A) and (B), when facing various baselines and maintaining identical fine-tuning data, the incorporation of motion information can enhance the model's effectiveness in both Anomaly Video Description Generation and Anomaly Video Question-Answering.\n\n>[1] Video-chatgpt: Towards detailed video understanding via large vision and language models. ACL 2024.\n\n>[2] VideoChat: Chat-Centric Video Understanding. CVPR 2024.\n\n>[3] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. EMNLP 2023 Demo.\n\n>[4] Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ICLR 2024.\n\n>[5] Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. Axriv 2024.\n\n---\n**Response to Q2: Demonstrating that the trained model outperforms GPT-4-based data generation.**\n\nWhile GPT-4 can assist in video labeling, \"Manual Checking\" remains a crucial step in the data annotation process. As depicted in Figure 2 of the main paper, we show the case with high-quality labeling, which is also subjected to \"Manual Checking\".\n\nIn line 496, \"Manual Checking\" involves correcting errors in the video's textual descriptions. This primarily includes rectifying inaccuracies in the descriptions, eliminating redundant information, and adding correct details regarding objects, motions, and scenes. The table below provides an example of before and after manual checking:\n\n---\n| Before Manual Checking (GPT-4) | After Manual Checking | \n| :--- | :--- |\n| The video depicts different scenes of people walking and sitting in front of a building and a train station. There are also shots of a woman with a red bag sitting on the ground, a man walking by with a shopping bag, and a young boy in a red jacket standing outside a building. There are also several shots of people standing at a bus stop, and a sign with a glowing red hand. | The video shows scenes in front of the subway entrance, with students and pedestrians coming and going or entering and exiting the subway. In addition, two school security guards patrol through, and at the subway entrance, two students are chatting and discussing. | \n---\n\nIn this example, the caption generated by GPT-4 included hallucinations (such as \"a man walking by with a shopping bag\" or \"a young boy in a red jacket\" ), which were corrected after Manual Checking.\nAdditionally, GPT-4's description was inaccurate (for instance, \"bus stop\" should have been more accurately described as \"subway entrance\").\n\nBased on this, only using the results generated by GPT-4 for video understanding is inaccurate and could lead to erroneous descriptions and hallucinations. Therefore, we invested human resources (a total of 150 hours) in Manual Checking for each video's annotations to ensure the dataset is of higher quality.\n\nFurthermore, to demonstrate that our model has better detection capabilities than GPT-4, we compared our model's results with the unchecked labels generated by GPT-4 on the same testset, as shown in the following table:\n\n---\n|Methods|Backbones|LLM Size|BLEU-1|BLEU-2|BLEU-3|BLEU-4|Reasonability|Detail|Consistency|\n|:---|---:| ---:|---:|---:|---:|---:|---:|---: |---: |\n|Data Engineering Pipeline| GPT-4 [1] | - | 0.188 | 0.098 | 0.056 | 0.034 | 0.189 | 0.313 |0.158|\n|Ours| LLaMA-2 7B | 7B | **0.270** | **0.139** | **0.074** | **0.043** | **0.283** | **0.320** | **0.218** |\n---\n\nClearly, the results indicate that our model can better assist in understanding video anomalies compared to GPT-4, achieving state-of-the-art (SOTA) performance in both text-level and GPT-guided evaluations.\n\n> [1] Gpt-4 technical report. Axriv 2023.\n\n---\n**Response to minor comment.**\n\nWe will revise the manuscript to address the typos and also conduct a comprehensive check in the final version to try to correct any minor problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMPddwrTNN", "reviewer": "Reviewer_Cy7X", "summary": "The manuscript proposes a new framework that uses an interactive large-scale visual language model (VLM) to accurately explain video anomalies. The framework can explicitly integrate motion modalities to enhance anomaly identification. The manuscript also constructs an auxiliary consistency loss to guide the video branch to focus on motion modalities. The manuscript annotates more than 8,000 anomaly videos with language descriptions, enabling effective training in different open-world scenarios, and creates 8,000 question-answer pairs for users' open-world questions. The final results show that HAWK achieves SOTA performance, surpassing existing baselines in both video description generation and question answering.", "questions": "As mentioned above.", "limitations": "Yes, the authors address possible limitations of their study.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This manuscript proposes a novel video language framework HAWK, which aims to understand video anomalies, and combines motion modalities to enhance its capabilities.\n2.This manuscript collects seven video anomaly datasets from different scenarios and generates rich language descriptions for each video. At the same time, considering the diversity of open-world questions, question-answer pairs are generated to solve potential user queries.\n3.The proposed method achieves state-of-the-art performance on three public anomaly detection datasets.", "weaknesses": "1. There are already many studies focusing on the background information in abnormal events. Have the authors considered describing the background information when describing the action information?\n[1] Scene-aware context reasoning for unsupervised abnormal event detection in videos[C]//Proceedings of the 28th ACM international conference on multimedia. 2020: 184-192.\n[2] Few-shot scene-adaptive anomaly detection[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020: 125 -141.\n[3] Hierarchical scene normality-binding modeling for anomaly detection in surveillance videos[C]//Proceedings of the 30th ACM international conference on multimedia. 2022: 6103-6112.\n2.The dataset proposed in the manuscript also supports question answering in open scenes, but the method does not reflect how to use these questions and answers to improve the model's anomaly detection and understanding capabilities.\n3.When constructing a dataset, if a description is generated for each second in the dataset, it may result in a lot of repeated descriptions. In order to avoid this redundancy problem, have you considered using keyframes instead of second-by-second descriptions to reduce the complexity and computational cost of data processing?\n4.Most previous studies pay equal attention to various parts of the video, such as background, motion information, character appearance, etc. This paper focuses on motion information. Section 4.3 mainly extracts language descriptions related to motion. Whether background information is not considered. Moreover, the 7 datasets proposed in the manuscript contain more abnormal scenes. I am very curious why the scene information is not paid attention to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your acknowledgment of our efforts, especially for highlighting our contributions to the framework, dataset, and experimental outcomes. Additionally, we are thankful for the precious suggestions you have offered, which will significantly aid in the enhancement of our manuscript. \n\nIn the following sections, I will systematically respond to the queries and concerns you have presented.\n\n---\n**Response to Q1 and Q4: Missing background information and other parts?**\n\nFirstly, although we have significantly enhanced motion information in Sections 4.2 and 4.3, this enhancement does not preclude our core task from maintaining vital scene information, which includes background details and appearances. \n\nAs delineated in Line 227 (Optimization Goal) of the main paper, our primary task is established upon all parts of the video, controlled via $t_0 = 1$. Motion information only serves as an augmentation for anomaly information, with its optimization weights set to $t_1 = 0.1$ and $t_2 = 0.1$. Thus, our optimization goal does not neglect the comprehension of background information or other parts.\n\nMoreover, to focus on motion information is due to the observation in other video understanding baselines (e.g., Video-LLaMA) that an excessive focus on understanding background information led to a diminished capability in grasping video anomalies (as shown in Figure 1 and Table 2(A) in the main paper). \nTherefore, to enhance the understanding of video anomalies, it is essential to enhance the representation of motion information within the network.\n\n**About Reference Paper.** Regarding the works you referenced, [1] and [3], these studies also utilize object-bounding boxes to prominently describe objects involved in anomalies and employ a dual-path strategy to amplify the representation of these objects within the network. These objects, typically humans or vehicles, are also in motion during anomalies, whereas background information is usually static.\n\n**Our Comment.** We agree that leveraging background information can provide a robust prior for understanding video anomalies, as [1,2,3] indicated. We will try to integrate information related to scenes, backgrounds, and objects in a large-model-based video anomaly understanding model.\n\n> [1] Scene-aware context reasoning for unsupervised abnormal event detection in videos[C]//Proceedings of the 28th ACM international conference on multimedia. 2020: 184-192.\n\n> [2] Few-shot scene-adaptive anomaly detection[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020: 125 -141.\n\n> [3] Hierarchical scene normality-binding modeling for anomaly detection in surveillance videos[C]//Proceedings of the 30th ACM international conference on multimedia. 2022: 6103-6112.\n\n---\n**Response to Q2:  Use generated questions and answers to improve the model's anomaly detection and understanding capabilities?**\n\nWhile we introduce the generation of question-answer (QA) pairs, it is crucial to clarify that the primary objective of generating these pairs was **NOT** intended as a means of data augmentation to enhance the model's capabilities in understanding anomalies. Instead, the improvement in our model's understanding of anomalies is **ONLY** through the accurate descriptions of anomaly videos and the motion-aware network.\n\n**Distinctively**, the generation of QA pairs aims to cover potential inquiries from users in an open-world setting, thereby facilitating improved user interaction with the network. Therefore, as shown in Table 1 (B), the Anomaly Video Question-Answering is evaluated as a separate task. This evaluation is specifically designed to simulate the performance of the system when utilized in the open-world environment.\n\n---\n**Response to Q3: Why not use keyframes to reduce the complexity and computational cost of data processing?**\n\n**Our Consideration.** First, our current methodology for dataset construction involves sampling the video at consistent one-second intervals. This technique is strategically chosen to ensure that all possible anomalies within the videos are comprehensively captured (Some anomalies only happened in 1-2 seconds), thereby significantly reducing the likelihood of missing critical accidents.\nWhile we realize that this may lead to a degree of caption redundancy, we still prioritize the facilitation of thorough annotation to ensure that all anomalies are detected.\n\nIn addition, we have leveraged the capabilities of GPT-4 for generating captions, especially for anomalous events (refer to Figure 2 Prompt). Due to GPT-4's advanced text generation and summarization abilities, it serves as an effective tool in minimizing redundancy, ensuring that the extracted captions are both high-quality and succinct.\n\nAfter the initial processing with GPT-4, we also undertake a manual checking process. This step is crucial for further reducing any residual redundancy and correcting possible errors within the captions, thereby ensuring the quality and accuracy of our dataset.\n\n**Future Work.** Certainly, we agree that the utility of keyframes is an effective strategy, especially for much longer videos, and believe its potential to significantly enhance data annotation efficiency. This will be the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HdBDxzzW4d", "reviewer": "Reviewer_kzaw", "summary": "This work proposes a novel video-language framework, HAWK, aiming at understanding video anomalies, which incorporates motion modality to enhance its capability. The work generates rich language descriptions for seven different video anomaly datasets, and also generates question-answer pairs to tackle potential user inquiries. The proposed framework demonstrates SOTA performance for video anomaly understanding and question-answering across multiple scenarios, which will advance the open-world anomaly understanding field.", "questions": "See the Weakness part.", "limitations": "The paper has discussed the limitations and potential impacts of the work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The work proposes a novel vision-language framework to address open-world video anomaly understanding, which is a very different pipeline from previous classification-based anomaly detection pipelines. To build a vision-language model for open-world video anomaly understanding, the work adopts seven different video anomaly datasets for training, where rich language descriptions and question-answer pairs are generated. Experiments for video anomaly understanding and question-answering across multiple scenarios demonstrate the effectiveness of the proposed framework. I believe the work will advance the field of open-world video anomaly understanding.", "weaknesses": "1. The work adopts Gunnar Farneback’s algorithm to obtain the motion modality. Is this algorithm efficient? \n2. The Ubnormal dataset consists of virtual anomaly videos. Do the authors consider the gap between real and virtual anomaly videos, which is mentioned in previous works [1, 2]?\n\n[1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022\n\n[2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our profound gratitude for your esteemed recognition of our paper. Furthermore, we sincerely appreciate your constructive feedback and suggestions, which have proven to be instrumental in enhancing our manuscript. \n\nWe are devoted to addressing all of your proposed problems, as shown in follows:\n\n---\n**Response to Q1: The Efficiency of Gunnar Farneback’s Algorithm for Motion Modality Extraction**\n\nFirstly, Gunnar Farneback's algorithm demonstrates remarkable efficiency in generating video optical flows—even on CPU platforms. For each frame, the efficiency of this algorithm surpasses that of other widely deployed methods, as illustrated in the table below:\n\n\n| Methods | Gunnar Farneback[1] | LDOF [2] | FlownetS [3] | FlownetC[3] |\n| :---: | :---: | :---: | :---: | :---: |\n| Second(s) per frame | 0.02 (CPU) | 65 (CPU) & 2.5 (GPU) | 0.08 (GPU) | 0.15 (GPU) |\n\nSecondly, for processing one video, the motion of multiple rounds of dialogue necessitates just a single iteration of video motion extraction. Consequently, the response time for processing one video (about 0.72 seconds) is significantly shorter than the time required to generate a single round of dialogue (about 1.5 seconds). This cost is deemed acceptable for a practical anomaly detection system for users.\n\nLastly, we concur that future research into more efficient methods for optical flow extraction, including end-to-end optical flow extraction strategies, will likely further augment the efficiency of our system.\n\n>[1] Two-frame motion estimation based on polynomial expansion. In Image Analysis: 13th Scandinavian Conference, SCIA 2003.\n\n>[2] Large displacement optical flow: descriptor matching in variational motion estimation. TPAMI, 2011.\n\n>[3] FlowNet: Learning Optical Flow with Convolutional Networks. ICCV 2015.\n\n---\n\n**Response to Q2: The gap between real and virtual anomaly videos within the Ubnormal dataset.**\n\nFirstly, we acknowledge that some existing papers considered the distinction between real-world and virtual anomalies. In our paper, to mitigate the potential discrepancies and bridge the gap between these two types of data, we have mixed all datasets to train an open-world model in the domain of both synthetic and real-world datasets, as shown in Appendix D (Data Distribution). This strategy ensures that our model is versatile and performs well across different data types, as shown in Appendix F.\n\nBesides, while the primary focus of our current work does not delve into the intricacies of how synthetic data can be leveraged to enhance the detection of real-world anomalies in [2], we recognize the importance of this aspect. The potential of synthetic data to boost the performance of anomaly detection systems is substantial, and we agree that exploring this potential further could yield significant advancements.\n\n>[1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022\n\n>[2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Firstly, we would like to express our heartfelt gratitude to all the reviewers for their insightful and constructive suggestions. These suggestions have been immensely helpful in refining our paper and have provided valuable guidance for our research direction.

---
**Restating Our Contributions**

The principal contribution of our work is the development of a novel framework aimed at understanding video anomalies. This framework, by integrating motion modality information, significantly enhances the capability to comprehend abnormal events.

Moreover, we have built a comprehensive dataset containing over 8,000 annotated video anomalies across various scenarios. The creation of this dataset enriches the training materials for the model, enabling it to better adapt to the needs of open-world anomaly understanding.

We believe that, with the valuable feedback from the reviewers to refine the paper, this research will bring innovative insights and practical solutions to the field of video anomaly understanding.

---
**Supplementary experiment for Reviewer b4dX and Reviewer K7jv**

To further demonstrate the effectiveness of the motion model, we fine-tuned other frameworks using the same dataset we introduced.

---
**(A) Anomaly Video Description Generation**

|Methods|Backbones|LLM Size|BLEU-1|BLEU-2|BLEU-3|BLEU-4|Reasonability|Detail|Consistency|
|:---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|Video-ChatGPT [1]|LLaMA 7B|7B|0.240|0.121|0.062|0.021|0.159|0.177|0.132|
|VideoChat [2]|Vicuna 7B|7B|0.123|0.101|0.048|0.019|0.198|0.271|0.144|
|Video-LLaMA [3]|LLaMA-2 7B|7B|0.250|0.135|0.069|0.038|0.279|0.319|0.195|
|LLaMA-Adapter [4]|LLaMA-2 7B|7B|0.253|0.136|0.070|0.040|0.240|0.190|0.172|
|Video-LLaVA [5]|Vicuna 7B|7B|0.150|0.072|0.031|0.010|0.101|0.201|0.118|
|Ours|LLaMA-2 7B|7B|**0.270**|**0.139**|**0.074**|**0.043**|**0.283**|**0.320**|**0.218**|
---

**(B) Anomaly Video Question-Answering**

|Methods|Backbones|LLM Size|BLEU-1|BLEU-2|BLEU-3|BLEU-4|Reasonability|Detail|Consistency|
|:---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|Video-ChatGPT [1]|LLaMA 7B|7B|0.188|0.109|0.063|0.042|0.612|0.565|0.527|
|VideoChat [2]|Vicuna 7B|7B|0.282|0.143|0.090|0.049|0.712|0.663|0.567|
|Video-LLaMA [3]|LLaMA-2 7B|7B|0.168|0.089|0.051|0.030|0.624|0.527|0.449|
|LLaMA-Adapter [4] |LLaMA-2 7B|7B|0.239|0.142|0.088|0.050|0.673|0.572|0.532|
|Video-LLaVA [5]|Vicuna 7B|7B|0.123|0.098|0.069|0.047|0.620|0.528|0.533| 
|Ours|LLaMA-2 7B|7B|**0.319**|**0.179**|**0.112**|**0.073**|**0.840**|**0.794**|**0.753**|
---

Meta-review: The manuscript received ratings of two weak accept, one accept and one reject. Reviewers appreciated the proposed benchmark, approach and experimental results but also raised some concerns such as the impact of motion modality and strong LLM backbone. Authors provided a detailed rebuttal with three reviewers remained positive. Given the reviewers feedback, rebuttal and reviewer discussions, the recommendation is accept. Authors are strongly encouraged to take into account all the reviewers feedback along with publicly releasing the dataset, code and models, when preparing the final draft.
Decision: Accept (poster)