Towards Multi-Domain Learning for Generalizable Video Anomaly Detection
Authors: MyeongAh Cho
Taeoh Kim
Minho Shim
Dongyoon Wee
Sangyoun Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Video anomaly detection
Multi-domain learning
Domain generalization
Primary area: machine_vision

Abstract: Most of the existing Video Anomaly Detection (VAD) studies have been conducted within single-domain learning, where training and evaluation are performed on a single dataset. However, the criteria for abnormal events differ across VAD datasets, making it problematic to apply a single-domain model to other domains. In this paper, we propose a new task called Multi-Domain learning forVAD (MDVAD) to explore various real-world abnormal events using multiple datasets for a general model. MDVAD involves training on datasets from multiple domains simultaneously, and we experimentally observe that Abnormal Conflicts between domains hinder learning and generalization. The task aims to address two key objectives: (i) better distinguishing between general normal and abnormal events across multiple domains, and (ii) being aware of ambiguous abnormal conflicts. This paper is the first to tackle abnormal conflict issue and introduces a new benchmark, baselines, and evaluation protocols for MDVAD. As baselines, we propose a framework with Null(Angular)-Multiple Instance Learning and an Abnormal Conflict classifier. Through experiments on a MDVAD benchmark composed of six VAD datasets and using four different evaluation protocols, we reveal abnormal conflicts and demonstrate that the proposed baseline effectively handles these conflicts, showing robustness and adaptability across multiple domains.

Towards Multi-Domain Learning for Generalizable Video Anomaly Detection
MyeongAh Cho Kyung Hee University
Taeoh Kim
Minho Shim NAVER Cloud
NAVER Cloud
maycho@khu.ace ac.kr
taeoh. www@navercorppcccomm.m.......... com
minho wwwnnaaeercrrppccooooooo com
Dongyoon Wee NAVER Cloud
Sangyoun Lee Yonsei University
dongyoon www.:owwweeeeeaeeerr....oo.
syleeeGyonsei.. ac
Abstract
Most of the existing Video Anomaly Detection (VAD) studies have been conducted within single-domain learning, where training and evaluation are performed on a single dataset. However, the criteria for abnormal events differ across VAD datasets, making problematic to apply a single-domain model to other domains. In this paper, propose new task called Multi-Doman learning for VAD (MDVAD) explore various real-world abnormal events using multiple datasets for general model. MDVAD involves training on datasets from multiple domains simultane- ously, and we experimentally observe that Abnormal Conflicts between domains hinder learning and generalization. The task aims address two key objectives: (i) better distinguishing between general normal and abnormal events across multiple domains, and (ii) being aware of ambiguou abnormal conflicts. This paper is the first to tackle abnormal conflict issue and introduces a new benchmark, baselines, and evaluation protocols for MDVAD. As baselines, we propose a framework with Instance Learning and an Abnormal Conflict classifier.
Null( Angular)-Multip Through experiments on a MDVAD benchmark composed of six VAD datasets and using four different evaluation protocols, we reveal abnormal conflicts and demonstrate that the proposed baseline effectively handles these conflicts, showing robustness and adaptability across multiple domains.
Introduction
Video Anomaly Detection (VAD) is identifying abnormal events diverse scenarios depicted in video and determining their temporal intervals frame-level. Nowadays, CCTVs ubiquitous, recording every moment of life, which aids in preventing accidents and responding to crimes promptly. However, human monitoring of every situation is highly inefficient, requiring significant labor and resources. Consequently, extensive research has been conducted to automate VAD through deep
learning by leveraging large amounts of surveillance data 33
24
15
55
44
43
50
21 7]
In VAD research, Weakly-supervised VAD (WVAD) (55 44 43
21 7 which involves learning normal and abnormal events with minimal supervision of the video-level annotation and detecting abnormal events at the frame-level during testing, has been studied a lot recently. This paper focuses on the WVAD setting (denoted as VAD), and summary of VAD research is explained in the
50
Work done while doing an internship NAVER Cloud. Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

uz uas Normal
D1
D2
TAB: Absormal
n D2
Abnormal
u D1 n
TAlk: Absormal
D3
Abnormal Conflicts
(a) Examples of Abnormal Conflict between datasets
(b) Venn diagrams of events
Figure (a) An example of abnormal conffict: Pedestrian on the road normal in UCFC dataset but abnormal in TAD. (b) Each circle represents each domain. MDVAD aims to design a general model that effectively considers abnormal conflicts to separate general normal and abnormal events.
supplementary material ($ three key questions:
ED. Unlike the conventional VAD research, we first address the following
Q1: What the problem with the existing VAD model? Most VAD models are trained single- domain where the training and evaluation dataset are the same. In the case of single-domain learning, application across different datasets (cross-domain evaluation) results performance degradation reported in [14 6 because VAD models are heavily influenced by the criteria for abnormality defined by each dataset.
Q2: Why do we need general VAD model? First, a single generalized model removes the need for multiple specific models for different domains, analogous to multi-task learning. Second, proper pre-training on multiple domains embodies generalized representation, and it can discriminate abnormal events according to their domain, leading to better performance in unseen target domains. Consequently, a general VAD model will be highly beneficial for applying VAD in practical scenarios.
Q3: Is possible create a general VAD model? The general VAD model aims handle multiple domains, but this is challenging because the definition of abnormal differs for each dataset, leading to conflicts between these abnormal events. For example, as shown Fig. [(a) in one dataset, pedestrian on the road is considered normal, while in another dataset, it is deemed abnormal. A general VAD cannot be solved with a naive muti-task learning because of this confusion across multiple domains, which this paper defines as 'Abnormal Conflict.' Therefore, for general VAD, it necessary to be aware of these abnormal conflicts (yellow region in Fig. [(b) and learn general
normal (green region in Fig. ((b) or abnormal (red region in Fig. [(b)) representations that are common across all domains.
Our goal is to construct a general VAD model by conducting multi-domain learning while recogniz- ing abnormal conflicts and exploring representations general normality and abnormality. To achieve this goal, we introduce new task called 1) Multiple Domain VAD (MDVAD), along with benchmark and new evaluation protocols. MDVAD involves concurrent training on multiple VAD datasets, each with its own definition of abnormality. Specifically, the MDVAD benchmark comprises six representative VAD datasets with balanced sampling ($ 1. We also propose four evaluation protocols: held-in, leave-one-out, low-shot domain adaptation, and full fine- fine-tunig. The held-in protocol is designed to evaluate the model's ability as unified model like multi-task model, while the leave-one-out, low-shot domain adaptation, and full fine-tuning protocols are intended to access the model's capability as a general pre-training for an unseen target domain.
As multi-domain learning is a novel concept in the field of VAD, we also introduce baselines and new learning methods. We design domain domain-specific multiple heads to mitigate abnormal conflicts and learn generality across domains. To facilitate multi-head learning without conflicts, we propose the 2) Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIII losses, which activate only the output of the head corresponding to the input domain, assigning inactive heads with Null values to prevent confusion. Additionally, we suggest the 3) Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts, leveraging the variance abnormal scores across the multiple heads (Fig. green and yellow region in Venn diagram). Through experiments with four protocols on the MDVAD benchmark, we reveal the limitations of multi-domain learning with abnormal conflict and demonstrate the effectiveness of our baselines in offering generalized and adaptive model.

---

1.1 Scope of research
Focusing on Multiple Domains. this paper, we focus on solving the problems multiple domains introduced above rather than solving problems in a single domain. Therefore, complex design of the backbone and head or achieving state-of-the-ars performance in a single domain can proceed orthogonally with our research. Instead, this paper raises the issue of abnormal conflict and focuses on the necessity of MDVAD.
Distinction from Open-set VAD Approaches. Unlike Open-set VAD 59
25 D. which separates
seen and unseen anomalies within a single-domain cannot achieve robust representation learning among multiple domains, this paper excels at handling abnormal conflicts and shows adaptability across domains with versatile evaluation protocols meeltin/outt and low-shot).
2 Observations
2.1
Datasets
In this paper, we use six representative VAD datasets: UCF-Crimes (UCFC) 43 XD-Violences (XD) [5T], Large-scale Anomaly Detection (LAD) (7)) UBI-Fights (UBIF) 9 Traffic Anomaly
Dataset (TAD) [9). and Shanghai-Tech Campus (ST) (24) As shown in Table ! each dataset has different environments (e.g. CCTV, Traffic, Campus), quantities, and abnormal categorics. Unlike other datasets, ST is unsupervised VAD benchmark whose training set comprises only normal More details are provided in
videos, so the training set has been reorganized following 21 45 57 the supplementary material (Sec. A)
2.2 Analysis
Based on observations of each dataset's properties, we aim to quantify how the different properties negatively affect domain shifts. Table presents cross-domain evaluation results, iee, single-domain models validated on different target datasets. The results reveal that while these models excel in- domain settings (diagonal elements of the table), they exhibit significant performance degradation in cross-domain scenarios. This implies that single-domain VAD model may be ineffective in most other environments unless the environment and user's intention are precisely identical. Therefore, leveraging diverse datasets for generalized feature learning is crucial, enabling the model to handle unknown domain and well adapted to unseen anomalies. This paper addresses two primary issues: 1) Abnormal conflict arising during the multi-domain learning process, and 2) Scene discrepancy occurring during the evaluation process on an unseen target domain.
Abnormal conflict. As illustrated in Fig. I abnormal conflicts indicate abnormal events that are considered abnormal in one (or some) domain(s) but are denoted as normal in other domains. As
Table I: Datasets and abnormal categories. Colored categories are shared abnormal categories with other datascts. Uncolored categories are considered normal eveuts in other datascts. Gray categorics are defined by ourselves while all other categories are provided datasets.
Table Anomaly detection performances (Area under curve, AUC) single-domain models. Diagonal elements domain results and ofl-diagonal elements are cross-domain
Eanget
UCFC
XD
LAD 37
UBIF
TAD
9
ST 24
Soaree
DCFC
X0
LAD
UBIF
3T
Scilings
DUR
R2.32
08106
75.73
71.12
73.75
39.25
CCTV,
XD
68.38
90.37
77.00
67.25
71.10
46.37
CCTV
Sports, Cartoon,
CCTV
PCTV Mobile
Thilki (CCTV)
Campux (CCTV)
UBIF
LAD
39.60
74.70
75.36
75.22
96.97
70.20
93.65
59.27
73.80
68.16
47.20
54.21
Movics,
TAD
30.83
45.38
53.02
61.95
90.71
41.58
News
55.
52.87
Training Set Volume
I8T0
8054
400
A Almormal Categories
Table Scene discrepancy between demaiins (Earth mover's distance, EMD), Yellow: Abmornaal Class wise. Blue: Nor- Using pretrained backhone model on the Kinetics datasct ( we extract embedding features for cach datasct between
Aluus
Altucc
Hun
Hegaa Turns
Chaenn
Aurest
Dose
Roud Spitls
Decpping
Aneen
Reteograde sil Moviun
Throwing
Awanti
Fulling
Illegd Ouceppatios
Running
Ausidaat
Burglay
Auxidens
Desroy
Czadi
Pedestraz
Ascileats
Mooocycle
Junping
and
vectors.
Explocio
Explosion
UCTC
XD
L.U
UBD
15.0
5Y
Stealing Shoplifting
Robbes
Shocting
Shoocing
Fighting
Violence
Thiesing
Crowd
Fighting
Robbery
UCRL
UBIF
LAD
XD
12.08
9.75
7.00
10.75
7.90
7.37
11.99
5:19
5.25
5.87
6.54
7.50
10.54
10/00
9.76
9.65
10:45
9.10
0.54
9.32
Vandalium
Rist
Trmppldd
Lettzing
Pasic
Lastering
TAD
ST
11.32
11.32
10.87
10.11
14.30
14.27
10.88
8.77
11.77
1065

---

Demain-Agnesti Layers
Single/Multiple Abnormal Heads (Sec. 3.1 & 3.2)
Video Backbone te
+
Feature Agg egation
Tempordl Apprepation
PC
Abrernal lloc
Top-K
B
Eapo
Abror Hoad sel
$2
Top-K
Input Video
(d-th Domain)
50y
Null
Nol--MI
Classifler (Sec. 3.3)
SD
Classiie
Train-Only
Norma Hoadlt
Top-K
AC Loss
Label
hornul Bes Wiw
6bx
Null
NullAng-MILL
Abnornal Conflict-
Abnomaa Conflict
NullAng-MIL
lIwE,
46.4
Null
I
[|vs,]
Figure 2: The overall framework of our MDVAD baselines that consists of domain -anaosic layers, single ahnormal head (Sec. 3.1 multiple abnormal heads (Sec. 3.2). and AC classifier (Sec. 3.3
shown in Table there is relatively little overlap of abnormal classes in ST with other datasets. In other words, this means that the abnormal conflict with other datasets is relatively large. This conflict leads to low cross-dataset performances of models trained or evaluated on ST in Table Taking UCFC as example (target domain: UCFC column in Table performance increases in the order of TAD, ST, LAD, XD, and UBIF, which is proportional to the number of abnormal conflict categories that exist in the source domain. Abnormal conflict makes the MDVAD unique the label spaces between domains literally conflict because each other's differing definitions.
Scene discrepancy. Scene discrepancy refers to differences in the visual settings of scenes, distinct from abnormal conflict arising from variations in the definition of abnormal classes. To quantify scene discrepancy, we utilize the Earth Mover's ' Distance (EMD) [38] introduced in [8] to calculate the distance between VAD datasets in the Table 3 In Table 3 the top-right section illustrates the
comparison of normal features, while the bottom-left provides numerical comparison of class-wise abnormal features. Lighter colors indicate a higher discrepancy between datasets. Unlike other datasets, TAD, which comprises traffic videos, exhibits a large distance from normal sample distances the dataset, while LAD, with diverse complex scenes for abnormal classes, shows the furthest distance from other datasets. This explains the results of domain adaptation experiments. (Table
Baselines
Fig. depicts the overall framework of our MDVAD baselines divided into Nomain-agnosiie layers, consisting of the video backbone and aggregation modules, Single 3.1) or Multiple heads ($ 3.2).
and AC classifier ($ 3.3).
Donain-agnostic layers. The input abnormal normal video Vo or v is divided into uniformly sampled snippets (V" E [vf,-.. ... v).)). These T snippets pass through a pre-trained video backbone, resulting in a C-dimensional feature B E RTxC that undergoes an aggregation module, fusing them from the feature level to the temporal level. The feature aggregation layer doubles the
channel size B E TC followed by split and max operation F Pagg.; max B: B Cel
-L.... to squeeze the channel size back C for the i-th snippet. Activating only the maximum element during gradient propagation enables the implicit differentiation of specific channels, allowing the model to establish discrepancies between classes 7 52). The temporal aggregation layer, with temporal kernel size of 3, produces the domain- aggregated feature F FDA B Tx/22

---

3.1 Single-domain learning
The final feature DA undergoes fully connected layers denoted as and the single abnormal head (wB, I followed by a sigmoid function to derive the final abnormal score sD, RR'x1
MIL. Due to the absence of temporal interval training labels for abnormal events, WVAD models rely on video-level labels for training, employing the Multiple Instance Learning (MIL) method. When the top-K K score set is represented as (s ). the Binary Cross Entropy-basee (BCE) classification loss function is formulated as presented Eq. where = {0. 13.
LAFIL Σ
(ylogsD + (1 ylog B),),,
(80, )
This loss function ensures that only snippets with high (Top-K) abnormal scores can contribute to the loss. This single-head model serves as the single-doman baseline (MIL in Fig. 2
3.2 Multi-domain learning: Multi-head learning
Null-MIL.. To address the abnormal conflict issue mentioned in Sec. in the MDVAD framework, the abnormal head is divided into multiple heads, each responsible for its own domain, allowing for the domain-wise prediction of the output score. Inspired by [18), the prediction score for the input snippet from the Da dataset is derived exclusively from the output of the Da-head, and the results from heads of other datasets are filled with Null values (Null-MIL in Fig. 2). Compared to Eq. I the Null-MIL loss is changed follows:
M
LN MIL
(ylogs + (1 y)o BB.)))
(2)
d=1 (
where M is the number of heads (domains). To avoid the abnormal confficts, the heads between datasets are separated where Da-head's weight w is independently trained for the corresponding dataset. In Eq. 2 only Sb among all output scores is added to the loss, thus the gradient becomes axb, bwb,
while other heads are not affected.
For the test, the abnormal score i-th snippet is s when the target dataset Da and max D by selecting the maximum value for unseen target data.
NuILAng-MIL.
MIL method that employs the angular margin to effectively diminish large variations among intra-class instances. In this case, multiple normal heads added (NullAng-MIL. in Fig. 2 When head classifier weight of each dataset Da is denoted as w and w D, and final embedding feature F, the final abnormal and normal scores represented by sb, F wD, and $b
We additionally propose a NullAngular-bue
F wD,,- respectively. Normalizing the head weight and feature vector to results in sh = |[ w/ |l cos 0% = cos Ob, and s" cos "b.. representing cosine similarity. Thus, in the cosine space, Eq.2 can be defined Eq. B requiring the maximum cosine similarity between the feature from dataset Da and the abnormal head w to be at least an angular margin of m greater than the normal head wD, that enlarging the gap between normal and abnormal.
max cos (B + m) >> ?
cos A f..
Denoting the top-K abnormal scores from the head of dataset Da (s DS i=topkindices" indices' rewriting Eq. 2 angular margin- based regression problem results in Eq. 4 Similar to Null-MIL, the loss is computed by the head associated with the input dataset, while scores from other heads have no impact on updating the model's 's weights.
LNuilAng-Ma MILL
M
cos (0% +m)
(ylog cos (0g
+m)
cos 0n
1=1
(ED
+ e
cos (07 +m)
+ (1 y)log C
(4)
cos 0a + cos (0 (0 m))
5

---

For the test, since the normal and abnormal heads are trained through angular margin learning, the abnormal score is shown in Eq. 5
Abnormal Score;
Dan + (1 8 Baad m Da.s + (1 - max, 8 Da.
source Da=te
(5)
sourcey/target
When the source domain the pre-trained general model different from the target domain, we determine the final score to reflect conflicts taking the maximum normal and abnormal score from multiple heads.
Unseen domain adaptation. After multi-domain learning, when a new target dataset or unseen condition appears, the final score is computed based on the similarity between the embedding feature of the input video and each source domain's 's head classifier. Therefore, it operates by considering domain similarity, which can address the performance degradation issue that occurs due to the scene discrepancy discussed in Sec. 2
Complexity. Only the final layer is added based on the number of datasets, with the head's weight denoted as WD, E RTx1, which is very small proportion of the entire model. Comparing a single and multiple heads (6 datasets), the training times 2.68 and 2.81 hours, and the inference times are 0.158 and 0.164 milliseconds per snippet, respectively, indicating negligible increase in complexity.
3.3 Abnormal Conflict (AC) classifier
In the domain-agnostic feature extraction phase, it is crucial to capture general features that can handle every domain. While we have divided the heads to avoid abnormal conflicts in multiple source datasets, the agnostic part extracts features from all datasets using a single branch, and inconsistent labels cause confusion. Therefore, we propose an Abnormal Conflict (AC) classifier for learning the final embedding feature F D that passes through the detector heads.
In Fig. 2 each classifier head distinguishes between Da and D& while the AC classifier intended to distinguish between elements that abnormal normal across all source datasets (green area) and elements that represent conflicts (yellow area) within the Venn diagram. The AC classifier takes the embedding feature FD as input, followed by two FC layers, predict the final conflict score ACC In Eq. 6, the AC label is generated based on the scores of all abnormal heads, where y = if the deviation between the scores is above a threshold
[max D min, Da.4 +]+
y/
AC
0
otherwise
(6)
The loss of the AC classifier, denoted as LAC and calculated using cross-entropy as follows:
T
LAC
+ +(1 Mog sAO)
The total objective function is L LNuutAng- MIL + ALAC.
During the testing phase, the auxiliary branch AC classifier is eliminated, and the output is calculated for each input snippet v, as the final Abnormal Score;
4 Experimental Results
4.1 MDVAD benchmark
As shown in Table [T, VAD comprises six representative datasets with diverse settings and volumes. The MDVAD is a task aimed at addressing domain shifts between datasets, and each dataset included the MDVAD benchmark should be structured so that it not biased toward any particular dataset or anomalies. Consequently, datasets should have an equal volume in the training set and encompass various abnormal categories and criteria. To achieve this, we sampled each dataset to align with the dataset with the smallest volume, ensuring that each abnormal category has similar proportion

---

Table 4: Single-domain results (AUC)): In-domain (diagonal ele-
Table S: El: Multi-domain training: held-in results (AUC).
ments) and cross-domain (off-diagonal elements) results.
MOVAD Benchmarks
Source
MDVAD Denchmarks
XB
LAB
1arget
ST
Single-dmmain
DCR:
TAD
lnreet
URIF
TAD
Aag.
UCFC
67.07
14.23
67.95
50.94
Aur Avg.
61.39
68.10
6683
66.37
61.90
49.21
61.43
XD
65.88
83:23
72.63
70.38
63.46
(e-demam))
(72.93)
(83.25)
(8382)
(92.60)
(90.75)
(90.79)
(86.53)
LAD
60.23
71.37
63.82
64.73
68.47
47.27
El: Hebdin
URIF
74.11
70:49
67.59
92.62
68.03
56.47
Head
ACT
UBIF
TAD
Ang.
TAD
56.21
45.08
59.67
67.83
90.75
41.92
MIL.
50.05
K1.77
85.76
$3.90
85.56
ST
50,49
61.53
60.91
36.38
41.60
90.79
80.11
K3.91
85.15
87.72
90.05
87.98
85.82
Our Avg
61.30
63.10
66.83
66.17
61 61.90
49.21
Nail-MIL
79.01
$1.96
85.06
93.06
90.57
91104
86.75
(In domaia)
(7.90)
(83.23)
(85 82)
(92.62)
(90.75)
(90.79)
(Ours)
82.95
85.82
92.41
91.16
86.36
Training: Single source conaset? Testing: Targei Galaset
62.74
86.13
Out Ang: Averge of cross-domain roults.
MiL(OOrs)
77.21
$2.09
8338
91.36
Training: drtnscts Testing: lagget daraact
Column-wisc coloring with increvsed intersity higher values
and conducted 3-lold experiments. Additionally, to handle small datasets like TAD and ST, which have minimal volumes, we combined them with the traffic dataset CADP [41] and campus dataset NWPU (3) respectively, by reorganizing their training sets. The volume of MDVAD benchmark is
unified at 386 videos per dataset. More details provided in the supplementary material (Sec. B
4.2 Empirical studies
This section presents empirical studies as follows: first, single-domai results for comparison, followed by the results of four evaluation protocols for the MDVAD benchmark. The models are trained on the training set of the source domain and evaluated on the test set of the target domain under the following settings: Held-in (EI), Leave-one-out (E2), Low-shot adaptation (E3), and Full fine- tuning (F:4). The Area Under Curve (AUC) used as the evaluation metric. The hyper-parameter are T = 32, À 10, 0.3, and m 0.3. Since the single-head baseline cannot assign AC labels using Eq. 6 pscudo-labels for the AC classifier are assigned based on the range of predicted abnormal scores. All detailed implementations, more results, and discussions are provided in the supplementary material.
Single-domain results. In the preliminary experiment, we examine the performance of the MDVAD benchmark under single-source and single-target conditions using a single-head MIL baseline. Comparing the in-domain (diagonal elements) results in Table (using the original training set) and Table 4 (using (using the sampled training set), we observe that the performance diminishes when datasets, other than TAD and ST, are reduced during sampling. Conversely, TAD and ST show improved performance, benefiting from increased diversity and data volume due to reorganization with CADP and NWPU for TAD and ST, respectively. The cross-domain (off-diagonal elements) results in Table 4 demonstrate consistent trends with both the MDVAD and VAD benchmarks. These trends primarily arise from challenges related to abnormal conflict and scene discrepancies, indicating that the issues confined to in-domain settings. Consequently, to uphold in-domain performance across diverse datasets and effectively address issues related to abnormal conflict and scene discrepancies, we emphasize the essential role of multi-domain learning in developing a general model for VAD.
4.2.1 Held-in results (E1)
In this section, we discuss the held-in evaluation results, where models are trained on all datasets of MDVAD, and the test set of each source dataset used as the target. Tabeee55presents the results obtained by training with the MIL baseline with single-head, which is commonly used in traditional WVAD, and the proposed Null(Ang)-MIL. baseline composed of multiple heads. Because UCFC, XD, and LAD exhibit low abnormal confficts, numerous similarities in abnormal categories, and minimal scene discrepancy, there are great performances with the single-head MIL bascline. However, when TAD and ST are targets, the results demonstrate that the Null(Ang)- MMIL baselines, which extract general domain features in the domain-agnostic part and avoid conflict by assigning Null values with multiple heads, outperform the MIL baseline.
Handling multi-domain with a single AuIl(Ang)-MIL baseline. The average performance of single Null-MIL and NullAng-MIL, is comparable to or even better than the average performance of individual in-domain models trained for each domain (86.52%) Moreover, the NulLAng-MILL baseline, which effectively performs inter/intra-class learning through cosine angular margin while avoiding conflict between each head, achieves superior results, with 01.82% (TAD) and 91 91.26% (ST) compared to the performances of single in-domain results of 90.75% (TAD) and 90.79% (ST),
7

---

Table 6: E2: Leave-one-out results
Table 7: E3: Low-shot adaptation resalts
MDVAD Benchmarks
MDVAD Benchmarks
Tageet
Target
UCFC
LAD
UBIF
TAD
ST
UCFC
XD
LAD
UBIT
TAD
ST
Single-domain
EJ: Low-shot Adapiation
Out Avg.
61.39
63.10
66.83
66.17
61.90
49:21
Head
AC
E2: Leare-one-owt
Head
AC
MIL
/
72.52
71.00
76.69
82.34
78.72
74.88
MIl.
V
15.98
78.49
76.87
78.67
81.81
78.39
65.66
NaIl-MIL
(Ours)
V
67.35
70.57
66.40
60.32
73.58
75.11
81.39
75.97
71.12
62.29
63.02
37.72
Noll-MIL.
62.38
59.63
64.91
55,42
66.28
45.60
NallAng
77.76
70.67
74,86
83.44
78.57
71.81
(Ours)
/
68.78
74.65
74.46
55.61
67.72
55.26
MIL(OOrs))
78.99
75.80
77.82
83.75
84.06
76.23
NuilAng
75.26
73.00
79.41
77.94
52.98
Training:
R22
target samples
MIL(Ours)
78.55
77.68
77.36
82.53
79.31
69.41
Testing: Target dataset
Training: Five datascts except the target dataset
Testing:
Target dataset
highlighting the effectiveness of learning with diverse and abundant data to enhance model generality, Furthermore, additional performance boosts are observed across all baselines with the auxiliary AC classifier to make the model conflict-aware.
4.2.2 Leave-one-out results (E2)
The E2 experiments, in Table O are conducted on the leave-one-out setting among multiple source datasets and evaluated on the unseen target dataset unlike E1, Each column represents the outcomes of models trained with the target dataset excluded. Compared with the held-in results, the evaluation results conducted without prior knowledge of abnormal boundaries for unseen target data lead performance gap between E1 and E2.
Effectiveness of multi-domain learning for unseen target domain. When comparing single-domain learning and multi-domain learning in the results of unseen target evaluation (Out Avg. in Table is evident that the performance with multiple datasets is significantly superior. Effectively addressing conflict and discrepancy issues, and learning from diverse and complex situations across multiple domains, leads to the development of a general model with better performance on unseen domains.
4.2.3 Low-shot adaptation results (E3)
In E3, we delve into low-shot learning, examining how well the model trained with multiple sources in the leave-one-out experiments (E2) adapts unseen targets. this experiment, utilized 10% of the target training set for low-shot learning (in 3-fold setting). For datasets with abnormal categories, an equal number of data per category are randomly sampled, while for uncategorized datasets like UBIF and ST, random sampling is employed.
Importance of general pre-trained baselines for adaptation. In Table 7 comparing with Table 6 notable performance improvements are observed across all baselines for target samples with severe domain conflicts and gaps, such UBIF, TAD, and ST, even with limited volume of training samples, This emphasizes the importance of building general model to adapt on unforeseen targets. While the MIL baseline with a single head holds an advantage in adapting specific single head when target samples are appropriately selected, it may become dependent on specific samples and is susceptible overfitting. On the other hand, NullAng-MIL, with conflict-aware learning and training on diverse domains, outperforms other methods in adaptation performance.
4.2.4 Full fine-tuning results (E4)
Multi-domain models superior to specific single- domain models. Table g shows the full fine-tuning results of single-domain and multi-domain trained baselines. The single-domain models reflect results from training and testing within a specific dataset. When evaluating domti- baselines trained on all datasets in the held-in (E1) setting, it demonstrates performance comparable to single-domain models that perform well by fitting to a single dataset. However, after full fine-tuning on the target dataset, the El model achieves superior performance in most cases. The El
Table 8: E4: Compurison between the single domain model and full fine-tuned models from the El and E2.
MIVAD Benchmarks
UCRC
LAD
UBIF
dingle-domain Single 17.93
83.23
92.62
9075
90.79
E::: Pull fin-ttning
E1
78.62
82.71
84.41
92.42
92.50
91.17
12
80.24
82.77
83.1
92.95
92.07
91.27
Finetuning: Tirgt
Testnng Target
8

---

Ours NIL
NIL
A
hmnn
bb)
Figure 3: (a) The plot of AC scores. Both seenes are from UCFC and normal in UCFC. (Top) Yellow box indicates abnormal conflict, which is abnormal ST. (Bottom) Normal scene. (b) Qualitative results. Red box indicates abnormal event in scene. (Top) Bicyclist on walkway abnormal event ST. (Bottom) Accident abnormal event in UCFC and Pedestrian on Road abnormal conflict in TAD.
and E2 models, which are well-explored across multiple domains, outperform the models specifically trained on a single domain after full fine-tuning.
4.2.5 Summary
Null-MIL and NullAng-MIL. The Null-MIL baseline achieves the best performance (86.86%) in the Held-in setting (E1), even surpassing the in-domain average results (86.52%). However, in the Leave- one-out setting (E2). which involves unknown target domains, it shows suboptimal performance. This is because is impossible determine which head's 's output score among the source dataset heads is more reliable, so multiple head baselines employ the maximum score for unknown domains. Conversely, NullAng-MIL, exhibits superior results in E2 due to similarity- NullAng-MIL. considers the cosine distance between the final feature and the weights of each source head, As result, when calculating the final score, the head with high similarity to the target data activated, yielding promising results. When pre-training the general VAD model without knowledge of the target, we observed that utilizing the NullAng-MILL baseline is beneficial. However, when conducting multi-domain learning with knowledge of the target, employing Null-MIL yields better results. We observed that when pre-training the general VAD model without prior knowledge the target, utilizing the NullAng-MIL bascline proves beneficial and employing Null-MIL yields better results when knowledge of the target is available during multi-domain learning.
score calculation.
Role of the AC Classifier. In multi-domain learning, the proposed AC classifier serves an auxiliary branch designed predict whether there is an abnormal conflict between domains, thus making the model aware of conflicts. Since it is not used during testing, it does not impact the final model's 's cost but provides performance gains in most experiments. Notably, in E2 and E3, training with the AC classifier shows significant boost effect, aiding in adaptation to unseen domains.
4.3 Discussions
Open-set VAD. We conducted experiments in open-set scenario using UBNormal (UBN) dataset []] The UBN is VAD benchmark proposed for open- set scenarios to handle unexpected abnormal events. Both normal and abnormal events are available during training, but the anomalies that occur during inference belong distinct set of anomaly types (categories). Unlike other VAD datasets, UBN consists of synthetic videos alleviate the difficulty of collecting abnormal event data in the real world. There are substantial ab- normal conflicts and differences in the visual settings of scenes compared to other domains. For multi-domain learning, we reorganized the training/ testing set of UBN to maintain the same number of normal and abnormal videos the training set with MDVAD, which is uniformly sampled and balanced the same amount of videos from all domains. As
Table Ablation studies Open-set VAD scenario
Single source: UBN/ Target: UBN
MIL
75.13
Multiple source: MDVAD UBN Target: UBN
MIL
67.93
MIL+AC
70.56
NuIl-MIL
72.
MIII-MIL+AC
70.94
NollAng
74.42
NullAng+AC
74.54

---

shown in Table 9 despite domain discrepancies and abnormal conflict, the model effectively handles multiple domain learning, demonstrating that general feature learning can adequately address unseen abnormal categories. Please refer to S. A5 for more details.
Comparison with WVAD models. Although this paper focuses on the analysis of multiple domain learning within the context of abnormal conflict issues, instead of exploring complex architecture designs for single-domain VAD models, we compare with various VAD mod-
Table 10: WVAD models on MDVAD Benchmark
MDVAD Benchmarks
Models
Target UBIT
UCFC
XD
LAD
TAD
ST
Avg.
EI: Held-in
MMIL
77.93
81.34
85.18
85.44
87.78
84.39
83.68
els. Table I0 presents the results of other MIIL-
ARNet
79.26
80.38
86.65
84.22
based WVAD models, MMIL [43] ARNet [46].
WSAL
76.47
78.35
75.44
86.41
85.62
82.45
80.79
COMO
80.41
82.75
86,24
85.82
90.13
89.76
85.85
WSAL 30, and COMO 7 the MDVAD task. Compared to the proposed baseline trained with the AC classifier, our method achieves the highest average AUC, particularly in datasets with severe abnormal conflicts and scene dis- crepancies, such as TAD and ST, in both the E2 and even more in the E3 settings. Various single-domain VAD models or backbones can be incorporated into the MDVAD task, show- ing direction for future generalization work. Please refer A6
Ours
77.21
82.09
83.88
91.90
91.36
91.12
86.26
E2: Leave-one-out
MMIL
76.68
74.92
67.39
82,40
67.61
61.86
71.81
ARNet
77.05
75.02
78.98
80.84
75.09
55.35 34
73.72
WSAL
76.39
73.75
76.71
79.86
77.68
55.5 54
73.36
COMO
77.07
76.64
77.43
76. 74
78.65
57.6 63
74.92
Ours
78.55
77.68
77.36
82.53
79.21
60.41
75.96
EE Law-shot Adapiation
MMIL
17.28
12.75
80,70
85.29
81.72
61.33
76.31
ARNet
75.48
72.18
79.90
81.70
79.43
68.24
16.16
WSAL
75.77
72.18
66.45
82.57
75.05
78.05
75.01
COMO
70.02
72.89
80.59
83.05
74.73
73.53
75,80
Ours
78.99
75.80
77,82
85.75
84.06
76.23
79.78
E4: Full FiRe-tuning
MMIL
80.26
82,51
86.34
89.88
90.32
89.34
86.48
ARNet
80.88
82.57
86.72
89.92
90.69
91.7
87.08
AC classifier. The AC classifier helps the model learn conflict-aware features, providing a clearer understanding of abnormalities. The proposed framework is composed of Domain-
WSAL
80.99
81.6
76.43
87.87
88.78
87.47
83.86
COMO
80.61
84.25
86.88
91.51
91.74
91.23
87.70
Ours
78.62
82.71
84.41
94.42
92.5
91.17
87.31
Agnostic Layers that learn general features across multiple domains, followed by Multiple Heads that predict abnormalities specific to each domain. When the Domain-Agnostic layers learn to perform AC classification, they capture whether the imput snippets relate to abnormal conflict or not. From the Heads' perspective, these features are separated into abnormal conflict and non-AC in the feature space, allowing the Heads to apply different criteria (decision boundaries) when distinguishing between normal and abnormal instances. For example, in classifying abnormalities, non-AC scenarios can be addressed more straightforwardly, while abnormal conflict scenarios require more careful exploration. Fig. 3(a) presents the results of the AC classifier of NullAng-MIL baseline from E1. shows abnormal conffict scores for two scenarios: (Top) A car the Sidewalk abnormal event in ST, which normal the UCFC. (Bottom) A normal situation of people shopping a grocery store. Through the multi-domain learning, the AC classifier outputs a high abnormal conflict score for the top sample, demonstrating that the model has learned to be conflict-aware.
Qualitative results. Fig. 3(b) illustrates abnormal conffict scene: (Top) Bicyclist on Walkway, abnormal in ST but normal other domains, and (Bottom) Pedestrian on Road, abnormal TAD but normal in UCFC. In these scenarios, the MIL baseline trained with multiple domains predicts (Top) false negative and (Bottom) false positive, while ours adaptively handles conflicts across different domains.
Conclusion, Limitation, and Future Works
In this paper, we propose new task called MDVAD, whose ultimate goal is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general VAD model. As a baseline, we propose Null(Ang)-ILL loss and AC classifier. These modules effectively handle abnormal conflicts between domains and show meaningful results in the MDVAD benchmark with diverse evaluation protocols.
multi-head framework with
Instead of exploring complex architecture design of single-domain VAD models, this paper focuses on resolving abnormal conflicts from multiple domains. Various single-domain VAD models or backbones can be applied into our novel framework to address the MDVAD task, representing a valuable direction for future generalization research.
10

---

Acknowledgments
This work was partly supported by the NAVER Cloud Corporation and the National Research Foundation of Korea (NRF) grant funded by the Korea
References
Andra Acsintoae, Andrei Florescu, Mariana-luliana Georgeseu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 20143-20153, 2022.
[2] Abhishek Aich, Kuan-Chuan Peng, and Amit K Roy-Chowdhury. Cross-domain video anomaly detection without target domain adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2579-2591, 2023.
[3] Congqi Cao, Lu, Peng Wang. Yanning Zhang. A new comprehensive benchmark semi- supervised video anomaly detection and anticipation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20392-20400 June 2023.
[4] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
Yunpeng Chang, Zhigang Tu, Wei Xic, and Junsong Yuan. Clustering driven deep autoencoder for vidco anomaly detection. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 329-345. Springer, 2020.
[6] AyeongAh Cho, Tacoh Kim, Wao Jin Kim, Suhwan Cha, and Sangyoun Lee. Unsupervised video anomaly detection via normalizing flows with implicit latent features. Pattern Recognition, 129:108703, 2022.
[7] MyeongAh Cho, Minjung Kim, Sangwon Hwang, Chaewon Park, Kyungjae Lee, and Sangyoun Lee, Look around for anomalies: Weakly-superrised anomaly delection via context-motion relational learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12137- 12146, 2023.
[8] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categoriza- tion and Nomain-speciffe transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109-41 2018.
[9] Bruno Degardin and Hugo Proença. Iterative weak/sdff-supervise classification framework for abnormal events detection. Pattern Recognition Letters, 145:50- 57, 2021.
[10] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas. Ben Glocker. Domain generalization via model-agnostic learning of semantic features. Advances in neural information processing systems, 32, 2019.
[11] Mark Dredze, Alex Kulesza, and Koby Crammer, Multi-domain learning by confidence-weighhed parameter combination. Machine Learning, 79:123-1499 2010.
[12] Masoud Faraki, Xiang Yu, Yi-Hsuan Tsai, Yumin Suh, and Manmohan Chandraker. Cross-domain similarity learning for face recognition in unseen domains. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition, pages 15292-15301, 2021.
[13] Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14009-14018, 2021.
[14] Mariana Iuliana Georgescu, Radu Tudor lonescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. background- agnossic framework with adversarial training for abuormal event detection in video. IEEE transactions on pattern analysis and machine intelligence, 44(9):4505
2021.
[15] Dong Gong. Lingqiao Liu, Vuong Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705- 1714, 2019.

---

[16] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. pmlr, 2015.
[17] Mahesh Joshi, Mark Dredze, William Cohen, and Carolyn Rose. domains matter? In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural I Language Learning, pages 1302-1312, 2012.
learning: when do
[18] Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki, Sparsh Garg, Manmohan Chandraker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In European Conference on Computer Vision, pages 20-36. Springer, 2022.
[19] Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evalua- tion time-series anomaly detection. Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 7194-7201, 2022.
[20] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint WrXiv:1412.660 69800, 2014.
[21] Shuo Li, Fang Liu, and Licheng Jiao. Self-training multi sequence learning with transformer for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intellignnce volume 36, pages 1395-1403, 2022.
[22] Yunsheng Li and Nuno Vasconcelos. Efficient multi-domain learning by covariance normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5424-5433, 2019.
[23] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE internutional conference computer vision, pages 2980-2988, 2017.
[24] Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection- - new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6536-6545, 2018.
[25] Wen Liu, Weixin Luo, Zhengxin Li, Peilin Zhao, Shenghua Gao, et al. Margin learning embedded prediction for video anomaly detection with few anomalies. In ICAAI, volume 3, pages 023-3, 2019.
[26] Zuhao Liu, Xiao-Ming Wu, Dian Zheng, Kun-Ye Yu Lin, and Wei-Shi Zheng, Generating anomalies for video anomaly detection with prompt-based feature mapping. In Proceedings the IEEE/CVF conference on computer vision and pattemn recognition, pages 24500-24510, 2023.
[27] Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy. and Yang Wang. Few-shot scene-adaptive anomaly detection. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part 16, pages 125-141. Springer, 2020.
[28] Haohan Luo and Feng Wang. A simulation based framework for urban traffic accident detection. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. IEEE, 2023.
[29] Hui Lv. Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Learning normal dynamics in videos with meta prototype network. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 15425 15434, 2021.
[30] Hui Lv, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yong Li. and Jian Yang. Localizing anomalies from wcakly-labeled videos. IEFE transactions on image processing, 30:4505-45155 2021.
[31] Zhen Ma, José JM Machado, and João Manuel RS Tavares. Weakly supervised video anomaly detection based on 3d convolution and Istm, Sensors, (1(2))75088 2021.
[32] Hyconscob Nam and Bohyung Han. Learning multi- convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4293 4302, 2016.
[33] Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance-motion correspondence. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1273-1283, 2019.
[34] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
12

---

[35] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[36]
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[37] Sylvestre- Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of nomai-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 81 19- 8127, 2018.
[38] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271), pages 59-66. TEEE, 1998.
[39] Alice Schoenauer-Sebag. Louise Heinrich, Mare Schoenauer, Michele Sebag, Lani F Wu, and Steve J Altschuler. Multi-domain adversarial learning. In International Conference on Learning Representations, 2018
[40] Sconguk Sco, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXII 16, pages 68-83. Springer, 2020.
[41] Ankit Parag Shah, Jcan-Bapstine Lamare, Tuan Nguyen-Anh, and Alexander Hauptmann. Cadp: A novel dataset for ectv traffic camera based accident analysis. In 2018 1.5th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1-9, IEEE, 2018
[42] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
[43] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 6479-6488, 2018.
[44] Shengyang Sun and Xiaojin Gong. Hierarchical semantic contrast for scene-aware video anomaly detection. In Proceedings of the IREF/CVP Conference on Computer Vision and Pattern Recognition, pages 22846- 22856, 2023.
[45] Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Careeir. Wieakyyssupervise video anomaly detection with robust temporal feature magnitude learning. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 4975 4986. 2021.
[46] Boyang Wan, Yuming Fang, Xue Xia, and Jiajie Mei. Weakly supervised video anomaly detection via center-guided discriminative learning. In 2020 IEEE international conference on multimedia and expo (ICME), pages 1-6, IEEE, 2020.
[47] Boyang Wan, Wenhui Jiang, Yuming Fang. Zhiyuan Luo, and Guanqun Ding. Anomaly detection in video sequences: A benchmark and computational model. IET Image Processing, 15(14)3454-3465, 2021.
[48] Li Wang, Dong Li, Han Liu, Jinzhang Peng, Lu Tian, and Yi Shan. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2487-2494, 2022.
[49] Xudong Wang. Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 7298. 2019.
[50] Peng Wu and Jing Liu. Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Transactions on Image Processing, 30:3513-3527, 2021.
[51] Peng Wa, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer Vision-ECCo ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XXX 16, pages 322-339. Springer, 2020.
[52] Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. A light enn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security. 13(11):2884 2896, 2018.
13

---

[53] Shuhan Yi, Zheyi Fan, and Di Wu. Batch feature standardization network with triplet loss for weakly- supervised video anomaly detection. Image and Vision Computing. 120:104397, 2022.
[54] Tianhe Yu, Saurahh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-555 5836, 2020.
[55] Chen Zhang, Guorong Li, Yuankai Qi, Shuhui Wang, Laiyun Qing, Qingming Huang, and Ming-Hsuan Yang. Exploiting completeness and uncertainty of pseudo labels for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16271-16280, 2023.
[56] Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 178-193. Springer, 2020.
[57] Jia Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H Li, and Ge Li. Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1237-1246, 2019.
[58] Yi Zhu and Shawn Newsam. Motion-aware feature for improved video anomaly detection. arXiv preprint arXiv: 1990.0002211 2019.
[59] Yuansheng Zhu, Wentao Bao, and Qi Yu. Towards open video anomaly detection. In European Conference on Computer Vision, pages 395-412. Springer, 2022.
14

---

Supplementary Material
Table A1: Detailed descriptions of VAD datasets used in the paper. N: The number of normal videos. A: The number of abnormal videos.
Videos
Frumes
Anomalies (Calggrcaas)
Settings
Sample
UCRC
1900 (188)) N: 930/ A. 950
7287 per video
N: 8007 A. 810
Tor:: 1610
Tes:: 2M0 N: A. 140
Abroar Ance, Arunn Avenilt Actident, Burpbwry
Explision, Fighting, Rubbery, Stawting Sazding,
Suppilling Vandklion
OCTV
XO
4756 (2178)
23407 2405
N:
Tou:: 2954
Tex:: 800 303000
Abuse, Cur Accident, Explosior, Fighting. Rirt, Shouting
CCTV Sports, Gamen Mosio, News
Croud Destroy, Depp, Filling,
180
Highting free, Lottenng, Theeing. Trappied
UBIF
N:216/ 754
1000
Trin: 933 7577 A. 176
A7:40
Fights
CCTV, Mobile
TAD
N: 250/ A. 250
500 (25kg
1075 video
Thin: 400 N: 2107.4 190
Tea:: 100 N: 4./0//A 6.
Vehicle Accidents Bllegal Turns, lllegal Occuputions Retingnade Morion, Polestion Road, Road Spills, The Else
Traffi CCTV
ST
N: 1303 100
497
776 per video
Trin: 208 N: A: 63
1SS//: A::44
Ties 199
Fightngg Raebery. Chasing, Jumping. Throwing.
Tarwing Ruining, Despiing, Motcrcycle. Skatcbound Ca,, Gan, Lottring
Campus CCTV
CAPP
N: 0: 1416
1416
Trun: 1418 D 1416
Traflic accislent
Traffic
Group cumilist, Trick, Climking farzz: can.
1:12
305 N:
Text:
Cycling on torguth. Dogs, Chasirg: Loiterng crossing square,
Campus
U-inn, Falling,
Summary The supplementary material is organized in the following order. First, it provides explanation of the characteristics of the Video Anomaly Detection (VAD) dataset and introduces the detailed configuration and experimental settings of the MDVAD benchmark. Following this, we present implementation details of the experiments and examples and explanations related to abnormal conflict. Lastly, we discuss the pseudo labeling equation, score comparison plots, and failure cases. Note that back-references in the supplementary material sections and tables are from the main manuscript.
Terminology
Domain/ Dataset: In this paper, 'domain' refers to the broader context environment, while 'dataset' refers to the specific collection of data used within that domain.
Single-domain: Training and evaluation data are the same, as in traditional VAD research. Multi-domain Learning/ General VAD: General VAD is the objective of detecting anoma- lies across various domains, while Multi-domain Learning is the method used achieve this by training on multiple domains.
Multi-head Learning: The baseline method proposed in this paper for Multi-domain Learning.
Multi-task Learning:
Performing multiple tasks with single model, related to E1 by integrating various domain's 's anomalies into one framework.
General Pre-training: A methodology of pre-training large data without target knowl- edge, related to E2 by training in a held-out setting and applying to an unseen target. Furthermore, general pre-trained models adapt well to low-shot learning in E3 and fine- tuning in E4.
A VAD Datasets
As mentioned in $2, the VAD task encompasses diverse datasets, as delineated in Table A1
UCFC 43), the prominent large-scale dataset for the Weakly-Supervised approach, consists of untrimmed surveillance videos. It categorizes anomalies into 13 classes related to public safety,
15

---

Compus Flle
Abona 1.5%
16.4%
Road Spitls
Aeddontt 12.5%
XD: Normal
1.2%
Pod. en Read
ST: Abnormal
1.5%
Traffic Rbe
Explesios 5.8%
3.5%
Illegal Oceepy
(h) Normal video from XD)
1.5%
Tranpled
1.3%
XD: Normel
Fightng
Absormal
Rabbery
(a)
(c) 'icycclist' abnormal video from ST
Figure A4: (a) Distribution of the training videos by abnormal category in the MDVAD benchmark. (b)(c) Examples of abnormal conflict between VAD datasets. The scenes colored by red and blue borders represent abnormal and normal situations, respectively, based on the labels from the corresponding dataset.
ranging from crimes such as abuse and arson to more complicated scenarios like stealing and shoplifting. The dataset consists of a total of 950 normal and abnormal videos each, with the training set maintaining a balanced distribution of 800 and 810 samples for normal and abnormal videos, respectively.
The XD dataset [51] stands out as the most extensive dataset with a variety of environments, including CCTV, movies, hand-he hand-held cameras, and car camera settings. It incorporates six categories of violence, such as abuse, riot, and explosion, defining anomalies. Within the dataset of 4,754 videos, there are 2,049 normal and 1,905 abnormal training videos.
The LAD dataset [7 defined by a detailed categorization of 14 anomaly classes, including crash, fire, and violence, consisting of 2,000 videos, with 958 normal and 482 abnormal videos.
The UBIF dataset is composed of various fighting-related anomalies extracted from our daily life videos. This dataset consists of 1,000 videos without a specific categorization for the fighting class, and the training consists of 757 normal and 176 abnormal videos, leading class imbalance issue.
The TAD dataset [19] is structured with traffic surveillance and Ist-person videos, encompassing anomalies related to traffic, such as accidents, illegal turns, and road spills. This dataset, totaling 500 videos, has a small volume with 210 and 190 numbers normal and abnormal training videos.
The ST 24 serves campus surveillance dataset, capturing anomalies occurring pedestrian walkways. exhibits heightened sensitivity abnormal events, ranging from fighting and throwing to running and jumping. With only 175 normal and 63 abnormal videos in the training set out of total of 437 videos, there is a notable imbalance between classes, and the quantity for learning. Abnormal categories in the training set cover all anomalies in the testing set on each dataset except for ST (ST remains indeterminate because the absence of category information), which means evaluations are exclusively conducted for seen abnormal events.
data is insufficient
Examining the dataset samples in Table AT reveals that TAD and ST exhibit visual gap compared to the other datasets, UCFC, XD, and LÄD. Such gap is attributed to visual elements such as scene settings and conditions that impact the visual characteristies of scenes, which is the scene discrepancy discussed in $2. While LAD (some videos) and ST are recorded in square campus settings to establish real-world scenario datasets, the remaining datasets are constructed online, such as YouTube and LiveLeak using text search queries. These datasets also exhibit significant domain distance, as such, the criteria for defining abnormalities and the visual characteristies of
indicated in Table scenes vary across different datasets.
B
MDVAD Benchmarks
B.1
Configuration
The five datasets introduced in Table AT have different training set volumes, leading challenges in handling multiple dataset learning, where larger datasets may overwhelm the learning process.
16

---

For instance, there is significant disparity in training set volumes, with TAD (400 videos), ST (238 videos), and XD (3954 videos) differing by almost 10 times. Additionally, the number of abnormal videos in XD differs roughly 10 and 30 times compared to TAD and ST, respectively.
The objective of the MDVAD task is to avoid abnormal conftict and build general model through balanced learning across diverse situations and anomalies. To control for other variables, we uniformly sampled and balanced the same amount of videos from all domains, equal to the number of videos in the smallest domain. Since TAD and ST have an insufficient number of videos for training, we recombine them with two new datasets, CADP 41 and NWPU [3], respectively, to balance the numbers. CADP is a dataset for traffic accident analysis, and it shares similarities with TAD in terms of setting and abnormal definition. Hence, following the approach in (28), we compose TAD's training set by adding abnormal videos from CADP. Additionally, incorporating the recently released NWPU dataset, which collects various anomalies of pedestrians on campus, we augment ST's dataset with NWPU data. For both datasets, we conducted random sampling for normal and abnormal videos, considering the smallest sample size, resulting in a number of 210 normal and 176 abnormal videos. In this process, for datasets with information on abnormal classes, we compose to achieve uniform number of instances per abnormal category on each dataset. For datasets with a sufficient amount, we organize 3-fold sets to minimize overlap. All reported performance metrics are the average results of evaluations conducted on the target dataset after training on each fold. For example, in the E2 leave-one-or setting, the average results are derived from models trained on each fold and evaluated on an E3 low-shot adaptation set for the corresponding fold.
The MDVAD benchmark incorporates the datasets introduced in Table A1 including abnormal categories: Abuse, Arrest, Arson, Assault, Accident, Burglary, Explosion, Fighting, Robbery, Shooting, Stealing. Shoplifting, Vandalism, Drop, Hurt, Fall Into Water, Falling, Destroy, Fire, Violence, Crowd, Thiefing, Panic, Loitering, Trampled, Illegal Turns, Illegal Occupations, Retrograde Motion, Traffic Else, Pedestrian Road, Road Spills, and Campus Else. The circular chart in Fig. AA((a) illustrates the distribution of categories. For datasets without abnormal categories like ST (named as Campus Else), TAD (where additional categories are labeled as Truffic Else). and UBIF (where categorization is not specified and all fall under Fighting), we provide appropriate category names. The averages across the 3-folds are calculated, and in the case of XD with multiple labels per video, we count the number of categories per video. The Fighting predominates in the majority of datasets, occupying the largest portion, followed by categories like Accident, Explosion, and others.
B.2 Experimental settings
VAD is predominantly evaluated using the Area Under the Curve (AUC) score based on abnormal scores, focusing on the trade-off between sensitivity and specificity, emphasizing the false positive rate (FPR). While XD employs the Average Precision (AP) metric to balance precision and recall and concentrate on positive samples for direct comparison in cross-dataset and held-in/out evaluations, and emphasizing False Positive Rate (FPR) (detailed explanation is in Section D) we utilize the AUC score across all evaluations.
The MDVAD benchmark comprises four protocols: Held-in (EI) involves multiple-source learning with all datasets, Leave-one-out (E2) employs a leave-one-out approach where a dataset left out during training is used for evaluation, low-shot adapation learning (E3) is training on the remaining datasets, and Full fine-tuning (E4) that fine-tunes multi-domain models on single-domain datasets. In the E3 setting, around 10% of the training set, 20 videos, is randomly sampled for low-shot examples. For datasets with anomaly category information, low-shot examples are selected uniformly to ensure an even distribution within each category. To mitigate the impact of randomness, the low-shot setting is also conducted with 3-fold cross- voss-valid,tion, obtaining low-shot examples from each fold. In all cases, including MIL, Null-MIL, and NullAng-MIL models, all weights except for the pretrained backbone are updated. Additionally, for abnormal videos, the score from the AC classifier is multiplied to the loss value.
C Implementation Details
All experiments are conducted on the MDVAD benchmark. We use 13D backbone feature which is
pretrained on Kinetics dataset 4 with C = 2048 dimension of RGB features. During the training
phase, we conduct consistent experiments with a batch size of 32, utilizing the Adam optimizer [0]
17

---

with learning rate of 5e 5 and weight decay of 5e - 4. The input image size is set to 224 x 224
and following [21 3T 45 7 we perform 10-crop augmentation when extracting backbone features. The backbone is used the same as RTFM [45] and CoMo 77. For input data, 16 frames are stacked for a snippet, and 7 snippets are uniformly sampled in the video during training. For testing, all snippets pass through the model, and the output score is assigned to all frames within the snippet. For MIL learning, the Top-K K snippets, where K equals 10% of the total, i.e. K = 3, are utilized. In the case of a single-headed MIL model in Table 5 (EI) setting, since pseudo-labels cannot be generated through multiple heads, the AC classifier is trained with label that و = for a score sh, within the (0.4.00 0.7] range. When computing EMD metric in $2, we use a pretrained I3D backbone model for extract embedding features of each dataset and measure the distance between feature vectors. We use test set with frame-level annotations for abnormalities. All models and experiments are implemented and evaluated end-to-end using PyTorch (34 with a single NVIDIA V100 GPU.
In -uaeey-suer VAD, maintaining class balance pivotal, so the number of normal and abnormal input videos within a batch is kept equal for the MIL-bbaed learning. Similarly, during multiple source learning, an equal number of input videos from each source domain within a batch is crucial. Despite each head is trained independently, the final feature FA is extracted from shared domainnagnostic part, necessitating the balance in the number of source domains. Moreover, when training the AC classifier, in order to address the imbalance between the number of abnormal conflict snippets and all normal/abnormal snippets, we apply focal loss [23] to Ac- During the testing phase, the AC classifier is eliminated, and the output is calculated for each input snippet v; as the final Abnormal Score,,
Table A4 provides information about the layers of the baseline model. Convid, fc, Max, BN, and DO represent ID convolutional layer, fully connected layer, element wise max activation, 1D batch normalization [16), and dropout [22, respectively, with a dropout probability set to = 0.7. In Convid(c, k. s) and fc(c), e, k, and denote the channel size, kernel size, and stride size, respectively. T input snippets pass through the backbone, resulting in backbone feature B with shape of (N, C). where N represents the batch size. The feature aggregation layer undergoes channel doubling followed by channel squeezing with element-wis max operation, resulting in an output feature F Fage with a shape of (N,T,C) C.). Subsequently, the temporal aggregation layer outputs FDA with shape of (N, T, C/2). These embeddings are then input into the AC classifier and fc layer. AC classifier, composed of consecutive fc layers, outputs AC scores sAC for each snippet with a shape of (N, T, 1). Additionally, the final feature F, processed through fc layer, is input into multiple domain heads WD, for NullAng-MIL. learning. During this process, both the feature and weights are normalized with respect to [|] and ||D,, ll for each domain.
D Abnormal Conflict
In Fig. A4 (a) represents a video labeled as normal in UCFC, while (c) is labeled as abnormal in TAD, both depicting a situation where people are present on the road. Similarly, (b) is labeled as a normal video in XD, whereas (d) is considered abnormal in ST; both scenes show bicycling on a
Table A2: Abnormal Conflict: Average Relative FPR Relative
Table Illustration of each layer proposed model.
FNR.
Part
Name, Notation
Layers
Oulpui
Target
Rackbone
130
(N,7.0)
Source UCK.
UCFC
0.204
XD
0.099
LAD
0.153
UBI
0.195
TAD
0.273
ST
Aggregation
Peciture
F
Coarid 2C,
Naa-Rel.I1.6
(N.7.C)
LAD
XD>
0.186
0.315
0.152
0.082
0.234
0.275
0.166
0.175
0.337
0.326
Damain
Apgresation
Temporal
Fba
Coavid[C)2. 1)
-Rel J--BM
(N,7,0/3
UBIF
0.074
0.237
0.119
0.281
0.150
0.265
0.266
0.258
0.236
0.286
0.343
Agnosic
AC Classitier
(Tain--hnly)
40
461512) NGLU
fe(1)-S1gmodd
(N,7.1)
layer
121128 3610
12.1.125)
Table A3: Ablation of Eq. 6 in manuscript.
Domain Specific
Mulippee
Heads
#c(D)
(N.
Target
UCFC
XD
LAD
UBIF
TAD
ST
Avg.
E1: Held-in results
Fixed
76.30
81.35
82.40
91.56
91.39
91.02
85.82
Std.
77.04
82.28
83.04
91.71
91.61
90.75
86.07
Difl.
77.21
82.09
83,88
91.90
91.36
91.12
86.26
18

---

walkway. We define such instances, where the criteria for abnormalities differ across datasets, as abnormal conflicts.
While cross-domain evaluation in Table 2 reveals a domain gap, it is difficult to attribute as abnormal conflicts directly. In Table A2 we calculate the average of the relative False Positive Rate (FPR) and relative False Negative Rate (FNR) to quantify the abnormal conflict between domains.
(F )) + (FNR.,, FNR
JC
(8)
2
where and j indicate source and target domain, repectively. First, FPR and FNR refer to the rate of misclassifying abnormal normal and normal as abnormal in the scenario of learning from source and testing on a target, respectively. However, since these include the FPR and FNR caused by the incompleteness of the model itself, this cannot accurately reflect the abnormal conflict between the source and target dataset. Therefore, we subtract the FPR and FNR when the source domain is the same as the target domain (F P and FN R,3)) from the cross-domain FPR and FNR to calculate the relative FPR and FNR caused by the domain shift. Table[A2]she (alleAAsswwws shows similar trends compared to Table 2, where values are not normalized. This shows that abnormal conflicts between domains are relevant to the reduction in generalization ability that can occur when transferring between domains.
E Related Works
As mentioned, proposing new VAD model is beyond the scope of this paper, this part is organized to aid understanding.
Unsupervised VAD Defining anomalies precisely is challenging as they can vary based on criteria and sensitivity, and it is impossible to categorize and collect datasets for every scenario. For this reason, Unsupervised VAD methods [33 5
24 15 have been developed to learn normal patterns using only normal training data and then detect anomalies as scenes with low normality during the test phase. However, this approach leads to a significant bias towards the normal samples, causing the detector to misclassify normal patterns that differ from the learned data as abnormal events, resulting in high false alarm rate. To address this issue, Weakly-supervise
VAD (WVAD) approaches [55 44 43 50 21 have been introduced to differentiate between normal and abnormal events with minimal supervision using video-level annotations, achieving significant performance improvements while incurring lower labeling costs compared to frame-level annotations.
Weakly-superviset VAD MMIL 43 is the first WVAD method pose the MIL ranking approach as a regression problem that achieves significant performance gains. Various learning methods, including magnitude feature learning 45], self-training approaches through pseudo labels [33 21), and distance learning [53] have been explored alongside traditional MIL methods. Furthermore, subsequent approaches have focused on capturing features by focusing temporal or motion informa- tion [58), leveraging additional audio signals [5T), learning relations between temporal [50] or motion and context information 77. These methods differentiate complex anomalies through contextual information; for example, the same motion may be classified as normal or abnormal depending on the surrounding environment. PFMP (26) proposed a method to utilize virtual data anomalies to reduce scene discrepancy with real-world data, addressing the issue of data scarcity in VAD. However, these approaches still cannot address abnormal conflicts, where the same situation is labeled differently as abnormal or normal based on the criteria of different datasets.
Generalization for VAD few studies have been conducted to generalize VAD. Domain adaptation refers to evaluating new domain that was not seen during training, and some works have proposed using some data from the target domain (2 29) or not using all (2). We also experiment multiple domains compared to existing
with domain adaptation, but the difference is that studies that only use a single domain during training. Open-set recognition aims to work well even in untrained classes (abnormal categories in VAD). Compared to our research involving multiple domains, Open-set VAD (59 focuses on learning and evaluation within a single domain.
learning Multi-Domain Learning (MDL) refers to a method for learning datasets
from multiple domains with various distributions together. MDL originated from natural language
processing [T [7 and has been applied to various tasks in computer vision 36 37 22 32 49
19

---

mmma
AC WCEEM
AC scon
(c)
(d)
Figure A5: The plot illustrates the abnormal conflict scores from the AC Classifier normal videos on UCFC. For clarity, all scores have been normalized using the minimum and maximum values of the entire score distribution produced by the AC Classifier.
48 56 18
The difficulty of MDL arises when the difference in distribution between domains is significant, and learning domain-invarian features is the basic approach (39]. In the fields of image classification [36 37 [22,, object tracking [32], detection [49], and segmentation [48], methods for learning multiple domains with multiple heads have been proposed. After that, methods have been proposed to have a unified label space [56] 18 but in this process, the problem that the labels different for each domain arises. T8 solved the label conflict problem by proposing class- independent loss using the Null class strategy. On the other hand, VAD has only two classes, normal and abnormal, so there is explicit label conftict problem, but there is an implicit abnormal conflict problem where the definition of a label different for each domain.
A similar method is multi-task learning [54], which considers differences between tasks rather than domains. addition, multi- multi-source domain adaptation (35] utilizes data from the target domain, and domain generalization [10 40 [12] requires adaptation an unseen target domain. The difference with these tasks is that they basically share the label space between the source and target domains, while MDL for VAD has different label definitions.
F
Discussions
F1
Pseudo label of AC Classifier
The purposes of the AC Classifier is the facilitation of feature learning with consideration of the discrepancies across multiple domains. cases where snippet is entirely normal or abnormal across all datasets, the AC score is low, making it difficult to directly use it for the abnormal score. Noisy samples can indeed have a negative impact on training in Eq. 6. However, in cases where even one of the multiple datasets has different definition of normal and abnormal, it is crucial to sensitively detect conflicts in such samples, leading us to decide that using a difference is preferable. In Table A3 we conducted ablation studies with Std. (7 0.1) and a fixed value (3 I where 0.3 D 0.7)
which revealed no significant differences between the methods.
F.2 Abnormal conflict score
Fig. A5 depicts the output abnormal conflict scores from the AC classifier of the E1 held-in model. While all events are considered normal in UCFC, (a) and (b) correspond to the Pedestrian on Road abnormal category in TAD (as shown in Fig. ((c)). Therefore, unlike (c) and (d), the normal situations of people shopping in a grocery store, (a) and (b) exhibit elevated abnormal conflict scores.
F.3 Abnormal score
Fig. A6 compares the abnormal scores of our model, trained with NullAng-MIL + AC classifier, and MII. model trained with MIL. loss on single head in the E2 multiple dataset setting. scenario (a), before the explosion, oil truck passes, causing the MIL model to generate high abnormal
20

---

100
1100
(a) Explosion' abnormal test vidco on UCFC
Ours MLL
3.
(b) 'Arrest' abnormal test video on UCFC
CuP'
ML
32
(c) Running person on walkway' abnormal test video on ST
(d) Carr on walkway' abnormal test video on ST
Figure A6: The plot abnormal scores with blue and magenta lines representing our model and the MIL bascline model (EI), respectively. The red region indicates the time when abnormal events occurred. The scenes are from input videos whose borders are colored red and blue for normal and abnormal scenes, respectively. For clarity, the scores from each model have been normalized.
score false alarm, while our model accurately predicts only the explosion abnormal situation. In scenario (b), where a person loitering the office, the MIL model outputs a high abnormal score, in contrast, model that trained with multiple heads to avoid domain conflicts categorizes only the arrest defined UCFC as an abnormal event.
Scenarios (c) and (d) involve abnormal videos in the ST dataset where running person or car appears on the walkway. Because these scenarios are classified as normal scenes in other datasets during training, the MII. model considers them as normal. However, our model, recognizing them as abnormal scenes corresponding to ST, outputs high abnormal scores. Thus, when learning from multiple datasets, it is crucial to avoid conflicts by simultaneously learning domain-agnostic features with the AC classifier and predicting domain-specifie scores using multiple heads according to each dataset's is criteria.
F.4 Multi-domain learning on Virtual dataset
The UBNormal (UBN) dataset is VAD benchmark proposed for open-set scenarios to handle unexpected abnormal events, where the abnormal categories the train set and test set do not over- lap. Unlike other VAD datasets which were collected from the real world, UBN consists of synthetic videos which Ieads
Table AS: Multi-souree learning MDVAD with UBN
Source
Target UBIF
MDVAD+UBN
UCFC
XD
LAD
TAD
ST
AVG.
MIL.
78.59
81.79
83.06
86.6
88.83
87.99
84.81
MIL-AC
78.14
81.78
84.92
85.08
90.77
89.12
84.96
NuIMML
78.74
83.34
85.93
91.28
90.05
88.69
86.34
NulMMLAAC
78.76
83.19
86.01
92.63
90.73
90.54
86.98
NollAngMIL
77.09
81.01
83.96
92.88
91.57
90.04
86.09
NullAngMHLLAAC
77.66
83.01
92.55
91.33
91.07
21

---

Guss MIL
34
183
2000
(a) 'Arrest' abnormal test vidco on UCFC
OMP
MLL
(b) Running person on walkway' abnormal test video on ST
MIL
32
(c) *Skatehoarder on walkway' abnormal test vidco on ST
Figure A7: The abnormal scores of our model and the MIL model, where (a) corresponds False Positive results, and (b)(c) represent False Negative results.
abnormal conflicts and differences in the visual settings of scenes. comprises 543 videos across 29 scenes with 22 types of anomaly categories. Same as MDVAD benchmark, we reorganize Training set with 210 number of normal videos and 176 number of abnormal videos where training anomaly categories are falling, dancing, walking injured, running injured, crawling and stumbling walk and testing anomalies are running, having a seizure, laying down, shuffling, walking drunk, people and car accident, car crash, jumping, fire, smoke, jaywalking and driving outside lane.
The Table A5 shows the performance in each target domain when trained with MDVAD and UBNor- mal under the E1: held-in setting. In the single-head MIL baseline, there is performance drop when trained with MDVAD+UBN, indicating difficulty in handling AC. However, the model trained with multiple heads and the AC classifier shows improved results. By leveraging the virtual dataset, we can overcome data limitations and create general model capable of handling diverse and complex scenes.
F.5 Baseline models
We conducted additional experiments ing the WSAL (30 model as base- line validate proposed method the MDVAD benchmark's four protocols. The Table presents the results, where all settings are consistent with the other experiments in the paper. The results show the addition of multi-head learning with NullAng-Mil and the AC Classifier to the WSAL model brings performance gains which effectively operate across different baselines. Notably, in the E3 and E4, which shows the target adaptation of the pre-trained general model, the method yielded competitive results. Future studies can be delve deeper into studying more sophisticated basclines for resolving AC within the MDVAD setting.
Table A6: Results of MDVAD with different baseline
Targel
Exp-
Models
UCFC
UBIF
ST
EI
WSAL
7647
18.55
75.44
86.41
85.62
82.44
80.79
81.89
86.27
E2
WSAL
76.39
73.75
76.71
79.36
77.68
55.54
73.36
76.67
72.41
WSAL
75.77
72.18
66.45
82.57
75.05
78.05
75.01
E3
+Ours
74.57
WSAL.
80.99
81.6
76.43
87.87
88.78
87.47
83.86
+Ours
90.61
84.38
22

---

F.6 Failure cases
Fig. (7)a) presents an abnormal test video from UCFC corresponding Arrest anomalies. However, this video includes scenes of Stealing, leading to the subsequent arrest and physical fighting. Both the MIL and our models output high abnormal scores for scenes other than the Arrest, as they correspond to Stealing and Fighting abnormal categories in UCFC. Models predict accurately in categories other than Arrest, this failure is observed because of single abnormal label for each video.
In Fig. A7(b) an abnormal event from ST is considered normal by both models, as other datasets generally classified normal. Similarly, in Fig. A7(c)] the skateboarder is deemed normal,
resulting in a false negative failure case. However, these situations are challenging to identify as severe abnormalities from general perspective. This failure case is able be addressed through low-shot learning or fine-tuning based on ST's criteria.
E.7 Broader Impacts
The development of Multi-Domain Learning for Generalizable VAD enhances the reliability and accuracy of security and surveillance systems across diverse environments, thereby improving public safety. This research advances Al and machine learning by addressing domain adaptation challenges, promoting robust and generalizable systems. Additionally, it underscores the importance of ethical considerations and privacy, ensuring that advanced surveillance technologies are deployed responsibly and transparently.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Abstract, $ and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In and F.6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
In and
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
25

---

Justification:
We will release the data and code when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptaips.cc/ubblic/guideesss /oodeSemmideSSmbim ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
and Table A4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: We do not reprot the error bars but all reported experimental results are average 3-fold setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification:
We have complied with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In F7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "JlL1NjZd91", "reviewer": "Reviewer_2cHp", "summary": "This paper proposes a new task called MDVAD, the goal of which is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general VAD model. To achieve this, the authors expand the traditional single-head framework to multiple-head framework for learning different knowledge and design an AC classifier to handle abnormal conflicts. The experimental results prove the effectiveness of the proposed method.", "questions": "1. (referring to weakness 1) In practical applications, what kind of scenarios conform to the task settings of MDVAD proposed by the authors?\n2. (referring to weakness 3) During test phase, is it necessary to know which dataset the sample comes from? If a certain anomalous event is found to have conflicts in different datasets, how should it be handled? Do the multi-dataset evaluation and the test procedures for other methods use the same test data?", "limitations": "I do not recognize obvious potential negative societal impact of this work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on the problem of learning a generalizable VAD model, which is an important task.\n2. The experiments conducted by the author are relatively compr", "weaknesses": "1. This paper proposes a new task called MDVAD to achieve generalizable VAD by resolving conflicts in anomaly definitions. However, for any VAD application, the definition of normal or abnormal events should be explicitly determined according to the scenario requirements, rather than simply combining multiple datasets and resolving the abnormal conflicts. I find it difficult to understand under what practical scenario a VAD model trained using multiple datasets with abnormal conflicts is needed.\n2. The writing of this paper is not clear enough, where some necessary training and inference details are missed. For example, the normal head training mentioned in NullAng-MIL is confusing.\n3. This paper lacks a detailed description of the experimental setup. For example, if an anomalous event is determined to be a conflict, how should the model handle such an event?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1, Q1] Practical scenarios of MDVAD\n\n**MDVAD’s practical relevance**\nIn real-world scenarios, performance degradation due to domain shift is a persistent issue for deep learning models. Consequently, various tasks have seen the introduction of domain adaptation and generalization methods. As reported in cross-domain evaluation performance studies [6, 7, 24], VAD models experience more than a 20~60% drop in AUC, largely due to the criteria for abnormality defined by each dataset. Models that perform well only within a single domain are significantly limited in their practical applicability.\nAdditionally, collecting data for every possible scenario in real-world applications is challenging, and abnormal events occur rarely, resulting in data scarcity. To address these challenges, our approach leverages multiple VAD datasets and aims to learn a general model that effectively mitigates the ambiguity in defining abnormalities across datasets.\n\n**Utilizing a generalizable VAD model**\nA general VAD model trained on multiple domains offers two key benefits:\n\n* First, when data from the target domain in the real world is available (E1), a general model trained on multiple source datasets, including the target domain, is able to explore robust and general features, similar to the effects of multi-task learning. Additionally, a single generalized model eliminates the need for multiple specific models for different domains.\n* Second, when target domain training data is not available (E2), proper pre-training on multiple domains allows the general VAD model to embody generalized representations, leading it to better performance in unseen target domains. Furthermore, when a real-world target dataset is provided (E3:, E4), the pre-trained general VAD model can adapt well to the new domain, which is highly beneficial for practical VAD applications.\nWe are grateful for the reviewer’s insightful comment and will incorporate this discussion on this aspect in Section 4.3 of our paper.\n\n# [W2] Training/Inference details of normal head in NullAng-MIL\n\nNullAng-MIL uses an angular margin to learn normal and abnormal features effectively. It adjusts the feature vector distances in the angular space both inter- and intra-class within each domain. \n\nRegarding the training of the normal head, when input training data is from domain $j$, the normal score$s^n $ is output from $j^{th}$ Normal head ${W}^n_{D_j}$ and  abnormal score $s^a $ is output from ${W}^a_{D_j}$.\n\n${s}^n_{D_j}=F \\cdot {W}^n_{D_j}=\\left\\| F \\right\\|\\left\\| {W}^n_{D_j} \\right\\|cos{\\Theta }^n_{D_j}$\n\nIn the equation, $F$ is the final embedding feature. By normalizing both the feature and the head, score $s$ is the angle in the angular space between the two vectors. For simplicity, the snippet index $i$ is omitted. When $F$ is the feature of a normal snippet, the angle with the normal head${W}^n_{D_j}$ should be narrower than the angle with the abnormal head. Conversely, when $F$  is from an abnormal snippet, the opposite should hold true.\n\n$\\begin{cases}cos{\\Theta }^a_{D_j} + m > cos{\\Theta }^n_{D_j} & \\textrm{when abnormal snippet}\\\\\\cos{\\Theta }^a_{D_j} < cos{\\Theta }^n_{D_j}+m & \\textrm{when normal snippet}\\end{cases} $\n\nTherefore, by adding the angular margin $m$, the feature learning process is designed to satisfy the above conditions. When training the normal head, other normal heads $\\\\{{W}^n_{D_d}\\\\}$ where $d \\neq j$ do not affect the gradient, which is calculated as $\\frac{ \\partial  s^n}{ \\partial {W}^n_{D_j}}$. \n\n# [W3, Q2] Missing detail description\n\n**Experimental setup**\nThe four protocols to verify the MDVAD task are as follows:\n\n* **E1 Held-in**: Integrating various domain’s anomalies into one framework, performing similarly to multi-task learning that handles multiple tasks with a single model. All source and target domains are accessed. A single model trained on $M$ (M=6) domains is evaluated on $M$ target domains.\n* **E2 Leave-one-out**: A methodology of pre-training on large data without target knowledge. The source and target are different, and the target is not accessed. Training with $M-1$ source domains in a held-out setting and applying to an unseen target dataset.\n* **E3 Low-shot adaptation**: Evaluates the adaptation ability of the general pre-trained model (E2). When few samples (10% of the training set) from the target domain are given, low-shot learning is performed, and the model is evaluated on the target domain.\n* **E4 Full fine-tuning**: Evaluates the performance of the general pre-trained model (E2) after full finetuning on the target domain’s training set.\n\n**Handling abnormal conflict**\nWhen performing multiple domain learning, inconsistent labels across domains can lead to abnormal conflicts, causing confusion in the model's training and making it difficult to develop a robust general model. Our proposed framework consists of domain-agnostic layers and multiple heads for  different domains. While the heads are divided to prevent conflicts, the agnostic part extracts features from all datasets using a single branch. Therefore, to explore general features while being aware of abnormal conflicts, an auxiliary branch, the AC classifier, predicts conflicts by leveraging the variance in abnormal scores across multiple heads. \n\n**Testing phase** During the testing phase, target domain information is unnecessary in the E2~E4 settings because the source domain of the pre-trained general model is different from the target domain, it does not have a head for the target domain. As in Eq. 5, under the E1 setting, since we have information about the target domain, the target branch score is used as the final score. As in Eq. 5, we determine the final score with both normal and abnormal score to reflect conflicts by taking the maximum normal and abnormal score from multiple heads. All empirical studies of the MDVAD and the comparison models were evaluated and reported on the same source and target domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQCqVJbfu2", "reviewer": "Reviewer_SwSy", "summary": "The manuscript addresses the limitations of existing Video Anomaly Detection (VAD) models that are confined to single-domain learning. The primary contribution of the paper is the introduction of a new task called Multi-Domain Learning for VAD (MDVAD), which aims to develop a general model capable of identifying abnormal events across multiple domains. The manuscript conducts experiments using the MDVAD benchmark and demonstrates the limitations of traditional multi-domain learning. It shows the effectiveness of the proposed baselines in handling abnormal conflicts and achieving robust performance across multiple domains.", "questions": "1. What is the training difficulty of MDVAD? The introduction of domain-specific multi-head mechanism and Null-MIL method greatly increases the complexity and computational cost of the model, can it meet the real-time requirements in practical applications?\n2. Have the MDVAD and evaluation protocols been subjected to comparative experiments with other state-of-the-art methods in order to objectively assess the relative advantages of the proposed methods?", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The manuscript proposes a new task, Multiple Domain Video Anomaly Detection (MDVAD), which solves the problem that the existing model is limited to a single domain and provides a new idea for the development of domain-generalized models.\n2.The MDVAD method proposes domain-specific multiple head mechanism and Null-Multiple Instance Learning Method (Null-MIL), which effectively solves the problem of anomaly conflict between different domains.\n3. The MDVAD method constructs a new benchmark containing six representative VAD datasets, which fills the gap of the lack of unified evaluation standard in multi-domain learning tasks.\n4. The MDVAD method designs four evaluation protocols (held-in, leave-one-out, low-shot domain adaptation, and full fine-tuning) to systematically evaluate the generalization ability of the model.", "weaknesses": "1. MDVAD introduces the domain-specific multi-head mechanism and the Null-MIL method, which increases the complexity and computational cost of the model, and may place higher demands on the computational resources in practical applications.\n2. The multi-domain learning task itself is difficult to train, and with the proposed method further increasing the complexity of training, MDVAD may require longer training time and higher technical requirements.\n3. Although the theoretical background and analysis are provided, the theoretical basis and derivation process of some of the methods of MDVAD are slightly weak and need to be further explored and verified in depth. Part of the theoretical analysis is based on specific assumptions, and these assumptions may not be fully valid in practical applications, affecting the applicability of the theoretical analysis.\n4. Although new benchmarks and assessment protocols are proposed, MDVAD lacks comparative experiments with other state-of-the-art methods, making it difficult to objectively assess the relative advantages of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1, W2, Q1] Complexity and computational cost\n\nIn our proposed framework, only the final layer, $W_{D_d}\\in \\mathbb{R}^{T\\times 1}$ where $T=128$, corresponds to the head and is added based on M number of datasets ($T \\times M$). This constitutes a very small parameter and computational load compared to the total weights of the VAD model. We kindly refer reviewer to Section 3.2 on Complexity.\n\n# [W2, Q1] Training difficulty and inference time\n\nRegarding difficulty of training convergence, the pseudo-labels of the AC classifier are assigned through multiple heads (Eq. 6), which may lead to lower label reliability in initial training. However, this issue resolves as training progresses. \n\nComparing a single head and multiple heads (when 6 datasets), the training times are 2.68 and 2.81 hours, and the inference times are 0.158 and 0.164 ms per snippet, respectively, indicating a negligible increase in complexity.\n\n# [W3] Theoretical analysis and applicability\nWe would like to elaborate on how the following points support the theoretical flow of our work:\n* **[Section 2.2 and Section D]** Our motivation and the necessity for the novel task MDVAD are derived from the analysis of the VAD benchmark and the cross-evaluation of single domain models (Section 2.2, Tables 1 and 2). We have identified two key issues: Abnormal Conflict and Scene Discrepancy. We validated our assumption regarding AC by computing $AC_{i,j}$ using Eq. 8, which measures the average of the relative False Positive Rate (FPR) and relative False Negative Rate (FNR) to quantify abnormal conflict between domains (Supplementary Material, Section D, Table A2). For the domain discrepancy assumption, we analyzed the Earth Mover’s Distance (EMD) (Table 3).\n* **[Section 4]** We defined protocols for four scenarios (E1~E4) considering practical applicability and conducted quantitative and qualitative experimental analyses to support our assumptions about AC and domain discrepancy:\nE1: When all source and target domains are accessible, performing similar to multi-task learning, handling multiple domains with a single model.\nE2: When the source and target domains are different, and the target is not accessible, pre-training on large data without target knowledge simulates the scenario where training data for the practical application domain is unavailable.\nE3 : When a few samples from the target domain are provided, adapting the pre-trained general model.\nE4: When the pre-trained general model is fine-tuned for the specific needs of the target domain.\n\n# [W4, Q2] Comparison experiments\n|    |        |          |       |       |       |        |       |       |        |   |   |\n|:---:|:------:|:--------:|:-----:|:-----:|:-----:|:------:|:-----:|:-----:|:------:|:---:|:---:|\n|    | **Models** | **Pub.**     |       |       |       | **Target** |       |       |        |   |   |\n|    |        |          | UCFC  | XD    | LAD   | UBIF   | TAD   | ST    | AVG.    |   |   |\n| **E1** | MMIL   | CVPR 18' | 77.93 | 81.34 | 85.18 | 85.44  | 87.78 | 84.39 | 83.68  |   |   |\n|    | ARNet  | ICME 20' | 79.26 | 80.38 | 85.27 | 84.18  | 89.57 | 86.65 | 84.22  |   |   |\n|    | WSAL   | TIP 21'  | 76.47 | 78.35 | 75.44 | 86.41  | 85.62 | 82.44 | 80.79  |   |   |\n|    | COMO   | CVPR 23' | 80.41 | 82.75 | 86.24 | 85.82  | 90.13 | 89.76 | 85.85  |   |   |\n|    | **Ours**   |     -     | 77.21 | 82.09 | 83.88 | 91.9   | 91.36 | 91.12 | **86.26**  |   |   |\n|    |        |          |       |       |       |        |       |       |        |   |   |\n| **E2** | MMIL   | CVPR 18' | 76.68 | 74.92 | 67.39 | 82.4   | 67.61 | 61.86 | 71.81  |   |   |\n|    | ARNet  | ICME 20' | 77.05 | 75.02 | 78.98 | 80.84  | 75.09 | 55.34 | 73.72  |   |   |\n|    | WSAL   | TIP 21'  | 76.59 | 73.75 | 76.71 | 79.86  | 77.68 | 55.54 | 73.36  |   |   |\n|    | COMO   | CVPR 23' | 77.07 | 76.64 | 77.43 | 76.74  | 78.67 | 57.63 | 74.92  |   |   |\n|    | **Ours**   |     -     | 78.55 | 77.68 | 77.36 | 82.53  | 79.21 | 60.41 | **75.96**  |   |   |\n|    |        |          |       |       |       |        |       |       |        |   |   |\n| **E3** | MMIL   | CVPR 18' | 77.28 | 72.75 | 80.7  | 85.29  | 81.72 | 61.33 | 76.51  |   |   |\n|    | ARNet  | ICME 20' | 75.48 | 72.18 | 79.9  | 81.7   | 79.43 | 68.24 | 76.16  |   |   |\n|    | WSAL   | TIP 21'  | 75.77 | 72.18 | 66.45 | 82.57  | 75.05 | 78.05 | 75.01  |   |   |\n|    | COMO   | CVPR 23' | 70.02 | 72.89 | 80.59 | 83.05  | 74.73 | 73.53 | 75.80  |   |   |\n|    | **Ours**   |    -      | 78.99 | 75.8  | 77.82 | 85.75  | 84.06 | 76.23 | **79.78**  |   |   |\n|    |        |          |       |       |       |        |       |       |        |   |   |\n| **E4** | MMIL   | CVPR 18' | 80.26 | 82.51 | 86.54 | 89.88  | 90.32 | 89.34 | 86.48  |   |   |\n|    | ARNet  | ICME 20' | 80.88 | 82.57 | 86.72 | 89.92  | 90.69 | 91.7  | 87.08  |   |   |\n|    | WSAL   | TIP 21'  | 80.99 | 81.6  | 76.43 | 87.87  | 88.78 | 87.47 | 83.86  |   |   |\n|    | COMO   | CVPR 23' | 80.61 | 84.25 | 86.88 | 91.51  | 91.74 | 91.23 | **87.70**  |   |   |\n|    | **Ours**   |     -     | 78.62 | 82.71 | 84.41 | 94.42  | 92.5  | 91.17 | 87.31  |   |   |\n|    |        |          |       |       |       |        |       |       |        |   |   |\n\nInstead of exploring complex architecture designs for single-domain VAD models, this paper focuses on the analysis of multiple domain learning within the context of AC issues. In response to the reviewer's concerns, we compared MDVAD with four representative WVAD models (Section E): MMIL [42], ARNet [45], WSAL [29], and CoMo [7]. In the Table, proposed simple baseline demonstrates competitive performance. Notably, the average results show superior results in the E2 and E3, indicating better generalization and adaptation to unseen target domains. Various single-domain VAD models or backbones can be incorporated into the MDVAD task, showing a direction for future generalization work (please refer to the W2 response of 926M).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XslE8Tdz6x", "reviewer": "Reviewer_beRv", "summary": "In this paper, authors proposed a new task called Multiple Domain VAD (MDVAD), along with a benchmark and new evaluation protocols. Authors' goal is to construct a general VAD model by conducting multi-domain learning while recognizing abnormal conflicts and exploring representations of general normality and abnormality. Authors introduced a baseline for MDVAD and proposed a new framework with multi-head to mitigate abnormal conflicts and proposed Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIL) losses for multi-domain training. Additionally authors suggested an Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts. Authors analyzed the primary issues of MDVAD and proposed a baseline for this new task.", "questions": "My main questions are shown in the weakness.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. According to the analysis, authors believed that the abnormal conflict and the scene discrepancy are the two main issues and designed a framework with multi-head to deal with these problems. \n\n2. Null-MIL and NullAng-MIL methods are designed for multi-domain learning, and an AC classifier is proposed for learning general features while abnormal conflicts exists.\n\n3. Authors provided sufficient experiment results for this task and create a new baseline.", "weaknesses": "1. The proposed framework with multi-head for multi-domain seems not flexible enough during the domain changes, such as adding a new dataset with extra abnormal conflicts. And for the abnormal conflicts, will the proposed method performs better compared to make anomaly categories classifications for all anomaly events type of all domains?\n2. In my opinion, traditional WS-VAD methods are designed to detect abnormal events in single domain without abnormal conflicts, and when abnormal conflicts exists, it will be better to use other paradigms such as temporal action localization or video grounding. And for the current WS-VAD datasets, the annotations are video-level, or even without category information, which is too weak for higher level anomaly detection. Training model with the current MDVAD paradigm is likely to not achieve good results.\n3. Maybe using visual-language model with multimodal alignment can deal with the above issues? These models contain more knowledges for more event categories and higher generalization ability, which are likely to have the ability to individually detect conflicting anomalies. Compared to multi-head regression, is VL alignment a better approach for MDVAD task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1-1] when adding a new dataset\nThank you for highlighting this important consideration. Our framework consists of domain-agnostic layers and domain-specific heads, with each head being the final layer of the entire model, $W_{D_d}\\in \\mathbb{R}^{T\\times 1}$ where $T=128$, which is a very small part. If a new domain is introduced and additional models need to be trained, we can flexibly add a final branch. Creating a single general model is more cost-effective than developing individual in-domain models for each domain and retraining them every time a new domain appears. Moreover, such as in the E3 and E4 settings, pre-training the general model (E2) with multiple source domains without target domains, when new target data is added, can adapt to the new domain by tuning the source heads, without adding new heads. We will include this perspective in Section 4.3 discussion.\n\n# [W1-2] Abnormal classification for AC\nEven with comprehensive information on all abnormal categories across all domains, the lack of consistency in abnormality criteria between domains would still result in abnormal conflicts, posing challenges for abnormal classification through multiple domain learning. Additionally, in real-world scenario, addressing the issue through classification would lead to a closed-set model, limiting its ability to handle unexpected anomalies. We will discuss this approach in Section E of the supplementary material.\n\n# [W2] WS-VAD limitations and alternative paradigms\nWe appreciate the reviewer's insightful comments. We would like to address each point raised:\n1. **Traditional WS-VAD Methods in Single Domain**\nWhile traditional WS-VAD methods are designed for single-domain anomaly detection without conflicts, the MDVAD approach specifically addresses the limitations of these methods, which are heavily influenced by the criteria for abnormality defined by each dataset. Models that perform well in in-domain settings require each domain to be learned separately, necessitating sufficient training data for each single domain. This limits the applicability of anomaly detection methods to real-world settings. Therefore, research on a general VAD model capable of multiple domain learning is necessary to effectively mitigate the challenges posed by abnormal conflicts across different domains.\n2. **Alternative Paradigms for AC (Temporal Action Localization or Video Grounding)** \nWe agree that temporal action localization and video grounding are effective paradigms for certain types of video analysis. However, even with precise temporal annotations or detailed category information, defining anomalies precisely across multiple domains is challenging due to the varying criteria and sensitivity across datasets. This makes it difficult to provide a clear solution for conflicts.\nWe hope this addresses your concerns. Thank you for your valuable feedback. Please let us know if any further considerations are needed.\n\n# [W3] Utilizing VL models\n\nWe thank the reviewer for their thoughtful suggestion regarding the use of visual-language models with multimodal alignment. We appreciate the opportunity to discuss the potential advantages and considerations of such an approach in MDVAD task.\n* **Advantages of Visual-Language Models**: Recent studies have demonstrated the powerful capabilities of VLMs, achieving performance gains by leveraging VLM backbones and multimodal alignment for VAD tasks. Additionally, the integration of text information allows for meaningful interpretations of anomalies.\n* **AC detection with VL models**: While VLMs offer significant advantages, they also have limitations in capturing complex scenes in surveillance videos. According to [iii], existing VLMs may face challenges such as inaccuracies in color recognition, difficulty identifying intricate scenes, and struggles in capturing subtle movements. Furthermore, the availability of textual descriptions or annotations for VAD datasets is often limited, making high-level information like abnormal conflict detection an additional challenge.\n* **Future Directions**: We acknowledge the potential of VLMs and multimodal alignment as a promising direction for future research. Incorporating such models could enhance the ability to detect and classify a broader range of anomalies, especially in scenarios where textual annotations are available. For future work, we plan to explore this avenue to further improve the robustness and generalization capabilities of our MDVAD framework.\n\n[iii] Yuan, Tongtong, et al. \"Towards Surveillance Video-and-Language Understanding: New Dataset Baselines and Challenges.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lw3aonvTQw", "reviewer": "Reviewer_926M", "summary": "This work proposes a new task named Multi-Domain Learning Video Anomaly Detection, which aims to learn a general VAD model across domains. The work finds that abnormal conﬂict is a critical challenge in the task. Then, the work establishes a new benchmark, designs an effective baseline and conducts extensive experiments to investigate this challenge. The results shown on the benchmark demonstrate that the abnormal conﬂict is alleviated.", "questions": "See the Weakness part.", "limitations": "The paper has discussed the limitations and potential impacts of the work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The work proposes a new task, which is interesting. \n2. The work establishes a new benchmark to evaluate the new task. \n3. The motivation of the proposed baseline, i.e., abnormal conﬂict, is clear and makes sense.", "weaknesses": "I have some concerns about the proposed method, and I think more comparison experiments are needed to demonstrate the effectiveness. Despite this, I think the abnormal conflict issue is interesting, thus I am willing to raise my rating if my major concerns are addressed. My concerns are as follows:\n\n1. Why the proposed Abnormal Conﬂict (AC) classifer can address the abnormal conﬂict problem? Why the label is determined by the discrepancy in Eq. (6)? It seems that there are some mistakes in the formula (inconsistent with that in Fig. 2). \n2. I would like to see the results of more baselines, in addition to MIL, Null-MIL and NullAng-MIL. \n3. More detailed discussions about related works are needed, e.g., virual video anomaly detection datasets [1] and related techniques utilizing virtual datasets [2]. \n\n[1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022\n\n[2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1] AC Classifier\nThank you for highlighting this important aspect.\n\n**Role of the AC Classifier:** By training with the AC Classifier, the domain-agnostic layer learns Conflict-Aware features, which helps in resolving conflicts.\n\nTo achieve a general VAD model through multiple domain learning while avoiding abnormal conflicts, our framework aggregates features through domain-agnostic layers and performs multiple head learning. This activates only the output of the head corresponding to the input domain, assigning inactive heads with Null values to prevent confusion. While the heads are divided to prevent conflicts, the agnostic part extracts features from all datasets using a single branch. Therefore, to explore general features while being aware of abnormal conflicts, the AC classifier predicts conflicts by leveraging the variance in abnormal scores across multiple heads. This auxiliary task provides performance gains in most experiments. Notably, in E2 and E3 settings, it shows a significant boost effect, aiding in adaptation to unseen domains without any additional cost to the final model. Additionally, as shown in Supplementary Material Fig. A5, the conflict score of the AC classifier plotted for the UCFC domain reveals high conflict scores in scenarios like jaywalking, which is normal in UCFC but abnormal in other domains (TAD, ST), demonstrating the classifier's awareness of AC in the domain-agnostic part.\n\n**Pseudo label of the AC Classifier:** As shown in Table A5, assigning pseudo labels in Eq. 6 was determined experimentally. \n\nThe purpose of the AC classifier is to facilitate feature learning with discrepancies across domains. Therefore, when the gap between the abnormal scores of multiple heads exceeds a threshold $\\tau$, it indicates different definitions of normal and abnormal between domains (e.g., abnormal in one domain and normal in another), and a conflict is considered to have occurred. When the score gap is small, it indicates consistent scores with no conflict, and the AC classifier generates a pseudo label accordingly. \n\nIn Fig. 2, the yellow score graph shows a large gap between scores $S^a_{D_2}$ and $S^a_{D_M}$​, indicating an abnormal conflict and assigning $Y^{AC}=1$ in Eq. 6. Conversely, the green score graph shows consistent scores $S^a_{D_i}$ across all domains (all normal or all abnormal), indicating no conflict and assigning $Y^{AC}=0$. To determine conflicts between domains, we conducted ablation studies comparing different methods, such as the difference between scores of multiple heads, Std. with τ = 0.1, and a fixed value . \n\nWe kindly refer reviewer to Section 4.2.5, \"Role of the AC Classifier,\" and Section 4.3, \"Discussions.\" If there are any mistakes in the figures, we would appreciate it if you could point them out for corrections.\n\n# [W2] Additional Baselines \n|    |           |       |       |       |        |       |       |        |\n|:---:|:---------:|:-----:|:-----:|:-----:|:------:|:-----:|:-----:|:------:|\n|    | **Models**    |       |       |       | **Target**|       |       |        |\n|    |           | UCFC  | XD    | LAD   | UBIF   | TAD   | ST    | AVG.   |\n| **E1** | WSAL      | 76.47 | 78.35 | 75.44 | 86.41  | 85.62 | 82.44 | 80.79  |\n|    | WSAL+Ours | 76.90 | 78.59 | 76.17 | 87.83  | 81.89 | 86.27 | 81.27  |\n|    |           |       |       |       |        |       |       |        |\n| **E2** | WSAL      | 76.59 | 73.75 | 76.71 | 79.86  | 77.68 | 55.54 | 73.36  |\n|    | WSAL+Ours | 76.67 | 73.69 | 65.54 | 85.85  | 70.1  | 72.41 | 74.04  |\n|    |           |       |       |       |        |       |       |        |\n| **E3** | WSAL      | 75.77 | 72.18 | 66.45 | 82.57  | 75.05 | 78.05 | 75.01  |\n|    | WSAL+Ours | 76.81 | 72.38 | 59.51 | 83.49  | 77.6  | 74.57 | 74.06  |\n|    |           |       |       |       |        |       |       |        |\n| **E4** | WSAL      | 80.99 | 81.6  | 76.43 | 87.87  | 88.78 | 87.47 | 83.86  |\n|    | WSAL+Ours | 79.21 | 81.17 | 78.19 | 90.98  | 86.09 | 90.61 | 84.38  |\n|    |           |       |       |       |        |       |       |        |\n\nWe conducted additional experiments using the WSAL [29] model as a baseline to validate our proposed method on the MDVAD benchmark's four protocols. The above table presents the results, where all settings are consistent with the other experiments in the paper. The results show the addition of multi-head learning with NullAng-MIL and the AC Classifier to the WSAL model brings performance gains which effectively operate across different baselines. Notably, in the E3 and E4, which shows the target adaptation of the pre-trained general model, the method yielded competitive results.\n\nWe focus more on highlighting the necessity of MDVAD and raising the issue of AC. Our paper introduces a novel task along with a benchmark, evaluation protocols, and a baseline and emphasizes analysis. Therefore, aspects like baseline architecture design or utilizing a powerful backbone were not the primary focus. However, future research will delve deeper into studying more sophisticated baselines for resolving AC within the MDVAD setting.\n\n# [W3] VAD with virtual datasets [i, ii]\nWe appreciate the thoughtful comment. Following the reviewer's suggestion, we have added an analysis using the virtual dataset UBNormal [i] (please refer to the general response above).\n\nPFMP [ii] proposed a method to utilize virtual data anomalies to reduce scene discrepancy with real-world data, addressing the issue of data scarcity in VAD. While this method can mitigate data scarcity, it still faces the problem of anomaly conflicts when the criteria for abnormal events are inconsistent across multiple datasets. \n\nThe conducted experimental analysis on utilizing virtual datasets [i] and a discussion on related work [ii] will be included in Section E of the supplementary material.\n\n[ii] \"Generating anomalies for video anomaly detection with prompt-based feature mapping.\" CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. We tried to address all the questions with references to weaknesses (**W**) and questions (**Q**). 

We are glad the reviewers found that
* Addressing an Important and Generalizable Problem
* Novelty of Method and Effectively Solving Identified Problems
* Thorough Experiments and a Systematic Protocols for Model Generalization

# Summary of rebuttal
In response to the reviewers’ feedback, we will improve the paper as follows:
- We have added additional virtual UBNormal datasets for multiple-domain learning and open-set VAD experiments with discussions (reviewer 926M).
- We have expanded experiments on additional baseline and state-of-the-art comparison methods with detailed discussions (reviewer SwSy and 926M).
- We have provided clear explanations and detailed descriptions regarding the theoretical analysis, practical applicability, experimental settings, and the role of the AC classifier (reviewers SwSy, 2cHp, and 926M).
- We have included further discussion about other alternative paradigms and future work with vision-language models (reviewer beRv).

# Results added using virtual data
In response to Reviewer 926M's comment regarding the discussion on utilizing virtual datasets, we have included the UBNormal dataset [i] alongside the MDVAD to conduct multi-domain learning and validate the model's generality in settings with significant scene discrepancy. 

1. **Multi-domain learning with virtual domain dataset:**

|                     |       |       |        |       |       |       |       |
|---------------------|-------|-------|--------|-------|-------|-------|-------|
| **Baseline**           |       |       | **Target** |       |       |       |       |
|                     | UCFC  | XD    | LAD    | UBIF  | TAD   | ST    | AVG.  |
| **Source: MDVAD**       |       |       |        |       |       |       |       |
| MIL                 | 80.05 | 83.77 | 86.01  | 85.76 | 88.92 | 88.82 | 85.56 |
| MIL+AC              | 80.11 | 83.91 | 85.15  | 87.72 | 90.05 | 87.98 | 85.82 |
| NullMIL             | 79.01 | 81.96 | 85.08  | 93.06 | 90.57 | 91.04 | 86.79 |
| NullMIL+AC          | 79.15 | 82.96 | 85.82  | 92.41 | 91.16 | 89.67 | 86.86 |
| NullAngMIL          | 76.32 | 82.74 | 82.32  | 92.30 | 91.82 | 91.26 | 86.13 |
| NullAngMIL+AC       | 77.21 | 82.09 | 83.88  | 91.90 | 91.36 | 91.12 | 86.26 |
|                     |       |       |        |       |       |       |       |
| **Source: MDVAD + UBN** |       |       |        |       |       |       |       |
| MIL                 | 78.59 | 81.79 | 85.06  | 86.6  | 88.83 | 87.99 | 84.81 |
| MIL+AC              | 78.14 | 81.78 | 84.92  | 85.08 | 90.77 | 89.12 | 84.96 |
| NullMIL             | 78.74 | 83.34 | 85.93  | 91.28 | 90.05 | 88.69 | 86.34 |
| NullMIL+AC          | 78.76 | 83.19 | 86.01  | 92.63 | 90.73 | 90.54 | 86.98 |
| NullAngMIL          | 77.09 | 81.01 | 83.96  | 92.88 | 91.57 | 90.04 | 86.09 |
| NullAngMIL+AC       | 77.66 | 82.09 | 83.01  | 92.55 | 91.33 | 91.07 | 86.29 |
|                     |       |       |        |       |       |       |       |

The UBNormal (UBN) dataset is a VAD benchmark proposed for open-set scenarios to handle unexpected abnormal events. Both normal and abnormal events are available during training, but the anomalies that occur during inference belong to a distinct set of anomaly types (categories). To alleviate the difficulty of collecting abnormal event data in the real world, UBN consists of synthetic videos, unlike other VAD datasets. There are substantial abnormal conflicts and differences in the visual settings of scenes compared to other domains.

The table shows the performance in each target domain when trained with MDVAD and UBNormal under the E1: held-in setting. In the single-head MIL baseline, there is a performance drop when trained with MDVAD+UBN, indicating difficulty in handling Abnormal Conflict (AC). However, the model trained with multiple heads and the AC classifier shows improved results. By leveraging the virtual dataset, we can overcome data limitations and create a general model capable of handling diverse and complex scenes.

2. **Openset VAD results**

|                            |                |
|----------------------------|----------------|
| **Open-set VAD**               |                |
| **Single source:**       UBN         |  / **Target:** UBN |
| MIL                        | 75.13          |
|                            |                |
| **Multi source:** MDVAD + UBN  | / **Target:** UBN  |
| MIL                        | 67.95          |
| MIL+AC                     | 70.56          |
| NullMIL                    | 72.14          |
| NullMIL+AC                 | 70.94          |
| NullAngMIL                 | 74.42          |
| NullAngMIL+AC              | 74.54          |
|                            |                |

We conducted experiments in an open-set scenario using UBN, where the abnormal categories in the train set and test set do not overlap. As shown in the table, despite domain discrepancies and AC, the model effectively handles multiple domain learning, demonstrating that general feature learning can adequately address unseen abnormal categories.

[i] "Ubnormal: New benchmark for supervised open-set video anomaly detection." CVPR. 2022.

Meta-review: This paper received 4 reviews from experts in the field. The paper received the following reviews:  3 Borderline Accepts and 1 Borderline Reject.

The main concerns presented by the reviewers involved the clarity of the presentation. The author rebuttal addressed the reviewers’ concerns. The AC agrees that the merits of this paper outweigh any clarity concerns and the decision for this paper is to accept.  Please consider the reviewer comments in your preparation of the camera ready version of the paper.
Decision: Accept (poster)