Title: Towards Multi-Domain Learning for Generalizable Video Anomaly Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Video anomaly detection

Multi-domain learning

Domain generalization
Abstract: Most of the existing Video Anomaly Detection (VAD) studies have been conducted within single-domain learning, where training and evaluation are performed on a single dataset. However, the criteria for abnormal events differ across VAD datasets, making it problematic to apply a single-domain model to other domains. In this paper, we propose a new task called Multi-Domain learning forVAD (MDVAD) to explore various real-world abnormal events using multiple datasets for a general model. MDVAD involves training on datasets from multiple domains simultaneously, and we experimentally observe that Abnormal Conflicts between domains hinder learning and generalization. The task aims to address two key objectives: (i) better distinguishing between general normal and abnormal events across multiple domains, and (ii) being aware of ambiguous abnormal conflicts. This paper is the first to tackle abnormal conflict issue and introduces a new benchmark, baselines, and evaluation protocols for MDVAD. As baselines, we propose a framework with Null(Angular)-Multiple Instance Learning and an Abnormal Conflict classifier. Through experiments on a MDVAD benchmark composed of six VAD datasets and using four different evaluation protocols, we reveal abnormal conflicts and demonstrate that the proposed baseline effectively handles these conflicts, showing robustness and adaptability across multiple domains.

Towards Multi-Domain Learning for Generalizable Video Anomaly Detection

Abstract

Most of the existing Video Anomaly Detection (VAD) studies have been conducted within single-domain learning, where training and evaluation are performed on a single dataset. However, the criteria for abnormal events differ across V AD datasets, making it problematic to apply a single-domain model to other domains. In this paper, we propose a new task called M ultiD omain learning for VAD ( MDVAD ) to explore various real-world abnormal events using multiple datasets for a general model. MDVAD involves training on datasets from multiple domains simultaneously, and we experimentally observe that Abnormal Conflicts between domains hinder learning and generalization. The task aims to address two key objectives: (i) better distinguishing between general normal and abnormal events across multiple domains, and (ii) being aware of ambiguous abnormal conflicts. This paper is the first to tackle abnormal conflict issue and introduces a new benchmark, baselines, and evaluation protocols for MDVAD. As baselines, we propose a framework with Null(Angular)-Multiple Instance Learning and an Abnormal Conflict classifier. Through experiments on a MDVAD benchmark composed of six VAD datasets and using four different evaluation protocols, we reveal abnormal conflicts and demonstrate that the proposed baseline effectively handles these conflicts, showing robustness and adaptability across multiple domains.

1 Introduction

Video Anomaly Detection (VAD) is identifying abnormal events in diverse scenarios depicted in a video and determining their temporal intervals at a frame-level. Nowadays, CCTVs are ubiquitous, recording every moment of life, which aids in preventing accidents and responding to crimes promptly. However, human monitoring of every situation is highly inefficient, requiring significant labor and resources. Consequently, extensive research has been conducted to automate VAD through deep learning by leveraging large amounts of surveillance data [33, 5, 24, 15, 55, 44, 43, 50, 21, 7].

In VAD research, Weakly-supervised VAD (WVAD) [55, 44, 43, 50, 21, 7], which involves learning normal and abnormal events with minimal supervision of the video-level annotation and detecting abnormal events at the frame-level during testing, has been studied a lot recently. This paper focuses on the WVAD setting (denoted as VAD), and a summary of VAD research is explained in the

Minho Shim NAVER Cloud

Sangyoun Lee †

UCFC: Normal

TAD: Abnormal

UCFC: Normal

TAD: Abnormal

D1

D2

D3

𝐷1

!

∩𝐷2

!

∩𝐷3

!

𝐷1 ∩ 𝐷2 ∩ 𝐷3

𝐷1 ∪ 𝐷2 ∪ 𝐷3 - 𝐷1 ∩ 𝐷2 ∩ 𝐷3

Normal

Abnormal

Abnormal Conflicts

(b) Venn diagrams of events

supplementary material (§ E). Unlike the conventional VAD research, we first address the following three key questions:

Q1: What is the problem with the existing VAD model? Most VAD models are trained in a singledomain where the training and evaluation dataset are the same. In the case of single-domain learning, application across different datasets (cross-domain evaluation) results in performance degradation as reported in [14, 6, 7] because V AD models are heavily influenced by the criteria for abnormality defined by each dataset.

Q2: Why do we need a general VAD model? First, a single generalized model removes the need for multiple specific models for different domains, analogous to multi-task learning. Second, proper pre-training on multiple domains embodies generalized representation, and it can discriminate abnormal events according to their domain, leading to better performance in unseen target domains. Consequently, a general VAD model will be highly beneficial for applying V AD in practical scenarios.

Q3: Is it possible to create a general VAD model? The general VAD model aims to handle multiple domains, but this is challenging because the definition of abnormal differs for each dataset, leading to conflicts between these abnormal events. For example, as shown in Fig. 1(a), in one dataset, a pedestrian on the road is considered normal, while in another dataset, it is deemed abnormal. A general VAD cannot be solved with a naive muti-task learning because of this confusion across multiple domains, which this paper defines as 'Abnormal Conflict.' Therefore, for general V AD, it is necessary to be aware of these abnormal conflicts (yellow region in Fig. 1(b)) and learn general normal (green region in Fig. 1(b)) or abnormal (red region in Fig. 1(b)) representations that are common across all domains.

Our goal is to construct a general V AD model by conducting multi-domain learning while recognizing abnormal conflicts and exploring representations of general normality and abnormality . To achieve this goal, we introduce a new task called 1) Multiple Domain VAD (MDVAD), along with a benchmark and new evaluation protocols. MDVAD involves concurrent training on multiple VAD datasets, each with its own definition of abnormality. Specifically, the MDVAD benchmark comprises six representative VAD datasets with balanced sampling (§ 4.1). We also propose four evaluation protocols: held-in, leave-one-out, low-shot domain adaptation, and full fine-tuning. The held-in protocol is designed to evaluate the model's ability as a unified model like multi-task model, while the leave-one-out, low-shot domain adaptation, and full fine-tuning protocols are intended to access the model's capability as a general pre-training for an unseen target domain.

As multi-domain learning is a novel concept in the field of V AD, we also introduce baselines and new learning methods. We design domain-specific multiple heads to mitigate abnormal conflicts and learn generality across domains. To facilitate multi-head learning without conflicts, we propose the 2) Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIL) losses , which activate only the output of the head corresponding to the input domain, assigning inactive heads with Null values to prevent confusion. Additionally, we suggest the 3) Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts , leveraging the variance in abnormal scores across the multiple heads (Fig. 2 green and yellow region in the Venn diagram). Through experiments with four protocols on the MDVAD benchmark, we reveal the limitations of multi-domain learning with abnormal conflict and demonstrate the effectiveness of our baselines in offering a generalized and adaptive model.

1.1 Scope of research

Focusing on Multiple Domains. In this paper, we focus on solving the problems in multiple domains introduced above rather than solving problems in a single domain. Therefore, complex design of the backbone and head or achieving state-of-the-art performance in a single domain can proceed orthogonally with our research. Instead, this paper raises the issue of abnormal conflict and focuses on the necessity of MDVAD.

Distinction from Open-set VAD Approaches. Unlike Open-set VAD [59, 25, 1], which separates seen and unseen anomalies within a single-domain cannot achieve robust representation learning among multiple domains, this paper excels at handling abnormal conflicts and shows adaptability across domains with versatile evaluation protocols (held-in/out and low-shot).

2 Observations

2.1 Datasets

In this paper, we use six representative V AD datasets: UCF-Crimes (UCFC) [43], XD-Violences (XD) [51], Large-scale Anomaly Detection (LAD) [47], UBI-Fights (UBIF) [9], Traffic Anomaly Dataset (TAD) [19], and Shanghai-Tech Campus (ST) [24]. As shown in Table 1, each dataset has different environments ( e.g. CCTV, Traffic, Campus), quantities, and abnormal categories. Unlike other datasets, ST is an unsupervised VAD benchmark whose training set comprises only normal videos, so the training set has been reorganized following [21, 45, 57]. More details are provided in the supplementary material (Sec. A).

2.2 Analysis

Based on observations of each dataset's properties, we aim to quantify how the different properties negatively affect domain shifts. Table 2 presents cross-domain evaluation results, i.e ., single-domain models validated on different target datasets. The results reveal that while these models excel indomain settings (diagonal elements of the table), they exhibit significant performance degradation in cross-domain scenarios. This implies that a single-domain VAD model may be ineffective in most other environments unless the environment and user's intention are precisely identical. Therefore, leveraging diverse datasets for generalized feature learning is crucial, enabling the model to handle unknown domain and well adapted to unseen anomalies. This paper addresses two primary issues: 1) Abnormal conflict arising during the multi-domain learning process, and 2) Scene discrepancy occurring during the evaluation process on an unseen target domain.

Abnormal conflict. As illustrated in Fig. 1, abnormal conflicts indicate abnormal events that are considered abnormal in one (or some) domain(s) but are denoted as normal in other domains. As

UCFC [43]

XD [51]

LAD [47]

UBIF [9]

TAD [19]

ST [24]

Settings

CCTV

CCTV,

Sports,

Cartoon,

Movies,

News

CCTV

CCTV,

Mobile

Traffic

(CCTV)

Campus

(CCTV)

Training Set Volume

1610

3954

1440

933

400

238

Abnormal Categories

Abuse

Abuse

Hurt

Illegal Turns

Chasing

Arrest

Road Spills

Dropping

Fall Into Water

Throwing

Assault

Falling

Illegal Occupations

Running

Accident

Car Accident

Crash

Vehicle Accidents

Jumping

Burglary

Destroy

Pedestrian on Road

Motorcycle

Explosion

Explosion

Fire

The Else

Skateboard

Fighting

Fighting

Fighting

Fighting

Fighting

Robbery

Violence

Robbery

Shooting

Shooting

Crowd

Gun

Stealing

Thieving

Car

Shoplifting

Loitering

Loitering

Vandalism

Panic

Trampled

Target

Source

UCFC

XD

LAD

UBIF

TAD

ST

UCFC

82.32

68.06

75.75

71.12

73.75

59.24

XD

68.38

90.87

77.60

67.23

71.10

46.87

LAD

59.60

75.26

86.97

59.27

73.80

47.29

UBIF

74.79

75.22

70.29

93.63

68.16

54.21

TAD

50.83

45.38

52.02

61.95

90.71

41.58

ST

55.75

52.87

48.96

59.00

48.57

91.88

UCFC

XD

LAD

UBIF

TAD

ST

UCFC

-

7.90

8.45

5.87

9.76

9.32

XD

9.76

-

6.19

6.94

9.65

9.10

LAD

12.78

10.76

-

7.90

10.09

10.43

UBIF

7.90

7.37

11.99

-

10.54

9.54

TAD

11.32

10.87

14.39

10.98

-

10.65

ST

11.32

10.31

14.27

8.77

11.77

-

Video

Backbone

❄

Feature

Aggregation

Temporal

Aggregation

AC

Classifier

Domain-Agnostic Layers

Single/Multiple Abnormal Heads (Sec. 3.1 & 3.2)

FC

Abnormal Head 1,

w

!

!

"

Abnormal Head d,

𝐰

!

"

"

Abnormal Head M,

𝐰

!

#

"

Normal Head 1,

𝐰

!

!

#

Normal Head d,

𝐰

!

"

#

Normal Head M,

𝐰

!

#

#

Input Video

(d-th Domain)

Train-Only

...

...

...

...

𝐬

!

!

"

AC Label

AC Loss

𝑠

!

!

,$

"

𝑠

!

"

,$

"

𝑠

!

#

,$

"

...

𝑠

!

$

,$

"

Abnormal Conflict = 1

Abnormal Conflict = 0

MIL

Null-MIL

NullAng-MIL

Top-K

𝑠

!

!

,$

"

𝑠

!

"

,$

"

𝑠

!

#

,$

"

...

𝑠

!

$

,$

"

𝑠

!

!

,$

"

𝑠

!

"

,$

"

𝑠

!

#

,$

"

...

𝑠

!

$

,$

"

Null

𝜽

𝐰

!

!

"

𝐰

!

!

#

𝐰

!

"#!

"

𝐬

!

%

"

𝐬

!

$

"

Top-K

Null

𝐬

!

!

%

Null

𝐬

!

%

%

𝐬

!

$

%

Top-K

Null

NullAng-MIL

𝚩

𝐅

"&&

𝐅

!'

𝐅

AC Classifier (Sec. 3.3)

shown in Table 1, there is relatively little overlap of abnormal classes in ST with other datasets. In other words, this means that the abnormal conflict with other datasets is relatively large. This conflict leads to low cross-dataset performances of models trained or evaluated on ST in Table 2. Taking UCFC as an example (target domain: UCFC column in Table 2), performance increases in the order of TAD, ST, LAD, XD, and UBIF, which is proportional to the number of abnormal conflict categories that exist in the source domain. Abnormal conflict makes the MDVAD unique as the label spaces between domains literally conflict because of each other's differing definitions.

Scene discrepancy. Scene discrepancy refers to differences in the visual settings of scenes, distinct from abnormal conflict arising from variations in the definition of abnormal classes. To quantify scene discrepancy, we utilize the Earth Mover's Distance (EMD) [38] introduced in [8] to calculate the distance between VAD datasets in the Table 3. In Table 3, the top-right section illustrates the comparison of normal features, while the bottom-left provides a numerical comparison of class-wise abnormal features. Lighter colors indicate a higher discrepancy between datasets. Unlike other datasets, TAD, which comprises traffic videos, exhibits a large distance from normal sample distances in the dataset, while LAD, with diverse and complex scenes for abnormal classes, shows the furthest distance from other datasets. This explains the results of domain adaptation experiments. (Table 7).

3 Baselines

Fig. 2 depicts the overall framework of our MDVAD baselines divided into domain-agnostic layers, consisting of the video backbone and aggregation modules, Single (§ 3.1) or Multiple heads (§ 3.2), and AC classifier (§ 3.3).

R

Domain-agnostic layers. The input abnormal or normal video V a or V n is divided into uniformly sampled T snippets ( V a ∈ { v a 1 , · · · , v a T } ). These T snippets pass through a pre-trained video backbone, resulting in a C -dimensional feature B ∈ R T × C that undergoes an aggregation module, fusing them from the feature level to the temporal level. The feature aggregation layer doubles the channel size ˆ B ∈ T × 2 C , followed by a split and max operation F = max [ ˆ B c , ˆ B C + c ]

c =1 , ··· ,C to squeeze the channel size back to C for the i -th snippet. Activating only the maximum element during gradient propagation enables the implicit differentiation of class-specific channels, allowing the model to establish discrepancies between classes [7, 52]. The temporal aggregation layer, with a temporal kernel size of 3 , produces the domain-agnostic aggregated feature F DA ∈ R T × C/ 2 .

agg,i

i

i

3.1 Single-domain learning

The final feature F DA undergoes fully connected layers denoted as FC and the single abnormal head ( w a D 1 ∈ R C/ 16 × 1 ) followed by a sigmoid function to derive the final abnormal score s a D 1 ∈ R T × 1 .

MIL. Due to the absence of temporal interval training labels for abnormal events, WV AD models rely on video-level labels for training, employing the Multiple Instance Learning (MIL) method. When the topK score set is represented as Ω k ( s a D 1 ) , the Binary Cross Entropy-based (BCE) classification loss function is formulated as presented in Eq. 1, where y = { 0 , 1 } .

This loss function ensures that only snippets with high (TopK ) abnormal scores can contribute to the loss. This single-head model serves as the single-domain baseline (MIL in Fig. 2).

3.2 Multi-domain learning: Multi-head learning

Null-MIL. To address the abnormal conflict issue mentioned in Sec. 2, in the MDVAD framework, the abnormal head is divided into multiple heads, each responsible for its own domain, allowing for the domain-wise prediction of the output score. Inspired by [18], the prediction score for the input snippet from the D d dataset is derived exclusively from the output of the D d -head, and the results from heads of other datasets are filled with Null values (Null-MIL in Fig. 2). Compared to Eq. 1, the Null-MIL loss is changed as follows:

where M is the number of heads (domains). To avoid the abnormal conflicts, the heads between datasets are separated where D d -head's weight w a D d is independently trained for the corresponding dataset. In Eq. 2, only s a D d among all output scores is added to the loss, thus the gradient becomes ∂ s a Dd ∂ w a Dd , while other heads are not affected.

For the test, the abnormal score of i -th snippet is s a D d ,i when the target dataset is D d and max d s a D d ,i by selecting the maximum value for unseen target data.

NullAng-MIL. We additionally propose a NullAngular-based MIL method that employs the angular margin to effectively diminish large variations among intra-class instances. In this case, multiple normal heads are added (NullAng-MIL in Fig. 2). When head classifier weight of each dataset D d is denoted as w a D d and w n D d and final embedding feature is F , the final abnormal and normal scores are represented by s a D d = F · w a D d and s n D d = F · w n D d , respectively. Normalizing the head weight and feature vector to 1 results in s a D d = ∥ F ∥ ∥ ∥ w a D d ∥ ∥ cos θ a D d = cos θ a D d and s n D d = cos θ n D d , representing cosine similarity. Thus, in the cosine space, Eq. 2 can be defined as Eq. 3, requiring the maximum cosine similarity between the feature from dataset D d and the abnormal head w a D d to be at least an angular margin of m greater than the normal head w n D d that enlarging the gap between normal and abnormal.

Denoting the topK abnormal scores from the head of dataset D d as Ω k ( s D d ) = { s a D d ,i , s n D d ,i } i=topk indices , rewriting Eq. 2 as an angular margin-based regression problem results in Eq. 4. Similar to Null-MIL, the loss is computed by the head associated with the input dataset, while scores from other heads have no impact on updating the model's weights.

For the test, since the normal and abnormal heads are trained through angular margin learning, the abnormal score is shown in Eq. 5.

̸

When the source domain of the pre-trained general model is different from the target domain, we determine the final score to reflect conflicts by taking the maximum normal and abnormal score from multiple heads.

Unseen domain adaptation. After multi-domain learning, when a new target dataset or unseen condition appears, the final score is computed based on the similarity between the embedding feature of the input video and each source domain's head classifier. Therefore, it operates by considering domain similarity, which can address the performance degradation issue that occurs due to the scene discrepancy discussed in Sec. 2.

Complexity. Only the final layer is added based on the number of datasets, with the head's weight denoted as w D d ∈ R T × 1 , which is a very small proportion of the entire model. Comparing a single and multiple heads (6 datasets), the training times are 2.68 and 2.81 hours, and the inference times are 0.158 and 0.164 milliseconds per snippet, respectively, indicating a negligible increase in complexity.

3.3 Abnormal Conflict (AC) classifier

In the domain-agnostic feature extraction phase, it is crucial to capture general features that can handle every domain. While we have divided the heads to avoid abnormal conflicts in multiple source datasets, the agnostic part extracts features from all datasets using a single branch, and inconsistent labels cause confusion. Therefore, we propose an Abnormal Conflict (AC) classifier for learning the final embedding feature F DA that passes through the detector heads.

In Fig. 2, each classifier head distinguishes between D d and D d c , while the AC classifier is intended to distinguish between elements that are abnormal or normal across all source datasets (green area) and elements that represent conflicts (yellow area) within the Venn diagram. The AC classifier takes the embedding feature F DA as input, followed by two FC layers, to predict the final conflict score s AC . In Eq. 6, the AC label is generated based on the scores of all abnormal heads, where y AC i = 1 if the deviation between the scores is above a threshold τ .

The loss of the AC classifier, denoted as L AC and calculated using cross-entropy as follows:

The total objective function is L = L NullAng -MIL + λ L AC .

During the testing phase, the auxiliary branch AC classifier is eliminated, and the output is calculated for each input snippet v i as the final Abnormal Score i .

4 Experimental Results

4.1 MDVAD benchmark

As shown in Table 1, VAD comprises six representative datasets with diverse settings and volumes. The MDVAD is a task aimed at addressing domain shifts between datasets, and each dataset included in the MDVAD benchmark should be structured so that it is not biased toward any particular dataset or anomalies. Consequently, datasets should have an equal volume in the training set and encompass various abnormal categories and criteria. To achieve this, we sampled each dataset to align with the dataset with the smallest volume, ensuring that each abnormal category has a similar proportion

MDVAD Benchmarks

Source

Target

UCFC

XD

LAD

UBIF

TAD

ST

UCFC

77.93

67.07

74.23

71.10

67.95

50.84

XD

65.88

83.23

72.63

70.58

63.46

49.55

LAD

60.23

71.37

83.82

64.73

68.47

47.27

UBIF

74.11

70.49

67.69

92.62

68.03

56.47

TAD

56.22

45.05

58.67

67.83

90.75

41.92

ST

50.49

61.53

60.91

56.58

41.60

90.79

61.39

(77.93)

63.10

(83.23)

66.83

(83.82)

66.17

(92.62)

61.90

(90.75)

49.21

(90.79)

(In-domain)

Training:

Single-source dataset / Testing: Target dataset

Out Avg.:

Average of cross-domain results.

MDVAD Benchmarks

Target

UCFC

XD

LAD

UBIF

TAD

ST

Single-domain

61.39

(77.93)

63.10

(83.23)

66.83

(83.82)

66.17

(92.62)

61.90

(90.75)

49.21

(90.79)

61.43

(86.52)

(In-domain)

E1: Held-in

Head

AC

UCFC

XD

LAD

UBIF

TAD

ST

-

80.05

83.77

86.01

85.76

88.92

88.82

85.56

MIL

!

80.11

83.91

85.15

87.72

90.05

87.98

85.82

Null-MIL

-

79.01

81.96

85.08

93.06

90.57

91.04

86.79

(Ours)

✓

79.15

82.96

85.82

92.41

91.16

89.67

86.86

NullAng

-

76.32

82.74

82.32

92.30

91.82

91.26

86.13

MIL

(Ours

)

✓

77.21

82.09

83.88

91.90

91.36

91.12

86.26

Training:

All six datasets /

Testing:

Target dataset

Column-wise coloring with increased intensity for higher values

and conducted 3-fold experiments. Additionally, to handle small datasets like TAD and ST, which have minimal volumes, we combined them with the traffic dataset CADP [41] and campus dataset NWPU [3], respectively, by reorganizing their training sets. The volume of MDVAD benchmark is unified at 386 videos per dataset. More details provided in the supplementary material (Sec. B).

4.2 Empirical studies

This section presents empirical studies as follows: first, single-domain results for comparison, followed by the results of four evaluation protocols for the MDVAD benchmark. The models are trained on the training set of the source domain and evaluated on the test set of the target domain under the following settings: Held-in (E1), Leave-one-out (E2), Low-shot adaptation (E3), and Full finetuning (E4). The Area Under Curve (AUC) is used as the evaluation metric. The hyper-parameters are T = 32 , λ = 10 , τ = 0 . 3 , and m = 0 . 3 . Since the single-head baseline cannot assign AC labels using Eq. 6, pseudo-labels for the AC classifier are assigned based on the range of predicted abnormal scores. All detailed implementations, more results, and discussions are provided in the supplementary material.

Single-domain results. In the preliminary experiment, we examine the performance of the MDVAD benchmark under single-source and single-target conditions using a single-head MIL baseline. Comparing the in-domain (diagonal elements) results in Table 2 (using the original training set) and Table 4 (using the sampled training set), we observe that the performance diminishes when datasets, other than TAD and ST, are reduced during sampling. Conversely, TAD and ST show improved performance, benefiting from increased diversity and data volume due to reorganization with CADP and NWPU for TAD and ST, respectively. The cross-domain (off-diagonal elements) results in Table 4 demonstrate consistent trends with both the MDVAD and VAD benchmarks. These trends primarily arise from challenges related to abnormal conflict and scene discrepancies, indicating that the issues are confined to in-domain settings. Consequently, to uphold in-domain performance across diverse datasets and effectively address issues related to abnormal conflict and scene discrepancies, we emphasize the essential role of multi-domain learning in developing a general model for V AD.

4.2.1 Held-in results (E1)

In this section, we discuss the held-in evaluation results, where models are trained on all datasets of MDVAD, and the test set of each source dataset is used as the target. Table 5 presents the results obtained by training with the MIL baseline with a single-head, which is commonly used in traditional WVAD, and the proposed Null(Ang)-MIL baseline composed of multiple heads. Because UCFC, XD, and LAD exhibit low abnormal conflicts, numerous similarities in abnormal categories, and minimal scene discrepancy, there are great performances with the single-head MIL baseline. However, when TAD and ST are targets, the results demonstrate that the Null(Ang)-MIL baselines, which extract general domain features in the domain-agnostic part and avoid conflict by assigning Null values with multiple heads, outperform the MIL baseline.

Handling multi-domain with a single Null(Ang)-MIL baseline. The average performance of single Null-MIL and NullAng-MIL is comparable to or even better than the average performance of individual in-domain models trained for each domain (86.52%). Moreover, the NullAng-MIL baseline, which effectively performs inter/intra-class learning through cosine angular margin while avoiding conflict between each head, achieves superior results, with 91.82% (TAD) and 91.26% (ST) compared to the performances of single in-domain results of 90.75% (TAD) and 90.79% (ST),

Training: Five datasets except the target dataset

Testing:

Target dataset

highlighting the effectiveness of learning with diverse and abundant data to enhance model generality. Furthermore, additional performance boosts are observed across all baselines with the auxiliary AC classifier to make the model conflict-aware.

4.2.2 Leave-one-out results (E2)

The E2 experiments, in Table 6, are conducted on the leave-one-out setting among multiple source datasets and evaluated on the unseen target dataset unlike E1. Each column represents the outcomes of models trained with the target dataset excluded. Compared with the held-in results, the evaluation results conducted without prior knowledge of abnormal boundaries for unseen target data lead performance gap between E1 and E2.

Effectiveness of multi-domain learning for unseen target domain. When comparing single-domain learning and multi-domain learning in the results of unseen target evaluation (Out Avg. in Table 6), it is evident that the performance with multiple datasets is significantly superior. Effectively addressing conflict and discrepancy issues, and learning from diverse and complex situations across multiple domains, leads to the development of a general model with better performance on unseen domains.

4.2.3 Low-shot adaptation results (E3)

In E3, we delve into low-shot learning, examining how well the model trained with multiple sources in the leave-one-out experiments (E2) adapts to unseen targets. In this experiment, we utilized 10% of the target training set for low-shot learning (in a 3-fold setting). For datasets with abnormal categories, an equal number of data per category are randomly sampled, while for uncategorized datasets like UBIF and ST, random sampling is employed.

Importance of general pre-trained baselines for adaptation. In Table 7, comparing with Table 6, notable performance improvements are observed across all baselines for target samples with severe domain conflicts and gaps, such as UBIF, TAD, and ST, even with a limited volume of training samples. This emphasizes the importance of building a general model to adapt on unforeseen targets. While the MIL baseline with a single head holds an advantage in adapting specific single head when target samples are appropriately selected, it may become dependent on specific samples and is susceptible to overfitting. On the other hand, NullAng-MIL, with conflict-aware learning and training on diverse domains, outperforms other methods in adaptation performance.

4.2.4 Full fine-tuning results (E4)

Multi-domain models superior to specific single- domain models. Table 8 shows the full fine-tuning results of single-domain and multi-domain trained baselines. The single-domain models reflect results from training and testing within a specific dataset. When evaluating multi-domain baselines trained on all datasets in the held-in (E1) setting, it demonstrates performance comparable to single-domain models that perform well by fitting to a single dataset. However, after full fine-tuning on the target dataset, the E1 model achieves superior performance in most cases. The E1

MDVAD Benchmarks

Target

UCFC

XD

LAD

UBIF

TAD

ST

E3: Low-shot Adaptation

Head

AC

-

75.19

68.20

79.18

82.13

82.80

71.65

MIL

!

72.52

71.00

76.69

82.34

78.72

74.88

Null-MIL

-

67.55

60.32

75.11

75.97

62.29

57.72

(Ours)

!

70.57

66.40

73.58

81.39

71.12

63.02

NullAng

-

77.76

70.67

74.86

83.44

78.57

71.81

MIL

(Ours)

!

78.99

75.80

77.82

85.75

84.06

76.23

Training: E2 + a few target samples

Testing: Target dataset

Finetuning: Target dataset / Testing: Target dataset

(a)

(b)

1.0

1.0

0.8

0.8

Ours

MIL

0.6

0.6

0.4

0.4

0.2

AC score

0.2

0°0

0'0

50

100

150

200

1.0

1.0

Ours

0.8

AC score

0.8

MIL

0.6

0.6

0.4

0.4

0.2

0.2

0.0

250

500

1000

1250

1500

1750

and E2 models, which are well-explored across multiple domains, outperform the models specifically trained on a single domain after full fine-tuning.

4.2.5 Summary

Null-MIL and NullAng-MIL. The Null-MIL baseline achieves the best performance (86.86%) in the Held-in setting (E1), even surpassing the in-domain average results (86.52%). However, in the Leaveone-out setting (E2), which involves unknown target domains, it shows suboptimal performance. This is because it is impossible to determine which head's output score among the source dataset heads is more reliable, so multiple head baselines employ the maximum score for unknown domains. Conversely, NullAng-MIL exhibits superior results in E2 due to its similarity-based score calculation. NullAng-MIL considers the cosine distance between the final feature and the weights of each source head. As a result, when calculating the final score, the head with high similarity to the target data is activated, yielding promising results. When pre-training the general V AD model without knowledge of the target, we observed that utilizing the NullAng-MIL baseline is beneficial. However, when conducting multi-domain learning with knowledge of the target, employing Null-MIL yields better results. We observed that when pre-training the general V AD model without prior knowledge of the target, utilizing the NullAng-MIL baseline proves beneficial and employing Null-MIL yields better results when knowledge of the target is available during multi-domain learning.

Role of the AC Classifier. In multi-domain learning, the proposed AC classifier serves as an auxiliary branch designed to predict whether there is an abnormal conflict between domains, thus making the model aware of conflicts. Since it is not used during testing, it does not impact the final model's cost but provides performance gains in most experiments. Notably, in E2 and E3, training with the AC classifier shows a significant boost effect, aiding in adaptation to unseen domains.

4.3 Discussions

Open-set VAD. We conducted experiments in an open-set scenario using UBNormal (UBN) dataset [1]. The UBN is a VAD benchmark proposed for openset scenarios to handle unexpected abnormal events. Both normal and abnormal events are available during training, but the anomalies that occur during inference belong to a distinct set of anomaly types (categories). Unlike other VAD datasets, UBN consists of synthetic videos to alleviate the difficulty of collecting abnormal event data in the real world. There are substantial abnormal conflicts and differences in the visual settings

of scenes compared to other domains. For multi-domain learning, we reorganized the training/ testing set of UBN to maintain the same number of normal and abnormal videos in the training set with MDVAD, which is uniformly sampled and balanced the same amount of videos from all domains. As

shown in Table 9, despite domain discrepancies and abnormal conflict, the model effectively handles multiple domain learning, demonstrating that general feature learning can adequately address unseen abnormal categories. Please refer to §. A5 for more details.

Comparison with WVAD models. Although this paper focuses on the analysis of multiple domain learning within the context of abnormal conflict issues, instead of exploring complex architecture designs for single-domain VAD models, we compare with various VAD models. Table 10 presents the results of other MILbased WVAD models, MMIL [43], ARNet [46], WSAL [30], and COMO [7] on the MDVAD task. Compared to the proposed baseline trained with the AC classifier, our method achieves the highest average AUC, particularly in datasets with severe abnormal conflicts and scene discrepancies, such as TAD and ST, in both the E2 and even more in the E3 settings. Various single-domain VAD models or backbones can be incorporated into the MDVAD task, showing a direction for future generalization work. Please refer to § A6.

AC classifier. The AC classifier helps the model learn conflict-aware features, providing a clearer understanding of abnormalities. The proposed framework is composed of Domain-

Agnostic Layers that learn general features across multiple domains, followed by Multiple Heads that predict abnormalities specific to each domain. When the Domain-Agnostic layers learn to perform AC classification, they capture whether the input snippets relate to abnormal conflict or not. From the Heads' perspective, these features are separated into abnormal conflict and non-AC in the feature space, allowing the Heads to apply different criteria (decision boundaries) when distinguishing between normal and abnormal instances. For example, in classifying abnormalities, non-AC scenarios can be addressed more straightforwardly, while abnormal conflict scenarios require a more careful exploration. Fig. 3(a) presents the results of the AC classifier of NullAng-MIL baseline from E1. It shows abnormal conflict scores for two scenarios: (Top) A car on the Sidewalk abnormal event in ST, which is normal in the UCFC. (Bottom) A normal situation of people shopping in a grocery store. Through the multi-domain learning, the AC classifier outputs a high abnormal conflict score for the top sample, demonstrating that the model has learned to be conflict-aware.

Qualitative results. Fig. 3(b) illustrates abnormal conflict scene: (Top) Bicyclist on Walkway , abnormal in ST but normal in other domains, and (Bottom) Pedestrian on Road , abnormal in TAD but normal in UCFC. In these scenarios, the MIL baseline trained with multiple domains predicts (Top) false negative and (Bottom) false positive, while ours adaptively handles conflicts across different domains.

5 Conclusion, Limitation, and Future Works

In this paper, we propose a new task called MDVAD, whose ultimate goal is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general V AD model. As a baseline, we propose a new multi-head framework with Null(Ang)-MIL loss and AC classifier. These modules effectively handle abnormal conflicts between domains and show meaningful results in the MDVAD benchmark with diverse evaluation protocols.

Instead of exploring complex architecture design of single-domain VAD models, this paper focuses on resolving abnormal conflicts from multiple domains. Various single-domain VAD models or backbones can be applied into our novel framework to address the MDVAD task, representing a valuable direction for future generalization research.

Acknowledgments

This work was partly supported by the NAVER Cloud Corporation and the National Research Foundation of Korea (NRF) grant funded by the Korea government(MSIT)(RS-2024-00456589).

References

Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 20143-20153, 2022.

Abhishek Aich, Kuan-Chuan Peng, and Amit K Roy-Chowdhury. Cross-domain video anomaly detection without target domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2579-2591, 2023.

Congqi Cao, Yue Lu, Peng Wang, and Yanning Zhang. A new comprehensive benchmark for semisupervised video anomaly detection and anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20392-20401, June 2023.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299-6308, 2017.

Yunpeng Chang, Zhigang Tu, Wei Xie, and Junsong Yuan. Clustering driven deep autoencoder for video anomaly detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16 , pages 329-345. Springer, 2020.

MyeongAh Cho, Taeoh Kim, Woo Jin Kim, Suhwan Cho, and Sangyoun Lee. Unsupervised video anomaly detection via normalizing flows with implicit latent features. Pattern Recognition , 129:108703, 2022.

MyeongAh Cho, Minjung Kim, Sangwon Hwang, Chaewon Park, Kyungjae Lee, and Sangyoun Lee. Look around for anomalies: Weakly-supervised anomaly detection via context-motion relational learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1213712146, 2023.

Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4109-4118, 2018.

Bruno Degardin and Hugo Proença. Iterative weak/self-supervised classification framework for abnormal events detection. Pattern Recognition Letters , 145:50-57, 2021.

Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. Advances in neural information processing systems , 32, 2019.

Mark Dredze, Alex Kulesza, and Koby Crammer. Multi-domain learning by confidence-weighted parameter combination. Machine Learning , 79:123-149, 2010.

Masoud Faraki, Xiang Yu, Yi-Hsuan Tsai, Yumin Suh, and Manmohan Chandraker. Cross-domain similarity learning for face recognition in unseen domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15292-15301, 2021.

Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14009-14018, 2021.

Mariana Iuliana Georgescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. A background-agnostic framework with adversarial training for abnormal event detection in video. IEEE transactions on pattern analysis and machine intelligence , 44(9):4505-4523, 2021.

Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1705-1714, 2019.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Mahesh Joshi, Mark Dredze, William Cohen, and Carolyn Rose. Multi-domain learning: when do domains matter? In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pages 1302-1312, 2012.

Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki, Sparsh Garg, Manmohan Chandraker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In European Conference on Computer Vision , pages 20-36. Springer, 2022.

Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7194-7201, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Shuo Li, Fang Liu, and Licheng Jiao. Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 1395-1403, 2022.

Yunsheng Li and Nuno Vasconcelos. Efficient multi-domain learning by covariance normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5424-5433, 2019.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6536-6545, 2018.

Wen Liu, Weixin Luo, Zhengxin Li, Peilin Zhao, Shenghua Gao, et al. Margin learning embedded prediction for video anomaly detection with a few anomalies. In IJCAI , volume 3, pages 023-3, 2019.

Zuhao Liu, Xiao-Ming Wu, Dian Zheng, Kun-Yu Lin, and Wei-Shi Zheng. Generating anomalies for video anomaly detection with prompt-based feature mapping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 24500-24510, 2023.

Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16 , pages 125-141. Springer, 2020.

Haohan Luo and Feng Wang. A simulation-based framework for urban traffic accident detection. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Hui Lv, Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Learning normal dynamics in videos with meta prototype network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15425-15434, 2021.

Hui Lv, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Localizing anomalies from weakly-labeled videos. IEEE transactions on image processing , 30:4505-4515, 2021.

Zhen Ma, José JM Machado, and João Manuel RS Tavares. Weakly supervised video anomaly detection based on 3d convolution and lstm. Sensors , 21(22):7508, 2021.

Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4293-4302, 2016.

Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance-motion correspondence. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1273-1283, 2019.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems , 30, 2017.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8119-8127, 2018.

Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271) , pages 59-66. IEEE, 1998.

Alice Schoenauer-Sebag, Louise Heinrich, Marc Schoenauer, Michele Sebag, Lani F Wu, and Steve J Altschuler. Multi-domain adversarial learning. In International Conference on Learning Representations , 2018.

Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16 , pages 68-83. Springer, 2020.

Ankit Parag Shah, Jean-Bapstite Lamare, Tuan Nguyen-Anh, and Alexander Hauptmann. Cadp: A novel dataset for cctv traffic camera based accident analysis. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) , pages 1-9. IEEE, 2018.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1): 1929-1958, 2014.

Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6479-6488, 2018.

Shengyang Sun and Xiaojin Gong. Hierarchical semantic contrast for scene-aware video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2284622856, 2023.

Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4975-4986, 2021.

Boyang Wan, Yuming Fang, Xue Xia, and Jiajie Mei. Weakly supervised video anomaly detection via center-guided discriminative learning. In 2020 IEEE international conference on multimedia and expo (ICME) , pages 1-6. IEEE, 2020.

Boyang Wan, Wenhui Jiang, Yuming Fang, Zhiyuan Luo, and Guanqun Ding. Anomaly detection in video sequences: A benchmark and computational model. IET Image Processing , 15(14):3454-3465, 2021.

Li Wang, Dong Li, Han Liu, Jinzhang Peng, Lu Tian, and Yi Shan. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2487-2494, 2022.

Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7289-7298, 2019.

Peng Wu and Jing Liu. Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Transactions on Image Processing , 30:3513-3527, 2021.

Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 322-339. Springer, 2020.

Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. A light cnn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security , 13(11):2884-2896, 2018.

Shuhan Yi, Zheyi Fan, and Di Wu. Batch feature standardization network with triplet loss for weaklysupervised video anomaly detection. Image and Vision Computing , 120:104397, 2022.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Chen Zhang, Guorong Li, Yuankai Qi, Shuhui Wang, Laiyun Qing, Qingming Huang, and Ming-Hsuan Yang. Exploiting completeness and uncertainty of pseudo labels for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16271-16280, 2023.

Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 178-193. Springer, 2020.

Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H Li, and Ge Li. Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1237-1246, 2019.

Yi Zhu and Shawn Newsam. Motion-aware feature for improved video anomaly detection. arXiv preprint arXiv:1907.10211 , 2019.

Yuansheng Zhu, Wentao Bao, and Qi Yu. Towards open set video anomaly detection. In European Conference on Computer Vision , pages 395-412. Springer, 2022.

Supplementary Material

Summary The supplementary material is organized in the following order. First, it provides an explanation of the characteristics of the Video Anomaly Detection (VAD) dataset and introduces the detailed configuration and experimental settings of the MDVAD benchmark. Following this, we present implementation details of the experiments and examples and explanations related to abnormal conflict. Lastly, we discuss the pseudo labeling equation, score comparison plots, and failure cases. Note that back-references in the supplementary material sections and tables are from the main manuscript.

Terminology

Domain/ Dataset: In this paper, 'domain' refers to the broader context or environment, while 'dataset' refers to the specific collection of data used within that domain.

Single-domain: Training and evaluation data are the same, as in traditional V AD research.

Multi-domain Learning/ General VAD: General VAD is the objective of detecting anomalies across various domains, while Multi-domain Learning is the method used to achieve this by training on multiple domains.

Multi-head Learning: The baseline method proposed in this paper for Multi-domain Learning.

Multi-task Learning: Performing multiple tasks with a single model, related to E1 by integrating various domain's anomalies into one framework.

General Pre-training: A methodology of pre-training on large data without target knowledge, related to E2 by training in a held-out setting and applying to an unseen target. Furthermore, general pre-trained models adapt well to low-shot learning in E3 and finetuning in E4.

A VAD Datasets

As mentioned in §2, the VAD task encompasses diverse datasets, as delineated in Table A1.

UCFC [43], the prominent large-scale dataset for the Weakly-Supervised approach, consists of untrimmed surveillance videos. It categorizes anomalies into 13 classes related to public safety,

16.4%

Road Spills

2.2%

Ped. on Road

1.5%

Traffic Else

3.5%

Illegal Occup.

1.5%

Trampled

1.3%

Vandalism

5.1%

Shooting

3.5%

Abuse

1.5%

Accident

12.5%

Explosion

5.8%

Fighting

22.3%

Robbery

1.3%

(a)

XD: Normal

ST: Abnormal

(b) Normal video from XD

XD: Normal

ST: Abnormal

(c)

'Bicyclist'

abnormal video from ST

ranging from crimes such as abuse and arson to more complicated scenarios like stealing and shoplifting. The dataset consists of a total of 950 normal and abnormal videos each, with the training set maintaining a balanced distribution of 800 and 810 samples for normal and abnormal videos, respectively.

The XD dataset [51] stands out as the most extensive dataset with a variety of environments, including CCTV, movies, hand-held cameras, and car camera settings. It incorporates six categories of violence, such as abuse, riot, and explosion, defining anomalies. Within the dataset of 4,754 videos, there are 2,049 normal and 1,905 abnormal training videos.

The LAD dataset [47] is defined by a detailed categorization of 14 anomaly classes, including crash, fire, and violence, consisting of 2,000 videos, with 958 normal and 482 abnormal videos.

The UBIF dataset [9] is composed of various fighting-related anomalies extracted from our daily life videos. This dataset consists of 1,000 videos without a specific categorization for the fighting class, and the training set consists of 757 normal and 176 abnormal videos, leading class imbalance issue.

The TAD dataset [19] is structured with traffic surveillance and 1st-person videos, encompassing anomalies related to traffic, such as accidents, illegal turns, and road spills. This dataset, totaling 500 videos, has a small volume with 210 and 190 numbers of normal and abnormal training videos.

The ST [24] serves as a campus surveillance dataset, capturing anomalies occurring on pedestrian walkways. It exhibits a heightened sensitivity to abnormal events, ranging from fighting and throwing to running and jumping. With only 175 normal and 63 abnormal videos in the training set out of a total of 437 videos, there is a notable imbalance between classes, and the quantity of data is insufficient for learning. Abnormal categories in the training set cover all anomalies in the testing set on each dataset except for ST (ST remains indeterminate because of the absence of category information), which means evaluations are exclusively conducted for seen abnormal events.

Examining the dataset samples in Table A1 reveals that TAD and ST exhibit a visual gap compared to the other datasets, UCFC, XD, and LAD. Such gap is attributed to visual elements such as scene settings and conditions that impact the visual characteristics of scenes, which is the scene discrepancy discussed in §2. While LAD (some videos) and ST are recorded in square or campus settings to establish real-world scenario datasets, the remaining datasets are constructed online, such as YouTube and LiveLeak using text search queries. These datasets also exhibit a significant domain distance, as indicated in Table 3. As such, the criteria for defining abnormalities and the visual characteristics of scenes vary across different datasets.

B MDVAD Benchmarks

B.1 Configuration

The five datasets introduced in Table A1 have different training set volumes, leading to challenges in handling multiple dataset learning, where larger datasets may overwhelm the learning process.

For instance, there is a significant disparity in training set volumes, with TAD (400 videos), ST (238 videos), and XD (3954 videos) differing by almost 10 times. Additionally, the number of abnormal videos in XD differs roughly 10 and 30 times compared to TAD and ST, respectively.

The objective of the MDVAD task is to avoid abnormal conflict and build a general model through balanced learning across diverse situations and anomalies. To control for other variables, we uniformly sampled and balanced the same amount of videos from all domains, equal to the number of videos in the smallest domain. Since TAD and ST have an insufficient number of videos for training, we recombine them with two new datasets, CADP [41] and NWPU [3], respectively, to balance the numbers. CADP is a dataset for traffic accident analysis, and it shares similarities with TAD in terms of setting and abnormal definition. Hence, following the approach in [28], we compose TAD's training set by adding abnormal videos from CADP. Additionally, incorporating the recently released NWPU dataset, which collects various anomalies of pedestrians on campus, we augment ST's dataset with NWPU data. For both datasets, we conducted random sampling for normal and abnormal videos, considering the smallest sample size, resulting in a number of 210 normal and 176 abnormal videos. In this process, for datasets with information on abnormal classes, we compose to achieve a uniform number of instances per abnormal category on each dataset. For datasets with a sufficient amount, we organize 3-fold sets to minimize overlap. All reported performance metrics are the average results of evaluations conducted on the target dataset after training on each fold. For example, in the E2 leave-one-out setting, the average results are derived from models trained on each fold and evaluated on an E3 low-shot adaptation set for the corresponding fold.

The MDVAD benchmark incorporates the datasets introduced in Table A1, including abnormal categories: Abuse, Arrest, Arson, Assault, Accident, Burglary, Explosion, Fighting, Robbery, Shooting, Stealing, Shoplifting, Vandalism, Drop, Hurt, Fall Into Water, Falling, Destroy, Fire, Violence, Crowd, Thiefing, Panic, Loitering, Trampled, Illegal Turns, Illegal Occupations, Retrograde Motion, Traffic Else, Pedestrian on Road, Road Spills, and Campus Else . The circular chart in Fig. A4(a) illustrates the distribution of categories. For datasets without abnormal categories like ST (named as Campus Else ), TAD (where additional categories are labeled as Traffic Else ), and UBIF (where categorization is not specified and all fall under Fighting ), we provide appropriate category names. The averages across the 3-folds are calculated, and in the case of XD with multiple labels per video, we count the number of categories per video. The Fighting predominates in the majority of datasets, occupying the largest portion, followed by categories like Accident , Explosion , and others.

B.2 Experimental settings

VAD is predominantly evaluated using the Area Under the Curve (AUC) score based on abnormal scores, focusing on the trade-off between sensitivity and specificity, emphasizing the false positive rate (FPR). While XD employs the Average Precision (AP) metric to balance precision and recall and concentrate on positive samples for direct comparison in cross-dataset and held-in/out evaluations, and emphasizing False Positive Rate (FPR) (detailed explanation is in Section D), we utilize the AUC score across all evaluations.

The MDVAD benchmark comprises four protocols: Held-in (E1) involves multiple-source learning with all datasets, Leave-one-out (E2) employs a leave-one-out approach where a dataset left out during training is used for evaluation, low-shot adapation learning (E3) is training on the remaining datasets, and Full fine-tuning (E4) that fine-tunes a multi-domain models on single-domain datasets. In the E3 setting, around 10% of the training set, 20 videos, is randomly sampled for low-shot examples. For datasets with anomaly category information, low-shot examples are selected uniformly to ensure an even distribution within each category. To mitigate the impact of randomness, the low-shot setting is also conducted with 3-fold cross-validation, obtaining low-shot examples from each fold. In all cases, including MIL, Null-MIL, and NullAng-MIL models, all weights except for the pretrained backbone are updated. Additionally, for abnormal videos, the score from the AC classifier is multiplied to the loss value.

C Implementation Details

All experiments are conducted on the MDVAD benchmark. We use I3D backbone feature which is pretrained on Kinetics dataset [4] with C = 2048 dimension of RGB features. During the training phase, we conduct consistent experiments with a batch size of 32, utilizing the Adam optimizer [20]

with a learning rate of 5 e -5 and weight decay of 5 e -4 . The input image size is set to 224 × 224 and following [21, 31, 45, 7], we perform 10-crop augmentation when extracting backbone features. The backbone is used the same as RTFM [45] and CoMo [7]. For input data, 16 frames are stacked for a snippet, and T snippets are uniformly sampled in the video during training. For testing, all snippets pass through the model, and the output score is assigned to all frames within the snippet. For MIL learning, the TopK snippets, where K equals 10% of the total, i.e. , K = 3 , are utilized. In the case of a single-headed MIL model in Table 5 (E1) setting, since pseudo-labels cannot be generated through multiple heads, the AC classifier is trained with a label that y AC = 1 for a score s a D 1 within the [0 . 4 , 0 . 7] range. When computing EMD metric in §2, we use a pretrained I3D backbone model for extract embedding features of each dataset and measure the distance between feature vectors. We use a test set with frame-level annotations for abnormalities. All models and experiments are implemented and evaluated end-to-end using PyTorch [34] with a single NVIDIA V100 GPU.

In weakly-supervised VAD, maintaining class balance is pivotal, so the number of normal and abnormal input videos within a batch is kept equal for the MIL-based learning. Similarly, during multiple source learning, an equal number of input videos from each source domain within a batch is crucial. Despite each head is trained independently, the final feature F DA is extracted from a shared domain-agnostic part, necessitating the balance in the number of source domains. Moreover, when training the AC classifier, in order to address the imbalance between the number of abnormal conflict snippets and all normal/abnormal snippets, we apply focal loss [23] to L AC . During the testing phase, the AC classifier is eliminated, and the output is calculated for each input snippet v i as the final Abnormal Score i .

Table A4 provides information about the layers of the baseline model. Conv1d , fc , Max , BN , and DO represent 1D convolutional layer, fully connected layer, element-wise max activation, 1D batch normalization [16], and dropout [42], respectively, with a dropout probability set to p = 0 . 7 . In Conv1d ( c, k, s ) and fc ( c ) , c , k , and s denote the channel size, kernel size, and stride size, respectively. T input snippets pass through the backbone, resulting in a backbone feature B with a shape of ( N,T,C ) , where N represents the batch size. The feature aggregation layer undergoes channel doubling followed by channel squeezing with element-wise max operation, resulting in an output feature F agg with a shape of ( N,T,C ) . Subsequently, the temporal aggregation layer outputs F DA with a shape of ( N,T,C/ 2) . These embeddings are then input into the AC classifier and fc layer.The AC classifier, composed of consecutive fc layers, outputs AC scores s AC for each snippet with a shape of ( N,T, 1) . Additionally, the final feature F , processed through an fc layer, is input into multiple domain heads w D d for NullAng-MIL learning. During this process, both the feature and weights are normalized with respect to ∥ F ∥ and ∥ w D d ∥ for each domain.

D Abnormal Conflict

In Fig. A4, (a) represents a video labeled as normal in UCFC, while (c) is labeled as abnormal in TAD, both depicting a situation where people are present on the road. Similarly, (b) is labeled as a normal video in XD, whereas (d) is considered abnormal in ST; both scenes show bicycling on a

walkway. We define such instances, where the criteria for abnormalities differ across datasets, as abnormal conflicts .

While cross-domain evaluation in Table 2 reveals a domain gap, it is difficult to attribute as abnormal conflicts directly. In Table A2, we calculate the average of the relative False Positive Rate (FPR) and relative False Negative Rate (FNR) to quantify the abnormal conflict between domains.

where i and j indicate source and target domain, repectively. First, FPR and FNR refer to the rate of misclassifying abnormal as normal and normal as abnormal in the scenario of learning from a source and testing on a target, respectively. However, since these include the FPR and FNR caused by the incompleteness of the model itself, this cannot accurately reflect the abnormal conflict between the source and target dataset. Therefore, we subtract the FPR and FNR when the source domain is the same as the target domain ( FPR j,j and FNR j,j ) from the cross-domain FPR and FNR to calculate the relative FPR and FNR caused by the domain shift. Table A2 shows similar trends compared to Table 2, where values are not normalized. This shows that abnormal conflicts between domains are relevant to the reduction in generalization ability that can occur when transferring between domains.

E Related Works

As mentioned, proposing a new VAD model is beyond the scope of this paper, this part is organized to aid understanding.

Unsupervised VAD Defining anomalies precisely is challenging as they can vary based on criteria and sensitivity, and it is impossible to categorize and collect datasets for every scenario. For this reason, Unsupervised VAD methods [33, 5, 24, 15] have been developed to learn normal patterns using only normal training data and then detect anomalies as scenes with low normality during the test phase. However, this approach leads to a significant bias towards the normal samples, causing the detector to misclassify normal patterns that differ from the learned data as abnormal events, resulting in a high false alarm rate. To address this issue, Weakly-supervised V AD (WV AD) approaches [55, 44, 43, 50, 21, 7] have been introduced to differentiate between normal and abnormal events with minimal supervision using video-level annotations, achieving significant performance improvements while incurring lower labeling costs compared to frame-level annotations.

Weakly-supervised VAD MMIL [43] is the first WVAD method to pose the MIL ranking approach as a regression problem that achieves significant performance gains. Various learning methods, including magnitude feature learning [45], self-training approaches through pseudo labels [13, 21], and distance learning [53], have been explored alongside traditional MIL methods. Furthermore, subsequent approaches have focused on capturing features by focusing temporal or motion information [58], leveraging additional audio signals [51], learning relations between temporal [50] or motion and context information [7]. These methods differentiate complex anomalies through contextual information; for example, the same motion may be classified as normal or abnormal depending on the surrounding environment. PFMP [26] proposed a method to utilize virtual data anomalies to reduce scene discrepancy with real-world data, addressing the issue of data scarcity in V AD. However, these approaches still cannot address abnormal conflicts, where the same situation is labeled differently as abnormal or normal based on the criteria of different datasets.

Generalization for VAD Afew studies have been conducted to generalize V AD. Domain adaptation refers to evaluating a new domain that was not seen during training, and some works have proposed using some data from the target domain [27, 29] or not using it at all [2]. We also experiment with domain adaptation, but the difference is that we use multiple domains compared to existing studies that only use a single domain during training. Open-set recognition aims to work well even in untrained classes (abnormal categories in VAD). Compared to our research involving multiple domains, Open-set VAD [59, 1] focuses on learning and evaluation within a single domain.

Multi-domain learning Multi-Domain Learning (MDL) refers to a method for learning datasets from multiple domains with various distributions together. MDL originated from natural language processing [11, 17] and has been applied to various tasks in computer vision [36, 37, 22, 32, 49,

(a)

(b)

(c)

(d)

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

ACscore

0.2

ACscore

0.0

20

60

80

100

0.0

10

OZ

OE

40

50

70

1.0

1.0

ACscore

ACscore

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

.5

5.0

7.5

10.0

12.515.017.5

20.0

48, 56, 18]. The difficulty of MDL arises when the difference in distribution between domains is significant, and learning domain-invariant features is the basic approach [39]. In the fields of image classification [36, 37, 22], object tracking [32], detection [49], and segmentation [48], methods for learning multiple domains with multiple heads have been proposed. After that, methods have been proposed to have a unified label space [56, 18], but in this process, the problem that the labels are different for each domain arises. [18] solved the label conflict problem by proposing classindependent loss using the Null class strategy. On the other hand, V AD has only two classes, normal and abnormal, so there is no explicit label conflict problem, but there is an implicit abnormal conflict problem where the definition of a label is different for each domain.

A similar method is multi-task learning [54], which considers differences between tasks rather than domains. In addition, multi-source domain adaptation [35] utilizes data from the target domain, and domain generalization [10, 40, 12] requires adaptation to an unseen target domain. The difference with these tasks is that they basically share the label space between the source and target domains, while MDL for VAD has different label definitions.

F Discussions

F.1 Pseudo label of AC Classifier

The purposes of the AC Classifier is the facilitation of feature learning with consideration of the discrepancies across multiple domains. In cases where snippet is entirely normal or abnormal across all datasets, the AC score is low, making it difficult to directly use it for the abnormal score. Noisy samples can indeed have a negative impact on training in Eq. 6. However, in cases where even one of the multiple datasets has a different definition of normal and abnormal, it is crucial to sensitively detect conflicts in such samples, leading us to decide that using a difference is preferable. In Table A3, we conducted ablation studies with Std. ( τ = 0 . 1 ) and a fixed value ( y AC i = 1 where 0 . 3 < s a D d ,i < 0 . 7 ) which revealed no significant differences between the methods.

F.2 Abnormal conflict score

Fig. A5 depicts the output abnormal conflict scores from the AC classifier of the E1 held-in model. While all events are considered normal in UCFC, (a) and (b) correspond to the Pedestrian on Road abnormal category in TAD (as shown in Fig. 1(c)). Therefore, unlike (c) and (d), the normal situations of people shopping in a grocery store, (a) and (b) exhibit elevated abnormal conflict scores.

F.3 Abnormal score

Fig. A6 compares the abnormal scores of our model, trained with NullAng-MIL + AC classifier, and the MIL model trained with MIL loss on a single head in the E2 multiple dataset setting. In scenario (a), before the explosion, an oil truck passes, causing the MIL model to generate a high abnormal

0.8

0.6

auoxS

0.4

0.2

Ours

MIL

500

1000

1500

2000

2500

3000

Framet

1.0

Ours

0.8

MIL

0.6

aJoxS

0.2

M

500

1000

1500

2000

2500

Frames

1.0

sino

0.8

MIL

0.6

0.4

0.2

0.0

50

100

Frames

150

200

250

300

1.0

0.8

MIL

0.6

0.4

0.2

0.0

50

100

150

200

250

Framet

score as a false alarm, while our model accurately predicts only the explosion abnormal situation. In scenario (b), where a person loitering the office, the MIL model outputs a high abnormal score, in contrast, our model that trained with multiple heads to avoid domain conflicts categorizes only the arrest defined in UCFC as an abnormal event.

Scenarios (c) and (d) involve abnormal videos in the ST dataset where a running person or car appears on the walkway. Because these scenarios are classified as normal scenes in other datasets during training, the MIL model considers them as normal. However, our model, recognizing them as abnormal scenes corresponding to ST, outputs high abnormal scores. Thus, when learning from multiple datasets, it is crucial to avoid conflicts by simultaneously learning domain-agnostic features with the AC classifier and predicting domain-specific scores using multiple heads according to each dataset's criteria.

F.4 Multi-domain learning on Virtual dataset

The UBNormal (UBN) dataset [1] is a VAD benchmark proposed for open-set scenarios to handle unexpected abnormal events, where the abnormal categories in the train set and test set do not overlap. Unlike other VAD datasets which were collected from the real world, UBN consists of synthetic videos which leads

0.8

MIL

0.6

Score

0.4

0.2

0.0

0

500

1000

1500

Frames

2000

2500

3000

1.0

0.8

MIL

0.6

auooS

0.4

0.2

0.0

0

50

100

150

200

250

300

Framet

1.0

Ours

0.8

MIL

0.6

auoog

0.4

0.2

0.0

0

100

200

0OE

400

abnormal conflicts and differences in the visual settings of scenes. It comprises

543 videos across 29 scenes with 22 types of anomaly categories. Same as MDVAD benchmark, we reorganize Training set with 210 number of normal videos and 176 number of abnormal videos where training anomaly categories are falling, dancing, walking injured, running injured, crawling and stumbling walk and testing anomalies are running, having a seizure, laying down, shuffling, walking drunk, people and car accident, car crash, jumping, fire, smoke, jaywalking and driving outside lane .

The Table A5 shows the performance in each target domain when trained with MDVAD and UBNormal under the E1: held-in setting. In the single-head MIL baseline, there is a performance drop when trained with MDVAD+UBN, indicating difficulty in handling AC. However, the model trained with multiple heads and the AC classifier shows improved results. By leveraging the virtual dataset, we can overcome data limitations and create a general model capable of handling diverse and complex scenes.

F.5 Baseline models

We conducted additional experiments using the WSAL [30] model as a baseline to validate our proposed method on the MDVAD benchmark's four protocols. The Table A6 presents the results, where all settings are consistent with the other experiments in the paper. The results show the addition of multi-head learning with NullAng-MIL and the AC Classifier to the WSAL model brings performance gains which effectively operate across

different baselines. Notably, in the E3 and E4, which shows the target adaptation of the pre-trained general model, the method yielded competitive results. Future studies can be delve deeper into studying more sophisticated baselines for resolving AC within the MDVAD setting.

F.6 Failure cases

Fig. A7(a) presents an abnormal test video from UCFC corresponding to Arrest anomalies. However, this video includes scenes of Stealing , leading to the subsequent arrest and physical fighting. Both the MIL and our models output high abnormal scores for scenes other than the Arrest , as they correspond to Stealing and Fighting abnormal categories in UCFC. Models predict accurately in categories other than Arrest , but this failure is observed because of a single abnormal label for each video.

In Fig. A7(b), an abnormal event from ST is considered normal by both models, as other datasets are generally classified as normal. Similarly, in Fig. A7(c), the skateboarder is deemed normal, resulting in a false negative failure case. However, these situations are challenging to identify as severe abnormalities from a general perspective. This failure case is able to be addressed through low-shot learning or fine-tuning based on ST's criteria.

F.7 Broader Impacts

The development of Multi-Domain Learning for Generalizable VAD enhances the reliability and accuracy of security and surveillance systems across diverse environments, thereby improving public safety. This research advances AI and machine learning by addressing domain adaptation challenges, promoting robust and generalizable systems. Additionally, it underscores the importance of ethical considerations and privacy, ensuring that advanced surveillance technologies are deployed responsibly and transparently.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In Abstract, § 1, and § 1.1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In § 5 and F.6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In § 4 and § C

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the data and code when the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In § 4, § C, and Table A4

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not reprot the error bars but all reported experimental results are average of 3-fold setting.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In § C

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have complied with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In § F.7

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In § A

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new task named Multi-Domain Learning Video Anomaly Detection, which aims to learn a general VAD model across domains. The work finds that abnormal conﬂict is a critical challenge in the task. Then, the work establishes a new benchmark, designs an effective baseline and conducts extensive experiments to investigate this challenge. The results shown on the benchmark demonstrate that the abnormal conﬂict is alleviated.\n\nSTRENGTHS:\n1. The work proposes a new task, which is interesting. \n2. The work establishes a new benchmark to evaluate the new task. \n3. The motivation of the proposed baseline, i.e., abnormal conﬂict, is clear and makes sense.\n\nWEAKNESSES:\nI have some concerns about the proposed method, and I think more comparison experiments are needed to demonstrate the effectiveness. Despite this, I think the abnormal conflict issue is interesting, thus I am willing to raise my rating if my major concerns are addressed. My concerns are as follows:\n\n1. Why the proposed Abnormal Conﬂict (AC) classifer can address the abnormal conﬂict problem? Why the label is determined by the discrepancy in Eq. (6)? It seems that there are some mistakes in the formula (inconsistent with that in Fig. 2). \n2. I would like to see the results of more baselines, in addition to MIL, Null-MIL and NullAng-MIL. \n3. More detailed discussions about related works are needed, e.g., virual video anomaly detection datasets [1] and related techniques utilizing virtual datasets [2]. \n\n[1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022\n\n[2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023\n\nQUESTIONS:\nSee the Weakness part.\n\nLIMITATIONS:\nThe paper has discussed the limitations and potential impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors proposed a new task called Multiple Domain VAD (MDVAD), along with a benchmark and new evaluation protocols. Authors' goal is to construct a general VAD model by conducting multi-domain learning while recognizing abnormal conflicts and exploring representations of general normality and abnormality. Authors introduced a baseline for MDVAD and proposed a new framework with multi-head to mitigate abnormal conflicts and proposed Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIL) losses for multi-domain training. Additionally authors suggested an Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts. Authors analyzed the primary issues of MDVAD and proposed a baseline for this new task.\n\nSTRENGTHS:\n1. According to the analysis, authors believed that the abnormal conflict and the scene discrepancy are the two main issues and designed a framework with multi-head to deal with these problems. \n\n2. Null-MIL and NullAng-MIL methods are designed for multi-domain learning, and an AC classifier is proposed for learning general features while abnormal conflicts exists.\n\n3. Authors provided sufficient experiment results for this task and create a new baseline.\n\nWEAKNESSES:\n1. The proposed framework with multi-head for multi-domain seems not flexible enough during the domain changes, such as adding a new dataset with extra abnormal conflicts. And for the abnormal conflicts, will the proposed method performs better compared to make anomaly categories classifications for all anomaly events type of all domains?\n2. In my opinion, traditional WS-VAD methods are designed to detect abnormal events in single domain without abnormal conflicts, and when abnormal conflicts exists, it will be better to use other paradigms such as temporal action localization or video grounding. And for the current WS-VAD datasets, the annotations are video-level, or even without category information, which is too weak for higher level anomaly detection. Training model with the current MDVAD paradigm is likely to not achieve good results.\n3. Maybe using visual-language model with multimodal alignment can deal with the above issues? These models contain more knowledges for more event categories and higher generalization ability, which are likely to have the ability to individually detect conflicting anomalies. Compared to multi-head regression, is VL alignment a better approach for MDVAD task?\n\nQUESTIONS:\nMy main questions are shown in the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript addresses the limitations of existing Video Anomaly Detection (VAD) models that are confined to single-domain learning. The primary contribution of the paper is the introduction of a new task called Multi-Domain Learning for VAD (MDVAD), which aims to develop a general model capable of identifying abnormal events across multiple domains. The manuscript conducts experiments using the MDVAD benchmark and demonstrates the limitations of traditional multi-domain learning. It shows the effectiveness of the proposed baselines in handling abnormal conflicts and achieving robust performance across multiple domains.\n\nSTRENGTHS:\n1.The manuscript proposes a new task, Multiple Domain Video Anomaly Detection (MDVAD), which solves the problem that the existing model is limited to a single domain and provides a new idea for the development of domain-generalized models.\n2.The MDVAD method proposes domain-specific multiple head mechanism and Null-Multiple Instance Learning Method (Null-MIL), which effectively solves the problem of anomaly conflict between different domains.\n3. The MDVAD method constructs a new benchmark containing six representative VAD datasets, which fills the gap of the lack of unified evaluation standard in multi-domain learning tasks.\n4. The MDVAD method designs four evaluation protocols (held-in, leave-one-out, low-shot domain adaptation, and full fine-tuning) to systematically evaluate the generalization ability of the model.\n\nWEAKNESSES:\n1. MDVAD introduces the domain-specific multi-head mechanism and the Null-MIL method, which increases the complexity and computational cost of the model, and may place higher demands on the computational resources in practical applications.\n2. The multi-domain learning task itself is difficult to train, and with the proposed method further increasing the complexity of training, MDVAD may require longer training time and higher technical requirements.\n3. Although the theoretical background and analysis are provided, the theoretical basis and derivation process of some of the methods of MDVAD are slightly weak and need to be further explored and verified in depth. Part of the theoretical analysis is based on specific assumptions, and these assumptions may not be fully valid in practical applications, affecting the applicability of the theoretical analysis.\n4. Although new benchmarks and assessment protocols are proposed, MDVAD lacks comparative experiments with other state-of-the-art methods, making it difficult to objectively assess the relative advantages of the proposed methods.\n\nQUESTIONS:\n1. What is the training difficulty of MDVAD? The introduction of domain-specific multi-head mechanism and Null-MIL method greatly increases the complexity and computational cost of the model, can it meet the real-time requirements in practical applications?\n2. Have the MDVAD and evaluation protocols been subjected to comparative experiments with other state-of-the-art methods in order to objectively assess the relative advantages of the proposed methods?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new task called MDVAD, the goal of which is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general VAD model. To achieve this, the authors expand the traditional single-head framework to multiple-head framework for learning different knowledge and design an AC classifier to handle abnormal conflicts. The experimental results prove the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper focuses on the problem of learning a generalizable VAD model, which is an important task.\n2. The experiments conducted by the author are relatively compr\n\nWEAKNESSES:\n1. This paper proposes a new task called MDVAD to achieve generalizable VAD by resolving conflicts in anomaly definitions. However, for any VAD application, the definition of normal or abnormal events should be explicitly determined according to the scenario requirements, rather than simply combining multiple datasets and resolving the abnormal conflicts. I find it difficult to understand under what practical scenario a VAD model trained using multiple datasets with abnormal conflicts is needed.\n2. The writing of this paper is not clear enough, where some necessary training and inference details are missed. For example, the normal head training mentioned in NullAng-MIL is confusing.\n3. This paper lacks a detailed description of the experimental setup. For example, if an anomalous event is determined to be a conflict, how should the model handle such an event?\n\nQUESTIONS:\n1. (referring to weakness 1) In practical applications, what kind of scenarios conform to the task settings of MDVAD proposed by the authors?\n2. (referring to weakness 3) During test phase, is it necessary to know which dataset the sample comes from? If a certain anomalous event is found to have conflicts in different datasets, how should it be handled? Do the multi-dataset evaluation and the test procedures for other methods use the same test data?\n\nLIMITATIONS:\nI do not recognize obvious potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}