Towards Multi-Domain Learning for Generalizable Video Anomaly Detection

Authors: myeongah  cho
taeoh  kim
minho  shim
sangyoun  lee
abnormal  head
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 16:12:31

Abstract: Most of the existing Video Anomaly Detection (VAD) studies have been conducted within single-domain learning, where training and evaluation are performed on a single dataset. However, the criteria for abnormal events differ across VAD datasets, making it problematic to apply a single-domain model to other domains. In this paper, we propose a new task called Multi-Domain learning forVAD (MDVAD) to explore various real-world abnormal events using multiple datasets for a general model. MDVAD involves training on datasets from multiple domains simultaneously, and we experimentally observe that Abnormal Conflicts between domains hinder learning and generalization. The task aims to address two key objectives: (i) better distinguishing between general normal and abnormal events across multiple domains, and (ii) being aware of ambiguous abnormal conflicts. This paper is the first to tackle abnormal conflict issue and introduces a new benchmark, baselines, and evaluation protocols for MDVAD. As baselines, we propose a framework with Null(Angular)-Multiple Instance Learning and an Abnormal Conflict classifier. Through experiments on a MDVAD benchmark composed of six VAD datasets and using four different evaluation protocols, we reveal abnormal conflicts and demonstrate that the proposed baseline effectively handles these conflicts, showing robustness and adaptability across multiple domains.

Main paper: 
 introduction: 
 Video Anomaly Detection (VAD) is identifying abnormal events in diverse scenarios depicted in a video and determining their temporal intervals at a frame-level. Nowadays, CCTVs are ubiquitous, recording every moment of life, which aids in preventing accidents and responding to crimes promptly. However, human monitoring of every situation is highly inefficient, requiring significant labor and resources. Consequently, extensive research has been conducted to automate VAD through deep learning by leveraging large amounts of surveillance data [33, 5, 24, 15, 55, 44, 43, 50, 21, 7] .In VAD research, Weakly-supervised VAD (WVAD) [55, 44, 43, 50, 21, 7] , which involves learning normal and abnormal events with minimal supervision of the video-level annotation and detecting abnormal events at the frame-level during testing, has been studied a lot recently. This paper focuses on the WVAD setting (denoted as VAD), and a summary of VAD research is explained in the supplementary material ( § E). Unlike the conventional VAD research, we first address the following three key questions:Q1: What is the problem with the existing VAD model? Most VAD models are trained in a singledomain where the training and evaluation dataset are the same. In the case of single-domain learning, application across different datasets (cross-domain evaluation) results in performance degradation as reported in [14, 6, 7] because VAD models are heavily influenced by the criteria for abnormality defined by each dataset.Q2: Why do we need a general VAD model? First, a single generalized model removes the need for multiple specific models for different domains, analogous to multi-task learning. Second, proper pre-training on multiple domains embodies generalized representation, and it can discriminate abnormal events according to their domain, leading to better performance in unseen target domains. Consequently, a general VAD model will be highly beneficial for applying VAD in practical scenarios.Q3: Is it possible to create a general VAD model? The general VAD model aims to handle multiple domains, but this is challenging because the definition of abnormal differs for each dataset, leading to conflicts between these abnormal events. For example, as shown in Fig. 1 (a), in one dataset, a pedestrian on the road is considered normal, while in another dataset, it is deemed abnormal. A general VAD cannot be solved with a naive muti-task learning because of this confusion across multiple domains, which this paper defines as 'Abnormal Conflict.' Therefore, for general VAD, it is necessary to be aware of these abnormal conflicts (yellow region in Fig. 1(b) ) and learn general normal (green region in Fig. 1(b )) or abnormal (red region in Fig. 1(b )) representations that are common across all domains.Our goal is to construct a general VAD model by conducting multi-domain learning while recognizing abnormal conflicts and exploring representations of general normality and abnormality. To achieve this goal, we introduce a new task called 1) Multiple Domain VAD (MDVAD), along with a benchmark and new evaluation protocols. MDVAD involves concurrent training on multiple VAD datasets, each with its own definition of abnormality. Specifically, the MDVAD benchmark comprises six representative VAD datasets with balanced sampling ( § 4.1). We also propose four evaluation protocols: held-in, leave-one-out, low-shot domain adaptation, and full fine-tuning. The held-in protocol is designed to evaluate the model's ability as a unified model like multi-task model, while the leave-one-out, low-shot domain adaptation, and full fine-tuning protocols are intended to access the model's capability as a general pre-training for an unseen target domain.As multi-domain learning is a novel concept in the field of VAD, we also introduce baselines and new learning methods. We design domain-specific multiple heads to mitigate abnormal conflicts and learn generality across domains. To facilitate multi-head learning without conflicts, we propose the 2) Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIL) losses, which activate only the output of the head corresponding to the input domain, assigning inactive heads with Null values to prevent confusion. Additionally, we suggest the 3) Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts, leveraging the variance in abnormal scores across the multiple heads (Fig. 2 green and yellow region in the Venn diagram). Through experiments with four protocols on the MDVAD benchmark, we reveal the limitations of multi-domain learning with abnormal conflict and demonstrate the effectiveness of our baselines in offering a generalized and adaptive model.Focusing on Multiple Domains. In this paper, we focus on solving the problems in multiple domains introduced above rather than solving problems in a single domain. Therefore, complex design of the backbone and head or achieving state-of-the-art performance in a single domain can proceed orthogonally with our research. Instead, this paper raises the issue of abnormal conflict and focuses on the necessity of MDVAD.Distinction from Open-set VAD Approaches. Unlike Open-set VAD [59, 25, 1] , which separates seen and unseen anomalies within a single-domain cannot achieve robust representation learning among multiple domains, this paper excels at handling abnormal conflicts and shows adaptability across domains with versatile evaluation protocols (held-in/out and low-shot).2 Observations 
datasets: 
 In this paper, we use six representative VAD datasets: UCF-Crimes (UCFC) [43] , XD-Violences (XD) [51] , Large-scale Anomaly Detection (LAD) [47] , UBI-Fights (UBIF) [9] , Traffic Anomaly Dataset (TAD) [19] , and Shanghai-Tech Campus (ST) [24] . As shown in Table 1 , each dataset has different environments (e.g. CCTV, Traffic, Campus), quantities, and abnormal categories. Unlike other datasets, ST is an unsupervised VAD benchmark whose training set comprises only normal videos, so the training set has been reorganized following [21, 45, 57] . More details are provided in the supplementary material (Sec. A).Based on observations of each dataset's properties, we aim to quantify how the different properties negatively affect domain shifts. Table 2 presents cross-domain evaluation results, i.e., single-domain models validated on different target datasets. The results reveal that while these models excel indomain settings (diagonal elements of the table), they exhibit significant performance degradation in cross-domain scenarios. This implies that a single-domain VAD model may be ineffective in most other environments unless the environment and user's intention are precisely identical. Therefore, leveraging diverse datasets for generalized feature learning is crucial, enabling the model to handle unknown domain and well adapted to unseen anomalies. This paper addresses two primary issues: 1) Abnormal conflict arising during the multi-domain learning process, and 2) Scene discrepancy occurring during the evaluation process on an unseen target domain.Abnormal conflict. As illustrated in Fig. 1 , abnormal conflicts indicate abnormal events that are considered abnormal in one (or some) domain(s) but are denoted as normal in other domains. As shown in Table 1 , there is relatively little overlap of abnormal classes in ST with other datasets.In other words, this means that the abnormal conflict with other datasets is relatively large. This conflict leads to low cross-dataset performances of models trained or evaluated on ST in Table 2 . Taking UCFC as an example (target domain: UCFC column in Table 2 ), performance increases in the order of TAD, ST, LAD, XD, and UBIF, which is proportional to the number of abnormal conflict categories that exist in the source domain. Abnormal conflict makes the MDVAD unique as the label spaces between domains literally conflict because of each other's differing definitions.Scene discrepancy. Scene discrepancy refers to differences in the visual settings of scenes, distinct from abnormal conflict arising from variations in the definition of abnormal classes. To quantify scene discrepancy, we utilize the Earth Mover's Distance (EMD) [38] introduced in [8] to calculate the distance between VAD datasets in the Table 3 . In Table 3 , the top-right section illustrates the comparison of normal features, while the bottom-left provides a numerical comparison of class-wise abnormal features. Lighter colors indicate a higher discrepancy between datasets. Unlike other datasets, TAD, which comprises traffic videos, exhibits a large distance from normal sample distances in the dataset, while LAD, with diverse and complex scenes for abnormal classes, shows the furthest distance from other datasets. This explains the results of domain adaptation experiments. (Table 7 ). V a or V n is divided into uniformly sampled T snippets (V a ∈ {v a 1 , • • • , v a T }).These T snippets pass through a pre-trained video backbone, resulting in a C-dimensional feature B ∈ R T ×C that undergoes an aggregation module, fusing them from the feature level to the temporal level. The feature aggregation layer doubles the channel size B ∈ R T ×2C , followed by a split and max operationF agg,i = max Bc i , BC+c i c=1,••• ,Cto squeeze the channel size back to C for the i-th snippet. Activating only the maximum element during gradient propagation enables the implicit differentiation of class-specific channels, allowing the model to establish discrepancies between classes [7, 52] . The temporal aggregation layer, with a temporal kernel size of 3, produces the domain-agnostic aggregated feature F DA ∈ R T ×C/2 . 
single-domain learning: 
 The final feature F DA undergoes fully connected layers denoted as FC and the single abnormal head (w a D1 ∈ R C/16×1 ) followed by a sigmoid function to derive the final abnormal score s a D1 ∈ R T ×1 . MIL. Due to the absence of temporal interval training labels for abnormal events, WVAD models rely on video-level labels for training, employing the Multiple Instance Learning (MIL) method. When the top-K score set is represented as Ω k (s a D1 ), the Binary Cross Entropy-based (BCE) classification loss function is formulated as presented in Eq. 1, where y = {0, 1}.EQUATIONThis loss function ensures that only snippets with high (Top-K) abnormal scores can contribute to the loss. This single-head model serves as the single-domain baseline (MIL in Fig. 2 ).Null-MIL. To address the abnormal conflict issue mentioned in Sec. 2, in the MDVAD framework, the abnormal head is divided into multiple heads, each responsible for its own domain, allowing for the domain-wise prediction of the output score. Inspired by [18] , the prediction score for the input snippet from the D d dataset is derived exclusively from the output of the D d -head, and the results from heads of other datasets are filled with Null values (Null-MIL in Fig. 2 ). Compared to Eq. 1, the Null-MIL loss is changed as follows:EQUATIONwhere M is the number of heads (domains). To avoid the abnormal conflicts, the heads between datasets are separated where D d -head's weight w a D d is independently trained for the corresponding dataset. In Eq. 2, only s a D d among all output scores is added to the loss, thus the gradient becomes∂s a D d ∂w a D d, while other heads are not affected.For the test, the abnormal score of i-th snippet is s a D d ,i when the target dataset is D d and max d s a D d ,i by selecting the maximum value for unseen target data.NullAng-MIL. We additionally propose a NullAngular-based MIL method that employs the angular margin to effectively diminish large variations among intra-class instances. In this case, multiple normal heads are added (NullAng-MIL in Fig. 2 , representing cosine similarity. Thus, in the cosine space, Eq. 2 can be defined as Eq. 3, requiring the maximum cosine similarity between the feature from dataset D d and the abnormal head w a D d to be at least an angular margin of m greater than the normal head w n D d that enlarging the gap between normal and abnormal.EQUATIONDenoting the top-K abnormal scores from the head of datasetD d as Ω k (s D d ) = s a D d ,i , s n D d ,i i=topk indices, rewriting Eq. 2 as an angular margin-based regression problem results in Eq. 4. Similar to Null-MIL, the loss is computed by the head associated with the input dataset, while scores from other heads have no impact on updating the model's weights. For the test, since the normal and abnormal heads are trained through angular margin learning, the abnormal score is shown in Eq. 5.Abnormal ScoreEQUATIONWhen the source domain of the pre-trained general model is different from the target domain, we determine the final score to reflect conflicts by taking the maximum normal and abnormal score from multiple heads.Unseen In the domain-agnostic feature extraction phase, it is crucial to capture general features that can handle every domain. While we have divided the heads to avoid abnormal conflicts in multiple source datasets, the agnostic part extracts features from all datasets using a single branch, and inconsistent labels cause confusion. Therefore, we propose an Abnormal Conflict (AC) classifier for learning the final embedding feature F DA that passes through the detector heads.In Fig. 2 , each classifier head distinguishes between D d and D d c , while the AC classifier is intended to distinguish between elements that are abnormal or normal across all source datasets (green area) and elements that represent conflicts (yellow area) within the Venn diagram. The AC classifier takes the embedding feature F DA as input, followed by two FC layers, to predict the final conflict score s AC . In Eq. 6, the AC label is generated based on the scores of all abnormal heads, where y AC i = 1 if the deviation between the scores is above a threshold τ .EQUATIONThe loss of the AC classifier, denoted as L AC and calculated using cross-entropy as follows:EQUATIONThe total objective function isL = L N ullAng-M IL + λL AC .During the testing phase, the auxiliary branch AC classifier is eliminated, and the output is calculated for each input snippet v i as the final Abnormal Score i .4 Experimental Results 
mdvad benchmark: 
 As shown in Table 1 , VAD comprises six representative datasets with diverse settings and volumes.The MDVAD is a task aimed at addressing domain shifts between datasets, and each dataset included in the MDVAD benchmark should be structured so that it is not biased toward any particular dataset or anomalies. Consequently, datasets should have an equal volume in the training set and encompass various abnormal categories and criteria. To achieve this, we sampled each dataset to align with the dataset with the smallest volume, ensuring that each abnormal category has a similar proportion and conducted 3-fold experiments. Additionally, to handle small datasets like TAD and ST, which have minimal volumes, we combined them with the traffic dataset CADP [41] and campus dataset NWPU [3] , respectively, by reorganizing their training sets. The volume of MDVAD benchmark is unified at 386 videos per dataset. More details provided in the supplementary material (Sec. B).This section presents empirical studies as follows: first, single-domain results for comparison, followed by the results of four evaluation protocols for the MDVAD benchmark. The models are trained on the training set of the source domain and evaluated on the test set of the target domain under the following settings: Held-in (E1), Leave-one-out (E2), Low-shot adaptation (E3), and Full finetuning (E4). The Area Under Curve (AUC) is used as the evaluation metric. The hyper-parameters are T = 32, λ = 10, τ = 0.3, and m = 0.3. Since the single-head baseline cannot assign AC labels using Eq. 6, pseudo-labels for the AC classifier are assigned based on the range of predicted abnormal scores. All detailed implementations, more results, and discussions are provided in the supplementary material.Single-domain results. In the preliminary experiment, we examine the performance of the MDVAD benchmark under single-source and single-target conditions using a single-head MIL baseline. Comparing the in-domain (diagonal elements) results in Table 2 (using the original training set) and Table 4 (using the sampled training set), we observe that the performance diminishes when datasets, other than TAD and ST, are reduced during sampling. Conversely, TAD and ST show improved performance, benefiting from increased diversity and data volume due to reorganization with CADP and NWPU for TAD and ST, respectively. The cross-domain (off-diagonal elements) results in Table 4 demonstrate consistent trends with both the MDVAD and VAD benchmarks. These trends primarily arise from challenges related to abnormal conflict and scene discrepancies, indicating that the issues are confined to in-domain settings. Consequently, to uphold in-domain performance across diverse datasets and effectively address issues related to abnormal conflict and scene discrepancies, we emphasize the essential role of multi-domain learning in developing a general model for VAD.In this section, we discuss the held-in evaluation results, where models are trained on all datasets of MDVAD, and the test set of each source dataset is used as the target. Table 5 presents the results obtained by training with the MIL baseline with a single-head, which is commonly used in traditional WVAD, and the proposed Null(Ang)-MIL baseline composed of multiple heads. Because UCFC, XD, and LAD exhibit low abnormal conflicts, numerous similarities in abnormal categories, and minimal scene discrepancy, there are great performances with the single-head MIL baseline. However, when TAD and ST are targets, the results demonstrate that the Null(Ang)-MIL baselines, which extract general domain features in the domain-agnostic part and avoid conflict by assigning Null values with multiple heads, outperform the MIL baseline.Handling multi-domain with a single Null(Ang)-MIL baseline. The average performance of single Null-MIL and NullAng-MIL is comparable to or even better than the average performance of individual in-domain models trained for each domain (86.52%). Moreover, the NullAng-MIL baseline, which effectively performs inter/intra-class learning through cosine angular margin while avoiding conflict between each head, achieves superior results, with 91.82% (TAD) and 91.26% (ST) compared to the performances of single in-domain results of 90.75% (TAD) and 90.79% (ST), highlighting the effectiveness of learning with diverse and abundant data to enhance model generality. Furthermore, additional performance boosts are observed across all baselines with the auxiliary AC classifier to make the model conflict-aware.The E2 experiments, in Table 6 , are conducted on the leave-one-out setting among multiple source datasets and evaluated on the unseen target dataset unlike E1. Each column represents the outcomes of models trained with the target dataset excluded. Compared with the held-in results, the evaluation results conducted without prior knowledge of abnormal boundaries for unseen target data lead performance gap between E1 and E2.Effectiveness of multi-domain learning for unseen target domain. When comparing single-domain learning and multi-domain learning in the results of unseen target evaluation (Out Avg. in Table 6 ), it is evident that the performance with multiple datasets is significantly superior. Effectively addressing conflict and discrepancy issues, and learning from diverse and complex situations across multiple domains, leads to the development of a general model with better performance on unseen domains.In E3, we delve into low-shot learning, examining how well the model trained with multiple sources in the leave-one-out experiments (E2) adapts to unseen targets. In this experiment, we utilized 10% of the target training set for low-shot learning (in a 3-fold setting). For datasets with abnormal categories, an equal number of data per category are randomly sampled, while for uncategorized datasets like UBIF and ST, random sampling is employed.Importance of general pre-trained baselines for adaptation. In Table 7 , comparing with Table 6 , notable performance improvements are observed across all baselines for target samples with severe domain conflicts and gaps, such as UBIF, TAD, and ST, even with a limited volume of training samples. This emphasizes the importance of building a general model to adapt on unforeseen targets. While the MIL baseline with a single head holds an advantage in adapting specific single head when target samples are appropriately selected, it may become dependent on specific samples and is susceptible to overfitting. On the other hand, NullAng-MIL, with conflict-aware learning and training on diverse domains, outperforms other methods in adaptation performance. Multi-domain models superior to specific singledomain models. Table 8 shows the full fine-tuning results of single-domain and multi-domain trained baselines. The single-domain models reflect results from training and testing within a specific dataset. When evaluating multi-domain baselines trained on all datasets in the held-in (E1) setting, it demonstrates performance comparable to single-domain models that perform well by fitting to a single dataset. However, after full fine-tuning on the target dataset, the E1 model achieves superior performance in most cases. The E1 and E2 models, which are well-explored across multiple domains, outperform the models specifically trained on a single domain after full fine-tuning.Null-MIL and NullAng-MIL. The Null-MIL baseline achieves the best performance (86.86%) in the Held-in setting (E1), even surpassing the in-domain average results (86.52%). However, in the Leaveone-out setting (E2), which involves unknown target domains, it shows suboptimal performance. This is because it is impossible to determine which head's output score among the source dataset heads is more reliable, so multiple head baselines employ the maximum score for unknown domains. Conversely, NullAng-MIL exhibits superior results in E2 due to its similarity-based score calculation. NullAng-MIL considers the cosine distance between the final feature and the weights of each source head. As a result, when calculating the final score, the head with high similarity to the target data is activated, yielding promising results. When pre-training the general VAD model without knowledge of the target, we observed that utilizing the NullAng-MIL baseline is beneficial. However, when conducting multi-domain learning with knowledge of the target, employing Null-MIL yields better results. We observed that when pre-training the general VAD model without prior knowledge of the target, utilizing the NullAng-MIL baseline proves beneficial and employing Null-MIL yields better results when knowledge of the target is available during multi-domain learning.Role of the AC Classifier. In multi-domain learning, the proposed AC classifier serves as an auxiliary branch designed to predict whether there is an abnormal conflict between domains, thus making the model aware of conflicts. Since it is not used during testing, it does not impact the final model's cost but provides performance gains in most experiments. Notably, in E2 and E3, training with the AC classifier shows a significant boost effect, aiding in adaptation to unseen domains. Comparison with WVAD models. Although this paper focuses on the analysis of multiple domain learning within the context of abnormal conflict issues, instead of exploring complex architecture designs for single-domain VAD models, we compare with various VAD models. Table 10 presents the results of other MILbased WVAD models, MMIL [43] , ARNet [46] , WSAL [30] , and COMO [7] on the MDVAD task. Compared to the proposed baseline trained with the AC classifier, our method achieves the highest average AUC, particularly in datasets with severe abnormal conflicts and scene discrepancies, such as TAD and ST, in both the E2 and even more in the E3 settings. Various single-domain VAD models or backbones can be incorporated into the MDVAD task, showing a direction for future generalization work. Please refer to § A6.The AC classifier helps the model learn conflict-aware features, providing a clearer understanding of abnormalities. The proposed framework is composed of Domain-Agnostic Layers that learn general features across multiple domains, followed by Multiple Heads that predict abnormalities specific to each domain. When the Domain-Agnostic layers learn to perform AC classification, they capture whether the input snippets relate to abnormal conflict or not. From the Heads' perspective, these features are separated into abnormal conflict and non-AC in the feature space, allowing the Heads to apply different criteria (decision boundaries) when distinguishing between normal and abnormal instances. For example, in classifying abnormalities, non-AC scenarios can be addressed more straightforwardly, while abnormal conflict scenarios require a more careful exploration. Fig. 3 (a) presents the results of the AC classifier of NullAng-MIL baseline from E1. It shows abnormal conflict scores for two scenarios: (Top) A car on the Sidewalk abnormal event in ST, which is normal in the UCFC. (Bottom) A normal situation of people shopping in a grocery store. Through the multi-domain learning, the AC classifier outputs a high abnormal conflict score for the top sample, demonstrating that the model has learned to be conflict-aware.Qualitative results. Fig. 3 (b) illustrates abnormal conflict scene: (Top) Bicyclist on Walkway, abnormal in ST but normal in other domains, and (Bottom) Pedestrian on Road, abnormal in TAD but normal in UCFC. In these scenarios, the MIL baseline trained with multiple domains predicts (Top) false negative and (Bottom) false positive, while ours adaptively handles conflicts across different domains.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: In § 4 and § C Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion, limitation, and future works: 
 In this paper, we propose a new task called MDVAD, whose ultimate goal is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general VAD model. As a baseline, we propose a new multi-head framework with Null(Ang)-MIL loss and AC classifier. These modules effectively handle abnormal conflicts between domains and show meaningful results in the MDVAD benchmark with diverse evaluation protocols.Instead of exploring complex architecture design of single-domain VAD models, this paper focuses on resolving abnormal conflicts from multiple domains. Various single-domain VAD models or backbones can be applied into our novel framework to address the MDVAD task, representing a valuable direction for future generalization research. The XD dataset [51] stands out as the most extensive dataset with a variety of environments, including CCTV, movies, hand-held cameras, and car camera settings. It incorporates six categories of violence, such as abuse, riot, and explosion, defining anomalies. Within the dataset of 4,754 videos, there are 2,049 normal and 1,905 abnormal training videos.The LAD dataset [47] is defined by a detailed categorization of 14 anomaly classes, including crash, fire, and violence, consisting of 2,000 videos, with 958 normal and 482 abnormal videos.The UBIF dataset [9] is composed of various fighting-related anomalies extracted from our daily life videos. This dataset consists of 1,000 videos without a specific categorization for the fighting class, and the training set consists of 757 normal and 176 abnormal videos, leading class imbalance issue.The TAD dataset [19] is structured with traffic surveillance and 1st-person videos, encompassing anomalies related to traffic, such as accidents, illegal turns, and road spills. This dataset, totaling 500 videos, has a small volume with 210 and 190 numbers of normal and abnormal training videos.The ST [24] serves as a campus surveillance dataset, capturing anomalies occurring on pedestrian walkways. It exhibits a heightened sensitivity to abnormal events, ranging from fighting and throwing to running and jumping. With only 175 normal and 63 abnormal videos in the training set out of a total of 437 videos, there is a notable imbalance between classes, and the quantity of data is insufficient for learning. Abnormal categories in the training set cover all anomalies in the testing set on each dataset except for ST (ST remains indeterminate because of the absence of category information), which means evaluations are exclusively conducted for seen abnormal events.Examining the dataset samples in Table A1 reveals that TAD and ST exhibit a visual gap compared to the other datasets, UCFC, XD, and LAD. Such gap is attributed to visual elements such as scene settings and conditions that impact the visual characteristics of scenes, which is the scene discrepancy discussed in §2. While LAD (some videos) and ST are recorded in square or campus settings to establish real-world scenario datasets, the remaining datasets are constructed online, such as YouTube and LiveLeak using text search queries. These datasets also exhibit a significant domain distance, as indicated in Table 3 . As such, the criteria for defining abnormalities and the visual characteristics of scenes vary across different datasets.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will release the data and code when the paper is accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In § A Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The five datasets introduced in Table A1 have different training set volumes, leading to challenges in handling multiple dataset learning, where larger datasets may overwhelm the learning process.For instance, there is a significant disparity in training set volumes, with TAD (400 videos), ST (238 videos), and XD (3954 videos) differing by almost 10 times. Additionally, the number of abnormal videos in XD differs roughly 10 and 30 times compared to TAD and ST, respectively.The objective of the MDVAD task is to avoid abnormal conflict and build a general model through balanced learning across diverse situations and anomalies. To control for other variables, we uniformly sampled and balanced the same amount of videos from all domains, equal to the number of videos in the smallest domain. Since TAD and ST have an insufficient number of videos for training, we recombine them with two new datasets, CADP [41] and NWPU [3] , respectively, to balance the numbers. CADP is a dataset for traffic accident analysis, and it shares similarities with TAD in terms of setting and abnormal definition. Hence, following the approach in [28] , we compose TAD's training set by adding abnormal videos from CADP. Additionally, incorporating the recently released NWPU dataset, which collects various anomalies of pedestrians on campus, we augment ST's dataset with NWPU data. For both datasets, we conducted random sampling for normal and abnormal videos, considering the smallest sample size, resulting in a number of 210 normal and 176 abnormal videos. In this process, for datasets with information on abnormal classes, we compose to achieve a uniform number of instances per abnormal category on each dataset. For datasets with a sufficient amount, we organize 3-fold sets to minimize overlap. All reported performance metrics are the average results of evaluations conducted on the target dataset after training on each fold. For example, in the E2 leave-one-out setting, the average results are derived from models trained on each fold and evaluated on an E3 low-shot adaptation set for the corresponding fold.The MDVAD benchmark incorporates the datasets introduced in A4 (a) illustrates the distribution of categories. For datasets without abnormal categories like ST (named as Campus Else), TAD (where additional categories are labeled as Traffic Else), and UBIF (where categorization is not specified and all fall under Fighting), we provide appropriate category names. The averages across the 3-folds are calculated, and in the case of XD with multiple labels per video, we count the number of categories per video. The Fighting predominates in the majority of datasets, occupying the largest portion, followed by categories like Accident, Explosion, and others.VAD is predominantly evaluated using the Area Under the Curve (AUC) score based on abnormal scores, focusing on the trade-off between sensitivity and specificity, emphasizing the false positive rate (FPR). While XD employs the Average Precision (AP) metric to balance precision and recall and concentrate on positive samples for direct comparison in cross-dataset and held-in/out evaluations, and emphasizing False Positive Rate (FPR) (detailed explanation is in Section D), we utilize the AUC score across all evaluations.The MDVAD benchmark comprises four protocols: Held-in (E1) involves multiple-source learning with all datasets, Leave-one-out (E2) employs a leave-one-out approach where a dataset left out during training is used for evaluation, low-shot adapation learning (E3) is training on the remaining datasets, and Full fine-tuning (E4) that fine-tunes a multi-domain models on single-domain datasets. In the E3 setting, around 10% of the training set, 20 videos, is randomly sampled for low-shot examples. For datasets with anomaly category information, low-shot examples are selected uniformly to ensure an even distribution within each category. To mitigate the impact of randomness, the low-shot setting is also conducted with 3-fold cross-validation, obtaining low-shot examples from each fold. In all cases, including MIL, Null-MIL, and NullAng-MIL models, all weights except for the pretrained backbone are updated. Additionally, for abnormal videos, the score from the AC classifier is multiplied to the loss value.All experiments are conducted on the MDVAD benchmark. We use I3D backbone feature which is pretrained on Kinetics dataset [4] with C = 2048 dimension of RGB features. During the training phase, we conduct consistent experiments with a batch size of 32, utilizing the Adam optimizer [20] with a learning rate of 5e -5 and weight decay of 5e -4. The input image size is set to 224 × 224 and following [21, 31, 45, 7] , we perform 10-crop augmentation when extracting backbone features. The backbone is used the same as RTFM [45] and CoMo [7] . For input data, 16 frames are stacked for a snippet, and T snippets are uniformly sampled in the video during training. For testing, all snippets pass through the model, and the output score is assigned to all frames within the snippet. For MIL learning, the Top-K snippets, where K equals 10% of the total, i.e., K = 3, are utilized. In the case of a single-headed MIL model in Table 5 (E1) setting, since pseudo-labels cannot be generated through multiple heads, the AC classifier is trained with a label that y AC = 1 for a score s a D1 within the [0.4, 0.7] range. When computing EMD metric in §2, we use a pretrained I3D backbone model for extract embedding features of each dataset and measure the distance between feature vectors. We use a test set with frame-level annotations for abnormalities. All models and experiments are implemented and evaluated end-to-end using PyTorch [34] with a single NVIDIA V100 GPU.In weakly-supervised VAD, maintaining class balance is pivotal, so the number of normal and abnormal input videos within a batch is kept equal for the MIL-based learning. Similarly, during multiple source learning, an equal number of input videos from each source domain within a batch is crucial. Despite each head is trained independently, the final feature F DA is extracted from a shared domain-agnostic part, necessitating the balance in the number of source domains. Moreover, when training the AC classifier, in order to address the imbalance between the number of abnormal conflict snippets and all normal/abnormal snippets, we apply focal loss [23] to L AC . During the testing phase, the AC classifier is eliminated, and the output is calculated for each input snippet v i as the final Abnormal Score i .Table A4 provides information about the layers of the baseline model. Conv1d, fc, Max, BN, and DO represent 1D convolutional layer, fully connected layer, element-wise max activation, 1D batch normalization [16] , and dropout [42] , respectively, with a dropout probability set to p = 0.7. In Conv1d(c, k, s) and fc(c), c, k, and s denote the channel size, kernel size, and stride size, respectively. T input snippets pass through the backbone, resulting in a backbone feature B with a shape of (N, T, C), where N represents the batch size. The feature aggregation layer undergoes channel doubling followed by channel squeezing with element-wise max operation, resulting in an output feature F agg with a shape of (N, T, C). Subsequently, the temporal aggregation layer outputs F DA with a shape of (N, T, C/2). These embeddings are then input into the AC classifier and fc layer.The AC classifier, composed of consecutive fc layers, outputs AC scores s AC for each snippet with a shape of (N, T, 1). Additionally, the final feature F, processed through an fc layer, is input into multiple domain heads w D d for NullAng-MIL learning. During this process, both the feature and weights are normalized with respect to ∥F∥ and ∥w D d ∥ for each domain.In Fig. A4, (a ) represents a video labeled as normal in UCFC, while (c) is labeled as abnormal in TAD, both depicting a situation where people are present on the road. Similarly, (b) is labeled as a normal video in XD, whereas (d) is considered abnormal in ST; both scenes show bicycling on a walkway. We define such instances, where the criteria for abnormalities differ across datasets, as abnormal conflicts.While cross-domain evaluation in Table 2 reveals a domain gap, it is difficult to attribute as abnormal conflicts directly. In Table A2 , we calculate the average of the relative False Positive Rate (FPR) and relative False Negative Rate (FNR) to quantify the abnormal conflict between domains.EQUATION)where i and j indicate source and target domain, repectively. First, FPR and FNR refer to the rate of misclassifying abnormal as normal and normal as abnormal in the scenario of learning from a source and testing on a target, respectively. However, since these include the FPR and FNR caused by the incompleteness of the model itself, this cannot accurately reflect the abnormal conflict between the source and target dataset. Therefore, we subtract the FPR and FNR when the source domain is the same as the target domain (F P R j,j and F N R j,j ) from the cross-domain FPR and FNR to calculate the relative FPR and FNR caused by the domain shift. Table A2 shows similar trends compared to Table 2 , where values are not normalized. This shows that abnormal conflicts between domains are relevant to the reduction in generalization ability that can occur when transferring between domains.As mentioned, proposing a new VAD model is beyond the scope of this paper, this part is organized to aid understanding.Unsupervised VAD Defining anomalies precisely is challenging as they can vary based on criteria and sensitivity, and it is impossible to categorize and collect datasets for every scenario. For this reason, Unsupervised VAD methods [33, 5, 24, 15] have been developed to learn normal patterns using only normal training data and then detect anomalies as scenes with low normality during the test phase. However, this approach leads to a significant bias towards the normal samples, causing the detector to misclassify normal patterns that differ from the learned data as abnormal events, resulting in a high false alarm rate. To address this issue, Weakly-supervised VAD (WVAD) approaches [55, 44, 43, 50, 21, 7] have been introduced to differentiate between normal and abnormal events with minimal supervision using video-level annotations, achieving significant performance improvements while incurring lower labeling costs compared to frame-level annotations.Weakly-supervised VAD MMIL [43] is the first WVAD method to pose the MIL ranking approach as a regression problem that achieves significant performance gains. Various learning methods, including magnitude feature learning [45] , self-training approaches through pseudo labels [13, 21] , and distance learning [53] , have been explored alongside traditional MIL methods. Furthermore, subsequent approaches have focused on capturing features by focusing temporal or motion information [58] , leveraging additional audio signals [51] , learning relations between temporal [50] or motion and context information [7] . These methods differentiate complex anomalies through contextual information; for example, the same motion may be classified as normal or abnormal depending on the surrounding environment. PFMP [26] proposed a method to utilize virtual data anomalies to reduce scene discrepancy with real-world data, addressing the issue of data scarcity in VAD. However, these approaches still cannot address abnormal conflicts, where the same situation is labeled differently as abnormal or normal based on the criteria of different datasets.Generalization for VAD A few studies have been conducted to generalize VAD. Domain adaptation refers to evaluating a new domain that was not seen during training, and some works have proposed using some data from the target domain [27, 29] or not using it at all [2] . We also experiment with domain adaptation, but the difference is that we use multiple domains compared to existing studies that only use a single domain during training. Open-set recognition aims to work well even in untrained classes (abnormal categories in VAD). Compared to our research involving multiple domains, Open-set VAD [59, 1] focuses on learning and evaluation within a single domain.Multi-domain learning Multi-Domain Learning (MDL) refers to a method for learning datasets from multiple domains with various distributions together. MDL originated from natural language processing [11, 17] and has been applied to various tasks in computer vision [36, 37, 22, 32, Figure A5 : The plot illustrates the abnormal conflict scores from the AC Classifier on normal videos on UCFC. For clarity, all scores have been normalized using the minimum and maximum values of the entire score distribution produced by the AC Classifier. 48, 56, 18] . The difficulty of MDL arises when the difference in distribution between domains is significant, and learning domain-invariant features is the basic approach [39] . In the fields of image classification [36, 37, 22] , object tracking [32] , detection [49] , and segmentation [48] , methods for learning multiple domains with multiple heads have been proposed. After that, methods have been proposed to have a unified label space [56, 18] , but in this process, the problem that the labels are different for each domain arises. [18] solved the label conflict problem by proposing classindependent loss using the Null class strategy. On the other hand, VAD has only two classes, normal and abnormal, so there is no explicit label conflict problem, but there is an implicit abnormal conflict problem where the definition of a label is different for each domain.A similar method is multi-task learning [54] , which considers differences between tasks rather than domains. In addition, multi-source domain adaptation [35] utilizes data from the target domain, and domain generalization [10, 40, 12] requires adaptation to an unseen target domain. The difference with these tasks is that they basically share the label space between the source and target domains, while MDL for VAD has different label definitions.The purposes of the AC Classifier is the facilitation of feature learning with consideration of the discrepancies across multiple domains. In cases where snippet is entirely normal or abnormal across all datasets, the AC score is low, making it difficult to directly use it for the abnormal score. Noisy samples can indeed have a negative impact on training in Eq. 6. However, in cases where even one of the multiple datasets has a different definition of normal and abnormal, it is crucial to sensitively detect conflicts in such samples, leading us to decide that using a difference is preferable. In Table A3 , we conducted ablation studies with Std. (τ = 0.1) and a fixed value (y AC i = 1 where 0.3 < s a D d ,i < 0.7) which revealed no significant differences between the methods. score as a false alarm, while our model accurately predicts only the explosion abnormal situation. In scenario (b), where a person loitering the office, the MIL model outputs a high abnormal score, in contrast, our model that trained with multiple heads to avoid domain conflicts categorizes only the arrest defined in UCFC as an abnormal event.Scenarios (c) and (d) involve abnormal videos in the ST dataset where a running person or car appears on the walkway. Because these scenarios are classified as normal scenes in other datasets during training, the MIL model considers them as normal. However, our model, recognizing them as abnormal scenes corresponding to ST, outputs high abnormal scores. Thus, when learning from multiple datasets, it is crucial to avoid conflicts by simultaneously learning domain-agnostic features with the AC classifier and predicting domain-specific scores using multiple heads according to each dataset's criteria. The Table A5 shows the performance in each target domain when trained with MDVAD and UBNormal under the E1: held-in setting. In the single-head MIL baseline, there is a performance drop when trained with MDVAD+UBN, indicating difficulty in handling AC. However, the model trained with multiple heads and the AC classifier shows improved results. By leveraging the virtual dataset, we can overcome data limitations and create a general model capable of handling diverse and complex scenes. In Fig. A7 (b), an abnormal event from ST is considered normal by both models, as other datasets are generally classified as normal. Similarly, in Fig. A7(c ), the skateboarder is deemed normal, resulting in a false negative failure case. However, these situations are challenging to identify as severe abnormalities from a general perspective. This failure case is able to be addressed through low-shot learning or fine-tuning based on ST's criteria.The development of Multi-Domain Learning for Generalizable VAD enhances the reliability and accuracy of security and surveillance systems across diverse environments, thereby improving public safety. This research advances AI and machine learning by addressing domain adaptation challenges, promoting robust and generalizable systems. Additionally, it underscores the importance of ethical considerations and privacy, ensuring that advanced surveillance technologies are deployed responsibly and transparently.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: This paper received 4 reviews from experts in the field. The paper received the following reviews:  3 Borderline Accepts and 1 Borderline Reject.

The main concerns presented by the reviewers involved the clarity of the presentation. The author rebuttal addressed the reviewers’ concerns. The AC agrees that the merits of this paper outweigh any clarity concerns and the decision for this paper is to accept.  Please consider the reviewer comments in your preparation of the camera ready version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a new task called MDVAD, the goal of which is to effectively learn from multiple domains with different data distributions and definitions of abnormality without confusion, resulting in a general VAD model. To achieve this, the authors expand the traditional single-head framework to multiple-head framework for learning different knowledge and design an AC classifier to handle abnormal conflicts. The experimental results prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. This paper proposes a new task called MDVAD to achieve generalizable VAD by resolving conflicts in anomaly definitions. However, for any VAD application, the definition of normal or abnormal events should be explicitly determined according to the scenario requirements, rather than simply combining multiple datasets and resolving the abnormal conflicts. I find it difficult to understand under what practical scenario a VAD model trained using multiple datasets with abnormal conflicts is needed.
2. The writing of this paper is not clear enough, where some necessary training and inference details are missed. For example, the normal head training mentioned in NullAng-MIL is confusing.
3. This paper lacks a detailed description of the experimental setup. For example, if an anomalous event is determined to be a conflict, how should the model handle such an event?
questions: questions: 1. (referring to weakness 1) In practical applications, what kind of scenarios conform to the task settings of MDVAD proposed by the authors?
2. (referring to weakness 3) During test phase, is it necessary to know which dataset the sample comes from? If a certain anomalous event is found to have conflicts in different datasets, how should it be handled? Do the multi-dataset evaluation and the test procedures for other methods use the same test data?
limitations: limitations: I do not recognize obvious potential negative societal impact of this work.
paper_summary: paper_summary: The manuscript addresses the limitations of existing Video Anomaly Detection (VAD) models that are confined to single-domain learning. The primary contribution of the paper is the introduction of a new task called Multi-Domain Learning for VAD (MDVAD), which aims to develop a general model capable of identifying abnormal events across multiple domains. The manuscript conducts experiments using the MDVAD benchmark and demonstrates the limitations of traditional multi-domain learning. It shows the effectiveness of the proposed baselines in handling abnormal conflicts and achieving robust performance across multiple domains.
strength_weakness: strength_weakness: 1. MDVAD introduces the domain-specific multi-head mechanism and the Null-MIL method, which increases the complexity and computational cost of the model, and may place higher demands on the computational resources in practical applications.
2. The multi-domain learning task itself is difficult to train, and with the proposed method further increasing the complexity of training, MDVAD may require longer training time and higher technical requirements.
3. Although the theoretical background and analysis are provided, the theoretical basis and derivation process of some of the methods of MDVAD are slightly weak and need to be further explored and verified in depth. Part of the theoretical analysis is based on specific assumptions, and these assumptions may not be fully valid in practical applications, affecting the applicability of the theoretical analysis.
4. Although new benchmarks and assessment protocols are proposed, MDVAD lacks comparative experiments with other state-of-the-art methods, making it difficult to objectively assess the relative advantages of the proposed methods.
questions: questions: 1. What is the training difficulty of MDVAD? The introduction of domain-specific multi-head mechanism and Null-MIL method greatly increases the complexity and computational cost of the model, can it meet the real-time requirements in practical applications?
2. Have the MDVAD and evaluation protocols been subjected to comparative experiments with other state-of-the-art methods in order to objectively assess the relative advantages of the proposed methods?
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: In this paper, authors proposed a new task called Multiple Domain VAD (MDVAD), along with a benchmark and new evaluation protocols. Authors' goal is to construct a general VAD model by conducting multi-domain learning while recognizing abnormal conflicts and exploring representations of general normality and abnormality. Authors introduced a baseline for MDVAD and proposed a new framework with multi-head to mitigate abnormal conflicts and proposed Null-Multiple Instance Learning (Null-MIL) and NullAngular-MIL (NullAng-MIL) losses for multi-domain training. Additionally authors suggested an Abnormal Conflict (AC) Classifier to explore general features while being aware of abnormal conflicts. Authors analyzed the primary issues of MDVAD and proposed a baseline for this new task.
strength_weakness: strength_weakness: 1. The proposed framework with multi-head for multi-domain seems not flexible enough during the domain changes, such as adding a new dataset with extra abnormal conflicts. And for the abnormal conflicts, will the proposed method performs better compared to make anomaly categories classifications for all anomaly events type of all domains?
2. In my opinion, traditional WS-VAD methods are designed to detect abnormal events in single domain without abnormal conflicts, and when abnormal conflicts exists, it will be better to use other paradigms such as temporal action localization or video grounding. And for the current WS-VAD datasets, the annotations are video-level, or even without category information, which is too weak for higher level anomaly detection. Training model with the current MDVAD paradigm is likely to not achieve good results.
3. Maybe using visual-language model with multimodal alignment can deal with the above issues? These models contain more knowledges for more event categories and higher generalization ability, which are likely to have the ability to individually detect conflicting anomalies. Compared to multi-head regression, is VL alignment a better approach for MDVAD task?
questions: questions: My main questions are shown in the weakness.
limitations: limitations: N/A
paper_summary: paper_summary: This work proposes a new task named Multi-Domain Learning Video Anomaly Detection, which aims to learn a general VAD model across domains. The work finds that abnormal conﬂict is a critical challenge in the task. Then, the work establishes a new benchmark, designs an effective baseline and conducts extensive experiments to investigate this challenge. The results shown on the benchmark demonstrate that the abnormal conﬂict is alleviated.
strength_weakness: strength_weakness: I have some concerns about the proposed method, and I think more comparison experiments are needed to demonstrate the effectiveness. Despite this, I think the abnormal conflict issue is interesting, thus I am willing to raise my rating if my major concerns are addressed. My concerns are as follows:

1. Why the proposed Abnormal Conﬂict (AC) classifer can address the abnormal conﬂict problem? Why the label is determined by the discrepancy in Eq. (6)? It seems that there are some mistakes in the formula (inconsistent with that in Fig. 2). 
2. I would like to see the results of more baselines, in addition to MIL, Null-MIL and NullAng-MIL. 
3. More detailed discussions about related works are needed, e.g., virual video anomaly detection datasets [1] and related techniques utilizing virtual datasets [2]. 

[1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022

[2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023
questions: questions: See the Weakness part.
limitations: limitations: The paper has discussed the limitations and potential impacts of the work.

Comments: comment: Thanks for the authors' clarification. My concern about the design of abnormal conflict classifier has been addressed. 

In my understanding, this work proposes a new task, designs a new benchmark, and develop an interesting solution to address the task. Although this work has some flaws, I believe it makes a good contribution to the field, with its strengths outweighing its weaknesses. Therefore, I will increase my rating to borderline accept. 

By the way, I strongly encourage the authors to thoroughly check their manuscript and polish their writing, regardless of whether this paper is accepted.
We apologize for the confusion caused by typo in our notation. As the reviewer correctly pointed out, **the notation should be $d$ instead of $i$ in Eq. 6.** We will revise the equation in the manuscript accordingly. 

$y_i^{AC}= 1$ where $[\max_{d} s^a_{D_d, i}-\min_{d} s^a_{D_d, i}-\tau]_+> 0$

$y_i^{AC}= 0$ where $[\max_{d} s^a_{D_d, i}-\min_{d} s^a_{D_d, i}-\tau]_+\le 0$

Once again, we appreciate reviewer for pointing out this typo.
Thanks for the authors' detailed response.

According to your illustration, in my understanding, the notation "$\max_{i} S^a_{D_d, i}$" should be "$\max_{d} S^a_{D_d, i}$" (also "$\min_{i} S^a_{D_d, i}$" should be "$\min_{d} S^a_{D_d, i}$"). This is because **the index $d$ denotes a domain and $i$ denotes a snippet**, and you argue that an abnormal conflict has more likely occurred as **the score gap between domains** is larger. I think there may be some typos in the formulas. Please check this.
We gratefully appreciate for providing thoughtful feedback on our paper.

# [Q2] AC Labels

When generating pseudo labels for the AC Classifier, we utilize the score gap between multiple heads:

**When Abnormal Conflict occurs between domains:** The abnormal scores produced by each domain's head will vary. For example, as shown in Fig 1, *'a pedestrian on the road'* is considered normal in the UCFC dataset but abnormal in the TAD dataset. In this case, the abnormal score from the UCFC head $s^a_{UCFC}$ would be low, while the abnormal score from the TAD head $s^a_{TAD}$ would be high. Therefore, the larger the score gap between domains, the more likely an abnormal conflict has occurred. This is represented by the following equation: 

$y_i^{AC}= 1$ where $[\max_{i} s^a_{D_d, i}-\min_{i} s^a_{D_d, i}-\tau]_+>0$

**When normal and abnormal are consistent across domains:** The score difference between each domain's head will be minimal. In scenario where all domains consider the event normal, they will all have low abnormal scores, and in the opposite case, they will all have high abnormal scores. Thus, the difference between the maximum and minimum scores will be small and AC label will assign by following equation:

$y_i^{AC}= 0$ where $[\max_{i} s^a_{D_d, i}-\min_{i} s^a_{D_d, i}-\tau]_+\le 0$

We conducted various experiments, including difference, Std. ($\tau = 0.1$), and a fixed value ($y_i^{AC}=1$ where $0.3<s^a_{D_d,i}<0.7$) approaches when assigning labels (Please refer to Supplementary Material Section F.1 with Table A3). Experimentally, we determined that using the score difference, as in Eq. 6, is the most effective method for generating labels for the AC Classifier.

# [Q3] Math notation
We have organized the key notations below.

* Abnormal video: $\\mathbf{V}^a\in \\left \\{ \\mathbf{v}^a_1, \cdots , \\mathbf{v}^a_i, \cdots , \\mathbf{v}^a_T \\right \\}$ where $i$ indicates snippet index.
* Abnormal heads: $\\mathbf{w}^a_{D_d}\\in\mathbb{R}^{C/16\times1}$ where $d$ refers domain index
* Abnormal score of $d$-th domain head: $\\mathbf{s}^a_{D_d}\\in\mathbb{R}^{T\times1}$ ($\\mathbf{s}^a_{D_d, i}$: $i$-th snippet abnormal score)
* Abnormal score from NullAng: $\mathbf{s}^a_{D_d}= \mathbf{F} \cdot \mathbf{w}^a_{D_d} =\left \|| \mathbf{F} \right \||\left \|| \mathbf{w}^a_{D_d} \right \||\cos\boldsymbol{\theta}^a_{D_d}=\cos\boldsymbol{\theta}^a_{D_d}$ where final embedding feature is $\mathbf{F}$
* Subset of Top-$K$ snippets' abnormal score: $\\Omega_k(\mathbf{s}^a_{D_{d}})$
* AC score: $s^{AC}$ ($s^{AC}_i$: $i$-th snippet AC score)

We hope our explanation has clarified any confusions, and we will also provide detailed descriptions of the mathematical notation in the Supplementary Material to ensure clarity.
We are pleased that our rebuttal has largely addressed the reviewer's concerns. We sincerely appreciate reviewer's thoughtful comment. 

# [W1] Practical Applications

**Problem Definition of Domain Generalization**

The goal of Domain Generalization (DG) is to learn a robust and generalizable predictive function from the $M$ training domains to achieve minimal prediction error on an unseen test domain that cannot be accessed during training. The difference between **Domain Adaptation (DA) and DG is that DA has access to the target domain data while DG cannot see them during training. This makes DG more challenging than DA but more realistic and favorable in practical applications [R1].** Various approaches, such as zero-shot learning (E2), adaptation learning (E3, E4), meta-learning, lifelong learning, and transfer learning, are employed to address the emergence of unknown target domains in real-world scenarios.

Foremost, we would like to clarify that the focus is on the fact that the **“Even without Target Domain information, the model performs comparably to an In-Domain (Single) Model”**, rather than on the idea that “Target Domain information is unnecessary.” In real-world scenarios that an unknown target domain emerges, the pre-trained general model learned from multiple source domains can operate using Eq. 5 without Target Domain information. Moreover, if samples from the target domain are provided, the model can adapt by tuning the multiple heads.

As mentioned previously, **the ability to handle multiple domains with a single model (E1) is also a practical scenario for generalization**. Because requiring a separate model and training for each domain is neither practical nor robust.

Similarly, issues such as Domain Shift, Overfitting, Label Conflict, and Efficiency are all challenges that need to be addressed for practical applications.

We acknowledge that our paper represents the first step toward practical VAD applications, and it opens up the possibility for various future works that could lead to valuable applications. We hope that the reviewer's concerns have been resolved, and we will include a discussion on this practical perspective in the manuscript.

[R1] Wang, Jindong, et al. "Generalizing to unseen domains: A survey on domain generalization." IEEE transactions on knowledge and data engineering 35.8 (2022): 8052-8072.
I greatly appreciate the authors' responses and the additional experiments, which largely addressed my concerns. My comments about their responses are listed as follows.

W1: The authors mention its application scenario as: "During the testing phase, target domain information is unnecessary in the E2~E4 settings because the source domain of the pre-trained general model is different from the target domain." However, I still remain skeptical about its practical significance.

W2: The authors have re-elaborated on the training methodology, and the overall process is relatively clear.

Based on the above considerations, I still have doubts regarding the practical relevance of the task proposed by the author. As such, I will maintain my current score for the time being and am open to hearing the opinions of other reviewers. If my doubts are resolved through further discussion, I will not refuse to increase my score accordingly.
Thanks for the authors' detailed response.

More specifically, I am confused that why AC labels are generated according to Eq. (6), i.e., why we should set the label to 1 only when the margin between max and min scores is larger than $\tau$. Could you please clarify this? 

By the way, the math notations are a little bit confusing in the manuscript. Please clarify them in your updated illustration.
I consider the methodology of this work to be another innovative approach to anomaly detection that is different from previous methods. Although there are some flaws in the work, it is a good starting point. I wish there were more types of approaches to anomaly detection. I will keep my marks. Good luck!
We are pleased that our rebuttal has addressed the reviewer's concerns.

# [Q1] How the AC Classifier Addresses the AC Problem

The AC classifier helps the model **learn conflict-aware features**. To provide a clearer understanding, we would like to explain this through a detailed step-by-step procedure.

We have designed our framework with **Domain-Agnostic Layers** that learn multi-domain general features, followed by **Multiple Heads** that predict abnormalities for each domain. When the Domain-Agnostic layers learn the ability to perform AC classification, **their features capture knowledge about whether the input snippets are related to AC or not**. From the perspective of the Heads, these features are divided between AC and non-AC in the feature space, **allowing the head to apply different criteria (decision boundaries) when distinguishing between normal and abnormal**. For instance, when classifying abnormalities, non-AC scenarios can be addressed more easily, while AC scenarios require a more careful exploration.

As a result, **1)** models trained with the AC Classifier handle AC more effectively in multi-domain learning (Table 5) and adapt better to unseen targets (Table 7). **2)** The AC Classifier learning was experimentally designed and validated (Table A3) and **3)** the AC score plot demonstrates that the model has effectively learned to predict AC (Fig. 3(a) and Fig. A5).

In summary, the AC Classifier plays a crucial role in enhancing the model's ability to address AC problems by learning conflict-aware features, thereby allowing the Heads to aware conflicts when classifying normal and abnormal instances, leading to more effective learning. Thank you for the valuable feedback, and we hope this response alleviates the reviewer's concerns.
Thanks for the authors' detailed response. After reading their response, some of my concerns are addressed. However, it is still not very clear for me that how the proposed Abnormal Conﬂict classifer addressed the abnormal conﬂict problem. I would like to see more detailed explaination from the authors, and I am open to hear from other reviewers.