Masked multi-prediction for multi-aspect anomaly detection
Authors: Yassine Naji
Romaric Audigier
Aleksandr Setkov
Angelique Loesch
Michèle Gouiffès
Venue: Accepted by TMLR ()

Abstract: In this paper, we address the anomaly detection problem in the context of heterogeneous normal observations and propose an approach that accounts for this heterogeneity. Although prediction-based methods are common to learn normality, the vast majority of previous work predicts a single outcome, which is generally not sufficient to account for the multiplicity of possible normal observations. To address this issue, we introduce a new masked multi-prediction (MMP) approach that produces multiple likely normal outcomes, and show both theoretically and experimentally that it improves normality learning and leads to a better anomaly detection performance. In addition, we observed that normality can be characterized from multiple aspects, depending on the types of anomalies to be detected. Therefore, we propose an adaptation (MMP-AMS) of our approach to cover multiple aspects of normality such as appearance, motion, semantics and location. Since we model each aspect separately, our approach has the advantage of being interpretable and modular, as we can select only a subset of normality aspects. The experiments conducted on several benchmarks show the effectiveness of the proposed approach.

Published in Transactions on Machine Learning Research (07/2024)
Masked multi-prediction for multi-aspect anomaly detection
Yassine Naji
yussine.najjiBeeu.
Université Paris-Saclos CEA, List, 91120, Palaiseau, France Université Paris- Sacia, CNRS, LISN, 91400, Orsay, France
Romaric Audigier
romaric.audigierrce..
Université Paris-Saclay, CEA, List, 91120, Palaiseau, France
Aleksandr Setkov
aleksandr.schhoocco...
Université Paris-Saclay, CEA, List, 91120, Palaisean, France
Angélique Loesch
ongelique.loescchcca..
Université Paris-Saclay, CEA, List, 91120, Palaiseau, France
Michèle Gouiffes
Université Paris-Saclay CNRS, LISN, 91400, Orsay, France
Reviewed on OpenReview: https: // openreview. net/ /trmm? id- Tuyb YcKipw
Abstract
In this paper, we address the anomaly detection problem in the context of heterogeneous normal observations and propose an approach that accounts for this heterogeneity. Although prediction baasdd methods are common to learn normality, the vast majority of previous work predicts single outcome, which is generally not sufficient to account. for the multiplicity of possible normal observations. To address this issue, introduce new masked multi- prediction (MMP) approach that produces multiple likely normal outcomes, and show both theoretically and experimentally that it improves normality learning and leads to a better anomaly detection performance. In addition, we observed that normality can be character- ized from multiple aspects, depending on the types anomalies to be detected. Therefore, we propose an adaptation (MMP-AMS) of our approach to cover multiple aspects of nor- mality such as appearance, motion, semantics and location. Since we model each aspect separately, our approach has the advantage of being interpretable and modular, select only subset of normalily aspects. The experiments conducted on several benchmarks show the effectiveness of the proposed approach.
can
1 Introduction
Video anomaly detection (VAD) is crucial for many applications such as video surveillance or autonomous driving for instance. However, it still an open research problem due to several challenges. The first one is the scarcily of anomaly examples and the lack of their corresponding annotations. Indeed, by definition, anomalics are unexpected and usually diverse, therefore, is infeasible samples. This makes classical supervised methods ineffective due to the class imbalance issue. Thus, this problem is often considered from the One-Clase perspective where a model of normality is learned from normal data only and detects anomalies outliers.
collect enough representative
Diverse approaches of modeling normal data exist in the One-Class anomaly detection literature. On the one hand, many existing methods model directly the distribution of normal data or a high-level representation of it,, denoted by P((())) where is a transformation applied to normal data samples X. This category inchudes probabilistic methods such diffusion models (Flaborea et al. (2023): WWatt et al. (2022)) or GANs Ravanbakhsh et al. (2017); Liu et al. (2018)) which learn probabilistic model of normality. At inference.

---

Published in Transactions on Machine Learning Research (07 (07/202) /024)
Normal observation
Normal observation
C
Multi-prediction
Single prediction
model
model
Prediction:
Predictions:
Figure An example illustrating the motivation behind performing multi-prodiction instead of a single prediction. We generated a synthetic dataset illustrating a road scenario where a car (in blue) is al an intersection, and can perform actions (turning left or right) which generates two different next states. The right figure shows single prodiction network trained to perform next position prediction, while the left figure shows another trained to perform multiple predictions using our proposed loss functions. Green arrow indicates correct matching between the ground truth observation at time & and model prediction, while red arrow indicates incorrect matching.
samples with low likelihood given the learned density function are considered as anomalies. Another popular family of approaches are distance-based methods (Ramachandra et al.. (2022) Singh et al. (2023) ). Those methods learn an embedding space and the corresponding metric, to ensure that abnormal data are far apart from the normal data. Other approaches perform clustering some low dimensional space (Ionescu et al.) (2019a b); Wang al. (2020)) to define normality regions, and the anomaly is deduced from the distance to them. Recent approaches such Park et al. (2020); Gong et al. (2019); Liu al. (2021) Bergaoui et al. 2022) model the normality via a discrete set of prototypes in the latent space. On the other hand, other approaches learn normal patterns by training a model on a pretext task, thereby learning features of normality such as appearance and motion (Georgescu et al. (2021a); Barbalau et al. (2022): Wang et al. (2022)) ). pretext task usually consists of learning (g) where and f are transformations of X, and g ((X) informative about f(X). A model trained on normal data only via those pretext lasks. For given test sample, the abnormality score can be inferred from the model's inability perform the task correctly. There are two main pretext tasks used in the VAD litcrature: reconstruction-baan based (Gong et al.)
Georgescu al. (2221)) and future prediction-based (Liu 2021 Dong et al. (2220) Nguyen & Meunier (2019); Taang et al. (2020 Reconstruction basde approaches generally train neural network which reproduces the normal training data from low dimensional space. The fundamental assumption that the model will not be able to generalize well to anomalies. Differently, future prediction-based approaches train model to predict a future outcome given the past. At inference, the anomaly score doduced from the prediction error.
(2019 : Bergaoui al. 2022): Park et al. 2020
et al. 2018): Naji et al. (2022) Liu et al.
The choice between reconstruction. based and prediction based methods involves trade-offs. While the former reconstruct training data well, they also tend to reconstruct anomalies due to the generalization abilities of neural networks (Gong et al. (2019)). ). On the contrary, the latter has the advantage of prodicting poorly anomalies, as the model cannol simply reproduce the input as with reconstruction--oeed. methods. However, these methods have difficulty predicting normal future scenarios because their diversity. Indeed, many existing futurcoprediction methods perform single prodiction, which is often not enough to characterize all possible future outcomes (Babacizadeh al. (2018)). To illustrate this point, let us consider road scenario (Figure D where the task consists in modeling the normal set of trajectories. We assume that the car is
2

---

Published in Transactions on Machine Learning Research (07/2024)
located at intersection at time t, and can turn left or right at time t 1. Note that both trajectories are normal in this case, but it is impossible to predict both possibilities with a single prediction. Even worse, this leads to an abnormal prediction (Figure ! right), which is different from all normal trajectories. In this paper, we aim to solve this problem by better modeling the distribution of possible future sccnarios. Thus, we propose a model that performs mulliple predictions instead of a single one. In order to cover normal possibilities, we use the nearest neighbor loss (Guzmán-rivera et al. (2012): BBhattacharya et al. (2018): Nguyen et al. (2018)), which is often used in the setting of multiple choice learning, and we introduce a new non-participation loss that ensures balanced training of all predictors. Thus, our approach offers the advantages of prediction-based methods in terms of poor anomaly recovery, and enhances normality learning through multi-preedition, improving anomaly detection performance.
Another VAD challenge is that, in order to detect anomalies, it is necessary to determine normality. However, the definition of what is considered normal depends on the context and the application, which also influences the choice of normality aspects to be modeled (c.g., appearance, motion, ctc.). While certain aspects of normality are not relevant. to detect anomalies for some applications (e-g., the location of a person on a sidewalk when the objective is to detect violence), they tend to be crucial in others (e.g., the location of person when the objective is to detect jaywalking). Therefore, we model several object-level aspects such as appearance, motion and class-semantics, as well as location-related anomalies. Our approach is interpretable and modular, since it assigns an anomaly score for each aspect. This allows us to adapt our method to applications that require only a subset of the aspects to be modeled while providing information about the anomaly type. In summary, our contributions are as follows: 1) a novel and generic masked multi- pretitppediition (MMP) approach for anomaly detection in the context. of heterogeneous normal data; 2) adaptation of MMP to model multiple normality aspects (MMP-AMS); 3) a new non-participation loss to betier model the multiplicily of normal scenarios; 4) a theoretical analysis and experiments showing the effectiveness of our methodology.
2 Related work
Multi-prediction learning: also known as multiple choice learning (Lee et al. (016)) 3 Dey et al. 2011 Lee
let al. (2017): Guzmán-rivera et al. (2012)) or multiple hypotheses learning (Rupprecht et al. (017)); ; Nguyen ct al. (2018))
task where multiple models are learned to produce diverse predictions. During training, samples are assigned to the minimum loss predictor. This technique has been used for tasks involving aleatoric
uncertainty such as future prediction [Bhattacharyya et al. (2018)), human pose estimation (Rupprecht et al.
(2017)), image segmentation (Guznén-rivera et al. (0012); Dey et al. (2017)). In the context of anomaly detection in images (a.k.a novelty detection), Nguyen et al. (2018) proposed a multiple hypothesis auto- encoder which performs multiple reconstructions. Differently from Nguyen et al. (2018), we perform masked multt-prediction by combining masking with multi-prediction in order to limit the capacity of the model to recover anomalies. We also propose a new non-parlicipation loss that penalizes only the predictors that do not participate in training, and show both theoretically and empirically that it improves the coverage of normal possibilitics. In addition, our framework learns multiple aspects of normality, cnabling it to detect the corresponding anomaly types in the context of videos. To our knowledge, our work is the first to propose multi-prediction approach for VAD.
Masked prediction for anomaly detection: the procedure of predicting a masked input has been studied in several applications of anomaly detection such as industrial inspection (Zavrtanik et al. (021] ; Tong et al. (2023) ; Huang et al.] (2022) : Jiang et al.) (2023) ), where a model trained to perform the inpainting task, hyperspectral imagery (Li al. (2023)) or time series (Fuu & Xue (2022)). In the context of VAD, most of previous studies perform temporal masking by performing tasks such as future frame prediction [Luu
et al. (2018) 2021); Dong et al. (2020); Tang et al. (2020)), middle frame prediction (Barbalau et al. (2022
Georgescu et al. (2021a) or video event completion (Yu et al. (2020)). Barbalau et al. (2022) proposed a spatial masking procedure which consists of random patch inpainting. Differently from those studies, we propose a joint spatio-temporal masking procedure in order to learn correlations between the spatial and temporal patterns of normal data. Morcover, we propose to perform multiple prodictions instcad of a single prediction in order to better learn diverse normal patterns and therefore improve anomaly detection performance.
3

---

Published in Transactions on Machine Learning Research (07/2024)
Video anomaly detection: most implicit VAD approaches use self-supervised learning to model normality, The model is trained on given task using normal data only and it is expected not to generalize well to abnormal samples. Usually, these tasks are designed to characterize a certain aspect of normal data such as appearance or motion, which allows the model to detect the corresponding anomaly types. Hasan et al. (2016) were one of the first to propose a reconstruction-base method by training an aulo-encoder to recover handerafted appearance and motion features. As pointed out by Gong et al. (2019), anto-encoders are able to reconstruct anomalies due to the extrapolation capabilities of deep learning models, which is not suitable for distinguishing between normal and abnormal samples. The reconstruction task can be further constrained using a memory module as proposed in (Gong et al. (2019)). Georgescu et al. (2021b) proposed to train a model via an adversarial objective function, where normal data is well reconstructed and some pseudo-anomalies are explicitly miste..nsstrutted A major self-supervised paradigm to learn normalily consists in training a model to perform future frame prediction. Liu et al. (2018) trained a generator using an adversarial objective function to prodict a future frame and its optical flow given few past frames.
Ravanbakhsh et al. (2017) trained two generators to perform image- translation between RGB and optical flow modalities in order to learn both appearance and motion normality, Liu et al. (2021) introduced hybrid framework for frame prediction and optical flow reconstruction at the object-level by making use of a pretrained object detector. During inference, the anomaly score is computed based on a sampled future object-level frame. However, one sample may not be representative of the full distribution of future scenarios. In order to cover various modes of this distribution, we propose to train our framework to produce diverse and representative predictions, using the nearest neighbor loss (Guzmán-rivers et al. (2012) and our novel non- participation loss. Similarly to Liu et al. (2021), we propose to model normality at the object-level to provide better robustness to scene changes and background variety. Recent works introduced other self-supervised tasks for object-level normality learning such spatio-temporal jigsaw puzzle ( (Wang et al. (2022) ) video
event completion (Yu et al. (2020)) or random patch inpainting (Barbalau et al. (2022)). ). Barbalau et al. (2022]; Georgescu et al. (2021a): Shi et al. (2023) proposed to combine multiple tasks such as arrow-of-time prodiction, motion irregularity, middle-frame prodiction and knowledge distillation to characterize object- level normality. Differently from these previous works, our approach performs multiple predictions instead of a single one for each normality aspect. Furthermore, our method can detect location- -reeddende anomalies which have not been addressod in the aforementioned methods. To our knowledge, only Doshi & Yilmaz 2020) addressed the modeling of location attributes at the object-level. The authors proposed non- parametric model of hand-crafted object-level features which included the object position. Differently, we model the distribution of normal bounding boxes separately for each object class which allows to detect class-wise location anomalies.
3 Masked multi-prediction for normality modeling: a preliminary study
In this section, we introduce masked pretiippediction (MMP) approach and motivate it theoretically and experimentally in the context of anomaly detection. We first demonstrate the importance of multi- prediction compared to single prediction. Then, we show the impact of the loss choice to ensure diversity and likelihood of predictions. Finally, we discuss the importance of spatial and temporal masking in improving anomaly detection performance. The proofs of all propositions are provided in the supplementary material. Preliminary experiments to support our analysis have been carried out on the following datasets:
This dataset is used to show the impact of performing multi-prediction instead of single prediction in the context of anomaly detection. Indeed, we generated synthetic dataset (Figure D illustrating a road scenario in which a car is at the intersection of m roads. In the normal case, the car moves towards one of the roads. More precisely, we assume that the x axis is oriented from left to right and that the y axis is oriented from bottom to top. We indicate the position of the car by its 2D coordinates (x, Our synthetic roads datasel consists of sequences of 2 pairs of coordinates indicating the position of the car at times t and ++1 1. Therefore a sample X is in the form : X = (Pc, Pt+1) = ((z1, (# Mt41)) E I[-1, 1]a For simplicity, we will assume that the car is always at position: Pt = (0, 0) at time t and that, in the normal
Synthetic roads dataset:
case, it uniformly moves to one of the following positions at time t + 1: (Pt41 (1, 2x((-1)
(()
1))
m-1
ie[t.m]
4

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
9
X
f
((
masking
X
XM
(m)
Figure 2: Overview of our methodology: the masked multi culltiprediction network (MMP) denoted by receives masked sample Xas and produces n likely predictions: f(XM) = (((()))E(L which are compared to the original sample X. The model is trained using LNN and Lxp. At inference only CNN is used.
The parameter m 2 indicates the number of roads the car can take. Therefore, the set of normal samples N is:
N N= (o Peta)(E) [L,m]) Pe-1 P.f1 (1, 2 (i-1) 1 -1) 1)}
m-1 1) }}}
The set of anomalics A is:
{ (p E [1, p41
2 (i (1,2x(i-1) m
A=
We assume that the models are trained on N by predicting the normal positions at time t+ 1: given the position time f, and tested on representative samples of NUAA, mcaning that all normal positions are in the test set and that abnormal positions cover sufficiently [-1,
1
sE[1.m]
MNIST [LeCun et al. 2010)):: this dataset is used to show the importance of masking. It contains handwritten digits from to 9. As this dataset
not designed for the one-class setting, it is adapted by considering particular class as normal in each case. We use the training-testing split of the dataset and perform training only on samples from the class considered normal. During inference, all test data are used. Test samples from the class on which the model has been trained are considered normal, and other classes are considered abnormal.
3.1 Multi-prediction vs. single prediction
In the following analysis, we will focus on the general problem of sample prediction (denoted by X) given a masked input Xa X @ M, where M the mask applied to X. The sample prediction problem boils down to learning the conditional probability distribution Q A P(X]Xar), which can be done using model that receives as input masked sample XM and generates candidate samples according P(X)MM) However, learning this distribution via single prediction model does not capture its multi-modalityy shown in Figure Indeed, a single prediction model is generally trained to predict a sample X by minimizing:
Leingle (N(XM), X) ll9(XM) - XII
(1)
However, since X is not deterministic given XM, the network minimizes:
Leingle (g(XM)) A Ex~o(Il((MM)) - X())
(2)
This loss can be interpreted the average distance across all possible samples X Q. If mcan squared error (MSE) is used, the global minimum of the loss is achieved by model g" that predicts the conditional expectation of X:

---

Published in Transactions on Machine Learning Research (07/2024)
Proposition 1. Let 9 be single prediction model trained using the L2 norm. The minimam loss is achieved for a model o² that predicts the condilional expectation:
9- (X = B(X(XM)
This shows that the output of single prediction model is an average of possible samples (X Q) in the best case, which is sub-optimal. In order to better cover normal possibilities, we propose to train a masked multi-prediction model (MMP): = (f(1), ( ((_)),,,_()))))))))))))))) = (f(4)) [=(l.n) to predict n likely possibilities using the nearest noighbor objective function (Guzmán-rivera et al. (2012)). An illustration of MMP is provided in Figure 2 The nearest neighbor loss penalizes only the distance to the closest prediction. Formally, the loss between the set of predictions f(XM) (f(h) (XM)))ee)) and a sample X can be written as follows:
X))
min (XM) - -XI [E[
(3)
The corresponding expected loss across all possible samples X ~ Q is:
(fx(f(XM) & Ex-o(Cxx(((XX))
(4)
Training a MMP model using the previous loss can achieve better fitring of normal data (i.e lower training loss) as shown in the following proposition:
Proposition 2. Let X sample from P, F the space of self-maps of [o, 11 C and f e mig
(fx(f(X))) The minimum expected loss is lower when using multi- pultii-rediction than when
using single prediction:
[Nx(f"(XM)) S Esingtels" (XM))
Moreover, in the case of MSE, by considering the index of the closest predictor to X: K arg min (X))((((X X12 we have:
Esingle (g" (XM)) - CNN = EK
Ex( EX XM) B(X K) K)
The last result shows that the prediction loss gap between a single prediction model and an MMP model increases with the distance betweon optimal single prediction and optimal multiple predictions. Consequently, this gap increases as distribution becomes "more" multi-modal. Moreover, MMP network trained using the nearest neighbor loss provides better anomaly detection performance than single prediction model, shown in section 5.4 and demonstrated in the case of the synthetic roads dataset. Indeed, Proposition 3 shows that MMP network trained using the nearest. neighbor objective function LNN can theoretically achieve a perfect anomaly detection score the synthetic roads dataset, if it reaches the global minimum of the training loss, whereas a single prediction model cannot, even if the global minimum is reached.
Proposition 3. A MMP model with a number of predictors corresponding to the number of roads m and reaching the global minimum training loss using CNN, achieves an AUC of 100% on the synthetic roads dalasel. On the other hand, single prediction model achieving the global minimum training loss using Exingle cannot achieve an AUC of 100%.
3.2
Impact of loss functions
3.2.1
Nearest neighbor loss
The choice of the nearest neighbor objective function is important to ensure diversity of predictions and a better fit. to normal data (Guzmán-rivera et al. (2012)). Indoed, if we use instead naive objective function Cnaive & 1 prediction, shown in the following proposition:
n Exelaa f (X The diversity of predictions is lost, which amounts to making single

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Proposition 4. Let (f the expected loss corresponding to Lnaive (f(X)))
1 E
=Ex=@
(XM) -
(5)
RE[LM!
In case of L2 norm, the minimum expected loss is achieved by MMP model: f"= (fo RE[[in] (fo(t)))
such that
(Vi E [I,n]) (x (Xx)) = MM) which is similar to perform single prediction.
3.2.2 Nonpparticipation loss
In practice, M( is often intractable, highly dimensional and we have only access to samples from it (o.g., the future frame observed in video). Therefore, it is infeasible to train the network via NN (f(XM) since it involves an expectation over all possible samples X, which is difficult to compute. Instead, we train the network via (f
where X the actual observed sample. This loss encourages the model to produce diverse predictions in order to reduce the distance between the sampled data point X and its nearest neighbor f(k') (Xx) where k' = arg ming (XM) X|I.
arg
Nevertheless, this objective function only optimizes the best among all predictions (Xxs) Thus, the model receives sparse signal which may lead to optimizing only predictors which are selected nearest neighbors during training. Other predictors remain far from the data subspace, as they are never selected and therefore never optimized. In order to overcome this issue, we introduce a novel object ive function called the non-participation loss in order to optimize these predictors. More specifically, we collect the indicos of unoptimized predictors C [1,n] that were not selected as nearest neighbors the last epoch of training (a small threshold 5 is used in practice, cf. Section 5.2 Then, we optimize those predictors (FF)) Ieu via the non-parte con-participation loss that can be written as:
Σ (1 XII pell
(6)
Thus, train the network via weighted combination of the two losses L LNN + XCxP. The following proposition shows that adding the non-participation objective to the nearest neighbor loss can only decrease the prediction error:
Proposition 5. Let X the training dataset composed from normal samples. We denote by f= (F(t))
a MMP model optimized via the nearest neighbor objective CNN and achieving a loss (f X) on sample X X. Let f (f(() (EEL,n) a MMP model resulting from training the non com-optimized predictors of f via the non-particioation loss LNP, and CNN (f(M)) the loss of f on sample X. We have (VX E X):
[NN(f(XM), S (xM()(XM),
Table 1: Influence of training loss choice. The metrics participation and prediction loss (x10²) are calculated as the average obtained by training each model ten times, each time on different class from MNIST.
Variant
Losses
Pretraining
À
Participation 1
Prediction loss J
vl (1 pred.)
Leinghe
= LNN = Lnaiue
1/1
7.04
v2 (10 pred.)
Lnaive
X
10/10
7.29
v3 (10 pred.)
LNN
X
1/10
7.09
v4 (10 pred.)
4.4/10
5.32
v5 (10 pred.)
0.01
10/10
4.57
v6 (10 )
Lxx + XLxP
x
0.1
10/10
4.26
v7 (10 )
1
10/10
4.42
v8 (10 pred.)
10
10/10
4.52

---

Published in Transactions on Machine Learning Research (07/2024)
In order to empirically verify Propositions 2, 4 and 5, we trained single prediction model composed of one predictor via Lisingee as well as three MMP models each compased of predictors, and each predictor has the same architecture as the single prediction model. The MMP models are trained via the following training objectives: LNN, LNN + ALNP and Lnaive. Since LNN optimizes only the best among all predictors, we observed that is very sensitive to the initialisation of the predictors. Thus, we performed two lypes of training when using this loss, the first consists in randomly initializing predictors (similarly to other losses), and the second consists in pretraining the predictors using Lnaive. More precisely, we train the predictors using Lnaive for the first epoch of training. The results shown in Table | First, we observe that when an MMP model is trained using CNN, it achieves either lower loss (v4 vs. v1) when the predictors are pretrained, or an equivalent loss in the worst case (v3 vs. vl), when only one predictor is optimized.
Second, we observe that when an MMP network is trained via Cnaive: it achieves a loss similar to Ceingle: which is line with proposition 4. Note that the loss is slightly higher when learning an MMP model using Cnoive than a singlo-prcdiction model, we explain by the fact that the former requires training all predictors instead of just one. When using LNN + XLNP, all predictors participate in the training, resulting in a lower prediction loss. To test the sensitivity of training to the À parameter, we experimented with different orders of magnitude vr5,v6,v7,v8) and observed slight changes in the prediction loss. One possible explanation for this low sensitivity to the À parameter is that the LNP loss optimizes predictor parameters that have not been optimized by LNN (by definition). Consequently, the two losses optimize different. sets of parameters since the predictors are independent. In addition, once all predictors start to be selected by CNN, CNP no longer used. Additional experiments on bascline models are provided in the supplementary material (section C.1).
3.3 Impact of masking
The role of masking is to ensure that the model does not. learn the trivial identity function, which would result in a good reconstruction of both normal and abnormal samples, and therefore a poor discrimination between them. Indeed, masking forces the model to learn the specificities of normal data in order to predict well normal samples while having a poor prediction of abnormal samples. This would result in a better detection of anomalies. It is important to note that masking can be applied either spatially or temporally. For example, patch masking, illustrated in Figure 3 is clear example of spatial masking. On the other hand, in the case of future frame prediction based on a current frame, masking is implicitly performed at the temporal level. In this case, the sample X can be considered as a pair of the current and next. frames: (Xp, Xr) and the masked sample XM is the current frame Xp. In order to illustrate the importance of masking, we carried out comparison between our masked multi-prediction model (MMP) and a reconstruetion network, both trained on single class from MNIST (Figure 3.. The training details are provided in the supplementary material. Different masking strategies are presented in Tables 2a and 26 We observe that when masking is used, the recovery of abnormal patterns is more difficult than that of normal patterns, resulting in better anomaly detection performance (e7 vs. e4 in Table 2b). Moreover, it can be seen that a high percentage of masking is advantageous for anomaly detection on MNIST. However, a compromise arises when selecting the patch size for masks. Specifically, as the patch size increases, the prodiction task becomes more challenging, since the model is required to predict more global information.
Table 2: Influence of masking strategies on anomaly detection performance.
(b) Influence of the percentage of pixels masked on anomaly detection performance (Mean AUC) on MNIST, using patch size of 16 x 16.
(a) Influence patch of masks on anomaly de- tection performance (Mean AUC) on MNIST, using 75% percentage masking.
Exp. Masking percentage
Mean AUC
Exp.
Patch size
Mcan AUC
0%
85.9%
el
8 8
89.0%
e5
25%
90.4%
e2
16 x 16
94.0%
e6
50%
91.3%
c3
32 x 32
90.2%
75%
94.0%
8

---

Published in Transactions on Machine Learning Research (07 (07/2024)
4 Multi-aspect normality modeling
In this section, present masked multi- prediction framework for modeling appearance, mo- tion and semantics HIMP-AMS) which is an adap- tation of MMP to VAD. An illustration of our frame- work presented in Figure !
Predictions
MNP
7
7
7
First we introduce the notations for this section:
Ress
7
7
30mml
NMM
7
7 7
We denote the normality aspects follows: F for future frame prediction (appearance), O for opti- cal flow prediction (motion); C for class prediction (semantics), and By ,B .w for the bounding boxes center coordinates, height and width respec- lively (location). We denote by one of these as- pects which can belong to the set of appearance, motion and semantics aspects: r = C) or lo- cation aspects: @ [Bx,By: Bh, }. )- fA is the network used for modeling work is used for multiple aspects we denote it as fA. where A is the set of modeled aspects. Let X a normal object extracted from a frame at time t, following the normality distríbution IP and having bounding box with center coordinates (X XBY ) height XB, and width XBw- The image of X at denoted respectively by (Xp, xF). The masked current im- age of X is denoled by XMP A Xp @ M, where M is the mask applied to Xp and @ the Hadamard product. The onc-hot encoding object classes and the object-level forward optical flow are denoted by Xc; Xo respectively.
7
Rec.
7
7
NMMP
7
4
4
4
Rec.
4
Abnorma
MHVP
2
Rec.
2
2
aspect A, if net-
Figure 3: Visualization of two normal samples (class 7) and two abnormal samples (classes 4 and 2) from MNIST. The first column indicates the input of the model, the second column shows the predictions of two models: MMP (ours) which receives masked input and preforms multiple predictions, and reconstrue- tion network (Rec.) which receives non-masked in- put and preforms single reconstruction. The nearest predictions are circled in green. The third column in- dicates the corresponding ground truth.
times (Present) and + (Future)
4.1 Overview of MMP-AMS
Our framework learns the appearance, motion and semantics aspects of normality, by predicting multiple future frames as well as the corresponding optical flow and class vectors given a masked current frame. this way, the model learns: appearance features via unmasking, motion features via future prediction and semantic features through class prediction. For each aspect, the model performs multiple predictions in order to take into account the diversity of normal data w.r.t. each aspect. We model the three aspects: r = (F,, O, C} via three MMP networks (fA)Aer (one for each aspect). Therefore, the full model can be written as: fr (fA)Aer = (1F) ((FA)))E([L))))) AEr The model predicts respectively different possible future
frames (& )eEI,,. optical flows (8(²) )<[[..] and classes (X ) given masked current frame XMP. The k-th prediction for an aspect A E I is given by: X- A ((k) (XMP). The full model fr is trained to minimize the nearest neighbor loss and the non-participation loss for each of the three aspects. Specifically, the following sum is minimized:
X)) & ENNN(F XA) AXxr(()(XMM))
(7)
AET

---

Published in Transactions on Machine Learning Research (07 (224)
he
h
h2
hn
XMP
fa
fA
fa
x
(9) A
(n) A
Xp XA
Figure 4: Overview the proposed masked multi-prediction framework applied to appearance, motion and semantics (MMP-AMS). We show the model predictions for an aspect A, which can be one of the following aspects: future frame prediction F (appearance), optical flow prediction O (motion) or class prediction C (semantics). We consider A F in the illustration. The network receives masked image at time & and produces multiple predictions (& (x)(k) ] for an aspect A. The model is trained using NNN and CNp. At inference only LNN used. Networks of the same color share parameters.
4.2 Network design
In order to allow the number of predictions be varied without changing the total number of model parameters, introduce novel architecture which consists of recurrent neural network R, first state predictor P as well as three state conditional networks (S^)Aer = (JA(XMM, ha ha))kep...)) Aer
(illustrated
in Figure 4b. For cach aspect A, we propose to replace the predictors )xE(in) by state conditional network, / which receives as input the masked frame XMP as well as a state hi generated using the recurrent neural network R. Therefore, k-th predictor (k) is equivalent to conditioning the network fA on the state hg. Formally, we have (AV E (F,O,C)) x = 1 (X fa(XMr, hx) = fa(NMr, R(hx-1)). The recurrent architecture ensures that the hidden state h4 contains information about previously predicted states, and encourages the model to explore new normality regions in order minimize the nearest neighbor error. The networks architectures, the number of parameters and training pseudo-code are provided in the supplementary material.
4.3
Location module
In order to detect location related anomalies, we learn the distribution of object positions for each normal class using simple Gaussian model. More specifically, given a class Xc, we model the distribution of bounding box centers (X Xny), height and width (Xng- Xnw) of objects belonging to Xc, using 4 Gaussians: (Xc), for A E θ [Bx, By, BH, Bw}. The mean (x(X)) and the standard deviation (Xx)) for a given dimension A are predicted by network fe using the negative log-likelihood loss:
Σ log (og()x(X))) (Xxx-a(XXx) 2
Lo(fe(Xc), X)
x & 2
(8)
(x()))
Aee
10

---

Published in Transactions on Machine Learning Research (07 (224))
4.4 Anomaly scoring
At inference lime, we perform the same pre-processing steps on a lest sample and compule the appearance, motion and semantics anomaly scores (r(X) as well as the location anomaly scores Se by summing the z-scores of the prediction errors across aspects. This allows balancing the contribution of each aspect to the anomaly score. More formally, given an object X at frame Fe:
WA LNN (fa((N)) Pr
LNLL LNLL (Xa:(aa,) p4
(r - Σ
(9) Se(X)= &
Σ
(10)
AA
0A
AET
Ace
where fA, 9A are the networks and parameters after training. μa and a are respectively the expectation and the standard deviation of the loss function which is either LNN or LNLL estimated from normal training data for a given aspect A. WA is the weight assigned to each aspect, which may vary depending on the application. The final anomaly score a weighted combination of the appearance, motion, semantics and location anomaly scores: S(X) e (r() + (X)).. The parameter y can be chosen depending on whether we aim to detect location anomalies. The frame level score for a frame is the maxinum object-level score in the frame.
5
Experimental study
5.1 Datasets and evaluation metrics
We adopt the following metrics: the Region- Basod Detection Criterion (RBDC) and the Track-Based De- tection Criterion (TBDC) introduced by Ramachandra & Jones (2020) as alternative to frame-level AUC widely used in the literature. The latter metric mcasures the anomaly detection performance at the temporal level only. However, it does not evaluate the capacity of the model to localize anomalies spatially because it does not penalize false positive regions detected in abnormal frames as pointed out by Ramachandra & Jones (2020). We perform experiments on the most commonly used datasets for the onc-class and object-centric scenario. UCSDped2 (Mahadevan et al. (2010)) is a single scene dataset which includes anomalies such as riding a bike and driving a vehicle on a sidewalk. [Ramachandra & Jones (2020) provided region-level and track-level annotations for the RBDC and TBDC metrics. ShanghaiTech (Luo al. (2017)) contains scenes of different backgrounds. Anomalies include jumping, running, stalking a sidewalk. The region- level and track-level annotations are provided Georgescu el al. (2021b). CUHK Avenue (Lu et al. 20033) is a single scene dataset which consists of videos with abnormal events such as running or walking towards the camera. We use the improved set of annotations proposed by Ramachandra & Jones (2020) which take into account some static anomalies that where not considered in the original annotations.
5.2 Implementation details
For fair comparison to other object-centric approaches, Yolov3 (Redmon & Farhadi MSCOCO is applied for object detection, using the implementation of MMDetection (Chen et al. (0019) with an objectness threshold of 0.5 for UCSDped2 since objects have low resolutions and 0.7 for Avenue and ShanghaiTech. The set of objects detected by the used implementation contained very small false positives which are filtered out based on their area (lower than 350 pixels). Optical flow maps are computed using the official implementation of FlowNet2 (Reda al. (2017)) ) as in (Liu et al. (2021)). For anomaly scoring, we keep only the optical flow magnitudes since the optical flow orientation maps are not precise enough to be predicted for small displacements. The detected objects well the corresponding optical flow maps are resized to 64x64. For the mask M, we remove 50% of pixels using grid of 4x4 pixels. Regarding the distance used for anomaly scoring in Section A we use the Lı distance for RGB and optical flow, as well as the cross entropy loss for class probabilities. We train the network for 150 epochs for UCSDped2 and Avenue and for 400 epochs for ShanghaiTech using Adam optimizer with a learning rate of 10-3 3 with a batch size of 640 for the biggest dataset ShanghaiTech and 64 for UCSDped2 and Avenue, We set (y, Wr, Wr, WC, W BB WBg WBw >) (0, 1, 1, 0.1) for UCSDped2 and ShanghaiTech
2018)) pretrained on
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Comparison of our approach to state-of-the-at object-centric VAD methods on RBDC and TEDC
(%). Best results are bold, second best are underlined.
UCSDped2
ShanghaiTedl
Avenue
Method
RBDC
TBDC
RBDC
TBDC
RBDC
TBDC
lonescu al.
2019a
52.8
72.9
20.7
44.5
15.8
27.0
Liu et al.
2021
41.1
86.2
Georgescu et
2021b
69.2
93.2
41.3
78.8
65.1
66.9
Georgescu et a
2021a
72.8
91.2
42.8
83.9
57.0
58.3
Bergaoui
2022
80.1
95.4
51.5
82.2
75.8
70.0
Naji et al. 2022
77.2
98.5
51.6
84.6
75.3
73.4
Georgescu et 22221a)
Ristea
22022
40.6
83.5
66.0
64.9
1 et al.
2018
Ristea et al. 2022
18.5
60.2
20.1
62.3
2021
Ristea
(2022
45.5
84.5
62.3
89.3
Bar Barbalau et al.
2002
47.1
85.6
47.8
85.2
Ours (MMP-AMS w w/o location module)
84.0
99.0
55.9
85.7
67.4
68.0
Ours (MMP-AMS w/ location module)
77.7
74.2
For the non- participation loss, select predictors which have participation below ô 5%. Regarding the inference time, our model processes batch of objects in a frame taken from Avenue in 18ms on single Nvidia- Titan-X GPU. Therefore, it satisfies the real-time constraints, given real-time object detector and optical flow extractor that can run in parallel. In our implementation, the inference time of Yolov3 is 50 ms and ms. Therefore, if optical flow estimation and object detection are parallelized, our
and 1111,,
for Avenue. We explain the parameters choices in Section 5.4
that of FlowNet2 pipcline can run at 13 FPS. Furthermore, the method requires accoss one future frame only computc the anomaly score which allows online application.
5.3 Evaluation results
This section presents the results (Table 3 of our method on the benchmarks UCSDped2, Avenue, Shang- haiTech. Since focus object-level anomalics and for fair comparison with state-of-the-art object- centric methods, we use the same baseline object-detector (Yolov3). Qualitative results are provided in the supplementary material.
UCSDped2. On this datasct, MMP-AMS outperforms previous works on RBDC (+3.9 p.p.), and reaches the stale-of-the-art on TBDC. can be seen that the optical flow prediction particularly relevant for this dataset (ef. ablation study in Section 5.4) due the fact that most anomalies have abnormal motion.
ShanghaiTech. Unlike other datasets, this contains multiple scenes for training and testing. Neverthe- less it shares similar normal context across scenes. Since method is object-centric, it is less sensitive to scene changes. MMP-AMS achieves significant improvement (+4.3 p.p.) in terms of RBDC and slightly outperforms other methods on TBDC. The improvements in terms of anomaly localization can be explained by two factors. First, the choice aspects allows us to detect appearance, semantics and behavior anomalies (cf. ablation study in Section 5.4 Second, we found that multi-prediction beneficial for this dataset, which is explained by a high scene complexity Icading
multiplicity of normal scenarios.
Avenue. This dataset is challenging because it contains several types of anomalies, such as human behavior and unusual objects. In addition, unlike previous datasets, it contains location-dependent anomalies. It is important to note that no method consistently outperforms the others in all metrics. However, MMP. AMS combined with the location module provides a good compromise between RBDC and TBDC. More specifically, our approach achieves the best performance in terms of RBDC (+1.9 p.p.). and a moderate TBDC. Nevertheless, our method achieves the bost TBDC (+0.8 p.p.) among methods which use a similar temporal window (Georgescu et al. (2021b); [Bergaoii al. (2022); Naji et al. (2022)). A more detailed analysis of the impact of increasing the temporal window is provided in the supplementary material. The
12

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 4: Comparison between different training loss functions using the metrics participation, diversity and prediction loss computed for normal samples from Avenue using the RGB modality. The participation the selection frequency of predictor. The diversity (x10²) is the average pixelwise distance between predictions from two different predictors (higher is better). The prediction loss is the nearest neighbor loss (lower is better). Green and red respectively indicate best and worst results.
Training metrics
Variant
Losses
Participation
Diversity f
Prediction loss 4
v1 pred.)
Lnuive
LNN
100%
0
0.164
v2 (3 pred.)
Lnaive
29%, 13%, 58%
0.03
0.165
v3 (3 pred.)
LNN
65%, 35%, 0%
1.4
0.159
v1 (3 pred.)
LNN ALNP
35%, 51 %,, 14%
1.8
0.157
good performance in RBDC is partly due to simultancously taking into account the appearance, motion and location aspects which are relevant for this dataset. Moreover, MMP-AMS alone oulperforms methods
designed for appearance and motion anomalies GGorresccu et al. [202bba) a): Barbalau et al. (2022) Liu et al. (2021b; Ristea et al. (2022); Ionescu et al.] (2019a)) in terms of RBDC. When combined with the location module, it outperforms all methods under consideration. This shows the complementarity of our two modules for this dataset.
5.4 Discussion
5.4.1 Impact of multi praliiipediition
As mentioned in the introduction, there are trade-olfs between reconstruction-basel methods and future prediction-based methods. While the former reconstruct training data well, they also tend to reconstruct anomalies. On the contrary, the latter predict anomalies poorly, however, they predict less well normal data. Our approach embraces advantages of both families. Indeed, as our model has only access masked current image, it cannot recover anomalies well, and thanks to multi-prediction, it fits nor- mal data better than single-prediction methods, as shown in Proposition 1. In order to empirically ver- ify our claims, we compared MMP-AMS performing only one prediction (vl in Table 4 with the same framework performing multiple predictions (v2,v3,v4 in Table 4.. We observe in Table 4 that a multi- prediction network achieves a lower prediction error than a single prediction network only when LNN is used ((v2,v3,v4) vs. vl). On the one hand, we observe that the model actually produces similar predic- tions when trained with Lnaine (v2 in Table 4), which is coherent with Proposition On the other hand, when train the model using CNN, produces a higher diversity of predictions (v3, v4 in Table Indeed, LNN encourages predictor specialization, since it penalizes only the best guess. This ob- servation is consistent with Proposition 1. Nev- ertheless, optimizing LNN alone leads to the non- participation (v3 in Table 4 some predictors in the training, since they are never selected as near- est neighbors. This explains the introduction of the non-participation loss LvP (v4 in Table 4 which en- sures that all branches get optimized. Empirically, we notice that it allows the model to better fit nor- mal data since it helps to decrease the prediction loss and increases diversity, In terms anomaly detection performance, we trained the MMP-AMS framework to predict up predictions on Avenue (Figure 5). We observe significant increase in
68
68
67-
5
Number of prediction
Rumber predictions
Figure 5: Influence of the number of predictions on MMP-AMS performance (RBDC, TBDC % scores on Avenue).
13

---

Published in Transactions on Machine Learning Research (07/2024)
Table 5: RBDC and TBDC scores in % obtained by incrementally combining the aspects. Best performances are bold. The normality aspects are denoted as follows: for future frame prediction, for optical flow prediction, C for class prediction.
UCSDped2
ShanghaiTechi
Avenue
Ablation
Location module
Set of aspects
RBDC
TBDC
RBDC
TBDC
RBDC
TBDC
a1
{C}
64.7
75.2
37.5
58.8
04.5
18.0
a2
{0}
82.5
99.5
48.7
84.2
17.2
54.5
a3
X
(w/o masking]
61.9
82.1
47.2
82.2
67.5
66.7
a4
X
{F}
66.8
85.1
48.4
83.4
68.2
66.7
a5
X
(F.C)
76.7
94.9
53.8
83.2
68.5
67.5
a6
(F,C,O)
84.0
99.0
55.9
85.7
67.4
68.0
a7
(F,C,0)
80.5
94.2
52.6
83.1
77.7
74.2
Table Influence of masking strategies on anomaly detection performance on UCSDped2.
(b) Influence of the percentage of pixels masked anomaly detection performance, using patch size of
(a) Influence of patch size of masks on anomaly de- fection performance, using 50% percentage masking.
x
Exp.
Masking percentage
RBDC
TBDC
Exp-
Patch size
RBDC
TBDC
e4
0%
61.9
82.1
x 4
66.8
85.1
e5
25%
66.6
82.3
c2
8 x 8
67.6
82.3
50%
66.8
85.1
16
65.3
79.2
e7
75%
61.4
79.8
all metrics (RBDC: +4.1 p.p., TBDC: +1.8 p.p.) until 3 predictions. However, in the case of 4 predictions, performance decreases slightly but remains superior to that of a single prediction. This suggests that 3 predictions are enough to model the diversity of normality for this dataset.
5.4.2 Impact of the choice of normality aspects
In MMP-AMS, we introduced multiple aspects to capture diverse types of normality patterns. This allows the model to jointly learn spatio-temporal fine grained patterns via unmasking and future prediction, as well as the object-level semantics through class prodiction. Those aspects are complementary, especially for datasets that contain diverse appearance, motion and semantics anomaly lypes such as ShanghaiTech. This results in performance improvement when incrementally adding more aspects in the normality modeling (a2, a3, a4 in Table 5). We notice that class and optical flow predictions are less relevant for Avenue dataset. This can be explained by two reasons: 1) most anomalics this datasct are done by humans for which the class information is not relevant. to detect anomalies; 2) optical flow predictions are not enough to characterize complex motion patterns in the scene that would require additional 3D information. Therefore, we give them less weight for anomaly scoring (cf. weights WA detailed in Section 5.2h. Regarding the masking of the input (cf. a2 vs. al in Table 5) we can observe that beneficial for all datasets especially for UCSDped2. This suggests that constraining the prediction task by masking the input makes the prediction even harder for abnormal objects, which leads a better discrimination between normal and abnormal samples. The inffuence of masking parameters is presented in Tables Sa and 8b We observe that optimal performances are achieved using less masking compared to MNIST, which can be explained by the higher complexity of the urban scenes compared to MNIST images. Concerning the location module, modcling the distribution of bounding boxes (cf. a6 vs. a7 Table significantly improves the results for Avenue dataset. This can be explained by the fact that some anomalies Avenue are related to the position with respect to the camera, which is not the case for UCSDped2 and ShanghaiTech, that do not include location-dependent anomalics the definition of what is considered as abnormal. For the sake of consistency with the definition of anomalies in these datasets, the location aspect not taken into account. (cf. parameter 0 detailed in Section
14

---

Published in Transactions on Machine Learning Research (07/2024)
We explain the performance drop when adding the location module MMP-AMS (cf. a6 a7 Table 5) for those datasets, by the fact that the full model (MMP-AMS w/ location module) becomes able to detect location related anomalies, however, those types of anomalies are not included in the definition of what is abnormal in both UCSDped2 and ShanghaiTech. This means that samples, for which only the location aspect is abnormal (cf. Figure 3 the supplementary material), are marked as normal by the ground truth for those datasets, but since the location module detects those samples as abnormal (since they have an abnormal location), they are considered an anomaly by the whole model (MMP-AMS w/ location module), therefore leading to false positive detection w.r.t. the ground truth for those datasets. Thus, even if the location module detects location related anomalies well, we still can observe drop in performance w.r.t. the ground truth for those datasets. These results show the importance of defining the aspects of normality that are relevant to each user application, in order to achieve optimal anomaly detection performance. As our approach models these aspects via separate networks, it allows aspects of normality be weighted according their relevance to the types of anomalics to be detected. This also provides an end-user explanation of which aspects cause an anomaly score that would trigger an alarm.
5.2
5.4.3 Limitations and Future work
One downside of our approach is that it depends on supervised object detectors which are usually trained in closed world manner. However, for some applications, anomaly detection can be aimed at finding objects not seen during training. To address this, it would be interesting expand our method using open-set object detectors, which are better at adapting to out-of-distributio objects. This could reduce the number of missed anomalies caused by missed detection. Moreover, our method can be improved by adding further normality aspects, which are useful for a given application. For example, would be interesting to model long- merm dependencies such as trajectories. This improvement is valuable for spotting anomalies like loitering, where having a longer context is crucial for an accurate identification.
5.4.4 Broader impact
Our work respects the othical guidclines applicd in contemporary computer vision rescarch. Although anomaly detection models can advantageously address user's needs in many applications (e.g. for safety, security and efficiency in autonomous vehicles, video-surveillance, industrial inspection, etc), some negative impacts could arise in the context of misuse or unintended uses, any machine learning model. Therefore, it is very important to take ethical considerations such as safety, fairness and privacy into account when deploying these models.
6 Conclusion
In this work, we addressed the problem of modeling a heterogeneous and multi-aspect normality. For this purpose, we proposed a masked multiple prediction approach (MMP) that is adapted to the multiplicity of possible scenarios. We showed both theoretically and experimentally that modeling the distribution of normal data via multiple predictions improves normality learning and anomaly detection performance. We also discussed the importance of determining the relevant aspects of normality for given application order to achieve satisfactory performance, and proposed to model several important aspects of normality such appearance, motion, semantics and localization. As model each aspect separately, approach has the advantage of being both interpretable and modular.
Acknowledgement
This publication was made possible by the use of the Factory-AI supercomputer, financially supported by the ne--e- France Regional Council.
15

---

Published in Transactions on Machine Learning Research (07 (024))
References
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational vidco prodiction. In ICLR, 2018.
Antonio Barbalau, Radu Tudor Ioneseu, Mariana-Iuliana Georgeseu, Jacob Ducholm, Bharathkumar Ramachandra, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B. Moeshund, and Mubarak Shah. SSMTL++:: Revisiting self-supervised multi-task learning for video anomaly detection. In CVTU, 2022.
Khalil Bergaoui, Yassine Naji, Aleksandr Setkov, Angelique Loesch, Michèle Gouiffès, and Romaric Audigier. Object-centric and memory-guides normality reconstruction for video anomaly detection. In ICIP, 2022.
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based "best of many"" sample objective. In CVPR, 2018.
Kai Chen, Jiaqi Wang, and et al. MMDetection: Open mmllab detection toolbox and benchmark. In arXiv, 2019.
Debadeepta Dey, Varun Ramakrishna, Martial Hebert, and J. Andrew Bagnell. Predicting multiple struc- tured visual interpretations. In ICCV, 2017.
Fei Dong, Yu Zhang, and Xiushan Nie. Dual discriminator generative adversarial network for video anomaly detection. In IEEE Access, 2020.
Keval Doshi and Yasin Yilmaz. Continual learning for anomaly detection in surveillance videos. In CVPR Workshop, 2020.
Alessandro Flaborca, Luca Collorone, Guido Maria D'Amely di Melendugno, Stefano D'Arrigo, Bardh Prenkaj, and Fabio Galasso. Multimodal motion conditioned diffusion model for skeleton-basel video anomaly detection. In ICCV, 2023.
Yiwei Fu and Feng Xue. MAD: Self-supervieed masked anomaly detection task for multivariate time series. In IJCNN, 2022.
Mariana-Luliana Iuliana Georgescu, Antonio Bärbäläu, Radu Tudor lonescu, Fahad Shahbaz Khan, Marius Claudiu Popescu, and Mubarak Shah. Anomaly detection in video via self-supervised and multi-task learning. In CVPR, 2021a.
Mariana Iuliana Georgescu, Radu Ionescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. A baokground-agnostic framework with adversarial training for abnormal event detection in video. Tn TPAMI, 2021b.
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In ICCV, 2019.
Abner Guzmán-rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. In NeurIPS, 2012.
M. Hasan, J. Choi, J. Neumann, A. K. Roy- Cho-Chhwy, and L. S. Davis. Learning temporal regularity in video sequences. In CVPR, 2016.
Chaoqin Huang, Qinwei Xu, Yanfeng Wang, Yu Wang, and Ya Zhang. Self-supervised masking for unsuper- vised anomaly detection and localization. In ArXiu, 2022.
Radu Tudor Ionescu, Fahad Shahbaz Khan, Aariana-Tuliana Georgescu, and Ling Shao. Object-centric aulo-encoders and dummy anomalies for abnormal event detection in video. In CVPR, 2019a.
Radu Tudor Ionescu, Sorina Smeureanu, Marius Popeseu, and Bogdan Alexe. Detecting abnormal events in video using narrowed normality clusters. In WACV, 2019b.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Jielin Jiang, Jiale Zhu, Muhammad Bilal, Yan Cui, Neeraj Kumar, Ruihan Dou, Feng Su, and Xiaolong Xu. Masked swin transformer unet for industrial anomaly detection. IEFE Transactions on Industrial Informatics, 2023.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. In ATT Labs [Online]. Avoilable: http://aann.eeun wwwp:e/..een..comeeeee com/erdb hrmm//iii/mmsst, 2010.
Kimin Lee, Changho Hwang, KyoungSoo Park, and Jinwoo Shin. Confident multiple choice learning. In ArXio, 2017.
Stefan Lec, Senthil Purushwalkam, Michael Cogswell, Viresh Ranjan, David J. Crandall, and Dhruy Batra. Stochastic multiple choice learning for training diverse deep ensembles. In ArXie, 2016.
Zhaoxu Li, Yingqian Wang, Chao Xiao, Qi Ling, Zaiping Lin, and Wei An. You only train once: Learning a general anomaly enhancement network with random masks for hyperspectral anomaly detection. IEEE Transactions on Geoscience and Remote Sensing, 2023.
W. Liu, D. Lian W. Luo, and S. Gao. Future frame prediction for anomaly detection - new baseline. In CVPR, 2018.
Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly detection framework via memory-augmente flow reconstruction and flow-guided frame prediction. In ICCV, 2021.
Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150 fps in matlab. In ICCV, 2013.
Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked rmn framework. In ICCV, 2017.
Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. Anomaly detection in crowded scenes. In CVPR, 2010.
Yassine Naji, Aleksandr Setkov, Angélique Loesch, Michèle Gouiffès, and Romaric Audigier. Spatio-temporal prodictive tasks for abnormal event detection in videos. In AVSS, 2022.
Duc Tam Nguyen, Zhongyu Lou, Michael Klar, and Thomas Brox. Anomaly detection with multiple- hypotheses predictions. In ICML, 2018.
Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance-motion correspondence. In ICCV, 2019.
Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detec- tion. In CVPR, 2020.
Bharathkumar Ramachandra and Michael Jones. Street scene: A new dataset and evaluation protocol for video anomaly detection. In WACV, 2020.
Bharathkumar Ramachandra, Michael Jones, and Ranga Raju Vatsavai. survey of single-scene video anomaly detection. In TPAMT, 2022.
Mahdyar Ravanbakhsh, Moin Nabi, Enver Sangineto, Lucio Marcenaro, Carlo Regazzoni, and Nicu Sebe. Abnormal event detection in videos using generative adversarial nets. In ICIP, 2017.
Fitsum Reda, Robert Pottorff, Jon Barker, and Bryan Catanzaro. flownet2-pytorch: Pytorch implementation of flownet 2.0: Evolution of optical flow estimation with deep networks. https://githu/ www/ 1owmet2-pytorch 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. In arXio, 2018.
Cicolac-Cttalin Ristea, Nechu Madan, Radu Tudor Ioneseu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B Moeslund, and Mubarak Shah. Self-supervised predictive convolutional attentive block for anomaly detection. In CVPR, 2022.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Christian Rupprecht, Iro Laina, Robert DiPictro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D. Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In ICCV, 2017.
C. Shi, C. Sun, Y. Wi, and Y. Jia. Video anomaly detection via. sequentially learning multiple pretext tasks. In ICCV, 2023.
Ashish Singh, Michael J Jones, and Erik G Learned-Miller. Eval: Explainable video anomaly localization. In CVPR, 2023.
Yao Tang, Lin Zhao, Shanshan Zhang, Chen Gong, Guangyu Li, and Jian Yang. Integrating prediction and reconstruction for anomaly detection. In Pallern Recognition Letters, 2020.
Guoxiang Grayson Tong, Quanquan Li, and Yan Song. Two-stage reverse knowledge distillation incorporated and sell-supervised masking strategy for industrial anomaly detection. Knowledge-Brad Based Systems, 2023.
Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. In ECCV, 2022.
Ziming Wang, Yuexian Zou, and Zeming Zhang. Cluster attention contrast. for video anomaly detection. MM '20 2020.
Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noisc. In CVPR Workshop, 2022.
Guang Yu, Siqi Wang, Zhiping Cai, En Zhu, Chuanfu Xu, Jianping Yin, and Marius Kloft. Cloze test helps: Effective vidco anomaly detection via learning to complete video events. In ACM Multimedia, 2020.
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Reconstruction by inpainting for visual anomaly detection. Pallern Recogrition, 2021.
18

---

Published in Transactions on Machine Learning Research (07 (07/2024)
A Propositions and proofs:
Proposition 1. Let g be single prediction model trained using the L2 norm. The minimum loss is achieved for a model o² that predicls the condilional expeclation:
g"(Mm) (
Proof. By definition, the conditional expectation is the minimizer of MSE, we have (Vg E F):
Esingte(E XM))
Ex Ex=e(1EE(X (X - xI)
S Ex~ell (1lg( [Xa))())))))))) - -(l) (by definition)
Sinnge (g( (XM))
Thus, the minimim is reached for:
9" (XM)) = B(X
Proposition 2. Let X sample from P, F the space of self-maps of [o, C and f e arg min (f The minimum expected loss is lower when using multi cuutt-prrdicttin than when f=(f())
using single prediction:
EN Laingle (Xa1)
Moreover, in the case of MSE, by considering the inder of the closest predictor to X: K arg min (1)) (XM) X12, have:
Esingie (g" (XM)) (N EK ( (X E(X]MM, K) K)
Proof. First we show that: (NN(F((X))) S Esingte (g" (X
The multi-prediction optimization problem using the nearest neighbor loss can be formulated as follows:
minimize (=ff))ee EF
(NN()(XM))
Where is multi-prediction model which maps the input XM to n predictions particular, if we consider a MMP model for which all predictors produce an optimal single prediction: = (X = gm.=g( = (XMM,, the loss for this particular model is:
Tn
(NN(g(XM)) = Exae
min llg (k) (XM) - X1| [E[
=Exa
min |1g (Xm)) xII [kE[I.n]
= Ex-q (1gg* (Xar) - X
= Lsingtelg" (g" (Xas))
19

---

Published in Transactions on Machine Learning Research (07/2024)
By definition, (xx(f"(Xx)) is the minimum value of the loss (xx()(XM)) Thus, have (Vf E FF)):
(xN(((XN)))
In particular for g:
(NN(f"(Xx)) Exw(g(XM)) Laiggt (9" (Xa())))
Second, show that: Lsingiel9"(Xiie (f Ex [x B(X -XXXXX K)1B(xxx))
Let I* E arg min Rep
(X and let K defined as: K = arg min me] ( - x112.
We first show that:
(xN))"(XM)) E[min [| K= k) x11 XM)
We have:
CN B(min [I (k) (XM) x11)
E(min || (k) (XM) - X X
E((E(min || (XM) - XI1 K)
-E(E(I)) (XM) K) XM)
-E(E(I)F*<)) (X.) B(X XM, K) + B(X K) - X
-E(E(I)F'0) (XM) E(X X.. K [X XII
+ x (f)())(MM) - [X K
K) ()1)))X)))
(((E(I)F((K)) (XM) |EX XM. K) (111111)) + ((EE(I K) (11114))
+ 2 x E(E(() (KM) - [EX (B(XIXX K) X)IK)(M))
-Ex(I(-(() (Xa) X M. ( + E[E K) X122
= (Ex(II((() (Xar)
|XX
Xar) B(E(min |B[X|XM K = XI
= (Ex(( - EXX X M, K x Emin
X XI XM)
=ENN (xxx))) (Xy definition)
Thus:
Exx(f((XM) = 1mmin k
| K- k) - -X1
We will show that Esingte (g" (Xx LNN [NN(("(XN))
Ex((IE(XXx)) - EXX K))B
20

---

Published in Transactions on Machine Learning Research (07 2024)
Esingle (g"((gg (X [([ X (2) (by definition)
B([ 12 X
= [[E ())
E(E(IE(X[[MM) E(X XMM. + -X
E(E([I[(X[XM) E(X XM K) (w, -
+ x (R[X]XM) X (B( K) -
= (X( - E(X K)) XM) E X K) -
+ 2 [(L((((X]XM) - IXX (B(X))))))/)))///)))) () - X)IK)|XM)
=Ex( X B(X M. K X ) E(min | K = k) -1
(x(( E(X KII XM) (N
Proposition 3. A MMP model with number of predictors corresponding the number of roads and reaching the global minimumn truining loss using LNN, achieves an AUC of 100% on the synthelic dalasel. On the other hand, single prediction model achieving the globel minimum training loss using Ssingee connot achieve an AUC of 100%.
Before moving on to the demonstration, we need to prove the following lemma:
Let N discrele and fonile set of normal samples, A set of abnormal samples and S the anomaly scores given by trained model. We have the following equivalence:
((V(pnPP)) EN x A): S(p.) < S(pa)) + AUC 100%
Proof. AUC 100% is equivalent to find threshold for the anomaly scores that separates perfectly normal and abnormal data. Which is equivalent to having:
(3A (R) E A) S(m)) S < S(aa)
(11)
Indeed, exits, the ROC curve passes through the point (FPR,TPR) (0, 1) (which corresponds to choosing the threshold 0) and therefore AUC acheives its maximum.
Equation 11
implics that:
(V(pn Pa) E A) :(pp)) < S(pa)
The reciprocal implication be verified by taking 0 max((S((p))ee) E EN).. This maximum exits since suppose that N is discrete and finite.
The following is the proof of proposition 3 for the case 2.
Proof.
The multi-prediction case:
First, we need to compute the predictions of the model when it achieves the global minimum. Indeed, the global minimum the training loss is acheived for model that predicts exactly the two normal
Training:
possibilities: Pe+1 Pr+1.Ptt1 -1 which minimizes the training loss since:
(1) (12)
CNN = 2
Σ
min
1p-- P = 0
(2))
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
The optimal set of predictions is therefore: (1) (2)
(Pt+1: Pi+1] (the uniqueness of the set can be easily demon-
strated), Figure 1 in the main paper illustrate the optimal predictions.
Anomaly scoring: Once we computed the best model predictions, we can evaluate the anomaly detection
performances. We denote by SuN the anomaly scores given by the multi-predietion model. We define the normal set N (1) P(41) and the abnormal set = - (pt P
(P J. We have:
(2)
EN)NN)= SNN(D)= = min
1p- =11 =
(2) pelritip/41)
While:
(Vp E :): SNN(p)= min pe (9)
|pp 61
Therefore:
(V(p,, Pa) E N x A) SNN(PM) < Spa(
We conclude by the lemma that AUC 100%.
The single prediction case:
Training: Given single prediction model, the equation 3 in the main paper shows that the conditional expectation is the optimal solution. In the case of the synthetic dataset, it consists in the average of the
next possible positions. Therefore the optimal prediction is p14) +r (1,0) (Figure in the main paper). Note that this position is abnormal and that the optimal training loss is non-mull:
(1)
2
Esingle
Σ
lp 1p-1i:1l
2
pC (
(1- 1)2
+ 0)2 +(1 (1 - (-1-0-10) 0)2
2
= =1>0
Anomaly scoring: We denote by Ssingle the anomaly scores given by the single prediction model. We will show that:
(3(pns Pa) E A): (pm)) 2 Sosngle (pa)
In fact, if we take Pa = p(a) =1,0) E A, we have:
Ssingle Ssingle (b(+))
= =11p(11 - 110)111
=0 0
While if we consider: Pa Pt41 N:
(1)
22

---

Published in Transactions on Machine Learning Research (07 (074))
Seinge((P)) = Ssingte (Ponge((2241) (Pt41)
= -11p - p(() -p(t)
= - 112 + (1-
= =1>0 0
We conclude using the lemma (by contrapositive) that: AUC 100%.
Finally, we have shown AUC 100% achieved only by the multi multi- prediction model trained via LNN.
Proposition 4. Let Leaive (f(xx)) the expected loss corresponding to Lnnive (f(
Enaine (J(x)) = Ex-e 1
(1((k) -X1]
(12)
RE[I.N]
In case of L2 norm, the minimumm expected loss is achieved by MMP model: f (ffa(t)))epp such that
(Vk E [I,n]) (*) (XM) = {x))MM), which is similar to perform single prediction.
Proof. We have:
1 (()M))
min
Ex
Σ
(X xXI -x1]
kE[T.n]
min
1
Σ
(x( (XMr) xII)
EF kEl
Σ
Ex (1) JR
min
(H (XM) - XI))
R kE
By definition of the conditional expectation E(X]XM), we have (v E F) for La norm:
E Ex-c(IE(XXXM) - XII
< ExxollI (XM) (X)m) -(1)
Therefore:
1
Σ
min Ex Ex(1)((( XII) JaeE
ke[L.n]
Σ
Ex- Ex~o<1 (1 XII)
Ex
ke[L,n]
= Ex (Exxe(EE(XX) (I XAs) XI))
Therefore, the minimum achieved for f (K)) verifying:
(Vk [1,n]) (
(X(s) = B(X[XM))
23

---

Published in Transactions on Machine Learning Research (07 (024))
Proposition 5. Let X the training dataset composed from normal somples. We denote by f f= (f(()))EL a MMP model oplimized
the nearest neighbor objective CNN and achieving a loss LNN (NN()(M))) X) on sample X E X. Let f f (f(N))E a MMP model resulting from training the non-optimized predictors of f via the non-participation loss CNP, and CNN CNN(f (f(x)) the loss of f on sample X. We have (VX E X):
()) & (x
Since training the model with NNN does not necessarily lead to optimizing all prodictors (local min- imum problem), the set of predictors that are not selected as nearest neighbors 2 C [[. n]] is not necessarily empty. Note that the demonstration also holds if 21 0.
Proof.
By definition of U, we have (VX X):
arg min mig (I - X|I u ke[1,n]
Therefore, re-write LNN (NN(f(XM),x) as follows (AX X)
(xx(
min keiL.x]ver
(XMP) - XII
Since the non-participation loss optimize only the predictors which belong to U1 the set of already optimized predictors (f(()) )e[[I.]]] obtained after training (f(²) (()))_[ using the non-participation loss CNP remains the same:
(Vk [1,] f(t) f(0)
Therefore, we have (VX x):
ke[i.n] (NN(f(XN))
min (X - -X1
ke]l,n]
min (()) (Xar) XII
min
(Nar) - XII
ke
min [1(()) - -X
kE]L.n]
=[NN(((XM)) X)
Thus (A) E x):
(x) (xN(F(XM),
24

---

Published in Transactions on Machine Learning Research (07/2024)
Table 7: MMP-AMS sub-networks number of parameters
Sub-networks
Number of parameters
fF
5.3M
fo
5.3M
Sc
2.2M
R
460K
P
2.0M
Total
15M
B
Implementation details
B.1
Pseudo-code
In order to facilitate the reproducibility of our approach, we provide the pseudo-code for training MMP-
AMS.
B.2 Network architectures
Regarding MMP-AMS, the architectures of JF and fo are made of two stacked auto-encoders, each one is composed of 4-strided convolutions with ReLU activations and strided transposed convolutions. The last layers are adapted for each modality and are followed by sigmoid activation. We use skip connection in the first auto-encoder to avoid the vanishing gradient issue. The hidden states h2 are concatenated at the bottleneck of each auto-encoder to add the latent information. The architecture of fc is composed of convolution layers followed by max-pooling with ReLU activation and 4 fully connected layers with softmax activation. fc produces class vectors which have the same size as YOLOv3 class vectors. The first state predictor P, assembles a series of 4 convolution layers, a maxpooling and ReLU activations. The recurrent neural network R is composed of fully connected layers of hidden dimensions 256 with ReLU activations and a skip connection to avoid gradients vanishing. For all networks, use a kernel size of 4x4 for convolutions and 2x2 for maxpooling. The architectures and the number of paramcters of cach module are shown in Figure and in Table 7 respectively.
Concerning the location module, we use a small fully connected network which takes as input a class vec- tor Xe and predicts the means: (aa(Xc))4ee and standard deviations: (SA(Xc))Ase The network is composed of 5 fully-connected layers with ReL.U activation and hidden size of 256. An illustration of the location module is shown in Figure 7
B.3 Implementation details MNIST
In order to adapt model to images, mask input image I with mask M. Then, we train model fr to predict the multiple possible completions of the image I:
(1 (1 MM)eee) f e M) - (1/((((
(((()))efl.
We use similar architecture as for future frame prediction, and train the network using the same objective function:
L= + X
(13)
We train the network for 200 epochs for both datasets using Adam optimizer with a learning rate of 10 10-4
with batch size of 128. We set À 0.1. We select predictors which have participation below 5%.
25

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm MMP-AMS training pseudo-code
Require: number of predictors n, non-participation loss weight X, selection threshold 6, mask M, number of training epochs npep, training dataset. X.
Ensure: Trained networks fo. R, P)
Init (Sr, Jo: Jc, R, P) for A (F, 0) do
Init the set of unoptimized predictors: U4 f- 0
Init the participation counters: clA) <- (O, 0) end for for E [l,n_ epl do for X E do
# Forward pass:
Apply mask: XMP + Xp 3 M
Compute first state: ho t P(XMP)
for k E [1. n|
Compute state: hgg R(hx-1)
for A CF,CCO) do Compute prediction:
{ + fX(Mp,h
A end for end for # Loss computation: Init total loss: Lr + 0
for A (F,C,O) do
Compute non participation loss:
LNP + Eteu, || XAll
Compute nearest neighbor:
K argmin|l
| XAll
kE[
Compute nearest ncighbor loss:
CNN + || - X
Update total loss:
Lr+ f- CNN + XLNP
Update counter:
C ++1
end for # Backward pass:
Backpropagate through (fr, fe. P) end for # Update unoptimized predictors:
for A E (F,C,O) do
Reset the set of unoptimized predictors:
UA <- 0
Compute participations: pl4) (sum((())))
for & E [1] do
(A) < then KPPEND(UA,K)
if Pk
end if end for
Reset the participation counters: ((A) + (0,..,0) 0) E TR"
end for end for
26

---

Published in Transactions on Machine Learning Research (07 2024)
fo,fo
fc
R
MP
XMP
3ecarr
pad
dec, uileut
JRNGH PC
cir isl, arila
180 voee ded
2d las pnd
til, miila-2
Bull::
NALA
SGi Trare Inl, saili--it
Rrll:
Frare
h4
h
fot, drid
200
Migrand
Sultou
lik
X C
P
Juble-t
XMP
Beill
fe
cilcam tel, drile-2
Chease
Xc
sax poll
10 PC THUT
BaM::
144ram dek
350 Touar bel, didee22 R:Ul
250224 PC
2:0 goil
1/
BxUU
Tcuar andlao
15
Nur Hearr dol, ardd=0
PC
(a PB.) (X,Y, (X,
ha
X (R)
Figure 6: Overview of the architectures used for different method components: fF: frame prediction network, for optical flow prediction network, fc: class prediction network, R: recurrent neural network, P: the first state predictor and fe: bounding boxes prediction network used for the location module. Notations: +: residual connection, |]: concatenation.
27

---

Published in Transactions on Machine Learning Research (07 2024)
Location module
X person
X
fo
Group by class
(B.) ( BB,)
(ae B (a BB,)
LNLL
Figure 7: Overview of the the location module, extract objects in the scene and group them by class. The class vector Xe is given to fe to predict the parameters of 4 Gaussians: N(as(Xc), (x((()) for AE [Bx, By, Bu, B.}. ). We optimize the network via the negative log likelihood-loss LNLL-
Table 8: Comparison of the impact non-participation loss on diffcrent bascline models.
Impact on the multi-prediction model
(b) Impact on the mixture of Caussian model (x10->)
Model LNP
Mean prediction loss (4)
Model
LNP
Mean log- likeliholdoo (t)
MP
3.52
MoG
2.41
MP
2.45
MoG
2.91
C Further experiments:
C.1 Impact of the an-paaticiiation loss on baseline models:
In order to test the impact of the non-portinnn con-partccipation on different baselines, we evaluated it on two standard models that used in the context of modeling multi-modal probability distributions: 1) mixture of Gaussian model (MoG) trained via maximum log-likelihood 2) multi-predictiol (MP) model which consists of n prototypes trained using the nearest neighbor loss NLNN). The dataset which both models are trained consists of feature level representation of objects extracted from UCSDped2 using a pre-trained auto-encoder. Models are trained in a similar way, using Adam optimizer with a lcarning rate of 10-3 and a batch size of 64 for 150 epochs. For each of the two models, we assess the impact of adding the non- con-participttion loss to training. For the Gaussian mixture model, the nonpparticipation loss consists of maximizing the likelihood of samples w.r.t. Gaussians which do not maximize the likelihood for enough samples (the threshold 8 5% is used). As observed in the case of MMP models, using LNP in training allows for lower prediction loss in the case of the multi-prediction model (MP) well as for higher likelihood for the mixture of Gaussians model (cf. Table 8
C.2 Impact of the temporal window on the performance of MMP-AMS on Avenue:
In order test the impact of increasing the window size, we experimented MMP-AMS with larger prediction windows for future frames and the optical flows (cf. Table g As the predictions concern the spatio-temporal
28

---

Published in Transactions on Machine Learning Research (07 (07/2024)
window centred around the object, consider the anomaly score for the full window. Nevertheless, since the same object can be detected multiple times in the same temporal window, we perform non-maximum suppression (NMS) to delete redundant detections. While increasing the time window results in better performance, it also requires additional computational costs and increases latency, as producing an anomaly score at time requires accessing the image a times | window-size
Table 9: Influence of window size on anomaly detection performance on Avenue.
Exp.
window size
RBDC
TBDC
2
77.7
74.2
e2
3
76.5
75.5
e3
4
77.8
75.6
e4
5
78.1
76.4
29

---

Published in Transactions on Machine Learning Research (07/2024)
D Qualitative results
We show anomaly detection examples taken from the 3 datascts: ShanghaiTech, Avenue and UCSDped2: The healmaps illustrate object-level anomaly scores S(X) and the graphs correspond to frame level scores S(F:). We can see that our anomaly scores correlate well with the ground truth for different anomaly types including: unusual test objects (cf. Figures 13
10
12
13
10 and unexpected behaviours (cf. Figures 11 12 Furthermore, we show in Figure 8 some frame predictions on test samples. Furthermore,
14
show example of case for which only the location aspect is abnormal, which not annotated as anomaly in UCSDped2 (Figure 9
Input
Predictions
RGB (RGB)
Normal
RGB (RGG)
RGB (RGG) RGB Diffs.(RGB)
Abnormal
Figure 8: Visualization of normal and abnormal lest examples. The first column indicates the masked input, the second column shows 3 predictions of the MMP-AMS framework. The nearest predictions are circled in green. The third column indicates the corresponding ground truth. We observe that the nearest predictions are closer to the ground truth for normal samples than for abnormal samples.
30

---

Published in Transactions on Machine Learning Research (07 (07/202) /204)
Figure 9: An example of case for which only the location aspect is abnormal, which is not annolated as anomaly in UCSDped2 (circled in green). The heatmap correspond to the density function learned by the location module.
Anomaly score
1.0
0.8
Ground truth
Frame-level anomaly scores
0.4
0.2
0.0
100
200
300
400
Frames
Figure 10: Qualitative results from video 01 0134 taken from ShanghaiTech dataset. The heatmaps illustrate object level anomaly scores S(X) and the graphs correspond to frame level scores S(F)). Red boxes indicate ground truth anomalies while green boxes indicate YOLOv3 delections. The anomaly consists of a person riding a bike.
31

---

Published in Transactions on Machine Learning Research (07 (07/2022 2024)
Anomaly score
1.0
Ground truth
0.8
Frame-level anomaly scores
0.6
0.4
0.2
0.0
200
600
800
1000
Frames
Figure 11: Qualitative results from video 15 taken from Avenue datasct. The hcatmaps illustrate object level anomaly scores S(X) and the graphs correspond to frame level scores S(F,). Red boxes indicate ground truth anomalies while green boxes indicate YOLOv3 detections. The anomaly consists in a person walking towards the camera.
Anomaly score
1.0
0.8
0.6
0.4
0.2
Ground truth
Frame-level anomaly scores
200
600
1000
1200 Frames
Figure 12: Qualitative results from video 09 taken from Avenue dataset. The heatmaps illustrate object level anomaly scores S(X) and the graphs correspond to frame level scores S(F)). Red boxes indicate ground truth anomalics while green boxes indicate YOLOv3 detections. The anomalics consist in a person standing in front of the camera, kid jumping and person throwing a bag
32

---

Published in Transactions on Machine Learning Research (07/2022 2024)
Anomaly score
1.0
Ground truth
0.8
Frame-level anomaly scores
0.4
0.2
0.0
100
125
150
Frames
Figure 13: Qualitative results from video Test001 taken from UCSDped2 dataset. The heatmaps illustrate object level anomaly scores S(X) and the graphs correspond frame level scores S(F)). Red boxes indicate ground truth anomalies while green boxes indicate YOLOv3 detections. The anomaly consists of person riding a bike
Anomaly score
1.0
0.8
0.6
0.4
0.2
Ground truth
Frame-level anomaly scores
0.0
0
25
50
75
100
125
150
175
Frames
Figure 14: Qualitative results from video Test007 taken from UCSDped2 dataset. The heatmaps illustrate object level anomaly scores S(X) and the graphs correspond frame level scores S(F)). Red boxes indicato ground truth anomalies while green boxes indicate YOLOv3 detections. The anomalies consist in skate- boarding and riding bike.
33


Reviews:
[{"review_id": "149ghZ9OBj", "reviewer": "Reviewer_cuob", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed multi-prediction multi-aspect approach is sensible for future prediction-based anomaly detection. \n- The method achieves strong results on three benchmarks.  \n\nWeaknesses:\n\n- The motivation is a bit unclear. First, the grouping of anomaly detection methods into explicit and implicit modeling lacks a clear justification. In particular, for video data, reconstruction and future prediction can be viewed as special cases of probabilistic methods under certain probability distribution assumptions (e.g. conditional Gaussian). Moreover, it is unclear why the paper focuses on the 'implicit' strategy. An 'explicit' model can naturally generate multiple outputs using multi-modal model distributions. \n\n- The limitation of using a single prediction for anomaly prediction seems obvious and well-understood since the anomaly probability distributions can be multi-modal. The theoretical analysis in Sec 3.1 lacks sufficient insights. \n\n- The design of the loss seems ad hoc by simply combining L_NN and L_NP with a fixed weight parameter $\\lambda$. It is difficult to adjust the weight under different data instances. A more natural way is to match the predicted distribution of normality with the sampled groundtruth distribution, which has been explored in the uncertainty modeling literature.\n\n- The overall technical novelty of the proposed strategy is incremental. The proposed loss is a minor extension of the NN loss from Guzman-rivera et al (2012); The masking strategy is widely used in representation learning; Using multi-aspect modeling is reasonable but it relies on three pre-defined aspects and thus has difficulty in generalizing to other unknown patterns. \n\n- The experiments are a bit lacking in several aspects: 1) As the proposed method uses multiple subnetworks for capturing various aspects, it would be clearer to include the model sizes in the comparison for fairness. 2) For the comparison on the Avenue dataset, it would be more convincing for the proposed method to also use a longer temporal context. 3) The analysis/comparison seems to lack some simple baselines, such as a mixture of probabilistic models or a traditional multi-choice model on the learned feature space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a prediction-based anomaly detection method that integrates multiple outputs into masked representation learning to cope with heterogeneity. It employs a modified nearest neighbors loss and introduces different modules for multiple aspects of the anomaly. The work evaluates the method on three video benchmarks with comparisons to several previous approaches and ablation studies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns in the weakness section.", "broader_impact_concerns": "Given its relevance to visual surveillance, it would be beneficial to include a concise discussion on its broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6mxelBxFUQ", "reviewer": "Reviewer_feNb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The example of multi-prediction of the road scenario is intriguing.\n- The authors theoretically discussed the multi- and single-prediction and the impact of loss functions.\n- MMP-AMS is applied complex video anomaly detection.\n- Most of the claims are supported and seem to be accurate.\n\nNeutral\n- Compared with existing studies on multi-prediction-based anomaly detection, this paper proposed masked multi-prediction.\nThis contribution itself seems to be not very innovative because there are existing studies on masked prediction for anomaly detection.\n\nV.Zavtnik, M.Kristan, D.Skocaj, Reconstruction by Inpainting for visual anomaly detection, Pattern Recognition, vol. 112, 2011.\n\nWeaknesses\n- The discussion of the synthetic road dataset is not clear. P4.``We generated a synthetic dataset ‘'.\nWhat is the training and test dataset? What is the anomaly? What is the model?\nObviously if the model predicts all anomal and normal sample correctly, even a single prediction can achieve 100% AUC. Thus, these conditions should be clarified.\n\n- In the state-of-the-art results of VAD, the latest method is the year 2022, which is considered old.\n\n- In the resutls of Table2, the location module significantly reduces performance on UCSDped2, and ShanghaiTech datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes anomaly detection using a masked multi-prediction framework for modeling appearance, motion, and semantics (MMP-AMS).\nIn the MMP approach, the importance of multi-prediction is discussed. MMP recovers multiple original images using recurrent neural networks, and the minimum distance of multiple predictions is utilized. This loss is lower than in the single prediction case. Also, a non-participation loss that penalizes only the predictors that do not participate in training is proposed.\nIn the AMS approach, appearance, motion, and semantic aspects are used for the MMP approach in video anomaly detection. Experiments conducted on the UCSDped2, ShanghaiTech, and Avenue datasets demonstrate the effectiveness of MMP-AMS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add a discussion of existing masked prediction methods.\n\n- Enhance the description of the road dataset.\n\n- Update the state-of-the-art results of VAD.\n\n- Evaluate the loss of the MNIST dataset to confirm the Proportion 2, 5 in Sec. 3.\n\n- Add the results of O, C to Table 4 to analyze the individual performance of these predictions.\n\n- Add the conference or journal names for all references.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nHrx8phbQa", "reviewer": "Reviewer_hQaC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. This paper is well-prepared, with comprehensive theoretical and experimental content.\n2. The experimenst conducted on several benchmarks show the effectiveness of the proposed approach.\n\n\nCons:\n1. The first paragraph of the Introduction section is too lengthy, while the specific background introduction is too brief, causing confusion for readers. Additionally, in this section, the motivation for the paper is not clearly articulated.\n2. In the second paragraph of the Introduction section, the abbreviation \"VAD\" is used without its full expansion, which hasn't been introduced previously.\n3. In Section 3, the experiments are all based on the MNIST dataset. However, since this paper focuses on video tasks, the disparity between real-world scenarios and MNIST is expected to be substantial.\n4.What is the time complexity of the method proposed in this paper? Is it necessary to apply masking to each frame?\n5. There is no open-source code available here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach called masked multiprediction (MMP) for anomaly detection in heterogeneous normal observations. MMP generates multiple likely normal outcomes, enhancing normality learning and improving anomaly detection performance. Additionally, the authors propose an adaptation of MMP, called MMP-AMS, which covers multiple aspects of normality such as appearance, motion, semantics, and location, leading to interpretable and modular anomaly detection. Experimental results validate the effectiveness of the proposed approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. According to the suggestions, revise Section 1.\n2. The experiments in  Section 3 should also evaluate on video datasets.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with video anomaly detection (VAD). The authors introduce a masked multi-prediction (MMP) approach based on combination of a nearest neighbour loss and a new non-participation loss. They also propose to extend the approach to model multiple normality aspects (MMP-AMS) and in order to refine anomaly detection. Theoretical results support the choices of the proposed framework, and preliminary experiments are provided on MNIST to justify the relevance of the proposed making approach over reconstruction methods on MNIST. Experiments are conducted on three real VAD datasets: UCSDped2, ShanghaiTech, Avenue datasets.

Initially, the reviewer appreciated the sensitivity of the approach, and acknowledge that most of the claims are validated by evidence. However, they also raised concern about the novelty and motivation behind the proposed approach, clarification of the positioning with respect to explicit vs implicit approaches, and justification of the different terms in the loss. They also question the experimental results, requesting clarifications on the toy dataset, requesting for more recent baselines, and a discussion on the impact of the location module. The authors' feedback was overall successful to answer to reviewer's concerns: the authors provided new experiments and modified the paper accordingly. After rebuttal, two reviewers recommended paper acceptance, while the third reviewer recommended rejection because they requested more recent baseline comparisons. 

The AE carefully reviewed the submission and discussions. The AE considers that the approach is meaningful and that each contribution is validated by evidence, theoretically or experimentally. Although intuitive, the theoretical justification of multiple prediction will be of interest for the TMLR audience. The importance of MMP has been validated on MNIST but also on real video datasets, and the importance of leveraging diverse aspect for representing normality on the context of VAD is convincing. Therefore, the AE recommends paper acceptance.
Decision: Accept (poster)