Real-world Video Anomaly Detection by Extracting Salient Features in Videos
Yudai Watanabe*
Makoto Okabe
Shizuoka University

Yasunori Harada
Naoji Kashima
Chubu Electric Power Co., Inc.

arXiv:2209.06435v1 [cs.CV] 14 Sep 2022

watanabe.yudai@jp.panasonic.com, m.o@acm.org, {Harada.Yasunori,kashima.naoji}@chuden.co.jp

Abstract
We propose a lightweight and accurate method for
detecting anomalies in videos. Existing methods used
multiple-instance learning (MIL) to determine the normal/abnormal status of each segment of the video. Recent
successful researches argue that it is important to learn thetemporal relationships among segments to achieve high accuracy, instead of focusing on only a single segment. Therefore we analyzed the existing methods that have been successful in recent years, and found that while it is indeed
important to learn all segments together, the temporal orders among them are irrelevant to achieving high accuracy.
Based on this finding, we do not use the MIL framework, butinstead propose a lightweight model with a self-attention
mechanism to automatically extract features that are important for determining normal/abnormal from all input segments. As a result, our neural network model has 1.3%of the number of parameters of the existing method. We
evaluated the frame-level detection accuracy of our methodon three benchmark datasets (UCF-Crime, ShanghaiTech,and XD-Violence) and demonstrate that our method canachieve the comparable or better accuracy than state-ofthe-art methods.

1. Introduction
The number of surveillance cameras in the world is increasing every year, and they are used for crime prevention in cities and for safety confirmation in factories, power
plants, and other large-scale facilities. However, since it isdifficult for humans to see and confirm all of these videos,
there is an urgent need to develop technology that enables
artificial intelligence to analyze the videos and automatically detect abnormal events on behalf of humans. Since
abnormal events are rarely observed, many methods havebeen proposed that use only the normal state as training
data, and judge whether the input video is normal or abnormal based on the criterion of how much it deviates from the* Current affiliation: Panasonic System Networks R&D Lab. Co., Ltd.

learned normal state when inferring [13, 2, 7, 19, 8]. However, these methods can only detect abnormalities based on
low-level features such as differences in the appearance andvelocity in the video. Therefore, recently, a method has
been proposed to train an anomaly detector using a weakly
supervised dataset that contains both normal and abnormal
videos [27, 28].
In the weakly supervised dataset, each video is labeled
as normal or abnormal. That is, the videos labeled as normal contain only the normal state throughout all frames. On
the other hand, the videos labeled as abnormal contain amixture of normal and abnormal frames. By using such adataset, it is no longer necessary to label each frame in thevideo as normal or abnormal, thus reducing the labeling effort.
Many existing methods treated multiple consecutive
frames as a single short-term segment and used multiple
instance learning (MIL) to determine normal/abnormal foreach segment [27]. However, recent successful methods
have argued that it is important to learn the temporal relationships among segments by taking all segments from thevideo as input together, rather than focusing on only a single
segment [28]. To confirm this, we analyzed these methods
on a dataset of video segments randomly sorted in time. We
then found that while it is indeed important to train all segments together, the temporal order among them is irrelevant
for high accuracy.
Based on this finding, we propose a novel model that
does not rely on MIL, but instead takes all segments as input and has a self-attention mechanism to automatically extract features important for determining normal/abnormal
from them. Despite the fact that the proposed neural network has 1.3% of the number of parameters of the existing
method [28], the proposed method can achieve the comparable or better accuracy than state-of-the-art methods. We
report the frame-level detection accuracy on three benchmark datasets (UCF-Crime [27], ShanghaiTech [18], and
XD-Violence [33]).
In summary, this paper has three contributions:• We analyzed the existing methods andfound that while it is important to learn

all segments together, the temporal orders
among them are irrelevant to achieving highaccuracy.
• Based on this finding, we propose alightweight and accurate method for video
anomaly detection. Our method outperforms existing methods for the UCF-Crime
dataset despite its lightweight model, a neural network with 1.3% trainable parameters. We also show that our method outperforms existing methods for the XDViolence dataset when the model is extended by adding a bi-directional LSTM.
• A detailed analysis of our method was performed: visualization and observation of theattention map show that our self-attention
mechanism works for extract salient features similarly to the top-k strategy in existing methods. The relationship between hyperparameters and accuracy was also investigated in detail.
Our method is a simple model with a self-attention mechanism and appears to be a commonplace model. However, our technical contribution is that we arrived at sucha lightweight model based on our observations and analysisof existing methods and found that this model is sufficientto achieve the comparable or better accuracy than state-ofthe-art methods.

2. Related work
In the real world, most of what we can observe are normal states, and abnormal events are rarely observed. For
this reason, many anomaly detection methods have been developed using unsupervised learning approaches that learn
only the normal state. In inference, the input video isjudged to be normal or abnormal based on how much it
deviates from the learned normal state. The normal state
can be learned using a set of mixture of dynamic texturesmodels [13], a space-time Markov random field (MRF)
model [12], Gaussian mixture models [2], and sparse dictionary learning [4, 17], etc. Deep learning approaches havebeen also proposed, such as a method for analyzing the
temporal changes in the CNN features [23], autoencoderbased methods [34, 7, 19, 8, 11] and GAN-based methods [24, 26], etc. A method that uses multi-task learning
has also been proposed [6]. However, these methods arebasically only able to detect anomalies based on low-levelfeatures such as differences in the appearance and velocitiesof the video.
Recently, in order to develop higher-level anomaly detectors, a number of methods have been proposed to learn

anomaly detectors using weakly supervised datasets containing both normal and abnormal videos [27, 28, 33, 39,
40, 36, 5, 22]. Usually, to train a frame-by-frame anomaly
detector, we need to label and train every frame of the video,
which is expensive to label. Therefore, Sultani et al. proposed a weakly supervised dataset where not each frame but
each video is labeled as normal or abnormal [27]. They also
developed an anomaly detector by introducing MIL, which
considers a video as a bag and selects the segment with thehighest anomaly score from the bag for training.
Most of the anomaly detection methods using weakly supervised datasets are based on MIL. MIL-based methods
have a problem that they have a negative impact on learning when they select normal parts of anomalous videos forlearning. To cope with this problem, Zhong et al. proposed
an approach that considers the weakly supervised dataset as
a dataset containing incorrect labels, and uses graph convolutional neural networks to correct the incorrect labels andlearn from them [40]. Tian et al. proposed a top-k strategy that calculates the difference in anomaly scores between
segments and selects the top k segments with the highest
scores for training [28]. Sapkota et al. proposed a modelthat does not require the selection of the parameter k by introducing distributionally robust optimization [25]. Zaheer
et al. does not rely on MIL-based approach but proposed amethod for detecting anomalies using global features of theentire video and local features of each segment using theattention mechanism and per-video clustering loss [36, 37].
These recent successful methods achieve high detection accuracy by efficiently using the features of multiple segmentsin the video instead of just a single segment.
Recently, Zaheer et al. proposed a method for unsupervised video anomaly detection and reported significant
improvements over existing unsupervised methods throughexperiments on the benchmark datasets (UCF-Crime andShanghaiTech) [38]. Acsintoae et al. proposed UBnormal, a new benchmark dataset for supervised video anomaly
detection consisting of virtual scenes and annotated at the
pixel level [1].

3. Method
We propose a lightweight and accurate learning methodfor detecting anomalies in videos. The proposed method
analyzes the entire video and automatically extracts andlearns the features that are important for determining normal/abnormal.
Let D = {(Vi , yi )} be the dataset. where Vi is the ith
video in the training dataset and yi is the label attached to
Vi . yi = {0, 1}, 0 indicates normal, and 1 indicates abnormal. In a video labeled as normal, only the normal state isrecorded in all frames. On the other hand, the video labeled
as abnormal contains a mixture of frames with normal andabnormal states. Vi is divided into T segments, and eachsegment is converted into a D-dimensional feature vector
Fi,j by the feature extractor: Fi,j represents the jth feature vector of Vi ; the feature extractor used throughout all
experiments was I3D [3], which has been trained on the Kineticts dataset.

Figure 1. An overview diagram of our method

Our method is a simple model that consists of a selfattention mechanism and two fully connected layers (Figure 1). In the self-attention mechanism, the input Fi istransformed by Atten-MLP1 multilayer perceptron into ada × T matrix A1 . At this time, the Tanh function is used
for activation. A1 is converted to A2 , a matrix of r × T , byAtten-MLP2 multilayer perceptron. At this time, the softmax function is used for activation. In addition, a dropout
regularization of 30% is also performed.
Using the weight matrix A2 obtained from the selfattention mechanism, we calculate M = Fi At2 . After reshaping M into a vector of D × r dimensions, it is transformed into the anomaly score through two fully connected
layers FC1 (32 units) and FC2 (1 unit). The activation function of FC1 is an identity function, and that of FC2 is asigmoid function. The binary cross entropy (BCE) function
was used as the loss function.
Note that our self-attention mechanism does not deal
with temporal orders. As we see in the operations throughAtten-MLP1 and Atten-MLP2, the vector in column j of theattention map A2 depends only on the feature vector Fi,j incolumn j. In other words, shuffling the input feature vectors Fi,j in the temporal direction (column direction) does
not change the video anomaly score, which is the output of
the model.

3.1. Motivation of our method
Several existing methods take into account the temporal relationships between segments in a video [27, 39, 28,
36, 37]. Sultani et al. introduced a term in the loss function to impose continuity of the anomaly score, as theanomaly score should vary continuously in the video [27].
Zaheer et al. also used the similar term in their loss function [36, 37]. Tian et al. introduced a multi-scale temporal network (MTN) to capture local and global temporalfeatures [28]. All of these methods achieve high accuracy
in anomaly detection. Therefore, we investigated how themechanism for capturing the temporal relationship between
segments contributes to the high accuracy.
Specifically, in Fi , the set of feature vectors obtained
from the training video Vi , the feature vectors are ordered
by default as {Fi,1 , Fi,2 , · · · , Fi,T }, but we randomly rearranged this order to create a new dataset and used it fortraining. The results of the experiments using the UCFCrime dataset [27] are shown in Table1. In both methods [27, 28], there was no degradation in accuracy due torandom reordering of feature vectors.
Table 1. Comparison of AUC performance using the UCF-Crime
dataset [27], where the feature vectors of each video are randomly
reordered. Here, we used not C3D but I3D with 10-crop augumentation for Sultani et al.’s method [27]MethodSultani et al. [27]
RTFM [28]Reorder
X
X

AUC(%)
81.39
81.54
84.30
84.26

This result indicates that capturing the temporal order
between segments does not contribute to the accuracy of
the anomaly detector.
The temporal smoothness term introduced by Sultani etal. minimizes the difference in anomaly scores between
temporally adjacent segments [27]. However, from theabove results, we infer that this term has the regularization effect of ensuring that all segments in a video have
similar anomaly scores, rather than constraining the temporal order between segments. Zaheer et al. reported a decrease in accuracy without this term [36, 37], which may
indicate the importance of the regularization effect that thetemporal smoothness term brings. Although the MTN andtop-k strategies used in Tian et al.’s method [28] are more
complex, and we would expect them to analyze more complex temporal relationships than just temporal order, theabove results indicate that temporal order is not important.
Based on the above observations, we hypothesized that thehigh anomaly detection accuracy achieved by these methods is due to the fact that they have a mechanism that allows all segments from the video to be trained together andextracts salient features that are important for determining
normal/abnormal.
Our method (Figure 1) is designed based on the above
insights. Our method is not an MIL framework but it is amodel that takes all segments in a video as input and determines whether the video is normal or abnormal. Since
we do not use MIL, the extraction of salient features can beachieved with a simple self-attention mechanism. For the
self-attention mechanism, we introduce a model inspiredby Lin et al.’s method [14]: their method targets sentenceclassification and can deal with variable length input. We
adopted this mechanism because we divide the video Vi
into T segments during training, but the number of segments during inference should be different dependent on thelength of the input video. Our method is accurate and also
lightweight because it does not have any mechanism to capture temporal orders.
Our self-attention mechanism is similar in concept tothat of Claws [36, 37], but there are some differences between them. Claws generates the attention map twice but
our method generates it once. While Claws’ paper statesthat the purpose of introducing the self-attention mechanism is to suppress normalcy, our purpose is to extract
salient features that are important in determining normality/abnormality.

3.2. Inference
Let Ve be the video to which we want to apply our
method for anomaly detection. First, we divide Ve into N
esegments {V1e , V2e , · · · , VN
} using 16 consecutive frames
as one segment. Each segment Vie is converted by I3D [3]into a D-dimensional feature vector Fei . Let the set of feature vectors be Fe = {Fe1 , Fe2 , · · · , FeN }. Let l be the split
size and we divide Fe into m = N/l bags for every l segments:
Fe = {B1 , B2 , · · · , Bm } ={{Fe1 , · · · , Fel }, {Fel+1 , · · · , Fe2l } · · · {FeN −l+1 , · · · , FeN }}.
Next, we input B1 , B2 , · · · , Bm one by one into ouranomaly detector, and we obtain the inference result
Sv = {sv1 , sv2 , · · · , svm }. svi is used as the anomaly scoreee}. When it is necfor each segment of {V(i−1)l+1, · · · , Vilessary to produce the results of frame-level anomaly detection, the obtained scores are assigned to all frames in eachsegment.

anomalies. The total number of videos is 1900, and the total
duration of the videos is 128 hours. 1610 of the 1900 videos
are for training and 290 are for testing. Each training videois labeled as normal/abnormal on a per-video basis. In eachtest video, each frame is labeled as normal or abnormal.
ShanghaiTech dataset is a medium-sized dataset ofvideos captured by fixed cameras installed in the university.
The dataset contains 437 videos captured by 13 fixed cameras. Of the 437 videos, 307 are normal videos and 130 arevideos with anomalies. The original dataset was intended
for the development of an anomaly detector based on unsupervised learning [18]. However, Zhong et al. labeled each
video so that it could be used as a dataset for weakly supervised learning [40]. We constructed a weakly supervised
dataset using the same procedure as Zhong et al. [40] andconducted experiments.
XD-Violence dataset is a large dataset containing various types of videos, such as movies and videos from video
sharing sites. The total number of videos is 4754, and thetotal duration of the videos is 217 hours. 2405 of the 4754
videos contain 6 types of anomalies: fighting, shooting, riot,
abuse, explosion, and car accident. 2349 of the 4754 videos
are normal. As in UCF-Crime dataset, each training video islabeled with a per-video label, and each test video is labeledwith a per-frame label.
Evaluation measure: for UCF-Crime and ShanghaiTech datasets, we used the Area Under the Curve (AUC)
with the Receiver Operating Characteristic (ROC) curve,
which is calculated based on the frame-level anomaly detection accuracy, as in the existing studies [27, 28, 39, 41,
40, 36, 33, 5, 8, 7]; a larger AUC value indicates a more accurate anomaly detector. For XD-Violence dataset, we used
Average Precision (AP) as the evaluation measure, as in theexisting study [33, 28]; a larger AP value indicates a moreaccurate anomaly detector.

4.2. Implementation detailTo evaluate the proposed method, we conducted
experiments on three weakly supervised datasets foranomaly detection: UCF-Crime dataset [27], ShanghaiTech
dataset [18], and XD-Violence dataset [33]. We compare
the detection accuracy with several existing methods.

Our method was developed and evaluated using PyTorch [21]. Radam [16] was used as the optimization algorithm, and the learning rate was set to 0.001. The batch size
was set to 64. As in the existing method [27], mini-batches
were created so that each mini-batch contained an equal
number of normal and abnormal videos, i.e., 32 normal
videos and 32 abnormal videos in our case. The hyperparameter T was set to 32. For UCF-Crime and ShanghaiTech
datasets, as in the existing method [28, 5], 10-crop augmentation was performed for each video. For XD-Violence
dataset, we performed 5-crop augmentation for each video
as in the existing method [33].

4.1. Dataset and evaluation measure

4.3. Analysis on Our Model

UCF-Crime dataset is a large dataset of real-world
surveillance video [27]. This dataset contains 13 types of

We have experimented with adding the bi-directional
long short-term memory (LSTM) to our model and re-

4. Experiments

moving the self-attention mechanism from our model. In
the context of sentence classification, the model of Lin etal. [14] achieves high classification accuracy, where theself-attention mechanism is used after the bi-directional
LSTM. Inspired by this, we also extended our model byadding the bi-directional LSTM and evaluated the accuracyof anomaly detection. Specifically, our model proposed in
Section 3 directly inputs Fi into the Atten-MLP1 (Figure 1).
In the model with the bi-directional LSTM, Fi is first input
to the bi-directional LSTM and the output from it is input tothe Atten-MLP1. The dimension of the hidden layer of the
LSTM was set to 256.
The experimental results are shown in Table 2. For
UCF-Crime and ShanghaiTech datasets, we used the features obtained by I3D, while for XD-Violence dataset, since
it contains audio information, we used the features obtained by VGGish [10] in addition to I3D. For UCF-Crime
and ShanghaiTech datasets, the model using only the selfattention mechanism without the bi-directional LSTM (themodel proposed in Section 3) achieved the highest detection
accuracy. For XD-Violence dataset, the model with both bidirectional LSTM and self-attention mechanism achieved
the highest detection accuracy. This may be due to the factthat each video in XD-Violence dataset contains audio information, and the bi-directional LSTM may have worked
effectively for audio information.
Table 2. Results of experiments on the addition of the bidirectional LSTM (BL) to our model and the removal of the selfattention mechanism (SA) from our model
BL
X
X

SA
X
X

UCF-Crime
81.72
84.91
83.28

ShanghaiTech
92.90
95.72
94.06

XD-Violence
75.92
75.46
82.89

4.4. Results on UCF-Crime
Table 3 shows the frame-level AUC performance on the
UCF-Crime dataset. The split size l is set to 28. Comparedto MIST [5] and Wu et al. [33], which use the same I3D
RGB features, our method achieves higher detection accuracy. Although our method is a simple model, it achieves
the highest detection accuracy 84.91% and this is 0.61%
higher than RTFM [28], which has the highest accuracy
among the existing methods.

4.5. Results on ShanghaiTech
Table 4 shows the frame-level AUC performance on the
ShanghaiTech dataset. The split size l is set to 21. Ourdetection accuracy is 95.72% and this is 1.49% inferior to
RTFM [28], which has the highest accuracy among the existing methods. However, our method achieves the second

Table 3. Comparison of frame-level AUC performance on the
UCF-Crime dataset. Blue is the highest value and red is the second highest value

Supervision

One-classclassifier

Supervised

WeaklySupervised

MethodSVM Baseline
Conv-AE [8]Lu et al. [17]BODS [30]
GODS [30]
NLN [31]
Lin et al. [15]Lin et al. [15]Sultani et al. [27]Zhang et al. [39]
Motion-Aware [41]GCN-Anomaly [40]
CLAWS Net [36]
CLAWS Net+ [37]
CLAWS Net+ [37]Wu et al. [33]MIST [5]RTFM [28]
RTFM [28]
Our (da =64,r=3)
Our (da =128,r=7)

Feature TypeNLN RGB
C3D RGBNLN RGB
C3D RGB
C3D RGBPWC Flow
TSN RGB
C3D RGB
C3D RGB
3DResNext
I3D RGBI3D RGB (Fine)

C3D RGBI3D RGBI3D RGB

AUC(%)
50.00
50.60
65.51
68.26
70.46
78.9
70.1
82.0
75.41
78.66
79.00
82.12
83.03
83.37
84.16
82.44
82.30
83.28
84.30
84.74
84.91

highest value and outperforms the other existing methods.
In addition, the accuracy of more than 95% is achieved, indicating that a sufficiently practical anomaly detector can betrained.
Table 4. Comparison of frame-level AUC performance on the
ShanghaiTech dataset. Blue is the highest value and red is thesecond highest value

Supervision
One-classclassifier

WeaklySupervised

MethodConv-AE [8]Frame-Pred [32]Mem-AE [7]
VEC [35]
GCN-Anomaly [40]Zhang et al. [39]
CLAWS Net [36]
CLAWS Net+ [37]
CLAWS Net+ [37]AR-Net [29]MIST [5]RTFM [28]
RTFM [28]
Our (da =64,r=3)

Feature TypeTSN RGBI3D RGB
C3D RGB
C3D RGB
3DResNext
I3D RGB&Flow
I3D RGB (Fine)

C3D RGBI3D RGBI3D RGB

AUC(%)
60.85
73.40
71.20
74.80
84.44
82.50
89.67
90.12
91.46
91.24
94.83
91.51
97.21
95.72

Table 6. Comparison of the number of trainable parameters

4.6. Results on XD-Violence
Table 5 shows the frame-level AP performance on the
XD-Violence dataset. The split size l is set to 9. Whenusing only I3D RGB features, our method was inferior to
Wu et al. [33] and RTFM [28]. Since XD-Violence dataset
contains audio information, we can also use audio features(VGGish) [10] in addition to I3D RGB features as Wu etal. [33] and Pang et al. [20] did. When using the audio features, as mentioned in Section 4.3, we extend our modelby adding the bi-directional LSTM in front of the selfattention mechanism, which is represented as “Ours†” inTable 5. This extended model achieves the highest detection accuracy 82.89%: this is 1.2% higher than that of Panget al. [20], which is a method dedicated to anomaly detection based on multimodal information.
Table 5. Comparison of frame-level AP performance on XDViolance dataset. Blue is the highest value and red is the second highest value. “Ours†” represents the model that adds bidirectional LSTM to our original modelSupervision
One-classclassifier

WeaklySupervised

Method
SVM baseline
Hasan et al. [9]Sultani et al. [27]Wu et al. [33]
RTFM [28]Ours (da =64,r=3)
Wu et al. [33]Pang et al. [20]Ours (da =64,r=3)
Ours† (da =64,r=3)
Ours† (da =128,r=1)

Feature TypeC3D RGBI3D RGBI3D RGB+VGGish

AP(%)
50.78
30.77
73.20
75.68
77.81
73.25
78.64
81.69
75.46
79.92
82.89

4.7. Comparison of the number oftrainable parameters
Table 6 shows the number of parameters that can betrained in the model for each method. Our method is anextremely lightweight model with a much smaller numberof parameters than the existing methods. Even though thenumber of parameters is only 1.3% of RTFM [28] whenda = 64 and r = 3, our method achieves higher accuracy
than RTFM [28] in Table 3, and achieves a comparable highaccuracy in Table 4. It is also the lightest model among theexisting methods even when da = 128 and r = 7, which
achieves even higher accuracy.

4.8. Visualization of Attention Map
It was described in Section 3 that our method does not
use the MIL framework and so the self-attention mechanism can be used for extracting salient features. Therefore,
we visualize A2 , the attention map, to investigate whether

MethodSultani et al. [27]Wu et al. [33]
RTFM [28]Ours (da = 64, r = 3)
Ours (da = 128, r = 7)

Number of Parameters
2,114,113
769,155
24,718,849
328,004
721,992

such salient features are really extracted. Here we visualize
A2 for “Buruglary030”, which is one of the training videos
in the UCF-Crime dataset. Figure 2 shows the heatmap of
A2 for the 50-th iteration, the 500-th iteration, and the highest detection accuracy. Note that one iteration here means
training on one mini-batch. As the training progresses, theweights are concentrated on specific segments, indicatingthat the self-attention mechanism is automatically learning
the segments of interest. The fact that the weights are concentrated on multiple segments instead of one indicates thatthe self-attention mechanism has an effect similar to the topk strategy of RTFM [28].

4.9. Performance on each anomaly class
Figure 3 shows the AUC performance of the proposed
method for each anomaly class on the UCF-Crime dataset.
Our method achieves higher or comparable detection accuracy compared to RTFM [28] for six classes of anomalies: Abuse, Arrest, Assault, Explosion, RoadAccidents,
and Stealing. In particular, for the four classes of anomalies, Abuse, Assault, Explosion, and Stealing, the detection
accuracy was improved by more than 6%. Since these fourclasses of anomalies cannot be detected without long-term
analysis of the motion of objects and people, we suspectthat these results indicate that the proposed method successfully captures the long-term features of the videos. For thefour classes of Arrest, Arson, RoadAccidents, and Shooting, our method performed comparably to RTFM. For thefour classes of Burglary, Fighting, Robbery, and Shoplifting, our method was inferior to RTFM. Our method may be
still difficult to classify instantaneous anomalies. In addition, RTFM seems to be a method that is goot at capturingfeatures on human movement.

4.10. Analysis on the hyperparameters da and r
We defined da and r as hyperparameters in Section 3. Bychanging the hyperparameters, the number of trainable parameters changes, and the detection accuracy also changes.
Therefore, using the UCF-Crime dataset, we investigated
how changes in hyperparameters affect the detection accuracy. The split size l was set to 32. The results are shown in
Figure 4. The points surrounded in red represent the highest detection accuracy for each da . Focusing only on r, if r
is larger than 3, detection accuracy is almost stable and its

Figure 2. Visualization of the attention map A2
Ours (da = 128, r = 7)
RTFM

89.4

90

82.0

AUC(%)

80

70

70.7

68.7
65.3

70.1

70.0

70.8
66.3

57.3

55.9

80

63.2

62.7

59.3 59.1

60

69.8

75.7

73.6

65.4

64.9

62.9

71.9

achieved when r = 1, regardless of the value of da . In thecase of the model with the bi-directional LSTM, usage ofa model that is too large for the self-attention mechanism
may cause overfitting.

58.4

79

55.9

52.0

ArrestArson

Assault

AP(%)

45.2
Abuse

Burglary Explosion FightingRoadAccidentsRobbery Shooting Shoplifting Stealing Vandalism

Abnormal classes in UCF-Crime

Figure 3. AUC performance by anomaly classes in UCF-Crime
dataset

76
75
1

2

3
r

4

5

Figure 5. Relationship between hyperparameters (da and r) andAP performance on the XD-Violence dataset

85.5

da = 128, r = 7
da = 64, r = 3

84.91

85.0
AUC(%)

variation is small. Focusing on the relationship between da
and r, if da is increased, r should also be increased to someextent to obtain better detection accuracy. However, if r ismade too large, the detection accuracy will decrease. Up
to 256, better detection accuracy could be obtained by increasing da , but when da was increased to 512, the overall
detection accuracy decreased regardless of the value of r.
This may be due to overfitting caused by making the model
too large.

84.5

84.74

84.0
83.5
83.0

85.0

84.68

84.59

84.51

84.5
AUC(%)

78.54

77

74

82.5

84.53

84.0
da=64
da=128da=256
da=512

83.5
83.0

da=64
da=128da=256
da=512

78

50

40

79.78
79.72
79.56

1

2

3

4

5

r

6

7

8

9

10

Figure 4. Relationship between hyperparameters (da and r) and
AUC performance on the UCF-Crime dataset

We also investigated the influence of hyperparameters on
the detection accuracy using the XD-Violence dataset. Here
we use the extended model with the bi-directional LSTM
and set the split size l to 32. The results are shown in
Figure 5. It can be seen that good detection accuracy is

1

16

32

48
64
Split Size(l)8096Figure 6. Relationship between the split size l and AUC performance during inference on the UCF-Crime dataset

4.11. Analysis on the split size l
We defined the split size l in Section 3.2. Since ourmethod divides N feature vectors into m = N/l bags during inference, each bag contains l feature vectors. Since thedetection accuracy varies depending on the split size l, we
used the UCF-Crime dataset to investigate how the change
in l affects the detection accuracy. The result for da = 64
and r = 3 and the result for da = 128 and r = 7 are shownin Figure 6. The points surrounded in red represent the highest detection accuracy for each hyperparameter set. The de-(a)Burglary061(b)Explosion033(c)Normal352(d)Shoplifting015

Figure 7. Visualization of the anomaly scores of our method. Black lines show the transition of anomaly scores. Orange blocks indicate
ground truth. Blue arrows indicate correctly detected anomalies. Red arrows indicate incorrectly detected anomalies

tection accuracy is increased until l is around 16. Since
our method analyzes the entire video, it requires a certain
length of the video, which means that a certain split size isnecessary. On the other hand, when the split size l is larger
than 16, the detection accuracy is stable, indicating that our
method can analyze the whole video efficiently if the video
is long enough.

4.12. Qualitative Analysis
Figure 7 shows the transition of anomaly scores predicted by our method for the videos of “Burglary061”,“Explosion033”, “Normal352”, and “Shoplifting015” in the
UCF-Crime datasets. The hyperparameters da and r are set
to 128 and 7, respectively, and the split size l is set to 16.
Our method successfully detects abnormal frames when aburglar is breaking the window glass and stealing items asshown in Figure 7-a, or abnormal frames when an explosion accident happens and then dust is flying as shown in
Figure 7-b. Our method correctly detects anomalies that are
occurring for a long period of time. Figure 7-c shows avideo at a gas station that does not contain any anomalies,
and our method does not cause any false positives. Figure 7d is an failure case. This is a video of a man shoplifting
items placed on a counter. Although our method is able todetect anomalies, there are moments when false positives
are detected. As described in Section 4.9, our method is notgood at detecting short-term anomalies. For a instantaneous
anomaly such as shoplifting, the frames before and after the
anomaly may be identified as anomalies in addition to themoment when the anomaly actually occurred.

5. Conclusion
We have proposed a lightweight and accurate weakly supervised learning method for anomaly detection from video.
Since MIL is not used, the extraction of salient features canbe achieved with a simple self-attention mechanism. We
show that the proposed model is simple and lightweight,
yet achieves the comparable or better accuracy than stateof-the-art methods.

References[1] Andra Acsintoae, Andrei Florescu, Mariana-Iuliana
Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor
Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly
detection. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), pages
20143–20153, June 2022.
[2] Arslan Basharat, Alexei Gritai, and Mubarak Shah. Learning
object motion patterns for anomaly detection and improved
object detection. In 2008 IEEE Conference on Computer
Vision and Pattern Recognition, pages 1–8, 2008.
[3] Joao Carreira and Andrew Zisserman. Quo vadis, action
recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 6299–6308, 2017.
[4] Yang Cong, Junsong Yuan, and Ji Liu. Sparse reconstruction
cost for abnormal event detection. In CVPR 2011, pages
3449–3456, 2011.
[5] Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist:
Multiple instance self-training framework for video anomaly
detection. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 14009–
14018, 2021.
[6] Mariana-Iuliana Georgescu, Antonio Barbalau, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, andMubarak Shah. Anomaly detection in video via selfsupervised and multi-task learning. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 12742–12752, 2021.
[7] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha,
Moussa Reda Mansour, Svetha Venkatesh, and Antonvan den Hengel. Memorizing normality to detect anomaly:
Memory-augmented deep autoencoder for unsupervised
anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705–1714,
2019.
[8] Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K
Roy-Chowdhury, and Larry S Davis. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
733–742, 2016.
[9] Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K
Roy-Chowdhury, and Larry S Davis. Learning temporal reg-ularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
733–742, 2016.
[10] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F
Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal,
Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131–135. IEEE, 2017.
[11] Radu Tudor Ionescu, Fahad Shahbaz Khan, Mariana-Iuliana
Georgescu, and Ling Shao. Object-centric auto-encoders anddummy anomalies for abnormal event detection in video. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7842–7851, 2019.
[12] Jaechul Kim and Kristen Grauman. Observe locally, inferglobally: A space-time mrf for detecting abnormal activities with incremental updates. In 2009 IEEE Conference
on Computer Vision and Pattern Recognition, pages 2921–
2928, 2009.
[13] Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos.
Anomaly detection and localization in crowded scenes. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
36(1):18–32, 2014.
[14] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos,
Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A
structured self-attentive sentence embedding. arXiv preprint
arXiv:1703.03130, 2017.
[15] Kun Liu and Huadong Ma. Exploring background-bias foranomaly detection in surveillance videos. In Proceedings ofthe 27th ACM International Conference on Multimedia, MM’19, page 1490–1499, New York, NY, USA, 2019. Association for Computing Machinery.
[16] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen,
Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint
arXiv:1908.03265, 2019.
[17] Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision, pages 2720–2727,
2013.
[18] Weixin Luo, Wen Liu, and Shenghua Gao. A revisit ofsparse coding based anomaly detection in stacked rnn framework. In Proceedings of the IEEE International Conference
on Computer Vision, pages 341–349, 2017.
[19] Trong-Nguyen Nguyen and Jean Meunier. Anomaly detection in video sequence with appearance-motion correspondence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1273–1283, 2019.
[20] Wen-Feng Pang, Qian-Hua He, Yong-jian Hu, and YanXiong Li. Violence detection in videos based on fusing visual and audio information. In ICASSP 2021 - 2021 IEEE
International Conference on Acoustics, Speech and Signal
Processing (ICASSP), pages 2260–2264, 2021.
[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer,
James Bradbury, Gregory Chanan, Trevor Killeen, Zeming
Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Ad-

vances in neural information processing systems, 32:8026–
8037, 2019.
[22] Didik Purwanto, Yie-Tarng Chen, and Wen-Hsien Fang.
Dance with self-attention: A new look of conditional random fields on anomaly detection in videos. In Proceedingsof the IEEE/CVF International Conference on Computer Vision (ICCV), pages 173–183, October 2021.
[23] Mahdyar Ravanbakhsh, Moin Nabi, Hossein Mousavi, Enver
Sangineto, and Nicu Sebe. Plug-and-play cnn for crowd motion analysis: An application in abnormal event detection. In
2018 IEEE Winter Conference on Applications of Computer
Vision (WACV), pages 1689–1698, 2018.
[24] Mahdyar Ravanbakhsh, Moin Nabi, Enver Sangineto, Lucio Marcenaro, Carlo Regazzoni, and Nicu Sebe. Abnormal
event detection in videos using generative adversarial nets.
In 2017 IEEE International Conference on Image Processing (ICIP), pages 1577–1581, 2017.
[25] Hitesh Sapkota, Yiming Ying, Feng Chen, and Qi Yu. Distributionally robust optimization for deep kernel multiple instance learning. In Arindam Banerjee and Kenji Fukumizu,
editors, Proceedings of The 24th International Conferenceon Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2188–2196.
PMLR, 13–15 Apr 2021.
[26] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein,
Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised
anomaly detection with generative adversarial networks toguide marker discovery. In International conference on information processing in medical imaging, pages 146–157.
Springer, 2017.
[27] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world
anomaly detection in surveillance videos. In Proceedings ofthe IEEE conference on computer vision and pattern recognition, pages 6479–6488, 2018.
[28] Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh,
Johan W Verjans, and Gustavo Carneiro. Weakly-supervised
video anomaly detection with robust temporal feature magnitude learning. In Proceedings of the IEEE/CVF international
conference on computer vision, 2021.
[29] Boyang Wan, Yuming Fang, Xue Xia, and Jiajie Mei.
Weakly supervised video anomaly detection via centerguided discriminative learning. In 2020 IEEE International
Conference on Multimedia and Expo (ICME), pages 1–6.
IEEE, 2020.
[30] Jue Wang and Anoop Cherian. Gods: Generalized one-classdiscriminative subspaces for anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8201–8211, 2019.
[31] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[32] Dongze Lian Wen Liu, Weixin Luo and Shenghua Gao. Future frame prediction for anomaly detection - - a new baseline. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018, pages 6536–6545, 2018.
[33] Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao,
Zhaoyang Wu, and Zhiwei Yang. Not only look, but also

listen: Learning multimodal violence detection under weaksupervision. In European Conference on Computer Vision,
pages 322–339. Springer, 2020.
[34] Dan Xu, Yan Yan, Elisa Ricci, and Nicu Sebe. Detecting
anomalous events in videos by learning deep representationsof appearance and motion. Computer Vision and Image Understanding, 156:117–127, 2017.
[35] Guang Yu, Siqi Wang, Zhiping Cai, En Zhu, Chuanfu Xu,
Jianping Yin, and Marius Kloft. Cloze test helps: Effective video anomaly detection via learning to complete video
events. In Proceedings of the 28th ACM International Conference on Multimedia, pages 583–591, 2020.
[36] Muhammad Zaigham Zaheer, Arif Mahmood, Marcella
Astrid, and Seung-Ik Lee. Claws: Clustering assisted weakly
supervised learning with normalcy suppression for anomalous event detection. In European Conference on Computer
Vision, pages 358–376. Springer, 2020.
[37] Muhammad Zaigham Zaheer, Arif Mahmood, Marcella
Astrid, and Seung-Ik Lee. Clustering aided weakly supervised training to detect anomalous events in surveillance
videos. arXiv preprint arXiv:2203.13704, 2022.
[38] M. Zaigham Zaheer, Arif Mahmood, M. Haris Khan, Mattia
Segu, Fisher Yu, and Seung-Ik Lee. Generative cooperative
learning for unsupervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), pages 14744–14754, June2022.
[39] Jiangong Zhang, Laiyun Qing, and Jun Miao. Temporal convolutional network with complementary inner bag loss forweakly supervised anomaly detection. In 2019 IEEE International Conference on Image Processing (ICIP), pages
4030–4034. IEEE, 2019.
[40] Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu,
Thomas H Li, and Ge Li. Graph convolutional label noisecleaner: Train a plug-and-play action classifier for anomaly
detection. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 1237–
1246, 2019.
[41] Yi Zhu and Shawn Newsam.
Motion-aware featurefor improved video anomaly detection. arXiv preprint
arXiv:1907.10211, 2019.