Year: 2021

Abstract:
The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds, such as running, bicycling, throwing from a height. The algorithm forms continuous video frames into a frame group and uses the frame-group feature extractor to obtain the spatio-temporal information. The implicit vector based attention mechanism will work on the extracted frame-group features to highlight the important features. We use fully connected layers to transform the space and reduce the computation. Finally, the group-pooling maps the processed frame-group features to the abnormal scores. The network input is flexible to cope with the form of video streams, and the network output is the abnormal score. The designed compound loss function will help the model improve the classification performance. This paper arranges several commonly used anomaly detection datasets and tests the algorithms on the integrated dataset. The experimental results show that the proposed algorithm has significant advantages in many objective metrics compared with other anomaly detection algorithms.

heading: 1 INTRODUCTION
text: Nowadays anomaly detection is useful to maintain social security and conduct legal forensics. Due to the ambiguous definition of abnormal events, it increases the difficulty of detection. For example, the appearance of a vehicle on a road is normal, while it is abnormal when a vehicle is on the sidewalk. So we make the goal clear in order to carry out consequent measures. Abnormal behavior is defined as rapid movements in slow moving crowds, such as cycling, running, throwing from a height, etc. Based on this definition, the commonly used datasets for anomaly detection are integrated and relabeled. The new fusion dataset contains more scenes and anomaly types, so it is more challenging for the anomaly detection. Subsequent experiments are performed on this new dataset. The test results show that the algorithm proposed in this paper has great advantages in many metrics.
The end-to-end anomaly detection network shown in Fig. 1 has the following contributions:
(1) The video group composed of consecutive multiple frames is the basic processing unit to extract expressive features with the designed group feature extractor. On the one hand, the spatial-temporal information could be retained comparing with single-image processing. On the other hand, the
abnormality score of a single frame can be easily obtained without the access of the whole video, which can copy with the situation where video streams are the input.
(2) The designed framework composed of group feature extractor and group score mapper can effectively obtain the abnormality score using the spatial-temporal information. Implicit vector-based attention mechanism is used to weight the frame-group features. The more important the feature is, the higher the weight is.
(3) The basic cross-entropy loss and the improved hinge loss are united to improve the performance of the network. The latter devote to make the score of the abnormal frames greater than that of normal frames.
The paper is organized as follows. Section 1 introduces the background of anomaly detection. Section 2 introduces the related work on anomaly detection. Section 3 mainly introduces the details of the proposed anomaly detection algorithm. Section 4 introduces the fusion dataset and then analyzes the experimental results. Section 5 gives the summary of the whole paper.

heading: 2 RELATED WORK
text: The challenges of video semantic analysis lie in the extraction and representation of video features. The video contains complex spatial texture information and time information. Multidimensional data provides more information but meanwhile it contains a lot of redundant information. How to extract low-redundant, comprehensive and representative video features is one of the research focuses. The manual extraction method of video features focuses on the extraction and analysis of low-level visual features, such as guided gradient histograms Xiao et al. (2014), optical flow maps Reddy et al. (2011), spatio-temporal points of interest Dollár et al. (2005), texture models Xiao et al. (2018), filtering models Zhang et al. (2018), etc. After obtaining the statistical information, the visual dictionary Roshtkhari & Levine (2013) and other methods will be used to save the normal distribution, and then calculate the similarity criterion to determine whether the target is abnormal. Luo & Wang (2019) explores the method of multi-stream manual features for video representation. It constructs a three-dimensional video representation structure composed of spatio-temporal vector and positional vector, and improves the encoding method to make the extracted video representation structure more powerful in representation.
With the rapid development of deep learning, automatic feature extraction using neural networks has become a research hotspot. Zhou et al. (2016) uses 3D convolutional networks to detect abnormal events in surveillance video. In Sabokrou et al. (2017), the video frame is divided into several small areas, and each small area is feed into a 3D self-encoding network combined with 3D convolutional neural network to extract features and detect anomaly. Medel (2016) proposes a long-short term memory network based on convolution, which simultaneously extracts spatial and temporal information. Xu et al. (2015) first uses stacked auto-encoders to learn and fuse the appearance and motion characteristics of abnormal individuals, and then trains multiple single-classifiers to calculate the abnormal score. Ionescu et al. (2019) uses one-to-many classifiers instead of the single-classifiers after obtaining multiple pseudo-anomaly classes from the trained normal behavior pattern. Hinami et al. (2017) multiple attributes of the same target to extract features.
Due to the lack of anomalous videos and various types of anomaly, it is difficult to find a general model that covers all anomalous events. The auto-encoding network Yuan et al. (2019) performs anomaly detection based on the reconstruction error. Hasan et al. (2016) uses a convolutional neural network to implement video anomaly detection. Since the convolutional layer operates on a twodimensional structure, time information will be lost. Chong & Tay (2017) designs a spatio-temporal autoencoder that encodes video sequence with spatial convolution and Convolutional Long-Short Term Memory (ConvLSTM) Shi et al. (2015) structure, and then uses the symmetric structure called the decoder, convert the video encoding into the image sequence. The abnormal score can be obtained from the calculation of the Euclidean distance between the decoded images and the original images. An & Cho (2015) proposes a variational autoencoder (Variational Autoencoder, VAE), which uses the results of video encoding to fit a distribution function. Jing & Yujin (2017) supplements a gradient difference constraint on the basis of the sparse denoising auto-encoding network, which is helpful to make the model more effective in detecting abnormal behavior.
For specific tasks, the model trained according to the specific task can often get better results. In Sultani et al. (2018), the video containing abnormal events is divided into several video segments to form multiple video instances. A fully connected network is designed to map the video features extracted by the C3D network Tran et al. (2015) into abnormal scores. The score of the abnormal instance is higher than the score of the instances with only normal frames. The experiments in this paper show that the network has achieved good results. But the detection process is disconnected. In order to minimize the interference to the input data and obtain the final score directly and timely, we designs an end-to-end network, which uses both positive and negative samples to make the model more targeted.

heading: 3 PROPOSED METHOD
text: The paper proposes an end-to-end anomaly detection network, which operates on the frame-group formed with consecutive frames to obtain the abnormality score. The whole framework is composed of group feature extractor and group score mapper. The former performs on the raw frame-group to obtain the spatial-temporal group-feature. The latter performs on the group-feature to obtain the abnormality score.

heading: 3.1 THE GROUP-FEATURE EXTRACTOR
text: The frame-group is defined as a structure consisting of consecutive τ frames, which contains rich spatial texture information and temporal change information. Fig. 2 shows the details of the groupfeature extractor, where In(0,Spa,Tem)t and Out (0,Spa,Tem) t represent the input and output matrixes in time t respectively. Spa and Tem mean the input and output feature map in the spatial feature extractor and the temporal feature extractor respectively.
Since the frame-group contains multiple continuous frames, in order not to destroy the time information within consecutive frames, we use a trainable convolution kernel to extract spatial information of each single-frame in the time dimension. Then batch normalization Ioffe & Szegedy (2015) is used to prevent the gradient dispersion and accelerate the model convergence. Activation function helps to make the model nonlinear. Two sets of ”convolution-regular-activation” structures are used in the experiments. The first group uses 128 convolution kernels with size of 5× 5 and step size of 3, and the second one uses 64 convolution kernels with size of 3× 3 and step size of 2. After obtaining the spatial feature maps, ConvLSTM will further extract the spatial and temporal information using gate structures. The implementation of hidden features is shown in Equation (1). Where the hidden feature HidTemt is used to record the accumulated state information up to time t. The symbol ⊗ represents the Hadamard product. σ and tanh represents the sigmoid and tanh nonlinear activation functions respectively. Conv is the convolution operation. This paper uses two cascading ConvLSTM layers to extract time flow information. The first layer uses 64 convolution kernels with size of 3 × 3 and the second one uses 32 same kernels. We use the ”same padding” to keep the size of input and output feature maps same.
HidTemt = σ(Conv(Out Tem t−1 , In Tem t ,Hid Tem t−1 )) ⊗ HidTemt−1
+ σ(Conv(OutTemt−1 , In Tem t ,Hid Tem t−1 ) ⊗ tanh(Conv(Outt−1, In Tem t )) (1)
While the final output is shown in Equation (2):
OutTemt = σ(Conv(Out Tem t−1 , In Tem t ,Hid Tem t )) ⊗ tanh(HidTemt ) (2)
In order to better extract the spatial and temporal information as well as high-level and low-level features of frame-group, we use a multi-level feature fusion structure to merge multi-level features of the frame-group. The output result of the multi-level feature fusion structure is regarded as the final feature representation of the frame-group. In the experiment, the structure was implemented using a 3D convolution Zhou et al. (2018) layer with size of 8× 1× 1.

heading: 3.2 THE GROUP-SCORE MAPPER
text: In the group-score mapper, the attention mechanism Ilse et al. (2018) is used to increase the decisive influence of useful features and weaken the effects of irrelevant features on the results. The fully connected network is used to make the encoding of the group-feature more expressive while reducing feature dimensions. The group-level pooling is applied to map the refined group-feature to the abnormality score of the video group. The specific process is shown in Fig. 3.
The implicit vector method based attention mechanism assigns different weights to different features in order that the key features have a more important impact on the result and the interference of noise can be suppressed. The trainable transformation matrix will be used to project the original group feature into the implicit space, and then the weight vector will be obtained by an inverse transformation matrix. Different from the attention mechanism aiming at multiple instances in Ilse et al. (2018), this paper focus on the attention of the group-feature. To be specific, the implicit vector is used to generate a weight vector of the original group, which has a dimension of 128 in the experiment. Fgrp is the flattened Out0t with length =1444. T represents transposition operation. V and W are the feature space transformation matrices. Ψnl is a non-linear transformation function. Therefore we can define the coefficient χk of the kth element F (k) grp as Equation (3).
χk = exp((WTΨnl(VF T grp))k)∑K
i=1 exp((W TΨnl(VF T grp))i)
, k ∈ [1,K] (3)
The weighted group feature F̃grp is as Equation (4).
F̃grp = (χ1F (1) grp, · · · , χkF(k)grp), k ∈ [1,K] (4)
The weighted group-feature next passes through two fully connected layers with Dropout operation to reduce the feature dimensions and the computation and enhance the feature expressive ability. The output dimensions of the two fully connected layers are 512, and theRelu activation function is used. The Dropout parameter is set to 0.5, that is, 50% of the neural units are discarded randomly and not participated in each iteration of training, which can reduce the risk of neural network overfitting.
Group-pooling is used to get the final group score. The trainable weight matrix followed by the sigmoid function is used to map the refined group feature to the abnormality score. The positive samples marked as 1 are the frame-groups containing anomalies, while negative samples contain only normal. As the trainable weight matrix of the refined group feature F̃grp is marked Φ and the sigmoid is marked σ the predicted group-score B̂(t) corresponding to the input (In0t , · · · , In 0 t+τ ) is defined as Equation (5).
B̂(t) = σ(F̃grp ·Φ) (5)

heading: 3.3 THE LOSS FUNCTION
text: The cross-entropy loss is the most commonly used loss in general logical classification problems, which considers the maximum likelihood estimation B̂(i) of the ith true group-score B(i) from the perspective of the information entropy. The two-class cross-entropy loss Loss(i)bc for a single sample is shown in Equation (6).
Loss (i) bc = −B (i) log B̂(i) − (1−B(i)) log(1− B̂(i)) (6)
The aim of the proposed algorithm is to predict high abnormality score when the video group contains at least one abnormal frame. Therefore we hope that the abnormality score of video group containing abnormal frames is higher than that of group with normal frames only, that is to say that the loss function should preferably increase the category gap so the network has a certain fault tolerance ability when dealing with a small amount of noise input. Thus we propose the improved hinge loss. The used two-class hinge loss function for a single-sample is shown in Equation (7).
Loss (i) bh = max(0, 1−B (i) · B̂(i) + (1−B(i)) · B̂(i)) (7)
With N as the total number of samples, the final loss is:
Loss = 1
N ( N∑ i=1 Loss (i) bc + N∑ i=1 Loss (i) bc ) (8)

heading: 4 EXPERIMENTS

heading: 4.1 THE DATASETS
text: The proposed anomaly detection network uses the videos taken with the fixed-angle surveillant cameras, to detect anomalous events, including running, cycling, throwing objects, etc. The commonly used anomaly detection datasets are Avenue, UMN and UCSD.
The Avenue dataset Lu et al. (2013) contains the video segments captured on CUHK Campus Avenue, and the anomaly is when at least one runs, walks in the opposite direction, or hangs out. The UMN dataset Mehran et al. (2009) contains different indoor and outdoor scenes with crowd burst. In indoor scenes, the overall sight is dark due to insufficient light. The UCSD dataset Mahadevan et al. (2010) has the anomaly include cycling, carts, and turf crossings. The viewing angle of Ped1 is perpendicular to the camera plane, while the two planes are parallel in Ped2.
Above datasets only contain abnormal videos in testing set, while both positive and negative samples are required for the proposed network training. Thus we merge and filter these testing videos to choose 40 videos with single abnormal event in each one as the new training set, and the remains as the new testing set. It is worth mentioning that the anomalies, such as walking in the wrong direction and turf crossings, are treated as normal since they are not belong to the strenuous motion
in the crowd. The details of the new dataset are shown in Table 1. Compared with the original dataset, the new dataset encounters more challenges: (1) More scenarios. There are six different indoor and outdoor scenes, with different camera positions and angles. (2) More types of abnormal events. The new dataset contains single-person and group abnormal events. Subsequent experiments and comparisons were performed on this dataset.

heading: 4.2 EXPERIMENTS
text: The training and testing codes are run under the centos7 system, using Intel i5-8600K @ 3.60GHz six-core CPU. The network is built using keras framework with tensorflow as the backend, supposed by python, opencv, h5py, etc.

heading: 4.2.1 THE MODELS IN TRAINING STAGE
text: The proposed algorithm uses sequential τ = 8 frames to generate a single video group, and the sliding stride is 1. The output group score is taken as the abnormal score of the first frame of the group. Due to the discontinuity of different video files, the last τ − 1 frames of each video file were discarded when the training set was generated, so the resulting training set contained 15, 618 video frame-groups. The Adam optimizer was used during training, and the learning rate was set to 0.0001; the batch-size was set to 32; each epoch contained 15, 618 sets of inputs during training. Fig. 4 shows the performance of models in training stage.
Fig. 4 (a) shows changes of parameters during training stage. The train loss represents the value of the compound loss function introduced in Section 3.3. The train accuracy is the ratio of the number of correctly classified samples to the total number of samples. We end the training stage when the train loss stays basically unchanged in 5 iterations. The accuracy rate basically does not change after the 12th epoch, so the best model occur after the 12th epoch.
The ROC (Receiver operating characteristic) curve and its corresponding area (AUC) Mahadevan et al. (2010) Saligrama & Chen (2012) are the two commonly used metrics in classification algorithms. We select models every 5 epochs for performance testing and then narrow the scope according to the test results to find the model that performs best on the testing set. Thus we draw the Fig. 4 (b) and (c). TPR indicates the probability of correct prediction in all positive samples, and FPR indicates the probability of incorrect prediction in negative samples. Therefore the EER can be defined as the average of the incorrect prediction in both positive and negative samples. The ideal
target is TPR = 1 and FPR = 0, so the closer the ROC curve is to the (0, 1) point, the better. Table 2 recorded the AUC and EER of the models to make the metrics more specific and easier to compare the performance.
Though the models before 10th epoch have higher AUC values, they are in a large fluctuation state from Fig. 4 (c). The extreme point after 12th epoch is at 20th epoch. Thus the model at 20th epoch is used as the representative model of the proposed algorithm in this paper to compare with other algorithms.

heading: 4.2.2 THE COMPARISON WITH OTHER ALGORITHMS
text: (1) Metrics in AUC and EER
We test the AUC and EER of the spatio-temporal autoencoder (ENC), variational encoder (VAE), multiple instance ranking framework (MIR) and the proposed algorithm (GRP) using the fused dataset introduced in Section 4.1, see Table 3. The GRP has the best performance with the highest AUC and the lowest EER, followed by the ENC. While the VAE based on implicit distribution prediction performs the worst due to the introduction of too much noise.
In order to make the comparison more intuitive, we draw the ROC curves of each algorithm, as shown in Fig. 5. VAE generates fewer and more concentrated abnormal scores, the candidate values of the category segmentation threshold are relatively less and the data has poor separability, resulting in a higher binary classification error rate and a concave state of the ROC curve. In contrast, the GRP considers more scene information and reflects it on the abnormal score, so the obtained abnormal score set performs better under the established rules.
(2)Metrics in frame-level and segment-level
The AUC is used in the situation where the testing set includes both positive and negative samples, which may not be satisfied in real scenes. Thus we refer to the metric accuracy, which indicates the ability of the algorithm to classify the samples. The accuracy is the proportion of correct detection in all the samples. Due to the similarity of the appearance under different actions, it is difficult to judge the border of the abnormal event, which will greatly affect the frame-level metrics. Therefore, the segment-level metric is supplemented, labeled as abnormal segment hit rate (ASH). For a segment containing abnormal events named as abnormal segment, a frame-level detection rate of not less than 60% is judged as detecting the abnormal segment. Define the abnormal segment according to the following two principles. (1) Two anomalous frames are counted as two abnormal segments when the distance between two anomalous frames exceeds 20 frames. (2) The division of an abnormal segment is not based on semantics, and consecutive adjacent anomalous frames belong to the same abnormal segment, even if two types of abnormal events occur in the two frames.
Fig. 5 shows the metrics of different algorithms on the testing set using the threshold selected every 0.01 between 0 and 1. accuracy is the proportion of samples that are detected correctly. The maximum accuracy of the GRP is the best of all algorithms, followed by the ENC. While when the threshold is low, the ASH of ENC is indeed higher than GRP, but it should be noted that each network model has a different focus on abnormality scores, that is the threshold to classify a score as anomaly is different. It can be found that the optimal thresholds of the two networks are basically in the high threshold area, where the ASH of GRP is greater than that of ENC, which is the reason why GRP is superior in other metrics. It is worth noting that the ASH of VAE is ladder-shaped and is relatively less affected by the threshold, which means that the score set of VAE is relatively sparse.
(3)Metrics in the detection effect
From the above metrics, the proposed algorithm and the spatio-temporal autoencoder are the two algorithms with the best performance in the comparison algorithms. Therefore, only the effects of these two algorithms are listed in the following video detection effect examples. Fig. 6 shows the specific score curve of the video. Fig. 6(a) is from the UMN dataset with a sudden burst of the crowds. The scores of the GRP are lower than that of the ENC in normal segments and higher in abnormal segments. Fig. 6(b) is from the UCSD dataset with a bumper car running. Although the GRP generates a high abnormal score at the end of the normal segment, it can detect most of the abnormal frames in the abnormal segment. So the overall effect is better than that of the ENC. Fig. 6(c) is from the Avenue dataset with a parabolic at high altitude. The GRP and the ENC both show a peak when the abnormal event occurs, but the peak time of GRP is slightly delayed. As a whole, the trend of the score curve of GRP and ENC is roughly the same.
The threshold corresponding to the maximum point of the accuracy in Fig. 5 is used as the hyperparameter of the algorithm to determine whether an abnormal event occurs in each frame. The threshold of spatio-temporal autoencoder is 0.68, and its corresponding accuracy is 0.725; the threshold of our algorithm is 0.6, and its corresponding accuracy is 0.746. We use Fβ score to evaluate the performance of the algorithms under the specified threshold. The definition of Fβ score is shown in Equation (9).
Fβ = (1 + β 2) · pre · rec
β2 · pre+ rec , β ∈ N (9)
Where pre represents the proportion of correctly detected samples in all the detected positive samples, and rec represents the proportion of correctly detected positive samples in all positive samples. Fβ score is the harmonic value of pre and rec with the weight of β. When β = 1, namely F1 score, it means that precision and recall are equally important. Fig. 7 calculates the confusion matrix and F1 score of the two algorithms at their respective thresholds.
The numbers in Fig. 7 indicate the number of frames in the corresponding case. It can be seen from Fig. 7 that the GRP detects more abnormal frames with a higher number of both correct and wrong detection than that of the ENC. Hence, the calculation of F1 score is necessary. The F1 score of the GRP is 0.577, while that of the ENC is 0.463. Therefore, when precision and recall are considered together with a same weight, the performance of the GRP is slightly better than that of the ENC.

heading: 5 CONCLUSION
text: This paper proposes an end-to-end anomaly detection network, which can be used to detect strenuous movements in slow moving crowds, such as running, bicycling, throwing from a height. We arrange the multiple images of continuous video frames in a sequence called a frame group as the input of the proposed network to capture the spatio-temporal information. Then the attention mechanism is used to weight the extracted features to highlight the important information and weaken the interference of irrelevant information on the detection results, and fully connected layers to transform space and reduce the dimensions of the features. Finally, group-level pooling is used to map video group-level features to anomalous scores ranging from 0 to 1. The higher the score is, the more probability the frame has abnormal behaviors. More results and source code reported in this paper can be downloaded from website https://github.com/xiaojs18/Anomaly-Detection/tree/main/fgan.

Review summaries:
This paper proposes a novel neural network that maps short chunks of eight video frames to a final probability score (0-1) of abnormality. The network is trained and evaluated in a supervised fashion, using the Avenue, UMN and UCSD data (all three datasets capture semantic anomalies e.g. walking in opposite directions of pedestrians, etc.). The network architecture comprises of various architectural concepts such as CNN, multi-level features, ConvLSTM, attention & transformer, etc. Basically the idea is to design a spatiotemporal feature from the chunk of video frames in an embedding where the features are easily being discriminated to 0/1 (normality/abnormality).

The paper presents an algorithm to detect abnormal events in video sequences. The proposed algorithm decomposes a video sequence into groups of consecutive frames ("frame groups"), and uses ConvLSTM to extract features from frame groups. A group level attention module is applied to focus on some most relevant portions of the extracted features, before feeding to fully connected layers for final anomaly score prediction. The algorithm is evaluated on a curated dataset accumulated from three popular benchmarks.      

The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds. The group-feature extractor is adopted. The proposed algorithm is tested on the integrated dataset.