ANOMALY DETECTION WITH FRAME-GROUP ATTENTION IN SURVEILLANCE VIDEOS

1 INTRODUCTION . Nowadays anomaly detection is useful to maintain social security and conduct legal forensics . Due to the ambiguous definition of abnormal events , it increases the difficulty of detection . For example , the appearance of a vehicle on a road is normal , while it is abnormal when a vehicle is on the sidewalk . So we make the goal clear in order to carry out consequent measures . Abnormal behavior is defined as rapid movements in slow moving crowds , such as cycling , running , throwing from a height , etc . Based on this definition , the commonly used datasets for anomaly detection are integrated and relabeled . The new fusion dataset contains more scenes and anomaly types , so it is more challenging for the anomaly detection . Subsequent experiments are performed on this new dataset . The test results show that the algorithm proposed in this paper has great advantages in many metrics . The end-to-end anomaly detection network shown in Fig . 1 has the following contributions : ( 1 ) The video group composed of consecutive multiple frames is the basic processing unit to extract expressive features with the designed group feature extractor . On the one hand , the spatial-temporal information could be retained comparing with single-image processing . On the other hand , the abnormality score of a single frame can be easily obtained without the access of the whole video , which can copy with the situation where video streams are the input . ( 2 ) The designed framework composed of group feature extractor and group score mapper can effectively obtain the abnormality score using the spatial-temporal information . Implicit vector-based attention mechanism is used to weight the frame-group features . The more important the feature is , the higher the weight is . ( 3 ) The basic cross-entropy loss and the improved hinge loss are united to improve the performance of the network . The latter devote to make the score of the abnormal frames greater than that of normal frames . The paper is organized as follows . Section 1 introduces the background of anomaly detection . Section 2 introduces the related work on anomaly detection . Section 3 mainly introduces the details of the proposed anomaly detection algorithm . Section 4 introduces the fusion dataset and then analyzes the experimental results . Section 5 gives the summary of the whole paper . 2 RELATED WORK . The challenges of video semantic analysis lie in the extraction and representation of video features . The video contains complex spatial texture information and time information . Multidimensional data provides more information but meanwhile it contains a lot of redundant information . How to extract low-redundant , comprehensive and representative video features is one of the research focuses . The manual extraction method of video features focuses on the extraction and analysis of low-level visual features , such as guided gradient histograms Xiao et al . ( 2014 ) , optical flow maps Reddy et al . ( 2011 ) , spatio-temporal points of interest Dollár et al . ( 2005 ) , texture models Xiao et al . ( 2018 ) , filtering models Zhang et al . ( 2018 ) , etc . After obtaining the statistical information , the visual dictionary Roshtkhari & Levine ( 2013 ) and other methods will be used to save the normal distribution , and then calculate the similarity criterion to determine whether the target is abnormal . Luo & Wang ( 2019 ) explores the method of multi-stream manual features for video representation . It constructs a three-dimensional video representation structure composed of spatio-temporal vector and positional vector , and improves the encoding method to make the extracted video representation structure more powerful in representation . With the rapid development of deep learning , automatic feature extraction using neural networks has become a research hotspot . Zhou et al . ( 2016 ) uses 3D convolutional networks to detect abnormal events in surveillance video . In Sabokrou et al . ( 2017 ) , the video frame is divided into several small areas , and each small area is feed into a 3D self-encoding network combined with 3D convolutional neural network to extract features and detect anomaly . Medel ( 2016 ) proposes a long-short term memory network based on convolution , which simultaneously extracts spatial and temporal information . Xu et al . ( 2015 ) first uses stacked auto-encoders to learn and fuse the appearance and motion characteristics of abnormal individuals , and then trains multiple single-classifiers to calculate the abnormal score . Ionescu et al . ( 2019 ) uses one-to-many classifiers instead of the single-classifiers after obtaining multiple pseudo-anomaly classes from the trained normal behavior pattern . Hinami et al . ( 2017 ) multiple attributes of the same target to extract features . Due to the lack of anomalous videos and various types of anomaly , it is difficult to find a general model that covers all anomalous events . The auto-encoding network Yuan et al . ( 2019 ) performs anomaly detection based on the reconstruction error . Hasan et al . ( 2016 ) uses a convolutional neural network to implement video anomaly detection . Since the convolutional layer operates on a twodimensional structure , time information will be lost . Chong & Tay ( 2017 ) designs a spatio-temporal autoencoder that encodes video sequence with spatial convolution and Convolutional Long-Short Term Memory ( ConvLSTM ) Shi et al . ( 2015 ) structure , and then uses the symmetric structure called the decoder , convert the video encoding into the image sequence . The abnormal score can be obtained from the calculation of the Euclidean distance between the decoded images and the original images . An & Cho ( 2015 ) proposes a variational autoencoder ( Variational Autoencoder , VAE ) , which uses the results of video encoding to fit a distribution function . Jing & Yujin ( 2017 ) supplements a gradient difference constraint on the basis of the sparse denoising auto-encoding network , which is helpful to make the model more effective in detecting abnormal behavior . For specific tasks , the model trained according to the specific task can often get better results . In Sultani et al . ( 2018 ) , the video containing abnormal events is divided into several video segments to form multiple video instances . A fully connected network is designed to map the video features extracted by the C3D network Tran et al . ( 2015 ) into abnormal scores . The score of the abnormal instance is higher than the score of the instances with only normal frames . The experiments in this paper show that the network has achieved good results . But the detection process is disconnected . In order to minimize the interference to the input data and obtain the final score directly and timely , we designs an end-to-end network , which uses both positive and negative samples to make the model more targeted . 3 PROPOSED METHOD . The paper proposes an end-to-end anomaly detection network , which operates on the frame-group formed with consecutive frames to obtain the abnormality score . The whole framework is composed of group feature extractor and group score mapper . The former performs on the raw frame-group to obtain the spatial-temporal group-feature . The latter performs on the group-feature to obtain the abnormality score . 3.1 THE GROUP-FEATURE EXTRACTOR . The frame-group is defined as a structure consisting of consecutive τ frames , which contains rich spatial texture information and temporal change information . Fig . 2 shows the details of the groupfeature extractor , where In ( 0 , Spa , Tem ) t and Out ( 0 , Spa , Tem ) t represent the input and output matrixes in time t respectively . Spa and Tem mean the input and output feature map in the spatial feature extractor and the temporal feature extractor respectively . Since the frame-group contains multiple continuous frames , in order not to destroy the time information within consecutive frames , we use a trainable convolution kernel to extract spatial information of each single-frame in the time dimension . Then batch normalization Ioffe & Szegedy ( 2015 ) is used to prevent the gradient dispersion and accelerate the model convergence . Activation function helps to make the model nonlinear . Two sets of ” convolution-regular-activation ” structures are used in the experiments . The first group uses 128 convolution kernels with size of 5× 5 and step size of 3 , and the second one uses 64 convolution kernels with size of 3× 3 and step size of 2 . After obtaining the spatial feature maps , ConvLSTM will further extract the spatial and temporal information using gate structures . The implementation of hidden features is shown in Equation ( 1 ) . Where the hidden feature HidTemt is used to record the accumulated state information up to time t. The symbol ⊗ represents the Hadamard product . σ and tanh represents the sigmoid and tanh nonlinear activation functions respectively . Conv is the convolution operation . This paper uses two cascading ConvLSTM layers to extract time flow information . The first layer uses 64 convolution kernels with size of 3 × 3 and the second one uses 32 same kernels . We use the ” same padding ” to keep the size of input and output feature maps same . HidTemt = σ ( Conv ( Out Tem t−1 , In Tem t , Hid Tem t−1 ) ) ⊗ HidTemt−1 + σ ( Conv ( OutTemt−1 , In Tem t , Hid Tem t−1 ) ⊗ tanh ( Conv ( Outt−1 , In Tem t ) ) ( 1 ) While the final output is shown in Equation ( 2 ) : OutTemt = σ ( Conv ( Out Tem t−1 , In Tem t , Hid Tem t ) ) ⊗ tanh ( HidTemt ) ( 2 ) In order to better extract the spatial and temporal information as well as high-level and low-level features of frame-group , we use a multi-level feature fusion structure to merge multi-level features of the frame-group . The output result of the multi-level feature fusion structure is regarded as the final feature representation of the frame-group . In the experiment , the structure was implemented using a 3D convolution Zhou et al . ( 2018 ) layer with size of 8× 1× 1 . 3.2 THE GROUP-SCORE MAPPER . In the group-score mapper , the attention mechanism Ilse et al . ( 2018 ) is used to increase the decisive influence of useful features and weaken the effects of irrelevant features on the results . The fully connected network is used to make the encoding of the group-feature more expressive while reducing feature dimensions . The group-level pooling is applied to map the refined group-feature to the abnormality score of the video group . The specific process is shown in Fig . 3 . The implicit vector method based attention mechanism assigns different weights to different features in order that the key features have a more important impact on the result and the interference of noise can be suppressed . The trainable transformation matrix will be used to project the original group feature into the implicit space , and then the weight vector will be obtained by an inverse transformation matrix . Different from the attention mechanism aiming at multiple instances in Ilse et al . ( 2018 ) , this paper focus on the attention of the group-feature . To be specific , the implicit vector is used to generate a weight vector of the original group , which has a dimension of 128 in the experiment . Fgrp is the flattened Out0t with length =1444 . T represents transposition operation . V and W are the feature space transformation matrices . Ψnl is a non-linear transformation function . Therefore we can define the coefficient χk of the kth element F ( k ) grp as Equation ( 3 ) . χk = exp ( ( WTΨnl ( VF T grp ) ) k ) ∑K i=1 exp ( ( W TΨnl ( VF T grp ) ) i ) , k ∈ [ 1 , K ] ( 3 ) The weighted group feature F̃grp is as Equation ( 4 ) . F̃grp = ( χ1F ( 1 ) grp , · · · , χkF ( k ) grp ) , k ∈ [ 1 , K ] ( 4 ) The weighted group-feature next passes through two fully connected layers with Dropout operation to reduce the feature dimensions and the computation and enhance the feature expressive ability . The output dimensions of the two fully connected layers are 512 , and theRelu activation function is used . The Dropout parameter is set to 0.5 , that is , 50 % of the neural units are discarded randomly and not participated in each iteration of training , which can reduce the risk of neural network overfitting . Group-pooling is used to get the final group score . The trainable weight matrix followed by the sigmoid function is used to map the refined group feature to the abnormality score . The positive samples marked as 1 are the frame-groups containing anomalies , while negative samples contain only normal . As the trainable weight matrix of the refined group feature F̃grp is marked Φ and the sigmoid is marked σ the predicted group-score B̂ ( t ) corresponding to the input ( In0t , · · · , In 0 t+τ ) is defined as Equation ( 5 ) . B̂ ( t ) = σ ( F̃grp ·Φ ) ( 5 )

Summary: This paper proposes a novel neural network that maps short chunks of eight video frames to a final probability score (0-1) of abnormality. The network is trained and evaluated in a supervised fashion, using the Avenue, UMN and UCSD data (all three datasets capture semantic anomalies e.g. walking in opposite directions of pedestrians, etc.). The network architecture comprises of various architectural concepts such as CNN, multi-level features, ConvLSTM, attention & transformer, etc. Basically the idea is to design a spatiotemporal feature from the chunk of video frames in an embedding where the features are easily being discriminated to 0/1 (normality/abnormality).