ANOMALY DETECTION WITH FRAME-GROUP ATTENTION IN SURVEILLANCE VIDEOS

1 INTRODUCTION . Nowadays anomaly detection is useful to maintain social security and conduct legal forensics . Due to the ambiguous definition of abnormal events , it increases the difficulty of detection . For example , the appearance of a vehicle on a road is normal , while it is abnormal when a vehicle is on the sidewalk . So we make the goal clear in order to carry out consequent measures . Abnormal behavior is defined as rapid movements in slow moving crowds , such as cycling , running , throwing from a height , etc . Based on this definition , the commonly used datasets for anomaly detection are integrated and relabeled . The new fusion dataset contains more scenes and anomaly types , so it is more challenging for the anomaly detection . Subsequent experiments are performed on this new dataset . The test results show that the algorithm proposed in this paper has great advantages in many metrics . The end-to-end anomaly detection network shown in Fig . 1 has the following contributions : ( 1 ) The video group composed of consecutive multiple frames is the basic processing unit to extract expressive features with the designed group feature extractor . On the one hand , the spatial-temporal information could be retained comparing with single-image processing . On the other hand , the abnormality score of a single frame can be easily obtained without the access of the whole video , which can copy with the situation where video streams are the input . ( 2 ) The designed framework composed of group feature extractor and group score mapper can effectively obtain the abnormality score using the spatial-temporal information . Implicit vector-based attention mechanism is used to weight the frame-group features . The more important the feature is , the higher the weight is . ( 3 ) The basic cross-entropy loss and the improved hinge loss are united to improve the performance of the network . The latter devote to make the score of the abnormal frames greater than that of normal frames . The paper is organized as follows . Section 1 introduces the background of anomaly detection . Section 2 introduces the related work on anomaly detection . Section 3 mainly introduces the details of the proposed anomaly detection algorithm . Section 4 introduces the fusion dataset and then analyzes the experimental results . Section 5 gives the summary of the whole paper . 2 RELATED WORK . The challenges of video semantic analysis lie in the extraction and representation of video features . The video contains complex spatial texture information and time information . Multidimensional data provides more information but meanwhile it contains a lot of redundant information . How to extract low-redundant , comprehensive and representative video features is one of the research focuses . The manual extraction method of video features focuses on the extraction and analysis of low-level visual features , such as guided gradient histograms Xiao et al . ( 2014 ) , optical flow maps Reddy et al . ( 2011 ) , spatio-temporal points of interest Dollár et al . ( 2005 ) , texture models Xiao et al . ( 2018 ) , filtering models Zhang et al . ( 2018 ) , etc . After obtaining the statistical information , the visual dictionary Roshtkhari & Levine ( 2013 ) and other methods will be used to save the normal distribution , and then calculate the similarity criterion to determine whether the target is abnormal . Luo & Wang ( 2019 ) explores the method of multi-stream manual features for video representation . It constructs a three-dimensional video representation structure composed of spatio-temporal vector and positional vector , and improves the encoding method to make the extracted video representation structure more powerful in representation . With the rapid development of deep learning , automatic feature extraction using neural networks has become a research hotspot . Zhou et al . ( 2016 ) uses 3D convolutional networks to detect abnormal events in surveillance video . In Sabokrou et al . ( 2017 ) , the video frame is divided into several small areas , and each small area is feed into a 3D self-encoding network combined with 3D convolutional neural network to extract features and detect anomaly . Medel ( 2016 ) proposes a long-short term memory network based on convolution , which simultaneously extracts spatial and temporal information . Xu et al . ( 2015 ) first uses stacked auto-encoders to learn and fuse the appearance and motion characteristics of abnormal individuals , and then trains multiple single-classifiers to calculate the abnormal score . Ionescu et al . ( 2019 ) uses one-to-many classifiers instead of the single-classifiers after obtaining multiple pseudo-anomaly classes from the trained normal behavior pattern . Hinami et al . ( 2017 ) multiple attributes of the same target to extract features . Due to the lack of anomalous videos and various types of anomaly , it is difficult to find a general model that covers all anomalous events . The auto-encoding network Yuan et al . ( 2019 ) performs anomaly detection based on the reconstruction error . Hasan et al . ( 2016 ) uses a convolutional neural network to implement video anomaly detection . Since the convolutional layer operates on a twodimensional structure , time information will be lost . Chong & Tay ( 2017 ) designs a spatio-temporal autoencoder that encodes video sequence with spatial convolution and Convolutional Long-Short Term Memory ( ConvLSTM ) Shi et al . ( 2015 ) structure , and then uses the symmetric structure called the decoder , convert the video encoding into the image sequence . The abnormal score can be obtained from the calculation of the Euclidean distance between the decoded images and the original images . An & Cho ( 2015 ) proposes a variational autoencoder ( Variational Autoencoder , VAE ) , which uses the results of video encoding to fit a distribution function . Jing & Yujin ( 2017 ) supplements a gradient difference constraint on the basis of the sparse denoising auto-encoding network , which is helpful to make the model more effective in detecting abnormal behavior . For specific tasks , the model trained according to the specific task can often get better results . In Sultani et al . ( 2018 ) , the video containing abnormal events is divided into several video segments to form multiple video instances . A fully connected network is designed to map the video features extracted by the C3D network Tran et al . ( 2015 ) into abnormal scores . The score of the abnormal instance is higher than the score of the instances with only normal frames . The experiments in this paper show that the network has achieved good results . But the detection process is disconnected . In order to minimize the interference to the input data and obtain the final score directly and timely , we designs an end-to-end network , which uses both positive and negative samples to make the model more targeted . 3 PROPOSED METHOD . The paper proposes an end-to-end anomaly detection network , which operates on the frame-group formed with consecutive frames to obtain the abnormality score . The whole framework is composed of group feature extractor and group score mapper . The former performs on the raw frame-group to obtain the spatial-temporal group-feature . The latter performs on the group-feature to obtain the abnormality score . 3.1 THE GROUP-FEATURE EXTRACTOR . The frame-group is defined as a structure consisting of consecutive τ frames , which contains rich spatial texture information and temporal change information . Fig . 2 shows the details of the groupfeature extractor , where In ( 0 , Spa , Tem ) t and Out ( 0 , Spa , Tem ) t represent the input and output matrixes in time t respectively . Spa and Tem mean the input and output feature map in the spatial feature extractor and the temporal feature extractor respectively . Since the frame-group contains multiple continuous frames , in order not to destroy the time information within consecutive frames , we use a trainable convolution kernel to extract spatial information of each single-frame in the time dimension . Then batch normalization Ioffe & Szegedy ( 2015 ) is used to prevent the gradient dispersion and accelerate the model convergence . Activation function helps to make the model nonlinear . Two sets of ” convolution-regular-activation ” structures are used in the experiments . The first group uses 128 convolution kernels with size of 5× 5 and step size of 3 , and the second one uses 64 convolution kernels with size of 3× 3 and step size of 2 . After obtaining the spatial feature maps , ConvLSTM will further extract the spatial and temporal information using gate structures . The implementation of hidden features is shown in Equation ( 1 ) . Where the hidden feature HidTemt is used to record the accumulated state information up to time t. The symbol ⊗ represents the Hadamard product . σ and tanh represents the sigmoid and tanh nonlinear activation functions respectively . Conv is the convolution operation . This paper uses two cascading ConvLSTM layers to extract time flow information . The first layer uses 64 convolution kernels with size of 3 × 3 and the second one uses 32 same kernels . We use the ” same padding ” to keep the size of input and output feature maps same . HidTemt = σ ( Conv ( Out Tem t−1 , In Tem t , Hid Tem t−1 ) ) ⊗ HidTemt−1 + σ ( Conv ( OutTemt−1 , In Tem t , Hid Tem t−1 ) ⊗ tanh ( Conv ( Outt−1 , In Tem t ) ) ( 1 ) While the final output is shown in Equation ( 2 ) : OutTemt = σ ( Conv ( Out Tem t−1 , In Tem t , Hid Tem t ) ) ⊗ tanh ( HidTemt ) ( 2 ) In order to better extract the spatial and temporal information as well as high-level and low-level features of frame-group , we use a multi-level feature fusion structure to merge multi-level features of the frame-group . The output result of the multi-level feature fusion structure is regarded as the final feature representation of the frame-group . In the experiment , the structure was implemented using a 3D convolution Zhou et al . ( 2018 ) layer with size of 8× 1× 1 . 3.2 THE GROUP-SCORE MAPPER . In the group-score mapper , the attention mechanism Ilse et al . ( 2018 ) is used to increase the decisive influence of useful features and weaken the effects of irrelevant features on the results . The fully connected network is used to make the encoding of the group-feature more expressive while reducing feature dimensions . The group-level pooling is applied to map the refined group-feature to the abnormality score of the video group . The specific process is shown in Fig . 3 . The implicit vector method based attention mechanism assigns different weights to different features in order that the key features have a more important impact on the result and the interference of noise can be suppressed . The trainable transformation matrix will be used to project the original group feature into the implicit space , and then the weight vector will be obtained by an inverse transformation matrix . Different from the attention mechanism aiming at multiple instances in Ilse et al . ( 2018 ) , this paper focus on the attention of the group-feature . To be specific , the implicit vector is used to generate a weight vector of the original group , which has a dimension of 128 in the experiment . Fgrp is the flattened Out0t with length =1444 . T represents transposition operation . V and W are the feature space transformation matrices . Ψnl is a non-linear transformation function . Therefore we can define the coefficient χk of the kth element F ( k ) grp as Equation ( 3 ) . χk = exp ( ( WTΨnl ( VF T grp ) ) k ) ∑K i=1 exp ( ( W TΨnl ( VF T grp ) ) i ) , k ∈ [ 1 , K ] ( 3 ) The weighted group feature F̃grp is as Equation ( 4 ) . F̃grp = ( χ1F ( 1 ) grp , · · · , χkF ( k ) grp ) , k ∈ [ 1 , K ] ( 4 ) The weighted group-feature next passes through two fully connected layers with Dropout operation to reduce the feature dimensions and the computation and enhance the feature expressive ability . The output dimensions of the two fully connected layers are 512 , and theRelu activation function is used . The Dropout parameter is set to 0.5 , that is , 50 % of the neural units are discarded randomly and not participated in each iteration of training , which can reduce the risk of neural network overfitting . Group-pooling is used to get the final group score . The trainable weight matrix followed by the sigmoid function is used to map the refined group feature to the abnormality score . The positive samples marked as 1 are the frame-groups containing anomalies , while negative samples contain only normal . As the trainable weight matrix of the refined group feature F̃grp is marked Φ and the sigmoid is marked σ the predicted group-score B̂ ( t ) corresponding to the input ( In0t , · · · , In 0 t+τ ) is defined as Equation ( 5 ) . B̂ ( t ) = σ ( F̃grp ·Φ ) ( 5 )

Summary: The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds. The group-feature extractor is adopted. The proposed algorithm is tested on the integrated dataset.