Detecting abnormal events in traffic video surveillance using superorientation optical flow feature

IET Image ProcessingVolume 14, Issue 9 p. 1881-1891 Research ArticleFree Access Detecting abnormal events in traffic video surveillance using superorientation optical flow feature Joshan Athanesious, Corresponding Author joathenz@gmail.com Department of Electronics Engineering, Madras Institute of Technology, Chromepetn, Chennai, IndiaSearch for more papers by this authorVasuhi Srinivasan, Department of Electronics Engineering, Madras Institute of Technology, Chromepetn, Chennai, IndiaSearch for more papers by this authorVaidehi Vijayakumar, Mother Teresa Woman's University, Kodaikanal, IndiaSearch for more papers by this authorShiny Christobel, Department of Electronics and Communication Engineering, Sri Ramakrishna Institute of Technology, Coimbatore, IndiaSearch for more papers by this authorSibi Chakkaravarthy Sethuraman, School of Computer Science and Engineering, Vellore Institute of Technology – Andhra Pradesh (VIT-AP) University, Amaravati, IndiaSearch for more papers by this author Joshan Athanesious, Corresponding Author joathenz@gmail.com Department of Electronics Engineering, Madras Institute of Technology, Chromepetn, Chennai, IndiaSearch for more papers by this authorVasuhi Srinivasan, Department of Electronics Engineering, Madras Institute of Technology, Chromepetn, Chennai, IndiaSearch for more papers by this authorVaidehi Vijayakumar, Mother Teresa Woman's University, Kodaikanal, IndiaSearch for more papers by this authorShiny Christobel, Department of Electronics and Communication Engineering, Sri Ramakrishna Institute of Technology, Coimbatore, IndiaSearch for more papers by this authorSibi Chakkaravarthy Sethuraman, School of Computer Science and Engineering, Vellore Institute of Technology – Andhra Pradesh (VIT-AP) University, Amaravati, IndiaSearch for more papers by this author First published: 08 June 2020 https://doi.org/10.1049/iet-ipr.2019.0549Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Detection of abnormal events in the traffic scene is very challenging and is a significant problem in video surveillance. The authors proposed a novel scheme called super orientation optical flow (SOOF)-based clustering for identifying the abnormal activities. The key idea behind the proposed SOOF features is to efficiently reproduce the motion information of a moving vehicle with respect to superorientation motion descriptor within the sequence of the frame. Here, the authors adopt the mean absolute temporal difference to identify the anomalies by motion block (MB) selection and localisation. SOOF features obtained from MB are used as motion descriptor for both normal and abnormal events. Simple and efficient K-means clustering is used to study the normal motion flow during the training. The abnormal events are identified using the nearest-neighbour searching technique in the testing phase. The experimental outcome shows that the proposed work is effectively detecting anomalies and found to give results better than the state-of-the-art techniques. 1 Introduction Recently, machine learning based on abnormal detection has become an interesting task in video analysis. This is appropriate in traffic video surveillance (TVS), which contains a busy traffic scene that includes different motion patterns occurring frequently or occasionally in the successive time frame. It is looked for to investigate the motion pattern and extract some form of high-level understanding [1]. Fig. 1 shows the example of vehicle motion pattern for a traffic scene. Determining such motion features will directly mine the scene to enable the task of scene analysis and detection of abnormal events such as a person walking in the active driving region, illegal U turn, vehicle deviating from the normal pattern. Identification of rare abnormal pattern in enormous traffic volume is very challenging and hence there is a need for an automatic anomaly detection system [2]. Fig. 1Open in figure viewerPowerPoint Examples of two traffic scenes with their motion patterns (a) City, (b) Highway Naturally, abnormal detection [3] is a subdomain of behaviour understanding in TVS. Abnormal detection methods will be learned from a normal pattern via training and anything that diverges from the normal behaviour is termed as anomaly. It is fundamentally done by applying some specific learning techniques on the extracted features. The primary data in video surveillance is a raw video (sequence of frames) from which relevant features are extracted and becomes inputs for abnormal detection [4]. The proposed work is based on pixel level features combined with the pattern-learning approach. Several methodologies for anomaly detection in TVS [3] are mainly focusing on foreground modelling [5], adaptive learning [6], classification [7] and classifier [8], which have limitations such as effective clustering centre, imbalance data (normal and abnormal events) and different types of objects in the traffic scene. In this paper, super orientation optical flow (SOOF) features are proposed to detect anomalies in video traffic data. In addition, the motion block (MB) is selected and localised using mean absolute temporal difference for computing the SOOF features in the learning phase. We also used the K-means clustering to group the normal data events and detect the anomaly using the NN search in the testing phase. The proposed work is experimented in video traffic scene such as highway road, traffic junctions to investigate the vehicle behaviour and to identify the anomalies. From this study, it is confirmed that the efficacy of proposed SOOF-based anomaly detection is better than the existing works. The main contributions of the paper are as follows: Mean absolute temporal difference is used to locate the motion region to extract the optical flow features to represent the motion orientation of the vehicle in the video. Also, the temporal information along with orientation is used in the feature extraction process which helps to estimate the motion vectors more accurately. SOOF features are proposed by calculated superorientation block using confidence score and range weight factors. The cluster labelling method is proposed to improve the clustering results. The rest of the paper is organised as follows: related work for several anomaly detection methods is briefly reviewed in Section 2. In Section 3, the proposed SOOF feature-based clustering is described. In Section 4, the experimental results on real-time traffic video datasets are analysed. In Section 5, the main contribution of the proposed work is summarised and the idea for future research is presented. 2 Literature survey In this section, various methodologies for detecting abnormal events in TVS are discussed. Table 1 describes some of the feature-based learning approaches for anomaly detection. Table 1. Abnormal event detection categorisation based on feature-based learning approach. S. no Methodology Scene Kind of anomalies Datasets Ref. 1 velocity distribution and cascade classifier traffic junction, highway road, vehicle parking person walking in active driving region, vehicle stopped in non-parking area I-LIDS [8] 2 OFFPM traffic junction, roundabout vehicle deviating from normal pattern AVSS-2007 [9] 3 density-based clustering technique traffic junction, highway road accident detection, wrong side driving Youtube datasets [5] 4 hierarchical motion pattern mining highway road, traffic junction vehicle moving along the crosswalk, changing lane when it is waiting in stop line NGSIM, MIT [10] 5 lossy count-based sequential temporal pattern pedestrian crossing, junction road vehicle in wrong direction AVSS - 2007 [11] 6 sparse optical flow highway road person walking in active driving region, wrong side driving UCSDPed1 [12] 7 histogram of optical flow orientation with SVM crowd behaviour abnormal walking pattern UMN, PETS2009 [7] 8 co-occurring matrix and Markov random field highway road illegal U-turn, person dropping a bag PETS2009 [13] 9 GMM and LDA traffic junction, highway road illegal U turn, vehicle in the opposite direction MIT, AVSS-2007 [6] 10 statistics with heuristic approach city road, traffic junction, lobby entrance illegal U turn, unusually staying long ARENA, CAVIAR, MIT [14] 11 convolution neural network (CNN), long short-term memory (LSTM) college roads, subways vehicle in pedestrian pathway, loitering UCSD, venue [15] 12 spatio-temporal adversarial networks (STAN) highways, college roads vehicle in pedestrian pathway UCSD Ped1, Ped2, avenue [16] Cui et al. [8] proposed an abnormal detection system using the local feature extraction. In their technique, foreground object in the traffic scene is detected and morphological operations are performed and the velocity of the object is estimated for object classification. Finally, cascade classifier is used for identifying the anomalies. Since local features are efficient in identifying the abnormal vehicles, overlapping between vehicles may still affect the detected region's shape and noise. Deepika et al. [9] proposed anomaly detection framework using optical flow-based frequent pattern mining (OFFPM). In addition, with spatial and direction features, temporal features are also determined from the video sequence. Frequent pattern mining is used to find the regular patterns and the patterns that diverge from the already trained pattern are detected as anomalies. The structure of roads is varying for different scene and it may unable to adopt and learn the patterns effectively. Ranjith et al. [5] utilised the density-based clustering technique which uses spatio-temporal features for detecting anomaly events in TVS. In this method, parameters such as epsilon (eps) and a minimum number of points (minpts) play a vital role for efficient abnormal detection. Clustering accuracy differs for varying datasets which in turn reduces its effectiveness. Song et al. [10] proposed the hierarchical motion pattern mining for classifying the vehicle motion as normal or abnormal. Here the optical flow technique is used for detecting the motion of the vehicle. Two-level motion pattern mining approach is proposed for effective anomaly detection. In Level 1, latent Dirichlet allocation (LDA) modelled activities are represented as visual words based on spatial location, direction and moving speed. In Level 2, traffic states are discovered based on the activity distribution of vehicles calculated by Level 1. This method lacks the ability to make correction while splitting and merging of clusters. Kumar and Vaidehi [11] proposed a lossy count-based sequential temporal pattern (LC-STP) mining approach which uses spatial and temporal symbols for location and time information. Spatio-temporal information along the direction of moving objects is determined using optical flow techniques and the normal events are examined to form a temporal pattern and events that deviate from these patterns are identified as anomalies. However, the foreground estimation-based abnormal detection may suffer from high computation. Hanlin et al. [12] proposed anomaly detection system using the sparse optical flow. In their technique, the efficiency of optical flow computation is improved by using foreground masking. Since the sparse features unable to learn the unexpected features efficiently, the authors extract the vehicle motion by obtaining aggregated histogram of optical flow. Tian and Snoussi [7] utilised the histogram of optical flow orientation as an image descriptor along with non-linear classification method for detecting anomalies in a traffic video scene. The non-linear one-class support vector machine (SVM) is used to study the normal pattern from the training video and detect abnormal events in the testing frame. Although SVM performs well, still there is a difficulty in identifying the normal and abnormal events. Yannick et al. [13] used low-level features for event modelling and abnormal event detection. In the first phase, the co-occurrence matrix is obtained using the spatio-temporal attributes of normal events. In the second phase, these matrices are used by Markov random field to examine the activity of upcoming data. Hawook et al. [6] proposed an LDA with Gaussion mixture model (GMM) for analysing the traffic pattern for detecting anomalies in the traffic scene. The corner points of the foreground objects are estimated and tracked using Kanade–Lucas–Tomasi. LDA is used to analyse the frequent occurrence of each vehicle which is taken as a feature to train the model. In addition, hidden Markov mode is used to find abnormal activities by using the temporal transition of the scene state. Luis and Ferryman [14] proposed statistical method with a heuristic approach for detecting anomalies in the video. This approach is mainly focused on context modelling approach to govern the activity zones trajectory. However, these statistical approaches [6, 13, 14] are more suitable for identifying the abnormalities and they have not considered the motion estimation by interrelations among the nearer objects. Sangmin et al. [16] proposed the spatio-temporal discriminator to determine whether the given video sequences are in normal or abnormal. The network is trained in an adversarial way by effectively encoding the spatio-temporal features of normal data. The deviation from the trained sequence is identified as abnormal using the GAN detectors. While it seems to be effective, learning based on ConvLSTM is computationally expensive for a large number of temporal contextual information. Hasan et al. [17] applied the deep autoencoder to learn the spatio-temporal cuboids using histograms of oriented gradients and histograms of optical flows. Further, reconstruction error is calculated to identify the abnormalities in traffic video. Likewise, Hajimolahoseini et al. [18] proposed the dynamic textures along with spatial anomaly to learn the normal pattern of the video. Although their methods [17, 18] are effective, they are not developed with temporal features and not useful to the video. In summary, the key challenges identified from the above researches are: Detecting abnormal events using foreground detection is computationally high and also degrades the performance in terms of accuracy [11]. Vehicle with varying speed produces a different set of points [5]. Traffic scene contains all kinds of objects such as vehicles, humans, animals etc., which is very difficult to track based on object classification [19]. Labelling of normal and abnormal data in a traffic scene is very difficult [7]. Often contains noise which increases the false alarm rate. In this paper, a novel SOOF feature is proposed to efficiently learn the motion flow of vehicles in the traffic scene by reducing the computation and also omitting the background modelling algorithms. The proposed SOOF, which represents the motion flow, helps to track the vehicles irrespective of the speed and motion. Moreover, we jointly consider the motion flow vector and interrelations among objects to compute the presence of suspicious activities in the traffic scene. The proposed SOOF feature is estimated based on centre orientation block using a confidence score and range weight factor. In addition, MB is obtained using the mean absolute difference method which helps to define the motion region. 3 Proposed methodology In this section, a novel scheme is proposed to represent the motion information of both normal and abnormal events by means of motion block-based optical flow features for detecting anomalies in the traffic scene. The proposed work is divided into training and testing phases. The training phase is divided into sub-tasks mainly: optical flow computation, estimating the SOOF features, clustering using K-means. In the testing phase, optical flow is computed for the test video (not used for training) and the abnormal events are identified using the nearest distance measure. The overall scheme for detecting anomalies using the proposed SOOF model is shown in Fig. 2. The optical flow technique is used to represent the motion characteristics for each pixel, and motion block formulation and selection is obtained using the mean absolute temporal difference method for the given training set. The spatio-temporal characteristics of the video being used to represent the SOOF feature. In addition, the MB selection for each frame using mean absolute temporal difference shows the effectiveness of the proposed scheme. Based on the MB information, motion descriptor is clustered using the unsupervised K-means algorithm. During the testing phase, optical flow is estimated for test data and anomalies are detected using the nearest distance measure. Fig. 2Open in figure viewerPowerPoint Overall framework for proposed SOOF features clustering for detecting anomaly in TVS The movement of objects in the traffic scenario can be considered by orientation and spatio-temporal features. So, the optical flow [20] technique is used to provide information on both spatial and temporal values. In this work, Gunner Farneback's optical flow methodology is used to describe the motion characteristics which work on polynomial expansion of two-frame motion approximation. Furthermore, the optical flow is defined as the apparent motion of the individual pixel and gives a good approximation on true physical motion projected onto the image plane. The estimation of orientation tensors gives a powerful representation of the motion which can be inferred from the intensity variations. These orientations are well suited for estimating motion models over larger regions [21]. Also, the motion of vehicles in vast area like traffic scene is very less and finds difficulties in predicting the displacement between the successive frames. So, the flow vector is computed between the first and the consecutive frames to obtain the motion more efficiently. 3.1 Block division and MB localisation In spite of potential growth in anomaly detection, there is still a need for an effective method to detect the anomaly in the traffic scene. It is found that the motion vector estimated from the optical flow at the pixel level is not constant in wide video traffic environment. So, the MB level optical flow is considered, which is estimated from the pixel level. Mean absolute temporal difference is used to acquire the block-wise optical flow vector which contains both spatial and temporal information. Input frame in the video is divided into a number of non-overlapping blocks which are represented as . The mean absolute temporal difference (MAT) to find motion block is given by (1)where is the pixel value at location (m, n) of block in the previous frame, is the pixel value at location (m, n) of block in the current frame and m n is the dimension of the MB. Fig. 3 shows the block division and MB localisation for each frame using the mean absolute temporal difference. Fig. 3a illustrates the non-overlapping division for a frame and the shaded region in Fig. 3b shows the non-motion blocks which will be discarded during training and testing: (2)where n = 1, 2, 3, …., the number of blocks in each frame and is the mean orientation threshold. Fig. 3Open in figure viewerPowerPoint Block division and MB localisation using the mean absolute temporal difference (a) non-overlapping division for aframe, (b)Non-motion blocks If , then the MB is selected for training or testing, else the block is discarded. The orientational features are estimated for each selected MB. Fig. 4 shows the temporal difference of each MB region for a frame using the MAT difference method. The motion blocks are identified in a frame and these blocks are used to compute the SOOF features. Fig. 4Open in figure viewerPowerPoint Illustration of the mean absolute temporal difference value for each block 3.2 SOOF features The movement of objects in road can be identified based on various factors such as nearby object movements, traffic flow and obstacles along the path. These interrelation characteristics are used to create novel SOOF features to localise the abnormal event activities. The blocks with motion are influenced by three factors such as object direction, speed and spatio-temporal attributes. With the impact of moving objects in each block, SOOF features are obtained using the following steps. Step 1: Cluster size estimation: Once the motion block is separated from the frame, the optical flow for centre of each MB is found. A flow vector with an angle is in the range of . Fig. 5 shows the cluster labelling based on orientation features. Fig. 5a represents the four ranges of orientation. Fig. 5b illustrates the graphical representation for estimating the cluster size in a traffic scenario. Feature vectors are denoted as a matrix where rows correspond to the orientation and spatio-temporal features. Fig. 5Open in figure viewerPowerPoint Cluster size estimation (a) Illustrates the four ranges of orientation , (b) Optical flow direction for eachcentre in the block for a matrix (8 × 4 × 2), (c) Estimating cluster size For example, 120° and (185, 20) are the orientation and spatio-temporal features, respectively. The orientation angle 120° is in range of . For the traffic scenario in Fig. 5c, the orientation value obtained is in between the range of and . So, the cluster size is defined as 2, which is shown in blue and green colour arrows. Step 2: Calculating confidence score: The orientation difference for centre of MB with the neighbouring centre is obtained by (3) where represents the centres of current MB and represents the eight neighbouring MB centres of , represents the angular distance between the current and neighbouring centres (signed difference), = angle size of the MB and N(.) is the eight neighbouring centres of the current MB . Range weight factor for block i to set block j is calculated as follows: (4)After computing the range weight of all the blocks, the matrix representing the motion vector that arises within the frame is finally constructed. Each element of the matrix signifies the SOOF feature of the block . is the centre of the quantitative motion vector of MB, whereas is the centre of the motion vector of neighbouring blocks of . Here, eight neighbouring are taken into consideration for each and is shown in Fig. 6a. Fig. 6Open in figure viewerPowerPoint Illustration of constructing SOOF features for different traffic scene (a) Illustration to calculate the confidence score for each centreof MB. is compared with eightneighbouring , (b) A moving car (normal), (c) Pedestrian in driving region(abnormal) Clustering is a separation of data into groups (cluster) of similar objects. Each cluster consists of objects that are similar between themselves and dissimilar to objects of other groups. K-means [3] is one such type of unsupervised clustering, which is based on the partitioning and relocation of cluster centroid. The algorithm for the proposed SOOF feature-based clustering is given in Algorithm 1 (Fig. 7). Fig. 7Open in figure viewerPowerPoint Algorithm 1: SOOF feature-based anomaly detection The vehicles are moving in a structured pattern and there will not be much change in the motion flow in each respective block. So, the centre pixel of the block is taken to compute the motion descriptor. Then the range weight of current MB is calculated by summing the motion impact of neighbouring eight MB. Using these characteristics of the proposed SOOF feature-based clustering, the abnormal activities are detected and localised in the testing phase. The MB in which abnormal activities are happened has a different motion vector compared to neighbouring MB. It is also noted that proposed SOOF features are obtained by concurrently considering the object direction, interrelations among nearby objects and spatio-temporal features. The motion flow of rigid objects such as car, lorry is relatively stable and follows the structured patterns in terms of orientation and magnitude when compared to non-rigid objects. Non-rigid objects have complicated motion directions. Additionally, the blocks occupied by the rigid objects are more than the non-rigid objects when computing the range weight which results in high values of SOOF features. For example, the presence of the pedestrian in the driving region provides a smaller sum of range weight than the other rigid objects in the traffic scene. Figs. 6b and c show the constructed SOOF features for various traffic scenarios. These rigid objects (Fig. 6b) make a stable motion over time compared to the non-rigid pedestrians (Fig. 6c). The proposed scheme utilised these representations to identify abnormal activity in the present frame. 4 Anomaly detection The significant role of anomaly detection is to find the point in the test data that do not follow regular data pattern. If the incoming patterns are similar to the trained pattern, then it is considered as normal data else it is declared as abnormal data. Detection of events such as driving on the wrong side, a person crossing the road illegally areidentified as anomalies. Nearest-neighbour search is used in this paper between thetest data and cluster label to identify the abnormal pattern in the traffic scene.The steps for the abnormal detection process are presented in Algorithm 2(Fig. 8). Fig. 8Open in figure viewerPowerPoint Algorithm 2: anomaly detection using nearestneighbour The frame is declared as abnormal when the distance between the test data(Test) and the cluster label (L) is greater than the threshold (thres). Thethreshold is chosen based on trial and error. 5 Experimental result and analysis This section presents the experimental outcome of the proposed methodology for detecting anomaly events using the real-life situation [12], AVSS iLIDS dataset [22] and UCF – Crime (road accident) [23] datasets. The model is developed using a python programming language with OpenCV library. 5.1 Computational complexity Let C be a number of centre pixels of each block in a frame, V the optical flow vector computation, MB the number of motion blocks in the frame and S the SOOF feature. The computational time for obtaining SOOF feature-based clustering is O(V + C1*S + L*t), where C1 is the selected MB of each frame, L is the number of clusters (Label) and t is the number of iterations used in K-means clustering. 5.2 Real-life dataset The efficacy of the proposed work is evaluated using the real-time dataset [12] containing the video of length 30 min (6000 frames and size of 960 × 540 pixels). To ease the experimental procedure, the dimension of real-life datasets is resized to 320 × 240 pixels. The dataset contains both normal and abnormal patterns which include both vehicles and pedestrians. The scene (highway) contains 6000 frames in which, 4500 frames are taken for the training phase and remaining for the testing phase. Motion vectors (orientation and direction) are computed between the first frame to the rest of the frame using the optical flow technique. The selection of specific MB is essential and it has an impact on the performance of the proposed work. We have chosen MB size of 12 × 16 for further analysis and experimentation. For motion vectors of each MB, the SOOF features are obtained and clustered using K-means technique described in Section 3.2. The selection of proper MB size is more significant in the discovery of the motion features which is also having a major impact on the detection rate. Therefore, experimentation is made for varying block size and the analysed results are presented in Table 2. It is noticed that 12*16 MB size is attaining good accuracy and it is used for further analysis. Figs. 9a–d show the SOOF feature-based clustering results of the sample frame in a real-life situation. Table 2. Parameter analysis of the proposed SOOF feature-based clustering approach for real-life datasets Traffic scenario Original frame Resized frame No. of rows and columns in block (R*H) MB size Distance threshold for each MB Accuracy Jaccoard coefficient (J%) highway road 960 × 540 320 × 240 05*05 48*64 0.9210 80.34 10*10 24*32 7.4609 86.41 20*20 12*16 14.419 94.65 city road 960 × 540 320 × 240 05*05 48*64 0.856 79.10 10*10 24*32 4.986 84.35 20*20 12*16 12.257 92.04 Bold values show the better performance among all the trails. Fig. 9Open in figure viewerPowerPoint SOOF feature based clustering (a) Optical flow computation, (b) MB localisation, (c) Illustration of range weight foreach block, (d) Clustering representation usingK-means It determines the closeness of the test data with each cluster centroid. In testing, the point which is not within any of the cluster is considered as an abnormal point. In Fig. 9c, the motion vector (direction) attained during the training is in the range of 90–270°). The flow of pedestrian crossing the road is towards the horizontal direction (180°), which is identified as an abnormal event. Figs. 10a–c show the pedestrian using the driving region for crossing the road which is identified as abnormal and it is noticed in frame nos. 5232, 5345 and 5438, respectively. Fig. 10a