Classification of human activity detection based on an intelligent regression model in video sequences

IET Image ProcessingVolume 15, Issue 1 p. 65-76 ORIGINAL RESEARCH PAPEROpen Access Classification of human activity detection based on an intelligent regression model in video sequences Natarajan Kumaran, Corresponding Author kmshre19@gmail.com Research Scholar, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu, India Correspondence Natarajan Kumaran, Research Scholar, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu 620015, India. Email: kmshre19@gmail.comSearch for more papers by this authorUyyala Srinivasulu Reddy, Assistant Professor, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu, IndiaSearch for more papers by this author Natarajan Kumaran, Corresponding Author kmshre19@gmail.com Research Scholar, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu, India Correspondence Natarajan Kumaran, Research Scholar, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu 620015, India. Email: kmshre19@gmail.comSearch for more papers by this authorUyyala Srinivasulu Reddy, Assistant Professor, Department of Computer Applications, National Institute of Technology, Trichy, Tamil Nadu, IndiaSearch for more papers by this author First published: 03 December 2020 https://doi.org/10.1049/ipr2.12006AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract The most critical objective in security surveillance is abnormal event detection in public scenarios. A scheme is presented for detecting abnormal behaviours in the activities of human groups based on social behaviour analysis. This approach efficiently models group activities than some of the previous strategies that use independent local features. This paper presents a feature descriptor method to signify the movement by implementing the optical flow through covariance matrix coding. The multi-RoI (region of interest) covariance matrix has some frames or patches which could represent the movement in high accuracy. Normal samples are plentiful in public surveillance videos, while there are only a few abnormal samples. For that, the model of a hybridised optical flow covariance matrix is represented in this paper. Optical flow (OF) in the temporal domain is measured as a critical feature of video streams. The logistic regression method is used to detect abnormal activities in a crowded scene. Finally, the behaviours of human crowds can be predicted using benchmark datasets such as UMN, UCSD as well as BEHAVE. The obtained experimental results show that the proposed approach can effectively detect abnormal events from the abandoned environment of surveillance videos. 1 INTRODUCTION Human action recognition has drawn an expanding consideration, and its applications are relevant in the extreme mechanism of video surveillance and security, video annotation and recovery, behavioural biometrics and human–computer connection, and so forth [1]. Vision-based human activity recognition is the way towards perceiving human activities in video sequences by using computer vision strategies. The term activity refers to basic movement patterns normally executed by solitary individuals and ordinarily extend for a small duration of time [2]. The objective of action recognition is to investigate progressing actions from an unknown video, consequently [3]. Accordingly, one of an adequate obstacle admired among human action recognition originate from tremendous conflicts such as great intra-class difference, scaling, impediment, and clutter. Video-based human activity detection has numerous applications in human-computer collaboration, observation, video ordering, and recovery. Activities or movements create some of the patterns which are fluctuating in terms of spatio-temporal forms in recordings, which could be utilised as feature descriptors for the detection of actions [4, 5]. In the earlier developments, action recognition has included utilising video successions caught by camcorders. Sequentially spatio-temporal highlights are broadly utilised for perceiving human actions and contrasted with some of the images which are conventional. In lighting conditions, the depth maps are very insensitive to changes and can give 3D data for recognising activities that are hard to portray the utilisation of traditional images [6, 7]. Space-time based strategies, for example, the highlights of local spatio-temporal are famous systems for video representation also demonstrates promising execution in action recognition. The intensity-based video images, which restrict the intensity of detecting an action [8]. Convolutional Neural Networks (CNNs) is a sort of deep models in which trainable channels and nearby neighbourhood pooling tasks are connected alternatingly on the raw input images, bringing about a chain of command of progressively complex highlights [9]. When prepared with suitable regularisation, CNNs can accomplish superior execution on visual object recognition tasks. Furthermore, CNNs have been appeared to be invariant to specific varieties, for example, posture, lighting, and encompassing clutter. As a class of deep models for highlight development, CNNs have been applied to 2D pictures [10, 11]. In the most recent decade, the progressions in computational capacities and the introduction of powerful methodologies with preparing deep neural system designs need to prompt their wide use to address different computer vision challenges [12]. The convolutional kernels mutually in space and time demonstrate with the 3D CNN because of a direct expansion of the well-known 2D CNN profound structures to the 3D spatio-temporal domain [13]. The techniques go for adapting long-extended movement by taking in a chain of command comprising of different layers of 3D spatio-temporal convolution parts by early fusion, late fusion, or slow fusion. The CNN architecture of two-stream functions acquires movement patterns utilising an extra CNN, and input given is the optical flow, which has been obtained from the progressive frames of video sequence [14]. In order to detect the abnormalities in the behaviour of the huge body with normal data, it is relevant to enhance the utilisation of normal data rather than the acknowledgment [15]. Automatically detecting abnormal exercises or occasions from long length video sequences is important for intelligent surveillance, behaviour analysis, and security applications. Specifically, abnormal behaviour detection in queued scenes is a problem because of the substantial number of people on foot nearby, the instability of individual appearance, the continuous incomplete impediments that deliver, and the irregular motion pattern of the group [16, 17]. Moreover, there are potential risky exercises in swarmed conditions, for example, crowd panic, stampedes, and accidents, including an extensive number of people, which makes computerised scene investigation in the most need [18]. Among numerous video detection works, identifying anomalous activities in video streams is fundamental significance as it is identified with certain topics in computer vision, which are fascinating, for example, visual saliency, imperative behaviour recognition, and interestingness prediction [19]. Tragically, in the video sequences, the anomalous action recognition is a troublesome test because of the instability of the classifications mutually for normality and abnormality. An illustrative method to address the detection problems of irregularity is to separate local features. After that handle, an exception identification errand where typical exercises in video scenes are displayed, also abnormal occasions are distinguished as fundamentally different as of the model [20]. In a public scene concerning social security, the detection of an abnormal event is important. The public scene could be considered a significant place where common and daily activities often occur. In the public scene, security surveillance concerns are connected to livelihood subjects. The security guards check the surveillance videos, and for them, it is very simple to visual and intellectual fatigue. Hence, it is not possible to perform rapid action towards unpredicted or abnormal events. Because in public scenarios, most monitors are placed for safety purposes, the automatically set alarm system helps the security guard to observe the events which are in an emergency from a large database of videos, which is critical. The optical flow, as well as the consequent partial derivatives, are fused with the help of the covariance matrix descriptor and are projected into a lower range of dimensional feature space. In the temporal domain, optical flow (OF) is measured using the Lucas Kanade method, and it is considered as a critical feature of video streams. The covariance is classified using logistic regression, and it probably patterns the dealings of the optical flow in the ROI. Although, as a linear method, the logistic regression is considered in which the predictions are renovated through logistic function. The abnormal event is detected using logistic regression with the proposed covariance matrix descriptor. Experimentations conducted in the standard datasets confirm that the proposed model does better than the existing methods. This research article designs the model of anomalous activity issues present in the complicated public scenarios, and advanced technologies are presented for the intelligent security surveillance system. They are remaining part of the article prearranged, as listed below. In Section 2, the related works are reviewed. Section 3 explains the proposed method with the subsections mentioned below. In Section 3.1, foreground detection by using the Gaussian mixture model (GMM) in which each pixel is patterned as a mixture of Gaussian. The classification is based on the mean and variance of both foreground and back ground. In Section 3.2, for optical flow estimation, the Lucas kanade approach is proposed, and in Section 3.3 the motion descriptor presents the movement details, Section 3.4 introduces the logistic regression analysis for abnormal behaviour detection with a stochastic gradient descent algorithm for cost optimisation is discussed. In Section 4, the experimental evaluations conducted in the datasets of BEHAVE, UMN, and UCSD are explained. At last, Section 5 concludes with the prospect of future work. 2 RELATED WORKS The humans action recognition has been applied in some domains like human–computer communication, game control, and smart surveillance. Liu et al. (2017) [21] proposed an improved visualisation strategy of skeleton meant for human activity detection based on the view-invariant sequence. At first, based on the sequences, a view-invariant change was made to devastate the impact of skeleton joints view variations from the spatio-temporal areas. Next, the skeletons changed were visualised as a sequence of colour images. Third, a model based on the convolutional neural system for colour images was received to isolate the strong as well as discriminative highlights. Using the choice level fusion of deep features, the final action class scores were produced. Ten CNNs hard determination and the weighted probability fusion was an old strategy that has less flexibility, and the enhancement of images needed to be more improved. Experiments conducted on standard datasets confirmed the durability of the proposed scheme compared to view variations, noisy skeletons, inter, and intra similarities between the skeleton sequences. The experiments, when conducted with the NTU RGB+D dataset, the proposed technique achieves approximately a gain of 10%. The effectiveness of the scheme was shown by this result associated with the modern LSTM-based approaches. In human behaviour subjects, the investigation means to comprehend the subject behaviour over time utilising the motion information. Ijjina et al. (2017) [22] specified an approach for perceiving human action because of RGB-D video based on the motion sequence information utilised deep learning. This technique does not perceive human behaviour, like hand movements and activities of a group. Multi-modal information usage, as well as the ConvNet features noise tolerance, provided the strength also the compliance of this method compared with other detection tasks. A combination of facts among several classes implied that best actions could be accomplished through the fusing facts across models highlighting certain temporal areas. Then the prevailing methods, the proposed approach is faster owing to the un-complicated calculation in figuring out the new representation (which could be parallelised) as well as the implementation is correspondently done for the ConvNet feature extraction. This approach's efficiency was verified on SBU Kinect interaction, MINIA action NATOPS gesture, and Weizmann datasets. Skeletons were usually accessible as a contribution for human action recognition. Hou et al. (2018) [23] depicted a strategy to encode the spatio-temporal data of a skeleton arrangement into colour texture images, referred to as skeleton optical spectra. It utilised a Convolutional Neural Networks (ConvNets) to attain the discriminative highlights for activity recognition. Such range portrayal made it conceivable to utilise a standard ConvNets design to take inappropriate powerful highlights from skeleton arrangements without preparing a huge number of parameters. In this technique, the ConvNets based end to end encoding of skeleton sequence was not available. To increase the accuracy rate, the Late score fusion was implemented, which involves the three orthogonal planes which were complementary to one another. On three standard used datasets, the results were obtained. In intelligent video surveillance, anomalous action recognition was also a challenging research concern. Sun et al. (2017) [24] presented Deep One-Class (DOC) by representing an end-to-end model that synchronised with the Convolutional Neural Network (CNN) using the one-class Support Vector Machine (SVM). The proposed model, related to the hierarchical models, finds the optimal global solution even though it simplifies the complication involved in the entire process. Approving DOC showed an easily accessible dataset, and this model has incredible execution; also, it is powerful for anomalous action recognition from surveillance videos. The video anomaly detection system should have been enhanced to acquire greater outcomes. Separating discriminative and powerful highlights in action recognition of humans from the video sequence is the first and most basic advance. Liu et al. (2016) [25] determined a genetic programming (GP) strategy; this strategy enhances the motion feature descriptor on a population of crude 3D operators. The optimal arrangement chosen by GP was considered as the ideal action descriptor which has been attained. Feature learning was influenced by the proposed method, which lets a computer to routinely collect the holistic feature extraction using a pool of primitive operators, and these operators were formulated based on the basic idea of feature extraction. As an optimisation concern, the feature extraction done based on the GP approach was reviewed on four standard action datasets, specifically KTH, HMDB51, UCF YouTube, and Hollywood2. In all four datasets, the Experimental results showed that the proposed feature learning method based on GP accomplished improved performance in detection associated with modern hand-crafted and machine-learned methods. 3 THE PROPOSED METHOD Herein the article intends a new anomaly detection technique for congested ones with a covariance matrix description model for Local and Global Abnormal Detection. In Figure 1, the plan of the proposed approach is encapsulated. Initially using the Gaussian Mixture Model algorithm (GMM), the extraction of the foreground objects is done. FIGURE 1Open in figure viewerPowerPoint Overview of proposed method To model the distribution of image intensities, the GMM is used per image location, then the foreground pixels in one locality were categorised into a region of interest (ROI). Then the optical flow is measured using Lucas–Kanade method as this method is easy to compare to another method, very fast calculation with accurate time derivatives. Then, a feature descriptor is proposed to perform as per the movement based on the usage of covariance matrix coding and equivalent partial derivatives of several frames in a connective state, otherwise the patches of the frames. The movements in a high range of accuracy is represented by the covariance matrix of multi-RoI (region of interest). The covariance matrix descriptor is combined with the optical flow and equivalent partial derivatives, then prompted into feature space of lesser dimension. The issues of common binary classification varied for the abnormal event detection, which contains some samples which are normal besides very less or null abnormal samples used for training. Thus, instead of using other classification approaches, the logistic regression method is used in this paper. 3.1 Foreground detection by using Gaussian mixture model The procedure involved in Foreground detection is extracting the object in motion (called foreground object) from the information, which is in static form termed background. In multistage computer vision systems, it remains the backbone and as a principal pace in various computer vision fields, namely video surveillance systems. So, the overall system performance is based on the result of foreground detection. GMM, among several algorithms, is one of the well-known ones because of its adaptive nature, good accuracy as well as less computational cost [26]. Each pixel in GMM [27] is modelled as a Gaussian mixture, then based on the mean and variance, it is categorised as the foreground or background. For identifying a specific pixel value, the probability is P ( X t ) = ∑ i = 1 K ω i , t η X t ; μ i , t , σ 2 i , t (1) From (1) the probability density function η is represented as: η ( X t ; μ i , t , σ 2 i , t ) = 1 σ i , t 2 π e − x − μ i , t 2 σ i , t 2 (2)in which, K, ω i , t, σ i , t 2 , are the number of Gaussian, weight estimation, the ith Gaussians mean and variance in the mixture at time ‘t’. The decision standards at time ‘t’, so the specific pixel as background or foreground is: X t − μ i , t > λ σ i , t (3) In (3), λ is a perpetual threshold equivalent to 2.5, with one of the K Gaussian modules, if a match is noticed, the pixel is grouped as background. Moreover its factors are updated as: ω i , t + 1 = ( 1 − α ) ω i , t + α (4) μ i , t + 1 = ( 1 − β ) μ i , t + 1 + β X t + 1 (5) σ i 2 , t + 1 = ( 1 − β ) σ i 2 , t + 1 + β ( X t + 1 − μ i , t ) ( X t + 1 − μ i , t ) T (6) β = α η X t + 1 ; μ i , t + 1 , σ i , t + 1 2 (7)where α and βare the learning rate, which is constant also second learning rate, respectively. The pixel is sorted as a foreground, if none of any match is observed among any of the K Gaussian, besides only weight is updated: ω i , t + 1 = ( 1 − α ) ω i , t (8) μ i , t + 1 = μ i , t (9) σ i , t + 1 2 = σ i , t 2 (10) Gaussian Mixture Model is applied for extracting the foreground based on shape. Next to the extraction process, for separating skeleton shape from persisting objects this analysis is used. Foreground region of interest (ROI) is termed to feature extraction process. For that, we use feature extraction based on optical flow process. Optical flow feature set is termed to denote the segmented object(s). The optical flow based feature vectors are calculated beside the boundary and the feature set includes the shape and instantaneous velocity data mined through the action performers boundaries. The extracted optical flow based features are fed to a logistic regression. 3.2 Lucas–Kanade method for optical flow measurement The Lucas–Kanade (LK) method is generally utilised for estimating optical flow as well as computer vision in a differential method. The simple optical flow equations are solved by this approach for every pixel in that neighbourhood, using the minimum squares principle. This approach is not able to provide the details of flow in the identical internal regions of the image; hence it is a local method. In this approach, the flow is supposed to be constant in a pixel of the local neighbourhood under consideration [28]. The equation based on the optical flow can be made to hold for every pixel within a window positioned at ‘P’. The LK method states that the movement of the image contents among two adjacent frames are in-significant and almost constant within the point P's neighbourhood [29]. I x p 1 V x + I y p 1 V y = − I t p 1 (11) I x p 2 V x + I y p 2 V y = − I t p 2 (12) I x p n V x + I y p n V y = − I t p n (13) Inside the window P1, P2…..Pn are the pixels, and the partial derivatives of the image ‘I’ are Ix(pn), Iy (pn), and It(pn) concerning the position (x, y) and time ‘t’, estimated at the point ‘Pn’ at the current time. In matrix form, the equations can be denoted: A v = b where A = I x ( p 1 ) I y ( p 1 ) I x ( p 2 ) I y ( p 2 ) . . . . I x ( p n ) I y ( p n ) , v = V x V y , b = − I t ( p 1 ) − I t ( p 2 ) . . − I x ( p n ) (14) By means of the least squares standard, the solution of LK is determined. Now AT AV = AT b can be expressed in another form and obtain V = (AT A)−1 AT b, where AT is the transpose matrix which can be written as: V x V y = ∑ i I x ( P n ) 2 ∑ i I x ( P n ) I y ( P n ) ∑ i I x ( P n ) I y ( P n ) ∑ i I y ( P n ) 2 − 1 × − ∑ i I x ( P n ) I t ( P n ) − ∑ i I y ( P n ) I t ( P n ) (15) Here, n = N to 1 The structure tensor of the image is characterised by the matrix AT A at the point ‘P’. 3.2.1 Weighted window The former presented plain least squares clarification provides a similar prominence to all N pixels Pn in the window. In the procedure, it is generally suitable to provide additional weight to the pixels that are adjacent to the central pixel P. For this purpose, the least squares equation's weighted version is used [30]. AT WAV = AT W b, this equation can be written so that to obtain V. In which ‘W’ is diagonal matrix of (n × n) with the weights Wii = wi, which is to be assigned to the equation of the pixel Pn, which computes the value of V. V x V y = ∑ i w i I x ( P n ) 2 ∑ i w i I x ( P n ) I y ( P n ) ∑ i w i I x ( P n ) I y ( P n ) ∑ i w i I y ( P n ) 2 − 1 × − ∑ i w i I x ( P n ) I t ( P n ) − ∑ i w i I y ( P n ) I t ( P n ) (16) The weight ‘wi’ is generally allocated to a Gaussian function of the distance concerning Pn and P. A weighted least squares (LS) fit of local first-order constraints has been applied by certain approaches such as Lucas and Kanade and others to model for ‘v’ which is a constant in every small spatial neighbourhood by minimising the below equation. V = ∑ w 2 ( x , y ) ∇ I ( x , y , t ) 2 (17) In (17), w(x, y) represents a window function that provides considerable influence to constrain at the neighbourhood's centre. 3.3 Motion descriptor The goal of finding the anomalous action present in a surveillance video, a feasible descriptor is used for describing the information of the movement, which could be processed through a machine learning approach. Optical flow is a form of object's motion visual formed based on the comparative motion concerning an observer and a scene. Using a basic feature, the optical flow is preferred to signify the movement among two image frames. As an objective function, the optical flow approach is optimised, which using a spatial term modelling, fuses with the brightness constancy limit, in whatever approach the flow probably varies across the image. Using a comprehensive constraint of smoothness, the aperture concerns could be solved. The Optical flow is formulated in Section 3.2. The Feature descriptor based on a covariance matrix dependent optical flow [31] combines the information of the spatio-temporal motion of some of the consecutive RoI (Region of Interest). Initially, the video clips are divided into classes comprising of n number of frames, then for each of the two sequential frames, the optical flow is computed. The RoI features of the pixels in every group are arranged from 1st to nth. As an example, from the kth group, the position of the pixel from (1, 1) to (h, w) are experimented using the height ‘h’ also width ‘w’ in a RoI. The coding related to the information of the movements are done using the matrix with (2 + l) columns and (h × w) rows, the optical flow relied on features length is denoted as ‘l’; also the height and width of the RoI are denoted as ‘h’ and ‘w’. One RoI‘s (R) feature matrix is represented, as shown in (18): R = 1 1 … f e a t u r e 1 1 … f e a t u r e . … . . … . . … . . h w f e a t u r e h × w (18) Every RoI present in a group is classified with the serial numbers from 1st to nth in a multi-RoI covariance feature, then one RoI groups feature matrix is represented as in Equation (19), Where1 is a vector with all the elements 1. One group matrix's dimension is (n × h × w) × (3 + l). The initial format of the column in the group is the RoI position, and further columns are the position of the pixel in one RoI. Based on the objective, the feature based on the optical flow is determined. For the problems related to the detection of an anomalous event, the feature based on the optical flow is presented in Table 1. TABLE 1. The constructed covariance descriptor based on the optical flow feature Feature Vector F F1(4*4) [y,x,U,V] F2(8*8) [y,x,U,V,Ux,Uy,Vx,Vy] F3(14*14) [y,x,U,V,Ux,Uy,Vx,Vy,Uxx,Uyy,Vxx,Vyy,Uxy,Vxy] The issues present in the feature selected for abnormal detection is described as: F ( X , Y ) = ϕ ( X , Y , U , V ) (19) In (20), the horizontal and vertical optical flow are denoted as U, V,ϕ is the ith feature's optical flow. In the dimension ( h × w ) × ( 3 + l ) , ‘F’ denotes the feature gained for a single group of RoI, which is described in Equation (18). With the feature F, the covariance matrix of dimension ( 3 + l ) × ( 3 + l ) is presented in (20) [32]: c = 1 M − 1 ∑ i = 1 m ( Z i − μ ) ( Z i − μ ) T (20) In (20) the number of the pixels sampled art denoted as ‘M’. For a group of RoI's it is signified as n × h × w. The mean of the ‘m’ feature vector is ‘μ’ and the ith point's feature vector is ‘Zi’. Therefore, the covariance matrix unites frequent features. The proposed feature's calculation is shown in Figure 2. A feature matrix ‘F’ is represented in a video stream ‘V’, with group ‘G’ having n number of frames, besides the multi-frame covariance matrix ‘c’ is computed. The intra-RoI information is defined by the pixels position, optical flow as well as the equivalent partial derivatives. Furthermore, the inter-RoI knowledge stamps by the RoI position in one group. Hence, the RoI movement property combined by the proposed covariance descriptor exposes the spatial-temporal feature. Due to symmetry, based on the covariance matrix ‘c’ that combines the ‘l’ features, it has l 2 + l 2 dissimilar values with the support of the integral image, the complications in creating a rectangular region covariance which is O ( l 2 ) . When the raw values are represented in a similar region, the feature dimension is n × l. Specifically, the number of the pixels are denoted by ‘n’. The feature's dimension is considered as B l , if histograms of the joint features are used, where ‘B’ denotes the number of histogram bins. Hence, when compared to other region descriptors, the covariance matrices are effective and is of low-dimensional. FIGURE 2Open in figure viewerPowerPoint The multi-RoI covariance matrix features calculation in progress. The optical flow of RoI is represented based on arrows. Blue, green and red colours denote different RoI's 3.4 Logistic regression analysis for abnormal behaviour detection The logistic regression analysis for the detection of abnormal behaviour is introduced in this section. Logistic regression, as it is known, might be bi- or multinomial. Binary otherwise binomial logistic regression is the case