# Online Detection of Abnormal Events in Video Streams

Authors: Tian Wang; Jie Chen; Hichem Snoussi
Journal: Journal of Electrical and Computer Engineering
 (2013)

---

## Abstract

We propose an algorithm to handle the problem of detecting abnormal events, which is a challenging but important subject in video surveillance. The algorithm consists of
an image descriptor and online nonlinear classification method. We introduce the covariance matrix of the optical flow and image intensity as a descriptor encoding moving information. The nonlinear online support vector machine (SVM) firstly learns a limited set of the training frames to provide a basic reference model then updates the model and detects abnormal events in the current frame. We finally apply the method to detect abnormal events on a benchmark video surveillance dataset to demonstrate the effectiveness of the proposed technique.


---

## Body

## 1. Introduction

Visual surveillance is one of the major research areas in computer vision. In a crowd image analysis problem, the scientific challenge includes abnormal events detection. For instance, Figure1(a) illustrates a normal scene where the people are walking. In Figure 1(b), all the people are suddenly running in different directions. This dataset imitates panic-driven scenes.Examples of the normal and abnormal scenes: (a) Normal lawn scene: all the people are walking. (b) Abnormal lawn scene, all the people are running.
(a)
Normal lawn scene
(b)
Abnormal lawn sceneTrajectory analysis of objects was described in [1–3]. The moving object was labeled by a blob in consecutive frames, and then a trajectory was produced. The deviation from the learnt trajectories was defined as abnormal events. Tracking based approaches are suitable for the sparse scenes with a few objects. The target might be lost due to occlusion.In [4, 5], abnormal detection approaches which used features encoding motion, texture, and size of the objects were introduced. Local image regions in a video were analyzed by employing background subtraction method; then a dynamic Bayesian network (DBN) was constructed to model normal and abnormal behavior, and finally a likelihood ratio test was applied to detect abnormal behaviors. In [6], a space-time Markov random field (MRF) model which detected abnormal activities in a video was proposed, mixture of probabilistic principal component analyzers (MPPCA) was adopted to model local optical flow. The prediction is based on probabilistic assumption techniques where an accurate model exists, but there are various situations where a robust and tractable model cannot be obtained; model-free methods are needed to be studied.Spatiotemporal motion features described by the context of bag of video words were adopted to detect abnormal events. In [7], the authors presented an algorithm which monitored optical flow in a set of fixed spatial positions, and constructed a histogram of optical flow. The likelihood of the behavior in a new coming frame concerning the probability distribution of the statistically learning behavior was computed. If the likelihood fell below a preset threshold, the behavior was considered as abnormal. In [8], irregular behavior of images or videos was detected by an inference process in a probabilistic graphical model. In [9, 10], the video pixels were densely sampled to form the feature. These methods are based on the partial information of images, such as small blocks in a frame, without fully exploiting the global information of the feature. In [11–13], spatiotemporal features modeled motion regions of the frame as background, and anomaly was detected by subtracting the newly sample to the background template. These works are similar to the change detection method when the background is not stable.In this paper, the proposed algorithm is composed of two parts. Firstly, a covariance feature descriptor is constructed over the whole video frame, and then a nonlinear one-class support vector machine algorithm is applied in an online fashion in order to detect abnormal events. The features are extracted based on the optical flow which presents the movement information. Experiments of real surveillance video dataset show that our online abnormal detection techniques can obtain satisfactory performance. The rest of the paper is organized as follows. In Section2, covariance matrix descriptor of motion feature is introduced. In Section 3, the online one-class SVM classification method is presented. In Section 4, two abnormal detection strategies based on online nonlinear one-class SVM are proposed. In Section 5, we present results of real-world video scenes. Finally, Section 6 concludes the paper.


## 2. Covariance Descriptor of Frame Behavior

The optical flow is a feature which presents the direction and the amplitude of a movement. It can provide important information about the spatial arrangement of the objects and the change rate of this arrangement [14]. We adopt Horn-Schunck (HS) optical flow computation method in our work. The optical flow of the gray scale image is formulated as the minimizer of the following global energy functional:
(1)E=∬[(Ixu+Iyv+It)2+α2(∥∇u∥2+∥∇v∥2)]dxdy,
where I is the intensity of the image, Ix, Iy, and It are the derivatives of the image intensity value along the x, y, and time t dimension, u and v are the components of the optical flow in the horizontal and vertical direction, and α represents the weight of the regularization term.We introduce the covariance matrix encoding the optical flow and intensity of each frame as the descriptor to represent the movement. The covariance feature descriptor is originally proposed by Tuzel et al. [15] for pattern matching in a target tracking problem. The descriptor is defined as
(2)F(x,y,i)=ϕi(I,x,y),
where I is the color information of an image (which can be gray, RGB, HSV, HLS, etc.), ϕi is a mapping relating the image with the ith feature from the image, F is the W×H×d dimensional feature extracted from image I, W and H are the image width and image height, and d is the number of chosen features. For each frame, the feature can be represented as d×d covariance matrix:
(3)C=1n-1∑k=1n‍(zk-μ)(zk-μ)⊤,
where n is the number of the pixels sampled in the frame, zk is the feature vector of pixel k, μ is the mean of all the selected points, and C is the covariance matrix of the feature vector F. The covariance descriptor C of each frame dose not have any information regarding the sample ordering and the number of points [15]. Because the feature F can be designed as different approaches to fuse features, the covariance matrix descriptor proposes a way to merge multiple parameters. Different choices of feature vectors extraction are shown in Table 1, where I is the intensity of the gray image, u and v are horizontal and vertical components of optical flow, Ix, ux, and vx are the first derivatives of the intensity, horizontal optical flow, and vertical optical flow in the x direction respectively, Iy, uy, and vy are the first derivatives of the corresponding feature in the y direction and respectively, Ixx, uxx, and vxx are the second derivatives in x direction, Iyy, uyy, and vyy are the second derivatives in y direction. The flowchart of covariance matrix descriptor computation is shown in Figure 2. The optical flow and corresponding partial derivative characterize the interframe information or can be regarded as the movement information. The intensity of the frame and partial derivative of the intensity describe the intraframe information; they encode the appearance information of the frame.Table 1
Different choices of featureF to construct the covariance descriptor.
Feature vectorF
F
1
(
6
×
6
)
[
y
x
u
v
u
x
u
y
]
F
2
(
6
×
6
)
[
y
x
u
v
v
x
v
y
]
F
3
(
8
×
8
)
[
y
x
u
v
u
x
u
y
v
x
v
y
]
F
4
(
12
×
12
)
[
y
x
u
v
u
x
u
y
v
x
v
y
u
x
x
u
y
y
v
x
x
v
y
y
]
F
5
(
17
×
17
)
[
y
x
u
v
u
x
u
y
v
x
v
y
u
x
x
u
y
y
v
x
x
v
y
y
I
I
x
I
y
I
x
x
I
y
y
]Figure 2
Covariance descriptor computation of a video frame.If proper parameters are given, the traditionally used kernel, such as Gaussian, polynomial, and sigmoidal kernel, has similar performances [19]. Gaussian kernel κ(xi,xj)=exp(-∥xi-xj∥2/2σ2) is chosen for our spatial features. The covariance matrix is an element in Lie group; the Gaussian kernel on the Euclidean spaces is not suitable for the covariance descriptors. The Gaussian kernel in Lie Group is defined as [20, 21]:
(4)κ(Xi,Xj)=exp(-∥log(Xi-1Xj)∥2σ2),(Xi,Xj)∈G×G,
where Xi and Xj are matrices in Lie Group G.


## 3. Online One-Class SVM

The essence of an abnormal detection problem is that only normal scene samples are available. The one-class SVM framework is well suitable to an abnormal detection problem. Support vector machine (SVM) is initially proposed by Vapnik and Lerner [22, 23]. It is a method based on statistical learning theory and has fine performance to classify data and recognize patterns. There are two frameworks of one-class SVM, one is support vector data description (SVDD) which is presented in [24, 25] and the other is ν-support vector classifier (ν-SVC) introduced in [26]. The SVDD formulation is adopted in our work. It computes a sphere shaped decision boundary with minimal volume around a set of objects. The center of the sphere c and the radius R are to be determined via the following optimization problem:
(5)minR,ξ,cR2+C∑i=1n‍ξi,≤R2+ξi,ξi≥0,∀iξi≥,(6)subjectto∥Φ(xi)-c∥2≤R2+ξi,ξi≥0,∀i,
where n is the number of training samples and ξi is the slack variable for penalizing the outliers. The hyperparameter C is the weight for restraining slack variables; it tunes the number of acceptable outliers. The nonlinear function Φ:𝒳→ℋ maps a datum xi into the feature space ℋ; it allows to solve a nonlinear classification problem by designing a linear classifier in the feature space ℋ. κ is the kernel function for computing dot products in ℋ, κ(x,x′)=〈Φ(x),Φ(x′)〉. By introducing Lagrange multipliers, the dual problem associated with (6) is written by the following quadratic optimization problem:
(7)maxα∑i=1n‍αiκ(xi,xi)-∑i,j=1n‍αiαjκ(xi,xj),subjectto0≤αi≤C,∑i=1n‍αi=1,c=∑i=1n‍αiΦ(xi).
The decision function is
(8)f(x)=sgn(R2-∑i,j=1n‍αiαjκ(xi,xj)+2∑i=1n‍αiκ(xi,x)-κ(x,x)).For the large training data, the solution cannot be obtained easily, and an online strategy to train the data is used in our work. Letc𝔇 denotes a sparse model of the center cn=(1/n)∑i=1n‍Φ(xi) by using a small subset of available samples which is called dictionary:
(9)c𝔇=∑i∈𝔇‍αiΦ(xi),
where 𝔇⊂{1,2,…,n}, and let N𝔇 denote the cardinality of this subset x𝔇. The distance of a mapped datum Φ(x) with respect to the center c𝔇 can be calculated by
(10)∥Φ(x)-c𝔇∥=∑i,j∈𝔇‍αiαjκ(xi,xj)-2∑i∈𝔇‍αiκ(xi,x)+κ(x,x).
A modification of the original formulation of the one-class classification algorithm that consists of minimizing the approximation error ∥cn-c𝔇∥ is [27, 28]
(11)α=argminαi,i∈𝔇∥1n∑i=1n‍Φ(xi)-∑i∈𝔇‍αiΦ(xi)∥2.
The final solution is given by
(12)α=K-1κ,
where K is the Gram matrix with (i,j)th entry κ(xi,xj) and κ is the column vector with entries (1/n)∑i=1nκ(xk,xi), k∈𝔇.In the online scheme, at each time step there is a new sample. Letαn denote the coefficients, Kn denote the Gram matrix, and κn denote the vector, at time step n. A criterion is used to determine whether the new sample can be included into the dictionary. A threshold μ0 is preset, for the datum xt at time step t, the coherence-based sparsification criterion [29, 30] is
(13)ϵt=maxi∈𝔇|κ(xi,xt)|.First Case  (ϵt>μ0). In this case, the new data Φ(xn+1) is not included into the dictionary. The Gram matrix Kn+1=Kn. κn changes online:
(14)κn+1=1n+1(nκn+b),αn+1=Kn+1-1κn+1=nn+1αn+1n+1Kn-1b,
where b is the column vector with entries κ(xi,xn+1), i∈𝔇.Second Case  (ϵt≤μ0). In this case, the new data Φ(xn+1) is included into the dictionary 𝔇. The Gram matrix K changes:
(15)Kn+1=[Knbb⊤κ(xn+1,xn+1)].
By using Woodbury matrix identity
(16)(A+UCV)-1=A-1-A-1U(C-1+VA-1U)-1VA-1,Kn+1-1 can be calculated iteratively:
(17)Kn+1-1=[Kn-100⊤0]+1κ(xn+1,xn+1)-b⊤Kn-1b×[-Kn-1b1][-b⊤Kn-11].
The vector κn+1 is updated from κn,
(18)κn+1=1n+1[nκn+b→κn+1],
with
(19)κn+1=∑i=1n+1‍κ(xn+1,xi).
Computing κn+1 as (19) needs to save all the samples {x}i=1n+1 in memory. For conquering this issue, it can compute as κn+1=(n+1)κ(xn+1,xn+1) by considering an instant estimation. The update of αn+1 from αn is
(20)αn+1=1n+1[nαn+Kn-1b0]-1(n+1)(κ(xn+1,xn+1)-b⊤Kn-1b)×[Kn-1b1](nb⊤αn+b⊤Kn-1b-κn+1).
Based on (20), we have an online implementation of the one-class SVM learning phase.


## 4. Abnormal Events Detection

In an abnormal event detection problem, it is assumed that a set of training frames{I1,…,In} (the positive class) describing the normal behavior is obtained. The general architectures of abnormal detection are introduced below.The offline training strategy refers to the case where all the training samples are learnt as one batch, as shown in Figure3(a). We propose two abnormal detection strategies; the difference between these two strategies is the time when the dictionary is fixed. These two strategies are shown in Figures 3(b) and 3(c). Strategy 1 is shown in Figure 3(b). The training data are learnt one by one. When the training period is finished, the dictionary and the classifier are fixed. Each test datum is classified based on the dictionary. Figure 3(c) illustrates Strategy 2. The training procedure is the same as Strategy 1. But in the testing period, the dictionary is updated if the datum xi satisfies the dictionary update condition. The details of these two strategies are explained in the following.Offline and two online abnormal event detection strategies based on one-class SVM. (a) Strategy offline. The training data are learnt as one batch offline. (b) Strategy 1. The dictionary is fixed when all the training data are learnt. (c) Strategy 2. The dictionary continues being updated through the testing period.
(a)
Strategy offline
(b)
Strategy 1
(c)
Strategy 2


### 4.1. Strategy 1

In Strategy 1, the dictionary is updated merely through the training period.Step 1.
The first step is calculating the covariance matrix descriptor of training frames based on the image intensity and the optical flow. This step can be generalized as(21){(I1,OP1),(I2,OP2),…,(In,OPn)}⟶{C1,C2,…,Cn},
where {(I1,OP1),(I2,OP2),…,(In,OPn)} are the image intensity and the corresponding optical flow of the 1st to nth frame. {C1,C2,…,Cn} are the covariance matrix descriptors.Step 2.
The second step consists of applying one-class SVM on the small subset of extracted descriptor of the training normal frames to obtain the support vectors. Consider a subset{Ci}i=1m, 1≤m≪n of data selected from the full training sample set {Ci}i=1n, without loss of generality, and assume that the first m examples are chosen. This set of m examples is called dictionary C𝔇:
(22){C1,C2,…,Cm},1≤m≪n⟶SVMsupportvector{Sp1,Sp2,…,Spo},
where the set {C1,C2,…,Cm} is the first m covariance matrix descriptors of the training frames; it is the original dictionary C𝔇. In one-class SVM, the majority of the training samples do not contribute to the definition of the decision function. The entries of a monitory subset of the training samples, {Sp1,Sp2,…,Spo}, o≤m, are support vectors contributing to the definition of the decision function.Step 3.
After learning the dictionaryC𝔇 which includes the first m, 1≤m≪n samples, the training samples {Cm+1,Cm+2,…,Cn} are learned online via the technique described in Section 3. This step can be generalized as(23){C𝔇,Ck},m<k≤n⟶SVMsupportvector{Sp1,Sp2,…,Spp},o≤p≤n,C𝔇:=C𝔇∪Ck,ifϵt≥μ0,where C𝔇 is the dictionary obtained through Step 2, Ck is a new sample in the remaining training dataset. According to the criterion introduced in Section 3, if the new sample Ck satisfies the dictionary updated condition, it will be included into the dictionary C𝔇.Step 4.
Based on the dictionary and the classifier obtained from the training frames, the incoming frame sampleCn+l is classified. The workflow of Strategy 1 is shown in Figure 4 and described by the following equation:
(24)f(Cn+l)=sgn(R2-∑i,j=1n‍αiαjκ(Ci,Cj)+2∑i‍αiκ(Ci,Cn+l)-κ(Cn+l,Cn+l))={1f(Cn+l)≥0-1f(Cn+l)<0,
where Cn+l is the covariance matrix descriptor of the (n+l)th frame needed to be classified and Ci and Cj are the samples of the dictionary C𝔇. “1” corresponds to the normal frame; “-1” corresponds to the abnormal frame.Figure 4
Major processing states of the proposed abnormal frame event detection method. The covariance of the frame is computed.


### 4.2. Strategy 2

In this strategy, the dictionary is updated through both training and testing periods. The feature extraction step (Step 1) and the online training steps (Steps 2 and 3) are the same as the ones presented in Strategy 1. The testing step is different. The new coming datum which is detected as normal but satisfies dictionary update condition should be included intoC𝔇. The dictionary is needed to be updated through the testing period to include new samples.Step  4: Strategy 2. If the incoming frame sample Cn+l is classified as normal (f(Cn+l)=1), the data is checked by the criterion described in Section 3. When the data satisfies the dictionary update criterion, this testing sample will be included into the dictionary. This step can be generalized by the following equation:(25)f(Cn+l)=sgn(R2-∑i,j=1n‍αiαjκ(Ci,Cj)+2∑i‍αiκ(Ci,Cn+l)-κ(Cn+l,Cn+l))={1f(Cn+l)≥0{ϵt≥μ0⟶C𝔇:=C𝔇∪Cn+lϵt<μ0⟶C𝔇:=C𝔇-1f(Cn+l)<0.


## 4.1. Strategy 1

In Strategy 1, the dictionary is updated merely through the training period.Step 1.
The first step is calculating the covariance matrix descriptor of training frames based on the image intensity and the optical flow. This step can be generalized as(21){(I1,OP1),(I2,OP2),…,(In,OPn)}⟶{C1,C2,…,Cn},
where {(I1,OP1),(I2,OP2),…,(In,OPn)} are the image intensity and the corresponding optical flow of the 1st to nth frame. {C1,C2,…,Cn} are the covariance matrix descriptors.Step 2.
The second step consists of applying one-class SVM on the small subset of extracted descriptor of the training normal frames to obtain the support vectors. Consider a subset{Ci}i=1m, 1≤m≪n of data selected from the full training sample set {Ci}i=1n, without loss of generality, and assume that the first m examples are chosen. This set of m examples is called dictionary C𝔇:
(22){C1,C2,…,Cm},1≤m≪n⟶SVMsupportvector{Sp1,Sp2,…,Spo},
where the set {C1,C2,…,Cm} is the first m covariance matrix descriptors of the training frames; it is the original dictionary C𝔇. In one-class SVM, the majority of the training samples do not contribute to the definition of the decision function. The entries of a monitory subset of the training samples, {Sp1,Sp2,…,Spo}, o≤m, are support vectors contributing to the definition of the decision function.Step 3.
After learning the dictionaryC𝔇 which includes the first m, 1≤m≪n samples, the training samples {Cm+1,Cm+2,…,Cn} are learned online via the technique described in Section 3. This step can be generalized as(23){C𝔇,Ck},m<k≤n⟶SVMsupportvector{Sp1,Sp2,…,Spp},o≤p≤n,C𝔇:=C𝔇∪Ck,ifϵt≥μ0,where C𝔇 is the dictionary obtained through Step 2, Ck is a new sample in the remaining training dataset. According to the criterion introduced in Section 3, if the new sample Ck satisfies the dictionary updated condition, it will be included into the dictionary C𝔇.Step 4.
Based on the dictionary and the classifier obtained from the training frames, the incoming frame sampleCn+l is classified. The workflow of Strategy 1 is shown in Figure 4 and described by the following equation:
(24)f(Cn+l)=sgn(R2-∑i,j=1n‍αiαjκ(Ci,Cj)+2∑i‍αiκ(Ci,Cn+l)-κ(Cn+l,Cn+l))={1f(Cn+l)≥0-1f(Cn+l)<0,
where Cn+l is the covariance matrix descriptor of the (n+l)th frame needed to be classified and Ci and Cj are the samples of the dictionary C𝔇. “1” corresponds to the normal frame; “-1” corresponds to the abnormal frame.Figure 4
Major processing states of the proposed abnormal frame event detection method. The covariance of the frame is computed.


## 4.2. Strategy 2

In this strategy, the dictionary is updated through both training and testing periods. The feature extraction step (Step 1) and the online training steps (Steps 2 and 3) are the same as the ones presented in Strategy 1. The testing step is different. The new coming datum which is detected as normal but satisfies dictionary update condition should be included intoC𝔇. The dictionary is needed to be updated through the testing period to include new samples.Step  4: Strategy 2. If the incoming frame sample Cn+l is classified as normal (f(Cn+l)=1), the data is checked by the criterion described in Section 3. When the data satisfies the dictionary update criterion, this testing sample will be included into the dictionary. This step can be generalized by the following equation:(25)f(Cn+l)=sgn(R2-∑i,j=1n‍αiαjκ(Ci,Cj)+2∑i‍αiκ(Ci,Cn+l)-κ(Cn+l,Cn+l))={1f(Cn+l)≥0{ϵt≥μ0⟶C𝔇:=C𝔇∪Cn+lϵt<μ0⟶C𝔇:=C𝔇-1f(Cn+l)<0.


## 5. Abnormal Detection Result

This section presents the results of experiments conducted to analyze the performance of the proposed method. A competitive performance through both Strategy 1 and Strategy 2 of UMN [31] dataset is presented.


### 5.1. Abnormal Visual Events Detection: Strategy 1

The results of the proposed abnormal events detection method via Strategy 1 online one-class SVM of UMN [31] dataset are shown below.The UMN dataset includes eleven video sequences of three different scenes (the lawn, indoor, and plaza) of crowded escape events. The normal samples for training or for normal testing are the frames where the people are walking in different directions. The samples for abnormal testing are the frames where people are running. The detection results of the lawn scene, indoor scene, and plaza scene are shown in Figures5, 6, and 7, respectively. Gaussian kernel for the Lie Group is used in these three scenes. Different values of σ and penalty factor C are chosen; the area under the ROC curve is shown as a function of these parameters [32]. The results show that taking covariance matrix as descriptor can obtain satisfactory performance for abnormal detection. And also, training the samples online can obtain similarly detection performance as training all the samples offline. Online one-class SVM is appropriate to detect abnormal visual events. There are 1431 frames in the lawn scene, and 480 normal frames are used for training. In the offline strategy, all the 480 frames covariance matrices should be saved in the memory. In Strategy 1, 100 frames covariance matrices are considered as the dictionary firstly. When F5-17×17 feature is adopted to construct the covariance descriptor, the variance of Gaussian kernel is σ=1, the preset threshold of the criterion is μ0=0.5, the dictionary size increases from 100 to 101, and the maximum accuracy of the detection results is 91.69%. In the indoor scene, there are 2975 normal frames and 1057 abnormal frames. In the plaza scene, there are 1831 normal frames and 286 abnormal frames. The processes of the experiments are similar to the ones of the lawn scene. When feature vector is F5-17×17, σ=1, μ0=0.5, the dictionary size of these two scenes remain 100. The online strategy keeps the memory size almost unchanged when the size of training dataset increases.Detection results of the lawn scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the lawn scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.9591. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.9581.
(a)
Normal lawn scene
(b)
Abnormal lawn scene
(c)
ROC train offline
(d)
ROC Strategy 1Detection results of the indoor scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the indoor scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.8649. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.8628.
(a)
Normal indoor scene
(b)
Abnormal indoor scene
(c)
ROC train offline
(d)
ROC Strategy 1Detection results of the plaza scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the plaza scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.9649. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.9632.
(a)
Normal plaza scene
(b)
Abnormal plaza scene
(c)
ROC train offline
(d)
ROC Strategy 1


### 5.2. Abnormal Visual Events Detection: Strategy 2

The results of the abnormal event detection method via Strategy 2 of UMN dataset are shown as follows. In the experiment process of the lawn scene, 100 normal samples from the training samples are learnt firstly, and then the other 380 training data are learnt online one by one. After these two training steps, we can obtain the basic dictionary from the training samples and also the classifier. In the following testing step, the dictionary is updated if the sample satisfies the dictionary update criterion. When a new sample is coming, it is firstly detected by the previous classifier. If it is classified as anomaly, the dictionary and the classifier are not changed. Otherwise, if the sample is classified as a normal one, the sparse criterion introduced in Section3 is used to check the correlation between the earlier dictionary and this new datum. It will be included into the dictionary when it satisfied the update condition. The dictionary will be updated through the whole testing period. The other two scenes, the indoor and plaza scene, are handled by the same methods. When F5-17×17 feature is adopted, the variance of the Gaussian kernel is σ=1, and the preset threshold of the criterion is μ0=0.5, and the dictionary size of the lawn, indoor, and plaza scene is increased from 100 to 106, 102, and 102, respectively. The ROC curve of detection results of these three scenes is shown in Figures 8(a), 8(b), and 8(c). Besides the merit of saving memory of Strategy 1, Strategy 2 also has the advantage of adaptation to the long duration sequence.ROC curve of UMN dataset. (a) ROC curve of different featuresF results via Strategy 2 of the lawn scene. The biggest AUC value is 0.9605. (b) Strategy 2 results of the indoor scene. The biggest AUC value is 0.8495. (c) Strategy 2 results of the plaza scene. The biggest AUC value is 0.9746. (d) The ROC curve of best performance of the lawn, indoor, and plaza scene when the training samples are learnt offline. The biggest AUC values of the lawn, indoor, and plaza are 0.9591, 0.8649 and 0.9649.
(a)
ROC Strategy 2 lawn scene
(b)
ROC Strategy 2 indoor scene
(c)
ROC Strategy 2 plaza scene
(d)
ROC train offlineThe results performances of offline strategy, Strategy 1, and Strategy 2 are shown in Table2. The performances of these two strategies results are similar to that of the results when all training samples are learnt together. When F4(12×12) or F5(17×17) are chosen as the features to form covariance matrix descriptor, the results have the best performance. These two features are more abundant to include movement and intensity information.Table 2
AUC of abnormal event detection method of different featuresF via original SVDD which learns training samples offline, Strategy 1 online one-class SVM, and Strategy 2 online one-class SVM. The biggest value of each method is shown in bold.
Features
Area under ROC
Lawn
Indoor
Plaza
Training samples are learned offline
F
1
(
6
×
6
d
u
)
0.9426
0.8351
0.9323
F
2
(
6
×
6
d
v
)
0.9400
0.8358
0.9321
F
3
(
8
×
8
)
0.9440
0.8375
0.9359
F
4
(
12
×
12
)
0.9591
0.8440
0.9580
F
5
(
17
×
17
)
0.9567
0.8649
0.9649
Strategy 1
F
1
(
6
×
6
d
u
)
0.9399
0.8328
0.9343
F
2
(
6
×
6
d
v
)
0.9390
0.8355
0.9366
F
3
(
8
×
8
)
0.9418
0.8377
0.9411
F
4
(
12
×
12
)
0.9581
0.8457
0.9573
F
5
(
17
×
17
)
0.9551
0.8628
0.9632
Strategy 2
F
1
(
6
×
6
d
u
)
0.9427
0.8237
0.9288
F
2
(
6
×
6
d
v
)
0.9370
0.8241
0.9283
F
3
(
8
×
8
)
0.9430
0.8274
0.9312
F
4
(
12
×
12
)
0.9605
0.8331
0.9505
F
5
(
17
×
17
)
0.9601
0.8495
0.9746The result performances of the covariance matrix descriptor based online one-class SVM method and the state-of-the-art methods are shown in Table3. The covariance matrix based online abnormal frame detection method obtains competitive performance. In general, our method is better than others except sparse reconstruction cost (SRC) [17] in lawn scene and indoor scene. In that paper, multiscale HOF is taken as a feature, and a testing sample is classified by its sparse reconstructor cost, through a weighted linear reconstruction of the overcomplete normal basis set. But computation of the HOF might takes more time than calculating covariance. By adopting the integral image [15], the covariance matrix descriptor of the subimage can be computed conveniently. So the covariance descriptor can be appropriately used to analyze the partial movement.Table 3
The comparison of our proposed method with the state-of-the-art methods for the abnormal event detection in the UMN dataset.
Method
Area under ROC
Lawn
Indoor
Plaza
Social force [16]
0.96
Optical flow [16]
0.84
NN [17]
0.93
SRC [17]
0.995
0.975
0.964
STCOG [18]
0.9362
0.7759
0.9661
COV online (ours)
0.9605
0.8628
0.9746


## 5.1. Abnormal Visual Events Detection: Strategy 1

The results of the proposed abnormal events detection method via Strategy 1 online one-class SVM of UMN [31] dataset are shown below.The UMN dataset includes eleven video sequences of three different scenes (the lawn, indoor, and plaza) of crowded escape events. The normal samples for training or for normal testing are the frames where the people are walking in different directions. The samples for abnormal testing are the frames where people are running. The detection results of the lawn scene, indoor scene, and plaza scene are shown in Figures5, 6, and 7, respectively. Gaussian kernel for the Lie Group is used in these three scenes. Different values of σ and penalty factor C are chosen; the area under the ROC curve is shown as a function of these parameters [32]. The results show that taking covariance matrix as descriptor can obtain satisfactory performance for abnormal detection. And also, training the samples online can obtain similarly detection performance as training all the samples offline. Online one-class SVM is appropriate to detect abnormal visual events. There are 1431 frames in the lawn scene, and 480 normal frames are used for training. In the offline strategy, all the 480 frames covariance matrices should be saved in the memory. In Strategy 1, 100 frames covariance matrices are considered as the dictionary firstly. When F5-17×17 feature is adopted to construct the covariance descriptor, the variance of Gaussian kernel is σ=1, the preset threshold of the criterion is μ0=0.5, the dictionary size increases from 100 to 101, and the maximum accuracy of the detection results is 91.69%. In the indoor scene, there are 2975 normal frames and 1057 abnormal frames. In the plaza scene, there are 1831 normal frames and 286 abnormal frames. The processes of the experiments are similar to the ones of the lawn scene. When feature vector is F5-17×17, σ=1, μ0=0.5, the dictionary size of these two scenes remain 100. The online strategy keeps the memory size almost unchanged when the size of training dataset increases.Detection results of the lawn scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the lawn scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.9591. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.9581.
(a)
Normal lawn scene
(b)
Abnormal lawn scene
(c)
ROC train offline
(d)
ROC Strategy 1Detection results of the indoor scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the indoor scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.8649. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.8628.
(a)
Normal indoor scene
(b)
Abnormal indoor scene
(c)
ROC train offline
(d)
ROC Strategy 1Detection results of the plaza scene. (a) The detection result of one normal frame. (b) The detection result of one abnormal panic frame. (c) ROC curve of different featuresF of the plaza scene results via one-class SVM. All the training samples are learned together offline. The biggest AUC value is 0.9649. (d) ROC curve of different features F results via Strategy 1 online one-class SVM. The biggest AUC value is 0.9632.
(a)
Normal plaza scene
(b)
Abnormal plaza scene
(c)
ROC train offline
(d)
ROC Strategy 1


## 5.2. Abnormal Visual Events Detection: Strategy 2

The results of the abnormal event detection method via Strategy 2 of UMN dataset are shown as follows. In the experiment process of the lawn scene, 100 normal samples from the training samples are learnt firstly, and then the other 380 training data are learnt online one by one. After these two training steps, we can obtain the basic dictionary from the training samples and also the classifier. In the following testing step, the dictionary is updated if the sample satisfies the dictionary update criterion. When a new sample is coming, it is firstly detected by the previous classifier. If it is classified as anomaly, the dictionary and the classifier are not changed. Otherwise, if the sample is classified as a normal one, the sparse criterion introduced in Section3 is used to check the correlation between the earlier dictionary and this new datum. It will be included into the dictionary when it satisfied the update condition. The dictionary will be updated through the whole testing period. The other two scenes, the indoor and plaza scene, are handled by the same methods. When F5-17×17 feature is adopted, the variance of the Gaussian kernel is σ=1, and the preset threshold of the criterion is μ0=0.5, and the dictionary size of the lawn, indoor, and plaza scene is increased from 100 to 106, 102, and 102, respectively. The ROC curve of detection results of these three scenes is shown in Figures 8(a), 8(b), and 8(c). Besides the merit of saving memory of Strategy 1, Strategy 2 also has the advantage of adaptation to the long duration sequence.ROC curve of UMN dataset. (a) ROC curve of different featuresF results via Strategy 2 of the lawn scene. The biggest AUC value is 0.9605. (b) Strategy 2 results of the indoor scene. The biggest AUC value is 0.8495. (c) Strategy 2 results of the plaza scene. The biggest AUC value is 0.9746. (d) The ROC curve of best performance of the lawn, indoor, and plaza scene when the training samples are learnt offline. The biggest AUC values of the lawn, indoor, and plaza are 0.9591, 0.8649 and 0.9649.
(a)
ROC Strategy 2 lawn scene
(b)
ROC Strategy 2 indoor scene
(c)
ROC Strategy 2 plaza scene
(d)
ROC train offlineThe results performances of offline strategy, Strategy 1, and Strategy 2 are shown in Table2. The performances of these two strategies results are similar to that of the results when all training samples are learnt together. When F4(12×12) or F5(17×17) are chosen as the features to form covariance matrix descriptor, the results have the best performance. These two features are more abundant to include movement and intensity information.Table 2
AUC of abnormal event detection method of different featuresF via original SVDD which learns training samples offline, Strategy 1 online one-class SVM, and Strategy 2 online one-class SVM. The biggest value of each method is shown in bold.
Features
Area under ROC
Lawn
Indoor
Plaza
Training samples are learned offline
F
1
(
6
×
6
d
u
)
0.9426
0.8351
0.9323
F
2
(
6
×
6
d
v
)
0.9400
0.8358
0.9321
F
3
(
8
×
8
)
0.9440
0.8375
0.9359
F
4
(
12
×
12
)
0.9591
0.8440
0.9580
F
5
(
17
×
17
)
0.9567
0.8649
0.9649
Strategy 1
F
1
(
6
×
6
d
u
)
0.9399
0.8328
0.9343
F
2
(
6
×
6
d
v
)
0.9390
0.8355
0.9366
F
3
(
8
×
8
)
0.9418
0.8377
0.9411
F
4
(
12
×
12
)
0.9581
0.8457
0.9573
F
5
(
17
×
17
)
0.9551
0.8628
0.9632
Strategy 2
F
1
(
6
×
6
d
u
)
0.9427
0.8237
0.9288
F
2
(
6
×
6
d
v
)
0.9370
0.8241
0.9283
F
3
(
8
×
8
)
0.9430
0.8274
0.9312
F
4
(
12
×
12
)
0.9605
0.8331
0.9505
F
5
(
17
×
17
)
0.9601
0.8495
0.9746The result performances of the covariance matrix descriptor based online one-class SVM method and the state-of-the-art methods are shown in Table3. The covariance matrix based online abnormal frame detection method obtains competitive performance. In general, our method is better than others except sparse reconstruction cost (SRC) [17] in lawn scene and indoor scene. In that paper, multiscale HOF is taken as a feature, and a testing sample is classified by its sparse reconstructor cost, through a weighted linear reconstruction of the overcomplete normal basis set. But computation of the HOF might takes more time than calculating covariance. By adopting the integral image [15], the covariance matrix descriptor of the subimage can be computed conveniently. So the covariance descriptor can be appropriately used to analyze the partial movement.Table 3
The comparison of our proposed method with the state-of-the-art methods for the abnormal event detection in the UMN dataset.
Method
Area under ROC
Lawn
Indoor
Plaza
Social force [16]
0.96
Optical flow [16]
0.84
NN [17]
0.93
SRC [17]
0.995
0.975
0.964
STCOG [18]
0.9362
0.7759
0.9661
COV online (ours)
0.9605
0.8628
0.9746


## 6.  Conclusions

A method for the abnormal event detection of the frame is proposed. The method consists of covariance matrix descriptor encoding the movement features, and the online nonlinear one-class SVM classification method. We have developed two nonlinear one-class SVM based abnormal event detection techniques that update the normal models of the surveillance video data in an online framework. The proposed algorithm has been tested on a video dataset yielding successful results to detect abnormal events.



---

*Source: 837275-2013-12-24.xml*