# Forest Fire Smoke Video Detection Using Spatiotemporal and Dynamic Texture Features

Authors: Yaqin Zhao; Zhong Zhou; Mingming Xu
Journal: Journal of Electrical and Computer Engineering
 (2015)

---

## Abstract

Smoke detection is a very key part of fire recognition in a forest fire surveillance video since the smoke produced by forest fires is visible much before the flames. The performance of smoke video detection algorithm is often influenced by some smoke-like objects such as heavy fog. This paper presents a novel forest fire smoke video detection based on spatiotemporal features and dynamic texture features. At first, Kalman filtering is used to segment candidate smoke regions. Then, candidate smoke region is divided into small blocks. Spatiotemporal energy feature of each block is extracted by computing the energy features of its 8-neighboring blocks in the current frame and its two adjacent frames. Flutter direction angle is computed by analyzing the centroid motion of the segmented regions in one candidate smoke video clip. Local Binary Motion Pattern (LBMP) is used to define dynamic texture features of smoke videos. Finally, smoke video is recognized by Adaboost algorithm. The experimental results show that the proposed method can effectively detect smoke image recorded from different scenes.


---

## Body

## 1. Introduction

Fires are a constant threat to forest ecological systems and human safety; moreover, forest fires are an important problem in regions which present hot climate. With the development of computer vision techniques, forest fire video surveillance has been one of research focuses in the field of forest fire prevention. Generally, methods for detecting fire video can be categorized as flame detection and smoke detection. However, most of the fire video systems are mainly designed for smoke detection, since the appearance of smoke is in most cases more visible than the fire itself. Smoke detection algorithms are divided into systems based on single and based on multiple frames. In the first case, smoke images are recognized by color [1, 2], texture [3, 4], and energy [5]. In the second case, smoke features are extracted by video sequence. For example, Favorskaya and Levtin [6] extracted spatiotemporal features of smoke video by grouping moving regions with a turbulence parameter connecting with fractal properties of smoke in order to track effectively a smoke propagation. In [7], wavelet transform is used to detect high frequency information of moving pixels so that smoke flickering features are analyzed. Interesting work was presented by Wang et al. [8] who firstly detected motion regions from video frames, and then four flutter features of the motion regions are extracted over a sliding time window, including the flutter direction and three types of flutter intensities.Smoke detection systems have made some achievements but cannot be used as a self-sufficient solution. They often have a high false rate and thus need an additional human confirmation for final decision. For bringing the performance of the detection systems closer to the results that could currently be obtained by human observers, this paper presents a novel forest fire smoke video detection based on spatiotemporal energy and dynamic texture features. First of all, candidate smoke regions are segmented and then are divided into small blocks. Afterwards, for each block, three features including two spatiotemporal features and one dynamic texture feature are extracted. Finally, Adaboost algorithm is used to recognize smoke video clips. Figure1 shows the basic process of the proposed scheme.Figure 1
The basic process of the proposed scheme.


## 2. Spatiotemporal Feature Extraction

Kalman filtering is firstly used to update video background [8] in order to detect motion regions. And then three features of early smoke of forest fire are extracted in terms of flutter analysis, energy analysis, and color analysis. Motion regions in terms of Section 2.1 are divided into blocks of the size 8 × 8. Then smoke features are extracted from the candidate smoke blocks.


### 2.1. Spatiotemporal Consistency Feature

When smoke diffuses in the scene, it covers part of the scene. The edges of one smoke region are blurred and high frequency information slowly changes. LettingEBG(bm) and EI(bm), respectively, denote the high frequency energy of corresponding background and current frame with the block bm, then square difference of high frequency energy is defined as(1)ΔE2=EIbm-EBGbm2.A smoke block is detected as candidate smoke block according to the following condition:(2)ΔE2≥ΔE2¯,where ΔE2¯ represents the mean of the square difference ΔE2 of the previous t-1 frames.For a smoke block, its spatially and temporally neighboring blocks have greater possibility of being smoke blocks. We count the number of neighboring smoke blocks of the blockbm of the video frame at the time t:(3)DY1t=∑i=t-10tNcandi,where Ncand(i) denote the number of 8-neighboring smoke blocks at the corresponding position bm of the video frame at the time i. Figure 2 shows the calculation process of the spatiotemporal consistency energy.Figure 2
The calculation process of the spatiotemporal consistency energy.


### 2.2. Flutter Feature

As we know, there is massive heavy fog in the forest, so heavy fog can easily cause false alarm. Fortunately, forest fire smoke has one characteristic that is different from heavy fog. Smoke moves from bottom to top because heat smoke has a lower density than air, which is significantly different from heavy fog. The motion direction of one block is mapped as a direction code by computing the centroid motion of one candidate smoke block. Thep+q order moment of one candidate smoke block is computed by the following formula:(4)mpq=∑i=WsWe∑j=HsHeipjqIi,j,where Ws and We denote starting and ending point of one smoke region width, respectively, and Hs and He denote starting and ending point of the region height. The centroid coordinates of one candidate smoke block are defined as(5)Cxt,k=m10m00,Cyt,k=m01m00,k=1,2,…,N,where N=Frate, Frate denotes one frame rate. Cx(t,k) and Cy(t,k) denote horizontal ordinate and vertical ordinate of the centroid of the kth candidate smoke block in tth time window. The moving direction angle of the kth candidate smoke block(6)θtk=arccos⁡Cxt,kCx2t,k+Cy2t,k.Flutter direction angle is defined as(7)θt=∑k=1Frateθtk,if∑k=1Frateθtk≥0,2π-∑k=1Frateθtk,otherwise.


### 2.3. Dynamic Texture Feature

The Local Binary Pattern (LBP) is a powerful means of texture description. The operator labels the pixels of an image region by thresholding the neighborhood of each pixel with the center value and considering the result as a binary number (binary pattern). The classical definition of LBP can be represented as follows:(8)LBPP,Rx=∑p=1PsIvp-Ix2P-1,su=1u≥00u<0,where I(x) corresponds to the gray value of the pixel at position x and I(vp)p=1,…,P to gray values of P equally spaced pixels vpp=1,…,P on a circle of radius R with the center at position x.LBMP uses the basic Local Binary Pattern (LBP) to extract both dynamic and appearance features of dynamic texture of candidate smoke region. We choose one search window of the size 5 × 5.Wk represents the search window in the current frame fk, and P0 is its central pixel. Let LBPk(P0) denote the LBP descriptor of the pixel P0 computed by (8); LBPk+1(Px) denote the LBP descriptor of one arbitrary pixel Px in the search window Wt+1 of adjacent frame ft+1. The matching point can be found by the formula(9)LBMPi,j,k=minx⁡LBPP0-LBPPx,where i ranges from 1 to 25 due to one search window of the size 5 × 5. For one block of the size 8 × 8, the dynamic texture is defined as(10)DY3=∑i,j,kδLBMPi,j,k,i,where δ(LBMP(i,j,k),i) is Kronecker’s delta that is defined as (11)δi,j=1,i=j,0,i≠j.


## 2.1. Spatiotemporal Consistency Feature

When smoke diffuses in the scene, it covers part of the scene. The edges of one smoke region are blurred and high frequency information slowly changes. LettingEBG(bm) and EI(bm), respectively, denote the high frequency energy of corresponding background and current frame with the block bm, then square difference of high frequency energy is defined as(1)ΔE2=EIbm-EBGbm2.A smoke block is detected as candidate smoke block according to the following condition:(2)ΔE2≥ΔE2¯,where ΔE2¯ represents the mean of the square difference ΔE2 of the previous t-1 frames.For a smoke block, its spatially and temporally neighboring blocks have greater possibility of being smoke blocks. We count the number of neighboring smoke blocks of the blockbm of the video frame at the time t:(3)DY1t=∑i=t-10tNcandi,where Ncand(i) denote the number of 8-neighboring smoke blocks at the corresponding position bm of the video frame at the time i. Figure 2 shows the calculation process of the spatiotemporal consistency energy.Figure 2
The calculation process of the spatiotemporal consistency energy.


## 2.2. Flutter Feature

As we know, there is massive heavy fog in the forest, so heavy fog can easily cause false alarm. Fortunately, forest fire smoke has one characteristic that is different from heavy fog. Smoke moves from bottom to top because heat smoke has a lower density than air, which is significantly different from heavy fog. The motion direction of one block is mapped as a direction code by computing the centroid motion of one candidate smoke block. Thep+q order moment of one candidate smoke block is computed by the following formula:(4)mpq=∑i=WsWe∑j=HsHeipjqIi,j,where Ws and We denote starting and ending point of one smoke region width, respectively, and Hs and He denote starting and ending point of the region height. The centroid coordinates of one candidate smoke block are defined as(5)Cxt,k=m10m00,Cyt,k=m01m00,k=1,2,…,N,where N=Frate, Frate denotes one frame rate. Cx(t,k) and Cy(t,k) denote horizontal ordinate and vertical ordinate of the centroid of the kth candidate smoke block in tth time window. The moving direction angle of the kth candidate smoke block(6)θtk=arccos⁡Cxt,kCx2t,k+Cy2t,k.Flutter direction angle is defined as(7)θt=∑k=1Frateθtk,if∑k=1Frateθtk≥0,2π-∑k=1Frateθtk,otherwise.


## 2.3. Dynamic Texture Feature

The Local Binary Pattern (LBP) is a powerful means of texture description. The operator labels the pixels of an image region by thresholding the neighborhood of each pixel with the center value and considering the result as a binary number (binary pattern). The classical definition of LBP can be represented as follows:(8)LBPP,Rx=∑p=1PsIvp-Ix2P-1,su=1u≥00u<0,where I(x) corresponds to the gray value of the pixel at position x and I(vp)p=1,…,P to gray values of P equally spaced pixels vpp=1,…,P on a circle of radius R with the center at position x.LBMP uses the basic Local Binary Pattern (LBP) to extract both dynamic and appearance features of dynamic texture of candidate smoke region. We choose one search window of the size 5 × 5.Wk represents the search window in the current frame fk, and P0 is its central pixel. Let LBPk(P0) denote the LBP descriptor of the pixel P0 computed by (8); LBPk+1(Px) denote the LBP descriptor of one arbitrary pixel Px in the search window Wt+1 of adjacent frame ft+1. The matching point can be found by the formula(9)LBMPi,j,k=minx⁡LBPP0-LBPPx,where i ranges from 1 to 25 due to one search window of the size 5 × 5. For one block of the size 8 × 8, the dynamic texture is defined as(10)DY3=∑i,j,kδLBMPi,j,k,i,where δ(LBMP(i,j,k),i) is Kronecker’s delta that is defined as (11)δi,j=1,i=j,0,i≠j.


## 3. An Adaboost Approach for Classification

### 3.1. Block-Based Statistic Characteristics

One image is divided into regular blocksbm of the fixed size M×M. Block-based statistic characteristics are computed as [9](12)SCb1=1n∑k=1nCbi,SCb2=∑k=1nCbin-∑k=n+1NCbiN-n,where N is the number of blocks and 1≤n≤N and Cb(i) represents moving direction MC or energy lowering ratio ELR or color feature CF of the ith block.


### 3.2. Adaboost Algorithm

Adaboost is superior to tradition neural networks on learning abilities and is applied in many fields of image processing, such as car license plate detection and face recognition. Adaboost algorithm is a training procedure for a collection of weak classifiers [10]. If the weak classifiers have the success rate about 0.5, they are boosted by suitable voting process to obtain a strong classifier. Because the collection of fire smoke videos is relatively difficult, the number of negative samples is much more than positive [11].Let the set of training samples be{(x1,y1),(x2,y2),…,(xi,yi),…,(xn,yn)}, where xi denotes training sample and yi∈{0,1} denotes positive sample and negative sample, respectively. Suppose m and l denote the number of positive samples and the number of negative samples, respectively. The implementation details are as follows.Step 1.
The weight of theith sample xi is initialized by formula (14):(13)W1i=1m,yi=1,1l,yi=0.Step 2.
Then, the weight of weak predictor is computed as follows:(14)at=12ln⁡1-etet,where et denotes the sum of predictive errors.Step 3.
Supposinggti, i=1,2,…,n, denote the predictive sequence, the iterative formula of the weight of training sample is computed by(15)Wt+1i=Wticte-at,gxi=yi,eat,gxi≠yi,where ct denote the normalization factor.Step 4.
Letr denote the number of weak predictors. All the weak classifiers form a strong classifier using the following formula:(16)hxi=1,βthtxi≥Th,0,otherwise.


## 3.1. Block-Based Statistic Characteristics

One image is divided into regular blocksbm of the fixed size M×M. Block-based statistic characteristics are computed as [9](12)SCb1=1n∑k=1nCbi,SCb2=∑k=1nCbin-∑k=n+1NCbiN-n,where N is the number of blocks and 1≤n≤N and Cb(i) represents moving direction MC or energy lowering ratio ELR or color feature CF of the ith block.


## 3.2. Adaboost Algorithm

Adaboost is superior to tradition neural networks on learning abilities and is applied in many fields of image processing, such as car license plate detection and face recognition. Adaboost algorithm is a training procedure for a collection of weak classifiers [10]. If the weak classifiers have the success rate about 0.5, they are boosted by suitable voting process to obtain a strong classifier. Because the collection of fire smoke videos is relatively difficult, the number of negative samples is much more than positive [11].Let the set of training samples be{(x1,y1),(x2,y2),…,(xi,yi),…,(xn,yn)}, where xi denotes training sample and yi∈{0,1} denotes positive sample and negative sample, respectively. Suppose m and l denote the number of positive samples and the number of negative samples, respectively. The implementation details are as follows.Step 1.
The weight of theith sample xi is initialized by formula (14):(13)W1i=1m,yi=1,1l,yi=0.Step 2.
Then, the weight of weak predictor is computed as follows:(14)at=12ln⁡1-etet,where et denotes the sum of predictive errors.Step 3.
Supposinggti, i=1,2,…,n, denote the predictive sequence, the iterative formula of the weight of training sample is computed by(15)Wt+1i=Wticte-at,gxi=yi,eat,gxi≠yi,where ct denote the normalization factor.Step 4.
Letr denote the number of weak predictors. All the weak classifiers form a strong classifier using the following formula:(16)hxi=1,βthtxi≥Th,0,otherwise.


## 4. Experimental Results

The proposed method is tested on 23 video clips of varying length and scene, including 13 positive samples and 10 negative samples, which are generally processed around 20 fps. 12 video clips (7 positive videos and 5 negative videos) are randomly chosen for training; the remaining 11 clips are used for testing. One part of the dataset is publicly available at websitehttp://imagelab.ing.unimore.it/visor/ and another part of dataset is recorded by ourselves.


### 4.1. Segmentation of Smoke Regions

Smoke region segmentation is key part for extracting effective features of smoke region. Figure3 shows the results of smoke region segmentation based on Kalman filter. In Figure 3, red points denote the centroids of smoke regions.Figure 3
The results of smoke region segmentation.
(a)
Original images
(b)
Segmented smoke regions


### 4.2. Centroid Motion of Smoke Regions

Figure4 shows centroid motion direction of smoke regions in one video clip. As we see, smoke often moves from bottom to top, which is significantly different from heavy fog.Figure 4
Centroid motion direction of smoke regions.


### 4.3. The Results of Smoke Detection

Figure5 shows the results of several frames in three smoke videos from different scenes. Figures 5(a) and 5(b) show the results of the videos that are recorded by the camera from close distance and far distance, respectively. Figure 5(c) shows the results of the video that includes obviously moving objects.Figure 5
The results of several frames in three smoke videos from different scenes.
(a)
The video recorded by the camera from close distance
(b)
The video recorded by the camera from far distance
(c)
The video that includes obviously moving objects


### 4.4. Performance of Smoke Detection

A smoke strong classifier of four-layer cascaded architecture is established, in which the numbers of weak classifiers in each layer are 1, 26, 69, and 126. Cost factors of each layer are 60, 6, 4, and 2, respectively. To validate the performance of our method, this section compares the proposed smoke detection method with other two methods: smoke detection using image energy and color information in [5] (EN-CI for short) and smoke detection method based on mixed Gaussian model and wavelet transformation in [12] (MGM-WT for short). We evaluate the performance of smoke detection method by computing true positive rate (TPR) and true negative rate TNR which are, respectively, defined by(17)TPR=Number of TP framesNumber of TP frames  +  Number of FN frames,TNR=Number of TN framesNumber of TN frames  +  Number of FP frames.Figure 6 shows the performance of the three methods, including the proposed one, in terms of TPR and TNR, where different datasets were used. Both TPR and TNR of our method are higher than the other two methods, as shown in Figure 6. It indicates that dynamic features extracted by the proposed method are effective in discriminating between smoke and nonsmoke video.Figure 6
Experimental results of different smoke detection methods in terms of TPR and TNR of each testing video.
(a)
TPR of positive testing videos
(b)
TNR of negative testing videos


## 4.1. Segmentation of Smoke Regions

Smoke region segmentation is key part for extracting effective features of smoke region. Figure3 shows the results of smoke region segmentation based on Kalman filter. In Figure 3, red points denote the centroids of smoke regions.Figure 3
The results of smoke region segmentation.
(a)
Original images
(b)
Segmented smoke regions


## 4.2. Centroid Motion of Smoke Regions

Figure4 shows centroid motion direction of smoke regions in one video clip. As we see, smoke often moves from bottom to top, which is significantly different from heavy fog.Figure 4
Centroid motion direction of smoke regions.


## 4.3. The Results of Smoke Detection

Figure5 shows the results of several frames in three smoke videos from different scenes. Figures 5(a) and 5(b) show the results of the videos that are recorded by the camera from close distance and far distance, respectively. Figure 5(c) shows the results of the video that includes obviously moving objects.Figure 5
The results of several frames in three smoke videos from different scenes.
(a)
The video recorded by the camera from close distance
(b)
The video recorded by the camera from far distance
(c)
The video that includes obviously moving objects


## 4.4. Performance of Smoke Detection

A smoke strong classifier of four-layer cascaded architecture is established, in which the numbers of weak classifiers in each layer are 1, 26, 69, and 126. Cost factors of each layer are 60, 6, 4, and 2, respectively. To validate the performance of our method, this section compares the proposed smoke detection method with other two methods: smoke detection using image energy and color information in [5] (EN-CI for short) and smoke detection method based on mixed Gaussian model and wavelet transformation in [12] (MGM-WT for short). We evaluate the performance of smoke detection method by computing true positive rate (TPR) and true negative rate TNR which are, respectively, defined by(17)TPR=Number of TP framesNumber of TP frames  +  Number of FN frames,TNR=Number of TN framesNumber of TN frames  +  Number of FP frames.Figure 6 shows the performance of the three methods, including the proposed one, in terms of TPR and TNR, where different datasets were used. Both TPR and TNR of our method are higher than the other two methods, as shown in Figure 6. It indicates that dynamic features extracted by the proposed method are effective in discriminating between smoke and nonsmoke video.Figure 6
Experimental results of different smoke detection methods in terms of TPR and TNR of each testing video.
(a)
TPR of positive testing videos
(b)
TNR of negative testing videos


## 5. Conclusions

In this paper, a novel smoke detection scheme using spatiotemporal and dynamic texture features is proposed. Three dynamic features, spatiotemporal energy, flutter features, and dynamic texture feature, are extracted for recognizing forest fire smoke video effectively and efficiently. The experimental results show the proposed approach provides higher accuracy of wildfire flame detection with comparable computational time. It is worth noting that there are several potential works for future development. One is to improve background model to effectively detect candidate smoke region in smoke-like scene; another is to extract more effective features for completely excluding the interference of heavy fog.



---

*Source: 706187-2015-07-02.xml*