# A Video-Based Fire Detection Using Deep Learning Models
## 3 Proposed Method
### The Spatial Features Extraction  
The coordinates of the bounding boxes are projected on the \(n\times n\times d\) activation map to extract the spatial features. Here, we extract them in the last layer of CNN, and \(n=14\), and \(d=1024\) when ResNet 101 is used as a base net. For the projected region, we take a scalar feature by taking a weighted average over each feature map. Note that the feature is extracted from the bounding boxes SRoFs, including flames and smoke, and non-fire objects, so that it may give the pure spatial features of diverse types of fire and non-fire objects. Figure 5 shows the part of this feature extraction in our proposed method, including the Faster R-CNN object detection model.  
The Faster R-CNN can provide more than one SRoF or non-fire object because an image can contain several bounding boxes. Faster R-CNN can detect multiple objects in a frame, where each object is enclosed by a bounding box with its class score and can intersect with each other. Therefore, we should carefully investigate the multiple areas to further consider the temporal behavior of a fire or non-fire object. Figure 6 shows the case where there are several SRoFs.  
In our proposed method, a weighted Global Average Pooling (GAP) scheme is adapted to extract the spatial features. After insignificant bounding boxes are filtered out by thresholding their own confidence score, the significant SRoFs and non-fire objects are selected. The image which does not contain any specific small bounding box is treated as a non-fire object whose bounding box covers the whole image with confidence score 1. Note that the significant SRoF or the non-fire object has its own confidence score which can be used to take the weighted GAP. Figure 5 shows the process to extract the spatial features from the last layer of CNN of the Faster R-CNN object detector with \(d\) feature maps, where \(d=1024\). From each feature map \(f_{i}\), the scalar feature value is determined as follows:  
Figure 5: The spatial feature extraction from Faster R-CNN.  
Figure 6: Frames with multiple suspected regions of fire (SRoFs). (**a**) Shows the case that the predicted flame and smoke area intersect, and (**b**) shows the case that the predicted flame and smoke area do not intersect.  
\[v_{i}=\frac{1}{Z}\sum\begin{array}{c}\text{for each bounding box $b$ of SRoFs}\\ \text{or non $-fire$ objects}\end{array}\quad w_{b}\cdot\sum\
olimits_{(x,y)\in b}f_{i}(x,y), \tag{1}\]  
where,  
\[Z=\sum\begin{array}{c}\text{for each bounding box $b$ of SRoFs}\\ \text{or non $-fire$ objects}\end{array}\quad w_{b}\cdot\sum\
olimits_{(x,y)\in b}1. \tag{2}\]  
The vector \(\mathbf{v}=(v_{1},\,v_{2},\,\dots,v_{d})\) represents the aggregated spatial feature for SRoFs or non-fire objects detected by Faster R-CNN in an image or a frame of a video. In general, the prominent features among \(d\) can be found projecting the bounding box of SRoFs or non-fire objects on the feature map similar to the class activation map [23]. Because they are merely spatial features which do not contain temporal information, the feature selection in our proposed method is transferred to the following LSTM stage of the temporal aggregation in a short-term.