# Plane Detection Using Appearance-Based Pre-Processing and Convolutional Neural Network
## 3 Experiment and Performance Analysis
### Experimental Setup and Training  
In the experimental environment, the CPU used the Intel i7-8700 processor, the GPU used the Geforce RTX 3070, and the OS was tested on Linux Ubuntu 18.04. Figure 7 shows the flame dataset training results of CNN, where Figure 7a is accuracy and Figure 7b is loss. The red curve is the dataset used for training among flame images collected by the training dataset, and shows the accuracy detected through this dataset. The blue curve is the test dataset which, unlike the training dataset, has not been used in the training process, and is a dataset used only for accuracy evaluation purposes. Therefore, if high accuracy and low loss are shown in the training dataset and relatively low accuracy and high loss in the test dataset, overfitting can be seen as occurring, and training has not progressed properly. However, the training results of this work show that both datasets have similarly high accuracy and low loss, which can be judged to be well-trained against flames. Additionally, the training was ended at 3000 steps, where the accuracy did not increase over a certain level and converged to some extent. Figure 8 shows the equipment used for the actual fire test. Figure 9 shows the results of the flame detection evaluation. For specific performance evaluation of the proposed flame detection algorithm, a fire test picture was taken, and pictures of the actual fire site in different environments were used: indoors, outdoors night, day, etc. This was composed of various evaluation pictures. The presented photos are some of the results of correct flame detection for the flame image, with the green-bound boxes indicating the areas finally determined to be flames in the CNN model. From the left, Figure 9a is the result of detection through the method presented in this study, Figure 9b is the result of detection using the Faster R-CNN, and Figure 9c is the result of detection using SSD. In the case of the model presented in this study, most of the flames were detected even when the size of the object occupied by the flame was small; the Faster R-CNN also correctly judged most flames, but in some cases, it incorrectly judged an object other than a flame. In the case of the SSD, it took less \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{**Training Dataset**} & \multicolumn{2}{c}{**Test Dataset**} \\ \hline
**Flame** & **Non-Flame** & **Flame** & **Non-Flame** \\ \hline
8152 & 8024 & 2001 & 2000 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of images in the dataset. Figure 7: Accuracy and loss curve of learning outcomes using CNN. (**a**) Accuracy curve of learning dataset and test dataset; and (**b**) loss curve of learning dataset and test dataset. computation time than the Faster R-CNN, but there were many cases where the object size was not detected in the picture. The test images comprised 100 fire and non-fire photographs, to evaluate the detection results with the following expressions for a specific and objective accuracy evaluation [30]. \[Accuracy=\frac{TP+TN}{TP+FN+FP+TN} \tag{10}\]  
\[Precision=\frac{TP}{TP+FP} \tag{11}\]  
\[F1-Score=2\times\frac{Precision\cdot Recall}{Precision+Recall} \tag{12}\]  
\[Recall=\frac{TP}{TP+FN} \tag{13}\]  
where \(TP\) is the correct detection of flames, \(FN\) is not detected from the image of flames, \(  
Figure 9: Flame detection results of the model presented in this study and two-object detection: (**a**) Flame detection results of the model presented in this study; (**b**) flame detection results using Faster R-CNN; and (**c**) flame detection results using SSD. Figures 10-12 show the receiver operating characteristic (ROC) curves and precision recall (PR) curves for specific accuracy comparisons of the three detection models. The ROC curves were expressed through two parameters, TPR (true positive rate) and FPR (false positive rate), and showed that the ROC curve changed by changing the threshold criteria.