# Research and Conceptual Design of Sensor Fusion for Object Detection in Dense Smoke Environments
## 1 Introduction
### Thermal IR Image Processing: Object Detection  
Most indoor fire sites are composed of typical structures such as walls, stairs, and floors, as well as atypical objects such as various pieces of furniture and broken or burnt objects. Therefore, thermal IR images taken at the sites show a mixture of all objects indoors.  
Appropriate image processing needs to be applied to these images (called query images) to distinguish between objects that may be obstacles and those that are not. Figure 6 shows an image processing overview for this distinction. This processing is based on the object detection method using a basic Canny algorithm and Hough transformation.  
After obtaining the query image from image sequences of the thermal IR camera, calibration and rectification are applied to the image for pre-processing of edge detection. The pre-processed image is utilized to detect edges using the Canny algorithm. We chose the Canny algorithm for the following reasons [37; 38]:  
Figure 5: The angle of view and detection area of the ultrasonic sensor system: (**a,b**) are side and front views, respectively, of a single 3D ultrasonic sensor module, (**c,d**) are side and top cross-sectional views, respectively, of multiple 3D ultrasonic sensor modules.  
1 The maximum number of edges can be detected.
2 It gives very good results for detecting horizontal and vertical edges.
3 Circular and corner edges can be detected.
4 Good localization.
5 Only one response to a single edge.  
Edge images are used to establish obstacle boundaries and planes for walls, stairs, and floors. By using Hough transformation to extract line features from these edge images, we classify vertical and horizontal lines as features of walls, stairs, and floors. After this, for object detection, we collect groups of objects such as walls, stairs, and floors using the image correlation technique of template matching.  
Image correlation using template matching is detailed in ref. [39]. The correlation is processed by calculating a correlation coefficient. The correlation coefficient is obtained by moving a filter mask over the query and template images and calculating the sum of the products at each location. The maximum value of the correlation coefficient occurs when regions of the template and the query images are identical. This is called the maximum correlation, i.e., the best possible match. On the other hand, minimum correlation occurs when the regions of the template and the query images are most dissimilar.  
After this template matching, everything except walls, stairs, and floors is considered an object. From these objects, small objects are discarded, and large objects are identified as obstacles because they can interfere with the heading path of robots or firefighters. Finally, a display shows the thermal IR images with 2D locations (\(u,v\)) and the size box of an object (\(w\), \(h\)) overlay.  
Note that for better computational cost, we use a relatively simple object detection algorithm (template matching algorithm) rather than a complex one. To use a template matching algorithm, we assume that we have obtained prior (or reference) data by pre-scanning the designated fire hazard area (or the other normal conditions inside buildings) with various types of walls, stairs, and floors. In this case, the template matching algorithm can work more effectively because the performance of the algorithm depends on how much data can be obtained. Nevertheless, if prior data do not fit appropriately in an environment, a traditional method such as SIFT [40] or a deep learning model such as YOLO [41] can be used instead of a template matching algorithm because these algorithms can provide better performance. However, detailed discussions about these algorithms themselves are outside the scope of this paper.