# Research and Conceptual Design of Sensor Fusion for Object Detection in Dense Smoke Environments
## 1 Introduction
### Data Fusion
#### 2.4.2 Calibration between Thermal IR Camera and 3D Ultrasonic Sensor System or FMCW Radar System  
As shown in Figure 8, (\(x_{s}\), \(y_{s}\), \(z_{s}\)), (\(x_{r}\), \(y_{r}\), \(z_{r}\)), and (\(x_{c}\), \(y_{c}\), \(z_{c}\)) are 3D ultrasonic sensor system coordinates, FMCW radar coordinates, and thermal IR camera coordinates. These coordinates are denoted as the coordinate frames {S}, {R}, and {C}, respectively, and (\(u\), \(v\)) is the image plane coordinate system. Additionally, (\(p_{\text{xs}}\), \(p_{\text{ys}}\), \(p_{\text{zs}}\)) and (\(r_{x}\), \(r_{y}\), \(r_{z}\)) are the positions of near and far objects as measured by the 3D ultrasonic sensor system and the FMCW radar system, respectively. These positions should correspond to (\(u_{s}\), \(v_{s}\)) and (\(u_{r}\), \(v_{r}\)) detected by image processing of the thermal IR camera image for short- and long-range objects, respectively. In order to map (\(p_{\text{xs}}\), \(p_{\text{ys}}\), \(p_{\text{zs}}\)) to (\(u_{s}\), \(v_{s}\)), we use the transformation equation between {S} and (\(u\), \(v\)) as  
\[\omega_{s} \begin{bmatrix}u\\ v\\ 1\end{bmatrix} =\mathbf{H} \begin{bmatrix}x_{s}\\ y_{s}\\ z_{s}\end{bmatrix}  \tag{19}\]  
where \(\mathbf{H}=[h_{ij}]_{i,j}=1,2,3\) is a \(3\times 3\) homography matrix and \(\omega_{s}\) is an unknown constant. By estimating \(\mathbf{H}\), we determine the transformation between the ultrasonic plane (\(\Pi_{\text{s}}\)) and the image plane (\(\Pi_{\text{t}}\)). The \(\omega_{s}\) is an unknown parameter, but we can assume that the objects are equidistant in two planes, so \(\omega_{s}=z_{s}\). This means that \(\mathbf{H}\) is the affine homography matrix with \(h_{32}=h_{32}=0\) and \(h_{33}=1\). Then, we obtain two algebraic equations:  
\[ \begin{array}{l}u=\frac{h_{11}}{z_{s}}x_{s}+\frac{h_{12}}{z_{s}}y_{s}+h_{13 }\\ v=\frac{h_{21}}{z_{s}}x_{s}+\frac{h_{22}}{z_{s}}y_{s}+h_{23}\end{array}  \tag{20}\]  
Note that \(z_{s}\) is not a constant, but we can treat it as a constant on each image frame, especially for static objects. Thus, for Equation (20), we get the following homogeneous linear square problem:  
\[ \begin{array}{l}\mathbf{A}_{u}\mathbf{H}_{s}=0,\;\mathbf{A}_{u}\mathbf{H}_ {s}=0\\ \mathbf{A}_{u}=\left[\begin{array}{cccccccc}x_{s}&y_{s}&1&0&0&0&0&0&-u\\ 0&0&0&x_{s}&y_{s}&1&0&0&-v\end{array}\right]\\ \mathbf{H}_{s}=\left[\begin{array}{cccccccc}\frac{h_{11}}{z_{s}}&\frac{h_{2 }}{z_{s}}&h_{13}&\frac{h_{21}}{z_{s}}&\frac{h_{22}}{z_{s}}&h_{23}&0&0&1\end{array} \right]^{T}\end{array}  \tag{21}\]  
Then, using Singular Value Decomposition (SVD), we estimate the homogeneous matrix, \(\mathbf{H}\), in Equation (19). Figure 8: Sensor geometry and transformation. To match \((r_{x},r_{y},r_{z})\) to \((u_{r},v_{r})\), we use the same approach described in Equations (19) to (21). The corresponding radius \(R\) and angle \(\theta\) are converted to Cartesian coordinates using \(y_{r}\) = \(R\sin\theta\) and \(z_{r}\) = \(R\cos\theta\). Additionally, since all FMCW radar data comes from somewhere in the radar plane (\(\Pi_{\text{r}}\)), we can take \(x_{r}\) = 0. Hence, the transformation between \(\{\text{R}\}\) and \((u,v)\) becomes  
\[\omega_{r} \begin{bmatrix}u\\ v\\ 1\end{bmatrix} =\mathbf{M} \begin{bmatrix}y_{r}\\ z_{r}\\ 1\end{bmatrix}  \tag{22}\]  
where \(\mathbf{M}=[m_{ij}]_{i,j\ =1,2,3}\) is a \(3\times 3\) homography matrix and \(\omega_{r}\) is an unknown constant. By estimating the \(\mathbf{M}\), we determine the transformation between the radar plane (\(\Pi_{\text{r}}\)) and the image plane (\(\Pi_{\text{i}}\)). We simplify \(m_{32}=m_{32}=0\) and \(m_{33}=1\) by assuming the affine homography. Thus, \(\omega_{r}\) = 1, and we obtain the following two algebraic equations:  
\[ \begin{array}{l}u=m_{11}y_{r}+m_{12}z_{r}+m_{13}\\ v=m_{21}y_{r}+m_{22}z_{r}+m_{23}\end{array}  \tag{23}\]  
Thus, for Equation (23), we get the homogeneous linear square problem of  
\[