# Real-Time Information Fusion System Implementation Based on ARM-Based FPGA
## 2 Information Fusion Method  
The information fusion system can generate an image in the line of sight of an observer that sees the image overlapped on their exhibit of interest. The side view of the information fusion system is shown in Figure 2. The system detects the positions of the observer and exhibit relative to that of the transparent display. Thus, the system employs two depth-color cameras to capture the color and depth images of the observer and exhibit. The system receives these images and feeds them into neural network models, which locate the observer and exhibit in the color images. The identified positions of the observer and exhibit in the color images are then used to determine their positions in the depth image corresponding to the color images. Thus, the positions of the observer and exhibit in the real world can be derived by mapping their positions in the image coordinates from the real world coordinates. Subsequently, the system calculates the displayed position and size of the fusion information according to the relative positions of the observer and exhibit. Finally, the system displays the fusion information on a transparent display to provide additional details for the exhibit. The workflow of the information fusion system is illustrated in Figure 3.  
The system performs two essential processes: recognition and position acquisition of the fusion information. In the position-acquisition process, the system performs image registration between the depth and color images. The depth-to-color image coordinate maps are derived using Equations (1)-(3), shown below:  
\[ \begin{bmatrix}X^{D}\\ Y^{D}\\ Z^{D}\end{bmatrix} =d \begin{bmatrix}f^{D}_{x}&0&p^{D}_{x}\\ 0&f^{D}_{y}&p^{D}_{x}\\ 0&0&1\end{bmatrix} ^{-1} \begin{bmatrix}u^{D}\\ v^{D}\\ 1\end{bmatrix}  \tag{1}\]  
\[ \begin{bmatrix}X^{C}\\ Y^{C}\\ Z^{C}\end{bmatrix} = \begin{bmatrix}r^{DC}_{DC}&r^{DC}_{DC}&r^{DC}_{3}&r^{DC}_{ DC}&t^{DC}_{DC}\\ r^{DC}_{3}&r^{DC}_{3}&r^{DC}_{3}&t^{DC}_{2}\\ r^{DC}_{31}&r^{DC}_{32}&r^{DC}_{33}&t^{DC}_{3}\\ 0&0&0&1\end{bmatrix}\begin{bmatrix}X^{D}\\ Y^{D}\\ Z^{D}\end{bmatrix}  \tag{2}\]  
\[Z^{C} \begin{bmatrix}u^{C}\\ v^{C}\\ 1\end{bmatrix} = \begin{bmatrix}f^{C}_{x}&0&p^{C}_{x}\\ 0&f^{C}_{y}&p^{C}_{y}\\ 0&0&1\end{bmatrix}\begin{bmatrix}X^{C}\\ Y^{C}\\ Z^{C}\end{bmatrix}  \tag{3}\]  
Figure 1: Devices used in the information fusion system. Equation (1) is the transformation from the two-dimensional depth image coordinate (\(u^{D},v^{D}\)) to the three-dimensional depth camera coordinate (\(X^{D}\), \(Y^{D}\), \(Z^{D}\)). \(f_{x}{}^{D}\) and \(f_{y}{}^{D}\) are the focal lengths of the depth camera; \(p_{x}{}^{D}\) and \(p_{y}{}^{D}\) are the central positions of the depth image; \(d\) is the depth data of the depth image. Equation (2) is the transformation from the depth camera coordinate to the three-dimensional color camera coordinate (\(X^{C}\), \(Y^{C}\), \(Z^{C}\)). \(r_{11}{}^{DC}\) to \(r_{3}{}^{DC}\) are the rotation parameters, and \(t_{1}{}^{DC}\) to \(t_{3}{}^{DC}\) are the translation parameters. Equation (3) is the transformation from the color camera coordinate to the two-dimensional color image coordinate (\(u^{C}\),\(v^{C}\)). After detecting the observer or exhibit in the color image, the position of observer or exhibit in the three-dimensional color camera coordinate can be derived from the inverse transformation of Equation (3). Next, the positions are mapped to the three-dimensional transparent display coordinate (\(X^{M}\), \(Y^{M}\), \(Z^{M}\)) according to Equation (4), expressed below:  
\[ \begin{bmatrix}X^{M}\\ Y^{M}\\ Z^{M}\end{bmatrix} = \begin{bmatrix}r_{11}^{CM}&r_{12}^{CM}&r_{13}^{CM}&t_{13}^{ CM}\\ r_{21}^{CM}&r_{22}^{CM}&r_{23}^{CM}&t_{23}^{CM}\\ r_{31}^{CM}&r_{32}^{CM}&r_{33}^{CM}&t_{33}^{CM}\\ 0&0&1\end{bmatrix}\begin{bmatrix}X^{C}\\ Y^{C}\\ Z^{C}\\ 1\end{bmatrix}  \tag{4}\]  
Here, \(r_{11}{}^{CM}\) to \(r_{33}{}^{CM}\) are the rotation parameters, and \(t_{1}{}^{CM}\) to \(t_{3}{}^{CM}\) are the translation parameters.