# Real-Time Information Fusion System Implementation Based on ARM-Based FPGA
## 3 Implementation  
The information fusion system mainly comprised a transparent display, two depth cameras, Realsense D435i, and a Xilinx Zynq UltraScale+ MPSoC ZCU102 development board. The depth camera provided depth and color images for object detection. The development board contained an ARM-based FPGA XCZU9 EG chip with a quad-core Arm(r) Cortex(r)-A53 (PS), dual-core Cortex-R5F real-time processors, Mali(tm)-400 MP2 graphics processing unit (MGPU), and a PL. The PS side received image data and sent images to the PL. Image registration between the color and depth images was performed on the PL instead of on the PS, because the image registration was faster on the PL than on the PS. The PL contained a deep learning processor unit (DPU) that executed the exhibit and observer identification. The image registration and detection results were fed back to the PS. Next, the PS calculated the size and position of the information displayed on the transparent display according to the coordinates and depth values of the observer's eyes and objects. The MGPU received the display information from the PS and updated the information to the transparent display. The computation architecture of the system is shown in Figure 5.  
The PS performed multi-thread tasks using a total of four threads, including DecodeThread, DupThread, SortingThread, and GuiThread. DecodeThread received the color and  
Figure 4: Schematic and coordinate definition of the information fusion system.  
Figure 5: Computation system block diagram.  
depth images and sent them to the PL for image registration. Then, DecodeThread received the image registration results and pushed the color images into a queue. DpuThread popped the images in the queue to the DPU for detecting the observer and exhibit and received the detection results, including the coordinates, sizes, and labels. SortingThread sorted the images in the queue according to the frame indices. GpuThread displayed the images at a specific position on the transparent display.  
The information fusion system employed the RetinaFace and YOLO models to detect the face of the observer and exhibit, respectively. The RetinaFace model was pretrained and supported by the VITIS AI, which could rapidly apply the RetinaFace model on the DPU. The YOLO model for detecting LEGOs was self-trained and converted into a DPU-executable xmodel format; the conversion flow is shown in Figure 6. The parameters of the trained YOLO model were modified, as shown in Table 1, to run it on the DPU. The size of the image sent as an input to the DPU was 512 \(\times\) 512 pixels. Moreover, because the DPU did not support the mish activation function, the leaky rectified linear unit activation function was used. The max-pooling sizes were reduced, because the maximum maxpool size of the DPU was limited to eight. Furthermore, the computing number format of the model was converted from floating-point to integer by the VITIS AI quantizer to increase the computation speed.  
The image registration function was programmed using C++, and the VITIS(tm) High-Level Synthesis tool was utilized to synthesize the function into the registration level (RTL) module for the PL implementation. The code is displayed in Table 1. The input image size is 640 \(\times\) 480 pixels, and the input parameter includes the intrinsic and intrinsic parameters of the color and depth cameras. Lines 10 to 19 in Table 1 describe the coordinate transformation from the depth camera coordinate to the color camera coordinate. Lines 20 and 21 in Table 1 describe the depth data alignment in the color camera coordinates.