# Real-Time Information Fusion System Implementation Based on ARM-Based FPGA
## 1 Introduction  
Augmented reality (AR) in large-area direct-view transparent displays is a type of information fusion system. This type of information fusion system can be used by an observer to interact with and directly access the corresponding information of their real-world target of interest. Such systems can be applied in retail, manufacturing, medical applications, service industries, etc. [1; 2; 3; 4] and are thus expected to be implemented in various fields in the near future. These systems can merge information by considering the relationships among the screen, background object, and the user's gaze direction to overlap the corresponding virtual information on a screen. Moreover, direct-view displays are more user-friendly and intuitive for users. As a result, research on the development and applications of direct-view transparent displays, which are considered an indispensable part of smart life applications, has gained traction [5; 6]. Such systems rely on artificial intelligence (AI) recognition technology to record the object and the user's gaze direction. Thus, the computing power capacity as well as the tradeoff between computing power and power consumption are the most critical issues for these systems. In AI recognition technology, GPU-based computers are primarily employed to establish and train deep learning models. However, GPU-based computers are not suitable for performing models in some field applications. Field-programmable gate array (FPGA)-based embedded systems, whose speed, power consumption, size, and flexibility surpass those of GPU-based computers, are advantageous for real-time deep-learning identification and classification models. FPGAs, consisting of programmable logic (PL) units, built-in memory, and floating point units, perform high-speed data exchange with peripheral hardware. As a result, a deep learning model can be run on the FPGA entity in a chip form [7]. The execution speed can be more than three to five times that of GPUs, and the average power consumption is also lower than that of GPUs. Therefore, FPGAs are more suitable for deep model identification or classification in prediction or inference extraction operations [8]. The implementation and performance summary of FPGA-based accelerators [9] has two discussion directions for accelerating convolutional neural network (CNN) model computations using FPGAs. The first approach is to compress the CNN model. Reducing memory footprint is realized by singular value decomposition, whereas filter clustering methods [10] simplify the connection numbers of convolution or fully connected layers through network pruning techniques [11; 12; 13] to reduce model complexity and prevent over-fitting. The method of enforcing weight sharing [14] is also used to reduce memory footprint. The other direction focuses on implementing CNN based on the architecture of FPGA hardware, including the hardware parallelization of data types and arithmetic units. In deep learning model computations, addition and multiplication operations are required for convolution, pooling, and fully connected layers. Typically, floating-point arithmetic is used to avoid precision sacrifice. However, the number of floating-point arithmetic units inside an FPGA is limited, whereas the amount of data to be temporarily stored is substantial, depending on the number of layers and parameters in the model. To date, numerous ways have been explored to reduce the number of bits or data operations without sacrificing precision. In previous studies [15; 16; 17], fixed-point formats, with 8 to 16 bits as the basic arithmetic format, were used to improve the data computation speed and reduce the amount of data without reducing prediction accuracy. However, this method requires additional optimization and adjustment or the addition of floating-point arithmetic in the data conversion step. VITIS AI is a development platform from Xilinx which is designed for drawing AI-based inferences on Xilinx hardware platforms.