# FPGA-QNN: Quantized Neural Network Hardware Acceleration on FPGAs
## 4 Definition of Models in FPGA-QNN
### FINN Framework  
Developed at Xilinx labs, FINN is a framework for building scalable and fast quantized networks on FPGAs. FINN is a powerful tool for synthesizing and implementing the deep learning model designed and trained using Pytorch and Brevitas on FPGAs. The main objective behind FINN-based accelerators is to perform a huge number of classifications per second, rendering it an ideal tool in real-time embedded practices. With the FINN framework, an accelerator for a deep learning model can be designed using four main groups of processing blocks, as presented in Figure 7. In the first phase, a deep learning network is designed and trained using Pytorch and Brevitas. The second step performs the FINN transformations on the model exported into ONNX format, making the network layers suitable for data flow. In the third stage, IP blocks are created on the FPGA using Vivado as well as importing bitstream files into the development environment. The final step includes the acceleration of the network by running the ZYNQ operating system.  
FINN has the potential to lower the convolutions to the multiplications of matrix-to-matrix. To achieve this goal, FINN employs two units: (1) a sliding window unit (SWU) and (2) a matrix vector threshold unit (MVTU). The SWU unit produces the image matrix from the input and weight matrices. The MVTU unit performs the actual matrix multiplication by varying the column vector of the image matrix continually. A typical MVTU comprises an array of processing elements (PEs), each of which includes a number of single-instruction multiple-data (SIMD) lanes. PE blocks, the most basic core of the FINN framework, perform the multiplication and accumulation operations of the input images converted into a data stream by the SWU. Networks with a quantization level of 1 bit use XNor instead of multiplication, while popcount is used for accumulation. The MVTU structure, consisting of an internal structure of PE blocks and PE SIMD blocks, can be seen in Figure 8.  
Figure 8: (**a**) Single processing element (PE) and (**b**) matrix–vector–threshold unit in FINN framework.  
Figure 7: The main body of the FINN framework with 4 steps.  
PE blocks perform as many parallel multiplication operations as the number of SIMDs. Increasing this number will result in more parallelism and faster speed, subject to an increased resource usage. The folding transformation is performed by determining the PE and SIMD parameters for each layer of the model. A tensor in FINN is associated with the dimension of NxHxWxC (batch number, height, width, channel). The number of channels (C) is equal to the multiplication of F (folding) and P (parallelization), setting the dimension of a folded tensor to NxHxWxCP. Another assumption is that PE should be an exact divisor of output attributes and SIMD is an exact divisor of input attributes. PE and SIMD divide the image and weight matrices into pieces and perform multiplication operations separately, with an example of four separate matrix multiplications presented in Figure 9.  
The heterogeneous flow architecture formed as a result of the folding process is the key performance indicator of the deep learning accelerator. This architecture increases the number of frames to be processed per second by parallelizing feedforward deep learning processes pipelined on the FPGA for each block. Thus, the developed hardware can run deep learning networks much faster than sequential processing hardware such as CPUs.