# FPGA architecture-based front-end processing for SLAM applications

Imad El Bouazzaoui, Sergio Rodríguez Flórez, Abdelhafid El Ouardi

► To cite this version:

Imad El Bouazzaoui, Sergio Rodríguez Flórez, Abdelhafid El Ouardi. FPGA architecture-based front-end processing for SLAM applications. Journal of Real-Time Image Processing, 2025, 22 (2), pp.73. 10.1007/s11554-025-01650-w . hal-04980724

HAL Id: hal-04980724

https://hal.science/hal-04980724v1

Submitted on 6 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FPGA architecture-based front-end processing for SLAM applications

Imad El Bouazzaoui<fnref n="1" /> Ⓒ · Sergio Rodríguez Flórez<fnref n="1" /> Ⓒ · Abdelhafid El Ouardi<fnref n="1" /> Ⓒ

Received: 16 February 2024 / Accepted: 15 February 2025
© The Author(s) 2025

## Abstract

Simultaneous Localization and Mapping is intended for robotic and autonomous vehicle applications. These targets require an optimal embedded implementation that respects real-time constraints, limited hardware resources, and energy consumption. SLAM algorithms are computationally intensive to run on embedded targets, and often, the algorithms are deployed on CPUs or CPU-GPGPU architectures. With the growth of embedded heterogeneous computing systems, research work is increasingly interested in the algorithm-architecture mapping of existing SLAM algorithms. The latest trend is pushing processing closer to the sensor. FPGAs constitute the perfect architecture for designing smart sensors by providing low latency suitable for real-time applications, such as video streaming, as they supply data directly into the FPGA without needing a CPU. In this work, we propose the implementation of the HOOFR-SLAM front end on a CPU-FPGA architecture, including both feature extraction and matching processing blocks. A high-level synthesis (HLS) approach based on OpenCL paradigm has been used to design a new system architecture. The performance of the FPGA-based architecture was compared to a high-performance CPU. This innovative architecture delivers superior performance compared to existing state-of-the-art systems.

**Keywords** Image processing · Features' extraction and matching · FPGA implementation · Embedded systems

## 1 Introduction

Feature-based SLAM systems are becoming increasingly popular due to their performance and robustness. Several feature extractors are used in various SLAM systems, such as ORB [1], SIFT [2], and SURF [3]. Although these extractors yield good matching results, the computational complexity of feature extraction and matching is a significant hurdle when embedding such SLAM algorithms on low-power architectures. Recently, Nguyen et al. [4] proposed an FPGA implementation of the HOOFR extractor while maintaining the same accuracy. However, the matching task remains the most time-consuming task in the processing flow. The design of an accelerated architecture for this functional block is mandatory to achieve on-the-fly processing on

a system-on-chip. Our challenge is to boost the algorithm's performance on low-power architectures to ensure on-the-fly processing. FPGAs are considered the best choice for stream processing. Compared to GPUs, which only provides parallelism for data processing and acceleration, FPGAs can provide pipeline parallelism and on-the-fly data processing which makes them more suitable for stream processing [5, 6], especially for embedded systems.

We achieve our objective through an algorithm architecture mapping applied to CPU-FPGA architectures. In practice, an algorithm is broken down into functional blocks, and each block is assigned to the appropriate processing unit, ensuring optimal performance. In this paper, we evaluate the performance of the matching block on different architectures, since it is the bottleneck of performance, concluding with the proposition of an optimal CPU-FPGA mapping for the RGB-D HOOFR-SLAM front-end.<fnref n="1" />

<figure><img src="image_2.png" /></figure>

Sergio Rodríguez Flórez
sergio.rodriguez@universite-paris-saclay.fr

Imad El Bouazzaoui
imad.elbouazzaoui.ee@gmail.com

Abdelhafid El Ouardi
abdelhafid.elouardi@universite-paris-saclay.fr

<footnote n="1">¹ Université Paris-Saclay, ENS Paris-Saclay, CNRS, SATIE, Gif-sur-Yvette 91190, France</footnote>

<footnote n="1">¹ The source code is available at https://github.com/Imel23/FPGA-Based-HOOFR-Front-End.</footnote>
# 2 Related works

The visual SLAM comprises two main blocks: a front-end block that takes charge of all the data processing and poses calculation and a back-end block that optimizes the map and the trajectory. There are two approaches which aim at embedding complex algorithms on dedicated architectures in the literature. The first focuses on embedding the front-end processing [4, 7, 8], and the second is concerned with the back-end [9, 10]. To bring the processing as near as possible to the sensor, we will focus on works performed on the SLAM front-end.

## 2.1 CPU-GPU-based SLAM

The CPU-GPU architectures are widely used in robotics, especially in computer vision, since a GPU can offer many cores for parallel Single Instruction, Multiple Data (SIMD) processing. Based on DTAM [11], Ondrúška et al. [12] exploited the GPU of various mobile phones to implement a pipeline that creates a connected 3D surface model directly on the device in real time. They assigned sequential tasks, including keyframe selection and dense camera alignment, to the CPU, as the camera alignment requires an accumulation of errors across the entire input image. Also, they used SIMD instructions which led to a processing of 4 pixels at a time. On the other hand, GPU was run in parallel carrying stereo depth computation, model update, and raycasting. Even though this architecture allows volumetric surface reconstruction and dense 6DoF camera tracking in real time, the GPU hardware constraints limit the voxel resolution. In the category of indirect approaches, Aldegheri et al. [13] modified ORB-SLAM2 to operate on the Nvidia Jetson TX2 in real time. Besides the current parallelization of the algorithm on CPU [Parallel PThreads on shared-memory multi-core CPUs and automatic parallel implementation (i.e., through OpenMP directives) of the bundle adjustment sub-block], the authors have added two layers of parallelism. The first one consists of a parallel implementation of the tracking sub-blocks on GPU. The second is the implementation of an 8-stage pipeline of such sub-blocks. The acceleration targets the feature extraction block as it is the bottleneck of the processing flow. For this purpose, they modeled the extraction block with a directed acyclic graph (DAG), adopting the OpenVX standard. Ma et al. [14] presented a front-end processing parallelization of the ORB-SLAM2 algorithm on the Jetson TX2. As front-end processing consumes more than half of computing resources and operates on images, so this part is well suitable for parallelization. The parallelization involves the feature extractor and the

local point selection. This latter is used to reduce the input data of the matching block. The feature extraction parallelization involves constructing the Gaussian pyramid of the image on the GPU, then feature detection, orientation calculation, and description are performed on the GPU. Because of the asynchronous operation of CPU and GPU, task and thread allocation were adjusted to decrease the idle time of the GPU and to increase the usage of the streaming multiprocessor (SM). Nguyen et al. [7] found that the features' matching block has a high computational cost and low data dependence, so they proposed to parallelize the HOOFR-SLAM feature matching block on GPU. On the other hand, they employed OpenMP to implement HOOFR feature extraction for two reasons: first, the extractor uses FAST detection, where a pixel can be rejected after one or two pixel tests, leading to a difference in processing cost for each pixel, so the GPU computation resources are not well used due to unbalanced complexity. Second, the Hessian filtering is much more rapid on CPU thanks to the binary classification, which needs a dynamic memory allocation that is not supported on GPU. The performance obtained on the Jetson TX1 is real time if we consider the KITTI [15] dataset's acquisition rate. Although these GPU implementations provide real-time processing, the energy consumption of such architecture is still a challenge for robotics and autonomous vehicle applications, where autonomy is a crucial asset.

## 2.2 CPU-FPGA-based SLAM

Several works have recently focused on implementing visual SLAM algorithms on embedded architectures. In particular, indirect-based algorithms are gaining attention due to their low complexity and ease of parallelization. Much effort has been devoted to accelerating the most time-consuming part of the algorithm, including feature extraction and matching. Fang et al. [16] proposed a design of ORB feature extractor, since it is the bottleneck of performance and energy consumption. They implemented the extractor on an Altera Stratix V FPGA and achieved 67 frames per second using VGA-resolution images. The design runs at 203 MHz frequency, reducing the energy consumption. Liu et al. [17] presented eSLAM, an implementation end-to-end ORB-based SLAM system on a SoC. The authors accelerated the extraction part and extended the hardware implementation to the matching block. First, the descriptor was reformulated into a rotationally symmetric pattern to simplify the hardware implementation. Then, the extractor and the matching blocks were implemented as a pipeline operating in two modes. The extractor and matching are launched in parallel with the pose estimation and optimization in normal frame mode. In the keyframes mode, feature extraction runs in parallel with pose estimation and optimization.

<figure><img src="image_3.png" /></figure>
The implementation has been conducted on a Xilinx Zynq XCZ7045 SoC architecture. The algorithm was evaluated on the TUM dataset [18], achieving a frame rate of 55 FPS for normal frames and 31 FPS for keyframes with a power consumption of 1.936 W. Vemulapati et al. [19] proposed a SoC-based ORB-SLAM by accelerating the feature extraction and matching on FPGA, since they consume 60–70% of the runtime of the algorithm. The rest of the algorithm is run on the ARM cores. To reduce resource utilization, they reduce the bit-depth of each pixel at different stage of the ORB pipeline and quantize the pixels to 6 bits per pixel with a 25% reduction of Flip-Flop resources and a drop of 3.9% in accuracy. Compared to eSLAM, the loss of accuracy is much lower. The algorithm was implemented on a Xilinx ZU3EG FPGA-SoC on-board with a quad-core ARM and evaluated on the TUM dataset [18]. The algorithm achieved 62 FPS with an energy consumption of 4.6 W. Nguyen et al. [4] proposed an HOOFR feature extraction architecture with a maintained complexity of the HOOFR algorithm to ensure a similar detection result on hardware as on software. The algorithm uses bucketing to ensure the homogeneous distribution of keypoints intended for SLAM applications. The algorithm was implemented on Arria 10 SoC-FPGA and achieved a frame rate of 26 FPS at 1280 × 720 pixels.

Following Nguyen et al.'s work [4], we propose an
architecture combining the HOOFR extractor and the

matching blocks since the high computational intensity
of feature extraction and matching makes running the algo-
rithm on low-power embedded platforms very challeng-
ing. The HOOFR-SLAM algorithm has been decomposed
into blocks and evaluated on different architectures to
achieve an optimized partitioning (to assign each block to
the appropriate processing unit), considering data transfer
constraints between processing units. Our ultimate goal
is to ensure processing at an adequate rate for autono-
mous vehicle applications (30 FPS) at a high-definition
resolution.

3 HOOFR-SLAM front-end overview

The HOOFR-SLAM algorithm is composed of two threads
running in parallel: Mapping thread and Loop detection
thread. The front end includes HOOFR extraction and the
mapping thread, as shown in Fig. 1. By measuring the
processing time of each functional block on the CPU, as
shown in Table 1, we notice that the most computationally
expensive blocks are detection (50% of front-end execu-
tion time) and matching (33% of front-end execution time).

<figure><img src="image_4.png" /><figcaption>**Fig. 1 HOOFR-SLAM front-end diagram, tasks that run on the front end are the HOOFR extraction (including detection, Hessian score computing, filtering, and description), Features Matching, and Relative Pose Computation. In red, the most computationally expensive blocks are detection (50% of front-end execution time) and matching (33% of front-end execution time)**</figcaption></figure>
<table><caption>**Table 1** Processing time of each functional block on CPU (AMD Ryzen 9 4900hs with 8 cores @ 3.0 GHz and 24GiB of RAM)</caption><thead><tr><th></th><th>Functional block</th><th>Runtime on CPU (ms)</th></tr></thead><tbody><tr><td rowspan="2">HOOFR Extraction</td><td>Feature detection (FAST/ Hessian/Filtering)</td><td><strong>14.29</strong></td></tr><tr><td>Description</td><td>1.68</td></tr><tr><td rowspan="2">Mapping thread</td><td>Matching</td><td><strong>9.69</strong></td></tr><tr><td>Pose estimation</td><td>3.5</td></tr><tr><td rowspan="2">Back-end</td><td>Loop detection</td><td>5.22</td></tr><tr><td>Map processing</td><td>0.126</td></tr></tbody></table>

Bold values highlight the highest processing times among the functional blocks

## 3.1 Bucketing-based HOOFR extractor

Nguyen et al. [4] proposed the implementation of a buckting-based HOOFR extractor on FPGA. The bucketing technique divides the image into a grid, as shown in Fig. 2. The number of cells in the grid depends on the image's resolution. For each cell, keypoints are detected by the FAST detector. Then, the Hessian score is calculated for each point, keeping only the points with the highest score. After filtering the points, the description of each point is computed. The bucketing technique has the benefit of ensuring a homogeneous distribution of keypoints over the entire image, which improves the localization accuracy of the SLAM [20, 21] and allows a pipeline operation at the cell level, as illustrated in Fig. 2, meaning that when a kernel finishes its task on the cell, the next kernel starts processing immediately on this cell.

The architecture of the HOOFR extractor on FPGA is illustrated in Fig. 3. The system includes four functional blocks (FAST, Hessian score, Filtering, and Description). These kernels are launched in parallel. At each image acquisition, the CPU copies the image in the global memory. Then, it consecutively launches the kernels for detection (FAST, Hessian score, and Filtering). The description block is launched once the integral of the image is computed and ready in the global memory. Finally, synchronization is performed when all the kernels are active, and the CPU waits to get the results. The FAST kernel and the Hessian score calculation have been duplicated, because they are the bottlenecks of the algorithm flow. Testing the pixels of the whole image in high resolutions (for example, 921,600 pixels in the case of HD resolution) makes the FAST kernel very cost-intensive. Moreover, the latter returns several keypoints, which makes computing the Hessian score very costly. The communication between these different kernels is done using the Intel channel extension. Intel channels provide a mechanism for direct data transfer and synchronization between kernels via FIFO buffers without interaction with the host processor.

## 3.2 Features' matching

In [7], the matching block was implemented to run on GPU. The algorithm matches the Previous Neighbor Frames' (PNFs) keypoints to the current frame. In the PNF, the cell index and 256-bit description are required for each keypoint. Four matrices are transferred to GPU global memory: $Pnf_{Dess}$, $Pnf_{Cels}$, $Cur_{Dess}$ and $Curr_{Distributions}$.

The $Pnf_{Dess}$ is the description of keypoints in previous neighbor frames, and is organized as an unsigned char matrix with a dimension of ($pnf_{np} \times 32$), where $pnf_{np}$ is

<figure><img src="image_5.png" /><figcaption>**Fig. 2** Image split into a grid. Image cells are processed in the pipeline. When a kernel finishes processing a cell, the following kernel takes over</figcaption></figure>

<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 3** HOOFR extractor architecture implements duplicated FAST kernel, duplicated Hessian score computing kernel, a filtering kernel, and the description kernel. Kernels communicate using a channel extension without interacting with the host processor</figcaption></figure>

the total number of keypoints in PNFs. The $Pnf_{Cels}$ is a $(pnf_{np} \times 1)$ matrix, where each row represents the cell number where each keypoint is located. For the current frame, $Cur_{Dess}$ is a current description matrix which is created same as $Pnf_{Dess}$ with a dimension of $(cur_{np} \times 32)$, where the $cur_{np}$ is the total number of keypoints in the current frame. The keypoints of the current frame are organized by the order of image cell. To keep track of the current keypoints located in each cell, a structure denoted Points_Distribution is used. This structure is composed of two elements: the first element ref is the position where the first keypoint of the cell is located in the whole set

and the second element nb is the number of keypoints of the cell. $Curr_{Distributions}$ is a matrix with the dimension of $(N_{CELLS} \times 2)$ for the N cells built using Points_Distribution structure. For each keypoint in the PNFs, the correspondence is searched in the current frame at the same cell and neighbor cells, as shown in Fig. 4. This process is so fast, since each cell contains a small number of keypoints suitable for handling by one work item. The GPU uses nine work items in a work group to find the matches in nine neighbor cells of the current frame. The nine results are stored in the local memory and are synchronized by the barrier function. After synchronizing, the final matching is

<figure><img src="image_8.png" /><figcaption>**Fig. 4** Features' matching strategy. The concept consists in searching for the corresponding keypoint of the previous neighbor images in the same cell and the neighboring cells in the current image</figcaption></figure>
extracted by one of the nine work items. Then, the match is validated by computing the Lowe's ratio [2].

## 4 Algorithm-architecture mapping

Starting from the Bucketing-based HOOFR extractor [4], we aim to embed the front end of the HOOFR-SLAM on FPGA in the context of pushing the processing as close as possible to the sensor (an RGB camera for instance). For this purpose, we implemented the matching algorithm on the FPGA, since it is the second most computationally heavy task. The implementation of matching on GPU [7] has shown outstanding performance. However, considering the power consumption, GPUs are highly power-consuming, making thermal management more challenging in embedded systems, especially for applications where energy autonomy is a crucial asset. Although GPUs (i.e., Jetson AGX Xavier™) begin getting close to FPGAs in terms of performance-per-watt, that does not mean they are the best solution for all applications. FPGAs represent an essential feature, which is any-to-any I/O connection, allowing connection to any

device, network, or storage without needing a host CPU. The FPGA is well suited for front-end processing in general-purpose processing, since it can be directly connected to high-speed sensors and offer very high bandwidth.

In our study, we first evaluated the GPU implementation of matching block on FPGA, since the algorithm was implemented in OpenCL [7] and thus can be run on both GPU and FPGA. When we ran the algorithm using NDRange on two images of a resolution of 1280 × 720 with 2000 keypoints, we got a very large runtime of 813.01 ms. This execution time is explained by the data dependency represented in waiting for the results of the work items to select the match with the minimum distance. The impact of synchronization is not seen on GPU, as the processing frequency is high and the work items on GPU are launched simultaneously, unlike in FPGA, where the work items are launched in parallel in a pipelined way. Also, FPGAs are flexible in terms of programming architecture and are able to provide performance for operations that contain conditionals and/or branches. These architectural differences have a significant impact on performance. For this reason, we used a single-work-item implementation, also called task kernel.

In Table 2, we see that with a single task kernel, we obtain a speedup factor of ×96 compared to the NDRange kernel. To reach similar performances as on GPU, we create four matching kernels, since they do not take a lot of hardware resources. With four matching kernels, we get a speedup of ×254.

Figure 5 represents the CPU-FPGA mapping for the HOOFR extractor and the matching block. To establish communication between the CPU and the FPGA, several OpenCL APIs are used to manage data transfers, task control, and

<table><caption>**Table 2** Comparison of NDRange and single-work-item execution time in milliseconds with different numbers of kernel task duplication</caption><tr><td>Matching NDRange kernel runtime (ms)</td><td>813.01</td></tr><tr><td>Matching task kernel duplication</td><td>1 2 3 4</td></tr><tr><td>Matching task runtime (ms)</td><td>8.39 5.02 3.99 <b>3.19</b></td></tr></table>

The bold value highlights the best single-work-item execution time for a given task kernel duplication factor

<figure><img src="image_9.png" /><figcaption>**Fig. 5** CPU-FPGA mapping of the HOOFR extractor and the matching block, the matching block is duplicated four times. The keypoints of the previous neighbor images are distributed on the four kernels</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>**Fig. 6** Matching kernels' duplication diagram. The indexes of the cells and the descriptors of the previous neighboring images are divided into four chunks and distributed over the four kernels. Each kernel finds the match in the current and neighboring cell from the descriptor matrix of the current image. Finally, each kernel stores the found matches into a vector</figcaption></figure>

memory management (Write/Read Buffers, NDRange Kernel's configuration). The online code allows identifying these APIs and understanding the exchanges between the CPU and the FPGA. Once the description is done, the results are stored in the global memory, and the four matching blocks are launched concurrently. The description and cell matrices

of PNFs are partitioned and distributed over the four kernels, and the description of the current frame keypoint and their distributions on cells are accessed by the four kernels, as shown in Fig. 6.
Algorithm 1 represents one of the four duplicated kernels on FPGA.

<caption>**Algorithm 1 OpenCL matching kernel on FPGA**</caption>

Input: Global arrays: $Pnf_{Dess}$, $Pnf_{Cels}$, $Curr_{Distributions}$, $Curr_{desc}$
Output: Global array: Correspondence

1 **Function Kernel: Matching(Input, Output):**
2   //Iteration on one quarter of the Pnf keypoints
3   **for** i ← 0 **to** $N_{ktps}$/4 **do**
4     //Get the cell coordinates in the grid (NX, NY) to which the Pnf keypoint belongs.
5     NX ← $Pnf_{Cels}[i].x$
6     NY ← $Pnf_{Cels}[i].y$
7     point_pnf_des ← Get_Kpt_Descrip($Pnf_{Dess}[32 \times i]$)
8     **return** sum
9   //Iteration over the neighbor cells
10  //NUM_SEARCH_X, NUM_SEARCH_Y represent the horizontal and vertical search radius respectively
11  **for** kx ← 0 **to** 2 × NUM_SEARCH_X + 1 **do**
12    **for** ky ← 0 **to** 2 × NUM_SEARCH_Y + 1 **do**
13      //Compute the neighbor cell coordinates (nx, ny)
14      nx ← NX + kx − NUM_SEARCH_X
15      ny ← NY + ky − NUM_SEARCH_Y
16      **if** (nx, ny) not exceeding Grid boundaries then
17        //Get the keypoints indexes belonging to the current cell (nx, ny) from Curr_{Distributions}
18        (id_{start}, id_{end}) = Get_idx_Distrib(Curr_{Distributions}, nx, ny)
19        **for** j ← id_{start} **to** id_{end} **do**
20          //Current keypoint descriptor
21          point_current_des ← Get_Kpt_Descrip(Curr_{desc}[32 \times j])
22          Correspondence[i] ← Find_1and2_matches(point_pnf_des, point_current_des)

# 5 Experimental results

We implement the HOOFR-SLAM front-end on a DE5a-Net DDR4 Arria 10 FPGA PCIe board connected to a desktop via PCIe Gen 3 × 8. The DE5a-Net Arria 10 FPGA uses the Intel Arria 10 External Memory Interfaces (EMIF) IP to

connect with DDR4 memory. This IP includes a hard memory controller within the FPGA to manage communication with DDR4. The hard memory controller implements the DDR4 protocol, handling all necessary command, address, and data transfer operations between the FPGA and DDR4 memory. The desktop's CPU is an Intel Xeon Silver 4108

<table><caption>**Table 3** FPGA resource utilization</caption><thead><tr><th>Kernel</th><th>ALUTs</th><th>FFs</th><th>RAMs</th><th>DSPs</th></tr></thead><tbody><tr><td>FAST Detection kernel 1</td><td>13,155 (2%)</td><td>19,048 (1%)</td><td>166.2 (6%)</td><td>0 (0%)</td></tr><tr><td>FAST Detection kernel 2</td><td>13,155 (2%)</td><td>19,048 (1%)</td><td>166.2 (6%)</td><td>0 (0%)</td></tr><tr><td>Hessian Score kernel 1</td><td>7320 (1%)</td><td>12,121 (1%)</td><td>62 (2%)</td><td>6 (0%)</td></tr><tr><td>Hessian Score kernel 2</td><td>7321 (1%)</td><td>12,256 (1%)</td><td>62 (2%)</td><td>6 (0%)</td></tr><tr><td>Filtering kernel</td><td>5901 (1%)</td><td>11,557 (1%)</td><td>75 (3%)</td><td>0 (0%)</td></tr><tr><td>Description kernel</td><td>24,397 (3%)</td><td>35,413 (2%)</td><td>236 (9%)</td><td>8.5 (1%)</td></tr><tr><td>Matching kernel 1</td><td>13,903 (2%)</td><td>19,962 (1%)</td><td>190 (7%)</td><td>3.5 (0%)</td></tr><tr><td>Matching kernel 2</td><td>14,282 (2%)</td><td>24,427 (1%)</td><td>206 (8%)</td><td>3.5 (0%)</td></tr><tr><td>Matching kernel 3</td><td>14,312 (2%)</td><td>24,419 (1%)</td><td>206 (8%)</td><td>3.5 (0%)</td></tr><tr><td>Matching kernel 4</td><td>14,290 (2%)</td><td>24,358 (1%)</td><td>206 (8%)</td><td>3.5 (0%)</td></tr><tr><td><strong>Kernel Subtotal</strong></td><td><strong>128,036 (15%)</strong></td><td><strong>202,609 (12%)</strong></td><td><strong>1,575 (59%)</strong></td><td><strong>34.5 (3%)</strong></td></tr><tr><td>Pipe and channel resources</td><td>55 (0%)</td><td>558 (0%)</td><td>5 (0%)</td><td>0 (0%)</td></tr><tr><td><strong>Available</strong></td><td><strong>854,400</strong></td><td><strong>1,708,800</strong></td><td><strong>2,713</strong></td><td><strong>1,518</strong></td></tr></tbody></table>

ALUTSs Adaptive Look-Up Tables, FFs Flip Flops, RAMs Random Access Memory blocks, DSPs Digital Signal Processing blocks. Between parenthesis, the usage percentage of the total available

Bold values highlight the subtotals of resource usage and the remaining FPGA resources
<table><caption>**Table 4** The number of cycles per match (CPM) of the matching block on PC CPU vs FPGA</caption><thead><tr><th>Keypoints</th><th>PC CPU (CPM)</th><th>FPGA (CPM)</th></tr></thead><tbody><tr><td>1000</td><td>49,953</td><td><strong>88</strong></td></tr><tr><td>1500</td><td>72,813</td><td><strong>78</strong></td></tr><tr><td>2000</td><td>88,925</td><td><strong>108</strong></td></tr><tr><td>2500</td><td>110,675</td><td><strong>127</strong></td></tr></tbody></table>

Bold values highlight the best CPM achieved

<table><caption>**Table 5** The runtime of the matching block on PC CPU vs FPGA</caption><thead><tr><th>Keypoints</th><th>PC CPU (ms)</th><th>FPGA (ms)</th><th>Speedup</th></tr></thead><tbody><tr><td>1000</td><td>16.65</td><td>1.76</td><td><strong>9.46</strong></td></tr><tr><td>1500</td><td>36.40</td><td>2.33</td><td><strong>15.62</strong></td></tr><tr><td>2000</td><td>59.28</td><td>4.32</td><td><strong>13.72</strong></td></tr><tr><td>2500</td><td>92.22</td><td>6.37</td><td><strong>14.47</strong></td></tr></tbody></table>

Bold values highlight the best speedup factors

processor with eight cores with a base frequency of 1.8 GHz. The PC used for comparison is a high-performance laptop equipped with an 8-core AMD Ryzen 9 CPU with a base frequency of 3 GHz. The frames used for the evaluation are outdoor sequences recorded with the Intel RealSense D455 camera with a frame rate of 30 FPS and a resolution of 1280 × 720. The images are resized with the resize function of OpenCV.

## 5.1 Resource usage

Our accelerator is implemented on a DE5a-Net DDR4 Arria 10 FPGA Development Kit (operating at 50 MHz) with 1150K LEs. The resource utilization of the proposed architecture is shown in Table 3. We followed some performance-improvement rules proposed by Intel®FPGA SDK for OpenCL™Pro Edition Best Practices Guide to save resources. By reducing the number of unrolled loops and the compute units, we decreased the resource usage and required memory bandwidth. Unrolling the outer loops of a nested-loop structure also increases resource utilization significantly. Therefore, we unrolled only the inner loops. By saving resources, we could incorporate the matching block into our architecture.

## 5.2 Matching block timings' analysis

The matching block performance has been evaluated on high-performance PC CPU (8-core AMD Ryzen 9 CPU @ 3 GHz) and FPGA architectures for a different number of

<table><caption>**Table 6** Comparison of the matching recall-precision between PC and FPGA</caption><thead><tr><th rowspan="2"></th><th colspan="2">PC (CPU)</th><th colspan="2">FPGA</th></tr><tr><th>Recall (%)</th><th>1-precision (%)</th><th>Recall (%)</th><th>1-precision (%)</th></tr></thead><tbody><tr><td>480 × 270</td><td>64</td><td>36</td><td>64.94</td><td>35.05</td></tr><tr><td>640 × 480</td><td>58.4</td><td>41.6</td><td>57</td><td>42.98</td></tr><tr><td>848 × 480</td><td>56.8</td><td>43.2</td><td>56.65</td><td>43.34</td></tr><tr><td>1280 × 720</td><td>57.35</td><td>42.65</td><td>56.94</td><td>43.05</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>(a) Images before matching</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Result after matching</figcaption></figure>

<caption>**Fig. 7** Matching two successive images from outdoor sequence 1 on FPGA</caption>

features. The CPU implementation of the matching block utilizes multi-core processing through OpenMP parallelization. Specifically, the matching computations are distributed across 4 threads, leveraging the multi-core capabilities of the AMD Ryzen 9 4900HS processor. This parallelization aims to provide an efficient software baseline for comparison with the FPGA implementation. To quantify the performance of each architecture, regardless of the frequency, we have computed the number of cycles per match (CPM) using the following equation:

$$CPM = \frac{t \cdot f}{N}, \qquad (1)$$

where *t* is execution time, *f* is the frequency of the architecture, and *N* is the number of features. We ran the matching block on the PC CPU, and on the DE5a-Net DDR4 Arria
<table><caption>**Table 7** Timing performance on different resolutions (FAST_thresh-old = 7, POINTS_PER|RCELL = 15)</caption><thead><tr><th>Resolution (px)</th><th>Number of Keypoints</th><th>NX × NY</th><th>Runtime (ms)</th><th>FPS</th></tr></thead><tbody><tr><td>480 × 270</td><td>2228</td><td>24 × 14</td><td>24.46</td><td>40.88</td></tr><tr><td>640 × 480</td><td>2158</td><td>24 × 14</td><td>26.21</td><td>38.15</td></tr><tr><td>848 × 480</td><td>2111</td><td>24 × 14</td><td>27.09</td><td>36.91</td></tr><tr><td>1280 × 720</td><td>2009</td><td>24 × 14</td><td>33</td><td>30.3</td></tr></tbody></table>

Bold values highlight the highest image resolution that meets runtime application constraints

10 FPGA. Our system is evaluated on images taken by an Intel RealSense D455 camera at a resolution of 1280 × 720. Tables 4, 5 show that the FPGA outperforms the PC CPU in CPM and in execution time. This hardware implementation allows a speedup of up to ×15.

## 5.3 Matching block accuracy analysis

To ensure that the accuracy of matching is maintained, we fix the FAST detection threshold at 7 and the hamming distance threshold at 20 and we compute the recall (number of correct matches/number of correspondences) and the 1-precision (number of false matches/number of matches) [22–24]. Both quantities are compared in software and hardware implementation. The results are summarized in Table 6, while the Fig. 7 shows the matching result on FPGA with images in a resolution of 640 × 480 pixels. These results show that the matching result on hardware is slightly similar to that on software.

## 5.4 Power consumption analysis

Our design demonstrates a total thermal power dissipation of 23.88 W, with the core dynamic power contributing 6.97 W and the transceiver dynamic power contributing 4.06 W. When compared to GPU-based systems, such as the NVIDIA Jetson series (which typically consumes 20 W to 40 W), and high-performance desktop GPUs (often exceeding 200 W), our FPGA solution shows significant power efficiency. This efficiency is achieved through the activation of only necessary circuitry for each specific task, particularly in computationally intensive operations like feature extraction and matching.

# 6 Overall performance evaluation

In this section, we evaluate the performance of the entire algorithm on different resolutions supported by the Intel RealSense D455 camera. The image is divided into a grid. NX and NY represent the number of horizontal and vertical cells. Each cell return a maximum number of keypoints (with the highest Hessian score) limited to POINTS_PER|RCELL. As shown in Table 7, our design reaches a frequency of 30 FPS at a resolution of 1280 × 720 pixels, generating and matching 2009 keypoints which is well suited to achieve good localization accuracy [7]. At 640 × 480 pixels resolution, our design reaches 38 FPS, which is sufficient to perform on-the-fly processing. The HD resolution can be helpful if an object detection task is needed, since object detection at more considerable distances becomes feasible [25].

# 7 Conclusion

This paper presents a design of FPGA-based near-sensor processing (matching incorporated with the bucketing-based HOOFR extractor). A matching algorithm has been implemented with FPGA programming paradigms to achieve real-time processing tailored to the 30 FPS camera frame rate. Also, we have shown that the algorithm's complexity has been preserved to guarantee the same quality of results in software as in hardware. Our design was implemented on DE5a-Net DDR4 Arria 10 FPGA architecture. The FPGA implementation shows that the OpenCL design is 9× to 14× faster than the C++ implementation running on a high-performance CPU. The achieved throughput is 30 FPS at 1280 × 720 pixels resolution and 38 FPS at 640 × 480 pixels resolution, which is sufficient to perform on-the-fly processing. The proposed architecture involves acquiring the images from the sensor on the on-chip memory, launching the keypoint extraction and matching on FPGA. Finally, the pose estimation and the loop closure detection tasks of the SLAM algorithm are launched on a parallel thread on the embedded system's CPU. We made our code source available at the following link https://github.com/Imel23/FPGA-Based-HOOFR-Front-End.

**Author Contributions** All authors contributed to the study conception and methodology. AE and SR had the idea for the article and supervised the process. SR was in charge of providing useful insights to the algorithmic aspects of the discussed SLAM method. AE was in charge of providing useful insights to the hardware aspects of the discussed SLAM system. IE contributed to the implementation of the HOOFR-SLAM front end on a CPU-FPGA architecture, including both feature extraction and matching processing blocks. He also participated in manuscript writing and editing to share the research findings. SR and
AE commented on previous versions of the manuscript and critically revised the work. All authors read and approved the final manuscript.

**Funding** Open access funding provided by Université Paris-Saclay.

**Data Availability** No datasets were generated or analyzed during the current study.

## Declarations

**Conflict of interest** The authors declare no competing interests.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## References

1. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: International Conference on Computer Vision, pp. 2564–2571 (2011)

2. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. **60**(2), 91–110 (2004)

3. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: speeded up robust features. In: Computer Vision – ECCV 2006, pp. 404–417 (2006)

4. Nguyen, D.D., El Ouardi, A., Rodriguez, S., Bouaziz, S.: FPGA implementation of HOOFR bucketing extractor-based real-time embedded SLAM applications. J. Real-Time Image Proc. **18**(3), 525–538 (2021)

5. Wu, S., Hu, D., Ibrahim, S., Jin, H., Xiao, J., Chen, F., Liu, H.: When fpga-accelerator meets stream data processing in the edge. In: IEEE 39th International Conference on Distributed Computing Systems, pp. 1818–1829 (2019)

6. Nakamura, K., Hayashi, A., Matsutani, H.: An FPGA-based low-latency network processing for spark streaming. In: IEEE International Conference on Big Data (Big Data), pp. 1818–1829 (2016)

7. Nguyen, D.-D., Elouardi, A., Rodriguez Florez, S.A., Bouaziz, S.: HOOFR SLAM system: an embedded vision SLAM algorithm and its hardware-software mapping-based intelligent vehicles applications. IEEE Trans. Intell. Transp. Syst. **20**(3), 1127–1139 (2019)

8. Chien, C.-H., Chien, C.-J., Hsu, C.-C.: Hardware-software co-design of an image feature extraction and matching algorithm. In: 2nd International Conference on Intelligent Autonomous Systems, pp. 37–41 (2019)

9. Latif, R., Saddik, A.: SLAM algorithms implementation in a UAV, based on a heterogeneous system: a survey. In: 4th World Conference on Complex Systems, pp. 1–6 (2019)

10. Dine, A., Elouardi, A., Vincke, B., Bouaziz, S.: Graph-based slam embedded implementation on low-cost architectures: a practical approach. In: IEEE International Conference on Robotics and Automation, pp. 1–6 (2015)

11. Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: DTAM: dense tracking and mapping in real-time. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2320–2327 (2011)

12. Ondrúška, P., Kohli, P., Izadi, S.: Mobilefusion: real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE Trans. Visual Comput. Graph. **21**(11), 1251–1258 (2015)

13. Aldegheri, S., Bombieri, N., Bloisi, D.D., Farinelli, A.: Data flow ORB-SLAM for real-time performance on embedded GPU boards. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1–6 (2019)

14. Ma, T., Bai, N., Shi, W., Wu, X., Wang, L., Wu, T., Zhao, C.: Research on the application of visual slam in embedded GPU. Wirel. Commun. Mob. Comput. **221**, 1–17 (2021)

15. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Robot. Res. **32**(11), 1231–1237 (2013)

16. Fang, W., Zhang, Y., Yu, B., Liu, S.: FPGA-based ORB feature extraction for real-time visual SLAM. In: International Conference on Field Programmable Technology, pp. 1–8 (2017)

17. Liu, R., Yang, J., Chen, Y., Zhao, W.: ESLAM: An energy-efficient accelerator for real-time orb-slam on fpga platform. In: Proceedings - Design Automation Conference, pp. 1–6 (2019)

18. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: IEEE/RSJ international conference on intelligent robots and systems, pp. 573–580 (2012)

19. Vemulapati, V., Chen, D.: ORB-based SLAM accelerator on SoC FPGA. Preprint arXiv:2207.08405 (2022)

20. Mur-Artal, R., Tardos, J.D.: ORB-SLAM2: an open-source slam system for monocular, stereo, and RGB-D cameras. IEEE Trans. Rob. **33**(5), 1255–1262 (2017)

21. Pire, T., Fischer, T., Civera, J., De Cristoforis, P., Berlles, J.J.: Stereo parallel tracking and mapping for robot localization. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1373–1380 (2015)

22. Nguyen, D.-D., El Ouardi, A., Aldea, E., Bouaziz, S.: HOOFR: An enhanced bio-inspired feature extractor. In: 23rd International Conference on Pattern Recognition, pp. 1–6 (2016)

23. Alahi, A., Ortiz, R., Vandergheynst, P.: Freak: Fast retina keypoint. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–517 (2012)

24. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. **27**(10), 1615–1630 (2005)

25. Schwiegelshohn, F., Gierke, L., Hubner, M.: FPGA based traffic sign detection for automotive camera systems. In: 10th International Symposium on Reconfigurable Communication-centric Systems-on-Chip, pp. 1–6 (2015)

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.