<figure><img src="image_1.png" /></figure>

HAL
open science

# Indoor Visual SLAM Dataset With Various Acquisition Modalities

Imad El Bouazzaoui, Sergio Alberto Rodriguez Florez, Bastien Vincke, Abdelhafid El Ouardi

► To cite this version:

Imad El Bouazzaoui, Sergio Alberto Rodriguez Florez, Bastien Vincke, Abdelhafid El Ouardi. In-door Visual SLAM Dataset With Various Acquisition Modalities. Data in Brief, 2021, pp.107496. 10.1016/j.dib.2021.107496. hal-03400312

HAL Id: hal-03400312

https://hal.science/hal-03400312

Submitted on 25 Oct 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
<figure><img src="image_2.png" /></figure>

Data Article

Indoor visual SLAM dataset with various
acquisition modalities

<figure><img src="image_3.png" /></figure>

Imad El Bouazzaoui*, Sergio Rodriguez, Bastien Vincke, Abdelhafid El Ouardi

SATIE - CNRS UMR 8029, Paris-Saclay University, France

ARTICLE INFO

Article history:
Received 10 June 2021
Revised 4 October 2021
Accepted 11 October 2021
Available online 19 October 2021

Keywords:
Depth map
Indoor localization
RGB-D cameras
Robotics

A B S T R A C T

The indoor Visual Simultaneous Localization And Mapping (V-SLAM) dataset with various acquisition modalities has been created to evaluate the impact of acquisition modalities on the Visual SLAM algorithm's accuracy. The dataset contains different sequences acquired with different modalities, including RGB, IR, and depth images in passive stereo and active stereo modes. Each sequence is associated with a reference trajectory constructed with an Structure From Motion (SFM) and Multi View Stereo (MVS) library for comparison. Data were collected using an intrinsically calibrated Intel RealSense D435i camera. The RGB/IR and depth data are spatially aligned, and the stereo images are rectified. The dataset includes various areas, some with low brightness, with changes in brightness, wide, narrow and texture.

© 2021 The Authors. Published by Elsevier Inc.

This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/)

E-mail addresses: imad.el-bouazzaoui@universite-paris-saclay.fr (I. El Bouazzaoui), sergio.rodriguez@universite-paris-saclay.fr (S. Rodriguez), bastien.vincke@universite-paris-saclay.fr (B. Vincke), abdelhafid.elouardi@universite-paris-saclay.fr (A. El Ouardi).

<footnote>* Corresponding author.</footnote>
Specifications Table

<table><tr><td>Subject</td><td>Computer Science</td></tr><tr><td>Specific subject area</td><td>Computer Vision and Pattern Recognition</td></tr><tr><td>Type of data</td><td>Image<br/>Reference trajectory : text file<br/>Timestamps files : text file<br/>Intrinsic parameters files (ORB-SLAM2 parameters format [1]) : yaml file</td></tr><tr><td>How data were acquired</td><td>Images were acquired using a Intel RealSense D435i camera connected to a machine equipped with an Intel Celeron N4100 Quad-Core CPU, 8G RAM and 512GB SSD memory, running Ubuntu 18.04 and Realsense viewer app for Digiteo_seq1 and Digiteo_seq2. Digiteo_seq3 was acquired using a laptop equipped with AMD® Ryzen 9 4900hs with 23GB RAM and 1TB SSD running Ubuntu 20.04 and ROS Noetic. The experiment was carried out using different acquisition modes. The sensor has been intrinsically calibrated in advance using Intel® RealSense™ D400 Series Dynamic Calibration Tool to obtain a spatial alignment between the RGB/IR frames and the depth frames, and to have already rectified stereo frames.</td></tr><tr><td>Data format</td><td>Analysed</td></tr><tr><td>Parameters for data collection</td><td>Indoor acquisitions were performed, taking into account different scene light conditions, various scene scale, textured, and low-textured environments.</td></tr><tr><td>Description of data collection</td><td>Indoor images were acquired using various acquisition modes, including IR stereo, RGB-D active stereo (IR projector on), RGB-D passive stereo (IR projector off), and IR-D passive stereo. IR-D was not recorded in active stereo because the IR patterns interfere with feature extraction, causing spurious detections. Images were recorded with a resolution of 1280x720 pixels and a frame rate of 30 FPS. RGB images have been converted to grayscale to reduce data size. The difference between the RGB images and the IR is the field of view and exposure modes. It is a rolling shutter for the RGB camera, while for the IR, it is a global shutter.</td></tr><tr><td>Data source location</td><td>Institution: SATIE Laboratory - Paris-Saclay University<br/>City/Town/Region: Gif-sur-Yvette<br/>Country: France<br/>Latitude: 48.71264, Longitude: 2.16825<br/>Repository name: Mendeley Data</td></tr><tr><td>Data accessibility</td><td>Data identification number:<br/>10.17632/7swv73drgr.3<br/>10.17632/tb9g7th9yz.2<br/>10.17632/c2gtvyxyt7.2<br/>10.17632/2n7j5pg2xj.2<br/>10.17632/kpps3854xm.2<br/>10.17632/5xmzkgcgg7.2<br/>Direct URL to data:<br/><a href="https://doi.org/10.17632/7swv73drgr.3">https://doi.org/10.17632/7swv73drgr.3</a><br/><a href="https://doi.org/10.17632/tb9g7th9yz.2">https://doi.org/10.17632/tb9g7th9yz.2</a><br/><a href="https://doi.org/10.17632/c2gtvyxyt7.2">https://doi.org/10.17632/c2gtvyxyt7.2</a><br/><a href="https://doi.org/10.17632/2n7j5pg2xj.2">https://doi.org/10.17632/2n7j5pg2xj.2</a><br/><a href="https://doi.org/10.17632/kpps3854xm.2">https://doi.org/10.17632/kpps3854xm.2</a><br/><a href="https://doi.org/10.17632/5xmzkgcgg7.2">https://doi.org/10.17632/5xmzkgcgg7.2</a></td></tr><tr><td>Related research article</td><td>I. El Bouazzaoui, S. Rodriguez and A. El Ouardi, "Enhancing RGB-D SLAM Performances Considering Sensor Specifications for Indoor Localization," in IEEE Sensors Journal, doi: 10.1109/JSEN.2021.3073676.</td></tr></table>

Value of the Data

* This dataset contains three sequences. Each sequence contains two to four acquisition modalities in the same environment, allowing to visualize the sensor's impact on the localization accuracy of SLAM algorithms.

* The dataset is relevant to the computer vision and robotics field, particularly for autonomous robot applications involved in localization in an indoor environment.
* The provided dataset can be used to evaluate visual SLAM algorithms or visual odometry algorithms, with different input types such as monocular, stereo, or RGB-D.

* The data is temporally and spatially aligned and ready to be used without further pre-processing.

# 1. Data Description

The simultaneous localization and mapping problem known as SLAM is considered one of the pillars of autonomy in robotics and autonomous vehicles, besides other applications that use it. This problem has been under hard work for over a decade. Several solutions have been proposed using various algorithms with a variety of sensors, such as [1–3]. With the development of SLAM algorithms, several datasets have been made available to researchers to evaluate their algorithms, particularly visual SLAM. Often these datasets are intended for evaluation of the algorithm with a single acquisition modality, such as [4,5]. As a result, there are no different acquisition modalities for the same sequence, allowing us to compare the sensor’s impact on the localization accuracy. In this work, we propose a dataset consisting of three sequences with different acquisition modalities. The dataset was the subject of a study done in [6]. The dataset includes three static sequences in two different environments. Two sequences are recorded in the laboratory’s corridors: Digiteo_seq1 and Digiteo_seq2, as shown in the Figs. 1a–1c. Furthermore, one sequence is recorded in the basement parking of the laboratory: Digiteo_seq3, as illustrated in Fig. 1d.

Digiteo_seq1 and Digiteo_seq2 have varied zones in texture and brightness, also characterized by their narrow space. While for Digiteo_seq3, we have a variety in brightness and a vast space. Digiteo_seq1 consists of three acquisition modalities: Stereo, IR-D, and RGB-D in passive stereo. Digiteo_seq1 forms a single loop closure over 83 m. It includes three modalities: IR-D, RGB-D, and Stereo in passive mode. Digiteo_seq2 is 58 m long with single loop closure. It has active RGB-D

<figure><img src="image_4.png" /><figcaption>(a) Well-textured and well-illuminated narrow environment</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Low-textured and low-illuminated narrow environment</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c) Well-textured and well-illuminated large environment</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d) Low-textured and well-illuminated large environment</figcaption></figure>

<caption>**Fig. 1.** Different images from Digiteo_seq1, Digiteo_seq2 and Digiteo_seq3.</caption>
<figure><img src="image_8.png" /><figcaption>**Fig. 2.** Dataset structure with the various acquisition modes.</figcaption></figure>

and passive RGB-D. The IR-D and Stereo could not be recorded because the patterns projected by the IR projector are visible, and this disturbs the detection of the primitives by the extractor. Finally, the Digiteo_seq3 recorded in the car park includes all the modalities presented before and makes the largest trajectory on 151 m with only one loop closure. The whole dataset is summarized in the Fig. 2.

Each dataset is provided with a file named `param.yaml` containing the intrinsic parameters of the camera, as well as the parameters of the ORB-SLAM2 algorithm. The dataset has a reference trajectory to make the comparison built using a Structure From Motion and Multiview-Stereo pipeline [7,8] as shown in Fig. 3. All reference trajectories have a format compatible with the evaluation tools available on TUM RGB-D Dataset [4].

The reprojection errors of the COLMAP point cloud reconstruction are shown in Fig. 4.

We configured the camera to acquire data at a rate of 30 FPS. The Figs. 5 and 6 represent the statistics of the acquisition rate for each sensor in the camera. The acquisition rate of RGB images with IR images ends up with a frame drop, in the case of Digiteo seq 1. Same for sequence 2. By using a laptop with more RAM, the frame rate is balanced between the three cameras. In contrast, the RGB-D active stereo is acquired alone at a rate of 30 FPS.

## 2. Experimental Design, Materials and Methods

The dataset was acquired using an Intel RealSense D435i camera. The camera provides stereo IR rectified, RGB-D and IR-D aligned data using realsense-ros package [9]. The images were temporally aligned after acquisition based on ROS Bag timestamps. The datasets were acquired using ROS and realsense-ros package at 30 FPS and 720p resolution without filters. A reference trajectory for comparison was created based on subsampled (subsampled by 1/5 for seq1 and seq2 and by 1/10 for seq3) monocular images of the environment using the SFM and MVS pipeline [7,8]. The provided images are synchronized IR/RGB images with the depth images. The images
<figure><img src="image_9.png" /><figcaption>**Fig. 3.** Reference trajectory in red using COLMAP.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 4.** Reprojection errors of the COLMAP point cloud reconstruction.</figcaption></figure>

used for the reference reconstruction are non-synchronized images that have been sub-sampled
while keeping the timestamp of each image. Subsampled images allow for faster processing
while providing sufficient visual overlap. The first step is to detect and extract features from
all images and describe them using a numerical descriptor. The feature extraction uses a pin-
hole camera model [10] with the camera's intrinsics and extrinsics parameters, as shown in the
Tables 1 and 2. The extractor used is SIFT and executed on GPU with a maximum number of
primitives of 8192.
<figure><img src="image_11.png" /></figure>

Digiteo seq1

<figure><img src="image_12.png" /></figure>

Digiteo seq2

<figure><img src="image_13.png" /></figure>

Digiteo seq3

<caption>**Fig. 5.** The histogram of the acquisition rate of each modality for each sequence.</caption>
FPS rate for different acquisition modes

<figure><img src="image_14.png" /></figure>

(a) Digiteo seq 1

FPS rate for different acquisition modes

<figure><img src="image_15.png" /></figure>

(b) Digiteo seq 2

FPS rate for different acquisition modes

<figure><img src="image_16.png" /></figure>

(c) Digiteo seq 3

<caption>**Fig. 6.** The Box plot of the acquisition rate of each modality for each sequence.</caption>
<figure><img src="image_17.png" /><figcaption>Fig. 7. Reference trajectory process using COLMAP.</figcaption></figure>

<table><caption>**Table 1**

Intrinsics of the D435i camera.</caption><thead><tr><th></th><th>RGB</th><th>IR</th></tr></thead><tbody><tr><td>Width (pixel)</td><td>1280</td><td>1280</td></tr><tr><td>Height (pixel)</td><td>720</td><td>720</td></tr><tr><td>PPX (pixel)</td><td>648.57</td><td>639.70</td></tr><tr><td>PPY (pixel)</td><td>363.66</td><td>356.18</td></tr><tr><td>Fx (pixel)</td><td>912.36</td><td>638.19</td></tr><tr><td>Fy (pixel)</td><td>910.26</td><td>638.19</td></tr><tr><td>Distortion</td><td>Inverse Brown Conrady</td><td>Brown Conrady</td></tr><tr><td>Coeffs</td><td>0 0 0 0 0</td><td>0 0 0 0 0</td></tr><tr><td>FOV (deg)</td><td>70.09 × 43.16</td><td>90.16 × 58.85</td></tr></tbody></table>
<table><caption>**Table 2**

Extrinsic of the D435i camera.</caption><thead><tr><th></th><th>"Color" to "Depth"</th><th>"Infrared 1" to "Depth"</th><th>"Infrared 2" to "Depth"</th></tr></thead><tbody><tr><td>Rotation</td><td>0.9999; -0.0089; -0.0044</td><td>1; 0; 0</td><td>1; 0; 0</td></tr><tr><td>Matrix</td><td>0.0089; 0.9999; 0.0009</td><td>0; 1; 0</td><td>0; 1; 0</td></tr><tr><td></td><td>0.0044; -0.0009; 0.9999</td><td>0; 0; 1</td><td>0; 0; 1</td></tr><tr><td>Translation Vector (m)</td><td>-0.0148; -0.0001; -0.0002</td><td>0; 0; 0</td><td>0.0502; 0; 0</td></tr></tbody></table>

<table><caption>**Table 3**

Feature extraction COLMAP parameters.</caption><thead><tr><th>Camera model</th><th>pinhole</th></tr></thead><tbody><tr><td>Shared for all images</td><td>Yes</td></tr><tr><td>Custom parameters</td><td>RGB: 912.360291, 910.268250, 648.570679, 363.666290<br>IR: 638.149170, 638.149170, 639.755127, 356.510254</td></tr><tr><td>Max_image_size</td><td>3200</td></tr><tr><td>Max_num_features</td><td>8192</td></tr><tr><td>First_octave</td><td>-1</td></tr><tr><td>Num_octaves</td><td>4</td></tr><tr><td>Octave_resolve</td><td>3</td></tr><tr><td>Peak_threshold</td><td>0.00667</td></tr><tr><td>Edge_threshold</td><td>10</td></tr><tr><td>Estimate_affineShape</td><td>No</td></tr><tr><td>Max_num_orientations</td><td>2</td></tr><tr><td>Upright</td><td>No</td></tr><tr><td>Domain_size_pooling</td><td>No</td></tr><tr><td>Dsp_min_scale</td><td>0.16667</td></tr><tr><td>Dsp_max_scale</td><td>3</td></tr><tr><td>Dsp_num_scales</td><td>10</td></tr><tr><td>Num_threads</td><td>-1</td></tr><tr><td>Use_gpu</td><td>Yes</td></tr><tr><td>GPU_index</td><td>-1</td></tr></tbody></table>

The intrinsic parameters of the camera are set manually and shared between all images. Then, the geometric matching and verification are performed using sequential matching, which is best suited for consecutive frames with sufficient visual overlap. The overlap is set to 20, with quadratic overlap and loop detection enabled. The values of the other parameters are kept as default. Tables 3 and 4 summarize all parameters values.

Loop closure detection is used through a pre-trained vocabulary tree. The GPU accelerates the matching process. Once the matching step is finished, the sparse reconstruction is launched. Data is loaded from the database into memory during this process, and the scene is expanded by incrementally registering the images from an initial image pair seed. Finally, a model can be exported, containing the camera information, the images including all the keypoints and the reconstructed pose of an image specified as the projection of the world to the camera coordinate system of an image using a quaternion and a translation vector, and finally the 3D points in the dataset. After the model is acquired, the reconstructed poses of the images are used to calculate the coordinates of the center of the projection/camera using Eq. (1).

$$c_c = -R^T t \quad (1)$$

where $c_c$ is the coordinates of the camera center, $R^T$ is the transpose of the rotation matrix obtained from the quaternions, and $t$ is the translation vector. For the scaling of the trajectory, we proceed to a dense reconstruction of the environment. This step consists of importing the sparse 3D model and launching the MVS, which first involves undistorting the images. The normal and depth maps are computed to be fused into a dense point cloud and finally estimating the dense surface using Poisson or Delaunay reconstruction. This dense point cloud will allow us to recover the distances of some objects with known dimensions to calculate the ratio between the distances on the point cloud and those measured with a rangefinder. This scale factor allowed
<table><caption>**Table 4**
Feature matching COLMAP parameters.</caption><tr><td>Overlap</td><td>20</td></tr><tr><td>QuadraticOverlap</td><td>Yes</td></tr><tr><td>Loop detection</td><td>Yes</td></tr><tr><td>Loop detection_period</td><td>10</td></tr><tr><td>Loop detection_num_images</td><td>50</td></tr><tr><td>Loop detection_num_nearest_neighbors</td><td>1</td></tr><tr><td>Loop detection_num_checks</td><td>256</td></tr><tr><td>Loop detection_num_images_after_verify</td><td>0</td></tr><tr><td>Loop detection_max_num_features</td><td>-1</td></tr><tr><td>Vocab_tree_path</td><td>32K visual words (for small-scale images)<br>256K visual words (for medium-scale images)</td></tr><tr><td>Num_threads</td><td>-1</td></tr><tr><td>Use GPU</td><td>Yes</td></tr><tr><td>GPU_index</td><td>-1</td></tr><tr><td>Max_ratio</td><td>0.8</td></tr><tr><td>Max_distance</td><td>0.7</td></tr><tr><td>Cross_check</td><td>Yes</td></tr><tr><td>Max_num_matches</td><td>32768</td></tr><tr><td>Max_error</td><td>4</td></tr><tr><td>Confidence</td><td>0.999</td></tr><tr><td>Max_num_trials</td><td>10000</td></tr><tr><td>Min_inlier_ratio</td><td>0.250</td></tr><tr><td>Min_num_inliers</td><td>15</td></tr><tr><td>Multiple_models</td><td>No</td></tr><tr><td>Guided_match</td><td>No</td></tr></table>

us to scale our reference trajectory. Fig. 7 summarizes the process of trajectory reconstruction using COLMAP.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships which have or could be perceived to have influenced the work reported in this article.

## References

[1] R. Mur-Artal, J.D. Tardos, ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras, IEEE Trans. Robot. 33 (5) (2017) 1255-1262 arXiv:1610.06475, doi:10.1109/TRO.2017.2705103, http://ieeexplore.ieee.org/document/7946260/

[2] M. Labbé, F. Michaud, RTAB-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation, J. F. Robot. 36 (2) (2019) 416-446, doi:10.1002/rob.21831.

[3] C. Campos Martínez, R. Elvira, J.J. Gómez Rodríguez, J.M. Montiel, J.D. Tardós, ORB-SLAM3: an accurate open-source library for visual, visual-inertial and multi-map SLAM, arXiv:2007.11898.

[4] J. Sturm, N. Engelhard, F. Endres, W. Burgard, D. Cremers, A benchmark for the evaluation of RGB-D SLAM systems, in: Proceedings of the IEEE/RSJ International Conference on Intelligent and Robotic Systems, IEEE, 2012, pp. 573-580, doi:10.1109/IROS.2012.6385773.

[5] A. Handa, T. Whelan, J. McDonald, A.J. Davison, A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM, in: Proceedings of the IEEE International Conference on Robotics and Automation, IEEE, 2014, pp. 1524-1531, doi:10.1109/ICRA.2014.6907054.

[6] I. EL BOUAZZAOUI, S. RODRIGUEZ FLOREZ, A. EL OUARDI, Digiteo_seq2, 2021, https://data.mendeley.com/datasets/tb9g7th9yz/2, doi:10.17632/tb9g7th9yz.1.

[7] J.L. Schonberger, J.M. Frahm, Structure-from-motion revisited, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, IEEE Computer Society, 2016, pp. 4104-4113, doi:10.1109/CVPR.2016.445.

[8] J.L. Schöninger, E. Zheng, J.M. Frahm, M. Pollefeys, Pixelwise view selection for unstructured multi-view stereo, in: Lecture Notes in Computer Science(including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9907 LNCS, Springer Verlag, 2016, pp. 501-518, doi:10.1007/978-3-319-46487-9_31.
[9] S. Dorodnicov, A. Grunnet-Jepsen, G. Wen, Projection, texture-mapping and occlusion with intel Re-alSense depth cameras (2019). https://dev.intelrealsense.com/docs/projection-texture-mapping-and-occlusion-with-intel-realsense-depth-cameras.

[10] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2004, doi:10.1017/cbo9780511811685. https://www.cambridge.org/core/books/multiple-view-geometry-in-computer-vision/0B6F289C78B2B23F596CAA76D3D43F7A.