Real-Time Dense Mapping for Self-Driving Vehicles using Fisheye Cameras
Zhaopeng Cui1, Lionel Heng2, Ye Chuan Yeo2, Andreas Geiger3, Marc Pollefeys1,4, Torsten Sattler5
1Department of Computer Science, ETH Zürich
3MPI-IS and University of Tübingen

2DSO National Laboratories

4Microsoft, Switzerland

5Chalmers University of Technology

Motivation•

Contributions

Real-time 3D mapping required to perceive and thus navigate in complex
environments.

•Higher-resolution 3D maps obtained from images compared to LiDAR.

•More well-developed scene understanding techniques for images than those for
LiDAR point clouds.

•

A practical system for real-time dense mapping purely using fisheye cameras.

•A new multi-scale strategy for fisheye depth map estimation to maintain bothaccuracy and efficiency.

•Evaluation of multiple depth filtering and local map pruning techniques with LiDAR
data.

System Overview
Synchronized
FramesDepth Map Estimation at Multiple Scales
Downsampled Image

Depth Map
Depth Map
Estimation

Downsample

Computed Depth

StereoUpsample

StereoReplace

TSDF-based DepthMap Fusion

CameraReferenceFrame

Dynamic Object
Detection

Dynamic Object
Masks

Full Resolution Image

Camera Poses

Localization

CropCropped Image• Step 1: Compute the depth map given multiple images captured at the same time.
• Step 2: Detect dynamic objects using YOLOv3 [1] with the finetuned model.
• Step 3: Integrate depth maps over time into a truncated signed distance function
volume using camera poses provided by a localization system.

Computed Depth• Two scales: 1) the downsampled fisheye image; 2) the cropped central area of thefisheye image (close-to-pinhole camera).
• Use the plane-sweeping stereo algorithm [2] for the depth map estimation.
• Reduce the running time by about 28% compared to processing original images.

Depth Map Filtering(a) Step 1

Fused Depth Map

TSDF-based Depth Map Fusion(b) Step 2(c) Step 3• Filter Step 1: Filter with the matching cost value of the best depth candidate for apixel.

• Adapt the depth map fusion pipeline [3] to support the fisheye camera model.

• Filter Step 2: Filter with the ratio between the first and second best cost values.

• Maintain a local map with a size of 60m × 60m × 3m centered at the current
vehicle position for online mapping.

• Filter Step 3: Filter with the local depth continuity checking.

• Consider voxel blocks with at least 3 observations only.

Experimental Evaluation
South Buona Vista

Experimental Setup

AutoVision [4] vehicle platform.

Component

Runtime

Depth map estimation (5 cameras)

60 ms

Dynamic object
detection

40 ms

Depth fusion

20 ms

One North

Evaluation of the 3D Mapping Stage

Average runtime.
Evaluation of mapping results with and without unreliable voxels removal.

Evaluation of the Depth Estimation Stage

Evaluation of Object Detection
Impact of the filtering stages on the depth error (in meter).

Reference image

Ground truth depth map

Error map for half resolution

Error map for multi-scale strategy

Performance with different camera configurations.

Recovered 3D points (left) without and (right) with moving object detection.

References[1] J. Redmon and A. Farhadi. YOLOv3: An incremental improvement. CoRR, abs/1804.02767, 2018.
[2] C. Häne, L. Heng, G. H. Lee, A. Sizov, and M. Pollefeys. Real-time direct dense matching on fisheye images using plane-sweeping stereo. In International Conference on 3D Vision (3DV), 2014.
[3] O. Kähler, V. A. Prisacariu, C. Y. Ren, X. Sun, P. Torr, and D. Murray. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Transactions on Visualization and Computer Graphics(TVCG), 21(11):1241–1250, 2015.
[4] L. Heng, B. Choi, Z. Cui, M. Geppert, S. Hu, B. Kuan, P. Liu, R. Nguyen, Y. C. Yeo, A. Geiger, G. H. Lee, M. Pollefeys, and T. Sattler. Project autovision: Localization and 3d scene perception for an autonomousvehicle with a multi-camera system. In IEEE International Conference on Robotics and Automation (ICRA), 2019.