John Sweetser

RealSense Group, Intel Corp
2

➢ What is Depth Quality and what makes a good depth camera?

o Quantitative & Qualitative assessment

➢ Depth Quality evaluation

o Performance Metrics (KPI’s)

o Test and Characterization Methods and Tools

o Performance Criteria (Standards)

➢ Sample depth quality test data

➢ Factors that influence depth quality 

➢ Kpi’snot currently tested

➢ effect of camera calibration on depth quality 

Discussion topics
3

What is Depth Image Quality?

• Depth Camera: adds the distance (Z) dimension to traditional 2D RGB or B&W image. 

attributes such as sharpness, distortion/uniformity, color fidelity, noise, and dynamic 

range, etc.

• Depth Image: typically represented as 2D ‘depth map’ or 3D ‘point cloud’.

• Depth Image Quality: Evaluation of the depth image quantitatively (using 

predefined metrics) or qualitatively (using visual clues). 

Typically, quantitative metrics are used in simplified scenes and controlled conditions 

and qualitative assessment is used in arbitrary or complex scenes
4

WHAT MAKES An ideal DEPTH CAMERA?

1. See everything:

a) All conditions: From darkness to bright sunlight

b) All materials & objects

c) All ranges

d) No interference

2. See it with little noise (high precision)

3. Get exact distance (high accuracy)

4. And cheap, small, low-power, wide field-of-view, high-speed, color...
5

Sample “Good” and “BAD” depth images
6

Image Quality Evaluation

Qualitative

Based on measurements performed on the camera, depth data can be analyzed to produce 

metrics designed to quantify performance. 

Based on visual inspection of the depth map or point cloud and assess quality based on known 

properties of the scene

Quantitative
7

Key Depth Metrics

Sample of raw depth data from a DS5 camera during a flat target test

A very good snapshot of depth performance can be seen from the histogram of flat target depth values. Ideally, this distribution

is narrow, centered near the known (ground truth) distance, and has a complete number of sample points.

• Fill Ratio: Percentage of “valid” (w/nonzero depth) pixels over ROI.

*Typical good value: >99%, <98% poor

• Z-Accuracy: Offset of mean/median 

depth from ground truth.

Typical good value: <1%, >2% poor

• RMS Error (Spatial Noise): 

Variation in depth over ROI.

Typical good values: <0.4% (~0.2 pixels), >1% poor 

• Temporal Noise: Variation in depth 

per pixel over time (frame-to-frame).

Typical good values <0.2%, >0.5% poor

# valid pixels -> Fill Ratio

Mean Z -> Accuracy

STD -> Spatial Noise

Center GT

Width

(spatial noise)

GT – Center

(accuracy)

Metric values may be expressed in absolute units, e.g., mm or as % of depth. 

*For D415 @ Z ~ 1m, HD resolution, center 40% ROI, Active

Consult datasheet for latest specifications
8

Image Quality Measurement –BasicZ-Performance 

• Quantitative depth quality is evaluated primarily based on “flat target” testing (either textureless or textured). 

This provides a simple, well-defined, and standardized environment to capture images and compute metrics.

• In all measurement methods, image data is captured and then analyzed, either off-line or in real-time, to 

compute the performance metrics. 

• Measurements are performed as a function of distance from the target and may be run for different 

resolutions, frame rate, depth settings, ambient conditions, etc. 

A typical camera characterization apparatus 

Textured target 

(projected or physical)
9

Depth Quality Measurement Tools

• Offline: Typically used for official validation. Depth data captured and analyzed later.

• Real-time: Test application that captures, analyzes data, and computes metrics in real-time (per frame or 

based on sequence). Metrics are usually a subset of full validation that contain only the key metrics needed for 

basic depth camera health check.

• For D400 cameras, Depth Quality Tool is the recommended tool for basic real-time measurements
SN 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Center 40% 2.62 5.28 4.21 13.19 9.41 21.20 35.95 36.28

Full FOV 4.60 9.12 12.29 20.85

mean Error (mm)

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Error (mm)

Distance (m)

Mean Error

Sample Depth Quality Data – ASR short range

1280x720, 30 FPS, P=210mW, AE

Target: Flat white wall, ~100-200 Lux fluorescent lighting

0.00

5.00

10.00

15.00

20.00

25.00

30.00

0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

StD (mm)

Distance (m)

Plane fit RMS

SN 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

Center 40% 2.13 3.46 3.71 7.77 7.69 16.59 26.68 26.69

Full FOV 3.13 5.39 10.16 13.85

std (mm)

Full

Full

Center

Center

2% error

10
11

Factors affecting Depth Quality

Product use cases largely drive environment in which depth quality is assessed from among factors below

Lighting - should be tested in different lighting conditions in which the product is used. 

Different technologies will behave differently in specific lighting conditions. 

Range or Distance

- Ex 1: A room scanning device designed to remain in the center of a large space and rotate while 

capturing walls, ceiling, and objects in detail will require accurate depth at long distance. 

- Ex 2: A robot or drone in motion can use depth at longer distances for path planning without 

requiring accurate depth at those distances.

Shape

- Quantitative testing currently done with flat targets due to Ground Truth availability.

- Qualitative testing looking at the point cloud for edge fidelity, flat or round surfaces, and proper 

angles on different geometric shapes.

Different Materials, Flat uniform surfaces (Ex. White board/wall), Textured Patterns

- Ex: Autonomous vacuum cleaners would test different floor materials such as light and dark tile, 

wood, carpet, and linoleum; body scanning would test materials that might be worn by the user and 

different colors/patterns of that material.
12

Additional 3d image Quality characteristics of interest

Types of quantitative testing/characterization not currently done (@ Intel)

• 2D (x,y) spatial resolution:

Resolution chart with variable width slots or features.

• Minimum detectable object size:

Targets with variable size objects (spheres, cylinders) - 3D resolution.

• Edge Fidelity:

Sharpness of edges (depth discontinuities). 

• Full 3D Object/Scene Reproduction:

Error in reproducing a specific scene or object (e.g., mannequin). RMSE from ground truth.
13

Effect of calibration on depth quality

• Approximately 27 parameters that are determined during a full calibration procedure: 

o Intrinsic – individual camera factors (PP, FL, distortion)

o Extrinsic – relative left-right camera position & orientation

• Post-factory re-calibration can usually be done by adjusting 1-2 intrinsic and/or extrinsic parameters:

o Px/Py – shift of lens-sensor position to adjust disparity or alignment of images (~0.2 – 1 pixel).

o Rx/Ry/Rz – rotation of camera for same purpose (<0.2 deg).

Most degradation of depth quality can be corrected quickly with proper adjustment of one or more of these 

parameters. 

Temperature variations, mechanical shock/vibration or stress can lead to degradation that requires recalibration. 

15

Other performance metrics

“Fill Factor” – Combines fill ratio, Z accuracy, and RMSE in to a single figure of merit.

Example: % of all pixels that are within 3% of ground truth distance. 
16

Facial 

Authentication Robotics Scanning Measurement Tracking

3D camera applications
17

Stereo Cameras

D415 D435

D435i D465

Tracking Camera

T265

LiDAR Camera

L515

Coded Light Camera

SR305

Stereo Modules

D410 D415

D420 D430 Coded Light Module

SR300

D4 Vision Processor and 

Board

Q1 2020 Launch Q1 2020 Launch
18

Depth Quality Dependence on Technology

• Depth quality evaluation methods and metrics in general do NOT depend on underlying technology, 

however the image quality itself may have technology-specific characteristic.

• Active depth Technologies such as Structured Light and TOF rely on projected light and therefore work well 

in scenes with little texture and low lighting, such as uniform walls in a factory or office environment. 

Therefore, these are the conditions recommended for evaluation.

• Stereo depth (such as D400 family) which does not rely solely on projected light and can benefit from natural 

texture and ambient lighting, may be evaluated in a variety of scenes and conditions.

Lighting Target Scene Distance

Low 

Light

Sunlight Indoor 

normal

light

Uniform, high 

reflectivity surface 

(Ex. Flat White wall)

Texture Geometric

shapes/edges

Materials (Low

Reflectivity, 

Diffuse, Dark)

Near Far

Stereo

Active Stereo

TOF

Structured

Light
19

Sample Depth Data – More examples
Intel® RealSenseTM D400 SERIES depth QUALITY

Intel RealSense Group

2D RGB Image

DEPTH MAP(1280x720) POINTCLOUD WITH COLOR TEXTURE

D415

3D test scene configured to include a variety of object types, textures and distances, captured through RealSenseTM Viewer 2.8.1.

HIGH DENSITY PRESET

HIGH ACCURACY PRESET

Ex. Usages: 

BGS

3D Enhanced 

Photography

Collision 

Avoidance 

Ex. Usages: 

3D Object 

Scan

 RealSenseTM 400 series provides excellent depth quality under all lighting conditions, and longer range

 Great configurability - Viewer and Depth Quality tools in SDK provide different Presets (High Density, High 

Accuracy, Close Range, Hand etc.) OR users can tune their own for their applications

20
Intel® RealSenseTM D415/D435 OUTDOORs

Intel RealSense Group

2D RGB DEPTH MAP 415

435 (Note the Wider FOV)

3D test scene outdoors captured through RealSenseTM Viewer 2.8.1 – 10m?.

 Stereo takes advantage of 

visible light for best 

Outdoor performance and 

Range

 Projector can be off with 

enough visible light and 

texture => low power!

21