Vzense DS Series Time-of-Flight Camera
Accuracy and Precision Testing Data

Nov 2022
Revision 1.0

Revision History

Revision
1.0

Description•

Initial VersionDate
Nov 20222Contents
1Introduction ............................................................................................................................ 4

2Test Condition ......................................................................................................................... 5

3

Introduction to DS Series ........................................................................................................ 7

4

DS Series Accuracy Testing ..................................................................................................... 85

DS Series Precision Testing ................................................................................................... 10

65.1

Relations between precision and distance .............................................................. 105.2

Relations between precision and exposure time .................................................... 11

More test Data reference in different environment ............................................................ 126.1

More data as supplement to the test result shown above ................................... 126.2

Items in a bin/box ............................................................................................. 126.3

Corn pile in a corner with sunlight ...................................................................... 126.4

Neatly stacked boxes in a warehouse ................................................................. 133

1Introduction
There are a variety of metrics that can be defined to evaluate depth quality. A common
environment that is compatible with all tests is recommended to be used, such as asmooth and flat surface with a matt finish uniform, aligned parallel to the depth camerathat is placed at a controllable and known distance from the camera under test. White or
light color board mounted to a rigid frame or structure is a suitable target.
Depth data accuracy and precision (also called temporal noise) are the primarycharacteristics for judging the depth camera image quality.
Accuracy
Accuracy is the mean difference between the measured distance and the true distance(ground truth). It is a description of systematic absolute errors.
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =𝑀𝑒𝑎𝑠𝑢𝑟𝑒𝑑 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 − 𝑡𝑟𝑢𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑡𝑟𝑢𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒

Precision
Precision is mostly defined as a standard deviation of accuracy. It is also called temporalnoise or repeatability. Precision measures the variation in depth values over a specificnumber of frames.
𝑁

1
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑 𝑑𝑒𝑣𝑖𝑎𝑡𝑖𝑜𝑛 = √ ∑(𝑥𝑖 − 𝑢)2
𝑁𝑖=1

The following diagrams illustrate the relation between the two aspects:

FIGURE 1: RELATION OF PRECISION AND ACCURACY4

2Test Condition••••••Target: flat 80cm*80cm 40% reflectivity grey board mounted on bar attached to motion
stage (refer to figure 1)
Camera positioning: mounted on a steel fixture (refer to figure 2), laser distance meterused to measure distance from case front to the grey board
Camera setting: default setting (refer to figure 3), camera warmed up for 20 minutes.
Imaging environment: 22℃ room temperature, 200Lux room light on during testing,black curtain used on one side to minimize reflections around (refer to figure 1)
Motion stage moves from 0.4meters to 4.5meters, and stopped in 0.4m, 1m,1.5m, 2m,2.5m, 3m, 3.5m, 4m, 4.5m. During each step measure depth over 10×10 pixel ROI(region of interest) at image center, repeat 32 times at each position.
Depth metrics are measured and recorded in real time by using the self-developed tool.
For the evaluation of absolute depth values we use the mean depth of the image set ineach pixel. The standard deviation is computed based on the deviation in an image set.

FIGURE 2: 80CM*80CM 40% REFLECTIVITY TARGET & IMAGING ENVIRONMENTFIGURE 3: CAMERA MOUNTED ON THE FIXTURE5FIGURE 4: DEFAULT CAMERA SETTING

63Introduction to DS Series
Vzense DS series features Sony depthsense continuous wave iToF sensor with 640*480
depth resolution and two infrared VCSEL emitters. DS series consists of a suite of cameras
including DS77 Pro, DS77 Lite and DS77C Pro, DS77C Lite.
DS series produces high quality depth data which is very helpful for a variety of market
segments including mobile robotics, volumetric scanning, target recognition, etc. The
depth data is calculated inside the camera and users can get point cloud via SDK from thehost.
Besides, DS77C Pro and DS77C Lite are configurated with a 1600*1200 resolution RGB
sensor which provides acquisition of mapped RGB and mapped depth images. Please referto the DS series datasheet for the detailed description of these modules and their features.
The difference between the models is listed below briefly.

DS77 Pro
DS77 Lite
DS77C Pro
DS77C Lite

RGB

PoE+
√IP67
√

√
√

√

√

TABLE 1: DIFFERENCE OF DS SERIES MODELSFIGURE 5: DS77 CONNECTORSFIGURE 6: DS77C CONNECTORS

7

4

DS Series Accuracy Testing
To achieve a result as accurate as possible, Vzense cameras are calibrated at the factory
and tested to guarantee a reliable accuracy. Vzense DS series cameras are calibrated fora measurement range of 0.4~4.5meters. Technically speaking, the camera is capable ofmeasuring distances as minimal as 0.15meters so if users have quite near measurement
requirement, we also offer products at a customized range.

Test Result
Below Table 2 shows the data of two units’ mean measurement error and accuracy basedon the test conditions listed above. Measurement Error keeps in millimeter or evensubmillimeter precise and accuracy is lower than 1%.
Unit 1:
Test
No.
123456789Ground
Truth(mm)5206201118
1617.52620
3123
3621.54121
4620

Mean Measurement
Value (mm)
515.5
616.8
1115.21614.32618.83118.23620.44120.14620.7

Mean Measure
Error (mm)
-4.50
-3.20
-2.80
-3.20
-1.20
-4.80
-1.10
-0.90
0.70

Accuracy(%)-0.865%-0.516%-0.250%-0.198%-0.046%-0.154%-0.030%-0.022%0.015%

Test
No.
123456789Ground
Truth(mm)4009951496
19972497
2998
35003999
4498

Mean Measurement
Value (mm)
401.6
995.6
1494.4
1999.22500.42999.4
3500.13999.54511.6

Mean Measure
Error (mm)
1.6
0.6
-1.6
2.2
3.4
1.4
0.1
0.5
2.2

Accuracy(%)
0.400%
0.060%-0.107%0.110%
0.136%
0.047%
0.003%
0.013%
0.049%

Unit 2:TABLE 2: TWO UNITS ACCURACY TEST DATA

8

15105Measurement
Error(mm)0-5-10-15
0.000

1.000

2.000

3.000

4.000

5.000

Distance(m)Unit 1

Unit 2

GRAPH 1: MEASUREMENT ERROR IN DIFFERENT DISTANCE

1.00%
0.80%

Accuracy 0.60%(%)
0.40%
0.20%
0.00%
0.000

0.500

1.000

1.500

2.000

2.500

3.000

3.500

4.000

4.500

5.000

Distance(m)Unit 1

Unit 2

GRAPH 2: ACCURACY IN DIFFERENT DISTANCE

95DS Series Precision Testing
Precision measures the variation in depth values over a specific number of frames, so it isalso called temporal noise or repeatability. The purpose of precision measurement is tounderstand the depth camera’s time-dependent aspects of per pixel Z-accuracy. It ismeasured per pixel and then an average or median is taken over the ROI.

Test Result
5.1

Relations between precision and distance

Following the test conditions described in Part 2, with the assistant of a self-developed
software tool, average accuracy is measured over 10×10 pixels ROI at image center,
repeat 32 times at each position then precision data as a standard deviation in depth
values is calculated.
Unit 1:
Test
No.
12
3
4
5
6
7
8
9

GroundTruth(mm)
520
620
1118
1617.52620
3123
3621.54121
4620

Mean Measurement
Value (mm)
515.5
616.8
1115.21614.32618.8
3118.23620.44120.14620.7

Precision
(mm)
1.70
2.00
3.10
5.90
9.30
15.50
21.70
25.50
34.10

Test
No.
12
3456
78
9Ground
Truth(mm)400995
1496
1997
2497
2998
3500
3999
4498

Mean Measurement
Value (mm)
401.6
995.6
1494.4
1999.22500.42999.4
3500.13999.54511.6

Precision
(mm)
1.40
3.10
6.50
7.20
12.00
14.10
20.20
30.90
25.60

Unit 2:TABLE 3: TWO UNITS PRECISION IN DIFFERENT DISTANCE (EXPOSURE TIME: 1MS)

10

40.0030.00

Precision
20.00
(mm)

10.00
0.00
0.0001.000

2.000

3.000

4.000

5.000

Distance(m)Unit 1

Unit 2

GRAPH 3: TWO UNITS PRECISION IN DIFFERENT DISTANCE(EXPOSURE TIME: 1MS)

5.2

Relations between precision and exposure time

To help users understand the factors to influence precision, in each distance, we tested
the precision with different exposure time setting at 1ms, 2ms, 3ms and 4ms. Below datashows that the longer exposure time the camera is set, the higher precision the camera
can get, as long as images are not over exposure.
40353025Precision20(mm)
151050
0.00

1.00

2.00

3.00

4.00

5.00

Distance(m)4ms exposure time

3ms exposure time

2ms exposure time

1ms exposure time

GRAPH 4: PRECISION INFLUENCE WITH DIFFERENT EXPOSURE TIME

116

More test Data reference in different environment

6.1

More data as supplement to the test result shown above
Saved images and data: RGB, IR, Depth, point cloud
Test target: flat 80cm*80cm 40% reflectivity grey board
Distance: 0.4~4.5meters
Ambient Light: room light from ceiling, ~200Lux
Download link:
https://drive.google.com/drive/u/1/folders/1qegY4Q8aAEJBadKtlav7wiwfnY0Q-aEP

6.2

Items in a bin/box
Saved images and data: RGB, IR, Depth, point cloud, registered color to depthDistance: 1.1~1.3meters
Test target: items in the bin and box including white boxes with/without poly films, packs
of batteries, power strips in poly film, plastic bottles
Ambient Light: room light from ceiling, ~200Lux
Download link:
https://drive.google.com/drive/u/1/folders/1a0gW2Chb_oepI--g5xlAMCGX4AKJUl3P

6.3

Corn pile in a corner with sunlight
Saved images and data: RGB, IR, Depth, point cloud12

Test target: corn pile
Ambient Light: sunlight, ~20,000Lux
Download link:
https://drive.google.com/drive/u/0/folders/1XOpmaScvUivulv_HbjRUTMQwmsVRnp_?hl=zh-TW

6.4

Neatly stacked boxes in a warehouse
Saved images and data: Depth, point cloud
Distance: 1.7meters
Test target: stacked boxes
Ambient Light: room light from ceiling, ~300Lux
Download link:
https://drive.google.com/drive/u/0/folders/1yHFlClzksA0WAFGm3U8s1VitFs__yFaT?hl=zh-TW

13