How to Choose a 3D Vision Technology
Chris Osterwood
Chief Technology Officer, Carnegie Robotics
osterwood@carnegierobotics.com

Copyright
Copyright©©2017
2017YourCarnegie
CompanyRobotics
Name1

Carnegie Robotics, LLC• Supplier of reliable robotic components and systems.
• We have particular focus on and capability in autonomous and semiautonomous mobile ground robots. This includes perception, autonomy,positioning, and safety — for size, weight, power, and cost constrainedapplications.

Copyright © 2017 Carnegie Robotics

2Outline• Criteria & Definitions• Overview of 3D Sensor Modalities : LIDAR, ToF Camera, Stereo• How do they work?
• What are their general advantages and disadvantages?
• Sample Data
• 3D Sensor Testing & Edge Cases• Not covered: Laser Triangulation, Structured Light, & Computed Tomography

Copyright © 2017 Carnegie Robotics

3

Evaluation Criteria - Performance• Field of View (FOV): Defines angular area of perceivable field of sensor.
• Density:• Angular step size between sample points.
• Can be different horizontally and vertically.
• Resolution: Generally FOV x Density.
• Depth Accuracy: Difference between measured range and actual range.
• Depth Resolution: Step size between possible measurement ranges (along measurementaxis)• Minimum and Maximum Range: Defines distances that are perceivable to the sensor. May
vary by object material, brightness, reflectivity, etc.
• Rate: Specified in “frames” or “points” per second, depending on modality.
Copyright © 2017 Carnegie Robotics

4Evaluation Criteria - Non-Performance• Size, Weight, & Power• Cost• Sealing: Dust proof, Water splash, Water streams, Immersion• Shock & Vibration Rating• Communication Interface: Ethernet, USB, Serial, CANBus, Firewire• Synchronization: None, Hardware, Software (broadcast trigger or network time sync)• Software Interface:• Published wire protocol?
• Drivers. Platform? Language? License?
• Example usage software?
• Other: Trigger, GPIO, Lighting, IMU, Temperature & Health Reporting

Copyright © 2017 Carnegie Robotics

5

3D Sensor Technologies

Copyright © 2017 Carnegie Robotics

3D LIDAR : How do they work?

• Building block is single-beam system above.
• 3D LIDARS can have:• 1 beam steered mechanically in 2 DOF• Multiple beams steered mechanically in 1 DOF• Steering can be done with rotating mirror, galvanometer, or directly upon
the emitter & detector.
Copyright © 2017 Carnegie Robotics

7

3D LIDAR• Advantages:• Constant error model — 1 sigma can be +/- 1cm.
• Long range. 10m to 2km possible depending on laser power and pulse
duration.
• General wide horizontal FOV due to rotating mirror or rotating emitters anddetectors.
• Can have large vertical FOV. Dependent on laser and mechanism details.
• Downsides:• Cost. Trends are encouraging here, but low cost systems are not yet
available.
• Scan time — no instantaneous capture of data. System has to compensate
for platform (or mechanism) motion during scan.
• Limited angular density & asymmetric density.
• Moving parts & non-planar windows are difficult to seal.
• Affected by obscurants in the environment.

Copyright © 2017 Carnegie Robotics

8

ToF Cameras : How do they work?
• Like LIDAR, but every pixel is a
pixel is a measurement point.
• Results in limited range or moretransmission power.
• Accuracy increases withmodulation frequency.
• Maximum range decreases withmodulation frequency.
• Systems generally use multiple
frequencies to allow for longrange and non-ambiguous range.

Copyright © 2017 Carnegie Robotics

9

ToF Cameras• Advantages:• Dense data regardless of scene texture.
• Instantaneous horizontal & vertical FOV.
• Linear error with respect to distance.
• Narrower shadow behind positive obstacles.
• Solid state — no moving parts.
• Fewer “camera” parameters.
• Downsides:• Low resolution.
• Long integration time (to increase SNR and resolve rangeambiguity) causes motion blur if platform or objects aremoving.
• Susceptible to multi-echo returns distorting data.
• Affected by obscurant in the environment.
• Limited FOV — generally both horizontal and vertical.
• May not have an “image” output.
Copyright © 2017 Carnegie Robotics

10

Stereo Camera : How do they work?
• Features matches are found in left and rightcameras. Different in lateral position(disparity) is inversely proportional to distance.
• Matching process (correspondence problem) is
critical to data accuracy and density.
• “Disparity search range” = number of horizontal
pixels in right camera searched before moving
onto the next left pixel. A larger search range
allows seeing objects closer to the camera, but
generally reduces 3D throughput.

Copyright © 2017 Carnegie Robotics

11

Active & Passive Stereo• Active Stereo: Correspondence problem aided through projection ofarbitrary texture into the scene.
• Advantages:• Very high resolution and density.
• Instantaneous horizontal & vertical FOV.
• Flexible FOV options (from 130° to 10°)
• Monochrome or color images are co-registered with 3D data.
• Many “camera” parameters.
• Passive solution is immune to obscurant in the environment.
• Solid state — no moving parts.
• Downsides:• Data density dependent on scene texture or patternprojection.
• Non-linear error model with respect to distance.
• Double shadow behind positive obstacles.
• Computationally expensive process increases BOM cost.

Copyright © 2017 Carnegie Robotics

12

Sensor Error Models• Understanding the stereo error
model is key to effective use.
• It is VERY different than every
other sensor.
• Stereo accuracy here assumes
matches with 1/2 pixel ofaccuracy. That is common inreal world environments, but
not guaranteed. Accuracy isaffected by scene texture.

Copyright © 2017 Carnegie Robotics

13

General Sweet Spots• LIDAR• High speed vehicles — well served by their long range
and high accuracy at range.
• Vehicles with high turning rates — require wide
horizontal FOV.
• ToF Camera• Indoor environments.
• Short range object scanning & gesture recognition.
• Stereo Camera• Outdoor applications with high levels of ambientobscurants.
• Multi-view applications (overlapping FOVs).
• Applications which require a non-emitting solution.

Copyright © 2017 Carnegie Robotics

14

Qualitative Comparison

Copyright © 2017 Carnegie Robotics

3D LIDAR Sample Data
• 1: 8 seconds of persisteddata.
• 2: 1 second of persisted
data, high rotation rate.
• Notice:• Wide FOV.
• Difficulty in seeing /identifying person.
• Density differences
horizontally & vertically.
• Mixed pixel returns.
Copyright © 2017 Carnegie Robotics

16

ToF Camera Sample Data : Low Cost
• 1: Axis Colored Scene• 2: Intensity Colored Scene• Notice:• Low resolution / Low Rate.
• Narrow FOV.
• Invisible / bent carthandle.
• “Flashlight effect”.
• Mid-air returns.
• 1/2 of floor missing.
Copyright © 2017 Carnegie Robotics

17

ToF Camera Sample Data : Industrial• 1 & 2: Axis colored scene.
• Notice:• High resolution / High rate.
• Distortion with near-fieldobject.
• Distortion to rear wall withforeground object.
• Multi-path distortion in floor.
• Mixed-pixel returns.

Copyright © 2017 Carnegie Robotics

18

Stereo Sample Data
• 1: Overview of scene.
• 2: Close-up of chess board.
• Post-matching cost threshold starts
high (all data), is lowered (only highconfidence data), and is raised again.
• Notice:• High density — can see pieces, cansee cart handle.
• High Rate & Wider FOV.
• Correlated scene color.
• Data below ground caused by specularreflections.
• False “spike” in rear wall caused bycart handle.

Copyright © 2017 Carnegie Robotics

19

Active versus Passive Stereo• IR Pattern Projector turnedon and off throughoutscene.
• Areas of low-confidence
have higher-confidencewith projector on. This
results in lower-noise andmore accurate data.
• Scene has RGB colormap ofmonochrome intensity.
Copyright © 2017 Carnegie Robotics

20

Stereo from Passive Thermal
OpticalStereoThermal
StereoCopyright © 2017 Carnegie Robotics

21

Testing 3D Sensors

Copyright © 2017 Carnegie Robotics

3D LIDAR : Lighting : Vendor A
300 lux : Fluorescent
8369755000 lux : Halogen
49547

Std. dev. of first row (mm)6LexanAcrylic
Black Tile

MirrorMeshWhite areas shifted
2.5cm closer to sensor
Copyright © 2017 Carnegie Robotics

233D LIDAR : Lighting : Vendor B
300 lux : Fluorescent
107791085000 lux : Halogen
119101213

Std. dev. of first row (mm)12

Polished
BrassLexan
Acrylic
Black Tile

MirrorMesh

50% more noise
than Vendor A

High IR content of Halogen
increase noise additional 40%
Copyright © 2017 Carnegie Robotics

24

ToF Camera : Lighting
300 lux : Fluorescent

5000 lux : Halogen

Polished
Brass

Lexan
Acrylic
Black Tile

MirrorMesh

Higher range noiseon dark objects

High IR content of Halogen
increases noise by 2x to 5x
Copyright © 2017 Carnegie Robotics

25

Stereo Camera : 7cm Baseline Range Accuracy
0.4 meter
4444

1.2 meter
44991210

Std. dev. of first row (mm)810

Polished
BrassLexanAcrylic
Black Tile

MirrorMesh

Higher range noise on
low-texture objects
Copyright © 2017 Carnegie Robotics

26

3D LIDAR : Error Changes Over Time
Unit A

Error to Color
Mapping
0.00 cm : Blue
0.50 cm : Green
1.00 cm : Yellow
1.25 cm: Orange
1.50+ cm: Red

January

Unit A

July

Unit B• Every laser (even same SKU) is different.
• Lasers change over time.
• Error car vary by emission angle & range.
Copyright © 2017 Carnegie Robotics

27

3D Resolution Testing• In some applications XYZ resolution
matters as much as depth accuracy.
• How do you measure 3D resolution?
Take cues from 2D camera testing.
• Raised Relief 3-D Resolution
Target• Derived from USAF 1951
resolution target• THE standard image quality testfor 60 years

Copyright © 2017 Carnegie Robotics

28

3D Resolution Testing Process• Capture Image• Identify and Isolate Target• Outputs:• Colorized Depth Image• Colorized Depth Error Image• RMS error on features andbetween features for eachgrouping size

Copyright © 2017 Carnegie Robotics

29

3D Resolution : Industrial ToF Camera

Error Image

Range Image
Copyright © 2017 Carnegie Robotics

30

3D Resolution : Stereo

Error Image

Range Image
Copyright © 2017 Carnegie Robotics

31

Take Aways• There is no perfect sensor. They all have limitations and edge cases.
• It is critical to understand the performance & non-performance criteria thatmatter for your application.
• Those weights must drive your sensor selection process.
• You may have to sacrifice one or more lesser characteristics to meet theones which really matter to you.
• Look for published test data. If none is available:• Ask the sensor manufacture,• Collect it yourself, or• Turn to those who have past experience in integration and downstream
software.
Copyright © 2017 Carnegie Robotics

32

Resources• Carnegie Robotics:• http://carnegierobotics.com/roscon-2017• http://carnegierobotics.com/support• Companies & Products:• Carnegie Robotics MultiSense• IFM O3D
• SICK 3vistor-T• pmd pico
• ZED Camera• Orbbec Persee• Structure Sensor• SICK TiM / LMS LIDARs• Velodyne 3D LIDARs• Hokuyo 2D & 3D LIDAR• Quanergy 3D LIDAR

Copyright © 2017 Carnegie Robotics

33