LiDAR and cameras: automation and autonomy
By: Marcus Gee
Tags: Technology, Autonomous Vehicles, Computer Vision, Lidar, Autonomous Cars

Photo by Jp Valery on Unsplash

Stereoscopic cameras and LiDAR

I hope it is becoming clear that comparing LiDAR to cameras is not apples to apples, its not even apples to oranges, its more like apples to tree trunks. These two technologies don’t stack against each other as they have completely different functions with only a minor overlap. However, they can intrude into each others territory, here comes the stereoscopic camera.

First two search results about Elon and stereoscopic cameras. This absolutist binarity makes it really difficult to find good information.

The big buzz was generated after the breakthrough development by Cornell researchers that improved object detection from previous 22% to a whooping 74%. This is a fantastic achievement, however, perhaps a bit undernoted, this is at 30 meters distance. 30 meters is the distance traveled over 1 second at highway speed. On top of that, they did not match the accuracy of a LiDAR but the accuracy of object identification which is different but perhaps related.

So what is the actual limitation of stereoscopy? Well, we measure distance to stars using a similar principle — image deviation based on camera position: we take a picture of a star at two points as earth travels around the sun and compare these images to find the distance. You see the trend — long optics and large distance between cameras (stereo-base) produces accurate distance estimation for far objects. Short optics, low resolution, low stereo-base, as you expect, high error becoming eventually infinite as the error grows exponentially towards an asymptote (to be precise, the error chart is discrete and at some point the error flips to infinity so not really asymptotic). Sample depth error estimation shown below.

Depth perception error chart for a 120 deg FoV, 2.0f optics camera with a 1/2" sensor and 2000x1500 resolution. Depth error increases exponentially

There is a range of factors affecting the error, I found a decent chart in a doctoral thesis about robotic aerial mapping vision-aided navigation systems (pdf link, page 116, author: Fadi Atef BAYOUD).

This chart is missing a critical piece of information, but I found it in the writing. The sensor size was a standard CCD 1/3" so the field of view of these cameras were 22 and 11 degrees respectively at 480p resolution. So with a significant zoom and a wide spread of the cameras we can get a decent depth estimation.

Stereoscopic camera vs LiDAR TL;DR

We reviewed 3 cases for stereoscopic setup, high resolution camera with 120 degree horizontal field of view and 2 lower resolution cameras with smaller sensors and narrow fields of view, 22 and 11 degrees. Their depth estimation error at 30 meters was 180 cm, 50 cm, 18 cm. Keep in mind that the error grows exponentially. For comparison, LiDAR depth accuracy at 120 meters is 5 cm for 360 degree horizontal field of view.

So next time you see something like this:

Headline of an article misleadingly describing researchers findings.

keep in mind the study limitations — benchmark at 30 meters range benchmarking average precision where LiDAR still produced 2x more precise results.

Implications

How important is low error distance estimation? It is not a big deal for driving autonomy level 2 or 3 but it becomes critical for level 4 and 5. You can follow the link below to a discussion about autonomy levels and why level 3 is a very tricky spot.

We are going to try an illustration scenario. Again, this is a little oversimplified (in particular, how error works) but serves as a decent example: the scenario is merging onto a high speed street or highway. Imagine estimating speed of incoming traffic with +/- 30% error margin — at this point it becomes unreliable. Let’s calculate at what distance such error potentially occurs:

Vehicle travelling at 60 mph may be perceived as 40 mph or 80 mph.

Converting this to meters, 26 mps speed can be perceiver as 17 or 35.

Taking worst case scenario (unlikely but possible) two consecutive distance estimation result in opposite error range reading [ (35–17)/2 = 9].

So, we have 6.5 m error. For the 120 deg hFOV camera, this occurs somewhere in the 70 m distance range.

Best case scenario tells us the car will reach us in 4.1 seconds and we need to accelerate to 40 mph in ballpark 6 seconds, easy.

Worst case: the vehicle reaches us in 2 seconds and we need to drive at 80 mph within 3 seconds.

Which one do you pick? If you wait for more data, your merge opportunity closes no matter what scenario it was and the guy behind you starts to boil.

All you know is that merge is something in between a causal and impossible turn, depending on the error swing.

On top of that, for 360 degree distance estimation we need 8 cameras and enough processing power to quickly chew the data. Improvement of distance estimation can be achieved by lowering the horizontal field of view, say at 70 degrees you need 16 cameras but for comfortable operation, 40 degree hFoV should be already satisfactory, so 32 cameras, decent optics, low latency, good network, automotive grade components, and a lot of processing power, perhaps FPGA(s). Where does that lead us? $6k? $9k?

It might seem like I am not a big fan of stereoscopic cameras. Quite the contrary, however. I believe the technology is perfectly suited for a lot of robotic and industrial application where most of the action occurs relatively close to the camera and in static, predictable, well illuminated setups. I would definitely chose a stereoscopic camera solution for short range sensing over a LiDAR or LiDAR + camera combo.