LiDAR and cameras: automation and autonomy
By: Marcus Gee
Tags: Technology, Tech, Autonomous Vehicles, Lidar, Autonomous Cars

Human Eye — Camera — LiDAR

Photo by Amanda Dalbjörn on Unsplash

Short intro to perception

We all have a working understanding of a camera image as the image representation maps well to how we perceive the world. Camera image is a 2D projection of a 3D scene where each pixel carries color/brightness information. Let’s assume an RGB pixel format: 3 values that combined represent practically all that we can decipher color-wise. Make it 4K HDR 30 fps and you get very close to a human eye. This is of course an oversimplification for sake of illustration as human eye has a very high resolution on the focused object and poor peripheral. Humans have a native ability to understand and infer information from a camera stream — we can distinguish distances, surfaces, 3D shapes based on light and its reflectivity, we can identify objects, estimate their relative sizes and predict their relative volumes (sort of). On top of that we have some very specialized skills like the ability to immediately spot a person looking at you in a crowd, perform ultrafast facial recognition, or detect repeating pattern with no training. It will take us a while to get there technologically. Cameras excel at capturing an image, usually aiming to realistically depict a scene to a human. However, cameras can easily beat human at capture speed. We recognize individual images at about 10 frames per second. In other words, we cant comprehend more than 10 images in a second and if shown, say 10 images in half a second, we would not be able to tell what we have seen on each of them. Cameras have an ability to unlock that frame rate much higher — effortlessly reaching 300 fps. At this point you may wonder whether can we effectively consume 300 frames provided by a camera in a second and we will get back to this question later. Its time for the resolution! As mentioned above, 4K (about 8 MP, 2000+px height) is roughly equivalent to human eyesight. Industrial grade cameras range from 0.4 MP (500px height) to 40+ MP (5000px height). And here comes a LiDAR with its 32 px height image, provided at 20 fps costing ballpark $4000. Medium size icon in Windows 10 is 48 pixels high. Quite a rip-off, it seems, for an imaging system to provide icon height images for $4000. However, we are no longer in the RGB pixel format and there is a lot of horizontal measurements to make sense of the space. This time we use 4 to 5 parameters to describe a single point in space using spherical coordinates — radius, elevation, azimuth (think of these as an equivalent of X,Y,Z), reflectivity and potentially noise reading (noise is equivalent to a monochromatic camera capture). Reflectivity can tell you about the surface that bounces off a light — a practical use case is street signs or reflective paint street lane markings. Alternatively, if you assume a somewhat uniform reflectivity of an object surface, say, a human, the reflectivity tells you the angle of the surface the light beam reflected from.

This is me holding a large tea cup described with 14 lidar channels. Cheers! Warmer color means higher reflectivity; notice the blue stroke — it implies rounded edge of an object as the beams reflect at a different angle.

Pay attention the the reflectiveness values on the image provided. This information allows accurately predicting 3D shapes of objects which happens to be very useful combined with exact location in space. On top of that, the laser beam works in full sun and complete darkness with (virtually) no accuracy loss. Driving towards the sun 1 hour before sunset makes camera’s performance less than stellar, similarly driving at night.