Towards Autonomous Vehicles 2021
By: Stephen Adhisaputra
Tags: Artificial Intelligence, Technology, Autonomous Vehicles, Future, Self Driving Cars

Vehicle Perception Technology

Perception is key to understanding the world. This section connects sensing and perception in automated driving systems. Based on its function, vehicle perception technology can be divided into two major categories:

Localization — locating the position of the vehicle on maps

locating the position of the vehicle on maps Recognition — object detection, target tracking, and motion estimation

AVs usually need to perform simultaneous localization and mapping (SLAM) to operate in a dynamic environment. Localization is considered part of the perception module here, while planning and control are part of the decision module.

Vehicle perception technology can be categorized based on its sensing medium:

Perception via radio waves (GPS, radar)

(GPS, radar) Perception via sound waves (ultrasonic sensors)

(ultrasonic sensors) Perception via light (LiDAR, Camera)

Most AV Companies rely on HD Maps for navigation. Given this map, an AV could easily perceive the environment using its technologies. The vehicles will compare and update for any changes in the map as they drive.

“If we want to have autonomous cars everywhere, we have to have digital maps everywhere,” — Amnon Shashua, CEO of Mobileye (The New York Times 2017)

Real roads are dynamic; they can change drastically due to construction or natural disaster. Some companies like Tesla and Apple set a high ambition of using the ‘mapless’ motion planning system. They rely completely on vision.

GPS/IMU

Localization can be realized using GNSS, short for the global navigation satellite system. This is more commonly known as GPS, which is a type of GNSS. The update rate of GPS is about 10 Hz.

Additionally, IMU devices are also used in AVs to enable real-time vehicle positioning in maps. IMU devices operate at around 200 Hz, a much higher update rate than GPS.

Some techniques, such as tracking filters and Sequential Monte Carlo methods, can be used to combine sensor measurements from multiple sources.

We cannot rely solely on GPS/IMU for localization because GPS is not accurate enough.

Radar

Radio detection and ranging system (Radar) in AVs can detect the speed and range of objects nearby. Radars consist of a transmitter and a receiver. AV Radars (millimeter-wave radar) emit and receive radio waves in extremely high frequency to sense the environment.

Radars measure the range of objects and relative velocity through pulse compression and Doppler shift, respectively. Radars in AVs can be long, medium, or short-ranged. All three types can be used together in an AV.

Doppler effect | Image via Wikimedia Commons

The biggest advantage of radar is that it can operate in all weather conditions. In driving applications, radars typically operate in pulses, linear frequency-modulated continuous-wave, and frequency-shift keying.

Common practical challenges of radars are time synchronization, interface communication, optimal allocation of digital signal processing, and minimizing interference. AVs usually use radars for the obstacle avoidance task alongside ultrasonic sensors.

Ultrasonic Sensors

Ultrasonic sound navigation and ranging (Sonar) also emit waves and convert the reflected waves to electric signals similar to radar with a resonant frequency of around 48 Hz. It consists of an ultrasonic transducer and a receiver circuit.

Ultrasonic sensors can ‘see’ through objects, suitable for recognizing slow-moving targets. This sensor is unaffected by weather conditions, though a rapid temperature change will affect its accuracy.

The range of objects detected by ultrasonic sensors can be calculated by measuring the elapsed time interval between emission and reflection.

Long-range ultrasonic sensors can detect objects from up to 8 meters away. Most AVs use ultrasonic sensors for safe driving applications such as automatic parking and collision warning.

There exist methods to generate images from airborne ultrasonic waves. This is a relatively new research field in medical imaging and robotics. The information content of ultrasonic images is however still inferior to LiDAR or camera. Therefore, the process of generating images from sonar is not necessary.

LiDAR

Light detection and ranging systems (LiDAR) consists of laser scanners and laser receivers. LiDAR uses ultraviolet, visible, or infrared light in the form of a pulsed laser to image objects, with up to 150,000 pulses per second.

There are three types of LiDAR: time-of-flight, triangulating, and phase-ranging. The world’s highest resolution LiDAR is the Velodyne VLS-128. It has a range of 300 meters and can record up to 2.4–9.6 million data points per second.

LiDAR calculates the distance of detected objects by measuring the pulse interval between emission and reflection of the light scattered by targets.

LiDAR technology is notorious for being quite expensive, but it has been undergoing the most technological progress in recent years. Innovations in 2020 include microelectromechanical mirrors (MEMS) and solid-state LiDAR. Those LiDARs don’t have mechanical components, thus reducing the unit cost.

MIT Spinoff Building New Solid-State Lidar-on-a-Chip System (12/2020) | Image by MIT Lincoln Laboratory

AVs can use LiDAR technology to scan the environment and create 3D digital representations. This 3D-data is useful for both localization and motion tracking.

LiDAR-based localization often also uses particle filter techniques. The data collected by LiDAR fused with the GPS/IMU data in sensor fusion. This method can achieve real-time position accuracy of within 10 cm.

The performance of liDAR strongly depends on the good calibration of the device. This involves many variables, such as the choice of parametrization and data segmentation.

A point cloud image of a vehicle in an intersection, projected from Velodyne’s HDL-64E LiDAR | source

Camera

Most AVs rely heavily on camera devices. AVs usually mount eight 1080p cameras capturing images at 60 frames per second. With multiple cameras, you could see all directions. You could also project images into a 360-degree field of view in 3D-dimension.

One argues that the camera is superior to radar and lidar because it could capture the color, texture, and contour of objects. The challenge with cameras is that it depends heavily on weather conditions.

“Once you solve cameras for vision, autonomy is solved; if you don’t solve vision, it’s not solved… You can absolutely be superhuman with just cameras.” — Elon Musk, TED2017

Most researches limit AVs to urban areas, where AVs would make a bigger impact. Lane-keeping assist systems is now a well-researched topic in this field. [6]

Image-based perception algorithms can be either heuristic or data-driven. In recent years, deep learning techniques have proven superior to conventional computer vision techniques for object detection and tracking. We can segment and classify images using deep convolutional neural networks (DCNN).

All current methods are relying on DCNN architectures, such as SPP-Net, ResNet-50, VGG-16, DarkNet-53. The open-source state-of-the-art for real-time object detection at the time of writing this is YOLOv4. Recurrent neural networks (RNNs) such as the Long-short-term-memory (LSTM) are also powerful tools when working with images with temporal features (video).

In practice, AV companies combine different machine learning methods into complex hybrid forms when designing their pipelines. The object recognition framework used by AVs can generally be divided into two types:

Single-stage detection framework — a single network to predict object location and classify objects

— a single network to predict object location and classify objects Region proposal detection framework — uses two distinct stages, where the region of interest is predefined before classified by separate networks.

The current research trend for vehicle perception is 360° video/image processing algorithms. This is necessary for higher-level automation [6].

Meanwhile, the task of camera-based localization is a broad topic that also found its applications in robotics, virtual reality, and augmented reality. In AVs, vision-based localization normally uses visual odometry techniques [3].

The general pipeline for visual (image-based) odometry is as the following: 1) obtain disparity map by triangulation of stereo image pair, 2) matching salient features, 3) estimate depth for each point between successive frames, 4) comparing the features to estimate motion and current vehicle position.