# Sensor-Based Environmental Perception Technology for Intelligent Vehicles

Authors: Biyao Wang; Yi Han; Di Tian; Tian Guan
Journal: Journal of Sensors
 (2021)

---

## Abstract

Environmental perception technology is the basis and premise of intelligent vehicle decision control of intelligent vehicles, a crucial link of intelligent vehicles to realize intelligence, and also the basic guarantee of its safety and intelligence. The accuracy and robustness of the perception algorithm will directly affect or even determine the realization of the upper function of intelligent vehicles. The wrong environmental perception will affect the control of the vehicle, thus causing safety risks. This paper discusses the intelligent vehicle perception technology and introduces the development status and control strategies of several important sensors such as machine vision, laser radar, and millimeter-wave radar. Target detection, target recognition, and multisensor fusion are analyzed in the optimized part of sensor results. The functions of the intelligent vehicle assistance system which has been applied to the ground at present are described, and the lane detection, adaptive cruise control (ACC), and autonomous emergency braking (AEB) are analyzed. Finally, the paper looks forward to the research direction of sense-based intelligent vehicle perception technology, which will play an important role in guiding the development of intelligent vehicles and accelerate the landing process of intelligent vehicles.


---

## Body

## 1. Introduction

Smart car is an entity concept that continues to grow and develop rapidly, with more and more content coverage, which is not uncommon in newspapers, the internet, and books. It is generally believed that [1] the intelligent vehicle is a comprehensive system with various functions, including integrating environmental perception, planning and decision-making, and multilevel autonomous driving. It focuses on computers, modern sensing, information integration, communication, artificial intelligence, and automated control technologies. It is a typical high-tech complex. The current research on intelligent vehicles is mainly dedicated to improving the safety and comfort of the vehicle and providing excellent human-vehicle interaction functions. Specifically, the so-called “intelligent car” is to add advanced information perception systems (such as radar, camera, Global Positioning System (GPS), and internet networking equipment), advanced control system, reliable actuators, and other devices based on ordinary cars. The intelligent car is also aimed at realizing intelligent information exchange through the vehicle sensing system and information terminal. The vehicle has intelligent environmental perception, which can automatically analyze the safety and dangerous state, make the vehicle reach the destination, and finally achieve the purpose of replacing human operation. In 2015, in the “Made in China 2025,” issued by the State Council, intelligent connected vehicles were included in the important field of national intelligent manufacturing development in the next decade. Automatic driving is the key technology to realize “intelligent automobile” and “intelligent transportation,” which is also the inevitable trend of automobile development. From the perspective of industrial development, autonomous driving will be the inevitable result of the integrated development of the Internet of Things, cloud computing, big data technology, and an important engine for many industrial developments.The intelligent car is the direction of the future development of cars, but it also has a broad space for market interests. Unmanned driving was originated in the DARPA (Defense Advanced Research Projects Agency) Grand Challenge, and Sebastian Thrun initiated the practical R&D programs in 2009. In the past decade, the automobile industry, technology, and even IT manufacturers have been involved in the field of intelligent vehicles, forming a flourishing situation. In 2010, Shelley, a driverless sports car, jointly developed by the Volkswagen Electronics Research Laboratory, Stanford Dynamic Design Laboratory, and Oracle, climbed Pike Peak in western Colorado without any driver intervention. In 2014, with the rise of artificial intelligence through deep learning, Uber, OEMs, Baidu, and other manufacturers have entered the game. At the 2015 Frankfurt Auto Show, Mercedes-Benz integrated many of the latest technological achievements in its concept car, IAA, which could realize automatic driving and information interaction between cars. In addition, car manufacturers, such as Audi, Cadillac, Nissan, and Toyota, are planning to launch vehicles with automatic steering, acceleration and deceleration, lane guidance, automatic parking, and adaptive cruise control, which fully reflect the achievements of technological innovation in various fields. As a result, intelligent car technology innovation has become the main technical competition for future automobile manufacturers.Among them, as the basis and premise of other problems, environmental sensing is a crucial link of intelligent vehicles and the basic guarantee for its safety and intelligence. The accuracy and robustness of the sensing algorithm will directly affect or even determine the implementation of the high-level function, while the wrong environmental perception will influence the control of the vehicle, resulting in safety risks. For example, the accuracy of the road area detection system is responsible for deciding whether the intelligent vehicle can normally drive in the driving road areas, and the performance of the lane line detection technology directly affects the operation of the lane maintenance system. Therefore, the research on environmental perception technology for the visual navigation of intelligent cars has important value for promoting the rapid development of intelligent vehicles and transportation.At present, the existing intelligent vehicle sensing technologies are mainly classified by algorithm, which cannot well summarize the current development status and application of each sensor. This paper takes sensors as the starting point and focuses on the intelligent vehicle perception technology from three aspects: sensors, multisensor fusion/result optimization processing, and application scenarios. In this paper, the application background and field of each sensor are discussed in detail, and the research direction of intelligent vehicle perception technology is proposed more effectively by integrating the existing intelligent driver assistance system, which provides important guidance for the large-scale production of intelligent vehicle.


## 2. Sensors

The ways of intelligent vehicles to realize environmental perception are mainly divided into visual perception, LiDAR perception, millimeter-wave radar perception, and infrared sensing [2]. The comparative analysis of various sensors used for intelligent vehicle environment perception is shown in Table 1, and each type of sensor is described separately in this paper.Table 1
Sensors.
SensorsFigureCost (RMB)AdvantagesDisadvantagesCamera500Low cost, rich information access, and mature hardware technologyVulnerable to weather, lighting conditions, and other factorsLiDAR≥50000High precision, accurate ranging, accurate positioning, and no illumination impactHigh price, complex processing algorithms, vulnerable to smoke, rain, snow, fog, and other interferenceMillimeter-wave radar1000Working all day, long detection distance, easy installationVulnerable to signal interference in some scenariosInfrared sensor1000Low price, accurate biological identification, and a good identification of tangential movementRadial motion has a poor discrimination ability and no angular measurement capability


### 2.1. Machine Vision (Video Camera)

With the speedy development of artificial intelligence technology, internet technology, computer technology, communication technology, and machine vision technology have gradually emerged in all aspects of life. Compared with other intelligent technologies, machine vision technology started late, but its application prospects are broad. Currently, machine vision has become a popular research object in the automobile-assisted driving industry. The “eye” of the automobile-assisted driving system mainly combines various sensors and GPS equipment to realize the distance alarm function.At present, deep learning (DL), the main force of machine learning (ML), has set off a fierce technological wave, bringing a new face of computer vision (CV) technology, thus providing a great opportunity for the real landing of autonomous driving. Various deep learning algorithms for self-driving cars are coming. Numerous learning networks, such as Recurrent Neural Network (RNN) [3], Deep Boltzmann Machine (DBN) [4], Generative Adversarial Networks (GAN), Long Short-Term Memory (LSTM), Region-Based Convolutional Neural Networks (RCNN), Single Shot MultiBox Detector (SSD), and You only look once (YOLO) network have broken through the limitations of traditional image processing algorithm and enabled the rapid development of autonomous driving in the industry.At present, the target detection algorithm adapts deep learning and follows the prediction process of its overall design. It can be roughly divided into two categories: (1) the two-stage object detection algorithm represented by RCNN and its variants [5–7]. The first stage is to obtain candidate boxes through various methods, and the feature extraction must be performed on each candidate box. The second stage is to classify the areas represented by the candidate box. Therefore, such algorithms have better accuracy. (2) The one-stage object detection algorithm using YOLO and its variant [8–10]. These types of algorithms abandon the regional classification in the two-stage algorithms. The category prediction directly predicts each positive sample on the selected feature diagram. As a result, these algorithms can detect faster in real-time applications and are usually the optimal choice.Object detection (OD) also absorbs nutrients from deep learning. The main purpose of object detection is to automatically predict the category and location of interested objects in the input image through the algorithms. This characteristic is urgently needed for autonomous driving. By providing enough images of the autonomous driving scene to train the model, the algorithm can extract features and identify targets in the autonomous driving scene, such as pedestrians, vehicles, and traffic lights. When the algorithm obtains the scene target information, it will share it with other sensors, allowing the automatic driving system to understand the current road environment of the vehicle, and finally reply to multiple interactive systems in the car for early warning. If needed, it can even directly control the steering or braking of the vehicle to deal with complex situations and emergencies on the road to ensure safety.There has been a lot of research on obstacle detection using only camera sensors [11]. A more common light-flow method makes use of dynamic information for the detection of the obstacle. Zhang et al. [12] only used a single camera but achieved an accuracy of over 90% through hierarchical detection, while Hane et al. [13] obtained a powerful recognition effect by analyzing wheel speed and detecting multiple-frame images. Other examples are as follows: the integral channel algorithm proposed by Piotr Dollar et al., the lane line detection algorithm presented by Zhao et al. [14], and the multiobjective trajectory tracking algorithm [15] developed by Piao et al.However, the autonomous driving scenes need to be used to detect targets, including pedestrians, vehicles, various scales, large changes in road background, light strength, and fuzzy occlusion interference [16]. The target detection algorithm for automatic driving scenes is still very challenging. There is still room for improvement in terms of accuracy and real-time performance. In summary, object detection and recognition for autonomous driving are of great significance.


### 2.2. LiDAR

LiDAR sensor is widely used in unmanned vehicles with high detection accuracy and resolution and provides rich 3D data description as well as information on the reflection intensity of the detected obstacle surface at the same time [17]. The current LiDAR is mainly divided into mechanical rotating LiDAR and solid-state LiDAR. Unlike monocular image data, the point cloud data of LiDAR has the characteristics of disorder, sparseness, and a limited amount of information. In order to use point cloud information more efficiently, 2D and 3D processing schemes are generally adopted. The LiDAR point cloud 2D processing scheme uses an image-oriented approach to project the point cloud data onto a two-dimensional plane according to some specific perspectives (the front or aerial perspective) instead of directly processing the three-dimensional point cloud data. The point cloud 3D processing scheme fully considers the invariance of the input point arrangement. It uses a neural network that directly calculates the point cloud, providing a unified architecture for applying target classification and component segmentation in scene semantic analysis.At present, the application of LiDAR in autonomous vehicles on the ground has two main directions: obstacle detection and target tracking and simultaneous localization and mapping (SLAM). The first direction is the main application used in obstacle detection and target tracking of unmanned car driving. It can provide a better understanding of the environment for driverless cars. The primary role of SLAM is to provide accurate position information for unmanned vehicles to compensate for the specific use defects of GPS, Inertial Measurement Unit (IMU), and other equipment.Due to the irregularities of the point cloud, many 3D detection algorithms convert the point cloud into 2D images to extract features in a 2D convolution manner and detect the method [18–20]. Zeng et al. [21] proposed a Real-Time 3D (RT3D) algorithm for converting 3D point clouds into 2D grids. The algorithm predicts the target vehicle location, direction, and size of Region Proposal Network (RPN) and classification networks. There are also studies regarding converting the point cloud into a voxel grid and then transferring the gridded point cloud to deep convolutional neural networks. Engelcke et al. [22] proposed the Vote3Deep, which discretizes the point clouds into sparse 3D grids and achieves object detection through a feature-centric voting scheme. In 2017, Zhao et al. [23] proposed a 3D point cloud segmentation algorithm called point net, which can directly process the 3D point cloud without converting the point cloud to other formats. The single LiDAR sensor can no longer meet the needs of normal driving of intelligent vehicles. Kumar et al. [24] proposed an exact fusion method to estimate the distance between autonomous vehicles and other obstacles. It is mainly based on geometric transformation and projection to perform low-level sensor fusion of the markers between the camera and LiDAR that detects the vehicle to the obstacle. However, the data collected by sensors have deviations and redundancy, making it difficult for some methods to predict future information accurately. Wen et al. [25] propose a semisupervised prediction model, which uses the improved unsupervised clustering algorithm to establish the fuzzy partition function, and then utilize the neural network model to construct the information prediction function to solve the time analysis of massive industry data effectively.


### 2.3. Millimeter-Wave Radar

The millimeter-wave radar can perceive the position and movement status of vehicles and pedestrians in real-time and is one of the main sensors in the field of intelligent vehicle perception. Due to its characteristic of weather resistance, the millimeter-wave radar can penetrate cloud and fog and has strong environmental applicability. It can also detect information about the target speed through the Doppler effect, making it very suitable for target detection and recognition. Automatic emergency braking (AEB) is one of the common functions of intelligent driving. It uses the millimeter-wave radar to measure the relative distance, speed, and azimuth angle of obstacles ahead, analyzes the potential danger of collision avoidance, and conducts early warning, mild braking intervention, or automatic emergency braking intervention. It has the purpose of providing a guarantee for safe driving. Adaptive cruise control (ACC) is developed based on the existing cruise control system. It scans the target or obstacles through the onboard radar to realize the desired speed or cruise at the preset speed.In the 1960s [26], Germany, Sweden, Japan, and other developed countries first carried out research work on millimeter-wave radar. In the research of 77 GHz millimeter-wave radar [27], Sweden and Germany developed the adaptive intelligent driving control system and the autonomous cruise control system in 1986 and 2003, respectively, and Germany’s Bosch initially realized the commercial use of 77 GHz radar. From 2004 to 2013, Japan first developed a 24 GHz millimeter-wave radar and installed it on a new crown car for trial use and then displayed a millimeter-wave radar in the 79 GHz band. The United States developed a 24 GHz collision avoidance system for forward detection and warning, which has been installed on the bus and can achieve multitarget detection within 106 m, reducing the bus accident rate by nearly 30%. Millimeter-wave radar develops rapidly around the world, and its technology has three main trends: high frequency [28], intelligent, and collaborative, so as to guarantee the safety of vehicles.Vehicle-mounted millimeter-wave radar has made some progress in the field of target detection. Kobayashi et al. processed the acquired signals with range-Doppler algorithm to obtain the two-dimensional image [29]. Bilik et al. of the General Motors Advanced Technology Center in Israel in 2016 developed a high azimuthal and high-resolution multiple input multiple output (MIMO) radar prototype with 16 TX and 16 RX antenna elements to address the challenge [30] for autonomous vehicles in complex urban environments. The team introduced the design of hardware and software modules for a multimode cascaded radar data processing system in 2018, with the proposed architecture enabling high-resolution imaging capability [31]. In 2019, Abdullah et al. proposed an improved SO-CFAR detector [32].


### 2.4. Infrared Sensor (Night Vision System)

One effective strategy to reduce the number of deaths and injuries from such road traffic accidents is to use the car night vision-assisted driving system to give early warning or help drivers make decisions in dangerous driving situations.Many automakers have installed visible light cameras in their generated vehicles to detect obstacles, such as the parking assist systems in some Audi, Volkswagen, and Toyota models. Due to the visible camera at night, under low visibility conditions, such as fog effect is not ideal for target detection, in order to overcome the deficiency of visible light, infrared night vision assistant system research more and more, the night vision system based on infrared sensor has many advantages [33–35], embodied in the following aspects: (1) Infrared night vision system receives infrared radiation imaging of external targets and does not depend on the lighting conditions of the scene. Any object whose temperature is higher than absolute zero will radiate infrared. (2) Compared with visible light camera, the resolution, imaging effect, and cost performance of infrared thermal imagers are constantly improving, and more and more surveillance scenes begin to use infrared cameras. (3) Infrared night vision goggles with all-weather work ability can significantly reduce the risk of driving at night, to help it in the whole night, rain, snow, fog haze weather, and opposite lights glare eye under the condition of low visibility and can output the clear thermal image of the conditions ahead, effectively improve the driver visual range, avoid the collision of vehicles, pedestrians, and obstacles, and effectively improve driving safety. Based on the above reasons, vehicular infrared night vision technology has been highly valued by domestic and foreign major automobile manufacturers and research institutions. With the maturity of the technology, the application of the vehicular infrared night vision system has been gradually promoted.The cameras used for infrared pedestrian detection are infrared cameras, which are divided into two types. One is an active system, using the near-infrared line, which is also known as the near-infrared system. The active system transmits infrared light to the front of the vehicle through an infrared light source and then uses a CCD camera to capture the diffuse reflection of infrared light to form an image. In addition, the use of a filter to filter out infrared light allows the system to work in good light conditions like a normal visible camera during the day. The images acquired by active systems have high quality and can usually clearly show objects up to 150 meters. However, due to the dependence on infrared light sources, the near-infrared line will greatly attenuate in rain, snow, fog, and other weather, resulting in normal operation. Another type of system is a passive system, also known as the far infrared system, which does not require an infrared light source, but instead uses the thermal radiation of the object to form an image. Because the thermal radiation is not affected by light and can penetrate rain and snow and haze, it can work even in inclement weather conditions. This advantage makes the passive infrared system better for vehicle night vision applications.According to the imaging principle, the night vision system is mainly divided into active night vision system and passive night vision system, in which low-light and infrared imaging are the most widely used night vision technology [36]. Dim light refers to faint light at night or at low luminosity conditions, with a wavelength of about 0.4~2 m. Low-light night vision technology, also known as image enhancement technology, converts weak or relatively low energy light into enhanced optical image through an image intensifier to achieve direct observation. Infrared night vision technology is divided into active infrared night vision technology and passive infrared night vision technology. Active infrared night vision technology is an infrared technology for observation through infrared light active irradiation and infrared light reflected back by the target, corresponding equipped with an active night vision system. Passive infrared night vision technology is an infrared technology that realizes observation by means of infrared radiation emitted by the target itself. It changes the temperature distribution on the target surface which cannot be directly seen by the human eye into a thermal image representing the temperature distribution which can be seen by the human eye. It is equipped with a thermal imager. According to the characteristics of each imaging technology, its corresponding advantages and disadvantages are summarized as shown in Table 2.Table 2
Comparison of different imaging technologies [37].
Night vision technologyAdvantagesDisadvantageLow-light level night visionLight weight, small size, good image quality, high cost performance, fast response, and wide applicationBlinking due to strong illumination and high gain, small contrast difference, limited gray levelActive night visionHigh contrast between target and background, clear image, and low priceThe working distance is limited by the power of the infrared light source. The infrared light is easy to be exposed and damagedThermal imagingNo need auxiliary light source. Seeing through smoke and smog and has a far night vision distanceNot sensitive to the surrounding environment, the cost is relatively high


## 2.1. Machine Vision (Video Camera)

With the speedy development of artificial intelligence technology, internet technology, computer technology, communication technology, and machine vision technology have gradually emerged in all aspects of life. Compared with other intelligent technologies, machine vision technology started late, but its application prospects are broad. Currently, machine vision has become a popular research object in the automobile-assisted driving industry. The “eye” of the automobile-assisted driving system mainly combines various sensors and GPS equipment to realize the distance alarm function.At present, deep learning (DL), the main force of machine learning (ML), has set off a fierce technological wave, bringing a new face of computer vision (CV) technology, thus providing a great opportunity for the real landing of autonomous driving. Various deep learning algorithms for self-driving cars are coming. Numerous learning networks, such as Recurrent Neural Network (RNN) [3], Deep Boltzmann Machine (DBN) [4], Generative Adversarial Networks (GAN), Long Short-Term Memory (LSTM), Region-Based Convolutional Neural Networks (RCNN), Single Shot MultiBox Detector (SSD), and You only look once (YOLO) network have broken through the limitations of traditional image processing algorithm and enabled the rapid development of autonomous driving in the industry.At present, the target detection algorithm adapts deep learning and follows the prediction process of its overall design. It can be roughly divided into two categories: (1) the two-stage object detection algorithm represented by RCNN and its variants [5–7]. The first stage is to obtain candidate boxes through various methods, and the feature extraction must be performed on each candidate box. The second stage is to classify the areas represented by the candidate box. Therefore, such algorithms have better accuracy. (2) The one-stage object detection algorithm using YOLO and its variant [8–10]. These types of algorithms abandon the regional classification in the two-stage algorithms. The category prediction directly predicts each positive sample on the selected feature diagram. As a result, these algorithms can detect faster in real-time applications and are usually the optimal choice.Object detection (OD) also absorbs nutrients from deep learning. The main purpose of object detection is to automatically predict the category and location of interested objects in the input image through the algorithms. This characteristic is urgently needed for autonomous driving. By providing enough images of the autonomous driving scene to train the model, the algorithm can extract features and identify targets in the autonomous driving scene, such as pedestrians, vehicles, and traffic lights. When the algorithm obtains the scene target information, it will share it with other sensors, allowing the automatic driving system to understand the current road environment of the vehicle, and finally reply to multiple interactive systems in the car for early warning. If needed, it can even directly control the steering or braking of the vehicle to deal with complex situations and emergencies on the road to ensure safety.There has been a lot of research on obstacle detection using only camera sensors [11]. A more common light-flow method makes use of dynamic information for the detection of the obstacle. Zhang et al. [12] only used a single camera but achieved an accuracy of over 90% through hierarchical detection, while Hane et al. [13] obtained a powerful recognition effect by analyzing wheel speed and detecting multiple-frame images. Other examples are as follows: the integral channel algorithm proposed by Piotr Dollar et al., the lane line detection algorithm presented by Zhao et al. [14], and the multiobjective trajectory tracking algorithm [15] developed by Piao et al.However, the autonomous driving scenes need to be used to detect targets, including pedestrians, vehicles, various scales, large changes in road background, light strength, and fuzzy occlusion interference [16]. The target detection algorithm for automatic driving scenes is still very challenging. There is still room for improvement in terms of accuracy and real-time performance. In summary, object detection and recognition for autonomous driving are of great significance.


## 2.2. LiDAR

LiDAR sensor is widely used in unmanned vehicles with high detection accuracy and resolution and provides rich 3D data description as well as information on the reflection intensity of the detected obstacle surface at the same time [17]. The current LiDAR is mainly divided into mechanical rotating LiDAR and solid-state LiDAR. Unlike monocular image data, the point cloud data of LiDAR has the characteristics of disorder, sparseness, and a limited amount of information. In order to use point cloud information more efficiently, 2D and 3D processing schemes are generally adopted. The LiDAR point cloud 2D processing scheme uses an image-oriented approach to project the point cloud data onto a two-dimensional plane according to some specific perspectives (the front or aerial perspective) instead of directly processing the three-dimensional point cloud data. The point cloud 3D processing scheme fully considers the invariance of the input point arrangement. It uses a neural network that directly calculates the point cloud, providing a unified architecture for applying target classification and component segmentation in scene semantic analysis.At present, the application of LiDAR in autonomous vehicles on the ground has two main directions: obstacle detection and target tracking and simultaneous localization and mapping (SLAM). The first direction is the main application used in obstacle detection and target tracking of unmanned car driving. It can provide a better understanding of the environment for driverless cars. The primary role of SLAM is to provide accurate position information for unmanned vehicles to compensate for the specific use defects of GPS, Inertial Measurement Unit (IMU), and other equipment.Due to the irregularities of the point cloud, many 3D detection algorithms convert the point cloud into 2D images to extract features in a 2D convolution manner and detect the method [18–20]. Zeng et al. [21] proposed a Real-Time 3D (RT3D) algorithm for converting 3D point clouds into 2D grids. The algorithm predicts the target vehicle location, direction, and size of Region Proposal Network (RPN) and classification networks. There are also studies regarding converting the point cloud into a voxel grid and then transferring the gridded point cloud to deep convolutional neural networks. Engelcke et al. [22] proposed the Vote3Deep, which discretizes the point clouds into sparse 3D grids and achieves object detection through a feature-centric voting scheme. In 2017, Zhao et al. [23] proposed a 3D point cloud segmentation algorithm called point net, which can directly process the 3D point cloud without converting the point cloud to other formats. The single LiDAR sensor can no longer meet the needs of normal driving of intelligent vehicles. Kumar et al. [24] proposed an exact fusion method to estimate the distance between autonomous vehicles and other obstacles. It is mainly based on geometric transformation and projection to perform low-level sensor fusion of the markers between the camera and LiDAR that detects the vehicle to the obstacle. However, the data collected by sensors have deviations and redundancy, making it difficult for some methods to predict future information accurately. Wen et al. [25] propose a semisupervised prediction model, which uses the improved unsupervised clustering algorithm to establish the fuzzy partition function, and then utilize the neural network model to construct the information prediction function to solve the time analysis of massive industry data effectively.


## 2.3. Millimeter-Wave Radar

The millimeter-wave radar can perceive the position and movement status of vehicles and pedestrians in real-time and is one of the main sensors in the field of intelligent vehicle perception. Due to its characteristic of weather resistance, the millimeter-wave radar can penetrate cloud and fog and has strong environmental applicability. It can also detect information about the target speed through the Doppler effect, making it very suitable for target detection and recognition. Automatic emergency braking (AEB) is one of the common functions of intelligent driving. It uses the millimeter-wave radar to measure the relative distance, speed, and azimuth angle of obstacles ahead, analyzes the potential danger of collision avoidance, and conducts early warning, mild braking intervention, or automatic emergency braking intervention. It has the purpose of providing a guarantee for safe driving. Adaptive cruise control (ACC) is developed based on the existing cruise control system. It scans the target or obstacles through the onboard radar to realize the desired speed or cruise at the preset speed.In the 1960s [26], Germany, Sweden, Japan, and other developed countries first carried out research work on millimeter-wave radar. In the research of 77 GHz millimeter-wave radar [27], Sweden and Germany developed the adaptive intelligent driving control system and the autonomous cruise control system in 1986 and 2003, respectively, and Germany’s Bosch initially realized the commercial use of 77 GHz radar. From 2004 to 2013, Japan first developed a 24 GHz millimeter-wave radar and installed it on a new crown car for trial use and then displayed a millimeter-wave radar in the 79 GHz band. The United States developed a 24 GHz collision avoidance system for forward detection and warning, which has been installed on the bus and can achieve multitarget detection within 106 m, reducing the bus accident rate by nearly 30%. Millimeter-wave radar develops rapidly around the world, and its technology has three main trends: high frequency [28], intelligent, and collaborative, so as to guarantee the safety of vehicles.Vehicle-mounted millimeter-wave radar has made some progress in the field of target detection. Kobayashi et al. processed the acquired signals with range-Doppler algorithm to obtain the two-dimensional image [29]. Bilik et al. of the General Motors Advanced Technology Center in Israel in 2016 developed a high azimuthal and high-resolution multiple input multiple output (MIMO) radar prototype with 16 TX and 16 RX antenna elements to address the challenge [30] for autonomous vehicles in complex urban environments. The team introduced the design of hardware and software modules for a multimode cascaded radar data processing system in 2018, with the proposed architecture enabling high-resolution imaging capability [31]. In 2019, Abdullah et al. proposed an improved SO-CFAR detector [32].


## 2.4. Infrared Sensor (Night Vision System)

One effective strategy to reduce the number of deaths and injuries from such road traffic accidents is to use the car night vision-assisted driving system to give early warning or help drivers make decisions in dangerous driving situations.Many automakers have installed visible light cameras in their generated vehicles to detect obstacles, such as the parking assist systems in some Audi, Volkswagen, and Toyota models. Due to the visible camera at night, under low visibility conditions, such as fog effect is not ideal for target detection, in order to overcome the deficiency of visible light, infrared night vision assistant system research more and more, the night vision system based on infrared sensor has many advantages [33–35], embodied in the following aspects: (1) Infrared night vision system receives infrared radiation imaging of external targets and does not depend on the lighting conditions of the scene. Any object whose temperature is higher than absolute zero will radiate infrared. (2) Compared with visible light camera, the resolution, imaging effect, and cost performance of infrared thermal imagers are constantly improving, and more and more surveillance scenes begin to use infrared cameras. (3) Infrared night vision goggles with all-weather work ability can significantly reduce the risk of driving at night, to help it in the whole night, rain, snow, fog haze weather, and opposite lights glare eye under the condition of low visibility and can output the clear thermal image of the conditions ahead, effectively improve the driver visual range, avoid the collision of vehicles, pedestrians, and obstacles, and effectively improve driving safety. Based on the above reasons, vehicular infrared night vision technology has been highly valued by domestic and foreign major automobile manufacturers and research institutions. With the maturity of the technology, the application of the vehicular infrared night vision system has been gradually promoted.The cameras used for infrared pedestrian detection are infrared cameras, which are divided into two types. One is an active system, using the near-infrared line, which is also known as the near-infrared system. The active system transmits infrared light to the front of the vehicle through an infrared light source and then uses a CCD camera to capture the diffuse reflection of infrared light to form an image. In addition, the use of a filter to filter out infrared light allows the system to work in good light conditions like a normal visible camera during the day. The images acquired by active systems have high quality and can usually clearly show objects up to 150 meters. However, due to the dependence on infrared light sources, the near-infrared line will greatly attenuate in rain, snow, fog, and other weather, resulting in normal operation. Another type of system is a passive system, also known as the far infrared system, which does not require an infrared light source, but instead uses the thermal radiation of the object to form an image. Because the thermal radiation is not affected by light and can penetrate rain and snow and haze, it can work even in inclement weather conditions. This advantage makes the passive infrared system better for vehicle night vision applications.According to the imaging principle, the night vision system is mainly divided into active night vision system and passive night vision system, in which low-light and infrared imaging are the most widely used night vision technology [36]. Dim light refers to faint light at night or at low luminosity conditions, with a wavelength of about 0.4~2 m. Low-light night vision technology, also known as image enhancement technology, converts weak or relatively low energy light into enhanced optical image through an image intensifier to achieve direct observation. Infrared night vision technology is divided into active infrared night vision technology and passive infrared night vision technology. Active infrared night vision technology is an infrared technology for observation through infrared light active irradiation and infrared light reflected back by the target, corresponding equipped with an active night vision system. Passive infrared night vision technology is an infrared technology that realizes observation by means of infrared radiation emitted by the target itself. It changes the temperature distribution on the target surface which cannot be directly seen by the human eye into a thermal image representing the temperature distribution which can be seen by the human eye. It is equipped with a thermal imager. According to the characteristics of each imaging technology, its corresponding advantages and disadvantages are summarized as shown in Table 2.Table 2
Comparison of different imaging technologies [37].
Night vision technologyAdvantagesDisadvantageLow-light level night visionLight weight, small size, good image quality, high cost performance, fast response, and wide applicationBlinking due to strong illumination and high gain, small contrast difference, limited gray levelActive night visionHigh contrast between target and background, clear image, and low priceThe working distance is limited by the power of the infrared light source. The infrared light is easy to be exposed and damagedThermal imagingNo need auxiliary light source. Seeing through smoke and smog and has a far night vision distanceNot sensitive to the surrounding environment, the cost is relatively high


## 3. Multisensor Fusion/Result Optimization Processing

### 3.1. Target Detection and Identification

There is no doubt that for intelligent vehicles, the perception of moving objects on the road is the premise of achieving safe autonomous driving, which mainly includes two tasks: the detection and recognition of moving objects.Dynamic object detection technology is very important and basic in the field of machine vision, which forms an important branch of machine vision [38, 39]. After decades of development, many dynamic object detection methods using 2D images have been developed, based on which the subsequent researches mainly focused on the improvement of these methods to suit for different and more complex application scenarios by adding new mathematical methods. The key to the success of dynamic object detection lies in the extraction of the characteristics of the region of the dynamic target as much as the dynamic target detection. The most common dynamic object detection methods based on 2D images include background subtraction, optical flow calculation [40], continuous frame difference [41], and feature matching. Among them, the background subtraction method mainly includes a background model construction method based on the image pixel value [42] and a background model construction method based on the image texture information. The continuous interframe difference method is available for the analysis of the motion characteristics of objects in the image sequence by using the brightness difference of each pixel or the depth difference of each pixel in the depth image. The detection method based on the light flow field is used to determine the pixel velocity vector change according to the pixel value changes of each pixel at different times in the video sequence and its correlation with the adjacent pixels in position. At early time, Reid proposed the multiple hypothesis tracking (MHT) algorithm [43] of which, however, the complexity is increased by the uncertainty of target number and target correlation measurements. To solve this problem, literature [44] proposed the Bayesian MHT approach which decomposes the tracking problem into events and target states to facilitate the processing. The literature [45] proposed a spline resampling particle filter method available to improve target tracking accuracy and obtain high tracking accuracy with fewer samples. Montemerlo et al. [46] used the nearest neighbor approach to address the data association problem of object location tracking, and the literature [47] proposed a Bayesian-based object tracking framework based on threshold segmentation, mathematical morphology, and perspective projection techniques.In the field of objective recognition, to obtain efficient and reliable classification results, modern mode recognition theory is generally used for classification design. There are several conventional classification methods, such as fuzzy method [48], statistical mode [49], support vector machine, syntactic, model, and neural network. The fuzzy classification method is originated from fuzzy mathematics, so the key point lies in the design of the membership function of the object class. Document [50] achieves fuzzy clustering of the horizon based on the Gustafson-Kessel (GK) algorithm, thereby identifying probabilistic optimal control horizon partition strategies and refining the prototype template. Statistical mode-based classification methods are able to obtain the eigenvector distribution of various categories by using probabilistic models and classify unknown samples. Document [51] presents a multiobjective recognition method for the recognition of pedestrian and vehicle based on single-line LiDAR and monocular visual integration. The method is used to detect location of obstacles and extract feature vectors from region of interest (ROI) by means of a histogram of directed gradient features; it also uses a preestablished support vector machine classifier model to identify and determine the type of obstacles. Document [52] is based on various operations in image improvement, gray scale transformation, bilateral filtering edge detection, and obtaining license plates from vehicle images. The literature [53] employs MobileNet-SSD based on the deep neural network algorithm to process the frames captured by the onboard sensor for objective recognition. The classification recognition algorithm based on the neural network is a classification method proposed for the simulation of the physiological tissue structure of the human brain. It has the advantage of adjusting the output to approximate any goal in the characteristic space. However, the mathematical interpretation must be more complex, and the design of the neural network is often needed and is improved through a large number of experiments.


### 3.2. Target Tracking

The tracking and prediction of dynamic objects are achieved by analyzing the continuous observation signal of the object, estimating the state of the object at the current moment, and predicting the future moment state, including the target position, motion speed, and acceleration. They are usually [54] based on information-rich sensors, such as vision or LiDAR.Vision-based motion target tracking computes the position of the target on each frame image and estimates the speed of the target and the position in the next frame image. The conventional tracking methods include the relevant tracking method [55] and the optical flow tracking method [56]. The correlation tracking algorithm does not require high image quality, which can work stably under the low signal-to-noise ratio conditions, and adapt to the more complex scene structure. Facing the large shortage of related tracking algorithms, many improved algorithms have been developed [57], such as Kalman filter-based method [58], three-dimensional point mean drift method [59], and particle filter tracking [60]. Optical flow tracking method utilizes the movement information of the target to avoid the influence of gray scale change on the target tracking, thus obtaining a good antinoise capability. In addition, some artificial intelligence algorithms, such as neural network [61], fuzzy logic [62], invariant moment matching, and gray scale feature matching method [63] are also applied into the tracking algorithms to realize the identification of tracking target features.LiDAR-based motion target tracking utilizes the rich and accurate ranging information of LiDAR for the estimation of the current and future states of the target. The conventional methods include Kalman filter series methods and particle filter methods. The former shows good operational performance. In the case that the motion noise and measurement noise of the target belong to Gaussian white noise, and the target motion model and measurement model are linear models, the Kalman filter technology is available to obtain the best estimation [64]. The multitarget model [65] is established if the number of targets is large, or a more accurate description of the various motion states is required. In addition, when the tracked target is not able to establish a linear system model, the dynamic tracking [66] can be implemented by using the extended Kalman filtering method.Many tracking activities of the motion target are based on the static platform. When the platform is in the motion state, the tracking of the target has to consider both the movement of the target and that of the platform itself, that is, the problem of motion compensation of the platform is also an important research direction for the dynamic target tracking based on the motion platform [67].In terms of the prediction of dynamic objects, the relatively simple prediction method is to take the ratio of the position change to time of the dynamic object in several recent observation cycles as the velocity [68] of the motion. This method contains the measurement error of the position, which makes it more applicable in the case of short observation period and low dynamics of the object. In the dynamic environment, Kalman filter is achievable by means of Kalman filter and particle filtering. The literature [69] adds enhanced motion prediction functions to behavior-based algorithms to improve the accuracy of dynamic prediction. And literature [70] combines the position-assisted task assignment framework algorithm and the grid-based multiobjective optimization mathematical model to propose a target allocation and path planning method for underwater multiple robots. Document [71] proposes a new video saliency model that enhances the CNN-LSTM architecture through a supervised attention mechanism to achieve fast end-to-end saliency learning. Document [72] proposes a method that combines the cellular neural networks with artificial potential fields for grid-based path planning. Document [73] implements an object-tracking algorithm based on basic particle filter, and an example of tracked pedestrians in contrasting camera and 3D LiDAR perspective is shown in Figure 1.Figure 1
A scene with several tracked pedestrians and cyclist with a basic particle filter on an urban road intersection. Past trajectories are shown in white with current heading and speed shown by the direction and magnitude of the arrow [73].


### 3.3. Multisource Sensor Information Fusion

Multisource information fusion is to use mathematical method and computer technology to treat multisource sensor perception information obtained at different time and space and then process and make comprehensive judgment following certain rules before generating the consistency description of the perceived objects, thereby providing a basis for subsequent analysis and decision-making [74]. For intelligent vehicles, the integration of multisource sensing information is mainly related to the use of driving environment sensing sensors (such as cameras, LiDAR, and GPS) information, in this manner to obtain a complete and correct driving environment, namely, the road conditions.As an important basic theoretical branch in the research of intelligent vehicles, the application achievement of multisource information fusion research is very fruitful. Martin Marietta has designed and manufactured the DARPA independent land car [75, 76] which uses sonar to determine the height and tilt of the car body itself and extract the geometric characteristics of road obstacles; it also uses color cameras to obtain road edge information and unify the measurement information under different coordinate systems to the public coordinate system, thereby forming an integrated road tracking track. The smart car at Braunschweig University of Science and Technology in Germany is able to detect the obstacles [77] by using the information fusion of various sensors, such as stereo cameras and laser scanners. The American Demom Smart Car is equipped with a millimeter-wave ranging sensor and machine vision system consisting [78] of color camera, binocular stereo camera, and infrared camera, which uses binocular stereo vision to detect obstacles; besides, color and infrared cameras are utilized for scene geometric feature classification. The robot of ASIMO, developed by Toyota, Japan, is a typical example of intelligent bionic [79] when the current multisensor information fusion technology is applied to mobile robots. In China, the National Key Laboratory of Intelligent Technology and System of Tsinghua University takes the quasi-structured and unstructured road environment as the research background. The serial unmanned vehicle system THMR is equipped with color cameras, laser distance finder, magnetic compass, optical code disk, and other positioning system, and the system enjoys a high level of action decision-making and planning ability [80]. HQ3 unmanned car of Red flag from Defense University of Science and Technology [81] and series intelligent cars from Jilin University, JLUIV, and DLIUV, as well as the intelligent vehicles from Shanghai Jiaotong University, Chang’an University, Hunan University, and Hefei Institute of Material Science, Chinese Academy of Sciences, also adopted a variety of environmental sensing equipment and multisource information fusion technology, realizing their own positioning, navigation, obstacle avoidance, and following functions [82]. Document [83] proposes Multi-View 3D Object Protection Network (MV3D) for a groundbreaking fusion of LiDAR point cloud data with RGB image information. Document [84] represents the 3D point cloud information with the front view and bird’s aerial view of the laser point cloud and integrates with the RGB image for the prediction of the directional 3D boundary frame. The network, as shown in Figure 2, consists of two subnetworks, i.e., a 3D proposal network and a region-based fusion network.Figure 2
MV3D perceptual fusion framework [83]: the network takes the bird’s eye view and front view of LiDAR point cloud as well as an image as input. It first generates 3D object proposals from bird’s eye view map and projects them to three views. A deep fusion network is used to combine region-wise features obtained via ROI pooling for each view. The fused features are used to jointly predict object class and do oriented 3D box regression. The bird’s eye view representation is encoded by height, intensity and density. This paper discretizes the projected point cloud into a 2D grid with resolution of 0.1 m. For each cell, the height feature is computed as the maximum height of the points in the cell. To encode more detailed height information, the point cloud is divided equally into M slices. A height map is computed for each slice; thus, we obtain M height maps.The research of multisource information fusion is developed based on various disciplines, such as statistics, information theory, operational research, computer, and artificial intelligence. The specific methods mainly include signal processing and estimation theoretical methods, such as applying wavelet transform, Gauss filter (GSF), Kalman filter, particle filtering, Markov chain, and desired Maximization algorithm, thus obtaining the optimal parameter estimate under the premise of the establishment of specific optimization indicators, including the typical minimal risk method, minimal energy method, the statistical inference methods, such as Bayes inference, support vector machine theory, classical reasoning, evidence inference, and random set theory, as well as information theory methods, such as entropy method and minimum description length method. It also includes the decision-making theory methods, AI methods, such as genetic algorithms, fuzzy logic, rule-based reasoning, neural network, expert system, logical template method, and mass factor method.


## 3.1. Target Detection and Identification

There is no doubt that for intelligent vehicles, the perception of moving objects on the road is the premise of achieving safe autonomous driving, which mainly includes two tasks: the detection and recognition of moving objects.Dynamic object detection technology is very important and basic in the field of machine vision, which forms an important branch of machine vision [38, 39]. After decades of development, many dynamic object detection methods using 2D images have been developed, based on which the subsequent researches mainly focused on the improvement of these methods to suit for different and more complex application scenarios by adding new mathematical methods. The key to the success of dynamic object detection lies in the extraction of the characteristics of the region of the dynamic target as much as the dynamic target detection. The most common dynamic object detection methods based on 2D images include background subtraction, optical flow calculation [40], continuous frame difference [41], and feature matching. Among them, the background subtraction method mainly includes a background model construction method based on the image pixel value [42] and a background model construction method based on the image texture information. The continuous interframe difference method is available for the analysis of the motion characteristics of objects in the image sequence by using the brightness difference of each pixel or the depth difference of each pixel in the depth image. The detection method based on the light flow field is used to determine the pixel velocity vector change according to the pixel value changes of each pixel at different times in the video sequence and its correlation with the adjacent pixels in position. At early time, Reid proposed the multiple hypothesis tracking (MHT) algorithm [43] of which, however, the complexity is increased by the uncertainty of target number and target correlation measurements. To solve this problem, literature [44] proposed the Bayesian MHT approach which decomposes the tracking problem into events and target states to facilitate the processing. The literature [45] proposed a spline resampling particle filter method available to improve target tracking accuracy and obtain high tracking accuracy with fewer samples. Montemerlo et al. [46] used the nearest neighbor approach to address the data association problem of object location tracking, and the literature [47] proposed a Bayesian-based object tracking framework based on threshold segmentation, mathematical morphology, and perspective projection techniques.In the field of objective recognition, to obtain efficient and reliable classification results, modern mode recognition theory is generally used for classification design. There are several conventional classification methods, such as fuzzy method [48], statistical mode [49], support vector machine, syntactic, model, and neural network. The fuzzy classification method is originated from fuzzy mathematics, so the key point lies in the design of the membership function of the object class. Document [50] achieves fuzzy clustering of the horizon based on the Gustafson-Kessel (GK) algorithm, thereby identifying probabilistic optimal control horizon partition strategies and refining the prototype template. Statistical mode-based classification methods are able to obtain the eigenvector distribution of various categories by using probabilistic models and classify unknown samples. Document [51] presents a multiobjective recognition method for the recognition of pedestrian and vehicle based on single-line LiDAR and monocular visual integration. The method is used to detect location of obstacles and extract feature vectors from region of interest (ROI) by means of a histogram of directed gradient features; it also uses a preestablished support vector machine classifier model to identify and determine the type of obstacles. Document [52] is based on various operations in image improvement, gray scale transformation, bilateral filtering edge detection, and obtaining license plates from vehicle images. The literature [53] employs MobileNet-SSD based on the deep neural network algorithm to process the frames captured by the onboard sensor for objective recognition. The classification recognition algorithm based on the neural network is a classification method proposed for the simulation of the physiological tissue structure of the human brain. It has the advantage of adjusting the output to approximate any goal in the characteristic space. However, the mathematical interpretation must be more complex, and the design of the neural network is often needed and is improved through a large number of experiments.


## 3.2. Target Tracking

The tracking and prediction of dynamic objects are achieved by analyzing the continuous observation signal of the object, estimating the state of the object at the current moment, and predicting the future moment state, including the target position, motion speed, and acceleration. They are usually [54] based on information-rich sensors, such as vision or LiDAR.Vision-based motion target tracking computes the position of the target on each frame image and estimates the speed of the target and the position in the next frame image. The conventional tracking methods include the relevant tracking method [55] and the optical flow tracking method [56]. The correlation tracking algorithm does not require high image quality, which can work stably under the low signal-to-noise ratio conditions, and adapt to the more complex scene structure. Facing the large shortage of related tracking algorithms, many improved algorithms have been developed [57], such as Kalman filter-based method [58], three-dimensional point mean drift method [59], and particle filter tracking [60]. Optical flow tracking method utilizes the movement information of the target to avoid the influence of gray scale change on the target tracking, thus obtaining a good antinoise capability. In addition, some artificial intelligence algorithms, such as neural network [61], fuzzy logic [62], invariant moment matching, and gray scale feature matching method [63] are also applied into the tracking algorithms to realize the identification of tracking target features.LiDAR-based motion target tracking utilizes the rich and accurate ranging information of LiDAR for the estimation of the current and future states of the target. The conventional methods include Kalman filter series methods and particle filter methods. The former shows good operational performance. In the case that the motion noise and measurement noise of the target belong to Gaussian white noise, and the target motion model and measurement model are linear models, the Kalman filter technology is available to obtain the best estimation [64]. The multitarget model [65] is established if the number of targets is large, or a more accurate description of the various motion states is required. In addition, when the tracked target is not able to establish a linear system model, the dynamic tracking [66] can be implemented by using the extended Kalman filtering method.Many tracking activities of the motion target are based on the static platform. When the platform is in the motion state, the tracking of the target has to consider both the movement of the target and that of the platform itself, that is, the problem of motion compensation of the platform is also an important research direction for the dynamic target tracking based on the motion platform [67].In terms of the prediction of dynamic objects, the relatively simple prediction method is to take the ratio of the position change to time of the dynamic object in several recent observation cycles as the velocity [68] of the motion. This method contains the measurement error of the position, which makes it more applicable in the case of short observation period and low dynamics of the object. In the dynamic environment, Kalman filter is achievable by means of Kalman filter and particle filtering. The literature [69] adds enhanced motion prediction functions to behavior-based algorithms to improve the accuracy of dynamic prediction. And literature [70] combines the position-assisted task assignment framework algorithm and the grid-based multiobjective optimization mathematical model to propose a target allocation and path planning method for underwater multiple robots. Document [71] proposes a new video saliency model that enhances the CNN-LSTM architecture through a supervised attention mechanism to achieve fast end-to-end saliency learning. Document [72] proposes a method that combines the cellular neural networks with artificial potential fields for grid-based path planning. Document [73] implements an object-tracking algorithm based on basic particle filter, and an example of tracked pedestrians in contrasting camera and 3D LiDAR perspective is shown in Figure 1.Figure 1
A scene with several tracked pedestrians and cyclist with a basic particle filter on an urban road intersection. Past trajectories are shown in white with current heading and speed shown by the direction and magnitude of the arrow [73].


## 3.3. Multisource Sensor Information Fusion

Multisource information fusion is to use mathematical method and computer technology to treat multisource sensor perception information obtained at different time and space and then process and make comprehensive judgment following certain rules before generating the consistency description of the perceived objects, thereby providing a basis for subsequent analysis and decision-making [74]. For intelligent vehicles, the integration of multisource sensing information is mainly related to the use of driving environment sensing sensors (such as cameras, LiDAR, and GPS) information, in this manner to obtain a complete and correct driving environment, namely, the road conditions.As an important basic theoretical branch in the research of intelligent vehicles, the application achievement of multisource information fusion research is very fruitful. Martin Marietta has designed and manufactured the DARPA independent land car [75, 76] which uses sonar to determine the height and tilt of the car body itself and extract the geometric characteristics of road obstacles; it also uses color cameras to obtain road edge information and unify the measurement information under different coordinate systems to the public coordinate system, thereby forming an integrated road tracking track. The smart car at Braunschweig University of Science and Technology in Germany is able to detect the obstacles [77] by using the information fusion of various sensors, such as stereo cameras and laser scanners. The American Demom Smart Car is equipped with a millimeter-wave ranging sensor and machine vision system consisting [78] of color camera, binocular stereo camera, and infrared camera, which uses binocular stereo vision to detect obstacles; besides, color and infrared cameras are utilized for scene geometric feature classification. The robot of ASIMO, developed by Toyota, Japan, is a typical example of intelligent bionic [79] when the current multisensor information fusion technology is applied to mobile robots. In China, the National Key Laboratory of Intelligent Technology and System of Tsinghua University takes the quasi-structured and unstructured road environment as the research background. The serial unmanned vehicle system THMR is equipped with color cameras, laser distance finder, magnetic compass, optical code disk, and other positioning system, and the system enjoys a high level of action decision-making and planning ability [80]. HQ3 unmanned car of Red flag from Defense University of Science and Technology [81] and series intelligent cars from Jilin University, JLUIV, and DLIUV, as well as the intelligent vehicles from Shanghai Jiaotong University, Chang’an University, Hunan University, and Hefei Institute of Material Science, Chinese Academy of Sciences, also adopted a variety of environmental sensing equipment and multisource information fusion technology, realizing their own positioning, navigation, obstacle avoidance, and following functions [82]. Document [83] proposes Multi-View 3D Object Protection Network (MV3D) for a groundbreaking fusion of LiDAR point cloud data with RGB image information. Document [84] represents the 3D point cloud information with the front view and bird’s aerial view of the laser point cloud and integrates with the RGB image for the prediction of the directional 3D boundary frame. The network, as shown in Figure 2, consists of two subnetworks, i.e., a 3D proposal network and a region-based fusion network.Figure 2
MV3D perceptual fusion framework [83]: the network takes the bird’s eye view and front view of LiDAR point cloud as well as an image as input. It first generates 3D object proposals from bird’s eye view map and projects them to three views. A deep fusion network is used to combine region-wise features obtained via ROI pooling for each view. The fused features are used to jointly predict object class and do oriented 3D box regression. The bird’s eye view representation is encoded by height, intensity and density. This paper discretizes the projected point cloud into a 2D grid with resolution of 0.1 m. For each cell, the height feature is computed as the maximum height of the points in the cell. To encode more detailed height information, the point cloud is divided equally into M slices. A height map is computed for each slice; thus, we obtain M height maps.The research of multisource information fusion is developed based on various disciplines, such as statistics, information theory, operational research, computer, and artificial intelligence. The specific methods mainly include signal processing and estimation theoretical methods, such as applying wavelet transform, Gauss filter (GSF), Kalman filter, particle filtering, Markov chain, and desired Maximization algorithm, thus obtaining the optimal parameter estimate under the premise of the establishment of specific optimization indicators, including the typical minimal risk method, minimal energy method, the statistical inference methods, such as Bayes inference, support vector machine theory, classical reasoning, evidence inference, and random set theory, as well as information theory methods, such as entropy method and minimum description length method. It also includes the decision-making theory methods, AI methods, such as genetic algorithms, fuzzy logic, rule-based reasoning, neural network, expert system, logical template method, and mass factor method.


## 4. Application Scenario

### 4.1. Lane Line Detection

The realization of lane line detection is to take the lane image information collected by the visual sensor as input, process the input information, and detect the lane line position in real time, and the detection results are used as the decision-making information to guide the automatic driving. Lane line detection is mainly applied to the path planning of autonomous vehicles such as navigation, positioning, and lane departure warning, which has an important impact on the development of autonomous driving.In the 1990s, lane line detection was an essential part of assisted driving systems. The GOLD system [85], proposed by scholars from the University of Palma in Italy realized the detection of lane lines and obstacles. The reverse perspective transformation method was used to convert the lane lines into parallel mode, and then, the template matching technology was used to detect the lane lines and determine the position of the lane lines. You et al. [86] proposed an algorithm for lane line detection at night, which used the image processing method to process the digital collected by CCD camera, used the multidirection search method to eliminate the noise of lane line boundary, and used the adaptive Hough transform to detect lane line information. The algorithm shows good reliability and robustness in nighttime lane line detection. Document [87] proposes a more powerful network called Ripple-GAN, by integrating Ripple Lane Line Detection Network (RiLLD-Net), confrontation training of Wasserstein generative adversarial networks, and multitarget semantic segmentation. Experiments show that, especially for complex or obscured lane lines, Ripple-GAN can produce a superior detection performance to other state-of-the-art methods.According to different methods of lane line extraction, the feature extraction method of lane line detection is divided into two types: traditional methods and deep learning methods. Traditional lane line detection methods rely on a highly specialized combination of handmade features and heuristics to identify lane lines. Feature extraction according to the image gray gradient change, color, texture, visual vanishing point, and other features is analyzed and designed. These algorithms are sensitive to light, weather, and other changes. When the driving environment changes significantly, the performance of lane line detection is not good. With the improvement of computer computing power and the rapid development of GPU, deep learning technology has been widely used in computer vision, image processing, and other fields to extract lane features. The researchers proposed to study lane line detection as a segmentation problem and use image segmentation model to extract lane features. The model extracts feature information from a large number of images with annotated information and inferences the corresponding pixel tags in the original image according to the information. Under this end-to-end training, the model can better extract the semantic information of lane lines and classify each pixel. Compared with traditional manual methods such as edge feature extraction, threshold segmentation, and watershed, the image segmentation method based on deep learning can extract richer lane line information and has been widely applied in lane line detection technology [88, 89].


### 4.2. Adaptive Cruise Control

As the growth of vehicles and the acceleration of urbanization, the urban traffic congestion problem becomes a burning issue in our society [90]. ACC is an important longitudinal tracking technology in traffic flow research of autonomous vehicles. The vehicle can obtain the driving state of the vehicle in front in real time through the onboard detection equipment and the vehicle-vehicle communication technology, which has more timely and accurate traffic condition perception ability than ordinary drivers, more stable and safer decision-making and judgment ability, and more economic and environmental protection power output control.ACC is a longitudinal tracking control technology that obtains real-time workshop distance and speed information from the vehicle in front through onboard measurement equipment and uses acceleration optimization algorithm to control the vehicle and the vehicle in front to maintain a stable workshop distance. It can be regarded as an important part of self-driving vehicle technology. Research on ACC began in the 1960s and has entered the phase of implementation [91] in the United States, Japan, and Europe since the 1990s. ACC control system focuses only on the longitudinal driving control of the vehicle along the lane line direction, and its control system is usually divided into upper and lower control. Among them, the upper control is responsible for the output of the target acceleration at the next moment according to the running state such as workshop distance and speed difference obtained by the vehicle-mounted equipment. The lower control is responsible for adjusting the internal power system of the vehicle to achieve the acceleration optimization goal of the upper control. Therefore, the ACC lower control system mainly studies the specific electromechanical control process of the internal power system of the vehicle.ACC system is the core of the longitudinal assisted driving system, and the control strategy is the key to the whole ACC system, so the selection of control algorithm is particularly important to the design of ACC system. At present, commonly used control theory algorithms include the following: Sliding Mode Control (SMC) theory, Classical Proportion Integration Differentiation (PID), Fuzzy Control Theory, Model Prediction Control Theory, Fuzzy PID Control Theory, Optimal Control, and Artificial Neural Nets. The above control algorithms have been widely used in adaptive cruise system.Corona et al. calculated the driver previewed trajectory based on model predictive control theory and then carried out predictive analysis and processing on the driving state of the vehicle. Finally, they analyzed the control target quantity, analyzed the advantages and disadvantages of model predictive control (MPC) and proposed the corresponding optimization strategy [92]. In literature [93], a multiobjective collaborative ACC system was designed based on model predictive control, which solved the dilemma that the traditional ACC system was only suitable for simple distance keeping in a single working condition, and further optimized the problems of high computational amount and nonfeasible solution of model predictive control. Fritz and Schiehlen, based on the sliding mode control algorithm, solved the influence of external disturbance on the instability of low-speed following and at the same time optimized the tracking of ACC system to improve the robustness of the system [94]. Naranjo et al. also designed the lower controller of the adaptive cruise system based on fuzzy control to realize the control of engine throttle opening and brake pedal opening [95, 96].


### 4.3. AEB

AEB is an active safety system that can automatically start the driving brake when the vehicle independently detects the risk of collision ahead, so as to reduce the speed of the vehicle and avoid the collision as much as possible [97]. At present, IT has received more and more attention from the national government, oEMS, component manufacturers, and scientific research institutes. Pedestrians are different from target objects with fixed features, such as vehicles and traffic signs, so it is relatively difficult to detect and track pedestrians. Pedestrians have a variety of clothing and poses that are often difficult to detect due to occlusion and background interference, so the algorithm must be able to guarantee a high enough accuracy. Pedestrians often appear in complex outdoor environments with different backgrounds such as weather, road conditions, and streets, so the robustness and adaptability of the algorithm are highly required. Traffic accidents in real life usually occur in a short time, so the algorithm has a high requirement on real-time performance.Volvo’s City Safety is a low-speed AEB control system for urban roads that uses laser sensors to monitor the traffic ahead. Audi’s Pre Sense Front system uses millimeter-wave radar and camera data fusion to detect road conditions up to 80 meters in front of the vehicle.In 2019, the American Automobile Association (AAA) published a simulation test of the AEB system, which showed that AEB had a 40% success rate at 32 km/h, and AEB essentially failed when the speed reached 48 km/h. In addition, the AEB system also has quite high false alarm rate and false braking rate [98], which seriously affects the driver’s driving comfort and driving safety. Effective and accurate identification of forward target is the prerequisite for the normal operation of AEB control strategy. Wang et al. [99] found that the sensor is limited by the accuracy of millimeter-wave radar, the vision field of camera, and the limitations of multisensor data fusion algorithm. The accuracy and speed of the environmental perception and the handling of the emergency situation are still the key points and difficulties in the development of AEB system [100].


## 4.1. Lane Line Detection

The realization of lane line detection is to take the lane image information collected by the visual sensor as input, process the input information, and detect the lane line position in real time, and the detection results are used as the decision-making information to guide the automatic driving. Lane line detection is mainly applied to the path planning of autonomous vehicles such as navigation, positioning, and lane departure warning, which has an important impact on the development of autonomous driving.In the 1990s, lane line detection was an essential part of assisted driving systems. The GOLD system [85], proposed by scholars from the University of Palma in Italy realized the detection of lane lines and obstacles. The reverse perspective transformation method was used to convert the lane lines into parallel mode, and then, the template matching technology was used to detect the lane lines and determine the position of the lane lines. You et al. [86] proposed an algorithm for lane line detection at night, which used the image processing method to process the digital collected by CCD camera, used the multidirection search method to eliminate the noise of lane line boundary, and used the adaptive Hough transform to detect lane line information. The algorithm shows good reliability and robustness in nighttime lane line detection. Document [87] proposes a more powerful network called Ripple-GAN, by integrating Ripple Lane Line Detection Network (RiLLD-Net), confrontation training of Wasserstein generative adversarial networks, and multitarget semantic segmentation. Experiments show that, especially for complex or obscured lane lines, Ripple-GAN can produce a superior detection performance to other state-of-the-art methods.According to different methods of lane line extraction, the feature extraction method of lane line detection is divided into two types: traditional methods and deep learning methods. Traditional lane line detection methods rely on a highly specialized combination of handmade features and heuristics to identify lane lines. Feature extraction according to the image gray gradient change, color, texture, visual vanishing point, and other features is analyzed and designed. These algorithms are sensitive to light, weather, and other changes. When the driving environment changes significantly, the performance of lane line detection is not good. With the improvement of computer computing power and the rapid development of GPU, deep learning technology has been widely used in computer vision, image processing, and other fields to extract lane features. The researchers proposed to study lane line detection as a segmentation problem and use image segmentation model to extract lane features. The model extracts feature information from a large number of images with annotated information and inferences the corresponding pixel tags in the original image according to the information. Under this end-to-end training, the model can better extract the semantic information of lane lines and classify each pixel. Compared with traditional manual methods such as edge feature extraction, threshold segmentation, and watershed, the image segmentation method based on deep learning can extract richer lane line information and has been widely applied in lane line detection technology [88, 89].


## 4.2. Adaptive Cruise Control

As the growth of vehicles and the acceleration of urbanization, the urban traffic congestion problem becomes a burning issue in our society [90]. ACC is an important longitudinal tracking technology in traffic flow research of autonomous vehicles. The vehicle can obtain the driving state of the vehicle in front in real time through the onboard detection equipment and the vehicle-vehicle communication technology, which has more timely and accurate traffic condition perception ability than ordinary drivers, more stable and safer decision-making and judgment ability, and more economic and environmental protection power output control.ACC is a longitudinal tracking control technology that obtains real-time workshop distance and speed information from the vehicle in front through onboard measurement equipment and uses acceleration optimization algorithm to control the vehicle and the vehicle in front to maintain a stable workshop distance. It can be regarded as an important part of self-driving vehicle technology. Research on ACC began in the 1960s and has entered the phase of implementation [91] in the United States, Japan, and Europe since the 1990s. ACC control system focuses only on the longitudinal driving control of the vehicle along the lane line direction, and its control system is usually divided into upper and lower control. Among them, the upper control is responsible for the output of the target acceleration at the next moment according to the running state such as workshop distance and speed difference obtained by the vehicle-mounted equipment. The lower control is responsible for adjusting the internal power system of the vehicle to achieve the acceleration optimization goal of the upper control. Therefore, the ACC lower control system mainly studies the specific electromechanical control process of the internal power system of the vehicle.ACC system is the core of the longitudinal assisted driving system, and the control strategy is the key to the whole ACC system, so the selection of control algorithm is particularly important to the design of ACC system. At present, commonly used control theory algorithms include the following: Sliding Mode Control (SMC) theory, Classical Proportion Integration Differentiation (PID), Fuzzy Control Theory, Model Prediction Control Theory, Fuzzy PID Control Theory, Optimal Control, and Artificial Neural Nets. The above control algorithms have been widely used in adaptive cruise system.Corona et al. calculated the driver previewed trajectory based on model predictive control theory and then carried out predictive analysis and processing on the driving state of the vehicle. Finally, they analyzed the control target quantity, analyzed the advantages and disadvantages of model predictive control (MPC) and proposed the corresponding optimization strategy [92]. In literature [93], a multiobjective collaborative ACC system was designed based on model predictive control, which solved the dilemma that the traditional ACC system was only suitable for simple distance keeping in a single working condition, and further optimized the problems of high computational amount and nonfeasible solution of model predictive control. Fritz and Schiehlen, based on the sliding mode control algorithm, solved the influence of external disturbance on the instability of low-speed following and at the same time optimized the tracking of ACC system to improve the robustness of the system [94]. Naranjo et al. also designed the lower controller of the adaptive cruise system based on fuzzy control to realize the control of engine throttle opening and brake pedal opening [95, 96].


## 4.3. AEB

AEB is an active safety system that can automatically start the driving brake when the vehicle independently detects the risk of collision ahead, so as to reduce the speed of the vehicle and avoid the collision as much as possible [97]. At present, IT has received more and more attention from the national government, oEMS, component manufacturers, and scientific research institutes. Pedestrians are different from target objects with fixed features, such as vehicles and traffic signs, so it is relatively difficult to detect and track pedestrians. Pedestrians have a variety of clothing and poses that are often difficult to detect due to occlusion and background interference, so the algorithm must be able to guarantee a high enough accuracy. Pedestrians often appear in complex outdoor environments with different backgrounds such as weather, road conditions, and streets, so the robustness and adaptability of the algorithm are highly required. Traffic accidents in real life usually occur in a short time, so the algorithm has a high requirement on real-time performance.Volvo’s City Safety is a low-speed AEB control system for urban roads that uses laser sensors to monitor the traffic ahead. Audi’s Pre Sense Front system uses millimeter-wave radar and camera data fusion to detect road conditions up to 80 meters in front of the vehicle.In 2019, the American Automobile Association (AAA) published a simulation test of the AEB system, which showed that AEB had a 40% success rate at 32 km/h, and AEB essentially failed when the speed reached 48 km/h. In addition, the AEB system also has quite high false alarm rate and false braking rate [98], which seriously affects the driver’s driving comfort and driving safety. Effective and accurate identification of forward target is the prerequisite for the normal operation of AEB control strategy. Wang et al. [99] found that the sensor is limited by the accuracy of millimeter-wave radar, the vision field of camera, and the limitations of multisensor data fusion algorithm. The accuracy and speed of the environmental perception and the handling of the emergency situation are still the key points and difficulties in the development of AEB system [100].


## 5. Summary and Prospects

This paper discusses the intelligent vehicle perception technology and introduces the development status and control strategies of several important sensors such as camera, laser radar, and millimeter-wave radar. Starting from the optimization of sensor results, three aspects of target detection, target recognition, and multisensor fusion are analyzed. The key scenarios of intelligent vehicle perception that have been applied to the ground are described, and lane detection, ACC, and AEB are analyzed. The research direction of intelligent vehicle perception technology is prospected, and the following points are pointed out:(1)
Single vision sensor can no longer meet the requirements of normal driving of intelligent vehicles. In the future, to solve the challenges faced by autonomous driving vehicles in the complex urban environment, the fusion of multisensors and vehicle road cloud will be the mainstream development in the future(2)
The detection, recognition, and tracking of moving objects are the main technical problems that the intelligent vehicle perception module needs to face. Deep learning gradually stands out among the current improved algorithms, but a large number of tests are needed to ensure the accuracy and timeliness of recognition(3)
Compared with intelligent vehicles, intelligent assisted driving system takes the lead in the application of products, such as ACC, AEB, and other functions. The intelligent assisted driving system has low cost and little interference factors, which will provide better test data for intelligent vehicles and accelerate the landing process of intelligent vehicles



---

*Source: 8199361-2021-09-03.xml*