# A Survey on Theories and Applications for Self-Driving Cars Based on Deep Learning Methods
## 4 Applications Overview of Deep Learning in the Field of Self-Driving Cars
### Obstacle Detection  
Obstacle detection technology is the most fundamental and core technical problem in self-driving systems, which is used to detect the target area that may block the normal driving of the vehicle, and guide the vehicle to avoid obstacles in time through the control system. In recent years, various sensors have been developed and equipped on self-driving cars to realize obstacle detection and recognition, such as the vision sensor, the ultrasonic sensor, the radar sensor, the lidar sensor, and so on [43, 44]. Compared with detection technology based on other sensors, the vision sensor-based detection technology has many advantages, such as fast sampling speed, a large amount of information, and relatively low price. Obstacle detection technology based on vision sensors mainly includes a monocular vision and binocular vision [45]. The traditional monocular visual detection technology mainly relies on the manual design features to construct the model, and the quality of the model depends on the prior knowledge of the designer, so the recognition accuracy of this kind of algorithm is not high. The development of deep learning has revolutionized machine vision, and there are lots of research results in this field. For example, [PERSON], et al. [46] proposed a CNN architecture, which can jointly finish the learning task for depth estimation and obstacle detection. [PERSON] [47] presented a monocular vision-based algorithm. It can detect obstacles and identify obstacle-aware regions by implementing a deep encoder-decoder network. [PERSON], et al. [48] proposed an improved CNNs based on the addition of a range estimation layer, which can accomplish obstacle detection, classification and ranking simultaneously. The advantage of binocular vision over monocular vision is that 3D information of the scene can be directly obtained, and the geometric relationship between obstacles and road surface can be supplemented, so as to serve as the segmentation basis to realize obstacle detection. The flow chart of obstacle detection using binocular vision technology is shown in Figure 10. First, binocular  
Figure 9: The classification of the deep learning methods used in the field of self-driving cars. images are acquired by the binocular camera, and a disparity map is obtained by the stereo matching algorithm. Then the calculation of the disparity map determines whether it is an obstacle point. Finally, the obstacle area is extracted. Typical stereo matching algorithms include the following four steps: matching cost computation, cost aggregation, disparity computation, and disparity refinement [49]. There are many research results on the binocular vision obstacle detection based on deep learning. For example, [PERSON], et al. [50] designed a twin convolution structure MC-CNN (Matching Cost-Convolutional Neural network), where the CNN was applied to the image similarity measure and matching cost calculation. The structure of this MC-CNN is shown in Figure 11.  
In the twin convolution structure of MC-CNN, there are many convolutional layers in the networks. Each layer is followed with a rectified linear unit. Then the obtained two vectors are connected in series and propagated by the fully-connected layers. At last, a single number is produced by the last fully-connected layer. This number denotes the similarity rate between the input patches. In the method of [50], the matching cost is directly initialized from the output of the network:  
\[C_{\text{CNN}}\left(\mathbf{p},d\right)=-s\left(<P^{L}\left(\mathbf{p}\right), P^{R}\left(\mathbf{p}-d\right)>\right), \tag{10}\]  
where \(s\left(\cdot\right)\) is the output function; \(P^{L}\left(\mathbf{p}\right)\) and \(P^{R}\left(\mathbf{p}-d\right)\) are the input patches from the left image and the right image respectively; \(\mathbf{p}\) denotes the position \(\left(x,y\right)\), and \(d\) is the correct disparity at this position. The cost aggregation process of stereo matching in MC-CNN is as follows:  
\[