# A Survey on Theories and Applications for Self-Driving Cars Based on Deep Learning Methods
## 4 Applications Overview of Deep Learning in the Field of Self-Driving Cars
### Obstacle Detection
\begin{array}{l}C_{\text{CBCA}}^{0}=C_{\text{CNN}}\left(\mathbf{p},d\right) \\ C_{\text{CBCA}}^{+}=\frac{1}{\left|U_{d}\left(\mathbf{p}\right)\right|}\sum \limits_{\mathbf{q}\in U_{d}\left(\mathbf{p}\right)}C_{\text{CBCA}}^{i-1}\left( \mathbf{q},d\right),\end{array}  \tag{11}\]  
Figure 11: The structure of MC-CNN proposed in [50]. Figure 10: Binocular visual obstacle detection flow chart. where, \(U_{d}\left(\mathbf{p}\right)\) denotes the combined support region for \(\mathbf{p}\); \(i\) is the number of iterations. The final matching cost \(C_{\text{SGM}}\) is defined as the average across of the four directions \(\mathbf{r}\), namely:  
\[C_{\text{SGM}}\left(\mathbf{p},d\right)=\frac{1}{4}\sum_{\mathbf{r}}C_{ \mathbf{r}}\left(\mathbf{p},d\right) \tag{12}\]  
(13)  
where \(P_{1}\) and \(P_{2}\) are the penalty parameters. Then the disparity image \(D\) is calculated by finding the disparity \(d\) that minimizes \(C\left(\mathbf{p},d\right)\), which is as follows:  
\[D\left(\mathbf{p}\right)=\operatorname*{arg\,min}_{d}C\left(\mathbf{p},d \right). \tag{14}\]  
After the disparity is calculated, the obstacle area can be detected, which makes it possible to detect the obstacle in the self-driving car. The experiment examples of disparity results using MC-CNN are shown in Figure 12.  
The solution of the stereo matching algorithm lays a solid foundation for obstacle determination. There are other methods based on deep learning used in the obstacle detection for self-driving cars. For example, [PERSON], et al. [51] presented a network structure, which is constructed by a wide context learning network and stacked encoder-decoder 2D CNNs. [PERSON], et al. [52] proposed an end-to-end multidimensional residual dense attention network, which focuses on more comprehensive pixel-level feature extraction. The network includes a two-dimensional residual dense attention network for feature extraction and a three-dimensional convolutional attention network for matching. [PERSON], et al. [53] presented a deep learning network for regressing the disparity of a pair of stereo images, where the context is incorporated directly from the data employing 3D convolutional network. [PERSON], et al. [54], proposed a method where a deep-stacked auto-encoders (DSA) model is used. In this DSA model, the greedy learning features are combined with the dimensionality reduction capacity. In addition, an unsupervised \(k\)-nearest neighbor algorithm is employed to detect the obstacles. A summary of the deep-learning-based obstacle detection algorithms is illustrated in Table 1, where the main methods used in the obstacle detection are given out and the corresponding references are also listed. Figure 12: Examples of the disparity results by MC-CNN in [50]: (**a**) error of 3.73%; (**b**) error of 2.36%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Category** & **Reference** & **Method** & **Remarks** & **Performances** \\ \hline \multirow{3}{*}{Monocular vision} & [PERSON], et al. & \multirow{3}{*}{CNN} & The input of the proposed network is an RGB image and the features are extracted by the VGG19 module. Then these features are used to obtain the dense depth maps and obstacles bounding boxes based on CNNs. & \multirow{3}{*}{Recall of 90.85\% on UnrealDataset dataset.} \\  & [PERSON], et al. (2019)[46] & & & \\  & [PERSON], et al. & \multirow{3}{*}{CNN} & The range of the object in meters and class labels for each detected object are & \multirow{3}{*}{mAP of 96.92\% on KITTI dataset.} \\  & (2019)[48] & & produced based on the proposed feed-forward CNN. & \multirow{3}{*}{RMSE of 6.863 on KITTI dataset.} \\  & [PERSON], et al. & \multirow{3}{*}{FCN} & A fully convolutional network (CNN) is proposed, which is used to obtain depth & \multirow{3}{*}{RMSE of 6.863 on KITTI dataset.} \\  & (2016)[55] & & estimation based on both images and optical flows. & \\  & [PERSON], et al. & \multirow{3}{*}{CNN and DBN} & The local information of a candidate block is generalized by CNN, and the global information of the whole image is generalized by DBN.