Title: MULTIPLE ENCODER-DECODERS NET FOR LANE DE-TECTION

Abstract: For semantic image segmentation and lane detection, nets with a single spatial pyramid structure or encoder-decoder structure are usually exploited. Convolutional neural networks (CNNs) show great results on both high-level and low-level features representations, however, the capability has not been fully embodied for lane detection task. In especial, it's still a challenge for model-based lane detection to combine the multi-scale context with a pixel-level accuracy because of the weak visual appearance and strong prior information. In this paper, we we propose an novel network for lane detection, the three main contributions are as follows. First, we employ multiple encoder-decoders module in end-to-end ways and show the promising results for lane detection. Second, we analysis different configurations of multiple encoder-decoders nets. Third, we make our attempts to rethink the evaluation methods of lane detection for the limitation of the popular methods based on IoU.

1. INTRODUCTION:

Autonomous driving has become a focus of computer vision nowadays. A key component of autonomous driving, optical image based lane detection, is also one of the most important task for feature maps producing and Semantic information utilizing research. In practice, camera-based lane detection still be a challenging task for several reasons. First, the local information of a lane such as sharp, edges, texture and color, can not provides distinctive features for lane detection. Second, it is common for vehicles to be in the complex optical scenarios like rainy, night, shadow, dazzle light conditions. Third, the incomplete lanes make it difficult to get the local informations while in a condition such as crossroad, crowded traffic, no line, etc. Furthermore, the strong global prior knowledge of lane detection put forward higher requirements for the perceptive field of CNNs.

End-to-end CNNs always give better results than systems relying on hand-crafted features. Although the low-level feature maps can supply the local information of the whole image, hand-crafted features based systems are limited to generate sharp features. Most existing algorithms for the segmentation of road boundaries and lane detection (e.g. (Borkar et al., 2012) , (Deusch et al., 2012) , (Hur et al., 2013) , (Jung et al., 2013) , (Tan et al., 2014) ) rely on detectable and meaningful parts of the local features. As aforementioned limitations, it insufficient to recognize and detect lanes using local features in lots of scenes. Highly hand-craft features based methods can only deal with harsh scenarios.

CNNs show the robust capacity to capture object localization on image classification (Krizhevsky et al., 2012) , (Sermanet et al., 2013) , (Simonyan & Zisserman, 2014) and object detection (Girshick et al., 2014) , (Liu et al., 2015) , (Redmon et al., 2016) , (He et al., 2016) , but less explored on Semantic Image Segmentation due to strong prior information is needed. Although High-level features have shown great capacity to understand Semantics, they lost localization accuracy of the object. Especially in lane detection task, while a human being is able to recognize the lane well under different scenes, could be very challenging for the weak visual appearance and strong prior information conditions, including crowded traffic, dim or dazzle light, etc. 

1.1. RELATED WORK:

For lane detection, traditional methods rely on highly-specialized, handcrafted features and heuristics from the background pixel by pixel to identify lane segments. These works detect generic sharps of markings and try to fit a line or a spline to localize lanes by Kalman filter (Nieto et al., 2011) , tracking-based method (Lipski et al., 2008) , B-snakes (Sawano & Okada, 2006) , (Wang et al., 2004) , hierarchical bi-partite graph (Nieto et al., 2008) , color-based features (Chiu & Lin, 2005) , the structure tensor (Loose et al., 2009) , the bar filter (Teng et al., 2010) , ridge features (Lpez et al., 2010) , etc. In general, these traditional approaches are prone to robustness issues due to road scene variations that can not be easily modeled by such model-based systems.

Popular choices of recent methods have replaced the feature-based methods with model-based methods. In an attempt to take advantage of the CNN features from multiple layers, (Huval et al., 2015) used object detection methods to detect lanes, (Pan et al., 2017) views rows or columns of feature maps as layers and applies convolution, (Neven et al., 2018) try to cast the problem as an instance segmentation problem, (Ghafoorian et al., 2018) applied a GAN framework to mitigate the discussed problem. (Cong et al., 2018) tackle this problem by using template matching with RGB-D camera.

There have been some other attempts to parse semantic information in a pixel level. (Visin et al., 2015) and (Bell et al., 2016) utilized recurrent neural networks(RNN) to pass information along each row or column, thus in one RNN layer each pixel position could only receive information from the same row or column. (Liang et al., 2016) proposed variants of LSTM to exploit contextual information in semantic object parsing, but such models are computationally expensive. Researchers also attempted to combine CNN with graphical models like MRF or CRF, in which message pass is realized by convolution with large kernels (Peng et al., 2017) .

1.2. OUR CONTRIBUTIONS:

Our contributions can be summarized to the following.

First, reduced localization accuracy due to the weak performance of combining the local information and global information effectively and efficiently. We propose an end-to-end net architecture which is applied to multiple encoder-decoders module. Multiple encoder-decoders net shows a successful result for lane detection.

Second, we analysis the capability of nets with different configurations to combine the location and semantic information. In both visual and statistical way, we compare the performances of nets with different encoder-decoders times and give an explanation of the performances. Third, the evaluation methods which are used in recent works for lane detection are transfered from other tasks. there exists some limitations of these methods for lane detection. We make our attempts to rethink these IoU based methods. CNNs have achieved great success on producing feature maps and utilizing semantic information.

For lane detection task, the challenge is to combine the local information and global information effectively and efficiently. We address the issue by merging the feature maps multiple times as shown in figure 2 .

The traditional methods are limited in the little receptive field of the small kernel and the huge amount of computation of the large kernel. But for multiple encoder-decoders nets, it is possible to use the small kernel for low-level features and the large kernel for high-level features. In practice, we use the 9*9 kernel for the lowest-level features for only once and the 3*3 or 1*1 kernel for the most feature maps, which is able to learn the spatial and semantic information well and reduce the computation greatly.

To merge the feature maps sufficiently, we not only apply multiple encoder-decoders net, but also place the residual layer and route layer instead of pooling layer and activity layer. It can be seen the combination of the different level feature comprises richer information, might be a better structure to model the lane representation.

During training we optimize the following loss function

loss = 1 N N i=1 S j=1 (x i,j -x i,j ) 2

N denotes the number of lanes of the input image, while S denotes the number of pixel. x i,j = 0.5 if a lane covers the pixel of i, j, and if a lane does not cover the pixel of i, j x i,j is set to 0. Considering the imbalanced label between background and lane markings, the loss of background is multiplied by 0.5.

3. EXPERIMENT:

In this section, we compare the influence of the different level encoder-decoder times and number of channels of a multiple encoder-decoders net on different scenarios. Furthermore, we give the vi-sualized results of comparison between our method and the state-of-the-art methods in figure 3 . We modify the figure from (Pan et al., 2017) directly so that the results is able to prove the significance of multiple encoder-decoders net. 

P recision = T P T P +:

F P Recall = T P T P + F N F -measure = (1 + β 2 ) P recisionRecall β 2 P recision + Recall

As figure 4 shown, predictions with the same IoU can be different extremely. while the red line is inaccurate, the blue line is an apparent better prediction. We have thought about adding geometrical constraint, evaluating by key points and some other probability theory methods, but these methods are less than ideal either. In order to judge whether a lane marking is successfully detected and discover the the ability to learn the correct and precise features of different nets, we have compared more than 500 probmaps of each level nets manually and count the accuracy of these probmaps as shown in figure 7 . 

3.3. NUMBER OF ENCODER-DECODER:

After abundant experiments, generally, we find that ability of the nets to acquire semantic information enhances along with the number of encoder-decoder growing, but the ability to extract the spatial information and pinpoint the localization of the objects declines. Nets with three or four times encoder-decoder model reach a balance between the local and global information which can be seen in figure 6 and figure 7 . To improve ability of the network, we propose a small quantity of channel to reduce overfitting by considering interdependencies among channels. We enable networks with tens of layers to train easier and get a better performance by stacking a small quantity of channel(eg., 4) more times in the entire network. 

4. CONCLUSIONS:

We propose multiple encoder-decoders net to extract the spatial information and pinpoint the localization of the lane detection. multiple encoder-decoders structure allows the net to improve localization and sharp performance simultaneously. Furthermore, to improve ability of the network, we propose a small quantity of channel to reduce overfitting by considering interdependencies among channels. Extensive experiments on CUlane dataset demonstrate the effectiveness of our proposed net. Last, we rethink the evaluation methods which are used in recent works and exist some shortcomings for lane detection.

5 REFERENCES

:



Figures and Tables:

Figure 1: Example results of our method on the task of lane detection.

Figure 2: (a)Nets with multiple encoder-decoders model can combine richer local and global information than nets with a single encoder-decoder model.

Figure 3: The visual results of comparison between our method and the state-of-the-art methods, which can tell the nets with multiple encoder-decoders model can combine richer local and global information than nets with a single encoder-decoder model.

Figure 5: (a)These samples are sampling randomly from the CUlane dataset of normal, night, crowded , diverging paths and dazzle light scenarios.

the nets we used , the twice encoder-decoders model has 56 layers, three times encoder-decoders model has 79 layers, four times encoder-decoders model has 102 layers and five times encoderdecoders model has 125 layers. As what can be seen in the the column chart, for normal scenarios, these nets achieve competitive result on predictions. On the contrary, nets with three or four times encoder-decoder model get much better results for complex optical conditions. On the other hands, the very deep structure of five times encoder-decoders model plays a positive role of modeling the global feature for diverging paths.

Figure 7: lane detection accuracy of multiple encoder-decoders nets with different configurations.

Figure 8: Example results of our method with three times encoder-decoder model and different number of channel.

Peer review (iclr2019):
 The paper addresses the problem of pixel-wise segmentation of lanes from images taken from a vehicle-mounted camera. The proposed method uses multiple passes through encoders decoders convnets, thereby allowing extract global features to inform better local features, and vice versa. Only qualitative baseline comparisons are presented by manually comparing the output of the network to reported results of other methods in [Pan et al.2017]. It is unclear to me if the proposed multiple encoder-decoder network is a novel architecture, or a known architecture applied to a novel use case. In case of the former, more details should be given on the design of the network, how it is trained, etc. for reproducibility. The biggest problem however is the subjective manual comparison to existing methods, which the authors do in favor of a quantitative comparison using well-understand objective metrics. While they point out problems with evaluating segmentation with conventional accuracy metrics, no attempt is made to make a better objective measure. We are left to judge the results on only a few selected example frames. It is also unclear how the method and evaluation strategy compares to methods which predict lanes as splines or other parameterized functions. E.g. see surveys on existing approaches, and discussion of different evaluation strategies, e.g. "Recent progress in road and lane detection: a survey" [Hillel et al.2014] and "Visual lane analysis and higher-order tasks: a concise review" [Shin, 2014]. Throughout the paper, various fuzzy and unclear statements are made (see detailed comments below). The paper would be in a better shape if more time is spend to improve the writing, provide more details on the method, and extend the experiments. Pros: + multiple encoder-decoder stages could be beneficial for lane segmentation Cons: - lacking evaluation and comparison to baseline methods - missing details on proposed network architecture, making it hard to reproduce - unclear what colors in figures for qualitative evaluation represent: are individual lanes also distinguished? Below are more detailed comments and questions: * Abstract * "the capability has not been fully embodied for" → Fuzzy statement, I don*t understand what this means. * "In especial" → check grammar * Sec 1.: Introduction * "the local information of a lane such as sharp, edges, texture and color, can not provides distinctive features for lane detection" local edges are not distinctive for lanes? Possibly local edges alone are not sufficient, but various lanes detection approaches rely on edge extraction as features. This statement therefore seems too strong. * "End-to-end CNNs always give better results than systems relying on hand-crafted features.". It is not possible to say that one type of classifier categorically better than another. The *best* classifier depends on the problem at hand, valid assumptions that can be made, and the amount of training data avaiable, among others. For instance, "How Far are We from Solving Pedestrian Detection?" [Zhang,CVPR16] demonstrates that CNNs do not always give better results than hand-crafted features for some tasks and datasets. The paper should be more careful with such strong statements. * "Highly hand-craft features based methods can only deal with harsh scenarios.". I don*t understand, is this statement intended as an argument against hand-crafted features? Isn*t it good to deal especially with harsh scenarios? * "but less explored on Semantic Image Segmentation due to strong prior information is needed." CNNs are extensively used for semantic image segmentation, e.g. see the well-known Cityscapes benchmark. * "recent methods have replaced the feature-based methods with model-based methods.". Not sure why the paper call CNNs "model-driven methods", but refer to the earlier classical methods with highly designed representations (Kalman filter, B-snakes, ...) as "feature-based methods". This seems diferent from what I typically see, where CNNs are referred to as *data-driven methods*, and the classical methods as "model-driven". * Sec 1.2: Contributions * "First, reduced localization accuracy due to the weak performance of combining the local information and global information effectively and efficiently". Instead of presenting a first contribution, the paper presents a problem. Do the authors mean that they "tackle the problem of reduced localization accuracy ..." ? That would still not make this contribution very concrete though ... * "We make our attempts to rethink these IoU based methods." → Please argue in favor of your new method. An in-depth comparison of evaluation methods, and why some metrics fail or could be redesigned would be good. However, the paper currently fails to present a new metric, and convince that it tackles shortcomings of established metrics. * Sec 2.: Multiple Encoder-Decoder Nets * Figure 2: Is this the first paper to propose this multiple encoder-decoder net? Or is the idea taken from other work, and is the novelty to apply it to this problem? If this general architecture was already proposed (for semantic segmentation?), please add citations and discuss it as related work. If this network design is completely novel, I would expect more details on how the network is constructed (e.g. dimensions of each layer, non-linear activation function used, batch normalization, strides, etc.). * "the following loss function:". Since it is a binary classification problem, and not a regression problem, why not use a (binary) cross entropy loss instead of a mean squared error? * Sec 3: Experiments * Figure 3: What is the "Baseline" method ? Where are the references to the other works, or is the reader required to read [Pan*2017] to understand your figures? * Figure 3: How are the colors in these figures determined? Is this also an instance segmentation problem? From your methodology section I though only binary classification was considered. Do you do some post-processing to separate individual lanes? I find this confusing, as I thought that the task was limited to binary segmentation. * "Recent works evaluated ..." please cite the works you refer to. * "we have compared more than 500 probmaps of each level nets manually and count the accuracy of these probmaps as shown in figure 7." So if I understand correctly, instead of using an objective evaluation metric, you have reverted to manual labor to visually judge lane detection quality. This is not really a metric, and not really a solution that *rethinks IoU based methods.* Problems of your approach is that it is unclear on what criteria results are judged, your evaluation is not objectively reproducible by others, and does not scale well for novel future evaluations. Why is this even needed? E.g. why not use some chamfer distance or Gaussian smoothing of the edge map if you want to evaluate near coverage instead of hard boundaries? Or, fit a function through the boundary, and evaluate distance (in meters) to true lane. I find the proper discussion and motivation for manual evaluation over objective metric evaluation lacking. * Figure 6: What are the Ground Truth images of each row ? E.g. in the fourth row from the top, should the right-most yellow lane be present or not? As it stands, I can*t interpret the columns and see which x times is visually *better*. * Sec 3.4: * "To improve ability of the network, we propose a small quantity of channel to reduce overfitting by considering inter-dependencies among channels." To improve relative to what? Where are the results comparing large amounts vs small amount of channels? Note that Figure 8 is not referred to in the text, and confusingly compares "18 layers" to "1 layers". Do you mean channels instead of layers? And, how many channels were to obtained the results in the preceding sections?