# Lane Detection Based on Adaptive Network of Receptive Field

Authors: YuFan Cai; YanYan Zhang; ChengSheng Pan
Journal: Security and Communication Networks
 (2021)

---

## Abstract

The difficulty of lane detection lies in the imbalance of the number of target pixels and background pixels. The sparse target distribution misleads the neural network to pay more attention to background segmentation in order to obtain a better loss convergence result. This makes it difficult for some models to detect lane line pixels and leads to the training fail (unable to output useful lane information). Increasing receptive field properly can enlarge the sphere of action between pixels, so as to restrain this trouble. Moreover, the interference information and noise existing in the real environment increase the difficulty of lane classification, such as vehicle occlusion, car glass reflection, and tree shadow. In this paper, we do think that the features obtained by the reasonable combination of receptive fields can help avoid oversegmentation of the image, so that most of the interference information can be filtered out. Based on this idea, Adaptive Receptive Field Net (ARFNet) is proposed to solve the problem of receptive field combination with the help of multireceptive field aggregation layers and scoring mechanism. This paper explains the working principle of ARFNet and analyzes several results of experiments, which are carried out to adjust network structure parameters in order to get better effects in the CuLane dataset testing.


---

## Body

## 1. Introduction

With the advent of 5G era, automatic driving as a new traffic concept has been developed rapidly. It requires the further development of lane detection technology based on computer vision, as it plays an important role in automatic driving technology. Current mainstream lane detection algorithms give up using single filter; rather, they choose methods of deep learning such as the convolution neural network or Transformer to transform the lane detection task into image segmentation. However, due to the special structural characteristics of lane lines, the number of distributions between background pixels and target pixels is seriously uneven. This sparsity seriously affects the quality of network results. In addition, the widespread interference information in the driving environment makes it difficult to distinguish lane line pixels from background pixels.To solve these problems, Lee et al. [1] chose to engage more human interventions in the training process, using vanishing points to guide the training of deep models, but this way requires more manpower. Pan et al. [2] added a spatial information transfer module in VGG16 [3] network to effectively resist the sparsity effect, and the lane position can be located more accurately by transferring the local information to the global. However, this message passing process increases the inference time greatly. To avoid a large increase in time, Hou et al. [4] used self-attention distillation technology, which encourages the network to extract more global information from feature graph by adding distillation loss function. Lee et al. [5] combined the advantages of the above two methods to transfer information in the horizontal and vertical directions in the distillation diagram. The ESA module abandons the method of using the average of distillation diagram to learn layer by layer but directly learns from the label via MSE loss. Qin et al. [6] grasped the special mathematical distribution characteristics of lane pixels, taking advantage of the structure-aware method to detect lane lines faster and better. However, all methods of distillation still need additional loss function to constrain the learning direction, which means that much more human testing is needed to adjust the function form, and it will bring more challenges as well.In addition to using Convolution Neural Network (CNN) and front view to model, Garnett et al. [7] not only input the bird’s eye view into the CNN, but also took advantage of 3D modeling method to deal with the complex fork road and curve situation. Liu et al. [8] replaced CNN model with Transformer and greatly improved both the accuracy and speed. In fact, these two methods have high requirements for hardware configuration, so it is not easy to train the model.In our work, without adding any additional loss function, ARFNet aggregates information from different receptive fields in the layer. It extracts the mean values of different feature graphs as global information, which will be scored in multilayer perceptron (MLP). This process does not add extra inference time, and at the same time, with the help of pruning [9], the network structure can be simplified without harming the performance according to the score. Moreover, we have done some detailed adaptive ability exploration experiments and achieved good results in lane detection.


## 2. Related Work

### 2.1. Network Structure Optimization Theory

CNN based deep learning models can be regarded as many feature streams constructed by multilayer filters, as shown in Figures1 and 2.Figure 1
Working process in one convolution layer.Figure 2
Feature streams in CNN.If the convolution operations in layers are ignored, the forward propagation process can be described by a path matrix:(1)1A1B⋯OUT1A2B⋯OUT⋮⋮⋮⋮1B2C⋯OUT.The number of rows in the matrix represents the number of paths, and the number of columns represents the depth of the network. The elements in the matrix represent the feature maps. Each independent path will have different impacts on the classification judgment of different categories. Any modification of the matrix will affect the learning ability of the network. Therefore, the network design can be intervened by changing the number of matrix elements and changing the value of elements.(1)
Changing the number of matrix elements. EfficientNet [10, 11] is a good example. It uses a simple and efficient composite coefficient to enlarge the baseline from three dimensions: depth, width, and resolution. This makes the network structure more reasonable without any cost of manual adjustment. In other words, the size of the path matrix is optimized.(2)
Changing the value of elements. As shown in Figure3, convolution process can be equivalent to matrix multiplication (Hadamard product). This means that each feature map can be described as shown in (2)Feature=Data∗∏l=1LWl.(3)
Although the values of elements in kernel space are random and variable, we still can change the initialization distribution, like Xavier [12], or use the constraint mechanism to guide the change process. DyNet [13] uses covariance prediction module and dynamic convolution kernel mechanism to guide the weighted feature map forming to improve convolution efficiency. In addition, changing the distribution of the kernel space will affect the feature maps as well. DCN [14, 15] has adopted this strategy, which adds the offset to adjust the position of convolution kernel. Moreover, pruning [9] also can be combined to make the matrix more sparse, thus reducing the amount of calculation.(4)
Changing the initial data. Generally, the initial image will be rotated, clipped, or do value transformation to increase the generalization ability of the model, such as CutMix [16], PBA [17], and autoaugmentation [18]. In addition, Tsinghua team also found that the data augmentation can still be finished via loss function (ISDA [19]).Figure 3
Equivalent expression.In our work, due to the limitation of hardware equipment, it is necessary to limit the model size artificially. Therefore, instead of adopting the first option, we will integrate the second and third options.


### 2.2. ARFNet

In lane detection, appropriately increasing the receptive field can effectively improve the accuracy of classification. Inspired by the structure of InceptionNet [20], convolution modules with different receptive fields will be fused in parallel in one layer. Meanwhile, receptive fields can be easily changed by dilated convolutions [21]. Hence, in theory, the convolution module of ARFNet should be as shown in Figure 4.Figure 4
ARF-module.By adding the squeeze-and-excitation-module [22, 23] (see Figure 5), We have achieved the scoring of information under different receptive fields. This effect will also be passed along with the accumulation of multiplication. Unfortunately, in the actual experiment, the number of parallel modules is limited; in other words, the dilation rate r is limited by the hardware storage capacity. Therefore, instead of adopting the strategy of structural search proposed in IC-conv [24] to implement self-adaption, our work chooses to manually adjust the dilation rate and use the scoring mechanism to let the network adapt to these features. At the same time, it is easier to find the effect of different receptive field aggregation on the results by manual adjustment. Specific details will be introduced in the experimental section.Figure 5
Se-module.


## 2.1. Network Structure Optimization Theory

CNN based deep learning models can be regarded as many feature streams constructed by multilayer filters, as shown in Figures1 and 2.Figure 1
Working process in one convolution layer.Figure 2
Feature streams in CNN.If the convolution operations in layers are ignored, the forward propagation process can be described by a path matrix:(1)1A1B⋯OUT1A2B⋯OUT⋮⋮⋮⋮1B2C⋯OUT.The number of rows in the matrix represents the number of paths, and the number of columns represents the depth of the network. The elements in the matrix represent the feature maps. Each independent path will have different impacts on the classification judgment of different categories. Any modification of the matrix will affect the learning ability of the network. Therefore, the network design can be intervened by changing the number of matrix elements and changing the value of elements.(1)
Changing the number of matrix elements. EfficientNet [10, 11] is a good example. It uses a simple and efficient composite coefficient to enlarge the baseline from three dimensions: depth, width, and resolution. This makes the network structure more reasonable without any cost of manual adjustment. In other words, the size of the path matrix is optimized.(2)
Changing the value of elements. As shown in Figure3, convolution process can be equivalent to matrix multiplication (Hadamard product). This means that each feature map can be described as shown in (2)Feature=Data∗∏l=1LWl.(3)
Although the values of elements in kernel space are random and variable, we still can change the initialization distribution, like Xavier [12], or use the constraint mechanism to guide the change process. DyNet [13] uses covariance prediction module and dynamic convolution kernel mechanism to guide the weighted feature map forming to improve convolution efficiency. In addition, changing the distribution of the kernel space will affect the feature maps as well. DCN [14, 15] has adopted this strategy, which adds the offset to adjust the position of convolution kernel. Moreover, pruning [9] also can be combined to make the matrix more sparse, thus reducing the amount of calculation.(4)
Changing the initial data. Generally, the initial image will be rotated, clipped, or do value transformation to increase the generalization ability of the model, such as CutMix [16], PBA [17], and autoaugmentation [18]. In addition, Tsinghua team also found that the data augmentation can still be finished via loss function (ISDA [19]).Figure 3
Equivalent expression.In our work, due to the limitation of hardware equipment, it is necessary to limit the model size artificially. Therefore, instead of adopting the first option, we will integrate the second and third options.


## 2.2. ARFNet

In lane detection, appropriately increasing the receptive field can effectively improve the accuracy of classification. Inspired by the structure of InceptionNet [20], convolution modules with different receptive fields will be fused in parallel in one layer. Meanwhile, receptive fields can be easily changed by dilated convolutions [21]. Hence, in theory, the convolution module of ARFNet should be as shown in Figure 4.Figure 4
ARF-module.By adding the squeeze-and-excitation-module [22, 23] (see Figure 5), We have achieved the scoring of information under different receptive fields. This effect will also be passed along with the accumulation of multiplication. Unfortunately, in the actual experiment, the number of parallel modules is limited; in other words, the dilation rate r is limited by the hardware storage capacity. Therefore, instead of adopting the strategy of structural search proposed in IC-conv [24] to implement self-adaption, our work chooses to manually adjust the dilation rate and use the scoring mechanism to let the network adapt to these features. At the same time, it is easier to find the effect of different receptive field aggregation on the results by manual adjustment. Specific details will be introduced in the experimental section.Figure 5
Se-module.


## 3. Methodology

### 3.1. Loss Function

Lane detection is commonly formulated as a semantic segmentation task. Specifically, we assign the corresponding class numberi (i = 1, 2, 3, …, N) to the lane line pixels to facilitate the completion of a multiclassification task. At the same time, the lane existence label is added to assist network learning that uses the binary labels to describe the distribution of lanes. The total loss function is described as (3)Loss=Lsegs,s^+0.5∗Lexistb,b^,where s^,b^ is the predict map, and s,b is the label. In order to highlight the importance of the target in the background, in the cross-entropy loss function, the weight of the labels will be reassigned after one-hot processing. The weight distribution is arranged in the following order:(4)Weight=Background, Lane1, Lane2, Lane3, Lane4.By actively suppressing attention to the background, the model can perform better under sparse labels. However, excessive restraint will bring more misjudgment that tends to distinguish more background pixels into lanes pixels. Based on this factor, after several testing and comparisons, we find that the weight distribution of0.2,2,2.5,2.5,2 can bring better prediction results of lane pixels than just using the weight of 1,1,1,1,1. Therefore, this paper will use the weight of 0.2,2,2.5,2.5,2 in the following experiments.


### 3.2. Dataset

Figure6 shows a sample of the CuLane [25] dataset. Although the dataset contains only four lanes, its sample range is wide including a large number of complex scenes with background interference. It is an extremely challenging training dataset, so that it can well test the learning ability and generalization ability of the model. The specific dataset description is shown in Table 1 (the color changes in Figure 6(b) represent the occlusion, and interference has occurred in different degrees).Figure 6
(a) CuLane sample. (b) Overlay.
(a)(b)Table 1
Basic information of CuLane dataset.
NameTrainValidationTestResolutionRoad typeOther labelLaneCuLane888809675346801640 × 590Urban, rural, highway✓4


### 3.3. Experiment Configuration and Strategy

In this paper, the only experimental GPU is RTX2080 super. Due to the limited video memory of 8G, we compress the original image to 800 × 216 and set the length of each batch to 10. In the experiment, we randomly selected 80 K images as the training set and 9 K images as the validation set. Moreover, we take advantage of SGD [26] algorithm and CosineAnnealing [27] algorithm to change the learning rate and optimize the parameter update. In the experiment, the initial learning rate is set to 1e − 3, the weight decay is set to 1e − 4, and the annealing period is set to 10 to enhance overfitting resistance. In addition, we retain the momentum part of 0.9 and allow the Nesterov [28] algorithm to correct the gradient direction. Because there are many comparison experiments in our experiments, we only normalize images as shown in equation (5) and will not use the dataset augmentation strategy or batch random processing in order to ensure the consistency of learning samples.(5)data=data−meanstd.Mean is the expectation of the data set, and std is the standard deviation of the data set. For the CuLane dataset, the parameter values are as(6)mean=0.3598, 0.3653, 0.3662,std=0.2573, 0.2663, 0.2756.


### 3.4. Evaluation Metrics

Different from the evaluation metrics in SCNN [2], which used IoU (see equation (7)) to increase fault tolerance and F1-score (see equation (8)) to evaluate the performance, we choose to use Accuracy (see equation (9)) and Recall (see equation (10)) of each lane line to comprehensively evaluate the performance.(7)F1−score=2∗Recall∗PrecisionRecall+Precision,(8)IoU=TPTP+FP+FN,(9)Recall=TPTP+FN,(10)Accuracy=NpredNgt,(11)Precision=TPTP+FP,where TP represents the number of lane pixels accurately classified, FN represents the number of lane pixels wrongly judged as background, FP represents the number of background pixels wrongly judged as lane pixels, Npred represents the number of pixels correctly classified, and Ngt represents the total number of pixels. In the experiment, we find that the Precision is always much larger than the Recall. This is because the sparsity of the target leads to misjudge lane pixels as background pixels. This makes the Precision unable to accurately describe the classification ability of the network, so we do not use the F1-score, which contains the Precision. In addition, this paper will not add any fault-tolerant means.


## 3.1. Loss Function

Lane detection is commonly formulated as a semantic segmentation task. Specifically, we assign the corresponding class numberi (i = 1, 2, 3, …, N) to the lane line pixels to facilitate the completion of a multiclassification task. At the same time, the lane existence label is added to assist network learning that uses the binary labels to describe the distribution of lanes. The total loss function is described as (3)Loss=Lsegs,s^+0.5∗Lexistb,b^,where s^,b^ is the predict map, and s,b is the label. In order to highlight the importance of the target in the background, in the cross-entropy loss function, the weight of the labels will be reassigned after one-hot processing. The weight distribution is arranged in the following order:(4)Weight=Background, Lane1, Lane2, Lane3, Lane4.By actively suppressing attention to the background, the model can perform better under sparse labels. However, excessive restraint will bring more misjudgment that tends to distinguish more background pixels into lanes pixels. Based on this factor, after several testing and comparisons, we find that the weight distribution of0.2,2,2.5,2.5,2 can bring better prediction results of lane pixels than just using the weight of 1,1,1,1,1. Therefore, this paper will use the weight of 0.2,2,2.5,2.5,2 in the following experiments.


## 3.2. Dataset

Figure6 shows a sample of the CuLane [25] dataset. Although the dataset contains only four lanes, its sample range is wide including a large number of complex scenes with background interference. It is an extremely challenging training dataset, so that it can well test the learning ability and generalization ability of the model. The specific dataset description is shown in Table 1 (the color changes in Figure 6(b) represent the occlusion, and interference has occurred in different degrees).Figure 6
(a) CuLane sample. (b) Overlay.
(a)(b)Table 1
Basic information of CuLane dataset.
NameTrainValidationTestResolutionRoad typeOther labelLaneCuLane888809675346801640 × 590Urban, rural, highway✓4


## 3.3. Experiment Configuration and Strategy

In this paper, the only experimental GPU is RTX2080 super. Due to the limited video memory of 8G, we compress the original image to 800 × 216 and set the length of each batch to 10. In the experiment, we randomly selected 80 K images as the training set and 9 K images as the validation set. Moreover, we take advantage of SGD [26] algorithm and CosineAnnealing [27] algorithm to change the learning rate and optimize the parameter update. In the experiment, the initial learning rate is set to 1e − 3, the weight decay is set to 1e − 4, and the annealing period is set to 10 to enhance overfitting resistance. In addition, we retain the momentum part of 0.9 and allow the Nesterov [28] algorithm to correct the gradient direction. Because there are many comparison experiments in our experiments, we only normalize images as shown in equation (5) and will not use the dataset augmentation strategy or batch random processing in order to ensure the consistency of learning samples.(5)data=data−meanstd.Mean is the expectation of the data set, and std is the standard deviation of the data set. For the CuLane dataset, the parameter values are as(6)mean=0.3598, 0.3653, 0.3662,std=0.2573, 0.2663, 0.2756.


## 3.4. Evaluation Metrics

Different from the evaluation metrics in SCNN [2], which used IoU (see equation (7)) to increase fault tolerance and F1-score (see equation (8)) to evaluate the performance, we choose to use Accuracy (see equation (9)) and Recall (see equation (10)) of each lane line to comprehensively evaluate the performance.(7)F1−score=2∗Recall∗PrecisionRecall+Precision,(8)IoU=TPTP+FP+FN,(9)Recall=TPTP+FN,(10)Accuracy=NpredNgt,(11)Precision=TPTP+FP,where TP represents the number of lane pixels accurately classified, FN represents the number of lane pixels wrongly judged as background, FP represents the number of background pixels wrongly judged as lane pixels, Npred represents the number of pixels correctly classified, and Ngt represents the total number of pixels. In the experiment, we find that the Precision is always much larger than the Recall. This is because the sparsity of the target leads to misjudge lane pixels as background pixels. This makes the Precision unable to accurately describe the classification ability of the network, so we do not use the F1-score, which contains the Precision. In addition, this paper will not add any fault-tolerant means.


## 4. Evaluation

### 4.1. Ablation Experiments

In this part, we mainly introduce the structure of the ARFNet and verify the effectiveness of the network through ablation experiments.As mentioned in RepVGG [29], single channel architecture has the advantages of saving video memory and good flexibility. At the same time, the calculation density of 3 × 3 convolution on GPU is better than other parameters. Therefore, this paper selects VGG network model as the baseline (the specific structure is shown in Table 2). In addition, a multilayer perceptron (MLP) is also added to construct a small classifier to predict the existence distribution of four lanes (see Figure 7).Table 2
ARFNet.
LayerTypeOutput-size1Conv216 × 800 × 642Maxpool108 × 400 × 643A-conv(r = 2)108 × 400 × 644Conv108 × 400 × 1285Maxpool54 × 200 × 1286A-conv(r = 2)54 × 200 × 1287Conv54 × 200 × 2568Maxpool27 × 100 × 2569A-conv(r = 2)27 × 100 × 25610Conv27 × 100 × 51211A-conv(r = 2)27 × 100 × 51212A-conv(r = 2)27 × 100 × 51213A-conv(r = 2)27 × 100 × 51214Conv27 × 100 × 12815Conv27 × 100 × 516Upsample216 × 800 × 5Figure 7
Lane existence prediction module.Figure8 shows the ARF-module we used in our experiment. In the following article, in order to distinguish it from normal convolution, the receptive field adaptive module will be marked as A-conv, r is the dilation rate, and C is the number of feature maps.Figure 8
A-conv.To verify the effectiveness of the module, we add Se-block to convolution in the normal VGG network as the control group. The Tables3 and 4is the comparison of the experimental result.Table 3
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepVGG-SE0.49160.7930.6630.6280.7640.7120.9188000ARFNet0.48170.8080.6790.6390.7750.7250.9168000Table 4
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepVGG-SE0.50070.7170.6140.5820.7030.6540.924900ARFNet0.45340.7040.7000.6880.7620.7130.914900We fixed the number of steps to ensure that the training degree of the model is consistent. Through the experiment, we find that although the two models show high Accuracy, their renderings (see Figure9) still fail to meet the expectations. This is because the visual judgment will be more affected if lane line pixels are misjudged as background. Based on this phenomenon, we give priority to the Recall and then consider the value of Accuracy.Figure 9
(a) Label. (b) ARFNet. (c) VGG-SENet.
(a)(b)(c)It can be seen from the data in the Table and the renderings that the normal network like VGG-SENet presents a large number of misjudgments for lane pixels, which makes the lane line broken obviously. However, ARFNet uses its information learning ability of different receptive fields to improve this phenomenon.In order to get better segmentation effect, we appropriately increase the number of feature maps in the module. Moreover, a weighted linear connector is added to enhance learning ability for aggregation. The modified ARF-module is shown as Figure10 and the result of ablation experiment is shown in Tables 5 and 6.Figure 10
Modified ARF-module.Table 5
Training results (the dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepWith linear connector0.45760.8230.6970.6690.7960.7460.9168000Without linear connector0.46630.8190.6950.6510.7880.7380.9168000Table 6
Test results (the dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepWith linear connector0.42750.8120.7550.7090.8330.7770.907900Without linear connector0.45220.7740.6310.6430.7960.7110.922900In our evaluation method, the module with linear connector is much better than the module with only increasing feature maps. In addition, we comparer = 1 and r = k (k is taken as 3 in this paper) to prove that the aggregation effect of different receptive fields is better than the aggregation of single receptive field. The result is shown in Tables 7 and 8. It is clear that, after different receptive field aggregation and adaptive process, the detection performance of each lane has been improved to a certain extent.Table 7
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepr = 10.48890.8010.6640.6240.7670.7140.9188000r = 30.45760.8230.6970.6690.7960.7460.9168000Table 8
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepr = 10.45860.7940.6170.6090.7710.6980.914900r = 30.42750.8120.7550.7090.8330.7770.907900


### 4.2. Adaptive Receptive Field Exploration Experiment

In this part, we make detailed experiments on the internal structure of ARF-module to find the best way to build it.


#### 4.2.1. The Way of Feature Aggregation

In addition to the direct concatenation in the channel direction, different receptive field features can also be aggregation by summation. According to the description of the theory part (see Figure3 and equation (2), the differences between the two are shown as (12)YConcatenation=αX1W1+X2W2,YSummation=αX1+X2W,where Y is the output of the module, a is the score tensor provided by SE-block, X is the feature under different receptive fields, and W represents kernels in the linear connector. This method can greatly reduce the number of parameters and improve the running speed as well. The ARF-module with summation is shown in Figure 11 and the result is shown in Tables 9 and 10.Figure 11
Summation module.Table 9
Training results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.46420.8200.6950.6540.7900.7400.9158000Concatenation0.45760.8230.6970.6690.7960.7460.9168000Table 10
Test results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.43670.7910.6970.6680.8200.7440.909900Concatenation0.42750.8120.7550.7090.8330.7770.907900Obviously, this result is consistent with the results described in the paper of ResNet [30, 31] and DenseNet [32] that, at the cost of weakening the performance, the summation module improves the speed of the network and reduces the amount of calculation.


#### 4.2.2. Optimization of Different Receptive Fields

Considering the factors of hardware and running speed, this paper will use the summation module in the optimal receptive field combination experiment. The receptive field combination in the summation module can be extended to(13)α∗xr1+β∗xr2+δ∗i,where α, β, δ represent the weight coefficients, r represents the dilation rate, i represents the input tensor, and x represents feature maps in the parallel structure. The experimental result is shown in Tables 11 and 12. In order to facilitate the analysis, we fix some parameters in the parallel structure, so that the first part does not occur dilation convolution, and we set all the coefficients to 1.Table 11
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.47420.8150.6850.6430.7800.7310.9168000x(1) + x(3)0.46420.8200.6950.6540.7900.7400.9158000x(1) + x(3)+i0.48690.8030.6780.6320.7680.7210.9158000x(1) + x(3)-i0.47400.8180.6830.6520.7830.7340.9148000x(1) + x(4)0.46090.8230.7020.6610.7970.7460.9138000Table 12
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.46230.7570.6730.6280.8370.7240.915900x(1) + x(3)0.43670.7910.6970.6680.8200.7440.909900x(1) + x(3)+i0.46210.7650.7200.5700.7550.7020.917900x(1) + x(3)-i0.44120.7810.6440.6520.7760.7130.913900x(1) + x(4)0.44360.7420.6580.6110.8100.7050.920900Through the experiment, we find that when the dilation rate exceeds 3, the Recall decreases obviously, but the Accuracy still increases. Therefore, we only use the result ofr = 4 to represent r > 3 and do not show other results of r > 4. The renderings are shown in Figure 12.Figure 12
(a) The result ofr = 2. (b) The result of r = 3. (c) The result of r = 4.
(a)(b)(c)It can be seen from the renderings that the resistance to interference information is only reflected in a certain appropriate range of dilation rate. That means despite the increase ofr, the number of lane line pixels correctly predicted has increased significantly the number of background pixels misjudged as lanes has increased as well. Hence, we choose r = 3, which has relatively better evaluation result and rendering, for further experiments (see Figure 13).Figure 13
(a) The result ofx (1) + x (3) + i. (b) The result of x (1) + x (3) − i. (c) Original data.
(a)(b)(c)Combined with the original data, we find that the double line of Lane 2 is not shown in the label (see Figure9(a)). In this case, the segmentation renderings can directly reflect the quality of the results. From the segmentation renderings, the effect of x (1) + x (3) − i is much smoother and clearer. However, there are obvious lane disappearance and distortion of straight line model in the prediction of vehicle occlusion. Compared with r = 3 and r = 4, the ARF-model with x (1) + x (3) − i has weaker generalization ability for lane lines when facing occlusion. Based on the above analysis, the combination of r = 3 and r = 1 is more suitable for CuLane dataset than other combinations.


### 4.3. Results

In this part, the optimized ARFNet is compared with current SOTA models.In the test, we choose the structure shown in Figure10 and set the dilation rate to 3 as the final version of ARFNet. Since there is no detailed public data of Recall and Accuracy in CuLane dataset, we will use a unified training strategy to train each model. Considering the limitation of hardware, we only test some lightweight SOTA models: ResNet-34 [30, 31], SCNN [2]. The experimental result is shown in Table 13 and the comparison of renderings with SCNN is shown in Figure 14 (the first graph is the original data, the second graph is the segmentation result of ARFNet, and the third graph is the segmentation result of SCNN).Table 13
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSCNN0.33910.8810.8200.7970.8630.8400.934900ARFNet0.37250.8430.7800.7530.8240.8000.941900ResNet-340.41500.834a0.7210.7510.8380.7860.919900Figure 14
(a) Result 1. (b) Result 2. (c) Result 3. (d) Result 4.
(a)(b)(c)(d)We selected four typical difficult scenarios for comparison. In result 1, the original data has four lane lines and are partially occluded by the tree shadow. In result 2, there are three clear lane lines, but the shooting angle is slightly different. In result 3, the original data has serious occlusion from people and vehicles. In result 4, the original data has obvious glass reflection, which causes a big trouble to predict the rightmost lane.From the experimental results, ARFNet has achieved relatively good results but still cannot exceed SCNN to reach the level of SOTA now. According to the comparative analysis of the renderings, we have found problems that ARFNet shows typical attention preference. Though it can well judge the obvious lane line pixels, it is extremely sensitive to occlusions. The network tends to distinguish the covering part as background and only segments those pixels with distinct lane characteristics, which reflects that the robustness and the reasoning ability of the network for global context are still defective. We will continue to update the network in the following experiments to improve the current deficiencies.Finally, we take 9 × 9 convolution kernel to check the segmentation map for smoothing filtering and add spline to select the appropriate prediction pixel points in order to bring better renderings (see Figure15).Figure 15
Lane prediction map postprocessing.


## 4.1. Ablation Experiments

In this part, we mainly introduce the structure of the ARFNet and verify the effectiveness of the network through ablation experiments.As mentioned in RepVGG [29], single channel architecture has the advantages of saving video memory and good flexibility. At the same time, the calculation density of 3 × 3 convolution on GPU is better than other parameters. Therefore, this paper selects VGG network model as the baseline (the specific structure is shown in Table 2). In addition, a multilayer perceptron (MLP) is also added to construct a small classifier to predict the existence distribution of four lanes (see Figure 7).Table 2
ARFNet.
LayerTypeOutput-size1Conv216 × 800 × 642Maxpool108 × 400 × 643A-conv(r = 2)108 × 400 × 644Conv108 × 400 × 1285Maxpool54 × 200 × 1286A-conv(r = 2)54 × 200 × 1287Conv54 × 200 × 2568Maxpool27 × 100 × 2569A-conv(r = 2)27 × 100 × 25610Conv27 × 100 × 51211A-conv(r = 2)27 × 100 × 51212A-conv(r = 2)27 × 100 × 51213A-conv(r = 2)27 × 100 × 51214Conv27 × 100 × 12815Conv27 × 100 × 516Upsample216 × 800 × 5Figure 7
Lane existence prediction module.Figure8 shows the ARF-module we used in our experiment. In the following article, in order to distinguish it from normal convolution, the receptive field adaptive module will be marked as A-conv, r is the dilation rate, and C is the number of feature maps.Figure 8
A-conv.To verify the effectiveness of the module, we add Se-block to convolution in the normal VGG network as the control group. The Tables3 and 4is the comparison of the experimental result.Table 3
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepVGG-SE0.49160.7930.6630.6280.7640.7120.9188000ARFNet0.48170.8080.6790.6390.7750.7250.9168000Table 4
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepVGG-SE0.50070.7170.6140.5820.7030.6540.924900ARFNet0.45340.7040.7000.6880.7620.7130.914900We fixed the number of steps to ensure that the training degree of the model is consistent. Through the experiment, we find that although the two models show high Accuracy, their renderings (see Figure9) still fail to meet the expectations. This is because the visual judgment will be more affected if lane line pixels are misjudged as background. Based on this phenomenon, we give priority to the Recall and then consider the value of Accuracy.Figure 9
(a) Label. (b) ARFNet. (c) VGG-SENet.
(a)(b)(c)It can be seen from the data in the Table and the renderings that the normal network like VGG-SENet presents a large number of misjudgments for lane pixels, which makes the lane line broken obviously. However, ARFNet uses its information learning ability of different receptive fields to improve this phenomenon.In order to get better segmentation effect, we appropriately increase the number of feature maps in the module. Moreover, a weighted linear connector is added to enhance learning ability for aggregation. The modified ARF-module is shown as Figure10 and the result of ablation experiment is shown in Tables 5 and 6.Figure 10
Modified ARF-module.Table 5
Training results (the dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepWith linear connector0.45760.8230.6970.6690.7960.7460.9168000Without linear connector0.46630.8190.6950.6510.7880.7380.9168000Table 6
Test results (the dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepWith linear connector0.42750.8120.7550.7090.8330.7770.907900Without linear connector0.45220.7740.6310.6430.7960.7110.922900In our evaluation method, the module with linear connector is much better than the module with only increasing feature maps. In addition, we comparer = 1 and r = k (k is taken as 3 in this paper) to prove that the aggregation effect of different receptive fields is better than the aggregation of single receptive field. The result is shown in Tables 7 and 8. It is clear that, after different receptive field aggregation and adaptive process, the detection performance of each lane has been improved to a certain extent.Table 7
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepr = 10.48890.8010.6640.6240.7670.7140.9188000r = 30.45760.8230.6970.6690.7960.7460.9168000Table 8
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepr = 10.45860.7940.6170.6090.7710.6980.914900r = 30.42750.8120.7550.7090.8330.7770.907900


## 4.2. Adaptive Receptive Field Exploration Experiment

In this part, we make detailed experiments on the internal structure of ARF-module to find the best way to build it.


### 4.2.1. The Way of Feature Aggregation

In addition to the direct concatenation in the channel direction, different receptive field features can also be aggregation by summation. According to the description of the theory part (see Figure3 and equation (2), the differences between the two are shown as (12)YConcatenation=αX1W1+X2W2,YSummation=αX1+X2W,where Y is the output of the module, a is the score tensor provided by SE-block, X is the feature under different receptive fields, and W represents kernels in the linear connector. This method can greatly reduce the number of parameters and improve the running speed as well. The ARF-module with summation is shown in Figure 11 and the result is shown in Tables 9 and 10.Figure 11
Summation module.Table 9
Training results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.46420.8200.6950.6540.7900.7400.9158000Concatenation0.45760.8230.6970.6690.7960.7460.9168000Table 10
Test results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.43670.7910.6970.6680.8200.7440.909900Concatenation0.42750.8120.7550.7090.8330.7770.907900Obviously, this result is consistent with the results described in the paper of ResNet [30, 31] and DenseNet [32] that, at the cost of weakening the performance, the summation module improves the speed of the network and reduces the amount of calculation.


### 4.2.2. Optimization of Different Receptive Fields

Considering the factors of hardware and running speed, this paper will use the summation module in the optimal receptive field combination experiment. The receptive field combination in the summation module can be extended to(13)α∗xr1+β∗xr2+δ∗i,where α, β, δ represent the weight coefficients, r represents the dilation rate, i represents the input tensor, and x represents feature maps in the parallel structure. The experimental result is shown in Tables 11 and 12. In order to facilitate the analysis, we fix some parameters in the parallel structure, so that the first part does not occur dilation convolution, and we set all the coefficients to 1.Table 11
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.47420.8150.6850.6430.7800.7310.9168000x(1) + x(3)0.46420.8200.6950.6540.7900.7400.9158000x(1) + x(3)+i0.48690.8030.6780.6320.7680.7210.9158000x(1) + x(3)-i0.47400.8180.6830.6520.7830.7340.9148000x(1) + x(4)0.46090.8230.7020.6610.7970.7460.9138000Table 12
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.46230.7570.6730.6280.8370.7240.915900x(1) + x(3)0.43670.7910.6970.6680.8200.7440.909900x(1) + x(3)+i0.46210.7650.7200.5700.7550.7020.917900x(1) + x(3)-i0.44120.7810.6440.6520.7760.7130.913900x(1) + x(4)0.44360.7420.6580.6110.8100.7050.920900Through the experiment, we find that when the dilation rate exceeds 3, the Recall decreases obviously, but the Accuracy still increases. Therefore, we only use the result ofr = 4 to represent r > 3 and do not show other results of r > 4. The renderings are shown in Figure 12.Figure 12
(a) The result ofr = 2. (b) The result of r = 3. (c) The result of r = 4.
(a)(b)(c)It can be seen from the renderings that the resistance to interference information is only reflected in a certain appropriate range of dilation rate. That means despite the increase ofr, the number of lane line pixels correctly predicted has increased significantly the number of background pixels misjudged as lanes has increased as well. Hence, we choose r = 3, which has relatively better evaluation result and rendering, for further experiments (see Figure 13).Figure 13
(a) The result ofx (1) + x (3) + i. (b) The result of x (1) + x (3) − i. (c) Original data.
(a)(b)(c)Combined with the original data, we find that the double line of Lane 2 is not shown in the label (see Figure9(a)). In this case, the segmentation renderings can directly reflect the quality of the results. From the segmentation renderings, the effect of x (1) + x (3) − i is much smoother and clearer. However, there are obvious lane disappearance and distortion of straight line model in the prediction of vehicle occlusion. Compared with r = 3 and r = 4, the ARF-model with x (1) + x (3) − i has weaker generalization ability for lane lines when facing occlusion. Based on the above analysis, the combination of r = 3 and r = 1 is more suitable for CuLane dataset than other combinations.


## 4.2.1. The Way of Feature Aggregation

In addition to the direct concatenation in the channel direction, different receptive field features can also be aggregation by summation. According to the description of the theory part (see Figure3 and equation (2), the differences between the two are shown as (12)YConcatenation=αX1W1+X2W2,YSummation=αX1+X2W,where Y is the output of the module, a is the score tensor provided by SE-block, X is the feature under different receptive fields, and W represents kernels in the linear connector. This method can greatly reduce the number of parameters and improve the running speed as well. The ARF-module with summation is shown in Figure 11 and the result is shown in Tables 9 and 10.Figure 11
Summation module.Table 9
Training results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.46420.8200.6950.6540.7900.7400.9158000Concatenation0.45760.8230.6970.6690.7960.7460.9168000Table 10
Test results (dilation rate is 3).
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSummation0.43670.7910.6970.6680.8200.7440.909900Concatenation0.42750.8120.7550.7090.8330.7770.907900Obviously, this result is consistent with the results described in the paper of ResNet [30, 31] and DenseNet [32] that, at the cost of weakening the performance, the summation module improves the speed of the network and reduces the amount of calculation.


## 4.2.2. Optimization of Different Receptive Fields

Considering the factors of hardware and running speed, this paper will use the summation module in the optimal receptive field combination experiment. The receptive field combination in the summation module can be extended to(13)α∗xr1+β∗xr2+δ∗i,where α, β, δ represent the weight coefficients, r represents the dilation rate, i represents the input tensor, and x represents feature maps in the parallel structure. The experimental result is shown in Tables 11 and 12. In order to facilitate the analysis, we fix some parameters in the parallel structure, so that the first part does not occur dilation convolution, and we set all the coefficients to 1.Table 11
Training results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.47420.8150.6850.6430.7800.7310.9168000x(1) + x(3)0.46420.8200.6950.6540.7900.7400.9158000x(1) + x(3)+i0.48690.8030.6780.6320.7680.7210.9158000x(1) + x(3)-i0.47400.8180.6830.6520.7830.7340.9148000x(1) + x(4)0.46090.8230.7020.6610.7970.7460.9138000Table 12
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepx(1) + x(2)0.46230.7570.6730.6280.8370.7240.915900x(1) + x(3)0.43670.7910.6970.6680.8200.7440.909900x(1) + x(3)+i0.46210.7650.7200.5700.7550.7020.917900x(1) + x(3)-i0.44120.7810.6440.6520.7760.7130.913900x(1) + x(4)0.44360.7420.6580.6110.8100.7050.920900Through the experiment, we find that when the dilation rate exceeds 3, the Recall decreases obviously, but the Accuracy still increases. Therefore, we only use the result ofr = 4 to represent r > 3 and do not show other results of r > 4. The renderings are shown in Figure 12.Figure 12
(a) The result ofr = 2. (b) The result of r = 3. (c) The result of r = 4.
(a)(b)(c)It can be seen from the renderings that the resistance to interference information is only reflected in a certain appropriate range of dilation rate. That means despite the increase ofr, the number of lane line pixels correctly predicted has increased significantly the number of background pixels misjudged as lanes has increased as well. Hence, we choose r = 3, which has relatively better evaluation result and rendering, for further experiments (see Figure 13).Figure 13
(a) The result ofx (1) + x (3) + i. (b) The result of x (1) + x (3) − i. (c) Original data.
(a)(b)(c)Combined with the original data, we find that the double line of Lane 2 is not shown in the label (see Figure9(a)). In this case, the segmentation renderings can directly reflect the quality of the results. From the segmentation renderings, the effect of x (1) + x (3) − i is much smoother and clearer. However, there are obvious lane disappearance and distortion of straight line model in the prediction of vehicle occlusion. Compared with r = 3 and r = 4, the ARF-model with x (1) + x (3) − i has weaker generalization ability for lane lines when facing occlusion. Based on the above analysis, the combination of r = 3 and r = 1 is more suitable for CuLane dataset than other combinations.


## 4.3. Results

In this part, the optimized ARFNet is compared with current SOTA models.In the test, we choose the structure shown in Figure10 and set the dilation rate to 3 as the final version of ARFNet. Since there is no detailed public data of Recall and Accuracy in CuLane dataset, we will use a unified training strategy to train each model. Considering the limitation of hardware, we only test some lightweight SOTA models: ResNet-34 [30, 31], SCNN [2]. The experimental result is shown in Table 13 and the comparison of renderings with SCNN is shown in Figure 14 (the first graph is the original data, the second graph is the segmentation result of ARFNet, and the third graph is the segmentation result of SCNN).Table 13
Test results.
NameLossLane 1 (recall)Lane 2 (recall)Lane 3 (recall)Lane 4 (recall)AverageAccuracyStepSCNN0.33910.8810.8200.7970.8630.8400.934900ARFNet0.37250.8430.7800.7530.8240.8000.941900ResNet-340.41500.834a0.7210.7510.8380.7860.919900Figure 14
(a) Result 1. (b) Result 2. (c) Result 3. (d) Result 4.
(a)(b)(c)(d)We selected four typical difficult scenarios for comparison. In result 1, the original data has four lane lines and are partially occluded by the tree shadow. In result 2, there are three clear lane lines, but the shooting angle is slightly different. In result 3, the original data has serious occlusion from people and vehicles. In result 4, the original data has obvious glass reflection, which causes a big trouble to predict the rightmost lane.From the experimental results, ARFNet has achieved relatively good results but still cannot exceed SCNN to reach the level of SOTA now. According to the comparative analysis of the renderings, we have found problems that ARFNet shows typical attention preference. Though it can well judge the obvious lane line pixels, it is extremely sensitive to occlusions. The network tends to distinguish the covering part as background and only segments those pixels with distinct lane characteristics, which reflects that the robustness and the reasoning ability of the network for global context are still defective. We will continue to update the network in the following experiments to improve the current deficiencies.Finally, we take 9 × 9 convolution kernel to check the segmentation map for smoothing filtering and add spline to select the appropriate prediction pixel points in order to bring better renderings (see Figure15).Figure 15
Lane prediction map postprocessing.


## 5. Conclusion

In this study, we propose ARF-module to realize the aggregation and adaptation of multiple receptive fields. Although it has achieved relatively good results in CuLane dataset by picking up information from different receptive fields, it cannot reach the level of SOTA now. This is mainly reflected in the lack of prediction generalization ability when dealing with large area occlusion. In other words, the model relies too much on the lane line pixels in the image, which weakens the ability to extend the existing pixels to the line or curve model. We will further optimize ARFNet to solve this problem and try to realize ARFNet mentioned in theory.



---

*Source: 6618459-2021-03-24.xml*