Title: A Siamese Transformer with Hierarchical Refinement for Lane Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Lane detection

Siamese Transformer

Hierarchical Refinement
Abstract: Lane detection is an important yet challenging task in autonomous driving systems. Existing lane detection methods mainly rely on finer-scale information to identify key points of lane lines. Since local information in realistic road environments is frequently obscured by other vehicles or affected by poor outdoor lighting conditions, these methods struggle with the regression of such key points. In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in complex road environments. Specifically, we propose a high-to-low hierarchical refinement Transformer structure, called LAne TRansformer (LATR), to refine the key points of lane lines, which integrates global semantics information and finer-scale features. Moreover, exploiting the thin and long characteristics of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines. Extensive experiments on three benchmark datasets of lane detection demonstrate that our proposed new method achieves state-of-the-art results with high accuracy and efficiency. Specifically, our method achieves improved F1 scores on the OpenLane dataset, surpassing the current best-performing method by 5.0 points.

A Siamese Transformer with Hierarchical Refinement for Lane Detection

Abstract

Lane detection is an important yet challenging task in autonomous driving systems. Existing lane detection methods mainly rely on finer-scale information to identify key points of lane lines. Since local information in realistic road environments is frequently obscured by other vehicles or affected by poor outdoor lighting conditions, these methods struggle with the regression of such key points. In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in complex road environments. Specifically, we propose a high-to-low hierarchical refinement Transformer structure, called LAne TRansformer (LATR), to refine the key points of lane lines, which integrates global semantics information and finer-scale features. Moreover, exploiting the thin and long characteristics of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines. Extensive experiments on three benchmark datasets of lane detection demonstrate that our proposed new method achieves state-of-the-art results with high accuracy and efficiency. Specifically, our method achieves improved F1 scores on the OpenLane dataset, surpassing the current best-performing method by 5.0 points.

1 Introduction

Lane detection is a fundamental task in Autonomous Driving Systems (ADS), which enables a vehicle to localize its relative position and avoid potential risks. It plays an important role in many downstream tasks, such as driving route planning, road tracking, and adaptive cruise control. Recently, lane detection methods based on computer vision have attained lots of achievements. Compared to methods that combine GPS/INS (Inertial Navigation System) [2], lane detection methods that incorporate only a camera are cheaper and safer to apply.

Early lane detection research focused on hand-crafted features and applied methods such as Hough Transform [16] or Kalman Filter [36] to filter out unreasonable lanes. However, manually extracted features often fail in complex scenarios. Convolutional Neural Network (CNN) based methods [10, 13, 38, 39] arose to cope with different scenarios, which greatly improved the accuracy of lane detection. These methods often rely on straight-line anchors [12, 15, 31, 32, 35] or parametric curves [18, 33, 9, 4] to detect lane lines. Although CNN-based methods can handle different scenarios, they still struggle with realistic road environments, especially when involving strong light, shadows, or dense traffic. Due to the thin and long characteristics of lane lines, lane detection requires a lot of contextual information, and CNN-based methods may be ineffective and incapable of clustering global semantics information and combining it with finer-scale features.

Recently, attention-based methods (e.g., CLRNet [41]) have shown promising capability in lane detection by describing a lane line as a series of key points. By taking the lane lines as a whole unit,

these methods can make use of global semantics information and finer-scale features, which help the detection of lane lines in complex environments. However, since these methods mainly rely on finer-scale information to identify the position of each key point, their detection results may have large deviations when there is local occlusion or blurring. In addition, former studies did not take into account the thin and long structure of lane lines during supervision, thus often leading to inaccurate detection when the lane line has a certain curvature.

In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Two attributes of our proposed method contribute to its universality for the lane detection task. First, to address the under utilization of finer-scale information when fine-tuning the positions of key points, we propose a novel Siamese Transformer structure with shared parameters, called LAne TRansformer (LATR), which can integrate global semantics information and finer-scale features. Simultaneously, we develop a high-to-low hierarchical refinement scheme to refine the key points of lane lines so that the network can fully learn information at different scales. Second, to take the thin and long structure of lane lines into account, we propose a novel Intersection over Union (IoU) loss called Curve-IoU (CIoU) for lane detection. Compared to the common IoU loss, we supervise the fit of lane lines at different locations for the thin and long structure of the lane lines, which helps accurately detect lane lines with curves.

We evaluate our proposed method on three benchmark datasets for lane detection, OpenLane [5], CULane [26], and Tusimple[34], and achieve state-of-the-art results. Our main contributions can be summarized as follows:

We propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.

We propose a high-to-low hierarchical refinement Transformer structure called LATR to refine the key points of lane lines so that the network can fully integrate global semantics information and finer-scale features.

Exploiting the thin and long structure of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines at different locations, which helps the regression of the curves.

We achieve state-of-the-art results on three benchmark datasets, with 5.0% improvement in F1 score compared to the best-known method on the OpenLane dataset.

2 Related Work

Early lane detection studies relied on hand-crafted features [7, 23]. Due to their limited feature capturing capability and low robustness, these methods often fail in complex conditions.

To improve the robustness of lane detection under different environments, segmentation-based methods [10, 13, 38, 39] were introduced to lane detection. These methods typically apply postprocessing operations such as curve fitting and clustering on pixel-level segmentation maps to generate final results. Compared to traditional methods, segmentation-based methods are able to capture more plentiful visual features and spatial structure information, thus achieving better performance than traditional detection methods. However, per-pixel-based segmentation methods incur high computational costs, have limited real-time capability, and struggle with learning lane line specific long and thin characteristics.

To address these issues, LaneNet [25] introduced a branched, multi-task architecture to cast the lane detection task as an instance segmentation problem. This method is more robust to variations in road conditions compared to the previous methods, but it is more time-consuming. RESA [40] proposed to aggregate spatial information by shifting sliced feature maps, which obtains good real-time results but still fails under complex road conditions. Furthermore, the output lane lines of most of the above methods may not be continuous.

To attain more continuous lane lines with higher efficiency, recently curve-based methods [18, 9, 4, 33] viewed the lane detection task as a polynomial regression problem and utilized parametric curves to fit lane lines. These methods depend heavily on the parameters of the curves (e.g., x = ay 3 + by 2 + cy + d , where ( x, y ) denotes the coordinates of a lane line pixel and a, b, c , and d are the parameters of a

LATR3

LATR2

LATR1

hierarchical features

Siamese Transformer

Curve FFN

Offset FFN

1

o

...

N

o

0

x

0

y



l

Curve-IoU

input image

lane anchor

ground truth

Swin level3

backbone

3

p

2

p

0

p

1

p

3

p

2

p

1

p

0

p

curve function). PloyLaneNet [33] first proposed an end-to-end deep polynomial regression method that directly outputs the parameters. To improve the stability and efficiency, BézierLaneNet [9] proposed a parametric Bézier curve to model the geometric shape of lane lines. However, because of the limited learning ability of global information, the accuracy of these curve-based methods is not satisfactory on large datasets, especially in complex road conditions.

Attention-based methods [32, 18] were introduced to the computer vision field, and were proven to be able to capture long-range information. LaneATT [32] introduced an attention mechanism to anchor-based lane detection methods. Based on PolyLaneNet [33], LSTR [18] was proposed with high inference efficiency but relatively low accuracy, especially in some complex road environments. PriorLane [30] improved the accuracy of prediction compared to LSTR with pre-training and local prior. However, there is still a gap in accuracy between the contemporaneous Transformer-based methods and CNN-based methods, and the reason can be attributed to the shortcut of the multi-head self-attention mechanism which neglects the characteristics of different frequencies [28].

3 Method

In this section, we present the proposed Siamese Transformer with hierarchical refinement for lane detection. First, we describe the overall architecture of our proposed network. Then we explain each key component of the proposed network, including the LAne TRansformer (LATR) and the proposed Curve-IoU loss. Finally, we provide the inference details of our network.

3.1 Overall Architecture

We present the overall architecture of our proposed method in Fig. 1. It consists of a multi-scale backbone and a Siamese Transformer structure named LATR. An input image R H × W × C is first fed into the backbone to obtain hierarchical features from high to low levels, which are then refined by LATR with the supervision of the Curve-IoU loss. Next, we use two different detection feedforward networks (FFN) to generate (1) lane line properties including the start point ( x 0 , y 0 ) , angle of inclination θ , and length l of each lane line, and (2) the offset map { o i } P i =1 . Finally, key points of the lane line are produced by post-processing, which can be expressed as:

where i denotes the i -th key point of the lane line. All the key points are sampled at equal intervals based on the y -axis, which can be expressed as y i = i H P +1 , where H and P represent the image height and the number of key points, respectively.

3.2 Lane Transformer

Transformer structure. Our proposed LATR employs the Siamese Transformer structure based on the hierarchical features extracted by the backbone of the network. Inspired by the recent Transformer-based networks with Siamese structures [1, 14], we develop a high-to-low refinement structure to obtain features of the predicted lane line. The detailed structure of LATR is illustrated in Fig. 2.

Specifically, we denote a lane query as Q d = { q j } N j =1 , where N represents the dimension of the decoder embeddings and d is the number of the scales. The lane query is a series of one-dimensional features of the lane lines, which can be further processed to obtain the key points of the lane lines. The input hierarchical features P d ∈ R H d × W d × C ′ are extracted by the multi-scale backbone (we use Swin Transformer [19] for it), and then are sorted from high-level to low-level. Next, we flatten it into M d ∈ R H d W d × C ′ with 2-D positional embedding.

Different from DETR [3], we integrate the lane query in the Transformer encoder and then employ Deformable Attention [42] as cross-attention in the decoder process. The whole process can be expressed as:

where Q ′ d and M ′ d represent the lane query and high-to-low hierarchical features after positional embeddings respectively, DA () denotes the Deformable Attention, and z d represents the sequence length.

Deformable Attention

Hierarchical

feature

from the

higher-scale

FFN

self-attention

k

q

v

Compared with DETR, our proposed Siamese Transformer integrates the lane query during the encoder. The multi-scale features from the backbone have different shapes. Note that integration during the encoder can unify the decoder's input scales with less information loss. When the scales are unified, the network can be made into a Siamese structure with shared parameters, which leads to a big efficiency improvement.

High-to-low refinement structure with shared parameters. Recent studies have shown that Vision Transformer (ViT) is good at extracting low-frequency information (in other words, global semantics information) from images [27]. However, lane lines are characteristically thin and long, and their accurate detection often requires integrating finer-scale features and global semantics information. High-level information can help determine the structural information of lane lines while low-level information can help adapt key points of the lane lines. Therefore, we adopt a high-to-low refinement structure to refine the lane query. Given hierarchical features P d ∈ R H d × W d × C ′ from the multi-scale backbone, we first flatten it into M d ∈ R H d W d × C ′ with 2-D positional embedding. Then we employ cross-attention to integrate M d with the lane query into S d ∈ R N × C ′ . Since S d shares the same shape, the same parameters can be used during refinement between multi-scales. By refining the lane query from high to low, the network can adapt the key points of the lane lines from coarse to fine, which helps the Transformer learn high-frequency information of the image. After the refinement, we employ a detection FFN to generate the lane anchor and finally attain the predicted lane lines with bi-linear interpolation between the key points.

3.3 Curve-IoU Loss

We represent a lane line as a sequence of key points, and we need a loss to supervise the regression of lane lines. In previous work, LIoU [41] was proposed as a similarity function to calculate the distance between predicted key points and the ground truth. Although the LIoU loss ensures the scale consistency of the lane shape, it does not match the L1 distance when there is a long distance between the prediction lane line and the ground truth. Fig. 3(a) shows a typical curve scenario that

Ground Truth

Prediction A

Prediction B

l

i

P

r

i

P

l

i

G

r

i

G

u

i

d

o

i

d

(a)

0

5

10

15

20

25



1.5

1.0

0.5

0.0

0.5

1.0















(b)

LIoU cannot describe correctly. In such cases, lane lines are inclined and curved, and the LIoU loss is unable to judge which prediction lane line is closer to the ground truth. In Fig. 3, the values for Prediction A and Prediction B are -0 . 6 and 0 . 6 , respectively. However, the L1 distances for them are 8 e and 1 . 5 e , respectively.

To bridge this gap, we propose Curve-IoU (CIoU). Different from the traditional IoU, we add a penalty term to IoU so that the spatial distance between each pair of lane lines will be represented more accurately, especially for those with a long distance between the prediction lane line and the ground truth. We use a sequence of points with a certain width e to present a lane line and calculate the Intersection over Union, which can be written as:

where P l i = x i -e and P r i = x i + e , and G l i and G r i are defined similarly. IoU can be calculated as IoU = d o i d u i . Then we add the L1 distance between each corresponding pair of lines as the penalty term to the overlap ranges, as:

where j is the first valid point in the lane line. Then the CIoU loss can be calculated as CIoU = 1 -L IoU .

Exploiting the thin and long structures of lane lines, our CIoU loss can supervise the regression of lane lines better. As shown in Fig. 3(b), the values of LIoU and CIoU for Prediction A are -0 . 6 and -1 . 6 respectively, which means that our CIoU can handle these scenarios more precisely.

3.4 Inference Details

In order to reduce the training costs, we generate only the lane anchor during training. The loss is calculated by comparing the lane anchor to the ground truth. Our method generates a fixed number of lane anchors, which is much larger than the maximum number of lane lines in the image. Besides the Curve-IoU loss, we also utilize a classification accuracy loss to determine whether the lane anchor is a lane line or the background. The classification accuracy loss can be described as:

where L ( · , · ) denotes the cross-entropy loss and L is the number of sequences. If the i -th predicted lane line matches the ground truth, then g i = 1 ( 0 otherwise). The objective function of our network can be represented as:

where λ cls and λ reg are importance weight values. We supervise each generated lane anchor during training, while during inference we use Non-Maximum Suppression (NMS) [24] to exclude incorrect lane lines. Finally, we utilize bi-linear interpolation to produce the predicted lane lines of the image.

4 Experiments

4.1 Datasets

To demonstrate the effectiveness of our proposed method in realistic road environments, we evaluate it on three benchmark datasets: OpenLane [5], CULane [26], and Tusimple [34]. OpenLane is a real-world large-scale lane detection dataset, which contains 160K and 40K images as the training and validation sets, respectively. The validation set consists of six realistic road scenarios and annotates 14 lane categories (including white dotted line, double yellow solid, left/right curb, and so on). CULane is a widely-used large dataset for lane detection including eight hard-to-detect scenarios in urban areas and on highways, with 88K and 34K images as the training and validation sets, respectively. Tusimple is also a widely-used dataset with images collected on US highways under clear weather, which contains 3K images for training and 2K for validation.

4.2 Evaluation Metrics

For OpenLane [5] and CULane [26], we adopt the F1-score measure proposed by SCNN [26] as the evaluation metric. Intersection-over-Union (IoU) between the ground truth (GT) label and each predicted lane line of the model is calculated to determine whether a sample is True Positive (TP), False Positive (FP), or False Negative (FN). The ways to calculate IoU and F1-score can be found in the supplemental material. For Tusimple [34], the evaluation metrics are composed of three official indicators: Accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). The way to calculate Accuracy can be found in the supplemental material.

4.3 Implementation Details

In the experiments, we adopt the Swin Transformer [19] as the pre-trained backbone of our network. We divide the versions of the backbone of the network based on the size of the Swin Transformer into three categories: tiny, small, and big, which are consistent with the work in [19]. For data augmentation, we adopt the affine transformation method (horizontal flip, rotation), brightness, and saturation addition method. All the input images are reshaped into 800 × 320 pixels each for both the training and inference stages. For the number of lane anchors in an image, we set it to 150. In the optimization process, we adopt AdamW [21] and the cosine decay learning rate strategy [20] with the initial learning rate set to 6e-4. A batch size of 32 and training epoch numbers of 10, 20, and 90 are used for OpenLane, CULane, and Tusimple, respectively. All the experiments are conducted on a machine with a single NVIDIA RTX3090 GPU with 24GB memory.

4.4 Experimental Results

4.4.1 Results on OpenLane

Comparisons of results by recent methods and our method on the OpenLane dataset are shown in Table 1. Our method achieves state-of-the-art results in F1 score. Specifically, our method achieves F1 scores of 68.3, 69.1, and 69.7, surpassing those of the best-known method CondLaneNet by 5.0, 3.5, and 1.9 points, respectively. Further, our method achieves the best performances in four out of six scenarios, showing the robustness of our method. Among these, the 'Up & Down' and 'Curve' categories are 7.2% and 1.1% higher than the previous best results, respectively. These results demonstrate that the proposed Siamese Transformer with the hierarchical refinement structure can deal with lane lines in realistic road environments very well. This is because our proposed LATR integrates global semantics information and finer-scale features, which help refine key points when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Simultaneously, the proposed Curve-IoU can supervise the regression of curves well, which helps accurate detection of lane lines.

4.4.2 Results on CULane

We compare the results by recent known methods and our method on the CULane dataset in Table 2. In the total F1 score, our method achieves state-of-the-art results, with 0.38% improvement over the best-known CLRNet. Among all the eight difficult scenarios, our method achieves the best results on five of them. Specifically, for the 'Crowded" and 'Hlight" scenarios, our method achieves

GT

CondLane

CLRNet

Ours

93.86

71.05

85.51

69.15

92.18

84.36

61.47

65.26

90.01

68.46

64.98

91.58

80.21% and 76.04% F1 scores respectively, surpassing those of CLRNet by 0.62% and 0.74% points, respectively. These results demonstrate that our method can deal with realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.

Fig. 5 shows some visual results of several known methods and our method on the CULane dataset. CondLaneNet is an open-source dynamic convolution-based method and CLRNet is the second best-performing method on CULane. The results of CondLaneNet and CLRNet are generated with ResNet18 as the backbone and ours are generated with Swin Transformer tiny as the backbone. The visual results show that when the roads are crowded and lane lines are covered by other vehicles, our method can detect lane lines better and the lane regression is closer to the ground truth. What's more, our method also outperforms these methods in night scenarios, proving that our method can adapt to different lighting environments with good robustness.

4.4.3 Results on Tusimple

Comparison results of recent methods and our method on the Tusimple dataset are given in Table 3. This dataset consists of images captured in different weather conditions. Our method achieves state-of-the-art results in F1 score, Accuracy, and False Positive Rate (FP), demonstrating that our method can be adapted to both complex urban environments and simple highway scenarios. Note that because the dataset was collected on US highways where the road conditions are relatively simple (lane features are more obvious and clear), the results of various methods are close.

4.5 Ablation Studies

4.5.1 LAne TRansformer (LATR)

To fully integrate global semantics information and finer-scale features, we propose a high-to-low hierarchical refinement Transformer structure for lane detection called LAne TRansformer (LATR), which helps identify key points especially when roads are crowded or affected by blurring. Fig. 4 shows some high-to-low attention maps of our LATR. From Fig. 4, one can see that attention is gradually focused on key points on both sides of the road from high to low. High-level attention is extended along the road from the near end to the far end of the road, initially defin-

level3

level2

ing the overall structure of the lane lines. Low-level attention focuses on finer-scale features of the lane lines, which refine the key points of the lane lines from higher-levels to lower-levels.

We compare the F1 scores of our Lane Transformer (LATR) and existing Transformer structures with the same backbone Swin Transformer tiny on CULane, and report the results in Table 4. Compared

with the existing Transformer structures, our method has a big advantage of extracting features at multi-scales, which play an important role in the process of lane detection. Compared with the Deformable Transformer, our method improves the detection accuracy by 4.36%. Further, as shown in Table 4, lane features cannot be fully obtained by relying only on high-level features p 0 or lowlevel features p 3 . The Siamese refinement structure from high to low can better integrate the global semantics information and finer-scale features, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.

Table 4: Ablation study results of the Lane Transformer on the CULane dataset with the same backbone (Swin Transformer tiny).

4.5.2 Curve-IoU Loss

To help the regression of lane lines, we propose the Curve-IoU loss. In Table 5, we compare our Curve-IoU with different types of IoU for lane detection. 'w/o IoU' denotes optimizing the network with no IoU loss, using only classification and L1 regression loss, which only yields an F1 score of 77.16%. LIoU was proposed by CLRNet as a similarity function to calculate the distance between the prediction and ground truth. However, LIoU does not match the L1 distance when there is a long distance between the prediction lane line and ground truth. Compared with LIoU, our Curve-IoU takes the L1 distance into account, which helps the regression of lane lines with curves. On the CULane dataset, Curve-IoU achieves an F1 score of 80.01%, which is 2.85% higher than w/o IoU and 0.77% higher than LIoU. Its effect on Transformer-based lane detection is validated.

5 Discussion

Our method achieves state-of-the-art results on three benchmark datasets and performs the best among all the Transformer-based methods, especially in some challenging and complex road conditions. However, we still find that the current known anchor-based methods are competitive in the 'Shadow", 'Noline", and 'Arrow" scenes on the CULane dataset, which is the opposite of the OpenLane dataset. Also, the improvement of our method is larger on the OpenLane dataset. For this interesting phenomenon, we assmue the reason behind it is that the CULane dataset contains 88K images for training while the Openlane dataset contains 160K, which is almost twice that of CULane and includes more complex and various scenes. Our Transformer-based method performs relatively better on a larger dataset compared to anchor-based and CNN-based methods, that is, our method has a much bigger potential for scaling up if the data and computation resources are sufficient. Considering the current state of affairs in vision-based autonomous driving systems, deep learning algorithms are advancing and models are expanding, yet the systems are still constrained by the size and diversity of datasets. We call for larger and better datasets featuring more diverse scenes for further research.

6 Conclusions

In this paper, we proposed a novel Siamese Transformer structure with hierarchical refinement, which achieves state-of-the-art results on three benchmark datasets. Specifically, we developed a high-to-low hierarchical refinement Transformer structure called LATR to refine key points of lane lines, which compensates for the Vision Transformer's deficiency in extracting finer-scale features. Also, we proposed a novel Curve-IoU loss tailored for the long and thin shape of lane lines, which helps supervise the regression of lane lines with different offsets. Extensive experiments confirmed that our model effectively handles complex scenarios, particularly when lane lines are heavily obscured by other vehicles or compromised by poor lighting conditions.

Table 5: Effect of our proposed Curve-IoU on OpenLane and CULane. 'w/o IoU' denotes optimizing with no IoU loss.

References

Wele Gedara Chaminda Bandara and Vishal M Patel. A Transformer-based Siamese network for change detection. In IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium , pages 207-210. IEEE, 2022.

Kenneth R Britting. Inertial navigation systems analysis . 2010.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, et al. End-to-end object detection with Transformers. In Computer Vision-ECCV 2020: 16th European Conference, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

Haoxin Chen, Mengmeng Wang, and Yong Liu. BSNet: Lane detection via draw B-spline curves nearby. arXiv preprint arXiv:2301.06910 , 2023.

Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. PersFormer: 3D lane detection via Perspective Transformer and the openlane benchmark. In European Conference on Computer Vision , pages 550-567. Springer, 2022.

Ziye Chen, Yu Liu, Mingming Gong, Bo Du, Guoqi Qian, and Kate Smith-Miles. Generating dynamic kernels via transformers for lane detection. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6812-6821, 2023.

P.M. Daigavane and P.R. Bajaj. Road lane detection with improved Canny edges using ant colony optimization. In 2010 3rd International Conference on Emerging Trends in Engineering and Technology , pages 76-80, 2010.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Weissenborn, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, et al. Rethinking efficient lane detection via curve modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17062-17070, 2022.

Mohsen Ghafoorian, Cedric Nugteren, Nóra Baka, Olaf Booij, and Michael Hofmann. EL-GAN: Embedding loss driven generative adversarial networks for lane detection. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pages 0-0, 2018.

Jianhua Han, Xiajun Deng, Xinyue Cai, Zhen Yang, Hang Xu, Chunjing Xu, and Xiaodan Liang. Laneformer: Object-aware row-column Transformers for lane detection. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 799-807, 2022.

Yuenan Hou, Zheng Ma, Chunxiao Liu, Tak-Wai Hui, and Chen Change Loy. Inter-region affinity distillation for road marking segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12486-12495, 2020.

Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection CNNs by self attention distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1013-1021, 2019.

Le Hui, Lingpeng Wang, Linghua Tang, Kaihao Lan, Jin Xie, and Jian Yang. 3D Siamese Transformer network for single object tracking on point clouds. In European Conference on Computer Vision , pages 293-310. Springer, 2022.

Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-CNN: End-to-end traffic line detection with line proposal unit. IEEE Transactions on Intelligent Transportation Systems , 21(1):248-258, 2019.

Guoliang Liu, Florentin Wörgötter, and Irene Markeli´ c. Combining statistical Hough transform and particle filter for robust lane detection and tracking. In 2010 IEEE Intelligent Vehicles Symposium , pages 993-997. IEEE, 2010.

Lizhe Liu, Xiaohao Chen, Siyu Zhu, and Ping Tan. CondLaneNet: A top-to-down lane detection framework based on conditional convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3773-3782, 2021.

Ruijin Liu, Zejian Yuan, Tie Liu, and Zhiliang Xiong. End-to-end lane shape prediction with Transformers. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3694-3702, 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Zinan Lv, Dong Han, Wenzhe Wang, and Cheng Chen. IFPNet: Integrated feature pyramid network with fusion factor for lane detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1888-1897, 2023.

Hiren M. Mandalia and Mandalia Dario D. Salvucci. Using support vector machines for lanechange detection. Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 49:1965 - 1969, 2005.

Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th International Conference on Pattern Recognition (ICPR'06) , volume 3, pages 850-855. IEEE, 2006.

Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Towards end-to-end lane detection: An instance segmentation approach. In 2018 IEEE Intelligent Vehicles Symposium (IV) , pages 286-291. IEEE, 2018.

Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial CNN for traffic scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast Vision Transformers with Hilo attention. Advances in Neural Information Processing Systems , 35:14541-14554, 2022.

Namuk Park and Songkuk Kim. How do Vision Transformers work? In International Conference on Learning Representations , 2022.

Zequn Qin, Huanyu Wang, and Xi Li. Ultra fast structure-aware deep lane detection. In Computer Vision-ECCV 2020: 16th European Conference, 2020, Proceedings, Part XXIV 16 , pages 276-291. Springer, 2020.

Qibo Qiu, Haiming Gao, Wei Hua, Gang Huang, and Xiaofei He. PriorLane: A prior knowledge enhanced lane detection approach based on Transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5618-5624. IEEE, 2023.

Jinming Su, Chao Chen, Ke Zhang, Junfeng Luo, Xiaoming Wei, et al. Structure guided lane detection. arXiv preprint arXiv:2105.05403 , 2021.

Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, et al. Keep your eyes on the lane: Real-time attention-guided lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 294-302, 2021.

Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, et al. PolyLaneNet: Lane estimation via deep polynomial regression. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 6150-6156. IEEE, 2021.

Tusimple. TuSimple benchmark, 2020. https://github.com/TuSimple/ tusimple-benchmark/, Accessed September, 2020.

Jinsheng Wang, Yinchao Ma, Shaofei Huang, Tianrui Hui, Fei Wang, et al. A keypoint-based global association network for lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1392-1401, 2022.

Pei-Chen Wu, Chin-Yu Chang, and Chang Hong Lin. Lane-mark extraction for automobiles under complex conditions. Pattern Recognition , 47(8):2756-2767, 2014.

Lingyu Xiao, Xiang Li, Sen Yang, and Wankou Yang. ADNet: Lane shape prediction via anchor decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6404-6413, 2023.

Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang, Xiaodan Liang, et al. CurveLane-NAS: Unifying lane-sensitive architecture search and adaptive point blending. In Computer VisionECCV 2020: 16th European Conference, 2020, Proceedings, Part XV 16 , pages 689-704. Springer, 2020.

Seungwoo Yoo, Hee Seok Lee, Heesoo Myeong, Sungrack Yun, Hyoungwoo Park, Janghoon Cho, et al. End-to-end lane marker detection via row-wise classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 1006-1007, 2020.

Tu Zheng, Hao Fang, Yi Zhang, Wenjian Tang, Zheng Yang, et al. RESA: Recurrent feature-shift aggregator for lane detection. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3547-3554, 2021.

Tu Zheng, Yifei Huang, Yang Liu, Wenjian Tang, Zheng Yang, et al. CLRNet: Cross layer refinement network for lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 898-907, 2022.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

A Evaluation Metrics

For OpenLane [5] and CULane [26], we adopt the F1 score proposed by SCNN [26] as the evaluation metric. Intersection-over-Union (IoU) between the ground truth (GT) and the predicted lane line of the model is calculated to determine whether a sample is True Positive (TP), False Positive (FP), or False Negative (FN). The IoU and F1 score are calculated as in the following formulas:

For Tusimple [34], the evaluation metrics are composed of three official indicators: accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). The accuracy is calculated as:

where C clip is the number of correct points and S clip is the number of ground truth (GT) points in an input image. If the accuracy of a predicted lane is greater than 85%, it will be considered a True Positive (TP). F1 score is also used in the evaluation.

B More Ablation Studies

B.1 Overall Ablation Study

To verify the roles of our proposed lane Transformer (LATR) and Curve-IoU in our method, we carry out an overall ablation study with the same baseline LSTR [18]. The results of the ablation study are shown in Table 6. LATR can greatly improve the detection accuracy of lane lines, which improves the F1 scores with 8.38% and 6.45% increase on OpenLane and CULane, respectively. This is because our proposed LATR can better integrate global semantics information and finer-scale features, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. What's more, our proposed Curve-IoU further improves the F1 scores by 1.23% and 1.66% on OpenLane and CULane, respectively. These results show that our proposed Curve-IoU can improve the accuracy of lane detection, especially for roads with curves.

Table 6: Results of the overall ablation study with the same backbone. We conduct the overall ablation study based on the same baseline LSTR [18].

B.2 Number of Lane Anchors

Our method generates a fixed number of lane anchors, much more than the maximum number of lane lines in the image. Here we conduct an ablation study on the number of lane anchors. As shown in Table 7, when we increase the number of lane anchors from 50 to 150, the accuracy of lane detection on both OpenLane and CULane is improved. This demonstrates that increasing the number of lane anchors is beneficial to the detection of lane lines. However, when we further increase the number of

Table 7: Results of the ablation study on the number of lane anchors. 'FPS" denotes the FPS on the CULane dataset.

CondLane

CLRNet

Our method

GT

四通物设

四通物液

四通物流

010-8411

010-84313

010-841130

010-841130

lane anchors, the enhancement is not obvious or even appears to be decreasing. This is because when the number of lane anchors reaches a certain value, adding more lane anchors will cause redundancy. 150 lane anchors are enough for the network to detect lane lines in the image.

B.3 More Studies on Backbone

In order to evaluate the bias caused by the backbone net, we conduct ablation experiments on backbones. We find that the Swin Transformer with hierarchical features can better extract image features for subsequent processing. Therefore, we choose the Swin Transformer as our backbone network, which is an open-source backbone widely used by other works. The results show that our approach achieves better results with higher FPS and lower GFlops. We also replace the backbone of previous methods (e.g., CLRNet) with Swin Transformer and train these models under equal situations. The results are shown below in Table 8.

B.4 Scaling up

We only use a single LATR module for each pyramid level, to find out the potential scaling-up ability of our model, we add more layers to each level, and the results are shown in Table 9 and 10. It can be seen from the results that more LATR added at the lower feature levels can improve performance, while not leading to much increase in parameters. As a result, we think it is effective to add two LATR modules at both the lowest and second-lowest feature levels.

Table 9: Performance with Different Numbers of LATR Modules.

C More Visualization Results

More visualization results of Ground Truth (GT), CondLaneNet [17], CLRNet [41], and our proposed method on the CULane dataset [26] are shown in Fig. 6. Compared with CondLaneNet and CLRNet, our method is more effective in detecting lane lines on congested roads and in different lighting environments, which demonstrates the robustness of our method.

Table 10: Impact of Adding 2 LATR Modules to lowest and second lowest feature levels

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly illustrate our proposed model LATR, and precisely show the experiment results we have achieved.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The paper thoughtfully discusses the limitations of the research in Section 5. We evaluate the robustness of our proposed method on three datasets in varying scenarios, and before that, we make assumptions and declare the inference details thoroughly in Section 3.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not propose theoretical results in this paper

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We illustrate our experiment details in our paper, and we will release our code and checkpoint, ensuring our work's reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code and checkpoint of this paper, and all the datasets we use are open access. The appendix includes extensive experiments to prove our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details are all illustrated thoroughly in Section 4, and our code will release.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We display the experiment results in Section 4 which can explain the variance, outliers, and anomalies, and we illustrate all the evaluation metrics we use in Section 4.2 and A

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provides detailed descriptions of the computational resources used for each experiment in Section 4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly examined the ethical guidelines and legal considerations, and confirm that all study aspects obey the rules.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We comprehensively discuss the broader societal impacts of the research in this paper, and no significant negative impacts are found.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not pose any such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper cites the original creators of all utilized assets, including code and datasets. The licensing information of our code is clearly outlined, ensuring full compliance with legal and ethical standards.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper documents all newly introduced assets, including code and datasets, and we have ensured that all assets are appropriately anonymized in line with submission guidelines.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Siamese Transformer with hierarchical refinement, named LAne TRansformer (LATR), to enhance lane detection accuracy in complex road environments. LATR integrates global semantic information with finer-scale features using a high-to-low hierarchical refinement structure. Additionally, the paper introduces a Curve-IoU loss to better supervise the fitting of lane lines, leveraging their thin and long characteristics.\n\nSTRENGTHS:\n1) The proposed method achieves relatively high performance on Openlane dataset.\n2) The proposed Curve-IoU is effective in performance improvement.\n\nWEAKNESSES:\n1) The motivation of this paper is not very clear. Moreover, the hierarchical architecture is not a particularly innovative idea, as it has been demonstrated in lane detection (e.g., CLRNet), image segmentation (e.g., Mask2Former), and even earlier works like FPN. Additionally, it is unclear why the authors refer to several shared-parameter transformer layers as a Siamese transformer, as this seems to be merely a case of parameter reuse.\n2) The proposed high-to-low transformer in this paper is very similar to Mask2Former, thus offering limited novelty.\n3) The paper conducts a quantitative comparison with CondLSTR on the CULane dataset.\n4) The paper uses a stronger image backbone (Swin vs. ResNet) compared to previous methods, making the comparison unfair. To provide a fair comparison, the authors should replace the backbone of previous methods (e.g., CLRNet) with Swin and then compare the results.\n5) The writing quality of the paper is poor. The structure of the methodology section is disorganized, making it very difficult to follow.\n\nQUESTIONS:\nMy most concerned questions are about the rationality of the motivation and novelty of the proposed method. Besides, the writing of this paper should be thoroughly improved.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a lane detection method based on transformers and utilises a hierarchical refinement of lane queries. The paper uses a high-to-low refinement strategy instead of the traditional low-to-high refinement, which saves the computation cost in transformer attention. The evaluation of three datasets shows consistent but marginal accuracy improvements while delivering a higher speed.\n\nSTRENGTHS:\n1. The hierarchical design for the application of lane detection is commendable. Ignoring the application and refining queries only from a decoder instead of the encoder-decoder mechanism of DETR is very interesting.\n\n2. The method's design seems inspired by DETR pipelines, and utilizing this idea of lane detection is interesting.\n\n3. The proposed Curve-IoU is again a robust objective and metric to supervise the training when the existing L-IoU fails to differentiate the proximity of two lanes w.r.t. the ground truth.\n\n4. Evaluation is extensive across three datasets and a variety of baselines.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\nCurrently, only one LATR module is used for each pyramid level. It is advisable to add more such modules to assess the performance, i.e. whether more LATR modules are redundant or improve the performance. \n\nSuggestion: I believe adding more LATR modules to high-resolution would quadratically increase the computations due to the O(N^2) complexity of the self-attention and cross-attention. Therefore, multiple LATR modules can be used at the lowest or second lowest feature level. It would be great to see the results during the rebuttal.\n\nLIMITATIONS:\nThe primary limitation of the paper is marginal improvements in the CULane dataset. Although this is shadowed by the improved FPS, considering the importance of the results, it is advisable to tune hyperparameters or add more decoder layers to deliver a stand-out performance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper introduces a novel Siamese Transformer with hierarchical refinement for lane detection. The core innovation is a high-to-low hierarchical refinement Transformer structure, LATR, which refines lane line key points to integrate global semantic information and finer-scale features fully. Additionally, it proposes a Curve-IoU loss to supervise the fitting of lane lines at various locations.

STRENGTHS:
- The paper is well-written and well-organized.
- The experiments use a comprehensive set of datasets, including three popular ones: TuSimple, CULane, and OpenLane.
- The experimental results are quite promising.

WEAKNESSES:
From the perspective of the topic selection, 2D lane detection is less valuable than 3D lane detection. More importantly, this topic is not fully suitable for the NeurIPS conference.

Compared to existing 2D lane detection networks, this paper lacks innovation. Its overall framework is quite similar to CLRNet, and the proposed structured Loss has also been preliminarily explored in CLRNet and UFAST networks.

More importantly, current lane detection research tends to use simple CNN-based backbones (for ease of deployment) and to better compare performance fairly, especially using the ResNet series (18, 34, etc.). However, the authors only reported results under the Swin Transformer, which makes me skeptical about the fairness of the SOTA comparison.

Additionally, in terms of efficiency, the FPS speed does not have an advantage, and the model size (params) is not reported.

QUESTIONS:
- Suggest enhancing the method's innovativeness.
- Recommend using common public backbones for ablation studies, though this might make it more similar to CLRNet.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LATR (LAne TRansformer), a Siamese Transformer model with hierarchical refinement for lane detection. The model effectively combines global semantic information with finer-scale features to accurately detect lanes, even in challenging scenarios like occlusions and poor lighting. It also introduces a novel Curve-IoU loss function specifically tailored for the curved nature of lane lines. The proposed method demonstrates state-of-the-art performance on multiple benchmark datasets, particularly excelling on the OpenLane dataset.\n\nSTRENGTHS:\nThe Siamese Transformer architecture with hierarchical refinement and the Curve-IoU loss are novel contributions to the field. The proposed method achieves state-of-the-art results on multiple benchmark datasets, demonstrating its superior performance in challenging scenarios.\nThe model's ability to handle occlusions and poor lighting conditions highlights its robustness and potential for real-world applications.\nThe authors conduct extensive experiments and provide a comprehensive analysis, including ablations and comparisons to prior work. Overall, a well written paper.\n\nWEAKNESSES:\nThe authors acknowledge that the model's performance is somewhat dependent on the size and diversity of the training dataset. This is a common limitation in deep learning-based approaches and could be addressed with further research on data augmentation or self-supervised learning techniques. \n\nWhile the method excels in the tested scenarios, further testing in diverse real-world environments would strengthen the claims of robustness. However, this is not a major weakness as the current evaluation results are already very promising.\n\nQUESTIONS:\nHave you considered using self-supervised learning techniques to reduce the dependency on large labeled datasets?\nHow does the model perform in extreme weather conditions (e.g., heavy rain, snow) that can significantly affect lane visibility?\nAre there any plans to release the code and trained models to facilitate reproducibility and further research?\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work, acknowledging the data dependency issue and the need for further testing in diverse real-world environments. However, in this field of work, there are a lot of other works that have various implementations exceeding the results from CondLaneNet such as ERF-Net. Looking into that might be of interest to better compare the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}