A Siamese Transformer with Hierarchical Refinement for Lane Detection
Authors: Zinan Lv
Dong Han
Wenzhe Wang
Danny Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Lane detection
Siamese Transformer
Hierarchical Refinement
Primary area: machine_vision

Abstract: Lane detection is an important yet challenging task in autonomous driving systems. Existing lane detection methods mainly rely on finer-scale information to identify key points of lane lines. Since local information in realistic road environments is frequently obscured by other vehicles or affected by poor outdoor lighting conditions, these methods struggle with the regression of such key points. In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in complex road environments. Specifically, we propose a high-to-low hierarchical refinement Transformer structure, called LAne TRansformer (LATR), to refine the key points of lane lines, which integrates global semantics information and finer-scale features. Moreover, exploiting the thin and long characteristics of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines. Extensive experiments on three benchmark datasets of lane detection demonstrate that our proposed new method achieves state-of-the-art results with high accuracy and efficiency. Specifically, our method achieves improved F1 scores on the OpenLane dataset, surpassing the current best-performing method by 5.0 points.

A Siamese Transformer with Hierarchical Refinement for Lane Detection
Zinan Lv
Dong Han Wenzhe Wang Danny Z. Chen
'Shanghai JiaoTong University 2Zhejiang University 3 University of Notre Dame
Abstract
Lane detection is an important yet challenging task in autonomous driving systems. Existing lane detection methods mainly rely on finer-scale information to identify key points of lane lines. Since local information in realistic road environments is frequently obscured by other vehicles or affected by poor outdoor lighting condi- tions, these methods struggle with the regression of such key points. In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection improve the detection accuracy in complex road environments. Specif- ically, we propose a high-to-low hicrarchical refinement Transformer structure, called LAne TRansformer (LATR), to refine the key points of lane lines, which integrates global semantics information and finer-scale features. Moreover, exploit- ing the thin and long characteristics of lane lines, we propose novel Curve-IoU loss to supervise the fit of lane lines. Extensive experiments on three benchmark datasets of lane detection demonstrate that our proposed new method achieves state-of-the-art results with high accuracy and efficiency. Specifically, our method achieves improved scores on the Openlane dataset, surpassing the current best-performing method by 5.0 points.
Introduction
Lane detection fundamental task in Autonomous Driving Systems (ADS), which enables vehicle to localize its relative position and avoid potential risks. It plays an important role in many downstream tasks, such as driving route planning, road tracking, and adaptive cruise control. Recently, lane detection methods based on computer vision have attained lots of achievements. Compared to methods that combine GPS/INS (Inertial Navigation System) (2] lane detection methods that incorporate only camera cheaper and safer to apply.
Early lane detection research focused on hand-crafted features and applied methods such as Hough Transform [6] or Kalman Filter (36] to filter out unreasonable lanes. However, manually extracted features often fail in complex scenarios. Convolutional Neural Network (CNN) based methods 39 arose to cope with different scenarios, which greatly improved the accuracy of lane
[10 13 detection. These methods often rely on straight-line anchors [12 15
38
32 35 or parametric curves to detect lane lines. Although CNN-based methods can handle different scenarios, they
31
18 33 9 still struggle with realistic road environments, especially when invol strong light, shadows, or dense traffic. Due to the thin and long characteristics of lane lines, lane detection requires lot of contextual information, and CNN-based methods may be ineffective and incapable of clustering global semantics information and combining it with finer-scale features.
Recently, attention- based methods (e.g., CLRNet 41 have shown promising capability lane detection by describing a lane line series of key points. By taking the lane lines as a whole unit
"Corresponding Author. Email: Thhandong@@utloook
CCoresponding Author. Email: wangwnezhe www.eehueeuju..uuccc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

these methods can make use of global semantics information and finer-scale features, which help the detection of lane lines in complex environments. However, since these methods mainly rely on finer-scale information to identify the position of each key point, their detection results may have large deviations when there is local occlusion or blurring. In addition, former studies did not take into account the thin and long structure of lane lines during supervision, thus often leading to inaccurate detection when the lane line has a certain curvature.
In this paper, we propose novel Siamese Transformer with hierarchical refinement for lane detection improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Two attributes of our proposed method contribute its universality for the lane detection task. First, address the under utilization of finer-scale information when fine-tuming the positions of key points, we propose novel Siamese Transformer structure with shared parameters, called L.Ane TRansformer (LATR),, which can integrate global semantics information and finer-scale features. Simultaneously, we develop high-to-low hierarchical refinement scheme to refine the key points of lane lines so that the network can fully learn information at different scales. Second, to take the thin and long structure of lane lines into account, we propose a novel Intersection over Union (IoU) loss called Curve-IoU (CIoU) for lane detection. Compared to the common IoU loss, we supervise the fit of lane lines at different locations for the thin and long structure of the lane lines, which helps accurately detect lane lines with curves.
We evaluate our proposed method on three benchmark datasets for lane detection, OpenLane [5], CULane [26]. and Tusimple [34,, and achieve state-of-the-an results. Our main contributions can be summarized as follows:
We propose novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.
We propose a high-to-low hierarchical refinement Transformer structure called LATR to refine the key points of lane lines so that the network can fully integrate global semantics information and finer-scale features.
Exploiting the thin and long structure of lane lines, we propose novel Curve-IoU loss to supervise the fit of lane lines different locations, which helps the regression of the curves.
We achieve state-of-the oof-the-art results on three benchmark datasets, with 5.0% improvement in Fl score compared to the best- koown method on the OpenLane dataset.
2
Related Work
Due to their limited feature capturing capability and low robustness, these methods often fail in complex conditions.
Early lane detection studies relied on hand-crafted features 7
23
To improve the robustness of lane detection under different environments, iegmentation-basee 38 39 were introduced to lane detection. These methods typically apply post- processing operations such as curve fitting and clustering on pixel-level segmentation maps to generate final results. Compared traditional methods, segmentation-bist based methods able capture more plentiful visual features and spatial structure information, thus achieving better performance than traditional detection methods. However, per-pixel per-pixel-based segmentation methods incur high computational costs, have limited real-time capability, and struggle with learning lane line specific long and thin characteristics.
methods 10,
13
To address these issues, LaneNet 255 introduced a branched, multi-task architecture to cast the lanc detection task as instance segmentation problem. This method is more robust to variations in road conditions compared to the previous methods, but is more time-consuming. RESA 40 proposed to aggregate spatial information by shifting sliced feature maps, which obtains good real-time results but still fails under complex road conditions. Furthermore, the output lane lines of most of the above methods may not be continuous.
To attain more continuous lane lines with higher efficiency, recently curve-based methods 18
viewed the lane detection task as polynomial regression problem and utilized parametric curves to fit lane lines. These methods depend heavily on the parameters of the curves (e.g., x = a + + c-y--cy-- +d, where (x, y) denotes the coordinates of a lane line pixel and a, b, c, and are the parameters of

---

ground truth
Siamese Transformer
lane query
LATR:
LATR:
LATR
LATR
Curve FFN
input image
Swin
Offict FFN
Curve-loU
Swin level:
Swin levels
backbone
hierarchical features
lane anchor
The overall architecture our proposed method. It includes a multi-scale backbone to extract hierarchical features from the input image, Siamese Transformer structure named LATR to integrate global semantics information and finer-scale features, and a Curve-IoU loss to supervise the fit of lane lines for model training. Swin level 1-3 denotes the multi-scale backbone using the Swin Transformer. FFN represents a feed-forward network.
Figure I:
curve function). PloyLaneNet 33 first proposed an end-to-end deep polynomial regression method that directly outputs the parameters. To improve the stability and efficiency. BézierLaneNet [9] proposed parametric Bézier curve to model the geometric shape of lane lines. However, because of the limited learning ability of global information, the accuracy of these curve-based methods is not satisfactory large datasets, especially in complex road conditions.
Attention-basee based methods 32 18 introduced the computer vision field, and were proven to be able to capture long-range information. ATT (32 introduced an attention mechanism to anchor-based lane detection methods. Based on PolyLaneNet [33), LSTR [18] was proposed with high inference efficiency but relatively low accuracy, especially in some complex road environments. PriorLane [30] improved the accuracy of prediction compared to LSTR with pre-training and local prior. However, there is still gap in accuracy between the contemporaneous Transformer -baseed methods and CNN-based methods, and the reason can be attributed the shortcut of the multi-head self-attention mechanism which neglects the characteristics of different frequencies [28
Method
In this section, we present the proposed Siamese Transformer with hierarchical refinement for lane detection. First, we describe the overall architecture of our proposed network. Then we explain each key component of the proposed network, including the LAne TRansformer (LATR) and the proposed Curve-IoU loss. Finally, we provide the inference details of our network.
3.1 Overall Architecture
We present the overall architecture of proposed method in Fig. consists of multi-scale backbone and a Siamese Transformer structure named LATR. An input image "I xC first fed into the backbone by LATR with the supervision of the Curve-IoU loss. Next, we use two different detection feed- forward networks (FFN) to generate (1) lane line properties including the start point (æo, yo), angle of inclination 0, and length of each lane line, and (2) the offset map {o:) Finally, key points of the lane line are produced by post- prss-prcesssngg which can be expressed
obtain hierarchical features from high to low levels, which then refined
2; = tan x (y: - yo) + Fo + O;:
where denotes the i-th key point of the lane line. All the key points are sampled at equal intervals based the y-axis, which can be expressed y: 7 where H and P represent the image height and the number of key points, respectively.
3.2 Lane Transformer
3

---

Transformer structure. Our proposed LATR employs the Siamese Transformer structure based on the hierarchical fea- tures extracted by the backbone of the network. Inspired by the recent Transformer-basee networks with Siamese structures [114 we develop high-to-low refinement structure obtain features of the predicted lane line. The detailed structure of LATR is illusttated in Fig. 2
Lane query
FFN
Specifically, we denote lane query as Qa = {q3 } where N represents the dimension of the decoder embeddings and is the number of the scales. The lane query is series of one-dimensional features of the lane lines, which can be further processed to obtain the key points of the lane lines. The input hierarchical features Pa E Ha xC' are extracted by the multi-scale backbone (we use Swin Transformer [19] for it), and then are sorted from high-level to low-level. Next, we flatten it into Ma E " with 2-D positional embedding.
Deformable A ttention
seff-attention
Different from DETR [3], we integrate the lane query in the Transformer encoder and then employ Deformable Attention 42 as cross-attention in the decoder process. The whole pro-
cess can be expressed as:
Lane query from the higher-scale
Hierarchical feature
Io MMa- + Q2-11
k: = Softmax
Ved Qa Q DA(K, 1'a-1) Qa-1,
(2)
Figure 2:
The detailed structure of our proposed LAne TRansformer (LATR). We employ a high-to-low refinement structure in which the input lane query refined by higher-scale features.
where Q'2 and M' represent the lane query and high-to-low hierarchical features after positional embeddings respectively. DA() denotes the Deformable Attention, and 24 represents the sequence length.
Compared with DETR, our proposed Siamese Transformer integrates the lane query during the encoder. The multi-scale features from the backbone have different shapes. Note that integration during the encoder can unify the decoder's input scales with less information loss. When the scales are unified, the network can be made into Siamese structure with shared parameters, which leads to a big efficiency improvement.
High-to-low refinement structure with shared parameters. Recent studies have shown that Vision Transformer (ViT) is good at extracting low-freeg wwwffreuenccy information (in other words, global semanties information) from images (27]. However, lane lines are characteristically thin and long, and their accurate detection often requires integrating finer-scale features and global semantics information. High-level information can help determine the structural information of lane lines while low-level information can help adapt key points of the lane lines. Therefore, adopt high-to-low refinement structure refine the lane query. Given hierarchical features Pa Ha from the multi-scale backbone, we first flatten it into Ma E H with 2-D positional embedding. Then we employ cross-attention to integrate Ma with the lane query into S. E R-NxC', Since S. shares the same shape, the same parameters can be used during refinement between multi-scales. By refining the lane query from high to low, the network can adapt the key points of the lane lines from coarse to fine, which helps the Transformer learn high-frequency information of the image. After the refinement, we employ a detection FFN to generate the lane anchor and finally attain the predicted lane lines with bi-linear interpolation between the key points.
3.3 Curve-IoU Loss
We represent a lane line as sequence of key points, and we need loss to supervise the regression of lane lines. In previous work, LIoU [4] was proposed as a similarity function to calculate the distance between predicted key points and the ground truth. Although the LioU loss ensures the scale consistency of the lane shape, it does not match the I.1 distance when there is long distance
between the prediction lane line and the ground truth. Fig. B a) shows typical curve scenario that

---

Graun Truth Prediction
(1400))
LioU
d
Prediction
Curve-laU
loU value
(4,044
(217,006)
p
21.
Distance between GT and Prediction
(b)
Figure 3: (a) typical failure scenario for LIoU, which cannot measure the distances between Ground Truth and Predictions A and B; (b) the values of LioU and Curve-IoU.
LIoU cannot describe correctly. In such cases, lane lines are inclined and curved, and the LIoU loss unable judge which prediction lane line is closer the ground truth. In Fig. 3 the values for Prediction A and Prediction B are -0.6 and 0.6, respectively, However, the L1 distances for them are Se and 1.5e, respectively.
To bridge this gap, we propose Corve-IoU (CloU). Different from the traditional IoU, we add a penalty term to IoU that the spatial distance between each pair of lane lines will be represented more accurately, especially for those with long distance between the prediction lane line and the ground truth. We sequence of points with certain width present lane line and calculate the Intersection over Union, which can be written as:
df min(P, GT) max(P; (P), G)),
d = max(P, GT) min(P), (P)..) G)),
where p! 2, e and P. = 24 + e, and G and Gj are defined similarly. IoU can be calculated as IOU di Then we add the LI distance between each corresponding pair of lines as the penalty the overlap ranges, as:
du.
LIoU =1 Ex,, (d& R
(4)
Σ dy
where j is the first valid point in the lane line. Then the CloU loss can be calculated as CloU 1 LIoU.
Exploiting the thin and long structures of lane lines, our CloU loss can supervise the regression of lane lines better. As shown in Fig. 3)). the values of LIoU and CloU for Prediction A are -0.6 and -1.6 respectively, which means that our CloU can handle these scenarios more precisely.
3.4 Inference Details
In order to reduce the training costs, we generate only the lane anchor during training. The loss is calculated by comparing the lane anchor the ground truth. Our method generates fixed number of lane anchors, which is much larger than the maximum number of lane lines in the image. Besides the Curve-IoU loss, we also utilize classification accuracy loss determine whether the lane anchor a lane line or the background. The classification accuracy loss can be described as:
L
lets
Σ C (p..gi), i=1
(5)
where C(:,) denotes the cross-entropy loss and T. is the number of sequences. If the i-th predicted lane line matches the ground truth, then 1 (0 otherwise). The objective function of our network can be represented as:
l Aatehata + AregClol)
(6)
where Xets and Areg are importance weight values. We supervise each generated lane anchor during training, while during inference we use Mon-Maximum Suppression (NMS) (4 exclude incorrect lane lines. Finally, utilize bi-linear interpolation produce the predicted lane lines of the image.
5

---

Method
Backbone
F1
Up
Down
Curve
Extreme Weather
Night
Insersection
Merge & Split
FPS(
GFllgel
LaneATT
KesNet
25.3
25.8
32.0
14:00
24.3
9.3
LaneATT
PensFommr
EMhiiemNNa887
42.0
40.7
46.3
41.7
36.
28,9
41.2
CondLaneNeet
1
ResNeti8
$2.3
55.3
$7.5
45.8
46.6
48.4
45.5
173
10.2
CondLameNett
12
ResNet34
55.0
58.5
99.4
49.2
50.7
47.8
19.6
CLRNet 011
ResNe118
52.3
55.3
$7.5
45.8
46.6
48.4
15.5
11.9
CLRNet 11
ResNer34
55.0
58.5
59.4
49.2
48.6
50.7
47.8
124
19.6
CLRNet 41
RasNeti01
59.1
62.1
62.9
$4.7
51.0
55.7
52.3
41.8
RaaNet18
13.7
CTR
ResNet34
66.7
57.2
57.5
23.2
CTR
ResNeti01
67.8
62.2
68.0
89.8
57.4
59.1
69.
50.2
Our method
68.3
69.5
65.3
32.4
30.9
56.3
69.6
10.2
small
Our method
buse
70.5
69.1
$5.7
52.2
39,9
71.6
16.8
Table 1: Comparison results of recent methods and our method on the OpenLane dataset. In order to compare the computation speeds in the same setting, we remeasure FPS on the same machine with an RTX3090 GPU using open-source code (if code available). The best results in each column marked as bold and the second best results are underlined.
Mellond
Hackhone
Total
Normal
Crowied
Flight
Shackww
Natine
Arnw
Cuivve
Closs
Night
FPSCTT
VGGT6
71.60
UFLD
RexNe118
68.40
87.70
38.
40.3
57.90
1743
62.10
341
8.4
LPL,0
60
Net34
72.30
90.70
70.25
59:50
69
44.40
85.70
69.50
2037
66.70
184
LSTR 18
ResNet18
68.72
86.78
67.34
56.63
59.82
10.10
78.
56.64
1166
59.92
126
2.9
ResNoit8
64.76
Res
67.77
Laneformer
11
ResNet50
77.06
91.77
75.4]
70.17
75.75
48.73
66.33
19
71.04
RESA 50
ResNet34
74.50
91.90
72. 40
66.50
72.00
46.30
S8. 10
68.60
1896
69.80
51
RESA
40
ResNei50
75.30
92.10
73.10
69:20
72.80
47.70
S8. 30.30
70.30
1503
69.90
ResNet18
ADNet
Ax
ResNet18
77.56
91.92
75.81
69.39
76.21
51.75
$7.71
68.84
1133
72.33
ADNet
3
ResNet34
78.94
92.90
77.45
71.71
79.11
$2.89
$9.90
70.54
1499
74.78
77
ConsLannNe
77
RexNet18
78.14
92.87
75.79
70.72
80.01
$2.39
89.37
72.40
1364
73.23
10.2
73.71
66
11.9
CLRNer
41
ResNetIDI
80.13
93.85
78. 78
72.49
82 33
54.48
89.79
75.57
1262
75.51
42.9
CLRNer
41
DLA33
80:47
92.73
79.59
75.30
82.51
54.58
90.62
74.13
1155
75.37
101
18.5
80.01
75.41
Our
1022
75.54
Our method
hase
80.85
93.92
80.21
76.04
81.65
$5.42
89.53
75.66
1043
75,81
168
Table 2: Comparison results of recent methods and our method on the CULane dataset.
4 Experiments
4.1 Datasets
To demonstrate the effectiveness of our proposed method in realistic road environments, we evaluate it on three benchmark datasets: OpenLane [5), CULane [26). and Tusimple [34]. OpenLane is real-world large-scale lane detection dataset, which contains 160K and 40K images as the training and
validation sets, respectively. The validation set consists of six realistic road scenarios and annotates 14 lane categories (including white dotted line, double yellow solid, left/right curb, and so on). CULane is widely-used large dataset for lane detection including eight hard-to-deted scenarios in urban areas and on highways, with 88K and 34K images the training and validation sets, respectively. Tusimple is also a widely-used dataset with images collected on US highways under clear weather, which contains images for training and 2K for validation.
4.2 Evaluation Metrics
For OpenLane 5 and CULane 26 we adopt the F1-score measure proposed by SCNN 2d as the evaluation metric. Intersection-oven -ver--nion (IoU) between the ground truth (GT) label and each predicted lane line of the model is calculated to determine whether sample is True Positive (TP). False Positive (FP), or False Negative (FN). The ways to calculate IoU and F1-score can found in the supplemental material. For Tusimple 34 J the evaluation metrics are composed of three official indicators: Accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). The way to calculate Accuracy can be found in the supplemental material.

---

Method
Backbone
F1
Acc
FP(L)
FN(()
SCNN
VGGI6
95.97
96.53
6.17
1.80
RESA
ResNet34
96.93
96.82
3.63
2.48
UFLD
ResNet18
87.87
95.82
19.05
3.92
UFLD
ResNet34
88.02
95.86
18.91
3.75
PolyLaneNet
EfficientNerBO
90.62
93.36
9.42
9.33
LaneATT
ResNet18
96.71
95.57
3,56
3.01
LaneATT
ResNet122
96.06
96.10
5.64
2.17
LSTR
ResNet18
96.84
96.18
2.91
3.38
ADNet
ResNet18
96.90
96.23
2.91
3.29
ADNet
ResNet34
97.31
96.60
2.83
2.53
PriorLane
ResNet18
97.15
96.58
3.91
2.95
Laneformer
ResNet18
97.30
96.54
4.35
2.36
Laneformer
ResNet34
97.41
96.56
5.39
3.37
Laneformer
ResNet50
97.56
96.80
5.60
1.99
CLRNet
ResNet18
97.41
96.84
2.28
1.92
CLRNet
ResNet101
97.68
96.83
2.37
2.38
Our method
tiny
97.76
96.85
2.55
2.03
Our method
small
98.01
96.79
2.13
2.17
Our method
base
97.85
96.95
1.86
2.34
Table 3: Comparison results on the Tusimple dataset.
4.3 Implementation Details
the pre-trained backbone of network. We divide the versions of the backbone of the network based on the size the Swin Transformer into three categories: tiny, small, and big, which consistent with the work in [19] For data augmentation, we adopt the affine transformation method (horizontal flip. rotation), brightness, and saturation addition method. All the input images reshaped into 800 320 pixels each for both the training and inference stages. For the number of lane anchors in an image, we set it to 150. In the optimization process, adopt AdamW (1 and the cosine decay learning rate strategy [20] with the initial learning rate set to 6e-4. A batch size of 32 and training epoch numbers of 10, 20, and 90 used for OpenLane, CULane, and Tusimple, respectively. All the experiments conducted machine with single NVIDIA RTX3090 GPU with 24GB memory.
In the experiments, adopt the Swin Transformer 19
4.4 Experimental Results
4.4.1 Results OpenLane
Comparisons of results by recent methods and method the OpenLane dataset shown in Our method achieves state-of-the-art results in Fl score. Specifically, our method achieves Fl scores of 68.3, 69.1, and 69.7, surpassing those of the best-known method LendLaneNet by 5.0, 3.5, and 1.9 points, respectively. Further, our method achieves the best performances in four of six scenarios. showing the robustness our method. Among these, the "Up & Down" and "Curve" categories are 7.2% and 1.1% higher than the previous best results, respectively. These results demonstrate that the proposed Siamese Transformer with the hierarchical refinement structure can deal with lane lines in realistic road environments very well. This is because our proposed LATR integrates global semantics information and finer-scale features, which help refine key points when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Simultaneously. the proposed Curve-IoU can supervise the regression of curves well, which helps accurate detection of lane lines.
Table I
4.4.2 Results on CULane
We compare the results by recent known methods and our method on CULane dataset Table In the total F1 score, our method achieves state-of-the-art results, with 0.38% improvement over the best-known CLRNet. Among all the eight difficult scenarios, our method achieves the best results on five of them. Specifically, for the "Crowded" and "Hlight" scenarios, our method achieves
7

---

GT
CondLane
CLRNet
Ours
Figure 5: Visualization results of Ground Truth (GT), CondLaneNet [17
(CondLane), CLRNet 41
and our method on CULane [26]. The results of Condl LaneNet and CLRNet are generated with ResNet18 and ours generated with Swin Transformer tiny. Different lane lines are represented by different colors. The Fl score for each predicted image is labeled in the top left corner of the image.
80.21% and 76.04% Fl scores respectively, surpassing those of CLRNet by 0.62% and 0.74% points, respectively. These results demonstrate that our method can deal with realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.
Fig. 5 shows some visual results several known methods and our method on the CULane dataset. Condl LaneNet is open-source dynamic convolution-base method and CLRNet is the second best-performing method on CUL Lane... The results of Condl.aneNe and CLRNet are generated with ResNet18 as the backbone and ours are generated with Swin Transformer tiny the backhone. The visual results show that when the roads are crowded and lane lines are covered by other vehicles, our method can detect lane lines better and the lane regression is closer the ground truth. What's more, our method also outperforms these methods in night scenarios, proving that our method can adapt to different lighting environments with good robustness.
4.4.3 Results on Tusimple
Comparison results of recent methods and our method on the Tusimple dataset are given in Table B This dataset consists of images captured in different weather conditions. Our method achieves state-of-the-ar results in Fl score, Accuracy, and False Positive Rate (FP), demonstrating that our method can be adapted both complex urban environments and simple highway scenarios. Note that because the dataset was collected on US highways where the road conditions are relatively simple (lane features are more obvious and clear), the results of various methods are close.
4.5
Ablation Studies
4.5.1
LAne TRansformer (LATR)
To fully integrate global semantics information and finer-scale features, we propose a high-to-low hier- archical refinement Transformer structure for lane detection called LAne TRansformer (LATRR) which helps identify key points especially when roads are crowded or affected by blurring. Fig. 4 shows some high-to-low attention maps of our LATR. From Fig. 4 one can see that attention is gradually focused on key points on both sides of the road from high to low. High-level attention is extended along the road from the near end to the far end of the road, initially defin- ing the overall structure the lane lines. Low-level attention focuses on finer-scale features of the lane lines, which refine the key points of the lane lines from higher-levels to lower-levels.
Figure 4: High-to tow attention maps our pro- posed LATR.
We compare the F1 scores of our Lane Transformer (LATR) and existing Transformer structures with the same backbone Swin Transformer tiny CULane, and report the results in Table 4 Compared

---

with the existing Transformer structures, our method has big advantage of extracting features at multi-scales, which play an important role the process of lane detection. Compared with the Deformable Transformer, our method improves the detection accuracy by 4.36%. Further, as shown in Table lane features cannot be fully obtained by relying only on high-level features Po or low- level features P3. The Siamese refinement structure from high to low can better integrate the global semantics information and siner-scale features, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.
Transformer
Refine level
Fl
FPS
F1
F1
Loss
ViT
8
72.46
OpenLane
CULane
DETR
3
74.
w/o IoU
64.28
77.16
Deform
75.65
smooth-Li
65.52
77.86
PriorLane
30
76.27
RIoU 22
67.28
79.01
Our method
Our method
Po
Ps
78.15
76.87
210
208
LIoU 41
67.75
79.24
Our method
Po - Ps
80.01
172
Curve-IoU 68.34
80.01
Table 4: Ablation study results of the Lane Trans- former on the CULane dataset with the same back- bone (Swin Transformer tiny).
Table Effect our proposed Curve-IoU on OpenLane and CULane. "w/o IoU" de- notes optimizing with no IoU loss.
4.5.2 Curve-IoU Loss
To help the regression of lane lines, we propose the Curve-IoU loss. In Table [] we compare our Curve-IoU with different types of IoU for lane detection. "w/o IoU" denotes optimizing the network with no IoU loss, using only classification and LI regression loss, which only yields F1 score of 77.16%. LIoU was proposed by CLRNet as a similarity function to calculate the distance between the prediction and ground truth. However, LIoU does not match the I.I distance when there a long distance between the prediction lane line and ground truth. Compared with LIoU, our Curve-IoU takes the L1 distance into account, which helps the regression of lane lines with curves. On the CULane dataset, Curve-IoU achieves an F1 score of 80.01 %, which is 2.85% higher than w/o IoU and 0.77% higher than LIoU. Its effect on Transformer-based lane detection is validated.
5
Diseussion
Our method achieves state-of-the-ard results on three benchmark datasets and performs the best among all the Transformer-basee methods, especially some challenging and complex road conditions. However, we still find that the current known anchor-based methods are competitive in the "Shadow", "Noline", and "Arrow" scenes on the CULane dataset, which the opposite of the OpenLane dataset. Also, the improvement of our method is larger on the OpenLane dataset. For this interesting phenomenon, we assmue the reason behind it is that the CULane dataset contains 88K images for training while the Openlane dataset contains I 160K, which is almost twice that of CULane and includes more complex and various scenes. Our Transformer- on larger dataset compared to anchor-based and CNN-based methods, that is, our method has much bigger potential for scaling up if the data and computation resources are sufficient. Considering the current state of affairs in vision-based autonomous driving systems, deep learning algorithms are advancing and models are expanding, yet the systems are still constrained by the size and diversity of datasets. We call for larger and better datasets featuring more diverse scenes for further research.
method performs relatively better
6 Conclusions
In this paper, we proposed a novel Siamese Transformer structure with hierarchical refinement, which achieves state-of-the- art results on three benchmark datasets. Specifically, we developed high-to-low hierarchical refinement Transformer structure called LATR to refine key points of lane lines, which compensates for the Vision Transformer's s deficiency in extracting finer-scale features. Also, we proposed a novel Curve-IoU loss tailored for the long and thin shape of lane lines, which helps supervise the regression of lane lines with different offsets. Extensive experiments confirmed that our model effectively handles complex scenarios, particularly when lane lines are heavily obscured by other vehicles compromised by poor lighting conditions.

---

References
[1] Wele Gedara Chaminda Bandara and Vishal M Patel. A Transformer-based Siamese network for change detection. In /GARSS 2022-2024 IEEE International Geoscience and Remote Sensing Symposium, pages 207-210. IEEE, 2022.
[2] Kenneth R Britttin. Inertial navigation systems analysis. 2010.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, et al. End-to-end object detection with Transformers. In Computer Vision-ECCV 2020: 16th European
Conference, 2020, Proceedings. Part I 16. pages 213-229. Springer, 2020.
[4] Haoxin Chen, Mengmeng Wang, and Yong Liu. BSNet: Lane detection via draw B-spline curves nearby. arXiv preprint arXiv:2301.066 109100,, 2023.
[5] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. PersFormer: 3D lane detection via Perspective Transformer and the openlane benchmark. In European Conference on Computer Vision, pages
550-567. Springer, 2022.
[6] Ziye Chen, Yu Liu, Mingming Gong, Bo Du, Guoqi Qian, and Kate Smith-Miles. Generating dynamic kernels via transformers for lane detection. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 68 0812-6821. 2023.
[7] P.M. Daigavane and P.R. Bajaj. Road lane detection with improved Canny edges using ant colony optimization. In 2010 3rd International Conference on Emerging Trends in Engineering and Technology, pages 76-80, 2010.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Weissenborn, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.1 19229, 2020.
[9] Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, et al. Rethinking efficient lane detection via curve modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17062-17070. 2022.
[10] Mohsen Ghafoorian, Cedric Nugteren, Nóra Baka, Olaf Booij, and Michael Hofmann. EL-GAN: Embedding loss driven generative adversarial networks for lane detection. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pages 0-0, 2018.
[1]] Jianhua Han, Xiajun Deng, Xinyue Cai, Zhen Yang. Hang Xu, Chunjing Xu, and Xiaodan Liang. Object-aware row-column Transformers for lane detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 799-807, 807, 2022.
Laneformer:
[12) Yuenan Hou, Zheng Ma, Chunxiao Liu, Tak- Wai Hui, and Chen Change Loy. Inter-region affinity distillation for road marking segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12486-12499 2020.
[13] Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection CNNs by self attention distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1013-1021, 2019.
[14] Le Hui, Lingpeng Wang, Linghua Tang, Kaihao Lan, Jin Xie, and Jian Yang. 3D Siamese Transformer network for single object tracking on point clouds. In European Conference on Computer Vision, pages 293-310. Springer, 2022.
[15] Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-CNN: End-to-end traffic line detection with line proposal unit. IEEE Transactions Intelligent Transportation Systems, 21(1):248-258
2019.
[16] Guoliang Liu, Florentin Wörgötter, and Irene Markelic. Combining statistical Hough transform and particle filter for robust lane detection and tracking. In 2010 IEEE Intelligent Vehices Symposium, pages 993-997. IEEE, 2010.
[17] Lizhe Liu, Xiaohao Chen, Siyu Zhu, and Ping Tan. NondLaneNet: A top-to-down lane detection framework based on conditional convolution. In Proceedings of the IEEE/CVE International
Conference on Computer Vision, pages 3773-3782, 2021.
[18) Ruijin Liu, Zejian Yuan, Tie Liu, and Zhiliang Xiong. End-to-end lane shape prediction with Transformers. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer
Vision, pages 3694-3702, 2021.
[19] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021. [20] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv:1608 1et.:66680...8.... 2016.
[21] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
ar11v:1711.01101, 2017.
10

---

[22] Zinan Lv. Dong Han, Wenzhe Wang, and Cheng Chen. IFPNet:
Integrated feature pyramid network with fusion factor for lane detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 1888-1897, 2023.
[23] Hiren M. Mandalia and Mandalia Dario D. Salvucci. Using support vector machines for lane- change detection, Proceedings of the Human Factors and Ergonomics Society Annual Meeting,
49:1965- 1969, 2005.
[24] Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th In- ternational Conference on Pattern Recognition (ICPR 066, volume 3, pages 850-855. IEEE, 2006.
[25] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Towards end-to-end lane detection: An instance segmentation approach. In 2018 IEEE Intelligent
Vehicles Symposium (IV), pages 286-291. IEEE, 2018.
[26] Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial CNN for traffic scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[27] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast Vision Transformers with Hilo attention.
Advances in Neural Information Processing Systems, 35:14541-14555 2022.
[28] Namuk Park and Songkuk Kim. How do Vision Transformers work? In International Confer-
ence on Learning Representations, 2022.
[29] Zequn Qin, Huanyu Wang. and Xi Li. Ultra fast structure- aware deep lane detection. In Computer Vision-ECCV 2020: 16th European Conference, 2020, Proceedings, Part XXIV 16, pages 276-291. Springer, 2020.
[30] Qibo Qiu, Haiming Gao, Wei Hua, Gang Huang, and Xiaofei He. PriorLane: A prior knowledge enhanced lane detection approach based on Transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5618-5624, IEEE, 2023.
[31] Jinming Su, Chao Chen, Ke Zhang, Junfeng Luo, Xiaoming Wei, et al. Structure guided lane detection. arXiv preprint arXiv:2105.0544
2021.
[32] Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue. Alberto F De Souza, et al.
Keep your eyes on the lane: Real-time attention-guides lane detection. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 294-302, 2021.
[33] Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue. Alberto F De Souza, et al.
PolyLaneNet: Lane estimation via deep polynomial regression. In 2020 25th International
Conference on Pattern Recognition (ICPR), pages 6150-6156. IEEE, 2021.
[34] Tusimple.
TuSimple benchmark. 2020.
https: h/tggtttuu..... wtw.chu...ootttttt.ee com/TuSimple/
bessmplee-becchark// Accessed September, 2020.
[35] Jinsheng Wang, Yinchao Ma. Shaofei Huang, Tianrui Hui, Fei Wang. et al. A keypoint- basedd based global association network for lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1392-1401, 2022.
[36] Pei-Chen Wu, Chin-Yu Chang, and Chang Hong Lin. Lane-mark extraction for automobiles under complex conditions. Pattern Recognition, 47(8):2756-2767 2014.
[37] Lingyu Xiao, Xiang Li, Sen Yang, and Wankou Yang. ADNet: Lane shape prediction via anchor decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6404-6413, 2023.
[38] Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang. Xiaodan Liang, et al. CurveLane-NAS: Unifying lane-sensitive architecture search and adaptive point blending. In Computer Vision- ECCV 2020: 16th European Conference. 2020, Proceedings. Part XV 16, pages 689-704.
Springer, 2020.
[39] Seungwoo Yoo, Hee Seok Lee, Heesoo Myeong, Sungrack Yun, Hyoungwoo Park. Janghoon Cho, et al. End-to-end lane marker detection via row-wise classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 1006-1007,2033
[40] Tu Zheng, Hao Fang, Yi Zhang, Wenjian Tang, Zheng Yang, et al. RESA: Recurrent feature-shiff aggregator for lane detection. In Proceedings of the AAAI Conference on Artificial Intelligence.
pages 3547-3554, 2021.
[41] Tu Zheng. Yifei Huang. Yang Liu, Wenjian Tang. Zheng Yang, et al. CLRNet: Cross layer refinement network for lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 898-907, 2022.
[42] Xizhou Zhu, Weijie Su. Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,
2020.
11

---

A Evaluation Metrics
For OpenLane [5] and CULane 26 we adopt the FI score proposed by SCNN 266 as the evaluation metric. Intersection-over -eee--nve--nnon (IoU) between the ground truth (GT) and the predicted lane line of the model is calculated to determine whether a sample True Positive (TP), False Positive (FP), or False Negative (FN). The IoU and Fl score are calculated as in the following formulas:
TP
TP
Precision
Recall
(7)
TP + FP
TP+FN
Intersection
IoU =
(8)
Union
Precision Recall
F1 Precision + Recall
(9)
For Tusimple [34]. the evaluation metrics are composed of three official indicators: accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). The accuracy is calculated as:
Accuracy = Eclip Celip
(10)
Eetip Sclip
where Cetip is the number of correct points and Solap is the number of ground truth (GT) points in an input image. the accuracy of a predicted lane is greater than 85%, it will be considered a True Positive (TP). F1 score is also used in the evaluation.
B
More Ablation Studies
B.1 Overall Ablation Study
To verify the roles of our proposed lane Transformer (LATR) and Curve-IoU in our method, we carry out overall ablation study with the same baseline LSTR [18]. The results of the ablation study are shown Ta LATR can greatly improve the detection accuracy of lane lines, which improves the Fl scores with 8.38% and 6.45% increase OpenLane and CULane, respectively. This is because our proposed LATR can better integrate global semantics information and finer-scale features, especially when roads obscured by other vehicles affected by poor outdoor lighting conditions. What's more, our proposed Curve-IoU further improves the Fl scores by 1.23% and 1.66% on OpenLane and CULane, respectively. These results show that our proposed Curve-IoU can improve the accuracy of lane detection, especially for roads with curves.
Number of Lane anchors
F1 OpenLane
Fl
Fl
Fl CULane
FPS
LATR Curve-IoU
CULanc
OpenLane
50
65.64
79.16
208
59.96
73.56
100
67.56
79.75
195
67.11
78.35
150
68.34
80.01
172
61.86
76.05
200
68.36
79.91
150
68.34
80.01
250
68. 18
80.05
131
Table 6: Results of the overall ablation study with the same backbone. We conduct the overall ab- lation study based on the same baseline LSTR
Table Results of the ablation study on num- ber of lane anchors. "FPS" denotes the FPS on the CULane dataset.
18
B.2 Number of Lane Anchors
Our method generates a fixed number of lane anchors, much more than the maximum number of lane lines the image. Here we conduct an ablation study on the number of lane anchors. As shown in Table 7 when we increase the number of lane anchors from 50 to 150, the accuracy of lane detection on both Lpen and CULane improved. This demonstrates that increasing the number of lane anchors is beneficial to the detection of lane lines. However, when further increase the number of
12

---

M
HID
CondLane
CLRNet
Our method
Figure 6: More visualization results of Ground Truth (GT), ConeLaneNet (CondLane), CLRNet 41, and our method on the CULane dataset [26). The results of CondLaneNet and CLRNet are generated with ResNet18 and ours are generated with Swin Transformer tiny. Different lane lines marked by different colors.
lane anchors, the enhancement is not obvious appears be decreasing. This is because when the number of lane anchors reaches a certain value, adding more lane anchors will cause redundancy. 150 lane anchors enough for the network to detect lane lines in the image.
B.3 More Studies on Backbone
In order evaluate the bias caused by the backbone net, conduct ablation experiments backbones. We find that the Swin Transformer with hierarchical features can better extract image features for subsequent processing. Therefore, we choose the Swin Transformer as our backbone network, which is an open-source backbone widely used by other works. The results show that our approach achieves better results with higher FPS and lower GFlops. We also replace the backbone of previous methods (e.g., CLRNet) with Swin Transformer and train these models under equal situations. The results are shown below in Table g
Method NondLaneNet
Backbone
Fl score on CULane
Swin-tiny
77.16
ConNee
Swin-base
77.84
CLRNet
Swin-tiny
79.05
CLRNet
Swin-base
79.73
Ours
Swin-ting
80.01
Ours
Swin-base
80.85
Table 8: Replace the backbones of previous methods with Swin Transformer.
13

---

B.4 Scaling up
We only use single LATR module for each pyramid level, find out the potential scaling- up ability of our model, we add more layers to each level, and the results are shown in Table and seen from the results that more LATR added at the lower feature levels can improve performance. while not leading to much increase in parameters. As result, we think it is effective to add two LATR modules both the lowest and second-lowest feature levels.
and [TO It can be
Number of LATR Module
Fl Score on CULane
Layer Level
Fl Score on CULLane
Params
80.01
80.10
29.186 M
80.32
2 & 2
80.15
19.186 M
80.43
80.22
29.592 M
Table 9: Performance with Different Numbers of LATR Modules.
Table 10: Impact of Adding 2 LATR Modules
to lowest and second lowest feature levels
C More Visualization Results
More visualization results of Ground Truth (GT), CondLaneNett [17], CLRNet [4 and our proposed method the CULane dataset (26) are shown in Fig. 0 Compared with CondLaneNet and CLRNet, our method is more effective in detecting lane lines on congested roads and different lighting environments, which demonstrates the robustness of our method.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly illustrate our proposed model LATR, and precisely show the experiment results we have achieved.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper thoughtfully discusses the limitations of the research in Section 5 We evaluate the robustness of our proposed method on three datasets in varying scenarios, and before that, we make assumptions and declare the inference details thoroughly in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not propose theoretical results in this paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We illustrate our experiment details in our paper. and we will release our code and checkpoint, ensuring our work's 's reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We will release our code and checkpoint this paper, and all the datasets we are open access. The appendix includes extensive experiments to prove results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pcccccpcccccooooooop htteggguSuddeeoeSuuubbiiiiiaa wwwtanssssonppo. ssinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details are all illustrated thoroughly in Section 4 and our code will release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We display the experiment results in Section 4 which can explain the variance, outliers, and anomalies, and we illustrate all the evaluation metries we use Section 4.2 And
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provides detailed descriptions of the computational resources used for each experiment in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips.. httppr//ipp.ccppppp/e/eeteccccct iace
Answer: [Yes]
Justification: We have thoroughly examined the ethical guidelines and legal considerations, and confirm that study aspects obey the rules.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We comprehensively discuss the broader societal impacts of the research in this paper, and no significant negative impacts are found.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose any such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
This paper cites the original creators of all utilized assets, including code and datasets. The licensing information of our code is clearly outlined, ensuring full compliance with legal and ethical standards.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswipphaeeoooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
This paper documents all newly introduced assets, including code and datasets,
and we have ensured that all assets are appropriately anonymized in line with submission guidelines.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "4QAz3v5q7Z", "reviewer": "Reviewer_UWBy", "summary": "This paper introduces LATR (LAne TRansformer), a Siamese Transformer model with hierarchical refinement for lane detection. The model effectively combines global semantic information with finer-scale features to accurately detect lanes, even in challenging scenarios like occlusions and poor lighting. It also introduces a novel Curve-IoU loss function specifically tailored for the curved nature of lane lines. The proposed method demonstrates state-of-the-art performance on multiple benchmark datasets, particularly excelling on the OpenLane dataset.", "questions": "Have you considered using self-supervised learning techniques to reduce the dependency on large labeled datasets?\nHow does the model perform in extreme weather conditions (e.g., heavy rain, snow) that can significantly affect lane visibility?\nAre there any plans to release the code and trained models to facilitate reproducibility and further research?", "limitations": "The authors adequately address the limitations of their work, acknowledging the data dependency issue and the need for further testing in diverse real-world environments. However, in this field of work, there are a lot of other works that have various implementations exceeding the results from CondLaneNet such as ERF-Net. Looking into that might be of interest to better compare the proposed method.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The Siamese Transformer architecture with hierarchical refinement and the Curve-IoU loss are novel contributions to the field. The proposed method achieves state-of-the-art results on multiple benchmark datasets, demonstrating its superior performance in challenging scenarios.\nThe model's ability to handle occlusions and poor lighting conditions highlights its robustness and potential for real-world applications.\nThe authors conduct extensive experiments and provide a comprehensive analysis, including ablations and comparisons to prior work. Overall, a well written paper.", "weaknesses": "The authors acknowledge that the model's performance is somewhat dependent on the size and diversity of the training dataset. This is a common limitation in deep learning-based approaches and could be addressed with further research on data augmentation or self-supervised learning techniques. \n\nWhile the method excels in the tested scenarios, further testing in diverse real-world environments would strengthen the claims of robustness. However, this is not a major weakness as the current evaluation results are already very promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! The detailed responses to each concern are given below.\n\n**[Have you considered using self-supervised learning techniques to reduce the dependency on large labeled datasets? ]**\n\nWe have considered using self-supervision to reduce the dependence on large labeled datasets and to further improve the recognition accuracy of our approach in different scenarios. We noticed that some self-supervised work performs well on different tasks (e.g. CLLD [1]). We’ll include this part in future work.\n\n**[How does the model perform in extreme weather conditions (e.g., heavy rain, snow) that can significantly affect lane visibility?]**\n\nThe model is able to recognize lane lines under extreme conditions. In the manuscript, \"Extreme Weather\" in Table 1 and \"Hlight\" in Table 2 show our lane detection performance under extreme weather conditions.\n\n**[Are there any plans to release the code and trained models to facilitate reproducibility and further research?]**\n\nWe’ll release the code and pre-trained models once the paper is accepted.\n\n**Reference**\n\n[1] CLLD： Smith, J., & Doe, A. (2023). An innovative approach to deep learning. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kehxbK1mnG", "reviewer": "Reviewer_UDCn", "summary": "This paper introduces a novel Siamese Transformer with hierarchical refinement for lane detection. The core innovation is a high-to-low hierarchical refinement Transformer structure, LATR, which refines lane line key points to integrate global semantic information and finer-scale features fully. Additionally, it proposes a Curve-IoU loss to supervise the fitting of lane lines at various locations.", "questions": "- Suggest enhancing the method's innovativeness.\n- Recommend using common public backbones for ablation studies, though this might make it more similar to CLRNet.", "limitations": "None", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper is well-written and well-organized.\n- The experiments use a comprehensive set of datasets, including three popular ones: TuSimple, CULane, and OpenLane.\n- The experimental results are quite promising.", "weaknesses": "From the perspective of the topic selection, 2D lane detection is less valuable than 3D lane detection. More importantly, this topic is not fully suitable for the NeurIPS conference.\n\nCompared to existing 2D lane detection networks, this paper lacks innovation. Its overall framework is quite similar to CLRNet, and the proposed structured Loss has also been preliminarily explored in CLRNet and UFAST networks.\n\nMore importantly, current lane detection research tends to use simple CNN-based backbones (for ease of deployment) and to better compare performance fairly, especially using the ResNet series (18, 34, etc.). However, the authors only reported results under the Swin Transformer, which makes me skeptical about the fairness of the SOTA comparison.\n\nAdditionally, in terms of efficiency, the FPS speed does not have an advantage, and the model size (params) is not reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback! We address your questions and concerns below. If any other concerns remain, we’ll be happy to discuss them further. If the concerns are addressed well, we would appreciate it if you could consider raising your score.\n\n**[From the perspective of the topic selection, 2D lane detection is less valuable than 3D lane detection. More importantly, this topic is not fully suitable for the NeurIPS conference.]**\n\n2D lane line detection in different scenarios, especially under extreme conditions, is an important yet challenging task. It helps in detection and in the field of autonomous driving. Work on this topic has been published in past NeurIPS conferences, e.g., CARLANE[1], Openlane[2], 3D-LaneNet+[3], and BEVFusion[4].\n\n**[Its overall framework is quite similar to CLRNet, and the proposed structured Loss has also been preliminarily explored in CLRNet and UFAST networks.]**\n\nBoth our network and CLRNet use multi-scale features extracted by the backbone. Many methods such as CondlaneNet use multi-scale features as input so that the network can learn features at different scales. But the next step is different. Our network employs a Siamese Transformer structure that feeds the different scale features directly into our designed Transformer structure called LATR with shared parameters. To follow up the structure of the Transformer so that the network is end-to-end, we employ Swin Transformer as the backbone of the entire network. In contrast, CLRNet employs FPN to further extract multi-scale features, and then uses a detection header to output lane line information, which is different from our work. For the loss, we propose a novel Curve-IoU loss to supervise the fit of lane lines at different locations, which helps the regression of the curves. Our proposed loss is dedicated to the fitting of distal curves, which is different from previous work.\n\n**[The authors only reported results under the Swin Transformer, which makes me skeptical about the fairness of the SOTA comparison.]**\n\nFor the choice of backbone network, we found that the Swin Transformer with hierarchical features can better capture image features for subsequent processing. Thus, we chose Swim Transformer as our backbone, which is an open-source backbone widely used by other works. Accordingly, we designed the LATR and Siamese Transformer structures to reduce the overall number of network parameters. The results show that our approach achieves better results with higher FPS and lower GFlops.\nFor a fair comparison, we also replace the backbone of the previous methods (e.g., CLRNet) with Swin Transformer and train these models in the same setting. The results are shown below.\n\nBackbone  | Method  |  F1 score on CULane\n-------- | -----  | :-----:\nSwin-tiny | CondLane | 77.16\nSwin-tiny | CLRNet     | 79.05\nSwin-tiny | ours           | 80.01\nSwin-base | CondLane | 77.84\nSwin-base | CLRNet     | 79.73\nSwin-base | ours           | 80.85\n\nThe results show that our method performs better with the same backbone compared with previous methods. We’ll add these results to the final vision of our paper if accepted.\n\n**[The FPS speed does not have an advantage, and the model size (params) is not reported.]**\n\nOur approach achieves the best results on many open-source datasets with higher FPS and lower GFlops. Typically, the size of GFlops reflects the size of the model parameter count. We’ll add the number of parameters of our model to the final paper version if it is accepted. The model size of our proposed network is shown below.\n\nBackbone   |  # params\n-------- | :-----:\nSwin-tiny | 28.780618 M\nSwin-small | 50.098522 M\nSwin-base | 88.004936 M\n\n**Reference**\n\n[1] CARLANE: Stuhr, B., Haselberger, J., & Gebele, J. (2023). CARLANE: A lane detection benchmark for unsupervised domain adaptation from simulation to multiple real-world domains. In NeurIPS.\n\n[2] Openlane: Wang, H., Li, T., Li, Y., ... Li, H. (2023). OpenLane-V2: A topology reasoning benchmark for unified 3D HD mapping. In NeurIPS.\n\n[3] 3D-LaneNet+: Efrat, N., Bluvstein, M., Oron, S., Levi, D., Garnett, N., & Shlomo, B. E. (2020). 3D-LaneNet+: Anchor Free Lane Detection using a Semi-Local Representation. In NeurIPS.\n\n[4] BEVFusion: Liang, T., Xie, H., Yu, K., Xia, Z., Lin, Z., Wang, Y., Tang, T., Wang, B., & Tang, Z. (2022). BEVFusion: A simple and robust LiDAR-camera fusion framework. In NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "shMs66H2iJ", "reviewer": "Reviewer_qpE6", "summary": "The paper proposes a lane detection method based on transformers and utilises a hierarchical refinement of lane queries. The paper uses a high-to-low refinement strategy instead of the traditional low-to-high refinement, which saves the computation cost in transformer attention. The evaluation of three datasets shows consistent but marginal accuracy improvements while delivering a higher speed.", "questions": "Currently, only one LATR module is used for each pyramid level. It is advisable to add more such modules to assess the performance, i.e. whether more LATR modules are redundant or improve the performance. \n\nSuggestion: I believe adding more LATR modules to high-resolution would quadratically increase the computations due to the O(N^2) complexity of the self-attention and cross-attention. Therefore, multiple LATR modules can be used at the lowest or second lowest feature level. It would be great to see the results during the rebuttal.", "limitations": "The primary limitation of the paper is marginal improvements in the CULane dataset. Although this is shadowed by the improved FPS, considering the importance of the results, it is advisable to tune hyperparameters or add more decoder layers to deliver a stand-out performance.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The hierarchical design for the application of lane detection is commendable. Ignoring the application and refining queries only from a decoder instead of the encoder-decoder mechanism of DETR is very interesting.\n\n2. The method's design seems inspired by DETR pipelines, and utilizing this idea of lane detection is interesting.\n\n3. The proposed Curve-IoU is again a robust objective and metric to supervise the training when the existing L-IoU fails to differentiate the proximity of two lanes w.r.t. the ground truth.\n\n4. Evaluation is extensive across three datasets and a variety of baselines.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! The detailed responses to each concern are given below.\n\n**[Currently, only one LATR module is used for each pyramid level. It is advisable to add more such modules to assess the performance, i.e. whether more LATR modules are redundant or improve the performance.]**\n\nWe add a LATR module for each pyramid level, and the results are shown below.\n\nNumber of LATR modules  | F1 score on CULane \n:-----: | :-----:\n1 | 80.01\n2 | 80.32\n3 | 80.43\n\nFrom the results, one can see that adding more LATR modules helps improve the accuracy of lane detection recognition. Considering the efficiency, we think using two LATR modules for each pyramid level is more appropriate.\n\n**[Multiple LATR modules can be used at the lowest or second lowest feature level. It would be great to see the results during the rebuttal.]**\n\nWe add 2 LATR modules for the lowest and second lowest feature levels. The results are shown below.\n\nLayer Level (each 2 LATR module)  | F1 score on CULane  |  # params\n:-----: | :-----: | :-----:\n1 | 80.10 | 29.186202 M\n2 | 80.15 | 29.186202 M\n1&2 | 80.22 | 29.591786 M\n\nIt can be seen from the results that more LATRs added at the lower feature levels can improve performance, while not leading to much increase in parameters. Hence, we think it is effective to add two LATR modules on both the lowest or second lowest feature levels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gFt6egEHG0", "reviewer": "Reviewer_oCFK", "summary": "The paper proposes a Siamese Transformer with hierarchical refinement, named LAne TRansformer (LATR), to enhance lane detection accuracy in complex road environments. LATR integrates global semantic information with finer-scale features using a high-to-low hierarchical refinement structure. Additionally, the paper introduces a Curve-IoU loss to better supervise the fitting of lane lines, leveraging their thin and long characteristics.", "questions": "My most concerned questions are about the rationality of the motivation and novelty of the proposed method. Besides, the writing of this paper should be thoroughly improved.", "limitations": "Not applicable.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1) The proposed method achieves relatively high performance on Openlane dataset.\n2) The proposed Curve-IoU is effective in performance improvement.", "weaknesses": "1) The motivation of this paper is not very clear. Moreover, the hierarchical architecture is not a particularly innovative idea, as it has been demonstrated in lane detection (e.g., CLRNet), image segmentation (e.g., Mask2Former), and even earlier works like FPN. Additionally, it is unclear why the authors refer to several shared-parameter transformer layers as a Siamese transformer, as this seems to be merely a case of parameter reuse.\n2) The proposed high-to-low transformer in this paper is very similar to Mask2Former, thus offering limited novelty.\n3) The paper conducts a quantitative comparison with CondLSTR on the CULane dataset.\n4) The paper uses a stronger image backbone (Swin vs. ResNet) compared to previous methods, making the comparison unfair. To provide a fair comparison, the authors should replace the backbone of previous methods (e.g., CLRNet) with Swin and then compare the results.\n5) The writing quality of the paper is poor. The structure of the methodology section is disorganized, making it very difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback! We address your questions and concerns below. If any other concerns remain, we’ll be happy to discuss them further. If the concerns are addressed well, we would appreciate it if you could consider raising your score.\n\n**[The motivation of this paper is not very clear.]**\n\nAttention-based methods (e.g., CLRNet) have shown promising capability in lane detection. However, since these methods mainly rely on finer-scale information to identify the position of each key point, their detection results may have large deviations when there are local occlusions or blurring. Furthermore, due to the shortcut to the multi-head self-attention mechanism which neglects the characteristics of different frequencies, there is a gap in accuracy between the contemporaneous Transformer-based methods and CNN-based methods. To tackle these issues, we propose a high-to-low hierarchical refinement Transformer structure to refine key points of lane lines, which integrates global semantics information and finer-scale features.\n\n**[It is unclear why the authors refer to several shared-parameter transformer layers as a Siamese transformer, as this seems to be merely a case of parameter reuse.]**\n\nTo fully integrate global semantics information and finer-scale features, we propose a high-to-low hierarchical refinement Transformer structure for lane detection with shared parameters, which helps identify key points especially when roads are crowded or affected by blurring. The high to low features extracted by the backbone are fed into the Transformer structure with shared parameters and we calculate the loss for all the layers, which is the same definition as the Siamese structure [1][2]. As shown in Fig.5 of the manuscript, our proposed method can refine the key points of lane lines from higher levels to lower levels with fewer parameters by using shared parameters.\n\n**[The proposed high-to-low transformer in this paper is very similar to Mask2Former, thus offering limited novelty.]**\n\nOur method is different from Mask2Former. Indeed, both methods use hierarchical features as input to the Transformer. However, Mask2Former inputs multi-scale features into a Transformer decoder, which includes 3 transformer blocks, resulting in a large number of parameters and low efficiency. Instead, we feed information from each layer to the Transformer block with shared parameters and design the LATR structure to unify the feature information of different scales. This design not only enables the network to learn the feature information of different scales from high to low but also maintains a low number of parameters and high network efficiency.\n\n**[The paper uses a stronger image backbone (Swin vs. ResNet) compared to previous methods, making the comparison unfair. To provide a fair comparison, the authors should replace the backbone of previous methods (e.g., CLRNet) with Swin and then compare the results.]**\n\nFor the choice of backbone network, we found that the Swin Transformer with hierarchical features can better capture image features for subsequent processing. Thus, we chose Swin Transformer as our backbone network, which is an open-source backbone widely used by other works. Accordingly, we designed the LATR and Siamese Transformer structures to reduce the overall number of network parameters. The results show that our approach achieves better results with higher FPS and lower GFlops.\nWe also replace the backbone of the previous methods (e.g., CLRNet) with Swin Transformer and train these models in the same setting. The results are shown below.\n\nBackbone  | Method  |  F1 score on CULane\n-------- | -----  | :-----:\nSwin-tiny | CondLane | 77.16\nSwin-tiny | CLRNet     | 79.05\nSwin-tiny | ours           | 80.01\nSwin-base | CondLane | 77.84\nSwin-base | CLRNet     | 79.73\nSwin-base | ours           | 80.85\n\nThe results show that our method performs better with the same backbone compared with previous methods. We’ll add these results to the final vision of our paper if accepted.\n\n**[The writing quality of the paper is poor.]**\n\nThank you for your advice. We’ll improve the writing of the paper.\n\n**Reference**\n\n[1] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional Siamese networks for object tracking. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14, pages 850–865. Springer, 2016.\n\n[2] He A, Luo C, Tian X, et al. A twofold siamese network for real-time object tracking[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4834-4843.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary

The paper introduces LATR, a Siamese Transformer model with hierarchical refinement for lane detection. LATR combines global semantic information with finer-scale features to accurately detect lanes in challenging scenarios like occlusions and poor lighting. It also introduces a novel Curve-IoU loss function for curved lane lines. The method demonstrates state-of-the-art performance on multiple benchmark datasets, particularly excelling on the OpenLane dataset. The study shows consistent but marginal accuracy improvements while delivering higher speed.

##Strengths
-  The Siamese Transformer architecture with hierarchical refinement and the Curve-IoU loss are improvements to the field.
- The proposed method shows superior performance on multiple benchmark datasets, demonstrating its robustness in challenging scenarios.
- The proposed method achieves high performance on the Openlane dataset and the Curve-IoU is effective in performance improvement.

## Weaknesses
- Further research on data augmentation or self-supervised learning techniques could improve robustness.
- Current evaluation results are promising, but 2D lane detection is less valuable than 3D lane detection.
- The paper lacks innovation compared to existing 2D lane detection networks.
- The authors only reported results under the Swin Transformer, raising questions about the fairness of the SOTA comparison.

## Recommendation

Although the paper should include all the comments addressed during rebuttal period, the paper should be accepted based on the results shown during rebuttal in terms of accuracy, fps and Gops and  the positive comments of reviewers.
Decision: Accept (poster)