A Siamese Transformer with Hierarchical Refinement for Lane Detection

Authors: zinan  lv
dong  han
wenzhe  wang
danny z chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 11:52:37

Abstract: Lane detection is an important yet challenging task in autonomous driving systems. Existing lane detection methods mainly rely on finer-scale information to identify key points of lane lines. Since local information in realistic road environments is frequently obscured by other vehicles or affected by poor outdoor lighting conditions, these methods struggle with the regression of such key points. In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in complex road environments. Specifically, we propose a high-to-low hierarchical refinement Transformer structure, called LAne TRansformer (LATR), to refine the key points of lane lines, which integrates global semantics information and finer-scale features. Moreover, exploiting the thin and long characteristics of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines. Extensive experiments on three benchmark datasets of lane detection demonstrate that our proposed new method achieves state-of-the-art results with high accuracy and efficiency. Specifically, our method achieves improved F1 scores on the OpenLane dataset, surpassing the current best-performing method by 5.0 points.

Main paper: 
 introduction: 
 Lane detection is a fundamental task in Autonomous Driving Systems (ADS), which enables a vehicle to localize its relative position and avoid potential risks. It plays an important role in many downstream tasks, such as driving route planning, road tracking, and adaptive cruise control. Recently, lane detection methods based on computer vision have attained lots of achievements. Compared to methods that combine GPS/INS (Inertial Navigation System) [2] , lane detection methods that incorporate only a camera are cheaper and safer to apply.Early lane detection research focused on hand-crafted features and applied methods such as Hough Transform [16] or Kalman Filter [36] to filter out unreasonable lanes. However, manually extracted features often fail in complex scenarios. Convolutional Neural Network (CNN) based methods [10, 13, 38, 39] arose to cope with different scenarios, which greatly improved the accuracy of lane detection. These methods often rely on straight-line anchors [12, 15, 31, 32, 35] or parametric curves [18, 33, 9, 4] to detect lane lines. Although CNN-based methods can handle different scenarios, they still struggle with realistic road environments, especially when involving strong light, shadows, or dense traffic. Due to the thin and long characteristics of lane lines, lane detection requires a lot of contextual information, and CNN-based methods may be ineffective and incapable of clustering global semantics information and combining it with finer-scale features.Recently, attention-based methods (e.g., CLRNet [41] ) have shown promising capability in lane detection by describing a lane line as a series of key points. By taking the lane lines as a whole unit, these methods can make use of global semantics information and finer-scale features, which help the detection of lane lines in complex environments. However, since these methods mainly rely on finer-scale information to identify the position of each key point, their detection results may have large deviations when there is local occlusion or blurring. In addition, former studies did not take into account the thin and long structure of lane lines during supervision, thus often leading to inaccurate detection when the lane line has a certain curvature.In this paper, we propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Two attributes of our proposed method contribute to its universality for the lane detection task. First, to address the under utilization of finer-scale information when fine-tuning the positions of key points, we propose a novel Siamese Transformer structure with shared parameters, called LAne TRansformer (LATR), which can integrate global semantics information and finer-scale features. Simultaneously, we develop a high-to-low hierarchical refinement scheme to refine the key points of lane lines so that the network can fully learn information at different scales. Second, to take the thin and long structure of lane lines into account, we propose a novel Intersection over Union (IoU) loss called Curve-IoU (CIoU) for lane detection. Compared to the common IoU loss, we supervise the fit of lane lines at different locations for the thin and long structure of the lane lines, which helps accurately detect lane lines with curves.We evaluate our proposed method on three benchmark datasets for lane detection, OpenLane [5] , CULane [26] , and Tusimple [34] , and achieve state-of-the-art results. Our main contributions can be summarized as follows:• We propose a novel Siamese Transformer with hierarchical refinement for lane detection to improve the detection accuracy in realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.• We propose a high-to-low hierarchical refinement Transformer structure called LATR to refine the key points of lane lines so that the network can fully integrate global semantics information and finer-scale features.• Exploiting the thin and long structure of lane lines, we propose a novel Curve-IoU loss to supervise the fit of lane lines at different locations, which helps the regression of the curves.• We achieve state-of-the-art results on three benchmark datasets, with 5.0% improvement in F1 score compared to the best-known method on the OpenLane dataset.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction clearly illustrate our proposed model LATR, and precisely show the experiment results we have achieved.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Early lane detection studies relied on hand-crafted features [7, 23] . Due to their limited feature capturing capability and low robustness, these methods often fail in complex conditions.To improve the robustness of lane detection under different environments, segmentation-based methods [10, 13, 38, 39] were introduced to lane detection. These methods typically apply postprocessing operations such as curve fitting and clustering on pixel-level segmentation maps to generate final results. Compared to traditional methods, segmentation-based methods are able to capture more plentiful visual features and spatial structure information, thus achieving better performance than traditional detection methods. However, per-pixel-based segmentation methods incur high computational costs, have limited real-time capability, and struggle with learning lane line specific long and thin characteristics.To address these issues, LaneNet [25] introduced a branched, multi-task architecture to cast the lane detection task as an instance segmentation problem. This method is more robust to variations in road conditions compared to the previous methods, but it is more time-consuming. RESA [40] proposed to aggregate spatial information by shifting sliced feature maps, which obtains good real-time results but still fails under complex road conditions. Furthermore, the output lane lines of most of the above methods may not be continuous.To attain more continuous lane lines with higher efficiency, recently curve-based methods [18, 9, 4, 33] viewed the lane detection task as a polynomial regression problem and utilized parametric curves to fit lane lines. These methods depend heavily on the parameters of the curves (e.g., x = ay 3 +by 2 +cy+d, where (x, y) denotes the coordinates of a lane line pixel and a, b, c, and d are the parameters of a curve function). PloyLaneNet [33] first proposed an end-to-end deep polynomial regression method that directly outputs the parameters. To improve the stability and efficiency, BézierLaneNet [9] proposed a parametric Bézier curve to model the geometric shape of lane lines. However, because of the limited learning ability of global information, the accuracy of these curve-based methods is not satisfactory on large datasets, especially in complex road conditions.Attention-based methods [32, 18] were introduced to the computer vision field, and were proven to be able to capture long-range information. LaneATT [32] introduced an attention mechanism to anchor-based lane detection methods. Based on PolyLaneNet [33] , LSTR [18] was proposed with high inference efficiency but relatively low accuracy, especially in some complex road environments. PriorLane [30] improved the accuracy of prediction compared to LSTR with pre-training and local prior. However, there is still a gap in accuracy between the contemporaneous Transformer-based methods and CNN-based methods, and the reason can be attributed to the shortcut of the multi-head self-attention mechanism which neglects the characteristics of different frequencies [28] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper thoughtfully discusses the limitations of the research in Section 5.We evaluate the robustness of our proposed method on three datasets in varying scenarios, and before that, we make assumptions and declare the inference details thoroughly in Section 3.4.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we present the proposed Siamese Transformer with hierarchical refinement for lane detection. First, we describe the overall architecture of our proposed network. Then we explain each key component of the proposed network, including the LAne TRansformer (LATR) and the proposed Curve-IoU loss. Finally, we provide the inference details of our network.We present the overall architecture of our proposed method in Fig. 1 . It consists of a multi-scale backbone and a Siamese Transformer structure named LATR. An input image R H×W ×C is first fed into the backbone to obtain hierarchical features from high to low levels, which are then refined by LATR with the supervision of the Curve-IoU loss. Next, we use two different detection feedforward networks (FFN) to generate (1) lane line properties including the start point (x 0 , y 0 ), angle of inclination θ, and length l of each lane line, and (2) the offset map {o i } P i=1 . Finally, key points of the lane line are produced by post-processing, which can be expressed as:EQUATIONwhere i denotes the i-th key point of the lane line. All the key points are sampled at equal intervals based on the y-axis, which can be expressed as y i = i H P +1 , where H and P represent the image height and the number of key points, respectively.Hierarchical featureLane query from the higher-scale We employ a high-to-low refinement structure in which the input lane query is refined by higher-scale features.Transformer structure. Our proposed LATR employs the Siamese Transformer structure based on the hierarchical features extracted by the backbone of the network. Inspired by the recent Transformer-based networks with Siamese structures [1, 14] , we develop a high-to-low refinement structure to obtain features of the predicted lane line. The detailed structure of LATR is illustrated in Fig. 2 .Specifically, we denote a lane query asQ d = {q j } N j=1, where N represents the dimension of the decoder embeddings and d is the number of the scales. The lane query is a series of one-dimensional features of the lane lines, which can be further processed to obtain the key points of the lane lines. The input hierarchical featuresP d ∈ R H d ×W d ×C ′are extracted by the multi-scale backbone (we use Swin Transformer [19] for it), and then are sorted from high-level to low-level. Next, we flatten it intoM d ∈ R H d W d ×C ′with 2-D positional embedding.Different from DETR [3] , we integrate the lane query in the Transformer encoder and then employ Deformable Attention [42] as cross-attention in the decoder process. The whole process can be expressed as:EQUATIONwhere Q ′ d and M ′ d represent the lane query and high-to-low hierarchical features after positional embeddings respectively, DA() denotes the Deformable Attention, and z d represents the sequence length.Compared with DETR, our proposed Siamese Transformer integrates the lane query during the encoder. The multi-scale features from the backbone have different shapes. Note that integration during the encoder can unify the decoder's input scales with less information loss. When the scales are unified, the network can be made into a Siamese structure with shared parameters, which leads to a big efficiency improvement.High-to-low refinement structure with shared parameters. Recent studies have shown that Vision Transformer (ViT) is good at extracting low-frequency information (in other words, global semantics information) from images [27] . However, lane lines are characteristically thin and long, and their accurate detection often requires integrating finer-scale features and global semantics information. High-level information can help determine the structural information of lane lines while low-level information can help adapt key points of the lane lines. Therefore, we adopt a high-to-low refinement structure to refine the lane query. Given hierarchical featuresP d ∈ R H d ×W d ×C ′ from the multi-scale backbone, we first flatten it into M d ∈ R H d W d ×C ′with 2-D positional embedding. Then we employ cross-attention to integrate M d with the lane query into S d ∈ R N ×C ′ . Since S d shares the same shape, the same parameters can be used during refinement between multi-scales. By refining the lane query from high to low, the network can adapt the key points of the lane lines from coarse to fine, which helps the Transformer learn high-frequency information of the image. After the refinement, we employ a detection FFN to generate the lane anchor and finally attain the predicted lane lines with bi-linear interpolation between the key points.We represent a lane line as a sequence of key points, and we need a loss to supervise the regression of lane lines. In previous work, LIoU [41] was proposed as a similarity function to calculate the distance between predicted key points and the ground truth. Although the LIoU loss ensures the scale consistency of the lane shape, it does not match the L1 distance when there is a long distance between the prediction lane line and the ground truth. Fig. 3 (a) shows a typical curve scenario that LIoU cannot describe correctly. In such cases, lane lines are inclined and curved, and the LIoU loss is unable to judge which prediction lane line is closer to the ground truth. In Fig. 3 , the values for Prediction A and Prediction B are -0.6 and 0.6, respectively. However, the L1 distances for them are 8e and 1.5e, respectively.To bridge this gap, we propose Curve-IoU (CIoU). Different from the traditional IoU, we add a penalty term to IoU so that the spatial distance between each pair of lane lines will be represented more accurately, especially for those with a long distance between the prediction lane line and the ground truth. We use a sequence of points with a certain width e to present a lane line and calculate the Intersection over Union, which can be written as:EQUATIONwhere P l i = x i -e and P r i = x i + e, and G l i and G r i are defined similarly. IoU can be calculated asIoU = d o i d u i. Then we add the L1 distance between each corresponding pair of lines as the penalty term to the overlap ranges, as:EQUATION)where j is the first valid point in the lane line. Then the CIoU loss can be calculated as CIoU = 1 -L IoU .Exploiting the thin and long structures of lane lines, our CIoU loss can supervise the regression of lane lines better. As shown in Fig. 3 (b), the values of LIoU and CIoU for Prediction A are -0.6 and -1.6 respectively, which means that our CIoU can handle these scenarios more precisely.In order to reduce the training costs, we generate only the lane anchor during training. The loss is calculated by comparing the lane anchor to the ground truth. Our method generates a fixed number of lane anchors, which is much larger than the maximum number of lane lines in the image. Besides the Curve-IoU loss, we also utilize a classification accuracy loss to determine whether the lane anchor is a lane line or the background. The classification accuracy loss can be described as:EQUATIONwhere L(•, •) denotes the cross-entropy loss and L is the number of sequences. If the i-th predicted lane line matches the ground truth, then g i = 1 (0 otherwise). The objective function of our network can be represented as:EQUATIONwhere λ cls and λ reg are importance weight values. We supervise each generated lane anchor during training, while during inference we use Non-Maximum Suppression (NMS) [24] to exclude incorrect lane lines. Finally, we utilize bi-linear interpolation to produce the predicted lane lines of the image.  
experiments 4.1 datasets: 
 To demonstrate the effectiveness of our proposed method in realistic road environments, we evaluate it on three benchmark datasets: OpenLane [5] , CULane [26] , and Tusimple [34] . OpenLane is a real-world large-scale lane detection dataset, which contains 160K and 40K images as the training and validation sets, respectively. The validation set consists of six realistic road scenarios and annotates 14 lane categories (including white dotted line, double yellow solid, left/right curb, and so on). CULane is a widely-used large dataset for lane detection including eight hard-to-detect scenarios in urban areas and on highways, with 88K and 34K images as the training and validation sets, respectively. Tusimple is also a widely-used dataset with images collected on US highways under clear weather, which contains 3K images for training and 2K for validation.For OpenLane [5] and CULane [26] , we adopt the F1-score measure proposed by SCNN [26] as the evaluation metric. Intersection-over-Union (IoU) between the ground truth (GT) label and each predicted lane line of the model is calculated to determine whether a sample is True Positive (TP), False Positive (FP), or False Negative (FN). The ways to calculate IoU and F1-score can be found in the supplemental material. For Tusimple [34] In the experiments, we adopt the Swin Transformer [19] as the pre-trained backbone of our network.We divide the versions of the backbone of the network based on the size of the Swin Transformer into three categories: tiny, small, and big, which are consistent with the work in [19] . For data augmentation, we adopt the affine transformation method (horizontal flip, rotation), brightness, and saturation addition method. All the input images are reshaped into 800 × 320 pixels each for both the training and inference stages. For the number of lane anchors in an image, we set it to 150. In the optimization process, we adopt AdamW [21] and the cosine decay learning rate strategy [20] with the initial learning rate set to 6e-4. A batch size of 32 and training epoch numbers of 10, 20, and 90 are used for OpenLane, CULane, and Tusimple, respectively. All the experiments are conducted on a machine with a single NVIDIA RTX3090 GPU with 24GB memory.Comparisons of results by recent methods and our method on the OpenLane dataset are shown in Table 1 . Our method achieves state-of-the-art results in F1 score. Specifically, our method achieves F1 scores of 68.3, 69.1, and 69.7, surpassing those of the best-known method CondLaneNet by 5.0, 3.5, and 1.9 points, respectively. Further, our method achieves the best performances in four out of six scenarios, showing the robustness of our method. Among these, the "Up & Down" and "Curve" categories are 7.2% and 1.1% higher than the previous best results, respectively. These results demonstrate that the proposed Siamese Transformer with the hierarchical refinement structure can deal with lane lines in realistic road environments very well. This is because our proposed LATR integrates global semantics information and finer-scale features, which help refine key points when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. Simultaneously, the proposed Curve-IoU can supervise the regression of curves well, which helps accurate detection of lane lines.We compare the results by recent known methods and our method on the CULane dataset in Table 2 .In the total F1 score, our method achieves state-of-the-art results, with 0.38% improvement over the best-known CLRNet. Among all the eight difficult scenarios, our method achieves the best results on five of them. Specifically, for the "Crowded" and "Hlight" scenarios, our method achieves 80.21% and 76.04% F1 scores respectively, surpassing those of CLRNet by 0.62% and 0.74% points, respectively. These results demonstrate that our method can deal with realistic road environments, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions.Fig. 5 shows some visual results of several known methods and our method on the CULane dataset.CondLaneNet is an open-source dynamic convolution-based method and CLRNet is the second best-performing method on CULane. The results of CondLaneNet and CLRNet are generated with ResNet18 as the backbone and ours are generated with Swin Transformer tiny as the backbone. The visual results show that when the roads are crowded and lane lines are covered by other vehicles, our method can detect lane lines better and the lane regression is closer to the ground truth. What's more, our method also outperforms these methods in night scenarios, proving that our method can adapt to different lighting environments with good robustness.Comparison results of recent methods and our method on the Tusimple dataset are given in Table 3 . This dataset consists of images captured in different weather conditions. Our method achieves state-of-the-art results in F1 score, Accuracy, and False Positive Rate (FP), demonstrating that our method can be adapted to both complex urban environments and simple highway scenarios. Note that because the dataset was collected on US highways where the road conditions are relatively simple (lane features are more obvious and clear), the results of various methods are close. To fully integrate global semantics information and finer-scale features, we propose a high-to-low hierarchical refinement Transformer structure for lane detection called LAne TRansformer (LATR), which helps identify key points especially when roads are crowded or affected by blurring. Fig. 4 shows some high-to-low attention maps of our LATR. From Fig. 4 , one can see that attention is gradually focused on key points on both sides of the road from high to low. High-level attention is extended along the road from the near end to the far end of the road, initially defining the overall structure of the lane lines. Low-level attention focuses on finer-scale features of the lane lines, which refine the key points of the lane lines from higher-levels to lower-levels.We compare the F1 scores of our Lane Transformer (LATR) and existing Transformer structures with the same backbone Swin Transformer tiny on CULane, and report the results in To help the regression of lane lines, we propose the Curve-IoU loss. In Table 5 , we compare our Curve-IoU with different types of IoU for lane detection. "w/o IoU" denotes optimizing the network with no IoU loss, using only classification and L1 regression loss, which only yields an F1 score of 77.16%. LIoU was proposed by CLRNet as a similarity function to calculate the distance between the prediction and ground truth. However, LIoU does not match the L1 distance when there is a long distance between the prediction lane line and ground truth. Compared with LIoU, our Curve-IoU takes the L1 distance into account, which helps the regression of lane lines with curves. On the CULane dataset, Curve-IoU achieves an F1 score of 80.01%, which is 2.85% higher than w/o IoU and 0.77% higher than LIoU. Its effect on Transformer-based lane detection is validated. 
discussion: 
 Our method achieves state-of-the-art results on three benchmark datasets and performs the best among all the Transformer-based methods, especially in some challenging and complex road conditions. However, we still find that the current known anchor-based methods are competitive in the "Shadow", "Noline", and "Arrow" scenes on the CULane dataset, which is the opposite of the OpenLane dataset. Also, the improvement of our method is larger on the OpenLane dataset. For this interesting phenomenon, we assmue the reason behind it is that the CULane dataset contains 88K images for training while the Openlane dataset contains 160K, which is almost twice that of CULane and includes more complex and various scenes. Our Transformer-based method performs relatively better on a larger dataset compared to anchor-based and CNN-based methods, that is, our method has a much bigger potential for scaling up if the data and computation resources are sufficient. Considering the current state of affairs in vision-based autonomous driving systems, deep learning algorithms are advancing and models are expanding, yet the systems are still constrained by the size and diversity of datasets. We call for larger and better datasets featuring more diverse scenes for further research.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We will release our code and checkpoint of this paper, and all the datasets we use are open access. The appendix includes extensive experiments to prove our results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusions: 
 In this paper, we proposed a novel Siamese Transformer structure with hierarchical refinement, which achieves state-of-the-art results on three benchmark datasets. Specifically, we developed a high-to-low hierarchical refinement Transformer structure called LATR to refine key points of lane lines, which compensates for the Vision Transformer's deficiency in extracting finer-scale features. Also, we proposed a novel Curve-IoU loss tailored for the long and thin shape of lane lines, which helps supervise the regression of lane lines with different offsets. Extensive experiments confirmed that our model effectively handles complex scenarios, particularly when lane lines are heavily obscured by other vehicles or compromised by poor lighting conditions.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The details are all illustrated thoroughly in Section 4, and our code will release.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We display the experiment results in Section 4 which can explain the variance, outliers, and anomalies, and we illustrate all the evaluation metrics we use in Section 4.2 and A Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provides detailed descriptions of the computational resources used for each experiment in Section 4Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We comprehensively discuss the broader societal impacts of the research in this paper, and no significant negative impacts are found.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not pose any such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This paper cites the original creators of all utilized assets, including code and datasets. The licensing information of our code is clearly outlined, ensuring full compliance with legal and ethical standards.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For OpenLane [5] and CULane [26] , we adopt the F1 score proposed by SCNN [26] as the evaluation metric. Intersection-over-Union (IoU) between the ground truth (GT) and the predicted lane line of the model is calculated to determine whether a sample is True Positive (TP), False Positive (FP), or False Negative (FN). The IoU and F1 score are calculated as in the following formulas:EQUATIONEQUATIONEQUATION)For Tusimple [34] , the evaluation metrics are composed of three official indicators: accuracy, False Positive Rate (FPR), and False Negative Rate (FNR). The accuracy is calculated as:EQUATION)where C clip is the number of correct points and S clip is the number of ground truth (GT) points in an input image. If the accuracy of a predicted lane is greater than 85%, it will be considered a True Positive (TP). F1 score is also used in the evaluation.To verify the roles of our proposed lane Transformer (LATR) and Curve-IoU in our method, we carry out an overall ablation study with the same baseline LSTR [18] . The results of the ablation study are shown in Table 6 . LATR can greatly improve the detection accuracy of lane lines, which improves the F1 scores with 8.38% and 6.45% increase on OpenLane and CULane, respectively. This is because our proposed LATR can better integrate global semantics information and finer-scale features, especially when roads are obscured by other vehicles or affected by poor outdoor lighting conditions. What's more, our proposed Curve-IoU further improves the F1 scores by 1.23% and 1.66% on OpenLane and CULane, respectively. These results show that our proposed Curve-IoU can improve the accuracy of lane detection, especially for roads with curves. Our method generates a fixed number of lane anchors, much more than the maximum number of lane lines in the image. Here we conduct an ablation study on the number of lane anchors. As shown in Table 7 , when we increase the number of lane anchors from 50 to 150, the accuracy of lane detection on both OpenLane and CULane is improved. This demonstrates that increasing the number of lane anchors is beneficial to the detection of lane lines. However, when we further increase the number of lane anchors, the enhancement is not obvious or even appears to be decreasing. This is because when the number of lane anchors reaches a certain value, adding more lane anchors will cause redundancy. 150 lane anchors are enough for the network to detect lane lines in the image.In order to evaluate the bias caused by the backbone net, we conduct ablation experiments on backbones. We find that the Swin Transformer with hierarchical features can better extract image features for subsequent processing. Therefore, we choose the Swin Transformer as our backbone network, which is an open-source backbone widely used by other works. The results show that our approach achieves better results with higher FPS and lower GFlops. We also replace the backbone of previous methods (e.g., CLRNet) with Swin Transformer and train these models under equal situations. The results are shown below in Table 8 . We only use a single LATR module for each pyramid level, to find out the potential scaling-up ability of our model, we add more layers to each level, and the results are shown in Table 9 and 10. It can be seen from the results that more LATR added at the lower feature levels can improve performance, while not leading to much increase in parameters. As a result, we think it is effective to add two LATR modules at both the lowest and second-lowest feature levels. More visualization results of Ground Truth (GT), CondLaneNet [17] , CLRNet [41] , and our proposed method on the CULane dataset [26] are shown in Fig. 6 . Compared with CondLaneNet and CLRNet, our method is more effective in detecting lane lines on congested roads and in different lighting environments, which demonstrates the robustness of our method.NeurIPS Paper ChecklistJustification: We have thoroughly examined the ethical guidelines and legal considerations, and confirm that all study aspects obey the rules.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: ## Summary

The paper introduces LATR, a Siamese Transformer model with hierarchical refinement for lane detection. LATR combines global semantic information with finer-scale features to accurately detect lanes in challenging scenarios like occlusions and poor lighting. It also introduces a novel Curve-IoU loss function for curved lane lines. The method demonstrates state-of-the-art performance on multiple benchmark datasets, particularly excelling on the OpenLane dataset. The study shows consistent but marginal accuracy improvements while delivering higher speed.

##Strengths
-  The Siamese Transformer architecture with hierarchical refinement and the Curve-IoU loss are improvements to the field.
- The proposed method shows superior performance on multiple benchmark datasets, demonstrating its robustness in challenging scenarios.
- The proposed method achieves high performance on the Openlane dataset and the Curve-IoU is effective in performance improvement.

## Weaknesses
- Further research on data augmentation or self-supervised learning techniques could improve robustness.
- Current evaluation results are promising, but 2D lane detection is less valuable than 3D lane detection.
- The paper lacks innovation compared to existing 2D lane detection networks.
- The authors only reported results under the Swin Transformer, raising questions about the fairness of the SOTA comparison.

## Recommendation

Although the paper should include all the comments addressed during rebuttal period, the paper should be accepted based on the results shown during rebuttal in terms of accuracy, fps and Gops and  the positive comments of reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduces LATR (LAne TRansformer), a Siamese Transformer model with hierarchical refinement for lane detection. The model effectively combines global semantic information with finer-scale features to accurately detect lanes, even in challenging scenarios like occlusions and poor lighting. It also introduces a novel Curve-IoU loss function specifically tailored for the curved nature of lane lines. The proposed method demonstrates state-of-the-art performance on multiple benchmark datasets, particularly excelling on the OpenLane dataset.
strength_weakness: strength_weakness: The authors acknowledge that the model's performance is somewhat dependent on the size and diversity of the training dataset. This is a common limitation in deep learning-based approaches and could be addressed with further research on data augmentation or self-supervised learning techniques. 

While the method excels in the tested scenarios, further testing in diverse real-world environments would strengthen the claims of robustness. However, this is not a major weakness as the current evaluation results are already very promising.
questions: questions: Have you considered using self-supervised learning techniques to reduce the dependency on large labeled datasets?
How does the model perform in extreme weather conditions (e.g., heavy rain, snow) that can significantly affect lane visibility?
Are there any plans to release the code and trained models to facilitate reproducibility and further research?
limitations: limitations: The authors adequately address the limitations of their work, acknowledging the data dependency issue and the need for further testing in diverse real-world environments. However, in this field of work, there are a lot of other works that have various implementations exceeding the results from CondLaneNet such as ERF-Net. Looking into that might be of interest to better compare the proposed method.
paper_summary: paper_summary: This paper introduces a novel Siamese Transformer with hierarchical refinement for lane detection. The core innovation is a high-to-low hierarchical refinement Transformer structure, LATR, which refines lane line key points to integrate global semantic information and finer-scale features fully. Additionally, it proposes a Curve-IoU loss to supervise the fitting of lane lines at various locations.
strength_weakness: strength_weakness: From the perspective of the topic selection, 2D lane detection is less valuable than 3D lane detection. More importantly, this topic is not fully suitable for the NeurIPS conference.

Compared to existing 2D lane detection networks, this paper lacks innovation. Its overall framework is quite similar to CLRNet, and the proposed structured Loss has also been preliminarily explored in CLRNet and UFAST networks.

More importantly, current lane detection research tends to use simple CNN-based backbones (for ease of deployment) and to better compare performance fairly, especially using the ResNet series (18, 34, etc.). However, the authors only reported results under the Swin Transformer, which makes me skeptical about the fairness of the SOTA comparison.

Additionally, in terms of efficiency, the FPS speed does not have an advantage, and the model size (params) is not reported.
questions: questions: - Suggest enhancing the method's innovativeness.
- Recommend using common public backbones for ablation studies, though this might make it more similar to CLRNet.
limitations: limitations: None
paper_summary: paper_summary: The paper proposes a lane detection method based on transformers and utilises a hierarchical refinement of lane queries. The paper uses a high-to-low refinement strategy instead of the traditional low-to-high refinement, which saves the computation cost in transformer attention. The evaluation of three datasets shows consistent but marginal accuracy improvements while delivering a higher speed.
strength_weakness: strength_weakness: See questions.
questions: questions: Currently, only one LATR module is used for each pyramid level. It is advisable to add more such modules to assess the performance, i.e. whether more LATR modules are redundant or improve the performance. 

Suggestion: I believe adding more LATR modules to high-resolution would quadratically increase the computations due to the O(N^2) complexity of the self-attention and cross-attention. Therefore, multiple LATR modules can be used at the lowest or second lowest feature level. It would be great to see the results during the rebuttal.
limitations: limitations: The primary limitation of the paper is marginal improvements in the CULane dataset. Although this is shadowed by the improved FPS, considering the importance of the results, it is advisable to tune hyperparameters or add more decoder layers to deliver a stand-out performance.
paper_summary: paper_summary: The paper proposes a Siamese Transformer with hierarchical refinement, named LAne TRansformer (LATR), to enhance lane detection accuracy in complex road environments. LATR integrates global semantic information with finer-scale features using a high-to-low hierarchical refinement structure. Additionally, the paper introduces a Curve-IoU loss to better supervise the fitting of lane lines, leveraging their thin and long characteristics.
strength_weakness: strength_weakness: 1) The motivation of this paper is not very clear. Moreover, the hierarchical architecture is not a particularly innovative idea, as it has been demonstrated in lane detection (e.g., CLRNet), image segmentation (e.g., Mask2Former), and even earlier works like FPN. Additionally, it is unclear why the authors refer to several shared-parameter transformer layers as a Siamese transformer, as this seems to be merely a case of parameter reuse.
2) The proposed high-to-low transformer in this paper is very similar to Mask2Former, thus offering limited novelty.
3) The paper conducts a quantitative comparison with CondLSTR on the CULane dataset.
4) The paper uses a stronger image backbone (Swin vs. ResNet) compared to previous methods, making the comparison unfair. To provide a fair comparison, the authors should replace the backbone of previous methods (e.g., CLRNet) with Swin and then compare the results.
5) The writing quality of the paper is poor. The structure of the methodology section is disorganized, making it very difficult to follow.
questions: questions: My most concerned questions are about the rationality of the motivation and novelty of the proposed method. Besides, the writing of this paper should be thoroughly improved.
limitations: limitations: Not applicable.

Comments: title: Official Comment by Reviewer UDCn
Response to the authors
comment: - **Perspective**：Actually, the recent papers published in NeurIPS listed by the author are all about 3D lane detection rather than 2D lane detection.
- **Proposed Lane Loss**: The proposed lane loss lacks novelty, as similar losses have been introduced in works like CLRNet and UFAST. The issue of distant curve fitting has also been discussed in non-representative lane networks. Ablation studies show an average 0.7% improvement with Curve-IoU loss, but it's unclear if this is due to improved curve fitting, as there's no direct evidence from Curve category data in CULane. This is not rigorous.
- **Backbone**: In the field of lane detection, using a standard CNN Backbone (like ResNet) is still more popular. This is because lane detection is a practical industrial task for driver assistance, requiring models that are more convenient to deploy. Moreover, after replacing the backbone in CLRNet with Swin (author implementation), the F1 improvement is less than 1%, which is not a significant advantage.
- **Efficiency**: In terms of efficiency, I agree with the author’s response. While the parameter count is ordinary, the speed and GFLOPs performance are impressive.

In summary, the author's rebuttal still fails to convince me, especially since the paper appears to be a combination of Swin and the SOTA CLRNet. However, the experiments and performance are decent. I can raise my score by one point, but I don't think a significantly higher score is justified.
Thanks a lot for the response.
I thank the authors for the detailed response and the new experimentation.

Thank you for incorporating my suggestions into the experimentation which has improved the accuracy of the model at negligible parameter overhead.

Overall I am satisfied with the paper and author rebuttal. 

I have also read comments from other reviewers, primarily on the issue of replacing the backbones which authors have successfully defended. Hence I keep my original rating for the acceptance.