DV-3DLane: End-to-end Multi-modal 3D Lane Detection with Dual-view Representation
Authors: Yueru Luo
Shuguang Cui
Zhen Li
Venue: ICLR 2024 poster (2024)
Keywords: 3D Lane Detection
Multi-modal
Primary area: applications to robotics, autonomy, planning

Abstract: Accurate 3D lane estimation is crucial for ensuring safety in autonomous driving. However, prevailing monocular techniques suffer from depth loss and lighting variations, hampering accurate 3D lane detection. In contrast, LiDAR points offer geometric cues and enable precise localization. In this paper, we present DV-3DLane, a novel end-to-end **D**ual-**V**iew multi-modal **3D Lane** detection framework that synergizes the strengths of both images and LiDAR points. We propose to learn multi-modal features in dual-view spaces, *i.e.*, *perspective view* (PV) and *bird's-eye-view* (BEV), effectively leveraging the modal-specific information. To achieve this, we introduce three designs: 1) A bidirectional feature fusion strategy that integrates multi-modal features into each view space, exploiting their unique strengths. 2) A unified query generation approach that leverages lane-aware knowledge from both PV and BEV spaces to generate queries. 3) A 3D dual-view deformable attention mechanism, which aggregates discriminative features from both PV and BEV spaces into queries for accurate 3D lane detection. Extensive experiments on the public benchmark, OpenLane, demonstrate the efficacy and efficiency of DV-3DLane. It achieves state-of-the-art performance, with a remarkable 11.2 gain in F1 score and a substantial 53.5% reduction in errors. Code is available on [github](https://github.com/JMoonr/dv-3dlane).

Published as a conference paper at ICLR 2024
DV-3DLANE: END-TO-END MULTI-MODAL 3D LANE DETECTION WITH DUAL-VIEW REPRESENTATION
Yueru Luo!?, Shuguangg Cui Zhen Li
FNii, CUHK-Shenzhen School of Science and Engineering, CUHK-Shenzhen (222010057@link, shuguangeui e, Lizhene Jacannnecckk... cuhk. edu. cn
ABSTRACT
Accurate 3D lane estimation crucial for ensuring safety in autonomous driving. However, prevailing monocular techniques suffer from depth loss and lighting variations, hampering accurate 3D lane detection. In contrast, LiDAR points offer geometric cues and enable precise localization. In this paper, we present DV- 3DLane, a novel end-to-end Dual-View multi-modal 3D Lane detection frame- work that synergizes the strengths of both images and LiDAR points. We propose to learn multi-modal features in dual-view spaces, i.e. perspective view (PV) and bird's-eye-veee s-eye-view (BEV), effectively leveraging the modal speciffc information. To achieve this, we introduce three designs: 1) A bidirectional feature fusion strat- egy that integrates multi-modal features into each view space, exploiting their unique strengths. 2) A unified query generation approach that leverages lane- aware knowledge from both PV and BEV spaces to generate queries. 3) 3D dual-view deformable attention mechanism, which aggregates discriminative fea- tures from both PV and BEV spaces into queries for accurate 3D lane detection. Extensive experiments cacy and efficiency of DV-3DLane. achieves ttate-of-ie satte-of-the--e the-the-rt art performance. with remarkable 11.2 gain in F1 score and substantial 53.5 % reduction in errors. The code is available at https:///tthhtni com/ JMoonr/dv-- 3dlane)
the public benchmark, OpenLane, demonstrate the effi-
INTRODUCTION
Faster
Autonomous driving (AD) technology in recent years has made remarkable strides, bringing us closer to the realization of fully self-driving ve- hicles. Within this field, one of the key chal- lenges is the accurate detection of 3D lanes, critical component for ensuring safe and reli- able navigation. 3D lane detection entails iden- tifying the 3D positions of lane boundaries in the environment, providing essential data for tasks like path planning and vehicle control.
65-
Ours-Large
Ours
Ours-Tiny
F1 Score
55
LATR
M MDLaneee
45
Perser EnB?
PersFormer
35
Anchor31) bffktt
Better
Anchor3DLane Res18
30
10
12
14
16
FPS
3D lane detection is proposed to mitigate the limitations posed by the absence of depth infor- mation in 2D prediction. Currently, the major- ity of 3D lane detection methods rely on vision- centric approaches, ie., monocular solutions, where some designs are naturally borrowed and benefit from advances in 2D lane methods. Tak- (PV) image as input,
Figure 1: FPS vs. F1 score. All models are tested on single V100 GPU, and Fl-score is evalu- ated with harsh distance threshold of 0.5m on the OpenLane- dataset. Our model sets new -fate-of-the-art and our tiny version surpasses all previous methods with the fastest FPS. More de- tails can be found in Table and our Appendix.
ing the perspective-vies these monocular methods mainly utilize the inverse perspective mapping (IPM) Mallot et al. (1991) technique to warp the PV features into BEV. However, there are misalignment issues in the IPM- based methods when encountering non-flat roads, due to the rigid flat assumption of IPM Nedevschi
et al. (2004): Yan et al. (002). While some recent efforts have been made to address this issue and
"Corresponding author.

---

Published as a conference paper at ICLR 2024
have shown promising results by directly predicting 3D lanes in PV Bai et al. (2022b) Huang al. (2023) ;
Luo et al. (2023). these monocular 3D approaches, as vision-centric solutions, inevitably get stuck in capturing the complexity of real-world driving scenarios, when encountering adverse weather and lighting conditions. In contrast, as an active sensor, LiDAR excels in spatial localization and 3D structure perception, complementing the capabilities of passive sensor cameras, and gets more widely used thanks to hardware ad vancements, A bunch of recent works in 3D object detection
have demonstrated the power of LiDARs Zhou & Tuzel 2018); Lang al. (2019); Yin al. (202 la) and multiple modalities Liang et al. (2019); Wang et al. 2021 Yang et al. (2022) Li et al. (2022b);
Chen al. (2023) in autonomous driving scenarios. W Whereas, fewer endeavors Bai al. (2018);
Luo et al. (2022) have been mude to exploit multi-modal strength for 3D lane detection. Albeit using extra LiDAR data, M2-3DLane Luo et (2022) failed to make full use of features in image space which is crucial to 3D lane performance. Besides, M2-3DLane employs naive fusion to aggregate
multi-modal features, resulting in inferior performance to the camera-only methods(e.g.. Luo et al.
2023
Given the rich semantics inherent in images and the accurate positional information afforded by the BEV representation Philion & Fidler (2020); Li al. (2022d), we strive to exploit the multi- modal features to enhance the performance of 3D lane detection. Existing methods tend to fuse two modalities into single space Liang al. (202); Liu al. (2023b), e.g., BEV, for feature extrac- tion and subsequent prediction. However, this approach constrains the model's capacity to harness modality-speciitn features. We contend that features represented in both PV space and BEV space bear significance, facilitating improved representation learning. Motivated by the above observa- tion, we introduce DV-3DLane, a novel end-to-end multi-modal 3D lane detection framework.
To maintain dual-view space representation, we adopt a symmetric backbone consisting of PV branch and BEV branch extract features in PV and BEV spaces, respectively. To leverage the merits of both images and points for comprehensive feature learning in each view, we design a bidi- rectional feature fusion (BFF) strategy. Subsequently, to effectively facilitate query-based detection using the retained dual-view features, we devise unified query generator (UQG). This generator initially produces two sets lane-aware queries; one from the PV space and the other from the BEV space. These two query sets are compelled to capture lane knowledge regarding semantics and spatiality, guided by auxiliary 2D segmentation supervision. Further, these two sets are then combined into unified set that serves the decoder. To achieve the unification of dual-view queries, we propose a lane-centric clustering technique. Besides, we employ a Transformer decoder to ag- gressively integrate discriminative features from both views into the unified queries. For effective feature aggregation across different view spaces, we introduce a 3D dual-view deformable attention mechanism that considers the inherent properties of 3D space, resulting in deformed 3D sample points. These 3D sample points are then projected onto the PV and BEV planes, yielding 2D sample points in each respective view space. These projected 2D points are utilized for feature sampling within their respective view spaces.
In summary, our contributions are threefold:
We introduce DV-3DLane, an end-to-end multi-modal 3D lane detection framework that har- nesses the power of dual-view representation.
We devise the BFF strategy to mutually fuse features across modalities, and design the UQG to merge lane-aware queries from dual views, yielding unified query set. Further, 3D dual-view deformation attention mechanism is introduced to aggregate dual-view features effectively.
We conduct thorough experiments on the OpenLane benchmark to validate the effectiveness of our method. Experimental results show that DV-3DLane surpasses previous methods signifi- cantly, achieving an impressive 11.2 gain in Fl score and a remarkable 53.5% reduction in errors. Moreover, a 3D dual-view deformation attention mechanism is introduced to aggregate dual-view features effectively.
RELATED WORK
2.1 2D LANE DETECTION
Recent works in 2D lane detection can be broadly categorized into four main approaches: 1) Segmentation-basel methods Lee al. (2017); Pan et al. (2017); Neven et al. (2018; Hou et al.
2

---

Published as a conference paper at ICLR 2024
Xu al. (2020); Zheng et al. (2021) devote to classifying pixels into lanes the back- steps (e.g., grouping and curve fitting) to produce lane instances. 2) Anchor-based methods, inspired by basion object detectors such Faster- RCNN Ren et al. (2013), employ line-like anchors to localize lanes Wang et al. (2018); 8): Li et al.
2019);
ground, necessitating further post-process
(2019) Tabelini et al. (2021a) To overcome the limitations of straight-line constraints, Jin et al.
2022 employ eigenlane space to produce diverse lane shape candidates. 3) Point-based methods Ko
al. (021); Qu et al. (02I:; Wang et al. (2022); Xu al. (2022) attempt to flexibly localize key points along each lane instance and subsequently group the points belonging to the same lane. 4) Parametric methods Van Gansbeke et al. (2019;; Tabelini
al.) (2022) formulate lane detection as a curve fitting problem, leveraging prior knowledge about lane
(2 Liu et al. (202T):
shapes by representing them using various parametric forms, such as polynomials and splines.
PP Spaze Forrer
Query Clustering
3D Dual-view Deformable Attention
Image Backbone
IAM
Jnl Choser Conters
Quee
Piver Cser
3D Rairans Posti
Bidirectional
Ru RuSrrnnee Poin!
Feature Fusion
Iestance Supervision
Naw
Veaters
TAM
Backbone
New Pleer Cluory
Churry
Dual-view
Feature Process
Query Generation
Decoder
3D Lune Prediction
Figure 2: Overview DV-3DLane. First, images and point clouds undergo separate processing by the image backbone and point backbone. In the middle stage of backbones, we introduce Bidirec- tional Feature Fusion (BFF) to fuse multi-modal features across views. Subsequently, the instance activation map (IAM) is utilized to produce lane-aware queries Qpe and Qbee. These queries then subjected to Dual-view Query Clustering, which aggregates dual-view query sets Qye and Que into unified query set C, further augmented with learnable point embeddings Epoints to form query Q. Additionally, we introduce 3D Dual-view Deformable Attention to consistently aggregate point features from both view features F pV and Feeu into Q. @ denotes broadcast summation. Notably, the @ Epoints operation is performed only in the first layer, while in the following layer, @ Q is utilized. Different colored boxes DMA denote queries targeting different lanes; dashed boxes ! represent the background, and box texture indicates features.
2.2 3D LANE DETECTION
Existing methods center on vision-centric solutions and draw inspiration from the 2D task. Typically, monocular approaches Garnett al. (2019): Efrat et al. (2020); Guo et al. (2020); Chen et al. (2022)
Wang et al.] (2023); Liu et al. (2022) Li et al. (202aa): Ai et al. (2023): Yao et al. (2023) construct surrogate representations using inverse perspective mapping (IPM), and perform predictions in this surrogate space. Nonetheless, IPM inherently introduces discrepancies between the perspective and the surrogate view non-ilat areas due to planar assumption. To address this limitation, recent efforts have endeavored to predict 3D lanes from the perspective view Yan et al. (2022); : Bai et al.
2022b); Huang et al. (023). Luo et al. (2023), or employ depth-aware projection enhance lane perception by incorporating LiDAR information Luo et al. (2022).
2.3 MULTI-MODAL DETECTION
Despite advancements in lane detection, multi-modal methods remain relatively underexplored. Pre- vious works typically utilize either BEV Bai et al. (2018): Yin et al. (2000): [uo et al. (2022) or PV Zhang et al. (202lb) as representation spaces for performing 2D lane segmentation Yin et al. 2020); Zhang et al. (202 based methods, Bai et al. (2018) rasterizes LiDAR points to create a BEV image and transforms PV images into BEV using the estimated ground height derived from the LiDAR data. Similarly, -2-ane Luo et al. (2022) utilizes the BEV space to fuse multi-modas features. To project PV features into BEV space, they lift compact 2D features into 3D space guided by the depth map and
3D lane detection Bai et al. (2018); ;) Luo et al. (2022). For BEV-
3

---

Published as a conference paper at ICLR 2024
further employ a pillar-based method Lang et al. (2019) to splat them into BEV. While these methods primarily focus on 3D tasks. Yin et al. (2020) leverages BEV space for fusing camera and LiDAR features, serving for 2D BEV lane segmentation. Conversely, Zhang et al. (2021b) adopts PV to fuse multi-modal features for 2D lane segmentation. In contrast to lane detection, multi-modal meth- ods have been extensively studied in 3D object detection, with most previous multi-modal methods attemptinn to fuse image features into BEV space due to its compactness and interoperability for ambient perception Ma et al. (2022). These methods either adopt point-level fusion [Sindagi et al. (2019); Wang et al. (2021): in et al. (2021b) to paint points, instance-level fusion to project 3D proposals to image space Yoo al. (2020); Bai al. (2022a). or feature-level fusion transform features from PV space into BEV space Liu et al. (2023b); Liang et al. (2022). However, few works consider both the perspective view and BEV simultaneously.
3
METHODOL.OGY
The overall framework of our DV-3DLane is depicted in Figure 2 Section 3.1 describes the bidi- rectional feature fusion module, which merges different modalities bidirectionally and constructs multi-modal features in both PV and BEV spaces. In Section 3.2, we present the unified query gen- erator, which generates two lane-aware query sets from dual views and unifies them into shared space lane-centric manner. 3ection ule, which effectively aggregates dual-view features into unified queries, serving for prediction.
the 3D dual-view deformable attention mod-
3.1 BIDIRECTIONAL FEATURE FUSION
Instead of merging different views into single space Bai et al. (2018): Luo al. (022 Liang et al. 2022) Li et al. 2022dd; Liu et al. 2023b), we propose to retain features in both PV and BEV spaces while incorporating multi-modal features for each view. To achieve this, we employ a dual branch to extract features for each view, using images and points as input, respectively. Interme- diately, we conduct bidirectional feature fusion between the symmetric branches enhance each view with multiple modalities, as shown in Figure aadnd summarized in Algorithm [
Algorithm 1 Bidirectional Feature Fusion (BFF)
Input: LiDAR points Ppt. image I, camera parameters T Output: mm-aware PV features pe, BEV features F Fbevs "mm" denotes multi-modal,
PV Foture
Scatter
Fel PillarNet-S1 (P))), Fe1 ResNet-S11()
Sumple
SI: stage one.
P ptpps ((u, uz)li E P} Pr.
F ptppo = Scatter ide = P p12po srC
Fo1]
Polle Frsture
Concal
points pixels.
F puppp Grid_Sample( (src Fs1
coords P pt2pu) > pixels + points.
F ResNee(CConcttFF per Fp12pe))
Figure 3: Bidirectional Feature Fu- sion (BFF), We represent the image fea- ture in green and points in blue.
F New PillarNet (Concat (F;,, p)
> dual-view mulii-modal feature extruction.
Concretely, place points and images in their designated branches. After obtaining low-level features within each branch, we perform bidirectional feature fusion. By projecting 3D points P pt ((14, yts 2 e P} onto the PV plane, we obtain their corresponding 2D coordinates pt2pp {(u,, oz)li E P). where P is the cardinality of the point set. 1) For points-to-pixels fusion, we utilize a Scatter operation to construct dense point feature grids F pt2pe: (depicted in the upper part of FigureB with blue cells denoting positions hit by the projected 3D) points). 2) For pixels-to-points fusion, we employ bilinear interpolation to sample features at 2D positions hit by the projection of 3D points, yielding F po2pt (shown in the lower part of Figure 3). The resulting cross-modal features in PV and BEV are concatenated with their respective original modal features. The fused multi-modal features in each view, i.e., PV and BEV, then fed into subsequent modules in the corresponding branch, generating F Fp and F bev. respectively. Notably, pu and F hew encapsulate multi-modal information represented in distinct spaces.
4

---

Published as a conference paper at ICLR 2024
3.2 UNIFIED QUERY GENERATOR
We introduce a unified query generator for end-to-end 3D lane detection. To this end, we first generate distinct lane-aware query sets, termed dual-view queries, from the previously obtained multi-modal features, F pv and bee- Then, we present a lane-centric clustering strategy to unify these dual-view queries into a cohesive of queries.
Dual-view Query Generation. To effectively capture semantic and spatial features related to lanes, which are termed as "lane-aware" knowledge, we utilize an instance activation map (IAM) Cheng et al. (2022)assissed -assisted method to generate lane-aware queries in PV and BEV spaces. Taking PV branch as an example, we produce a set of IAMs, denoted as Apu. via the following equation:
A pC = (FF(Concat F pe: Spp))),
where Apr E N Epc xCCxHpuXWa, N denotes query number, is the sigmoid function, Concat represents concatenation operation, and Spo comprises two-channel spatial local- ization features for each pixel Liu et al. (2018). The lane-aware query Qpe assisted by IAMs is generated via:
Qpe Ap & FT
pe
where Qpe E NR @ denotes the matrix product. Similarly, lane-aware BEV query Qbev N.NCC is formed using:
Quee = oFF(( Shew))) F.....
To force the query segmentation for each branch on top of the query set. Labels for the auxiliary segmentation are generated in pairs for these two branches, which are further assigned predictions using mask- based bipartite matching Cheng et al. 2022). as illustrated in Figure 4 (a) and (b).
learn lane-aware features, during training, we employ auxiliary instance
Dual-view Query Clustering. Given dual-view query sets Qpe and Queu. we propose employing lane-centric clustering technique to generate unified query set for end-to-end lane Yu ct al. (022)) previously used k-means cross-attention to group pixels into distinct clusters, ie., instance masks, approach focuses on unifying queries from different views. Queries from Qpe and Quee targeting the same lane are merged within the same cluster. Specifically, we initiate lane cluster centers C' E R NC with Qpes. and assign each query in Quey its nearest cluster center among C. Notably, cluster centers can be chosen from either Que Qsey. Empirically, we found that using Qpu produces better results. To achieve clustering, we perform attention between C (query) and Qsee (key), while applying argmax along the cluster center (query) dimension Yu et al. (2022) as follows:
Groand Trads Lanc
Greand Tradh Lance
PV Lane Prakike
BAN Lane Prolein
detection. While
BEV
Felae Lase
Lane
0 Buckground
GT Askgunen
Positive Pair
Nogative Pair
Figure 4: Illustration of one-to-one matching and lane- centric clustering. (a) and (b) show the assignment for BEV and PV predictions, respectively. (c) depicts the pairing of the clustering. where queries targeting the same lane treated as positive pair, otherwise negative.
A argmax(C
Queu), Č Č-A-Q... -uce +C
N
where Ć E NxC refers updated centers unifying queries from dual views. In practise,
gumbel-softmax Jang et al. 2016); Liang et al. 2023) to substitute argmax.
Considering the variation and slenderness of lanes, employ refined point query scheme Luo ct al. (2023) to enhance lane detection. Instead of using a single query for each lane, multiple-
point queries
employed for more precise capture Luo et al. (2023); Liao et al. (2022); Zhang
et al. 202 a Liu et al. 2023a). Consequently, in the first layer, we construct point-based queries
Q 1 Nxxxxx xC with Q C + Epoints where @ denotes broadcast sum, Epoints E IR. MC is the
leamablee point embedding, and in the subsequent layer, update Q by Q Č @ Q.
Supervision on Query Clustering. Given the critical importance of deep supervision for the clus-
tering Yu et al. (2022). leverage the InfoNCE loss Oord al. (2018) to supervise the query
5

---

Published as a conference paper at ICLR 2024
clustering in a lane-centric manner, as illustrated in Figure 4 (c) and formulated as:
exp(q k 7TT
CNCE = -log exp(q-kt/r) + + EN expla.k<77
where t is a temperature hyper-parameter Wu et al. (2018), q denotes one query, k+ indicates the positive sample w.r.t. q. and .V' denotes the collection of all negative samples from the different query set relative to the one containing Notably, queries assigned to the background do not incur penalties in the clustering learning process. With this supervision, queries from different views grouped together when matched to the same ground truth lane. Consequently, lane-aware knowledge residing in two view spaces is synergized into the unified query.
3.3 3D DUAL-VIEW DEFORMABLE ATTENTION
Apart from informative query generation, feature aggregation plays a crucial role in DV-3DLane. Instead of projecting points from densely sampled grids Chen et al. 2022 their lifted pillars Li] et al. (2022d) onto the PV plane for feature sampling, as shown in Higure (a), we adopt sparse queries to sample features from different views. Moreover, our approach distinguishes itself from several existing sparse query methods, as depicted in Figure (b) and (c). For instance, Deepintera- tion Yang et al. (2022) (Figure 5 (b)) employs a sequential method to sample PV and BEV features, while FUTR3D Chen et al. (2023) (Figure (c)) projects 3D points into different spaces, sampling features individually for each space.
In contrast, as outlined in Algorithm 2 we leverage the inherent properties of 3D space by predicting both 3D reference points and their 3D offsets using queries, forming 3D deformed points. These 3D deformed points are then projected into each space, establishing a consistent feature sampling strategy across spaces, as depicted Figure [ Consequently, features corresponding to the same 3D points from different views are effectively sampled and integrated into the query.
Algorithm 3D DV Deformable Attention
Input:
unified query set Q, PV features F Fvv. BEV fea- turss Feee, camera parameters T. Output: updated unified query Q.
(N)
layer +1
Refad (QP
3D reference points.
RRe 3d = (Q) Sad = [ Vi, 2i N} ARef + Ref
(6)
[ou]]
>> deformed 3D positions.
Retimeec
Poyacton
D po DeformAAtu(Projeetty. (San, T), Fm)) project 3D deformed points to PV. Dec Deform (trotnnPrre.t......... (Saa), Fbce)
Figure 5: Illustration comparing 3D dual-
view deformable attention with other ap- proaches.
Q = SE,Dpu, pv, Doco)
3.4 PREDICTION AND Loss
Auxiliary Tasks. During training, we incorporate two auxiliary tasks: 1) 2D instance segmenta- tion Luo et al. (2023); Cheng al. (2022) loss Ceeg for both PV and BEV branches, aiding in extracting discrimintive lane fcatures in each view; 2) Depth estimation for the PV branch, which guides effective 3D structure-aware feature extraction of pr. Depth labels are generated from Li- DAR points, and the loss Ldepth is calculated following BEVDepth Li et al. (202cc).
3D Lane Prediction and Loss. As we adopt point-based queries Q E (NNXM) each query naturally corresponds to a 3D point, and every group of M points constructs a complete 3D lane. Thus, predict x, Z, and visibility for each point query on the predefined coordinates Chen et al. 2022 Luo et al. (2023) and classification probability for each lane. Overall, the total loss is:
Ltane W. Cx + W. WyCn + WeLes
Laur = Wseghseg + Wdepth Eetephh
Listal Ltane Luaar-
where w. denotes different loss weights. We adopt the L1 loss C, and L. to learn the x, positions, focal loss Lin et al. (201)) L. to learn the lane category, and BCELoss C., to learn visibility.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 DATASETS
We evaluate our method on OpenLane Chen
2022). the sole public 3D lane dataset featuring multi-modal sources, OpenLane is
Score
0.4
a large-scale dataset built on Waymo Open
Dataset Sun et al.] (2020), comprising 200K frames and 880K lanes across six driving sce- narios and 14 lane categories. The LiDAR data, collected using 64-beam LiDARs, is sampled at 10Hz. This extensive dataset provides solid foundation for evaluating 3D lane algorithms comprehensively.
PosFurmeer Ra300
AschuADLaas ENB3
Anchort Rests
011
LATR
9.0
Oun Hase
2
Threshold
Figure 6: F1 score vs, Distance Threshold. Our method consistently achieves superior results un- der more stringent criteria.
4.2 METRICS
We adopt the evaluation metries established by OpenLane Chen et al. (2022). framing 3D lane detection evaluation matching problem based on the edit distance between predictions and ground truth. Successful matching results in computed metrics, including F-Score, category accuracy, and error in X/Z-axes. A successful match for each predicted 3D lane is defined when at least 75% of its points have a distance to the ground truth below the predefined threshold Dthre-
4.3 IMPLEMENTATION DETAILS
Models. In base version of DV-3DLane, we employ ResNet34 He et al. (2016) and Pillar- Net34 (Shi et al. (2022) as the backbones for our camera and LiDAR branches, respectively. For the lite version, we utilize ResNet18 and PillarNet18. The base version features two decoder layers, while the lite version employs single decoder layer. Following LATR Luo al. (2023). set the number of lane queries to 40, and we employ deformable attention with 4 heads, 8 sample points, and 256 embedding dimensions.
Training. We use the Adam optimizer Kingma & Ba (014) with a weight decay of 0.01. The learning rate is 2e-4, and our models undergo training for 24 epochs with batch size of 32. We employ the cosine annealing scheduler Loshchilov & Hutter (2016) with Tmaz = 8. Our input images are of resolution 720x960, and we adopt voxel size of (0.2m, 0.4m) for the and Y axes.
4.4 MAIN RESULTS
It's important to note that the existing metrics use a rather lenient distance threshold of Dure =1.5m However, in the context of ensuring safety in AD, this value, although commonly used for assess- ment purposes, may be considered overly permissive. Following extend our evaluation include a more stringent threshold, DrAre 005mm. Further, we illus- trate the relationship between the Fl score performance and different distance thresholds for various models, as shown in Figure 6 Notably, our method consistently achieves superior results, even when evaluated under a much more stringent criterion of Dthre=0.1m. In contrast, other approaches experience a noticeable decline in performance as the distance threshold decreases. These findings confirm the robustness of our method across varying distance thresholds, particularly highlighting its advantage in precise localization.
Luo et al. 2022).
We present the main results in Table obtained from experiments conducted on the OpenLane- 1K dataset. The evaluation uses both D and Dthr.=0.5m criteria, allowing for a compre- hensive and insightful comparison. It is evident that DV-3DLane consistently outperforms previous state-of-ther -hrt--rr (SoTA) methods across all metrics. Notably, when applying a more strict 0.5m threshold, DV-3DLane demonstrates substantial 11.2% improvement in the Fl score. It is note- worthy that our method excels localization accuracy, leading to significant performance improve- ments. Specifically, method achieves remarkable reductions in localization errors: 52%/50% for

---

Published as a conference paper at ICLR 2024
Table 1: Comprehensive 3D Lane evaluation comparison on OpenLane with variable metrics.
denotes the results obtained using their provided models. Simage-Branch" and - refer to our image and LiDAR branches, respectively. "LATR LiDAR" denotes the model that combines the SOTA method LATR with LiDAR input, projecting all points into the image space and using them as additional features in the network.
Dist.
Methods
Backhone
Modality
F1
Acc. 1
X error (m)) 1
error (m) 1
Jar
3DLaneNet Gamett al.
2019
VGG-16
44.1
0.593
0.494
0.140
0.195
GenLaneNel Luuo el an
2020
ERFNet
C
32.3
0.591
0.684
0.411
0.521
PersFoorme Chen
20224
50.5
89.5
0.319
0.325
0.141
Anchor3DLanc Huang
2023
Bf
52.8
89.6
0.408
0.349
0.186
0.143
Mi-3DLaneNet Luo et al
EffNet-B7
C+L
55.5
88.2
0.283
0.256
0.078
0.106
1.5
Anchor3DI and Huang et
M021
ResNet-18 18
C
50.7
89.3
0.422
0.349
0.188
0.146
PersFormer Chei 2022
2022
ResNet-50
52.7
88.4
0.307
0.319
0.083
0.117
LATR at ( 2025
ResNet-50
61.9
92.0
0.219
0,259
0.075
0.104
DV-3DLane-
Lane-Tinyy Tiny (Oars)
ResNet- 18
C+I.
63.4
91.6
0.137
0.159
0.034
0.063
DV-3DLane Base (Ours)
ResNet
C+I
65.4
92.4
0.118
0.131
0.032
0.053
DV-3DI aaaannn-aaa (Ours)
Re
C+I.
66.8
93.3
0,115
0.134
0.029
0.049
Improvement
14.9
T7.3
10.10M
10.122
.0.046
10.055
PersFormer Chen el 2022
EffNet B7
36.5
87.8
0.343
0.263
0.161
0.115
Ancbor3DLane Huang et 2023
EffNet-B3
34.9
88.5
0.344
0,264
0.181
0.134
Nen Luo ct al XM2
EffNer-Bi B7
C+L
48.2
88.1
0.217
0.203
0.076
0.103
Anchor3DL Lan Huang el
X023
ResNet-18
32.8
87.9
0.350
0.266
0.183
0.137
PersForrer Chen
ResNet-50
43.2
87.8
0.229
0.245
0.078
0,106
0.5 m
LATR Luo 21 20.25
2025
ResNet-50
$4.0
91.7
0.171
0,201
0.072
0,099
LATR + LIDAR
ResNet-50
C+L
57.4
92.1
0.167
0,185
0.071
0.088
Image- Branchh (Ours)
ResNet-34
$2.9
0.173
0.212
0.069
0.098
LiDAR Branch (Ours)
PillarN 34
L
$4.1
84.4
0.282
0.191
0.096
0.124
DV.3DI Jan- Tiny (Ours)
ResNet-18 18
60.9
0.097
0.062
DV-3DLane Base (Ours)
ResNet-34
C+L
63.5
92.4
0.090
0.102
0.031
0.053
DV-3DI Large (Ours)
ResNet-50
C+I.
65.2
93.4
0.082
0.101
0.028
0.048
Improvement
1/1.2
1..7
10.089
10.100
30.044
10.051
X near/far, and 61%/5%% for Z. near/far. Due to space limitations, results in various scenarios and studies about robustness concerning calibration noise are included in our Appendix.
Effect of Multiple Modalities. To explore the impact of individual modalities, we conduct experi- ments using single modalities, as outlined in the "Image-Branch" and "LiDAR-Branch" rows of Ta- [I The results illustrate that DV-3DLan significantly enhances performance compared to using images alone or relying solely on LiDAR data. Notably, our method significantly surpasses config- urations that simply equip LATR with LiDAR input across all metrics, underscoring the substantial improvements achieved by DV-3DLane leveraging information from both modalities. Moreover, to evaluate the effect of dual-view, we conduct experiments using single-modality input but trans- forming features extracted from the backbone into another view, yielding single-modal dual-view features. Then, our dual-view decoder is applied, and the results are detailed in Appendix. Addi- tionally, we conduct experiments using our IImageBBranch on the Apollo Guo et al. (2020) dataset, which exclusively contains image data. The results are provided in our Appendix.
Qualitative Results. We present a qualitative comparison between DV-3DLane and LATR et (2023) in Figure 7 demonstrating that our method achieves more robust and accurate predictions across various scenarios. More visualization results are included in our Appendix.
4.5 ABLATION STUDIES
We conduct all ablation studies on OnenLane-300 following established practices et al. (2022); Luo et al. 2023); y Huang et al. (2023). while adopting 0.5m threshold Dihre for evaluation.
Effect of Bidirectional Feature Fusion. The corresponding experiments are included in our Ap- pendix, due to space limitations. We kindly direct the readers to refer to the Appendix for details. The results confirm the effectiveness of the proposed bidirectional feature fusion approach.
Effect of Unified Query. We study the effect of our unified queries generation strategy in Table where "Random" means random initialization using nn. Embedding, "Qpe" deeotees using only PV queries, and "Quev" refers to using only BEV queries. Replacing unified queries with randomly initialized ones Carion et al. (2020);Zhuu I al. (2020): Li al. (202dd) results a decrease of 1.0 in the F1 score compared to approach. Interestingly, employing single space instance-
8

---

Published as a conference paper at ICLR 2024
(9)
(c)
(d)
Figure 7: Qualitative Results. We present the projection of 3D lanes from ground truth, predictions DV-3Daane and the SOTA method LART Luo et al. (2023) in rows (a), (b), (c), respectively. Row (d) depicts the comparison between ground truth (red) and ours (green) in 3D space. We highlight the differences with colored arrows. Best viewed in color and zoom for details.
aware query yields even lower Fl scores of 69.6%/69.15 1% for PV/BEV, respectively, than random initialization. This underscores the inadequacy of single-space lane-aware query in capturing complex 3D lane features comprehensively existing in both PV and BEV spaces, However, our dual-view strategy, generating lane-aware queries w.r.t. both views, improves overall performance to 70.7, achieving the best result. This demonstrates that our method effectively integrates the strengths of features from two spaces, forming cohesive query set.
Effeet of 3D Dual-view Deformable Attention. To evaluate the efficacy of our proposed Dual-view Deformable Attention, we conduct ablation studies in Table where "PV space" and "BEV space" mean using single space in the decoder. "DeepInteration" Yang et al. (2022) denotes sequential fusion of features from different spaces, and "FUTR3D" Chen et al. (2023) refer to a modality- agnostic approach where sampling locations differ across views. We compare DV-3DLanc against alternative approaches, including single-view fused method, as well as methods proposed in Deep- Interation and FUTR3D, described in Section 3.3 The results underscore the significance of our approach. In detail, sampling only PV space features leads to a notable drop (70.7-63.6) in performance, showing the importance of BEV space due to its advantages in localization. Besides, our method outperforms the sequential approach of DeepInteration with a substantial 2.0 gain in F1 score. Furthermore, compared to the modality-agnostic approach proposed in FUTR3D, our method achieves 0.5 improvement, emphasizing the importance of consistent sampling locations in deformable attention across different spaces.
Table 3: Effect of 3D dual-view deformable attention.
Table 2: Effect of unified query.
error (m)
error (m)
X error (m)
Z error (m)
Methods
F1
Methods
F1
near
für
near
fur
PV space
63.6
0.150
near
0,202
far
0.060
near
0.081
far
Random
69.7
0.123
0.151
0.059
0.081
BEV space
68.5
0.127
0.151
0.064
0.087
Que
69.6
0.124
0.155
0.059
0.079
DeepInteration
68.7
0.126
0.157
0.059
0.081
Queu
69.1
0.122
0.145
0.058
0.077
FUTR3D
70.2
0.118
0.145
0.057
0.077
Ours
70.7
0.123
0.146
0.058
0.078
Ours
70.7
0.123
0.146
0.058
0.078
CONCLUSION
In this work, we introduce DV-3DLane, novel end-to-end multi-modal 3D lane detection frame-
work that leverages the strengths of both PV and BEV spaces. To this end, we propose three novel modules that effectively utilize dual-view representation different levels, consistently enhancing performance. Extensive experiments substantiate the outstanding advancements achieved by DV. 3DD establishing state of the OpenLane.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by NSFC with Grant No. 62293482, by the Basic Research Project No. HZQB-KCZYZ--021061 of Hetao Shenzhen HK S&T Cooperation Zone, by Shen- zhen General Program No. JCY1202205014360000,, by Shenzhen-Hong Kong Joint Funding No. SGDX202111233 12401002, by the National Key R&D Program of China with grant No. 0018YF81000800 by the Shenzhen Outstanding Talents Training Fund 202002, by Guangdong Research Project No. 20172T07X152 and No. 2019CX01X104, by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B 121 1212010001), by the Guang- dong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen, by the NSFC 61931024& 12326100, by the Shenzhen Key Laboratory of Big Data and Artificial Intelligence (Grant No. ZDSYS20170725 (490900 and the Key Area R&D Program of Guangdong Province with grant No. 2018803033800, by Tencent&Huavel Open Fund.
REFERENCES
Jianyong Ai, Wenbo Ding, Jiuhua Zhao, and Jiachen Zhong. Ws-3d-lane: Weakly supervised 3d lane detection with 2d lane labels. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5595-5601. IEEE, 2023.
Min Bai, Gellert Mattyus, Namdar Homayounfar, Shenlong Wang, Shrinidhi Kowshika Laksh- mikanth, and Raquel Urtasun. Deep multi-sensor lane detection. 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3102-3109, IEEE, 2018
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar- camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1090-1099, 2022a. 2022a.,
Yifeng Bai, Zhirong Chen, Zhangjie Fu, Lang Peng, Pengpeng Liang, and Erkang Cheng. Curve- former: 3d lane detection by curve propagation with curve queries and attention. arXiv preprint arXiv:2209.07790 07989, 2022b.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp- 213-229. Springer, 2020.
Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Con- ghui He, Jianping Shi, Yu Qiao, and Junchi Yan. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In European Conference on Computer Vision (ECCV), 2022.
Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: unified sensor fusion framework for 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 172-181, 2023.
Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wenqiang Zhang, Qian Zhang, Chang Huang, Zhaoxiang Zhang, and Wenyu Liu. Sparse instance activation for real-time instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4433-4442, 2022.
Netalce Efrat, Max Bluvstein, Shaul Oron, Dan Levi, Noa Garnett, and Bat El Shlomo. 3d-lanenet:: Anchor free lane detection using a semi-local representation. arXiv preprint arXiv:2011.01535 2020.
Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, and Lizhuang Ma. Rethinking efficient lane detection via curve modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17062-17070, 2022.
Noa Garnett, Rafi Cohen, Tomer Pe'er, Roee Lahav, and Dan Levi. 3d-lanenet: end-to end 3d multiple lane detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2921-2930. 2019.
10

---

Published as a conference paper at ICLR 2024
Yuliang Guo, Guang Chen, Peitao Zhao, Weide Zhang, Jinghao Miao, Jingao Wang, and Tae Eun Choe. Gen-lanenet: A generalized and scalable approach for 3d lane detection. In European Conference Computer Vision, pp. 666-6666699 Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection cnns by self attention distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1013-1021, 2019.
Shaofei Huang, Zhenwei Shen, Zehao Huang, Zi-han Ding, Jiao Dai, Jizhong Han, Naiyan Wang, and Si Liu. Anchor3dlane: Learning to regress 3d anchors for monocular 3d lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
17451-17460, 2023.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiio with gumbel-softmax. arXiv preprint arXiv:1611. 1.0111014444.. 2016.
Dongkwon Jin, Wonhui Park, Seong- -Gygg-Gunnnnu Jeong, Heeyeon Kwon, and Chang- Su Kim. Eigenlanes: Data-driven lane descriptors for structurally diverse lanes. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 17163-17171, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1:Xiv:1412.69980 6980, 2014.
Yeongmin Ko, Younkwan Lee, Shoaib Azam, Farzeen Munir, Moongu Jeon, and Witold Pedryez. Key points estimation and point instance segmentation approach for lane detection. IEEE Trans- actions on Intelligent Transportation Systems, 23(7):8949-8958 2021.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
Seokju Lee, Junsik Kim, Jae Shin Yoon, Seunghak Shin, Oleksandr Bailo, Namil Kim, Tae-Hee Lee, Hyun Seok Hong, Seung-Hoog Han, and So Kweon. Vpgnet: Vanishing point guided network for lane and road marking detection and recognition. In Proceedings of the IEEE international conference on computer vision, pp. 1947-1955, 2017.
Chenguang Li, Jia Shi, Ya Wang, and Guangliang Cheng. Reconstruct from top view; A 3d lane de- tection approach based on geometry structure prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4370-4379, 2022a.
Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-cnn: End-to-end enn traffic line detection with line proposal unit. IEEE Transactions on Intelligent Transportation Systems, 21(1):248-22) 2019.
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng. Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi- modal 3d object detection. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17182-17191, 2022b.
Yinhao Li, Zheng Ge. Guanyi Yu. Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206. 10092, 2022c.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's s-eee-iww representation from multi-camera images via spa- tiotemporal transformers. In Computer VisionFECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pp. 1-18. Springer, 2022d.
James Liang, Tianfei Zhou, Dongfang Liu, and Wenguan Wang. Clustseg: Clustering for universal segmentation. arXiv preprint arXiv:2305.02187. 2023.

---

Published as a conference paper at ICLR 2024
Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-senso fusion for 3d object detection. In Proceedings the
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7345-7355 2019.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:10221-10444 2022.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construc- tion. arXiv preprint arXiv:2208. 0000008./4443 /4437, 2022.
Tsung-V Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp- 2980-2988, 2017.
Huan Liu, Qiang Chen, Zichang Tan, Jiang-Jiang Liu, Jian Wang, Xiangbo Su, Xiaolong Li, Kun Yao, Junyu Han, Errui Ding, et al. Group pose: A simple baseline for end-to-end multi-person pose estimation. arXiv preprint ar arXiv:2308.0m T.t::2308.073333. 2023a.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018.
Ruijin Liu, Zejian Yuan, Tie Liu, and Zhiliang Xiong. End-to-end lane shape prediction with trans- formers. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3194-36022 2021.
Ruijin Liu, Dapeng Chen, Tie Liu, Zhiliang Xiong, and Zejian Yuan. Learning to predict 3d lane shape and camera pose from single image via geometry constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1765-1772, 2022.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's s-eye view representation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 2774-2781. IEEE, 2023b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Yueru Luo, Xu Yan, Chaoda Zheng, Chao Zheng, Shuqi Mei, Tang Kun, Shuguang Cui, and Zhen Li. M 2-3dlanenet: Multi-modal 3d lane detection. arXiv preprint arXiv:2209.05996, 2022.
Yueru Luo, Chaoda Zheng, Xu Yan, Tang Kun, Chao Zheng, Shuguang Cui, and Zhen Li. Latr: 3d lane detection from monocular images with transformer, 2023.
Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang, Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, Dinesh Manocha, and Xinge Zhu. Vision-centric bev perception: A survey. arXiv preprint arXiv:2208.00 027997 2022.
Hanspeter Mallot, Heinrich H Bülthoff, JJ Little, and Stefan Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics, 64(3):177- 185, 1991.
Sergiu Nedevschi, Rolf Schmidt, Thorsten Graf, Radu Danescu, Dan Frentiu, Tiberiu Marita, Florin Oniga, and Ciprian Pocol. 3d lane detection system based on stereovision. Proceedings. The 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No. 04TH8749), pp. 161-166. IEFE, 2004.
Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Towards end-to-end lane detection: an instance segmentation approach. In 2018 IEEE intelligent vehicles symposium (IV), pp. 286-291. IEFF, 2018
12

---

Published as a conference paper at ICLR 2024
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
X Pan, J Shi, Luo, X Wang, and X Tang. Spatial deep: spatial can for traffic scene understand- ing. 2017. Pan Shi J Luo P Spatial As Deep: Spatial CNN for Traffic Scene Understanding, 10, 2017.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210. Springer, 2020.
Zhan Qu, Huan Jin, Yang Zhou, Zhen Yang, and Wei Zhang. Focus local: Detecting lane marker from bottom up via key point. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 14120- 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillar-based 3d object detection. European Conference on Computer Vision, pp. 35-52. Springer, 2022.
Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. In 2019 International Conference on Robotics and Automation (ICRA). pp. 7276-7282. IEEE, 2019.
Pei Sun, Henrik Kretzsehmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 2446-2455 2020.
Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto De Souza, and Thi- ago Eliveira-Santos Keep your eyes the lane: Real-time attention attention-guided lane detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 294- 302, 2021a.
Lucas Tabelini, Rodrigo Berriel, Thiago Paixao, Claudine Badue, Alberto De Souza, and Thi- ago Oliveira-Santos. Polylanenet: Lane estimation via deep polynomial regression. In 2020 25th International Conference on Pattern Recogmition (ICPR), pp. 6150-6156. IEEE, 2021b.
Wouter Van Gansbeke, Bert De Brabandere, Davy Neven, Marc Proesmans, and Lue Van Gool. End- to-end lane detection through differentiable least-squares fitting. In Proceedings the International Conference Computer Vision Workshops, pp. 0-0, 2019.
the IEEE/CVF
Chunwei Wang. Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmen- tation for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11794-1 1803, 2021.
Jinsheng Wang. Yinchao Ma, Shaofei Huang, Tianrui Hui, Fei Wang, Chen Qian, and Tianzhu Zhang. A keypoint-based global association network for lane detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1392-1401, 2022.
Ruihao Wang, Jian Qin, Kaiying Li, Yaochen Li, Dong Cao, and Jintao Xu. Bev-lanedet: An effi- cient 3d lane detection based virtual camera via key-points. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1002-1011, 2023.
Ze Wang, Weiqiang Ren, and Qiang Qiu. Lanenet: Real-time lane detection networks for au- tonomous driving. arXiv preprint arXiv: 1807.01760 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
13

---

Published as a conference paper at ICLR 2024
Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Curvelane- nas: Unifying lane-sensitive architecture search and adaptive point blending. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XV 16, pp. 689-704. Springer, 2020.
Shenghua Xu, Xinyue Cai, Bin Zhao, Li Zhang, Hang Xu, Yanwei Fu, and Xiangyang Xue. Rclane: Relay chain prediction for lane detection. In European Conference on Computer Vision, pp. 461- 477. Springer, 2022.
Fan Yan, Ming Nie, Xinyue Cai, Jianhua Han, Hang Xu, Zhen Yang, Chaoqiang Ye, Yanwei Fu, Michael Bi Mi, and Li Zhang. Once-3dlanes: Building monocular 3d lane detection. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17143- 17152, 2022.
Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. Advances in Neural Information Processing Systems, 35:1992-2000 2005, 2022.
Chengtang Yao, Lidong Yu, Yuwei Wu, and Yunde Jia. Sparse point guided 3d lane detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8363-8372, 2023.
Ruochen Yin, Yong Cheng, Huapeng Wu, Yuntao Song, Biao Yu, and Runxin Niu. Fusionlane: Multi-sensor fusion for lane marking semantic segmentation using deep neural networks. JEEE Transactions on Intelligent Transportation Systems, 23(2):143-1553, 2020.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11784 11793, 2021a.
Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. Ad- vances in Neural Information Processing Systems, 34:16494-16507, 2021b.
Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In Computer Vision- ECCV 2020; 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXVII 16, pp. 720-736. Springer, 2020.
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. k-means mask transformer. European Conference on Computer Vision, pp. 288-307. Springer, 2022.
Jianfeng Zhang, Yujun Cai, Shuicheng Yan, Jiashi Feng, et al. Direct multi-view multi-person 3d pose estimation. Advances in Neural Information Processing Systems, 34:13153-13164 2021a a.
Xinyu Zhang, Zhiwei Li, Xin Gao, Dafeng Jin, and Jun Li. Channel attention in lidar-camera fusion for lane line segmentation. Pattern Recognition, 118:108020, 2021b.
Tu Zheng, Hao Fang, Zhang, Wenjian Tang, Zheng Yang. Haifeng Liu, and Deng Cai. Resa: Recurrent feature-shift aggregator for lane detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 3547-3554, 2021.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4490- 4499, 2018
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
14


Reviews:
[{"review_id": "Ju3R2Fez4g", "reviewer": "Reviewer_nYY6", "summary": "This paper presents a LiDAR Camera 3D-Lane detection model. This model consists of 4 main blocks: 1) two backbones (image in camera view (CV) and LiDAR (BEV) in bird-eye-view) linked by a bidirectional feature fusion, 2) a dual view query generation with clustering between BEV and CV queries, 3) a decoder with query clustering that produce point queries and 4) a 3D dual-view deformable attention producing a 3D lane prediction. The experimental part shows that the proposed model outperforms SOTA, including the last ICCV2023 papers.", "questions": "Did you try to experiment your models on other datasets like  OpenLane-Huawei: a dataset with more horizontal lines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "One original contribution is the bidirectional feature fusion module used to train the backbones in both BEV and CV leveraging for each view information from both sensors. The experimental part shows that this mechanism increases the F1 score about 2%. \nAnother important part of the pipeline is a Unified Query Generation process that win less than 1% F1 score. \nThe authors a 3D Dual-view Deformable Attention model that slightly improves the F1 score of the model. \nRegarding the ablation study, the sensor fusion win more than 10% regarding the LiDAR only and about 20% regarding only the camera. \nExperiments have been achieved on the public dataset OpenLane. The proposed model outperforms SOTA.", "weaknesses": "there are no really weakness in the paper. \nI should be interesting to experiment your models on other datasets like  OpenLane-Huawei: a dataset with more horizontal lines.\n\nminor typo : \n\ninstruction <-> introduction\nIncrease or remove line (d) of fig 7. (figures are too small)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1B2Y4JW0g", "reviewer": "Reviewer_7Rt7", "summary": "This paper target 3D lane detection problem by proposing a dual view representation method utilizing both camera and Lidar as input. Their proposed method has achieve remarkable improvement over prior work with high efficiency.", "questions": "Please refer to weakness. In general I feel like this paper is well written, happy to raise my score if question addressed.\nSome related papers from prior venue is not properly cited, please consider citing the following works\n\n@inproceedings{liu2022learning,\n  title={Learning to predict 3d lane shape and camera pose from a single image via geometry constraints},\n  author={Liu, Ruijin and Chen, Dapeng and Liu, Tie and Xiong, Zhiliang and Yuan, Zejian},\n  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n  volume={36},\n  number={2},\n  pages={1765--1772},\n  year={2022}\n}\n\n@inproceedings{yao2023sparse,\n  title={Sparse Point Guided 3D Lane Detection},\n  author={Yao, Chengtang and Yu, Lidong and Wu, Yuwei and Jia, Yunde},\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n  pages={8363--8372},\n  year={2023}\n}\n\n@inproceedings{li2022reconstruct,\n  title={Reconstruct from top view: A 3d lane detection approach based on geometry structure prior},\n  author={Li, Chenguang and Shi, Jia and Wang, Ya and Cheng, Guangliang},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={4370--4379},\n  year={2022}\n}\n\n@inproceedings{ai2023ws,\n  title={WS-3D-Lane: Weakly Supervised 3D Lane Detection With 2D Lane Labels},\n  author={Ai, Jianyong and Ding, Wenbo and Zhao, Jiuhua and Zhong, Jiachen},\n  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},\n  pages={5595--5601},\n  year={2023},\n  organization={IEEE}\n}", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Compared to prior work that mostly use only camera and feature learning in either Perspective View or Bird Eye View, this work fill the gap by fully investigate feature interaction of both sensor(Lidar+Camera) in both view(PV+BEV). And shows the superiority of their method by showing large improvement over prior work. \n2. The work is very well written, clearly introduce all related work as well as their problems, and how this work target those problem by proposing new solutions. \n3. This work include detail ablation study, make it clear to see the improvement over each step.", "weaknesses": "1. There’re some inconsistent in the evaluation result: 1). Although this method have explicitly filling the gap of missing Lidar modality in prior work, it might not seem as an apple to apple comparison in the table. In table 1 1.5m, compare with prior SOTA LATR, the proposed method didn’t achieve overwhelming improvement despite using one extra modality(Lidar). Does that mean the improvement in this range had been saturated to some extent? Also DV-3Dlane use a weaker backbone(ResNet-18/34) compare to ResNet50 LATR, make it even harder to see the full potential of proposed method compare to prior art. 2). Why Use OpenLane1000 in table 1(main experiment) but OpenLane300 for the ablation? Make it incomparable with result in Table1. \n2. For onboard application, it might not always viable, to access both Lidar and Camera information for joint feature, does this method also provide implications for camera only/Lidar only methods? Maybe it’s more clear in ablation study if we could compare Table1 and Table2 directly. A follow up question, if camera-only ablation is possible, could it also show result on Apollo dataset? This is the dataset most commonly used in prior work. \n3. For Figure1, please also consider adding more methods to be more completed, for instance light weight model like GenLane.\n4. What is the ‘lane-aware prior knowledge’ in the abstract? I feel like ‘prior knowledge’ it is not discussed in the main text. Please explain or consider rephrase it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4FZ1LxLhkD", "reviewer": "Reviewer_ECVn", "summary": "This paper presents a novel 3D lane detection algorithm that exploits a dual-view representation. The proposed algorithm consists of bi-directional feature fusion to aggregate view-specific features, unified query generation that focuses on coherent lane features, and 3D dual-view deformable attention to associate information across the viewpoints. The authors provided outperformed experimental results on OpenLane dataset.", "questions": ".", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Achieved the best performance on a 3D lane detection benchmark\n- Leveraged multi-modal features to generate a unified query for 3D lane detection", "weaknesses": "- It appears that the 3D DV deformable attention mechanism lifts PV features to 3D using known camera parameters. Also, when the authors concatenate DV features, they depend on these camera parameters. Have the authors attempted to test the tolerance of calibration parameters to noise?\n- Since the evaluation was only performed on OpenLane, it is difficult to check the generalizability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O3IDcI7Iuu", "reviewer": "Reviewer_Z8i9", "summary": "This paper presents a multi-modal 3D lane detection method. A bidirectional feature fusion approach is proposed to incorporate multi-modal features into each view space. A unified query generation module is adopted that provides lane-aware prior knowledge from both views. A 3D dual-view deformable attention mechanism that combines discriminative features from both PV and BEV into queries for precise 3D lane detection. Comprehensive experiments on the public benchmark, OpenLane, prove the efficacy and efficiency of DV-3DLane.", "questions": "*  The main performance improvement primarily comes from multi-modalities, but multi-modalities inherently achieve higher points than a single modality. Therefore, it would be best to prove that the multi-modal performance of existing methods is inferior to this paper. The camera result is not much competitiveness, especially when compared to some recent methods, such as Group Lane.\n*  Using Deformable attention to aggregate multi-modal features lacks innovation.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "*  The performance of this article exceeded that of existing methods. A significant increase was also achieved in the multimodal feature fusion at the backbone level. \n*  The paper is well written and easy to follow.", "weaknesses": "*  Most methods in Table 1 are based on camera. As can be seen from the results in Table 3, the performance of camera-only is not competitive. Multimodal inputs should be introduced to some methods for a comparative performance evaluation. Otherwise, presenting multimodal fusion as a key contribution is somewhat insufficient.\n\n* In Table 4, the performance of the queries generated based on PV and BEV is not as high as random queries, indicating that the adaptive generation of queries doesn't work. Although the queries after clustering have achieved performance improvement, it remains to be seen whether this improvement is brought about by the extra network. \n\n* The feature fusion in the backbone and decoder is quite tricky, making it difficult to be viewed as a major contribution point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7sdh8UAtQ", "reviewer": "Reviewer_9oXi", "summary": "This paper covers lane detection task using attention mechanism from multi-modal queries of vision and LiDAR.", "questions": "NA", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Really novel idea that does this task of lane detection using query clustering from multiple perspective. \n* SOTA results, and code should be available online soon \n* Takes care of runtime optimization as well.", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)