LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving
Authors: Tianyu Li
Peijin Jia
Bangjun Wang
Li Chen
KUN JIANG
Junchi Yan
Hongyang Li
Venue: ICLR 2024 poster (2024)
Keywords: Laneline detection
autonomous driving
topology reasoning
Primary area: applications to robotics, autonomy, planning

Abstract: A map, as crucial information for downstream applications of an autonomous driving system, is usually represented in lanelines or centerlines. However, existing literature on map learning primarily focuses on either detecting geometry-based lanelines or perceiving topology relationships of centerlines. Both of these methods ignore the intrinsic relationship of lanelines and centerlines, that lanelines bind centerlines. While simply predicting both types of lane in one model is mutually excluded in learning objective, we advocate lane segment as a new representation that seamlessly incorporates both geometry and topology information. Thus, we introduce LaneSegNet, the first end-to-end mapping network generating lane segments to obtain a complete representation of the road structure. Our algorithm features two key modifications. One is a lane attention module to capture pivotal region details within the long-range feature space. Another is an identical initialization strategy for reference points, which enhances the learning of positional priors for lane attention. On the OpenLane-V2 dataset, LaneSegNet outperforms previous counterparts by a substantial gain across three tasks, i.e., map element detection (+4.8 mAP), centerline perception (+6.9 DET$_l$), and the newly defined one, lane segment perception (+5.6 mAP). Furthermore, it obtains a real-time inference speed of 14.7 FPS. Code is accessible at https://github.com/OpenDriveLab/LaneSegNet.

Published as a conference paper at ICLR 2024
In memoriam Prof. Xiao' ou Tang
LANESEGNET: MAP LEARNING WITH LANE SEGMENT PERCEPTION FOR AUTONOMOUS DRIVING
Tianyu Li Peijin Jia
Bangjun Wang
Li Chen
Kun Jiang
Junchi Yan' Hongyang Li
Fudan University "OpenDriveLab
Tsinghua University
*Shanghai Jiao Tong University
ABSTRACT
A map, as crucial information for downstream applications of an autonomous driving system, is usually represented in lanelines or centerlines. However, ex- isting literature
map learning primarily focuses on either detecting geometry- based lanelines or perceiving topology relationships of centerlines. Both of these methods ignore the intrinsic relationship of lanelines and centerlines, that lane- lines bind centerlines. While simply predicting both types of lane in one model is mutually excluded in learning objective, advocate lane segment representation that seamlessly incorporates both geometry and topology infor- mation. Thus, we introduce LaneSegNet, the first end-to-end mapping net- work generating lane segments to obtain a complete representation of the road structure. Our algorithm features two key modifications. One is lane at- tention module to capture pivotal region details within the long-range feature space. Another is an identical initialization strategy for reference points, which enhances the learning of positional priors for lane attention. On the OpenLane- V2 dataset, LaneSegNet outperforms previous counterparts by a substantial gain across three tasks, i.e., map element detection (+4.8 mAP), centerline perception (+6.9 DET,), and the newly defined one, lane segment perception (+5.6 mAP). Furthermore, obtains real-time inference speed of 14.7 FPS. Code is accessible at https: //gi thub. www. Oeemm/ppeDDDrrr wew.riieeebbbbbbbeee www.aaa.aaaeaeeeeeeeee httpa//cc///eeeeeeeeaaaaaa//aaee
new
INTRODUCTION
Perceiving road information holds significant importance for autonomous vehicles. High-definition (HD) maps serve as a rich resource, providing comprehensive geometry and topology information about the road. However, the widespread deployment of HD maps is often hindered by high annota- tion costs and limited maintainability. To address these challenges, learning-based online mapping methods (Li et al., 2022a; Liao al., 2023b; Li et al., 2023c) have been proposed. These approaches allow autonomous vehicles construct HD maps in real time with onboard sensors and the task of reconstructing the road structure boils down to either map element detection or centerline perception.
The existing literature (Li et al., 2022a; Liao et al., 2023b) focusing the task of map element detection primarily emphasizes the detection of road geometry, as depicted in Fig. ( The map representation is rather oversimplified. Such formulation of HD maps results in the omission of critical details, such as lane direction, topological structure, laneline type, etc., which are vital im- portance for downstream applications such as trajectory planning. To cope with this caveat, certain post-processing techniques are introduced to restore the missing detailed information. Nevertheless, could incur additional computation costs, which are particularly challenging given the tight budget for real-time deployment in massive production.
Another line of studies (Can al., 2021; Li al., 2023c) concentrate on the centerline perception task, aiming to recognize lane topology. as described in Fig. I(c). These methodologies formulate centerlines an abstract or virtual representation of lanes and infer centerline connectivity con- struct the lane graph. The resultant graph feeds the subsequent planning module with topological
"Equal contribution. Primary contact Tianyu Li via t tnayul 1230m. fudan. edu.ch

---

Published as a conference paper at ICLR 2024
road boundary
ped crossing
solid
laneline
ped prossing
conterline
lane segment
nonviisiile
(b) Planform vie of an intercection yyng in the Bay Area.
(6) Map Element Detection
(c) Centerline Perception
(d) Lane Segrmett Perception (Ours)
Figure 1: Comparison of three formulations for online mapping. (a) A realistic scenario of an intersection in the Bay Area, CA. (b) Map Element Detection. It detects lanelines, pedestrian cross- ings, and road boundaries, but fails to adequately indicate topology relationships at intersections and/or situations where lanes diverge/merge. (c) Centerline Perception. It depicts lane topology; yet is a virtual representation and lacks crucial physical geometry information necessary for ensuring safe driving practices. (d) Lane Segment Perception. The proposed formulation captures geometric boundaries and semantic lane types in (b), as well as topological graphs in (c). Note that the pedes- trian crossing is deemed as special case transverse "lane") in the context of lane segment.
information (Liang et al., 2020). However, it lacks detailed geometry information, which is essential for ensuring safety and precise maneuvering during sophisticated driving behaviors.
We position this work to propose unified framework that encompasses a comprehensive description of the map information by simultaneously addressing element detection and centerline perception. One straightforward solution is to use two separate branches, one for learning road geometry and the other for modeling lane topology. However, it is empirically observed that such a multi-branche approach turns out to be redundant in handling correlated tasks.
To this end, we propose a new mapping formulation called lane segment perception as an alternative. As illustrated in Fig. (d)) lane segment includes the geometric boundaries, and retains the directed
topology connection required to construct lane graph. Meanwhile, inherently carries semantic information by encoding line types (dashed, solid, non-visible). Such line types could indicate the crossability of a lane boundary. By doing so, we are able to enhance the complete description of online mapping via interactions with wide diversity of road information.
To embrace the lane segment philosophy, this work introduces an online end-to-end mapping net- work, namely LaneSegNet, to leverage both geometry and topology modeling. LaneSegNet sists three parts, encoder, decoder, and predictor. These components work together to predict lane segments and reconstruct the complete road structure. Due the extra demand recognize lane segments in challenging scenarios, we devise two non-trivial modifications in the decoder. The first is lane attention module to capture important local details within the long-range feature space. This is achieved through a strategy for reference points. contrast the conventional practice of distributed initialization, where each query is initialized with multiple reference points, such a maneuver excels at learning positional priors for lane attention.
mechanism. The second is an identical initialization
LaneSegNet is validated to be effective on three tasks, i.e., map element detection, centerline percep- tion, and the newly proposed lane segment perception on OpenLane- V2 dataset (Wang et al., 2023). A unanimous improvement of 4.8 mAP, 9.6 OLS, and 5.6 mAP for three tasks has been witnessed respectively. Furthermore, LaneSegNet exhibits superior efficiency for online inference (14.7 FPS) with decoder latency decreased by 31.4% compared to the have the potential to further decrease post- processonns time for massive production in autonomous driving. The primary contributions are summarized follows:
baseline. It is deemed to
2

---

Published as a conference paper at ICLR 2024
We introduce lane segment perception new map learning formulation. It incorporates both geometry and topology essentials. We hope it would bring valuable insights to the community.
We propose LaneSegNet, an end-to-end network curated for lane segment perception. Two novel modifications have been proposed, including a lane attention module with a heads-to-regions mechanism for capturing long-range attention, and an identical initialization strategy for refer- ence points to enhance the learning of positional priors for lane attention.
RELATED WORK
Centerline Perception (identical to lane graph learning in this paper) from vehicle-mounted sensor data has garnered significant attention recently. STSU (Can et al., 2021) proposes DETR-like network detect centerlines, followed by multi-layer perceptron (MIP) module to determine their connectivity. Building upon STSU, Can et al. (2022) introduce additional minimal cycle queries to ensure the proper order of overlapping lines. CenterLineDet (Xu et al., 2023) treats centerlines as vertices and designs a graph-updating model trained through imitation learning. Notably, Tesla (2022) proposes the concept of "language of lanes" to represent the lane graph as sentence. Their attention-based model recursively predicts lane tokens and their connec- tivity. In addition to these piece-wise methods, LaneGAP (Liao et al., 2023a) introduces path-wise approach to recover the lane graph using an additional conversion algorithm. Aiming at a complete and diverse driving scene graph, TopoNet (Li et al., 2023c) explicitly models the connectivity of centerlines within the network and incorporates traffic elements into the task. In this work, adopt the piece-wise methodology to construct lane graphs. However, we distinguish ourselves from pre- vious approaches in modeling lane segments instead of centerlines as the vertices of the lane graph, which allows for convenient integration of segment-level geometry and semantic information.
Centerline Perception:
Map Element Detection: In prior works (Chen al., 2022; Huang et al., 2023), attention was paid to elevating map element detection from the camera plane to the 3D space to overcome the projec- tion error. With the trending popularity of BEV perception (Li et al., 2023b), recent works (Philion & Fidler, 2020; Li et al., 2022b; Zhou & Krähenbühl, 2022; Hu et al., 2023; Gao et al., 2023; Liao et al., 2023c) focus on learning HD maps with segmentation and vectorized methods. Map seg- mentation predicts the semantic meaning of each BEV grid, such lanelines, pedestrian crossings, and drivable areas. These works differentiate from each other mainly in the perspective view (PV) to BEV transform module. However, the segmented map cannot provide direct information to be employed by downstream modules. HDMapNet (Li et al., 2022a) handles the problem by grouping and vectorizing the segmented map with complicated post-processings.
Though dense segmentation provides pixel-level information, it still cannot touch down the complex relationship of overlapping elements. VectorMapNet (Liu et al., 2023) proposes directly represent- ing each map element as a sequence of points, using coarse key points to decode laneline locations modeling approach
sequentially. MapTR (Liao et al., 2023b) explores unified permutation- for the sequence of points to eliminate the modeling ambiguity and improve performance and effi- ciency. PivotNet (Ding et al., 2023) further models map elements with pivot-based representation in set prediction framework to reduce redundancy and improve accuracy. StreamMapNet (Yuan et al., 2024) employs multi-point attention and temporal information to facilitate the stability of long-range map element detection. In fact, since vectorization also enriches the direction informa- tion for lanclines, vectorization-base alternating the supervision. In this work, we propose a unified and faaming-friinnde representation, lane segment, for all HD map elements on the road.
methods could be easily adapted to centerline perception by
METHODOLOGY
3.1 TASK STATEMENT OF LANE SEGMENT PERCEPTION
An instance of the lane segment incorporates both geometric and semantic aspects of the road, As for geometry, it can be represented as line section composed of the vectorized centerline and its corresponding lane boundaries: V = {Voeteer Viets Vigha Each line is defined ordered set of N₂ points in 3D space. Moreover, the geometry can also be depicted as a closed polygon that defines the drivable area within this lane.
3

---

Published as a conference paper at ICLR 2024
LaneSeg Encoder
LaneSeg Decoder
LaneSeg Predictor
BEV Feature
Feed-forwart Network
Lane Attention
PV-to-BEV Encoder
clasa
Centerrline
offset
Self-Attention
line
Surrounding ng Muh-View images
topology
Lane Segment Queries
Prodiction MLPs
Lane Segment
Figure Pipeline of LaneSegNet. LaneSegNet is the first end-to-end network curated for the lane segment perception task. It consists of three parts. The LaneSeg encoder generates bird's-eye- view (BEV) features. The core LaneSeg decoder adopts our proposed lane attention module, which employs heads-to-regions mechanism well points. The LaneSeg predictor assembles the elements predicted from different MLPs to form a series of lane segments, thus generating complete description of the road structure.
identical initialization strategy for reference
In terms of semantics, it includes lane segment classification (e.g., lane segment, pedestrian cross- ing) and line type of the left/right lane boundary (e.g., non-visible, solid, dashed): {-Auet Angha )- These details provide autonomous vehicles with essential insights regarding deceleration require- ments and the feasibility of lane changes.
Additionally, topological information plays a vital role in path planning. To represent this informa- tion, lane graph denoted = (V, E) is constructed for lane segments. Each lane segment serves as a node in this graph, represented by the set V, while the edges in set E depict the connectivity among lane segments. We an adjacency matrix to store this lane graph, where the matrix element (i,j) is set to 1 only when the j-th lane segment follows the i-th one; otherwise, it remains 0.
3.2 FRAMEWORK OF LANESEGNET
The overall pipeline of LaneSegNet is depicted in Fig. 2. LaneSegNet takes surrounding view im- ages input to perceive lane segments certain BEV range. In this section, first brief the LaneSeg encoder for generating the BEV feature. Then, we introduce the lane segment decoder and rmed with lane attention. Lastly, we present the lane segment predictor. and training loss.
3.2. LANESEG ENCODER
The encoder transforms multi-view images into a BEV feature, denoted B E H which is used for lane segment extraction. We utilize standard ResNet-50 backbone (He et al., 2016) to derive multi-view feature maps from the raw images. Subsequently, the PV-to-BEY encoder module from BEVFormer (Li et al., 2022b) is employed for view transformation.
3.2.2 LANESEG DECODER
Transformer-based detection methods utilize decoder to collect features from the BEV feature and update the decoder queries through multiple layers. Each decoder layer utilizes self-attention, cross- attention mechanisms, and feed-forward network to update queries. Additionally, learnable positional query is employed. The updated queries are then output and fed to the subsequent stage.
Collecting long-range BEV features is crucial for the online mapping task, owing complex and elongated map geometry. Prior works (Liao et al., 2023b; Luo et al., 2023) utilize hierarchical custance-pointe decoder queries and deformable attention (Zhu et al., 2020) extract the local feature for each point query. While this method evades capturing long-range information, high computational cost ensues due the increased number of queries.
4

---

Published as a conference paper at ICLR 2024
As instance representation of lanes to construct the scene graph, lane segment obsesses superior characteristic on instance-level. Instead of using multiple point queries, we aim at adopting a single instance query represent the lane segment. Consequently, the core challenge lies in how to use single instance query to cross-attend to the global BEV feature.
Lane Attention: In object detection, deformable attention leverages the positional prior of object and attends only to small set of attention values near object's 's reference point as pre-filter, greatly accelerating convergence (Zhu al., 2020). The reference point placed the centriod of predicted object during layer iteration to refine the sampling locations of attention values, which are dispersed around the reference point via learnable sampling offset. The intentional initialization of sampling offiets incorporates geometry prior of 2D objects. By doing so, the features from each direction are captured well with the multi-branch mechanism, as shown in Fig. 3(a).
In the context of map learning, Li et al. (2023c) use naive deformable attention to predict centerline. However, depicted in Fig. 3(b), it may fall short at long-range attention with the naive placement of ref- erence points. Furthermore, due to our tar- get's elongated shape and complex visual cues (e.g., precisely predict the breaking point between solid and dashed laneline), this process requires additional adaptive design for our task. Taking all of those characteristics into consideration, it is nec- essary for networks to possess the abil- ity to not only attend to long-range con- texts but also accurately extract local de- tails. Accordingly, it is advisable dis- tribute sampling locations across a large region to effectively perceive long-range information. On the other hand, local de- tails should remain easily distinguishable for identifying the pivot points. Notably, while competitive interactions are observed among value features within an attention head, the value features between different heads still remain distinct. Therefore, it is promising to explicitly utilize this attribute to facilitate the attention to local features in specific regions.
* point
pamplig locoton
updated position
Ubeuconutta
Tenderborgger
local arrrction
Objjent Detection
Centerline Perception
[4] Lasn (Durs)
Figure 3: Advantage of lane attention. (a,b) Each head with native reference point extracts features from various shape-irrelevan directions, which accounts for why deformable attention can not perform long-range attention for line-shape objects. (c) By the aid of the mechanism, our lane attention mod-
ule effectively gathers long-range context along the elongated lane shape and preserve local details.
To this end, we propose heads- re-iions mechanism. We begin by distributing multiple reference points uniformly within the region of lane segment. The sampling locations are then initialized revolving around each reference point in a local region. To preserve the intricate local details, we utilize the multi-branch mechanism, where each head attends to a specific set of sampling locations within a local region, as illustrated in Fig. 3(c).
Taking into consideration the intuitions above, now provide mathematical description of the lane attention module. Given BEV feature B #xx the i-th lane segment query feature qi E RC and set of reference points p; R Mx2 as input, the lane attention is calculated by:
LaneAtin (g:: Pi, B)
Σ
W
Σ
di,m, W Bi-linear (B, Pi,m Aprim.s)
(1)
where m indexes the attention head, k indexes the sampling locations. The W E RC.xC and W. RCXC, are learnable weights that project the value feature split channel (C, = C/M) and combine the multi-branch feature, thus keep the information of each head. The attention weights
as,m, is predicted via linear projection of q:, and normalized as Σ Oj.m,k = 1 with soft-max function. The final sampling location is combined of reference point pi, and sampling offset Apr.m. which is also predicted via linear projection of q. The sampling offset is intentionally initialized to ensure that K sampling locations are evenly centered around the reference points as prior. The flexibility of learnable sampling offset is still reserved, allowing it to attend to details in the local region or capture information from long- long-range contexts. More implementation details are given in Appendix A.1.3.
5

---

Published as a conference paper at ICLR 2024
Identical Initialization of Reference Points:
The placement of reference points is determinant to the functionality of lane attention module. In order to align each instance query 's attention area with its actual geometry and position, the reference points p of each instance query are distributed based on the lane segment prediction from the previous layer, as illustrated in Fig. 3(c), and the prediction is iteratively refined as well, inspired by Zhu et al. (2020).
Previous work hold that the reference points provided to the first layer ought to be separately ini- tialized with the learnable prior deduced from the positional query embedding. However, since positional query is unrelated to the input images, such initialization method could conversely con- fine the model's capability of memorizing both the geometrical and positional priors and the falsely generated initialization positions will also pose an obstacle for training.
As such, for the first layer of the lane segment decoder, we propose an identical initialization strategy. In the first layer, every head adopts an identical reference point generated from the positional query. Compared to the distributed initialization of reference points as conventional approaches do, i.e, initializing multiple reference points for each query, identical initialization would make the learning of positional prior more stable by filtering out the distraction of intricate geometries. To be noticed, the identical initialization is seemingly counter-intuitive and yet observed to be effective. As detailed in Sec. 4.3, a performance drop has been witnessed in the ablations about using the distributed initialization.
3.2.3 LANESEG PREDICTOR
We employ MLPs in multiple prediction branches to generate the final predicted lane segment from the lane segment query, taking into account geometric, semantic, and topological aspects.
For geometry, we first devise a centerline regression branch to regress the vectorized points position of centerline 3D coordinates. The output Veenter E 2R is formatted as [ yo, ...] Owing
to the symmetry of the left and right lane boundaries, we introduce an offset branch to predict the offset Votiser E I. N33, which is formatted as [ (Axo. Ayo. Azo)....) Thus, the coordinates of left and right lane boundaries can be calculated with Viea = Veener + Votteet and Viihht
Veeneer Votiser
Given that the lane segment can be conceptualized as the drivable area, we integrate an instance segmentation branch into the predictor. The instance mask prediction part follows Cheng et al. (2021). As for the semantics, three classification branches predict the classification score for C, the scores for A and Aright in parallel. The topology branch takes the updated query features input and outputs a weighted adjacent matrix for the lane graph G using MLPs, following Li et al. (2023c).
3.3 TRAINING LOss
Adopting the DETR-like paradigm, LaneSegNet efficiently computes the one-to-one optimal assign- ment between predictions and ground truth with the Hungarian algorithm. Training losses are then computed based on the assignment results. The loss function is composed of four parts: geometric loss, classification loss, laneline type classification loss, and topological loss.
Geometric loss supervises the geometry of each predicted lane segment. According to the bipar- tite matching result, each predicted vectorized lane segment V assigned with a GT lane seg- ment V. The vectorized geometric loss is defined as the Manhattan distance computed between the assigned lane segment pairs. Specifically, the vectorized geometric loss is calculated between V = Veenter Vrigh) } and Y = (Vin, Veenter Viigh) accordingly. Taking full advantage of geo- metric constraints, the explicit manner of supervision on left and right lane boundaries can further centerlines. Besides, a composite loss function consisting of a Cross- Entropy loss and dice loss (Milletari et al., 2016) is applied for the supervision of predicted masks following Cheng et al. (2022)'s manner. As for the supervision for laneline type prediction, we introduce laneline type classification loss which applies cross-entropy loss on (Ate Aright) } and Arighe) correspondingly. Furthermore, inspired by Li et al. (2023c), we introduce the topo- logical loss, which designates the focal loss (Lin et al., 2017b) calculated based on the topology relationship information and & and serves supervise the relationship among lane segments.
benefit the regression

---

Published as a conference paper at ICLR 2024
Table I: Lane segment perception. LaneSegNet surpasses other approaches by a large margin. For consistency, we opted not to add topology head to Map'IR series or a line type head to either MR series or TopoNet, preventing any modifications that might influence the original design.
Method
MAPt
APa1
AP APpaa
TOP
ABtypt
ABa
FPS
TopoNet
23.0
23.9
22.0
1.0
0.769
10.5
MapTR
27.0
25.9
28.1
0.695
14.5
MapTRv2
28.5
26.6
30.4
0.702
13.6
LantSegNet-tiny
28.5
28.2
28.7
6.8
10.6
0.710
16.2
Neg
32.6
32.3
32,9
8.1
9.2
0.673
14.7
EXPERIMENTS
4.1 DATASET AND METRICS
Dataset: We evaluate LaneSegNet on the popular OpenLane- V2 dataset (Wang et al., 2023). This dataset contains 1,000 scenes, with each scene lasting approximately 15 seconds. The training set includes about 27,000 frames, and the validation set includes about 4,800 frames. Labels are further claborately re- annotated our proposed lane segment manner, and the map data from the Argoverse 2 dataset is further processed to align with proposed representation. All lane segments within [-50m, +50m] along the x-axis and [-25m, + +25mm along the y-axis are annotated in the 3D space. Each line in V is represented by ordered set of 10 points during training and evaluation. Details of our data processing approach can be found in Appendix A.2. The data will be released with code.
Metries:
Since the models we are comparing are specifically designed for different tasks, we categorize the evaluation metrics into three parts; map element detection, centerline perception, and lane segment perception, ensure fair comparison.
The evaluation metrics for centerline perception use existing benchmarks presented in OpenLane- V2 (Wang al., 2023). Specifically, the DET, score serves as the average precision (AP) for assess- ing centerline perception performance, primarily relying on the Fréchet distance. The TOP, score measures the performance of topology reasoning. The OLS as a weighted sum also included'.
In the map element detection benchmark, we adopt AP for lane divider (or laneline) and pedes- trian crossing to evaluate the map construction quality under Chamfer distance thresholds of {0.5, 1.0, 1.5) meters. The mean AP is computed the average of the AP values for each class.
Building upon the established metrics for these two mapping tasks, lane segment perception adopts specifically designed metries. lane segment distance Die, corresponding average precision APte and Aeped, the mean AP computed as the average of APIs and peped topology metric TOP and another average errors ABtype and AEgist. All of these metrics are defined in Appendix A.4.
4.2 MAIN RESULTS
Lane Segment Perception: In Tab. 1, we compare LaneSegNet with several state-of-ther meth- ods, MapTR (Liao al., 2023b), MapTRv2 (Liao al., 2023c) and TopoNet (Li al., 2023c), on the newly introduced lane segment perception benchmark. We re-train their model with our lane segment label. The implementation details can be found in Appendix A.3. LaneSegNet outperforms other methods up to 9.6% on mAP, and the average distance error is relatively reduced by 12.5%. LaneSegNet-ting also surpasses previous approaches, with a higher FPS of 16.2.
Qualitative results: As shown in Fig. 4, LaneSegNet is capable of handling complex intersections. For additional qualitative results, please refer to Appendix C.
Map Element Detection: To perform more fair comparison with the map element detection methods (Liu et al., 2023; Liao et al., 2023b), we decompose the predicted lane segments of Lane-
'OLS is short for OpenLane Score which is comprehensive metric defined in OpenLane-Ve V2 (Wang et al., 2023). DET represents the detection metric of certain attributes which are usually given in subscript and calculated in the way of mAP.

---

Published as a conference paper at ICLR 2024
Map element detection. The perfor- mance of LaneSegNet is even superior to ap- proaches that were natively trained on this task. Reshaping the prediction from the lane seg- ment detection model.
Table 2:
Table 3: Centerline perception. Nege outperforms other methods on both centerline perception and topology reasoning. The OLS is calculated between DET, and TOPp. ": Extract- ing centerlines from the lane segment results.
:;
Method
mAPt
A
AP APpedt
Method
OLSt
DET
TOP
VectorMapNet
8.3
5.5
11.1
STSU
13.2
12.6
1.9
MapTR
22.7
16.0
29.4
TopoNet
20.0
24.9
2.3
LameSegNet*
27.5
21.8
33.2
LaneSegNet"
29.7
31.8
7.6
Surrounding Views
GT
LaneSegNet
MapTR
TopoNet
Figure 4: Qualitative results. LaneSegNet is capable of reconstructing the topology of intersections and identifying pedestrian crosswalks. Various types of laneline are also categorized well. However, due to limitations in camera perspective, there are errors in estimating the number of lateral lanes on both sides of the intersection.
SegNet into pairs of lanelines and then compare them with saute-of-the-see methods using map el- ement detection metrics. We feed the disassembled laneline and pedestrian crossing labels into several state-of-the-art methods for re-training. The experimental results, as presented in Tab. 2, reveal that LaneSegNet consistently surpasses other approaches for the map element detection task. Under a fair comparison, LaneSegNet can restore the road geometry better with extra supervision. This indicates that the lane segment learning representation is adept at capturing road geometric information.
Centerline Perception: We also compare LaneSegNet with state-of-the-art centerline perception methods (Can et al., 2021; Li et al., 2023c) in Tab. 3. For consistency, centerlines are also extracted from lane segments for re-training. could be concluded that LaneSegNet's performance is drasti- cally above other methods in lane graph perception tasks. With additional geographical supervision, LeneSegNet also demonstrates superior topology reasoning ability. It proves that reasoning ability is closely correlated with strong localization and detection ability.
4.3 ABLATION STUDY
Lane Segment Formulations: In Tab. 4, we provide ablations to verify the designing merits and training efficiency of our proposed lane segment learning formulation. Compared with separated training models on the first two rows, the joint training of both centerline and map element brings an all-rounded and on average 1.3 improvement for the two main metrics as is presented in row 4, prov- ing the feasibility of multi-task training. However, the vanilla approach to training both centerline and map element in single branch by adding extra categories leads to an obvious performance drop. Our model trained with lane segment label gains salient performance enhancement over vanilla sin- gle branch approaches mentioned above (+7.2 on OLS and +4.44 on mAP for comparison between row 3 and row 5) which validates the positive interaction among various road information in our map learning formulation. Our model even surpasses the multi-branch approach, particularly in cen- terline perception (+4.8 on OLS). This suggests that geometry can guide topology reasoning in our map learning formulation, where the multi-branch model only marginally excels over the CL-only model (+0.6 OLSS between row and row 4). As for the minor decrease on APr... it comes from the reshaping process of our prediction results and is caused by the error line type classification,
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison among different learning formulations. "CL-only" (or "ME-only") refers to using a single branch to learn the centerline perception (or map element detection) task. "CL ME" denotes multi-task formulation to perform both tasks. All the models follow the same architecture without our specific designs mentioned in Sec. 3.2.2.
Formulation
DET,
TOP
mAP
APoi
piod
Latency (ms)
Param.
centerline (CL) only
21.1
22.9
3.7
16.29
5.64M
map clement (ME) only
21.8
17.3
26.2
14.41
5.18M
CL ME, single branch
19.3
22.5
2.6
20.4
17.0
23.7
16.97
5.64M
CL + ME, multi-branch
21.7
23.6
3.9
23.9
19.3
28.4
30.14
10.81M
Lane Segment
26.5
27.2
6.6
24.8
19.1
30.5
20.67
6.74M
Table 5: Comparison among different cross- attention designs. "Deform. Attn." refers to de- formable attention, "inst." means using instance query, "hie." refers to hierarchical query design.
Ablation on the designs in LaneSeg decoder, including heads-to-regions (heads2r.) and identical initialization (ident. init.) of ref- erence points.
Table 6:
Method
Rref
mAP
TOP
Exp.
heads2r.
ident. init.
mAP
TOP1sts
Deform. Attn. (inst.)
28.7
6.9
1
28.8
5.6
Deform. Attn. (hie.)
N,
29.1
5,2
2
29.1
6.2
Multi-Point Attn.
N
24.6
6.9
3
30.6
7.9
Lane Attn. (Ours)
8
32.6
8.1
4
V
32.6
8.1
which is fully acceptable. Besides, the latency of the model adopting lane segment is 10ms quicker than the multi-branch approach (31.4% latency improvement) and the number of parameters is also reduced by 37. 37.% proving that learning formulation has stricken the balance between robust performance and training efficiency.
Lane Attention Module: Ablations of our presented attention module are presented in Tab. 5. In order to facilitate fair comparison, we substitute the lane attention module in our framework with alternative attention designs. With our elaborated designs, LaneSegNet with lane attention significantly outperforms these approaches, showing substantial improvement (+3.9 on mAP and +1.2 on TOP; compared to row 1). Additionally, compared to the hierarchical query design, the decoder latency can be further reduced (from 23.45ms 20. 20.96ms), due to reduction in the number of queries.
Specifically, the ablations several designs proposed in Sec. 3.2.2 are showcased in Tab. 6. The heads-to- to togriion mechanism consistently enhances the geometry prediction, yielding a +2.0 mAP improvement when comparing row 3 to row 4. This demonstrates its superior ability to preserve local details. Furthermore, identical initialization improves mAP by 3.5 by accelerating the convergence. The full results for ablations and more details are provided in the Appendix B.2.2.
CONCLUSION
In this paper, present lane segment perception LaneSegNet, a tailored end-to-end network to address the problem. Along with the network, two innovative enhancements are proposed, including lane attention module, which employs heads- to-regions mechanism to capture long-range attention, and an identical initialization strategy for reference points enhance the learning of positional priors for lane attention. Experiment results on the OpenLane- V2 dataset demonstrate the effectiveness of our designs.
map learning formulation and propose
Limitations and Future Work. Due to the computational limitation, we have not expanded the proposed LaneSegNet to more additional backbones. More popular datasets mentioned in Li al. (2023a), such nuScenes and Waymo are not incorporated, due the missing lane segment annotation. The formulation of lane segment perception and LaneSegNet can potentially benefit downstream tasks which is worth future exploration.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
OpenDriveLab is the autonomous driving team affiliated with Shanghai AI Lab. This work was supported by National Key R&D Program of China (2022200160104), NSFC (62206172), and Shanghai Committee of Science and Technology (23YF1462000), We extend our gratitude to Huijie Wang, Chonghao Sima, and Jia Zeng for their profound discussions.
REFERENCES
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Structured bird seyds--ye---- view traffic scene understanding from onboard images. In ICCV, 2021. 3,
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Lue Van Gool. Topology preserving local road network estimation from single onboard camera image. In CVPR, 2022. 3
Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, and Junchi Yan. PersFormer: 3D lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurlPS, 2021. 6
Bowen Cheng, Ishan Misra, Alexander G. Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 6
Wenjie Ding, Limeng Qiao, Xi Qiu, and Chi Zhang. PivotNet: Vectorized pivot learning for end-to- end hd map construction. In ICCV, 2023. 3
Yulu Gao, Chonghao Sima, Shaoshuai Shi, Shangzhe Di, Si Liu, and Hongyang Li. Sparse dense fusion for 3d object detection. In IROS, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016. 4, 12, 13
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. canming-orientee autonomous driving. In CVPR, 2023.
Shaofei Huang, Zhenwei Shen, Zehao Huang, Zi-han Ding, Jiao Dai, Jizhong Han, Naiyan Wang, and Si Liu. Anchor3dlane: Learning to regress 3d anchors for monocular 3d lane detection. In CVPR, 2023.3
Hongyang Li, Yang Li, Huijie Wang, Jia Zeng, Pinlong Cai, Huilin Xu, Dahua Lin, Junchi Yan, Feng Xu, Lu Xiong, Jingdong Wang. Futang Zhu, Kai Yan, Chunjing Xu, Tiancai Wang. Beipeng Mu, Shaoqing Ren, Zhihui Peng, and Yu Qiao. Open-sourced data ecosystem in autonomous driving: the present and future. arXiv preprint arX1v:2312.03408, 2023a.
Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, et al. Delving into the devils of bird' s-eee-veww perception: A review, evaluation and recipe. PAMI, 2023b.
Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online map construction and evaluation framework. In ICRA, 2022a. 1,3
Tianyu Li, Chen, Hujjie Wang, Yang Li, Jiazhi Yang, Xiangwei Geng, Shengyin Jiang, Yuting Wang, Hang Xu, Chunjing Xu, Junchi Yan, Ping Luo, and Hongyang Li. Graph-based topology reasoning for driving scenes. arXiv preprint arXiv.2304.05277, 2023c. 1, 3, 5, 6, 7, 8, 12, 15
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. BEVFormer: Learning bird's red's-eye-view representation from multi-co multi-camera images via spa- tiotemporal transformers. In ECCV, 2022b. 3, 4, 12, 15
10

---

Published as a conference paper at ICLR 2024
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng. and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
Bencheng Liao, Shaoyu Chen, Bo Jiang, Tianheng Cheng, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Lane graph as path: Continitty-reseerving path-wise modeling for online lane graph construction. arXiv preprint arXiv:2303.088// 2023a.
Bencheng Liao, Shaoyu Chen, Xinggang Wang. Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: Structured modeling and learning for online vectorized HD map con- 1, 4, 7, 15, 16
struction. In ICLR, 2023b.
Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. MapTRR2: end--t--end framework for online vectorized hd map construction. arXiv preprint www.v:2308.......m 05736, 2023c. 3, 15
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017a. 12, 13
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b. 6, 14
Yicheng Liu, Yue Wang. Yilun Wang, and Hang Zhao. map learning. In ICML, 2023. 3,7
VectorMapNet: End-to-end vectorized hd
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 14
Yueru Luo, Chaoda Zheng, Xu Yan, Tang Kun, Chao Zheng, Shuguang Cui, and Zhen Li. LATR: 3d lane detection from monocular images with transformer. In ICCV, 2023. 4
Fausto Milletari, Nassir Navab, and Seyed- Aemedd Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 3DV, 2016. 6,
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting 3D. In ECCV, 2020. 3
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312. 14150 2023. 13
Tesla. Tesla AI Day. https: / www. www.yyuuubee...mmmmmmmmmmm.mm.b
2022. 3
Huijie Wang, Tianyu Li, Yang Li, Li Chen, Chonghao Sima, Zhenbo Liu, Yuting Wang, Shengyin Jiang, Peijin Jia, Bangjun Wang, Feng Wen, Hang Xu, Ping Luo, Junchi Yan, Wei Zhang, and Hongyang Li. Openlane-v2: A topology reasoning benchmark for scene understanding in au- tonomous driving. In NeurIPS Track Datasets and Benchmarks, 2023. 13
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandel- wal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS, 2021. 13
Zhenhua Xu, Yuxuan Liu, Yuxiang Sun, Ming Liu, and Lujia Wang. Centerl Road lane centerline graph detection with vehicle-mounted sensors by transformer for high-definition map creation. In ICRA, 2023.
Tianyuan Yuan, Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao. StreamMapNet: Streaming mapping network for vectorized online hd map construction. In WACV, 2024. 3, 16
Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR, 2022. 3
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020. 4, 5, 6, 12
11

---

Published as a conference paper at ICLR 2024
Appendix
A IMPLEMENTATION DETAILS
A.1 LANESEGNET
A.1.1 BEV FEATURE EXTRACTOR
To obtain image features, we employ ResNet-50 (He al., 2016) with FPN (Lin et al., 20) Three stages of multi-scale feature maps from ResNet-50 are inputted into FPN for fusion and combination. These three features are the downsampling results of the image at scales Sax, Si6x, S32x. The final output features have four levels with an additional S64x level and each level has 256 output channels. Afterward, we utilize three encoder layers, as proposed in BEVFormer (Li al., 2022b) to transform the multi-scale image features into BEV features. The final BEV grid is set to 200 x 100, corresponding to the perception range of +00 m X #:5 m.
A. 1.1 LANE SEGMENT DECODER
In the decoder, utilize six decoder layers based Deformable DETR (Zhu al., 2020), which consists by of three components: a self-attention layer, a cross-attention layer, and a feed-forward network (FFN). The self-attention layer employs attention heads. The cross-altention uses lane attention module which also has 8 attention heads and incorporates 32 offset points around reference points. We adopt two-layer FFN with feed-forward channel size of 512. The initial query em- beddings represented as q = 1qp. q:). consists of 256 channels qp for generating the initial reference point and the other 256 channels qi for the initial instance query. The number of queries set to 200 for lane segments.
A.1.3 LANE ATTENTION
In the process of terds--,--egions, it critical to guarantee that reference points are uniformly dis- persed within lane segment. To achieve this, methodically allocate four reference points along two predicted boundaries of a lane segment. Consequently, lane attention directed to each local area along the lane segment through distinct heads, enabling long range attention and precise local feature extraction. During the sampling offset initialization stage, for each reference point, the num- ber of sampling locations K set to 32. We center around reference point, and distribute four sam- pling locations from near to far in each of the eight directions, following Deformable DETR's (Zhu al., 2020) manner.
A.1.4 LANE SEGMENT PREDICTOR
The three classification branches all consist of three-layer MLP with LayerNorm and ReLU in between, which predicts the confidence score of the lane segment class and its left/right line type. The two regression heads for centerline and offset are both three-layer MLPs with ReLU, which predicts the ordered point set of 10 X 3 for the centerline and its corresponding offset. For mask prediction, use MLP convert the query embeddings q N into mask embeddings Emust E N xC,, Then the N possibly instance mask predictions S can be obtained via a dot product between mask embeddings &mask and BEV features followed by a sigmoid activation.
= sigmoid
X B).
(2)
The topology branch takes the matched query features Q as input and outputs weighted adjacent matrix A for the lane graph For each pair of qs q, E Q, the weight of edge At,, can be calculated as follows:
q: = MLP pre (q:) 9 = MLP suc (qz),
(3)
A,, sigmoid (MLP top (concat(g': (())a)((':g()))
where the parameters are not shared between MLPs for predecessors and for successors, following Li al. (2023c).
12

---

Published as a conference paper at ICLR 2024
boundayy
soldd
lanaline ped prossing
center
Jane seggment
ped crossing
Map Element Detection
(6) Canterline Perception
Oure
Figure 5: Different online mapping formulations.
A.2 DATA PROCESSING
The conventional data format for map element detection, as illustrated in Fig. 5(a), includes visible road geometry such as road boundary, laneline, and pedestrian crossing. On the other hand, the centerline perception, depicted in Fig. 5(b), consists of centerlines and their corresponding topology relationships. In our lane segment formulation, as shown in Fig. 5(c), we combine these two for- mats and introduce line types to create a comprehensive representation. The line type is extremely important for a general agent to make driving decisions (Sima et al., 2023).
We construct the lane segment data based on the Argoverse 2 dataset (Wilson et al., 2021). How- the map Argoverse 2 contains unnecessary breakpoints that do not conform to actual driving rules. To address this problem, we apply a depth-first search (DFS) merging process inspired by OpenLane- V2 (Wang et al., 2023). However, unlike OpenLane-V2 account both topology relationships and line types, the search boundaries for merging expand to encounter diverge, merge, intersection, and changes in the laneline type. Additionally, our data pro- cessing applies not only to the centerline but also to the lanelines of the lane instance, treating them as a whole during the merging process. Regarding line types, while the data processing currently focuses on the most fundamental line types, such as dashed, solid, and non-visible, it is also flexible enough to support the addition of other line types if required.
algorithm takes into
Considering the richness of online maps, we incorporate pedestrian crossing into data and simplify into lane segment format. We treat pedestrian crossing as two edges along its principal axis. Due to the inconsistency in the directions of pedestrian crossing, which can negatively affect the training, we uniformly reorganize the direction the principal axis to face the top-left direction in the BEV coordinate system and both lines should be pointing in nominally the same direction. In the aforementioned processing of lane segments, the boundary information is implicitly included, eliminating the need for separate processing of this class.
Furthermore, we migrate our data to two subtasks: map detection and centerline perception. For map element detection, to maintain consistency with previous conventional algorithms, we extract visible lanelines from lane segments for learning. For centerline perception, we extract the corresponding centerline for learning. Additionally, to align with the learning of TopoNet, the new relationships between traffic elements and centerlines are obtained by finding the corresponding relationship be- tween the centerline OpenLane- V2 and the lane segment, and then modifying the corresponding matrix.
A.3 TRAINING DETAILS
In the BEV feature extraction module, we initially use ResNet-50 (He et al., 2016) as the founda- tional image backbone to extract image features. Subsequently, apply FPN (Lin et al., 2017a) to obtain lulti-level feature maps. Following this, we incorporate 3 encoder layers to facilitate view transformation. Within the lane segment detection module, we utilize methodology based on the
13

---

Published as a conference paper at ICLR 2024
Deformable DETR structure, incorporating stack of 6 decoder layers. During the training process, we employ the Adam W optimizer (Loshchilov & Hutter, 2018) in conjunction with cosine anneal- ing schedule. The initial learning rate is set to 2 10 4. LaneSegNet is trained using 8 NVIDIA Tesla V100 GPUs, with a total batch size 8, over the course of 24 training epochs. Regarding the loss, the final loss function is defined as:
L AvecCvec + Xeeg + Aotohols + Atype + Atophtops
(4)
where Leeg Ace + AdiceCaicns and the hyperparameters are defined as: Xvec 0.025, Xseg 3.0, Ace 1.0, Adice = 1.0, Adl 1.5, Atype 0.01, Atop = 5.0.
To be specific, adopting the DETR-like paradigm, LaneSegNet infers a fixed number of N predic- tions in single pass through the decoder, and the bipartite matching acts as the prerequisite for instance-specific loss optimization. We denote by the ground truth of lane segments, and the N predicted segment sets. bipartite matching can be formulated as an optimization problem that requires finding permutation of N elements with the lowest cost:
ô - arg min oEGN
Σ Emanch (y: - yo.),
(5)
where GN represents the vector space composed of permutations of N elements and maachh (14, No:) is the pair-wise matching cost between ground truth and prediction with in- dex og, This optimized assignment can be efficiently computed with the Hungarian algorithm or Sinkhorn algorithm, following DETR's designing idcology.
In light of so many properties embodied by a single lane segment, one of the primary training diffi- culties lies in finding a proper definition of matching cost between predictions and ground truths of lane segments. LageSegNet, our matching cost takes into account both the semantic classification and the similarity of predicted and ground truth lane segments.
For the semantic classification part, the matching cost is composed of both classification loss Lets and laneline type classification loss Ltype- Lets employs focal loss (Lin et al., 2017b) the pre- dicted class score and the actual class label C. Ltype applies cross-entropy loss on {Ateft Aright) and Arighed correspondingly.
N-1 Lats Σ
Crocal C,).
(6)
The similarity between lane segments can be measured from two dimensions: geographical distance and geometric variance. For the former one, we define fvee as the & loss computed on the predicted vectorized lane V and the GT V. For the latter one, we apply the composite loss consisting of a Cross-Entropp loss and dice loss (Milletari et al., 2016) on the predicted instance mask and corresponding GT, represented as Laeg Xenhee + Adice Ldice-
As for training loss, we augment our matching cost with a topological loss component Lus which designates a Focal loss calculated based on the topological relationship information & and and serves supervise the topological relationships among lane segments.
A.4 METRICS FOR LANE SEGMENT PERCEPTION
Building upon the established metrics for these two mapping tasks, we present the evaluation metrics for lane segment perception. First, we define the lane segment distance weighted sum of the distances between the left/right lane boundaries and centerlines, also considering the lane direction. The resulting lane segment distance, denoted Di., between the prediction V and the ground truth expressed as follows:
(V, V = 0.5 [Chamfer(Meen) Vrightl, (Penn Vrigha) + Fréchet (Veenter, Veemee)
Using the aforementioned distance function, we can calculate an average precision, denoted as APts> over three matching thresholds given by T (1.0, 2.0, 3.0) meters. The AP APped is determined solely
14

---

Published as a conference paper at ICLR 2024
based the Chamfer distance, aiming to assess the non-directional pedestrian crossing, which aligns with thresholds in map element detection. The mean AP (mAP) is computed as the average of AP1s and AP ped- Additionally, a topology metrie, TOP is adapted from T to assess the similarity between the prediction and ground truth lane graphs. It is defined as the averaged vertice mAP between (V, B) and (V', VV,,A E') over all vertices:
ENEN'' P(W)I(W'' E N(c))
1 TOP
Σ VEV
[V
[N())
where (v) denotes the ordered list neighbors vertex e ranked by confidence and P(v) is the precision of the i-th vertex in the ordered list. The TOP is for topology among lane segments on graph (Vs,, Eists).
B EXPERIMENTS
B.1 BASELINES
For all models re-implemented for all the tasks, we uniformly employ ResNet-50 the founda- tional image backbone and apply FPN to generate multi-level feature maps. Regarding the view transformation from surrounding multi-views the bird's-eye s-eee-vwww,, the same encoder adopted from BEVFormer (Li et al., 2022b) is applied.
Lane Segment Perception: In the lane segment perception task, MapTR (Liao et al., 2023b), MapTRv2 (Liao et al., 2023c) and TopoNet (Li et al., 2023c) retrained as our baselines. For im- plementation, different approaches are applied. Specifically, we modify the prediction of 20 points for single laneline in MapTR series into predicting 10 points for the left and right boundaries respectively. The centerline is obtained by averaging the positions of the left and right lane lines. Furthermore, since lane segments have unidirectional nature, abandon the approach that siders different permutations. For the adaptation of TopoNet, we introduce an additional branch to predict the offset and to predict the lane segment as the centerline plus or minus the offset. The traffic element branch is kept for TopoNet. The results of FPS are benchmarked on an NVIDIA A100 GPU.
Map Element Detection: In the task of detecting map elements, both MapTR and LaneSegNet trained using their original methods. However, a slight modification needs to be made regarding the modeling approach for pedestrian crossings, which is changed from polygons to polylines. During the evaluation process, LaneSegNet's predictions are additionally post-processed to convert them into the format of map elements.
Centerline Perception: We select STSU and TopoNet our baseline models for the centerline perception task. To ensure fair comparison, both STSU and TopoNet are trained under the same conditions with the newly extracted centerline labels.
B.2
ABLATION STUDY
B.2.1 LANE SEGMENT FORMULATIONS
This section introduces the specific content of Tab. 4 in experimental details and evaluation details. To fairly illustrate the efficiency of lane segment formulation, we remove specific designs and only utilize the basic Deformable DETR architecture for comparison. For centerline (CL) only, it predicts the centerline parts in the lane segment. For map element (ME) only, it predicts the divider and pedestrian crossing extracted from the lane segment. In the combined training setting, "single- head" means using a shared decoder followed by a classification head to determine the prediction belonging, while "multi-branch" indicates using two separate decoders and heads to learn CL and ME respectively. As for Lane Segment, we use our proposed formulation to train and further process the predictions into corresponding CL and ME. Concerning the metric APaio. leverage the lane type from the lane segment to filter out non-vin non-visible elements and perform further evaluation. As for the latency and the number of parameters,
that compared to ME only, CL only adds
15

---

Published as a conference paper at ICLR 2024
Table 7: Comparison among different cross-attentions. "Deform. Attn." refers deformable
attention, "inst." means using instance query, "hie." refers to hierarchical query design.
Method
inst. query
Ryef
mAPt
A 1
AP pedt
1OP
AErype
ABgiat
Deform, Attn. (init.)
28.7
27.8
29.6
6,9
10.9
0.705
Deform, Attn. (hic.)
Ng
29.1
29.1
29.2
5.2
8.6
0.668
Multi-Point Attn.
Np
24.6
26.1
23.2
6.9
12.9
0.711
Lane Attn. (Ours)
32.6
32.3
32.9
8.1
9.2
0.673
Table 8: Ablation on the designs in LaneSeg decoder, including heads-to to-riions (heads2r.) and
identical initialization (ident. init.) of reference points.
Exp.
heads2r.
ident init.
mAPt
AP
APpdl prdt
AFsypet
AFgi.t
28.8
28.7
28.9
5.6
9.7
0.692
2
29.1
28.6
29.6
6.2
9.4
0.687
3
V
30.6
30.6
30.7
7.9
9.4
0.677
32.6
32.3
32,9
8.1
9.2
0.673
a topology head which increases the model size. These metrics for the multi-branch model are double compared to the single-task model. Additionally, due to the inclusion of offset and line type predictions in the lane segment, its performance in these two aspects is slightly higher compared to the single-task model.
B.2.2 LANE ATTENTION MODULE
Tab. shows the detailed metrics of the ablation the cross-attention module. Initially, employ a native instance query design of deformable attention as baseline, with only one reference point placed at the centroid of lane segment. Opting for the hierarchical query design (Liao al., 2023b) results in marginal improvement in the mAP (+0.44 for comparison between row and row 2), but the topology reasoning performance drops, which can be attributed either inadequate instance rep- resentation or point query issues. Though slightly lagging behind in AEdist- This may be attributed the fact that the hierarchical query provides more learnable space within the internal structure of the lane. In concurrent work, Yuan et al. (2024) propose Multi-Point Attention, which utilizes pre- dicted geometry to adjust the placement reference points and employs instance queries. However, their approach merely relies distributed initialization for reference points without considering the offset of sampling points. This leads to a 4.1 decline in mAP compared to the baseline, suggesting that indiscriminately adding more reference points is not beneficial.
We also provide the full results of ablations on the designs in LaneSeg decoder in Tab. 8. It can be observed that with the inclusion of heads-to-regions and identical initialization mechanisms, the model's 's performance improves in all aspects. Fig. 6 presents the loss with identical initialization and without identical initialization. It can be speculated that the better performance with the inclusion of identical initialization is possibly due to its demonstrated largely improved loss convergence.
B.2.3 IMPACT MASK SUPERVISION
Regarding the ablation experiment on mask supervision, the mask supervision improves the perfor- mance (+2.0 AP and +0.6 tsss. This is likely because incorporating instance mask prediction enhances feature representation within the lane segment region. Additionally, the variations in hy- perparameter Asng do not have significant impact on the results.
C QUALITATIVE VISUALIZATION
As shown in Fig. 7, select three different road structures to showcase the algorithm's perfor- mance as the visualization results. The first structure includes a typical intersection, lane segment diverge, and merge. The second structure represents a long-tail scenario of road around the park with complex ingress and egress. The third structure depicts two T-junctions. LaneSegNet demon- strates superior capability in capturing the geometric variations of these situations compared to other approaches.
16

---

Published as a conference paper at ICLR 2024
Training Loss
160
LaneSegNet w/o ident. init. LaneSegNet
140
100
80
60
40
20
10000
20000
30000
40000
50000
60000
70000
Step
Figure 6: The loss curves with and without identical initialization.
Surrounding
GT
LaneSegNet
MapTR
TopoNet
Figure 7: Qualitative results under different road structures.
17


Reviews:
[{"review_id": "FhDMNshdBz", "reviewer": "Reviewer_VcjB", "summary": "The work presents a lane-segment based approach for map constructions. The goal is to generate a comprehensive representation [ Map elements and centerline combined] of the lanes within the map. The results are competitive with previous approaches with a single model generalizing to multiple lane based tasks. The method achieves non-marginal results on different tasks.", "questions": "It would be nice to see the impact of such a map on downstream tasks such as motion planning/prediction.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written. Detailed explanation of each component. Starting from the introduction to a comprehensive related work review. Then section 3.2 details the approach. \n- Experiments are comprehensive, and non marginal improvments in the evaluation metrics can be seen\n- Ablation study for the different choices of attention mechanism and other design choices validating the model architecture is available. \n- The regression in AP_div is discussed in section 4.3\n\n— \nI increased my score after taking a look into rebuttal and other reviewers. The work is sound in terms of value and a novelty of the concept. It’s true that the new concept is having a form of disadvantage but the results are promising. The results indeed generalize to other datasets as shown in the rebuttal.", "weaknesses": "- It seems there is only one dataset that contains all of the tasks. What about the performance on other datasets where such tasks individually are available. \n- No ablation on the choice for the BEV encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3D6tHwTPV", "reviewer": "Reviewer_eWbz", "summary": "This paper proposes LaneSegNet to annote lane segment for online end-to-end map learning. The network leverages both geometry and topology. It introduces a lane detection module with heads-to-regions for long range attention and identical initialization of reference points to stablize the training.", "questions": "1. How long is each lane segment ?\n2. For a lane segment {A_left, A, A_right}, each appears in 3 lane segments, how to combine them to get final result for e.g. A ?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The results look strong. \n2. Abalation studies were conducted to compare different choices of attentions and initialization.", "weaknesses": "1. The model uses one-to-one optimal assignment between predictions and ground truth with the Hungarian algorithm, which is usually slow and unstable and difficult to train, particularly in the beginning. \n2. While metrics look promising, the metrics are still pretty low. It would be more interested in showing what type of lanes the model can do  so well that it can be used for autonomous driving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uDwNE5fgHy", "reviewer": "Reviewer_jPp9", "summary": "This work introduces LaneSegNet for predicting lane segments from multi-view camera images.  \nThey propose a new representation for lane segments and develop two new techniques for map-prediction architectures.  \nTheir lane attention module is an alternative to deformable cross-attention that aims to better capture long-range interactions, and their reference point initialization is a way to better capture spatial priors of map elements.\nThey perform experiments on the OpenLane-V2 dataset and show improvements on map detection, centerline prediction, and lane segment segmentation.", "questions": "* Take for example a simple road with two lanes (V1, V2 from left to right) - with the proposed representation, this would be represented as V1 = {V1_left, V1_center, V1_right} and V2 = {V2_left, V2_center, V2_right}.  \nHow is the consistency of V1_right and V2_left enforced?  \nIt seems redudant to regress additional left/right lanes instead of simply regressing centerline + adding left/right lane id attributes.\n\n* Why is Argoverse 2 mentioned on page 7?\n\n* Metrics: it would be helpful to readers to define the acronyms for OLS and DET", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "* Strong results on the new OpenLane-V2 dataset. Numbers are solid and their tiny model outperforms  the other baselines.\n* I like the general trend of work that directly predicted structured representations for mapping.", "weaknesses": "* Motivation for a new task - I'm not convinced this task is necessary. Can the authors elaborate why this is a more suitable representation as opposed to the representations in VectorMapNet, MapTR, others?\n* Writing - the majority of Section 3 (method) was not detailed enough. Section 3.3 (training loss) in particular describes several losses very briefly and with no citations. \n* Comparison with prior work - OpenLane-V2 is relatively new and baseline numbers are from authors implementations. It would help calibrate numbers if they applied their architecture to the original tasks conducted in nuScenes.\n* Figures should be more informative - Figure 2 looks like a generic map prediction architecture and Figure 3 does not help illustrate the lane attention module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uWucHJngMN", "reviewer": "Reviewer_afCW", "summary": "This paper presents a novel approach to map learning for autonomous driving systems by utilizing the commonly used PV2BEV feature transformation. In contrast to existing methods, this approach introduces a new representation called \"lane segment,\" which incorporates both geometry and topology information. The proposed model is built upon the BEVFormer architecture and incorporates two key modifications: a lane attention module and an identical initialization strategy for reference points, aimed at enhancing the model's prediction capabilities. The prediction branches of the model consist of multiple MLPs, which collectively generate the final predicted lane segment, including the centerline, laneline, laneline type, and adjacent matrix for the lane topology. The effectiveness of the proposed method has been demonstrated through validation on the OpenLaneV2 dataset, showcasing a significant improvement over other existing approaches.", "questions": "1. MapTRV2 is a stronger baseline but the relative results are not compared in the experiment section.\n2. How to innitialized multiple reference points uniformly inside a lane segment through positional query is not discussed detaily.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces an innovative end-to-end approach to jointly predict the centerline and laneline scheme, which is a unique contribution compared to existing methods.\n2. In contrast to MapTR, which employs hierarchical queries for map element prediction, this method utilizes a single query for both centerline and laneline prediction. The authors propose a heads-to-regions mechanism and distribute reference points evenly within a lane segment, thereby enhancing feature aggregation for both long-range and local image features.\n3. By employing an identical initialization strategy, the model achieves remarkable performance on the OpenLaneV2 dataset, demonstrating its effectiveness and high accuracy.", "weaknesses": "1. The definition of \"long range\" is not clearly defined in the paper. The authors mention that the OpenLaneV2 dataset is reannotated using the proposed lane segment manner, resulting in lanes being broken into multiple segments. By using shorter lane segments, I don't think \"long range\" is challange.\n2. Additionally, manually dividing a lane into segments without any visual cues may introduce unnecessary challenges for the model's prediction, whichi is also not critical for autonomous driving systems.\n3. In the context of map learning, the novelty of topology prediction in this paper is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)