OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping
Authors: Huijie Wang
Tianyu Li
Yang Li
Li Chen
Chonghao Sima
Zhenbo Liu
Bangjun Wang
Peijin Jia
Yuting Wang
Shengyin Jiang
Feng Wen
Hang Xu
Ping Luo
Junchi Yan
Wei Zhang
Hongyang Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D Lane Detection
Traffic Element Recognition
Topology Reasoning
Scene Understanding
Autonomous Driving
Dataset and Benchmark
TL;DR: We introduce the task of scene structure understanding with the newly proposed OpenLane-V2 dataset.

Abstract: Accurately depicting the complex traffic scene is a vital component for autonomous vehicles to execute correct judgments. However, existing benchmarks tend to oversimplify the scene by solely focusing on lane perception tasks. Observing that human drivers rely on both lanes and traffic signals to operate their vehicles safely, we present OpenLane-V2, the first dataset on topology reasoning for traffic scene structure. The objective of the presented dataset is to advance research in understanding the structure of road scenes by examining the relationship between perceived entities, such as traffic elements and lanes. Leveraging existing datasets, OpenLane-V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. It comprises three primary sub-tasks, including the 3D lane detection inherited from OpenLane, accompanied by corresponding metrics to evaluate the model’s performance. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on OpenLane-V2 to indicate future avenues for investigating topology reasoning in traffic scenes.

OpenLane- V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping
Huijie Wang'", Tianyu Li Yang Li Li Chen', Chonghao Sima', Zhenbo Liu', Bangjun Wang', Peijin Jia', Yuting Wang', Shengyin Jiang', Feng Wen", Hang Xu², Ping Junchi Yan', Wei Zhang", Hongyang Li
OpenDriivel.b,, Shanghai Al Lab
Huawei Noah's Ark Lab
Abstract
Accurately depicting the complex traffic scene is vital component for autonomous vehicles to execute correct judgments. However, existing benchmarks tend to oversimplify the scene by solely focusing lane perception tasks. Observing that human drivers rely on both lanes and traffic signals to operate their vehicles safely, we present OpenLane-Ve the first dataset on topology reasoning for traffic scene structure. The objective of the presented dataset is to advance research in understanding the structure of road scenes by examining the relationship between perceived entities, such as traffic elements and lanes. Leveraging existing datasets, OpenLane- V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. It comprises three primary sub-tasks, including the 3D lane detection inherited from OpenLane, accompanied by corresponding metrics to evaluate the model's I's performance. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on Openl.ane- V2 indicate future avenues for investigating topology reasoning in traffic scenes.
Introduction
In recent years, the availability of large-scale datasets and benchmarks has greatly facilitated research on autonomous driving. A critical aspect would be understanding the complex driving environment,
which is the prerequisite for reasonable decisions. Many datasets [1, 10, 18, 44, 45] focus on perceiving visible lanelines to keep vehicles on the right track, while others [13, 14, 37, 39, 46] specified acquiring traffic information through detecting traffic signals. Nevertheless, this separation of tasks represents a limited understanding of the driving scene. For instance, when driving into a crossroad without any visible laneline, an autonomous vehicle might wonder which direction to go. Mcanwhile, when a vehicle proceeds into an intersection where there is a green light presented, is still possible that the traffic signal does not control the lane which the car is driving. this work, we build strong association among traffic elements and lanes, aiming to create topology relationship of the physical world and thus facilitate decision-makkin
the downstream tasks.
To keep autonomous vehicles driving in the correct position, the concept of lanes needs to be introduced. The perception of lanelines, which are the visible separation of lanes, well explored. Previous datasets [1, 18, 25, 45] annotate lanelines on images in the perspective view. Such a 2D representation is insufficient to fulfill real-world requirements. When projecting 2D laneline into bird's s-ew (BEV) space, lane direction would diverge/converge if the height dimension is ignored, leading to improper action decisions in the planning and control module in challenging
"Equal Contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Bird's Bird's-eye-eee WWW--
Front View
Everything okay, keep on this lone.
Ahb, can that Wait, there seem to traffic lights, but can't road tham.
Okay, There are in red. We need to slow down.
But which direction does it lead to?
Figure 1: Motivation and Overview of OpenLane-V2. The dataset comprises various types of annotations, including instances and topology relationships. The directed centerlines provide trajectories for self-driving cars, and their connectivities build the lane network. Traffic elements with semantic labels deliver real-time traffic information. The associations between centerlines and traffic elements imply that traffic element controls some particular lanes based on traffic rules.
scenarios. Recent works [8, 10, 44] define lanelines in the 3D space but still limit the labeling range within the front-view image. Studies on HD map learning [27, 28] incorporate multi-view images to perceive visible road entities, namely lanelines, road boundaries, and pedestrian crossings. However, serving as separations of neighboring lanes, the visible lanelines might not benefit downstream tasks directly. In common circumstances, vehicles follow the center of lanes, i.e., lane centerlines, to drive on the road. To generate this type of invisible and conceptual trajectories, post-processing techniques are required based on the perceived lanelines. However, the desired trajectory becomes empty and the vehicle loses guidance when lanclines are absent, such as driving into a crossroad that typically does not have markings.
Similarly, the perception of traffic signals is formulated classic 2D detection problem on front- view images. Though traffic elements on the roads, such as traffic lights and road signs, provide practical and real-time information, existing formulations [13, 14] emphasize the accuracy of their positions but ignore proper guidance for cars on the road. The reason is that one traffic signal may control one or several lanes according to predefined traffic rules. Given that traffic elements within scene are perceived simultaneously, there exists the possibility for vehicles to be confused about which is the appropriate traffic instruction to obey. Hence, topology relationships between centerlines and traffic elements established to assign traffic information to a particular lane.
As depicted in Figure 1, we seek to unify the aforementioned tasks and provide a comprehensive understanding of driving scenes, including the static entities such as lanes and traffic elements, along with their topology relationships. the task of scene structure perception and reasoning. The requirement of perception is to obtain correct instance-level information, such as positions and semantic meanings, from captured scenes, while reasoning is to deduce topology relationships of perceived entities to generate reasonable understanding of the environment. For the newly defined task, we strive to make our metric capable of covering all aspects of the task. The OpenL Lane-V2 V2 Score (OLS) summarizes model performances with its component of DET and TOP scores for perception and reasoning respectively. Section describes the proposed tasks and metrics.
this end, we propose the OpenLane- V2 dataset to shed light on
Inherited from the OpenLane dataset [8], which is the first real-world and large-scale 3D lane dataset, OpenLane-V2, provides lane annotations in 3D space to reflect their properties in the real world. The directed lane centerlines and their connectivity serve as map-like perception results to facilitate downstream tasks. In addition to the annotations of traffic elements, we establish relationships between centerlines and traffic elements. That is, the correspondence between lane and traffic element is denoted as valid if and only if the traffic element controls the lane. With these representations, self-driving vehicles understand the current driving scenarios and know where to go or whether to accelerate. For more details the proposed dataset, please refer to Section 3.

---

2D Lane Detection
3D Lane Detection
Scene Understanding
Caltech
-AppolloScape
the third dimension
tooology
TuSimmle
-BDD100K
OpenLane ONCE-3DLanes
VVI--100
-CULANE
OpenLane-VZ
VPG
CurveLanee
OpenDenseLane
LLAMAS
OpenLane
Figure 2: Roadmap of lane detection datasets. Most of the previous works provide only 2D labels. Benefiting from the pioneering OpenLane dataset [8]. lane annotations in 3D space have gained great popularity recent years. Taking one step further, the OpenLane- V2 dataset extends the annotation range of 3D lanes to encompass multi-view images and includes topology relationships to promote the task of scene understanding.
To sum up, our contributions are as follows:
We present the Openl ane-V V2 dataset for benchmarking the task of scene understanding. To the best of our knowledge, OpenI.ane- Lan- is the first dataset that focuses on topology reason- ing in the autonomous driving domain. Tasks and corresponding metrics dedicatedly designed evaluate model performance on the proposed benchmark.
Built on top of awesome benchmarks, OpenLane-V2 includes massive images collected from various cities worldwide. It contains 2. instance-level annotations and 1.9M positive topology relationships. All annotations are carefully validated.
We provide a development kit for easy access to the proposed dataset. Besides, plug-ins to prevail deep learning frameworks for training models would be jointly maintained with the community. The test server and leaderboard will also be maintained for fair comparisons.
2 Related Work
2.1 Lane Detection
The task of lane detection has been pursued for several years (Figure 2). Previous works [1, 2, 25, 40, 42, 43] provided 2D laneline annotations in the perspective view. CULANE [31] collected large scale of data and manually annotated the occluded lane markings with cubic splines. With multiple sensors, AppolloScape [i8] included per-pixel lane mark labeling in 35 classes. BDDI0N [45] labeled lanes attributes of continuity (full dashed) and direction (parallel perpendicular) massive amount of data. However, the scope of annotation is still limited in the 2D space on front- view images. OpenLane [8] was the first large-scale, real-world 3D laneline dataset. It is equipped with a wide span of diversity in both data distribution and task applicability. In the spirit of it, the OpenLane- V2 dataset provides 3D annotations of lanes, which the ego vehicle. Instead of focusing on the visible lanelines, annotations of conceptual centerlines in the proposed dataset serve the trajectory guidance for downstream tasks. Moreover, human drivers also observe situations from backward, we provide lane annotations in all directions of the ego car within long range.
the whole surrounding area of
2.2 Traffic Element Recognition
Over the last decade, existing datasets have annotated traffic elements images from the driving scenarios. Table summarizes the relevant counterparts. Most of the works in the early 2010s [4, 16, 30, 37, 39] comprised small amount of data. GTSRB [37] collected data from multiple German landscapes and showed that neural networks could outperform human test persons detecting traffic signs. MTSD [13] made step forward in both scale and diversity that contained 100K street-level

---

Table Comparison current traffic element datasets. "# Img.", "# Cls.", and "# Anno." denote the number of images, classes, and annotations respectively. "Track." implies that a traffic element has a unique tracking ID in different frames. "Corr." indicates whether the correspondences between lanes and traffic elements are annotated. We decompose semantic labels of traffic elements into base
attributes and omit elements that require OCR to acquire their meanings for vision-centrie perception.
Dataset
# Img-
# Cls.
Anno.
Track.
Corr.
Resolution
Region
Year
LaRA [11]
11K
4
9K
X
640x480
France
2009
Stereopolis [4]
847
4
251
X
960x 1080
France
2010
GTSRB [37]
5K
43
39K
X
1360x 1024
Germany
2012
LISA [30]
6K
7K
X
1280
USA
2012
GTSDB [16]
900
43
IK
X
1360
Germany
2013
Belgium [39]
9K
62
13K
X
1628 1236
Belgium
2013
RTSD [34]
179K
156
104K
X
Russia
2016
TT100K [46]
100K
221
26K
2048 2048
China
2016
BSTLD [31
13K
15
24K
X
1280 720
USA
2017
DTLD [14]
344
230K
X
2048 1024
Germany
2018
MTSD [13]
100K
313
325K
X
Worldwide
2020
OpenLanee-V
466K
13*
258K
Worldwide
2023
images worldwide from diverse scenes, geographical locations, and varying weather and lighting conditions. Though with dedicated labels, previous datasets mainly pay attention to the correct location of traffic elements and are limited in the understanding of traffic elements. In addition to the positional label of traffic elements, provide annotations topology relationships of presented objects, enabling autonomous vehicles to have an understanding of the driving environment.
2.3 Scene Understanding
Understanding the driving scene plays a vital role in autonomous driving, especially in complicated scenarios. Few datasets focus on the comprehension of captured scenes. Current datasets [19, 23, 24, 29] comprised 2D images on which there are only small amount of objects. Datasets in the human-object interaction domain [7, 15] limited the labeled relationship to the interactions between human beings and detected objects. The aforementioned datasets include annotations such "cat- ride-snowboard" which are relationships between closely located objects. However, in our case, traffic light may correspond
lane in the distance rather than a closer one. To predict the correct. relationships, models required to have understanding of the predefined traffic rules.
In the field of autonomous driving, previous works try to understand the intention of perceived entities. Tian et al. [38] provided pairwise relationships between moveable objects, e.g., vehicles and pedestrians. Singh er al. [36] defined events as triplets, which comprise agents with their actions and locations. Other datasets paid attention to the intention and future behavior of driver [21, 32] or monnordre non-driver [22, 33] agents. While most of the existing tasks focus on the behavior of foreground movable objects, understanding the static background is also important for the downstream planning module [9. 17, 20, 35). In this work, we emphasize the understanding of the driving scene, which provides trajectory information for self-driving vehicles.
3 OpenLane- V2
In this section, we give overview of the OpenLane-Vi dataset, which is publicly available in our repository. Built on top of the Argoverse [41] and nuScenes [5] datasets, which are both distributed under the CC BY-NC-SA 4.0 license, the proposed dataset includes images 2,000 scene segments collected worldwide under different challenging environments, covering noon and night, sunny and rainy days, downtown and suburbs. Based on the provided HD maps and through a dedicated labeling process, we deliver high-quality annotations with the help of experienced annotators and multiple validation stages. The proposed dataset is under the CC BY-NC-SA 4.0 license, while the code under the Apache License 2.0.

---

Table Statisties of OpenLame-V2. All frames are accompanied by annotations. The annotation range is larger to the front and back compared to that in current methods, which is commonly set to 30m. # is an abbreviation for the number of. Frent-view images are transposed 1550 x 2048.
subset
aubset 83
Split
Train
Val
Test
Train
Val
Test
Sample Rate Annotation Range
2Hz
50m (K-axis), =5m (y-axis)
# Camera
Image Resolution
2048
1550*
Avg. Duration of Scene Segments
15r
20c
# Scene Segment
700
150
150
700
150
1.00
Avg. # Centerline per Frame
26.
26.44
26.50
24.32
24.80
23.82
Traffic Element per Frame
3.70
3.69
2,80
3.58
3,76
3.25
Avg. # Connection per Centerline
1,90
1.89
1.89
1.83
1.79
1.84
Avg. Corresponled Centerline per Element
3.1 Raw Data Acquisition
As camera-centric methods attract large amount of attention in academia and industry, we incorpo- rate multi-view images from original datasets. Due to differences sensor setups, whereby image data is independently collected in Argoverse 2 [41] and nuScenes [5], we divide the proposed dataset into subset_A and subsel_ B respectively, as described Table 2. The subset_A comprises scenes from six cities: Austin (3.1%) Detroit (11.7%), Miami (35,4%), Pittsburgh (35.0%), Palo Alto (2.2%), and Washington D.C. (12.6%), while the subset is collected from two cities: Boston (55,0%) and Singapore (45.0%) The subset A includes 3.0% night scenes and 1.1% rain scenes, while the subset_ B includes 1 11.7% night scenes and 17.4% rain scenes. Despite discrepancies in camera settings, the coordinate system is unified and right-handed. For ego coordinate, the x-axis positive forwards, the y-axis is positive to the left, and the z-axis positive upwards. Camera intrinsics, extrinsics, and ego-vehicle poses in the global coordinate system are provided.
3.2 Centerlines and Their Connectivity
In the provided HD maps, map elements are represented as lane segments, containing boundary, mark type, neighbors, predecessors, successors, etc. The problem is that lanelines are divided based on rules for constructing HD maps but not visually apparent marks, as the primary objective is map the world rather than facilitate autonomous driving directly. This characteristic introduces unnecessary noise and hinders the learning process of models. In this work, we represent single lane as an instance. To generate the ground truth of centerlines, we first regress their locations using the boundary information from HD maps. We then merge lanes with only one predecessor or successor ensure the continuity of lanes. Lanes are separated into different instances if and only if in the cases of intersection, fork, and merge. Topology relationship is then provided on the merged lanes.
The annotation of centerline is provided in 3DD space through ordered list of points. Specifically, for a centerline -P.,...,,,,, P1 = (11, M/, 21) represents the starting point of the lane, while Pa = (In: Mos 2n) denotes the ending point. Note that the ego car is located (0, O, 0) for each frame, and values of the z-axis of subset_Bb are set to 0, as its HD maps exclude the height information. We set n 201 in the given data, but subsample 11 points for each lane for efficient evaluation. The direction of centerline, from the starting point to the ending point, denotes that a vehicle should follow the direction when driving on this lane based on the predefined traffic rules. Topology relationships are provided as adjacency matrixes for each frame based on the ordering of centerlines. Since lane is directed and represented as a list of points, the connection two lanes means that the ending point of a lane is connected to the starting point of another lane. Statistically, about 90% of frames have more than 10 centerlines, while about 10% have more than 40. Most lanes have one predecessor or successor, but in complex scenarios such as crossroads, the number can be up to 7.
3.3 Traffic Elements and Their Correspondence to Centerlines
Traffic elements, such as traffic lights, road markings, and road signs, provide valuable instructions for autonomous vehicles. As critical traffic elements usually exhibited in the front view, and
5

---

their accurate 3D locations are not required for guiding autonomous vehicles, we only annotate them in 2D format on the frow images. Each traffic element is annotated with a 2D bounding box (x1,31, 31; 22, 1/2), where (I1, /1) is the top-left corner and (#2, 12) is the bottom-right corner. Additionally, we label the attributes of each traffic element. detail, the attribute of elements, whose semantic meaning is unobservable, is set to unknown, while valid elements are annotated as red, green, yellow, go go_straight, turn turn_left, turn_ rinht, no no_left_ture _eff___urn, no no_right__ ro__rihht_turn, _urt, u_ u_turn, no _t ___uun,, or slight_ slight_right, right, resulting in 13 various attributes in total. Note that those traffic elements having composite attributes would be divided into multiple annotations with decomposed attributes sharing the same bounding box. For instance, road sign, which is at the position of (æ1. M1 22, V2) and with the meaning of "go straight and turn left", is divided into two bounding boxes, namely (21. y1, T2, y2. go_ vo_straight) and (I1, V1, 22, V2, turn_ arn__eft). The class imbalance should be noticed in that specific traffic elements, such as u u_turn, are much rarer than the common ones like traffic lights in red or green.
The correspondence of a lane and a traffic element forms a regulation for vehicles driving in a particular lane. The construction of relationships between spatially relevant lanes and traffic elements is straightforward. For instance, a lane is controlled by the road marking which is located within its boundary. However, most of the centerlines and traffic elements do not fit into this case. We utilize the following principles for the labeling process. For those traffic element which does not contain directional information, it is associated with centerlines which only going straight is permitted. Traffic elements with directional information, such as traffic lights in the shape of left arrow, control the corresponding lanes going in the same direction. Note that traffic elements only associated with centerlines outside the intersection.
4 Task Definition & Evaluation Metric
In this section, we introduce the tasks and metrics in OpenLane-V2. V2. The primary task of the proposed benchmark scene structure perception and reasoning. which requíres the model to recognize lanes and their dynamic drivable states in the surrounding environment. The challenge includes detecting lane centerlines and traffic elements, recognizing the attributes of traffic elements, and reasoning about the topology relationships perceived entities. We further divide the primary task into three subtasks: 3D lane detection, traffic element recognition, and topology recognition. The OpenLane-V2 V2 Score (OLS), which is the average of various metrics from different subtasks, is defined to describe the overall performance of the primary task:
1 OLS 4
DET: + DET, + ((TOP)) + I(TOP.))
where f is scale function to emphasize the task of topology reasoning.
4.1 3D Lane Detection
In the spirit of the OpenLane dataset [8], which the first real-world and the largest scaled 3D lane dataset to date, we provide lane annotations in 3D space. We define the subtask of 3D lane detection as perceiving directed 3D lane centerlines from the given multi-view images covering a fully panoramic field-of-view (FOV).
Given pair of curves, namely ground truth vz = Ipi, Pa] and prediction in = -f.,,... px). their geometric similarity is measured by the discrete Fréchet distance [12]. Specifically, coupling L is defined as a sequence of pairs between points in v and 0:
(Pay ph, ..... (Pam Ph
where 1 - < 04 < aj < am n and 1 = b, < b, < by S bm = k for all i < j. Then the norm ||Z]| of a coupling L is defined as the distance of the most dissimilar pair in L. The Fréchet distance of a pair of curves is the minimum norm of all possible coupling:
Driecher (vt, tt) = min [ V possible L).
We define threshold t T that pair of centerlines would be regarded unmatched if their distance is greater than t. Then DET, is averaged over match thresholds of T = {1.0, 2.0, 3.0}:
1
DET
APt-
(4)

---

The AP score is the area under the precision-recall curve, defined as So p(r)dr, where and denote precision and recall respectively. Note that as the defined annotation range is relatively large compared to previous datasets, accurate perception of lanes in the distance would be challenging. Thus, the matching thresholds are relaxed for centerlines a distance based on the distance between the ground truth lane and the ego For instance, lane distance would be thresholded on 1.2t while another lane in a closer region would be thresholded on 1.1t for the same threshold E T.
4.2 Traffic Element Recognition
Traffic elements and their attribute provide crucial information for autonomous vehicles. The attribute represents the semantic meaning of traffic element, such the red color of traffic light. In this subtask, on the given image in the front view, the location of traffic elements and their attributes are demanded to be perceived simultaneously. Compared to typical 21) detection datasets, the challenge is that the size of traffic elements is tiny due to the large scale of outdoor environments.
To preserve consistency to the distance mentioned above, modify the common IoU (Intersection over Union) measure as a distance that:
DIoU (us, 01) = -1- 10 n de4
(5)
10 6z
where i, and 0g are the ground truth and predicted bounding box respectively. We consider IoU distance as the affinity measure with match threshold of 0.75. The DET, score is utilized to measure the performance of traffic elements detection and is averaged over different attributes A that:
1 DET,
Σ APa. oE
(6)
[
4.3 Topology Reasoning
We first define the task of recognizing topology relationships in the field of autonomous driving. On the perceived entities, the topology relationships are built. For simplicity, we divide the graph on all entities into two subgraphs. The connectivity of directed lanes establishes map-like network and is denoted as the lane graph (Vi, Fu)). Note that the edge set Fu S Vi x Vi is asymmetric, as the incoming and outgoing edges of a lane represent the connection on its starting and ending points respectively. An entry (i,j) in En is positive if and only if the ending point of the lane is connected to the starting point of Vy. Besides, the undirected graph (V/ Uv, Ett) describes the correspondence between centerlines and traffic elements. It can be seen as bipartite graph that positive edges only exist between Vi and V₁.
The TOP score, which is an mAP metric adapted from link prediction in the graph domain, is utilized to evaluate model performance on the matched graphs. Given a ground truth graph G = (V, E) and a predicted one Ô (V, E), it is possible that the number of predicted vertices is not equal to that of ground truth. We first build a projection between predictions and ground truth to preserve true positive vertices, according to the entity similarity measures, namely Fréchet and loU distances for centerlines and traffic elements respectively. The resulting vertex set Í" needs to fulfill the requirements such that V = V" and V C V U {va), where {ea} is a set of dummy vertices. Then the TOP score is the averaged mAP between (V, E) and (V", Æ') over all vertices:
EWEN' P( E N()))
1 TOP
IV
[N(w]
where (v) denotes the neighbors of vertex v, P(u) is the precision of vertex v the ordered list ranked by predicted confidences, and positive edges are those whose confidence is greater than 0.5. The TOP, is for topology among centerlines on the graph (Vi, FR), while the TOP for topology between lane centerlines and traffic elements the graph (V/ U V4, En).
7

---

Table Quantitative results on the OpenLane- V2 val split. is observed that the model design of how to represent centerlines in the network has an impact on model performance. "Instance" denotes that a centerline represented as a single query the network, while "Point Set" indicates that centerline is described by a set of independent points.
subset_ A
subset __
Method
Design
OLS
DET,
TOPa
TOP..
OLS
DET;
DET,
TOPu
TOP:
STSU
25,4
43.0
8.2
VeciorMapNNt (28]
Point Set
20.8
11.1
41.7
5.9
16.3
3.5
49.1
0.0
1.4
MapTR (27)
Point Set
20.0
8.3
43.5
0.2
5.9
21.1
8.3
54.0
0.1
3.7
TopoNet [26]
Instance
35.4
29.2
48.0
4.1
19.3
33.2
24.3
55.0
2,5
14.2
5
Experiments
In this section, we udapt and evaluate multiple state-of-the-ee methods on the proposed OpenLane- V2 dataset. Visualizations and analysis are then reported investigate the impact of different design choices model performance.
5.1 Baselines
In our experiments, various models are involved, including STSU [6]. VectorMapNet [28]. MapTTR [27], and TopoNet [26]. STSU utilizes DETR-like neural network to detect centerlines and then derive their connectivity by a successive MI.P module. Since is designed for monocular image inputs and contains the BEV representation as intermedia results, we adapt it to multi-view inputs by concatenating BEV embeddings from different views. VectorMapNet directly represents each map element as sequence of points and predicts positional information in auto-regressive manner. MapTR models each map element as a point set with a group of equivalent permutations to deal with geometrical ambiguity. For both methods, the BEV range is expanded to fit dataset requirements. As for training, we supervise VectorMapNet and MapTR with centerlines and use the element queries in the Transformer decoder instance queries to produce topology relationships. TopoNet is specifi- cally designed for the task of scene structure understanding that it utilizes instance-level queries for both centerlines and traffic elements handle their locations and topology relationships. Note that except for TopoNet, methods mentioned above are further appended with heads for predicting traffic elements and topology.
5.2 Results
Quantitative results on the OpenLane-V dataset are illustrated in Table 3. It is not surprising that the DET, scores are low for all methods, since the task of centerline perception is challenging for existing networks. Different from detecting visible lanclines. the centerline is invisible, which requires models to deduce their position by obtaining references from the neighboring lanelines. Additionally, the direction of centerlines is determined by the entire scene, adding extra complexity to the task. For traffic elements recognition, the tiny size of traffic elements, depicted in Figure 3, also introduces difficulty for the models. As the TOP scores require match between ground truths and predictions, performance on the connectivity between centerlines not ideal due the unsatisfactory perception results of centerlines. To increase model performance on topology reasoning, it is must to generate perception results that are sufficiently accurate compared to the ground truth.
In Figure 3, we present visualizations of predicted results a complex crossroad. Although all models provide an approximate shape of the intersection, their performance on the position and shape of centerlines, as well as their topology relationship, can still be improved. STSU generates large number of false positive centerline predictions and fails to attach traffic information centerlines. As MapTR represents points of lane independently in the networks, the predicted centerlines are not in the shape of driving trajectories, which are commonly smooth. Meanwhile, although VectorMapNet utilizes point set to describe a centerline as well, its auto-regressive design ensures proper shape of centerlines. VaptorMapNet, MapTR,, and TopoNet can only provide partially correct results on the semantic information of lanes, namely the correspondence between centerlines and traffic elements. A potential solution could be introducing more prior, such as knowledge of traffic rules, to the network for reasoning about the association between traffic elements of lanc.
8

---

Front View
Bird aodd'-eey-viier vivllllwww
GT
STSU
VectorMapNet
MapTR
TopoNet
Figure 3: Qualitative results of various algorithms. Traffic lights in red and green emphasized with red and green boxes respectively. The bird s-eye-view is truncated at +:25m along the x-axis.
Conclusion
In this paper, we introduce OpenL Lane-aa V2, aiming to facilitate the task of scene structure perception and reasoning. The lane network is represented by lane centerlines and their connectivity, while traffic information described by traffic elements with semantic meanings and their association with lanes. Tasks and metrics are described in detail for future research usage. We adapt various methods and demonstrate their performance on the proposed dataset. We hope this dataset will encourage the research community to design and develop neural networks on the defined tasks, and further promote research in the field of autonomous driving.
Limitations. Due to limitations in available resources, the proposed dataset is built on top of existing datasets, and data scale the same as previous datasets. We believe that including more driving scenes will further increase its diversity. Moreover, although the lane networks with traffic information benefit the downstream tasks, we do not include the planning task in the proposed dataset, as also requires knowledge of moveable objects, such leave the integration of static and dynamic entities to future work.
and pedestrians, for collision avoidance. We
Impact. Based on previous sections, it is evident that the released dataset is used for research purposes. Models trained or evaluated on this dataset should not be directly used for direct deployment or any real-world application. It should be noted that the proposed dataset does not provide any guarantee, particularly safety-critical situations.
Acknowledgements
This work was supported by National Key R&D Program of China (2022200160104) and NSFC (62206172). We thank reviewers for their fruitfuul comments and the research community for partici- pating in the OpenLane Topology Challenge 2023,
References
[1] Mohamed Aly. Real time detection of lane markers in urban streets. In IEEE IV, 2008
[2] Karsten Behrendt and Ryan Soussan. Unsupervised labeled lane markers using maps. In ICCV Workshop, 2019.
[3] Karsten Behrendt, Libor Novak, and Rami Botros. deep learning approach to traffic lights: Detection, tracking, and classification. In ICRA, 2017.

---

[4] Rachid Belaroussi, Philippe Foucher, Jean- Sean-Philipps Tarel, Bahman Soheilian, Pierre Char- bonnier, and Nicolas Paparoditis. Road sign detection in images: A case study. In ICPR. 2010.
[5] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: Á multimodal dataset for autonomous driving. In CVPR, 2020.
[6] Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Lue Van Gool. Structured bird's e-ee-viww traffic scene understanding from onboard images. ICCV, 2021.
[7] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, 2018.
[8] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022.
[9] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306.1660 16927, 2023.
[10] Xiaolei Chen, Wenlong Liao, Bin Liu, Junchi Yan, and Tao He. Opendenselane: dense lidar-based dataset for hd map construction. In ICME, 2022.
[11] Raoul De Charette and Fawzi Nashashibi. Real time visual traffic lights recognition based on spot light detection and adaptive traffic lights templates. In IEEE IV, 2009.
[12] Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. 1994.
[13] Christian Ertler, Jerneja Mislej. Tobias Ollmann, Lorenzo Porzi, Gerhard Neuhold, and Yubin Kuang. The mapillary traffic sign dataset for detection and classification on a global scale. In ECCV, 2020.
[14] Andreas Fregin, Julian Muller, Ulrich Krebel, and Klaus Dietmayer. The driveu traffic light dataset: Introduction and comparison with existing datasets. In ICRA, 2018.
[15] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint 1ene::0550044444 2015.
[16] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In IJCNN, 2013.
[17] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-orientee autonomous driving. In CVPR, 2023.
[18] Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE TPAMI, 2019.
[19] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[20] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end au- tonomous driving. In ICCV, 2023.
[21] Julian FP Kooij, Fabian Flohr, Ewoud AT Pool, and Dariu Gavrila. Context-based path prediction for targets with switching dynamics. IJCV, 2019.
[22] Julian Francisco Pieter Kooij, Nicolas Schneider, Fabian Flohr, and Dariu M Gavrila. Context- based pedestrian path prediction. In ECCV, 2014.
10

---

[23] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2017.
[24] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[25] Scokju Lee, Junsik Kim, Jae Shin Yoon, Seunghak Shin, Oleksandr Bailo, Namil Kim, Hee-Hee Lee, Hyun Seok Hong, Seung-Hoon Han, and In So Kweon. Vpgnet: Vanishing point guided network for lane and road marking detection and recognition. In ICCV, 2017.
[26] Tianyu Li, Li Chen, Huijie Wang, Yang Li, Jiazhi Yang. Xiangwei Geng, Shengyin Jiang, Yuting Wang, Hang Xu, Chunjing Xu, Junchi Yan, Ping Luo, and Hongyang Li. Graph-based topology reasoning for driving scenes, arXiv preprint arXiv:2304.05211 2023.
[27] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng. Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: Structured modeling and learning for online vectorized HD map construction. In ICLR, 2023.
[28] Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao. Vectormapnet: End-to-end vectorized hd map learning. arXiv preprint arXiv:2206. 08920, 2022.
[29] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016.
[30] Andreas Mogelmose, Mohan Manubhai Trivedi, and Thomas B Moeslund. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE TITS, 2012.
[31] Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial enn for traffic scene understanding. In AAAI, 2018.
[32] Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene understanding: dataset for learning driver behavior and causal reasoning. In CVPR, 2018.
[33] Amir Rasouli, Iuliaa Kotseruba, and John K Tsotsos. Are they going to cross? benchmark dataset and baseline for pedestrian crosswalk behavior. In ICCV Workshop, 2017.
[34] Vladislav Igorevich Shakhuro and AS Konouchine. Russian traffic sign images dataset. Com- puter optics, 2016.
[35] Chonghao Sima, Wenwen Tong, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, and Hongyang Li. Scene occupancy. In ICCV, 2023.
[36] Gurkirt Singh, Stephen Akrigg. Manuele Di Maio, Valentina Fontana, Reza Javanmard Ali- tappeh, Salman Khan, Suman Saha, Kossar Jeddisaravi, Farzad Yousefi, Jacob Culley, al. Road: The road event awareness dataset for autonomous driving. IEEE TPAMI, 2022.
[37] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 2012.
[38] Yafu Tian, Alexander Carballo, Ruifeng Li, and Kazuya Takeda. Road scene graph: A semantic graph- scene representation dataset for intelligent vehicles. arXiv preprint 1.Ki:20011.13888 2020.
[39] Radu Timofte, Karel Zimmermann, and Lue Van Gool. Multi-view traffic sign detection, recognition, and 3d localisation. Machine vision and applications. 2014.
[40] TuSimple. https: htops:://githbb http:::/githhb..c/////mmmmm httpa//..mmm//mmmm////s/m//////a http:///hsppp/tuuuiineeeeeeeeeee 2017.

---

[41] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint ..Xiv:3301.00493 2023.
[42] Tao Wu and Ananth Ranganathan. A practical system for road marking detection and recognition.
In IEEE IV, 2012.
[43] Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Curvelane- nas: Unifying lane-sensitive architecture search and adaptive point blending. In ECCV, 2020.
[44] Fan Yan, Ming Nie, Xinyue Cai, Jianhua Han, Hang Xu, Zhen Yang, Chaoqiang Ye, Yanwei Fu, Michael Bi Mi, and Li Zhang. Once-3dlanes: Building monocular 3d lane detection. In CVPR, 2022.
[45] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[46] Zhe Zhu, Dun Liang. Songhai Zhang, Xiaolei Huang, Baoli Li, and Shimin Hu. Traffic-sign detection and classification in the wild. In CVPR, 2016.
12


Reviews:
[{"review_id": "T5z75iX9Uz", "reviewer": "Reviewer_hUvy", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Comprehensive dataset: The OpenLane-V2 dataset unifies lanes, traffic elements, and topology relationships, providing a more realistic and holistic map representation of driving scenarios.\n\n2. 3D lane annotations: The authors introduce 3D lane centerlines and their connectivity in a bird's-eye-view space, reflecting real-world driving properties more accurately and overcoming the limitations of 2D lane annotations.\n\n3. Holistic metric: The OpenLane-V2 Score (OLS) combines perception and reasoning components, offering a comprehensive evaluation of model performance on the proposed benchmark.\n\n4. Extensive annotations: The dataset contains a substantial number of instance-level annotations and positive topology relationships, making it a valuable resource for training and evaluating autonomous driving models.\n\n5. Community involvement: The provision of a development kit, plug-ins for deep learning frameworks, and a test server with a leaderboard demonstrates the authors' commitment to community involvement and fair comparisons.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xr8F2DKQ4i", "reviewer": "Reviewer_wxkw", "summary": "", "questions": "", "limitations": "The authors should provide insights into the diversity of conditions under which data was collected. This includes various weather conditions, times of the day, geographical locations, and traffic scenarios. A diverse dataset is crucial for training robust autonomous systems.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "* OpenLane-V2 dataset  is an effort to bridge the gap between various disjointed datasets in autonomous driving, offering a comprehensive understanding of driving scenes by integrating both lanelines and traffic signals.\n* OpenLane-V2 is also the first dataset that works on topology reasoning which is useful for autonomous driving system\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "K0Stog5FXG", "reviewer": "Reviewer_79Ej", "summary": "", "questions": "", "limitations": "The limitations were discussed in the paper.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper tackles an important problem and it is very easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "uv6QBzxYeI", "reviewer": "Reviewer_bqmb", "summary": "", "questions": "", "limitations": "The authors indicated the limitations of their work (class imbalance in traffic elements, diversity needs to be increased, planning task is not included in the dataset) as well as the possible negative societal impacts of their work. The comments in the 'Opportunities for Improvement' part of the review that are not covered by the 'Limitations' section in the paper might result in improvements in the dataset.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Creating a dataset that couples visible lane perception and traffic signals enable a richer scene understanding than treating them separately. Understanding the topologic relations of the roads and traffic elements in the physical world allows the development of models that simplify decision-making in the downstream tasks.\n- OpenLane-V2 made use of well-known, publicly available datasets. This solution enables the easy utilization of the proposed dataset for the research community.\n- The association of the traffic elements with corresponding centerlines (i.e., scene structure perception and reasoning) is a vital problem of autonomous driving without publicly available datasets built for this purpose. This problem might be alleviated with OpenLane-V2.\n- The centerline is a good representation.\n- Real-world, large-scale dataset (2000 scenes, 2.1M instance-level annotations, 1.9M topology relationships).\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents a dataset for autonomous driving called OpenLane-V2 (extending the previous OpenLane) which augments lane annotations with traffic elements and topology relationship annotations. 2K road scenes have been annotated. 3 tasks and multiple competent baseline methods have been benchmarked on the proposed dataset. While there are some concerns from the reviewers regarding the clarify of the title, the difference compared to HD Map Learning, and many details that require further clarification, most of which have been well addressed during the rebuttal. The reviewers and AC agree mostly on recognizing the contributions of this paper on the dataset and annotation contribution, and its significance and importance to the field of study. Therefore, the AC recommends accepting this paper.
Decision: Accept (Poster)