SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huijie Wang $^{1*}$ , Tianyu Li $^{1*}$ , Yang Li $^{1*}$ , Li Chen $^{1}$ , Chonghao Sima $^{1}$ , Zhenbo Liu $^{2}$ , Bangjun Wang $^{1}$ , Peijin Jia $^{1}$ , Yuting Wang $^{1}$ , Shengyin Jiang $^{1}$ , Feng Wen $^{2}$ , Hang Xu $^{2}$ , Ping Luo $^{1}$ , Junchi Yan $^{1}$ , Wei Zhang $^{2}$ , Hongyang Li $^{1}$
<sup>1</sup>OpenDriveLab, Shanghai AI Lab  <sup>2</sup>Huawei Noah's Ark Lab
https://github.com/OpenDriveLab/OpenLane-V2
Accurately depicting the complex traffic scene is a vital component for autonomous vehicles to execute correct judgments. However, existing benchmarks tend to oversimplify the scene by solely focusing on lane perception tasks. Observing that human drivers rely on both lanes and traffic signals to operate their vehicles safely, we present OpenLane-V2, the first dataset on topology reasoning for traffic scene structure. The objective of the presented dataset is to advance research in understanding the structure of road scenes by examining the relationship between perceived entities, such as traffic elements and lanes. Leveraging existing datasets, OpenLane-V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. It comprises three primary sub-tasks, including the 3D lane detection inherited from OpenLane, accompanied by corresponding metrics to evaluate the model's performance. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on OpenLane-V2 to indicate future avenues for investigating topology reasoning in traffic scenes.
In recent years, the availability of large-scale datasets and benchmarks has greatly facilitated research on autonomous driving. A critical aspect would be understanding the complex driving environment, which is the prerequisite for reasonable decisions. Many datasets [1, 10, 18, 44, 45] focus on perceiving visible lanes to keep vehicles on the right track, while others [13, 14, 37, 39, 46] are specified in acquiring traffic information through detecting traffic signals. Nevertheless, this separation of tasks represents a limited understanding of the driving scene. For instance, when driving into a crossroad without any visible laneline, an autonomous vehicle might wonder which direction to go. Meanwhile, when a vehicle proceeds into an intersection where there is a green light presented, it is still possible that the traffic signal does not control the lane in which the car is driving. In this work, we build a strong association among traffic elements and lanes, aiming to create a topology relationship of the physical world and thus facilitate decision-making in the downstream tasks.
To keep autonomous vehicles driving in the correct position, the concept of lanes needs to be introduced. The perception of lanelines, which are the visible separation of lanes, is well explored. Previous datasets [1, 18, 25, 45] annotate lanes on images in the perspective view. Such a 2D representation is insufficient to fulfill real-world requirements. When projecting 2D laneline into bird's-eye-view (BEV) space, lane direction would diverge/converge if the height dimension is ignored, leading to improper action decisions in the planning and control module in challenging
Figure 1: Motivation and Overview of OpenLane-V2. The dataset comprises various types of annotations, including instances and topology relationships. The directed centerlines provide trajectories for self-driving cars, and their connectivities build the lane network. Traffic elements with semantic labels deliver real-time traffic information. The associations between centerlines and traffic elements imply that a traffic element controls some particular lanes based on traffic rules.
scenarios. Recent works [8, 10, 44] define lamines in the 3D space but still limit the labeling range within the front-view image. Studies on HD map learning [27, 28] incorporate multi-view images to perceive visible road entities, namely lanes, road boundaries, and pedestrian crossings. However, serving as separations of neighboring lanes, the visible lanes might not benefit downstream tasks directly. In common circumstances, vehicles follow the center of lanes, i.e., lane centerlines, to drive on the road. To generate this type of invisible and conceptual trajectories, post-processing techniques are required based on the perceived lanes. However, the desired trajectory becomes empty and the vehicle loses guidance when lanes are absent, such as driving into a crossroad that typically does not have markings.
Similarly, the perception of traffic signals is formulated as a classic 2D detection problem on front-view images. Though traffic elements on the roads, such as traffic lights and road signs, provide practical and real-time information, existing formulations [13, 14] emphasize the accuracy of their positions but ignore proper guidance for cars on the road. The reason is that one traffic signal may control one or several lanes according to predefined traffic rules. Given that all traffic elements within a scene are perceived simultaneously, there exists the possibility for vehicles to be confused about which is the appropriate traffic instruction to obey. Hence, topology relationships between centerlines and traffic elements are established to assign traffic information to a particular lane.
As depicted in Figure 1, we seek to unify the aforementioned tasks and provide a comprehensive understanding of driving scenes, including the static entities such as lanes and traffic elements, along with their topology relationships. To this end, we propose the OpenLane-V2 dataset to shed light on the task of scene structure perception and reasoning. The requirement of perception is to obtain correct instance-level information, such as positions and semantic meanings, from captured scenes, while reasoning is to deduce topology relationships of perceived entities to generate a reasonable understanding of the environment. For the newly defined task, we strive to make our metric capable of covering all aspects of the task. The OpenLane-V2 Score (OLS) summarizes model performances with its component of DET and TOP scores for perception and reasoning respectively. Section 4 describes the proposed tasks and metrics.
Inherited from the OpenLane dataset [8], which is the first real-world and large-scale 3D lane dataset, OpenLane-V2, provides lane annotations in 3D space to reflect their properties in the real world. The directed lane centerlines and their connectivity serve as map-like perception results to facilitate downstream tasks. In addition to the annotations of traffic elements, we establish relationships between centerlines and traffic elements. That is, the correspondence between a lane and a traffic element is denoted as valid if and only if the traffic element controls the lane. With these representations, self-driving vehicles understand the current driving scenarios and know where to go or whether to accelerate. For more details on the proposed dataset, please refer to Section 3.
2D Lane Detection
- Caltech Lanes
-TuSimple
VIL-100
- VPG
- LLAMAS
- ApolloScope
-BDD100K
- CULANE
- CurveLanes
-OpenLane
the third dimension
3D Lane Detection
- OpenLane
-ONCE-3DLanes
-OpenDenseLane
topology
- OpenLane-V2
Figure 2: Roadmap of lane detection datasets. Most of the previous works provide only 2D labels. Benefiting from the pioneering OpenLane dataset [8], lane annotations in 3D space have gained great popularity in recent years. Taking one step further, the OpenLane-V2 dataset extends the annotation range of 3D lanes to encompass multi-view images and includes topology relationships to promote the task of scene understanding.
Scene Understanding
- We present the OpenLane-V2 dataset for benchmarking the task of scene understanding. To the best of our knowledge, OpenLane-V2 is the first dataset that focuses on topology reasoning in the autonomous driving domain. Tasks and corresponding metrics are dedicatedly designed to evaluate model performance on the proposed benchmark.
- Built on top of awesome benchmarks, OpenLane-V2 includes massive images collected from various cities worldwide. It contains 2.1M instance-level annotations and 1.9M positive topology relationships. All annotations are carefully validated.
- We provide a development kit for easy access to the proposed dataset. Besides, plug-ins to prevail deep learning frameworks for training models would be jointly maintained with the community. The test server and leaderboard will also be maintained for fair comparisons.
2.1 3D Lane Detection
The task of lane detection has been pursued for several years (Figure 2). Previous works [1, 2, 25, 40, 42, 43] provided 2D laneline annotations in the perspective view. CULANE [31] collected a large scale of data and manually annotated the occluded lane markings with cubic splines. With multiple sensors, ApolloScape [18] included per-pixel lane mark labeling in 35 classes. BDD100K [45] labeled lanes attributes of continuity (full or dashed) and direction (parallel or perpendicular) on a massive amount of data. However, the scope of annotation is still limited in the 2D space on front-view images. OpenLane [8] was the first large-scale, real-world 3D laneline dataset. It is equipped with a wide span of diversity in both data distribution and task applicability. In the spirit of it, the OpenLane-V2 dataset provides 3D annotations of lanes, which cover the whole surrounding area of the ego vehicle. Instead of focusing on the visible lanelines, annotations of conceptual centerlines in the proposed dataset serve as the trajectory guidance for downstream tasks. Moreover, as human drivers also observe situations from backward, we provide lane annotations in all directions of the ego car within a long range.
2.2 Traffic Element Recognition
Over the last decade, existing datasets have annotated traffic elements on images from the driving scenarios. Table 1 summarizes the relevant counterparts. Most of the works in the early 2010s [4, 16, 30, 37, 39] comprised a small amount of data. GTSRB [37] collected data from multiple German landscapes and showed that neural networks could outperform human test persons in detecting traffic signs. MTSD [13] made a step forward in both scale and diversity that contained 100K street-level
Table 1: Comparison of current traffic element datasets. "#Img.", "#Cls.", and "#Anno." denote the number of images, classes, and annotations respectively. "Track." implies that a traffic element has a unique tracking ID in different frames. "Corr." indicates whether the correspondences between lanes and traffic elements are annotated. * We decompose semantic labels of traffic elements into base attributes and omit elements that require OCR to acquire their meanings for vision-centric perception.
Dataset	#Img.	#Cls.	#Anno.	Track.	Corr.	Resolution	Region	Year
LaRA [11]	11K	4	9K	✓	X	640×480	France	2009
Stereopolis [4]	847	4	251	X	X	960×1080	France	2010
GTSRB [37]	5K	43	39K	✓	X	1360×1024	Germany	2012
LISA [30]	6K	49	7K	X	X	1280×960	USA	2012
GTSDB [16]	900	43	1K	X	X	1360×800	Germany	2013
BelgiumTS [39]	9K	62	13K	X	X	1628×1236	Belgium	2013
RTSD [34]	179K	156	104K	✓	X	-	Russia	2016
TT100K [46]	100K	221	26K	X	X	2048×2048	China	2016
BSTLD [3]	13K	15	24K	✓	X	1280×720	USA	2017
DTLD [14]	-	344	230K	X	X	2048×1024	Germany	2018
MTSD [13]	100K	313	325K	X	X	-	Worldwide	2020
OpenLane-V2	466K	13*	258K*	✓	✓	-	Worldwide	2023
images worldwide from diverse scenes, geographical locations, and varying weather and lighting conditions. Though with dedicated labels, previous datasets mainly pay attention to the correct location of traffic elements and are limited in the understanding of traffic elements. In addition to the positional label of traffic elements, we provide annotations on topology relationships of presented objects, enabling autonomous vehicles to have an understanding of the driving environment.
2.3 Scene Understanding
Understanding the driving scene plays a vital role in autonomous driving, especially in complicated scenarios. Few datasets focus on the comprehension of captured scenes. Current datasets [19, 23, 24, 29] comprised 2D images on which there are only a small amount of objects. Datasets in the human-object interaction domain [7, 15] limited the labeled relationship to the interactions between human beings and detected objects. The aforementioned datasets include annotations such as "catride-snowboard", which are relationships between closely located objects. However, in our case, a traffic light may correspond to a lane in the distance rather than a closer one. To predict the correct relationships, models are required to have an understanding of the predefined traffic rules.
In the field of autonomous driving, previous works try to understand the intention of perceived entities. Tian et al. [38] provided pairwise relationships between moveable objects, e.g., vehicles and pedestrians. Singh et al. [36] defined events as triplets, which comprise agents with their actions and locations. Other datasets paid attention to the intention and future behavior of driver [21, 32] or non-driver [22, 33] agents. While most of the existing tasks focus on the behavior of foreground movable objects, understanding the static background is also important for the downstream planning module [9, 17, 20, 35]. In this work, we emphasize the understanding of the driving scene, which provides trajectory information for self-driving vehicles.
3 OpenLane-V2
In this section, we give an overview of the OpenLane-V2 dataset, which is publicly available in our repository. Built on top of the Argoverse 2 [41] and nuScenes [5] datasets, which are both distributed under the CC BY-NC-SA 4.0 license, the proposed dataset includes images in 2,000 scene segments collected worldwide under different challenging environments, covering noon and night, sunny and rainy days, downtown and suburbs. Based on the provided HD maps and through a dedicated labeling process, we deliver high-quality annotations with the help of experienced annotators and multiple validation stages. The proposed dataset is under the CC BY-NC-SA 4.0 license, while the code is under the Apache License 2.0.
Table 2: Statistics of OpenLane-V2. All frames are accompanied by annotations. The annotation range is larger to the front and back compared to that in current methods, which is commonly set to  $\pm 30m$ . # is an abbreviation for the number of. * Front-view images are transposed to  $1550 \times 2048$ .
Split	subset_A	subset_B
Train	Val	Test	Train	Val	Test
Sample Rate	2Hz
Annotation Range	±50m (x-axis), ±25m (y-axis)
# Camera	7	6
Image Resolution	2048 × 1550*	1600 × 900
Avg. Duration of Scene Segments	15s	20s
# Scene Segment	700	150	150	700	150	150
Avg. # Centerline per Frame	26.34	26.44	26.50	24.32	24.80	23.82
Avg. # Traffic Element per Frame	3.70	3.69	2.80	3.58	3.76	3.25
Avg. # Connection per Centerline	1.90	1.89	1.89	1.83	1.79	1.84
Avg. # Corresponded Centerline per Traffic Element	0.71	0.83	0.91	0.54	0.52	0.58
3.1 Raw Data Acquisition
As camera-centric methods attract a large amount of attention in academia and industry, we incorporate multi-view images from original datasets. Due to differences in sensor setups, whereby image data is independently collected in Argoverse 2 [41] and nuScenes [5], we divide the proposed dataset into subset_A and subset_B respectively, as described in Table 2. The subset_A comprises scenes from six cities: Austin (3.1%), Detroit (11.7%), Miami (35.4%), Pittsburgh (35.0%), Palo Alto (2.2%), and Washington D.C. (12.6%), while the subset_B is collected from two cities: Boston (55.0%) and Singapore (45.0%). The subset_A includes 3.0% night scenes and 1.1% rain scenes, while the subset_B includes 11.7% night scenes and 17.4% rain scenes. Despite discrepancies in camera settings, the coordinate system is unified and right-handed. For ego coordinate, the x-axis is positive forwards, the y-axis is positive to the left, and the z-axis is positive upwards. Camera intrinsics, extrinsics, and ego-vehicle poses in the global coordinate system are provided.
3.2 Centerlines and Their Connectivity
In the provided HD maps, map elements are represented as lane segments, containing boundary, mark type, neighbors, predecessors, successors, etc. The problem is that lanelines are divided based on rules for constructing HD maps but not visually apparent marks, as the primary objective is to map the world rather than facilitate autonomous driving directly. This characteristic introduces unnecessary noise and hinders the learning process of models. In this work, we represent a single lane as an instance. To generate the ground truth of centerlines, we first regress their locations using the boundary information from HD maps. We then merge lanes with only one predecessor or successor to ensure the continuity of lanes. Lanes are separated into different instances if and only if in the cases of intersection, fork, and merge. Topology relationship is then provided on the merged lanes.
The annotation of a centerline is provided in 3D space through an ordered list of points. Specifically, for a centerline  $[p_1, \dots, p_n]$ ,  $p_1 = (x_1, y_1, z_1)$  represents the starting point of the lane, while  $p_n = (x_n, y_n, z_n)$  denotes the ending point. Note that the ego car is located at  $(0, 0, 0)$  for each frame, and values of the z-axis of subset\_B are set to 0, as its HD maps exclude the height information. We set  $n$  to 201 in the given data, but subsample 11 points for each lane for efficient evaluation. The direction of a centerline, from the starting point to the ending point, denotes that a vehicle should follow the direction when driving on this lane based on the predefined traffic rules. Topology relationships are provided as adjacency matrixes for each frame based on the ordering of centerlines. Since a lane is directed and represented as a list of points, the connection of two lanes means that the ending point of a lane is connected to the starting point of another lane. Statistically, about  $90\%$  of frames have more than 10 centerlines, while about  $10\%$  have more than 40. Most lanes have one predecessor or successor, but in complex scenarios such as crossroads, the number can be up to 7.
3.3 Traffic Elements and Their Correspondence to Centerlines
Traffic elements, such as traffic lights, road markings, and road signs, provide valuable instructions for autonomous vehicles. As critical traffic elements are usually exhibited in the front view, and
their accurate 3D locations are not required for guiding autonomous vehicles, we only annotate them in 2D format on the front-view images. Each traffic element is annotated with a 2D bounding box  $(x_{1},y_{1},x_{2},y_{2})$ , where  $(x_{1},y_{1})$  is the top-left corner and  $(x_{2},y_{2})$  is the bottom-right corner. Additionally, we label the attributes of each traffic element. In detail, the attribute of elements, whose semantic meaning is unobservable, is set to unknown, while valid elements are annotated as red, green, yellow, go_straight, turn_left, turn_right, no_left_turn, no_right_turn, u_turn, no_u_turn, slight_left, or slight_right, resulting in 13 various attributes in total. Note that those traffic elements having composite attributes would be divided into multiple annotations with decomposed attributes sharing the same bounding box. For instance, a road sign, which is at the position of  $(x_{1},y_{1},x_{2},y_{2})$  and with the meaning of "go straight and turn left", is divided into two bounding boxes, namely  $(x_{1},y_{1},x_{2},y_{2},go\_straight)$  and  $(x_{1},y_{1},x_{2},y_{2},turn\_left)$ . The class imbalance should be noticed in that specific traffic elements, such as  $u\_turn$ , are much rarer than the common ones like traffic lights in red or green.
The correspondence of a lane and a traffic element forms a regulation for vehicles driving in a particular lane. The construction of relationships between spatially relevant lanes and traffic elements is straightforward. For instance, a lane is controlled by the road marking which is located within its boundary. However, most of the centerlines and traffic elements do not fit into this case. We utilize the following principles for the labeling process. For those traffic element which does not contain directional information, it is associated with centerlines on which only going straight is permitted. Traffic elements with directional information, such as traffic lights in the shape of a left arrow, control the corresponding lanes going in the same direction. Note that traffic elements are only associated with centerlines outside the intersection.
4 Task Definition & Evaluation Metric
In this section, we introduce the tasks and metrics in OpenLane-V2. The primary task of the proposed benchmark is scene structure perception and reasoning, which requires the model to recognize lanes and their dynamic drivable states in the surrounding environment. The challenge includes detecting lane centerlines and traffic elements, recognizing the attributes of traffic elements, and reasoning about the topology relationships on perceived entities. We further divide the primary task into three subtasks: 3D lane detection, traffic element recognition, and topology recognition. The OpenLane-V2 Score (OLS), which is the average of various metrics from different subtasks, is defined to describe the overall performance of the primary task:
$$
\mathrm {O L S} = \frac {1}{4} \left[ \mathrm {D E T} _ {l} + \mathrm {D E T} _ {t} + f \left(\mathrm {T O P} _ {l l}\right) + f \left(\mathrm {T O P} _ {l t}\right) \right], \tag {1}
$$
where  $f$  is a scale function to emphasize the task of topology reasoning.
4.1 3D Lane Detection
In the spirit of the OpenLane dataset [8], which is the first real-world and the largest scaled 3D lane dataset to date, we provide lane annotations in 3D space. We define the subtask of 3D lane detection as perceiving directed 3D lane centerlines from the given multi-view images covering a fully panoramic field-of-view (FOV).
Given a pair of curves, namely a ground truth  $v_{l} = [p_{1},\dots,p_{n}]$  and a prediction  $\hat{v}_{l} = [\hat{p}_{1},\dots,\hat{p}_{k}]$ , their geometric similarity is measured by the discrete Fréchet distance [12]. Specifically, a coupling  $L$  is defined as a sequence of pairs between points in  $v$  and  $\hat{v}$ :
$$
\left(p _ {a _ {1}}, \hat {p} _ {b _ {1}}\right), \dots , \left(p _ {a _ {m}}, \hat {p} _ {b _ {m}}\right), \tag {2}
$$
where  $1 = a_{1} \leq a_{i} \leq a_{j} \leq a_{m} = n$  and  $1 = b_{1} \leq b_{i} \leq b_{j} \leq b_{m} = k$  for all  $i < j$ . Then the norm  $||L||$  of a coupling  $L$  is defined as the distance of the most dissimilar pair in  $L$ . The Fréchet distance of a pair of curves is the minimum norm of all possible coupling:
$$
D _ {\text {F r e c h e t}} \left(v _ {l}, \hat {v} _ {l}\right) = \min  \left\{\left| \left| L \right| \right| \mid \forall \text {p o s s i b l e} L \right\}. \tag {3}
$$
We define a threshold  $t \in \mathbb{T}$  that a pair of centerlines would be regarded as unmatched if their distance is greater than  $t$ . Then  $\mathrm{DET}_l$  is averaged over match thresholds of  $\mathbb{T} = \{1.0, 2.0, 3.0\}$ :
$$
\mathrm {D E T} _ {l} = \frac {1}{| \mathbb {T} |} \sum_ {t \in \mathbb {T}} A P _ {t}. \tag {4}
$$
The  $AP$  score is the area under the precision-recall curve, defined as  $\int_0^1 p(r)\mathrm{d}r$ , where  $p$  and  $r$  denote precision and recall respectively. Note that as the defined annotation range is relatively large compared to previous datasets, accurate perception of lanes in the distance would be challenging. Thus, the matching thresholds are relaxed for centerlines at a distance based on the distance between the ground truth lane and the ego car. For instance, a lane at a distance would be thresholded on  $1.2t$  while another lane in a closer region would be thresholded on  $1.1t$  for the same threshold  $t\in \mathbb{T}$ .
4.2 Traffic Element Recognition