SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Traffic elements and their attribute provide crucial information for autonomous vehicles. The attribute represents the semantic meaning of a traffic element, such as the red color of a traffic light. In this subtask, on the given image in the front view, the location of traffic elements and their attributes are demanded to be perceived simultaneously. Compared to typical 2D detection datasets, the challenge is that the size of traffic elements is tiny due to the large scale of outdoor environments.
To preserve consistency to the distance mentioned above, we modify the common IoU (Intersection over Union) measure as a distance that:
$$
D _ {I o U} \left(v _ {t}, \hat {v} _ {t}\right) = 1 - \frac {\left| v _ {t} \cap \hat {v} _ {t} \right|}{\left| v _ {t} \cup \hat {v} _ {t} \right|}, \tag {5}
$$
where  $v_{t}$  and  $\hat{v}_{t}$  are the ground truth and predicted bounding box respectively. We consider IoU distance as the affinity measure with a match threshold of 0.75. The  $\mathrm{DET}_t$  score is utilized to measure the performance of traffic elements detection and is averaged over different attributes  $\mathbb{A}$  that:
$$
\mathrm {D E T} _ {t} = \frac {1}{| \mathbb {A} |} \sum_ {a \in \mathbb {A}} A P _ {a}. \tag {6}
$$
4.3 Topology Reasoning
We first define the task of recognizing topology relationships in the field of autonomous driving. On the perceived entities, the topology relationships are built. For simplicity, we divide the graph on all entities into two subgraphs. The connectivity of directed lanes establishes a map-like network and is denoted as the lane graph  $(V_{l},E_{ll})$ . Note that the edge set  $E_{ll}\subseteq V_l\times V_l$  is asymmetric, as the incoming and outgoing edges of a lane represent the connection on its starting and ending points respectively. An entry  $(i,j)$  in  $E_{ll}$  is positive if and only if the ending point of the lane  $v_{i}$  is connected to the starting point of  $v_{j}$ . Besides, the undirected graph  $(V_{l}\cup V_{t},E_{lt})$  describes the correspondence between centerlines and traffic elements. It can be seen as a bipartite graph that positive edges only exist between  $V_{l}$  and  $V_{t}$ .
The TOP score, which is an mAP metric adapted from link prediction in the graph domain, is utilized to evaluate model performance on the matched graphs. Given a ground truth graph  $G = (V,E)$  and a predicted one  $\hat{G} = (\hat{V},\hat{E})$ , it is possible that the number of predicted vertices is not equal to that of ground truth. We first build a projection between predictions and ground truth to preserve true positive vertices, according to the entity-specific similarity measures, namely Fréchet and IoU distances for centerlines and traffic elements respectively. The resulting vertex set  $\hat{V}'$  needs to fulfill the requirements such that  $V = \hat{V}'$  and  $\hat{V}' \subseteq \hat{V} \cup \{v_d\}$ , where  $\{v_d\}$  is a set of dummy vertices. Then the TOP score is the averaged mAP between  $(V,E)$  and  $(\hat{V}',\hat{E}')$  over all vertices:
$$
\mathrm {T O P} = \frac {1}{| V |} \sum_ {v \in V} \frac {\sum_ {\hat {n} ^ {\prime} \in \hat {N} ^ {\prime} (v)} P \left(\hat {n} ^ {\prime}\right) \mathbb {1} \left(\hat {n} ^ {\prime} \in N (v)\right)}{| N (v) |}, \tag {7}
$$
where  $N(v)$  denotes the neighbors of vertex  $v$ ,  $P(v)$  is the precision of vertex  $v$  in the ordered list ranked by predicted confidence, and positive edges are those whose confidence is greater than 0.5. The  $\mathrm{TOP}_{ll}$  is for topology among centerlines on the graph  $(V_l, E_{ll})$ , while the  $\mathrm{TOP}_{lt}$  is for topology between lane centerlines and traffic elements on the graph  $(V_l \cup V_t, E_{lt})$ .
Table 3: Quantitative results on the OpenLane-V2 val split. It is observed that the model design of how to represent centerlines in the network has an impact on model performance. "Instance" denotes that a centerline is represented as a single query in the network, while "Point Set" indicates that a centerline is described by a set of independent points.
Method	Design	subset_A	subset_B
OLS	\( \text{DET}_l \)	\( \text{DET}_t \)	\( \text{TOP}_{ll} \)	\( \text{TOP}_{lt} \)	OLS	\( \text{DET}_l \)	\( \text{DET}_t \)	\( \text{TOP}_{ll} \)	\( \text{TOP}_{lt} \)
STSU [6]	Instance	25.4	12.7	43.0	0.5	15.1	21.2	8.2	43.9	0.0	9.4
VectorMapNet [28]	Point Set	20.8	11.1	41.7	0.4	5.9	16.3	3.5	49.1	0.0	1.4
MapTR [27]	Point Set	20.0	8.3	43.5	0.2	5.9	21.1	8.3	54.0	0.1	3.7
TopoNet [26]	Instance	35.4	29.2	48.0	4.1	19.3	33.2	24.3	55.0	2.5	14.2
In this section, we adapt and evaluate multiple state-of-the-art methods on the proposed OpenLane-V2 dataset. Visualizations and analysis are then reported to investigate the impact of different design choices on model performance.
In our experiments, various models are involved, including STSU [6], VectorMapNet [28], MapTR [27], and TopoNet [26]. STSU utilizes a DETR-like neural network to detect centerlines and then derive their connectivity by a successive MLP module. Since it is designed for monocular image inputs and contains the BEV representation as intermedia results, we adapt it to multi-view inputs by concatenating BEV embeddings from different views. VectorMapNet directly represents each map element as a sequence of points and predicts positional information in an auto-regressive manner. MapTR models each map element as a point set with a group of equivalent permutations to deal with geometrical ambiguity. For both methods, the BEV range is expanded to fit dataset requirements. As for training, we supervise VectorMapNet and MapTR with centerlines and use the element queries in the Transformer decoder as instance queries to produce topology relationships. TopoNet is specifically designed for the task of scene structure understanding that it utilizes instance-level queries for both centerlines and traffic elements to handle their locations and topology relationships. Note that except for TopoNet, methods mentioned above are further appended with heads for predicting traffic elements and topology.
Quantitative results on the OpenLane-V2 dataset are illustrated in Table 3. It is not surprising that the  $\mathrm{DET}_l$  scores are low for all methods, since the task of centerline perception is challenging for existing networks. Different from detecting visibleanelines, the centerline is invisible, which requires models to deduce their position by obtaining references from the neighboringanelines. Additionally, the direction of centerlines is determined by the entire scene, adding extra complexity to the task. For traffic elements recognition, the tiny size of traffic elements, as depicted in Figure 3, also introduces difficulty for the models. As the TOP scores require a match between ground truths and predictions, performance on the connectivity between centerlines is not ideal due to the unsatisfactory perception results of centerlines. To increase model performance on topology reasoning, it is a must to generate perception results that are sufficiently accurate compared to the ground truth.
In Figure 3, we present visualizations of predicted results in a complex crossroad. Although all models provide an approximate shape of the intersection, their performance on the position and shape of centerlines, as well as their topology relationship, can still be improved. STSU generates a large number of false positive centerline predictions and fails to attach traffic information to centerlines. As MapTR represents points of a lane independently in the networks, the predicted centerlines are not in the shape of driving trajectories, which are commonly smooth. Meanwhile, although VectorMapNet utilizes a point set to describe a centerline as well, its auto-regressive design ensures a proper shape of centerlines. VectorMapNet, MapTR, and TopoNet can only provide partially correct results on the semantic information of lanes, namely the correspondence between centerlines and traffic elements. A potential solution could be introducing more prior, such as knowledge of traffic rules, to the network for reasoning about the association between traffic elements of a lane.
Figure 3: Qualitative results of various algorithms. Traffic lights in red and green are emphasized with red and green boxes respectively. The bird's-eye-view is truncated at  $\pm 25m$  along the x-axis.
In this paper, we introduce OpenLane-V2, aiming to facilitate the task of scene structure perception and reasoning. The lane network is represented by lane centerlines and their connectivity, while traffic information is described by traffic elements with semantic meanings and their association with lanes. Tasks and metrics are described in detail for future research usage. We adapt various methods and demonstrate their performance on the proposed dataset. We hope this dataset will encourage the research community to design and develop neural networks on the defined tasks, and further promote research in the field of autonomous driving.
Limitations. Due to limitations in available resources, the proposed dataset is built on top of existing datasets, and its data scale is the same as previous datasets. We believe that including more driving scenes will further increase its diversity. Moreover, although the lane networks with traffic information benefit the downstream tasks, we do not include the planning task in the proposed dataset, as it also requires knowledge of moveable objects, such as cars and pedestrians, for collision avoidance. We leave the integration of static and dynamic entities to future work.
Impact. Based on previous sections, it is evident that the released dataset is used for research purposes. Models trained or evaluated on this dataset should not be directly used for direct deployment or any real-world application. It should be noted that the proposed dataset does not provide any guarantee, particularly in safety-critical situations.
This work was supported by National Key R&D Program of China (2022ZD0160104) and NSFC (62206172). We thank reviewers for their fruitful comments and the research community for participating in the OpenLane Topology Challenge 2023.
[1] Mohamed Aly. Real time detection of lane markers in urban streets. In IEEE IV, 2008.
[2] Karsten Behrendt and Ryan Soussan. Unsupervised labeled lane markers using maps. In ICCV Workshop, 2019.
[3] Karsten Behrendt, Libor Novak, and Rami Botros. A deep learning approach to traffic lights: Detection, tracking, and classification. In ICRA, 2017.
[41] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493, 2023.
[42] Tao Wu and Ananth Ranganathan. A practical system for road marking detection and recognition. In IEEE IV, 2012.
[43] Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Curvelanenas: Unifying lane-sensitive architecture search and adaptive point blending. In ECCV, 2020.
[44] Fan Yan, Ming Nie, Xinyue Cai, Jianhua Han, Hang Xu, Zhen Yang, Chaoqiang Ye, Yanwei Fu, Michael Bi Mi, and Li Zhang. Once-3dlanes: Building monocular 3d lane detection. In CVPR, 2022.
[45] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[46] Zhe Zhu, Dun Liang, Songhai Zhang, Xiaolei Huang, Baoli Li, and Shimin Hu. Traffic-sign detection and classification in the wild. In CVPR, 2016.