Driving Scene Understanding with Traffic Scene-Assisted Topology Graph Transformer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:11:06

Abstract: Driving scene topology reasoning aims to understand the objects present in the current road scene and model their topology relationships to provide guidance information for downstream tasks. Previous approaches fail to adequately facilitate interactions among traffic objects and neglect to incorporate scene information into topology reasoning, thus limiting the comprehensive exploration of potential correlations among objects and diminishing the practical significance of the reasoning results. Besides, the lack of constraints on lane direction may introduce erroneous guidance information and lead to a decrease in topology prediction accuracy. In this paper, we propose a novel topology reasoning framework, dubbed TSTGT, to address these issues. Specifically, we design a divide-and-conquer topology graph Transformer to respectively infer the lane-lane and lane-traffic topology relationships, which can effectively aggregate the local and global object information in the driving scene and facilitate the topology relationship learning. Additionally, a traffic scene-assisted reasoning module is devised and combined with the topology graph Transformer to enhance the practical significance of lane-traffic topology. In terms of lane detection, we develop a point-wise matching strategy to infer lane centerlines with correct directions, thereby improving the topology reasoning accuracy. Extensive experimental results on Openlane-V2 benchmark validate the superiority of our TSTGT over state-of-the-art methods and the effectiveness of our proposed modules.

Main paper: 
 introduction: 
 Driving scene topology reasoning aims to detect lane and traffic elements in multi-view images captured by onboard cameras, and to construct the topology relationships between them. This emerging task of scene understanding provides a more natural integration for perception and planning tasks in autonomous driving, attracting significant attention in the research community. The driving scene topology reasoning task can provide information about drivable areas and traffic signals, so as to offer clear navigation signals for downstream tasks such as motion prediction and planning. Compared to the lane detection task and the 3D object detection task, driving scene topology reasoning is more challenging because of the difficulty in understanding the topology relationships of objects in complex scenes.Recently, Wang et al. [44] proposed a dataset called Openlane-V2 that defines the objectives of the driving scene topology reasoning task. Specifically, given multi-view images, topology reasoning aims to learn the vectorized road graph between centerlines and traffic elements. It includes four sub-tasks, namely lane centerline detection, traffic element detection, lane-lane topology reasoning and lane-traffic topology reasoning. Some explorations are advancing the field. For instance, Li et al. [26] proposed the TopoNet, which uses a GCN [23] to construct topology relationships between heterogeneous features. Wu et al. [48] proposed the state-of-theart TopoMLP, where the concept of "first-detect-then-reason" is introduced and a simple MLP is employed to build the topology relationships of target objects.Despite the impressive performance of some existing methods, there are still some drawbacks that need to be addressed. First, an important challenge in this task is how to construct the topology relationships of lane-lane and lane-traffic, while previous methods are not perfect in this aspect. As shown in Fig. 1 (a) , TopoNet [26] adopted a graph convolutional network (GCN) to construct topology relationships. However, using a single graph model to simultaneously build topology relationships between homogeneous and heterogeneous features may lead to confusion of object information. In addition, simultaneous detection and reasoning may result in the loss of object information. Therefore, TopoMLP [48] proposed the strategy of "first-detect-then-reason" and utilized two separate sets of MLPs to predict the topology relationships between different types of objects, as shown in Fig. 1 (b ). Nevertheless, from a graph perspective, MLPs can only aggregate information between pairs of nodes and cannot capture information from local and global nodes. This makes it difficult to effectively acquire intrinsic connections between objects. Second, solely relying on data without considering the practical significance of objects to construct topology relationships cannot effectively address complex driving scenes. Effective constraints on topology reasoning results based on traffic rules and the potential topology relationships provided by the Previous methods, such as MapTR [30] , employed hierarchical bipartite matching to constrain the orientation of HD Map objects.In contrast, designing an approach to ensure the correct lane orientation in the driving scene topology reasoning task remains to be developed.In this study, we propose TSTGT (Traffic Scene-Assisted Topology Graph Transformer), an innovative driving scene topology reasoning network, to address the aforementioned issues, as illustrated in Fig. 1 (c ). Inspired by GPS [40] , we first utilize lane embeddings and traffic element embeddings as graph node features inputs, and then design a divide-and-conquer topology graph Transformer to reason lane-lane topology and lane-traffic topology. In the topology graph Transformer, the lane-lane part accepts lane features and processes them as a directed graph, while the lane-traffic part is treated as an undirected bipartite graph. After aggregating local information through message-passing graph neural networks (MPNNs) and global information via global attention layer in the two parts, the model employs MLPs to gather features of adjacent nodes and edge features to predict topology relationship categories. In this way, the model can learn the underlying guiding relationships between target objects, and thus obtain more accurate topology reasoning results. Moreover, we design a traffic scene-assisted reasoning module and incorporate it after the topology graph Transformer, allowing the model to constrain topology reasoning based on the practical significance of traffic objects.To ensure the correct lane orientation, we further develop a point-wise matching strategy and integrate it into the lane detection module. Concretely, the point-wise matching strategy constructs ground truth for lane curve point sets with multiple equivalent arrangements. Through lane instance-level matching and curve point-wise matching constraints, the model decodes more accurate lane instance embeddings, thereby improving its performance in lane detection and lane-lane topology reasoning.The experimental results on two subsets of the benchmark dataset Openlane-V2 [44] demonstrate that TSTGT achieves state-of-theart performance in driving scene topology reasoning task. Ablation studies are conducted to validate the effectiveness of our framework. The main contributions of this work can be summarized as follows:• We propose an innovative topology reasoning framework, TSTGT, which can accurately detect target objects in multiview images and derive precise topology relationships by employing divide-and-conquer graph information aggregation and imposing constraints on the orientation of lane targets. TSTGT achieves state-of-the-art performance on the benchmark Openlane-V2 dataset of this task. • We devise a traffic scene-assisted topology graph Transformer to infer lane-lane topology and lane-traffic topology.It could effectively aggregate both local and global information of traffic objects and enhance the practical significance of lane-traffic topology, thus predicting high-quality topology relationships. • We develop a point-wise matching strategy in the lane detector to constrain the orientation of detected lane objects. This encourages the detector to identify more accurate lane objects from visual features and improves the performance of lane-lane topology reasoning. 
related work: 
 Lane Topology Learning. Lane Topology Learning has received abundant attention due to its pivotal role in autonomous driving.Earlier works used aerial images to generate a road graphs [1, 12, 18] or lane graphs [3, 17, 19, 52] . However, aerial images suffer from problems such as untimely updates and roads obscured by obstacles like trees, resulting in inaccurate results that do not match realworld driving conditions. As a result, it is becoming increasingly popular to use vehicle-mounted sensors to directly detect lane topology. STSU [6] proposes a DETR-like [9] neural network to detect centerlines and objects, and then derive them into a directed graph by a successive MLP module. On this basis, Can et al. [7, 8] introduce a minimal circle queries to provide additional supervision of the relationship, improving the estimation results of the lane graph. LaneGAP [29] designs a heuristic-based algorithm to learn from a set of lanes and build a lane topology. CenterLineDet [49] and TopoNet [26] use the road centerlines as vertices and design the graph model to update the topology. TopoMLP [48] designs a unified query-based framework for lane topology, which considers lane-traffic topology and lane-lane topology at the same time. HD Map Perception. HD Map Perception is designed to understand the layout of driving scenarios, such as lanes, pedestrian crossings, and other traffic elements. In recent years, with the development of 2D-to-BEV methods [36] , studies have focused on segmentation and vectorized methods. Chen et al. [11] ; Zhou & Krähenbühl [50] ; Li et al. [27] ; Liu et al. [33] generate a rasterized map by performing BEV semantic segmentation. HDMapNet [25] uses heuristics and complex post-processing to group and vectorize the segmented map. VectorMapNet [32] serves as the first end-to-end framework, which adopts a two-stage framework and directly represents each map element as a sequence of points then utilizes auto-regressive decoder to predict them. MapTR [30] further explores a unified permutation-based modeling approach for the sequence of points to eliminate the modeling ambiguity. Since vectorization also enriches the direction information of the lanelines, the vectorization-based method can easily adapt to the perception of the centerline by alternate supervision. InstaGraM [42] builds map elements as a graph by predicting the vertices first and then using a GNN method to detect edges. TopoNet [26] leverage instance-level feature transmission with graph neural networks to extract map prediction hints. Graph Neural Network. Graph Neural Networks, such as graph convolutional network (GCN) [23] , GraphSAGE [15] , GAT [43] , and Graph Transformer (GPS) [40] , are widely adopted to aggregate vertex features and extract information from graph data [41] . And they have contributed to remarkable achievements in various fields, e.g. recommender systems and video semantic segmentation. [10, 14, 37, 38] . Researchers in the autonomous driving community are also trying to utilize it to process unstructured data. Weng et al. [45, 46] introduce GNN to analyze the interaction between agent features for 3D multi-target tracking. LaneGCN [28] extracts the lane graph from the HD map. Others [13, 21, 22] model the relationship between moving agents and lanes as graphs to improve the trajectory prediction performance. TopoNet [26] uses GCN for driving scene understanding tasks, strengthens feature interaction, and introduces knowledge graphs to fuse semantic information.Our method introduces a divide-and-conquer topology graph Transformer, which enhances the model's ability to explore potential relationships between objects by alternately aggregating local features and global features through MPNN and global attention layers, thereby leading to improved topology reasoning results. 
method 3.1 overview: 
 The overview of our proposed TSTGT is illustrated in Fig. 2 . It mainly consists of four parts: the image backbone and FPN, the lane detector with point-wise matching, the traffic element detector, the divide-and-conquer topology graph Transformer with the traffic scene-assisted reasoning module. During inference, given multiview images from camera sensors, the image backbone first extracts the multi-view features 𝐹 ∈ 𝑅 𝑉 ×𝐶 ×𝐻 ×𝑊 and collects the frontview features 𝐹 0 ∈ 𝑅 1×𝐶 ×𝐻 ×𝑊 among them, where 𝑉 , 𝐶, 𝐻 and 𝑊 represent the view number, channel, height, and width of the features respectively. Then the multi-view features are fed into the lane detector and the front-view features are fed into the traffic detector. For the divide-and-conquer topology Transformer, the lane-lane part aggregates topology information in the lane instance embeddings output from the lane detector. The lane-traffic part is designed to understand the underlying control relationship between lane instance embeddings derived from the lane detector and the traffic element instance embeddings obtained from traffic detector. It leverages a traffic scene-assisted reasoning module to gather pertinent scene data, aiding in the process of topology modeling. Finally, task-specific prediction heads are utilized separately for detecting precise target objects and reasoning accurate topology relationships.For the surrounding images from multi-camera, we adopt a shared image backbone to extract the multi-level visual features of each image independently. Then we feed these multi-level features into the FPN to aggregate rich semantic information. Finally, the pyramid features are upsampled to the same size and stacked together as outputs.Lane Detector. We adopt a structure similar to PETR [31] as the lane decoder, which encodes 3D coordinates thus transforming multi-view features into 3D space, and incorporates 3D position embedding in order to enhance the perception of the query object in 3D space. Specifically, for the input multi-view features 𝐹 ∈ 𝑅 𝑉 ×𝐶 ×𝐻 ×𝑊 , we first encode them using 3D position embedding to generate visual perceptual features 𝐹 3𝑑 , which are input as the key-value pairs into the Transformer. Then a set of learnable anchor points 𝑄 𝐿 ∈ 𝑅 𝑁 𝐿 ×3 are randomly initialized, and after applying position encoding to them, they are projected into the feature space 𝑅 𝑁 𝐿 ×𝐶 by using a Linear layer in order to serve as the query embeddings for the PETRTransformer [31] . The lane query then interacts with the visual perceptual features 𝐹 3𝑑 to generate a set of 𝑁 𝐿 lane instance embeddings. The previous approach used a set of Bézier curves to model lane. For the lane instance embedding from PETRTransformer, it chose to use two sets of MLPs to predict the lane categories and Bézier control points respectively, and convert the control points to lane points to make predictions about their position coordinates. However, lane objects are inherently sequential in character, and only using instance-level matching may make the direction of lane appear head-to-tail reversed, thus reducing the accuracy of topology prediction. Therefore, we design a point-wise matching strategy to improve the accuracy of lane matching.Point-Wise Matching. Bézier curve is aimed to represent parametric curves by using an ordered set of control points 𝑃 0 through 𝑃 𝑛 :EQUATIONwhere 𝑛 represents the degree of the curve, and 𝐵 𝑖,𝑛 (𝑡) is referred to as the Bernstein basis polynomial of degree n, as follows:EQUATIONPredicting the control points of Bézier curves with anchors of the Transformer can effectively abstract the curve features for better matching results [39] . However, the ground truth provided by instance-level matching consists of lane points with fixed arrangements, imposing restrictions on the model to learn the correct direction of the lanes. Inspired by MapTR [30] , we re-interpolated and sampled the ground truth. For the given 𝑁 ground truth points, we use 3D linear interpolation sampling to obtain 𝑁 𝑝 curve lane points, and the two equivalent alignments of lane points were employed as the ground truth 𝐴 𝑝 for point-wise matching. The equivalent alignments of the lane points are as follows:EQUATIONThe predicted lane instance sequences ŷ are assigned with ground truth lane instance sequences 𝑦 to locate the best prediction sequence as the positive sample ŷ𝑝𝑜𝑠 . After that, the lane control points predictions are converted into 𝑁 𝑝 curved lane points P𝑝𝑜𝑠 . Then, we perform point-wise matching on each lane instance assigned a positive label to find the optimal point match γ between the predicted curved lane points P𝑝𝑜𝑠 (𝑖 ) and the ground truth point set 𝐴 𝑝 (𝑖 ) . To constrain the matching results, the following point-wise matching cost is utilized:EQUATIONwhere 𝐷 𝑀𝑎𝑛ℎ𝑎𝑡𝑡𝑎𝑛 ( p 𝑗 , 𝑝 𝛾 ( 𝑗 ) ) represents the Manhattan distance between the 𝑗-th point of the predicted lane curve points set P𝑝𝑜𝑠 (𝑖 ) and the ground truth points set 𝐴 𝑝 (𝑖 ) with equivalent arrangement 𝛾.Traffic elements are perceived from the front-view features, so this task can be considered as a 2D object detection task. Here we adopt the Deformable-DETR [51] as the traffic elements detector, which employs a set of randomly initialized reference points embeddings as the query 𝑄 𝑡 to interact with the front-view feature 𝐹 0 to generate a set of 𝑁 𝑡 traffic element instance representations. Two MLPs are employed to them to predict the category and the bounding box of the traffic elements respectively.A natural idea for topology reasoning tasks is to model them using graph methods. Inspired by GPS [40] , we design a divide-and conquer topology graph Transformer (DCTGT) to model lane-lane and lane-traffic topology relationships respectively. The graph Transformer is stacked by several DCTGT layers, as shown in Fig. 3 . Each of them consists of a message-passing graph neural network (MPNN), a Transformer-like global attention layer and several MLPs. The MPNN, utilizing GINE [20] for the lane-lane part and Gat-edGCN [2] for the lane-traffic part, is employed to aggregate local neighborhood node information. Meanwhile, the global attention layers consolidate global node features, which allows for information propagation among all nodes in the graph ,thereby addressing to the extent issues such as over-smoothing and over-squashing of node information caused by the MPNN. Such the alternating learning approach between local and global information facilitates enhanced information perception capability for each node in the graph. The DCTGT layer could be formulated as follows: where 𝐹 𝑙 𝑛𝑜𝑑𝑒 ∈ 𝑅 𝑁 ×𝐷 , 𝐹 𝑙 𝑒𝑑𝑔𝑒 ∈ 𝑅 𝑁 ×𝑁 ×𝐷 denote the 𝐷-dimensional node and edge features, respectively; 𝑀𝑃𝑁 𝑁 and 𝐺𝑙𝑜𝑏𝑎𝑙𝐴𝑡𝑡𝑛 refer to a MPNN and a global attention mechanism at layer 𝑙, each with its set of learnable parameters. 𝑀𝐿𝑃 𝑙 stands for a 2-layer MLP block. The edge adjacency matrix 𝑀 𝑎𝑑 𝑗 ∈ 𝑅 𝑁 ×𝑁 of the MPNN layer is defaluted to all ones to ensure that nodes fully utilize information exchange.EQUATIONLane-Lane Part. For the lane-lane topology reasoning, it can be considered as a problem of predicting edges categories from the known nodes features. We feed the lane instance embeddings 𝑄 𝑙 ∈ 𝑅 𝑁 𝑙 ×𝐶 output from the lane detector into the lane-lane part of DCTGT (LLTGT) as the input node features 𝐹 𝑛𝑜𝑑𝑒 ∈ 𝑅 𝑁 𝑙 ×𝐷 , and the initial edge sets are set to be the edge features 𝐹 𝑒𝑑𝑔𝑒 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑙 ×𝐷 , which are aggregated from the neighbouring nodes features. After clustering the information from the 𝑁 𝑙𝑙 DCTGT layers, we feed the output node features 𝐹 ′ 𝑛𝑜𝑑𝑒 ∈ 𝑅 𝑁 𝑙 ×𝐷 into the LL topology head composed of a 3-layer MLP to obtain predicted edge categories 𝐶 𝑙𝑙 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑙 , which serve as the prediction results for the lanelane topology relationship. The whole process can be represented by the following equation:EQUATIONLane-Traffic Part. Due to the distinct semantic meanings represented by lane node features and traffic element node features, lane-traffic topology reasoning should be viewed as a problem of predicting edge categories in a bipartite graph based on known node features. Specifically, we concatenate the lane instance embeddings 𝑄 𝑙 ∈ 𝑅 𝑁 𝑙 ×𝐶 and the traffic elements instance embeddings 𝑄 𝑡 ∈ 𝑅 𝑁 𝑡 ×𝐶 and then feed them into the lane-traffic part of DCTGT (LTTGT) as the node features 𝐹 𝑛𝑜𝑑𝑒 ∈ 𝑅 (𝑁 𝑙 +𝑁 𝑡 ) ×𝐷 . The edge sets between lane instances and traffic element instances are initialized as the edge features 𝐹 𝑒𝑑𝑔𝑒 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 ×𝐷 , which are aggregated from adjacent node features. The edge adjacency matrix 𝑀 𝑎𝑑 𝑗 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 are set to all-ones, allowing each lane-traffic element pair to have the opportunity to learn each other's information. Then, 𝐹 𝑒𝑑𝑔𝑒 and 𝑀 𝑎𝑑 𝑗 are expanded to the shape of (𝑁 𝑙 + 𝑁 𝑡 ) × (𝑁 𝑙 + 𝑁 𝑡 ) to fit the input of MPNN layer. Since the connection between lane nodes and traffic element nodes can be viewed as an undirected graph, meaning the topology relationship applies to both lane-traffic and traffic-lane pairs, 𝐹 𝑒𝑑𝑔𝑒 and 𝑀 𝑎𝑑 𝑗 are symmetrically initialized for the lane-traffic element and traffic element-lane parts. Other parts are set to all-zeros to ensure the correct bipartite graph interaction. Due to the difference between lane instances and traffic element instances, we set them as two sets of node instances to ensure that they interact with global information in the global attention layers separately. After interacting with the information from the 𝑁 𝑙𝑡 LTTGT layers, we use the LT topology head consisting of two sets of 3-layer MLP to aggregate 𝐹 ′ 𝑛𝑜𝑑𝑒 and 𝐹 ′ 𝑒𝑑𝑔𝑒 from the LTTGT, in order to obtain the predicted edge categories 𝐶 𝑙𝑡 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 . The whole process can be represented by the following equation:EQUATIONTraffic Scene-Assisted Reasoning Module. Just reasoning the topology relationship based on data without incorporating the practical meaning will make the accuracy of topological inference decrease. In the previous method [48] , only the lane-lane topology relationship is optimised, which uses L1 loss to constrain the positional coordinates of lane instances with topology associations. Nevertheless, the operation of optimising the lane-traffic topology relationship in conjunction with practical implications is missing. In this regard, we design a traffic scene-assisted reasoning module according to the construction way of Openlane-V2 [44] dataset and the traffic rules in the actual scenarios to overcome this difficulty, as shown in Fig. 4 .Specifically, excluding the semantically unobservable category 𝑢𝑛𝑘𝑛𝑜𝑤𝑛, the traffic elements in the Openlane-V2 dataset can be classified into two groups, the traffic light class and the road sign class [44] . Among them, the traffic light class generates topology relationship for all lane instances within their control area, while the road sign class only takes effect on the lane instances in the corresponding direction within their control range. The judgement method for the control range of traffic elements and the direction of lane is outlined as follows.Traffic element control range: We first transfer the lane points coordinates from the ego coordinate system to the front-view image coordinate system, and then calculate the minimum Manhattan distance between the centre coordinates of the traffic element bounding box 𝐵 𝑡 𝑓 and the set of the lane curves points 𝑃 𝑙𝑎𝑛𝑒 . The equations are as follows:EQUATION)If 𝑀 𝑑𝑖𝑠 is in the 𝜎 of the width of the image coordinate system, we consider that the lane instance is in the control range of the traffic element instance. Then we obtain the control range result 𝐶 𝑐𝑜𝑛 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 . Direction of lane: The lane direction can be determined by the position coordinates of its start and end points. We obtain lane start point 3D coordinates (𝑥 0 , 𝑦 0 , 𝑧 0 ) and end point 3D coordinates (𝑥 𝑛 , 𝑦 𝑛 , 𝑧 𝑛 ) from the lane detection result. For the vehicle ego coordinate system of Openlane-V2 dataset [44] , the x-axis is positive forwards, the y-axis is positive to the left, and the z-axis is positive upwards. The lane direction is judged only in BEV, which means that the effect of the z-axis can be ignored. We use the threshold 𝜏 to delineate lane direction 𝐷 𝑙𝑎𝑛𝑒 as follows:EQUATIONAfter obtaining the lane instance direction 𝐷 𝑙𝑎𝑛𝑒 , we input traffic element category 𝐶 𝑡 𝑓 and lane direction 𝐷 𝑙𝑎𝑛𝑒 into the traffic rule matrix 𝑀 𝑡𝑟 to obtain the traffic rule constraint topology result 𝐶 𝑡𝑟𝑐 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 . The detailed introduction of 𝑀 𝑡𝑟 is in the Appendix. The traffic rule constraint topology result 𝐶 𝑡𝑟𝑐 is multiplied by the control range result 𝐶 𝑐𝑜𝑛 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 to obtain the reference topology reasoning results for the traffic scene assistance 𝐶 𝑟𝑒 𝑓 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 . We compute the Manhattan distance between 𝐶 𝑟𝑒 𝑓 and the ground truth of the lane-traffic topology relationship 𝐺 𝐿𝑇 to apply constraint, and multiply 𝐶 𝑟𝑒 𝑓 by 𝐶 𝑙𝑡 to obtain the final lane-traffic topology reasoning result 𝐶 𝑙𝑡 𝑓 ∈ 𝑅 𝑁 𝑙 ×𝑁 𝑡 .Our model combines detection matching and topology reasoning matching for training, with the overall loss consisting of four components, 𝑖.𝑒. lane detection loss, traffic detection loss, lane-lane topology reasoning loss and land-traffic topology reasoning loss:EQUATION)Lane detection loss L 𝑑𝑒𝑡 𝑙 is decomposed into a classification loss, a L1 loss and a point2point loss for lane point regression:EQUATIONwhere L 𝑐𝑙𝑠 is a focal loss, and L 𝑝𝑡𝑠 is a point2point loss of 𝑁 𝑙 lane instance and 𝑁 𝑝 curve points for each lane instance with positive class label (𝐶 ≠ ∅), akin to MapTR [30] :EQUATIONTraffic detection loss L 𝑑𝑒𝑡 𝑡 consists of a focal loss for classification, a L1 loss and a GIoU loss for bounding box regression:EQUATIONLane-lane topology reasoning loss L 𝑡𝑜𝑝 𝑙𝑙 consists of a focal loss for binary classification, a L1 loss for constraining the coordinates of the start and end nodes of lanes with topology relationships:EQUATIONLane-traffic topology reasoning loss L 𝑡𝑜𝑝 𝑙𝑡 consists of a focal loss for binary classification and a distance loss for traffic scene assistance topology reasoning:EQUATIONwhere the L 𝑡𝑜𝑝 𝑙𝑡 is defined as the Manhattan distance computed between traffic scene assistance topology result 𝐶 𝑟𝑒 𝑓 and the ground truth of the lane-traffic topology relationship 𝐺 𝐿𝑇 :EQUATION 
experiences 4.1 datasets and metrics: 
 Datasets. The experiments are conducted on the OpenLane-V2 [44] , a comprehensive dataset designed for perception and reasoning tasks in autonomous driving scenes. OpenLane-V2 consists of two subsets, labeled as 𝑠𝑢𝑏𝑠𝑒𝑡_𝐴 and 𝑠𝑢𝑏𝑠𝑒𝑡_𝐵, which are derived from Argoverse 2 [47] and nuScenes [4] datasets, respectively. Each subset contains 1,000 scenes annotated at a rate of 2Hz. It is noteworthy that 𝑠𝑢𝑏𝑠𝑒𝑡_𝐴 encompasses seven views, whereas 𝑠𝑢𝑏𝑠𝑒𝑡_𝐵 comprises six views. Evaluation Metrics. The evaluation metrics of OpenLane-V2 dataset are divided into two parts: perception and reasoning. For the perception metrics, the DET score represents the standard mean average precision (mAP) used to evaluate the instance-level perception performance. In particular, DET 𝑙 employs the Fréchet distance to quantify similarity, averaging over match thresholds set at {1.0, 2.0, 3.0}, while DET 𝑡 calculates similarity using Intersection over Union (IoU) and averages across different traffic categories. Reasoning metrics also utilize an mAP metric known as the TOP score, tailored specifically for graph data. To encapsulate the combined impact of primary detection and topology reasoning, the OpenLane-V2 Score (OLS) is employed as follows:EQUATIONwhere DET and TOP evaluate performance in perception and reasoning, respectively, while 𝑓 denotes the square root function.Model Settings. We employ different image backbones, including ResNet-50 [16] , VOV [24] , and Swin-B [34] for training and inference and realize a fair comparison with TopoMLP [48] . For feature extractor, each image undergoes resizing to a uniform resolution of 1550×2048, followed by downsampling at a ratio of 0.5, and the output channels number is set to 𝐶 = 256. For lane detection, referencing TopoMLP [48] , the specified region spans from -51.2m to 51.2m along the X-axis, from -25.6m to 25.6m along the Y-axis, and Backbone Epoch DET 𝑙 DET 𝑡 TOP 𝑙𝑙 TOP 𝑙𝑡 OLS STSU [6] ResNet-50 24 12.7 43.0 0.5 15.1 25.4 VectorMapNet [32] ResNet-50 24 11.1 41.7 0.4 5.9 20.8 MapTR [30] ResNet-50 24 17.7 43.5 1.1 10.4 26.0 TopoNet [26] ResNet-50 24 from -8m to 4m along the Z-axis. The lane query number is set to 𝑁 𝑙 = 300, and the number of lane Bézier control points is configured as 4. Throughout the instance-level matching process, lane control points are transformed into 11 curve points for loss calculation.The configuration of PETRTransformer is based on PETR [31] . For traffic detection, all settings are identical to TopoMLP [48] . For topology reasoning, The lane-lane part employs 3 DCTGT layers, while the lane-traffic part employs 6. In the traffic scene-assisted reasoning module, the control range threshold 𝜎 is set to 0.2, and the lane direction threshold 𝜏 is set to 0.1. Training Details. All experiments are conducted on 8 NVIDIA A800 GPUs. Unless otherwise specified, the batch size of the model is set to 8, and the number of training epochs is set to 24 for fair comparison with TopoMLP [48] . The model is optimized using the AdamW optimizer [35] with a weight decay of 0.01. The initial learning rate of 4 × 10 -5 for the image backbone, and 2 × 10 -4 for the rest parts. Throughout the inference process, our model provides a maximum of 300 lane outputs for evaluation purposes. For lane detection loss 𝐿 𝑑𝑒𝑡 𝑙 , the weights for each component loss are set as 𝜆 𝑐𝑙𝑠 = 1.5, 𝜆 𝐿1 = 0.02, 𝜆 𝑝𝑡𝑠 = 5 × 𝑒 -3 . For traffic detection loss 𝐿 𝑑𝑒𝑡 𝑡 , the weights allocated to each component of the loss are as follow: 𝜆 𝑐𝑙𝑠 = 1.0, 𝜆 𝐿1 = 2.5, 𝜆 𝐺𝐼𝑜𝑈 = 1.0. For the lane-lane topology loss, the coefficient for the classification part is set as 𝜆 𝑐𝑙𝑠 = 5, and for the L1 loss is set as 𝜆 𝐿1 = 0.1. The coefficients for the lane-traffic topology loss are set as 𝜆 𝑐𝑙𝑠 = 5, 𝜆 𝑑𝑖𝑠 = 0.075.Openlane-V2 set. We conduct a comprehensive comparison between our method, TSTGT, and several state-of-the-art approaches including STSU [6] , VectorMapNet [32] , MapTR [30] , TopoNet [26] , In Table 2 , the performance evaluation on OpenLane-V2 𝑠𝑢𝑏𝑠𝑒𝑡_𝐵 further reinforces our findings. Our innovative TSTGT outperforms its counterparts across all metrics while employing the ResNet-50 backbone. Notably, it demonstrates a considerable advantage over TopoMLP in terms of topology performance, with substantial differences evident: 27.5 compared to 26.6 on DET 𝑙 , 60.5 compared to 58.3 on DET 𝑡 , 13.7 compared to 7.6 on TOP 𝑙𝑙 , and 18.9 compared to 17.8 on TOP 𝑙𝑡 . Additionally, the integration of more potent backbones leads to further performance enhancements. Overall, these results validate the effectiveness of our proposed TSTGT for driving scene topology reasoning task and demonstrate that our method achieves state-of-the-art performance.Fig. 5 presents the visualization of TSTGT. The lane detection results are displayed in multi-view images, while the lane-lane topology reasoning results are shown in the BEV. The traffic detection results and the lane-traffic topology reasoning results are displayed in the front-view image. Despite the complexity of the traffic scenes, TSTGT can still accurately perform detection and topology reasoning.In this part, we perform extensive ablation experiments to investigate the influence of the core components of our TSTGT as well as the impact of different model settings. All of the experiments are conducted on the Openlane-V2 𝑠𝑢𝑏𝑠𝑒𝑡_𝐴 dataset.Components Analysis. To explore the influence of the key components of our model, we first build a baseline model similar to TopoMLP [48] , which is the TSTGT without point-wise matching strategy and divide-and-conquer topology graph Transformer (DCTGT). As shown in Table 3 , the baseline model obtains an OLS score of 38.2. When we add point-wise matching strategy on the baseline model, the special TSTGT achieves an OLS score of 39.0, which is 0.8 higher than the baseline. The improvement proves the effectiveness of the point-wise matching strategy. When only the DCTGT is imposed on the baseline, the OLS score of the special TSTGT reaches to 39.8, which means the module brings a 1.6 gain and validates the superiority of the module. Equipped with both components, our TSTGT can achieve the best performance.Point-Wise Matching. In this study, we explore the impact of different settings in the point-wise matching strategy. A critical setting involves the determination of the number of sampled points along the lane curve when converting Bézier control points to lane curve points. As shown in Table 4 , there is a noticeable improvement in lane detection performance and other task performances when the number of sampled points increases from 11 to 30. However, it's worth noting that further increases do not enhance performance. This is because an excessive number of sampled points increases the burden on the model to match lane points while decreasing the abstraction level of the Bézier curve representation for the lane. Therefore, the number of sampled points along the lane curve is set to 30 to strike a balance between model efficiency and optimal performance.Divide-and-Conquer Topology Graph Transformer. The impact of different settings of divide-and-conquer topology graph Transformer (DCTGT) on topology inference performance is also worth noting. First is the initialization method for edge features in the graph Transformer. We designed two initialization methods: one, named "Aggregated", utilizes MLPs aggregation to aggregate node features adjacent to the edge as the initial edge features, while the other, named "All-one", adopts a similar approach to ARGNP [5] , utilizing all-ones features as the initial edge features. The former setting is adopted in TSTGT and the later is a common paradigm. The experimental results are reported in Table 4 . It can be seen that the TSTGT with the default setting performs better than the one with the later setting, which means that the "Aggregated" strategy enables better perception of node features, leading to superior topology reasoning performance.The second setting is the interaction mode of the lane-traffic part of DCTGT (LTTGT). In our model, We perform global attention operations respectively on the lane features and traffic element features inputted into the LTTGT as heterogeneous features. Another approach involves treating them as homogeneous features and applying global attention operations uniformly. The results are presented in Table 4 . It can be observed that employing the heterogeneous strategy for global attention helps to avoid confusion of information from different types of objects. 
conclusion: 
 In this paper, we propose a novel framework for driving scene topology reasoning, termed TSTGT, to address the issue of insufficient utilization of object and scene information in the existing topology modeling process. Our approach involves designing a divide-andconquer topology graph Transformer, which effectively aggregates local and global information of traffic objects, consequently obtaining more accurate topology reasoning results. Moreover, we devised a traffic scene-assisted reasoning module to enhance the practical significance of lane-traffic topology. For lane detection, we developed a point-wise matching strategy to ensure the accuracy of lane directions, further improving the performance of topology reasoning and detection. Experimental results on Openlane-V2 dataset validate the superiority of our TSTGT over state-of-the-art methods and the effectiveness of our proposed modules. We aspire for this work to inspire subsequent research endeavors. 
 
 Appendix:  


Decision: The paper introduces TSTGT, focuses on driving scene understanding task. The authors propose a traffic scene-assisted topology graph Transformer with divide-and-conquer design for topology reasoning and a point-wise matching strategy for accurate lane detection. Experiments on OpenLane-V2 datasets prove the effectiveness of the proposed method.

The preliminary ratings are (ba br br ba). Key concerns are:
1. Backbone and hyperparameters settings.
2. Missing visual comparision.
3. Marginal improvement on TOP_lt and OLS by the proposed module.
4. Marginal improvement from previous SOTA TopoMLP.

After author rebuttal, the final ratings are (ba ba r ba). Reviewer RzwB and neD9 keep their scores and reviewer syKy raise his score as their concerns are resolved. Reviewer gyez seems to be against the acceptance of this paper as 1) missing visualization and 2) unready claim of generalization of the proposed method. In the view of AC, 1) is viable and could be expected given enough rebuttal time, and actually, the visual comparision between TopoMLP and proposed methods was submitted in the supplementary material and 2) is not standing as the paper does not claim generalization across the two datasets.

Based on the comments above, AC decides to accept the paper since the paper shows significant improvement thanks to the addressed concerns. The authors should improve the manuscripts according to the comments above.

Reviews: review: paper_summary: paper_summary: This paper proposes a new topology reasoning framework TSTGT. It uses a divide-and- conquer topology graph Transformer to infer lane-lane and lane-traffic topology relationships separately. In addition, this paper also designed a traffic scene-assisted reasoning module and combined it with a topology graph Transformer. Meanwhile, in terms of lane detection, a point-wise matching strategy has been developed to infer lane centerlines with the correct direction.
strength_weakness: strength_weakness: The paper demonstrates that TSTGT achieves state-of-the-art performance on the Openlane-V2 benchmark, indicating the effectiveness of the proposed methods.
The paper introduces a novel topology reasoning framework, TSTGT, which combines a divide-and-conquer strategy with a graph transformer for improved traffic scene understanding.
limitations: limitations: In the Openlane-V2 subset_A dataset, when you set the epoch to 48, the metric is further improved. Can you inform us of the best achievable indicators as the epoch increases? Why did not an experiment with epoch=48 be conducted in Openlane-V2 subset_B?
Can you provide experiments using other backbones rather than just Resnet-50? And it would be better to explain why you chose Resnet-50 as your backbone? What’s more, providing the experiments of different backbones in paper’s body part rather than appendix may be better.
Visual comparisons with SOTA methods on different datasets should be given.
How are the hyperparameters in each loss selected? Including the weights for each component in    loss 𝐿𝑑𝑒𝑡𝑙 and  𝐿𝑑𝑒𝑡𝑡, and the coefficients for other two losses.
paper_summary: paper_summary: In the task of Driving Scene Topology Reasoning, the authors propose a complex graph reasoning module, the Divide-and-Conquer Topology Graph Transformer and scene-assisted reasoning module, to effectively utilize the associations between scene information and traffic objects to model topological relationships. Additionally, the authors introduce a point-wise matching method to enhance Lane Detection performance. These two innovations together improve Driving Scene Topology Reasoning and achieve state-of-the-art results, with promising experimental outcomes.
strength_weakness: strength_weakness: 1.The paper is articulated with clarity and effectively outlines the technical workflow in a coherent and logical sequence.
2.The method shows notable performance improvement in the Lane-Lane topology metric (Top_ll), which is very promising. According to ablation study, the method achieves a significant improvement in the Lane-Lane topology metric (Top_ll) (11.0 compared to 7.2), indicating that the complex Lane-Lane topology module design indeed works.
limitations: limitations: 1. As it mentioned that one of contributions is the inference of direction between lanes, yet the paper does not provide sufficient experimental results or visual demonstrations to directly showcase the effectiveness of this reasoning. I recommend incorporating visualizations that can clearly illustrate how these directional relationships are inferred and validated within the framework.
2. Lacks the computational efficiency of the proposed method to assess the applicability in real-world scenarios. Comparisons with existing methods on these aspects would also be helpful.
3. A detailed discussion comparing the performances of the two subsets is needed. The results show that the subset with more views performs better. What differences do the additional views introduce, and which module contributes most to this performance?
4. The improvement in Lane-Traffic Topology prediction is limited. Despite the introduction of a Traffic Scene-Assisted Reasoning Module and a Lane-Traffic Part Layer—key differences from TopoMLP[1] and primary innovations—these components do not significantly enhance Lane-Traffic Topology prediction performance. This limitation is evident from the ablation study, where the Top_lt metric increased only marginally from 22.8 to 23.4. The authors need to explain the reasons behind the discrepancy in the improvements observed in Top_ll and Top_lt.
5. According to TopoMLP[1] insights, the performance of Topology prediction is limited by the traffic detector performance, leading them to use YOLOv8 to generate proposals. Therefore, is the limited improvement in Lane-Traffic Topology prediction due to the detector? If the authors also use YOLOv8 and achieve remarkable improvement in Top_lt, then the designed methodology will be truly convincing.
6. Evaluation on the Enhanced Top metric is needed. TopoMLP[1] claims that the previous TOP score has a loophole where setting the confidence of unmatched instances to 1 can achieve remarkable improvements. The authors of TopoMLP [1] proposed an enhanced Top score, which has been accepted by the challenge organizers of OpenLane-V2. It seems that the initial Top score was used; it would be better to provide an evaluation using the enhanced Top score.
7. A discussion on hyper-parameters, such as the control range threshold, is needed. The authors introduce a threshold to limit the effect of road sign classes on lane instances only within a certain range. However, it is unclear why setting this threshold to 0.2 is considered reasonable for all types of road signs.
[1]Dongming Wu, Jiahao Chang, Fan Jia, Yingfei Liu, Tiancai Wang, and Jianbing Shen. 2023. TopoMLP: An Simple yet Strong Pipeline for Driving Topology Reasoning. arXiv preprint arXiv:2310.06753 (2023).
paper_summary: paper_summary: This paper presents a solution for Driving Scene Understanding, which is in detail with a traffic Scene-Assisted Topology Graph Transformer. The performance on the  Openlane-V2 𝑠𝑢𝑏𝑠𝑒𝑡_𝐴 dataset has shown the improvements. But Tables 1 and 2 seem different from Table 3 in the reference [44]. This makes the paper less convincing.
strength_weakness: strength_weakness: 1 Using the divide-and-conquer strategy to address lane-lane topology and lane-traffic topology seems interesting.
limitations: limitations: 1 There are concerns concerning the results provided by the author, in Table 3 of [44],  and the results in Tables 1 and 2 of the paper are different, which makes the results less convincing. Could the author explain the reason why the difference in results exists?

[44] Huijie Wang, Tianyu Li, Yang Li, Li Chen, Chonghao Sima, Zhenbo Liu, Bangjun
Wang, Peijin Jia, Yuting Wang, Shengyin Jiang, et al . 2024. Openlane-v2: A
topology reasoning benchmark for unified 3d hd mapping. Advances in Neural
Information Processing Systems 36 (2024).
paper_summary: paper_summary: This paper proposes a novel topology reasoning model called TSTGT. By employing divide-and-conquer graph information aggregation and imposing constraints on the orientation of lane targets, TSTGT achieves competitive performance on OpenLane-V2 benchmark. In detail, a traffic scene-assisted topology graph Transformer is devised to infer lane-lane topology and lane-traffic topology and a point-wise matching strategy is proposed to constrain the orientation of detected lane objects.
strength_weakness: strength_weakness: 1. The main idea of the paper is clear, focusing on the incorporation of scene information into topology reasoning. Many design choices in the paper are also well-motivated.

2. The paper presents a well-organized related work and states how the contributions of this work contrasts w.r.t. relevant literature. The Fig. 1 is particularly helpful as it clearly illustrates how the proposed method differs from the existing work.

3. The paper includes abundant experiments to showcase the effectiveness of the proposed method. It reports the results on real-world datasets, including OpenLane-V2 subset_A and subset_B, all achieving competitive performance.
limitations: limitations: 1. Even though TSTGT surpasses some classic models (TopoNet, TopoMLP), it is still far from SOTA performance. In Tab. 1, the OLS score of TSTGT on OpenLane-V2 subset_A is only 42.8 while the OLS scores of SOTA models [1] on the official benchmark [2] are over 55.

2. The performance improvements brought by the designed modules are too weak. In Tab. 3, both PWM and DCTGT only bring about a 1~2 improvement in OLS score, which is negligible and can be achieved by parameter tuning.

3. As an online perception task, reporting runtime is necessary.


[1] Wu D, Jia F, Chang J, et al. The 1st-place solution for cvpr 2023 openlane topology in autonomous driving challenge[J]. arXiv preprint arXiv:2306.09590, 2023.

[2] https://opendrivelab.com/challenge2023/#openlane_topology