GSLAMOT: A Tracklet and Query Graph-based Simultaneous Locating, Mapping, and Multiple Object Tracking System

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 05:13:56

Abstract: For interacting with mobile objects in unfamiliar environments, simultaneously locating, mapping, and tracking the 3D poses of multiple objects are crucially required.  This paper proposes a Tracklet and Query Graph based framework, i.e., GSLAMOT to address this challenge. GSLAMOT represents the dynamic scene by a combination of semantic map,  agent trajectory, and an online maintained Tracklet Graph (TG). TG tracks and predicts the 3D poses of the detected active objects. A Query Graph (QG) is constructed in each frame by object detection to query and to update TG, as well as the semantic map and the agent trajectory. For accurate object association,  a Multi-criteria Subgraph Similarity Association (MSSA) method is  proposed to find matched objects between the detections in QG and the predicted tracklets in TG. Then an Object-centric Graph Optimization (OGO) method is proposed to optimize the TG,  the semantic map,  and the agent trajectory simultaneously.  It triangulates the detected objects into the map to enrich the map's semantic information. We address the efficiency issues to handle the three tightly coupled tasks in parallel. Experiments are conducted on KITTI, Waymo, and an emulated Traffic Congestion dataset that highlights challenging scenarios including congested objects.  Experiments show that GSLAMOT enables accurately crowded object tracking while conducting SLAM accurately in challenging scenarios, demonstrating more excellent performances than the state-of-the-art methods.

Main paper: 
 related work: 
 The most closely related works are reviewed, which are categorized into two areas: Object-Oriented SLAM (OOSLAM) and 3D Multiple Object Tracking (3D MOT).Early semantic SLAM systems typically focus on removing dynamic objects to reduce the influence of dynamic features on pose estimation, including DynaSLAM [3] , BlitzSLAM [10] , and DynaVINS [30] . However, these systems do not provide poses of dynamic objects. Object-oriented SLAM (OOSLAM), as an important branch in the research of object-level scene understanding, aims to extract, model, and track the static and dynamic objects in the environment [1] . Early OOSLAM systems [28, 32] typically maintain a database of objects. These systems utilize RGB-D cameras as inputs and employ ICP losses along with pose graph optimization to solve for the camera and object poses. However, this model-based approach requires prepared object models in advance and cannot exhibit robust scalability to unknown scenes. Recent advancements in OOSLAM have increasingly focused on improving object feature extraction and modeling. In QuadricSLAM [24] , objects are modeled as ellipsoids or quadric geometry, enabling more effective optimization of shape and scale. CubeSLAM [36] aligns the detection bounding boxes with the image edges to enhance object proposals. DSP-SLAM [33] leverages the signed distance function (SDF) to reconstruct the objects after detection. The learned shape embedding serve as priors during the reconstruction, and the shapes of the objects are jointly optimized in the bundle adjustment process. In MOTSLAM [41] , Zhang et al. apply three neural networks to extract 2D bounding boxes, 3D bounding boxes, and semantic segmentation, respectively.While the multi-network approach can extract accurate information about an object from different scales, it does demand large computational resources. DynaSLAM II [2] employs instance semantic segmentation and tracks ORB features on dynamic objects. Similarly, VDO-SLAM [42] leverages instance semantic segmentation, utilizing dense optical flow to maximize the number of tracked points on moving objects. Furthermore, dense optical flow is applied to ensure consistent tracking across multiple objects.However, most of OOSLAM systems only utilize objects as landmarks for observation, lacking a complete multi-object tracking process and failing to output trajectories for surrounding objects. For the noises in object detection and camera pose tracking, object tracking remains a challenging problem in SLAM settings, especially in object-crowded environments.2D multi-object tracking, which involves tracking object bounding boxes in the image plane, has been extensively studied [40, [43] [44] [45] . 3D multi-object tracking remains a challenging problem due to the involvement of spatial motion and the 3D appearance of the objects [26, 29] .The framework of 3D MOT is mainly divided into tracking by detection and learning-based algorithms. The former mainly uses Kalman filtering to estimate objects and then relies on specific indicators such as Intersection of Union (IoU) [34] , Mahalabnobis distance [7] , or Generalized IoU [25] for association.For learning-based algorithms, many works model 3D MOT as GNN, representing the association information of objects by predicting edges [4, 35, 39] . More relevant to this paper is the approach based on geometric information. PolarMOT [16] only uses 3D boxes as inputs to GNN to learn the geometric features of objects. BOTT [46] relies on self-attention to represent global context information and associate boxes.Existing 3D MOT approaches generally work either in the egomotion frame or under the assumption of known ego-motion. However, SLAM and 3D MOT are specially required in unknown environments or in the presence of ego-motion noise. Therefore, our GSLAMOT presents a practical approach. 
approach overview: 
 The proposed GSLAMOT framework is shown in Figure 1 . We consider the agent is conducting self-locating, environment mapping, and multiple object 3D pose tracking in an unfamiliar environment. The agent is equipped with a stereo camera that captures RGB image pairs, and a 3D LiDAR. The stereo images are processed by the visual odometry (VO) system and the LiDAR point clouds are used for 3D object detection [19, 20] .For the 𝑡-th frame, the sensor input is denoted as Ω 𝑡 . The 3D detector provides the object detection results O 𝑡 . The VO frontend outputs the current ego-motion pose 𝑇 𝑡 . The detections O 𝑡 are transformed into the world coordinate frame by Q 𝑡 = 𝑇 𝑡 O 𝑡 , and the Query Graph G 𝑄 is constructed.EQUATIONEQUATIONEQUATIONIn processing this new frame, the active tracklets J 𝑡 -1 are predicted to time 𝑡 based on motion model, denoted asJ 𝑝𝑟𝑒𝑑 𝑡 . Then a tracklet graph G 𝑇 is constructed based on J 𝑝𝑟𝑒𝑑 𝑡. Subsequently, we employ Multi-criteria Star Graph Association (MSGA) to match between G 𝑄 and G 𝑇 to obtain the matching relationship M 𝑡 .EQUATIONEQUATIONEQUATIONThen, We employ Object-centric Graph Optimization (OGO) to optimize the ego-motion 𝑇 𝑡 , tracklets J 𝑡 , and the map points 𝑃 𝑡 . OGO operates within a window size 𝑤, incorporating Object-Centric Optimization Window (OCOW) and Object-Ego Fusion Window (OEFW). The optimized results are used to update the Tracklet Graphs.EQUATION𝑇 * 𝑡-𝑤:𝑡 , J * 𝑡-𝑤:𝑡 , 𝑃 * 𝑡-𝑤:𝑡 = 𝑓 𝑂𝐸𝐹𝑊 (𝑇 𝑡-𝑤:𝑡 , M 𝑡-𝑤:𝑡 , J 𝑡-𝑤:𝑡 , Q 𝑡-𝑤:𝑡 ) (8) We use Pointpillars [19] for LiDAR-based 3D object detection, and use ORB-SLAM3 [5] for visual odometry front-end. Their results, i.e., O 𝑡 and 𝑇 𝑡 are inputs of following steps. 
multi-criteria star graph association: 
 We firstly introduction the construction of TG and QG, and the Multi-criteria Star Graph Association (MSGA). (3)Tracklet Graph Matching ResultAt frame 𝑡, we select the active tracklets from frame 𝑡 -𝑤 to frame 𝑡 -1 as J 𝑡 -1 , where 𝑤 is the OGO window size (detailed in Section 5). The critical task at this stage is to correctly associate the objects in O 𝑡 with those in J 𝑡 -1 . The association is challenging for the following reasons: (1) the agent and the objects are moving; (2) the objects may be crowded; and (3) the detections may be noisy. Thus, we firstly predict the poses of the tracklets, i.e., J 𝑝𝑟𝑒𝑑 𝑡 = 𝑓 𝑚𝑜𝑡𝑖𝑜𝑛 (J 𝑡 -1 ), where 𝑓 𝑚𝑜𝑡𝑖𝑜𝑛 is based on Kalman Filter [25] using the objects' estimated motion velocities and historical trajectories. Then we construct a graph for both the detections O 𝑡 and the predicted poses of the tracklets, named Query Graph (QG) and Tracklet Graph (TG) respectively.In particular, in constructing QG, we firstly transform the poses of the detected 3D boxes to the world frame, obtaining Q 𝑡 . Then for each detection 𝑖 in Q 𝑡 , the nearest 𝐾 detections within distance 𝐿 are chosen as 𝑖's neighbor set N 𝑖 . We build edges between each detection and its neighbors. This converts the detection set Q 𝑡 to a graph G 𝑄 . Each detection 𝑖 also forms a local star-graph G 𝑄 𝑖 = (V 𝑖 , E 𝑖 ), where V 𝑖 = {𝑖} ∪ N 𝑖 , and E 𝑖 includes the edges from 𝑖 to each element in N 𝑖 .In constructing TG, for each 𝑗 in J 𝑝𝑟𝑒𝑑 𝑡 , the nearest 𝐾 predicted object poses within distance 𝐿 are chosen as 𝑗's neighbor set N 𝑗 . We also build edges between each tracklet and its neighbors, and converts J 𝑝𝑟𝑒𝑑 𝑡 to a graph G 𝑇 . Each predicted object 𝑗 also forms a local star-graph G 𝑇 𝑗 = (V 𝑗 , E 𝑗 ), where V 𝑗 = { 𝑗 } ∪ N 𝑗 , and E 𝑗 includes the edges from 𝑗 to its neighbors.The purpose of association is to determine whether a detection 𝑖 ∈ Q 𝑡 and a tracklet 𝑗 ∈ J 𝑝𝑟𝑒𝑑 𝑡 are corresponding to the same object in the environment. We access this by multi-criteria of G 𝑄 𝑖 and G 𝑇 𝑗 . Two subgraphs with similar structures and features should exhibit a high graph feature consistency [23] . We particularly consider consistency evalution from three aspects: (1) neighborhood consistency, which can be considered as edge similarity; (2) spatial consistency, evaluated by Normalized Generalized Intersection over Union (NGIoU) [27] and (3) shape consistency, evaluated by Iterative Closest Point (ICP) [15] between the point clouds in the object bounding boxes. EQUATIONwhere T 𝑖𝑚 𝑡 is the transformation of 𝑒 𝑖𝑚 𝑡 and (T 𝑗𝑛 𝑡 ) -1 is the inverse transformation of 𝑒 𝑗𝑛 𝑡 . ∥ • ∥ 𝑭 represents the Frobenius norm. However, at this stage, we have not yet completed the object association, meaning that for the query and tracklet graphs, the correspondence of leaf vertices (𝑚 and 𝑛 in Equation ( 9)) is unknown. This presents a classic chicken-and-egg problem. We propose a Greedy Neighbor Strategy to address this issue. For EQUATIONwhere [34] or inter-distance [7] . However, traditional methods based on interdistance or IoU between the current detection and tracklets may fail when the distance or IoU values exceed thresholds. This is generally when the agent or objects move at high speed or when there are detection errors. Furthermore, IoU cannot handle cases where two bounding boxes do not overlap, leading to redundant objects in the tracking results.To address the issues above, we utilize Normalized Generalized Intersection over Union (NGIoU) as our spatial consistency metric. For two centering vertices 𝑖 and 𝑗 corresponding to two star graphs (for instance, detection 𝑖 and tracklet 𝑗), we suppose their 3D bounding boxes are 𝐵 𝑖 and 𝐵 𝑗 respectively. Then the value of GIoU is calculated as follows:EQUATIONWe further normalize GIoU to ensure its range lies within [0, 1]:EQUATIONExperimental results in Section 7 demonstrate that this metric enhances object tracking performances in challenging scenarios.Traditional 3D object tracking methods [7, 34] rarely leverage semantic information within the bounding boxes, while deep learning-based approaches rely on implicit features. We find that the point cloud within the bounding box inherently represents the shape of the object, which can be considered as semantic information and is an important clue for evaluating object consistency. Although the shapes of the same object in consecutive frames are similar, the view angles maybe different. So we propose a point cloud registration based methods to evaluate the point cloud shape similarity.For detection 𝑖 in G 𝑄 and tracklet 𝑗 in G 𝑇 , the point clouds within their respective bounding boxes are denoted as 𝑃 The ICP algorithm with RANSA [47] is used to compute the pointto-point correspondence between 𝑃 𝑄 𝑖 and 𝑃 𝑇 𝑗 . The fitness score calculated for the two point clouds is given by:EQUATIONwhere 𝑛 𝑐 represents the number of successfully registered points.For each G 𝑄 𝑖 and its central vertex 𝑖, we first collect the candidata set C 𝑖 . For each tracklet star-graph in C 𝑖 , the distance between its central vertex and 𝑖 is less than 𝐿. For a query star-graph G 𝑄 𝑖 and a tracklet star-graph G 𝑇 𝑗 in C 𝑖 , the multi-criteria feature consistency is given by: 𝑆 (𝑖, 𝑗) = 𝜆 1 𝑆 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 (𝑖, 𝑗) + 𝜆 2 𝑆 𝑁 𝐺𝐼𝑜𝑈 (𝑖, 𝑗) + 𝜆 3 𝑆 𝑓 𝑖𝑡 (𝑖, 𝑗) (14) If 𝑆 (𝑖, 𝑗) falls below a threshold 𝜏, then we consider 𝑖 and 𝑗 should not be associated, leading to the removal of 𝑗 from the candidate set C 𝑖 . If the candidate set C 𝑖 is empty, this indicates that 𝑖 is a newly detected object does not originally exist in the map. We create a new tracklet, which is added as the only element in C 𝑖 . Finally, we utilize the Kuhn-Munkres algorithm to compute the final one-to-one detection-to-tracklet matching by seeking maximum consistency:EQUATIONwhere Ψ 𝑡 represents the set of all possible matches and 𝑗 = M 𝑖 𝑡 represents the association of detection 𝑖 with tracklet 𝑗 under the matching relationship M 𝑡 .  
object-centric graph optimization: 
 Graph optimization is the widely employed back-end in SLAM (Simultaneous Localization and Mapping) [8, 13] . It represents egomotion states and sensor measurements as nodes and edges in a graph, utilizing optimization algorithms to estimate the agent's trajectory and the environment map. However, we observed that this approach performs well primarily in static scenes. For tracking dynamic objects, the ego-motion errors and the object pose errors coexist, affecting the convergence speed and accuracy of the graph optimization. To enable optimization tailored for 3D tracklets, we propose a graph optimization framework named Object-centric Graph Optimization (OGO). We divide the sliding window into two parts: Object-centric Optimization Window (OCOW) and Object-Ego Fusion Window (OEFW), and two windows adopt different optimization strategies. 19). Our proposed OGO strategy can better optimize the system.Before graph optimization, we obtain initial ego-motion poses and map points for each frame from the visual odometry. Additionally, initial tracklets and the matching result M 𝑡 for each frame are also available after tracking and association.In the object-centric optimization window, we adopt a two-stage optimization strategy (Figure 3 ). In the first stage, we solely utilize residuals from static environment landmarks in SLAM and egomotion poses to estimate ego-motion. At this point, the ego motion serves as a relatively reliable initial value. The objective function of the first stage is: After the first stage, the estimated ego-motion is reliable due to the elimination of dynamic objects in estimation. Then, we fix the ego-motion and solely optimize the object poses using residuals from object detection. The object detection residual at frame 𝑡 is:EQUATION)where 𝑗 = M 𝑖 𝑡 and T 𝑎𝑏 𝑡 represents the transformation from 𝑎 to 𝑏 at frame 𝑡. The objective function for the sliding window is:𝑠𝑡𝑎𝑔𝑒 2 : J * = arg min J ∑︁ 𝑡 ∈ T 𝑂𝐶𝑂𝑊 𝐸 𝑜𝑏 𝑗𝑒𝑐𝑡 𝑡 (18)We combine reliable tracklets and ego-motion poses for joint optimization in OEFW, thereby enhancing the accuracy of both.When the number of frames where a static object is observed exceeds a threshold 𝑤, we move it from the object-centric optimization window (OCOW) to the object-ego fusion window (OEFW). Once all detections in a frame have been moved to the OEFW, we transfer that frame's ego-motion pose and map points to the OEFW. In OEFW, objects and tracklets have undergone sufficient multi-frame observations, possessing good initial values and low system error. Reliable observation and joint optimization can help correct cumulative errors and improve the accuracy of locating and tracking. We employ a tightly coupled optimization strategy within the window, jointly optimizing ego-motion poses, map points, and tracklet poses. The objective function of OEFW is:T * , J * , P * = arg min T, J,P ∑︁ 𝑡 ∈ T 𝑂𝐸𝐹𝑊 𝐸 𝑜𝑏 𝑗𝑒𝑐𝑡 𝑡 + 𝐸 𝑚𝑎𝑝 𝑡 (19)where T, J , P represent the set of ego-motion poses, the set of object poses, and the set of landmark positions, respectively. T 𝑂𝐸𝐹𝑊 is the set of frames in OEFW. 
system implementation details 6.1 system acceleration: 
 GSLAMOT demands high real-time performance for localization and tracking. To enhance system efficiency, we leverage parallel operations, effectively reducing the processing time for each frame of data. Specifically, tasks including visual front-end localization, environment map construction, object detection, and tracking are processed concurrently by different threads. The VO front-end determines whether a frame is a keyframe [5] . Keyframes are processed by MSGA and OGO. Localization and mapping for non-keyframes proceed in other parallel threads, thus eliminating the need to wait for time-consuming keyframe operations. Specifically, the odometry, mapping, detection, and submodules of OGO and MSGA all run on separate threads (Figure 5 ). In addition, we utilize parallel acceleration for point cloud operations and registration [47] during the computation of shape features. Experimental results demonstrate that our system achieves real-time performance and incurs less time overhead compared to other OOSLAM systems (Section 7.6). Highly dynamic and congested traffic environments are ubiquitous in the real world, but few data sets emphasize these scenarios. To address this challenge, we use the Carla simulator [9] to simulate dynamic and crowded scenes, and generate the Traffic Congestion Dataset (TCD). We collected several sequences from four maps and the sensors on the vehicle include stereo cameras and 64-channel LiDAR data.In addition to the ego vehicle, hundreds of other dynamic or static vehicles are on the map to simulate dynamic and crowded environments in reality. In each segment sequence of the Waymo Open Dataset [31] , the average number of vehicles is less than one hundred, while in our Traffic Congestion dataset, the number of vehicles per segment sequence ranges from 200 to 300. The dense and dynamic flow of vehicles pose challenges to accurate selflocalization and MOT. We will open-source this dataset to support further research in localization and 3D object tracking. 
experiments 7.1 experimental settings: 
 We perform evaluations of the proposed algorithm and compare it with the state-of-the-art algorithms, including the SLAM-based systems: ORBSLAM3 [5] , DynaSLAM [3] , VDO-SLAM [42] , DSP-SLAM [33] and MOT algorithms: AB3DMOT [34] , ProbTrack [7] , CenterPoint [37] , BOTT [46] , SimpleTrack [25] , TrajectoryFormer [6] . All experiments are conducted on a computer with a CPU of i7-12700, GPU of RTX 3060Ti, and 16G RAM. We adopt ORB-SLAM3 as our visual odometry front-end. We use Pointpillars [19] for 3D object detection. We evaluated our system on the KITTI benchmark [12] , Waymo Open Dataset [31] and our emulated Traffic Congestion dataset. We evaluate the results of 3D MOT on both the Traffic Congestion Dataset (TCD) (Table 2 ) and the widely used Waymo Open Dataset (WOD) (Table 1 ). For the WOD, we report the mismatch metrics and the MOTA metrics for each of the three categories (vehicle, pedestrian, and cyclist).Researches on 3D MOT typically assume known ego-motion poses. However, our system computes the ego-motion in real-time, making the comparisons unfair. Therefore, we provide two system implementations: one calculates ego-motion poses simultaneously (GSLAMOT) and another utilizes ground truth ego-motion poses as inputs (GSLAMOT*). Results show that our methods outperform other methods in both known ego-motion and unknown ego-motion scenarios. The environment conditions, like rainy, snowy, and foggy days, may interfere with hardware devices and algorithms and then may produce noises. However, few studies have focused on these noises in 3D object tracking. We add Gaussian noise to the ego-motion and 3D detection boxes to test the algorithm's robustness to noise on the Traffic Congestion dataset. Specifically, gaussian noise with zero mean and standard deviation 𝜎 is added to the sampled detections. We gradually increase the value of 𝜎. Experimental results (Figure 8 ) show that our system maintains superior tracking results compared to other algorithms even as noise increases, demonstrating strong robustness.We comprehensively evaluate the trajectory accuracy on two benchmark datasets: the KITTI dataset and the Traffic Congestion dataset. We employ standard metrics and compare them against the stateof-the-art SLAM systems. Trajectory results are processed using the EVO tool [14] . Specifically, we consider both the Absolute Pose Error (APE) and Relative Pose Error (RPE) for each trajectory. The Root Mean Square Error (RMSE) is chosen as it provides a more robust evaluation than the mean value. The evaluation results are detailed in Table 3 for the TCD and Table 4 for the KITTI dataset. The results show the superior performance of our approach across different datasets. For the performances on KITTI, our algorithm has achieved top-1 or top-2 in most sequences. Although the vehicles in the scenes of the KITTI dataset are not crowded, our proposed association method still achieves a modest improvement. It performs particularly better in sequences with a larger number of objects, such as in KITTI-07. In the TCD dataset, our algorithm has demonstrated significantly superior performance due to the presence of crowded objects in the scenes.We test the efficiency of the proposed modules in ablation experiments, including the Multi-criteria Star Graph Association (MSGA) and Object-centric Graph Optimization (OGO). The experiments are conducted on the proposed TCD dataset to better facilitate the evaluation of both 3D MOT and SLAM.In the ablation experiments of OGO (Table 6 ), the tightly coupled OEFW significantly reduces APE, while the two-stage real-time optimization of OCOW reduces RPE. For the MSGA (Table 5 ), spatial, neighborhood, and shape consistency all contribute to the improvement of MOT. To evaluate the system's efficiency and real-time applicability, we measure the average processing time per frame, encompassing all stages, including ego-motion pose estimation, object analysis, association, optimization, and map integration. We conduct experiments over sequences from the KITTI and TCD. The average computation time is reported. The evaluation results in Table 7 affirm GSLAMOT's real-time capability and computational efficiency. We evaluate the running time of each main module in the system. Note that some modules, such as detection, tracking, and optimization, run only on keyframes selected by the VO front-end. To facilitate a more intuitive time comparison, we calculate the average time expenditure of all modules across all frames. The results are shown in Table 8 . Due to the parallel computing strategy, the total time expenditure per frame is significantly less than the time consumed by serial computation of individual modules. Time Per Frame(s)↓ ORBSLAM3 [5] 0.025 DSP-SLAM [33] 0.12 DynaSLAM [3] 0.086 VDO-SLAM [42] 0.14 GSLAMOT( ours) 0.082  
conclusion: 
 This paper investigates the problem of Simultaneous Locating, Mapping, and 3D Multiple Object Tracking. We propose the MSGA algorithm, which computes features from query graphs and tracklet graphs to achieve detection and tracklet association. After association, we design the OGO strategy to optimize ego-motion and tracklet poses in multiple stages. Additionally, we create a simulation dataset to explore the challenges of 3D MOT in highly congested, dynamic scenes. In the future, we plan to extend GSLAMOT to multi-agent collaborative scenarios to leverage the information from multiple agents and viewpoints to enhance system performance. 
: 
 Multiple factors work collectively making this problem very challenging. (1) SLAM and 3D MOT need to be conducted concurrently and depend on each other. SLAM relies on object detection to eliminate the impacts of the dynamic objects for accurately tracking the agent poses and for static mapping, whereas 3D MOT relies on the accurate pose of the agent for triangulating the 3D poses of the mobile objects; (2) The concurrent movements of both the agent and the dynamic objects pose challenges for locating and object tracking; (3) Errors in possible object detection algorithms and sensor noise also contribute to inaccuracies in localizing surrounding objects; (4) Factors such as occlusions and high-speed movement also introduce difficulty in object matching, leading to locating and tracking errors.Existing works in 3D multiple object tracking (3D MOT) [25, 34, 46] often assume that the ego-motion is known and free from noise or the sensor is fixed in the world frame. However, in real-world applications, it is general that both the agent and surrounding objects are in motion, requiring concurrent SLAM and 3D MOT. Furthermore, due to motion, the agent may produce inconsistent observations of the same object in different frames, leading to inaccuracies of the bounding boxes (Figure 3 ) and the difficulty in object matching across frames. Object-Oriented SLAM (OOSLAM) [3, 28, 33, 48] is another closely related area, but OOSLAM primarily focuses on estimating the ego-motion trajectory. It does not explicitly address the optimization of the detailed object 3D trajectories. But many applications do require real-time tracking of the 3D object poses [11, 18, 21] .To address the above challenges, this paper presents GSLAMOT, a graph matching and graph optimization based system for conducting SLAM and 3D MOT simultaneously, which takes stereo images and LiDAR point cloud sequences as input. To the best of our knowledge, it is the first work that outputs ego trajectory and object trajectories concurrently and accurately, even in challenging scenarios with crowded and highly dynamic objects. In particular, GSLAMOT represents the dynamic scene by a combination of (1) a semantic map representing the environment, (2) the agent trajectory, and (3) an online maintained Tracklet Graph representing the 3D MOT trajectories. TG tracks the 3D poses of multiple objects and can predict their poses at time 𝑡. A Query Graph (QG) is constructed in each frame based on object detection to query the predicted object poses. For accurate object association, a novel Multi-criteria Star Graph Association (MSGA) method is proposed for robust association to deal with the matching challenges in dynamic, congested, and noisy environments. MSGA evaluates neighborhood consistency, spatial consistency, and shape consistency between TG and QG, significantly improving multi-object tracking compared to using only spatial features.After finding associations, an Object-centric Graph Optimization (OGO) method is proposed to simultaneously optimize the TG, the map, and the agent trajectory. We divide the optimization In OCOW, a two-stage optimization strategy is adopted. Then, the well-optimized ego-motion, environmental points, and tracklets will be fused in the OEFW in a tightly coupled manner. Experiments show this object-centric graph optimization achieves better convergence speed (Figure 4 ) and accuracy than the ego-centric methods.In system implementation, to enable the localization, 3D MOT, and semantic mapping to run concurrently and efficiently, We employ multi-thread parallel, executing modules such as visual front-end, mapping, detection, feature extraction, and tracking concurrently across threads. The main contributions of this work are as follows:• We present GSLAMOT, which utilizes multimodel information, including stereo images and LiDAR point clouds, as inputs, and uses graph matching and graph optimization to conduct SLAM and 3D MOT simultaneously even in challenging scenarios. • Multi-criteria Star Graph Association (MSGA) is proposed to match QG and the predicted TG. • Object-Centric Graph Optimization (OGO) is proposed to estimate tracklets' poses. Object-Ego Fusion Window (OEFW) is proposed to jointly optimize the object poses and the egomotion poses in a long-term sliding window. • We exploit parallel threads to address the concurrency requirements in implementing the GSLAMOT system. Experiments demonstrate that GSLAMOT exhibits better realtime performance than the OOSLAM systems without object tracking. • We have also created a Traffic Congestion dataset for objectlevel scene understanding using the Carla simulator [9] . The dataset encompasses a wide range of maps and has a varying number of dynamic and static objects, providing a valuable resource for research and development in this field. 
 
 Appendix:  


Decision: All four reviewers are positive about the work and aprreciated the technqiue contribution and experimental validation in general. Some concerns were raised and were late mostly addressed by the authors' rebuttal. The final ratings are borderline accepts or weak accept. After checking all the reviews and rebuttal, we agree with the reviwer team and would like to recommend the paper to ACMMM.

Reviews: review: paper_summary: paper_summary: This paper focuses on ego trajectory and object trajectory concurrently for the first time. Specifically, it first proposes Multi-criteria Star Graph Associatoin to accurately 
match query graph and tracklet graph based on neighborhood consistency, spatial consistency and shape consistency. 
Then, it utilizes Object-centric Optimization Window  and Object-Ego Fusion Window from the respect of object detection and ego-motion for further optimization.
strength_weakness: strength_weakness: 1.This paper associates two independent tasks (i.e., Object-oriented SLAM and 3D Multiple Object Tracking) for the first time and achieves outstanding performance.
2. The experimental results are relatively superior on widely used KITTI benchmark and Waymo Open Dataset.
3. The structure of this paper is good and it is easy to follow this paper.
limitations: limitations: 1. Section 7.3 Robustness Evaluation only proves that the proposed GSLAMOT has higher tracking accuracy compared to other methods with Gaussian detection noise, which may be related to your method combining ego-motion and object detection to make tracking predictions. I think a specialized ablation experiment needs to be designed to demonstrate that GSLAMOT can prevent interference from detection errors (Section 5).
2. Section 7.2 3D Object Tracking Evaluation lacks of comparison on KITTI which is also commonly used on 3D MOT with available point cloud and stereo images.
3. Paper Writing has tense inconsistency. For example, in line 641 and 665, collected -> collect, evaluated -> evaluate
paper_summary: paper_summary: The paper proposes GSLAMOT to simultaneously locate, map, and track the 3D poses of multiple moving objects. The framework utilizes camera and LiDAR information to represent dynamic scenes through semantic maps, proxy tracks, and tracklet graphs maintained online. It uses a multi-standard star graph association method and an object-centered graph optimization method to accurately correlate and optimize objects, semantic maps, and proxy trajectories. Experimental results show that GSLAMOT performs better than existing methods.
strength_weakness: strength_weakness: 1.This method can output ego trajectory and object trajectories concurrently, which is novel.
2.The experiment is relatively sufficient.
3.A new dataset proposed, which is focused on scene with dense and dynamic flow.
limitations: limitations: 1.Does the method make a fair comparison with other methods on those existing datasets, e.g., fair inputs, fair data, fair pre-trained models?
2.TrajectoryFormer[6] have significantly higher results that have not been reported on the Waymo Open Dataset.
3.The objective function and errors are not clear in Sec.5. What parameters is trained? Why can such a design achieve the effect claimed by the author?
4.The details of the proposed dataset are neglected. And, I think this dataset might be of more interest to the community. I think the contribution of the paper needs to be reorganized.
paper_summary: paper_summary: The paper introduces GSLAMOT, designed for the tasks of locating, mapping, and tracking the 3D poses of multiple objects in unfamiliar settings. It leverages both camera and LiDAR multimodal data, employing a Tracklet and Query Graph-based framework to effectively handle dynamic scenes and achieve precise object association. Results indicate that the proposed method outperforms state-of-the-art techniques in accurately tracking crowded objects and conducting SLAM in difficult conditions.
strength_weakness: strength_weakness: 1. GSLAMOT can effectively leverages multimodal information and employs graph matching and optimization for simultaneous SLAM and 3DMOT even in challenging scenarios. 
2. The  Multi-criteria Star Graph Association (MSGA) and Object-Centric Graph Optimization (OGO) improve object matching and pose estimation. Object-Ego Fusion Window (OEFW) optimizes object and ego motion poses in a long-term sliding window, enhancing overall performance. The authors validated the effectiveness of these modules in the experiments.
3. The article's textual descriptions are clear.
limitations: limitations: 1. In Section 4.1, the authors obtain object tracklets using traditional methods. Would replacing this with a deep learning-based tracking methods work better?
2. In the speed testing experiment in Section 7.6, the authors should specify the hardware conditions used for testing.
3. The author's page count does not fill the last page completely; the length should be appropriately adjusted to fill the final page.
paper_summary: paper_summary: The work presents a graph matching and graph optimization-based method for performing SLAM (self-locating) and 3D MOT (tracking) jointly providing both ego trajectory and object trajectories even in crowded and highly dynamic scenes. Moreover, the method outputs also mapping results together with other outputs. This makes the proposed method a locating, mapping, and MOT system.
strength_weakness: strength_weakness: - The idea of jointly performing localization, tracking, and mapping is novel. The work shows competitive results even in challenging scenarios like crowded scenes.
- A multi-criteria star graph association is proposed to match different modalities. 
- The work also addresses the runtime issue considering the multi-task nature of the system.
limitations: limitations: - The work is validated on a self-created dataset based on a possible unrealistic simulator. A performance drop is observed in real-world scenes like the Waymo Open dataset. More discussion on the performance gap can be further addressed.  
- The used motion model in Line 291 can be further detailed.
- The dependency of the object detector is not addressed while the proposed method depends on the performance of the used object detector.