SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Path Efficiency. Lastly, we also measure agent path efficiencies, defined as the ratio of the distance between trajectory endpoints to the trajectory length [25]. Intuitively, the closer to  $100\%$ , the closer the trajectory is to a straight line. As can be seen in Fig. 15 in the appendix, most path efficiency distributions are uniformly distributed, with peaks near  $100\%$ , echoing earlier straight-line findings. However, the INTERACTION [39] dataset is an outlier in that its agent trajectories are predominantly straight lines with much less curved motion than other AV and pedestrian datasets.
4.3 Annotation Quality
While analyzing datasets' true annotation accuracy would be best, neither we nor the original data annotators have access to the underlying real-world ground truth. As a proxy, we instead analyze the self-consistency of annotations in the form of incidence rates of collisions between agents, off-road driving, and uncomfortable high-acceleration events (using  $0.4g$  as a standard threshold [44, 45]).
Figure 9: Agent speed distributions per dataset and agent type.
Figure 10: Changes in heading relative to an agent's first timestep.
Virtually all observed agent data is free of collisions and off-road driving, save for rare one-offs (e.g., the INTERACTION dataset contains a minor car accident [39]). We denote bounding box intersections between agents as collisions, and agent center-of-mass exiting the road boundary as off-road driving. Collisions typically indicate errors in bounding box annotations, whereas off-road driving can indicate erroneous bounding box dimensions, missing map coverage, or harsh driving that, e.g., cuts corners during a right turn.
As can be seen in Fig. [11] (left), most vehicles in datasets experience collision rates below  $5\%$ . Of particular note is the fact that state-of-the-art autolabeling systems (e.g., used in Waymo [17]) are nearly matching the accuracy of human annotations (e.g., used in nuscenes [18]) in terms of resulting collision rates. However, detecting agents from a near-ground perspective (even with 3D LiDAR) is a very challenging task, and current performance still lags behind high altitude viewpoints. In particular, The INTERACTION [39] dataset achieves orders of magnitude lower vehicle collision, off-road, and harsh acceleration rates owing to its drone-based data collection strategy. In theory, SDD [40] should enjoy a similar advantage, but it only provides axis-aligned bounding box annotations (which overestimate agent extents) and Stanford's college campus contains much more interactive agents than other urban environments. More generally, the notion of bounding box intersections as collisions does not transfer exactly to pedestrians as they can enter/exit cars and walk in close groups, and further study is needed to robustly distinguish between errant motion and normal interactive motion.
In Fig. [11](middle), we find that vehicles in general experience very few ( $< 1\%$ ) harsh acceleration events, with Waymo [17], Lyft [19], and nuScenes [18] all having the highest incidence, commensurate with their earlier-discussed long-tail acceleration distributions. Lastly, we find in Fig. [11](right) that the INTERACTION [39] and nuPlan [27] agent annotations are well-aligned onto their maps, whereas nuScenes [18] suffers from poor map coverage away from main roads (there are many annotated parked cars next to the main road) and Lyft [19] suffers from high false positive detections next to the main road (the majority of which take the Unknown class).
5 Conclusions and Recommendations
The recent releases of large-scale human trajectory datasets have significantly accelerated the field of AV research. However, their unique data formats and custom developer APIs have complicated multi-dataset research efforts (e.g., [20, 21]). In this work, we present trajdata, a unified trajectory data loader that aims to harmonize data formats, standardize data access APIs, and simplify the process of using multiple AV datasets within the AV research community with a simple, uniform, and efficient data representation and development API. We used trajdata to comprehensively compare existing trajectory datasets, finding that, in terms of annotation self-consistency, drone-based data collection methods yield significantly more accurate birds-eye view bounding box annotations than even state-of-the-art AV perception stacks with LiDAR (albeit with much less spatial coverage), modern
Figure 11: Self-consistency failure rates per dataset and agent type, in the form of collision (left), high vehicle acceleration (middle), and off-road (right) rates.
autolabeling pipelines are nearing human annotation performance, and smaller-scale pedestrian datasets can still be useful for investigations requiring high-agent-density scenarios.
As concrete recommendations, we saw that some datasets artificially limit the distance agents are autolabeled. Instead, it would be more useful to the long-range detection community to remove such restrictions, but add autolabeler-output uncertainties to long-range detections, supporting uncertain perception research along the way. Further, incorporating explicit self-consistency checks within autolabeling pipelines and catching, e.g., collisions, prior to release can both improve the autolabeling method as well as the resulting data labels.
More broadly, providing researchers with access to more data comprised of various agent types from diverse geographies should help in modeling rare agent types and behaviors, in addition to aiding in the generalization of methods to multiple geographies. However, as we have seen in prior sections, there is an overwhelming bias towards straight line driving, and one capability missing from trajdata is the ability to (re)balance data on a semantic (behavioral) level. Finally, even if lower-level trajectory classes (e.g., driving straight, turning left/right, slowing down, speeding up, etc) are balanced, an important higher-level consideration during original dataset curation time is to ensure that AV datasets explore all geographic regions within an environment, and not only those of certain socioeconomic statuses or transportation access.
Future work will address the current limitations of trajdata (e.g., expanding the number of supported datasets and new capabilities such as geometric map element associations to support Waymo-like map formats [17]). Further, incorporating sensor data would also enable perception research as well as joint perception-prediction-planning research, an exciting emerging AV research field.
We thank all past and present members of the NVIDIA Autonomous Vehicle Research Group for their code contributions to trajdata and feedback after using it in projects. We additionally thank Leon De Andrade, Alex Naumann, and Stepan Konev for their contributions to trajdata on GitHub.
[1] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras, "Human motion trajectory prediction: A survey," Int. Journal of Robotics Research, vol. 39, no. 8, pp. 895-935, 2020.
[2] General Motors, "Self-driving safety report," 2018, Available at https://www.gm.com/content/dam/company/docs/us/en/gmcom/gmsafetyreport.pdf.
[3] Uber Advanced Technologies Group, "A principled approach to safety," 2020, Available at https://uber.app.box.com/v/UberATGSafetyReport.
[4] Lyft, "Self-driving safety report," 2020, Available at https://2eg1kz1onwfq1djllo2xh4bb-wpengine.netdna-ssl.com/wp-content/uploads/2020/06/Safety_Report_2020.pdf
[5] Waymo, "Safety report," Waymo LLC, 2021, Available at https://waymo.com/safety/safety-report.
[42] Bureau of Transportation Statistics, "National Transportation Statistics. Number of U.S. Aircraft, Vehicles, Vessels, and Other Conveyances," U.S. Dept. of Transportation, Tech. Rep., 2023.
[43] E. Tolstaya, R. Mahjourian, C. Downey, B. Varadarajan, B. Sapp, and D. Anguelov, "Identifying driver interactions via conditional behavior prediction," in IEEE Int. Conf. on Robotics and Automation, 2021.
[44] B. G. Simons-Morton, J. D. Ouimet, J. Wang, S. G. Klauer, S. E. Lee, and T. A. Dingus, “Hard braking events among novice teenage drivers by passenger characteristics,” Driving Assessment Conference, vol. 5, pp. 236-242, 6 2009. [Online]. Available: https://pubs.lib.uowa.edu/driving/article/id/28044/
[45] S. G. Klauer, T. A. Dingus, V. L. Neale, J. D. Sudweeks, and D. J. Ramsey, “Comparing real-world behaviors of drivers with high versus low rates of crashes and near-crashes,” National Highway Traffic Safety Administration, Tech. Rep. DOT HS 811 091, 2009.