trajdata: A Unified Interface to Multiple Human Trajectory Datasets
Authors: Boris Ivanovic
Guanyu Song
Igor Gilitschenski
Marco Pavone
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: multi-dataset benchmarking
data standardization
autonomous vehicles
pedestrian motion
TL;DR: In this work, we present trajdata, a unified interface to multiple human trajectory datasets, as well as a comprehensive empirical evaluation of the trajectory datasets underpinning much of AV and pedestrian motion forecasting research.

Abstract: The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, real-world human trajectory datasets for autonomous vehicles (AVs) and pedestrian motion tracking. While such datasets have been a boon for the community, they each use custom and unique data formats and APIs, making it cumbersome for researchers to train and evaluate methods across multiple datasets. To remedy this, we present trajdata: a unified interface to multiple human trajectory datasets. At its core, trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data. As a demonstration of its capabilities, in this work we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a rich understanding of the data underpinning much of current pedestrian and AV motion forecasting research, and proposing suggestions for future datasets from these insights. trajdata is permissively licensed (Apache 2.0) and can be accessed online at https://github.com/NVlabs/trajdata.

trajdata: A Unified Interface to Multiple Human Trajectory Datasets
Boris Ivanovic'
Guanyu Song²
Igor Gilitschenski
Marco Pavone 1.3
1NVIII Research
2 University of Toronto
SStanford University
Abstract
The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, real-world human trajectory datasets for autonomous vehicles (AVs) and pedestrian motion tracking. While such datasets have been a boon for the community, they each use custom and unique data formats and APIs, making it cumbersome for researchers to train and evaluate methods across multiple datasets. To remedy this, we present trajdata: unified interface to multiple human trajectory datasets. At its core, trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data. As a demonstration of its capabilities, in this work we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a rich understanding of the data underpinning much of current pedestrian and AV motion forecasting research, and proposing suggestions for future datasets from these insights. trajdata is permissively licensed (Apache 2.0) and can be accessed online at /ectps://githng com hatm///aaa///traaaaaa
Introduction
Research in trajectory forecasting (i.e., predicting where an agent will be in the future) has grown significantly in recent years, partially owing to the success of deep learning methods on the task MI: availability of new large-scale, real-world datasets (see Fig. ; and investment its deployment within domains such as autonomous vehicles (AVs)
y and social robots [00
In addition, recent dataset releases have held associated prediction challenges which have periodically [6). While this has been a boon for research progress, each dataset has unique data format and development API, making it cumbersome for researchers to train and evaluate methods across multiple datasets. For instance, the recent Waymo Open Motion dataset employs binary TFRecords [17 which differ significantly from nuScenes' foreign-key format [8] and Woven Planet (Lyft) Level 5's compressed zarr files [9 The variety of data formats has also hindered research on topics which either require greatly benefit from multi-dataset comparisons, such as prediction model generalization (c.g.. 20 (11). To remedy this, we present trajdata: a unified interface multiple human trajectory datasets.
benchmarked the field and spurned new developments [133] 14 15
Contributions. Our key contributions are threcfold. First, we introduce a standard and simple data format for trajectory and map data, well extensible API to access and transform such data for research use. Second, we conduct a comprehensive empirical evaluation existing trajectory datasets, providing users with richer understanding of the data underpinning much of pedestrian and AV motion forecasting research. Finally, we leverage insights from these analyses to provide suggestions for future dataset releases.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

1080
Arnoattion Nethed
Haman Lebelers
Hours
100
Offilin Autolabel
Perception
Untabeted
1
Sonantic "
KETT]
R3D (Honds)
INTERACTION
Argoversel
nušcenes
Apal Apal Irajectoric
Cityscapes
A+
800106K POT
lyrt Level
Oxferd
RTK
KITTI 368
Argoverse2
Yandes
Shifts
Moyno Open
nuft an
Collection
Ornpard Sensors
Not
2
Fixed Carera
2019
2628
2821
28221
Figure 1: Recent datasets provide access to thousands of hours of autonomous driving data, albeit with different data formats and APIS, complicating the use of multiple datasets in research projects.
2 Related Work
Human Trajectory Datasets. Initial trajectory forecasting research employed video motion tracking datascts for benchmarking, primarily due to the availability of annotated agent positions over time. Of these, the ETH [22] and UCY 2 pedestrian datasets were among the most widely-used M. containing total of 1536 pedestrians and challenging behaviors such couples walking together, groups crossing each other, and groups forming and dispersing. Soon after the successful applica- tion of deep learning models to pedestrian trajectory forecasting [24], and as data needs grew in autonomous driving research and industry, numerous large-scale datasets have emerged containing significantly
heterogeneoou--aeet interactive scenarios (e.g., between vehicles and pedestrians) in urban environments. Fig. D visualizes the scale, collection, and annotation strategy such datasets, with comprehensive review of earlier human motion datasets available in gradual shift from human annotation to autolabeling can be seen, with the recent large-scale Yandex Shifts 26 Waymo Open Motion 17. and nuPlan 27 datasets employing powerful autolabeling pipelines to accurately label sensor data collected by vehicle fleets at scale.
25 In particular, the
Multi-Dataset Benchmarking. While the increase in datasets and associated challenges has bol- stered research, their unique formats increase the complexity of evaluating methods across datasets, complicating efforts analyze, e.g.. prediction model generalization. To address this issue for pedestrian motion data, Open (55 created dataloaders for different pedestrian motion datasets part of its effort evaluate and compare motion complexity across pedestrian datasets. More recently, TrajNet++ (23] and Atlas 29 present multi-dataset benchmarks to systematically evaluate human motion trajectory prediction algorithms in a unified framework. While these efforts have provided the community with multi-dataset benchmarks, they are primarily focused on pedestrian data. In contrast, traj data tackles the standardization of both pedestrian and autonomous vehicle datasets, including additional data modalities such as maps.
trajdata
3 trajdata: A Unified Interface to Multiple Human Trajectory Datasets
trajdata is software package that efficiently compiles multiple disparate dataset formats into one canonical format, with an API to access and transform that data for use in downstream frameworks (e.g., PoTorh [0), which is natively supported). Currently, trajdata supports 8 diverse datasets, comprising 3,216 hours of data, 200+ million unique agents, and 10+ locations across 7 countries (see Table D. To date, trajdata has been extensively used in research on trajectory forecasting 21
pedestrian 31 and vehicle 22
+
simulation, and AV motion planning 34 35
3.1 Standardized Trajectory and Map Formats
Trajectories. For each dataset, trajdata extracts position, velocity, acceleration, heading, and extent (length, width, height) information for all agents in standard SI units (see Fig. 2 In order to support a variety of dataset formats, trajdata has minimal base data requirements: As long as agent positions (i.e., x, y coordinates) are provided, all other dynamic information can be derived automatically. If entire dynamic quantities (e.g., velocity) are not captured in the original dataset, trajdata uses finite differences to compute derivatives by default. Further, missing data between

---

Table 1: Datasets currently supported by trajdata. More details can be found in the appendix.
Dataset
Size
Locations
Maps?
Dataset
Size
Locations
Maps?
ETH P2
0,4h
2
No
INTERACTION
33
16.5h
4
Yes
UCY PJ
0.3h
No
Lyft Level
19
1118h
Yes
SDD
40
5h
1
No
Waymo Open
570h
6
Yes
nuScenes
5.5h
2
Yes
nuPlan 27
1500h
Yes
Urinown
Index
Optional
Vorkdo
Podestrion
apen__0
scere_ts
Molorcydle
Past Nolion
Future Notion
Center
fees
Sidewaiko
Figure 2: Left: trajdata adopts tabular representation for trajectory data, leveraging advanced indexing to satisfy user data queries. Right: Agent trajectories from the nuScenes [Tg] dataset visualized on the scene's VectorMap, containing all of trajdata's core map elements.
timesteps is imputed via linear interpolation. trajdata internally represents and stores trajectory data tabular data frames, allowing for advanced indexing and data grouping depending and the use of efficient spencs tabular data storage frameworks such as Apache Arrow [B6]] Note that each of these default choices (finite differences, linear interpolation, and tabular data frames) can be changed by the end user.
queries
Maps. To retain the most information from high-defiittion (HD) dataset maps, trajdata adopts a polyline representation for map data. This choice matches the vast majority of modern trajec- tory datasets which provide vector map data and makes them immediately compatible with format. Currently, there are four core map elements: RoadLane, RoadArea, PedCrosswalk, and PedWalkway. As illustrated in Fig. 2 RoadLane represents driveable road lane with centerline and optional left and right boundaries. RoadArea represents other driveable areas of roads which not part of lanes, e.g., parking lots or shoulders. A PedCrosswalk denotes marked area where pedestrians can cross the road. Finally, a PedWalkway marks sidewalks adjacent to roads. Of these, required be extracted, other elements optional (they absent in
only RoadLane elements some datasets). Our map format additionally supports lane connectivity information the form of left/right adjacent lanes (i.e., lanes accessible by left/right lane changes) and successoorpreedeesssor lanes (i.c., lanes that continue from lead into the current lane following the road direction).
Each map element is designed to be compatible with popular computational geometry packages, such as Shapely (37,, enabling efficient set-theoretic queries to calculate, e.g., road boundary violations. By default, trajdar serializes map data using Google protocol buffers (38), and, in particular, only stores neighboring position differences for efficiency, similar to the implementation used [19]. Dynamic traffic light information is also supported, and trajdata makes use of separate data frame to link the traffic signal shown per timestep with the lane ID) being controlled.
3.2 Core trajdata Functionalities
Multi-dataset training and evaluation. One of trajdata's s core functionalitie: iis aggregating data from multiple datasets in a UnifiedDataset object (a PyTorch 300 Dataset subclass by default).
from trajdata import UnifiedDataset.
dataset UnifiedDataset(
tammiiiiibsston" "sdd "sdd-train"), Resired_dt=0.11 desired_data= daddddaaaa"ss_mmiiii centric=" "ggent,, (1ttory_ee=(110,, 3.0), seuure_sec-(4.... 4.
These settings
used to create Figure 2.
Detailedd demonstrations of trajdata's capabilities cappbiitisc
be found in repository's examples/ folder.

---

History
Agent
Future
Other Historfes
-2
+1
-5
55
-5
Batch Idk
Net ghbor
Only Neighbers
Batch
Idk
Batch Idx
Batch Idx
Available data
Padding
(Unavatlable data)
Figure 3: trajdas data can provide agent-centric (or scene-centrie) batches of trajectory data for model training and evaluation associated AgentBatch (or SceneBatch) objects. The indexing and padding strategy of few core AgentBatch tensors are visualized here.
The example above creates a dataset that provides agent-centric data batches (i.e., each batch el- ement contains data for one agent at one timestep, see Fig. 3 sourced from only Boston in nuScenes mini dataset ("nusc_ming "rini-bostonn)
well as the Stanford Drone Dataset's entire train- ing split ("sdd-trane -train"),, with time upsampling ensuring all data is at 10Hz (desired_dtee history_sec= (1.0, 3. 0) specifies that the predicted agent's s trajectory must have at least 1.0s of history available, with padding for any missing data up to 3.0s (see Fig. 3 Similarly, future_sec= (4.0, 4. requires that the predicted agent's 's trajectory have 4.0s of future available.
1).
trajdata provides many other capabilities in addition the above, including scene-centric batches (i.e,, data for all agents in scene at the same timestep), semantic search (e.g.. nuScenes [18] provides text descriptions for each scene), agent filtering (e-g., only vehicles), coordinate frame standardiza- tion (i.e., making trajectories relative to the predicted agent's frame the current timestep), map rasterization (e.g., encoding scene context with convolutional architecture), data augmentations (e.g., additive Gaussian noise to past trajctories), and general data transforms via custom functions.
Map APL. trajdata's standardized vector map object is VectorMap. In addition providing access to individual map elements (e.g., lanes, sidewalks), it also leverages precomputed spatial indices make nearest neighbor queries very efficient.
from trajdata import MapAPI, VectorMap
_ec_map: VectorMap = (apAPI(<=>) _ep_ ("nusc mini boston seaport lane vec_map gett closesclosst__ee lane (np- array ([50.0, 100.0, 0.0])
In the example above, the polyline map of Boston's 's seaport neighborhood (from nuScenes 1 is loaded from the user's trajdata cache (its path would be specified instead of <=>) and queried for the closest RoadLane to a given x, y, position.
Simulation Interface. trajdata also provides a simulation interface that enables users to initialize a scene from real-world data and simulate agents from specific timestep onwards. Simulated agent motion is recorded by trajdata and be analyzed with library of evaluation metrics (e.g.. collision and offroad rates, statistical differences to real-world data distributions) or exported to disk. This functionality extensively used to benchmark Carning-based traffic models in 32
from trajdata.simulaticn simulation import SimulationScene sim_scene SimulationScene (<=>) (<=>) Specify initial scene to use. obs sim_scene () Initialized from real agent states in data. for range (10) Simulating # Compute the new state sim agents.
timesteps this example.
new new_state_dice obs = sim scene. step (new_state_dict)
In this example, a Simulati onScene initialized from scene in an existing dataset (specified with the <=> arguments), after which it can be accessed similarly to an OpenAl Gym 4 reinforcement learning environment, using methods like reset and step.
Dataset Comparisons and Analyses
In this section, we leverage trajdata's standardized trajectory and map representations to directly compare many popular AV and pedestrian trajectory datasets along variety of metrics. Our goal is to provide a deeper understanding of the datasets underpinning much of human motion research by analyzing their data distributions, motion complexity, and annotation quality.

---

10a
Dataset
nuSc
Number Agooes
10
80%
WOMD
INT
30a
Propartin
60%
INT
Lyft
10F
40%6
nuPlen
104
20%6
FTH
10
UNKNOWN
VEHICLE
PEDESTRIAN
BICYCLE
MOTORCYCLE
Agent Type
Figure 4: Left: Number of unique agents per dataset. Right: Distribution of agent types per dataset.
Note that only analyze dataset training and validation splits, since these are the splits predominantly used by methods for development. We explicitly do not analyze test splits since they are either not available publicly or because doing so may harm existing benchmark validity. Further, while trajdata supports data frequency up- and down-scaling via interpolation and down-sampling, all of the following analyses were conducted in each dataset's native data resolution. All analyses were performed using the latest version of trajdata at the time of writing (v1.3.2) on desktop computer with 64 GB of RAM and an AMD Ryzen Threadripper PRO 3975WX 32-core CPU. For larger datasets, an NVIDIA DGX-1 1 server with 400 GB of RAM and 64 CPU cores was used.
4.1 Agent Distributions
Population. To build fundamental understanding of the considered datasets, we first analyze and compare agent populations. Fig. visualizes overall agent counts and proportions per dataset. As can be expected, modern large-scale AV datasets such as Waymo [17 and Lyft Level 5 [99 contain multiple orders of magnitude more agents than earlier pedestrian datasets SDD 440, ETH [2 or UCY 23.. However, as we will show later, pedestrian datasets still provide value in terms of agent diversity, density, and motion complexity in popular social robotics settings such as college campuses.
As can be seen in Fig. 1 (right), the vast majority of agents in AV datasets are vehicles or pedestrians, with the exception of Lyft Level [9 where 71.8% of agents have unknown types. In contrast, bicy- cles (a relatively niche category many datasets) account for 41% of all agents in SDD 440 (indeed, biking is popular method of transportation around Stanford's 's large campus). Such imbalances in agent populations are indicative real-world distributions, e.g., motorcycles make up only 3.5% of vehicles in the USA 42] similar to their proportion in nuScenes [18] (1.6%).
Density and Observation Duration. In addition to which agent types are captured in scenes, the amount and density of agents can be an important desiderata (c.g.. for research on crowd behavior) or computational consideration (c.g., for methods whose runtime scales with the number of agents). Fig. 5 visualizes the distribution of the number of agents observed per scene per timestep (left), as well as the maximum number of simultaneous agents per scene (right). As can be seen, urban scenarios captured in modern AV datasets frequently contain 100+ detected agents (with a long tail extending 250++ agents). In this respect, ETH UCY 23 and INTERACTION 39 are limited by their fixed-camera and drone-based data-collection strategies compared to the comprehensive on-vehicle sensors used in nuScenes [18] Waymo [7]. Lyft 19 and nuPlan 27 However, while ETH (22), 39 do not contain many agents, they consistently provide the
UCY 23 and INTERACTIN highest-density scenes (see Fig. 6 especially for pedestrians and bicycles. We compute agent density by dividing the number of agents in scene by their overall bounding rectangle area, in 225]
Each dataset supported by trajdata adopts different scenario lengths and corresponding agent observation durations. As can be seen in Fig. 7 AV datasets are comprised of scenarios with lengths ranging from 4s in INTERACTION each AV dataset duration distribution are caused by the Kdways-presene ego-vehicle (for Vehicles) as well as other agents detected throughout the scene (common in steady traffic, parking lots, or at an intersection with stopped traffic and pedestrians waiting to cross). One can also see that Lyft Level [19] agent detections are much shorter-lived compared to other AV datasets' relatively uniform distributions (Waymo [r] nuScenes [8] and nuPlan 27] This could be caused by Lyft's
25s in Lyft Level 5
119. The peaks at the right of
5

---

nuSc
WOMD
Datiooet
S
10
10
Proparton
lyf
10
nuFlan
S20
ETH
30*4
UGY
250
50
250
Simulareuus Agents
Max.
Simulumeous Agents
Figure 5: Left: Number of agents present per timestamp and scene. Right: Maximum number of agents present at the same time per scene.
Unkmowss
Velidee
Pedvettions
Bicyc
Meistydee
WCMD
134
12-
ETH
10 10n1
10 10
10t1 1057
18t1
10"
10th
10ah
13"h
10"1
10 (0
R0TT 18 18t1 13"1
Ageri sqgeav))
Ageri Ageaarn
AgerzOera jgpram))
Ageri sipeaan))
Agerz Densly (ipeate))
Figure 6: Agent density per timestep and scene.
(which are affected by noise and occlusions) vs human annotators [8] or autolabeling [27 17 which can leverage data from past and future timesteps be more robust to such errors. We conduct additional comparisons between data collection methodologies in Section
annotations being collected from an onboard perception system
19
When developing AV perception systems, important consideration the sensor range(s) necessary to facilitate the desired prediction and planning horizons as well as provide advanced warning of critical situations (e.g., stopped traffic on highway). In Fig. M compare the distribution of ego-agent distances and find that, while nuSeenes 1 and Lyft Level 5 [9 have long-tailer distributions extending past 200m. Waymo [7 and nuPlan 2 appear to have artificial cut-offss at 75-80m. potentially to maintain data quality by avoiding poor data from distant agents. However, it would be more useful to maintain distant detections and add uncertainty outputs from the autolabeler to support uncertain long-range detection research in addition to improving autolabeling.
Ego-Agent Distances.
Mapped Areas. HD maps are core component of many AV datasets, frequently leveraged in trajectory forecasting and motion planning research to provide scene context and geometric lane information (e.g.. for global search-based planning and trajectory optimization). Current AV dataset maps are very large (see Table in the appendix) and comprehensive, spanning multiple neighbor- hoods in different cities. However, not all HD maps are created equal, commonly differing along three axes: Area complcteness, lane definitions, and traffic lights. While most AV datasets provide complete HD maps of neighborhoods, Waymo [7 differs by only providing local map crops per scenario without common reference frame across scenarios This also significantly increases the storage requirements of Waymo 17 maps compared to other datasets.
Lane definitions can also differ significantly between datasets, with intersections being a notable differentiator. For instance, the nuScenes dataset [18 does not annotate intersections fully, opting for only lane centerlines without associated edges (Fig. shows an example). Lyft Level 5 [9 and nuPlan 27 both include full lane center and edge information for all possible motion paths through an intersection. Waymo [77 maps are unique in that they provide full lane center and boundary information. but there are many gaps in the associations between lane centerlines and boundaries, making it difficult to construct lane edge polylines or lane area polygons B As a result, we exclude Waymo maps from map- analyses this work.
4.2 Motion Complexity
Measuring the complexity of driving scenarios is an important open problem in the AV domain, with a variety of proposed approaches ranging from heuristic methods 25 to powerful conditional behavior
htt.ss////hhhoooooorrrrrrr////as for visualizations.
for visualizations.

---

Unkinoenn
Vehicles
Podestriand
Bicycles
14:52
1214
WCMD
20-4
10
kym
10-4
Eyr
207
UDV
500
221
30
15
20
35
35
20
35
15
20
30
15
20
50
15
75
Agem Oeserval Langhh b4
Agem Obserca Lenghh 34
Agent Obseraa Langhh 30
Agert Dbmervation Langhh 30
Agert Deservationnnnnm Lengh 10
Figure 7: Distributions of the length of time agents are observed in each scene.
Unionows
Pedeabiaaa
Blyysko
Motorcys 66
Agert Ego Dersince 71
Agrre-ppo Dersunce PMD
Agere Ege Deraace PM
Agen--gg Dessunce
Agent Ege Drisunce (1))
Figure 8: Distribution of distances between agents and data-collecting ego-vehicle in AV datasets.
prediction models 4%]
To avoid potential biases in analyzing datasets with an externally-trained model, we employ simple and interpretable heuristics similar to [25].
Motion Diversity. We first analyze distributions of dynamic agent quantities (e.g., speed, acceleration, jerk). As can be seen in Fig. y the majority of speed distributions have high peaks at zero (no motion). This is corroborated by Table 3 in the appendix, which shows that significant portion of agents are stationary in many datasets, especially for nuScenes [ (17.5%) and Waymo [T7] (53.6% (53.6)). After the initial peak, agent speed distributions drop sharply to roughly uniform plateau (up to 20m/s for vehicles) before dropping completely around 30m/s (a common highway speed around the world).
While SDD 40 and ICTION 39 have sensible vehicle speeds, their pedestrian speeds can be too high. Such high speeds may be caused by annotations near the edge of drone camera view or by rectification artifacts near the image border. Additionally, the very long-tailed distribution of Lyft [19] and Waymo (17) vehicle, pedestrian, and bicycle speeds (exceeding 60m/s show remaining area of improvement for state-of-the-ar AV perception systems and autolabeling pipelines. Comparisons of acceleration and jerk can be found in the appendix. Overall, from dynamic quantities alone. Waymo (17) and Lyft [9 provide the most diversity in agent motion. If such long-tailed data undesirable, the INTERACTION 33 dataset provides the most realistic set of vehicle speeds.
Trajectory Nonlinearity. To analyze the spatial diversity of agent trajectories, we first compare each agent's heading to their initial timestep. As can be seen in Fig- TO and reiterating earlier analyses, the vast majority of human movement is straight and linear (Ah 0). Moving away from the center, we also see repeated symmetric peaks at + (capturing left and right turns) and tkT in some datasets. One possible reason for these periodic peaks in the distribution is an artifact of the autolabeling methods used in the datasets (since only datasets that autolabel sensor data are affected), another is that their respective scene geometries contain more roundabouts, cul-de-sacs, and repeated turns than other datasets (more detailed heading distributions can be found in the appendix). We can also see that pedestrians' distributions are more uniform as they do not have to adhere to rigid road geometry.
Lastly, we also measure agent path efficiencies, defined as the ratio of the distance between trajectory endpoints to the trajectory length (25). Intuitively, the closer to 100%, the closer the trajectory is to straight line. As can be seen in Fig. L3 in the appendix, most path efficiency distributions are uniformly distributed, with peaks near 100%, echoing earlier straight-line findings. However, the INTERACTION 39 dataset is outlier in that its agent trajectories are predominantly straight lines with much less curved motion than other AV and pedestrian datasets.
Path Efficiency.
4.3 Annotation Quality
While analyzing datasets" true annotation accuracy would be best, neither we nor the original data annotators have access to the underlying real-world ground truth. As proxy, we instead analyze the self-consistency of annotations in the form of incidence rates of collisions between agents, off-road driving, and uncomfortable high-acccleration events (using 0.4g as a standard threshold 44 45 )
7

---

Unlonowss
Vohiclis
Pedestrians
Matorcycles
11500
1824
Eyn
M000
10
12
banset
INT.
WOMD
22
530
10
10
30
Figure 9: Agent speed distributions per dataset and agent type.
Unlonswns
Vahiccis
Padicitrinn
Bicyckis
Mestocycckss
1:50
2010
MOW0
10
WOND
10+1
36
Paggrtion
10-
201a
20-4
23-4
500
UCY
ETH
1070
500
20 0
26
40
21
41
21
201
4/1
4E
211
0
20
40
41 2n
25 40
Rel Heading Change (adland)
Rei Heading Change podian))
Rei Heoding Change (odlind)
Re. Heedtnggg Change (msclane))
Hwading Change (axdaan))
Figure 10: Changes in heading relative to agent's first timestep.
Virtually all observed agent data is free of collisions and off-road driving, save for rare one-offs (e.g., the INTERACTION dataset contains a minor car accident (3)). We denote bounding box intersections between agents as collisions, and agent center-of-mass exiting the road boundary as off-road driving. Collisions typically indicate errors in bounding box annotations, whereas off-road driving can indicate erroneous bounding box dimensions, missing map coverage, or harsh driving that, e.g., cuts corners during right turn.
As can be seen in Fig. I1 (left), most vehicles in datasets experience collision rates below 5%. Of particular note is the fact that state-of-the-an autolabeling systems (e-g., used in Waymo [17 nearly matching the accuracy of human annotations (e.g., used in nuscenes (18) in terms of resulting collision rates. However, detecting agents from near-ground perspective (even with 3D LiDAR) is very challenging task, and current performance still lags behind high altitude viewpoints. In particular, The INTERACTION 39) dataset achieves orders of magnitude lower vehicle collision, off-road, and harsh acceleration rates owing to its drone-based data collection strategy. In theory, SDD 40 should enjoy similar advantage, but only provides axis-aligned bounding box annotations (which overestimate agent extents) and Stanford's 's college campus contains much more interactive agents than other urban environments. More generally, the notion of bounding box intersections as collisions does not transfer exactly to pedestrians as they can enter/exit cars and walk in close groups, and further study is needed to robustly distinguish between errant motion and normal interactive motion.
In Fig. 11 (middle), we find that vehicles in general experience very few (< 1%) harsh acceleration events, with Waymo [17] Lyft [19]. and nuScenes [1]] all having the highest incidence, commen- surate with their earlier-discusse long-tail acceleration distributions. Lastly, we find in Fig. I (right) that the INTERACTION maps, whereas nuScenes [8] suffers from poor map coverage away from main roads (there are many annotated parked cars next to the main road) and Lyft [19] suffers from high false positive detections next to the main road (the majority of which take the Unknown class).
3 and nuPlan
2 agent annotations are well-aligned onto their
5 Conclusions and Recommendations
The recent releases of large-scale human trajectory datasets have significantly accelerated the field of AV research. However, their unique data formats and custom developer APIs have complicated multi- dataset research efforts (e.g., (20 21). In this work, we present traj trajdata, a unified trajectory data loader that aims to harmonize data formats, standardize data access APIs, and simplify the process of using multiple AV datasets within the AV research community with a simple, uniform, and efficient data representation and development API. We used trajdata to comprehensively compare existing trajectory datasets, finding that, in terms of annotation self-consistency, drone-based data collection methods yield significantly more accurate birds-eye view bounding box annotations than even state-of-the- tht-h---tttttttt AV perception stacks with LiDAR (albeit with much less spatial coverage), modern
8

---

WOUD
PCORSIM
MCTOBONCI
Figure II: Self-consistency failure rates per dataset and agent type, in the form of collision (left), high vehicle acceleration (middle), and off-road (right) rates.
autolabeling pipelines are nearing human annotation performance, and smaller-scale pedestrian datasets can still be useful for investigations requiring high-agent-density scenarios.
As concrete recommendations, we saw that some datasets artificially limit the distance agents autolabeled. Instead, it would be more useful to the long-range detection community to remove such restrictions, but add autolabeler-output uncertainties to long-range detections, supporting uncertain perception research along the way. Further, incorporating explicit self self-consistency checks within autolabeling pipelines and catching, e.g., collisions, prior to release can both improve the autolabeling method as well as the resulting data labels.
More broadly, providing researchers with access more data comprised of various agent types from diverse geographies should help in modeling rare agent types and behaviors, in addition to aiding the generalization of methods to multiple geographies. However, we have seen in prior sections, there is an overwhelming bias towards straight line driving, and one capability missing from trajdata is the ability (re)balance data on semantic (behavioral) level. Finally, even lower-level trajectory classes (e.g., driving straight, turning left/right. slowing down, speeding up, etc) are balanced, an important higher-level consideration during original dataset curation time is to ensure that AV datasets explore all geographic regions within an environment, and not only those of certain socioeconomic statuses or transportation access.
Future work will address the current limitations of trajdatt (e.g., expanding the number of sup- ported datasets and new capabilities such as geometric map element associations support Waymo- like map formats (17)). Further, incorporating sensor data would also enable perception research as well as joint prapptioon-rediiiio-plaaaaaae research, exciting emerging AV research field.
Acknowledgments and Disclosure of Funding
We thank all past and present members of the NVIDIA Autonomous Vehicle Research Group for their code contributions to trajdata and feedback after using it in projects. We additionally thank Leon De Andrade, Alex Naumann, and Stepan Konev for their contributions trajdata on GitHub.
References
[1] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras, "Human motion trajectory prediction: A survey," Int. Journal of Robotics Research, vol. 39, 8, pp. 895-935, 2020.
[2] General Motors, "Self-driving safety report," 2018, Available at https://www.gm..com//mnnnntt
[3] Uber Advanced Technologies Group, "A principled approach to safety," 2020, Available at
[4] Lyft, "Self-driving
safety report," 2020.
Available
at https://
Report 2020.pdfi
[5] Waymo, "Safety report," Waymo LLC,, 2021, Available hutps:/wwaymo.com/sseeyy safety-report

---

[6]
Argo AI, "Developing self-driving system you can trust." Apr. 2021, Available at hitps:
[7]
Motional, sa BOO0gUCUNNNSSS//RR////////CCOOO
safety self-esssssssssssseett"""" 2021, Available at https://dive.ggggg..cooo//////ea
[8] Zoox, "Safety report volume 2.0," 2021, Available at https://zoox.com/ssttyy
[9]
NVIDIA, "Self-driving safety report." " 2021, Available at altts://mmggssnnddi.cmmmmomomo
self deffddriving-ceee caars//afety--rpoo//auur prittt-ppiittt-ttteee hate
[10] T. Kruse, A. K. Pandey, R. Alami, and A. Kirsch, "Human-aware robot navigation: A survey." Robotics and Autonomouy Systems, vol. 61, no. 12, pp. 1726-1743, 2013.
[11] S. F. Chik, C. F. Yeong, E. L. M. Su, T. Y. Lim, Y. Subramaniam, and P. J. H. Chin, "A review of social-aware navigation frameworks for service robot in dynamic human environments." " Journal of Telecommunication, Electronic and Computer Engineering, vol. 8, no. 11. pp. 41-50, 2016.
[12] A. Lasota, T. Fong, and A. Shah, "A survey of methods for safe human-robot interaction." Foundations and Trends in Robotics, vol. 5, no. 4, pp. 261-349, 2017.
[13] nu nuTonomy, "nuscenes prediction challenge," https//wwww.masseee...///mliciii
2020.
[14] Lyft Level 5, "Lyft motion prediction for autonomous vehicles,"
https//wwww.aggllc.oo
competitions/llt-mooiiii prtees/lltttmmmiiiiorrcccioooooo putdicciioo-uuutooooouuuuuuuuu -utonooous--ehiccleee vehicles, 2020.
[15] Waymo, "Waymo open dataset motion prediction challenge," https:/wwaymo.coommppnn challenges/ 2021.
[16] Yandex Research, "Shifts challenge: Robustness and uncertainty under real-world distributional shift," https://rssearchyandddes www.ercccccyacc..ccocmmm yawtsecoooo...oo.itttttt... 2021.
[17] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and Anguelov, "Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset," in IEEE Int. Conf. on Computer Vision, 2021.
[18] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuScenes: A multimodal dataset for autonomous driving." IEEE Conf. on Computer Vision and Pattern Recognition, 2020.
[19] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, A. Jain, S. Omari, V. Iglovikov, and Ondruska, "One thousand and one hours: Self-driving motion prediction dataset," in Conf. on Robot Learning. 2020.
[20] T. Gilles. S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde, "Uncertainty estimation for cross-dataset performance in trajectory prediction," in IEEE Int. Conf. on Robotics and Automation Workshop on Fresh Perspectives on the Future of Autonomous Driving. 2022.
[21] B. Ivanovic, J. Harrison, and M. Pavone, "Expanding the deployment envelope of behavior prediction via adaptive meta-learning." in IEEE Ini. Conf. on Robotics and Automation, 2023.
[22] S. Pellegrini, A. Ess. K. Schindler, and L. v. Gool, "You"l 'Il never walk alone: Modeling social behavior for multi-target tracking." in IEEE Int. Conf. on Computer Vision, 2009.
[23] A. Lerner, Y. Chrysanthou, and D. Lischinski, "Crowds by example," Computer Graphics Forum, vol. 26, no. 3, pp. 655-664, 2007.
[24] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, "Social LSTM: Human trajectory prediction in crowded spaces," " IEEE Conf. on Computer Vision and Pattern Recognition, 2016.
10

---

[25] J. Amirian, B. Zhang, F. V. Castro, J. J. Baldelomar. J.-B. Hayet, and J. Pettré, "OpenTraj: As- sessing prediction complexity in human trajectories datasets," " in Asian Conference on Computer Vision, 2020.
[26] A. Malinin, N. Band, Y. Gal, M. Gales, A. Ganshin, G. Chesnokov, A. Noskov, A. Ploskonosov, L. Prokhorenkova, I. Provilkov, V. Raina, V. Raina, D. Roginskiy, M. Shmatova, P. Tigas, and B. Yangel, "Shifts: A dataset of real distributional shift across multiple large-scale tasks," in Conf. on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[Online]. Available: https://peaarrvooowooo/o/mmmmdmd
[27] H. Cacsar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, "nuPlan: A closed- ML-based planning benchmark for autonomous vehicles," 2021, Available at https://axxvvorr/ab//22000 11810
[28] P. Kothari, S. Kreiss, and A. Alahi, "Human trajectory forecasting in crowds: A deep learning perspective," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 7386-7400, 2022.
[29] A. Rudenko, L. Palmieri, W. Huang, A. J. Lilienthal, and K. O. Arras, "The atlas benchmark: An automated evaluation framework for human motion prediction," in IEEF Int. Conf. on Robot and Human Interactive Communication, 2022.
[30] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vito,, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, "Automatic differentiation in PyTorch," in Conf. Neural Information Processing Systems - Autodiff Workshop. 2017.
[31] D. Rempe, Z. Luo, X. B. Peng. Y. Yuan, K. Kitani, K. Kreis, S. Fidler, and O. Litany, "Trace and Pace: Controllable pedestrian animation via guided trajectory diffusion," " in IEEE Conf. on Computer Vision and Pattern Recognition, 2023.
[32] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone, "BITS: Bi-level imitation for traffic simulation." in IEEE Int. Conf. on Robolics and Automation, 2023.
[33] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone, "Guided conditional diffusion for controllable traffic simulation," IEEE Int. Conf. on Robotics and Automation, 2023.
[34] Christianos, Karkus, B. Ivanovic, S. V. Albrecht, and M. Pavone, "Planning with occluded traffic agents using bi-level variational occlusion models," in IEEE Int. Conf. on Robotics and Automation, 2023.
[35] Y. Chen, P. Karkus, B. Ivanovic, X. Weng, and M. Pavone, "Tree-structured policy planning with learned behavior models," in IEEE Int. Conf. on Roboties and Automation, 2023.
[36] The Apache Software Foundation, "Apache arrow," 2023, Available https://github.com/ apache/arrow
[37] S. Gillies, C. van der Wel, J. Van den Bossche, M. W. Taves, J. Amott, B. C. Ward, and others, "Shapely," " 2023, Available at
[38] Google Inc., "Protocol buffers- google's data interchange format," 2023, Available at https: ggithub.ccm/ppooooobbfff///oooop
[39] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kümmerle, H. Königshof, C. Stiller, A. de La Fortelle, and M. Tomizuka, "INTERACTION Dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps." 2019, Available https://aki..og///////111..
[40] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, "Learning social etiquette: Human trajectory prediction in crowded scenes," in European Conf. on Computer Vision, 2016.
[41] G. Brockman, V. Cheung, I.. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. (2016) OpenAl Gym. Available at bttpps//aaxii.orrr/////66666..
11

---

[42] Bureau of Transportation Statistics, "National Transportation Statistics, Number U.S. Aircraft, Vehicles, Vessels, and Other Conveyances, U.S. Dept. of Transportation. Tech. Rep., 2023.
Dept.
[43] E. Tolstaya, R. Mahjourian, C. Downey, B. Varadarajan, B. Sapp, and D. Anguelov, "Identifying driver interactions Vía conditional behavior prediction," in IEEE Int. Conf. Robotics and Automation, 2021.
[44] B. G. Simons-Morton, J. D. Ouimet, Wang. S. G. Klauer, S. E. Lee, and A. Dingus, "Hard braking events among novice teenage drivers by passenger characteristics," Driving Assessment Conference, vol. 5, pp. 236-242, 6 2009. [Online]. Available:
htts::/ppbs..i hutps//pubs.lib.uoww..aee htw.://wwwwwwwwuuutttiiirrrrrrrr
[45] S. G. Klauer, Dingus, V. L. Neale, D. Sudweeks, and D. Ramsey, "Comparing real-world behaviors of drivers with high versus low rates of crashes and near-crashes," National Highway Traffic Safety Administration, Tech, Rep. DOT HS 811 091, 2009.
12


Reviews:
[{"review_id": "at6DW3fM9m", "reviewer": "Reviewer_65pW", "summary": "", "questions": "", "limitations": "The authors did not state the limitations of this work.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Multiple datasets are integrated into one unified format, enhancing diversity in use cases and training scenarios.\n\n- A large volume of data has been compiled from 8 diverse datasets, including over three thousand data points and 200 million agents.\n\n- Evaluations of existing datasets provide valuable insights for practitioners in this field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "WZgykCULif", "reviewer": "Reviewer_LYJj", "summary": "", "questions": "", "limitations": "Within the scope of the manuscript, I cannot see any serious limitations. However, the authors were well-positioned to extend the data-loader to an evaluation testbed (e.g., closed-loop with nonreactive agents) and evaluation metrics for model predictions (e.g., ADE, heading error, collisions). While it is possible for others to create their own testbeds on top of the proposed dataloader, the standardization of an evaluation methodology would ensure that practitioners not only have access to the all of the same data but also the same evaluation method.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors have made it significantly easier to train forecasting models on 8 well-studied datasets, which will be appreciated by many researchers.\nThey have also provided helpful insights on how the datasets differ, which can aid in both debugging and analyzing prediction results on these datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xznJQqP2Zw", "reviewer": "Reviewer_8txH", "summary": "", "questions": "", "limitations": "The authors should expedite their plans to incorporate sensor data into their tool. This would broaden its utility and relevance to the wider AV research community, especially those working in perception research.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Significance of the Contribution.The unified data loader \"trajdata\" is a significant contribution as it simplifies and standardizes the process of using and comparing multiple AV datasets. This tool can expedite research by eliminating the complexities associated with managing and analyzing datasets in different formats and using different APIs.\nRelevance to the Broader Research Community. The research is highly relevant to the autonomous vehicle (AV) research community, as it offers a comprehensive comparison of existing trajectory datasets. Such an analysis aids in identifying strengths and weaknesses of different data collection and annotation methods and can guide future data collection efforts.\nQuality of the Research.The research is thorough, offering a comprehensive analysis of various datasets, and providing clear recommendations for future improvements. It also proposes potential future enhancements for the tool, indicating a commitment to continuous development and improvement.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "OHIkdTSHwY", "reviewer": "Reviewer_fsaD", "summary": "", "questions": "", "limitations": "In this submission, the authors have not shown the experiment results of existing methods testing on the proposed trajdata interface. Since trajdata involves some modification of the original data formats from the existing datasets, it would be necessary to make sure the validity of those modifications/transformations by testing existing methods on the modified data.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The introduction of Trajdata will make the researchers who work on trajectory forecasting tasks focus more on the algorithm/system design rather than dataset processing, given the following reasons:\n1. It standardizes the trajectory data representation and the HD map, which used to be very different for different datasets. \n2. The functions it supports are necessary and valuable. For example, it promotes multi-dataset benchmarking and agent-centric batch & scene-centric batch generation, which all highly align with the current research interests of the community.\n3. The simulation interface is also well-motivated since data shortage is always one of the biggest problems in the trajectory forecasting community.\n\nMeanwhile, the analysis of the existing datasets is meaningful. It provides in-depth statistic analysis from several perspectives, including the diversity of the agent types, the diversity of the trajecotires, and the agent density. This reviewer can tell the authors are very familiar with the topic and understand what information is that the researchers need when designing the systems. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors have made training forecasting models on eight well-studied datasets significantly easier. The authors present a unified interface for various human and vehicle trajectory datasets, such as ETH/UCY, nuScenes, Waymo, and L5Kit (8 total datasets). The authors also evaluate the datasets, comparing them across different metrics.
Decision: Accept (Poster)