Data-Driven Traffic Simulation for an Intersection in a Metropolis

Authors: chengbo  zang
kerem  turkcan
gil  zussman
javad  ghaderi
zoran  kostic
Venue: thecvf_com_CVPR_2024_Workshop_POETS
Published: 2024-04-13 19:01:59

Abstract: We present a novel data-driven simulation environment for modeling traffic in metropolitan street intersections. Using real-world tracking data collected over an extended period of time, we train trajectory forecasting models to learn agent interactions and environmental constraints that are difficult to capture conventionally. Trajectories of new agents are first coarsely generated by sampling from the spatial and temporal generative distributions, then refined using state-of-the-art trajectory forecasting models. The simulation can run either autonomously, or under explicit human control conditioned on the generative distributions. We present the experiments for a variety of model configurations. Under an iterative prediction scheme, the way-point-supervised TrajNet++ model obtained $0.36$ Final Displacement Error (FDE) in $20$ FPS on an NVIDIA A100 GPU.

Main paper: 
 introduction: 
 Accurate modeling and reconstruction of traffic flows in simulation environments is important for solving transportation problems in modern cities [10] . Simulation of traffic trajectories within intersections of a metropolis involves consideration of realistic car movements, human decisions and interactions, environmental constraints, and various forms of social regulations.Conventional simulation systems are often built bottomup where the state space, rules of interactions, and policies are unambiguously defined beforehand. This can be challenging given the complex nature of real-world applications. Moreover, most existing simulation systems target traffic flow control and optimization, while lacking realistic fine-grained details of interactions between traffic participants. It is also quite challenging for such systems to model human decisions, where the behavior of each agent can be spontaneous, or affected by other agents as well as environmental constraints in the scene.To address these challenges, this study uses a data-driven approach leveraging the data acquired from a real traffic intersection situated in a busy urban environment. We utilize statistical priors and deep-learning-based trajectory forecasting models to capture the complex dynamics of traffic participants in real-world scenarios. 
related work: 
 Traffic Simulation. Conventional traffic modeling methods evolved largely from statistical physics [7] . These methods require heavy simplification assumptions and precise rule definitions. Modern approaches involve Deep Rein-forcement Learning [21] , evolutionary algorithms [24] , or other state-space models [2] . However, these techniques often struggle in real-world scenarios due to the intractable size of states and policies. Most simulation systems focus on vehicle flows and exclude the role of pedestrians. More recent work includes the modeling of vehicle-pedestrian interaction such as Social Force Model [4] , which has been adopted for the prediction of pedestrian motions [15, 33] .Trajectory Forecasting. Deep Neural Networks (DNNs) are used for predicting future motions of pedestrians and vehicles [28] . The key architectural component is often a sequential model (e.g. Recurrent Neural Network or Transformer) which autoregressively generates future predictions based on past observations [3, 12] . Some models take a generative approach and predict the embeddings of future trajectories from latent distributions to account for varying data patterns or noise using Generative Adversarial Networks (GANs) [14, 19] , Generative Adversarial Imitation Learning (GAIL) [6, 8] or Conditional Variational Auto-Encoders (CVAEs) [5, 23, 30, 33] . Specially designed modules are introduced when modeling interactions in multi-agent scenarios by pooling [3, 13] , attention operation [12, 29] , or Graph Neural Networks [20, 23, 30] . Many architectures choose to incorporate auxiliary supervision using coarse way-points [9, 22] or final destinations [17, 31] of agent trajectories to boost model performances. 
data collection: 
 We utilize a high-elevation camera overlooking a metropolis intersection. We fine-tuned a YOLOv8 object detection model [16] for pedestrians and vehicles, then collected real-world trajectory data under the tracking-by-detection paradigm featuring the BoT-SORT algorithm [1] .To underline the entry and exit locations for each agent for statistical analysis, we pre-processed the collected data by filtering out the trajectories that unexpectedly terminate in the middle of the intersection (due to occlusions or failure of the detection-tracking models). The filtered trajectories were then uniformly resampled to align at 30 FPS. Fig. 1a shows several processed trajectories overlaid on top of each other. Details about the dataset are described in Sec. 4.1.The distributions of pedestrian and vehicle trajectories exhibit clear dependencies both spatially and temporally (Fig. 1b and Fig. 2 ). It is intuitive to model them using conditional generative models, where the new agents would be generated by sampling from the distributions during simulation. At this stage of the study, we adopted Gaussian Mixture Model (GMM) for this purpose. We will explore Figure 2 . Distribution of agent densities over 24 hours. The bars show the collected number of agents while the dashed lines delineate the fitted pedestrian and vehicle frequencies, respectively. The x-axis is the ToD shifted to begin at 8:00 and end at 7:00 the next day, and the y-axis is the hourly average pedestrian and vehicle counts in the entire dataset. models such as conditional GANs [25] or CVAEs [9] in future studies.Temporal Agent Density. Fig. 2 gives the distribution of agent densities traveling through the intersection over different time-of-day (ToD). We assume that the ToD when agents enter the intersection is centered around a few peak hours (e.g. getting to work during the daylight or returning home at nighttime) and fit the mean pedestrian and vehicle densities using two GMMs (with 4 components for pedestrians and 3 components for vehicles, values determined by experiments). We denote their time-dependent distribution byEQUATIONwhere N t is the total number of agents at time t.In the case of urban intersections, the agent trajectories are generally more confined to follow specific patterns dictated by the layout of intersections, traffic rules, social regulations, and environmental constraints [11] . We propose to characterize the trajectory of each agent by: 1) the position and velocity at the point of entry into the intersection x(0), x ′ (0) ∈ R 2 ; 2) the position and velocity at its exit from the intersection x(T ), x ′ (T ) ∈ R 2 ; 3) the total time elapsed T ∈ R between its entry and exit; and 4) |K| = 20 way-points sampled evenly along the trajectory x(K) ∈ R 2|K| , with sampling time T /K. Thus a vectorized representation of each agent can be given by z = x(0), x ′ (0), x(T ), x ′ (T ), x(K), T ∈ R 2|K|+9 . We model the distribution of different types of trajectories us-ing a GMM with M = 12 components fitted respectively for pedestrians and vehicles, denoted asEQUATIONwhere N (µ, Σ) is a multivariate Gaussian and w m is the weight of component m. Examples of categorized trajectories from some GMM components are illustrated in Fig. 1b . Pedestrians and vehicles from six different components are plotted in different colors. for i ← 1 to N do 3: z (i) C ∼ p gmm (• | C) ▷ Agent sampling 4:x(i) pr ← SPLINE(z (i) C ) ▷ Resampling 5:end for 6:return x(1:N ) prThe algorithm for the generation of new agents during simulations is illustrated in Algorithm 1. We start by sampling pedestrians and vehicles from their corresponding GMMs. Auxiliary conditions can be provided to insert more control into the sampling process. For example, if one wishes to sample agents from specific GMM components (i.e. pedestrians or vehicles going in specific directions) in some set C, then the GMM can be modified asEQUATIONwhere ŵm = w m / n∈C w n is the adjusted component weight. This is exemplified in Sec. 4.2.The sampled z can serve as good priors that provide high-level control over agent motions. However, this is not fine-grained enough due to the basic limitation that GMMs take no consideration of agent interactions or other environmental constraints. Sec. 3.4 describes a deep-learningbased refinement approach.Note that the sampling times of the way-points T (i) /K are not uniform across different agents because their trajectories may have drastically different time elapsed T (i) . Given that z also contains the position and velocity at both ends, we fit the trajectory with Cubic Splines [32] , obtaining a piece-wise interpolating polynomial with time-continuous acceleration. We then evaluate the polynomial with a fixed time interval ∆t = 0.4s (2.5 FPS), obtaining a prior trajectory x pr as inputs to deep-learning-based trajectory forecasting models.Fig. 1c illustrates the prior trajectories of several pedestrians and vehicles. Their way-points are generated from the GMM components shown in Fig. 1b , followed by interpolation as described above.To model agent interactions and other latent patterns in their motions, we adopt the TrajNet++ model [17] , a DNN featuring an LSTM and a grid-based pooling module that deals with agent interactions. The model takes L ob = 8 steps (3.2s) of past observations to predict L pd = 12 steps (4.8s) into the future. The model operates in a goal-supervised manner, i.e. the agent positions at the end of the prediction window are also provided to the model as auxiliary inputs. The choice of L ob and L pd in our dataset follows from public benchmarks [18, 26, 27] . Sec. 5 presents more experiments comparing different L pd -s.At each time-step t, we combine L ob steps of previous trajectories from all agents in the scene as x ob := x(t ob ) (we use the sampled x pr in case of newly generated agents with no past predictions), along with the temporal target locations (i.e. goals) of trajectories taken from x pr at the end of the prediction window x tg := x pr (t + L pd ∆t) as model inputs. The model then predictsEQUATIONwithEQUATION)The model iteratively takes previous predictions as inputs while being supervised by temporal target locations taken from the priors. Fig. 1d shows the agent trajectories refined from Fig. 1c . We also explored several other model architectures and supervision schemes in Sec. 5.The simulation algorithm is summarized in Algorithm 2. It maintains a set of active agents A ac . At each iteration, we (i) obtain the expected total number of agents N t from p tod , (ii) generate prior trajectories of new agents x pr from p gmm accordingly, and (iii) add them into A ac . Then the observations x ob and target locations x tg of all agents are sliced from A ac (as in Eq. ( 4)) to construct DNN inputs and generate refined trajectories x pd , which are then concatenated to the historical data in A ac . Any agent whose current position x (i) t := x (i) (t) reaches its expected destination x (i)T := x (i)pr (T (i) ) will be considered to have exited the intersection and removed from A ac . N t ∼ p tod (• | t) 5: if |A ac | < N t then 6: N ← N t -|A ac | 7:x pr ← PRIORGEN(N, C)A ac ← A ac ∪ x pr 9:end if 10:x ob , x tg ← SLICE(t, A ac , L ob , L pd )11:x pd ← DNN(x ob , x tg ) ▷ DNN refinement 12:for x (i) in A ac do 13:x (i) ← CONCAT(x (i) , x (i) pd ) 14: if x (i) t -x (i) T < ϵ then ▷ Check status 15:A ac ← A ac \ x (i)16:end if17:end for 18: end loop 4. Experiments 
dataset and evaluations: 
 The collected data from the intersection were organized to fit different purposes. For object detection and tracking, 13k annotated images were collected sporadically over 5 years from a high-elevation camera overlooking the intersection. The fine-tuned YOLOv8 obtained 91.6 mAP for pedestrians and 98.7 mAP for vehicles, respectively.For trajectory forecasting, tracked objects were collected for over 30 days containing time and bounding-box locations for 510k pedestrians and 250k vehicles. We uniformly sample 10k of 20-frame (8s) scenes and 10k of 40frame (16s) scenes for trajectory forecasting model training and evaluation. Additionally, complete trajectories of 176k pedestrians and 215k vehicles were extracted from the collected data for the statistical analysis described in Sec. 3.2.We trained trajectory forecasting models on the 20-frame scenes (with L ob = 8, L pd = 12) using smooth-L1 loss and adopted common performance metrics of Average-Displacement-Error (ADE, the RMSE between the predictions and ground-truths over all agents at all time-steps) and Final-Displacement-Error (FDE, the RMSE over all agents evaluated only at the last time-step of the prediction window) [17] . The models were trained on the standard scenes and evaluated in an iterative prediction scheme (described in Sec. 3.4) The proposed simulation system can coarsely control agent trajectories with Eq. ( 3) in terms of where they enter and exit the intersection as well as a prior trajectory to follow. In Fig. 3a we purposefully sample south-bound pedestrians and left-turning vehicles whose prior trajectories meet at the middle of the crosswalk (the red ellipse), to see whether the trajectory forecasting model will correctly react to this situation.As illustrated in Fig. 3b , the trajectory forecasting model forces both the vehicle and the crowd of pedestrians to slow down and deviate from their prior trajectories (denoted by the dashed red lines) to avoid a collision. This is a common practice that respects social norms and is expected to be observed in real-world scenarios.Without inserting auxiliary conditions or other human control, the simulator is able to run autonomously and mimic different agent densities following Eq. ( 1) and spatial locations following Eq. (2). 
outliers: 
 Adopting conditional generative models for trajectory categorization allows for the identification of outliers in the collected trajectories by calculating their likelihoods (Eq. ( 2)). In Fig. 3c , we show the outliers in pedestrians whose loglikelihoods are more than 20 times of standard deviations away from the dataset mean. Some of these outliers show a pedestrian making a turn-around; others show a pedestrian staying still in one location for an exceptionally long period of time. Beyond the standard metrics of ADE/FDE calculated over a predefined prediction window length L pd , the measurement of simulation quality requires more careful considerations.Here we provide a brief discussion on system and model configurations based on current results.Goal Supervision. We compared the refined trajectories from TrajNet++ before and after adding goal supervision (Fig. 3d vs. Figs. 3e and 3f ). Significant improvements in refinement quality can be observed, where agents deviate less from their prior trajectories without external forces. Relavant results are also quantified in Tab. 1.It is worth noting that the supervised models are often trained with fixed-length sequences and the agents are expected to reach their destinations in exactly L pd steps. This raises considerable issues in real-world deployment, since although it might not be difficult to know the destination of an agent (in our cases they are directly sampled), it can be challenging to know exactly when they will get there. Supervising the model with final destinations (Fig. 3e ) resulted in notable overshoot in cases when an agent needs a longer time window to reach the destination than L pd , while, in other cases, undershoot when they need a shorter window.This can be mitigated by substituting the destinations with way-points taken from their prior trajectories at L pd steps ahead, with results shown in Fig. 3f . Alternating to smooth-L1 loss instead of MSE also dampens the overshoot. Agent motions tend to be more controllable by the statistical model (i.e. p gmm ), while being able to react and make deviations when necessary (Sec. 4.2).Choice of L pd . Using the 40-frame (16s) scenes, we further compared the performances of predicting iteratively by training the model with L pd = 12 (4.8s) and taking previous outputs as new inputs, vs. directly training the model to predict in one-shot with L pd = 32 (12.8s). The results are given in Tab. 1. For TrajNet++, Wpts. denotes way-point supervision and Dest. destination supervision, as opposed to LSTM and Trajectron++ using no supervisions.By comparison, the destination-supervised TrajNet++ model achieved the lowest ADE of 0.59, while the waypoint supervised version had a higher ADE but the lowest FDE of 0.36. Higher FPS was generally obtained under larger L pd due to fewer operations beyond model inference (e.g. data preparation). Considering the aforementioned complexities of choosing appropriate L pd and applying destination supervision in practice, it is thus reasonable to use way-point supervision with shorter L pd = 12 during actual applications, giving our chosen model for the experiments in Sec. 4. 
conclusion and future work: 
 In this study, we propose a data-driven methodology for simulating the movement (trajectories) of agents within an intersection in a metropolis. We show that trajectory forecasting models are able to realistically govern agent motions under proper supervision by the statistical priors. The Tra-jNet++ model with way-point supervision was able strike a balance between the length of the prediction window and overall simulation quality by performing the predictions iteratively, achieving an FDE of 0.36 under controlled experiments. However, we note that the presented models were trained and evaluated within a single traffic intersection, raising reasonable concerns on potential overfitting since traffic conditions may vary drastically across different locations. More comprehensive evaluation is needed to address the issue.Future work will include (a) evaluation of alternative trajectory forecasting architectures and configurations, (b) incorporation of a larger number of intersections and more diverse traffic scenarios for better generalization, (c) exploration of other potential cases of agent interactions under controlled simulation, and (d) investigations on how to connect broader aspects of applications (e.g. collision alert, traffic light control, and more efficient deployment). We intend to incorporate the model with graphics engines where we can reconstruct the traffic scenarios of the intersection in the digital world. 
 
 Appendix:  


Decision: The reviews collectively appreciate the paper's innovative, data-driven approach to modeling traffic dynamics at urban intersections using real-world data. Strengths include the detailed methodology, practical relevance, and clear visual documentation. Weaknesses involve potential overfitting, the need for broader testing scenarios, and generalization concerns. Also, the discussion of the role of virtual humans in the traffic is asked to better match this workshop.
Overall, the paper is recommended for acceptance due to its significant potential impact on urban planning and traffic management, with suggestions for clearer explanations and an expanded literature review.

Reviews: review: title: title: The paper provides a data-driven traffic simulation by a series standard operations and provide insights for future work.
main_review: main_review: Overall, the paper has good illustration and interpretation, the project has practical relevance, and an innovative approach is designed to model complex urban traffic scenarios, making it a positive candidate for acceptance.

### Quality
1. The paper demonstrates a comprehensive methodology for traffic simulation in urban intersections. It utilizes real-world trajectory data for training trajectory forecasting models, and the method and training process are well-constructed with clear steps in data collection, processing, and model application. 

2. The paper is also well organized with a clear interpretation of the result and case analysis. It is a good paper for reading.

### Clarity

1. The paper's figures and tables effectively support the textual content, providing clear visual summaries of the methodology and results. 
2. However, some sections could benefit from more detailed explanations to avoid ambiguity: such as in table 1. what does the `Wpts` mean? 
3. Some relevant work is suggested to include to provide a more comprehensive survey, such as for trajectory forecasting, another branch of method named Generative Adversarial Imitation Learning (GAIL):  
[1] Choi S, Kim J, Yeo H. TrajGAIL: Generating urban vehicle trajectories using generative adversarial imitation learning[J]. Transportation Research Part C: Emerging Technologies, 2021, 128: 103091.  
[2] Da L, Wei H. CrowdGAIL: A spatiotemporal aware method for agent navigation[J]. Electronic Research Archive, 2022, 31(2). 


### Originality
The paper is of good originality.

### Significance

The significance of this work lies in its potential impact on urban planning, traffic management, and autonomous vehicle systems: 
1. It helps in providing a more accurate and realistic simulation of traffic dynamics at intersections, 
2. This research could help in designing better traffic control systems
3. This paper may have a potential influence in improving the safety and efficiency of urban transportation.
title: title: Review of submission 4
main_review: main_review: **Strengths**

- Novel Data-Driven Approach: The paper introduces a novel approach to simulate traffic at urban intersections using data-driven methods. The use of real-world data to train trajectory forecasting models is commendable, as it aims to reflect real-world complexities in traffic dynamics.
- Comprehensive Methodology: The methodology is thorough, encompassing data collection, statistical analysis, generation of prior trajectories, and trajectory refinement. The integration of Gaussian Mixture Models (GMMs) and deep-learning models like TrajNet++ for refining trajectories enhances the simulation's realism.
- Clear Visualization and Documentation: The paper includes detailed figures that illustrate the workflow and results effectively, enhancing understanding of the complex processes involved.
- Important potential use: the discussion of this work's future work of incorporating the model with graphics engines is a very important and valuable direction.

**Weaknesses**

- Topic of The Workshop. This work discusses the simulation of humans and traffic. However, the discussion of the role of humans in the traffic is missing. Will humans be important in this research direction? How will humans affect the traffic (an interesting multi-agents question)?
- Model Complexity and Computation Cost: The reliance on sophisticated models and high-end GPUs (like NVIDIA A100) might limit the accessibility of the proposed system for users with fewer computational resources.
- Generalization Concerns: While the simulation provides good results in a controlled environment, the paper does not thoroughly discuss how the system performs across various types of intersections or under different traffic conditions.
- Potential for Overfitting: Given the detailed customization of the models to specific datasets, there is a risk of overfitting. More robust validation using diverse external datasets could strengthen the findings.
title: title: Review
main_review: main_review: In this paper, the authors propose a pipeline for modeling and simulating traffic at street intersections. The trajectory models are trained using real-world tracking data. When simulating an agent, a coarse trajectory is first sampled from a GMM and then refined by an LSTM.

Strengths:

The authors expend significant effort in preparing real-world data for the data-driven method, including five years of data for object detection and tracking, and 30 days for trajectory forecasting.

The GMM + LSTM model is sensible, where the GMM can generate diverse coarse trajectories, while the LSTM is capable of modeling nonlinear interactions.

Based on the existing data and the methodological backbone, several interesting follow-up works are proposed in the future work section, such as collision avoidance and traffic light control.

Weaknesses:

The diversity of the testing scenarios could be improved. Currently, only one intersection is showcased, while traffic patterns could vary significantly in different places.

The presentation of this work could be enhanced if the authors provided some videos.

Summary:

This paper presents a data-driven traffic modeling and simulation pipeline based on their collected data. The entire pipeline appears reasonable with promising future work. The project could be made more comprehensive with a better variety of tasks and data sources.