Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels
Authors: Zhizheng Liu
Joe Lin
Wayne Wu
Bolei Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Pedestrian Movement Analysis
Human Motion Dataset
Human Motion Generation
Primary area: generative models

Abstract: Understanding and modeling pedestrian movements in the real world is crucial for applications like motion forecasting and scene simulation. Many factors influence pedestrian movements, such as scene context, individual characteristics, and goals, which are often ignored by the existing human generation methods. Web videos contain natural pedestrian behavior and rich motion context, but annotating them with pre-trained predictors leads to noisy labels. In this work, we propose learning diverse pedestrian movements from web videos. We first curate a large-scale dataset called CityWalkers that captures diverse real-world pedestrian movements in urban scenes. Then, based on CityWalkers,  we propose a generative model called PedGen for diverse pedestrian movement generation. PedGen introduces automatic label filtering to remove the low-quality labels and a mask embedding to train with partial labels. It also contains a novel context encoder that lifts the 2D scene context to 3D and can incorporate various context factors in generating realistic pedestrian movements in urban scenes. Experiments show that PedGen outperforms existing baseline methods for pedestrian movement generation by learning from noisy labels and incorporating the context factors. In addition, PedGen achieves zero-shot generalization in both real-world and simulated environments. The code, model, and data are available at https://genforce.github.io/PedGen/.

Published as a conference paper at ICLR 2025
LEARNING TO GENERATE DIVERSE PEDESTRIAN MOVEMENTS FROM WEB VIDEOS WITH NOISY LABELS
Zhizheng Liu, Joe Lin, Wayne Wu, Bolei Zhou
Department of Computer Science, University of California, Los Angeles
The Snesky
Figure 1: Pedestrian Movement Generation. Our method can generate diverse pedestrian movements in real-world (top row) and simulated (bottom row) urban environments.
ABSTRACT
Understanding and modeling pedestrian movements in the real world is crucial for applications like motion forecasting and scene simulation. Many factors influence pedestrian movements, such as scene context, individual characteristics, and goals, which often ignored by the existing human generation methods. Web videos contain natural pedestrian behavior and rich motion context, but annotating them with pre-trained trained predictors leads to noisy labels. In this work, we propose learning diverse pedestrian movements from web videos. We first curate a large-scale dataset called City Walkers that captures diverse real-world pedestrian movements in urban scenes. Then, based on City Walkers, we propose generative model called PedGen for diverse pedestrian movement generation. PedGen introduces automatic label filtering to remove the low-quality labels and mask embedding to train with partial labels. It also contains a novel context encoder that lifts the 2D scene context to 3D and incorporate various context factors in generating realistic pedestrian movements in urban scenes. Experiments show that PedGen outperforms existing baseline methods for pedestrian movement generation by learning from noisy labels and incorporating the context factors. In addition, PedGen achieves zero-shot generalization in both real-world and simulated environments. The code, model, and data are available at wtltttss:/ggenootttt github io io/PedGen/
INTRODUCTION
In bustling city, office workers rush through the crosswalk to nearby buildings, tourists wander around along storefronts looking items on display, and friends and enjoy coffee on the outdoor patio. Pedestrians essential participants in urban spaces. Their movements represent the social lives and interactions with the surrounding environments. Understanding and modeling pedestrian movements is critical many applications. For example, city designers simulate pedestrian move- ments to optimize public areas and transportation systems (Mehta) 2008); forecasting the future pedestrian path is crucial for the safe deployment of autonomous vehicles (Lyssenko et al. 202T).
Generating diverse and realistic pedestrian movements remains challenging. Multiple context factors affect pedestrian movements in the scenes. The first factor is the surrounding environment. As

---

Published as a conference paper at ICLR 2025
people constantly interact with the environment, it is important model the scene context where the interaction happens. For example, objects like trash bins, plants, and other pedestrians in the scene can influence the walking behavior of pedestrian (Daamen & Hoogendoorn 2003), while the space types like crosswalks and waiting zones decide the overall movement pattern (Sime) 1995). The second factor is the individual characteristics. Studies have shown that walking speed and gait influenced by age (Ostrosky et al. 1994), gender (Yamasaki et al. 1991), and body weight (Heglund & Taylor, 1988) and can also reflect the pedestrian's fitness level (Dridi 2013). The last factor the goal of the pedestrian, which decides the walking route in the scene (Hoogendoorn & Bovy 2004).
Existing human motion generation research mainly focuses on the breadth of activities (Mahmood et al. (019). but few have studied generating natural real-world pedestrian movements in diverse scene contexts, For example, existing outdoor human motion datasets have limited scenes and human subjects with unnatural motion performed by actors with specific instructions (Kaufmann et al. 2023 Dai et al. 2023). Moreover, current motion generation methods (Guo et al., 2020 Tevet et al., 2022) lean toward generating complex motions from clean MoCap data, where many motion categories are rare in daily scenes. Few have considered learning the diverse motion contexts from noisy labels.
Web videos, captured by many people walking and touring in different cities worldwide and shared on the YouTube website, contain diverse scene contexts and pedestrian movements in the most natural forms. However, labeling pedestrian movements in these web videos with pretrained predictors leads to inevitable label noise. How to harness the web data with large noise yet rich context becomes the pivot of modeling and generating diverse pedestrian movements. To this end, we first collect City Filkkrs a large-scale real-world dataset containing pedestrians in urban scenes, annotated with pseudo-labels by off-the-shelf 4D human motion estimation models. Walkess captures diverse real-world pedestrian movements regarding various moving speeds, gaits, headings, and local motions. Each movement is also paired with labels of context factors, such as the pedestrian' 's body shape. route destinations, and the environment's s semantics and geometry.
We then develop new diffusion-basee generative model PedGen for learning context-aware pedes- trian movements with the noisy pseudo-labels from the Wttyeas dataset. PedGen has two key designs: 1) To mitigate the anomaly and incomplete labels from pseudo-labeling techniques, PedGen adopts data iteration strategy to identify and remove low-quality labels from the dataset automati- cally and a motion mask embedding to train with partial labels; 2) To model the important context factors, PedGen considers the surrounding environment, the individual characteristics, and the goal points as input conditions to generate realistic and long-term pedestrian movements in urban scenes. As web videos only contain scene context labels in 2D), we propose a novel Context Encoder that can lift the environment context from 2D images into a 3D local scene representation with geometry and semantic information and also encode the other context factors help generate realistic and long-term 3D pedestrian movements. We show some randomly sampled results of PedGen in Fig. [
Experiment results on the CityWalken validation set, the real-world Waymo open dataset (Sun) et al. 2020) and CARLA simulator (Dosovitskiy et al. (017) show that PedGen can predict more realistic and accurate future pedestrian movements than existing human motion generation methods and achieve better zero-shot generalization by generating high-quality context-aware movements. Additional experiments and ablation studies demonstrate the effectiveness of PedGen addressing noisy labels and incorporating the key context factors. It enables the application of forecasting pedes- trian movements in the real world and populating simulated environments with realistic pedestrians. We summarize our contributions as follows: 1) A new task of context-aware pedestrian movement generation from web videos with unique challenges in dealing with label noises and modeling various motion contexts. 2) A new large-scale real-world pedestrian movement dataset City Walkers with pseudo-labels of diverse pedestrian movements and motion contexts. 3) The context-awan generative model PedGen that can learn from noisy pseudo-labels to generate diverse pedestrian movements.
2 RELATED WORK
Pedestrian Movement Analysis. Pedestrian behaviors have been extensively studied in transporta- tion and social science. A hierarchical structure is defined for pedestrian behavior analysis (Hoogen doom & Bovy] 2004 Feng et al.) (221)) from the high-level strategic behavior, the middle-level tactical behavior, to the low-level operational behavior. Pedestrian movements belong to operational behav- iors, where pedestrians continuously make short-term movement decisions their route to respond
2

---

Published as a conference paper at ICLR 2025
to their immediate environment (Daamen 2002 Duives 2016). Many works analyze different factors
influencing pedestrian movements (Dridi, 2015). Some critical environmental factors include types of space (Sime) 1995). objects in the environment (Daamen & Hoogendoorn 2003), and movement of other pedestrians (van den Berg. 2016). Pedestrian movements also depend on their biometric data, like age (Ostrosky al.) 1994). gender Yamasaki et 1991), and body size (Heglund & 'aaylor 1988). Different from most existing works in pedestrian movement analysis that collect data from field observations (Shields & Boyce 2000) or controlled experiments (Haghani & Sarvi 2018 and analyze them using statistical approaches (Tong & Bode (023), we extract pedestrian movements from web videos and learn a generative model to facilitate pedestrian movement modeling.
Human Motion Datasets. AMASS (Mahmood et al. 2019) is one of the most popular human
motion datasets with diverse motions annotated with SMPL (Loper et al. 2023) parameters. The main issue of AMASS is the lack of context information related to the motion. Later datasets have focused on adding more human subjects (Cheng et al. 2023b), text descriptions (Guo et al. 2022).
human-object interactions (Bhatnagar et al. (022) and human-scene interactions (Hassan et al. 2019
et al. 2022 Jiang et al.] (2024) to the labels. Nevertheless, most of these datasets are captured in a controlled environment with the subjects asked to follow specific action instructions, and many motions, such as jump jacks and martial arts, are rarely seen in urban scenes. In-the-wild videos serve as a more suitable source for studying human movement with their richness in individuals and environments. Another line of work (Von Marcard et al. 2018: Guzov et al. 2021 Dai et al., 2023 Kaufmann et al. 2023) focuses on collecting human motion in outdoor places from in-the-wild videos, but their additional sensor requirements like IMUs limit their scalability collect lcage datasets. [hhu et al.] (2021); Zheng et al.] (2022) propose benchmarks for gait recognition in the wild, but these datasets lack diverse scene contexts. Some other datasets (Vendrow et al., 2023 Robicquet et al.] 2016) label human social behaviors and trajectories from street videos as key points or bounding boxes. Still, these labels have worse motion granularity than the SMPL parameters for pedestrian movement analysis. Our proposed dataset, City Walkers, consists of large-scale web videos of pedestrians in diverse urban environments. City CilyWsskers provides both SMPL movement labels and context pseudo-labels, including the body shape of the pedestrians, their route destinations, and the semantics and geometry of the scene.
Human Motion Generation. Human motion generation has been accelerated by large-scale motion datasets and rapid advancements in generative models, with diffusion models (Ho et al.) (2200 being the most successful architecture for its high generation quality and multi-modal modeling capacity. To generate motion with more fine-grained control, various input conditions have been used, including
action labels Tevet al. 2022). ) texts (Zhang et al., 2024), audios (Dabral et al.] 2023). and history motions (Chen et al. (2023)). The most relevant motion generation models to ours are the ones that condition on the indoor scenes (Huang et al.) 2023 Yi al. 2024 Diang et al., 2024). However, there is a huge gap between indoor and outdoor environments, and generating pedestrian movement requires considering more context factors other than the surrounding environment, like route destinations and pedestrian characteristics. Some works (Tripathi et al. 20244] Xue & | propose to condition the motion on the body shape but do not consider the scene context. |eempe et al. (2023) animate pedestrian movements by generating high-level trajectories and then training a policy control the pedestrian movements in simulation. Nevertheless, their movement data for training the RL policy comes from subset of AMASS (Mahmood al.] 2019) with limited diversity. [Wang et al. (2024) can generate diverse pedestrian animations while following the given trajectory, but the motion is not learned from real-world pedestrians. [Shan et al. (2023) tackle pedestrian movement generation with rule-based approaches by combining path planning algorithms (Treuille et al., 2006) and manually designed animations, lacking the diversity and realism of real-world pedestrian movements. On the contrary, our proposed PedGen model learns to generate diverse and realistic pedestrian movements conditioned on the context factors from large-scale real-world data with noisy labels.
CAPTURING DIVERSE REAL-WORLD PEDESTRIAN MOVEMENTS
This section introduces our effort to capture real-world pedestrian movements from web videos. Existing human motion datasets rarely capture natural pedestrian movements, lack diversity in scenes and human subjects, and do not provide the critical context factors of pedestrian movements, such as surrounding environments, individual characteristics, and route destinations. To support the task
3

---

Published as a conference paper at ICLR 2025
Diverse Pedestrian Movements
leaning on stroler
taking out phone to take picture
Diverse Individual Characteristics
pointing wal
furnnn walking with hands behind back Diverse Surrounding Environments
moving items
while sitting
Diverse Route Destinations
CityWalker dataset. Top Left: The diverse pedestrian movements. Top Right: The diverse body shapes of the pedestrians. Bottom Left: The diverse surrounding environments from depth- unprojected images and the 4D pedestrian movement labels. Bottom Right: The diverse route destinations shown on the depth labels and semantic maps of the seene. The background showcases pedestrians in bustling cities from where we construct the CityWalker dataset.
Figure 2: Samples
of context-aware pedestrian movement generation, construct CityWalkers a large-scale dataset with real-world pedestrian movements in diverse urban environments annotated by pscudo-labeling techniques. Sec. B.T shows overview of CityWalkers, and Sec. 3.2 describes our data collection and annotation procedure. Please refer to the appendix for more details about City Walkers.
3.1 CITY DATASET
The City Walkers dataset is collected from YouTube's city tour videos. Our data source consists of high-quality web videos of walking in cities worldwide posted by content creators on YouTube, such as the YouTube channel POPtravel (Sczepansky 2024). We manually picked videos so they incorporate a wide spectrum of urban public places in different regions and cultures with various scene contexts and real-world pedestrian movements. We label each video's 's scene attributes, including weather, location, crowd density, and time of the day. using an off-the-shelf VLM (Chen al.) 2023b). We then detect (Jocher et al., (023) and track (Cheng et al.) 2023a) each pedestrian in the video, and label the pedestrian movements as SMPL parameters and scene context as depth and semantic segmentation maps. These labels contain the crucial context factors for learning pedestrian movements. total, CityWalker Walker contains 30.8 hours of high-quality videos, including 120,914 pedestrians and 16,215 scenes across 227 cities and 49 countries, making it the most diverse human motion dataset regarding scene context and human subjects. As shown in Fig. Fig.2] City Walkers contains variety of pedestrian movements, scene contexts, pedestrian body shapes, and route destinations.
3.2 DATA PROCESSING AND AUTOLABELING
We adopt WHAM (Shin al. 2023 recover 4D pedestrian movement pseudo-labels. Given tracked pedestrians and their body key points, WHAM outputs their global human motion with SMPL (Loper et al. 2023) parameters (tc. Φ,, θr, 3}} at every timestep t, E RR3 Φ. SO(3) are the root translation and orientation of the SMPL model, and 0, E SO(3) 3 E 10 are the body pose and shape parameters. We further use the state-of-the-a monocular depth estimation model ZeeDepth (Bhat et al.) 2023) and semantic segmentation model SegFormer (Xie et al.) (021) to automatically label scene depth and semantics. To filter low-quality pedestrian movement labels brought by model noise, we threshold the detection confidence score, human bounding box size, and keypoint confidence scores. We also discard the occluded motion by thresholding the number of human key points that within the 2D human segmentation mask. To further clean the labels and curate a more accurate evaluation dataset, we manually check the label quality and adjust the labels, such 2D key points, if necessary. We want to stress that though have tried best to improve
4

---

Published as a conference paper at ICLR 2025
the quality of pseudo labels by using state-of-the- models and filtering wrong predictions, label noise from web videos is still inevitable. We show in the experiments that learning from the noisy labels still benefits pedestrian movement generation. We also apply techniques in generation model to further mitigate the effect of label noise. A benchmark result of the accuracy of our data autolabeling pipeline is provided in the appendix.
The final label includes future pedestrian movements and their corresponding context factors. {ær = {(4,, Φr, 0t)3, y [Z, Id, G3, I tr]). where æ is the pedestrian movement label and y is the context label. Z, I. Ia are images, depth labels, and segmentation maps of the scene, 33 represents the pedestrian's personal characteristics, and è tr are the starting and goal positions.
4 GENERATING CONTEXT-AWARD PEDESTRIAN MOVEMENTS FROM NOISY LABELS
This section introduces our method, PedGen, to address the label noise and incorporate the context factors from the Walkers dataset. PedGen a diffusion- generative model and the first method for the new task of context-aware pedestrian movement generation. An overview of PedGen shown in Fig. 3 Sec. defines the task of context-aware pedestrian movement generation. We introduce the overall architecture of PedGen in Sec. 4.2 Sec. introduces two simple and effective strategies to deal with the low-quality and incomplete labels from web videos, respectively. As is crucial to model the context factors, including the surrounding environments, the pedestrian's 's characteristics, and the goal points during generation, we design a novel Context Encoder in Sec. 4.4
4.1 TASK DEFINITION
We define the task of context-aware pedestrian movement generation as follows: Given the initial 3D position t [ 3/1 21] of pedestrian and the 2D image of an urban scene, our goal is to generate the pedestrian's 's future movements as the SMPL. root translation 4, root orientation &., and body pose O,. The following context factors are also provided to help generation: 1). The urban scene context represented as the 2D image I, semantic mask I", and depth label Id. 2). The SMPL. human shape parameter B3, which is latent representation that can indicate pedestrian's 's characteristics, such as height, weight, and body shape. 3). The 3D goal position tr [ET, yr, 27] the pedestrian's 's route destination in the scene.
[t.. Φe. 0, 0,TT The movement at each timestep is represented
4.2 PEDGEN MODEL
PedGen follows the conditional diffusion framework (Ho et al. 2020 Ho & Salimans 2022). Given sampled movement æ from the dataset, the forward diffusion a Markov noising process (ar* k defined from the noise scheduling parameter { k==1- The goal is to train reverse denoising model a F(**, k c) that predicts the sampled movement given the noisy movement æ* the diffusion timestep k. and the condition factor e. Our denoising model architecture follows state-of-the-art transformer-based human motion generation models (Tevet et al., 2022 Tiang et al. 2024 Chen et al. 2023a), where movements at different timesteps are encoded as separate tokens in the transformer. We represent & velocity v, t; t,- and rotation Φ.. O, with 6D rotation (Zhou et al., 2019) representation. Different from existing approaches that encode all parts of the motion into a single token, we find it helpful treat the velocity and the rotation of the motion as different tokens in the transformer, they have different representations and scales in the data. Our loss function for training the denoising model is as follows:
Ok=0
f(x, Φ)- = ExE[I.K],, WWrehhree + Weraj Ltraj + Wgeo Cgeu),
where D is the training dataset. free = || 2112 is the standard diffusion reconstruction loss. Luraj ET f, tollı is the trajectory loss the reconstructed global translation f, Et Ú. and Egeo- = [==IIFK(()) - FK())) the geometric loss (Tevet et al. 2022) that computes the joint position eиror with forward kinematics the SMPL body pose. In ablation experiments, we will show the effectiveness of separating velocity and rotation tokens, as well as adding the trajectory and geometry losses.
5

---

Published as a conference paper at ICLR 2025
During inference, we start from the k K step by sampling 2x from the standard Gaussian distribution. We then pass it to the denoising model to predict the clean motion & and add noise again to obtain the noised sample & This process is repeated K times to get the final generated pedestrian movement á. Our model can also facilitate the efficient generation of infinitely long movements by concatenating the short motion intervals. Please refer to the appendix for more details.
4.3 ADDRESSING NOISY LABELS
Pseudo-labels from pre-trained predictors have inevitable noise. Unlike existing human motion generation approaches that focus on datasets with clean MoCap data, our PedGen model aims to address noisy labels from web videos. There are two sources of label noise in City Walkers. The first is the inherent compound noise from the data and models used for pseudo-labeling, leading to low-quality labels. As it is difficult examine the label quality of 4D human motion from only 2D videos, many anomaly labels remain in the dataset even after rule-based filtering and manual checks. We propose to automatically identify these low-quality labels using reconstruction- based unsupervised anomaly detection techniques (Livernoche et al. 2023 Wolleb et al. 2022). Specifically, first train a PedGen model without context the training data of City Walkess. We then partially add noise with half of the diffusion steps K/2 for each sample and denoise it using the trained model. We use the reconstruction error between the original and the denoised sample as a metric for anomaly labels and filter labels with errors greater than a certain threshold. We then iterate the above process by re-training the model with the remaining labels and filtering based on the new reconstruction error.
Another source of label noise is the discontinuous or lost tracks of pedestrians due to occlusions and missed detections. As a result, more than half of the labels are incomplete and only annotated on subset of frames. These labels provide partial supervision that can benefit learning more diverse context-aware pedestrian movements. To train with these partial labels, we replace the missing timesteps for these partial labels with a learnable motion mask embedding m to the denoising transformer. We first define a label mask M, where M, == indicates the label at timestep is missing. Then we add the mask embedding the original noisy sample as x x= (1 M) + M to replace the missing timesteps with the mask embedding m and feed to the network to output the denoised prediction & similar to Sec. 4.2 We then update the loss with the masked predictions and masked ground truth as L = L(x(M) ) de M)) so it only operates on the labels at the available timesteps. All losses will guide the training with the masked outputs.
4.4 CONTEXT ENCODER
Our context encoder module outputs condition embedding e from the provided context factors. To encode the scene context, we first unproject the 2D depth label Id and semantic map Ie into 3D point cloud P {p- [P2.Pr Py, Pz. Pel). where Pe is the semantic label. Next, extract points within a local neighborhood of the starting location t1, resulting in the local point cloud Plocal = (p E P |lp.. .21 || Az; Ilp,, y1 ll < Ay, llp= - 21|l < Az) and voxelize it into a 3D grid. The class label of each voxel is either empty or determined by majority voting of the points within the voxel. The voxel is then processed with a single cross-attention layer to get the scene context embedding. The scene encoding is then added with embeddings from other conditions, including the human shape B and the goal position tr, to get the final context embedding c.
As web videos only contain 2D images, a natural idea is to directly encode the scene context using the 2D image feature extractors (e.g., Dino-V2 V2 (Oquab et al.) (2023)) We find that our proposed context encoder can encode the scene context more effectively than encoding the scene context in 2D. Since pedestrian movement represented in the 3D space and generating it requires 3D information about the scene, it is easier to reason about the surrounding environment using 3D representation than 2D image features. Furthermore, it is hard to disentangle the pedestrian and its surrounding context from the 2D image features. As result, the model may exploit the ego pedestrian instead of the scene context to generate future movements. Our context encoder could address this issue by unprojecting the pixels into 3D point clouds and discarding point clouds belonging to the ego pedestrian. In our ablation experiments, we show the importance of using the 3D scene representation and appending the semantic labels in encoding the scene context.

---

Published as a conference paper at ICLR 2025
Data: CityWalkers
Model: PedGen
Generated Pedestrian Movement
Label Fiterng
Anomaly Labels
Context
Attention
Raw Labels
Labels
Context
MLP M_P
Embrrdding.
Labels
Body Shape P
Denoising Transformer
True Fnewwkiing
Masik Embeddinn
Goal
MLP PE
Velecity (o1) Rotanion ( Re..
Figure 3: Our method. We discard the anomaly labels with iterative automatic label filtering procedure and add the partial labels to training data. We then train PedGen with Context Encoder to represent crucial context factors. The scene context is obtained by lifting the 2D depth and semantic labels the 3D space and converting them into a local voxel representation. The encoded scene context is combined with other context factors, including the body shape and the goal to get the context embedding c. The context embedding c and the timestep embedding & are then used to guide the Denoising Transformer predict the clean motion from the noised one. We use learnable motion mask embedding m address the partial labels during training.
EXPERIMENTS
We compare the performance of PedGen to the other baselines on the real-world Citywalkers and Waymo datasets and in simulated CARLA environments in Sec. 3.11 Sec. 5.2 shows experiments
on the effect of training with noisy labels and different context factors. Sec. |53 demonstrates the ablation study of our data and model. The diverse pedestrian movements generated by PedGen shown in Fig. We provide additional experimental details in the appendix.
Datasets. We train PedGen on the proposed City Walkers dataset. For each pedestrian movement trajectory, we sample the initial timestep at an interval of 30 frames and keep at most the future 60 frames (2 seconds) as the ground truth movement. The training set has 104,192 samples, including 53,405 partial labels that have at least 30 frames of annotation. The validation set has 13,039 samples and only contains complete labels. We split the validation set to contain novel scenes with completely different locations and human subjects never
in the training set.
To verify the performance of PedGen on real-world pedestrian movement prediction with ground truth labels, we use the Waymo open dataset (Sun al., (2020) with -annottdd 3D human keypoint labels, which is only used for testing. Waymo is critical as it is the largest urban dataset that captures diverse pedestrian motions with sparse 3D keypoint labels 10 Hz. In total, selected 80 test samples that (1) spans 2 seconds and (2) includes at least 6 sparse human keypoint labels.
We also collect additional test set in the CARLA Simulator (Dosovitskiy et al. 2017) to demonstrate the application of PedGen in simulation. We sample initial locations, goal locations, and camera view angles in different maps and render the ground truth images, depth labels, and segmentation maps in simulation. We manually check that all the sampled locations are valid and that all the camera views are not occluded by obstacles, resulting in a simulated test set with 262 diverse samples.
Evaluation metrics. For experiments on the City Walkers and Waymo dataset, compare the generated pedestrian movement with the ground truth, and follow the metrics used in human motion prediction (Chen et al. 2023a) to compute Average Displacement Error (ADE) and Final Displace- ment Error (FDE) on the realism of the predicted motion. We generate 50 movements for each data sample and report both minimum and average ADEs (MADE, aADE) and FDEs (mFDE, aFDE) among all samples. For experiments on the simulated test set, we evaluate the context awareness and the physical plausibility of the generated motion. Our metrics include the Collision Rate (CR), which measures the ratio of the generated movement that collides with other objects in the environment, and the Foot Floating Rate (FFR), which measures the ratio of the generated movement whose feet are either floating or penetrating with the ground greater than given threshold (20cm).
Baselines. We compare our method with several recent human motion diffusion models, which have different input conditions and motion representations. Note that since context-aware pedestrian
7

---

Published as a conference paper at ICLR 2025
NOW
Figure 4:
Visualizations of the generated pedestrian movements. The top row shows results real scenes from the City Walkers dataset, the middle row shows results in the real-world Waymo test set, and the bottom row shows results in simulated scenes from the CARLA test set.
Table I: Comparison to baselines. consider two cases where the model is given the goal condition not. We evaluate on the validation set of City Walkers, the real-world Waymo test set, and the simulated CARLA test set. We evaluate our model (PedGen) compared with the other baselines.
Goal Condition
Method
Walkers
Waymo
CARLA
aFDE
IMADE
SADE
mFDE
aFDE
FFR
x
HumanMAC
4.67
1.86
3.19
5.29
10.36
2.5%
10.2%
x
MDM
1.33
4.55
1.93
8.41
3.03
5.35
5.66
10.60
2.1%
3.2%
x
PedGen
1.13
4.08
1.61
7.56
2.90
5.15
5.52
10.11
1.6%
2.6%
/
TRUMANS 0.73
1.26
0.56
1.13
2,01
2.37
1.41
1.94
0.6%
0.6%
/
2.18
0.0%
movement generation is a newly defined task, no models could directly support it; we thus made minimum adjustments to make them compatible with our problem setting. MDM (Tevet al. 2022) uses texts or action labels as the condition and proposes several geometric losses to improve motion quality. We did not use the foot contact loss proposed in their method as CityWalkers does not provide ground-truth foot contact labels. HumanMac (Chen el al., 2023a) conditions the future motion generation on the history motion sequences, which uses the DCT transform to ensure smooth motion generation. TRUMANS (Jiang et al.] (024) uses the initial human pose, the 2D BEV goal position, the indoor scene context, and the frame-wiss action labels as the input condition. We modify TRUMANS so it is compatible with our scene context and goal point conditions.
5.1 COMPARISON TO BASELINES
We compare PedGen with the other basclines on the validation set of CityWalkers the real-world Waymo test set, and the simulated CARLA test We separately evaluate whether the goal condition is given, as it is deterministic factor for pedestrian movement. The benchmarking results are shown in Tab. T It can be observed that PedGen outperforms other baseline models by clear margin on City Walkers. PedGen also achieves the best zero-shot generalization ability on Waymo and CARLA test sets, further showing the superiority of PedGen the other baseline.
5.2 EFFECT OF NOISY LABELS AND CONTEXT FACTORS
Table 2a demonstrates the effectiveness of PedGen in addressing noisy labels. We can see that removing the anomaly labels with the proposed automatic label filtering can help generate more realistic pedestrian movements and improve aADE by 2.9%. Moreover, adding the partial labels as additional training data can improve the aADE by 5.8%. highlighting the value of partial labels in web videos. Combining both strategies for noisy labels leads to the best performance of 4.08 aADE.
8

---

Published as a conference paper at ICLR 2025
a)
Context
With Context
With Context (Long- Term)
b)
Context
With Context
Context
Figure 5: Qualitative comparison of training with context factors. We compare the generated movements of PedGen trained with without context factors in real-world environments (a) and in simulation (b).
Table Results with noisy labels and context factors. We experiment the City CalyWs validation set and the CARLA test set and study effect training with noisy labels (a) and different context factors (b).
(a) Evaluation of training with noisy labels. We evaluate PedGen with context trained with or without anomaly and partial labels.
(b) Evaluation of the context factors. We evaluate PedGen condi- tioned each context factor, including the surrounding environ- ment (scene), the pedestrian's 's own characteristics (human), and the goal points (goal).
Anomaly Partial
City
Labels Labels
mADE
sADE mFDE aPDE 1
Idx
Context Factor
City Walkers
CARLA
x
1.17
4.45
1.64
8.31
MADE
aADE J mFDE
aFDE
JJCR
FHR
1.13
1.60
8.09
1
No
1,13
4.08
1.61
7.56
2.1%
5.2%
v
1.15
4.19
1.62
7.79
2
Scene
1.11
3.75
1.55
6.92
1.6%
2.6%
3
Human
1,09
3,24
1.61
5.95
1.9%
0.7%
Scene + Human
1.00
3.12
1.43
5.65
1.5%
0.3%
5
Goal
0.60
1.09
0.47
1.00
00.0%
0.0%
0.46
0.99
7
Human + Goal
0.55
1.01
0.43
0.93
0.0%
0.0%
8 Scene Human + Goal
0.54
0.96
0.43
0.91
0.0%
0.0%
We further examine the effect of each context factor on the pedestrian movement generation perfor- mance Tab. 2b) The comparisons between "setting vs. 2" and between "setting vs. 3" show that adding the surrounding environment and the pedestrian's 's own characteristics as conditions are helpful to pedestrian movement generation and can reduce the aADE by 6.7% and 19.4%, respectively. Setting demonstrates that incorporating both context factors achieves even better results. From settings and 5, we find that the goal points are the most important context factor for the final performance and can significantly reduce aADE by 72.9%. The comparison between "setting 5 vs. 6" and between "setting vs. 7" further proves that the other two context factors (scene and human) also help movement generation after the goal is provided. Finally, setting 8 demonstrates that using all three context factors leads to the smallest generation errors. From the results of the CARLA test set, we can see that the scene context can contribute both lower collision rate of 1.6% and lower foot floating rate of 2.6%, while the human context is more usefuu in reducing the foot floating rate only 0.7%. Adding both the scene and the human context can further improve the physical plausibility of the generated movements. Using the goal context is the most crucial factor and can reduce the failure rate 0. This further proves the effectiveness of each context factor.
Some qualitative comparison results in real-world environments are shown in Fig. 5 (a). We can observe that PedGen trained without context factors generate arbitrary movements that walk off the sidewalk. By conditioning the generation on the context factors, the movement becomes context- aware, and the model can further generate long-term pedestrian walking behaviors on the sidewalk. Visualization results in simulation are shown in Fig. (b). From the left two figures, without the context factors, the generated movements have the feet floating from the ground, while that PedGen trained without
see that
adding context factors fix this issue. From the right two figures, context generated a walking pose that is in collision with the bench, while it successfully generated a sitting pose the bench after considering the context factors.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation experiment results. We ablate on the training data of PedGen (a) and the PedGen model's key components (b) City Walkers validation set.
(a) Ablation on training data of PedGen.
(b) Ablation on model components of PedGen.
Context Factor
Training
Metric
Context Factor
Metric
Data
Method
MADE
aADE
mFDE 4. aFDE 1
mADE
aADE | mFDE | aFDE J
SLOPER4D
161
6.04
2.42
11.65
PedGen (-rajj/geo)
1.33
4.47
1.97
8.29
No
City (ityWaleess(500)
1.16
4.19
1.66
7.76
No
PodGen (-sep. token)
1.32
4.26
2.01
7.95
CityWallers(100%
1.13
4.08
1.61
7.56
PedGen
1.13
4.08
1.61
7.56
ST,OPERAD
6.45
2.19
PedGen (-3D rep.)
1.13
4.26
1.60
7.95
Scene
(0))
1.14
4.02
1.57
7.47
Scene
PedGen (-semantic)
1.12
3.86
1.56
7.09
City Cityallker<(0003)
1.11
3.75
1.55
6.92
PedCen
1.11
3.75
1.55
6.92
SLOPER4D
3.82
10.39
7.11
20.62
Human
City (5yyWlleees(000))
3.34
1.65
6.18
CityWalkers(100%)
1.09
3.24
1.61
5.95
5.3 ABLATION STUDY
Ablation on training data of PedGen. Table 3a shows an ablation study on the training data of PedGen. To demonstrate the effectiveness and necessity of using web videos for pedestrian movement generation, we train PedGen on SLOPER4D (Dai et al., 2023). SLOPER4D is one of the largest outdoor 4D human datasets annotated from LiDAR point clouds. Still, its scale and diversity in human subjects and scene contexts are much less than City Walkers, and its motion is captured from human actors instead of real-world pedestrians. We can observe that training on City Walkers leads to significant performance improvement compared to training on SLOPER4D, even though Cityesseess annotated with pseudo-labels, Notably, training with the human context on SLOPPR44 only achieves 3.82 mADE, as the model can easily overfit the 12 human subjects in the dataset and results in degraded generalization performance. On the contrary, with more training data of City Walkers, models conditioned different context factors all perform better than those trained with less data, highlighting the value of capturing large-scale pedestrian movement data using web videos.
Ablation model components of PedGen. Table 3b shows ablations key model components of PedGen. -traj/geo means the model is trained only with the diffusion reconstruction loss without the trajectory and geometry losses. -sep. token means the motion each timestep is represented
single token in the transformer instead of separate tokens for the rotation and velocity. -31) rep. means the scene context is encoded with 2D feature using depth anything (Yang et al. 2024) pre-trained image backbone DINOv2 (Oquab et al. 2202).. -semantic means the 3D voxel only encodes the occupancy without semantic labels. The results show the importance of using 3D scene context with semantic labels for pedestrian movement generation, the effectiveness of separating velocity and rotation tokens, and adding the geometry and trajectory losses in the motion diffusion model.
CONCLUSION
We study a new task of generating context-aware pedestrian movements by learning from web videos with noisy labels. To facilitate this study, we collect large-scale dataset Walkes with diverse real-world pedestrian movements in urban scenes. We further propose PedGen, a generative model that addresses the noisy labels in context factors: the scene context, the personal characteristics, and the goal position. Experiments show that PedGen can generate realistic context-aware pedestrian movements in both real-world and simulation environments. We hope this study will present new opportunities and facilitate future research modeling pedestrian movements in real-world settings.
and models the three important
Limitations. PedGen only considers static scene context at the starting frame, while pedestrian movement also depends dynamic scene contexts such as the history trajectories of other pedestrians. Modeling these dynamic objects would be an interesting future direction. In addition, PedGen only generates the movements of single pedestrian at one time, while modeling group activities in urban scenes can help generate more realistic real-world behaviors.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We are fully aware of the potential privacy and ethical issues when using street-view videos, and we take these concerns as our highest priority. We acknowledge that we will be solely responsible for any legal violations with respect our collected dataset and that we accept all the associated risks. We have taken several measures to mitigate these issues;
Legal Compliance with YuuTube's Terms of Service and Creative Commons Licensing: Our dataset comprises YouTube videos that are explicitly labeled under the Creative Commons Attribution (CC-BY-SA") license. This license allows for redistribution and adaptation of the material for any purpose, including research, as long as proper attribution is provided to the original creators. We have carefully reviewed YouTube's 's Terms of Service and Creative Commons guidelines to ensure that our use of these videos is legally permissible. YouTube's 's Terms of Service explicitly permit redistribution of Creative Commons-licensee excludes videos marked under the "Standard YouTube License" any content that carries additional restrictions. We do not download videos through unauthorized means but instead rely on publicly available Creative Commons content accessible via YouTube's official API, ensuring compliance with both YouTube's platform policies and broader copyright laws.
content, provided proper attribution is given. Our dataset strictly
Privacy Protection and Anonymization Measures: We follow standard protocols used by street datasets such as Waymo and Ego4D (Sun et al. 2020 Grauman et al. (022) to ensure complete anonymization of identifiable features. As the first step of our data preprocessing, we apply mosaics to faces and license plates to prevent re-identification of individuals in the videos. Additionally, we omit any copyright-related information, such as logos, channel owner details, or other copyrighted materials, during data processing to protect intellectual property rights. Our model is trained exclusively on preprocessed images, ensuring that identifiable features and copyrighted materials are neither stored nor accessible within our dataset. Furthermore, our IRB-approved protocol ensures compliance with human subjects protection guidelines, giving individuals in the videos the right to request data removal if privacy concerns arise.
Upon data release, we will provide clear and transparent terms of use for the dataset, outlining ethical guidelines, usage restrictions, and legal obligations that users must comply with. We will implement safeguards on our dataset webpage, including detailed user agreements, encryption measures to protect personal information, and access controls to monitor and prevent unauthorized data use. To align with privacy protection best practices from existing YouTube datasets such as YouTube-8M and YouTube- (Abu-Ei-Haija al. 2016 Xu et al. 2018), we will not provide processed video clips directly. Instead, users will be redirected to the original YouTube videos via a link and will be required follow our preprocessing scripts to prepare the dataset themselves. Additionally, as our labels are generated using off-the-so the--hheehhfff models, we will not release direct annotations but will provide scripts that allow users to generate their own secudo-labels, To further prevent misuse, we will only release the trained model weights, allowing researchers to utilize the dataset for downstream tasks such as pedestrian movement simulation without exposing raw video data.
Ethical Use and Responsible Data Sharing:
Institutional Review Board (IRB) Oversight and Legal Compliance: Our dataset will be overseen by the UCLA Institutional Review Board (IRB) with security protocols in place to protect against unauthorized access and misuse. The UCLA IRB will review our data collection, ensuring compliance with ethical and legal standards. Specifically, the IRB will assess the legal permissibility of using publicly available, CC-licensed videos and will confirm that our approach aligns with applicable laws and institutional policies. The human subjects in the videos have the right to view, correct, and request deletion of personal information in the dataset. The review board will continue to evaluate the collection, use, and sharing of the data to ensure alignment with best practices in data privacy and ethics. Any legal concerns raised post- post-publicn cust-publication will be addressed under UCLA's data governance framework, ensuring continued compliance and ethical responsibility.
By implementing these measures, we ensure that our dataset adheres to both legal requirements and ethical best practices. We remain committed to continuously reviewing and improving our compliance protocols in response to emerging ethical and legal considerations.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The project was supported by the NSF grants CNS-2235012 and IIS-2339769.
REFERENCES
Sami Abu-El- El-Haaa,,,,, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.0000 2016.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
Bharat Lal Bhatnagar. Xianghui Xie, llya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
15935-15946, 2022.
Ling-Hao Chen, Jiawei Zhang. Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9544-9555, 2023a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- lingul-inguiitic tasks. arXiv preprint arXiv:2312. 14238, 2023b.
Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. Omnire: Omni urban scene reconstruction. arXiv preprint arXiv:2408. 16760, 2024.
Ho Kei Cheng, Scoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. ICCV, 2023a.
Wei Cheng, Ruixiang Chen, Siming Fan, Wanqi Yin, Keyu Chen, Zhongang Cai, Jingho Wang, Yang Gao, Zhengming Yu, Zhengyu Lin, et al. Dna-rendering: A diverse neural actor repository for high-fidelity human-centric rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19982-19993, 2023b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Winnie Daamen. Modelling passenger flows in public transport facilities. 2002.
Winnie Daamen and Serge P Hoogendoorn. Experimental research of pedestrian walking behavior. Transportation research record, 1828(1):20-30) 2003.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mofusion:
A framework for denoising seresinnndfffusoo--baae conference on computer vision and pattern recognition, pp. 9760-99 07709997,, 2023.
motion synthesis. In Proceedings of the IEEE/CVF
Yudi Dai, Yitai Lin, Xiping Lin, Chenglu Wen, Lan Xu, Hongwei Yi, Siqi Shen, Yuexin Ma, and Cheng Wang. Sloper4d: A acene-aware dataset for global 4d human pose estimation in urban environments. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 682-692, June 2023.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Mohamed H Dridi. List parameters influencing the pedestrian movement and pedestrian database. Int'l J. Soc. Sci. Stud., 3:94, 2015.
12

---

Published as a conference paper at ICLR 2025
Dorine Duives. Analysis and modelling of pedestrian movement dynamics large-scale events. 2016.
Yan Feng, Dorine Duives, Winnie Daamen, and Serge Hoogendoorn. Data collection methods for studying pedestrian behaviour: A systematic review. Building and Environment, 187:107329, 2021.
Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference Multimedia, pp. 2021-20029 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5152-5161, June 2022.
Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. Human poseitioning system (hps): 3d human pose estimation and self-localization in large scenes from body-mounted sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 4318 4329, 2021.
Milad Haghani and Majid Sarvi. Crowd behaviour and motion: Empirical methods. Transportation research part B: methodological. 007:253-294 2018.
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael Black. Resolving 3d human pose ambiguities with 3d scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2282-2292, 2019.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11334-1 1384, 2021.
Norman C Heglund and C Richard Taylor. Speed, stride frequency and energy cost per stride: how do they change with body size and gait? Journal of Experimental Biology, 138(1)3001-318, 1988.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Serge Hoogendoorn and Piet HL Bovy. Pedestrian route-choice and activity scheduling theory and 2004.
models. Transporiation Research Part B: 380i
Chun-Hac P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13274-13285, 2022.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion- based generation, optimization, and planning in 3d scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scemu interaction modeling, arXiv preprint wwXiv:243.008629 2024.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, January 2023. URL https 7/gi wttp:/ccoo//tttoeeo wwwrralytissulllltte wttpissslttttattti.......
13

---

Published as a conference paper at ICLR 2025
Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengeheng Tang, Juan José Zárate, and Otmar Hilliges. Emdb: The electromagnetic database of global 3d human pose and shape in the wild. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14632-14643, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. arXiv preprint arXiv:2305. 18593, 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
Maria Lyssenko, Christoph Gladisch, Christian Heinzemann, Matthias Woehrle, and Rudolph Triebel. From evaluation to verification: Towards task-oriented relevance metrics for pedestrian detection in safety-critical domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 38-45, 2021.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 5442-5451, 2019.
Vikas Mehta. Walkable streets: pedestrian behavior, perceptions and attitudes. Journal of Urbanism, 1(3):217-245, 2008.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXi::230..07193. 2023.
Karen Ostrosky, Jessie M VanSwearingen, Ray Burdett, and Zena Gee. A comparison gait characteristics in young and old subjects. Physical therapy, 74(7):637-644, 1994.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13756 13766, 2023.
Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory prediction in crowded scenes. In European Conference on Computer Vision (ECCV), volume 2, pp. 5, 2016.
Daniel Sczepansky. Poptravel. https: wttps///wwww...bbb......mmm com wtt.c///ppppp/eee/cooo 2024. Ac- cessed: 2024-05-21. Licensed under CC
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as generative prior. arXiv preprint arXiv:2303.01418, 2023.
Mengyi Shan, Brian Curless, Ira Kemelmacchr--Shizzeman. and Steve Seitz. Animating street view. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-12, 2023.
TJ Shields and Karen Boyce. A study of evacuation from large retail stores. Fire safety journal, 35 (1):25-49, 2000.
Soyong Shin, Juyong Kim, Eni Halilaj, and Michael Black. Wham: Reconstructing world-grounde humans with accurate 3d motion. arXiv preprint arXiv:2312.07531, 2023.
Jonathan D Sime. Crowd psychology and engineering. Safety science, 21(1):1-14, 1995.
14

---

Published as a conference paper at ICLR 2025
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1446-2456 2020.
Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. Advances in Neural Information Processing Systems, 2023.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-On -r,, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Yunhe Tong and Nikolai WF Bode. An investigation of how context affects the response of pedestrians to the movement of others. Safety science, 157: 157:105919, 2023.
Adrien Treuille, Seth Cooper, and Zoran Popović. Continuum crowds. ACM Transactions On Graphics (TOG), 2563):1100-1168. 2006.
Shashank Tripathi, Omid Taheri, Christoph Lassner, Michael Black, Daniel Holden, and Carsten Stoll. Humos: Human motion model conditioned on body shape. arXiv preprint arXiv:2409.0394 2024.
Mignon van den Berg. The Influence of Herding on Departure Choice in Case of an Evacuation: Design and Analysis of a Serious Gaming Experimental Ser-up. TRAIL, 2016.
Edward Vendrow, Duy Tho Le, Jianfei Cai, and Hamid Rezatofighi. Jrdb-pose: A large-scale dataset for multi-person pose estimation and tracking. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 481 14820, 2023.
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. Proceedings of the European conference computer vision (ECCV), pp. 601-617, 2018.
Jingbo Wang, Zhengyi Luo, Ye Yuan, Yixuan Li, and Bo Dai. Pacer+: On-demand pedestrian anima- tion controller driving scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 718-728 2024.
Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assistedd intervention, pp. 35-45. Springer, 2022.
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090 2021.
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03337. 2018
Yuanyuan Xu, Wan Yan, Genke Yang, Jiliang Luo, Tao Li, and Jianan He. Centerface: joint face detection and alignment using face point. Scientific Programming, 2020(1):7845344, 2020.
Kebing Xue and Hyewon Sco. Shape conditioned human motion generation with diffusion model. arXiv preprint arXiv:2405.06771 2024.
Masahiro Yamasaki, Takashi Sasaki, and Masafumi Torii.
Sex difference in the pattern of lower limb movement during treadmill walking. European journal of applied physiology and occupational physiology, 62:99-103, 1991.
Ming Yan, Yan Zhang, Shuqiang Cai, Shuqi Fan, Xincheng Lin, Yudi Dai, Siqi Shen, Chenglu Wen, Lan Xu, Yuexin Ma, et al. Relil Id: A comprehensive multimodal human motion dataset and method. arXiv preprint a403 19501, 2024.
15

---

Published as a conference paper at ICLR 2025
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
Hongwei Yi, Justus Thies, Michael J. Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. arXiv:2404. 2024.
Mingyuan Zhang. Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao Hc, Chenggang Yan, and Tao Mei. Gait recognition in the wild with dense 3d representations and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10228-202333 2022.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5745-5753, 2019.
Zheng Zhu, Xianda Guo, Tian Yang. Junjie Huang, Jiankang Deng, Guan Huang, Dalong Du, Jiwen Lu, and Jie Zhou. Gait recognition in the wild: A benchmark. In Proceedings of the IEEF/CVF international conference computer vision, pp. 14789-14799, 2021.
16


Reviews:
[{"review_id": "gT56JPCyRS", "reviewer": "Reviewer_EYrc", "summary": "This paper introduces a context-aware generative model for realistic pedestrian movement prediction. It leverages a conditional diffusion framework that uses 3D point clouds to capture spatial scene context. PedGen offers a solution suitable for applications in autonomous systems, crowd simulation, and urban planning.", "questions": "1- Since pedestrian path generation is done in static settings, the social attributes of pedestrians are not taken into account. Are other pedestrians considered as objects when calculating the collision measure? \n\n2- How is the collision rate  affected by the ablations?\n\n3- The training of masks *m* in Fig. 3 is unclear. How are these masks trained, and which part of the loss function guides this training?\n\n3- Looking at Table 3.b, it appears that the goal has a significant effect on error reduction. However, in practice, the goal of a pedestrian is generally unknown when predicting the path. Why is the goal handled as context in this work? Shouldn’t the model predict the goal as part of the path prediction process? I am also curious to see visualizations with an ablated version where the goal is not provided as input.\n\n4- mADE is mentioned to be measured across 50 movements. Does this mean that 50 possible paths were generated?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Context is an important factor in pedestrian trajectory prediction, which poses a challenge due to the difficulty of identifying and measuring it while predicting future paths. This paper proposes a valuable idea, supported by clear visualizations and thorough ablation studies.", "weaknesses": "1- I find Fig. 3 confusing. Based on the context of the paper, it appears that only one timestep is observed, and the rest are predicted. However, Fig. 3 suggests that the model is fed with the timesteps from t=1 to t=T. \n\n2- The learnable mask *m* needs to be explained more in the paper. How is this mask learned?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mjLjRppPt8", "reviewer": "Reviewer_M4G7", "summary": "The authors propose a method for learning pedestrian movements from web videos by using pre-trained predictors to generate pseudo-labels through off-the-shelf 4D human motion estimation models, despite the inherent noise in the labels. To refine these noisy labels, they introduce the PedGen model, which filters out noise and incorporates conditional inputs that may influence pedestrian behavior, thereby lifting the 2D scene into a 3D representation. The authors intend to provide open access to both the dataset and the model.", "questions": "1. How does the proposed method handle scenarios where conditional inputs are unavailable or unreliable, as might be the case in applications like autonomous driving?\n2. Could the authors elaborate on why only three baselines were chosen for comparison, and whether additional baselines might provide a more comprehensive evaluation?\n3. Could you clarify the specific novel aspects of the automated labeling and filtering pipeline? Additionally, is there potential for further innovation in this pipeline to enhance its originality, or were there particular design constraints that influenced its current implementation?\n\nI am open to reconsidering my final rating if the authors address the concerns raised.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on using noisy labels to learn pedestrian motion, a novel approach with the potential to benefit various research areas.\n2. The authors contribute a dataset accompanied by a label generation and filtering strategy, addressing the challenge of noise in automated labeling pipelines.\n3. The results demonstrate performance improvements over baselines, and comprehensive ablation studies are conducted to validate the approach.", "weaknesses": "1. Although the automated labeling and filtering pipeline is essential, it is a fairly common approach, limiting the novelty of this contribution.\n2. The baselines used in the comparison experiments appear weak, with only three included, potentially limiting the robustness of the results.\n3. While using the goal as a conditioning factor is crucial and enhances pedestrian movement prediction, some conditions are often not visible or are difficult to capture in practical applications, such as autonomous driving. This raises concerns about the real-world applicability of the proposed setting and whether alternative solutions might address this limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ASsFaYwqc", "reviewer": "Reviewer_yQtW", "summary": "In this work, the authors propose to learn pedestrian movements from web videos. To this end, they curate a large-scale dataset called CityWalkers that captures real-world pedestrian movements in urban scenes from YouTube, with some extra efforts to generate pseudo-GT labels and remove the low-quality labels. Given this dataset, the authors propose a generative model where a context encoder is introduced to incorporate various context factors, including goal, human, and scene, in generating realistic pedestrian movements in urban scenes. \n\nExperiments show that the proposed outperforms existing baseline methods for pedestrian movement generation by learning extra data and incorporating the context factors.", "questions": "Please address my concerns in weakness as well as in below.\n\n1. Given that the ground truth (GT) is primarily generated using existing methods, how do the authors ensure consistency across these various methods? For example, is the generated depth map aligned with the generated 4D human pose? If there is a discrepancy, what potential drawbacks arise from this mismatch, and how can the proposed method address them?\n2. How do the authors convert the depth labels into a 3D point cloud without knowing the camera parameters (l.310)? Am I missing any assumptions here?\n3. It appears that only static objects and elements are considered in the scene context, as there is no explicit modeling of dynamics in the context encoder. Do the authors intentionally exclude dynamics during scene context modeling, or are these dynamics treated as static objects? Is it valid to assume that other road participants, such as pedestrians, are not significant in the modeling process for pedestrian motion generation? I find this assumption questionable, especially since collisions are used as evaluation metrics in this study.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The concept of integrating contextual information into pedestrian generation is sound and has already demonstrated effectiveness in related trajectory generation tasks.\n2. The ablation studies examining each factor are thorough and well-executed.\n3. The visual examples provided in both the main paper and supplementary materials are satisfactory.", "weaknesses": "1. Unclear Task Definition: Section 4.1 outlines the overall task definition, but it seems that only the 3D location of pedestrians and the 2D image of the scene in the first frame (t_1) are provided, while all other elements are predicted. Is this interpretation correct? If so, what is actually learned from this setup, considering there could be millions of plausible results? Or do we learn the bias in the training set? Is this a reasonable setup?\n2. Clarification of Contributions: The overall contributions should be clarified. Lines 97-101 describe the contributions in a confusing manner. For instance, line 98 states, “1) A new task of context-aware pedestrian movement generation from web videos with unique challenges in dealing with label noise and modeling various motion contexts.” This seems contradictory since the dataset with noisy labels is presented as a contribution in “A new large-scale real-world pedestrian movement dataset, CityWalkers, with pseudo-labels of diverse pedestrian movements and motion contexts” (l. 99). I have significant doubts about the dataset's quality and question whether a noisy dataset can genuinely be considered a contribution.\n3. Need for More Explanations About Results: Based on Table 2(b), incorporating scene information yields only minor improvements. The priority order appears to be Goal, Human, and Scene, which raises questions about the usefulness and necessity of including Scene in the overall model. The authors should provide an explanation for this observation.\n4. Visual Results for Complex Scenes: More visual examples are needed to illustrate how pedestrians navigate around obstacles, such as chairs, trees, or cars, to reach their targets. Without these examples, the scene context seems to have limited utility, as indicated by the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qyFKYc9BdC", "reviewer": "Reviewer_XLS3", "summary": "This paper studies a new task of generating context-aware pedestrian movements by learning from web videos with noisy labels. Different from previous human motion generation task that focuses on in-door human motion generation, the proposed method can generate outdoor human movements conditioned on scene information and person identity information. To achieve this goal, this paper also introduces a new dataset named as CityWalker, which contains outdoor urban pedestrian movements from web videos. Experiments on several benchmarks demonstrate the effectiveness of the proposed method.", "questions": "In weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The contribution of this paper is comprehensive because it contains both dataset and method, which can provide reference for future research in this area.\n2. The focus of this paper is interesting, generating human motion in outdoor environment is not well-studied by previous work and this paper present a solution to this task, which may be useful for autonomous driving and related field.\n3. This paper is well-written and easy to understand.", "weaknesses": "Although this paper clarifies their task as motion generation, the proposed method and evaluation metrics are more like motion prediction, i.e., output person movements by accepting current condition inputs. The evaluation metrics also does not involve the diversity evaluation of the generated human motion. Therefore I have doubts about the task type of this paper, maybe motion prediction is more accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)