Pedestrian Motion Reconstruction: A Large-scale Benchmark via Mixed Reality Rendering with Multiple Perspectives and Modalities

Authors: Yichen Wang, Yiyi Zhang, Xinhao Hu, Li Niu, Jianfu Zhang, Yasushi Makihara, Yasushi Yagi, Pai Peng, Wenlong Liao, Tao He, Junchi Yan, Liqing Zhang
Primary area: datasets and benchmarks
Keywords: Pedestrian Dynamics, Mixed Reality

Abstract:
Reconstructing pedestrian motion from dynamic sensors, with a focus on pedestrian intention, is crucial for advancing autonomous driving safety. However, this task is challenging due to data limitations arising from technical complexities, safety, and cost concerns. We introduce the Pedestrian Motion Reconstruction (PMR) dataset, which focuses on pedestrian intention to reconstruct behavior using multiple perspectives and modalities. PMR is developed from a mixed reality platform that combines real-world realism with the extensive, accurate labels of simulations, thereby reducing costs and risks. It captures the intricate dynamics of pedestrian interactions with objects and vehicles, using different modalities for a comprehensive understanding of human-vehicle interaction. Analyses show that PMR can naturally exhibit pedestrian intent and simulate extreme cases. PMR features a vast collection of data from 54 subjects interacting across 12 urban settings with 7 objects, encompassing 12,138 sequences with diverse weather conditions and vehicle speeds. This data provides a rich foundation for modeling pedestrian intent through multi-view and multi-modal insights. We also conduct comprehensive benchmark assessments across different modalities to thoroughly evaluate pedestrian motion reconstruction methods.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730358063872

thread_type:
official_review

<thread_object>
creator: reviewer_zsyg

<content>
summary:
The paper introduces the Pedestrian Motion Reconstruction (PMR) dataset, a large-scale benchmark designed to advance autonomous driving safety by focusing on pedestrian intention and behavior. The dataset is created using a mixed reality platform that combines real-world realism with simulation accuracy, capturing pedestrian interactions with objects and vehicles from multiple perspectives and modalities. The PMR dataset includes data from 54 subjects across 13 urban settings with 7 objects, encompassing 12,138 sequences under diverse weather conditions and vehicle speeds. The paper also presents comprehensive benchmark assessments across different modalities to evaluate pedestrian motion reconstruction methods.

strengths:
1. The PMR dataset is the first of its kind to incorporate multi-perspectives and multi-views for modeling pedestrian intention in diverse outdoor scenes, including rare scenarios like collisions with safety concerns.
2. The dataset provides a rich foundation for modeling pedestrian intent through insights from third-person perspective RGB videos, LiDAR data, and egocentric perspectives.
3. The mixed reality platform reduces data collection costs and risks while ensuring ground-truth alignment with the global coordinate system, capturing pedestrian interactions realistically.
4. The paper conducts thorough evaluations across different modalities, providing a valuable resource for assessing pedestrian motion reconstruction methods.
5. The dataset is publicly available, promoting further research and development in the field.

weaknesses:
1. While the mixed-reality data offers high-quality labels, there may be a domain gap between simulated and real-world data, which could affect the generalizability of models trained on the PMR dataset. The domain gap not only lies the rendering but also the motion diversity.
2. The reliance on a mixed reality platform with VR headsets, MoCap systems, and the CARLA simulator might limit the reproducibility of the data collection process in other research settings.
3. The dataset is collected from a limited number of subjects, which may not fully represent the global diversity in pedestrian behavior and motion.

questions:
1. How does the PMR dataset address the domain gap between simulated and real-world data, and what measures are taken to ensure the dataset's applicability to real-world scenarios? The author should discuss more about how could this dataset working together with real captured dataset, for example waymo-open-dataset.
2. What are the limitations of the current data collection process, and how might these be overcome in future work to increase the diversity and representativeness of the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533014288

thread_type:
official_review

<thread_object>
creator: reviewer_xglt

<content>
summary:
This paper presents a large-scale pedestrian motion reconstruction (PMR) dataset developed using a mixed reality platform. It simulates the authenticity of the real world while reducing the costs and risks associated with dataset creation. The dataset includes third-person perspective RGB videos of moving vehicles, LiDAR data from the vehicles, and self-centered views of pedestrians, along with both unimodal and multimodal pedestrian pose estimation. This dataset contributes to the advancement of the community.

strengths:
1.	The dataset is designed based on the simulation of pedestrian intentions in the real world and features a pipeline for pedestrian motion capture utilizing VR and MoCap technologies. It replicates various pedestrian actions found in real-world scenarios, providing a reference for dataset collection and production. 
2.	The substantial amount of data offers a rich foundation for modeling pedestrian intentions through multi-view and multimodal insights.
3.	The dataset also simulates the interaction between pedestrians and vehicles under different environmental conditions, which is very suitable for practical applications.

weaknesses:
1.	I noticed that interactions between pedestrians and objects are quite limited, with a greater focus on pedestrian-vehicle interactions. Did the authors consider including specific scenarios, such as pedestrian conversations, telephone calls, and the use of umbrellas on rainy days?
2.	The comparison datasets in the paper are not up-to-date. To strengthen the comparison with datasets gathered from real-world scenarios, you may refer to some of the most recent multimodal human body reconstruction datasets. such as RELI11D(CVPR2024) [1], HiSC4D (TPAMI 2024) [2].
3.	In the collection pattern of this dataset, the encounters between pedestrians and vehicles in the environment are all predetermined by fixed program designs, which may result in slightly limited comprehensiveness of the dataset. Have the authors considered incorporating randomness in procedural generation techniques to accommodate the diversity of interactions?


**Reference**

[1] RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method, CVPR 2024

[2] HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR, PAMI 2024

questions:
1. There are already numerous existing human motion datasets; could you provide the latest datasets for comparison?
2. The picture annotation text in Figure 6 is blocked by the picture.
3. What is the frequency of camera and Lidar acquisition?
4. The ground truth used in this article is sampled from the mocap system to 120hz, please explain the rationality of the GT.
5. What are the shape distribution and the speed distribution of virtual avatars?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676290160

thread_type:
official_review

<thread_object>
creator: reviewer_7xae

<content>
summary:
The paper introduces the Pedestrian Motion Reconstruction (PMR) dataset, a large-scale benchmark designed to improve the reconstruction of pedestrian motion and intention for autonomous driving applications. The dataset is generated using a mixed-reality platform that combines real-world realism with simulation-based data collection, reducing costs and safety risks. PMR features multi-modal and multi-perspective data, including third-person RGB videos, LiDAR data, and egocentric perspectives from 54 subjects across 13 urban settings. The dataset provides 12,000+ sequences under various weather conditions and vehicle speeds, capturing complex pedestrian-vehicle interactions, including rare scenarios like collisions. The main contributions include: 1) a mixed-reality platform of collecting real-world like pedestrian motion and corresponding sensor data 2) benchmark evaluations across different modalities such as third-person/first-person RGB and LiDAR for reconstructing pedestrian poses, and 3) insights into pedestrian behavior in dangerous and rare scenarios.

strengths:
- The proposed mixed-realtiy platform combining real-world MoCap and VR simulation is interesting, and could provide a safe and efficient way to collect real-world pedestrian motion data and rare scenarios.
- It also provides a good benchmark for global human and camera motion estimation, with motion distribution closer to the real-world. Previous dataset often use existing MoCap motion data which often cannot reflect real-world human motion distributions.
- The dataset also includes real-world object interactions and tracking.

weaknesses:
- The main tasks of the benchmark seem to focus on human pose estimation, which I think discard most of the scene context such as vehicles, weather, etc. It does not evaluate the main strengths of the dataset, which includes pedestrian vehicle interactions. I think having some pedistrian behavior generation/forecasting tasks would be beneficial since it will measure models’ ability to capture vehicle pedestrain interactions in rare or dangerous scenarios.
- To validate the domain gap between the proposed synthetic dataset and real-world data, the paper should evaluate on its main tasks, human pose estimation. For example, using the proposed dataset as additional training to improve the human pose estimation performance on existing benchmarks (3DPW, EMDB, RICH etc.). It is a bit weird that the paper evaluates the domain gap on a complete different task of 3D object detection.
- Finally, it would be nice if the paper can provide videos to showcase the rare cases and highlight the quality of the dataset.

questions:
- How does the dataset decide which objects should be placed in real-world, and which should be placed in simulation?
- The proposed mixed-reality data capture platform is pretty neat. Any plans to open source it to expand the dataset further?
- What instructions are given to the subject to interact with the VR environment?
- Are there any audios such as traffic sounds provided to the subject to let them behavior more realistically?

Typo:
- l363: “taccount”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692608998

thread_type:
official_review

<thread_object>
creator: reviewer_mpm7

<content>
summary:
The paper presents the Pedestrian Motion Reconstruction (PMR) dataset, a large-scale, mixed-reality dataset created to support research on pedestrian motion reconstruction, particularly focusing on pedestrian intent and behavior in urban environments. This dataset combines real-world elements with virtual simulations, allowing for extensive data capture with reduced safety and cost risks compared to real-world experiments.

strengths:
The wide variety of urban setting, and the real data collected also with over 50 VR participants for first person perspective and realistic motions.  It also includes LIDAR. Totaling 12,138 sequences. The validation and the labeling of the dataset is good. All in all makes a very complete contribution.

weaknesses:
I don't see major weaknesses. However, there could be domain Gaps from the Egocentric Data in PMR as it is collected through VR headsets in a simulated setting, which differs from real-world egocentric videos captured with physical head-mounted cameras. 
I wonder how something like this could be done more similar to Ergo4D.
This domain discrepancy could limit the effectiveness of models trained on PMR for real-world applications, as the simulated egocentric data may not generalize well to real-world environments. This isn't a major flaw, but perhaps limits the impact of the dataset.

questions:
I would like to see in the discussion how the data could also be used for causal implementations. Beyond just the current dataset. Many simulators recreate events by replicating configuration files. In fact a prior interesting work on a similar space but running on Airsim instead of CARLA, was looking into creating realistic behaviours based on motion capture and personality recreation. 

Wang, Cheng Yao, et al. "CityLifeSim: A High-Fidelity Pedestrian and Vehicle Simulation with Complex Behaviors." 2022 IEEE 2nd International Conference on Intelligent Reality (ICIR). IEEE, 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732528348852

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate sincerely for all your precious reviews and positive ratings. We have updated both the pdf file and supplementary materials, where we marked the revised and newly added parts in red. Please check out our latest version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents the Pedestrian Motion Reconstruction dataset , a large-scale benchmark aimed at enhancing the reconstruction of pedestrian motion and intention for autonomous driving. Using a mixed-reality, it combines real-world realism with simulation to collect diverse multi-modal data, including RGB videos, LiDAR, and egocentric perspectives, capturing sequences across various urban scenarios and rare interactions. Key contributions include the innovative data collection platform, benchmark evaluations across modalities, dataset to get insights into pedestrian behavior in complex and hazardous situations.