SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pedestrian Trajectory Prediction. The pedestrian trajectory prediction benchmark is presented to evaluate the accuracy of trajectory prediction. Average displacement error (ADE) and final displacement error (FDE) are widely used as evaluation metrics [1, 22]. ADE represents the average error in predicted trajectories compared to ground truth over all time steps, and FDE quantifies the deviation between the predicted final position and ground truth at the last time step. Using the preceding 2-second trajectory as input, the trajectory prediction models generate a set of  $K$  7-second future trajectories, representing  $K$  distinct modes of future trajectory distribution. To evaluate the prediction accuracy of these  $K$  trajectories, we use  $\mathrm{ADE}_K$  and  $\mathrm{FDE}_K$ , which represent the minimum ADE and FDE values among the  $K$  trajectory candidates.
End-to-End Motion Prediction. In the end-to-end motion prediction task, the models take raw sensor data as input and generate future 3D bounding boxes, accompanied by the corresponding object trajectories. We consider a 7-second future horizon for trajectory prediction. We use  $\mathrm{mAP}_f$  metric proposed in FutureDet [23] along with the conventional ADE and FDE as performance metrics.
Methods	Map	ADE5↓	FDE5↓	ADE20↓	FDE20↓
Social-LSTM [1]	1.638	3.121	1.630	3.103
Y-Net [19]	1.527	2.802	0.836	1.878
Y-Net [19]	✓	1.361	2.624	0.675	1.547
NSP-SFM [38]	1.346	2.261	0.634	1.087
NSP-SFM [38]	✓	1.061	1.818	0.517	0.925
Table 4: Evaluation of pedestrian trajectory prediction baselines based on  $\mathrm{ADE}_5$ ,  $\mathrm{FDE}_5$ ,  $\mathrm{ADE}_{20}$ , and  $\mathrm{FDE}_{20}$ .  $\downarrow$ : the lower, the better.
Methods	mAP ↑	\( \mathrm{mAP}_{\mathrm{f}} \uparrow \)	\( \mathrm{ADE}_{5} \downarrow \)	\( \mathrm{FDE}_{5} \downarrow \)
FaF [18]	0.490	0.079	1.915	3.273
FutureDet-P [23]	0.209	0.037	2.532	4.537
FutureDet-V [23]	0.408	0.053	2.416	4.409
Table 5: Evaluation of end-to-end motion prediction baselines.  $\uparrow$  : the higher, the better.  $\downarrow$  : the lower,the better.
5.2 Performance of Baseline Models
In this section, we present the performance of several baseline models for each benchmark. All experiments were performed on 4 NVIDIA RTX3090 GPUs with 2 Intel Xeon CPUs. Detailed configurations of the baseline models can be found in the Supplementary Material.
3D Pedestrian Detection. Table 2 presents the performance of four baseline methods, FCOS3D [30], PointPillars [14], CenterPoint [36], and TransFusion [2] evaluated in 3D Pedestrian Detection benchmark. CenterPoint-P and TransFusion-P utilized the PointPillars as encoding backbone, while CenterPoint-V and TransFusion-V employed SECOND [34] as the voxel encoding backbone. Note that camera-LiDAR fusion yields better performance compared to models relying on a single modality.
3D Pedestrian Tracking. We considered AB3DMOT [31] and the CenterPoint tracker as the baseline models for 3D MOT. We applied AB3DMOT to PointPillar and CenterPoint detectors. Table 3 presents the performance of the baselines evaluated on the SiT dataset.
Pedestrian Trajectory Prediction. Table 4 presents the performance of Social-LSTM [1], Y-Net [19], and NSP-SFM [38]. For Y-Net and NSP-SFM, the scene encoding was conducted utilizing the semantic maps offered from our dataset. In Table 4, we confirm the benefit of utilizing scene contexts for trajectory prediction.
End-to-end Motion Prediction. Table 5 presents the performance of two end-to-end motion prediction baseline models: Fast and Furious [18] and FutureDet [23]. FutureDet-P uses PointPillars as the encoding backbone while FutureDet-V utilizes SECOND as the voxel encoding backbone.
In this paper, we introduced the SiT dataset, a new pedestrian trajectory dataset designed to facilitate the development of perception and motion prediction models for social navigation robots. The SiT dataset comprises pedestrian trajectories gathered from a variety of social interaction scenarios that can be encountered by robots during navigation in the real world. Our analysis revealed that the SiT dataset effectively captured the behavior of pedestrians in close proximity to the robot within a shared semantic space on the map. The SiT dataset also offered high-quality annotations for 2D and 3D object detection and tracking, enabling the design of end-to-end motion prediction models in the upstream pipeline. These features make the SiT dataset unique when compared to existing pedestrian trajectory datasets. We hope that our proposed dataset and baselines provide a strong foundation for future research in pedestrian perception and have the potential to accelerate the development of fully autonomous robots.
One limitation of our dataset is that we were unable to scale the size of the dataset up to hundreds of thousands of frames due to time constraints. However, we are planning to address this limitation by adding an additional dataset in the second phase release, called SiT dataset v2, to significantly expand the scale of our dataset.
This work was partly supported by 1) National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2020R1A2C2012146), 2) Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2020-0-01373, Artificial Intelligence Graduate School Program (Hanyang University) and 3) No.2022-0-00957, Distributed on-chip memory-processor model PIM (Processor in Memory) semiconductor technology development for edge applications), and 4) Hyundai Motor Company.
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961–971, 2016.
[2] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai. TransFusion: Robust LiDAR-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1090-1099, 2022.
[3] K. Bernardin, A. Elbs, and R. Stiefelhagen. Multiple object tracking performance metrics and evaluation in a smart room environment. In Sixth IEEE International Workshop on Visual Surveillance, in conjunction with ECCV, volume 90. CiteSeer, 2006.
[4] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11621-11631, 2020.
[5] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays. Argoverse: 3D tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8748-8757, 2019.
[6] Y. Chen, Y. Luo, C. Yang, M. O. Yerebakan, S. Hao, N. Grimaldi, S. Li, R. Hayes, and B. Hu. Human mobile robot interaction in the retail environment. Scientific Data, 9(1):673, 2022.
[7] Z. Chen, C. Jiang, and Y. Guo. Pedestrian-robot interaction experiments in an exit corridor. In Proceedings of the International Conference on Ubiquitous Robots (UR), pages 29–34. IEEE, 2018.
[8] Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, and F. Zhao. Deformable feature aggregation for dynamic multi-modal 3D object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 628-644. Springer, 2022.
[9] P. Cong, X. Zhu, F. Qiao, Y. Ren, X. Peng, Y. Hou, L. Xu, R. Yang, D. Manocha, and Y. Ma. STCrowd: A multimodal dataset for pedestrian perception in crowded scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19608-19617, 2022.
[10] H. Darweesh. Assure mapping tools. https://github.com/hatem-darweesh/assuremappingtools, 2021.
[11] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354-3361, 2012.
[12] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17853-17862, 2023.
[13] B. Kim, S. H. Park, S. Lee, E. Khoshimjonov, D. Kum, J. Kim, J. S. Kim, and J. W. Choi. LaPred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14636-14645, 2021.
[30] T. Wang, X. Zhu, J. Pang, and D. Lin. FCOS3D: Fully convolutional one-stage monocular 3D object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 913–922, 2021.
[31] X. Weng, J. Wang, D. Held, and K. Kitani. 3D multi-object tracking: A baseline and new evaluation metrics. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10359–10366, 2020.
[32] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2021.
[33] S. Xu, D. Zhou, J. Fang, J. Yin, Z. Bin, and L. Zhang. FusionPainting: Multimodal fusion with adaptive attention for 3D object detection. In Proceedings of the IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3047-3054, 2021.
[34] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018.
[35] D. Yang, L. Li, K. Redmill, and Ü. Özgün. Top-view trajectories: A pedestrian dataset of vehicle-crowd interaction from controlled experiments and crowded campus. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), pages 899–904, 2019.
[36] T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3D object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11784-11793, 2021.
[37] J. H. Yoo, Y. Kim, J. Kim, and J. W. Choi. 3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736. Springer, 2020.
[38] J. Yue, D. Manocha, and H. Wang. Human trajectory prediction via neural social physics. In Proceedings of the European Conference on Computer Vision (ECCV), pages 376-394. Springer, 2022.
[39] B. Zhang, J. Amirian, H. Eberle, J. Pettré, C. Holloway, and T. Carlson. From HRI to CRI: Crowd robot interaction—understanding the effect of robots on crowd motion: Empirical study of pedestrian dynamics with a wheelchair and a pepper robot. International Journal of Social Robotics, 14:631-643, 2021.