SiT Dataset: Socially Interactive Pedestrian Trajectory Dataset for Social Navigation Robots
Authors: Jongwook Bae
Jungho Kim
Junyong Yun
Changwon Kang
Jeongseon Choi
Chanhyeok Kim
Junho Lee
Jungwook Choi
Jun Won Choi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: robotics
human interaction

Abstract: To ensure secure and dependable mobility in environments shared by humans and robots, social navigation robots should possess the capability to accurately perceive and predict the trajectories of nearby pedestrians. In this paper, we present a novel dataset of pedestrian trajectories, referred to as Social Interactive Pedestrian Trajectory (SiT) dataset, which can be used to train pedestrian detection, tracking, and trajectory prediction models needed to design social navigation robots. Our dataset includes sequential raw data captured by two 3D LiDARs and five cameras covering a 360-degree view, two inertial measurement units (IMUs), and real-time kinematic positioning (RTK), as well as annotations including 2D & 3D boxes, object classes, and object IDs. Thus far, various human trajectory datasets have been introduced to support the development of pedestrian motion forecasting models. Our SiT dataset differs from these datasets in the following three respects.  First, whereas the pedestrian trajectory data in other datasets were obtained from static scenes, our data was collected while the robot navigated in a crowded environment, capturing human-robot interactive scenarios in motion. Second, unlike many autonomous driving datasets where pedestrians are usually at a distance from vehicles and found on pedestrian paths, our dataset offers a distinctive view of navigation robots interacting closely with humans in crowded settings.Third, our dataset has been carefully organized to facilitate the training and evaluation of end-to-end prediction models encompassing 3D detection, 3D multi-object tracking, and trajectory prediction. This design allows for an end-to-end unified modular approach across different tasks. We introduce a comprehensive benchmark for assessing models across all aforementioned tasks and present the performance of multiple baseline models as part of our evaluation. Our dataset provides a strong foundation for future research in pedestrian trajectory prediction, which could expedite the development of safe and agile social navigation robots. The SiT dataset, development kit, and trained models are publicly available at: https://spalaboratory.github.io/SiT/

SiT Dataset: Socially Interactive Pedestrian Trajectory Dataset for Social Navigation Robots
Jongwook Bae" Jungho Kim" Junyong Yun' Changwon Kang"
Jeongseon Choi
Chanhyeok Kim
Junho Lee
Jungwook Choi Jun Won Choit
Hanyang University
(jwbae, junghokim, jyyun, changwonkang, jschoi, chkim, hanaaaaaaa.h hanyang kr {choij, : junschoi)@hanyang ac.
Abstract
To ensure secure and dependable mobility in environments shared by humans and robots, social navigation robots should possess the capability to accurately perceive and predict the trajectories of nearby pedestrians. In this paper, we present a novel dataset of pedestrian trajectories, referred Pedestrian Trajectory (SiT) dataset, which can be used to train pedestrian detection, tracking. and trajectory prediction models needed to design social navigation robots. Our dataset includes sequential raw data captured by two 3D LiDARs and five cameras covering a 360-degree view, two inertial measurement writs (IMUs), and real-time kinematic positioning (RTK), as well as annotations including 2D & 3D boxes, object classes, and object IDs. Thus far, various human trajectory datasets have been introduced to support the development of pedestrian motion forecasting models. Our SiT dataset differs from these datasets in the following three respects. First, whereas the pedestrian trajectory data in other datasets were obtained from static scenes, our data was collected while the robot navigated a crowded environment, capturing human-robot interactive scenarios in motion. Second, unlike many autonomous driving datasets where pedestrians are usually at distance from vehicles and found on pedestrian paths, dataset offers distinctive view of navigation robots interacting closely with humans crowded settings. Third, our dataset has been carefully organized facilitate the training and evaluation of end-to-end prediction models encompassing 3D detection, 3D multi-object tracking, and trajectory prediction. This design allows for end-to- end unified modular approach across different tasks. We introduce comprehensive benchmark for assessing models across all aforementioned tasks and present the performance of multiple baseline models as part of our evaluation. Our dataset provides strong foundation for future research in pedestrian trajectory prediction, which could expedite the development safe and agile social navigation robots. The SiT dataset, development kit, and trained models publicly available https///pplbbooroooggggtbbbo/oo
Social Interactive
Introduction
With advances in technology in a variety of fields, social navigation robots are emerging as versatile vehicles that can provide variety of services, including last-mile delivery, street cleaning, manufac- turing, and security patrols. These mobile robots need to move safely and smoothly while interacting with humans in shared ways without collisions. A popular method of collision avoidance to create an occupancy grid map and represent the location of obstacles on the map. In complex and crowded
"qqual contribution.
t "orrespondiigg author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Ciouad
Pedestrton
Car
Rabott
Person Try.
Future of Invtance
Try. Rabot
Figure 1: Visualization of pedestrian trajectory data collected from diverse social interactive envi- ronments in outdoor scene (Cafe_Streeee Street_3t The first row showcases five different cameras, collectively capturing a full 360-degree view. In the lower section, point clouds, the mobile robot's trajectory, individual object trajectories, and their respective bounding boxes are displayed.
settings, traditional mobile robots often move at slow pace and behave passively because they perceive people as obstacles and adopt conservative planning strategies to ensure safe distance. For agile and safe navigation, robots need to perceive humans individual entities in three-dimensional space, predict their movements, and utilize these predictions to avoid collisions, plan optimal paths, and navigate smoothly.
Recent advances deep learning have led to significant improvements in performance for perceiving and predicting the motions of dynamic objects in diverse contexts. Several datasets have been made available to develop models designed specifically for predicting future human trajectories. While Argoverse S 322 nuScenes 4. and Waymo Open 277 datasets provide large-scale trajectory data capturing dynamic and interactive behaviors among agents, they were primarily designed within the context of autonomous driving scenarios, rather than for social navigation robots. ETH (22). UCY [16,, and SDD (24] provide trajectories of pedestrians, mainly utilized for benchmarks as human trajectory prediction tasks. However, these trajectory datasets were collected only from static top-view scenes captured by cameras positioned at high vantage points such as rooftops, buildings, drones hovering in the sky. Consequently, these datasets do not capture human-robot interactive scenarios. Furthermore, they do not provide raw sensor data with annotations required to train models for upstream tasks such as 3D object detection and multi-object tracking. STCrowd data p was collected from the sensors from a person-view perspective: however, the location of sensors also remained fixed position throughout the data collection process, resulting in lack of variation in the scenes across different samples. Consequently, these datasets fail to capture the dynamic nature of human-robot interaction. JRDB (20) provides the trajectory data collected in interactive scenarios, but the data were not arranged in a trajectory form, and the multiple sensor data were not fully synchronized time, limiting the potential of multi-sensor fusion.
Multiple studies in the field of human-roban interaction (HRI) have presented evidence revealing the impact of robot's actions pedestrians' walking behaviors [6 71 29 39). We aim to expand the research area for pedestrian perception by introducing the HRI-included dataset that reflects the interactive behaviors between pedestrians and mobile robot in the real world. In this paper, we present the Social Interactive Pedestrian Trajectory (SiT) dataset, which offers trajectories of pedestrians collected by a robot navigating in diverse range of socially interactive scenarios as shown in Figure I

---

Dataset
Platform
Sensor Data
Task
Sync.
Map
Location
UCY
16
Fixed
Top-view Cam
T, P
Outdoor
ETH
Fixed
Top-view Cam
T, P
Outdoor
SDD 2
Top-view Cam
P P
Outdoor
CITR-DUT
Fixed
Top-view Cam
Outdoor
nuSeenes
4
Vehicle
360-view Cam, LiDAR, IMU, RTK
D, T,
Outdoor
Waymo Open
M
Vehicle
multi-view Cam, LIDAR, IMU, RTK
D, T.
Outdoor
Argoverse
5
32
Vehicle
360-view Cam, LiDAR, IMU, RTK
D, T,
vt
Outdoor
JRDB 20
Robot
360-view Cam, LiDAR, IMU
D.T
Indoor&Outdoor
STCrowd
Fixed
Front-view Stereo Cam, LiDAR
D. T.
Outdoor
SiT(Ours)
Robot
360-view Cam, LiDAR, IMU, RTK D, T,
Indoor&Outdoor
Table Comparison of several pedestrian trajectory datasets. Dashes "-" represent attributes that are either inapplicable or unavailable. Sync. stands for Multi-Senson synchronization. D, T, and P mean detection, tracking, and prediction, respectively. E2E indicates whether the dataset supports the joint design of upstream tasks. (f) includes layti-l map information.
The SiT dataset consists of
Raw data acquired by two 3D scanning LiDARs, five cameras covering a 360-degree view, two inertial measurement units (IMUs), and real-time kinematic positioning (RTK)
2. 20 seconds of sequential data sampled at 10 Hz with indoor and outdoor scenes obtained from a mobile robot,
3. 9 seconds of trajectory data represented by vectors poses, namely (r, =)
4. Annotations for 2D & 3D bounding boxes and object IDs
5. Ego-motion of the robot obtained by RTK (for outdoor settings) and the simultaneous localization and mapping (SLAM) algorithm (for indoor settings)
6. Multi-laycred semantic maps obtained from LiDAR point clouds using the SLAM algorithm.
Our entire dataset comprises 60 scenes, totaling 60K images and 12K point cloud frames. This encompasses approximately 470K 2D annotations and 320K 3D annotations.
Table highlights the distinctive features of the SiT dataset compared existing human trajectory datasets. The main contributions of our dataset are summarized as follows.
Our SiT dataset provides large-scale real-world pedestrian trajectory data obtained through a robot's navigation in densely populated indoor and outdoor environments, including the interiors of buildings, campuses, crosswalks, public pedestrian roads, and more.
SiT dataset offers the flexibility design trajectory prediction models using various con- textual information including appearance features and ego-motion of robot, and semantic map data, which could not be supported the existing datasets.
SiT dataset supports precise time synchronization between multi-modal sensors using centralized sensor triggering method. This facilitates the development of efficient sensor fusion models tailored for perception and localization tasks.
SiT dataset provides semantic map data for both indoor and outdoor scenes, encompassing multi-layered scene information. This comprehensive semantic map data can enhance the capability of motion prediction models that use static scene contexts around the robot.
SiT dataset offers sequential raw data accompanied by 2D and box annotations to support joint modular design approach [12]. We have curated a benchmark for 3D object detection task, 3D multi-object tracking (MOT) task, and trajectory prediction task as well as an end-to-end prediction task that covers perception to motion forecasting.
SIT dataset is publicly available. The SiT dataset has the potential to open up research opportunities in the development of learning- 3D detection, 3D MOT, and trajectory prediction models in the context of social navigation robots.

---

NIGHT
Velodyne VIP-16 (tep)
FRONT_LER
REARLIEFT
Velodyne VLP-16 LIDAR [Bottom)
X-eois
Front
REar
Front
Z-aals
Base Unk
Downward
Upward to a-is
The Husky UGV platform (25] equipped with five cameras providing 360-view coverage, two 16-channel LiDARs, two IMUs, and RTK.
Figure 2:
Related Work
24 are popular pedestrian trajectory datasets that have been extensively utilized for developing pedestrian trajectory prediction models. These datasets were collected using static cameras positioned at elevated locations, capturing the (z, y) coordinates that represent the spatial locations of pedestrians. Hence, these datasets do not specifically cater to the development of perception and motion prediction models for social navigation robots. Recently, g has been introduced facilitate the development of pedestrian perception models by providing camera and LiDAR sensor data. However, the viewpoint of these sensors was also fixed, which led to a limited variation in the scenes captured across different samples. Consequently, the aforementioned datasets fail to capture the dynamic nature of human-robot interaction. Argoverse 32 nuScenes 44, and Waymo Open 27 datasets provide trajectories vehicles and pedestrians collected using sensors installed on moving vehicles. These datasets primarily emphasize capturing vehicle-to-vehille and vehicle-to- tehicle-to-pedeestrian interactions in driving scenarios. As a result, they may not be best suited to address the problems of social navigation robots. JRDB [20] offers camera and LiDAR sensor data collected from a mobile robot. However, it does not provide a benchmark specifically for the trajectory prediction task. Additionally, precise temporal synchronization between the camera and LiDAR data was not achieved in this dataset.
ETH [22]. UCY [16], and SDD
STCrowd
Table provides summary of the distinctive features offered by our SiT dataset in comparison to existing trajectory datasets. Our trajectory data is collected in scenarios where robot navigates through crowded indoor and outdoor areas, actively interacting with pedestrians. The SiT dataset provides raw sensor data accompanied by annotations necessary for developing the models spanning from pedestrian perception motion prediction. Furthermore, the SiT dataset achieves accurate time synchronization between different sensor modalities and provides semantic maps that describe the various scenes.
SiT Dataset
3.1 Robot Setup
We remotely operated Clearpath's Husky unmanned ground vehicle (UGV) platform 25 to collect different scenes of robot driving data. The robot has four large wheels that allow it to operate in variety of environments. As shown in Figure 2, the robot was equipped with the following sensors:
2 x Velodyne VIP-16 rotating 3D LiDARs with 16 channels, 0.09 degree angular resolution, 30 degree vertical field of view, 2 em distance accuracy, generating approximately 1.3 million points per second, and operating at 10 Hz.

---

(a) Hallway
(b) Cafeteria
(c) Outdoor Alley
(d) Crosswalk
Figure 3: We have collected data from variety of locations, both indoor and outdoor, that are densely populated with pedestrians.
5 Basler a2A 1920-51 ge PRO GigE, cameras capturing images a high resolution of 1920 x 1200, a field of view of 54.3 degree (vertical) x 79 degree (horizontal), and covering the entire 360-degree area at 10 Hz.
x MTi-680G IMU & RTK offering pose and global positioning data at 20 Hz 10 mm +1 ppm circular error probability (CEP), 0.5 degree RMS error of yaw, and 0.2 degree RMS error roll and pitch.
x VectorNav VN-100 IMU providing pose data 200 Hz with 1.0 degree RMS error of pitch and roll, and 2.0 degree RMS error of yaw.
Our mobile robot runs Ubuntu 18.04 and robot operating system (ROS) Melodic. The data captured by the aforementioned sensors were recorded in the rosbag file format. The sensor data is available in both rosbag and raw data formats. Additionally, both intrinsic and extrinsic calibration parameters are included.
Building
Car, rand
Crosswalk
Walkway
Sharedway
Road, sllpe
slope
Stadi obilacke
Stair
Previous studies have demonstrated that 3D object detection models based on multiple sensors produce more robust detection results than single sensor-based methods 2 17, 28
200m
33 37). Sensor fusion achieves max- imum benefits when the camera and LiDAR sensors are precisely synchro- nized in time. To ensure this synchro- nization, we employed pulse per sec- ond (PPS) signal generator to trigger two 3D LiDARs and five cameras. This method enables us to obtain accurately synchronized data from multiple cameras and LiDARs.
Figure 4: An example of a 12-layered semantic map from the outdoor scene. Distinct colors are used to represent different layers.
3.2
Data Collection
The data was collected the downtown area of Scoul, South Korea. We selected a variety of indoor and outdoor locations including hallways, cafeterias, university buildings, streets, and crosswalks, as shown in the examples in Figure 3) Our dataset contains a diverse and representative set of real-worle driving scenarios. We manually operated robot platform throughout densely populated areas. We directed the robot to adhere to social norms when interacting with pedestrians, ensuring avoided collisions and respected personal space.
3.3 Robot Localization
The robot's pose needs to be determined to compensate ego-motion for trajectory generation and utilize semantic maps for the motion prediction task. We used different localization methods for indoor and outdoor scenes. We used RTK to estimate the pose of the robot outdoor scenes. We implemented the LiDAAR-inertial SLAM algorithm described in 26 for indoor scenes. By leveraging these localization techniques, we obtained accurate absolute location information for pedestrians, facilitating object tracking and prediction tasks.
5

---

3.4 Semantic Map
Using scene context information has been proven beneficial in enhancing trajectory prediction models To facilitate this, the SiT dataset offers multi-layered semantic maps that encompass a wide range of scene information. The semantic maps were generated through a two-step process. Initially, point cloud maps were constructed, followed by manual segmentation using the ASSURE mapping tools [0O]. As shown in Figure these maps use a twelve-layer hicrarchical structure categorized by varying levels of detail.
13 13 [9].
3.5
Annotations
The SiT dataset contains annotations consisting of 2D bounding boxes of images captured by multi-view cameras covering 360- degree, and 3D cuboids expressed in 3D world coordinates. The 2D boxes were generated based on the existing 3D cuboids sharing object IDs. While our primary focus is to offer a pedestrian trajec- tory dataset, we have broadened the annotated classes to include car, bus, truck, cyclist, and motorcyclist. The distribution of class categories in our dataset is presented in Figure 5| To maintain high standard of annotations, frames taken at a 5HZ sampling rate were manually labeled by expert annotators. Subsequently, the labels were interpolated to higher frequency of 10Hz. This detailed gran- ularity is crucial to meet the real-time requirements of perception and prediction in navigation robots.
10a
of Instancees
10
100
pod Peda
Car
Cycllat
Bus
Truck
Motorcyclist
Figure 5: Histogram of 3D cuboids annotated for each class of objects present in the SiT Dataset.
Trajectory samples were produced by compensating for the ego- motion in the pedestrian tracks observed over certain duration.
3.6 Structure of Dataset
Train-Val Test
Train-Val
The SiT dataset was split into training, validation, and test subsets. We split the dataset so that each subset encom- passes different scenes. The distribu- tion of pedestrian distances from the robot and their velocities are shown in infFiuure 6) Consistent pedestrian be- haviors can be seen across the differ- ent subsets. Our dataset has been pro- cessed facilitate the training of com- prehensive end-to-end models cover- ing multiple tasks [12].
0.04
hormalteed Nitedd Denety
Nomal lomaaledd Denctty
1.0
0.03
9.4
0.01
20
40
60
0.0
0.5
1.0
1.3
2.0
Pedestrian Distance from Ego Robet [m]
Pedestrian Velocity [mm]
(a) Distance
(b) Velocity
Figure 6: Distribution of distance and velocity for Training- Valid and Test subsets.
3.7 Privacy
To prioritize privacy, all images have been processed to blur pedestrian faces and license plates. This careful curating ensures the responsible use of the data while maintaining the integrity of the information for research purposes. Detailed information on privacy concerns can be found in the Supplementary Material.
4 Data Analysis
In this section, we analyze the characteristics exhibited by the trajectory data in the SiT dataset. Figure 7]shows the distribution pedestrians' locations surrounding the robot in our dataset, where (0. 0) denotes the location of the robot or ego-vehicles. To provide basis for comparison, the distributions from two widely used self-driving datasets, Waymo Open 227 and nuScenes 4 datasets are also included. In the SiT dataset, pedestrians are predominantly located closer to the robot compared to other autonomous driving datasets. This can be attributed the fact that both the robot and pedestrians share the same path area. In contrast, in the Waymo Open and nuScenes datasets,

---

30
Forware Di Distance from Robot (ml
50
30
Forward Ditance from Robot
20
20
Forward Dietance from
10
10
10
Dense
Dense
Dense
10
-10
-10
20
-20
-20
30
-30
Sparea
Sparsa
Sparns
30 -30 Distance from Robot
20 Honzontal Distance Robot(m)
-30
-20
10 Robot(m)
Horizortal
(a) SiT (ours)
(b) Waymo Open
(c) nuScenes
Figure 7: Comparison of the spatial distribution of pedestrians relative to the ego-vehicle across the SiT, Waymo Open, and nuScenes datasets using kernel density estimates (KDEs). The red cross mark on the origin point (0, 0) in each plot is designated the center of the robot ego-vehicle.
Normalized Density
250k
Waymo
Waymo Open
0.03
Argoverse2
Instances
200k
Argoverse2
nuScenes
nuScenes
0.02
100k
0.01
50k
0
20
40
Space Sharing Proximity Sharing&Proxi.
Pedestrain Distance from Ego Vehicle [m]
Interaction Metrics
(a) Normalized Distance Density
(b) Pedestrian -gg Vchicle Interactions
Figure 8: Comparison with other autonomous vehicle datasets. (a) represents the normalized density graph of the distance between the robot (or ego-vehicle) and surrounding pedestrians, while (b) illustrates the number of instances that satisfy some criteria. Space sharing condition refers to instances where the robot and pedestrians occupy the same semantic space, Proximity condition indicates cases where the distance between the robot and pedestrian is within meters, and Sharing & Proxi. indicates cases that fulfill both conditions.
ego-vehicles and pedestrians typically occupy separate roads, resulting in fewer close interactions. observe that pedestrians are often positioned alongside ego-vehicles, rather than directly in front or behind them. Conversely, in our dataset, pedestrians can be found surrounding the robot from all angles.
Figure 8 demonstrates other attributes of the SiT dataset. Fiigure shows the distribution of the distance of pedestrians from the robot. This confirms that our dataset, the robot engaged in close-distance interactions with pedestrians. Figure 8b compares the number of trajectory instances that satisfy two conditions; 1) space sharing and 2) proximity. Space sharing condition indicates cases where the robot and pedestrian occupy the same semantic domain. We check this condition by examining the areas the semantic map where the robot and pedestrians are located. Typically, when the robot traverses roadway while pedestrian walks along sidewalk, their behaviors tend to remain unaffected by each other due to the physical separation of their paths. On other hand, interactive scenarios arise when they occupy the same semantic space. Proximity condition refers to cases where the distance between the robot and pedestrian is within 2 meters, indicating close physical proximity. Figure 8b/show, that the SiT dataset exhibits significantly higher number of instances that satisfy these two conditions compared to the other three autonomous driving datasets. This indicates that trajectory datasets designed for autonomous driving may adequately represent the human-robot interactions for social navigation robots.

---

Methods
Modality
mAP
AP(0.25) 1
AP(0.5) f
AP(1.0) + AP(2.0) f
FCOS3D
30
Camera
0.244
0.024
0.159
0.329
0.463
PointPillars
14
LiDAR
0.351
0.260
0.354
0.374
0.418
Centerpoint-P
LiDAR
0.414
0.300
0.424
0.446
0.486
Centerpoint-V 36
LiDAR
0,518
0.397
0.531
0.553
0.592
TransFusion-P LiDAR+Camera
0.390
0.248
0.371
0.437
0.507
TransFusion-V LiDAR+Camera
0.531
0.318
0.536
0.607
0.665
Table 2: Evaluation of 3D pedestrian detection baselines. mAP is obtained from the average of AP (0.25), AP (0.5), AP (1.0), and AP (2.0). f the higher, the better.
Methods
SAMOTA t AMOTA AMOTP(m) + MOTA T MOTP(m) + IDS
PointPillars
[4]
AB3DMOT 31
0.4110
0.1047
0.3580
0.4086
1.0277
1048
Centerpoint Detector
3G
AB3DMOT [1
0.4841
0.1398
0.3958
0.4586
0.9836
554
Centerpoint Tracker
16
0.6070
0.2007
0.2679
0.4760
0.5140
1136
Table 3: Evaluation of 3D pedestrian tracking baselines. t: the higher, the better. 4: the lower, the better.
Benchmarks and Experiments
In this section, we introduce the benchmarks designed for evaluating various 3D perception and motion prediction models. We also present the performance results of several baseline models on these benchmarks.
5.1 Benchmarks
Detailed information on the computation of these performance metrics can be found in the Supple- mentary Material.
3D Pedestrian Detection. The SiT dataset offers 3D pedestrian detection benchmark. The models generate 3D cuboids that encompass pedestrians in the scene for a given input frame. To evaluate the performance of 3D object detectors, mean average precision (mAP) metrie is widely used. The success of detection can be determined based on different criteria. For example, KITTI dataset [II] used intersection over Union (IoU) metric while nuScenes dataset [4] used the distance between two cuboid centers. In the SiT dataset, we adopt AP based on the distance metric using the thresholds of 0.25, 0.5, 1.0, and 2.0m.
3D Pedestrian Tracking. The SiT dataset offers the benchmark for evaluating 3D pedestrian tracking algorithms. Given sequence of data frames, the 3D pedestrian tracking models generate object tracks that indicate the pedestrians' movements over time. We adopt widely used performance metrics including SAMOTA, AMOTA, AMOTP, MOTA, MOTP, and identity switches (IDS) B 31 to evaluate the performance of tracking models.
The pedestrian trajectory prediction benchmark is presented to evaluate the accuracy of trajectory prediction. Average displacement error (ADE) and final displacement error (FDE) are widely used as evaluation metrics [ 22) ADE represents the average error in predicted trajectories compared to ground truth over all time steps, and FDE quantifies the deviation between the predicted final position and ground truth at the last time step. Using the preceding 2-second trajectory as input, the trajectory prediction models generate a set of K 7-second future trajectories, representing K distinct modes of future trajectory distribution. To evaluate the prediction accuracy of these K trajectories, we use ADEx and FDEK, which represent the minimum ADE and FDE values among the K trajectory candidates.
Pedestrian Trajectory Prediction.
End-to-BBl End Motion Prediction. the end-to-end motion prediction task, the models take raw sensor data input and generate future 3D bounding boxes, accompanied by the corresponding object trajectories. We consider a 7-second future horizon for trajectory prediction. use mAP, metric proposed in FutureDet [23] along with the conventional ADE and FDE as performance metrics.

---

Methods
Map
ADEs
FDE
ADE
FD
Methods
mAP 1 mAPr f ADE; & FDE, 4
Social-LSIM
1.638
3,121
1.630
3.103
FaF [8
0.490
0.079
1.915
3.273
Y-Not 19
1.527
2,802
0.836
1,878
FutureDet-P
23
0.209
0.037
2.532
4,537
Y.Net 19
1,361
2.624
0.675
1.547
FutureDet- 23
0.408
0.053
2,416
4.409
NSP-SFM 35
1.346
1,887
NSP-SFM
1.061
1818
0.517
0.925
Table 5: Evaluation of end-to-end motion predic- tion baselines. t the higher, the better. J: the lower, the better.
Table 4: Evaluation of pedestrian trajectory prediction baselines based on ADES, FDEE, ADE and FDE U: the lower, the better.
5.2 Performance of Baseline Models
In this section, we present the performance of several baseline models for each benchmark. All experiments were performed on 4 NVIDIA RTX3090 GPUs with 2 Intel Xeon CPUs. Detailed configurations of the baseline models can be found in the Supplementary Material.
3D Pedestrian Detection, Table 2 presents the performance of four baseline methods, FCOS3D [30. PointPillars [14), CenterPoint [36]. and TransFusion 2 evaluated in 3D Pedestrian Detection benchmark. CenterPoint-P and TransFusion-P utilized the PointPillars as encoding backbone, while CenterPoint-V and TransFusion-V employed SECOND 34 as the voxel encoding backbone, Note that camera-LiDAR fusion yields better performance compared to models relying on single modality,
3D Pedestrian Tracking. We considered AB3DMOT 311 and the CenterPoint tracker the baseline models for 3D MOT. We applied AB3DMOT to PointPillar and CenterPoint detectors. Table presents the performance of the baselines evaluated on the SiT dataset.
Pedestrian Trajectory Prediction. Table 4 presents the performance of Social-LSTTM D. Y-Net 11). and NSP-SFM (38). For Y-Net and NSP-SFM, the scene encoding was conducted utilizing the semantic maps offered from our dataset. In Table 4 we confirm the benefit of utilizing scene contexts for trajectory prediction.
presents the performance of two end-to-end motion prediction baseline models: Fast and Furious 18] and FutureDet 23 FutureDet-P uses PointPillars as the encoding backbone while FutureDet- utilizes SECOND as the voxel encoding backbone.
End-to-end Motion Prediction. Table
6 Conclusions
In this paper, we introduced the SiT dataset, a new pedestrian trajectory dataset designed to facilitate the development of perception and motion prediction models for social navigation robots. The SiT dataset comprises pedestrian trajectories gathered from a variety of social interaction scenarios that can be encountered by robots during navigation in the real world. Our analysis revealed that the SiT dataset effectively captured the behavior of pedestrians in close proximity to the robot within a shared semantic space the map. The SiT dataset also offered high-quality annotations for 2D and 3D object detection and tracking, enabling the design of end-to-cnd motion prediction models the upstream pipeline. These features make the SiT dataset unique when compared existing pedestrian trajectory datasets. We hope that our proposed dataset and baselines provide a strong foundation for future research in pedestrian perception and have the potential accelerate the development of fully autonomous robots.
One limitation of our dataset is that we were unable to scale the size of the dataset up to hundreds of thousands of frames due time constraints. However, we are planning to address this limitation by adding an additional dataset in the second phase release, called SiT dataset v2, to significantly expand the scale of our dataset.

---

Acknowledgment
This work was partly supported by 1) National Research Foundation of Korea(NRF) grant funded by the Korea gOVIT (No.2020R|) A2C201122)))) 2) Institute of Information & commu- nications Technology Planning & Evaluation (ITP)) grant funded by the Korea goVerament(MSIT) (No N000000000 Artificial Intelligence Graduate School Program (Hanyang University) and 3) No. No.2022-0-00957, Distributed on-chip memory-proceseee model PIM (Processor in Memory) semi- conductor technology development for edge applications), and 4) Hyundai Motor Company,
References
[1] A. Alahi, K. Goel, V. Ramanathan, Robiequet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961-971, 2016.
[2] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L.. Tai. TransFusion: Robust LiDAR- camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1090-1099, 2022.
[3] K. Bernardin, A. Elbs, and R. Stiefelhagen. Multiple object tracking performance metrics and evaluation in a smart room environment. In Sixth IEEE International Workshop on Visual Surveillance, in conjunction with ECCV, volume 90, Citeseer, 2006.
[4] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, Baldan, and O. Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11621-11631, 2020.
[5] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and Hays. Argoverse: 3D tracking and forecasting with rich maps. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8748-8757, 2019.
[6] Y. Chen, Y. Luo, C. Yang, M. O. Yerebakan, S. Hao, N. Grimaldi, S. Li, R. Hayes, and B. Hu. Human mobile robot interaction in the retail environment. Scientific Duta, 9(1)663, 2022.
[7] Z. Chen, C. Jiang, and Y. Guo. Pedestrian-robot interaction experiments in exit corridor. In Proceedings of the International Conference on Ubiquitous Robots (UR), pages 29-34. IEEE, 2018.
[8] Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, and F. Zhao. Deformable feature aggregation for dynamic multi-modal 3D object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 628 644.. Springer, 2022.
[9] P. Cong, X. Zhu, F. Qiao, Y. Ren, X. Peng, Y. Hou, L. Xu, R. Yang, D. Manocha, and Ma. STCrowd: A multimodal dataset for pedestrian perception in crowded scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19608-19617, 2022.
[10] H. Darweesh. Assure mapping tools.
https: bttps://githut http:///hhuu.cccmaaaaeaaaaaeeeee
sssuremappingtools 2021.
[11] A. Geiger, P. Lenz, and R. Urtasun. Are ready for autonomous driving? the KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognittin CVPR)) pages 3354-3361, 2012.
[12] Y. Hu, J. Yang, I.. Chen, K. Li, C. Sima, X. Zhu, Chai, S. Du, T. Lin, W. Wang, et al. Planning- oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17853-17862, 2023.
[13] B. Kim, S. H. Park, S. Lee, E. Khoshimjonov, D. Kum, J. Kim, J. S. Kim, and J. W. Choi. LaPred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition (CVPR), pages 14636-14645 2021.
10

---

[14] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. PointPillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12697-12705, 2019.
[15] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker. DESIRE: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 336-345, 2017.
[16] A. Lerner, Chrysanthou, and D. Lischinski. Crowds by example. Computer Graphics Forum, 26(3):655-60 664,, 2007.
[17] Z. Liu, H. Tang, A. Amini, X. Yang. H. Mao, D. Rus, and S. Han. BEVFusion: Multi-task multi- sensor fusion with unified s-eye view representation. arXiv preprint arXiv:2205.13542. 2022.
[18] W. Luo, B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3D detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3569-3577, 2018.
[19] K. Mangalam, Y. An, H. Girase, and J. Malik. From goals, waypoints & paths long term human trajectory forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15233-15242, 2021.
[20] R. Martin, M. Patel, H. Rezatofighi, A. Shenoi, J. Gwak, E. Frankel, A. Sadeghian, and S. Savarese, JRDB: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 45(6):6748-6775, 2021.
[21] C. Mavrogiannis, A. M. Hutchinson, J. Macdonald, P. -liveiOl and R. A. Knepper. Effects of distinct robot navigation strategies on human behavior in crowded environment. In Proceedings of the ACM/IEEE International Conference on Human-Robal Interaction (HRI), pages 421-430, 2019.
[22] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You'l '1l never walk alone: Modeling social behavior for multi-target tracking. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 261-268, 2009.
[23] N. Peri, J. Luiten, M. Li, A. Ošep. L. Taixé, and D. Ramanan. Forecasting from LiDAR via future object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17202-17211, 2022.
[24] A. Robiequet, Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 549-565. Springer, 2016.
[25] C. Robotics. Husky-unmanned ground vehicle. Technical Specifications, Clearpath Robotics, Kitcener, Ontario, Canada, 2013.
[26] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and R. Daniela. LIO-SAM: Tightly- coupled LiDAR inertial odometry via smoothing and mapping. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5135-5144 5142, 2020.
[27] Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2446-24 2204-22544424 2020.
[28] S. Vora, A. H. Lang, B. Helou, and O. Beijbom. PointPainting: Sequential fusion for 3D object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4604 4612, 2020.
[29] M. L. Walters, K. Dautenhahn, K. Koay, C. Kaouri, S. Woods, C. Nehaniv, R. te Boekhorst, D. Lee, and I. Werry. The influence of subjects' personality traits on predicting comfortable human-robot approach distances'. In Proceedings of Cog Sci 2005 Workshop: Toward Social Mechanisms of Android Science, pages 29-37,11 2005.

---

[30] T. Wang, X. Zhu, J. Pang, and D. Lin. FCOS3D: Fully convolutional one- monocular 3D object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 913-922, 2021.
[31] X. Weng, J. Wang, D. Held, and K. Kitani. 3D multi-object tracking: A baseline and new evaluation metrics. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10359-10366 2020.
[32] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2021.
[33] S. Xu, D. Zhou, J. Fang, J. Yin, Z. Bin, and L. Zhang. FusionPainting: Multimodal fusion with adaptive attention for 3D object detection. Proceedings of the IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3047-3054, 2021.
[34] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018.
[35] D. Yang. L. Li, K. Redmill, and Ü. Özgüner. Top-view trajectories: A pedestrian dataset of vehicle-crowd interaction from controlled experiments and crowded campus. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), pages 899-904, 2019.
[36] T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3D object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17784-1179,, 2021.
[37] J. H. Yoo, Y. Kim, J. Kim, and J. W. Choi. 3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736. Springer, 2020.
[38] J. Yue, D. Manocha, and H. Wang. Human trajectory prediction via neural social physics. In Proceedings of the European Conference on Computer Vision (ECCV), pages 376-394. Springer, 2022.
[39] B. Zhang, Amirian, H. Eberle, J. Pettré, C. Holloway, and T. Carlson. From HRI CRI: Crowd robot anteraction-undersaaniin the effect of robots on crowd motion: Empirical study of pedestrian dynamics with wheelchair and pepper robot. International Journal of Social Robotics, 14:631-660 2021.
12


Reviews:
[{"review_id": "W9Em20TwHa", "reviewer": "Reviewer_UimK", "summary": "", "questions": "", "limitations": "L294-297 discuss the challenges of scaling up the dataset. It would be beneficial to further explore limitations, such as privacy concerns and biases in real-world data collection.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset consists of real-world pedestrian trajectory data **collected by a real robot** navigating both indoor and outdoor environments.  \nIt also includes semantic map data, providing flexibility for designing trajectory prediction models.  \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LzpXeFFBWZ", "reviewer": "Reviewer_b9eM", "summary": "", "questions": "", "limitations": "For the prediction task, I believe there needs to be a stronger justification against HTP datasets. The fixed perspective is not limiting within the scope of the prediction task and these datasets provide valuable higher-density scenarios for learning to predict human trajectories.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors comprehensively justify the need for their dataset, provide a thorough analysis of the dataset's scenes, and report the performance of baseline models on their benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "oSoEYc2CQP", "reviewer": "Reviewer_YC2B", "summary": "", "questions": "", "limitations": "The authors have outlined their work, but they have not explicitly addressed all of its potential limitations.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The SiT dataset significantly advances the field of social navigation robots by providing a rich resource for training and evaluating models that perceive and predict pedestrian trajectories. Unlike existing datasets that are based on static scenes, SiT captures the dynamic human-robot interactive scenarios in motion, better mimicking real-world applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sbxrO3C212", "reviewer": "Reviewer_qffA", "summary": "", "questions": "", "limitations": "1. This paper contains real-world data. This reviewer may concern about the potential ethical issues during the data collection.\n\n2. The authors mentioned it is hard to scale the size of the dataset up to hundreds of thousands of frames due to the time constraints but they would add more data in the future. ", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper effectively addresses the research gap in pedestrian trajectory forecasting by introducing a valuable dataset of social navigation robot scenarios. The unique setting, where the robot is in motion while pedestrians are widely dispersed (as shown in Fig 7.), makes it distinct from common autonomous driving datasets like Waymo and Argoverse or surveillance video datasets like ETH/UCY. The significance of social navigation robots in trajectory forecasting and tracking systems further underscores the importance of this dataset. I am glad to see the authors collect and annotate data for this specific scenario.\n\n2. In addition to its strong motivation, the dataset exhibits high-quality characteristics. The inclusion of multi-modal sensors, diverse locations, and semantic maps aligns well with the research community's interests. The authors' provision of benchmark results enhances the dataset's potential impact on future research in related areas.\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "XpZQgLmnyT", "reviewer": "Reviewer_RCVX", "summary": "", "questions": "", "limitations": "The authors have addressed potential negative societal impacts. As for the dataset size limitation, the authors promise they will enlarge the size in the future V2 version.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "It provides a valuable resource for developing perception and motion prediction models for social navigation robots. The paper is well-organized and provides a thorough evaluation of the dataset. The SiT dataset has the potential to open up research opportunities in the field of social navigation robots and enable the development of more effective social navigation robots.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work introduces the Social Interactive Trajectory dataset, designed for detecting, tracking, and predicting pedestrian trajectories in real-world scenarios. Collected by a robot navigating diverse indoor and outdoor environments, the dataset also incorporates a benchmark for related tasks and offers baseline measurements.

In the rebuttal process, several reviewers raised several concerns, including privacy consideration. The authors have adequately addressed these issues. Consequently, all reviewers agree to accept.
Decision: Accept (Poster)