Title: Pedestrian-Centric 3D Pre-collision Pose and Shape Estimation from Dashcam Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Pedestrian Pre-collision pose

Human pose and shape estimation

Dashcam Perspective

Pedestrian-Vehicle Collision Pose dataset
Abstract: Pedestrian pre-collision pose is one of the key factors to determine the degree of pedestrian-vehicle injury in collision. Human pose estimation algorithm is an effective method to estimate pedestrian emergency pose from accident video. However, the pose estimation model trained by the existing daily human pose datasets has poor robustness under specific poses such as pedestrian pre-collision pose, and it is difficult to obtain human pose datasets in the wild scenes, especially lacking scarce data such as pedestrian pre-collision pose in traffic scenes. In this paper, we collect pedestrian-vehicle collision pose from the dashcam perspective of dashcam and construct the first Pedestrian-Vehicle Collision Pose dataset (PVCP) in a semi-automatic way, including 40k+ accident frames and 20K+ pedestrian pre-collision pose annotation (2D, 3D, Mesh). Further, we construct a Pedestrian Pre-collision Pose Estimation Network (PPSENet) to estimate the collision pose and shape sequence of pedestrians from pedestrian-vehicle accident videos. The PPSENet first estimates the 2D pose from the image (Image to Pose, ITP) and then lifts the 2D pose to 3D mesh (Pose to Mesh, PTM). Due to the small size of the dataset, we introduce a pre-training model that learns the human pose prior on a large number of pose datasets, and use iterative regression to estimate the pre-collision pose and shape of pedestrians. Further, we classify the pre-collision pose sequence and introduce pose class loss, which achieves the best accuracy compared with the existing relevant \textit{state-of-the-art} methods. Code and data are available for research at https://github.com/wmj142326/PVCP.

Pedestrian-Centric 3D Pre-collision Pose and Shape Estimation from Dashcam Perspective

Abstract

Pedestrian pre-collision pose is one of the key factors to determine the degree of pedestrian-vehicle injury in collision. Human pose estimation algorithm is an effective method to estimate pedestrian emergency pose from accident video. However, the pose estimation model trained by the existing daily human pose datasets has poor robustness under specific poses such as pedestrian pre-collision pose, and it is difficult to obtain human pose datasets in the wild scenes, especially lacking scarce data such as pedestrian pre-collision pose in traffic scenes. In this paper, we collect pedestrian-vehicle collision pose from the dashcam perspective of dashcam and construct the first Pedestrian-Vehicle Collision Pose dataset (PVCP) in a semi-automatic way, including 40k+ accident frames and 20K+ pedestrian pre-collision pose annotation (2D, 3D, Mesh). Further, we construct a Pedestrian Pre-collision Pose Estimation Network (PPSENet) to estimate the collision pose and shape sequence of pedestrians from pedestrian-vehicle accident videos. The PPSENet first estimates the 2D pose from the image (Image to Pose, ITP) and then lifts the 2D pose to 3D mesh (Pose to Mesh, PTM). Due to the small size of the dataset, we introduce a pre-training model that learns the human pose prior on a large number of pose datasets, and use iterative regression to estimate the pre-collision pose and shape of pedestrians. Further, we classify the pre-collision pose sequence and introduce pose class loss, which achieves the best accuracy compared with the existing relevant state-of-the-art methods. Code and data are available for research at https://github.com/wmj142326/PVCP.

1 Introduction

Pedestrian pre-collision pose refers to the emergency actions pedestrians take when facing potential hazards before collision accidents, affecting both the severity of pedestrian injuries and subsequent injury assessments (1). Using collision simulation software to reconstruct pedestrian-vehicle accidents is a popular and effective method for analyzing pedestrian injuries (2; 3; 4; 5). However, the current input for initial pose still relies on predefined gait sequence templates (6; 2; 3; 4; 5) or manual measurement of pose angles from accident images, the former cannot represent the posture of pedestrians in real accidents, and the latter is inefficient. Computer vision-based pedestrian pose estimation methods can directly estimate pose information such as joint positions or limb angles from images in real-time (7; 8; 9; 10; 11; 12; 13). Existing pose estimation methods are trained and applied on multiple datasets for various scenarios, adapting to different downstream tasks. However, unlike common poses, pedestrian pre-collision poses in traffic scenes are specific, with differences in spatial-

temporal characteristics and challenges such as dynamic backgrounds, sudden scene changes, and occlusions of lower limbs (14; 15). Directly applying existing algorithms to pedestrian pre-collision pose estimation in traffic scenarios does not achieve perfect adaptation effects.

Training a proprietary network with specific datasets can effectively improve the pose estimation performance of the network in that scenario. Precise 2D pose annotations can be obtained through time and manpower-intensive efforts, indoor motion capture (Mocap) systems (16; 17) utilize markers and sensors to acquire high-quality 3D motion data. However, acquiring ground truth (GT) 3D joint positions in-the-wild is nearly impossible (12). Existing in-the-wild datasets either do not contain human pose (18; 19) labels or only include limited movements of daily activities (20; 21; 14). Furthermore, training a model using a large amount of data is costly, and video data of pedestrianvehicle collisions belongs to small sample scarce data in traffic scenes, making dataset collection difficult. Dashcams or public surveillance devices are the only sources of data (22; 23; 24; 15), further constrains the approaches to dataset creation, thereby enhancing the difficulty and complexity of producing such datasets.

In this work, we constructed a Pedestrian-Vehicle Pre-collision Pose (PVCP) dataset and proposed a simple framework for Pedestrian Pre-collision Pose and Shape Estimation (PPSENet) in collision accident videos. We collected dashcam videos and used existing pose estimation algorithms to obtain rough 2D keypoints and 3D mesh initialization results, followed by manual correction using specialized annotation tools. Specifically, we designed an SMPL annotation tool (25) to align the initial results with image contours, resulting in approximately 40K+ frames of accident images and 20K+ instances of pedestrian emergency poses with both 2D and 3D annotations. Our PPSENet estimates the 2D pose from images (Image to Pose, ITP) and lifts the 2D pose to the 3D mesh (Pose to Mesh, PTM). We used a pre-trained model (12) to capture prior knowledge of human actions and employed iterative regression (26; 11; 27) to estimate pedestrian pre-collision poses and shapes. Additionally, we classified emergency poses and introduced pose class loss, achieving superior accuracy compared to existing methods.

The main contributions of this paper are summarized below:

We constructed a pedestrian pre-collision pose dataset, PVCP, by collecting dashcam videos of pedestrian-vehicle collisions. Through algorithm initialization and manual annotation, we obtained rich pose representation annotations, including 2D, 3D keypoints and SMPL mesh.

We propose a two-stage pedestrian pre-collision pose and shape estimation network, PPSENet, which first estimates the 2D pose from the image and then lifts the 2D pose to the 3D pose. A pretrained encoder with pose estimation and an iterative regression decoder are combined, and introduce a collision pose class loss.

Our framework achieved promising results on the PVCP dataset, outperforming other methods of human pose estimation. This provides both data and algorithmic support for pedestrian pre-collision pose estimation and active safety protection for pedestrians.

2 Related Work

Pedestrian Pre-collision Pose. Pedestrian pre-collision pose is crucial for studying collision damage, as the initial posture at the time of impact directly affects the severity and nature of the injuries (28; 6; 29; 30; 3; 31). Early studies estimated collision poses by collecting post-accident data from pedestrians and vehicles (32). Cadaver tests (33) became effective for biomechanical damage studies but are limited by ethics, sample size, and high costs. Currently, collision simulation software is the most convenient and effective method to assess damage under various poses (2; 3; 4; 5). However, initial collision poses are often fixed templates or simple categories (6; 2; 3; 4; 5), differing significantly from real pre-collision poses. One method to obtain pre-collision poses is using motion capture in virtual environments with volunteers (34), but this is limited by device constraints and lack of real danger. Another method captures collision sequences from real accident videos, manually measuring posture angles or adjusting dummies to match real collision poses (33). This method is closer to real accident scenarios but is labor-intensive, time-consuming, and lacks standardized testing, limiting its use to single accident reconstructions. With advances in deep learning and computer vision, some research has employed human pose estimation algorithms to automatically extract collision poses from accident images (15), providing a new approach to acquiring pre-collision poses

in real accidents. Rapid and accurate acquisition of pedestrian pre-collision poses supports research on collision damage and active safety protection.

Human Pose Estimation. Human Pose Estimation (HPE) is a fundamental task of computer vision, which aims to obtain human pose information such as joint positions and angle from images and videos (35). It can be simply classified into 2D human pose estimation and 3D human pose estimation. 2D HPE regresses pixel coordinates ( x, y ) of joints, while 3D HPE includes depth to obtain threedimensional coordinates ( x, y, z ) (7; 8; 36). Though 3D coordinates can be regressed directly from images (37; 38; 39; 40; 41), using 2D pose as intermediate supervision before lifting to 3D often achieves higher accuracy (9; 42). Additionally, the SMPL (Skinned Multi-Person Linear Model) (43) has gained popularity for providing pose and morphological information, along with prior knowledge of body structure, avoiding issues with limb length changes (44; 11; 12; 27). This rotation-based model is particularly useful in biomechanical research (15; 45), which benefits the study of pedestrian emergency poses. In our research, we used 2D-to-3D lifting to estimate the pre-collision pose and shape of pedestrians from real accident videos.

Accident and Pedestrian Datasets. Collecting 3D pose datasets in complex traffic scenes poses challenges due to the dynamic environments and uncertain pedestrian poses. While existing largescale datasets focus on 2D poses(20; 51; 52), 3D pose datasets are often confined to indoor settings using Motion Capture (Mocap) systems(16; 17) or estimated via Inertial Measurement Units (IMUs) for outdoor poses(21). Models trained on indoor datasets do not adapt well to other in-the-wild tasks. Advanced pose estimation methods can generate pseudo-datasets to construct 3D pose datasets in the wild. Although pseudo-3D labels from semi-automatic(44; 53) or fully-automated methods(54; 55) are less accurate than Mocap data and may contain noise, they significantly improve regression-based methods(56). Using a semi-automatic method, we collected dashcam videos of collisions to create a pedestrian-vehicle collision pose dataset, offering rich annotation information and contributing to pedestrian protection tasks. Table 1 highlights its advantages over other datasets.

3 PVCP Dataset

3.1 Data Collection

Dashcams or public surveillance devices are the main sources of crash data (22; 23; 24; 15). Dashcam views dominate vision-based Traffic Accident Anticipation (TAA) datasets due to the high potential for collision avoidance through vehicle control (57). Our PVCP dataset are all derived from the vehicular perspective of dashcam, and videos are sourced from two primary origins. Similar to previous works (47; 22; 23; 24; 15), we collected videos of pedestrian and vehicle collisions from online platforms such as YouTube, using 'pedestrian-vehicle collision' as a keyword. In addition, a small number of videos are derived from existing open-source traffic datasets (47; 22; 23), which were primarily developed for tasks related to driver attention and the prediction of sudden accidents. All of the collected videos were reduced to individual accident footage, recording a complete pedestrian

H36M

COCO

PVCP

20

10

0

-10

-20

-20

-10

0

10

20

30

H36M

COCO

PVCP

25

20

15

10

0

5

-5

-10

-15

-10

-5

0

5

10

15

20

(c) Pose and Shape parameters distribution

(b) Pre-collision pose class definition

(a) PVCP dataset annotation pipeline

Pose

Shape

B

β

+

and vehicle collision, resulting in 209 pedestrian-vehicle accident videos, totaling 42,511 frames of images and about 19,533 pre-collision poses.

3.2 Data Annotations

For the application of pedestrian-vehicle collision accident reconstruction and vehicle active protection in traffic collision scenarios, PVCP provides a rich pedestrian pose representation, including pedestrian clipping images, pedestrian Bounding Box, track id, 2D and 3D keypoints and SMPL mesh label. The entire annotation pipeline is shown in Figure 1(a).

Pedestrian Pre-collision Pose. Manually annotating human keypoints is tedious and labor-intensive. We address this by initializing the annotation process with pose estimation models and refining the results manually. First, we identify collision-involved pedestrians in each video using a tracking network (59) and manual filtering, followed by pose annotation. We use a 15 keypoint representation similar to the JHMDB dataset (60) for efficient pose depiction. ViTPose (10) provides rough 2D pose annotations, which we manually adjust for accuracy. For occluded limbs, we estimate positions to complete the 2D skeleton, excluding body parts beyond the frame. Annotating 3D human poses in-the-wild remains challenging with images as the only source. Unlike Pseudo-GT annotators (61; 62), we use SPIN (11) to initialize predictions from cropped images, then refine the SMPL model parameters θ ∈ R 24 × 3 and β ∈ R 10 using our specially designed SMPL annotation tool (25) for better pixel alignment. This yields mesh pose annotations M ( θ, β ) ∈ R 6890 × 3 for pre-collision pedestrians. Finally, we apply the pre-defined joint regression matrix J ∈ R J × 6890 (44) to obtain 3D keypoints X 3 D ∈ R J × 3 from J ( M ( θ, β )) , where J = 17 (16), as shown in Figure 1(a).

Pedestrian Motion Class. Throughout the course of a pedestrian-vehicle collision event, pedestrians often undergo a series of rapid evasive action changes in a short period. Effectively and accurately distinguishing and predicting these imminent changes is crucial for the proactive safety features of vehicle driving systems. Through the observation and analysis of all collected accident videos, as shown in Figure 1(b), we categorize pedestrian behaviors in collision sequences into four types:

Normal pose : Represents the pedestrian's pose under normal, non-emergency conditions. This includes upright body pose and natural stances, reflecting the general behavior of pedestrians when not faced with emergencies.

Run pose: Characterized by the pedestrian's body leaning forward with rapid alternation of arms and legs, this pose is an active measure to prevent vehicle collisions. It reflects a preemptive action to swiftly move away from potential threats, serving as a strategic pose to avoid accidents.

Avoid pose: This pose is adopted by pedestrians upon detecting an imminent collision or other emergency situations. It includes potential actions such as jumping, quickly turning around, and dodging, reflecting the emergency response of pedestrians when recognizing potential danger.

Image

Spatial

Attention

Temporal

Attention

Spatial

Attention

Temporal

Attention

FC

2D Skeletons

Crop Bbox

Image

w

h

𝑿𝒄

…

…

MotionBERT Encoder

Backbone

PVCP

Pedestrian-Vehicle Collision Pose

Dataset

2D Skeletons

…

Feat pose

Feat shape

Pre-training

Image to Pose

Pose to Mesh

+

Head pose

Head shape

+

𝜃

𝛽

Iteration Decoder

N

×

N

×

Loss Function

Feature Extraction

ℒ𝑀𝑜𝑡𝑖𝑜𝑛

ℒ𝐶𝑙𝑎𝑠𝑠

ℒ𝑆𝑀𝑃𝐿

ℳ(𝜃,𝛽)

Head class

𝑐

ℐ

M

×

ℒ𝑘𝑝𝑡2𝑑

FC

𝒮1

𝒯 1

𝒯 2

𝒮2

FC

𝛼

Collision pose: Represents the pose of a pedestrian post-collision with a vehicle. It encompasses possible actions such as losing balance, falling, and sustaining injuries, reflecting the change in the physical state of pedestrians after a collision.

The sequences of two behavior annotations are illustrated in Figure 1(b). It is important to note that a complete collision incident does not necessarily encompass all four types of behavior, nor is there a fixed sequence. Because some accidents do not result in a final collision due to timely measures taken, and some pedestrians may not even be aware of the approaching vehicle. To distinguish between the four types, we employ a four-digit one-hot encoding for pose annotation. For some critical changes in pose, we use three annotators and take the majority's annotation result as the final Ground Truth.

3.3 Dataset attribute distributions

PVCP dataset contains the pose sequences during pedestrian collision. We visualized the pose and shape distribution of SMPL labels in PVCP dataset and compared them with those in commonly used and influential datasets in human pose estimation, including an indoor MoCap pose dataset Human3.6M (16), a in-the-wild dataset PW3D (21), a pseudo-3D labels dataset MSCOCO (20) and a traffic scenario pedestrian dataset PedX (14) are shown in the Figure 1(c). The distribution of pose parameters in our dataset is comparable to PW3D and significantly larger than that of everyday poses in MSCOCO. Our dataset contains over 200 individual collision pedestrians, and its shape parameter distribution is more extensive than other datasets that use a limited number of actors, such as 5 actors in the Human3.6M dataset. The most similar to PVCP is PedX, but its distribution is sparse due to its small number of poses. To sum up, our PVCP has good generalization in pose and shape parameters, representing pedestrian pose under a variety of collision conditions. Compared to existing pose datasets, PVCP demonstrates significant differences and advantages in terms of scene specificity, action space variation, and temporal continuity.

4 Network Architecture

In this section we present a pipeline for pedestrian Pre-collision Pose and Shape Estimation (PPSE), as shown in Figure 2, adopts a top-down two-stage strategy. Instead of estimating the pose of all pedestrians in the whole images, we pre-select the Bbox of collision pedestrians obtained by the detect and track networks (59). The crop of a single pedestrian was input into the Image to Pose (ITP) network to extract the image features of the collision pedestrian and estimate the pedestrian's 2D pose. Then, the 2D pose was lifted to 3D mesh through the Pose to Mesh (PTM) network.

4.1 Pedestrian Pre-collision Pose and Shape Estimation

Image to Pose . Estimating 2D human pose from images is a basic and mature task, and many works have achieved very effective results in different datasets. We take image frame I and corresponding

Bbox as input, simply use ResNet50 (63) as the backbone of feature extraction, and use a transposed convolution and a heat map regression head H as the 2D pose estimation network, which is also the classic paradigm of 2D pose estimation (8).

Pose to Mesh . PTM is a network architecture for 2D-to-3D lifting, we take the 2D pose sequence P L 2 D ∈ R T × J × C in as input. First, a Dual-stream Spatio-temporal Transformer (DSTformer) is used as an encoder to extract the spatio-temporal features of the pose sequence F m ∈ R T × J × C f . Then we designed an iterative regression decoder to obtain pose parameters and shape parameters respectively. At the same time, we added an additional regression head of pose class to coordinate with pose class loss to further improve the precision of pose regression. Where T represents the length of the pose sequence and J represents the number of pedestrian joints. J = 17 takes the Human3.6M dataset (16) joint format and we generate additional nodes from P 2 d ∈ R 15 × 2 .

Inspired (12), PTM uses the backbone of an pre-trained model, MotionBERT (12), and combines PVCP dataset to estimate pedestrian pre-collision pose in collision scenarios. MotionBERT is a pre-training model of human motion representations. Firstly, it learns the prior knowledge of human motion poses under the training of a large number of datasets, which is suitable for the further improvement of small datasets such as PVCP and special pose types such as pre-collision poses. Secondly, its training mode simulates the detection results by randomly masking and adding noise to 2D pose sequences. In the same way, we train the situation of vehicles shielding pedestrian's lower limbs in collision environment. This encoder stacks spatial and temporal Multi-Head Self-Attention (MHSA) blocks in different orders to form two parallel computation branches:

Where i ∈ 1 , . . . , M and M represents network depth. S i and T i represent Spatial MHSA and temporal MHSA of different depth layers, respectively. Adaptive fusion weights α ST , α TS fuses the output features of the two branches using adaptive weights predicted by an attention regressor. W f denotes linear layer, ⊕ denotes concatenation.

We divide the fusion feature F m ∈ R T × J × C f obtained by DSTFormer into three branches using different linear layers. That is, pose feature F θ ∈ R T × C h , shape features F β ∈ R T × C h and class features F c ∈ R T × C h . Then the three features are predicted by three different heads, outputting the pose parameter θ ∈ R T × 24 × 6 , shape parameter β ∈ R 10 and class probability c ∈ R T × 4 . Here, we use the 6D rotation representation to converge the pose parameter more quickly (64). Further, we adopted the idea of iterative regression to add the fusion features of pose and form with the predicted results and iteratively output:

Where k ∈ 1 , . . . , N and N represents the number of iterations. W θ , W β and W c are three linear transformations heads, ⊕ denotes concatenation, + denotes add.

4.2 Loss Function

The loss function of ITP, defined as the Mean Squared Error (MSE), is applied for comparing the predicted heatmaps ˆ H and the ground truth heatmaps H . The heatmap for joint k is generated by applying a 2D Gaussian centered on the k th joint's location.

The PTM loss function consists of three parts: SMPL loss, motion loss, and the pose class loss introduced by us. The final loss function is calculated as:

SMPL loss: The loss function of SMPL-based 3D human mesh usually consists of three parts:

Where L θ = ∥ ∥ ∥ ˆ θ -θ ∥ ∥ ∥ 1 , L β = ∥ ∥ ∥ ˆ β -β ∥ ∥ ∥ 1 , L norm = ∥ ∥ ∥ ˆ θ ∥ ∥ ∥ 2 + ∥ ∥ ∥ ˆ β ∥ ∥ ∥ 2 represents pose loss, shape loss and normalization loss respectively.

Motion Loss: The human body is a complex rigid structure interconnected between joints, and the continuous frame sequence of human body movements possesses certain temporal characteristics. Therefore, introducing motion loss as

Where L kp 3 D = ∥ ∥ ∥ ˆ X -X ∥ ∥ ∥ 1 represents the loss of 3D keypoints. L v = ∥ ∥ ∥ ˆ V -V ∥ ∥ ∥ 1 represents speed loss, and V = X t +1 -X t , ˆ V = ˆ X t +1 -ˆ X t .

Pose class loss: The pedestrian pose of the collision sequence has obvious categories, namely normal pose, running pose, avoiding pose and collision pose as described in Sec. 3.2, the class loss of pose is defined as follows:

where L Cross Entropy ( ˆ C,C ) epresents the cross entropy loss between the predicted pose class and the GT pose class. λ θ , λ β , λ n , λ k , λ v , λ c are the constants of the balance weight loss.

5 Experimental and Results

5.1 Dataset, Evaluation Metric and Implementation Details

Dataset. The PVCP dataset consists of over 20K+ pedestrian pre-collision poses, with 19,533 poses annotated with category labels. Subsequently, we selected 164 video sequences as the trainset and 45 video sequences as the testset. To ensure the effectiveness of the pose sequences, only poses with the number of keypoints N kpt ≥ 10 were selected, resulting in 15,458 poses for training ( Normal :7,912; Run :5,044; Avoid :2,289; Collision :213) and 5,503 poses for testing ( Normal :3,383; Run :1,431; Avoid :631; Collision :58). The entire training process only utilized the PVCP dataset, while the pre-training model weights were obtained from MotionBERT (12) trained on the AMASS (17), Human3.6M (16), PW3D (21), and MSCOCO (20) datasets.

Evaluation Metric. We evaluated the estimation of 3D human pose and shape using the following metrics: MPJPE( mm, ↓ ), PA-MPJPE( mm, ↓ ), MPVE( mm, ↓ ), PA-MPVE( mm, ↓ ). Further, we test the errors of 14 keypoints ( X _ 14 j ) shared by 2D and 3D pose representations (21) and 17 keypoints ( X _ 17 j ) represented only by 3D pose representations (16) respectively.

Implementation Details. PyTorch (65) was used for the entire experimental environment, four NVIDIA RTX 2080Ti GPUs for all training, and batchsize was uniformly set to 32. In the training stage, we only use the images from the PVCP trainset and the corresponding 2D ground truth keypoints as the input of the two models. We first train the ITP network by loading a pre-trained model of the MPII dataset (51) and training 40 epoches. For PTM, we use sequence length T = 16 and train 100 epochs in about two hours. In the test stage, in addition to the 2D ground truth keypoints of the testset, we also take the image of the testset as input to the ITP model, and then take the estimated 2D keypoint results as input to the PTM model. The effects of the same PTM training model with two different inputs are compared.

5.2 Effects of Dataset and Pose priors

We first evaluate the effects of PVCP dataset on improving the pedestrians pre-collision pose estimation. Compared to large-scale human pose datasets, our PVCP dataset is not numerically dominant, so we use a pre-trained model that learns human pose priors and fine-tune it based on that. We ran tests on the original MotionBERT (12) to compare scratch training and the PVCP trainset with together pre-trained models. We take the detected 2D pose sequence (2D Det) as input, and compare the errors of four pre-collision pose (Normal, Run, Avoid, Collision) and all pose (All). As shown in the Table 2, the results of only-pretrain model or only-PVCP trainset are relatively poor. Due to the difference between the pre-collision pose and the daily pose, the error of using only-pretrain model ( MPVE det = 335 . 11 mm ) is even worse than that of only-PVCP dataset ( MPVE det = 315 . 64 mm ). When the pre-trained model and the PVCP dataset are trained together, The minimum error ( MPVE det = 282 . 50 mm ) is obtained.

When 2D ground truth pose sequence (2D GT) is used as the input, the result without using the pre-trained model is relatively poor ( MPVE gt = 300 . 04 mm ), and the error after using only-pretrain model is significantly decreased ( MPVE gt = 165 . 90 mm ), because the number of keypoints ( ≥ 10) of the 2D GT pose sequences are relatively complete compared with the 2D Det pose sequences, its input is not affected by lighting conditions, background appearance, clothing, and weather conditions. Similarly, when the pre-trained model and the PVCP dataset are trained together, The minimum error ( MPVE gt = 145 . 77 mm ) is obtained. This shows that our PVCP dataset has different features from ordinary pose, and the pose prior of the pre-trained model can effectively promote the precision of the pre-collision pose.

5.3 Ablation Study

In the stage of PTM, we added an iterative decode, which does not directly predict the output once but gradually approximates the optimal solution with multiple iterations. At the same time, an pose classification regression head is added to use class loss as supervision. As shown in Table 3, we compared the impact of different components on network performance, loaded the pre-trained model each time, and set the optimal number of iterations to 3 (as shown in Table 4).

Due to the difference between 2D Det pose sequences and 2D GT pose sequences in the number and correct position of keypoints, when only iterative decode is used, the input error of 2D Det decreases significantly ( MPVE det = 282 . 50 mm → 266 . 20 mm ), while that of 2D GT decreases slightly ( MPVE gt = 145 . 77 mm → 145 . 75 mm ), which may be because multiple iterations improve the pose regression ability of incomplete pose. When only pose class loss is added, the error reduction space of the 2D GT input is significantly stronger ( MPVE gt = 145 . 75 mm → 141 . 28 mm ) than that of the 2D Det input ( MPVE det = 266 . 20 mm → 259 . 05 mm ), possibly because the pose that is complete and correctly positioned at the keypoints has a stronger correlation with the pose class label. Under the combined action of iterative regression decoder and loss function, both 2D pose sequence inputs achieve the minimum error ( MPVE det = 257 . 75 mm,MPVE gt = 140 . 43 mm ).

5.4 Comparison with the state-of-the-art

Quantitative comparison. Similar to (69), Table 5 reports results for multiple baselines on the PVCP testset using the evaluation metrics described in Sec. 5.1. We compare the classic baseline methods of two paradigms: the one-stage method, which involves direct regression from image to mesh, and the two-stage method, which involves regression from 2D pose to 3D mesh. Following (66; 67), in the one-stage method, we used ResNet-50 (63) to extract the feature f i ∈ R 2048 of the collision pedestrian clip-off image in each frame. Following (68), We use DarkPose (70) to extract 2D poses in COCO format (20). For MotionBERT (12) and our method, 2D pose is converted from 15 keypoints of JHMDB (60) to the corresponding 17 keypoints of Human3.6 (16) as input. Because the PVCP dataset contains only pose and shape annotations, there is a lack of spatial arrangement in the 3D scene. Therefore, the error of MPVE and MPJPE is large, but in PAMPVE and PAMPJPE, our method achieves the best accuracy. In addition, in the one-stage method, the effect of PARE (67) is relatively excellent, because PARE has optimized the occlusion of pedestrians and is well adapted

Image

PARE

2D pose

VIBE

Ours

Crop

k

to the real situation of the occlusion of pedestrians' lower limbs in the collision scene. In addition, the two-stage method needs to detect the 2D pose first, and the missing keypoints detection caused by occlusion will continue the error to the subsequent Mesh regression stage. However, in the case of single-class pre-collision poses ( Run, Avoid, Collision ), our method still has great advantages.

Qualitative comparison. A qualitative comparison of different methods is shown in Figure 3. Ignoring the global orientation and position, we manually adjust the Mesh output of different methods to the outline of the pedestrian in the picture, focusing on comparing the pre-collision pose itself. It can be seen that our method is most close to the real pedestrian pose in the image, and at the same time close to the estimated 2D pose, especially for the most complex pedestrian upper limb (row 4).

6 Conclusion

In this work, we construct the first Pedestrian-Vehicle Collision Pose (PVCP) dataset from the perspective of dashcams, which contains a variety of pedestrian pose representation annotation. At the same time, we propose a framework called PPSENet for the estimation of pedestrian precollision pose and shape. Specifically, a two-stage method is adopted: first, the pedestrian's 2D pose is estimated from the image, and then the pedestrian's 3D mesh is estimated from the 2D pose. Furthermore, we adopt the pose prior of the pre-trained model and the idea of iterative regression, introducing the pose class loss to achieve the minimum error on the PVCP testset, effectively estimating the pedestrian pre-collision pose in traffic collision scenarios. We hope that this work will offer new insights into human pose estimation and active pedestrian safety protection.

Limitations. Due to the difficulty of collecting the dataset, the dataset is small in size and lacks real camera parameters, vehicle speed information, global position and direction of pedestrians. Additionally, our task involves two-stage of pose estimation, and the final pose error is largely influenced by the 2D pose estimation results from the first stage. Our method is not real-time at present, because our input is Image and pre-selected Bbox sequence of collision pedestrian targets.

Future Work. From the comparison with state-of-the-art (SOTA) methods, it is evident that methods like PARE (67), which directly estimate 3D meshes of pedestrians from images, can also achieve good results. Future work can focus on new improvements in one-stage methods. Furthermore, we hope that the introduction of more modal information can further improve the accuracy of the estimation. The purpose of our work is to provide pose data support for the study of vehicle active and passive protection system, so as to facilitate subsequent accident reconstruction, pedestrian injury assessment and vehicle structural design.

References

J. Elliott, C. Simms, and D. Wood, 'Pedestrian head translation, rotation and impact velocity: The influence of vehicle speed, pedestrian speed and pedestrian gait,' Accident Analysis Prevention , vol. 45, pp. 342-353, 2012.

L. Shi, Y. Han, H. Huang, Q. Li, B. Wang, and K. Mizuno, 'Analysis of pedestrian-to-ground impact injury risk in vehicle-to-pedestrian collisions based on rotation angles,' Journal of Safety Research , vol. 64, pp. 37-47, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0022437517303705

T. Zou, A. Zha, Q. Liu, and C. Simms, 'Pedestrian gaits observed from actual pedestrian-vehicle collisions,' International Journal of Crashworthiness , vol. 27, no. 1, pp. 1-23, 2022.

S. Shang, C. Masson, M. Llari, M. Py, Q. Ferrand, P.-J. Arnoux, and C. Simms, 'The predictive capacity of the madymo ellipsoid pedestrian model for pedestrian ground contact kinematics and injury evaluation,' Accident Analysis Prevention , vol. 149, p. 105803, 2021.

G. Crocetta, S. Piantini, M. Pierini, and C. Simms, 'The influence of vehicle front-end design on pedestrian ground impact,' Accident Analysis and Prevention , vol. 79, no. Jun., pp. 56-69, 1 2015.

C. D. Untaroiu, M. U. Meissner, J. R. Crandall, Y. Takahashi, M. Okamoto, and O. Ito, 'Crash reconstruction of pedestrian accidents using optimization techniques,' International Journal of Impact Engineering , vol. 36, no. 2, pp. 210-219, 2009.

Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, 'Realtime multi-person 2d pose estimation using part affinity fields,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2017.143

B. Xiao, H. Wu, and Y. Wei, 'Simple baselines for human pose estimation and tracking,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 466-481.

J. Martinez, R. Hossain, J. Romero, and J. J. Little, 'A simple yet effective baseline for 3d human pose estimation,' in 2017 IEEE International Conference on Computer Vision (ICCV) , Oct 2017. [Online]. Available: http://dx.doi.org/10.1109/iccv.2017.288

Y. Xu, J. Zhang, Q. Zhang, and D. Tao, 'Vitpose: Simple vision transformer baselines for human pose estimation,' Advances in Neural Information Processing Systems , vol. 35, pp. 38 571-38 584, 2022.

N. Kolotouros, G. Pavlakos, M. Black, and K. Daniilidis, 'Learning to reconstruct 3d human pose and shape via model-fitting in the loop,' in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2019. [Online]. Available: http://dx.doi.org/10.1109/iccv.2019.00234

W. Zhu, X. Ma, Z. Liu, L. Liu, W. Wu, and Y. Wang, 'Motionbert: A unified perspective on learning human motion representations,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 085-15 099.

Z. Qiu, Q. Yang, J. Wang, H. Feng, J. Han, E. Ding, C. Xu, D. Fu, and J. Wang, 'Psvt: Endto-end multi-person 3d pose and shape estimation with progressive video transformers,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 21 254-21 263.

W. Kim, M. S. Ramanagopal, C. Barto, M.-Y. Yu, K. Rosaen, N. Goumas, R. Vasudevan, and M. Johnson-Roberson, 'Pedx: Benchmark dataset for metric 3-d pose estimation of pedestrians in complex urban intersections,' IEEE Robotics and Automation Letters , vol. 4, no. 2, pp. 1940-1947, 2019.

M. Schachner, B. Schneider, W. Sinz, and C. Klug, 'Extracting quantitative descriptions of pedestrian pre-crash postures from real-world accident videos,' International Research Council on Biomechanics of Injury (IRCOBI) , 2020.

C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, 'Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,' IEEE transactions on pattern analysis and machine intelligence , vol. 36, no. 7, pp. 1325-1339, 2013.

N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black, 'Amass: Archive of motion capture as surface shapes,' in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2019. [Online]. Available: http://dx.doi.org/10.1109/iccv.2019.00554

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, 'nuscenes: A multimodal dataset for autonomous driving,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 621-11 631.

A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The kitti dataset,' The International Journal of Robotics Research , vol. 32, no. 11, pp. 1231-1237, 2013.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, 'Microsoft coco: Common objects in context,' in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer, 2014, pp. 740-755.

T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll, 'Recovering accurate 3d human pose in the wild using imus and a moving camera,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 601-617.

F.-H. Chan, Y.-T. Chen, Y. Xiang, and M. Sun, 'Anticipating accidents in dashcam videos,' in COMPUTER VISION - ACCV 2016, PT IV , ser. Lecture Notes in Computer Science, S. Lai, V. Lepetit, K. Nishino, and Y. Sato, Eds., vol. 10114, 2017, pp. 136-153, 13th Asian Conference on Computer Vision (ACCV), Taipei, TAIWAN, NOV 20-24, 2016.

Y. Yao, M. Xu, Y. Wang, D. J. Crandall, and E. M. Atkins, 'Unsupervised traffic accident detection in first-person videos,' in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2019, pp. 273-280.

W. Bao, Q. Yu, and Y. Kong, 'Uncertainty-based traffic accident anticipation with spatiotemporal relational learning,' in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 2682-2690.

M. Wang, 'Smpl annotation tool,' https://github.com/wmj142326/SMPL_Tools, 2024.

J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik, 'Human pose estimation with iterative error feedback,' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2016. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2016.512

F. Baradel, R. Brégier, T. Groueix, P. Weinzaepfel, Y. Kalantidis, and G. Rogez, 'Posebert: A generic transformer module for temporal 3d human modeling,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

C. K. Simms and D. P. Wood, 'Effects of pre-impact pedestrian position and motion on kinematics and injuries from vehicle and ground contact,' International Journal of Crashworthiness , vol. 11, no. 4, pp. 345-355, 2006. [Online]. Available: https://doi.org/10.1533/ijcr.2005.0109

M. Hamacher, L. Eckstein, and R. Paas, 'Vehicle related influence of post-car impact pedestrian kinematics on secondary impact,' in Proceedings of the International Research Council on the Biomechanics of Injury conference , vol. 40. International Research Council on Biomechanics of Injury, 2012, pp. 717-729.

J. Tang, Q. Zhou, B. Nie, T. Yasuki, and Y. Kitagawa, 'Influence of pre-impact pedestrian posture on lower extremity kinematics in vehicle collisions,' SAE International journal of transportation safety , vol. 4, no. 2, pp. 278-288, 2016.

K. Gildea, D. Hall, C. R. Cherry, and C. Simms, 'Forward dynamics computational modelling of a cyclist fall with the inclusion of protective response using deep learning-based human pose estimation,' Journal of biomechanics , vol. 163, p. 111959, 2024.

J. S. Robertson, A. J. Mclean, and G. A. Ryan, 'Traffic accidents in adelaide, south australia (1963 - 1964),' Automobiles , 1966.

S. Shang, C. Masson, D. Teeling, M. Py, Q. Ferrand, P.-J. Arnoux, and C. Simms, 'Kinematics and dynamics of pedestrian head ground contact: A cadaver study,' Safety science , vol. 127, p. 104684, 2020.

Q. Li, S. Shang, X. Pei, Q. Wang, Q. Zhou, and B. Nie, 'Kinetic and kinematic features of pedestrian avoidance behavior in motor vehicle conflicts,' Frontiers in bioengineering and biotechnology , vol. 9, 2021.

W. Liu, Q. Bao, Y. Sun, and T. Mei, 'Recent advances of monocular 2d and 3d human pose estimation: a deep learning perspective,' ACM Computing Surveys , vol. 55, no. 4, pp. 1-41, 2022.

K. Sun, B. Xiao, D. Liu, and J. Wang, 'Deep high-resolution representation learning for human pose estimation,' in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2019. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2019.00584

S. Li and A. B. Chan, 3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network , Jan 2015, p. 332-347. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-16808-1_23

G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis, 'Coarse-to-fine volumetric prediction for single-image 3d human pose,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2017.139

G. Moon and K. M. Lee, I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image , Jan 2020, p. 752-768. [Online]. Available: http://dx.doi.org/10.1007/978-3-030-58571-6_44

Z. Qiu, Q. Yang, J. Wang, and D. Fu, 'Ivt: An end-to-end instance-guided video transformer for 3d pose estimation,' in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 6174-6182.

--, 'Dynamic graph reasoning for multi-person 3d pose estimation,' in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 3521-3529.

C.-H. Chen and D. Ramanan, '3d human pose estimation = 2d pose estimation + matching,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2017.610

M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, 'Smpl: a skinned multi-person linear model,' ACM Transactions on Graphics , p. 1-16, Nov 2015. [Online]. Available: http://dx.doi.org/10.1145/2816795.2818013

F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black, 'Keep it smpl: Automatic estimation of 3d human pose and shape from a single image,' in Computer VisionECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 2016, pp. 561-578.

K. Gildea, C. Mercadal-Baudart, R. Blythman, A. Smolic, and C. Simms, 'Kinepose: A temporally optimized inverse kinematics technique for 6dof human pose estimation with biomechanical constraints,' arXiv preprint arXiv:2207.12841 , 2022.

W. Luo, W. Liu, and S. Gao, 'A revisit of sparse coding based anomaly detection in stacked rnn framework,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 341-349.

J. Fang, D. Yan, J. Qiao, J. Xue, and H. Yu, 'Dada: Driver attention prediction in driving accident scenarios,' IEEE Transactions on Intelligent Transportation Systems , vol. 23, no. 6, p. 4959-4971, 2022.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, 'The cityscapes dataset for semantic urban scene understanding,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 3213-3223.

S. Zhang, R. Benenson, and B. Schiele, 'Citypersons: A diverse dataset for pedestrian detection,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 3213-3221.

A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, 'Mot16: A benchmark for multiobject tracking,' arXiv preprint arXiv:1603.00831 , 2016.

M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, '2d human pose estimation: New benchmark and state of the art analysis,' in 2014 IEEE Conference on Computer Vision and Pattern Recognition , Jun 2014. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2014.471

U. Iqbal, A. Milan, and J. Gall, 'Posetrack: Joint multi-person pose estimation and tracking,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2017.495

C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler, 'Unite the people: Closing the loop between 3d and 2d human representations,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2017.500

H. Joo, N. Neverova, and A. Vedaldi, 'Exemplar fine-tuning for 3d human pose fitting towards in-the-wild 3d human pose estimation,' arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition , Apr 2020.

G. Moon and K. Lee, 'Neuralannot: Neural annotator for in-the-wild expressive 3d human pose and mesh training sets.' Nov 2020.

Y. Tian, H. Zhang, Y. Liu, and L. Wang, 'Recovering 3d human mesh from monocular images: A survey,' IEEE transactions on pattern analysis and machine intelligence , 2023.

J. Fang, J. Qiao, J. Xue, and Z. Li, 'Vision-based traffic accident detection and anticipation: A survey,' IEEE Transactions on Circuits and Systems for Video Technology , 2023.

L. McInnes, J. Healy, and J. Melville, 'Umap: Uniform manifold approximation and projection for dimension reduction,' arXiv preprint arXiv:1802.03426 , 2018.

Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, 'Bytetrack: Multi-object tracking by associating every detection box,' in European conference on computer vision . Springer, 2022, pp. 1-21.

H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, 'Towards understanding action recognition,' in International Conf. on Computer Vision (ICCV) , Dec. 2013, pp. 3192-3199.

Z. Li, J. Liu, Z. Zhang, S. Xu, and Y. Yan, 'Cliff: Carrying location information in full frames into human pose and shape estimation,' in European Conference on Computer Vision . Springer, 2022, pp. 590-606.

J. Lin, A. Zeng, H. Wang, L. Zhang, and Y. Li, 'One-stage 3d whole-body mesh recovery with component aware transformer,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 21 159-21 168.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, 'On the continuity of rotation representations in neural networks,' in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2019. [Online]. Available: http://dx.doi.org/10.1109/cvpr.2019.00589

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, 'Automatic differentiation in pytorch,' Oct 2017.

M. Kocabas, N. Athanasiou, and M. J. Black, 'Vibe: Video inference for human body pose and shape estimation,' in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2020. [Online]. Available: http://dx.doi.org/10.1109/cvpr42600.2020.00530

M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black, 'Pare: Part attention regressor for 3d human body estimation,' in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2021. [Online]. Available: http://dx.doi.org/10.1109/iccv48922.2021.01094

H. Choi, G. Moon, and K. M. Lee, Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose , Jan 2020, p. 769-787. [Online]. Available: http://dx.doi.org/10.1007/978-3-030-58571-6_45

P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmann, S. Tripathi, and M. J. Black, 'Agora: Avatars in geography optimized for regression analysis,' in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2021. [Online]. Available: http://dx.doi.org/10.1109/cvpr46437.2021.01326

F. Zhang, X. Zhu, H. Dai, M. Ye, and C. Zhu, 'Distribution-aware coordinate representation for human pose estimation,' in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2020. [Online]. Available: http://dx.doi.org/10.1109/cvpr42600.2020. 00712

J. Brooks, 'Coco annotator,' https://github.com/jsbroks/coco-annotator/, 2019.

Y. Name, 'Human model viewer,' https://github.com/Lemon-XQ/human_model_viewer, 2020.

G. Moon, J. Y. Chang, and K. M. Lee, 'Camera distance-aware top-down approach for 3d multi-person pose estimation from a single rgb image,' in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2019. [Online]. Available: http://dx.doi.org/10.1109/iccv.2019.01023

C. Zheng, W. Wu, C. Chen, T. Yang, S. Zhu, J. Shen, N. Kehtarnavaz, and M. Shah, 'Deep learning-based human pose estimation: A survey,' ACM Computing Surveys , vol. 56, no. 1, pp. 1-37, 2023.

Appendix / supplemental material

Overview

This supplementary material presents more details and additional results not included in the main paper due to page limitation. The list of items included are:

Details of PVCP Dataset in Sec. A

Details of the Network Architecture in Sec. B

Details of Experiment and Results in Sec. C

A PVCP Dataset

A.1 Data Collection.

Our PVCP dataset are all derived from the vehicular perspective of dashcam, and videos are sourced from two primary origins. Similar to previous works (47; 22; 23; 24; 15), we collected videos of pedestrian and vehicle collisions from online platforms such as YouTube, using 'pedestrian-vehicle collision' as a keyword. In addition, a small number of videos are derived from existing open-source traffic datasets (47; 22; 23), which were primarily developed for tasks related to driver attention and the prediction of sudden accidents. From these datasets, we selectively extracted videos that depicted incidents of pedestrian and vehicle collisions. Furthermore, a multi-step filtering process was instituted to ascertain the high quality of the data. Firstly, videos of pedestrian and vehicle collisions within traffic scenarios were selected based on the recorded times and locations. Secondly, we opted for videos shot from a first-person perspective, specifically from the viewpoint of a dashboard camera. Lastly, a resolution analysis was conducted on the videos, those with resolutions too low for human discernment were discarded to ensure the clarity and visibility of the content. All of the collected videos were reduced to individual accident footage, recording a complete pedestrian and vehicle collision, resulting in 209 pedestrian-vehicle accident videos, totaling 42,511 frames of images and about 19,533 pre-collision poses. The statistical results associated with video clips and image frames are shown in Figure A1(a) and Figure A1(b).

1095 (2.6%)

244 (0.6%)

10534 (24.8%)

10791 (25.4%)

19847 (46.7%)

(1280 ×  720)

(1584 ×  660)

(1920 ×  1080)

(1246 ×  720)

(640 ×  480)

337

2964

6498

9734

0

5

10

15

20

25

30

35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

21

22

Video Count

Duration (seconds)

(b) Frame resolution

(a) Video duration

(c) Pose class count

A.2 Data Annotations

In scenarios involving collisions, the pedestrian, the vehicles, and the interconnected data between the two are the pivotal components that must be attend. For the application of pedestrian-vehicle collision accident reconstruction and vehicle active protection in traffic collision scenarios, PVCP provides a rich pedestrian pose representation, including pedestrian clipping images, pedestrian Bounding Box, track id, 2D and 3D keypoints and SMPL mesh label.

A.2.1 Pose Annotations

SMPL Annotation Tool . We used COCO-annotation (71) as a tool for 2D pose annotations and specifically designed an annotation tool (25) for SMPL mesh, which was developed based on (72). By importing the cropped pedestrian background and the initial results estimated by SPIN (11), we ensured that the mesh aligned as closely as possible with the pedestrian's background outline in the image. The final exported mesh model was used as ground truth labels. The annotation interface and

File Export View

FileExport View

92

Shape

29

Shape

Expressior

Expression

Pose

Pose

et

1

nkleL

nikle R

12M

inpesi

三

ClaiceL

ClwideR

weritR

Fingesk

KneeR

Claide R

wristR

Female

Male

Camera

ReuetExpre

0.00

Position

0.00

0.00

Female

Male

Camera

ResetExpre

0.00

Position

0.00

0.00

frame_000031

aip

ftame_000031

Skip

Before annotation

After annotation.

the comparison before and after annotation are shown in the Figure A2. By adjusting the weight of the image background and the mesh foreground, it is clearly displayed to facilitate annotation.

A.2.2 Vehicle Annotations

Distance between the pedestrian and the vehicle, as well as the vehicle's speed, are crucial factors in collision accidents. In real traffic scenarios, these parameters can often be acquired through the sensors equipped on the vehicle. However, in our collected accident videos, obtaining the distance and speed through sensors is not feasible (as these data are not provided). Therefore, we refer to RootNet (73) to obtain the distance between the pedestrian and the vehicle as our dataset's distance annotation. Similarly, vehicles are often fitted with additional sensors to get accurate speed, but for our crash video dataset it's not perfect and the speed information is missing. Therefore, we use the estimated pedestrian-vehicle distance and video frame rate to obtain an approximate speed label.

A.2.3 Other Annotations

Beyond the aforementioned annotations, we have also annotated the environment of the collision, the cause of the accident, and whether a collision ultimately occurred. This additional annotation information serves as a supplement to our dataset, facilitating its use in related tasks.

A.2.4 Visualization Comparison

Compared to existing pose datasets, PVCP demonstrates significant differences and advantages in terms of scene specificity, action space variation, and temporal continuity. As shown in the Figure A3, we visualize a comparison of the differences between PVCP and other pose datasets.

MSCOCO Dataset (20) contains annotations for object detection, panoptic segmentation, and keypoint detection. The images are collected from websites including Google, Bing, and Flickr. The annotations are performed by workers on Amazon's Mechanical Turk (AMT). The dataset contains over 200K images and 250K person instances (35). Although the size of COCO datasets for 2D HPE is large enough for normal pose estimation (e.g., standing, walking, running), these datasets have limited training data for unusual poses, such as falling. The data imbalance may cause model bias, resulting in poor performance on those poses (74). Furthermore, the COCO dataset lacks continuous video frames for pose estimation, which presents a challenge for studying sequential pose variations related to temporal features.

Human3.6M Dataset (16) is the most widely used multi-view single person 3D human pose benchmark. The dataset is captured in a 4m×3m indoor space using 4 RGB camera, 1 time-of-flight sensor, and 10 motion cameras. It contains 3.6 million 3D human poses and the corresponding videos (50 FPS) in 15 scenarios, such as discussion, sitting on a chair, taking a photo, etc. Especially, both 3D positions and angles of keypoints are available (35). However, these poses belong to everyday activities and differ in spatial and temporal characteristics from specific poses such as pre-collision poses. Moreover, the dataset, captured in an indoor environment, lacks dynamic or static scene backgrounds, which are important factors that can affect the robustness of pose estimation algorithms.

MSCOCO

H36M

Our   PVCP

Image background is also one of the key factors influencing the performance of pose estimation methods (56).

PW3D Dataset (21) is a single-view multi-person in-the-wild 3D human pose dataset that contains 60 video sequences (24 train, 24 test, and 12 validation) of rich activities, such as climbing, golfing, relaxing on the beach, etc. The videos are captured in various scenes, such as forest, street, playground, shopping mall, etc. They leverage IMU to obtain accurate 3D pose despite the complexity of scenes. Especially, PW3D contains abundant 3D annotations, including 2D/3D pose annotations, 3D body scanning, and SMPL parameters. In some crowded scenes (e.g. on the street), PW3D only provides the label of the target person, ignoring the pedestrians passing by. Generally, the entire dataset is used for evaluation, without any fine-tuning (35). Similar to PW3D, we also annotate pedestrian 2D and 3D keypoints as well as SMPL parameters. However, due to scene constraints, this method of data collection is difficult to replicate in collision scenarios.

PedX Dataset (14) is a multi-sensor dataset focused on pedestrian-vehicle interactions at urban intersections. It includes over 5,000 stereo image pairs and 2,500 frames of 3D LiDAR data, all calibrated and time-synchronized. Collected at three four-way intersections, the dataset captures more than 14,000 pedestrian instances from a driver's perspective using roof-mounted stereo cameras. Each pedestrian is annotated with 2D and 3D labels, including 18 keypoints, and unique tracking IDs across frames. Using SMPL parameterization, PedX provides accurate 3D models of pedestrian pose, shape, and global position based on stereo images and LiDAR data. PedX is a pedestrian pose dataset in a traffic scene similar to our PVCP. However, it maintains a fixed viewpoint at intersections, and the pedestrian poses it captures are primarily limited to normal walking postures.

PVCP Dataset focuses on dashcam perspective, capturing pedestrians' emergency behaviors during vehicle movement, such as sudden road crossings or running into the street, within dynamic traffic environments and changing fields of view. It highlights rapid, unpredictable motions like accelerating, swerving, or jumping to avoid vehicles. PVCP categorizes these pre-collision poses into four types, and the PPSENet model leverages pose category loss to learn their spatial distinctions. Pedestrian pre-collision poses are closely linked to the vehicle's time series, occurring in rapid succession as the vehicle approaches, emphasizing the importance of time sensitivity and continuity in algorithms. PVCP provides continuous pre-collision emergency poses of pedestrians, this allows for timely pose predictions and warnings by capturing pose variations from a continuous context.

left_shoulder right_shoulder

ID

Joint name

0

neck

1

belly

2

head

3

right_shoulder

4

left_shoulder

5

right_hip

6

left_hip

7

right_elbow

8

left_elbow

9

right_knee

10

left_knee

11

right_wrist

12

left_wrist

13

right_ankle

14

left_ankle

ID

Joint name

0

pelvis

1

right_hip

2

right_knee

3

right_ankle

4

left_hip

5

left_knee

6

left_ankle

7

spine

8

neck

9

nose

10

head

11

12

left_elbow

13

left_wrist

14

15

right_elbow

16

right_wrist

(a) 15 keypoints of JHMDB

16

12

13

0

10

13

B Network Architecture

B.1 Image to Pose

Estimating 2D human pose from images is a basic and mature task, and many works have achieved very effective results in different datasets. We take image frame I and corresponding Bbox as input, simply use ResNet50 (63) as the backbone of feature extraction, and use a transposed convolution and a heat map regression head H as the 2D pose estimation network, which is also the classic paradigm of 2D pose estimation (8). The resulting 2D pose P 2 D ∈ R 15 × 2 will be used as input for the next stage. The testset accuracy of ITP is shown in the Table A1. The accuracy of Ankle is significantly lower because the lower limbs of the pedestrian are heavily shielded during the collision.

B.2 Pose to Mesh

We refer to the JHMDB (60) dataset for annotating human 2D pose keypoints (15 keypoints) and participate in the first stage of ITP training. For PTM training, we use the Human3.6M (16) pose representation (17 keypoints). As shown in Figure A4, there are differences in joint positions and numbering between the two. We aligned the common joint positions and calculated the additional joints for Human3.6M:

B.3 Loss Function

The loss function of ITP, defined as the Mean Squared Error (MSE), is applied for comparing the predicted heatmaps ˆ H and the ground truth heatmaps H . Following (12), the PTM loss function utilizes SMPL loss and motion loss. Furthermore, The pedestrian pose of the collision sequence has obvious categories, namely normal pose, running pose, avoiding pose and collision pose as described in 3.2. So we introduce pose class loss. The final loss function is as follows:

where L Cross Entropy ( ˆ C,C ) epresents the cross entropy loss between the predicted pose class and the GT pose class. λ c are the constants of the balance weight loss.

C Experimental and Results

C.1 Evaluation Metric

We evaluated the estimation of 3D human pose and shape using the following metrics, which are briefly described as follows.

MPJPE( mm, ↓ ). Mean Per Joint Position Error measures the average Euclidean distance between predicted 3D pose and ground truth after root (pelvis joint) matching, which comprehensively evaluates the predicted poses and shapes, including the global rotations.

PA-MPJPE( mm, ↓ ). Procrustes-Aligned Mean Per Joint Position Error denotes MPJPE after rigid alignment of the predicted 3D pose and ground truth, which eliminates the discrepancies in scale and global rotation.

MPVE( mm, ↓ ). Mean Per-vertex Error , initially aligns similarly to MPJPE. It is characterized by the average point-to-point Euclidean distance between predicted mesh vertices and corresponding ground truth vertices.

PA-MPVE( mm, ↓ ). Procrustes-Aligned Mean Per-vertex Error represents MPVE after applying a Procrustes alignment between predicted mesh vertices and ground truth vertices, similar to how PA-MPJPE is calculated.

Further, we test the errors of 14 keypoints ( X _ 14 j ) shared by 2D and 3D pose representations (21) and 17 keypoints ( X _ 17 j ) represented only by 3D pose representations (16) respectively. For 2d pose estimation, we use PCK as metrics.

PCK( % , ↑ ). Percentage of Correct Keypoints measures the accuracy of joint positioning in the body. If the candidate body joint falls within the threshold pixel of the GT joint, it is considered correct.

C.2 Ablation Study

Component of system. In the stage of lifting 2D poses to 3D meshes, we incorporated an iterative decoder. Instead of predicting the output in a single step, we iteratively refined the results to gradually approach the optimal solution. Additionally, we included a pose classification detection head and used a class loss as supervision. We compared the impact of different components on the network's performance, reloading and training the model each time, and set the optimal number of iterations to 3. Under the combined effect of these two modules, the pose sequences from both types of input achieved minimal error. As shown in Table A2, we provide a detailed comparison of the errors for different pre-collision pose class. The iterative decoder significantly improved the accuracy across all pose categories, and the introduction of the pose category loss function further enhanced the accuracy of pre-collision poses.

Number of iterations. As shown in the Table A3, we combined all modules to verify the accuracy of different iterations of 2D GT input to select the optimal number of iterations. We verify the test results of increasing the number of iterations from 2 to 6 in turn, and it can be seen that the test results of different iterations are excellent. When iter =3 or 4, the average error of each metric is the lowest value. We chose iter =3 as the final number of iterations for experimental comparison.

Without Fine-Tuning. As shown in the Table A4, we trained on the COCO and PW3D dataset respectively and validated on the PVCP testset. Because these datasets do not have pose category annotations, we set them uniformly to a single category. The results show that training only on another large human pose dataset without fine-tuning for PVCP does not perform well on PVCP this particular pose dataset. The conclusion that can be drawn is that training solely on other large human pose datasets, such as COCO and PW3D, without fine-tuning for PVCP, does not yield good performance on the PVCP dataset. This indicates that there are significant differences between datasets, particularly when dealing with specific scenarios and unique poses, and that fine-tuning and optimization tailored to the PVCP dataset are necessary to improve model performance.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our work provides data and algorithmic support for pedestrian pre-collision pose estimation and active safety protection for pedestrians

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are discussed in detail in the Paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We correctly cited all the theorems, formulas, and proofs in our paper and numbered them accordingly.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detailed the experimental procedures in our paper, and we are committed to releasing our dataset and code as soon as possible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our dataset and code will be made publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We detailed the experimental procedures in our paper, and we are committed to releasing our dataset and code as soon as possible.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted ablation experiments, qualitative experiments and quantitative experiments to demonstrate the advantages of the work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We detailed the experimental procedures in our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper conforms to the NeurIPS Code of Ethics in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the open-source code and datasets mentioned in the paper and adhered to the terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We are committed to releasing our dataset and code as soon as possible.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research without human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research without human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a dataset (PVCP) followed by a solution (PPSENeT), which is the first one that focuses on the 3D pre-collision pose of pedestrians. The dataset is collected by dashcam that contains various poses with both static and dynamic backgrounds. Various types of annotations are provided, including 2D, 3D, and human mesh, leading to diverse evaluation protocols. Following the dataset, the method (PPSENet) is a top-down method, that first crop the human subjects with a bounding box and then predict the 2D poses. The 2D pose will be lifted to a 3D mesh with the help of iterative regression and a novel pose class loss, following MotionBERT backbone. All combined, the method achieves SOTA performance on the proposed PVCP dataset.\n\nSTRENGTHS:\n1. The proposed dataset contains various types of annotations, including 2D, 3D, and human mesh. This contains abundant ways of evaluation for every method.\n\n2. The topic is in general interesting. It is a very good intention to expand the scope of the daily human pose to the pose in a specific setting. Especially, in such a setting related to the safety of daily life.\n\n3. The paper is generally good-written and easy to follow.\n\nWEAKNESSES:\nThe major problem addressed by the paper is somewhat trivial to me. Typically, there are two logical holes that the author fails to fulfill before they propose this dataset and solutions.\n\n(1). Are the pre-collision poses really so different from normal or daily poses? This is intuitively not correct in my perspective. The pre-collision pose is a subset of the daily pose. In this spirit, the model trained on a large human pose set (such as COCO), should work decently on the pre-collision pose. The pose such as running or avoiding, are covered in COCO (as COCO also contains various). COCO has even more diverse scenes (indoor and outdoor), while PVCP presumably only contains outdoor scenes (as it is for pre-collision poses). In this case, authors should give solid evidence to support their claim. Moreover, in the author's Fig1(c), the proposed dataset PVCP, seems even more general than COCO, which is contradictory to my intuition. \n\n(2). Are the pre-collision poses really an important factor in terms of avoiding collision? There are many factors that could influence the rate of avoiding collision, such as the speed of vehicles or the light conditions as the authors also mentioned in the appendix. Are there any experimental results to support having the pre-collision poses will significantly reduce the rate of accidents?\n\nIf the above two questions are not answered properly, then it seems this problem is not really a significant problem to be investigated, which will reduce the necessity of the proposed dataset and method.\n\nThe proposed method, on the other hand, follows a top-down paradigm, which is known as not compatible with real-time applications, not to mention that architecture is used to produce 3D mesh. However, the pre-collision pose estimation definitely requires real-time processing.\n\nQUESTIONS:\n1. What is the processing time (inference time) of the proposed method? Is it a real-time method?\n\n2. What is the performance of the method only trained on COCO or another large human pose dataset without finetuning on the PVCP?\n\nFor the other two questions, please see the weaknesses. These questions are the key to changing the rating.\n\nLIMITATIONS:\nNo negative societal impact is found.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
The paper conducts the first Pedestrian-Vehicle Collision Pose dataset(PVCP) including the pedestrian-vehicle collision pose from the dashcam perspective and pose annotations. Moreover, the paper presents the method(PPSENet) to estimate the collision pose and shape based on PVCP.

STRENGTHS:
1. The first Pedestrian-Vehicle Collision Pose dataset, annotated by algorithm and manual method.
2. The paper proposes the framework(PPSENet) to estimate the 2D pose and lift it to the 3D pose. The method achieves SOTA in PVCP

WEAKNESSES:
Because the dataset annotation largely depends on manual methods. The paper may need more discussions or a more convincing evaluation, such as more visualization of the dataset or a video.

QUESTIONS:
Will the SMPL annotation tool be made publicly available?

LIMITATIONS:
The author addresses the limitations in the apendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper collects a new dataset that includes pedestrian-vehicle collision poses from the dashcam perspective and presents a two-stage framework for estimating human pose and shape parameters. The dataset has the potential to benefit the study of pedestrian injuries. Given the challenge of obtaining ground truth pose and shape parameters in the wild, the authors employ the two-stage framework to generate initial results. These are then refined through manual adjustment using their annotation tool.\n\nSTRENGTHS:\nThis paper makes a significant contribution by collecting real-world pedestrian pose and shape parameters before collisions. The datasets include useful modalities such as bounding boxes and tracking information. Since it is difficult to obtain human poses in uncontrolled environments, the authors collect online dashcam videos and introduce a straightforward framework to estimate initial results. This framework iteratively regresses the pose and shape parameters.\n\nWEAKNESSES:\nSince the dataset uses videos collected from the Internet, it is difficult to obtain the ground truth pose and shape parameters. The accuracy of a dataset is important. The paper could elaborate more on the procedures to ensure accuracy such as how many annotators are recruited, how they are trained before the annotation, whether there is a post-processing stage, etc.\n\nQUESTIONS:\n1. How does the author ensure the motion smoothness of results during the manual annotation refinement stage?\n2. How many annotators are assigned to each video clip?\n3. How annotators are recruited and trained to ensure accuracy?\n\nLIMITATIONS:\nThe dataset is relatively small in size and no camera parameters are provided. Accurate vehicle speed and the global position of pedestrians could also be helpful for collision damage analysis.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the pre-collision posture of pedestrians in real traffic scenarios. The authors semi-automatically constructed the first pedestrian-vehicle collision pose dataset PVCP by collecting pedestrian-vehicle collision poses from the perspective of dashcams, which includes more than 40,000 accident frames and more than 20,000 pedestrian pre-collision pose annotations. Furthermore, this work constructs a pedestrian pre-collision pose estimation method(PPSENet) to estimate the collision pose and morphological sequence of pedestrians from pedestrian-vehicle accident videos. PPSENet first estimates the 2D pose from the image (image to pose, ITP), and then promotes the 2D pose to a 3D mesh (pose to mesh, PTM). Experiments show the relative advantages of PPSENet.\n\nSTRENGTHS:\n1. I consider the dataset collected in this work is meaningful. It can provide a well guide for many autonomous driving methods. Although the pose before the collision may not even last for a second, the human pose data at this moment is indeed worth studying.\n2. The author designed an SMPL annotation tool to align the initial results with the image contour. The PVCP dataset finally obtained about 40,000 accident images and more than 20,000 2D and 3D annotations of pedestrian emergency poses.\n\nWEAKNESSES:\n1. I consider  the PPSET method is the weak point of this work. From a methodological perspective, except for the robust 2D pose estimation method in the first stage, the second stage is almost similar to the backbone network of MotionBERT. There is almost no innovation. From an experimental perspective, the results of 2D pose sequences in Table 2 are really poor. Although PVCP is a brand new dataset for pre-collision human poses, the network input is sequences, and the overall 2D joint index, MPJPE is greater than 100. In the results of Table 5, Pose2Mesh performs better than MotionBERT without being trained on PVCP. I highly doubt this experimental result. However, PARE, a one-stage method that has not been retrained, can outperform PPSET in the first four indicators. We know that PARE is from the work in 2021. From this result, I think PPSET is completely at a disadvantage.\n2. Although PVCP has collected a relatively large number of pedestrian-vehicle collision pose datasets, it is not the first one. [1] As early as 2020, there was a description of the relevant dataset, and the video in it is the same as PVCP. Although I saw the relevant reference, it was glossed over. The statement in line 110 of Related Work should be modified.\n3. Although I think the annotation application of SMPL is an innovation, the author also cleverly hid the relevant github content. However, the relevant GUI interface had similar content as early as 2020 [2].\nTypo:\n1. The two time spans in Year in Table 1 should be based on \"2017\" instead of \"1017\"\n2. The use of \"impractical\" in line 46 is easy to cause ambiguity. Because PVCP is doing such content.\n\nQUESTIONS:\nI hope the author will emphasize the breakthrough and innovation brought by the proposed method PPSET. I think the starting point of the dataset is very good, and the content of the dataset is also very good. But I think the corresponding method should be equally good to meet the standards of NeurIPS.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}