HUM3DIL: Semi-supervised Multi-modal 3D HumanPose Estimation for Autonomous Driving
Authors: Andrei Zanfir
Mihai Zanfir
Alex Gorban
Jingwei Ji
Yin Zhou
Dragomir Anguelov
Cristian Sminchisescu
Venue: CoRL 2022 Poster (2022)
Keywords: autonomous  driving
perception
human  pose
key  points
skeletal representation

Abstract: Autonomous driving is an exciting new industry, posing important research questions. Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. While hardware systems and sensors have dramatically improved over the decades -- with cars potentially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information -- not much work has been done to harness these novel signals for the core problem of 3D human pose estimation. Our method, which we coin HUM3DIL (HUMan 3D from Images and LiDAR), efficiently uses of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. It is a fast and compact model for onboard deployment. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages. Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-the-art results on the task of 3D pose estimation.

HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving
Andrei Zanfirt
Mihai Zanfir
Alexander Gorban
Jingwei Ji
Yin Zhou
Dragomir Anguelovi
Cristian Sminchisescu'
Google Research
(andreiz, mihaiz, maiinchiseacu/@goolle.cco
Waymo Research
(gorban, jingveij, yinzhou, dragomir 00ayyo.
Abstract: Autonomous driving is exciting new industry, posing important search questions. Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. While hardware sys- tems and sensors have dramatically improved over the decades with cars poten- tially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information not much work has been done to harness these novel signals for the core problem of 3D human pose estimation. Our method, which coin HUM3DIL (HUMan 3D from Images and LiDAR), efficiently makes use of these complementary sig- nals, in a ceni-supervised fashion and outperforms existing methods with a large margin. It is fast and compact model for onboard deployment. Specifically, we embed LiDAR points into pixel-aligned multi-mon features, which we pass through a sequence of Transformer refinement stages. Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-the- art results on the task of 3D pose estimation.
Keywords: autonomous driving, perception, human pose, key points, skeletal representation
Introduction
Robotic systems which operate in environments with humans are required to avoid collisions with people and benefit from analysing their actions and forecasting future behaviors. Human pose derstanding is a well established research direction in computer vision with numerous industrial applications [1, 2, 3, 4]. In this work we focus 3D human pose understanding for the autonomous vehicle (AV) industry and robotic applications in general.
Safety is the top priority for the AV industry. Many robotic platforms sensors in different modal- ities (e.g. cameras, LiDARs, radars, audio, etc.) to improve safety by analyzing more signals about the environment. Using RGB cameras coupled with LiDAR sensors could be considered dard sensor suite for most robotic platforms. While many studies have shown impressive results for estimating human poses using RGB imagery, there is paucity of methods which can effectively both modalities [5].
stan-
Recent studies have made great headway into estimating 3D human pose in controlled environments [1]. but many real-world and safety-critical robotic applications require estimating human poses in uncontrolled environments, where subjects may be captured under different levels of occlusions, from various perspectives and across any ranges. There are several approaches to estimate 3D hu-
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

man poses in uncontrolled environments [5], but they have insufficient accuracy to unlock their full potential, especially for AV applications.
Another limiting factor for robotic applications of available methods of 3D pose detection is their computational complexity. There are fast methods for detecting a 3ID bounding box which contains a single person [6. 7]. Thus most robotic applications represent humans with 3D bounding boxes. While this crude representation of people allows such applications to meet basic safety requirements and avoid collisions, it is not sufficient for understanding complex human body gestures. There are methods which output feature rich representations and estimate parameters of full body meshes [8], but they are relatively slow. Representing 3D human body pose as a sequence of locations of 3D key points inside the body could be considered as a balanced trade off between fast compute boxes and slow full body models. The main goal of this work is to provide fast method for human pose estimation in uncontrolled environments which efficiently uses sensor modalities common for AV industry and outputs representations rich enough to enable analysis of complex human behaviors.
One of the key factors enabling research and development of methods for human pose understanding is the availability of high-quality ground truth 3D data with human poses and sensor data. There are few ways collect such data: marker or markerless camera based motion capture systems (suitable for controlled indoor environments); IMU based motion capture systems (suitable for both indoor and outdoor, but also controlled environments) [1] and manual human labeling (suitable for all environments, but expensive and error prone). To the best of our knowledge, there is no large-scale outdoor dataset with human poses collected in an uncontrolled environment with ground truth 2D and 3D keypoints with RGB and LiDAR for a fully supervised training mode. Waymo recently released version of their Waymo Open Dataset (WOD v1.3.2) with a large amount of camera (2D) keypoints and small amount of laser (3D) keypoints (enough for fine-tuning and evaluation purposes) which is suitable for weakly and/or semi-supervised training modes. In this work we use the WOD v1.3.2 dataset to demonstrate that our method can reliably predict 3D human pose in uncontrolled and challenging AV scenarios, and we compare our approach with several state-of-the-art methods [9, 10, 11] after adapting them for multi-modal applications.
We propose HUM3DIL, a light- 3D human joints prediction network, that leverages RGB information with LiDAR points, in novel fashion, by computing pixel-aligned [12] multi-modal features with the 3D positions of the LiDAR signal. These features are then used by subsequent Transformer- bassed refinement stages, to produce the desired 3D joints. We train our model in a semi- sun-ssupervieed manner, to maximize the utility of both 2D annotations (less expensive to collect, available in larger volumes) and 3D labels (expensive to collect, accurate, but with limited coverage). Quantitative results on Waymo Open Dataset indicate state-of-the-ar performance. Being accurate, fast and lightweight, HUM3DIL can be deployed into onboard autonomous driving systems to pro- vide real-time perception signals of human road users. We believe downstream tasks can greatly benefit from these fine-grained signals.
Related Work: There are considerable amount of prior works in 3D human pose reconstruction, mostly focused on estimation from RGB images alone. There are two main classes of methods, the first of which model based [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] and relies on statistical human body models like SMPL [24] or GHUM [25]. These methods do not estimate 3D pose directly, but instead regress the parameters of statistical body model, which has built-in anatomical and kinematic constraints. This leads to more natural predictions, with body shape usually estimated as well, even for poses not encountered during training. The second class of methods skeleton- based [26, 27, 28, 22], where the 3D pose is represented by 3D joint positions and these are to be regressed or detected directly from the input. These methods have the advantage of usually being more accurate and faster, but they are not guaranteed to produce anatomically correct human skeletons (c.g. the left arm may be reconstructed with different length than the right arm). Our proposed model falls in the latter category, as our goal is to estimate pedestrians as accurately as possible in real-time. Mixed approaches have started to recently emerge, like in e.g. [9], where 3D positions are inferred directly, but anatomically regularized through a statistical model.
There are works that use depth information, either separately or in combination with an RGB image, to reconstruct the 3D pose [29, 30, Approaches that utilize LiDAR information are hard to come by, mostly because of the lack of ground-truth 3D human pose paired with LiDAR data. There are a few datasets that, to different degrees, do provide 3D annotations. The PedX dataset [32] offers 14, 000 3D automatic pedestrian annotations obtained using model fitting on different modalities,

---

gathered from three different real scenes. The Waymo Open Dataset [33] has a similar amount of 3D annotations as [32], but features many more different environments (2.030 real scenes, 7,650 different people) with high-quality 2D and 3D manual annotations. Even with the existence of these datasets, the few works on 3D pose reconstruction published in this space mostly rely on weak supervision, by lifting 2D pose information in 3D. [34] trains on 2D ground-truth pose annotations and uses a reprojection loss for the 3D pose regression task. [1I] creates pseudo ground-trule 3D joint positions from the projection of annotated 2D joints, by considering neighboring LiDAR points in the projection space. During training, they directly compare the predicted 3D positions against this pseudo ground-truth.
2 Methodology
2.1 Problem Formulation
We focus on the task of key points localization and formulate the problem as estimating 3D locations for a set of key points Y E x3 inside human body, given the ground truth or predicted bounding box of human image I E [o, 1|"x [=XX8 x3 and a point cloud E R N3, consisting of N, LiDAR points from a single scan.
well as multi-modal inputs from camera and LiDAR sensors: an RGB camera
Camera Model. For correct LiDAR points to camera image projections, we use differential imple- mentation of the Waymo Open Dataset [33] camera model, with rolling shutter effect compensated. We denote the intrinsic information (e.g. lens distortion, shutter speed, focal length, focal point etc.) as a vector K, and the extrinsic parameters (e.g. vehicle pose, camera pose, linear and rotation speed) as K.. The complete camera information will be denoted as K = [K. Ki] E I The associated camera image projection operator will be denoted by 11(*, K), where 3D input will be correctly projected into the image space.
2.2 HUM3DIL
Our network. deemed HUM3DIL (see fig. 1), receives as input the RGB camera image I, LiDAR point cloud P. camera intrinsics K, the paired 2D and 3D bounding boxes, and outputs the pre- dicted 3D human keypoints Y. Our goal is to use image input to better exploit or disambiguate the structure present in the 3D point cloud, while at the same time use the 3D points to anchor imagery evidence. Because we have access to the ground-truth camera intrinsics, we can move between 3D space and 2D image space by projection, and vice-versa, by back-projection Motivated by recent advancements in 3D pose estimation and point-cloud processing [36, 37, 9]. we use a Transformer- based architecture [38] to process 3D and image-based structural information the same time, compared to typical approaches which use disconnected PointNet 3D [39, 40) embeddings and/or image features.
Enriching LiDAR points with image evidence We construct a depth-image projection D E xx into the space of LiDAR point cloud P. We concatenate it with the raw RGB image, DI and pass the derived tensor through a convolutional architecture. Thus, we simultaneously inform the convolutional layers of the regions of interest the image, i.e. sparse locations on the silhouette of the person, and make depth information available from the start. Adding the depth map channel helps disambiguate the task of key point prediction in cases with heavy occlusions or in crowded scenes where multiple people are in the frame depth map channel will have non zero values only for a person of interest. For the convolutional architecture, we employ a lightweight U-Net network [41], that outputs dense feature map representation F E #x xD,, This map is used to query from, based on the projection of the LiDAR points, by bilinear interpolation:
F; F[I[PP, K)] E 1x
for each point P. E P. We thus obtain pixel-aligned image features for LiDAR points.
LiDAR points embedding Aside from the per-point depth-infused image features, we also process the initial 3D LiDAR points. We use Random Fourier Features (RFF) [42] embed P in higher- dimensional space, capturing high-frequency behaviour of the signal. We use a random Gaussian matrix B E 3,/2, /2 with each entry independently drawn from a normal distribution (0, o²). The transformed points will have the form P (2 sin(2 E ND
3

---

ROB Camera Image
johnts
Mutt-modal Featuree
MLP
R
L (Transfforme Encoder)
Projected LIDA
LIDAR 30 Points
P N
Baszón Eirbeediigg
Training loss
C XC2D LaD
R g intrinaces
Figure 1: Overview of our proposed HUMSDIL architecture. It estimates the 3D joint positions of sin- gle person from multi-modal input representation. We encode LiDAR points P through Random Fourier Embedding [35], to produce representations P. The LiDAR points are also used to compute a depth-map representation D. which are concatenated with input RGB image I. We first (U-Net) that will from the produced map F. We construct token sequence of size equal the number points. in the beginning, will have information relating to the image features, camera intrinsics and Random Fourier Embeddings. L sequences of Transformer Encoder will act on the tokens. We read the final N₂ tokens and regress the 3D joints through an MLP.
image feature extractor the concatenation of D and I. The projected LiDAR points will further read features
token,
Transforming the LiDAR points In order to regress Y, we employ Transformer architecture [43]. We define input i-th token M, RIID with D Dy D, NR,
M, F,
the concatenation of the camera intrinsics, the per-point image feature and the per-point Fourier representation. We will apply the Transformer on fixed sequence of Np tokens. In order to work with variable number of tokens, we use a fixed maximum size for the token sequence, Nmma We shuffle and trim excess points, and pad with zeros if we have fewer number of points. The complete input sequence is M E Nmmz D This sequence is at first linearly embedded by using a learnable matrix E E Do Here, Dp is the operating dimensionality of the Transformer architecture. We additionally concatenate learnable joints tokens, M E N2,, N,N, Do, Similar to [9], we use cascaded block of L Transformer encoder layers, and collect the predicted 3D keypoints Y from an MLP applied on the transformed joints tokens:
M M
(3)
ME
M TL((M-))
(4)
Y MIP(MM
(5)
Losses Labeling 3D keypoints is significantly more expensive and slower than 2D keypoints in uncontrolled real-world environments. As a result, we usually collect dataset containing many more 2D annotations than 3D labels. To maximize the utility of all available labels (2D and 3D) and boost performance, we use a mixture of weakly and fully supervised losses to train our model. We denote the ground-truth -tunt 2D joints keypoints
IR, NgN and define the 2D reprojection, and 3D

---

samples
subset
subjects
total
w 2D keypoints
W 3D keypoints
w both
training
6999
149683
144866
9472
4655
validation
1651
28614
27382
2137
905
Table 1: Human Key Points in WOD v1.3.2
reconstruction losses as:
N
L2D =
ly: IY,,, K)
(6)
N i=1
N
1
CDD
Y Y.1l2
N
where || ##12 is the e² vector norm i.. the more robust euclidean distance between predictions. We add small during training, the function is not differentiable at O. Our final loss is given by:
L = LaD XL2D
where À is scalar factor used weigh the losses.
Semi-supervised support In order efficiently train with mixed 2D and 3D labels, we also set training batch to contain pre-defined fraction of 2D-to-3D annotations. This will allow the network to not forget about 3D, when the dataset drastically biased towards 2D annotations. We show study on the effect of the percentage of 3D and the loss balancing factor in figure 4, left.
3 Experimental Results
Datasets Waymo Open Dataset v1.3.2 (WOD) [33] contains RGB and LiDAR range images cap- turing various road users. Recently camera (2D) and laser (3D) key points annotations on portion of human subjects (pedestrians and cyclists) in WOD have been released, namely Waymo Human Key Points dataset v1.3 v1.3.2 (WHKP). We benchmark HUM3DIL and other baselines on the WHKP. As the official WOD/WHKP testing subset is hidden from the public, randomly select 50% of subjects from the WOD validation subset as our validation split, and the rest 50% fall into the testing split for benchmarking. In our experiments, we use the ground-truth camera and LiDAR bounding boxes during training and evaluation, for two reasons: a) disentangling the evaluation of the key points localization from the object detection task; b) setting nasy-tooreproduce baseline for future research works.
Metrics Where applicable, will report two metries: the MPJPE) between predicted and ground-truthh 3D joints, and a similar one, for the 2D case. As the ground-truth is not available for every frame, even for every joint, will use visibility indicator v/ 10, This signals if have a ground-truth annotation for particular joint of particular testing sample j. The MPJPE over a particular dataset will then have the value:
per-joint position error (i.c.
Σ V - Y
Σ
Evaluation on WHKP We train three different methods on the WHKP training subset: ours (i.c. HUM3DIL), THUNDR [9] reimplemented following the original paper, THUNDR with also an ad- ditional depth image as input, ContextPose [10] using publicly available code, and the multi-modal approach [11], for which we report their number similar version the dataset. These are all sfate-of-the-an methods in 3D pose estimation for single persons, from an RGB image. We tried to make comparisons against pure RGB image methods as fair as possible, but there modeling gap that cannot be breached- our architecture naturally exploits LiDAR signal with ease. Also note that we have 1/5 1/14th of the number of parameters of competing methods. We report results on the test split in Table. 2.
5

---

Implementation details In all our experiments we use U-Net backbone [41], with randomly initialized weights. The encoder/decoder convolutional filter sequences are [32, 64, 128, 256] and (256, 128, 64, 32], respectively. The backhone has 2, 095,9 392 parameters. For the Transformer ar- chitecture, we use
stages, an embedding size 256 and 8 heads for the Mul Multt:HeaddAteetiioo layer. We train the network for 50 epochs, with batch size of 16, base learning rate of 1e 4 and exponential decay 0.99. We set the maximum number of LiDAR points to 1024. Our Transformer architecture consists of 3, 229, 696 parameters, with final MLP of 771 neurons. We validate and À le - 2. The complete architecture has 5, 325, 859 parameters. All of our networks were trained single V100 GPU with 16GB of memory. Our code is implemented in TensorFlow. We test the network in inference mode on an Nvidia RTX 2080 GPU, for batch a single example. One pass is done in 8 milliseconds. The main performance/memory bottleneck resides in computing the attention matrix (which is z 1000 x 1000) in the Transformer architecture.
10
Method
MPJPE (cm) +
MPJPE 2D (pixels)
+
ContextPose [10]
10.82
12.95
Multi-modal [TT]*
10.32
N/A
THUNDR [9]
9.62
14.81
THUNDR [9] w/ depth
9.20
13.53
HUM3DIL (Ours)
6.72
8.33
Table Performances of different 3D joint predictors on the WHKP [33] test split. Our full multi-
modal approach is the best performer. (*) Note that [11] was evaluated on different subset of WOD.
Figure 2: Qualitative predictions the WHKP test subset. In each image, from left to right, we have: the input RGB image, the overlayed LiDAR points, and our joints predictions. Our method achieves plau- sible reconstructions even in challenging settings: low lighting conditions, cluttered environments, extreme occlusions or partial views and non-trivial poses. Note that in the case of partial views (e.g. second row from top, second column from left), our method outputs an anatomically plausible human prediction, even with an incomplete RGB signal.

---

Figure 3: Failure the WHKP subset. In each image, from left right, have: the input RGB image, the overlayed LiDAR points, and our 3D joints predictions. Most points of failure relate unusual person appearances or poor capture conditions, limited image support and extreme occlusions.
10.0
9.0
30
6.0
Error (cm)
11000
12.0
Number of visibile joints
7.5
1e:4
99%
se-4
90%
2D Error weight
1e:3
3D in batch (%)
30%
50.3
50%
50:2
20%
1e-1
10%
5e-1
5%
7.5
12.5
15
22.5
(m)
Figure 4: Visualizing network performance on the WHKP validation subset, with respect to semi-supervised choices, distance to LiDAR and keypoint occlusions. Left. We plot a 3D emror surface, where on the X-axis have the 2D loss weight X, and the Y-axis have the percentage of 3D samples in mixed-supervision training batch. The plot shows the importance under-utilize them. Bottom-right. We plot 3D error curve, where on the X-axis we have the distance to the LiDAR clont. Errors were computed for 20 equally concentrated bins, w.r.t distance. The dotted points represent the centers of those bins. Note how the near far away. Top-right. We plot 3D expected, the method degrades when more parts the target human are not visible (due to partial views or self-occlusions).
samples, as performance gradually drops when we
gracefully decays when the target subject either with respect the number of visible joints. As

---

3.1 Ablation studies
In table 3, we ablate different high-level methodological choices in our proposed architecture and report results on the validation set of WHKP. First, we disable the weakly-supervised loss by setting ==0 0. We notice that the error increases, expected, substantially in the 2D MPJPE, but also in the 3D MPJPE. This showcases the importance of using the available 2D training signal, even for inferring 3D joints. Next, we disable the Random Fourier Embedding for the 3D LiDAR points, by replacing it with an identity embedding. This has a fairly low impact on the performance. We also disable the depth input, leaving only the RGB image through the U-Net backbone. The performance drop is not dramatic in this case, as LiDAR point positions are already available as tokens. How- ever, this signals the fact that the feature processing done by the backbone is not redundant, as a gap in performance still exists. When we disable the RGB, we get & 1.3 em drop in performance. We also ablate with the PointNet[40] and PointNet++(33) architectures, instead of Tranformer, and performance is worse. The best performing method has all the components activated, showing their complementary impact.
Method
MPJPE (cm) 7
MPJPE 2D (pixels)
HUM3DIL w/ A/=
8.62
15.61
HUM3DIL w PointNet
8.16
9.96
HUM3DIL w/o RGB
8.06
11.41
HUMBDIL w/o depth
7.63
9.13
HUM3DIL w PointNet++
7.71
9.36
HUM3DIL w/o RFF
7.01
8.79
HUM3DIL full
6.72
8.33
Table 3: Performance of our method with different architectural choices. The model with full fea-
tures including multi-modality, semi-supervised training and Transformer architecture is the best performer.
4 Limitations
Failure cases. In figure 3, we randomly select and show six examples of results where the error exceeds 15 cm. Extreme occlusions and partial views are the most difficult cases to handle. Please note that the results are still, generally, anatomically plausible. Performance decay. We also show the performance decay w.r.t. the distance to the LiDAR human point-cloud (which also controls the sparsity of the LiDAR signal). As we can see from figure 4, bottom-right, our performance drops too far, but still produces reasonable results. We are seeing zs 3 cm
when the subject is too error gap between the best and worst conditions (as captured by the dataset). We also see a perfor- mance degradation (see figure top-right) when the target subject is heavily occluded, due to partial views or self-occlusions. The error increases almost two-fold when going from full-view to severely occluded. General applicability. For now, our network can only process each human instance by individually cropping, so it cannot use information about multiple humans at once (e.g. [17,44, 45]). improve the error and processing speed. Also, we do not utilize temporal information (c.g. [46]) which could further stabilize predictions and improve errors under occlusions.
5 Conclusions
We have presented novel deep neural network architecture, which has been tailored for the needs of modern autonomous driving vehicles: fast, lightweight and accurate, for the problem of 3D human pose estimation from color and 3D signals. Our novel architecture, deemed HUM3DIL, makes usage of both RGB and LiDAR data, by gathering pixel-aligned multi-modal features, that are then fed into a sequence of Transformer stages. The network is thus informed by multi-modal signals, which complement each other in achieving state-of-the art performance. We also train in a semi- semi-supervised regime, with limited annotated 3D data, but with an abundance of 2D labels, almost orders of magnitude more. This makes data collection and annotation easier, as 3D signals are non-trivial and tedious to annotate precisely. The performance of the network is supported by quantitative evaluation on one of the largest relevant datasets in the literature, with methodical ablation studies. For future work, we will explore temporal consistencies between the predictions and include them in motion forecasting and analysis.
8

---

References
[1] J. Wang, S. Tan, X. Zhen, S. Xu, Zheng. Z. He, and L. Shao. Deep 3D human pose estima- tion: A review. Comput. Vis. Image Underst., 210: 2021.
[2] C. Zheng, W. Wu, C. Chen, Yang, S. Zhu, J. Shen, N. Kehtarnavaz, and M. Shah. Deep Learning-Basee human pose estimation: A survey, Dec. 2020.
[3] Y. Chen, Y. Tian, and M. He. Monocular human pose estimation: A survey of deep learning- based methods. Computer Vision and Image Understanding, 192: 102897, 2020. ISSN 1077- 3142. /wt.://ww//coooorr///o.o.o..ooaa com/s http://c/e//.riieeeea///////////
wtR..tttss///. //ww://wwwwwao/ sciencedirect.
[4] B. I. I. A. K. Nikolaos Sarafianos, Bogdan Boteanu. 3D human pose estimation: A review of the literature and analysis of covariates. Comput. Vis. Image Underst., 152:1-20, Nov. 2016.
[5] M. Fürst, S. T. P. Gupta, R. Schuster, O. Wasenmüller, and D. Stricker. HPERL: 3D human pose estimation from RGB and LiDAR. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7321-7327, 2021.
[6] K. Huang, B. Shi, X. Li. X. Li, S. Huang, and Y. Li. Multi-modal sensor fusion for auto driving perception: A survey. Feb. 2022.
[7] D. Fernandes, Silva, R. Névoa, Simöes, D. Gonzalez, M. Guevara, Novais, J. Monteiro, and P. Melo-Pinto. Point-cloud based 3D object detection and classification methods for self- driving applications: A survey and taxonomy. Inf. Fusion, 68:161-191, 2021.
[8] Y. Tian, H. Zhang. Y. Liu, and L. Wang. Recovering 3D human mesh from monocular images: survey. Mar. 2022.
[9] M. Zanfir, A. Zanfir, E. G. Bazavan, W. T. Freeman, R. Sukthankar, and C. Sminchisescu. Thundr: Transformer-basse 3d human reconstruction with markers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12971-12980, 2021.
[10] X. Ma, Su, C. Wang, H. Ci, and Y. Wang. Context modeling in 3d human pose estimation: unified perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6238 6247, 2021.
[11] J. Zheng, X. Shi, A. Gorban, J. Mao, Y. Song, C. R. Qi, T. Liu, V. Chari, A. Commann Y. Zhou, al. Multi-modal 3d human pose estimation with 2d weak supervision in autonomous driving. arXiv preprint ar Xiv: 12. 2021.
[12] S. Saito, 7. Huang. R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel- aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2304-2314, 2019.
[13] A. Zanfir, F. Marinoiu, and C. Sminchisescu. Monocular 3d pose and shape estimation of multiple people in natural scenes-the importance of multiple scene constraints. In CVPR, 2018.
[14] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis. Learning to reconstruct 3d hu- man pose and shape via model-fitting in the loop. In Proceedings of the IEEE International Conference on Computer Vision, pages 2252-2261, 2019.
[15] R. A. Guler and I. Kokkinos. Holopose: Holistic 3d human reconstruction in-the-wild. In CVPR, pages 10884-10894, 2019.
[16] G. Moon and K. M. Lee. 121-meshnet: maaee-tollxxd prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In European Conference on Computer Vision (ECCV), 2020.
[17] W. Jiang, N. Kolotouros, G. Pavlakos, X. Zhou, and K. Daniilidis. Coherent reconstruction of multiple humans from single image. In CVPR, pages 5579-5588, 2020.

---

[18] B. Biggs, S. Ehrhadt, H. Joo, B. Graham, A. Vedaldi, and D. Novotny. 3d muulti-bodies Fitting sets of plausible 3d human models to ambiguous image data. arXiv preprint arXiv:201 1.00980, 2020.
[19] Y. Xu, S.-C. Zhu, and T. Tung. Denserac: Joint 3d pose and shape estimation by dense render- and-compare. In Proceedings of the IEEE International Conference on Computer Vision, pages 7760-7770, 2019.
[20] T. Zhang, B. Huang, and Y. Wang. Object-occluded human shape and pose estimation from single color image. In CVPR, pages 7376-7385, 2020.
[21] A. Arnab, C. Doersch, and A. Zisserman. Exploiting temporal context for 3d human pose estimation in the wild. In CVPR, pages 3395-3404, 2019.
[22] W. Zeng, W. Ouyang, Luo, W. Liu, and Wang- 3d human mesh regression with dense correspondence. In CVPR, pages 7054-7063, 2020.
[23] G. Georgakis, R. Li. S. Karanam, T. Chen, J. Košecká, and Z. Wu. Hierarchical kinematic human mesh recovery. In ECCV, pages 768-784. Springer, 2020.
[24] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi- person linear model. SIGGRAPHI, 2015.
[25] H. Xu, E. G. Bazavan, A. Zanfir, Freeman, R. Sukthankar, and C. Sminchiseseu. GHUM & GHUML: Generative 3D human shape and articulated pose models. CVPR, 2020.
[26] G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer, I. Laptev, and C. Schmid. BodyNet: Volumetric inference of 3D human body shapes. In ECCV, 2018.
[27] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 529-545, 2018.
[28] U. Iqbal, P. Molchanov, and J. Kautz. Weakly-supervised 3d human pose learning via multi- view images in the wild. In CVPR, pages 5243-5252. 2020.
[29] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore. Real-time human pose recognition in parts from single depth images. Commu- nications of the ACM, (6(1):116-122) 2013.
[30] G. MoMon, J. Y. Chang, and K. M. Lee. V2v-posenet: voxellto-voxel prediction network for accurate 3d hand and human pose estimation from single depth map. In Proceedings of the IEEE conference computer vision and pattern Recognition, pages 5079-5088. 2018.
[31] C. Zimmermann, T. Welschehold, C. Dornhege, W. Burgard, and T. Brox. 3d human pose estimation in rgbd images for robotic task learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1986-1992. IEEE, 2018.
[32] W. Kim, M. S. Ramanagopal, C. Barto, M.-Y. Yu, Rosaen, Goumas, R. Vasudevan, and PedX: Benchmark dataset for metric 3D pose estimation of pedestrians complex urban intersections. Sept. 2018
M.
[33] Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2020.
[34] M. Fürst, S. T. Gupta, R. Schuster, O. Wasenmüller, and D. Stricker. Hperl: 3d human pose estimation from rgb and lidar. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7321-7327. IEEE, 2021.
[35] A. Rahimi and B. Recht. Random features for large-scale kernel machines. J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Sys- tems, volume 20. Curran Associates, Inc., 2007. URL https: htttap:://proceedine www..inniisss......m..m.m
Papss----pp....e .df.
10

---

[36] K. Lin, L. Wang, and Z. Liu. End-to-end human pose and mesh reconstruction with transform- ers. CVPR, pages 1954-1963, 2021.
[37] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.JJ Mu, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, t.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Ni, W. Zhang, X. Zhu, and Q. Cao. Pointnet++ grasping: Learning an end-to-end spatial grasp generation algorithm from sparse point clouds. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3619-3625, IEEE, 2020.
[40] C. R. Qi, H. Su, K. Mo, and Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[41] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference Medical image computing and computer- assisted intervention, pages 234-2414 Springer, 2015.
[42] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency func- tions in low dimensional domains. NeurIPS, 2020.
[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NIPS, 2017. URL https://arxiv.ortt org/ /106. 03762. fdf.
[44] M. Fieraru, M. Zanfir, T. Szente, E. Bazavan, V. Olaru, and C. Sminchisescu. Remips: Physi- cally consistent 3d reconstruction of multiple interacting people under weak supervision. Ad- vances in Neural Information Processing Systems, 34:19385-19397, 2021.
[45] G. Moon, J. Y. Chang, and K. M. Lee. Camera distance top-down approach for 3d multi- person pose estimation from a single rgb image. In Proceedings of the IEÉE/CVF international conference on computer vision, pages 10133-10142, 2019.
[46] M. Kocabas, N. Athanasiou, and M. Black. Vibe: Video inference for human body pose and shape estimation. CVPR, 2020.


Reviews:
[{"review_id": "tyUi0_lU3E2", "reviewer": "Reviewer_ZEP4", "summary": "HUM3DIL proposes an approach for 3D human pose estimation that combines RGB and depth (from lidar) with a transformer backbone. They use both 2D keypoint annotations and 3D keypoint annotations to supervise the network during training and cast the problem as a case of semi-supervised learning (i.e. the majority of exemplars have only 2D keypoint annotations). The primary novelty of the paper is the use of a Fourier embedding for the lidar points which, experimentally, appears to have a large effect on the performance of the system.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and clear.\n- The related-work section is fairly thorough and does a good job contextualizing this work.\n- The empirical results are convincing and clearly presented.\n- Good qualitative results are provided, including failure cases.\n\nWeaknesses\n- The paper is fairly incremental. The main novelty is the Fourier depth embedding and, while it has a large impact on this particular task,\na version of the paper that focuses on this encoding's effect on a variety of downstream tasks would be much more compelling. As-is, there is nothing about the approach itself that is particularly tied to the task of human pose detection. An identical setup could be used for arbitrary keypoint detection tasks and a very similar setup (with a different head and loss) could be evaluated on many robotic perception tasks using RGBD input such as segmentation, object detection, object pose estimation, and potentially even localization.\n- The paper lacks a straightforward comparison to the obvious pointnet++ baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I advocate for acceptance of this paper. It's well written, the evaluations are thorough, and it appears scientifically sound. I wish the authors had considered a broader variety of tasks, not just human pose estimation, as the resulting work would likely be more broadly impactful, but this is still a worthwhile contribution in its current form.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Why is a similar approach not tested using pointnet++ as a baseline? It seems like a very obvious comparison and would strengthen the paper. Is there a particular reason this comparison was deemed inappropriate by the authors?\n"}}, {"review_id": "e9JiYkTYUCz", "reviewer": "Reviewer_fkxa", "summary": "This paper proposes to fuse LiDAR with RGB observations for 3D human joints prediction. Pixel-aligned multi-modal features are fed into the transformer. Semi-supervised learning is adopted to utilize abundant 2D labels as well as limited 3D annotations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The paper is clearly presented and easy to follow. \n\n+ The proposed method demonstrates great performance over baseline methods.\n\n+ The ablation study is comprehensive, and qualitative results are presented.\n\n+ The limitation is well discussed.\n\nWeaknesses\n\n+ One claim of the paper is that the proposed HUM3DIL is fast and lightweight. However, in the experiments section, the authors only mention that the model is lightweight without giving the exact numbers  (L188) and providing inference time. It is suggested to provide a comparison of model parameters and inference time with baseline models in Table 2 to clearly demonstrate the efficiency of HUM3DIL.\n\n+ The evaluation is only conducted on one dataset.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well motivated and proposes reasonable methods to address this problem. The improvement on Waymo Open Dataset is large compared with existing methods.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "7L95q-iT7Ub", "reviewer": "Reviewer_cQjf", "summary": "This paper proposes an architecture for 3D human keypoint detection from RGB and Lidar. In comparison to existing methods operating in 2D, the proposed approach performs much better. The proposed architecture, based on a Transformer, works by aligning network features coming from different modes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "One strength of the approach is that it addresses a relevant computer vision problem for the automotive industry. AVs are usually equipped with multiple sensors: it makes sense to use them all and fuse them. In addition, it is important in these settings to detect people in 3D, and not in 2D, since this observation is important for trajectory generation and tracking.\nA second strength is the novelty of the architecture, which, despite being highly inspired by transformers, is very interesting. I find it surprising that the camera extrinsic need to be repeated on all lidar features, I would have not expected that.\nFinally, the paper presents good results on the waymo dataset, outperforming other methods. This comparison is not always fair since the baselines use only 2D images as input, but sometimes an effort was made to include depth in one baseline.\n\nMy main concern is that it is difficult to understand the value of the paper's main contribution. This is due to the lack of simple and interpretable baselines. For example, something that would be interesting is using off-the-shelf trackers (as [1]) and lifting the prediction in 3D using the LIDAR points ( simple averaging in a local neigh. would do I guess ). This leads to the question: what is doing the heavy-lifting in the detection problem? Looking at the results in Table 3, it seems that the LIDAR points (i.e. depth) play the smallest role in the system's performance. This weakness is important since it contrasts what this work aims to do, i.e. using multi-modal inputs for person detection. \nAnother weakness is the strong assumption that the LIDAR bounding boxes are available for evaluation. This kind of assumes that the detection problem is solved a prior, and quite surely the network strongly profits from it. I disagree with the two reasons justifying this choice, since the (a) point location and object detection are part of the same task, they can't be disentagled, and  (II) future work will have a much easier time to reproduce the results if the masks are not groud-truth. In my opinion, the paper would improve with either of these two methods: (a) using the raw lidar map, letting the network do the tracking implicitly, or (b) pre-process the lidar map with some person detection algorithm and only passing the noisy detections to the network.\n\n---- After Rebuttal ----\n\nThe new results seem to show that even without GT-masks the approach seems reasonable for fusing these two modalities.\n\n\n[1] Tracking People by Predicting 3D Appearance, Location & Pose, Rajasegaran et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend rejection based on my concerns about the evaluation. What if (i) a simpler baseline would work as good or better?, or (ii) the primary signal comes from the ground-truth bounding boxes available from the lidar? I believe these questions to be important to understand the impact of the work.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The main issues I would like to see addressed are on the evaluation. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "3VLhVmXnMrM", "reviewer": "Reviewer_xAs9", "summary": "This paper studies the 3D human pose estimation problem in the context of autonomous driving. The authors propose to fuse the camera and lidar information, and leverage the 2D annotations as weak supervision. Although the results are good, the novelty is questionable.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. The experiment results are valid.\n2. The method does not have technical flaws.\n3. The writing is clear.\n\n### Weakness:\n\nThe novelty of this paper is questionable:\n\n1. Fusing camera and lidar depth by U-Net pixel-wise to get multi-modal features is not a novel idea. Fourier domain embedding is also an existing and well-evaluated method. The performance improvement is not surprising.\n\n2. The semi-supervised learning with 2D annotations by minimizing the 3D-2D reprojection error is a very standard method. Could the authors explain the novelty here?\n\n3. I am confused why the authors claim the pixel-aligned feature fusion is a novel contribution. As the authors pointed out, this idea is already proposed in [12]. It is true that the current manuscript focuses on a different application from [12], the usage and technical basis of this feature fusion is not different from [12]\n\nIn summary, this is a valid but not novel work. Not recommended for publication.\n\n----------------------------------------------\n\n### Comments after the Authors' Response:\n\nI appreciate the authors' response. If the authors' clarification is valid, some new questions will arise. The authors clarified that the novelty is applying the random fourier transform and the transformer attention methods in 3D human keypoint estimation task, rather than inventing the concept of random fourer transform or transformer. But:\n\nThe details of applying random fourier transform (which should be very important because the authors claim it is novel) in 3D human keypoint detection is not discussed. In the method section, the random fourier transform only occupies a few lines. But there are many tricks in random fourier transform such as how to carefully choose the parameter distribution and how to choose the dimensionality. Just the task itself, applying random fourier transform in human 3D keypoint detection, under diverse settings and data modality (e.g., depth maps, lidar point clouds), deserves in-depth exploration. But what the paper gives is too shallow.\n\nThe authors claim many novelties but the details is not sufficiently discussed. I would suggest the authors to focus on one or two novelty and give a very insightful discussion of it. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Not recommended for publication. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Although the results are valid, the contribution is incremental. The novelty is weak and not convincing.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)