Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to fuse LiDAR with RGB observations for 3D human joints prediction. Pixel-aligned multi-modal features are fed into the transformer. Semi-supervised learning is adopted to utilize abundant 2D labels as well as limited 3D annotations.
strengths_and_weaknesses: Strengths
+ The paper is clearly presented and easy to follow. 

+ The proposed method demonstrates great performance over baseline methods.

+ The ablation study is comprehensive, and qualitative results are presented.

+ The limitation is well discussed.

Weaknesses

+ One claim of the paper is that the proposed HUM3DIL is fast and lightweight. However, in the experiments section, the authors only mention that the model is lightweight without giving the exact numbers  (L188) and providing inference time. It is suggested to provide a comparison of model parameters and inference time with baseline models in Table 2 to clearly demonstrate the efficiency of HUM3DIL.

+ The evaluation is only conducted on one dataset.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well motivated and proposes reasonable methods to address this problem. The improvement on Waymo Open Dataset is large compared with existing methods.

issues: See weaknesses
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper studies the 3D human pose estimation problem in the context of autonomous driving. The authors propose to fuse the camera and lidar information, and leverage the 2D annotations as weak supervision. Although the results are good, the novelty is questionable.
strengths_and_weaknesses: ### Strengths:

1. The experiment results are valid.
2. The method does not have technical flaws.
3. The writing is clear.

### Weakness:

The novelty of this paper is questionable:

1. Fusing camera and lidar depth by U-Net pixel-wise to get multi-modal features is not a novel idea. Fourier domain embedding is also an existing and well-evaluated method. The performance improvement is not surprising.

2. The semi-supervised learning with 2D annotations by minimizing the 3D-2D reprojection error is a very standard method. Could the authors explain the novelty here?

3. I am confused why the authors claim the pixel-aligned feature fusion is a novel contribution. As the authors pointed out, this idea is already proposed in [12]. It is true that the current manuscript focuses on a different application from [12], the usage and technical basis of this feature fusion is not different from [12]

In summary, this is a valid but not novel work. Not recommended for publication.

----------------------------------------------

### Comments after the Authors' Response:

I appreciate the authors' response. If the authors' clarification is valid, some new questions will arise. The authors clarified that the novelty is applying the random fourier transform and the transformer attention methods in 3D human keypoint estimation task, rather than inventing the concept of random fourer transform or transformer. But:

The details of applying random fourier transform (which should be very important because the authors claim it is novel) in 3D human keypoint detection is not discussed. In the method section, the random fourier transform only occupies a few lines. But there are many tricks in random fourier transform such as how to carefully choose the parameter distribution and how to choose the dimensionality. Just the task itself, applying random fourier transform in human 3D keypoint detection, under diverse settings and data modality (e.g., depth maps, lidar point clouds), deserves in-depth exploration. But what the paper gives is too shallow.

The authors claim many novelties but the details is not sufficiently discussed. I would suggest the authors to focus on one or two novelty and give a very insightful discussion of it. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Not recommended for publication. 
issues: Although the results are valid, the contribution is incremental. The novelty is weak and not convincing.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an architecture for 3D human keypoint detection from RGB and Lidar. In comparison to existing methods operating in 2D, the proposed approach performs much better. The proposed architecture, based on a Transformer, works by aligning network features coming from different modes.
strengths_and_weaknesses: One strength of the approach is that it addresses a relevant computer vision problem for the automotive industry. AVs are usually equipped with multiple sensors: it makes sense to use them all and fuse them. In addition, it is important in these settings to detect people in 3D, and not in 2D, since this observation is important for trajectory generation and tracking.
A second strength is the novelty of the architecture, which, despite being highly inspired by transformers, is very interesting. I find it surprising that the camera extrinsic need to be repeated on all lidar features, I would have not expected that.
Finally, the paper presents good results on the waymo dataset, outperforming other methods. This comparison is not always fair since the baselines use only 2D images as input, but sometimes an effort was made to include depth in one baseline.

My main concern is that it is difficult to understand the value of the paper's main contribution. This is due to the lack of simple and interpretable baselines. For example, something that would be interesting is using off-the-shelf trackers (as [1]) and lifting the prediction in 3D using the LIDAR points ( simple averaging in a local neigh. would do I guess ). This leads to the question: what is doing the heavy-lifting in the detection problem? Looking at the results in Table 3, it seems that the LIDAR points (i.e. depth) play the smallest role in the system's performance. This weakness is important since it contrasts what this work aims to do, i.e. using multi-modal inputs for person detection. 
Another weakness is the strong assumption that the LIDAR bounding boxes are available for evaluation. This kind of assumes that the detection problem is solved a prior, and quite surely the network strongly profits from it. I disagree with the two reasons justifying this choice, since the (a) point location and object detection are part of the same task, they can't be disentagled, and  (II) future work will have a much easier time to reproduce the results if the masks are not groud-truth. In my opinion, the paper would improve with either of these two methods: (a) using the raw lidar map, letting the network do the tracking implicitly, or (b) pre-process the lidar map with some person detection algorithm and only passing the noisy detections to the network.

---- After Rebuttal ----

The new results seem to show that even without GT-masks the approach seems reasonable for fusing these two modalities.


[1] Tracking People by Predicting 3D Appearance, Location & Pose, Rajasegaran et al.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend rejection based on my concerns about the evaluation. What if (i) a simpler baseline would work as good or better?, or (ii) the primary signal comes from the ground-truth bounding boxes available from the lidar? I believe these questions to be important to understand the impact of the work.
issues: The main issues I would like to see addressed are on the evaluation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: HUM3DIL proposes an approach for 3D human pose estimation that combines RGB and depth (from lidar) with a transformer backbone. They use both 2D keypoint annotations and 3D keypoint annotations to supervise the network during training and cast the problem as a case of semi-supervised learning (i.e. the majority of exemplars have only 2D keypoint annotations). The primary novelty of the paper is the use of a Fourier embedding for the lidar points which, experimentally, appears to have a large effect on the performance of the system.
strengths_and_weaknesses: Strengths
- The paper is well-written and clear.
- The related-work section is fairly thorough and does a good job contextualizing this work.
- The empirical results are convincing and clearly presented.
- Good qualitative results are provided, including failure cases.

Weaknesses
- The paper is fairly incremental. The main novelty is the Fourier depth embedding and, while it has a large impact on this particular task,
a version of the paper that focuses on this encoding's effect on a variety of downstream tasks would be much more compelling. As-is, there is nothing about the approach itself that is particularly tied to the task of human pose detection. An identical setup could be used for arbitrary keypoint detection tasks and a very similar setup (with a different head and loss) could be evaluated on many robotic perception tasks using RGBD input such as segmentation, object detection, object pose estimation, and potentially even localization.
- The paper lacks a straightforward comparison to the obvious pointnet++ baseline.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I advocate for acceptance of this paper. It's well written, the evaluations are thorough, and it appears scientifically sound. I wish the authors had considered a broader variety of tasks, not just human pose estimation, as the resulting work would likely be more broadly impactful, but this is still a worthwhile contribution in its current form.
issues: - Why is a similar approach not tested using pointnet++ as a baseline? It seems like a very obvious comparison and would strengthen the paper. Is there a particular reason this comparison was deemed inappropriate by the authors?
review_title: null

Meta-review: metareview: The addressed problem is very important and relevant for robotics and autonomous driving, and the proposed approach shows a very good performance over current methods. There are different opinions within the reviewers regarding the novelty of the approach, which means that the work should be placed into context more precisely with a stronger focus on the novel ideas. Also, the experiments could be enriched by adding a comparison of model parameters and also a further data set.  

Post-rebuttal:
The authors have given detailed answers to the reviewers and also shown new results on non-GT detection masks, which has significantly improved the quality of the submission. Still, the authors are advised to put more emphasis on the Random Fourier Embedding technique, both in the technical description and in the experimental evaluation part (see comments by xAs9 on parameter distribution and dimensionality). 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)