Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Human mesh recovery methods commonly resort to 2D keypoint supervision to be able to handle in-the-wild images, as these often lack 3D ground truth. This submission argues that binary silhouettes are a better source of weak supervision and that classical segmentation techniques are less susceptible to domain shifts that 2D keypoint detectors. As such, silhouettes can be used to train shape and pose regressors on more heterogenous data without the need for annotating and retraining keypoint detectors, which this submission does: A SPIN-like method (Kolotouros et al., 2019) is proposed that replaces the keypoint-based loss and refinement, with a silhouette-based loss that allows for comparisons between the model and observed silhouettes. Key here is a differentiable skeletonisation operation that pares the silhouette down before applying the loss. The proposed method compares favourably against the state-of-the-art on a variety of datasets. The submission additionally demonstrates that the resulting scheme is more robust to adversarial perturbations than competing approaches.
main_review: Overall, I find this to be an interesting approach. I am somehow impressed that the submission makes the case for background subtraction as a replacement for keypoint-based fitting. In principle, I actually agree that keypoints by themselves are not reliable enough especially due to the domain adaptation problem as pointed out by the authors. The results also partly bear this out, especially when adversarial perturbations are applied. In general, incorporating low-level information can be very useful for pose estimation and indeed help address domain shifts. Silhouettes also provide information that keypoints don't include (e.g. shape, depth relations). 

Nonetheless, I completely disagree with the following: [L41]: "traditional FG [foreground] segmentation techniques are mostly unaffected by input domain shifts (Fig. 1C) and thus the FG masks are usually robust and easy to obtain in diverse target scenarios". Even if we assume this to be the case, it's notoriously hard to get reliable foreground segmentations without learning. It's especially hard in crowded scenarios or in poor viewing conditions. There's also the problem of self-occlusion. This is briefly mentioned at the very of the paper in the "Limitations" paragraph, but it's a really major problem that should somehow be addressed much more explicitly when the method is being introduction. In most of the visual examples, the depicted people conveniently do not self-occlude much, but in Fig. 1 C/D, the 3rd example demonstrates the problem: The skeletonisation procedure misses the right arm entirely. 

So while I agree that keypoint detection has its limitations that can partially be overcome by alternate sources of supervision, foreground segmentation (without learning) has different and partially more severe limitations. The submission also argues (or at least implies) that the lack of point-to-point correspondences in the proposed loss is an advantage over keypoint-based losses (e.g. L213), but this is not the case. If your 3D regressors provides you with a poor initialisation, the lack of correspondences is a problem. It is not a problem and can even be helpful when you are close to the solution and do not want to fit your model to observation noise but there is a trade-off.

The results are good enough for me, but I'm not sure about the fairness of the comparison against the state-of-the-art. Table 2 compares against other methods on H36M, and this approach outperforms others in the weakly- and unsupervised setting. However, it is very important especially in this area to have some comparisons that control for differences in the training data. Having the right pose distribution is at least as important as being able to handle shifts in appearance. As such, it would be good to have direct comparisons against HMR or SPIN using the same training setup. Given that SPIN is used for the ablation study later, that would probably be a good choice. The comparison against 3DPW is similarly problematic. Why aren't HMR and SPIN also compared in the weakly-supervised setting? It is not very surprising that SMPLify is significantly worse, given that it relies on optimising the method. Otherwise I think the rest of the experiments are nice, e.g. the ablation study comparing different losses as well as the ones with adversarial perturbations / challenging low-resolution images. What I'm missing however is a description of how the silhouettes are obtained on the UAP-H36M and LR-3DPW datasets. This is a critical detail that will help me judge these results better.

Silhouettes have a very long history in this area, and it's great that the submission cites some older papers. However, there are a lot of other relevant references that need to be taken into consideration. These need not all be cited, but I mention them here because of certain inaccuracies in the text: One very early method that is relevant is the 2002 paper of Sminchisescu and Telea "Human Pose Estimation from Silhouettes: A Consistent Approach Using Distance Level Sets". If I'm not mistaken, they also use something very similar to the "Distance-map" described in L153-L159.  Figure 3 in that paper shows the "outwards distance-map" as it is referred to in this submission, and the full objective is described in section 4. As such, it is inaccurate to that this has been "introduce[d]" here (L152). I also think the description of "chamfer[-]based distance[s]" (L145-L149) is inaccurate, as many rely on distance transforms as well, e.g. Gavrila & Davis (1996) "3-D Model-based tracking of humans in action: a multi-view approach" (Fig. 6 and section 4.1), described in more detail I think in Gavrila (1998) "Multi-feature Hierarchical Template Matching Using Distance Transforms": "The chamfer distance consists thus of a correlation between T and the distance image of I, followed by a division." A couple of newer methods that use silhouettes include Alldieck et al., (2017) "Optical Flow-based 3D Human Motion Estimation from Monocular Video" who use a distance-transform-based loss as well, and Omran et al. (2018) who use part segmentations as an intermediate step before lifting to 3D.

Just to be clear, I understand that a key difference here is the skeletonisation step, rather than the direct comparison of distance transforms applied to silhouettes, and that this extra step is helpful here (Table 4). The text however still needs to be amended as suggested. It would also probably not hurt to refer to a few works on skeletonisation based on distance transforms, as this also has a long history, e.g. Wright et al. (1995) "Skeletonization using an extended Euclidean distance transform" or Latecki et al. (2007) "Skeletonization using SSM of the Distance Transform". Maybe there are more appropriate references. What appears to be novel here as well is the differentiable skeletonisation but I am not sufficiently familiar with this problem area not to mention the use of distance transforms and skeletonisation outside of pose estimation, so I'm not sure.

I'm not sure why the binary activation function described in section 3.2 is necessary. The submission justifies this by stating that "a non-binary silhouette is not suitable for distance-map and topology definitions" (L176). Why would a "silhouette obtained via a fast differentiable renderer" (L174) be non-binary? If every mesh vertex has the same colour and no lighting is enabled, the resulting image will be binary and would not break differentiability.

Editing suggestions:
- the caption in Figure 2: The description of the topological-skeleton extraction in B is also in 3.1.3 (not just 3.2).

limitations_and_societal_impact: The authors address the potential misuses of 3D mesh recovery for surveillance and caution against such applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a method for human pose and shape estimation that uses silhouette information as an intermediate representation that is well suited for domain shifts. Albeit not new as a concept, the proposed implementation contains several novelties (computation of distance map, efficient approximation of Chamfer-like distance metric), is sound, and effective.
main_review: Missing related work (pose estimation from silhouette, as proposed here):
Creatures great and SMAL: Recovering the shape and motion of animals from video

The approximation of distance maps via convolution and pooling operation is creative, effective, and novel. Are there any corner cases (corners?) where the approximation fails? Could you quantify the approximation to a true distance map and visualize some examples qualitatively too?

How much faster is this method to the non-convolutional computation of distance maps and chamfer distance (each measured individually). I don't doubt that it is faster, but like to know how much.

The ablation study is extensive in terms of accuracy.

Overall the paper has sufficient novelty, is easy to follow, and the evaluation is sufficient; showing clear improvements of model components and fairs well in comparison to related work.
limitations_and_societal_impact: n
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an online adaptation method to adapt pretrained models to new datasets through the silhouette and their proposed topological skeleton via inward distance map and outward distance map. They also propose a Chamfer-inspired topology-alignment objective to align the silhouette better. 
main_review: The results of the paper are compelling and the generalization is good. The optimization on Thermal and low-res images is also amazing. However, as the loss function is only limited to T, I am not sure why not added the previous D and S as loss functions?
limitations_and_societal_impact: Will the method work on occluded cases?
like images in https://arxiv.org/abs/1905.07718 or https://prox.is.tue.mpg.de ?

Why the loss function only involves T as Eq 4?
Can D and S as loss function?

The writing and organization are poor and a little hard to follow. 
For examples, citation [5] [6] is the same person, however, the names are cited differently.
line 197: double the.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a novel method to tackle the lack of annotated data for 3D human pose recovery. The core observation is that silhouette estimation is not prone to domain gaps. Hence, the authors develop the topological skeleton, obtained from silhouette, to guide the training. 
main_review: Strengths:
- The paper is well-written. It is a joy reading this paper.

- The proposed method is sound and interesting. Specifically, 1) transferring chamfer distance, a normally 3D loss to images is innovative; 2)  computation of the ridge-lines via convolution-friendly operations is carefully designed and beautifully implemented. 

- Thorough ablation studies. 

- Insightful discussions. The observation on the robustness of silhouette estimation across domains is highly thought-provoking. Also, it is less relevant but important to highlight that articulation degradation is the key result of the domain gap (Line 345). 

Weaknesses:
-  Please explain the setting used for HMR and SPIN in Table 2, under the weak sup. setting. What does "unpaired" mean?

- Optimization in the loop is used. However, this involves using a well-established regressor. It raises a concern that the network regressor may play an over-important role in determining the performance of the method, because the regressor is trained in a supervised way, usually making it much better even if there exist domain gaps.

- As stated in the Limitation section that "inter-part occlusion" remains a challenge, is this an inherent limitation of the proposed method? i.e. impossible to solve?

- Unlike many fields in which unsupervised learning can produce close results to the supervised learning counterpart, the unsupervised learning in 3D human pose recovery is, however, far from being practical since there is a large performance gap. I would like to hear the authors' comments on this.

Minor:
- Line 332: it may be more convincing to provide some qualitative evidence to support the statement.
limitations_and_societal_impact: The authors have stated clearly the limitations and societal impact, and provide sound solutions to address them.
review_title: null

Meta-review: This submission has received 4 positive final ratings: 6, 7, 6, 7.
The reviewers appreciated overall novelty, extensive experiments, strong empirical performance and clear presentation.
The remaining minor questions and concerns were addressed by the authors in the rebuttal, as acknowledged by the reviewers.
As a result, the final recommendation is to accept for poster presentation.
Decision: Accept (Poster)