Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper looks at learning animatable avatars given a small number of depth images with associated known SMPL pose transformations. The proposed approach (similar to prior work e.g. SCANimate) learns: a) forward/inverse skinning weight predictors,  which given a query point x in canonical/posed space respectively predict the forward/inverse LBS weights, and b) a canonical space implicit SDF function, which conditioned on SMPL pose, predicts a pose-dependent shape s.t. its forward transformation looks correct.

While the paper uses a fixed network for a) above, the key contributions relate to how the pose-dependent canonical shape is modeled. This paper proposes to capture each object via a hyper-network which predicts the weights of the SDF function conditioned on SMPL pose parameters. It shows that such a hypernetwork can be meta-learned using training data with strong supervision, but can then be finetuned using only a small number of depth images in varying poses for a novel instance.

The experiments convincingly show the benefits of this approach across different datasets and in particular show how using only a small number of depth images suffice.

main_review: This is a very well-motivated and a very-well executed paper which looks at learning an avatar for a novel instance given a small number of depth images with known poses. In particular:

a) I really liked the insight of modeling the pose-dependent shape variations of each instance via a  hyper-network, and the paper convincingly demonstrates that this network can be effectively finetuned via a small number of depth images (unlike prior methods which use full scans/meshes).

b) The insight of factoring this hyper-network into a base network, only whose residual weights are predicted is also an elegant one.

c) The approach for predicting generalizable skinning weights via a convolutional network is also well-motivated, although I am not sure why a similar hyper-network based approach would not be suitable (especially for forward skinning weights).

d) The experiments, and the visualizations in the supplementary convincingly show that this approach allows more detailed reconstruction over the baselines - similar to prior methods which overfit to a single instance, it can capture details of the 3D shape, while only requiring a fraction of data or optimization time.

e) The ablations, in particular the MLP, PosEnc, and SIREN baselines clearly justify the benefits of learning a hypernetwork to model each instance instead of a 'normal' conditional network to predict shape by concatenating the SMPL pose.

Overall, this is a good paper that proposes a novel and insightful solution for learning animatable avatars from a small number of depth images, and I would argue for acceptance.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper aims to create generalizable and controllable neural signed distance fields from monocular depth observations. The authors propose a hypernetwork to represent neural SDFs across different human shapes and cloth types. For fast fine-tuning, they additionally leverage meta-learning to learn an initialization of the hypernetwork. This method achieves impressive reconstruction results and demonstrates a cool application that reconstruct high-quality human shapes from 8 monocular depth frames in 2 minutes.
main_review: Originality:

1. This paper propose a novel representation for human shapes. It first learns a meta-SDF to fit canonical 3D subjects with diffrent genders, body shapes, cloth types, and poses. To model the shape deformation caused by the poses, it then meta-learn a hypernetwork that takes the poses as input and predicts the residuals to the parameters of the previously learned meta-SDF.
2. This representation can be seen a combination of coarse and fine representations, which effectively and efficiently models the shape deformation.

Quality:

1. The submission is technically sound.
2. The claims are well supported.
3. The comparisons and ablation studies are sufficient.

Clarity:

1. The submission is clearly written and well organized.

Significance:

1. This paper proposes a method that quickly reconstructs high-quality human shapes from monocular depth frames, which is very practical and largely promotes the field of human reconstruction. It will probably enlighten follow-up works.
limitations_and_societal_impact: I do not see potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a framework that can learn an animatable implicit avatar for clothed humans from very few observations. The key idea is using meta-learning to learn a prior (specifically speaking, a good initialization) to enable fast fine-tuning at inference time. To enable animations of various cloth types, this paper proposes to perform meta-learning on a hyper-network which controls the parameters of the neural implicit function conditioned on pose configurations. Results show that the proposed method is able to generate clothed human avatars from few shot observations. 

main_review: ### Strengths:
* The idea of meta-learning a pose-conditioned hyper-network is interesting and inspiring. To the best of my knowledge, this paper is the first one that applies meta-learning in human modeling. The usage of meta-learning can accelerate network optimization by a large margin, e.g., from 10 hours to 1.6 hours as presented in the paper. The design of meta-learning a hyper-network is also a valuable idea for the community. 

* This paper is well motivated and addresses an important problem: current methods for neural SDFs-based clothed avatars take too much optimization time. Compared to SOTA methods like SCANimate, the proposed method can generate comparable and even better results, while being much more efficient. 

* This paper is overall well-written and easy to follow. 

### Weaknesses:
* My major concern is about the claim that this method requires "only" depth frames as input.  Although the authors keep emphasizing that their method only needs partial depth observations for fine-tuning, in fact the method assumes perfect SMPL registrations and ground-truth bone transformations, which is only possible when complete 3D scans are available. In the experiments, the authors only use depth rendering from *complete* CAPE meshes. In the supplemental material, the authors conduct an additional experiment on raw scans, but that experiment still requires ground-truth SMPL registrations to clean and complete the scans, and also to calculate the bone transformations. In one word, I don't think the experiments can support the claim in L55-56 ("given only a few depth observations of an unseen clothed human as inputs") because of its dependence on ground-truth SMPL registrations. It would make this paper stronger if the authors can conduct experiments on *real* partial depth inputs, i.e., raw depth images captured by an RGBD sensor like Microsoft Kinect or Intel RealSense. 

* The numerical evaluation in Table 2 is not very convincing. The numbers in the last 3 columns are really closed to each other. Table 2 makes me feel that the performance gains mainly stem from the usage of SIREN activations, and that the slight improvement in the last column is due to the increased number of network parameters. 


### Minor:
* L226: Although reader can guess that the meshes are rotated around the yaw axis, it would be better if the authors can explicitly specify it. 

* L311: to representation -> to represente

limitations_and_societal_impact: The authors have discussed the limitations in the supplemental document and the potential societal impact in Section 6. No more improvement is needed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel learning method to generate controllable neural SDF for non-rigid clothed human body from a few depth scans. The entire pipeline consists of three main parts, i.e., canonicalization, neural SDF, and hyper-net. The neural SDF takes canonicalized points as inputs and learns a static representation. The hyper-net is used to fine-tune the static neural SDF and generate dynamic SDF efficiently for forward LBS based animation.

main_review: As far as I am concerned, the method proposed in this work is novel and technically sound. The target application can be useful for animation of clothed 3D human body from a few depth incomplete input. The presented animation results demonstrate some advantages over those obtained via state-of-the-art methods in certain cases from the CAPE dataset.

On the other hand, I think this paper needs some major revisions on paper writing, especially Section 4. I suggest to add an overview section or paragraph to present the entire pipeline and clearly describe the input and output of each stage/component. The inference part should be put into an individual subsection. In Section 5, the CAPE dataset contains 15 human objects but only 14 are used for the experiments (among which 10 for training, 4 for validation and test). Is there any particular reason for this choice? Also, the depth frames are synthetically rendered from the CAPE dataset. There are no examples using input from real depth sensor or dataset other than CAPE.

Overall, I think this work is technically sound but the presented results are not convincing enough.
limitations_and_societal_impact: Please see the main review part.
review_title: null

Meta-review: This submission introduces a method that enables the generation of realistic clothed human avatars from monocular depth images. While the initial reviews are mixed, after rebuttal, all reviewers are positive and recommend acceptance.  The AC agrees.  The authors should try to address the reviewers' concerns in the camera-ready version. This includes adding the results from the rebuttal period, clarifying the concerns on meta-learning as reviewer XKUG suggested, among others.
Decision: Accept (Poster)