Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to refine the 3D human poses. To this end, the authors equip neural radiance fields with human skeletons, where they design a skeleton-relative embedding based on the human skeleton. By training the A-NeRF on the video, the human poses are optimized to fit the observations.
main_review: Originality:

1. This paper proposes a novel skeleton-relative embedding, which enables the neural radiance field to represent dynamic humans.
2. A-NeRF allows the human pose refinement during training, which improves the quality of rendering and estimated poses.

Quality:

1. Although the results in the submission look good, the proposed skeleton-relative embedding does not make sense to me. Why does relative distance work better than the bone-relative position. I think that reducing a position to a distance may cause ambiguity, since points on a spherical surface have the same distance.
2. For experiments, the results of NeuralBody always have black blobs, which seems strange. Is the NeuralBody trained with black background and visualized with the white background?
3. The training and test data is unclear. What are the training views and test views?
4. The motivation of this submission is unclear. At the first, I thought that it aims to refine the human poses with differentiable rendering. However, the experimental results are mostly about the view synthesis and animation. So I thought that it aims for view synthesis. However, the quantitative comparisons are only presented in the supplementary material, and the authors only compare their method with NeuralBody.

Clarity:

1. This submission is poorly written. Although I can understand the overall idea, it is still difficult for me to learn about the technical details. The authors could leave some unrelated contents in the supplementary material and describe their core method firstly.
2. For example, the readers probably cares about how to combine NeRF with human skeletons, so after describing the basic NeRF, the authors could introduce how to improve the input of NeRF based on 3D skeletons. The position encoding (line 163), 4D space-time encoding (line 169), and conditioning on pose (line 172) are unrelated with the skeleton-relative embedding and should be put in other places.
3. Similarly, the NeRF and articulated skeleton pose model are well-known techniques, which can be described more simply and can be put in the supplementary material.
4. The quantitative comparisons on view synthesis and the ablation studies are important, which should be put in the experiment section, while not in the supplementary material.

Significance:

1. This paper develops a human representation for view synthesis and animation from sparse camera views. It probably enlighten follow-up works in the field of human view synthesis.

Typos:

1. Line 201: "Much simpler to compute are distances from q to all bones m".
limitations_and_societal_impact: I do not see potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents A-Nerf which is a generative neural body model capable of rendering a human under novel viewpoints and poses. A-Nerf is an extension of Nerf to capture dynamic human bodies. It's achieved by conditioning Nerf on the skeletal body pose information. The paper demonstrates that naively conditioning Nerf on the body pose information does not lead to optimal performance and therefore proposes a set of relative encodings of the pose information that results in significantly better renderings without the need of a full-body mesh model. The experiments are performed for motion capture as well as novel view synthesis where the proposed method demonstrates superior performance as compared to existing methods. 
main_review: **Pros:**

+ The paper is well written and easy to read. 
+ The proposed pose relative embeddings are novels and intuitive. I like the idea of windowed positional encodings to reduce the influence of parts that are far away from the sampled point. 
+ The supplementary material provides an extensive ablation study of the proposed approach. 
+ The proposed method provides better renderings as compared to the state-of-the-art method NeuralBody even though it doesn't require surface mesh. 
+ I like the narrative of A-Nerf as a "Neural Body Model". Unlike existing body models, e.g., SMPL, A-Nerf can also capture additional finer details such as Caps, glasses, etc. that are otherwise not easily possible with existing human body models. 
+ The paper demonstrates the ability to refine initial pose estimates by rendering pose-conditioned images using A-Nerf. This is a solid contribution of the paper. Unlike, other differentiable rendering-based methods, A-Nerf does not require a mesh model hence it can inspire future research into test-time optimization with model-free differentiable rendering. 
+ The proposed approach doesn't require MV data to work and relies only on monocular video data. 
+ The experiments are performed for motion capture as well as novel-view synthesis. The evaluation protocols are well-defined and reproducible. 


**Cons:**

I cannot think of any significant limitation of the paper except its slightly incremental nature i.e., it mainly augments Nerf with pose conditioned embeddings. 

**Questions to authors:**

- Have the authors tried learning the cutoff parameters?  
- Have the authors visualized the underlying geometry of the person? Does it show anything reasonable? 
-  Since most available methods for skeletal human pose estimation only provide joint positions, it will be great if the authors can comment on what will happen if the body pose information only consists of joints positions, rather than joint rotation angles and a T-pose? What has to be changed? 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission presents a way for adding articulation to neural radiance fields. The main contribution is an encoding of the query points and view directions relative to a skeleton. The proposed formulation is trained per subject on a video sequence and optimizes for the 3D skeleton configuration in each frame and a global MLP which represents the NeRF. This defines the motion tracking throughout the input sequence and the recovered model can be reposed by rendering frames with new joint locations.
main_review: Strengths:
- Adding an articulation model to a NeRF seems a novel and significant contribution
- Novel poses can be rendered photorealistically.

Weaknesses:
- The formulation seems very expensive to get the motion tracking of a single subject 60 hours of training on 2 high end GPUs is needed.
- It is unclear if the model can work with very short sequences. How long are the videos and how much pose variation in the videos is needed to avoid overfitting of the NeRF? My concern is that this might behave similar to when a vanilla NeRF is trained on sparse views and just overfit to the small number of poses.
- To me it is not clear how with the skeleton relative encoding the NeRF is still able to reason about view dependent lighting effects given the NeRF does not have access to the skeleton configuration. How is the network still able to recover the relative configuration of the lighting environment, view direction, surface position and orientation in the original world coordinate system. It seems without this it would be hard for the network to recover view dependent lighting effects.
- Similar to the comment above it is not clear to me how the network could understand how the lighting behaves under camera motion, given everything is expressed relative to the camera. From the pictures in the submission it is hard to tell if view dependent lighting effects are recovered. Example renderings that show how the appearance changes if the view direction input to the network is changed but the actual rendered view direction stays constant would be helpful to better understand the capabilities of the proposed network.


Summary:
May main concern with this paper is that the effect of the skeleton relative encoding on the view dependent effects is not discussed. Given that the view dependent effects is one of the key properties of NeRF I find it important that this would be discussed and evaluated better. At this point it is not clear to me if the proposed formulation is even still able to generate view dependent effects or if this part of the formulation could even be removed. 


After Author Reply:
In the author's replies it turns out that the problem of modeling illumination more accurately is non-straight forward and the model proposed in this paper even though in my opinion slightly unintuitive works better than obvious alternatives. I don't think these issues need to be resolved for publishing this work. In my opinion the original submission is lacking a discussion of this limitation but I feel confident that this can be incorporated in the final version and left for future work. I adjusted my rating and time spent on the review accordingly. 
limitations_and_societal_impact: I think potential risks of creating models without consent of people is addressed. One point that could be emphasized more is that this method allows to render people in unseen poses which adds the risk of generating fakes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents modelling articulated neural radiance fields for human dynamics from monocular videos. To model NeRF for dynamic human, it encodes 3d spatial points and ray directions w.r.t the articulated skeleton poses, which are fine-tuned together when training the NeRF field. Various skeleton-based feature encoding are experimented in consideration of both performance and memory. 
main_review: There are 2 major contributions from the work. It first extends the static NeRF modeling to dynamic human motion by encoding the spatial points w.r.t. the bone transformations. Secondly, it utilizes the differential NeRF rendering to fine tune the estimated human pose that better aligns with image pixels. i am not fully convinced with either of them and have quite a few concerns (see detailed comments below). 

- I do not find any sentences in the paper describing how the background is removed from the NeRF rendering. A foreground mask is needed for training NeuralBody and how do you obtain good masks for the evaluation datasets? Could the dark noise around the person come from imperfect segmentation? Figure A1 shows both your results and NeuralBody with a background and how do you obtain them? Related to this background question, is your image metrics (SSIM and PSNR) evaluated over the whole image or only for the region of interest (around the target human)? All of these important details are missing in the paper.

- The paper provides comparisons to NeuralBody and with ablation studies, demonstrates it is helpful to fine tune the estimated skeleton poses. I agree for in-the-wild videos, fine-tuning the poses seem a good idea. However,  it does not demonstrate the superiority of the proposed skeleton-based feature encoding over other approaches, such as anchoring over statistical human 
surface as in NeuralBody. It seems to me that NeuralBody can also be easily extended to fine tune the skeleton but i realize it might be an unfair comparison to this submission. However, I would like to see both methods compared on synthetic dataset or dataset with ground-truth pose estimations for a better understanding on different feature encoding approaches.  

- Pose estimation:  how important the regularization term is? The pose estimation error could theoretically be compensated by the NeRF field (overfitting). How do you ensure they are decoupled well? Besides, such a pose estimation procedure with NeRF training is very expensive and it does not seem a very fair comparison by comparing to feed-forward approaches (such as SPIN which the method is initialized from). I would suggest to also compare with some nonlinear optimization based methods, such as Simplify or Zanfir et al ECCV 2020, followed with SPIN initialization. 

- Qualitative results:  it will be good to visualize the 3d geometry reconstructions from the trained NeRF field. The appearance looks very noisy for the subject in fig 1 and 3 (with clothing color diluted into the skin).  Any reasons?  

- I think there could be a fundamental problem by transforming spatial points w.r.t. the skeleton as NeRF feature encodings. Suppose there is a constant lighting source in the scene and now how the subject is located w.r.t. this lighting source should largely affects its appearance. However, the proposed encoding scheme will result in global transformation invariant features and will the ANeRF produce right lighting effects, say the subject is rigidly translating in the scene?

- Table A7: for this monocular and multiview comparison, it is not clear from which camera views you evaluated the approach?  

- Some missing related works:
   * D-NeRF: Neural Radiance Fields for Dynamic Scenes
   * Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

limitations_and_societal_impact: Seems ok.
review_title: null

Meta-review: The paper proposes A-Nerf, a generative neural body model capable of rendering a human under novel viewpoints and poses. A-Nerf is an extension of Nerf to capture dynamic human bodies. This is achieved by conditioning Nerf on the skeletal body pose information. The paper demonstrates that naively conditioning Nerf on the body pose information does not lead to optimal performance and therefore proposes a set of relative encodings of the pose information that results in significantly better renderings without the need of a full-body mesh model.

Reviewers raised concerns regarding the similarity or advantages to the proposed formulation with NeuralBody, which the rebuttal addressed to some extent. Reviewers agree on the novelty of the formulation, specifically the spatial encodings for ray and query point with respect to the skeleton bones. The paper is recommended for publication.  

Decision: Accept (Poster)