Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a method to synthesize a novel view image of a person given the sparse multiview images. The paper proposes time-augmented skeletal features---the pixel-aligned features that encode NeRF-like rendering fields by aggregating image features across  time and views, parametrized by a human mesh model. They use transformer networks to learn fusing multiview image features over time. The resulting features are used to generate an implicit function that can predict RGB and opacity. They demonstrate that the proposed method is generalizable to unseen identities and poses, validated on ZJU-MoCap and AIST datasets. 

main_review: Strength

(+Novelty) This paper is built upon Neural Body [31] framework (NeRF+SMPL) that parametrizes the neural radiant field using the SMPL vertices, which synthesizes a novel view image given a set of multiview images. This paper takes a new step to learn a visual representation of humans using pixel-aligned features that can be used to generalize to new subjects. In this sense, it combines NB and PIFu, which is novel and shown to be very effective. The image feature is in particular useful for occluded regions of body because the learned image features are rather viewpoint invariant.  Further, they propose a novel way to fuse multiview and time features through self-guided attention through transformers, which shows stronger performance compared to existing average or max-pooling (similar idea for multiview fusion has been used in [a] though). 

[a] Stable view synthesis, CVPR 2020

(+Performance) The experiments with multiple baselines (PixelNeRF, NB) show that the proposed image feature learning is highly effective on generalization of subjects and poses in particular, for unseen subjects and poses in terms of PSNR and SSIM metrics. The result in the supplementary video well illustrates the strengths of the proposed method. Further, the paper includes substantial ablation study that shows the impact of each component.  


Weakness:

(Justification of transformer) Learning feature fusion over views is interesting but not sure how effective it is (do we really need to learn, or can use a weighted average based on viewing directions to fuse similar to [a]?). I suspect that performance wise, it will be pretty similar. If it is not way better than such weighted average skeme, having transformer over views is overkill and may limit the generalizability to a new camera configuration.   

(Justification of temporal aggregation) Table 1-c illustrates the performance upper bound of the proposed method, which is on par with NB. This indicates temporal aggregation is not that effective. I thought the main claim of using temporal aggregation is that it can observe occluded regions as the performer changes poses so the features on the occluded regions can be taken into account. However, this is not reflected in the Table. I would suggest to conduct experiments that can justify the temporal aggregation for such a controlled experiment (this is different from the one in the ablation study).,



More comments:
-R^{3->2} to R^{3}->R^2
-L141: what does it mean by second axis?
-Since the network is supposed to learn human appearance agnostic to camera poses, it would be nice to demonstrate cross-dataset evaluation, e.g., learning from ZJU and testing on AIST. 
-Adding the qualitative results for Table 1-(c) would be useful to see the performance upper bound of the proposed rendering method.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a method that can synthesize free-viewpoint videos for dynamic humans in a sparse multi-camera system.  As is claimed and demonstrated in the paper, by introducing transformer modules to aggregate spatio-temporal information in the sparse multi-view data, the learned neural radiance fields are generalizable to unseen human identities and unseen poses. 
The proposed method is tested on public multi-camera datasets, showing promising results.
main_review: The novelty of this work is good. Although the generalizability to unseen poses has also been achieved in a few concurrent works, the unseen identity generalizability makes the key difference to those per-scene optimization methods.

Some questions and comments,
1. it seems like that it is pretty straightforward to incorporate the Transformer for learning the generalizable radiance fields. So I am afraid that the technical contribution in this paper may not be very significant.

2. the exposition of the method section needs more effort to be improved. 
     I would appreciate it much if the authors could make the method section much neater. The method section is full of tons of symbols, of which some may not be necessary, e.g., eq. 4 in Lin 178.
     Fig. 1 failed to do its job, I think the illustration of the method is supposed to be self-contained, it would be also much better to elaborate in the caption, instead of referring the readers back to each specific section.

3. regarding the details of the SMPL estimate. How many cameras are used for estimating the SMPL models of both ZJU-MoCap and AIST data? In Line 192, it is written that three methods are used to obtain these estimates, this confuses me. Also, the AIST dataset contains corresponding 3D keypoints and SMPL parameters (as described in the supplementary), are these data being used when running on AIST?

4. the paper did not show the failure cases. Is it true that the Transformers can always exploit the information in the memory bank to produce good imagery? What if the motions in the training data differ significantly from those in test data so that the memory bank fails. It is necessary to evaluate the transferability of the method in such a scenario, e.g., train on ZJU-MoCap, test on AIST, or vice-versa.

5. In Line 180, the samples are bounded within the bounding box of the SMPL mesh. But the SMPL mesh can be inaccurate, which means the bounding box derived from it can be inaccurate as well, e.g., the box may not be sufficiently large to enclose the GT shape.

Overall, this submission is good to me, and I hope the issues could be addressed in the rebuttal to make it a more solid submission.
limitations_and_societal_impact: Please see the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a generalizable NeRF network that can produce novel view synthesis for novel dynamic human scenes from sparse camera views, in a feed-forward manner. It leverages visual features from tracked parametric body model and similar to PixelNeRF, it conditions the NeRF representation on pixel-aligned image features for appearance generalization. A novel combination of a temporal Transformer and a multiview Transformer is proposed to integrate multi-frame and multi-view visual observations. It demonstrated stronger generalization capability than prior work. 

main_review: Creating a free-viewpoint video synthesis using a general model is a highly challenging problem.  The paper learns a generalizable NeRF for various dynamic human scenes, with a major contribution on a novel combination of a temporal Transformer and a multiview Transformer that enables better integration of visual cues across time and views. Even though the components of leveraging geometric surface prior (SMPL as done in NeuralBody) and pixel-aligned image features (as in PixelNeRF) for human NeRF learning are not new, the work has achieved stronger generalization capability and modeling power for dynamic human scenes with the proposed learning framework.  

The paper is well written and easy to follow. The experiments are extensive and well structured, demonstrating strong generalization capability to novel poses and identities compared with SOTA approaches. 

There are a few places that could be improved.
- It is not clear how important the accuracy of the parametric body tracking is, during at both training and testing stage. The paper demonstrates the experiments on AIST and ZJUMocap, both of which are indoor scenes with 3d mocap data. Therefore I assume it is less a problem in obtaining accurate SMPL parameters estimation. However, for in-the-wild video datasets, the body tracking accuracy might largely degrade and I will be curious to understand how the proposed learning framework will perform. 

- For the image evaluation metrics, it is not clear they are reported on novel camera views or the learned camera views. 

- It will be good to see the cross-dataset generalization capability, even just between AIST and ZJUMocap datasets. 
limitations_and_societal_impact: Well addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper solves a problem similar to Neural Body (reconstruction / free-view rendering of clothed humans from videos based on SMPL fits) but does not require training a neural network per scene; instead, it generalises from training sequences and, at test time, runs only a feed-forward model conditioned on keyframes. This is important for many applications since training a neural net per scene is not practical, and also the proposed extension is not trivial: the paper brings novel ideas (using attention for time-wise and camera-wise aggregation). Experimental evaluation is comprehensive and the quality is even superior to per-scene optimised Neural Bodies. Given this, I suggest to accept the paper, although there is some room for improvement.
main_review: Novelty.

(+) The paper significantly improves over Neural Body [31] (which was formally published after NeurIPS submission deadline). To be able to generalise to varying scenes, it conditions on the images filmed by different cameras and/or at different timestamps. This idea is exploited in recent pre-prints but for generic object categories (PixelNerf, PVA, WCE); this paper for the first time 1) exploits parametric body model to make it applicable to an articulated category of humans, 2) proposes to use transformers for aggregation across time and cameras instead of simpler average pooling. This is novel even w.r.t. unpublished work.

Clarity.

(+) The paper is well written, it mostly provides enough details.

(−) The main miss is that it does not describe the procedure of fitting SMPL to videos, which is not easy to do in a time-consistent manner (is it reconciled across cameras and time?). The paper only refers to [7, 31, 9], which is not enough. Also it would be good to clarify early on (before Section 4) that the method uses estimated SMPL parameters as it affects applicability.

(−) The paper can be more straightforward about limitations, see the next section for details.

(−) Skeletal features are sampled from projected SMPL vertices. How does the method deal with occlusions? The paper also says the query skeletal features are obtained by trilinear sampling but does not provide more details. I understand that the function is defined on the SMPL template surface; how exactly are the features extrapolated for the outside volume?

(+) Related work is comprehensive.

Significance.

(+) Training a neural network per scene is impractical in many applications due to time or storage constraints. Making a generalisable model is a life saver in these cases.

(+) The method can be used both for free-view rendering and reconstruction of clothed humans. As shown, the competing methods produce wrong 3D even if the rendering looks reasonable. The output also looks temporarily smooth in videos, which is also a desirable property.

Quality.

(+) Experimental results are comprehensive. The paper ablates the contributions, including turning off pixel-aligned and skeleton-defined features separately, and it shows that both transformer architectures bring improvement over average pooling.

(=) In the NHR baseline, is it fair to use just SMPL vertices? Will the method improve significantly with higher resolution?

(=) AIST numbers (Table 2b) are significantly lower than ZJU ones; is it still meaningful to compare results with such low PSNR?

Notes to the authors (no effect on the rating):
* In the time-wise transformer, would it make sense to use positional encoding of time?
* line 165: the different geometry patterns ← different geometry patterns;
* line 184: Is the loss indeed just L2 norm (RMSE), or squared (MSE)? 
* line 223: a health margin ← a healthy margin;
* lines 277, 279: “Table. 4”.
* in Tables 4 and 5, along with reconstructions, it would be interesting to show the SMPL fits, to see how far the methods deviate.

=====

UPD. I am satisfied with clarifications in the rebuttals, and raise the rating to 8 assuming they will be included in the paper.
limitations_and_societal_impact: In general, limitations section is good. I would suggest several improvements:
* please be clear upfront about the requirement to have SMPL fits as input; they are not given for granted, while Introduction does not mention this requirement. For other categories like animal species such parametric models are not available, so it limits the applicability of the method;
* the problem statement requires capture from static cameras only (extrinsics are not time-dependent): please make it clear and discuss whether it would be difficult to extend the method to support capture from e.g. one or several phones or AR devices;
* similar for prediction-time: the experiment al protocol seems to fix cameras: is it a big limitation? The supplementary videos seem to show free-view trajectories; could you comment how well it generalises? E.g. what if the cameras are sampled outside of a hemisphere manifold?
review_title: null

Meta-review: This submission received 4 positive final ratings: 8, 7, 7, 8.
The reviewers appreciated overall novelty of the approach, its generalization properties and strong empirical performance. The remaining concerns were mostly around clarity, justification of individual components and lack of analysis of failure cases. These were mostly addressed in the rebuttal, as acknowledged by the reviewers. The final recommendation is therefore to accept as a spotlight.
Decision: Accept (Spotlight)