Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a human full-body avatar model that combines human NeRF and image-based rendering methods. The proposed model achieves significant improvement for unseen pose and unseen person synthesis. Extensive quantitative and qualitative evaluations have been conducted to measure the model's effectiveness.
strength_and_weaknesses: Paper strengths:
- The proposed method is novel in combining human NeRF and image-based rendering.
- The proposed method shows clearly superior performance both qualitatively and quantitatively.
- Experimental evaluations are extensive and comprehensive.

Paper weaknesses:
- It seems unclear how much the current model's performance depends on accurate SMPL fitting. Would the proposed model be compatible with in-the-wild selfie videos followed by a single-view SMPL estimation?
- I am a bit confused by the feature pooling in Equation (1) which currently does not consider visibility, should invisible views be ignored here?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly-written and shows high quality. I believe the work is original as far as I'm aware.
summary_of_the_review: Overall speaking I think this is a solid paper. The method is novel, the results are significantly better than previous state-of-the-art, and the evaluations are thorough. My concerns are minor and can be addressed by additional discussions. Therefore, my initial rating for the paper is accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines two rendering modules for synthesizing novel views and novel poses of human bodies from sparse multi-view images. 
The first one is a body representation based on neural radiance fields and SMPL, extracting pixel-to-image features from each view, and attaching features to SMPL vertices. In this way, a coarse feature map for each view can be constructed, which is further converted into dense feature voxels through convolutions. Multi-view features are obtained through view-wise self-attention processing, and colors are finally obtained through MLPs.
The second one is a neural image-based rendering module, which can improve the details of the result by directly obtaining the pixel color from the source view image.
Results of the two modules are finally combined together with the prediction weights.
strength_and_weaknesses: **Strength**    
* This method can achieve good results on unseen views and unseen poses. Thanks to the image-based rendering method, it has good generalization performance across datasets.
* The ablation experiments show that the fusion of implicit body representations and image-based rendering plays a complementary role.
* Sufficient experiments have proved the effectiveness of the method.


**Weaknesses**   
* When the number of viewing angles is reduced to 1, the results are greatly degraded. Which part of this degradation comes from deserves further analysis, e.g. with regard to multi-view consistency.
* The appearance blending module shows effectiveness in visualization and quantitative studies. However, why it works or the mechanism it works is not demonstrated clearly.
* The method directly combines two rendering methods, which both exist in previous works. Any new insight about the pipeline/module design needs to be clarified.
clarity,_quality,_novelty_and_reproducibility: * **Clarity**: Most contents are clear and easy to follow.
* **Quality**: The proposed method has been evaluated and studied on substantial benchmarks and settings. Especially, experiments are also performed on cross-dataset settings.
* **Novelty**: The method directly combines two rendering methods, which both exist in previous works. This makes the novelty a little bit in discount.
* **Reproducibility**: Most implementation details are provided. It is likely to reproduce the method. Code is not available.
summary_of_the_review: Through extensive experiments, this paper shows better performance than previous methods. And made a preliminary exploration on cross-dataset settings. Some questions/issues need to be fixed. The influence of the view number and the mechanism of appearance blending need to be clarified. Any new insight about the pipeline/module design needs to be further discussed to rebut the novelty concern.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to obtain Neural Image-based Avatars (NIA) for synthesizing novel views and poses of an unseen human from sparse input images. NIA is derived from a neural blending network to integrate 1) NeRF-based body representation conditioned on parametric SMPL human body model and 2) neural image-based rendering.
strength_and_weaknesses: [Strengths]
+ The motivation is well-written and easy to follow. 
+ The citations are sufficient.
+ The authors provide several visualizations for realizing their rendering results.

[Weaknesses] 
- The proposed idea novelty is limited. The framework of NIA is similar to [Zhao et al. (2022)] since the two major steps of both methods are generalizable implicit body representation and sparse views blending. The core design of NIA is the neural appearance blending module (sec. 3.2), yet no ablation study to discuss and analyze why such a design helps.
- Three assumptions exist in this work: 1) the calibration parameters of the multi-view input images are known; 2) the foreground human region masks are known; 3) the fitted SMPL models are available as prior. The first two assumptions are also seen in other methods; however, yet the third assumption is new in this work. The third assumption seems too strong since the SMPL model estimation is a vital factor affecting human NeRF rendering. Therefore, while assuming the fitted SMPL models are available as a prior, it is doubtful that the comparison experiments are fair enough compared with other methods. Notice that Table 4 (i) shows that the NIA w/o image-based rendering (with the third assumption) has already better than most other methods.
- In sec 3.3, the description of “the 3D coordinates of the posed SMPL vertices in both reference and target spaces are known by the nature of motion-tracked SMPL model” is another strong assumption that makes the NIA available to avoid the finetuning step. It is unclear why the such assumption is available while dealing with the unseen subject with unseen poses.
- The comparison experiments are not convincing. It seems that some existing methods are reimplemented since the reported results of these methods (degraded) in this paper are not the same as their published ones. It is better to annotate which results are reimplemented and which are not.

- Some implementation details are not clear:
1) The limitations of the sparse input images. Are there any constraints for view directions?
2) The feature representation of P_n is not defined;
3) The design of two MLPs in equation (1) is different from NeRF [Mildenhall et al. (2020)]. Why is such a design reasonable, and what is its advantages?
4) The architecture of F_B in equation (2). Does (2) means that it has one single model for a specific value of N?
5) SE(3), x^can, k-th part sampling method, function R(), function t() in sec. 3.3;

Reference:
[Zhao et al. (2022)] Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In CVPR, 2022.

clarity,_quality,_novelty_and_reproducibility: Clarity is not sufficient.

The framework of NIA is similar to [Zhao et al. (2022)], weakening its novelty.

Reproducibility is limited due to its complex system.
summary_of_the_review: The primary concern of this paper is its limited novelty. The key idea is to learn to adaptively combine the parametric body model-based NeRF and image-based rendering techniques, which largely overlap the method proposed by Zhao et al. (2022). Besides, the assumptions of "the fitted SMPL models are available as prior" and "the 3D coordinates of the posed SMPL vertices in both reference and target spaces are known" are too strong to compare with the existing methods for the fairness concern.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose Neural Image-based Avatars (NIA), a method for synthesizing novel views and poses. Its main advantages are the implicit body NeRF representation and the image-based rendering. They claim to outperform the current state-of-the-art on the ZJU-MoCap and MonoCap datasets.
strength_and_weaknesses: Main Strengths:
- The method gathers two complementary state-of-the-art approaches (i.e. neural radiance fields and neural image-based rendering).
- The NIA is generalizable for different subjects.

Main Weaknesses: 
- The approach relies on sparse view images and the authors do not refer to the LOLNeRF (Rebain et al., CVPR22), a related and recent method based on single views.
- Most of the previous approaches rely only on PSNR and SSIM metrics. Despite that, it would be interesting to see additional metrics on the evaluation, or at least to read the authors' comments on why other metrics were not adopted.


clarity,_quality,_novelty_and_reproducibility: The paper is clear and technically sound. The overall pipeline is new, but it strongly relies on Kwon et al. (2021) and Wang et al. (2021). The authors describe the implementation details in the supplementary material and claim that the code will be published upon publication.

Minor suggestions:

- section(s) -> Section(s); sec. -> Sec..
- Section 4.1.1 could be on top of page 6.
- Eval. -> Evaluation/Evaluated (Tab.1).
- References could be included in Table 1, or at least in its caption.
summary_of_the_review: The paper introduces a new approach to generalized novel views and poses synthesis. Although it mainly put together existing methods (Kwon et al. (2021) and Wang et al. (2021)), it achieves state-of-the-art results. Therefore, I believe the contribution is worthy of publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by three experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2023. All the reviewers acknowledged the quality achieved by the proposed method for generalized novel view and pose synthesis. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper, e.g., adding more discussions and ablations to similar works such as Zhao et al. (2022). The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!

justification_for_why_not_higher_score: The overall pipeline is relatively incremental, which strongly relies on Kwon et al. (2021) and Wang et al. (2021).
justification_for_why_not_lower_score: The proposed method has achieved compelling results on generalized novel view and pose synthesis.
Decision: Accept: poster