Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling
Authors: YoungJoong Kwon
Dahun Kim
Duygu Ceylan
Henry Fuchs
Venue: ICLR 2023 poster (2023)
Keywords: Generalizable human radiance fields
Human performance capture
Human NeRF
Neural radiance fields

Abstract: We present a method that enables synthesizing novel views and novel poses of arbitrary human performers from sparse multi-view images. A key ingredient of our method is a hybrid appearance blending module that combines the advantages of the implicit body NeRF representation and image-based rendering. Existing generalizable human NeRF methods that are conditioned on the body model have shown robustness against the geometric variation of arbitrary human performers. Yet they often exhibit blurry results when generalized onto unseen identities. Meanwhile, image-based rendering shows high-quality results when sufficient observations are available, whereas it suffers artifacts in sparse-view settings. We propose Neural Image-based Avatars (NIA) that exploits the best of those two methods: to maintain robustness under new articulations and self-occlusions while directly leveraging the available (sparse) source view colors to preserve appearance details of new subject identities. Our hybrid design outperforms recent methods on both in-domain identity generalization as well as challenging cross-dataset generalization settings. Also, in terms of the pose generalization, our method outperforms even the per-subject optimized animatable NeRF methods.

Published as a conference paper at ICLR 2023
NEURAL IMAGE-BASED AVATARS: GENERALIZABLE RADIANCE FIELDS FOR HUMAN AVATAR MODELING
Youngjoong Kwon', Dahun Kim², Duygu Ceylan', Henry Fuchs!
University of North Carolina at Chapel Hill. ?Google Research, Brain Team. 3 Adobe Research. (youngjeong, fuchs 8es. edu
(meahny ) 8google. (ceylan) @adobe com
ABSTRACT
We present method that enables synthesizing novel views and novel poses of arbitrary human performers from sparse multi-view images. A key ingredient of our method is a hybrid appearance blending module that combines the advantages the implicit body NeRF representation and image-based rendering. Existing gener- alizable human NeRF methods that are conditioned on the body model have shown robustness against the geometric variation of arbitrary human performers. Yet they often exhibit blurry results when generalized onto unseen identities. Meanwhile, image-based rendering shows high-quality results when sufficient observations are available, whereas it suffers artifacts in sparse-view settings. We propose Neural Image-based based Avatars (NIA) that exploits the best of those two methods: to maintain robustness under new articulations and self-occlusions while directly leveraging the available (sparse) source view colors to preserve appearance details of new subject identities. Our hybrid design outperforms recent methods on both in-domain iden- tity generalization as well as challenging cross-dataset generalization settings, Also, in terms of the pose generalization, our method outperforms even the per-subject optimized animatable NeRF methods.
INTRODUCTION
Acquisition of 3D renderable full-body avatars is critical for applications to virtual reality, telepresence and human modeling. While early solutions have required heavy hardware setups such as dense camera rigs or depth sensors, recent neural rendering techniques have achieved significant progress to more scalable and low-cost solution. Notably, neural radiance fields (NeRF) based methods facilitated by the parametric body prior Loper et al. (2015) require only sparse camera views to enable visually pleasing free-view synthesis Peng al. (2021b); Kwon et al. (202T); Zhao et al. (2022); (2021a): Su et al. (2021) of the human avatar.
Cheng et al. (2022) or pose animation Peng et al.
Still, creating full-body avatar from sparse images (e.g., three snaps) of person is challenging problem due to the complexity and diversity of possible human appearances and poses. Most existing methods [Png al. (2021b a) therefore focusing on person-specific setting which requires dedicated model optimization for each new subject it encounters. More recent methods explore generalizable human NeRF representations (Pena al. (2021b); Raj et al. (2021b); Kwon al. (202T); Zhao et al. (2022) by using pixel-aligned features in a data-driven manner. Among them, Kwon et al. (2021) and Chen al. (2022) specifically exploit the body surface feature conditioned NeRF (i.e., pixel-aligned features anchored at the SMPL vertices) which helps robustness to various articulations while obviating the need for the 3D supervision Zhao et al. (2022); Cheng et al. (2022). Nevertheless. these body surface feature conditioned NeRFs still suffer blur artifacts when generalizing onto unseen subject identities with complex poses. Also, an extra effort on video-level feature aggregation required in Kwon et al. [202I) to compensate for the sparsity of input views.
In this paper, we propose Neural Image-based Avatars (NIA) that generalizes novel view synthesis and pose animation for arbitrary human performers from sparse set of still images. It is hybrid framework that combines body surface feature conditioned NeRF (e.g.. Kwon et al. (2021) I)) and image-basee rendering techniques (e.g., Wang et al. (202I)). While the former helps in robust representation of different body shapes and poses, the image- based rendering helps preserving the color and texture details from the source images. This can complement the NeRF predicted colors

---

Published as a conference paper at ICLR 2023
which are often blur and inaccurate in generalization settings (cross-identies as well as cross- generalization) as shown in figure and figure[5] To leverage the best of both worlds, we propose a neural appearance blending scheme that learns to adaptively blend the NeRF predicted colors with the direct source image colors. Last but not least, by deforming the learned NIA representation based on the skeleton-driven transformations Lewis et al. (2000): Kavan et al. 2007), we enable plausible pose animation of the learned avatar.
To demonstrate the efficacy of our NIA method, we experiment on ZJU-MoCap Peng al. (202 and MonoCap Habermann et al. 2020; 202T) datasets. First, experiments show that our method outperforms the state-of-the -he--hattttttt art Neural Human Performer Senooomerrwwonn [won al. (202T) and GP-NeRF Chen et al. (2022) in novel view synthesis task. Furthermore, we study the more challenging cross dataset generalization by evaluating the zero-shot performance on the MonoCap Habermann et al. (2020) 202T) datasets, where we clearly outperform the previous methods. Finally, we evaluate on the pose animation task, where our NIA tested on unseen subjects achieves better pose generalization than the per-subject trained A-NeRF Su et al. (2021) and Animatable-NeRR Peng al. (202 la) that are tested on the seen training subjects. The ablation studies demonstrate that the proposed modules of our NIA collectively contribute to the high-quality rendering for arbitrary human subjects.
2 RELATED WORK
Combined with Neural Radiance Fields (NeRF) Mildenhall et al. (2020), human reconstruction research has shown unprecedented development Pumarola et al. [2020) Park et al. (202 b). Human priors are utilized to enable robust reconstruction of face and body Gao et al. (020); Gafni et al. (2021): Peng al. (2021b). However, these methods are per-subject optimized, and cannot model the motions that are not seen during training. Therefore, subsequent works have been focusing on generalization in two directions: pose and subject identity.
Pose generalization. Su et al. (202T) utilize a joint-relative encoding for dynamic articulations.
Noguchi et al.] (2021) explicitly associate 3D points to body parts. Chen et al. (2021) and Peng et (2021a) deform the target pose space queries into the canonical space to obtain the color and density values. Liu et al. (202T) leverages normal map as the dense pose cue. Xu al. (202T) learns deformable signed distance field. [Weng et al. (2022) and [Peng et al. (2022) decompose the human deformation into articulation-driven and non non-rigid deformations, (Su al. (2022) and Zheng
et al. (202)) utilize the joint-specific local radiance fields. Despite the significant progress in pose generalization, they still focus on a subject-specific setting which requires training a single model for each subject. In this paper, we tackle generalization across both poses and subject identities.
The use of image-conditioned or pixel-aligned features Yu et al. (2020); Wang et al. (2021) has allowed generalized neural human representations from sparse camera views. Raj et al. (2021b) use camera-encoded pixel-aligned features for face view synthesis. Kwon et al. (2021) aggregate temporal features by anchoring them onto the SMPL body vertice to complement the sparse input views. [Chnn et al. (2022) also leverages body surface features to enable full-body synthesis. /Zhao al. (2022) and Cheng et al. (2022) also propose the neural blending of NeRF prediction with source view colors. Specifically, they use the pixel-aligned features- conditioned NeRF their implicit body representation. However, pixel-aligned features alone are prone to errors under complex poses as reported in Kwon et al. (2021). Therefore, their method require 3D supervision (e.g., depth, visibility) or per-subject finetuning. In contrast these methods, we aim at generalizing human modeling without relying on the 3D supervision or per-subject finetuning, but by using only RGB supervision.
Subject identity generalization.
Other (non HeRF-based methods. Deferred neural rendering based methods Thies et al. (2019);
Raj et al. (0011)) Grigorev et al. (202T) enable fast synthesis of person-specific avatar by combining the traditional graphics pipeline with the neural texture maps. Grigorev et al. further finetunes to deal with unseen subjects. contrast, we focus on the generalization without any finetuning. Saito et al. (2019 2020); He et al. (2021) leverage pixel-aligned features to enable 3d reconstruction from a single image. However, they require 3D groundtruth supervision. Habermann et al. (2019); Liu
eet al. (2020); Habermann et al. 12021); [Baguudiiovv al. (2021) generate high-quality non-rigid deformation, but they require the template mesh optimization. Aliev et al. (2019;; | Wu et al. (2020)
anchor features on the point clouds and render them with differentiable rasterizer. However, they
2

---

Published as a conference paper at ICLR 2023
reforence
target
Novel synthesis
Reference sprce
Targel space
reference targel
Posc unimation
d
Our syuthesis
Sec.3.1
Repose +
NeRF
scc.3.3
Hody sampled
d'
Co: color
Implicit body representarion
Deformed body representation
up confidence
O dansitt
Pisc--alggee liaturce p sampled at x 10, d')
Appearance blender Fgg sec.
Volume rendering
Sec.3.4
Source view colons
Figure 1: Overview of Neural Image- Avatars (NIA) method. Given sparse view images of an unseen person in the reference space, NIA instantly creates human avatar for novel view synthesis and pose animation. NIA is hybrid approach combining the SMPL-based implicit body representation and the image-basee rendering method. Our appearance blender learns adaptively blend the predictions from two components. Note that if the reference space and the target pose spaces are identical (i.e. no pose difference, I = x', d d'), the task is novel view synthesis. Otherwise, it is a pose animation task where we deform the NIA representation repose the learned avatar.
require sufficient amount of point clouds that cannot be obtained from our sparse input view setting. In this work, we mainly focus on and compare with the NeRF-base methods.
METHOD
The overview of our NIA is given in figure Our goal is to obtain Neural Image-based Avatars (NIA) which after training, can directly synthesize novel views and novel poses of an unseen person in a feed-forward manner from very sparse input views. To deal with complex and diverse human appearances and poses, we propose a hybrid framework that leverages the advantages of two representations. The first is neural radiance fields based body representation conditioned on a parametric human model (SMPL Loper et al. (2015)). which can provide the body geometry and dependent deformations (section Inspired by Wang al.] (202I), the second is obtained by neural image-based rendering which can improve the fine-details of the result by directly fetching the pixel colors from the source view images. A core of design is neural appearance blending scheme that learns to blend predictions from the two representations and outputs the final synthesis results (section 3.2 Given a of sparse view images of arbitrary person, our learned NIA only performs novel view synthesis, but also reposing by deforming the representation of the avatar capture time into the desired target pose (section 3.3).
We note that we assume the calibration parameters of the multi-view input images and the foreground human region masks are known, and the fitted SMPL models are available.
3.1 BASELINE: GENERALIZABLE IMPLICIT BODY REPRESENTATION
Inspired by NHP K won et al. (22T) and GP-NeRF Chen et al. (202 our implicit body NeRF representation follows the design of body surface feature conditioned NeRF (i.e., pixel-aligned features anchored at SMPL vertices.
Given N available source views, we extract pixel-aligned image features Saito et al. (2019) from each view and attach them to the SMPL vertices, to construct per-view coarse feature set P,, Each vertex location is projected to the image feature plane and the corresponding pixel-alignes feature computed by bilinearly interpolating the grid pixel values. Since P,, is sparse in 3D represent the whole human body, we use 3D sparse convolutions [iu et al. (2015) to diffuse the features into denser feature volume. The sparse convolutions learn extrapolate the sparse feature points P,, build dense body feature volume P,, that better adapts to the target body shape. For each query point x, sample the corresponding body feature from P.. The sampled features from N source
3

---

Published as a conference paper at ICLR 2023
views then processed by view-wise cross-attention with the pixel-aligned features to obtain the mieti-view aware body feature Zn. Besides the final density a and color value Co. we also propose to predict the confidence & E [0, 1] of its color prediction
= Fo Σ N)),
Co. vi h Feo E (Zni (d))/MN,
(I)
where Fo and Fen MLPs consisting of four linear layers respectively. and 7 : RR3-6x1 is positional encoding of viewing direction d E R as in Mildenhaal et al. (2020) with x I different basis functions. h is the intermediate color feature extracted from the second to last layer of Peo that is later used in the appearance blending (section 3.2).
3.2 LEARNABLE APPEARANCE BLENDING
While our implicit body NeRF representation helps robustness to geometric variation of arbitrary human shapes, it is difficult for such general NeRF representation convey all the high-frequency details (color and textures) present in the source view images. To remedy this, we leverage ideas from image-based rendering techniques Chen & Williams 1993); Buchler et al. (2001); Gortler et al. (1996): [evoo & Hanrahan (996); Debever et (199)) Hedman et al. (2018); Wang et al. (2021). In particular, query point can be projected to N source views to directly retrieve the color values. We propose an appearance blending module FB that learns to predict blending weights among the N source view colors C. and the NeRF generated color Co.
The inputs to this network are the pixel-aligned features ps, for x, the relative viewing direction d, which is defined by the difference and dot product between the target viewing direction d and the source viewing direction d,, the visibility o, of x with respect each source view, the intermediate NeRF color feature h and the color confidence 16 from F... We define the view index for the implicit body NeRF representation n=0. Formally, the blending weights woah are computed as:
rgb N { ( Jn= = FB((Pn dn; on J h, 2)
(2)
Note that our visibility o, is computed without the need for any ground truth 3D geometry. To pursue scalable and practical setting without the reliance the 3D supervision, underlying parametric body model to approximate the visibility for each query point. Specifically, we borrow the vertex visibility of the SMPL model. As in Huang et al. (2020): [Bhatnagar al. (2020) Peng et al.] (2021a), we first search the SMPL mesh surface that is closest to the query point. Then, we define the visibility
propose to exploit the
float value o, E [0, 1] computed by the barycentric interpolation of the 0-or-1 1 visibility values of three vertices on the corresponding mesh facet. This is different from Cheng et al. (2022) where visibility trained with groundtruth is required. Also, our design differs from Zhao et al. depth is utilized to sample source view colors. The final color for x is computed via softmax
20022 where the
N
0 (exp( (o (n))))),)))))))) exp(w;""))
3.3 RE-POSING NIA FOR POSE ANIMATION
We describe how our proposed NIA can be extended to animate an unseen subject given new target poses inference without any further training. Different from the previous works Chen al. (2021) Peng et al. 1202 2022), where a canonical feature space for appearance retrieval is optimized in a per-subject manner, our generalization approach aims reposing the NIA avatars that created on the fly given a set of sparse reference images of unseen human subjects. Since the available source images the reference space cannot be directly used in the target pose space (i.e., observation space), the construction of our NIA representations should be considered with the deformation between the target and reference spaces.
To build the implicit body feature volume in the target pose space, we retrieve the relevant pixel- aligned features from the location of the reference SMPL vertices. By using the known SMPL correspondences, these pixel-aligned feature points are relocated and attached to the corresponding target pose SMPL vertices. The following sparse convolutions will result in the feature volume the target pose space, where each body feature zter is sampled from. The density and color in the target space are computed similarly to Eq. (I). except we use star that transformed into the reference space, instead of zn,
4

---

Published as a conference paper at ICLR 2023
The appearance blending is defined similarly to Eq. using the target-torrferrnne transformed query location and viewing direction d', instead of x and d. The visibility is computed with respect to x' and the reference SMPL body.
Deformation modeling. We model the deformation from the target pose space (observation space) reference space the skeleton-driven deformation, which is the composition of the inverse and forward linear-blend skinning Lewis et al.] (2000). Specifically, we can transform the query x in the target space into the reference space location x' through T(x) (T T (x). where T and T wd are defined as:
K
K
-1
Trnv Σ
tar (x)Gk W
==
x can
Toudd
Σ
can can
x
Gk
x cen xenn=xx =x, (3)
=
where [Gk} E (E(3) is the K' transformation matrices produced by skeleton with K body parts, and wfar and wgen are the blending weights for k-th part sampled in the target and canonical space, respectively. Similar to the visibility computation (section 3.2). the blending weight is interpolated from the blending weights of the nearest SMPL vertices.
The deformation T for the viewing direction, which transforms the target-space viewing direction d to the reference space viewing direction is defined a similar manner. We denote the weighted sum of transformation matrices of T as [[(x); [x)] Then, ((d) is defined as (dd R(x)d = d',
3.4 VOLUME RENDERING AND LOSS FUNCTION
Volume rendering. To decide the color of each ray, we accumulate the density and color predictions along the ray r(t) = + td for t E [fueary tfre as defined in NeRF Mildenhall et al. (2020) as
follows:
(- (-f" a(r(s))ds)
C(r)
T(t)a(r(())c(((() d)dt, where T(t) = exp
(4)
facur
In practice, we uniformly sample set of 64 points ~ [bnear; L Lfar). The bounds for ray sampling tnears are drived by computing the 3D bounding box of the SMPL body.
Loss function. Together with the final blended appearance c, we also supervise the NeRF regressed co with a simple photometric loss improve the prediction quality: L |[co e || E||2 where è denotes the groundtruth pixel color.
4 EXPERIMENTAL RESULTS
To demonstrate the effectiveness of our proposed NIA method, we conduct experiments for novel view synthesis and pose animation tasks. We use 7JU-MoCap) Peng et al. (202Tb) for both tasks and ablation studies. Then we study our cross-dataset generalization ability by training on ZJU-Mocap and testing on MonoCap datasets without any finetuning.
4.1 IDENTITY GENERALIZATION
Competing baselines. For novel view synthesis, the state-of-the-an competitors are NHP Kwon jet al. (2021) and GP-NeRF Chen et al. (2022) which robustly perform generalization with the body surface feature. Other generalizable methods are pixel-NeRF Yu et al. (2020), PVA Raj et al. (2021b) lb). Image-Based Rendering Network (IBRNet) Wang et al.) (2021). and Keypoint NeRF Mihajlovic et al.) (2022). We also compare with per-subject optimization method Neural Body Peng al. (2021b). Neural Textures (NT) Thies et al. (2019), and Neural Human Rendering (NHR) Wu et al. (2020).
For pose animation task, compare with the per-subject animatable NeRF methods which Neural Body |Png et al.] (2021b), A-NeRF Su et al. (2021) and Animatable- Peng et al. (202 Ia).
Note that the comparison with pur-subject methods in both tasks is for reference purpose and places our (and other) generalizable method disadvantage. These methods require one model (network) trained for single subject, iee. as many models the number of testing subjects, and they
5

---

Published as a conference paper at ICLR 2023
Neural Body't
PixeI-NeRF
IBRNet
NHP
GP-NeRE
Ours
GT
Figure 2: Novel view synthesis results on ZJU-MoCap dataset. t: Neural Body Peng et al. 2222 is per-subject optimized method, and tested the subjects with unseen poses. PixeI-NeRI Yu et al. (202)) IBRNet Wang et al. (202T), NHP Kwon al. (202I) and GP-NeRFCChed al. (2022) are generalizable methods which are tested on unseen subjects with unseen poses.
NT
NHR
NB
NHP
Ours
GT
Figure 3: Results on the unseen poses of seen subjects on MJU-MoCap dataset. NT: Neural Texture Thies et al. (2019), NHR: Multi-view Neural Human Rendering Wu et al. (2020), NB: Neural Body Peng et al. (202 NHP. Neural Human Performer Kwon et al. (2021). and NIA (ours). NT, NHR, NB per-subject optimized methods.
method
PSNR
SSIM
method
PSNR
SSIM
Training: per-subject
PVA
23.15
0,8663
NT
22.28
0.8720
Pixel-NeRF
23.17
0.8693
NHR
22.31
0,8710
IBRNct
24.54
0.8935
NB
23.79
0.8870
NHP
24.75
0,9058
Training: generalizable
GP-NeRF
24.49
0,9012
NHP
26.94
0.9290
Keypoint NeRF
25.03
0.8969
NIA (ours)
27.57
0.9398
NIA (ours)
25.79
0.9200
Eval. subjects, unseen poses.
b. Training: generalizable (one model for multiple subjects); Eval. unseen subjects, unseen poses.
Table I: Novel view synthesis on ZJU-MoCap. All are evaluated on the same testing sequences.
evaluated the training subjects with unseen poses. On the other hand, the generalizable methods train only one model which is evaluated on all unseen subjects with unseen poses.
4.1.1 NOVEL VIEW SYNTHESIS
Setup. The ZJU-MoCap dataset contains 10 human subjects with 23 synchronized cameras. We follow the same training and testing protocols in Kwon al. (202).
The quantitative comparison of our NIA and other methods shown in Table. I Our model achieves state-of of- of-the-art performance ZJU-Mocap dataset. Especially, the previous state-of- the-art NHP /HP/Kwon et al. (202T) relies on temporally aggregated features to compensate the sparse input views. Our NIA outperforms NHP by +1.0 dB PSNR and + +1.5% SSIM, while utilizing only
Results.

---

Published as a conference paper at ICLR 2023
Neural Body't A-NeRF
Ani-NeRFf
Ours
GT
Ours
GT
Reference views
Figure Pose animation results on ZJU-MoCap dataset. t: Neural Body Peng et al. (2021b). A-NeRF et al. (202T), and Ani-NeRF Peng et al. (2021a) are per-subject optimized methods, and are evaluated in novel pose synthesis for the seen subjects. Our NIA is the only generalizable method, and is evaluated in novel pose synthesis given only three-view reference images of unseen subjects. Column and are our results for another poses, and the ground truths. The last columns are the reference image input to our NIA.
still images. NIA also outperforms the most recent sate-offtthear methods GP-NeRF Chen al. (0022) and Keypoint NeRF Mihajlovic et al. (0022) by +1.3 / +0.7 dB PSNR and +2.0 +2.5% SSIM.
In Table. D and figure
show the performance the unseen poses of the seen subjects. Despite of the disadvantageous generalizable setting (one network for all subjects), our method significantly outperforms other baselines including the per-subject optimized methods.
As shown in figure 2 our NIA exhibits more high-free freeuereyunnyyy details compared to NHP and GP-NeRF. Other generalizable methods that utilize image-conditioned features (pixel-NeRF, PVA, and IBRNet) often suffer noisy surface prediction, while NIA shows more robust results on different human shapes.
4.1.2 POSE ANIMATION
Setup. We evaluate our NIA for pose animation task on ZJU-MoCap dataset. Given a set of sparse view reference images of unseen person, our on-the-fly generated avatar for that person is reposed to new target poses. We also use three input views for this task.
While we do perform generalization onto unseen subject identities, we compare with per-subject methods. Note that this comparison puts our method on disadvantage and thus is only to provide a reference level, since the competing methods have seen the testing subjects (no identity generalization for baselines).
Specifically, we compare with Neural Body Peng al. (2021b), A-NeRF Su et al. (021 and Animatable-Nert Peng et al. (2021a) which made their code public and reported their results on ZJU-MoCap dataset. Meanwhile. we acknowledge there are more recent and advanced per-subject
animation methods (Su et al. (2022); Zheng et al. (2022); Peng et al. (2022); Weng et al. (2022). For all methods, we evaluate on the same testing sequences as in the novel view synthesis task.
Results. Table. shows the quantitative comparison. Remarkably, our NIA model that is generalized onto unseen subjects outperforms the per-subject optimized competitors tested on seen subjects by healthy margins: +1.2 dB / +2.0% over Neural Body, +0.1 dB / +1.2% over A-NeRF and +0.1 dB / +1.1% over Animatable-NeRF PSNR and SSIM, respectively. In figure we observe that both Neural Body and Animatable-NeRR produce noisy and blurry texture output on novel poses, even though their models are trained to memorize the appearance of the same human subject. Our NIA method preserves relatively more high-frequency details like clothes wrinkles and textures. Note that it is a very challenging setting that requires our model instantly create animatable avatars on the fly from only three reference view images of unseen human subjects.
7

---

Published as a conference paper at ICLR 2023
input Viows
NHP
Ours
GT
inpot views
NHP
Ours
GT
Our pose animation
images
Figure 5: Cross-dataset generalization results without finetuning. Our NIA outperforms NHP Kwon et al. (2021) on novel view synthesis, when trained on 7JU-MoCap and tested MonoCap datasets (left). Our NIA also f perform pose animation of unseen subjects given only sparse set of still images (right).
method
training
evaluation
PSNR
SSIM
Neural Body
22.88
0.8800
A-NeRF
per-subject
seen subjects,
23.91
0.8893
method
PSNR
SSIM
mnimtabble-NRR
unseen poses
23.88
0.8901
NHP
17.51
0.7457
NIA (ours)
19.12
0.7804
unseen subjects,
NIA (ours)
generalizable
24.04
0.9000
unseen poses
Table 3: Cross-dataset gen-
eralization (train on ZJU- MoCap, and test on MonoCap.
Table 2: Pose animation on ZJU-MoCap dataset. All methods are evaluated on the same testing sequences and target poses. A-NeRF all. (02T), Animatable-NeRF Peng et al. (2021a), and Ours.
4.2 CROSS-DATASET GENERALIZATION
For the out-of domminn cross-dataset generalization, we train a model on the ZJU-MoCap dataset, and test on the MonoCap dataset without any finetuning. MonoCap consists of DeepCap Habermann et al.. (2020) and DynaCap Habermann et al. (202T) dataset sequences that are captured by dense camera views. Given the different lighting and capture settings between the 7JU-MoCap and MonoCap datasets (e.g., dark v. bright studio), our NIA improves the cross-dataset generalization over NHP by clear margins of + 1.66 dB PSNR and +4.5 SSIM in Table. 3
As shown in the visual results (figure 5 the implicit body NeRF prediction of NHP especially suffers from the different color distribution between these datasets.
On the other hand, our NIA is hybrid model combining the implicit body NeRF with the direct pixel fetching from the input images f.e.image--asee rendering). which helps to generalize to out- of-domain MonoCap datasets. Given only three snaps of person, our NIA is able to perform plausible pose animation as shown in figure (right: "pose animation').
4.3 ABLATION STUDIES AND ANALYSIS
We provide ablation studies on ZJU-MoCap dataset by evaluating novel view synthesis performance of our NIA variants on unseen subject identities with unseen poses.
NIA architecture designs. We study the contribution of different modules of our NIA network in Table. 4 and figure 6 First, we ablate the two main components of our method: implicit body representation (a) and image-based rendering (b). By removing each component, we observe performance drop of - 1.63 dB / -3.80% (a) and -1.55 dB -1.49% (b) in PSNR and SSIM respectively. figure 6 visualizes how each component contributes the rendering quality. Without the implicit body representation, the model tends to suffer inaccurate shape prediction especially the regions fully occluded from all source views (e.g., ghosting artifact near the right hand). On the other hand, removing the image-basee based rendering leads to blurry results and the loss of high-frequency texture details. Our full NIA model-(e) leverages these complementary merits and shows the best results both quantitatively and gualitatively.
We further investigate the impact of our neural appearance blending module. learns to blend predictions from the implicit body representation (i.e. NeeF-peedidd color) and the color values
8

---

Published as a conference paper at ICLR 2023
Implicit body rep.
view #1
View +2
view #3
chanmel RD channel #1
channel H2
channel #3
wio implicit b. wio imagee simple
d. weighted averuging
full model blending)
GT
body rep.
based rendering
averaging
Figure Input views (#1,2,3) and the corresponding channels of learned neural blending weight maps. Channel #0 corresponds to the prediction from the implicit body representation (NeRF). Remaining columns show the results of ablation items in Table. Ti and the ground truth.
NIA variant
PSNR
SSIM
w/o implicit body representation
24.62
0,8888
views
PSNR
SSIM
w/o image-based rendering
24.70
0.9119
24.71
0,9006
w/o neural appearance blending (simple averaging)
25.28
0,9168
2
25.53
0.9150
w/o neural appearance blending (weighted averaging)
25.70
0.9236
26.25
0.9268
full model
26.25
0.9268
ii. number of input views.
i. NIA network components.
Table 4: Ablation studies on ZJU-MoCap dataset; novel view synthesis task.
from the source images (i.e., image-based rendering), adaptively to different regions render. Instead of our learnable design, we experiment by replacing it with two plausible deterministic functions. The first is naive simple averaging of different color predictions (c), and the second is a weighted averaging based on the cosine similarity between each source view's viewing direction and the target view (d). The comparison in rows-(c, d, e) verifies the benefit of our learnable appearance blender, demonstrated by the gains of +0.97 dB / +1.0% and +0.45 dB +0.3% in PSNR and SSIM over the two deterministic variants respectively. figure also shows the contribution of adaptive blending that helps resolving the blurs and ghosting artifacts.
Number of camera views. Table. ii shows that the performance of our NIA gradually increases with more source views. Increasing the number of source views can provide our model with more relevant observations from views closer to the target view.
Visualizations. In figure 6 we visualize the learned blending weight maps for all source views (#1, 2, and 3) and the color prediction from the learned implicit body representation (denoted as channel #0). Among the source views, we observe that the view #2 which is the most similar to the target view, is highly weighted in large area. Meanwhile, the implicit body representation takes charge of the overall content regions while especially highlighting the occluded regions that not visible from all source views, (e.g., the left boundary of the torso, and behind the right hand). With our neural appearance blending, NIA able properly blend the available observations and the learned NeRF prediction.
CONCLUSION
We introduce Neural Image-based Avatars (NIA), novel approach for feed-forward avatar creation from sparse view images of unseen persons. Our key idea to learn to adaptively combine the parametric body model-based NeRF and image-based rendering techniques. Our rendering results in novel view synthesis and pose animation tasks exhibit improved robustness and fine details on different human subjects and poses. Notably, NIA outperforms the state-of-the-arr competitors, and even per-subject optimized methods in both tasks. We believe our approach is step forward in generalizing the creation of human avatars capable of their free-view synthesis and pose animation.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGMENTS
We thank Sida Peng of Zhejiang University for many very helpful discussions on variety of implementation details of the Animatable-NeRe We thank Shih- Yang Su of University of British Columbia for helpful discussions on the A-NeRF details. We thank Prof. Helge Rhodin of UBC and his group for the insightful discussions on the human performance capture. This work was partially supported by National Science Foundation Award 2107454.
REFERENCES
Kara-Ali Aliev, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. arXiv preprint TrXiv:/906.08200 2(3):4, 2019.
Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. sriving-signal aware full-body avatars. ACM Transactions Graphics (TOG), -0(,) 2021.
Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. Advances in Neural Information Processing Systems, 33: 12999-12222,, 2020.
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstructured lumigraph rendering. In Proceedings of the 28th annual conference Computer graphics and interactive techniques, pp. 42044224-44 432, 2001.
Jianchuan Chen, Ying Zhang. Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, and Huchuan La. Animat- able neural radiance fields from monocular rgb videos, 2021.
Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Jiashi Feng, and Shuicheng Yan. Geometry- guided progressive nerf for generalizable and efficient neural human rendering. ECCV, 2022.
Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pp. 279-288, 1993.
Wei Cheng, Su Xu, Jingtan Piao, Chen Qian, Wayne Wu, Kwan-Yo Yee Lin, and Hongsheng Li. Generalizable neural performer: Learning robust radiance fields for human novel view synthesis. arXiv preprint arXiv:2204. 1798, 2022.
Paul Debevec, Yizhou Yu, and George Borshukov. Efficient view-dependent image-based rendering with projective texture-mapping. Eurographics Workshop on Rendering Techniques, pp. 105-1 16. Springer, 1998.
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8649-8658, 2021.
Chen Gao, Yichang Shih, Wei-Sheng Lai, Chia-Kai Liang, and Jia-Bin Huang. Portrait neural radiance fields from a single image. arXiv preprint arXiv:2012.0590) 2020.
Steven Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael Cohen. The lumigraph. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pp. 43-54, 1996.
Artur Grigorev, Karim Iskakov, Anastasia lanina, Renat Bashirov, Ilya Zakharkin, Alexander Vakhitow, and Victor Lempitsky. Stylepeople: A generative model of fullbody human avatars. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5151-5160, 2021.
Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Livecap: Real-time human performance capture from monocular video. ACM Transactions On Graphics (TOG), 38(2):1-17, 2019.
10

---

Published as a conference paper at ICLR 2023
Mare Habermann, Weipeng Xu, Michael Zollhofer, Gerard Pons-Mol and Christian Theobalt. Deepcap: Monocular human performance capture using weak supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5052-5063, 2020.
Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll and Christian Theobalt. Real-time deep dynamic characters. ACM Transactions on Graphics (TOG), 40(4):1-16, 2021.
Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation-ready clothed human reconstruction revisited. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11046-11066, 2021.
Peter Hedman, Julien Philip. True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (TOG), 37 2018
Zeng Huang. Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruction of clothed humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3093-3102, 2020.
Ladislav Kavan, Steven Collins, Jiří Zára, and Carol O'Sullivan. Skinning with dual quaternions, In Proceedings of the 2007 symposium on Interactive 3D graphics and games, pp. 39 49-44,,666 2007.
Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. Advances in Neural Information Processing Systems, 34, 2021.
Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference Computer graphics and inteructive techniques, pp. 31-42, 1996.
John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: unified approach to shape interpolation and skeleton-driver deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 162-172, 2000.
Baoyuan Liu, Min Wang. Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp. 806 814, 2015.
Lingjie Liu, Weipeng Xu, Mare Habermann, Michael Zollhöfer, Florian Berard, Hyeongwoo Kim, Wenping Wang, and Christian Theobalt. Neural human video rendering by learning dynamic textures and rendering-to-vine translation. arXiv preprint arX1v:2001.04444,, 2020.
Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Transactions on Graphics (TOG), 4066):1-16, 2021.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. Keypointnerf: Generalizing image-basee based volumetric avatars using relative spatial encoding of keypoints. In European Conference on Computer Vision (ECCV 2022), 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, pp. 405-421. Springer, 2020.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5762-5772, 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin- Burtiin-rruaaa... Nerfies: Deformable neural radiance fields. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 5865-5874, 2021a.
11

---

Published as a conference paper at ICLR 2023
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven Seitz. Hypemerf: A higher-ddmmnnsional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13226 2021b.
Sida Peng. Junting Dong. Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, 2021a.
Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujum Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021
Sida Peng, Shangzhan Zhang, Zhen Xu, Chen Geng, Boyi Jiang, Hujun Bao, and Xiaowei Zhou. Ani- matable neural implicit surfaces for creating avatars from videos. arXiv preprint a3X1v:2203.08/33, 2022.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011. 1.1361,,,1 2020.
Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll, and Christoph Lassner. Anr: Articulated neural rendering for virtual avatars. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3722-3731, 2021a.
Amit Raj, Michael Zollhoefer. Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, and Stephen Lombardi. Pva: Pixel-aligned volumetric avatars. arXiv preprint arXiv:2101.02697 2021b.
Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-alignel implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84-93, 2020.
Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. Advances Neural Information Processing Systems, 34, 2021.
Shih- Shin-Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disentangled articulated neural body representations via graph neural networks. arXiv preprint arXiv:2205.01666 2022.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4): 1-12, 2019.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. arXiv preprint 1xx.:i:2122.10000 2021.
Ricardo
Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. arXiv preprint 22t.22222....,,,. 2022.
Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu. Multi-view neural human rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. (682-1691,22 2020.
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances Neural Information Processing Systems, 34, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. arXiv preprint arXiv:2012.0211 2020.
12

---

Published as a conference paper at ICLR 2023
Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7743-7753 2022.
Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu. Structured local radiance fields for human avatar modeling. arXiv preprint arXiv:2203. 14478, 2022.
13


Reviews:
[{"review_id": "jWM2ow2g08I", "reviewer": "Reviewer_Ke1f", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose Neural Image-based Avatars (NIA), a method for synthesizing novel views and poses. Its main advantages are the implicit body NeRF representation and the image-based rendering. They claim to outperform the current state-of-the-art on the ZJU-MoCap and MonoCap datasets.", "summary_of_the_review": "The paper introduces a new approach to generalized novel views and poses synthesis. Although it mainly put together existing methods (Kwon et al. (2021) and Wang et al. (2021)), it achieves state-of-the-art results. Therefore, I believe the contribution is worthy of publication.", "strength_and_weaknesses": "Main Strengths:\n- The method gathers two complementary state-of-the-art approaches (i.e. neural radiance fields and neural image-based rendering).\n- The NIA is generalizable for different subjects.\n\nMain Weaknesses: \n- The approach relies on sparse view images and the authors do not refer to the LOLNeRF (Rebain et al., CVPR22), a related and recent method based on single views.\n- Most of the previous approaches rely only on PSNR and SSIM metrics. Despite that, it would be interesting to see additional metrics on the evaluation, or at least to read the authors' comments on why other metrics were not adopted.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CBsrgObwkuE", "reviewer": "Reviewer_4ShL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a human full-body avatar model that combines human NeRF and image-based rendering methods. The proposed model achieves significant improvement for unseen pose and unseen person synthesis. Extensive quantitative and qualitative evaluations have been conducted to measure the model's effectiveness.", "summary_of_the_review": "Overall speaking I think this is a solid paper. The method is novel, the results are significantly better than previous state-of-the-art, and the evaluations are thorough. My concerns are minor and can be addressed by additional discussions. Therefore, my initial rating for the paper is accept.", "strength_and_weaknesses": "Paper strengths:\n- The proposed method is novel in combining human NeRF and image-based rendering.\n- The proposed method shows clearly superior performance both qualitatively and quantitatively.\n- Experimental evaluations are extensive and comprehensive.\n\nPaper weaknesses:\n- It seems unclear how much the current model's performance depends on accurate SMPL fitting. Would the proposed model be compatible with in-the-wild selfie videos followed by a single-view SMPL estimation?\n- I am a bit confused by the feature pooling in Equation (1) which currently does not consider visibility, should invisible views be ignored here?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1hNw9dPGbq", "reviewer": "Reviewer_28Gy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines two rendering modules for synthesizing novel views and novel poses of human bodies from sparse multi-view images. \nThe first one is a body representation based on neural radiance fields and SMPL, extracting pixel-to-image features from each view, and attaching features to SMPL vertices. In this way, a coarse feature map for each view can be constructed, which is further converted into dense feature voxels through convolutions. Multi-view features are obtained through view-wise self-attention processing, and colors are finally obtained through MLPs.\nThe second one is a neural image-based rendering module, which can improve the details of the result by directly obtaining the pixel color from the source view image.\nResults of the two modules are finally combined together with the prediction weights.", "summary_of_the_review": "Through extensive experiments, this paper shows better performance than previous methods. And made a preliminary exploration on cross-dataset settings. Some questions/issues need to be fixed. The influence of the view number and the mechanism of appearance blending need to be clarified. Any new insight about the pipeline/module design needs to be further discussed to rebut the novelty concern.", "strength_and_weaknesses": "**Strength**    \n* This method can achieve good results on unseen views and unseen poses. Thanks to the image-based rendering method, it has good generalization performance across datasets.\n* The ablation experiments show that the fusion of implicit body representations and image-based rendering plays a complementary role.\n* Sufficient experiments have proved the effectiveness of the method.\n\n\n**Weaknesses**   \n* When the number of viewing angles is reduced to 1, the results are greatly degraded. Which part of this degradation comes from deserves further analysis, e.g. with regard to multi-view consistency.\n* The appearance blending module shows effectiveness in visualization and quantitative studies. However, why it works or the mechanism it works is not demonstrated clearly.\n* The method directly combines two rendering methods, which both exist in previous works. Any new insight about the pipeline/module design needs to be clarified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1XfBqsyl7w", "reviewer": "Reviewer_Pdgp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to obtain Neural Image-based Avatars (NIA) for synthesizing novel views and poses of an unseen human from sparse input images. NIA is derived from a neural blending network to integrate 1) NeRF-based body representation conditioned on parametric SMPL human body model and 2) neural image-based rendering.", "summary_of_the_review": "The primary concern of this paper is its limited novelty. The key idea is to learn to adaptively combine the parametric body model-based NeRF and image-based rendering techniques, which largely overlap the method proposed by Zhao et al. (2022). Besides, the assumptions of \"the fitted SMPL models are available as prior\" and \"the 3D coordinates of the posed SMPL vertices in both reference and target spaces are known\" are too strong to compare with the existing methods for the fairness concern.  ", "strength_and_weaknesses": "[Strengths]\n+ The motivation is well-written and easy to follow. \n+ The citations are sufficient.\n+ The authors provide several visualizations for realizing their rendering results.\n\n[Weaknesses] \n- The proposed idea novelty is limited. The framework of NIA is similar to [Zhao et al. (2022)] since the two major steps of both methods are generalizable implicit body representation and sparse views blending. The core design of NIA is the neural appearance blending module (sec. 3.2), yet no ablation study to discuss and analyze why such a design helps.\n- Three assumptions exist in this work: 1) the calibration parameters of the multi-view input images are known; 2) the foreground human region masks are known; 3) the fitted SMPL models are available as prior. The first two assumptions are also seen in other methods; however, yet the third assumption is new in this work. The third assumption seems too strong since the SMPL model estimation is a vital factor affecting human NeRF rendering. Therefore, while assuming the fitted SMPL models are available as a prior, it is doubtful that the comparison experiments are fair enough compared with other methods. Notice that Table 4 (i) shows that the NIA w/o image-based rendering (with the third assumption) has already better than most other methods.\n- In sec 3.3, the description of “the 3D coordinates of the posed SMPL vertices in both reference and target spaces are known by the nature of motion-tracked SMPL model” is another strong assumption that makes the NIA available to avoid the finetuning step. It is unclear why the such assumption is available while dealing with the unseen subject with unseen poses.\n- The comparison experiments are not convincing. It seems that some existing methods are reimplemented since the reported results of these methods (degraded) in this paper are not the same as their published ones. It is better to annotate which results are reimplemented and which are not.\n\n- Some implementation details are not clear:\n1) The limitations of the sparse input images. Are there any constraints for view directions?\n2) The feature representation of P_n is not defined;\n3) The design of two MLPs in equation (1) is different from NeRF [Mildenhall et al. (2020)]. Why is such a design reasonable, and what is its advantages?\n4) The architecture of F_B in equation (2). Does (2) means that it has one single model for a specific value of N?\n5) SE(3), x^can, k-th part sampling method, function R(), function t() in sec. 3.3;\n\nReference:\n[Zhao et al. (2022)] Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In CVPR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster