Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
Authors: YoungJoong Kwon
Dahun Kim
Duygu Ceylan
Henry Fuchs
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Novel view synthesis
Neural radiance fields
Human performance capture
Generalizable Neural radiance fields

Abstract: In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses.

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
Youngjoong Kwon', Dahun Kim?, Duygu Ceylan', Henry Fuchs'
University of North Carolina at Chapel Hill. ²KAIST. Adobe Research. (youngjoong fuchs)0cs unc. edu (mcahny)okaiist.... ac. {ceylan) com
Abstract
In this paper, we aim at synthesizing a free-view vieepvien video an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of particular human. In parallel, some work proposed use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, novel approach that learns generalizable neural radiance fields based on parametric human body model for robust performance capture. Specifically, we first introduce temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, multi-view transformer is proposed to perform cross-attention between the temporally- fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the 7JU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code available at hutps://youngooongune htw.:ngiioonnuun.gguui...iih.aii githubbio/nhp.
Introduction
Free-viewpoint video of human performer has variety of applications in the area of telepres- ence, mixed reality, gaming and etc. Conventional free-viewpoint video systems require extremely expensive setups such as dense camera rigs [4, 7, 4,43 or accurate depth sensors [6, 13], to capture person person-specific appearance information. In this paper, we aim at a scalable solution for free-viewpoint human performance rendering that generalizes across different human performers and requires only sparse camera views. However, representing and rendering arbitrary human performances ex- tremely challenging when the observations highly sparse (up three to four views) due to heavy self-occlusions and dynamic articulations the body parts. In particular, an effective solution needs to coherently aggregate appearance information from sparse multi-view observations across time as the body undergoes a 3D motion. Furthermore, the solution needs to generalize to unseen motions and characters at test time.
Recently, neural radiance fields (NeRF) [26, 11, 17, 30, 33, 35, 36, 47, 50, 52, 53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36, 52, 47] have been proposed which use inage-conditioned, pixel-aligned features and achieve feed- feed-forward view synthesis from sparse input views [36, 52]. Direct application of these methods to complex and non-rigid human motion is not straightforward, however, and naive solutions suffer from significant artifacts as shown in Fig. 3. Some existing methods [37, 52] aggregate image features across multiple views by simple average pooling, which often leads to over-smoothed outputs when details observed from multiple
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

views (e.g., front and side views) differ due to self occlusions of humans. Alternatively, several methods [23, 33] have proposed learn person-specific global appearance features from multi-view observations. However, such methods are not able to generalize to new human performers.
To address these challenges, we propose Neural Human Performer, novel approach that learns generalizable radiance fields based on a parametric 3D body model for robust performance capture. In addition exploiting a parametric body model as geometric prior, the core of our method is a combination of temporal and multi-view transformers which help to effectively aggregate spatio-temporal observations to robustly compute the density and color of query point. First, the temporal transformer aggregates trackable visual features based on the input skeletal body motion across time. The following multi-view transformer performs tross-attention between the temporally- augmented skeletal features and the pixel-aligned features for each time step. The proposed modules collectively contribute to the adaptive aggregation of multi-time and multi-view information, resulting in significant improvements in synthesis results in different generalization settings of unseen motions and identities.
We study the efficacy of Neural Human Performer on two multi-view human performance capture datasets, ZJU-MoCap [33] and AIST [16]. Experiments show that our method significantly out- performs recent generalizable radience field (NeRF) methods. Furthermore, we compare ours with -peenitt--peccfic methods [33, 44, 49] that also utilize 3D human body model prior. Surprisingly. our generalized method achieves better rendering quality than the person-specific dedicated methods when tested on novel poses demonstrating the effectiveness of our transformer-bee based generalizable representation.
To summarize, our contributions are:
We present a new beed-forward method of synthesizing novel-view videos of arbitrary human performers from sparse camera views. We propose Neural Human Performer that learns generalizable neural radiance representations by leveraging 3D body motion prior.
We design a combination of temporal and multi-view transformers that can aggregate information on the fly over video frames across multiple views to render each frame of the novel-view video.
We show significant improvements over recent generalizable NeRF methods on unseen identities and poses. Moreover, our generalization results can outperform even person- specific methods when tested on unseen poses.
2 Related works
Human performance capture. Novel view synthesis of human performance has long history in computer vision and graphics. Traditional methods rely on complicated hardware such as dense camera rigs [4, 7, 43] or accurate depth sensors [6, 13]. To enable fiee-view video from sparse camera views, template-base
methods [3, 5, 10, 42] exploit pre-scanned human models to track the motion of person. However, their synthesis results not photo-realistic and pre-scanned human models are not available in most cases. Recent methods [27, 37, 38, 54] learn 3D human geometry priors along with pixel aligned features to enable detailed 3D human reconstructions even from single images. However, these methods often suffer under complex human poses that are never seen during training and hence cannot be directly used for our purpose of human performance synthesis.
Neural 3D representations. Recently, there has been great progress in learning neural networks to represent the shape and appearance of scenes. The 3D representations are learned from 2D images via differentiable rendering networks. Convolutional neural networks are used to predict volumetric representations via 3D voxel-grid features [40, 23, 29, 25, 14, 15], point clouds [l, 49], textured meshes [18, 21, 44] and multi-planc images [8, 55J. The learnt representations are projected by a 3D-to-2D operation to synthesize images. Some methods [14, 15] introduce consistency loss to improve the multi-vier and temporal consistency. However, these methods often have difficulty in scaling to higher resolution due to memory restrictions.
To eschew these problems, implicit function-based methods [20, 22, 28, 41] learn a multi-layer perceptron that directly translates a 3D positional feature into pixel generator. The more recent NeRF [26] learns implicit fields of density and color with a volume rendering technique and achieves photo-realistic view synthesis. Among many following NeRF extensions, [30, 31, 35, 53, 17, 50]

---

d
Skeletal feature bank
Cans-1 Cam-2
NoRF
(rgb.
Fused quary feature
Temporal Transformer See 3.2
Multi-view Transformer See 3.3
Fixel-aligned features query
Trilineur sampling
Time-mugmented
Sampled skeletal
Car
skeletal Features
feature w.r.t
Figure 1: Given sparse set of multi- view videos person, Neural Human Performer uses temporal and multi-view transformers to learn pixel aligned features on the surface a deformable body template such as SMPL. Such features are utilized to predict the color and density value of query points to render novel views of possibly unseen poses of the person.
focus on dynamic scenes. While showing impressive results, it's an extremely under-constrained problem jointly learn NeRF and highly dynamic deformation fields. To regularize the training, Neural Body [33] combines NeRF with a deformable human body model (e.g. SMPL [24]). Despite the promising results, these general deformable NeRF [17, 53] and human-specific NeRF [11, 33, 32] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36, 47, 52] try avoid the expensive per-scene optimization by image-conditioning using pixel-aligned features. However, directly extending such methods to model complex and dynamic 3D humans is straightforward when available observations are highly sparse. Unlike existing works, our method exploits temporal and multi-view information on-the-fit fly and achieves free-vis viewpitt human rendering feed-forware manner, also generalizing new, unseen human identities and poses.
3 Method
3.1 Neural Human Performer
Problem definition. In our setting, given a sparse set (e.g., 3 or 4) multi-view cameras I...., C, input videos of arbitrary human performance Le,1-T captured for each camera view defined by (Ke, [ We assume that the 3D human body model fit corresponding to each frame is given. Our goal is to synthesize a novel view video fgr for query viewpoint q defined by (K,, [-
Le.,1, Lerr are
Overview. To compensate for the sparsity of available input views, we propose exploit temporal information across video frames. In practice, we sample memory frames from the original input videos to augment each queried timestep t. Our goal is to learn generalizable 3D representations of human performers from multi-time (M) and multi-view (C) observations.
To this end, the Neural Human Performer is proposed with two main components illustrated in Fig. 1. First, we construct the time-augmann auggeneeeented skeletal features {s}. We exploit a human body model (SMPL [24]) to construct 3D skeletal features by projecting all the SMPL vertices onto cach memory frame and picking up the pixel-aligned image features [36, 37, 52] at the projected 2D locations. The skeletal features sampled from all memory frames to construct the skeletal feature bank. Inspired by Transformers(2, 46, 48), we propose a temporal Transformer that aggregates these memory features into the time augmentdd skeletal features {s'}.
In the second stage, given query 3D point x, skeletal features are sampled at x. addition, pixel-aligned features (p} at each time / are sampled by directly projecting x onto the input images (7).. The multi-view Transformer proposed learn the correlation between the pixel-aligned features {p) and the time-augmented skeletal features {s') and to adaptively fuse the multi-view

---

information. The final fused feature of the query point x is fed into the radiance field module to predict its color and density.
3.2 Construction of time-augmented skeletal features
Unlike static scenes, video inputs of moving characters inherently contain more visual cues as the occluded regions frame may be visible other (potentially distant) frames. To take advantage of this temporal information, we first build up the skeletal feature bank from memory frames by leveraging parametric body model (see Fig. 1). Then, we propose temporal Transformer module that aggregates such collected features.
For each view and time t, we first build frame-level skeletal features Sc,t R 1xd by sampling image features at the projection of the SMPL vertices in each image Le,t Hx W33 L denotes the number SMPL vertices and d is the dimension of image features.
After collecting all the skeletal features from all memory frames, we aggregate the time information in an attention- aware manner, instead of using simple average pooling. Specifically, for any 1th skeletal feature vertex s E IRd, the proposed temporal Transformer casts attention over all other features contained in the vertex's memory bank s S,tt ( S Se,t ... st Se,ts) } R Mxd In particular, we compute soft weights for all memory feature vertices in a non-local manner with respect to the current timestep I. Then, the values of the memory features are weighted summed as
Lalte = 15( (V 2 a(s'e)) ((Fo
Latta E RRIXM
(1)
"'c, LLatt' v t
5e, E ER Vi
where yb represents the softmax operator along the second axis, g(-), k(.) and u(-) are learnable query, key and value embedding functions Rd-+do of the temporal Transformer.
In other words, the representation s' of each skeletal vertex at time t is computed through dynamically weighted combination of all its previous and next representations in the memory frames. This allows network to incorporate helpful information and ignore irrelevant ones from other timesteps. In practice, the temporal Transformer operation in Eq. (1) is performed by a batch matrix multiplication for all skeletal vertices L and all available viewpoints C.
3.3 Multi-view aggregation of skeletal and query features
Given a query 3D point x IRS, we retrieve the corresponding (timeaugmmeted) skeletal feature o',," x E Rd at the queried location via trilinear interpolation in the SMPL space with SparseCon- vÑet [19], following [39, 34, 51, 33].
In addition, we sample pixel-aligned image feature ph, via direct 3 projection of the query point x on Le,t. is important note that the pixel-aligned feature Pč,, is time-specific and represents the exact query location of x, while the skeletal feature s'c.," is time-augmented (w.r.t t) and contains inherent geometric deviations in the SMPL vertices and the following trilinear interpolations. We propose to combine these two complementary features, which will be shown to be effective in Sec, 4.4.
Given the two sets of view features, skeletal $1.,., x = { % = 1...., C} E RCdd and pixel-aligned pi.c.. (pč,/c 1,.... C) RCxd we propose multi-view Transformer that performs cross-attention from skeletal to pixel-aligned features. Specifically, the values of pixel- aligned features from all viewpoints is re-weighted based on how much compatible they are with each skeletal features. The noc-l cross- attentttnn me att is constructed as:
me mmt_alfy = vb( 1 M(sh..e)) ) k(pt.c.)"),
me me_atty E C
Van
(2)
.... mw_att, u(Pt.c.) + uushic, "
* E RCdd
2x E Rld
where 1 represents the softmax operator along the second axis. Note that and v are new layers independent from those in the temporal Transformer. The confident observations in each view will have large weights and be highlighted, and vice versa. Finally, we use the view-wise mean of
xx t/4 E. xx E ddd as meta-time and meta-view representation of the query point

---

The final density (x) and color values rgb, at time t are computed as:
(x) (2))
(g),(X) M Σ Ta
(3)
where MTP,, and rgg consist of four and two linear layers respectively, and a 3 is positional encoding of viewing direction g in [26] with 2 x different basis functions. More details on the network architecture can be found in the supplementary material.
3.4 Volume Rendering
The predicted color of a pixel IR2 for target viewpoint q in the focal plane of the camera and center ro IR3 obtained by marching rays into the scene using the camera-to world projection
matrix, P R [tg ' with the direction of the rays given by d = P pp-ra
p-rroll
We then accumulate the radiance and opacity along the ray r(z) ru zd for E Znear zarr as defined in NeRF [26] as follows:
Flar
T())o(((3))c(((()) d)dz, where T(z) exp (-f² a(r(s))ds)
(p)
(4)
Pasar
Racar
In practice, we uniformly sample a set of 64 points z [Znear, farl. We set X = r(2) and use the quadrature rule to approximate the integral. We compute the 3D bounding box of the SMPL parameters at time and derive the bounds for ray sampling Znear, 2far.
3.5 Loss Function
Given the ground truth target images Ig,1, train both the radiance field and feature extraction networks using simple photometric reconstruction loss L [ 1.
4 Experiments
We present novel view synthesis and 3d reconstruction results of human performances in different generalization scenarios. We compare our method against the current best view-synthesis methods from two classes: per-subject optimization methods that also and generalizable NeRF methods (Sec. 4.2). We experiment on the ZJU-MoCap [33] and AIST datasets [45, 16]. For training and testing our model well as the baselines, we remove the background using the foreground masks that are either provided by the dataset or pre-computed using an off-the-sher shelf method. Unless otherwise specified, we sample two memory frames (t - 20, + 20} at time (i.c.. three timesteps in total) and take three canonical input views all experiments. The details of the implementation, datasets, training process are provided in the supplementary material.
human body model prior (Sec. 4.
4.1 Comparison with body model-based, per-subject optimization methods.
Baselines. We perform comparisons with the state-of-tt ofee----thttttt art Neural Body (NB) [33] that combines the body model prior SMPL and NeRF in a per-subject optimization setting. Neural Textures (NT) [44] renders a coarse mesh with latent texture maps and uses a 2D CNN to render target images. We use the SMPI. mesh as the coarse mesh input to NT in our experiments. NHR [49] extracts 3D features from input point clouds and renders them into 2D images. Since dense point clouds are difficult obtain from sparse camera views, we use the SMPL vertices input point clouds. These methods have reported that they can adapt to new poses of the same performer, hence we compare to them for the task of novel pose synthesis.
Setup. We experiment with ZJU-MoCap dataset [33] which provides performance captures of 10 human subjects captured from 23 synchronized cameras, human body model parameters as well as foreground mask corresponding to each frame. Each video contains complex motions such as kicking and Taichi and is between 1000 to 2000 frames long. We first split the dataset into two parts: source and target videos. In all comparisons, the first 300 frames of either source or target videos are used during training, and the remaining next frames (unseen poses) are held out for testing. Note that
5

---

Method
PSNR
SSIM
Method
PSNR
SSIM
Method
PSVR
SSIM
Trained on source subjects
Trained on target subjects
Trained subjects
NB
23.79
0.887
NB
22.88
0.880
NB
28.51
0,947
NHR
22.31
0.871
NHR
22.03
0.875
NHR
23.95
0.897
NT
23.86
0.896
NT
22.28
0.872
NT
21.92
0.873
Trained on source subjects
Trained on source subjects
Trained on source subjects
Ours
28.73
0.936
Ours
26.94
0.929
Ours
24.75
0.906
per-subject
31,57
0.966
a Test results on unseen poses of b. Test results on unseen poses of c. Test results on seen poses of source source subjects
target subjects
subjects
Table 1: Comparison with other body model-based, per-subject optimization methods.
View
81
View #2
GT
NT
NHR
NB
Ours
GT
NT
NHR
NB
Ours
Figure 2: Pose generalization comparison with per-scene optimization methods that use human body prior. Results of NT(Neural textures) [44], NHR (Neural human rendering) [49]. (Neural body) [33] and ours. Novel view synthesis on the ZJU-MoCap for unseen poses of source subjects. While ours is a single model trained on all source subjects; baselines are trained in a per-subject manner.
the baseline methods are always trained in a per-subject manner. To validate whether the training reproducible, experiment with 5 independent runs with random train/test splits and observe variance of 0. PSNR, showing that the results are consistent. In each independent run, we used seven models for training and the other three for testing.
Results. We present three different comparison settings to validate our method. We would like to point out that all the comparison settings place method ('ours') ) in disadvantage. This is because our model is trained on all the source subjects at once (one network for all subjects), while the competing methods are per-subject trained on the subject to be tested (one network for one subject) easier setting. First, we evaluate the pose generalization capability (Table. 1a and Fig. 2) where we train on the training splits of the source subjects and test the testing splits, i.e., unseen poses, of the same source subjects. Our method significantly outperforms all the baselines and the state-of-the-art Neural Body [33] by +3.15 PSNR and +0.042 SSIM scores. We next evaluate the identity generalization capability (Table. 1b) by testing on the unseen poses of target subjects. While method is trained source subjects only, baselines
trained target subjects. Note that this comparison is disadvantageous to us since unlike our method the baselines have seen the testing subjects as they must be trained separately for each subject. Surprisingly. however, our method still outperforms all the baselines by a healthy margin of +1.87 PSNR and +0.026 SSIM scores. These results indicate that our proposed architecture with the temporal and the multi-view transformers generalizes well across unseen identities and poses, and produces photo-realistic results. Finally, in Table. 1c, we evaluate our method and baselines on seen poses of seen subjects. Even when we train single model of our method for all the source sobjects ('ours'), ours achieves comparable results with the state-of-the-art per-subject based methods [33]. To make a more fair comparison, we also train our method in the same per-subject setting similar to the baselines. Table. Ic shows that in this case, our model ('ours per-subject') achieves a significant improvement over the pest-performin baseline [33] by +3 PSNR and +1.4% SSIM.

---

View #1
View #2
GT
Pixel-Nerf
PVA
Ours
GT
Pixel-Nert
PVA
Figure 3: Identity-and-pon pose generalization comparison with generalizable NeRF methods. Results of Pixel-Nerf [52], PVA aPixel-aligned volumetric avatar) [36] and ours. Novel view synthesis on the ZJU-MoCap dataset where all methods are trained on source subjects and tested on unseen poses of target subjects.
Method
PSNR
SSIM
Method
PSNR
SSIM
PeRe
23.17
0.8693
Pixel-NeRF
18.06
0.7304
PVA
23.15
0.8663
PVA
17.82
0.7211
Ours
24.75
0.9058
Ours
19.03
0.8390
a Generalization results on ZJU-MoCap.
b. Generalization results on AIST.
Table 2: Comparison with generalizable NeRF methods.
4.2 Comparison with generalizable NeRF methods.
Baselines.
Among the recent generalizable NeRF methods [36, 52, 47], we compare with Pixel- NeRF [52] and PVA [36] which focus on very sparse (up to or 4) input views. We reimplement [36] since the code is not available.
Setup. In addition to the ZJU-MoCap dataset (see Sec. 4.1), we experiment on the larger AIST dataset [45, 16] to further evaluate the generalization abilities of different methods. AIST dataset provides dance videos of 30 human subjects captured from 9 cameras, together with the corresponding SMPL fits. We extract the foregroud mask of each image using an off-the-shelf human parser [12]. AIST dataset contains highly diverse motions, slow to fast, simple to complex. We split the dataset into 20 and 10 subjects for training and testing respectively, where the testing dataset contains unseen poses of unseen subjects.
Novel view synthesis results. As shown in Table. 2, across all the datasets and all the metrics, our method consistently outperforms the baselines by healthy margins of +1.6 PSNR and +0.037 SSIM scores. Fig. and Fig. 5 present the same tendency in visualizations. Pixel-NeRI and PVA aggregate multi-view observations via average pooling without explicitly considering the correlation between the views. In contrast, our temporal and multi-view transformers learn model the correlation between input views and integrate different observations to help the NeRF module to produce higher quality results. Another advantage of our method is that the used body model prior provides a robust geometric cue to handle the self-occlusion of the human body.
3D reconstruction results. We also evaluate the 3D reconstruction quality of generalizable NeRF methods and our method on the ZJU-MoCap (Fig. 4) and AIST datasets (Fig. 5) given three input views. The visualizations show that 3D reconstructions align well with the input images, and is more accurate than even the per-subject method [33] (e.g., the shape of the upper cloth in Fig. 4).
Overall, these results verify that since humans are complex and prone to occlusions, more sophis- ticated designs compared to simple imagc-conditioning are required to learn robust and accurate appearance representations.
4.3 Cross-dataset generalization.
We further study the generalization ability of our method across different datasets by training on one datasct and testing another one shown in Table. 4a. We would like to point out that the two
7

---

NB per person optimized
Impal views
Pixel-NeRF
PVA
Ours
Figure 4: 3D reconstruction on Z.U-MoCap. Tested on unseen poses of unseen subjects (except Neural Body which is a subject specific method). NB (Neural Body) [33], PVA (Pixel volumetric avatar) [36]. Pixel-NeRF [52] and ours.
View
View #2
GT
Pixcl-Nerf
Ours
GT
Pixcl-Nerf /
Ours
Figure 5: Generalization results on AIST. Novel view synthesis and 3D reconstruction results on unseen poses of unseen subjects.
datasets [33, 16] have significantly different statistics both in terms of color distribution (background, lighting) and distance of the camera to the subject, making the cross-dataset generalization task extremely challenging. Nevertheless, already outperform the baselines fully-trained on the target dataset. 16-minute fine-tuning performs on par with our model fully-trained the target dataset.
found that only 8-minute fine-tuning on the target dataset can
4.4 Ablation studies
Table. 3 shows the ablation study the ZJU-MoCap dataset for unseen subjects and unseen poses, using three time-steps and three camera views as input. Note that all the variations without either temporal or multi-view transformer modules use simple average pooling instead, fuse temporal multi-view observations respectively.
Complementariness of skeletal and pixel-aligned query features. 'Sk' uses only time- augmented skeletal features (Sec. 3.2) without time-specific pixel-aligned features, while 'Px' uses the contrary. Both 'only' models show the largest
only the time-specific pixel-aligned features, drops compared to our full model, and 'Sk + Px' model improves them by +1.2 PSNR and +0,9 PSNR respectively. This validates the complementariness between the skeletal and pixel-aligned features in that one is time-augmented but involves slight geometric deviations, while another is time-specific and represents exact query location, discussed in Sec. 3.3.
Impact of temporal and multi-view transformers. 'Sk Px' uses no transformers so far, falling behind our full model by -1.3 PSNR score. Then 'Sk Px + T adds the temporal transformer and improves +0.7 PSNR score, showing its effectiveness in aggregating information over video frames. 'Sk + Px MV" uses the multi-view transformer module and shows the largest gain of +1.0 PSNR, indicating the efficacy of learnt cross-attention between the skeletal features and pixel-aligned features, well the importance of learnt inter-view correlations. Our full model 'Sk Px T + MV' shows the best use of all the proposed components and achieves 24.75 PSNR and 0.9058 SSIM.
Impact of number of camera views. Table. 4b shows the performance of our method when tested with different number of input views, fixing the number timesteps to one. As expected, the performance degrades slightly with fewer input views (c.g., as few as one).

---

Variant
Skeletal
Pixel-aligned
T-transformer
MV-transformer
PSNR
SSIM
Sk
22.31
0.8865
Px
22.58
0.8780
Sk + Px
23.47
0.8906
Sk + +
24.21
0.9016
Sk S MV
24.44
0.9034
P
24.75
0.9058
Table Ablation study. Results the ZJU- dataset for subjects and unseen poses. Sk: skeletal features, Px: pixel-aligned features, temporal transformer, MV: multi-view transformer.
Exp. protocol
Fine-tune
PSNR
SSIM
# Timestops
# Views
PSNR
SSIM
Trained AIST Fine-tuned on MJc-p
8-min
24.25
0.8954
I
20.13
0.835
16-min
24.73
0.9023
2
21.82
0.871
Trained Moc-p
8-min
18.63
0.8242
3
23.33
0.906
Fine-tuned on AIST
16-min
18.83
0.8374
23.51
0.913
a Cross-dataset generalization.
b. Impact of number of camera views.
Table 4: Cross-dataset generalization (left) and impact of different number of camera views tested the ZJU-Mocap dataset for unseen poses of unseen subjects (right).
Limitations
While our novel generalizable human performance rendering method outperforms recent per-subject and generalizable NERF methods, there are remaining challenges yet to be explored. 1) While we show that our method can generalize across datasets with finetuning, the generalization capability will be limited as the distribution of the datasets become significantly different. 2) The performance of our method will be affected as the SMPL parameter accuracy degrades. This is the reason for lower performance on the AIST dataset where the accuracy of the SMPL fits are low due to highly complex motion sequences. It is an interesting direction to jointly refine the SMPL parameters using differentiable rendering within our framework. 3) Our algorithm does not have an explicit assumption of static cameras. However, in practice, might be hard to estimate the inputs to method (SMPL fits, camera parameters, foreground masks) with moving cameras due to motion blur, changing background, lighting, and synchronization issues etc. We consider this as orthogonal problem and expect that any advancements in unconstrained multi-view capture setups will help to generalize our method in-the-wild settings.
Societal impact
We discuss the potential societal impact of our work. On a positive note, the human performance synthesis is the key component of realizing telepresence, which has become more important especially in this pandemic era. In the future, people physically apart can feel like they are in the same space and feel connected with a few inexpensive webcams and AR/VR headsets thanks to the development of the telepresence. Like any other technology, our method can also be used for other reasons by corporations, c.g., for identity recognition from a small number of surveillance cameras. We strongly hope that our research will be used in positive directions.
7 Conclusion
We present Neural Human Performer, generalizable radiance field network based on a parametric body model that can synthesize viewpottt videos for arbitrary human performers from sparse camera views. Leveraging the trackable visual features from the input body motion prior, we propose a combination of temporal and multi-view Transformer that integrates multi-tine time and multi-view observations in feed-forward manner. Our method can produce photo-realistic view synthesis of unseen poses and identities at test time. In various generalization settings on the ZJU-MoCap and AIST datasets, our method achieves state-of-the-ee performance compared to other per-subject methods that utilize human body prior as well as the generalizable NeRF methods.

---

Acknowledgments and Disclosure of Funding
We thank Sida Peng of Zhejiang University, Hangzhou, China, for many very helpful discussions on a variety of implementation details of the Neural Body [33]. We thank Ruilong li and Alex Yu of UC Berkeley for many discussions on the + dataset [16] and Pixel-NeRF [S2] details. We thank Prof. Alex Berg of UNC for the generous offer of computational resources and Misha Shvets of UNC for a useful tutorial on it. This work was partially supported by National Science Foundation Award
1840131.
References
[1] Kara-Ali Aliev, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. arXiv preprint arXiv: 0xXiv:906.002400 2(3):,, 2019.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213 229. Springer, 2020.
[3] Joel Carranza, Christian Theobalt, Marcus A Magnor, and Hans-Peter Seidel. Free-viewpoint video of human actors. ACM transactions on graphics (TOG), 22(3):569-577, 2003.
[4] Alvaro Collet, Ming Chuang. Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics (ToG), 34(4):1-13, 2015.
[5] Edilson De Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun, Performance capture from sparse multi-view video. ACM SIGGRAPHI 2008 papers, pages 1-10. 2008.
[6] Paul Debevec, Tim Hawkins, Chris Tehou, Haarm-Pieted Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145-156, 2000.
[7] Mingsong Dou, Samch Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al. Fusion4d; Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG), 35(4):1-13, 2016.
[8] John Flynn, Michael Broxton, Paul Dehevec, Matthew Du Vall, Graham Fyffe, Ryan Overbeck, Noah Snavely. and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367-2376. 2019.
[9] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8649 8658, 2021.
[10] Juergen Gall, Carsten Stoll, Edilson De Aguiar, Christian Theobalt, Bodo Rosenhahn, and Hans-Peter Scidel. Motion capture using joint skeleton tracking and surface estimation. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1746-1753 IEEE, 2009.
[11] Chen Gao, Yichang Shih, Wei-Sheng Lai, Chia-Kai I Liang, and Jia-Bin Huang. Portrait neural radiance fields from a single image. arXiv preprint arXiv:2012.05900 2020.
[12] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang. and Liang Lin. Instance-level human parsing via part grouping network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 770-785, 2018
[13] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Gcoff Harvey, Sergio Orts Escolano, Rohit Pandey, Jason Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Transactions on Graphics (TOG), 38(6): 2019.
[14] Youngjoong Kwon, Stefano Petrangeli, Dahun Kim, Haoliang Wang, Henry Fuchs, and Viswanathan Swaminathan. Rolationally-consisten novel view synthesis for humans. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2308-23 2020.
[15] Youngjoong Kwon, Stefano Petrangeli, Dahun Kim, Haoliang Wang, Eunbyung Park, Viswanathan Swaminathan, and Henry Fuchs. cogationally-temporee consistent novel view synthesis of human performance video. In European Conference on Computer Vision, pages 387-402. Springer, 2020.
10

---

[16] Ruilong Li, Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In International Conference on Computer Vision, 2021.
[17] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3d video synthesis. arXiv preprint arXiv:2103.02554 2021.
[18] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5871-5880 2020.
[19] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 806 806-814, 814, 2015.
[20] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. arXiv preprint arXiv:2007. 11571, 2020.
[21] Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt. Neural rendering and reenactment of human videos. ACM Transactions on Graphics (TOG), 38(5):1144, 2019.
[22] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[23] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint 1oXxxv:9006.0777 2019.
[24] Matthew Loper, Naurcen Mahmood, Javier Romero, Gerard Pons-Moll and Michael Black. Smpl: A skinned multi-person lincar model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[25] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[27] Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Mor- ishima. Siclope: Silhouette-hased clothed people. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4480-4490, 2019.
[28] Michacl Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Diffcrentiable volumetric rendering: Learning implicit 3d representations without 3d supervision, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[29] Kyle Olszewski, Sergey Tulyakov, Oliver Woodford, Hao Li, and Linjie Luo. Transformable bottleneck networks. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 7648- 76557, 2019.
[30] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo-Martin Brualla. Deformable neural radiance fields. arxiv preprint arXiv:2011.12945 2020.
[31] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A cally varying neural radiance fields. arXiv preprint arXiv:2 f. 13228, 2021.
representation for topologi-
[32] Sida Peng. Junting Dong. Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13323, 2021.
[33] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.

---

[34] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. arXiv preprint arXiv:2003.04618, 2, 2020.
[35] Albert Pumarola, Enric Corona, Gerard Pons-Moll. and Francese Moreno-Noguer, D-nerf: Neural radiance fields for dynamic seenes. arXiv preprint arXiv:201 1.39611, 2020.
[36] Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, and Stephen Lombardi. Pva: Pixel-aligned volumetric avatars. arXiv preprint (tXix:2/01.02665 2021.
[37] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314 2414, 2019.
[38] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84 84-93, 2020.
[39] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remm: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529- 10538, 2020.
[40] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2437-2446, 2019.
[41] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d structure-awant -warre neural scene representations. arXiv preprint arXiv:1906.00 /906/996660688. 2019.
[42] Carsten Stoll, Juergen Gall, Edilson De Aguiar, Scbastian Thrun, and Christian Theobalt. Video-based reconstruction of animatable human characters. ACM Transactions on Graphics (TOG), 29(6):-10, 2010.
[43] Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, et al. Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd camera. Springer, 2020.
[44] Justus Thies, Michael Zollhöfer, and Matthias NieBuer. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
[45] Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In ISMIR, pages 501-510, 2019.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:/706.000 2017.
[47] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Marain-Brualla. Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.
[48] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7794-7803, 2018. 8.
[49] Minye Wu, Yuchao Wang, Qiang Hu, and Jingyi Yu. Multi-view neural human rendering. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1682-1691, 2020.
[50] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. arXiv preprint arXiv:2011.1295 129500. 2020.
[51] Yan Yan, Yuxing Mao, and Bo Li. 18(10):3337, 2018.
Second: Sparsely embedded convolutional detection. Sensors,
[52] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[53] Wentao Yuan, Zhanyang L.v, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. arXiv preprint 1011v:2101.01602 2020.
[54] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from a single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7739-7749, 2019.
2019.
[55] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 185. 2018.
12


Reviews:
[{"review_id": "iXSD26-GQlR", "reviewer": "Reviewer_r7DT", "summary": "This paper presents a method to synthesize a novel view image of a person given the sparse multiview images. The paper proposes time-augmented skeletal features---the pixel-aligned features that encode NeRF-like rendering fields by aggregating image features across  time and views, parametrized by a human mesh model. They use transformer networks to learn fusing multiview image features over time. The resulting features are used to generate an implicit function that can predict RGB and opacity. They demonstrate that the proposed method is generalizable to unseen identities and poses, validated on ZJU-MoCap and AIST datasets. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n(+Novelty) This paper is built upon Neural Body [31] framework (NeRF+SMPL) that parametrizes the neural radiant field using the SMPL vertices, which synthesizes a novel view image given a set of multiview images. This paper takes a new step to learn a visual representation of humans using pixel-aligned features that can be used to generalize to new subjects. In this sense, it combines NB and PIFu, which is novel and shown to be very effective. The image feature is in particular useful for occluded regions of body because the learned image features are rather viewpoint invariant.  Further, they propose a novel way to fuse multiview and time features through self-guided attention through transformers, which shows stronger performance compared to existing average or max-pooling (similar idea for multiview fusion has been used in [a] though). \n\n[a] Stable view synthesis, CVPR 2020\n\n(+Performance) The experiments with multiple baselines (PixelNeRF, NB) show that the proposed image feature learning is highly effective on generalization of subjects and poses in particular, for unseen subjects and poses in terms of PSNR and SSIM metrics. The result in the supplementary video well illustrates the strengths of the proposed method. Further, the paper includes substantial ablation study that shows the impact of each component.  \n\n\nWeakness:\n\n(Justification of transformer) Learning feature fusion over views is interesting but not sure how effective it is (do we really need to learn, or can use a weighted average based on viewing directions to fuse similar to [a]?). I suspect that performance wise, it will be pretty similar. If it is not way better than such weighted average skeme, having transformer over views is overkill and may limit the generalizability to a new camera configuration.   \n\n(Justification of temporal aggregation) Table 1-c illustrates the performance upper bound of the proposed method, which is on par with NB. This indicates temporal aggregation is not that effective. I thought the main claim of using temporal aggregation is that it can observe occluded regions as the performer changes poses so the features on the occluded regions can be taken into account. However, this is not reflected in the Table. I would suggest to conduct experiments that can justify the temporal aggregation for such a controlled experiment (this is different from the one in the ablation study).,\n\n\n\nMore comments:\n-R^{3->2} to R^{3}->R^2\n-L141: what does it mean by second axis?\n-Since the network is supposed to learn human appearance agnostic to camera poses, it would be nice to demonstrate cross-dataset evaluation, e.g., learning from ZJU and testing on AIST. \n-Adding the qualitative results for Table 1-(c) would be useful to see the performance upper bound of the proposed rendering method.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "e5OmaCp4djX", "reviewer": "Reviewer_8dkm", "summary": "This paper presents a generalizable NeRF network that can produce novel view synthesis for novel dynamic human scenes from sparse camera views, in a feed-forward manner. It leverages visual features from tracked parametric body model and similar to PixelNeRF, it conditions the NeRF representation on pixel-aligned image features for appearance generalization. A novel combination of a temporal Transformer and a multiview Transformer is proposed to integrate multi-frame and multi-view visual observations. It demonstrated stronger generalization capability than prior work. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Creating a free-viewpoint video synthesis using a general model is a highly challenging problem.  The paper learns a generalizable NeRF for various dynamic human scenes, with a major contribution on a novel combination of a temporal Transformer and a multiview Transformer that enables better integration of visual cues across time and views. Even though the components of leveraging geometric surface prior (SMPL as done in NeuralBody) and pixel-aligned image features (as in PixelNeRF) for human NeRF learning are not new, the work has achieved stronger generalization capability and modeling power for dynamic human scenes with the proposed learning framework.  \n\nThe paper is well written and easy to follow. The experiments are extensive and well structured, demonstrating strong generalization capability to novel poses and identities compared with SOTA approaches. \n\nThere are a few places that could be improved.\n- It is not clear how important the accuracy of the parametric body tracking is, during at both training and testing stage. The paper demonstrates the experiments on AIST and ZJUMocap, both of which are indoor scenes with 3d mocap data. Therefore I assume it is less a problem in obtaining accurate SMPL parameters estimation. However, for in-the-wild video datasets, the body tracking accuracy might largely degrade and I will be curious to understand how the proposed learning framework will perform. \n\n- For the image evaluation metrics, it is not clear they are reported on novel camera views or the learned camera views. \n\n- It will be good to see the cross-dataset generalization capability, even just between AIST and ZJUMocap datasets. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Well addressed.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "_Fu399qfWHj", "reviewer": "Reviewer_5d4Y", "summary": "The paper solves a problem similar to Neural Body (reconstruction / free-view rendering of clothed humans from videos based on SMPL fits) but does not require training a neural network per scene; instead, it generalises from training sequences and, at test time, runs only a feed-forward model conditioned on keyframes. This is important for many applications since training a neural net per scene is not practical, and also the proposed extension is not trivial: the paper brings novel ideas (using attention for time-wise and camera-wise aggregation). Experimental evaluation is comprehensive and the quality is even superior to per-scene optimised Neural Bodies. Given this, I suggest to accept the paper, although there is some room for improvement.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty.\n\n(+) The paper significantly improves over Neural Body [31] (which was formally published after NeurIPS submission deadline). To be able to generalise to varying scenes, it conditions on the images filmed by different cameras and/or at different timestamps. This idea is exploited in recent pre-prints but for generic object categories (PixelNerf, PVA, WCE); this paper for the first time 1) exploits parametric body model to make it applicable to an articulated category of humans, 2) proposes to use transformers for aggregation across time and cameras instead of simpler average pooling. This is novel even w.r.t. unpublished work.\n\nClarity.\n\n(+) The paper is well written, it mostly provides enough details.\n\n(−) The main miss is that it does not describe the procedure of fitting SMPL to videos, which is not easy to do in a time-consistent manner (is it reconciled across cameras and time?). The paper only refers to [7, 31, 9], which is not enough. Also it would be good to clarify early on (before Section 4) that the method uses estimated SMPL parameters as it affects applicability.\n\n(−) The paper can be more straightforward about limitations, see the next section for details.\n\n(−) Skeletal features are sampled from projected SMPL vertices. How does the method deal with occlusions? The paper also says the query skeletal features are obtained by trilinear sampling but does not provide more details. I understand that the function is defined on the SMPL template surface; how exactly are the features extrapolated for the outside volume?\n\n(+) Related work is comprehensive.\n\nSignificance.\n\n(+) Training a neural network per scene is impractical in many applications due to time or storage constraints. Making a generalisable model is a life saver in these cases.\n\n(+) The method can be used both for free-view rendering and reconstruction of clothed humans. As shown, the competing methods produce wrong 3D even if the rendering looks reasonable. The output also looks temporarily smooth in videos, which is also a desirable property.\n\nQuality.\n\n(+) Experimental results are comprehensive. The paper ablates the contributions, including turning off pixel-aligned and skeleton-defined features separately, and it shows that both transformer architectures bring improvement over average pooling.\n\n(=) In the NHR baseline, is it fair to use just SMPL vertices? Will the method improve significantly with higher resolution?\n\n(=) AIST numbers (Table 2b) are significantly lower than ZJU ones; is it still meaningful to compare results with such low PSNR?\n\nNotes to the authors (no effect on the rating):\n* In the time-wise transformer, would it make sense to use positional encoding of time?\n* line 165: the different geometry patterns ← different geometry patterns;\n* line 184: Is the loss indeed just L2 norm (RMSE), or squared (MSE)? \n* line 223: a health margin ← a healthy margin;\n* lines 277, 279: “Table. 4”.\n* in Tables 4 and 5, along with reconstructions, it would be interesting to show the SMPL fits, to see how far the methods deviate.\n\n=====\n\nUPD. I am satisfied with clarifications in the rebuttals, and raise the rating to 8 assuming they will be included in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In general, limitations section is good. I would suggest several improvements:\n* please be clear upfront about the requirement to have SMPL fits as input; they are not given for granted, while Introduction does not mention this requirement. For other categories like animal species such parametric models are not available, so it limits the applicability of the method;\n* the problem statement requires capture from static cameras only (extrinsics are not time-dependent): please make it clear and discuss whether it would be difficult to extend the method to support capture from e.g. one or several phones or AR devices;\n* similar for prediction-time: the experiment al protocol seems to fix cameras: is it a big limitation? The supplementary videos seem to show free-view trajectories; could you comment how well it generalises? E.g. what if the cameras are sampled outside of a hemisphere manifold?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "UCD9DktOetS", "reviewer": "Reviewer_ZRhs", "summary": "This work proposes a method that can synthesize free-viewpoint videos for dynamic humans in a sparse multi-camera system.  As is claimed and demonstrated in the paper, by introducing transformer modules to aggregate spatio-temporal information in the sparse multi-view data, the learned neural radiance fields are generalizable to unseen human identities and unseen poses. \nThe proposed method is tested on public multi-camera datasets, showing promising results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this work is good. Although the generalizability to unseen poses has also been achieved in a few concurrent works, the unseen identity generalizability makes the key difference to those per-scene optimization methods.\n\nSome questions and comments,\n1. it seems like that it is pretty straightforward to incorporate the Transformer for learning the generalizable radiance fields. So I am afraid that the technical contribution in this paper may not be very significant.\n\n2. the exposition of the method section needs more effort to be improved. \n     I would appreciate it much if the authors could make the method section much neater. The method section is full of tons of symbols, of which some may not be necessary, e.g., eq. 4 in Lin 178.\n     Fig. 1 failed to do its job, I think the illustration of the method is supposed to be self-contained, it would be also much better to elaborate in the caption, instead of referring the readers back to each specific section.\n\n3. regarding the details of the SMPL estimate. How many cameras are used for estimating the SMPL models of both ZJU-MoCap and AIST data? In Line 192, it is written that three methods are used to obtain these estimates, this confuses me. Also, the AIST dataset contains corresponding 3D keypoints and SMPL parameters (as described in the supplementary), are these data being used when running on AIST?\n\n4. the paper did not show the failure cases. Is it true that the Transformers can always exploit the information in the memory bank to produce good imagery? What if the motions in the training data differ significantly from those in test data so that the memory bank fails. It is necessary to evaluate the transferability of the method in such a scenario, e.g., train on ZJU-MoCap, test on AIST, or vice-versa.\n\n5. In Line 180, the samples are bounded within the bounding box of the SMPL mesh. But the SMPL mesh can be inaccurate, which means the bounding box derived from it can be inaccurate as well, e.g., the box may not be sufficiently large to enclose the GT shape.\n\nOverall, this submission is good to me, and I hope the issues could be addressed in the rebuttal to make it a more solid submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Around 36 hours over several days"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 positive final ratings: 8, 7, 7, 8.
The reviewers appreciated overall novelty of the approach, its generalization properties and strong empirical performance. The remaining concerns were mostly around clarity, justification of individual components and lack of analysis of failure cases. These were mostly addressed in the rebuttal, as acknowledged by the reviewers. The final recommendation is therefore to accept as a spotlight.
Decision: Accept (Spotlight)