Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

1 Introduction . Free-viewpoint video of a human performer has a variety of applications in the area of telepresence , mixed reality , gaming and etc . Conventional free-viewpoint video systems require extremely expensive setups such as dense camera rigs [ 4 , 7 , 43 ] or accurate depth sensors [ 6 , 13 ] , to capture person-specific appearance information . In this paper , we aim at a scalable solution for free-viewpoint human performance rendering that generalizes across different human performers and requires only sparse camera views . However , representing and rendering arbitrary human performances is extremely challenging when the observations are highly sparse ( up to three to four views ) due to heavy self-occlusions and dynamic articulations of the body parts . In particular , an effective solution needs to coherently aggregate appearance information from sparse multi-view observations across time as the body undergoes a 3D motion . Furthermore , the solution needs to generalize to unseen motions and characters at test time . Recently , neural radiance fields ( NeRF ) [ 26 , 11 , 17 , 30 , 33 , 35 , 36 , 47 , 50 , 52 , 53 ] have shown photo-realistic novel view synthesis results in per-scene optimization settings . To avoid the expensive per-scene training and improve the practicality , generalizable NeRFs [ 36 , 52 , 47 ] have been proposed which use image-conditioned , pixel-aligned features and achieve feed-forward view synthesis from sparse input views [ 36 , 52 ] . Direct application of these methods to complex and non-rigid human motion is not straightforward , however , and naive solutions suffer from significant artifacts as shown in Fig . 3 . Some existing methods [ 37 , 52 ] aggregate image features across multiple views by simple average pooling , which often leads to over-smoothed outputs when details observed from multiple 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . views ( e.g. , front and side views ) differ due to self occlusions of humans . Alternatively , several methods [ 23 , 33 ] have proposed to learn person-specific global appearance features from multi-view observations . However , such methods are not able to generalize to new human performers . To address these challenges , we propose Neural Human Performer , a novel approach that learns generalizable radiance fields based on a parametric 3D body model for robust performance capture . In addition to exploiting a parametric body model as a geometric prior , the core of our method is a combination of temporal and multi-view transformers which help to effectively aggregate spatio-temporal observations to robustly compute the density and color of a query point . First , the temporal transformer aggregates trackable visual features based on the input skeletal body motion across time . The following multi-view transformer performs cross-attention between the temporallyaugmented skeletal features and the pixel-aligned features for each time step . The proposed modules collectively contribute to the adaptive aggregation of multi-time and multi-view information , resulting in significant improvements in synthesis results in different generalization settings of unseen motions and identities . We study the efficacy of Neural Human Performer on two multi-view human performance capture datasets , ZJU-MoCap [ 33 ] and AIST [ 16 ] . Experiments show that our method significantly outperforms recent generalizable radience field ( NeRF ) methods . Furthermore , we compare ours with identity-specific methods [ 33 , 44 , 49 ] that also utilize a 3D human body model prior . Surprisingly , our generalized method achieves better rendering quality than the person-specific dedicated methods when tested on novel poses demonstrating the effectiveness of our transformer-based generalizable representation . To summarize , our contributions are : • We present a new feed-forward method of synthesizing novel-view videos of arbitrary human performers from sparse camera views . We propose Neural Human Performer that learns generalizable neural radiance representations by leveraging a 3D body motion prior . • We design a combination of temporal and multi-view transformers that can aggregate information on the fly over video frames across multiple views to render each frame of the novel-view video . • We show significant improvements over recent generalizable NeRF methods on unseen identities and poses . Moreover , our generalization results can outperform even personspecific methods when tested on unseen poses . 2 Related works . Human performance capture . Novel view synthesis of human performance has a long history in computer vision and graphics . Traditional methods rely on complicated hardware such as dense camera rigs [ 4 , 7 , 43 ] or accurate depth sensors [ 6 , 13 ] . To enable free-view video from sparse camera views , template-based methods [ 3 , 5 , 10 , 42 ] exploit pre-scanned human models to track the motion of a person . However , their synthesis results are not photo-realistic and pre-scanned human models are not available in most cases . Recent methods [ 27 , 37 , 38 , 54 ] learn 3D human geometry priors along with pixel aligned features to enable detailed 3D human reconstructions even from single images . However , these methods often suffer under complex human poses that are never seen during training and hence can not be directly used for our purpose of human performance synthesis . Neural 3D representations . Recently , there has been great progress in learning neural networks to represent the shape and appearance of scenes . The 3D representations are learned from 2D images via differentiable rendering networks . Convolutional neural networks are used to predict volumetric representations via 3D voxel-grid features [ 40 , 23 , 29 , 25 , 14 , 15 ] , point clouds [ 1 , 49 ] , textured meshes [ 18 , 21 , 44 ] and multi-plane images [ 8 , 55 ] . The learnt representations are projected by a 3D-to-2D operation to synthesize images . Some methods [ 14 , 15 ] introduce consistency loss to improve the multi-view and temporal consistency . However , these methods often have difficulty in scaling to higher resolution due to memory restrictions . To eschew these problems , implicit function-based methods [ 20 , 22 , 28 , 41 ] learn a multi-layer perceptron that directly translates a 3D positional feature into a pixel generator . The more recent NeRF [ 26 ] learns implicit fields of density and color with a volume rendering technique and achieves photo-realistic view synthesis . Among many following NeRF extensions , [ 30 , 31 , 35 , 53 , 17 , 50 ] focus on dynamic scenes . While showing impressive results , it ’ s an extremely under-constrained problem to jointly learn NeRF and highly dynamic deformation fields . To regularize the training , Neural Body [ 33 ] combines NeRF with a deformable human body model ( e.g. , SMPL [ 24 ] ) . Despite the promising results , these general deformable NeRF [ 17 , 53 ] and human-specific NeRF [ 11 , 9 , 33 , 32 ] methods must be optimized for each new video separately , and generalize poorly on unseen scenarios . Generalizable NeRFs [ 36 , 47 , 52 ] try to avoid the expensive per-scene optimization by image-conditioning using pixel-aligned features . However , directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse . Unlike existing works , our method exploits temporal and multi-view information on-the-fly and achieves free-viewpoint human rendering in a feed-forward manner , also generalizing to new , unseen human identities and poses . 3 Method . 3.1 Neural Human Performer . Problem definition . In our setting , given a sparse set ( e.g. , 3 or 4 ) of multi-view cameras c = 1 , . . . , C , input videos of an arbitrary human performance Ic,1 : T : = { Ic,1 , Ic,2 , ... , Ic , T } are captured for each camera view c defined by { Kc , [ R|t ] c } . We assume that the 3D human body model fit corresponding to each frame is given . Our goal is to synthesize a novel view video Îq,1 : T for a query viewpoint q defined by { Kq , [ R|t ] q } . Overview . To compensate for the sparsity of available input views , we propose to exploit temporal information across video frames . In practice , we sample M memory frames from the original input videos to augment each queried timestep t. Our goal is to learn generalizable 3D representations of human performers from multi-time ( M ) and multi-view ( C ) observations . To this end , the Neural Human Performer is proposed with two main components as illustrated in Fig . 1 . First , we construct the time-augmented skeletal features { s } . We exploit a human body model ( SMPL [ 24 ] ) to construct 3D skeletal features by projecting all the SMPL vertices onto each memory frame and picking up the pixel-aligned image features [ 36 , 37 , 52 ] at the projected 2D locations . The skeletal features are sampled from all memory frames to construct the skeletal feature bank . Inspired by Transformers [ 2 , 46 , 48 ] , we propose a temporal Transformer that aggregates these memory features into the time-augmented skeletal features { s′ } . In the second stage , given a query 3D point x , skeletal features are sampled at x . In addition , pixel-aligned features { p } at each time t are sampled by directly projecting x onto the input images { I } t. The multi-view Transformer is proposed to learn the correlation between the pixel-aligned features { p } and the time-augmented skeletal features { s′ } , and to adaptively fuse the multi-view information . The final fused feature of the query point x is fed into the radiance field module to predict its color and density . 3.2 Construction of time-augmented skeletal features . Unlike static scenes , video inputs of moving characters inherently contain more visual cues as the occluded regions in a frame may be visible in other ( potentially distant ) frames . To take advantage of this temporal information , we first build up the skeletal feature bank from memory frames by leveraging a parametric body model ( see Fig . 1 ) . Then , we propose a temporal Transformer module that aggregates such collected features . For each view c and time t , we first build frame-level skeletal features sc , t ∈ RL×d by sampling image features at the projection of the SMPL vertices in each image Ic , t ∈ RH×W×3 . L denotes the number of SMPL vertices and d is the dimension of image features . After collecting all the skeletal features from all memory frames , we aggregate the time information in an attention-aware manner , instead of using simple average pooling . Specifically , for any ith skeletal feature vertex sic , t ∈ Rd , the proposed temporal Transformer casts attention over all other features contained in the vertex ’ s memory bank sic , t1 : tM = { s i c , t1 , s i c , t2 , ... , s i c , tM } ∈ R M×d . In particular , we compute soft weights for all memory feature vertices in a non-local manner with respect to the current timestep t. Then , the values of the memory features are weighted summed as t_attic , t = ψ ( 1√ d0 q ( sic , t ) · k ( sic , t1 : tM ) T ) , t_attic , t ∈ R1×M s′c , t i = t_attic , t · v ( sic , t1 : tM ) + s i c , t , s ′ c , t ∈ RL×d ∀i ( 1 ) where ψ represents the softmax operator along the second axis , q ( · ) , k ( · ) and v ( · ) are learnable query , key and value embedding functions Rd→d0 of the temporal Transformer . In other words , the representation sic , t of each skeletal vertex at time t is computed through a dynamically weighted combination of all its previous and next representations in the memory frames . This allows our network to incorporate helpful information and ignore irrelevant ones from other timesteps . In practice , the temporal Transformer operation in Eq . ( 1 ) is performed by a batch matrix multiplication for all skeletal vertices L and all available viewpoints C .

Summary: This paper presents a generalizable NeRF network that can produce novel view synthesis for novel dynamic human scenes from sparse camera views, in a feed-forward manner. It leverages visual features from tracked parametric body model and similar to PixelNeRF, it conditions the NeRF representation on pixel-aligned image features for appearance generalization. A novel combination of a temporal Transformer and a multiview Transformer is proposed to integrate multi-frame and multi-view visual observations. It demonstrated stronger generalization capability than prior work.