Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

1 Introduction . Free-viewpoint video of a human performer has a variety of applications in the area of telepresence , mixed reality , gaming and etc . Conventional free-viewpoint video systems require extremely expensive setups such as dense camera rigs [ 4 , 7 , 43 ] or accurate depth sensors [ 6 , 13 ] , to capture person-specific appearance information . In this paper , we aim at a scalable solution for free-viewpoint human performance rendering that generalizes across different human performers and requires only sparse camera views . However , representing and rendering arbitrary human performances is extremely challenging when the observations are highly sparse ( up to three to four views ) due to heavy self-occlusions and dynamic articulations of the body parts . In particular , an effective solution needs to coherently aggregate appearance information from sparse multi-view observations across time as the body undergoes a 3D motion . Furthermore , the solution needs to generalize to unseen motions and characters at test time . Recently , neural radiance fields ( NeRF ) [ 26 , 11 , 17 , 30 , 33 , 35 , 36 , 47 , 50 , 52 , 53 ] have shown photo-realistic novel view synthesis results in per-scene optimization settings . To avoid the expensive per-scene training and improve the practicality , generalizable NeRFs [ 36 , 52 , 47 ] have been proposed which use image-conditioned , pixel-aligned features and achieve feed-forward view synthesis from sparse input views [ 36 , 52 ] . Direct application of these methods to complex and non-rigid human motion is not straightforward , however , and naive solutions suffer from significant artifacts as shown in Fig . 3 . Some existing methods [ 37 , 52 ] aggregate image features across multiple views by simple average pooling , which often leads to over-smoothed outputs when details observed from multiple 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . views ( e.g. , front and side views ) differ due to self occlusions of humans . Alternatively , several methods [ 23 , 33 ] have proposed to learn person-specific global appearance features from multi-view observations . However , such methods are not able to generalize to new human performers . To address these challenges , we propose Neural Human Performer , a novel approach that learns generalizable radiance fields based on a parametric 3D body model for robust performance capture . In addition to exploiting a parametric body model as a geometric prior , the core of our method is a combination of temporal and multi-view transformers which help to effectively aggregate spatio-temporal observations to robustly compute the density and color of a query point . First , the temporal transformer aggregates trackable visual features based on the input skeletal body motion across time . The following multi-view transformer performs cross-attention between the temporallyaugmented skeletal features and the pixel-aligned features for each time step . The proposed modules collectively contribute to the adaptive aggregation of multi-time and multi-view information , resulting in significant improvements in synthesis results in different generalization settings of unseen motions and identities . We study the efficacy of Neural Human Performer on two multi-view human performance capture datasets , ZJU-MoCap [ 33 ] and AIST [ 16 ] . Experiments show that our method significantly outperforms recent generalizable radience field ( NeRF ) methods . Furthermore , we compare ours with identity-specific methods [ 33 , 44 , 49 ] that also utilize a 3D human body model prior . Surprisingly , our generalized method achieves better rendering quality than the person-specific dedicated methods when tested on novel poses demonstrating the effectiveness of our transformer-based generalizable representation . To summarize , our contributions are : • We present a new feed-forward method of synthesizing novel-view videos of arbitrary human performers from sparse camera views . We propose Neural Human Performer that learns generalizable neural radiance representations by leveraging a 3D body motion prior . • We design a combination of temporal and multi-view transformers that can aggregate information on the fly over video frames across multiple views to render each frame of the novel-view video . • We show significant improvements over recent generalizable NeRF methods on unseen identities and poses . Moreover , our generalization results can outperform even personspecific methods when tested on unseen poses . 2 Related works . Human performance capture . Novel view synthesis of human performance has a long history in computer vision and graphics . Traditional methods rely on complicated hardware such as dense camera rigs [ 4 , 7 , 43 ] or accurate depth sensors [ 6 , 13 ] . To enable free-view video from sparse camera views , template-based methods [ 3 , 5 , 10 , 42 ] exploit pre-scanned human models to track the motion of a person . However , their synthesis results are not photo-realistic and pre-scanned human models are not available in most cases . Recent methods [ 27 , 37 , 38 , 54 ] learn 3D human geometry priors along with pixel aligned features to enable detailed 3D human reconstructions even from single images . However , these methods often suffer under complex human poses that are never seen during training and hence can not be directly used for our purpose of human performance synthesis . Neural 3D representations . Recently , there has been great progress in learning neural networks to represent the shape and appearance of scenes . The 3D representations are learned from 2D images via differentiable rendering networks . Convolutional neural networks are used to predict volumetric representations via 3D voxel-grid features [ 40 , 23 , 29 , 25 , 14 , 15 ] , point clouds [ 1 , 49 ] , textured meshes [ 18 , 21 , 44 ] and multi-plane images [ 8 , 55 ] . The learnt representations are projected by a 3D-to-2D operation to synthesize images . Some methods [ 14 , 15 ] introduce consistency loss to improve the multi-view and temporal consistency . However , these methods often have difficulty in scaling to higher resolution due to memory restrictions . To eschew these problems , implicit function-based methods [ 20 , 22 , 28 , 41 ] learn a multi-layer perceptron that directly translates a 3D positional feature into a pixel generator . The more recent NeRF [ 26 ] learns implicit fields of density and color with a volume rendering technique and achieves photo-realistic view synthesis . Among many following NeRF extensions , [ 30 , 31 , 35 , 53 , 17 , 50 ] focus on dynamic scenes . While showing impressive results , it ’ s an extremely under-constrained problem to jointly learn NeRF and highly dynamic deformation fields . To regularize the training , Neural Body [ 33 ] combines NeRF with a deformable human body model ( e.g. , SMPL [ 24 ] ) . Despite the promising results , these general deformable NeRF [ 17 , 53 ] and human-specific NeRF [ 11 , 9 , 33 , 32 ] methods must be optimized for each new video separately , and generalize poorly on unseen scenarios . Generalizable NeRFs [ 36 , 47 , 52 ] try to avoid the expensive per-scene optimization by image-conditioning using pixel-aligned features . However , directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse . Unlike existing works , our method exploits temporal and multi-view information on-the-fly and achieves free-viewpoint human rendering in a feed-forward manner , also generalizing to new , unseen human identities and poses . 3 Method . 3.1 Neural Human Performer . Problem definition . In our setting , given a sparse set ( e.g. , 3 or 4 ) of multi-view cameras c = 1 , . . . , C , input videos of an arbitrary human performance Ic,1 : T : = { Ic,1 , Ic,2 , ... , Ic , T } are captured for each camera view c defined by { Kc , [ R|t ] c } . We assume that the 3D human body model fit corresponding to each frame is given . Our goal is to synthesize a novel view video Îq,1 : T for a query viewpoint q defined by { Kq , [ R|t ] q } . Overview . To compensate for the sparsity of available input views , we propose to exploit temporal information across video frames . In practice , we sample M memory frames from the original input videos to augment each queried timestep t. Our goal is to learn generalizable 3D representations of human performers from multi-time ( M ) and multi-view ( C ) observations . To this end , the Neural Human Performer is proposed with two main components as illustrated in Fig . 1 . First , we construct the time-augmented skeletal features { s } . We exploit a human body model ( SMPL [ 24 ] ) to construct 3D skeletal features by projecting all the SMPL vertices onto each memory frame and picking up the pixel-aligned image features [ 36 , 37 , 52 ] at the projected 2D locations . The skeletal features are sampled from all memory frames to construct the skeletal feature bank . Inspired by Transformers [ 2 , 46 , 48 ] , we propose a temporal Transformer that aggregates these memory features into the time-augmented skeletal features { s′ } . In the second stage , given a query 3D point x , skeletal features are sampled at x . In addition , pixel-aligned features { p } at each time t are sampled by directly projecting x onto the input images { I } t. The multi-view Transformer is proposed to learn the correlation between the pixel-aligned features { p } and the time-augmented skeletal features { s′ } , and to adaptively fuse the multi-view information . The final fused feature of the query point x is fed into the radiance field module to predict its color and density . 3.2 Construction of time-augmented skeletal features . Unlike static scenes , video inputs of moving characters inherently contain more visual cues as the occluded regions in a frame may be visible in other ( potentially distant ) frames . To take advantage of this temporal information , we first build up the skeletal feature bank from memory frames by leveraging a parametric body model ( see Fig . 1 ) . Then , we propose a temporal Transformer module that aggregates such collected features . For each view c and time t , we first build frame-level skeletal features sc , t ∈ RL×d by sampling image features at the projection of the SMPL vertices in each image Ic , t ∈ RH×W×3 . L denotes the number of SMPL vertices and d is the dimension of image features . After collecting all the skeletal features from all memory frames , we aggregate the time information in an attention-aware manner , instead of using simple average pooling . Specifically , for any ith skeletal feature vertex sic , t ∈ Rd , the proposed temporal Transformer casts attention over all other features contained in the vertex ’ s memory bank sic , t1 : tM = { s i c , t1 , s i c , t2 , ... , s i c , tM } ∈ R M×d . In particular , we compute soft weights for all memory feature vertices in a non-local manner with respect to the current timestep t. Then , the values of the memory features are weighted summed as t_attic , t = ψ ( 1√ d0 q ( sic , t ) · k ( sic , t1 : tM ) T ) , t_attic , t ∈ R1×M s′c , t i = t_attic , t · v ( sic , t1 : tM ) + s i c , t , s ′ c , t ∈ RL×d ∀i ( 1 ) where ψ represents the softmax operator along the second axis , q ( · ) , k ( · ) and v ( · ) are learnable query , key and value embedding functions Rd→d0 of the temporal Transformer . In other words , the representation sic , t of each skeletal vertex at time t is computed through a dynamically weighted combination of all its previous and next representations in the memory frames . This allows our network to incorporate helpful information and ignore irrelevant ones from other timesteps . In practice , the temporal Transformer operation in Eq . ( 1 ) is performed by a batch matrix multiplication for all skeletal vertices L and all available viewpoints C .

Summary: This paper presents a method to synthesize a novel view image of a person given the sparse multiview images. The paper proposes time-augmented skeletal features---the pixel-aligned features that encode NeRF-like rendering fields by aggregating image features across  time and views, parametrized by a human mesh model. They use transformer networks to learn fusing multiview image features over time. The resulting features are used to generate an implicit function that can predict RGB and opacity. They demonstrate that the proposed method is generalizable to unseen identities and poses, validated on ZJU-MoCap and AIST datasets.