ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

1 Introduction . Reconstructing the world from a sequence of monocular frames is a long-standing task in computer vision . While there has been tremendous progress in reconstructing rigid scenes ( via SfM and SLAM [ 7 , 39 , 43 ] , or recent techniques based on neural rendering [ 28 ] ) , reconstructing dynamic scenes with articulated objects remains elusive . For example , given a monocular video , it is still challenging to reconstruct an everyday scene of a moving person with loose clothing . In this work , we tackle the problem of estimating the deforming mesh of articulated objects given a segmented monocular video of that object . Our method avoids the use of any mesh templates or category-specific priors and generalizes to unknown deformable articulated objects in the wild . Nonrigid shape recovery is highly under-constrained due to fundamental ambiguities between shape , appearance , and time-varying deformation . Current approaches for addressing these challenges fall into two camps : better data “ likelihoods ” or better “ priors ” . The first camp extracts richer sensor data , via multi-camera studio setups [ 15 ] or depth sensors [ 30 ] , but requires substantial efforts to work in the wild . The second camp makes use of category-level priors over object shapes [ 18 , 20 ] and is particularly effective for human reconstruction . However , building such models requires considerable offline efforts in the form of registered 3D scans [ 26 ] or manual keypoint annotations [ 12 ] , both of which are difficult to scale to arbitrary object categories . ∗Work done at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we use a practical but less explored variant of the data-likelihood camp : we use multiple frames of a video rather than multiple cameras or depth sensors . This considerably complicates analysis for dynamic , non-rigid scenes . Nonrigid structure-from-motion ( NRSfM ) [ 4 , 38 ] attempts to constrain the problem by relying on motion correspondences such as 2D point tracks . While 2D correspondences over short time scales ( i.e. , optical flow ) are relatively robust to extract , correspondences over long time scales are notoriously difficult to estimate because of appearance variations arising from viewpoint changes , occlusion and fast motion . In practice , this limits the applicability of NRSfM methods to controlled lab sequences . We propose ViSER ( Video-Specific Surface Embeddings for Reconstruction ) , which establishes long-range correspondence and reconstructs articulated 3D shapes from a monocular video . Fig . 1 shows a sample outdoor video and the corresponding ViSER results . The key insight behind ViSER is to force long-range video pixel correspondences to be consistent with an underlying canonical 3D mesh through the use of video-specific embeddings that capture the pixel appearance of each surface point . These embeddings behave as a continuous set of keypoint descriptors defined over the surface mesh , learned with coordinate-based MLPs that are fit to each video via self-supervised losses . ViSER simultaneously optimizes the image CNN , surface MLP , and 3D shape so as to fit the observed video frames . It reconstructs state-of-the-art articulated 3D shape and 3D trajectories without using category-specific priors , making it easily scalable to diverse videos including humans with challenging clothing and poses as well as animals . Lastly , we demonstrate that ViSER recovers meaningful part segmentation and blend skinning weights from videos , which typically require considerable manual effort from 3D artists . 2 Related Work . Low-level correspondence . Optical flow is a well-studied representation for short-term correspondence between adjacent frames of a video . After decades of research , recent CNN models [ 40 , 42 , 48 ] for optical flow have achieved an impressive level of accuracy as evidenced by the Sintel and KITTI benchmarks [ 5 , 9 ] . However , it is challenging to concatenate optical flow for reliable long-range correspondence due to occlusions and strong appearance changes [ 33 , 37 , 41 ] . ViSER does not concatenate optical flow but use it as a constraint to establishes long-range correspondence . The layered approach [ 6 , 14 , 45 ] segments a video into different moving objects with coherent motion , thereby establishing long-range correspondence for every frames through the shared layers . Early layered methods assume parameter motion for each layer and can only handle limited scenes . Unwrap Mosiacs [ 32 ] uses a dense 2D-to-2D mapping from a texture map to every input frame , and editing operations on the texture map naturally transfers to each individual frame . However , the 2D representation can not flexibly model complex 3D phenomena , such as occlusions . Dense pose and surface mappings . DensePose [ 12 ] directly maps pixels to the 3D surface of a human body model . It requires large amounts of training data with annotated image-to-surface correspondence and is hard to generalize to other categories . Articulation-aware Canonical Surface Mapping ( A-CSM ) [ 20 ] uses geometric cycle consistency for learning to map pixels to corresponding points on a template shape without using keypoint annotations . However , it requires a pre-defined template shape for each category . Continuous Surface Embeddings ( CSE ) [ 29 ] establishes dense correspondences between image pixels and 3D object geometry by predicting an embedding vector of the corresponding vertex in the object mesh for each pixel in a 2D image . While applicable to multiple categories , CSE requires annotations and only applies to categories in the training set . ViSER requires neither a template shape nor annotations to work on categories in the wild . Nonrigid shape reconstruction . One way to accurately reconstruct articulated shapes is to rely on rich sensor data , e.g. , multi-view [ 15 ] or depth sensors [ 30 ] , which requires substantial efforts to setup and reconstruct objects in the wild . For monocular videos/images , one popular approach is to adopt strong 3D shape and pose priors [ 18 , 26 , 35 , 36 , 53 , 54 ] but it works well only on limited categories , whose 3D data are easy to collect . To deal with more nonrigid object categories , a recent trend is to learn a category-level 3D shape model from a collection of images or videos with 2D annotations , such as keypoints and object silhouettes [ 10 , 16 , 20 , 22 , 23 , 44 , 46 , 50 ] . Although they are able to reconstruct more object categories , such as birds and quadruped animals , the reconstruction usually lacks details , and the level of deformation recovered tends to be low . Category-agnostic methods , such as nonrigid structure from motion ( NRSfM ) methods [ 4 , 11 , 19 , 38 ] reconstruct nonrigid 3D shapes from a set of 2D point trajectories . However , due to the difficulty in obtaining accurate long-range correspondences [ 37 , 41 ] they do not work well for videos in the wild . A recent work , LASR [ 49 ] , uses two-frame optical flow to reconstruct articulate shapes from a monocular video with differentiable rendering . Despite the promising results , LASR does not reason about long-range correspondences and can only reliably reconstruct what is visible in a short video . ViSER establishes reliable long-range correspondence that are robust to moderate shape variations and appearance changes . Thus , ViSER can obtain much higher-quality reconstruction by using either a long video or several videos of a category . 3 Approach . Fig . 2 provides an overview of our approach , which follows a typical framework of differentiable rendering [ 16 , 25 ] . Borrowing the notation from LASR [ 49 ] , we formalize our task as follows . Given a set of video observations including RGB pixel color , segmentation masks , and optical flow estimates { It , St , ut } t= { 0 , ... , T } , our goal is to recover a set of shape and motion parameters { S , Dt } that produce reconstructions { Ît , Ŝt , ût } t= { 0 , ... , T } that match the video observations . We refer to supplementary material for a complete list of notations defined in the paper . 3.1 Preliminaries . We represent an object ’ s shape as a triangular mesh S = { V̄ , F } with canonical vertices V̄ ∈ R3×N and a fixed topology ( edge connectivity ) F ∈ R3×M . To render an object , we displace mesh vertices with motion parameters Dt , apply a perspective projection with camera intrinsics Kt , and rasterize . We model vertex motion with root body transformations G0 and object articulations { G1 , · · · , GB } using linear blend skinning ( LBS ) [ 20 , 21 ] . LBS constrains vertex motion by linearly blending B rigid “ bone ” transformations with a skinning weight matrix W ∈ RB×N , transforming the canonical shape into frame t as Vi , t = G0 , t ( ∑ b Wb , iGb , t ) V̄i ( 1 ) where i is the vertex index , b is the bone index . Similar to LASR , the root body and bone transformations are represented as the outputs of a pose CNN given an input image , ( G0 , · · · , GB ) = ψp ( It ) . We define a set of surface properties for rendering , including vertex 3D coordinates , textures and features , and rasterize them in a differentiable manner [ 25 ] . We denote the differentiable rendering function that renders the property C defined on a canonical surface to an image asR ( C ; V , W , G ) , which executes the blending skinning function in Eq . ( 1 ) and softly blends the surface property based on their depth and barycentric coordinates [ 25 ] . For simplicity , we omit the shape , skinning , and motion parameters parameters and write the differentiable rendering function asR ( C ) . To render optical flow , we rasterize and project vertex coordinates in two consecutive frames and compute their 2D displacements [ 49 ] . Such renderings are compared against video observations to compute gradients for updating model parameters . 3.2 Video-specific Surface Embedding . Pixel-surface embeddings . We learn pixel and surface embeddings that map corresponding pixels in different frames to the same point on a canonical 3D surface . Intuitively , consider a particular region on the canonical surface mesh that is the “ nose ” of an articulated human . The surface embedding captures a descriptor for the nose , which can then be matched to pixel-level descriptors at each frame . Given an input image It , the pixel-wise descriptor embedding is computed by a U-Net [ 34 ] encoder : FI [ x , y , t ] = ψe ( It ) [ x , y ] ∈ R16 , ( 2 ) where [ x , y , t ] are pixel locations at frame t. The surface embedding is computed by a positionencoded MLP : FS ( X , Y , Z ) = φe ( X , Y , Z ) ∈ R16 , ( 3 ) where φe ( · ) is an MLP defined over 3D points ( X , Y , Z ) in the canonical space , augmented with Fourier positional encoding [ 28 ] . The two embeddings are optimized on test videos such that pixels representing the same surface location in different frames are mapped to the same canonical surface point [ 20 ] . Correspondence via soft-argmax regression . Given the pixel and surface embeddings , we construct a per-frame cost volume D ( FI , FS ) of size H×W×Ns over pixels and surface points ( we randomly sample Ns = 200 surface points at each step ) by considering their cosine feature distances , D ( FI , FS ) [ x , y , i ] = 1− cos ( FI [ x , y ] , FS ( Xi , Yi , Zi ) ) . ( 4 ) Normalizing the cost volume over the surface point dimension yields a softmax “ heatmap ” over surface points that potentially match to pixel ( x , y ) , as shown in Fig . 3 ( Left ) : σ ( x , y ) [ i ] = e−D [ x , y , i ] /τ∑ j e −D [ x , y , j ] /τ , ( 5 ) where τ is a temperature scaling parameter that is jointly optimized with the feature embeddings . To output a single surface point for pixel ( x , y ) , we can compute a “ soft ” argmax [ 17 , 48 ] by taking the expectation of the softmax distribution over the 3D locations of the points samples , Ŝ [ x , y ] = ∑ i σ ( x , y ) [ i ] ( Xi , Yi , Zi ) , ( 6 ) where ( Xi , Yi , Zi ) is the i-th sampled surface point and σ ( x , y ) [ i ] is the matching probability of pixel ( x , y ) over the sampled points i ∈ { 1 , 2 , . . . , Ns } . We can also normalize the H×W×Ns cost volume over spatial positions to capture a distribution of pixel locations that match to each surface point ( Xi , Yi , Zi ) : σ ( Xi , Yi , Zi ) [ x , y ] = e−D [ x , y , i ] /τ∑ [ x , y ] e −D [ x , y , j ] /τ . ( 7 ) and compute a similar soft argmax mapping of surface points to pixels , as shown in Fig . 3 ( Right ) . Relation to keypoints . The output of classic keypoint detectors are often represented as K-channel heatmaps over the pixel grid , where K is the number of keypoints . To define dense keypoints , one may increase the number of channels , which is computationally heavy . Similar to CSE [ 29 ] , we represent dense keypoints as low-dimensional pixel-surface embeddings , which establishes a mapping between pixels and a canonical 3D surface , but far more efficiently . DensePose [ 12 ] and CSM [ 20 ] use an alternative pixel-to-surface mapping that regresses a surface coordinate at every pixel . In contrast , our pixel-surface embedding captures multimodal uncertainties over keypoints ; for example , σ ( x , y ) [ i ] can capture the fact that a particular pixel matches well to both the left and right ankle , as visualized in Fig . 3 , while a regressor may “ regress ” to the mean of the two surface coordinates .

Summary: VISER tackles the problem of 3D shape and pose reconstruction from an input video with masks and flows. The main competitor on this task is the recent LASR approach (CVPR 2021) The main technical novelty in VISER is joint embedding of pixels and surface points learned with a self-supervised contrastive loss. These embeddings match each surface points to all corresponding pixels in the video, thus addressing long-range correspondences in the video. ViSER outperforms the state-of-the-art on human and animal videos and the supplementary videos are very impressive.