ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

1 Introduction . Reconstructing the world from a sequence of monocular frames is a long-standing task in computer vision . While there has been tremendous progress in reconstructing rigid scenes ( via SfM and SLAM [ 7 , 39 , 43 ] , or recent techniques based on neural rendering [ 28 ] ) , reconstructing dynamic scenes with articulated objects remains elusive . For example , given a monocular video , it is still challenging to reconstruct an everyday scene of a moving person with loose clothing . In this work , we tackle the problem of estimating the deforming mesh of articulated objects given a segmented monocular video of that object . Our method avoids the use of any mesh templates or category-specific priors and generalizes to unknown deformable articulated objects in the wild . Nonrigid shape recovery is highly under-constrained due to fundamental ambiguities between shape , appearance , and time-varying deformation . Current approaches for addressing these challenges fall into two camps : better data “ likelihoods ” or better “ priors ” . The first camp extracts richer sensor data , via multi-camera studio setups [ 15 ] or depth sensors [ 30 ] , but requires substantial efforts to work in the wild . The second camp makes use of category-level priors over object shapes [ 18 , 20 ] and is particularly effective for human reconstruction . However , building such models requires considerable offline efforts in the form of registered 3D scans [ 26 ] or manual keypoint annotations [ 12 ] , both of which are difficult to scale to arbitrary object categories . ∗Work done at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we use a practical but less explored variant of the data-likelihood camp : we use multiple frames of a video rather than multiple cameras or depth sensors . This considerably complicates analysis for dynamic , non-rigid scenes . Nonrigid structure-from-motion ( NRSfM ) [ 4 , 38 ] attempts to constrain the problem by relying on motion correspondences such as 2D point tracks . While 2D correspondences over short time scales ( i.e. , optical flow ) are relatively robust to extract , correspondences over long time scales are notoriously difficult to estimate because of appearance variations arising from viewpoint changes , occlusion and fast motion . In practice , this limits the applicability of NRSfM methods to controlled lab sequences . We propose ViSER ( Video-Specific Surface Embeddings for Reconstruction ) , which establishes long-range correspondence and reconstructs articulated 3D shapes from a monocular video . Fig . 1 shows a sample outdoor video and the corresponding ViSER results . The key insight behind ViSER is to force long-range video pixel correspondences to be consistent with an underlying canonical 3D mesh through the use of video-specific embeddings that capture the pixel appearance of each surface point . These embeddings behave as a continuous set of keypoint descriptors defined over the surface mesh , learned with coordinate-based MLPs that are fit to each video via self-supervised losses . ViSER simultaneously optimizes the image CNN , surface MLP , and 3D shape so as to fit the observed video frames . It reconstructs state-of-the-art articulated 3D shape and 3D trajectories without using category-specific priors , making it easily scalable to diverse videos including humans with challenging clothing and poses as well as animals . Lastly , we demonstrate that ViSER recovers meaningful part segmentation and blend skinning weights from videos , which typically require considerable manual effort from 3D artists . 2 Related Work . Low-level correspondence . Optical flow is a well-studied representation for short-term correspondence between adjacent frames of a video . After decades of research , recent CNN models [ 40 , 42 , 48 ] for optical flow have achieved an impressive level of accuracy as evidenced by the Sintel and KITTI benchmarks [ 5 , 9 ] . However , it is challenging to concatenate optical flow for reliable long-range correspondence due to occlusions and strong appearance changes [ 33 , 37 , 41 ] . ViSER does not concatenate optical flow but use it as a constraint to establishes long-range correspondence . The layered approach [ 6 , 14 , 45 ] segments a video into different moving objects with coherent motion , thereby establishing long-range correspondence for every frames through the shared layers . Early layered methods assume parameter motion for each layer and can only handle limited scenes . Unwrap Mosiacs [ 32 ] uses a dense 2D-to-2D mapping from a texture map to every input frame , and editing operations on the texture map naturally transfers to each individual frame . However , the 2D representation can not flexibly model complex 3D phenomena , such as occlusions . Dense pose and surface mappings . DensePose [ 12 ] directly maps pixels to the 3D surface of a human body model . It requires large amounts of training data with annotated image-to-surface correspondence and is hard to generalize to other categories . Articulation-aware Canonical Surface Mapping ( A-CSM ) [ 20 ] uses geometric cycle consistency for learning to map pixels to corresponding points on a template shape without using keypoint annotations . However , it requires a pre-defined template shape for each category . Continuous Surface Embeddings ( CSE ) [ 29 ] establishes dense correspondences between image pixels and 3D object geometry by predicting an embedding vector of the corresponding vertex in the object mesh for each pixel in a 2D image . While applicable to multiple categories , CSE requires annotations and only applies to categories in the training set . ViSER requires neither a template shape nor annotations to work on categories in the wild . Nonrigid shape reconstruction . One way to accurately reconstruct articulated shapes is to rely on rich sensor data , e.g. , multi-view [ 15 ] or depth sensors [ 30 ] , which requires substantial efforts to setup and reconstruct objects in the wild . For monocular videos/images , one popular approach is to adopt strong 3D shape and pose priors [ 18 , 26 , 35 , 36 , 53 , 54 ] but it works well only on limited categories , whose 3D data are easy to collect . To deal with more nonrigid object categories , a recent trend is to learn a category-level 3D shape model from a collection of images or videos with 2D annotations , such as keypoints and object silhouettes [ 10 , 16 , 20 , 22 , 23 , 44 , 46 , 50 ] . Although they are able to reconstruct more object categories , such as birds and quadruped animals , the reconstruction usually lacks details , and the level of deformation recovered tends to be low . Category-agnostic methods , such as nonrigid structure from motion ( NRSfM ) methods [ 4 , 11 , 19 , 38 ] reconstruct nonrigid 3D shapes from a set of 2D point trajectories . However , due to the difficulty in obtaining accurate long-range correspondences [ 37 , 41 ] they do not work well for videos in the wild . A recent work , LASR [ 49 ] , uses two-frame optical flow to reconstruct articulate shapes from a monocular video with differentiable rendering . Despite the promising results , LASR does not reason about long-range correspondences and can only reliably reconstruct what is visible in a short video . ViSER establishes reliable long-range correspondence that are robust to moderate shape variations and appearance changes . Thus , ViSER can obtain much higher-quality reconstruction by using either a long video or several videos of a category . 3 Approach . Fig . 2 provides an overview of our approach , which follows a typical framework of differentiable rendering [ 16 , 25 ] . Borrowing the notation from LASR [ 49 ] , we formalize our task as follows . Given a set of video observations including RGB pixel color , segmentation masks , and optical flow estimates { It , St , ut } t= { 0 , ... , T } , our goal is to recover a set of shape and motion parameters { S , Dt } that produce reconstructions { Ît , Ŝt , ût } t= { 0 , ... , T } that match the video observations . We refer to supplementary material for a complete list of notations defined in the paper . 3.1 Preliminaries . We represent an object ’ s shape as a triangular mesh S = { V̄ , F } with canonical vertices V̄ ∈ R3×N and a fixed topology ( edge connectivity ) F ∈ R3×M . To render an object , we displace mesh vertices with motion parameters Dt , apply a perspective projection with camera intrinsics Kt , and rasterize . We model vertex motion with root body transformations G0 and object articulations { G1 , · · · , GB } using linear blend skinning ( LBS ) [ 20 , 21 ] . LBS constrains vertex motion by linearly blending B rigid “ bone ” transformations with a skinning weight matrix W ∈ RB×N , transforming the canonical shape into frame t as Vi , t = G0 , t ( ∑ b Wb , iGb , t ) V̄i ( 1 ) where i is the vertex index , b is the bone index . Similar to LASR , the root body and bone transformations are represented as the outputs of a pose CNN given an input image , ( G0 , · · · , GB ) = ψp ( It ) . We define a set of surface properties for rendering , including vertex 3D coordinates , textures and features , and rasterize them in a differentiable manner [ 25 ] . We denote the differentiable rendering function that renders the property C defined on a canonical surface to an image asR ( C ; V , W , G ) , which executes the blending skinning function in Eq . ( 1 ) and softly blends the surface property based on their depth and barycentric coordinates [ 25 ] . For simplicity , we omit the shape , skinning , and motion parameters parameters and write the differentiable rendering function asR ( C ) . To render optical flow , we rasterize and project vertex coordinates in two consecutive frames and compute their 2D displacements [ 49 ] . Such renderings are compared against video observations to compute gradients for updating model parameters . 3.2 Video-specific Surface Embedding . Pixel-surface embeddings . We learn pixel and surface embeddings that map corresponding pixels in different frames to the same point on a canonical 3D surface . Intuitively , consider a particular region on the canonical surface mesh that is the “ nose ” of an articulated human . The surface embedding captures a descriptor for the nose , which can then be matched to pixel-level descriptors at each frame . Given an input image It , the pixel-wise descriptor embedding is computed by a U-Net [ 34 ] encoder : FI [ x , y , t ] = ψe ( It ) [ x , y ] ∈ R16 , ( 2 ) where [ x , y , t ] are pixel locations at frame t. The surface embedding is computed by a positionencoded MLP : FS ( X , Y , Z ) = φe ( X , Y , Z ) ∈ R16 , ( 3 ) where φe ( · ) is an MLP defined over 3D points ( X , Y , Z ) in the canonical space , augmented with Fourier positional encoding [ 28 ] . The two embeddings are optimized on test videos such that pixels representing the same surface location in different frames are mapped to the same canonical surface point [ 20 ] . Correspondence via soft-argmax regression . Given the pixel and surface embeddings , we construct a per-frame cost volume D ( FI , FS ) of size H×W×Ns over pixels and surface points ( we randomly sample Ns = 200 surface points at each step ) by considering their cosine feature distances , D ( FI , FS ) [ x , y , i ] = 1− cos ( FI [ x , y ] , FS ( Xi , Yi , Zi ) ) . ( 4 ) Normalizing the cost volume over the surface point dimension yields a softmax “ heatmap ” over surface points that potentially match to pixel ( x , y ) , as shown in Fig . 3 ( Left ) : σ ( x , y ) [ i ] = e−D [ x , y , i ] /τ∑ j e −D [ x , y , j ] /τ , ( 5 ) where τ is a temperature scaling parameter that is jointly optimized with the feature embeddings . To output a single surface point for pixel ( x , y ) , we can compute a “ soft ” argmax [ 17 , 48 ] by taking the expectation of the softmax distribution over the 3D locations of the points samples , Ŝ [ x , y ] = ∑ i σ ( x , y ) [ i ] ( Xi , Yi , Zi ) , ( 6 ) where ( Xi , Yi , Zi ) is the i-th sampled surface point and σ ( x , y ) [ i ] is the matching probability of pixel ( x , y ) over the sampled points i ∈ { 1 , 2 , . . . , Ns } . We can also normalize the H×W×Ns cost volume over spatial positions to capture a distribution of pixel locations that match to each surface point ( Xi , Yi , Zi ) : σ ( Xi , Yi , Zi ) [ x , y ] = e−D [ x , y , i ] /τ∑ [ x , y ] e −D [ x , y , j ] /τ . ( 7 ) and compute a similar soft argmax mapping of surface points to pixels , as shown in Fig . 3 ( Right ) . Relation to keypoints . The output of classic keypoint detectors are often represented as K-channel heatmaps over the pixel grid , where K is the number of keypoints . To define dense keypoints , one may increase the number of channels , which is computationally heavy . Similar to CSE [ 29 ] , we represent dense keypoints as low-dimensional pixel-surface embeddings , which establishes a mapping between pixels and a canonical 3D surface , but far more efficiently . DensePose [ 12 ] and CSM [ 20 ] use an alternative pixel-to-surface mapping that regresses a surface coordinate at every pixel . In contrast , our pixel-surface embedding captures multimodal uncertainties over keypoints ; for example , σ ( x , y ) [ i ] can capture the fact that a particular pixel matches well to both the left and right ankle , as visualized in Fig . 3 , while a regressor may “ regress ” to the mean of the two surface coordinates .

Summary: This paper presents an optimization-based framework that recovers 3D shape, articulated pose, and weak texture from a monocular video of an articulated object. The proposed method extends a recent method (LASR) with a surface embedding matching module for establishing long-range correspondences, and achieves significantly better results on highly deformable objects, such as break dancers.  This is an extremely challenging task, and has a long history in computer vision. Current methods either require extra information, eg depth or keypoints, or rely on category-specific template shape models. The results achieved by this method is very impressive, relying only on off-the-shelf category-agnostic segmentation and optical flow models.  The paper also demonstrates the possibility of extending the method with different instances in multiple videos, which presents exciting potential of capturing category-level priors for fast inference.