ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
Authors: Gengshan Yang
Deqing Sun
Varun Jampani
Daniel Vlasic
Forrester Cole
Ce Liu
Deva Ramanan
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: articulated shape from videos
nonrigid shape reconstruction
dense correspondence
differentiable rendering

Abstract: We introduce ViSER, a method for recovering articulated 3D shapes and dense3D trajectories from monocular videos.  Previous work on high-quality reconstruction of dynamic 3D shapes typically relies on multiple camera views, strong category-specific priors, or 2D keypoint supervision. We show that none of these are required if one can reliably estimate long-range correspondences in a video, making use of only 2D object masks and two-frame optical flow as inputs. ViSER infers correspondences by matching 2D pixels to a canonical,  deformable 3D mesh via video-specific surface embeddings that capture the pixel appearance of each surface point.  These embeddings behave as a continuous set of keypoint descriptors defined over the mesh surface, which can be used to establish dense long-range correspondences across pixels.  The surface embeddings are implemented as coordinate-based MLPs that are fit to each video via self-supervised losses.Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses as well as animals videos from DAVIS and YTVOS. Project page: viser-shape.github.io.

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
Gengshan Yang'
Deqing Sun?
Varun Jampani
Daniel Vlasic
Forrester Cole?
Ce Li
Deva Ramanan
CCameeiee Mellon University
Google Research
Argo AI
Microsoft Azure Al
ViSER-webpage
Abstract
We introduce ViSER, method for recovering articulated 3D shapes and dense 3D trajectories from monocular videos. Previous work on high-quality reconstruction of dynamic 3D shapes typically relies on multiple synchronized cameras, strong category-speciff priors, or 2D keypoint supervision. We show that none of these are required if one can reliably estimate long-range correspondences in video, making use of only 2D object masks and two-frame optical flow inputs. ViSER infers correspondences by matching 2D pixels to a canonical, deformable 3D mesh via video-specift surface embeddings that capture the view-independent appearance features of each surface point. These embeddings behave as a continuous set keypoint descriptors defined over the mesh surface, which can be used to establish dense long-rang correspondences across pixels. The surface embeddings are implemented coordinate- based MLPs that fit to each video via self-super losses. Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses well as animal videos from DAVIS and YTVOS.
Introduction
Reconstructing the world from a sequence of monocular frames is a long-standing task in computer vision. While there has been tremendous progress in reconstructing rigid scenes (via SfM and SLAM
39 43 or recent techniques based on neural rendering (28)). reconstructing dynamic scenes with articulated objects remains elusive. For example, given monocular video, is still challenging to reconstruct an everyday scene of a moving person with loose clothing. In this work, tackle the problem of estimating the deforming mesh of articulated objects given segmented monocular video of that object. Our method avoids the use of any mesh templates or category-specine priors and generalizes unknown deformable articulated objects the wild.
7
Nonrigid shape recovery is highly mnder-constrainet due to fundamental ambiguities between shape, appearance, and varying deformation. Current approaches for addressing these challenges fall into two camps: better data "likelihoods" or hetter "priors". The first camp extracts richer sensor data, via multi-camera studio setups (15) depth sensors [60, but requires substantial efforts work in the wild. The second camp makes use of category-level priors over object shapes [18,20 and is particularly effective for human reconstruction. However, building such models requires considerable offline efforts in the form of registered 3D scans [26) or manual keypoint annotations [12) both of which are difficult to scale arbitrary object categories.
Work done at Google.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(V.F) Verlex tupology
ML.Prear
Embedding
matching
F = CNN()
Shape
Trajectory
Parts
Figure I: Given a long video (or multiple short videos), ViSER jointly learns articulated 3D shapes (represented mesh with vertices V and faces F) and joint pixel-surface embeddings (including surface embedding Fs and pixel embedding F1) that establishes dense long-range pixel correspondences over time. As a result, ViSER produces accurate shapes, long term trajectories and meaningful part segmentation.
In this work, we use practical but less explored variant of the iata-likelihodd camp: we use multiple frames of video rather than multiple cameras depth sensors. This considerably complicates analysis for dynamic, non-rigid scenes. Nonrigid seuuctre-frrmmmotte to constrain the problem by relying motion correspondences such 2D point tracks. While 2D correspondences over short time scales (i.e., optical flow) are relatively robust to extract, correspon- dences over long time scales notoriously difficult estimate because of appearance variations arising from viewpoint changes, occlusion and fast motion. In practice, this limits the applicability of NRSfM methods controlled lab sequences.
(NRSIM) 38] attempts
We propose ViSER (Video-Specific Surface Embeddings for Reconstruction), which establishes longerange correspondence and reconstructs articulated 3D shapes from monocular video. Fig. shows a sample outdoor video and the corresponding ViSER results. The key insight behind ViSER to force long-range video pixel correspondences be consistent with underlying canonical 3D mesh through the use of video-specific embeddings that capture the pixel appearance of each surface point. These embeddings behave continuous set of keypoint descriptors defined over the surface mesh learned with coordinate-basee MLPs that are fit to each video via self-supervised losses. ViSER simultaneously optimizes the image CNN, surface MLP, and 3D shape as fit the observed video frames. reconstructs state-of-the-an articulated 3D shape and 3D trajectories without using category-specific priors, making it easily scalable diverse videos including humans with challenging clothing and poses as well as animals. Lastly, we demonstrate that ViSER recovers meaningful part segmentation and blend skinning weights from videos, which typically require considerable manual effort from 3D artists.
Related Work
Low-level correspondence. Optical flow is well-studied representation for short-term correspon- dence between adjacent frames of video. After decades of research, recent CNN models 40 for optical flow have achieved an impressive level of accuracy as evidenced by the Sintel and KITTI benchmarks [5] 9I. However, it is challenging to concatenate optical flow for reliable long-range correspondence due to occlusions and strong appearance changes 33 37 41 ViSER does not
42 48
concatenate optical flow but use it constraint establishes long-range correspondence.
The layered approach 6 14 45
segments video into different moving objects with coherent motion, thereby establishing long-range correspondence for every frames through the shared layers. Early layered methods assume parameter motion for each layer and can only handle limited scenes. Unwrap Mosiacs editing operations on the texture map naturally transfers each individual frame. However, the 2D representation cannot fiexibly model complex 3D phenomena, such occlusions.
[2 uses dense 2D-to-2D mapping from texture map every input frame, and

---

Embedding Matching S [r, y
Pixel embedding Fi = (b()
Surface embedding Fs - (VV)
L Lmatching
Rendered
surface: R(() R
r( (S| S|,, yl
L reproj
S(ar, v)
Differentiable Rendering R--)
Input image: I,
Rest shape: (V,F)
We learn joint pixel-surfaee embedding space for dense correspondentt between pixels in video frames 1, and points on canonical 3D surface (V, F). Such embedding space is optimized through "top-down" differentiable rendering R(-) and "bottom-up" correspondence matching $ y] (Sec 3.2). We introduce 3D matching loss to optimize the embeddings, where the matched surface locations are encouraged to be close to the rendered surface locations. The embedding further enables articulated shape optimization through 2D-3DD2D cycle reprojection: pixel [ar, vl + matched surface S[x, vl - re-projected pixel = ((((r, v)) (Sec. 3.3). reprogeee
Figure 2:
Dense pose and surface mappings. DensePose 12 directly maps pixels to the 3D surface of human body model. It requires large amounts of training data with annotated image-to-suriee correspondence and is hard to generalize to other categories. Articulation-awal Canonical Surface correspoon
Mapping (A-CSM) [20] uses geometric cycle consistency for learning to map pixels to corresponding points on a template shape without using keypoint annotations. However, it requires a pre-defined
template shape for each category. Continuous Surface Embeddings (CSE) (29) establishes dense correspondences between image pixels and 3D object geometry by predicting an embedding vector of the corresponding vertex in the object mesh for each pixel in a 2D image. While applicable mullipe to multiple categories, CSE requires annotations annotation and only applies to categories in the training set. ViSER requires neither a template shape nor annotations to work on categories in the wild.
Nonrigid shape reconstruction. One way to accurately reconstruct articulated shapes is to rely on rich sensor data, e.g., multi-view [15] depth sensors (30, which requires substantial efforts to setup and reconstruct objects in the wild. For monocular videos/images, one popular approach is to adopt strong 3D shape and pose priors [8 26
53 54 but works well only on limited categories, whose 3D data are easy to collect. To deal with more nonrigid object categories, recent trend is to learn a category-Level 3D shape model from a collection of images or videos with 2D annotations, such keypoints and object silhouettes [10] [6 20 22 23 44 46 50). Although they are able to reconstruct more object categories, such as birds and quadruped animals, the reconstruction usually lacks details, and the level of deformation recovered tends to be low.
35
36
Category-agnostic methods, such as nonrigid structure from motion (NRSIM) methods 4 19 38 reconstruce in obtaining accurate long-range correspondences rreeppoondeee 37 41
reconstruct nonrigid 3D shapes from set of 2D point trajectories. However, due to the difficulty they do not work well for videos in the wild. A recent work, LASR [49] uses two-frame optical flow to reconstruct articulate shapes from monocular video with differentiable rendering. Despite the promising results, LASR does not reason about long-range correspondences and can only reliably reconstruct what is visible in a short video. ViSER establishes reliable long-rang carrespondenn that are robust to moderate shape variations and appearance changes. Thus, ViSER obtain much higher-quality reconstruction by using either a long video or several videos of a category.
3 Approach
Fig. 2 provides an overview of our approach, which follows a typical framework of differentiable rendering [16 25
Brorrowing the notation from LASR 49) we formalize our task as follows.
Given a set of video Inservaatinn including RGB pixel color, segmentation desmentation masks, and optical flow
estimates [,, S, t== our goal is to recover set of shape and motion parameters (S, D.
that produce reconstructions (fe, St, ât )= that match the video observations. We refer to supplementary material for complete list of notations defined in the paper.
complet
3

---

3.1 Preliminaries
We represent an object's shape as triangular mesh (V.F) with canonical vertices V R and fixed topology (edge connectivity) F E R3xM M. To render an object, we displace mesh vertices with motion parameters D,, apply a perspective projection with camera intrinsics K,, and rasterize.
We model vertex motion with root body transformations Go and object articulations (GI.... GB) using linear blend skinning (LBS) (20 21). LBS constrains vertex motion by linearly blending B rigid "bone" transformations with a skinning weight matrix W E RBx,, transforming the canonical shape into frame as
VA, = Go., ( W... V:
(1)
where is the vertex index, b is the bone index. Similar LASR, the root body and bone transforma-
tions are represented as the outputs of a pose CNN given an input image, (Go, GB) = (T,).
We define set of surface properties for rendering, including vertex 3D coordinates, textures and features, and rasterize them in a differentiable manner [25). We denote the differentiable rendering function that renders the property C defined on canonical surface to an image as R(C; V, W, G). which executes the blending skinning function in Eq. ① and softly blends the surface property based on their depth and barycentric coordinates [25] For simplicity,
omit the shape, skinning, and motion parameters parameters and write the differentiable rendering function as R(C). To render optical flow, rasterize and project vertex coordinates in two consecutive frames and compute their 2D displacements [49]. Such renderings are compared against video observations to compute gradients for updating model parameters.
3.2 Video-specific Surface Embedding
Pixel-surface embeddings. We learn pixel and surface embeddings that map corresponding pixels in different frames to the same point on a canonical 3D) surface. Intuitively, consider particular region on the canonical surface mesh that is the "nose" of an articulated human. The surface embedding captures descriptor for the nose, which can then be matched to pixel-level descriptors at each frame.
Given an input image 4t, the pixel-wise descriptor embedding computed by a U-Net [34 encoder:
Fr jz, v. 4] = (5, vl E R
(2)
where [r., y, are pixel locations frame The surface embedding is computed by position- encoded MLP:
Fs(X,V, de. Y, Y, 2.
where (-) is an MLP defined over 3D points (X, Y, Z) the canonical space, augmented with Fourier positional encoding (28). The two embeddings optimized test vidcos such that pixels representing the same surface location in different frames are mapped to the same canonical surface point 20
Correspondence via soft-argmax regression. Given the pixel and surface embeddings, we construct per-frame cost volume (Fr, Fs) of size H x W x N, over pixels and surface points (we randomly sample N. 200 surface points at each step) by considering their cosine feature distances,
(Fr [r, y. = - cos [F v). Fs (X,, Y,, Z.))
(4)
Normalizing the cost volume over the surface point dimension yields softmax "heatmap" over surface points that potentially match to pixel (x,, y), as shown in Fig. (Left):
(a. [+]
where r is a temperature scaling parameter that is jointly optimized with the feature embeddings. To output a single surface point for pixel (z, y). we can compute "soft" argmax [17 48) by taking the expectation of the softmax distribution over the 3D locations of the points samples,
Š( w] = Σ (a Y., 2.,
(6)
where (X,, Y, Z:) is the i-th sampled surface point and (I. v) is the matching probability of pixel (z, y) over the sampled points
21,2,...,NN N.).

---

query pixel (2)
surface heatmap ((a)) X, Y, 2]
query surface point (X.YZ)
pixel heatmap (xXY 22) (2, M)
Figure 3: Pixel-surface embeddings establish continuous mapping between pixels and points on canonical surface. Left: Given query pixel at (x-y), we match it to set of canonical surface points, where the matching distribution is used regress continuous mapping to the canonical surface. Right: Given query surface point (X, (X,Y)), matching distribution over pixels can computed. Warm color indicates high matching prohability.
We can also normalize the H W x XN, cost volume over spatial positions to capture a distribution of pixel locations that match to each surface point (X,, Y,, 2;):
Eprool P(a.vl//
and compute a similar soft argmax mapping of surface points to pixels, as shown Fig. 3 (Right).
Relation to keypoints. The output of classic keypoint detectors are often represented as K K-channel heatmaps over the pixel grid, where K is the number of keypoints. To define dense keypoints, one may increase the number of channels, which is computationally heavy. Similar to CSE [29] we represent dense keypoints as low-dimensional pixel-surface embeddings. which establishes a mapping between pixels and a canonical 3D surface, but far more efficiently. DensePose [12] and CSM [20]
use an alternative pixel-to-surface mapping that regresses surface coordinate at every pixel. In contrast, our pixel-surface embedding captures multimodal uncertainties over keypoints; for example, ((.,)) can capture the fact that particular pixel matches well to both the left and right ankle, as
visualized in Fig. 3 while regressor may "regress" to the mean of the two surface coordinates.
3.3 Learning Embeddings and Articulated Shapes
Next we will introduce the loss functions that enable learning both embeddings and articulated shapes from monocular videos without pre-defined shape template or annotated correspondence. To learn non-degeee embeddings and overcome the local optima issue in differertiable renderers, we carefully construct a 3D matching loss and a 2D cycle loss.
3D match loss. Arguably, the simplest loss to learn embeddings is minimize the difference between the rendered surface features and the observed pixel features:
Lteatue-consistency
= Σ cos (R F(l.,))) P.M
(8)
where cos(-) denotes the inner product between two normalized vectors, and R(Fs) is the differen- tiably rendered surface descriptors. However, the feature consistency loss admits trivial solution, where all pixel and surface features are the same constant (yielding zero error). To address this, we introduce a 3D matching loss that ensures pixel embeddings only match to surface embeddings rendered at the pixel location: Ematching
Σ R(V) -$(2, v/ 2.M
where R( R(V) is the rendered 3D surface location and $|z, is the estimated pixel-to-surface mapping from Eq. (6). computed through sampling and computing the softmax distributions ji] over surface points [17. To minimize the loss, the embeddings of surface points that do not project to (æ, y) will be pulled away from the pixel embedding of (I, y) in contrastive way 13
2D cycle loss. The match loss aims to learn pixel- pixel-surface embeddings that are consistent over video frames and discrininative over difference surface locations. However for articulation optimization, the match loss suffers from bad local optima issue similar to other losses based on differentiable rendering 25). For instance, when the rendering of body part is outside the ground-truth object silhouette, gradient update of articulation parameters would likely not incur lower loss.
5

---

To guide articulated 3D shape learning using the learned pixel-surface embeddings. we further define cycle-based re- projecteooon loss, inspired by prior approaches in 3D model fitting with keypoints [3] and canonical surface mappings [20]. Given an input image, we establish a 2D>3DD mapping by extracting a pixel embedding and matching it to surface embedding. Then, we compute the expected surface
coordinate S1z, v/ at every pixel using Eq. (6). and ensure the differentiably rendered canonical surface coordinate lands back on the original pixel coordinate (x.
Preproj = Σ
R(S[ ul) (1. v.)
(10)
Reconstruction loss. Finally, we make use of reconstruction losses to ensure that generated images, masks, and flows match their estimated counterparts:
Trecon 81 ||5t S.1lē + =.11 T.1 + Ssoilla) mill Sapdist( T²)
(11)
where (,,,... 34] are weights empirically chosen, d; the normalized confidence map for flow measurement, and pdist(-:-) is the perceptual distance [51] measured by an ImageNet- ImageNet-prerrained pretrained AlexNet. The reconstruction losses ensure the match between rendered and observed optical flow, texture and silhouette images.
Regularization. To avoid degenerate shapes, enforce the recovered shape to be smooth, and as-rigid-as-posiblle (ARAP) regularization to enforce the deformation be locally rigid [44]. Different from prior work that only preserves the length of edges after articulation, we encourage both the area and length of faces to be the same after articulation. The area preserving term is defined
mesh Laplacian regularization 16 49
E
LARAParruu
Σ
ΣE
E
E x 1
(12)
1=1 JEN;
where E is the number of edges and N; the indices of neighbouring edges.
3.4 Representing Surface Properties with MLPs
By extending surface embedding MLPs with additional dimensions, we can model other surface properties including textures and even surface-based geometric deformations. Compared to explicitly defined textures, such continuous implicit representations have the capacity to encode arbitrary amount of details and are empirically easier to optimize.
Surface appearance. The appearance of the object is represented by a coordinate-Mil queried points on the canonical mesh surface. To handle view-dependent appearance (such as shadow and lighting), we further concatenate the Fourier features of the (X, Y, %) coordinates with frame appearance code, as the input to the texture ML.,,
(13) where V, is the i-th canonical mesh vertex, which passed through Fourier encoder F(-) as used in NeRF (28). and concatenated with wr, a 64-dimensional frame appearance code associated each image frame predicted from a R8sNet-18, as wi Viea (T)).
C₁A = &rea (FVV)) ER"
Instance shape deformation fields. To deal with videos of multiple instances of the same category. as experiments Sec. 4. we model shape variations across instances by continuous surface deformation field defined on the canonical surface. Similar to the surface texture, we represent the surface deformation field by shape MLP,
VI V + Oshape (F( os) e RR"
(14)
where Vx is the rest shape of instance and 04 is video-specific 64-dimensional shape code that randomly initialized and optimized together with the shape MI..
4 Experiments
We evaluate ViSER in three different scenarios where objects are highly articulating, making challenging reconstruct and estimate long-range correspondences. First, we consider long human

---

Table 1: 2D Keypoint transfer accuracy athletic videos. Methods with Best results are in bold.
keypoint annotations to train.
Method
break-1 1
break-2 2
dance
parkour
ballet-1 1
ballet-2
ballet-3
Ave.
"DensePose CSE 29
56.0
13.2
77.2
85.9
45.6
49.0
64.5
55.9
"VIBE+SMPLitf
18
37.1
8.2
70.4
83.8
55.4
53.0
78.8
55.2
LASR 49
29.1
18.1
56.6
49.8
44.5
47.4
48.6
42.0
VISER (Ours)
70.5
22.5
80.7
62.9
52.7
56.1
59.9
57.9
Table 2: 2D Keypoint transfer curacy on multiple elephant videos. Best results are in bold.
Table 2D Keypoint transfer accuracy BADIA dataset. Best results are in bold.
Method
camel
dog
cows
horse
bear
Ave.
Method
inner
across
CSE
48.8
38.6
63.8
60.2
76.6
$7.6
CSE 29
55.7
52.2
Flow-VCN
48
47.9
25.7
60.7
14.4
63.8
42.5
Plow-VCN 48
51.1
41.2
N-NRSIM 38
67.8
17.9
70.0
8.7
60.2
44.9
LASR 49
57.8
LASR 49
81.9
65.8
83.7
49.3
85.1
73.2
ViSER (Ours)
80.4
68.9
ViSER (Ours)
80.1
73.8
82.9
76.3
87.3
80.1
videos with loose clothing and unusual poses. Next, we evaluate on videos of articulated animals for which accurate shape templates are missing. Finally, we analyze a multi-video variant ViSER that learns single model from multiple videos of the same category. All scenarios require jointly establishing long-ran rangrange correspondences and reconstructing articulated 3D shapes at the same time.
Optimization details We use the AdamW [27] optimizer with a batch of 4 consecutive image pairs. We reconstruct a long video sequence in an incremental manner similar classic SfM. First, an initial set of around 20 consecutive frames to initialize the shape and pixel surface embeddings. The initial set is selected such that the viewpoint coverage is large enough. Then we gradually add in new frames. When a new frame is added, we first apply the 2D cycle loss Lieproš to optimize its articulations, and then jointly optimize all frames with all losses. Empirically, simultancously optimizing all the video frames produces unstable results of root body poses (or equivalently camera poses).
4.1 Athletic Video Reconstruction
Dataset. To evaluate ViSER on long-videos, we construct an athletic video dataset that is challenging due to loose clothing and unusual body poses. consists four videos from DAVIS (31 and three ballet videos. All videos are segmented and manually annotated with keypoints following the MSCOCO format (24] We only use keypoint annotations for evaluation purposes.
Metrics. Due to the lack of ground-truu truth 3D data for challenging athletic human videos, we use 2D keypoint transfer as proxy metric [ [2). Given any two frames from video, the goal is to transfer an annotated 2D keypoint from one frame to another. The accuracy is measured by percentage of correctly transferred keypoints over all T(T-1) pairs frames T-frame video. A transferred keypoint is marked correct when its distance to the ground-truth annotation is lower than de. 0.2./1S). where |S| is the area of the ground-truth silhouette 2 In general, a more accurate reconstruction leads to a higher transfer accuracy.
Baselines. To compare with template-hased approaches for video human reconstruction, we use VIBE with SMPLify temporal smoothing [18] To compare with template-free methods, we use LASR [49]. which also reconstructs articulated shapes using the same input setting as ours. To transfer keypoints from a reference frame to a target frame, we back-project the annotated keypoint in the reference frame to the canonical surface, and then project the intersected 3D point to the target frame. We also compare against Densepose CSE 29), which produces dense Axxe-ttosuurface correspondences for given category, but does not produce 3D reconstructions. To transfer keypoints for Densepose CSE, we compute pixelwise surface mappings for both frames and find the best matching w.r.t.geodeicc distance on the surface. We further qualitatively compare against state-of- the-art human reconstruction method, PiFUHD 36 in Fig. 4 which only produces reconstruction, but not correspondence.
7

---

Reference image
VIBE+SMPLity
PIFaHID
LASR
Ours
Alternate viewpoint
Figure 4: Qualitative comparisons for athletic video articulated shape reconstruction. Compared to methods that uses shape and pose priors (VIBE+SMPLify and PiFUHD), our method achieves comparable performance for common appearance and poses, and does much better on unusual poses such as traak-daceers
Reference image
LASR
Alternate view
Ours multi-video Alternate view
Figure 5: Qualitative comparisons for elephant shape reconstruction from multiple videos. Notice that ViSER able to take advantage of multiple videos to improve the category-level shape reconstruction but also reconstruct instance-specific details (as shown in red circles).
Figure Part segmentation results. Colors are determined by hard assigning vertices the closest rigid bones.

---

Reference image
Ours-single video
Ours-multi video
Figure 7: Comparison between single video ViSER and multi-video ViSER terms of reconstructing YTVOS elephants. We find using multiple videos helps reconstructing the body parts that may be occluded in a single vidco. While single-video ViSER reconstructs flattened shape and misses the hidden rear leg of the elephant, multiple video ViSER reconstructs more plausible shape and recovers both the rear limbs.
shows visual reconstruction results sample videos and for different techniques. ViSER estimates reconstructions that are more faithful to the input than the baselines, especially when the humans have unusual poses like in the first two rows. The accurate Long-range correspondence enables ViSER to reconstruct finer details than LASR that does not explicitly try to estimate long- range correspondences. We summarize quantitative comparisons in Tab. I There is moderate performance gap between ViSER and template- template-based methods when the input fits the latter, such as parkour with tight clothing and usual pose. Note that the supervised Densepose CSE and OpenPose methods fail on breakdance videos due to the novel pose, and also do not work well on ballet dancers due to loose clothing. As a result, template-based approaches that rely on accurate pose recognition, such as VIBE [8] fails. In contrast. our method does not suffer from such poor -ittrr--iittiittooon generalization. By establishing long-range correspondences, ViSER achieves higher keypoint transfer accuracy and better 3D reconstruction than LASR.
Results. Fig. 4
4.2 Reconstructing Animals from a Video
We use BADJA 2 to evaluate VISER on animal vidcos including camel, cow, dog, bear and horse. Similar to the athletic human video dataset, compare against template-free methods such LASR and neural-dense-ND -RRSM (N-NRSfM) [3]. Similar to LASR and our setup, N-NRSfM learns a video- sieeo-specific model for object shape, deformation and camera parameters from multi framm- optical flow estimations [8]. We further report performance comparison with dense correspondence methods such CSE and an optical flow method, VCN. We the CSE model trained on corresponding animal categories (except that we use the horse model for camel), and the "robust" model of VCN M.. which is the input to our method. As shown in Tab. 3 ViSER achieves better or similar accuracy on all five animal videos compared to LASR and N-NRSIM. While the imput optical flow is not robust at estimating long-range correspondences, our method integrates local optical flow to dense long-range correspondences via canonical shape, and achieves much better keypoint transfer accuracy. Note that CSE performs well for categories has been trained on, such as cow, horse and bear, but performs poorly on novel animal categories, such as camel and novel breed of dog.
4.3 Multi-video Shape and Correspondence
We curate a set of seven videos of different elephants from YTVOS 47 for multi-video shape and correspondence recovery. The annotations will be released for further research. We treat multiple videos as a single long video with strong appearance changes and shape variations. In the multi-video setup, We evaluate keypoint transfer accuracy on both the same instance (with video frames) and over different instances (across video frames), as denoted by "inner" and "across". Quantitative results in Tab. [ shows that ViSER is more accurate than the baseline methods in both cross-video keypoint transfer and inner video keypoint transfer by a large margin, without using any keypoint annotations or pre-defined shape templates. Fig. [5 show visual result comparisons. While LASR recovers the visible surfaces in video, it cannot infer the invisible parts. In contrast, our method is able to take advantage of multiple videos from the same category and produce a much better shape reconstruction. Note that LASR cannot handle multiple videos as it requires optical flow computed between every adjacent frame pairs. VISER, on the other hand, also uses correspondences via estimated 3D shape, thereby allowing the use of multiple videos even when the optical flow is missing across videos.

---

Table Ablation study on keypoint transfer. Best results are in bold.
Method
break-1
elephants
elephants- cross
Full
70.5
80.4
68.9
w/o matching loss Fmnatching Eq.
9
36.2
51.3
42.6
w/o reprojection loss Freeprajy Eq.
T0
38.3
80.1
62.5
CSM regression
20
47.1
77.4
63.3
Benefit of Using Multiple Videos. To examine the benefits of using multiple videos, we further
compare multi-video ViSER with single-video ViSER, shown in Fig. 7 We find using multiple videos helps reconstructing the body parts that may be occluded in a single video.
4.4 Part Discovery and Ablations
Part discovery. ViSER can discover detailed 3D part segmentation without any manual annotation, as shown in Fig. 6 After training either on a collection of videos or a long video, ViSER can segment the 3D shape into meaningful parts, such as the trunk of the elephants and the feet of the dancer.
Ablation study. We perform ablation study break-1 and elephants, shown in Tab. Without the contrastive matching loss, the pixel-surface embedding converges to a trivial solution with a significant decrease of accuracy. Removing the re-projection loss leads to much lower keypoint transfer (KPT) accuracy on the breakdance-1 sequence and cross-video KPT accuracy on the elephant videos. Likely the surface reprojection loss plays an important role in learning correct articulation that follows the bottom-up dense keypoint predictions. This may effectively avoid the local minimum issue for the differentiable rendering optimization. Finally, replacing the pixel-surface embedding with direct CSM regression 20 does not reason about distribution of possible matches and results in worse performance.
Limitations. We find ViSER to be sensitive to the random initialization of network parameters. We optimization with different random seeds for initializing the network parameters and find some perform considerably worse than the others, due to the convergence to bad local optima. Although in practice, could spot the convergence and re-run the optimization with a different random seed, an automatic method for selecting the best model parameters over different trials is desired. We leave how to make the optimization of ViSER robust for future research.
bad local optimum by visualizing the articulated shapes
ViSER also relies optical flow to kick-start with reasonable initial shape and pose for learning pixel-surface embeddings. Although recent optical flow models generalize well in many scenarios, they may fail when video is of low resolution or contains significant motion blur. In such challenging cases, using category shape and pose priors to initialize ViSER would be promising direction.
Conclusions
We have introduced ViSER. method to reconstruct articulate shapes, dense trajectories, and object parts from monocular videos. ViSER establishes long-range correspondence by matching 2D pixels to canonical 3D mesh via learned video-specific surface embeddings. Experimental results show that ViSER, without a template shape or keypoint annotations, compares favorably against prior work challenging human and animal videos. ViSER shows that it could be fruitful to reconstruct articulate shapes for categories in the wild, and we hope to see more work in this direction.
Broader impact. ViSER has many potential applications, e,g., in robotics, AR/VR, and film industry, but may be used for malicious purposes, e.g., producing fake videos or extracting bio-metric informa- tion without prior consent. ViSER is only suitable to offline applications as it takes about several hours to process a 80-frame video on one NVIDIA P100 GPU.
Acknowledgments
This work was supported by Google Cloud Platform (GCP) awards received from Google and the CMU Argo AI Center for Autonomous Vehicle Research. We thank William Freeman and many others from CMU and Google for valuable feedback.
10

---

References
Robust vision challenge. www. www.robustvision.co 2020.
[2] B. Biggs, T. Roddick. A. Fitzgibbon, and R. Cipolla, Creatures great and smal: Recovering the shape and motion of animals from video. In ACCV, pages 3 19. Springer, 2018.
[3] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from single image. In ECCV, 2016.
[4] C. Bregler, A. Hertzmann, and H. Biermann. Recovering non-rigid 3d shape from image streams, In CVPR, volume 2, pages 690-696. IEFF, 2000.
[5] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon al. (Eds.), editor, ECCV, Part IV, LNCS 7577, pages 611-625. Springer-Verlag. Oct, 2012.
[6] T. Darrell and A. P. Pentland. Cooperative robust estimation using layers of support. TPAMI, 17(5): 474-487, 1995.
[7] J. Fngell, T. Schöps, and D. Cremers. Lsd-slam: Large-scale direct monocular slam. In ECCV, pages 834-849. Springer, 2014.
[8] R. Garg, A. Roussos, and L. Agapito. A variational approach to vidco registration with subspace constraints. IJCV, 2013.
[9] A. Geiger, P. Lenz, and R. Urtasun. Are ready for autonomous driving? The KITTI vision benchmark suite. In Proc. CVPR, pages 3354-3361, IEEE, 2012.
[10] S. Goel, A. Kanazawa, and J. Malik. Shape and viewpoints without keypoints. In ECCV, 2020.
[11] P. F. Gotardo and A. M. Martinez. Non-rigid structure from motion with complementary rank-3 spaces. In CVPR, pages 3065-3072. IEEE, 2011.
[12] R. A. Güler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, pages 7297-7306, 2018.
[13] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, volume 2, pages 1735-1742. IEEE, 2006.
[14] A. Jepson and M. Black. Mixture models for optical flow computation. In CVPR, pages 760-761. IEBE, 1993.
[15] H. Joo, T. Simon, X. Li, H. Liu, L. Tan, L. Gui, S. Banerjee, T. Godisart, B. Nabbe, I. Matthews, et al. Panoptic studio: A massively multiview system for social interaction capture. TPAMI, (111:1990-204 2017.
[16] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik. Learning category-specific mesh reconstruction from image collections. In ECCV, 2018.
[17] A. Kendall. H. Martirosyan, S. Dasgupta. P. Heary, R. Kennedy, A. Bachrach, and A. Bry. End-to-end learning of geometry and context for deep stereo regression. In ICCV, pages 66-75, 2017.
[18] M. Kocabas, N. Athanasiou, and M. J. Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, June 2020.
[19] C. Kong and S. Lucey. Deep non-rigid structure from motion. In ICCV, pages
2019. [20] N. Kulkarni, A. Gupta, D. F. Fouhey, and S. Tulsiani. Articulation canonical surface mapping. In CVPR, pages 452 0224611 2020.
[21] J. P. Lewis, M. Cordner, and N. Fong. Pose space deformation: a unified approach to shape interpolation and skeleton-drivee deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 165 172, 2000.
[22] X. Li, S. Liu, S. De Mello, K. Kim. X. Wang. M.-H. Yang, and J. Kautz. Online adaptation for consistent mesh reconstruction in the wild. In NeurIPS, 2020.
[23] X. Li, S. Liu, K. Kim. S. De Mello, V. Jampani, M.-H. Yang, and J. Kautz. Self self-supervised single-view 3d reconstruction via semantic consistency, ECCV, 2020.

---

[24] T.-Y. Lin. M. Maire, S. Belongie. J. Hays. P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[25] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-hased 3d reasoning. ICCV, Oct 2019.
[26] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person linear model. SIGGRAPH Asia, 34(6):248:1 1-0:244:1:1281666 Oct. 2015.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
[28] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes as neural radiance fields view synthesis. In ECCV, pages 405-421. Springer, 2020.
[29] N. Neverova, D. Novotny, V. Khalidov, M. Szafranice, P. Labatut, and A. Vedaldi. Continuous surface embeddings. In NewrIPS, 2020.
[30] R. A. Newcombe, S. Izadi, Hilliges, Molyneaux, D. Kim, A. Davison, P. Kohi, Shotton, S. Hodges, and A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pages 127-136 136. IEEE, 2011.
[31] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Mornnnn. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016.
[32] A. Rav-Acha, P. Kohli, C. Rother, and A. Fitzgibbon. Unwrap mosaics: A new representation for video editing. In SIGGRAPH, pages 1-11, 2008.
[33] Z. Ren, O. Gallo, D. Sun, M.-H. Yang, E. B. Sudderth, and J. Kautz. A fusion approach for multi- frame optical flow estimation. In WACV, pages 2077-2086. IEEE, 2019.
[34] O. Ronneberger. P. Fischer. and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234- 241. Springer, 2015.
[35] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, pages 2304-2314, 2019.
[36] S. Saito, T. Simon, J. Saragih, and H. Joo. Pifuhd: Multi-level pixel-aligned implicit function for high- resolution 3d human digitization. CVPR, 2020.
[37] P. Sand and S. Teller. Particle video: Long-range motion estimation using point trajectories. IJCV, 80(1):
72, 2008.
[38] V. Sidhu, E. Tretschk, V. Golyanik, A. Agudo, and C. Theobalt. Neural dense non-rigid structure from motion with latent space constraints. In ECCV, pages 204-222. Springer, 2020.
[39] N. Snavely, S. M. Seitz, and R. Szeliski. Modeling world from internet photo collections. IJCV, 80(2): 189-210, 2008
[40] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR, June 2018.
[41] N. Sundaram, T. Brox, and K. Keutzer. Dense point trajectories by gpu-accelerated large displacement optical flow. In ECCV, pages 438 -38-451. Springer, 2010.
[42] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[43] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: a factorization method. IJC,, 92) 1992.
[44] S. Tulsiani, N. Kulkarni, and A. Gupta. Implicit mesh reconstruction from unannotated image collections. In arXiv, 2020.
[45] J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Transactions on Image P Processing, 3(5).225-638 Sept. 1994.
[46] S. Wu, T. Jakab, C. Rupprecht, and A. Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv preprint arXiv:2107.10844 2021.
12

---

[47] N. Xu, L. Yang. Y. Fan, J. Yang. D. Yue, Y. Liang, B. Price, S. Cohen, and T. Huang. YouTube-VOS: ssquence-fo-sequenee video object segmentation. In ECCV, pages 583-601, 2018.
[48] G. Yang and D. Ramanan. Volumetric correspondence networks for optical flow. In NeurlPS, pages 794 805, 2019.
[49] G. Yang, D. Sun, V. Jampani. D. Vlasic, F. Cole, H. Chang, D. Ramanan, W. T. Freeman, and C. Liu. LASR: Learning articulated shape reconstruction from monocular video. In CVPR, 2021.
[50] Y. Ye, S. Tulsiani, and A. Gupta. Shelf-supervised mesh prediction in the wild. In CVPR, pages 8843-8852, June 2021.
[51] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pages 586-595, 2018.
[52] T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros. Learning dense correspondence via 3d-guided cycle consistency. In CVPR, 2016.
[53] S. Zuffi, A. Kanazawa, D. Jacobs, and M. Black. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, July 2017.
[54] S. Zuffi, A. Kanazawa, and M. J. Black. Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images. In CVPR, 2018.
13


Reviews:
[{"review_id": "svoh3ltJMWm", "reviewer": "Reviewer_VYGA", "summary": "VISER tackles the problem of 3D shape and pose reconstruction from an input video with masks and flows. The main competitor on this task is the recent LASR approach (CVPR 2021) The main technical novelty in VISER is joint embedding of pixels and surface points learned with a self-supervised contrastive loss. These embeddings match each surface points to all corresponding pixels in the video, thus addressing long-range correspondences in the video. ViSER outperforms the state-of-the-art on human and animal videos and the supplementary videos are very impressive.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows impressive results in the supplementary videos and clearly outperforms the state-of-the-art. The main novelty is the joint pixel and surface embeddings, and how it is used to learn long-range correspondences. The main weakness is writing which is hard to follow.\n\n# Writing :\n\nThough the results are impressive, the quality of the writting could be improved. Some parts are still not very clear to me :\n\n-- Please highlight the similarities and differences between VISER and LASR. Is it reasonnable to think that the core of VISER is the feature embeddings i.e. you could take LASR code and data, add a CNN head to predict per pixel features, an MLP head to predict per surface point features, add the embedding losses and improve over LASR? If so, it would greatly help understand how the paper relates to the literature to clarify this.  \n\n-- Please clarify the exhaustive list of optimized parameters and their shape as done in LASR.’\n\n-- Please clarify equation (8) and (9) with a figure. In general I find the notation R(F) or R(V) uncommon. Maybe it would help to clarify it a bit more beforehand.\n\n-- The words “top-down” and “bottom-up” are used multiple times and confuse me.\n\n--Equation 13 and 14 are not clear. If V_k is a rest shape, then it belongs to R^(N,3) instead of R^3. What is instance k? Do you mean time-step k?\n\n--L191 : “position-and-image-encoded MLPs”→ I don’t know what you mean, a sentence would be better\n\n-- Please add the losses in the pipeline figure. They are critical to get an intuition on why the method works and brings an improvement over LASR\n\n-- Are the motion parameters G_0 … G_B, K represented as the output of a CNN, like in LASR, or explicit parameters of the optimization problem?\n\n\n# Losses :\n\nI briefly describe the additional losses over LASr as a sanity-check that I understood the approach.\n\n**consistency** : Each 3d point should have the same features as the pixel it projects to.\n\n**constrastive** : each 3D point should follow this cycle : 3d -> projection to pixel space -> estimated corresponding 3D point via cost volume matching. \n\n**Lreproj** : is the same as the constrastive loss except the cycle is 2d->3d->2d instead of 3d->2d->2d . Why place them in separate paragraphs?\n\nPlease clarify what you mean by “Different from geometric cycle consistency loss [19] that suffers from depth ambiguity along the back-projected ray, our formulation leverages graphics rasterization to automatically handle self-occlusion during joint embedding learning.”\n\n# Small comments\n\n**Figure 5:** Some parameters of LASR could be shared across several videos of the same category to compare more fairly with VISER in 4.3 in particular the template shape V and the skinning weights.\n\n**Figure 2 caption :** “Given a <test\\> video.\" There is no training nor testing set in your approach, it is a per-video optimization. Referring to “test” here is confusing.\n  \n# Feedback  \nAn additional interesting loss you could consider on the embeddings is enforcing that the distance between two embeddings equals the geodesic distance of the corresponding surface points, as is done in HumanGPS which in their own words “empirically leads to accurate, smooth, and robust results.”\n\n# Post-Rebuttal comment\nI have read the other reviews and thank the authors or their responses. I thank the authors for clarifying a number of points that were not clear to me initially. I maintain my initial rating and recommend this paper for acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Tnj_lV_Omh", "reviewer": "Reviewer_H2YS", "summary": "This paper presents a new method for reconstructing a 3D shape template and dense 3D part trajectories from a monocular video. This work is very similar to a recent work LASR. The main difference, which is also the main contribution, is to establish long-range correspondences by matching the features between each 2D frame to a 3D canonical shape. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work solves an interesting and important problem, i.e. reconstructing 3D templates and dense trajectories without any 3D coarse geometry or prior knowledge. The idea of establishing correspondences via pixel-to-model registration makes sense, which avoids the drifting issue caused by integrating optical flows frame by frame. \nThis method is very close to LASR but achieves better quality. \nMy main concern is that this method lacks novelty. Most of the pipeline components are quite similar to LASR. The main difference is how to establish correspondence. Instead of supervising with frame-to-frame optical flows, this method introduces a shared canonical space for employing the pixel-to-model registration. However, this pixel-to-model matching is widely used in classical computer vision and graphics techniques for registration. \nBesides, a comparison with \"Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes\" is expected. \nAlso, the authors are expected to release the code and data. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above for the limitations. No negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "QBhd-lt9iH", "reviewer": "Reviewer_mXo1", "summary": "Authors propose a method for reconstructing deforming objects from videos (with the target object pre-segmented). The output is a mesh with bone assignments, plus temporal deformation parameters (i.e. bone transformations). The approach predicts correspondences between frame pixels and points on the mesh by mapping them into a joint embedding space, using a contrastive loss. The method is demonstrated on videos of humans and animals, where it out-performs a recent baseline.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Method:\n\n++ The proposed approach is novel. More specifically: there is significant commonality with LASR, but I feel there's enough technical novelty in the present work to justify publication.\n\n++ Different parts of the model are clearly motivated and described.\n\n++ The idea of learning a joint embedding space for the 3D surface and the pixels is elegant.\n\nExperiments:\n\n++ The qualitative results on reconstruction are impressive, including on fairly challenging data\n\n++ Quantitative results (though only on keypoint transfer) exceed baselines (which are sensibly-selected modern approaches), apparently by a significant margin\n\n-- There is no discussion of error bars (due e.g. to different initialisations) -- this should be added.\n\n-- There is no direct evaluation of the geometric reconstruction performance. While this is challenging for natural data (lack of gt), it would be valuable to see such results on synthetic data, plus a comparison to baselines.\n\n-- Part-segmentation evaluation is not evaluated numerically -- only qualitative results are shown (which are ok but do not seem particularly accurate). A quantitative evaluation, against a suitable baseline, would be valuable.\n\n-- The predicted textures are never shown.\n\nMisc.:\n\n- Fig.1: typo: \"inlcuding\"\n- 269: typo: \"alblations\"\n\n---\n\n## Post-rebuttal\n\nThe authors' response addresses my concerns sufficiently, and I remain in favor of acceptance having read the other reviews (and respective responses). The sensitivity to initialisation (as reported in the authors' comment below) seems significant, and so this should be discussed in the revised paper. I still think more quantitative evaluation (of geometric reconstruction) would strengthen the paper -- this could be done on synthetic data, or human data with full annotations (e.g. Human3.6M).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion of limitations. There is brief (but adequate) discussion of broader societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Dr9KTb_G5v", "reviewer": "Reviewer_jCtU", "summary": "This paper presents an optimization-based framework that recovers 3D shape, articulated pose, and weak texture from a monocular video of an articulated object. The proposed method extends a recent method (LASR) with a surface embedding matching module for establishing long-range correspondences, and achieves significantly better results on highly deformable objects, such as break dancers.\n\nThis is an extremely challenging task, and has a long history in computer vision. Current methods either require extra information, eg depth or keypoints, or rely on category-specific template shape models. The results achieved by this method is very impressive, relying only on off-the-shelf category-agnostic segmentation and optical flow models.\n\nThe paper also demonstrates the possibility of extending the method with different instances in multiple videos, which presents exciting potential of capturing category-level priors for fast inference.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n### S1 - Method\nThe proposed extension from LASR is very powerful. It aims at establishing long-range correspondences by matching pixels in different frames through 3D surface embeddings. Specifically, pixels correspondences are established indirectly by matching CNN-predicted pixel embeddings to the same 3D surface embeddings.\nAt a high level, instead of relying on pixel colors for matching correspondences, it matches correspondences in a learned embedding space, which is more robust against pixel variations resulting from shading, occlusion, deformation etc.\nThis idea is carefully implemented in this paper and achieves impressive results.\n\n\n### S2 - Results\n- The reconstruction results show significant improvement over the LASR baseline, especially on longer videos, confirming the effectiveness of this embedding correspondence matching module.\n- The evaluation and comparison with prior arts are extensive and thorough, presenting lots of technical insights.\n- Moreover, the results on multi-video optimization also look very promising, demonstrating the possibility of extracting category-level shape priors from a video collection, which could potentially be extended to unlimited Internet videos at scale.\n\n\n## Weaknesses\n### W1 - Complicated optimization pipeline\nOne major limitation of the proposed method is its overwhelming complexity. The method is built on top of LASR which already has a very complicated optimization framework, and this new version further adds in even more complexity. It is very impressive that the authors manage to produce those results with this gigantic framework, but it still poses concerns on generalization. However, the paper already shows results on a diversity set of examples, which suggests it's still fairly robust.\n\n### W2 - Texture\nAnother weakness of the method is the poor texture prediction. I wonder why the optimized texture looks very coarse as shown in the supplementary results.\nBut as the method mainly focuses on shape reconstruction, and the model bypasses establishing correspondences via pixel colors but rather through embeddings, I think this is not a big issue.\n\n\n### W3 - Model (more like a question)\nIntuitively, I thought the _contrastive matching loss_ $L_\\text{ct}$ and the _re-projection loss_ $L_\\text{reproj}$ have very similar effects. They are both enforcing the embeddings to be discriminative. The former goes from 2D pixel to 3D surface and compare the rendered 3D coordinate, whereas the latter also goes from 2D pixel to 3D surface, but then back to 2D pixel (also via rendering) and compare the 2D coordinate. But the ablation results in Table 4 suggests using $L_\\text{cs}$ alone without $L_\\text{reproj}$ also performs much worse than the full model in some cases. Could the authors elaborate on the difference of the two losses?\n\n\n## Additional comments\n- In Fig 5, it would be more convincing for demonstrating the benefit of aggregating multiple videos, if the multi-video results are also compared to single-video results. It would be even more exciting if the authors can show that this learned category prior enables fast adaptation on new videos or even with still images.\n- There is a tiny weird formatting issue in the caption of Fig 3.\n- It would be helpful to draw a diagram of the three matching losses $L_\\text{cs}$, $L_\\text{ct}$ and $L_\\text{reproj}$.\n- Line 273: \"rbreak\" -> \"break\"\n\n\n---\n## Post-Rebuttal\nI appreciate the authors' effort in the rebuttal, which has addressed all of my questions. The additional results with higher resolution and comparison between single and multiple videos provide further insights. This is a solid submission with a very interesting approach and great results on an extremely challenging task. I remain strongly in favor of acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A few failure modes are discussed in the supplementary material, but it can be helpful to also show some visualizations of them.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see immediate ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 positive final ratings: 7, 7, 8, 6. The reviewers mostly appreciated novelty (while noting high similarity with LASR), clear presentation and strong empirical performance.
The remaining questions and concerns seemed to be addressed in the rebuttal, as acknowledged by the reviewers.
The final recommendation is therefore to accept as a spotlight.
Decision: Accept (Spotlight)