LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery
Authors: Chun-Han Yao
Wei-Chih Hung
Yuanzhen Li
Michael Rubinstein
Ming-Hsuan Yang
Varun Jampani
Venue: NeurIPS 2022 Accept (2022)
Keywords: Articulated shape
sparse-view optimization
3D part discovery

Abstract: Creating high-quality articulated 3D models of animals is challenging either via manual creation or using 3D scanning tools. 
Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful. In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse). Contrary to existing works that rely on pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view or temporal information. Moreover, each input image ensemble can contain animal instances with varying poses, backgrounds, illuminations, and textures. Our key insight is that 3D parts have much simpler shape compared to the overall animal and that they are robust w.r.t. animal pose articulations. Following these insights, we propose LASSIE, a novel optimization framework which discovers 3D parts in a self-supervised manner with minimal user intervention. A key driving force behind LASSIE is the enforcing of 2D-3D part consistency using self-supervisory deep features. Experiments on Pascal-Part and self-collected in-the-wild animal datasets demonstrate considerably better 3D reconstructions as well as both 2D and 3D part discovery compared to prior arts. Project page: https://chhankyao.github.io/lassie/

LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery
Chun-Han Yao!
Wei-Chih Hung?
Yuanzhen Li
Michael Rubinstein
Ming- -sugn Yang!
Varun Jampani
UC Merced
Waymo
Google Research
Yonsei University
Abstract
Creating high-quality articulated 3D models of animals challenging either via manual creation or using 3D scanning tools. Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful. this work, we propose practical problem setting to estimate 3D pose and shape animals given only a few (10-30) in-the-wild images of particular animal species (say, horse). Contrary existing works that rely pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view temporal information. Moreover, each input image ensemble can contain animal instances with varying poses, backgrounds, illuminations, and textures. Our key insight is that 3D parts have much simpler shape compared to the overall animal and that they are robust w.r.t. animal pose articulations. Following these insights, framework which discovers 3D parts in a self-supervised manner with minimal user intervention. A key driving force behind LASSIE is the enforcing of 2D-3D part consistency using self- self-supervisory deep fcatures. Experiments on Pascal-Part and self-collected in-the-wild animal datasets demonstrate considerably better 3D reconstructions as well as both 2D and 3D part discovery compared to prior arts. Project page: https: http///hhhaaaaaacaaa/ github. io o/lassie/
propose LASSIE, novel optimization
Introduction
The last decade has witnessed significant advances in estimating human body pose and shape from images [26, 2, 15, 19, 18, 34, 17]. Much progress is fueled by the availability of rich datasets [14, 29, 39] with 3D human scans in a variety of shapes and poses. In contrast, 3D scanning wild animals is quite challenging, and existing animal datasets such as SMALR [48] rely on man-made animal shapes. Manually creating realistic animal models for a wide diversity animals (e.g. mammals and birds) is also challenging and time-consuming. In this work, we propose to automatically estimate 3D shape and articulation (pose) of animals from only a sparse set of image ensemble (collection) without using any 2D or 3D ground-truth -truh annotations. Our problem setting is highly practical each image ensemble consists of only a few (10-30) in- wnllhe-wild images of a specific animal species. Fig. I shows sample zebra images which forms our input. In addition, skeleton (left in Fig. 1) that can be quite rough with incorrect bone lengths and this mainly provides the connectivity of animal parts. For instance,
assume human- saman--peeified 3D
use the same 3D skeleton for all the quadruped animals despite considerable shape variations across species, e.g., giraffe vs. zebra vs. elephant. Manually specifying rough skeleton is trivial task which only takes few minutes.
Estimating articulated 3D shapes from in in-the-wild image ensemble highly ambiguous and challenging problem. There are several varying factors across images: backgrounds; lighting; camera
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Generic
skeleton
Optimized and re-posed skeleton
Neural part surfaces
Image ensemble
Part and textured outputs
Additional examples
Figure I: Articulated shape optimization from sparse images in-the-wild. Given 10-30 images of articulated class and generic 3D skeleton, optimize the shared skeleton and neural parts as well as the instance-specific camera viewpoint and bone transformations. Our method is able to produce high-quality outputs without any pre-defined shape model instance-specific annotations. The part-based representation also allows applications like texture and pose transfer, animation, etc.
viewpoints; animal shape. pose and texture, etc. In addition, our highly practical problem setting does allow access any 3D animal models, per-image annotations such as keypoints and silhouettes, or multi-view images.
In this work, we present LASSIE, an optimization technique Learn Articulated Shapes from Sparse Image Ensemble. Our key observation is that shapes are composed of 3D parts with the following characteristics: 1) 3D parts are geometrically and semantically consistent across different instances. 2) Shapes of 3D parts remain relatively constant w.r.t changes in animal poses. 3) Despite the complex shape of overall animal body, 3D parts are usually made of simple convex shapes. Following these observations, LASSIE optimizes 3D parts and their articulation instead of directly optimizing the overall animal shape. Specifically, LASSIE optimizes 3D skeleton and part shapes that are shared across instances; while also optimizing instance instance-specific camera viewpoint, pose articulation, and surface texture. To enforce semantic part consistency across different images, we leverage deep features from a vision transformer [10] (DINO- ViT [6]) trained in self-supervised fashion. Recent methods [1, 38] demonstrate good 2D feature correspondences DINO- ViT features and we make use of these correspondences for 3D part discovery. To regularize part shapes, we constrain the optimized parts to a latent shape manifold that learned using 3D geometric primitives (spheres, cylinders, cones etc.). Being self-supervised at the image level, LASSIE easily generalize to wide range of animal species with minimal user intervention.
We conduct extensive experiments on image ensembles from the Pascal-Part Dataset [7] with quanti- tative evaluations using 2D ground-truth part segmentation and keypoints. We also collect in-the-wild web image ensembles with more animal species and annotate 2D keypoints for evaluation. Both qualitative and quantitative results demonstrate considerably better 3D reconstructions compared to the prior arts that use even more supervision. In addition, estimation of explicit 3D parts enables easy editing and manipulation of animal parts. Fig. shows sample 3D reconstructions with discovered parts from LASSIE. The main contributions of this work are:
To the best of our knowledge, this is the first work that recovers 3D articulated shapes from in-the-wild image ensembles without using any pre-defined animal shape models or instance-specific annotations like keypoints or silhouettes.
We propose novel optimization framework called LASSIE, where we incorporate several useful properties mid-level 3D part representation. LASSIE aategory-agnostee and can easily generalize to wide range of animal species.
LASSIE can produce high-quality 3D articulated shapes with state-of-the- results on existing benchmarks as well self-collected in-the-wild web image ensembles.
Related Work
Animal shape and p estimation. Estimating 3D pose and shape of animal bodies from in in-the-will images is highly ill-posed and challenging due to large variations across different classes, instances,

---

viewpoints, articulations, etc. Most existing mesh reconstruction work [21, 24, 11, 36, 44] focus more on compact shapes like birds and cars, and thus cannot handle articulations of animal bodies. methods, on the other hand, deal with simplified scenario by assuming
Recent articulation-awanl pre-defined class-level statistical model, instance-specific images, or accessible human annotations. For instance, Zuffi et al. [49] build statistical shape model, SMAL, for common quadrupedal animals, similar to the SMPL [26] model for human bodics. Follow-up work either optimize the SMAL parameterss based on human annotated images [48] or train neural network on large-scale image datasets to directly regress the parameters [47, However, the output shapes are limited by the SMAL shape space training images which contain one few animal classes. Kulkarni er al. [20] propose A-CSM technique that align a known template mesh with skinning weights to input 2D images. Other recent approaches [41, 40, 42] exploit the dense temporal correspondence in video frames to reconstruct articulated shapes. In contrast to existing methods, we deal with a novel and practical problem setting: learn without any pre-defined shape model, instance-level human annotations, or temporal information to leverage.
Part discovery from image collections. Deep feature factorization (DFF) [9] shows that one could automatically obtain consistent part segments by clustering intermediate deep features of a classification network trained on ImageNet. Inspired by DFF, SCOPS [13] learns a 2D part segmentation network from image collections, which can be used as semantic supervision for 3D reconstruction [24]. Choudury et al. [8] proposed to use contrastive learning for 2D part discovery. Lathuilière et al. [22] exploit motion cues in videos for part discovery. Recently, Amir et al. [1] demonstrate that clustering supervvsrvyylll learned vision transformers (ViT) such as DINO [6] features can provide good 2D part segmentations. In this work, we make use of DINO features for 3D part discovery instead of 21) segmentation. In the 3D domain, Tulsiani al. [37] use volumetric cuboids as part abstractions to learn 3D reconstruction. Mandikal et al. [28] predict part-segmented 3D reconstructions from a single image. In addition, Luo et al. [27] and Paschalidou et al. [31, 32] learn to form object parts by clustering 3D points. Considering that most 3D approaches require ground-truth 3D shape of whole object or its parts as supervision, Yao el al. [43] propose to discover and reconstruct 3D parts automatically by learning a primitive shape prior. Most of these works assume some form of supervision in terms of 3D shape or camera viewpoint. In this paper, we aim to discover 3D parts of articulated animals from image ensembles without any 2D or 3D annotations, which, to the best of our knowledge, is unexplored the literature.
3D reconstruction from sparse images. Optimizing a 3D scene or object from multi-view images one of the widely-studics problems in computer vision. The majority of recent breakthroughs based on the powerful Neural Radiance Field (NeRF) [30] representation. Given a set of multi-view images, NeRF can learn a neural volume from which one can render high-quality novel views. Closely related to our work, NeRS [45] optimizes a neural surface representation from a sparse set of image collections and also only requires rough camera initialization. In this work, we use a neural surface representation to represent animal parts. Existing methods assume multi-view images captured in the same illumination setting and also the object appearance (texture) be same across images. Another set of work [3, 46, 5] propose neural reflectance decomposition on image collections captured in varying illuminations. But they work on rigid objects with ground-truth segmentation and known camera poses. A concurrent work, SAMURAI [4] jointly reasons about camera pose along with shape and materials of rigid objects from image collections. In contrast, the input for our method consists of in-the-will animal images with varying textures, viewpoints and pose articulations and captured different environments.
3 Approach
Given a generic 3D skeleton and few images of an articulated animal species, LASSIE optimizes the camera viewpoint and articulation for each instance as well the resting canonical skeleton and part shapes that are shared across all instances. We introduce our neural part representation in Section 3.1 and the optimization framework in Section 3.2. An overview of LASSIE is illustrated in Fig. 2.
3.1 Articulated Shapes with Neural Part Surfaces
3D skeleton. Constructing template mesh or statistical shape model for specific animal classes requires either the ground-truth 3D shapes or extensive human annotation, which are both hard to obtain in real-world scenarios. A 3D skeleton, on the other hand, is easy to annotate since it
3

---

Neural Part Representation
(w.r)
Unit sphene
Part MLPs (P2')
features
Input images
Primitive part encodings (e)
Prior MLP
de Jc
(Ax, Ay, de)
Sample texture
LASSIE optimization
Nel
Resting pose
Canonical
Camera space articulation
articulation
Primitive parts
Part octputs
Textured octputs
Figure 2: Neural part surface representation. Based on an optimized 3D skeleton, we reconstruct the articulated shape by optimizing the primitive latent codes and part deformation decoders. The final output is the composition of neural part surfaces with textures sampled from the input images.
only contains a set of sparse joints connected by bones. Furthermore, it generalizes well across classes, and one can effortlessly modify certain joints or bones to adapt to distinct animal types (e.g.. mammals vs. birds). Assuming simple and generic 3D skeleton is given, we propose part-based representation for articulated shapes. A skeleton can be defined by a set of 3D joints P E R and bones B = {PP: P3)}² 1 in tree structure, where p and b p - 1 are numbers of joints and bones/parts, respectively. For instance, we use the same base skeleton for all four-legged animals in our experiments. The base skeleton specifies the connection of 16 joints and the initial joint coordinates in the root-relative 3D space. The skeleton is shared by all instances in the same class and can be optimized by scaling the bone lengths and rotating the bones with respect to the parent joints. Unlike the linear or neural blend skinning models [40, 42] where bones are unconstrained Gaussian components, skeleton- based representation allows to regularize bone transformations, discover meaningful and connected parts, and produce realistic shape animations.
Neural part surfaces. Given skeleton which specifies the 3D joints and bones, we represent the articulated shape as composition of several neural parts. Specifically, each part defined as a neural surface wrapped around a skeleton bone. Motivated by object representation in NeRS [45], we represent part surface as a multi-layer perceptron (MLP) network that predicts the 3D deformation of given continuous UV coordinate on the unit sphere. A key advantage of modeling parts instead of whole animal that part shapes are usually of simple convex shapes and can be represented well with deformed sphere. Compared to explicit mesh representation, the neural representation enables efficient mesh regularization on the output shapes while producing high resolution surfaces as MLPs can work with meshes of arbitrary vertex resolution. Each part surface is first reconstructed in the canonical space, then scaled by the corresponding bone length, rotated by the bone orientation, and centered the bone centroid. This ensures that the parts transformations and also easy to repose. Formally, let X E R3xm denote the m uniformly distributed 3D vertices unit sphere. Further let R, E R333 t; E R and s; E R denote the global rotation, translation and scaling of the jth part (i E {0. ....... b)), where denotes the number of parts. We first deform the sphere obtain part shape using the part MLP, X Fi(X); followed by global transformation to obtain part vertices V, E 3xmm in the global coordinate frame:
connected under various bone
V s; (X) t.
Latent part prior. Although fitting neural part surfaces on sparse images can produce reasonable outputs that are faithful to the input view, we observe that part shapes tend to be non-uniform or unrealistic from other novel views. To further regularize the part shapes, we decompose the part MLP F; into prior MLP, J and part deformation MLP, FAA. The prior MLP is trained to produce the base shapes that are close to geometric primitives such as spheres, cylinders, etc. The deformation MLP is then used to provide additional deformations on top of the base shapes. Fig, 2 illustrates these

---

two MLPs. As shown in the figure, the prior MLP takes as imput the surface coordinate as well as d-dimensional latent shape code e, E E. Both the latent space and the prior MLP are trained using geometric primitive shapes. Concretely, we train Variational Auto-Encoder ( (VAE) which takes a primitive mesh as input, predicts the latent shape encodings and reconstructs the primitive through a conditional neural surface decoder which forms our prior MLP. The primitive meshes in this work are randomly sampled geometrie primitives (spheres, ellipsoids, cylinders, and cones). With the use of prior and deformation MLPs, Eq. now becomes:
Vi = s; s; ei) + F,A (X)) tt.
(2)
where e; denotes the latent shape encodings of part i.
3.2 Discovering 3D Neural Parts from Image Ensemble
Optimization setting. The input to our optimization is sparse set (typically 20-30) of in-the- images {1,} along with generic/rough 3D skeleton P E R2x3 that has joints and bones/parts. The only image-level annotations for optimization are from the self-supervisory DINO [6] features. All the instances in the input image ensemble are of same species, but could have varying appearance due to texture, pose, camera angle, lighting, etc. We also assume that the entire animal body is visible in each image without truncated or occluded by another object although some degree of self-occlusion is allowed. Let denote the index over images j E {1,,, n) and denote the index over parts E {1,., 6). For each image, LASSIE optimizes the camera viewpoint n' (Ro, to) and part rotations R E R6x3x3 In addition, LASSIE optimizes these variables for each of the 8th part that are shared across all the animal instances: bone length scaling s; E R. latent shape encodings e, E Rd and part deformation MLPs F.A. That is, we assume that part shapes are shared across all instances whereas their articulation (pose transformation) is instance dependent. Although this may not be strictly valid in practice, this provides an important constraint for our highly ill-posed optimization problem.
Analysis by synthesis. Since we do not have access to any form of 3D supervision, we use analysis- by-synthesis to drive the optimization. That is, we use differentiable rendering render the optimized 3D articulated shape and define loss functions w.r.t. input images. Using Eq. 2, we first obtain the complete articulated shape for each image as a combination of different part meshes. We then use a differentiable mesh renderer [25] to project the 3D points onto 2D image space via the learnable camera viewpoints. A key challenge that we do not have access to any from of 2D annotations such as keypoints or part segmentation.
Self-superyisory deep features to the rescue. Since different instances in the ensemble can have different texture, we do not model and render image pixel values. Instead, we rely on learned deep features for our self-supervisory that deep features from DINO [6] vision transformer are semantically descriptive, robust to appearance variations, and righeroresolution compared to CNN features [1, 38). In this work, we use the key features from the last layer of the DINO network as semantic visual features for each image. We assume that these DINO features are relatively similar for same parts across different instances. In addition to using DINO features for consistent semantic features across images, we also compute 2D parts and foreground animal mask following similar steps as in [1]. Concretely, we take the class tokens from the last layer, compute the mean attention of class tokens as saliency maps, and sample the keys of image patches with high saliency scores. We then collect the salient keys from all images and apply an off-the-shelf K-means clustering algorithm to form semantic part clusters. Finally, we obtain foreground silhouettes by simply thresholding the pixels with minimum distance to cluster centroids. Fig. 3 shows the DINO part clusters on sample images from different image ensembles. We observe that using 4 clusters works well in practice. With the use of self- self-super features (keys) and rough silhouettes, we alleviate the dependency of ground- -tuth segmentation mask and keypoint annotations in our optimization framework.
maalysi--b--sytthee
framework. Recent works [1, 12] demonstrate
Silhouette loss. We compute the silhouette loss Lmaas by rendering all part surfaces with differ- entiable renderer [25] and comparing with the pseudo ground-truth foreground masks obtained via clustering DINO features: Lmask = Σ, | - .M I|2, where M and AL' are the rendered and pseudo ground-truth silhouettes of instance j. respectively. Although these silhouette-based losses encourage the overall shape to match the 2D silhouettes, their supervisory signal is often too coarse to resolve the ambiguity caused by different camera viewpoints and pose articulations. Therefore, we design novel semantic consistency loss to more densely supervise the reconstruction.
5

---

2D-3D semantic consistency. We propose a novel strategy to make use of the semantically consistent 2D DINO features for 3D part discovery. At a high-level, we impose the 2D feature consistency (HM)
across the images via learned 3D parts. For this, we propose an Sispetation-Maximizatii like optimization strategy where we alternatively estimate the 3D part features (E-step) from image ensemble features and then use these estimated 3D part features to update the parts (M-step). More formally, let KJ E wxx denote the f-dimensional DINO key features from jth image and Q E R=x/ denote the corresponding features on the 3D shape with total vertices. In the E- step, we update Q by projecting the 3D coordinates onto all the 2D image spaces via differentiable rendering under the current estimated shape and camera viewpoints. We aggregate the 2D image features from the image ensemble corresponding to each 3D vertex to estimate Q. In the M-step, we optimize the 3D parts and camera viewpoints using 2D-3D semantic consistency loss Leem- To define Lem,, we discretize the foreground 2D coordinates in each image {p|af' (p) 1}. Likewise, we sample a set of points {o E } on the 3D part surfaces and obtain their 2D projected coordinates T (v). Leem is then defined the Chamfer distance in a high-dimensional space:
Leem = C Σ
min D(p,v) DEVI
Σ min VEVS
D(p.v)). p
(p)=
For each instance j, the distance D between an image point p and 3D surface point e is defined as: D(p, v) (1) + a
(4)
Goometric distance
Semantic distance
where a is a scalar weighting for semantic distance. In effect, Zeem optimizes the 3D part shapes such that the aggregated 3D point features in the E-step would project closer to the similar pixel features in the image ensemble. We alternate between E and M steps every other iteration in the optimization process. In Eq. 4, the image coordinates p and semantic features K, Q are fixed in each optimization iteration, hence Leem can effectively push the surface points towards their corresponding 2D coordinates by minimizing the overall distance. Since this EM-style optimization is prone to local-minima as we jointly optimize camera viewpoints, shapes and pose articulations, we find that good initialization of 3D features is important. For this, we manually map each part index to one of the DINO clusters (4 in our experiments) and then initialize the 3D features Q by assigning the average 2D cluster features to the corresponding 3D part vertices. This manual 3D part to 2D cluster assignment takes minimal effort since it is category-level (all instances share the same 3D parts). Alternatively, one could roughly initialize the camera viewpoints like in recent works such as NeRS [45] which allows the computation of initial Q features from 2D features. Nonetheless, 2D-3D assignment is easier and faster compared to camera viewpoint initialization as 2D-3D assignment only needs to be done once per image ensemble whereas camera initialization is required for each instance in the ensemble. To make the optimization process fully automatic, we also propose to use simple heuristics for the 2D-3D part assignment using intuitions like animal head are usually above the torso and legs at the bottom. We refer to the LASSIE optimization with this heuristic-based 3D feature initialization 'LAsSIE-hcurrstic' and describe the details in the supplemental material.
Comparison related methods. Our use of 2D-3D semantic consistency is closely related to prior works, UMR [24] and BANMo [42]. At a high-level, LASSIE and UMR both start from 2D part segmentations and then lift them onto 3D. key difference is that LASSIE directly discovers parts in 3D using 2D feature consistency, whereas UMR uses 2D part consistency via canonical UV maps. More specifically, UMR uses 2D parts (not 2D features) and 2D UV maps as common canonical part representations. Instead, LASSIE directly optimizes 3D part features with feature based loss (not 2D parts) and LASSIE does not use UV maps like UMR. We only use 2D parts for feature initialization and not during optimization. That is, we discover parts directly in 3D with skeleton constraints, without much reliance on intermediate 2D part discovery. Hence, LASSIE is less sensitive to the issues in 2D part segmentations compared to UMR. BANMo, on the other hand, learns canonical feature embedding by enforcing the consistency between feature matching and geometric warping, which is made tractable by the dense temporal correspondence (optical flow) between video frames. The semantic loss in BanMo enforces the 2D-3D cycle consistency between 2D coordinates and canonical 3D points. Considering that image ensemble is sparse and un-correlated, we coarsely initialize and regularize the 3D surface features at the leve- The proposed semantic loss allows us first localize the 3D parts then refine the detailed part shapes in our challenging setting.
Regularization. Our skeleton-based representation allows us to conveniently impose pose prior or regularization. Specifically, we calculate pose prior loss Epese to minimize the bone rotation

---

deviations from the resting pose as: Epose - E1 RRj?, where R is the bone rotations of instance and R denotes the bone rotations of shared resting pose. We also impose a regularization Lang on joint angles on the animal legs to limit their sideway rotations:
Lang Σ
Σ
R + R
(5)
j EEley bones
where Rig and Rs,s are the rotations of bone i with respect to y and 2 axes respectively. Since the skeleton faces the + direction in the canonical space (+I right, ty up, +2 out), minimizing the y and 2-axis rotations essentially constrains the sideway movements of the bones. Note that both Goose and Lang are generic to all quadrupeds in our experiments, which we find erucial avoid unrealistic poses. Finally, we apply common mesh regularization terms to encourage smooth surfaces, including the Laplacian loss Ltap and surface normal loss Lnorm. Ltap regularizes 3D surfaces by pushing each vertex towards the center of its neighbors, and Lnorm encourages neighboring mesh faces to have similar normal vectors. Note that we apply regularization losses to each part surface individually since the part shapes should be primitive-Hin like and usually do not have sharp or pointy surfaces.
Optimization and textured outputs. The overall optimization objective is:
(6) where { are weighting hyper-parameters. We first pre-train the part prior MLP F with 3D geometric primitives and freeze it during the optimization on given image ensemble. For each image ensemble of an animal species, we perform multi-stage optimization on the camera, pose, and shape parameters until convergence. That is, we update the camera viewpoints and fix the rest first, then optimize the bone transformations, and finally the latent part codes as well as part deformation MLPs. In each iteration, we first update the semantic features of 3D surfaces, then use the updated features to update 3D surfaces, forming an EM-style optimization. We implement the framework in Py PyTorch [33] and update all the learnable parameters using an Adam optimizer [16]. To obtain the final textured outputs, we densely sample vertices from the optimized neural surfaces and directly sample the colors of visible vertices from individual images via 2D projection, where the visibility information is obtained from naive rasterization. For the invisible (self-occluded) vertices, we assign the color by finding their left-right symmetries or nearest visible neighbors. More implementation details can be found in the supplemental material.
Lmaak + Cuem + A2pnsse + Azhang + Aahtap + A5
4 Experiments
Datasets. Considering the novel problem of sparse- immge optimization, we select a subset of the Pascal-Part dataset [7] and collect web images of additional animal classes to evaluate LASSIE and related prior arts. The Pascal-Part dataset contains diverse images several animal classes which are annotated with 2D part segmentation masks. We automatically select images of horse, cow, or sheep with only one object covering 1-50% area of the entire image, and find the keypoints by calculating the centers/corners of ground-truth part masks. In addition, for analysis categories, we collect sparse image ensembles (with CC-licensed images) of some other animals from the internet and manually annotate the 2D keypoints for evaluation. The classes include quadrupeds like zebra, tiger, giraffe, elephant, as well as bipeds like kangaroo and penguin. We filter out the images where the animal body is heavily occluded or truncated, resulting in roughly 30 images per category. The LASSIE optimization and evaluations are performed on each image ensemble separately. The self-collected data does not contain personally identifiable information offensive content, and will be released to public.
more diverse animal
Baselines. Due to the lack of prior works on our problem setting (sparse image optimization for articulated animal shapes), we mainly compare LASSIE with lcarning-basse mesh reconstruction methods. Among these methods, we find 3D Safari [47] and A-CSM [20] to be most comparable to LASSIE since they also model articulation for animal classes of our interest, Other recent mesh reconstruction methods, on the other hand, either cannot handle articulations [21, 24, 11, 36, 44] or assume different inputs [23, 40, 42]. Note that both 3D Safari and A-CSM are trained on large-scale image sets while LASSIE optimizes dataset, we use the released models trained on the closest animal classes. For instance, the zebra model of 3D Safari and the horse model of A-CSM are used to evaluate on similar quadrupeds.
sparse image ensemble. To evaluate their methods on
Visual comparisons. Fig. shows the qualitative results of LASSIE against 3D Safari and A-CSM on the self-collected animal images. The 3D Safari [47] model is trained on zebra images and does
7

---

Input
DINO [1]
3D Safari [47]
A-CSM [20]
LASSIE (part)
LASSIE (texture)
Figure 3: Qualitative results on the in-the-wild image collections. We show example results the baselines as well as the part and texture reconstruction by LASSIE on the self-collected animal image ensembles. The results demonstrate the semantic consistency of discovered parts across diverse classes and the high-quality shape reconstruction that allows dense texture sampling.
Table 1: Keypoint transfer evaluations. We evaluate all the source-target image pairs and report the percentage of correct keypoints under two different thresholds (PCK 0.1/ PCK @.055..
Pascal-Part datasct
Our datasct
Method
Honse
Cow
Sheep
Zebra
Tiger
Giraffe
Elephant
Kangaroo
Penguin
3D Safari [47]
57.8/
63.4/ 50.3
62.6/ 50.5
80.8/ 62.1
63.4/ 50.3
57.6/ 32.5
25.4/
35.57 20.7
49.3/28.9
A-CSM [20]
69.3/ 55.3
68.8/ 60.5
67.4/ 54.7
78.51 60.3
59.1/
52.2
67.3/ 39.5
42.1/ 26.9
53.7/
LASSIE wlo Crem
60,4/45,6
58.91 43.1
55,3//2,3
62.7/47.5
53.61 40.0
54.7/
52.0V 25.6
33.8/ 19.8
50.6/ 25.5
LASSIE w/o JPP
56.1/56.3
69.0V 59.5
68.9/ 52.2
77.0V 60.2
71.5/ 59.2
79.3/ 57.8
66.6 37.1
44.3/ 30.1
63.3/ 38.2
LASSIE
73.0/ 58.0
71.3 62.4
70.8/ 55.5
79.9/
73.3 62.4
80.5/ 60.5
68.7/40.3
47.0/ 31.5
65.5/ 40.6
LASSlE-heuristic
72.1/57.0
69.3/ 61.1
69.7/ 55.3
78.9/ 61.9
71.91 60.0
80.4/ 60.1
66.3/ 38.7
45.6/ 30,9
60.8/ 37.6
not generalize well to other animal classes. A-CSM [20] assumes high-quality skinned model of an object category which enables detailed shapes of animal bodies. However, its outputs do not align well with the given 2D images. Our results demonstrate that LASSIE can effectively learn from sparse image ensemble to discover 3D articulated parts that are high-quality, faithful input images, and semantically consistent across instances. Note that our 3D reconstructions also improve the 2D part segmentation from DINO-ViT feature clustering [1].
Keypoint transfer metrics. As there is no ground-truth 3D annotation in our datasets, we quan- titatively evaluate using 2D keypoint transfer between each pair of images which is standard practice [47, 20]. We map given set of 2D keypoints on a source image onto the 3D part surfaces, and then project them target image using the optimized camera viewpoints, shapes and pose articulations. Since this keypoint transfer goes through the 2D-to-3D and 3D-to-2D mappings, successful keypoint transfer requires accurate 3D reconstruction on both the source and target images. We calculate the percentage of correct keypoints (PCK) under two different thresholds: 0.1 xmxxa (h, w) and 0.05 x.xhhm (h, w). where h and w are image height and width, respectively. Table I shows that LASSIE achieves higher PCK on the Pascal-pant [7] images and most other classes in our in-the-wild image collections. We also show ablative results of LASSIE without using the semantic feature consistency loss Lrem part prior MLP JP, Note that L is essential to fit the skeleton and part surfaces faithfully the input images, and FP provides effective regularization on part surfaces to generate realistic shapes from various viewpoints. LASSIE- heuristic. by using

---

Input
A-CSM [20] DINO [1]
LASSIE
GT (7)
Source
Target
Transferred
Figure 4: 2D segmentations. We show the overall masks (A-CSM) and part masks (DINO clusters, LASSIE results, and Pascal-part part GT) overlaid on the sample input images.
Figure 5: Keypoint transfer results using LASSIE from source target images.
Table Quantitative evaluations on the Pascal-part images. We report the overall foreground IOU, part mask IOU, and percentage of correct pixels (PCP) under dense part segmentation transfer between all pairs of source-target images.
Overall IOU
Part IOU
Part transfer (PCP)
Method
Horse
Cow
Sheep
Horse
Cow
Sheep
Horse
Cow
Sheep
SCOPS [13]
62.9
67.7
63.2
23.0
19.1
26.8
DINO clustering [1]
81.3
85.1
83.9
26.3
21.8
30.8
3D Safari [47]
72.2
71.3
70.8
71.7
69.0
69.3
A-CSM [20]
72.5
73.4
71.9
73.8
71.1
72.5
LASSIE w/o Leem
65.3
67.5
60.0
29.7
23.2
31.5
62.0
60.3
59.7
LASSIE w/o FFP
81.0
86.5
85.0
37.1
33.7
41.9
76.1
75.6
72.4
LASSIE
81.9
87.1
85.5
38.2
35.1
43.7
78.5
77.0
74.3
automatic 3D feature initialization, achieves slightly lower PCK but still performs favorably against prior arts overall. Example visual results of keypoint transfer are shown in Fig. 5.
2D overall/part IOU metrics. Our part-based representation can generate realistic 3D shapes and can even improve 2D part discovery compared to prior arts. In Table and Fig. 4, we show the qualitative and quantitative comparisons against prior 3D works as well as 2D co-part discovery methods of SCOPS [13] and DINO clustering [1]. For SCOPS [13] and DINO clustering [1], we set the number of parts N, 4 (which we find to be optimal for these techniques) and manually assign each discovered part to the best matched part in the Pascal-part annotations. Note that higher part IOU against Pascal-par -part annotations may not strictly indicate better part segmentation since the self-discovered parts need not correspond to human annotations. Compared to prior arts on articulated shape reconstruction, LASSIE results match the Pascal-pan segmentation masks better and achieves higher overall IOU. On co-part segmentation, our 3D reconstruction captures the semantic parts in 2D while being more geometrically refined than DINO clustering. Moreover, our results can separate parts with similar semantic features, e.g., animal legs, and thus resulting in much higher part IOU.
Part transfer. While keypoint transfer and part IOU are commonly used in the literature, we observe that they are either sparse, biased towards certain body parts (e.g., animal faces), or not directly comparable due to part mismatch. To address this issue, we propose part transfer metric for better evaluation of 3D reconstruction consistency across image ensemble. This part transfer metric is similar to keypoint transfer but uses part segmentations. That is, densely transfer the part

---

Pose transfer/ motion re-targeting (top: target pose, bottom: re-posed output)
Texture transfer
(top: target texture, bottom: re-textured output)
Figure 6: Applications of neural part surfaces. We use the LASSIE results zebra images (top) as target and apply pose transfer (bottom left) and texture transfer (bottom right) to the giraffe shapes. These applications demonstrate the realistic part surfaces discovered by LASSIE.
segmentation in source image target image through forward and backward mapping between the 2D pixels and the canonical 3D surfaces. A pixel is considered transferred correctly if and only it is mapped to the same 2D part in the target image where it belongs in the source image. Similar to PCK in keypoint transfer evaluation, we calculate the percentage of correct pixels (PCP) for each source-target pair. Since the part segmentation only covers the visible surfaces and not the occluded regions, we ignore the pixels mapped to a self-occluded surface when calculating the PCP metric. The quantitative results shown in Table 2, which demonstrate the favorable performance of LASSIE against prior arts.
Applications. Our 3D neural part representation not only produces more faithful and realistic shapes but also enables various applications like part manipulation, texture transfer, motion re-targeting, etc. Specifically, we can generate new shape by swapping or interpolating certain parts, transfer the sampled surface texture from one object class to another using the same base skeleton, or specify the bone rotations to produce desirable re-posed shape or animation. We show some example results of pose transfer and texture transfer between different animal classes in Fig.
Limitations. LASSIE relies heavily on the 3D skeleton and strong pose and shape regularization. As the first attempt address this novel and highly ill iol-posed problem, we find the proposed pose and shape constraints essential to produce good quality results and avoid unrealistic outputs. Moreover, LASSIE can not handle image collections with heavy truncation, occlusions, or extreme pose variations since we leverage self-supervisory DINO- ViT features sparse image ensemble as our main supervision. For instance, partial-body or noisy DINO features can cause ambiguities in camera pose and part localization. Finally, we observe that LASSIE currently struggles with a) highly articulated parts like elephant trucks and b) fluffy animals that appear with more instance variations and ambiguous articulations.
5 Conclusion
In this paper, we study novel and practical problem of articulated shape optimization of animals from sparse image collections in-the-wild. Instead of relying on pre-defined shape models human annotations like keypoints segmentation masks, we propose neural part representation based on generic 3D skeleton, which is robust to appearance variations and generalizes well across different animal classes. In LASSIE, our key insight is to reason about animal parts instead of whole shape as parts allow imposing several constraints. We leverage self-supervisory dense ViT features to provide both silhouette and semantic part consistency for supervision. Both quantitative and qualitative results on Pascal-Pai -arrt and our own in-the-wild image ensembles show that LASSIE can effectively Iearn from few in tn-the-wild images and produce high-quality results. Our part-based representation also enables various applications such as pose interpolation, texture transfer, and animation. We hope that this work will facilitate future research efforts on learning, inferring, and manipulating articulated shapes from image ensembles in the wild.
10

---

References
[1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep VIT features as dense visual descriptors. arXiv preprint arXiv:2112.05814 2021. 3, 8,8,9
[2] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In ECCV, pages
561-578, 2016. 1
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. NeRD:
Neural reflectance decomposition from image collections. In CVPR, pages 12684-12694, 2021. 3
[4] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen I.,, Deqing Sun, Jonathan T Barron, Hendrik Lensch, and Varun Jampani. SAMURAT: Shape and material from unconstrained real-world arbitrary image collections. NeurIPS, 2022.
[5] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-PII.: Neural pre-integrated lighting for reflectance decomposition. NeurIPS, 34, 2021. 3
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- supervise vision transformers. In ICCV, pages 9650-9666) 2021. 2, 3,
[7] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pages 1971-1978, 2014. 2, 7, 8,
[8] Subhahrata Choudhury, Tro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. NeurlPS, 14:28104-281 18, 2021. 3
[9] Edo Collins, Radhakrishna Achanta, and Sahine Susstrunk. Deep feature factorization for concept discovery.
In ECCV, pages 336-352,, 2018. 3
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19929, 2020. 2 [1I] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In ECCV, pages 804, 2020. 7.7
[12] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In ICLR, 2022. 5
[13] Wei-Chil Hung. Varun Jampani. Sifei Liu, Pavlo Molchanov, Ming-HHuan Hsuan Yang. and Jan Kautz. SCOPS:
Self-supervised co-part segmentation. In CVPR, pages 869-878, 2019. 3,
[14] Catalin lonescu. Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.66 6.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. PAMI, 36(7):1325-133) 2013. I
[15] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human
shape and pose. In CVPR, pages 7122-7131, 2018. 1
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900, 2014.7 7
[17] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. VIBE: Video inference for human body
pose and shape estimation. In CVPR, pages 5253-5263, 2020. 1
[18] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis, Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, pages 2252-2261 2019.
[19] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single- image human shape reconstruction. In CVPR, pages 4501-4510 2019. I
[20] Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation aware canonical
surface mapping. In CVPR, pages 261, 2020. 3, 7, 8,
[21] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In ICCV, pages 2202 2211, 2019. 3, 7
[22] Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe, et al. Motion- supervised co-part segmenta-
tion. arXiv preprint arxiv:2004.003 033.34, 2020. 3
[23] Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang. and Jan Kautz. Online adaptation for consistent mesh reconstruction in the wild. NeurlPS, 33:1 53:99500-15119 2020. 7 [24] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang. and Jan Kautz. Self-supervised single-view 3D reconstruction via semantic consistency. In ECCV, pages 677-693, 2020.
3, 6, 7
[25] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based
3D reasoning. CVPR, pages 7718-7717, 2019. 5
[26] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. TOG, 3466):1116 2015. 1.3
[27] Tiange Luo, Kaichun Mo, Zhiao Huang. Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning to group:
A bottom-up framework for 3D part discovery in unseen categories. arXiv preprint arXiv:2002.06447,
2020. 3
11

---

[28]
Priyanka Mandikal, Navaneet KL, and R Venkatesh Babu. 3D-PSRNet: Part segmented 3D point cloud reconstruction from single image. In ECCV, pages 0-0, 2018. 3
[29]
Dushyant Mehta, Oleksandr Sotaychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons- Moll, and Christian Theobalt. Single-shot multi-person 3D pose estimation from monocular RGB. In 3DV,
pages 120-130, 2018. 1
[30] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020. 3
[31] Despoina Paschalidou, Luc Van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition of 3D objects from a single RGB image. In CVPR, pages 1060-1070, 2020.
[32] Despoina Paschalidou. Angelos Katharopoulos, Andreas Geiger. and Sanja Fidler. Neural parts: Learning expressive 3D shape abstractions with invertible neural networks. In CVPR. pages 3204-3215, 2021.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chaman, Trevor Killeen, Zeming Lin, Natalia Gimelshein. Luca Antiga, al. PyTorch: An imperative style. high-performance deep learning library. NewrIPS, pages 8024-8035, 2019. 7
[34] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani. Timo Bolkart, Ahmed AA Osman. Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from single image. In CVPR. pages 10975-10988 2019.
[35] Nadine Rueege, Silvia Zuffi, Konrad Schindler, and Michael J Black. BARC: Learning to regress 3D dog
shape from images by exploiting breed information. arXiv preprint arXiv:2203. 15536, 2022. 3
[36] Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unannotated image collections. arXiv preprini urXiv:2007.008: 005544 2020. 3.
[37] Shubham Tulsiani. Hao Su, Leonidas Guibas, Alexei A Efros, and Jitendra Malik. Learning shape
abstractions by assembling volumetric primitives. In CVPR, pages 2635-2643, 2017. 3
[38] Narek Tumanyan, Omer Bar- Tal, Shai Bagon, and Tali Dekel. Splicing VIT features for semantic appearance transfer. arXiv preprint arXiv:2201 000004.00444 2022. 2,5
[39] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recov- ering accurate 3D human pose in the wild using IMUs and moving camera. In ECCV, pages 601-617.
2018.
[40] Gengshan Yang. Deqing Sun, Varun Jampani. Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. LASR: Learning articulated shape reconstruction from monocular video. CVPR, pages 15980-15989, 2021. 3. 7.
[41] Gengshan Yang. Deqing Sun, Varun Jampani, Daniel Vlasic. Forrester Cole, Ce Liu, and Deva Ramanan. VISER: Video-specific surface embeddings for articulated 3d shape reconstruction. NeurlPS, 34, 2021. [42] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. BANMo: Building animatable 3D neural models from many casual videos. arXiv preprint arXiv:2112.12700 2021.
3.4, 6.7
[43] Chun-Han Yao, Wei-Chih Hung, Varun Jampani, and Ming-Hsuan Hsuan Yang. Discovering 3d parts from image collections. In ICCV, pages 12981-12990, 2021. 3
[44] Yufei Ye. Shubham Tulsiani, and Abhinav Gupta. supee-supe mesh prediction in wild. In CVPR. pages 8843-8852, 2021. 7.7
[45] Jason Zhang. Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. NeRS: Neural reflectance surfaces for sparse-view 3D reconstruction in the wild. NewrIPS, 34, 2021. 3, 6
[46] Kai Zhang. Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting. In CVPR, pages 5453-5462, 2021. [47] Silvia Zufti, Angjoo Kanazawa, Tanya Berger- Wolf, and Michael J Black. ThreeD Safari: Learning to estimate zebra pose, shape. and texture from images "in the wild". In ICCV, pages
2019. 3, 7,
8.9
[48] Silvia Zufti, Angjoo Kanazawa, and Michael J Black. Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images. In CVPR, pages 3955-396 2018 1,
[49] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael Black, 3D menagerie: Modeling the shape and pose of animals. In CVPR, pages
2017. 3
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] proprictary.
The code and the data are
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] Since the scope of our paper is within 3D reconstruction of animal bodies, there is no potential negative social impact that are aware of.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] In our optimization framework, the random seed initialization does not affect the outputs much since the most sensitive parameters (e.g.
camera, bone scaling, pose) are not randomly initialized.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "qTHSMJFN7Gh", "reviewer": "Reviewer_LjEQ", "summary": "This paper presents an optimization-based methods that reconstructs 3D part based articulated shapes of animals from only a small collection of in-the-wild images (~30 instances) of different instances of an articulated object category, eg, horse, giraffe, elephant, penguin etc. The key idea is leverage self-supervised pretrained ViT features (DINO) to establish semantic correspondences between the images of different instances with various camera viewpoints, pose articulations, texture and environment. By constraining the 3D shapes with a set of pretrained part primitives (ellipse, cylinder, cone) on a pre-defined 3D skeleton (eg, quadrupedal or bipedal), the model is able to reconstruct compelling articulated 3D shapes of different instances, significantly better than existing learning-based models trained on only one specific category (zebra or horse).", "questions": "Apart from a few questions that need clarifications listed above, the only additional result I might suggest is to also perform test-time optimization with previous methods for a slightly fairer comparison, maybe in the final version. The settings are different enough though, so this may not really affect the evaluation of the paper.", "limitations": "The paper has included a discussion on the limitations of the model as well as illustrative examples in the supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n### S1 - Compelling results on an interesting and challenging task\n- The task of reconstructing articulated 3D animals from only a small collection of in-the-wild images of various articulated instances is highly ill-posed and very challenging. The proposed method seems to work well on a diverse set of examples (9 different animals demonstrated).\n- Based on the examples shown in Fig 9 in the supplementary material, the images in each category collection are also very diverse in terms of camera viewpoints, poses, and environment and illumination conditions.\n\n### S2 - Good ideas and careful implementations\n- There are two main ideas. One is to leverage self-supervised pretrained image features (DINO ViT) to establish semantic correspondences between different image instances via simple clustering. This is more powerful than previously studied SCOPS which still requires ImageNet pretraining and large training datasets.\n- The second idea is to use part-based 3D representation with a pre-defined generic 3D skeleton. This significantly reduces the complexity of this ill-posed task, and yet requires only minimal priors.\n- These two ideas are carefully implemented and validated through ablation studies. In particular, the use of an neural surface representation and VAE pretraining of the 3D parts seems quite effective in achieving more regularized shapes.\n\n### S3 - Good writing\n- The paper is very well written, with clear motivations, sufficient technical explanations and illustrative visualizations.\n\n\n## Weaknesses\n### W1 - Optimization-based method\n- The method optimizes over a small collection of images and does not generalize to new images out of the box. It would also be interesting to extend the idea to a learning-based pipeline that would allow inference on novel images.\n- This also makes the comparison against previous learning-based methods unfair, as the results of other methods (I assume) are obtained by one forward pass without any finetuning. It would be slightly fairer to also perform some test time finetuning on the other methods.\n\n### W2 - Heavy regularization\n- Due to the ill-posedness of the task, the model relies on a number of heavy constraints, including a simplistic part-based shape representation which also requires pretraining using a set of primitives. This largely limits the expressivity of the model and hence does not lead to fine geoemtric details.\n- The need of hand-crafted 3D skeleton is also a major limitation. Although it may seem effortless on one category, it is still quite cumbersome to scale up to all kinds of objects. Moreover, the model only works well on animals that have limited articulated poses, where the skeleton is clearly visible. I can imagine that it would still struggle for animals that highly articulated, like cats (in which case, only a small number of images may not be sufficient).\n- There are also a number of specific constraints in the model, eg, minimizing the sideway rotation angles on the leg bones, and a specific optimization procedure that alternates between viewpoints, bones, parts and features.\n\n\n### (Minor) Clarifications\n- Are the part prior latents $\\mathbf{e}_i$ randomly initialized and optimized for each part during full training (after VAE pretraining)?\n- I am confused by the visualization of the part primitives in Fig 2. Aren't the output of the neural surface networks $\\mathcal{F}_i$ supposed to be elongated shapes in the canonical space (without bone orientations)? Fig 2 seems to suggest that the part shapes predicted by the networks are already oriented.\n- Line 257: the optimization procedure seems to involve three steps: 1) camera viewpoints, 2) bone transformations, 3) latent part codes and part deformations. Is that correct? What about the 3D part features? When are they updated (the E step)?\n- It seems pretty easy for the model to confuse head and tail by looking at only the 4 clusters visualized in the zebra example in Fig 2. Since camera viewpoint is optimized in the first step, will it get confused initially and fall into local minima. And if so, how often does it occur?\n- Were the 2D keypoints used in keypoint transfer evaluation manually labelled?\n\n\n### (Minor) Typos\n- Line 146: I believe $k$ should be $b$.\n- Line 171: add $t^j \\in \\mathbb{R}^{b \\times 3}$ for completeness. Also, make the notations consistent -- $\\mathbf{t}$ vs. $t$.\n- Line 286: fix typos.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ecz3KPbPn_O", "reviewer": "Reviewer_rsf7", "summary": "This paper studies self-supervised learning of articulated shapes under a very challenging setting where only a few (~30) in-the-wild images of an animal category are available. This setting is much more challenging than prior art as silhouettes and templates are not used and the number of images used for training is very few. To make this possible, the authors propose to model articulated objects with a 3D skeleton, and each part is modeled with a simple shape. The model is trained via analysis by synthesis, where DINO features are used to provide silhouette and 2D-3D semantic consistency. Experiments on Pascal Part and self-collected web images demonstrate that the proposed method LASSIE achieves considerably better 3D reconstructions and 2D&3D part discovery compared to baselines.", "questions": "Please clarify why [39] and [17] are selected instead of the newer ones as mentioned in weaknesses.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method LASSIE is novel and reasonable. The design of modeling each part using simple shape primitives effectively regularizes the solution space and makes the problem more tractable. The 2D-3D semantic consistency loss makes use of the semantically consistent DINO features to help discover 3D parts. The regularization and training procedure are reasonable. I believe these new insights and designs are valuable to the community.  \n\n2. Thanks to the proposed method, it is the first time that the self-supervised articulated shape learning task can be addressed under such a challenging setting where only a few (~30) in-the-wild images are available. LASSIE also considerably outperforms previous methods, and has broad applications such as semantic part refinement, pose/texture transfer.  \n\n3. The paper is well written.  \n\n\nWeaknesses:\n\nThis paper is generally fine. The following weaknesses are not significant.\n\n1. In line 278, it is mentioned that there exists no closer framework other than 3D Safari [39] and A-CSM [17]. However, it seems to me that these are not the state-of-the-art method for self-supervised articulated shape learning. For example, the following papers are later than the two baselines. Please clarify why [39] and [17] are selected instead of these newer ones.  \n[1] Tulsiani et al, Implicit Mesh Reconstruction from Unannotated Image Collections, NeurIPS2020.  \n[2] Ye et al, Shelf-Supervised Mesh Prediction in the Wild, CVPR2021.  \n[3] Li et al, Self-supervised Single-view 3D Reconstruction via Semantic Consistency, ECCV2020.  \n\nTypos and grammar issues:  \nLine 12: We -> we.  \nLine 146: k denotes -> b denotes.  \nLine 198: The use of ... -> Through the use of ... or By using ...  \nLine 223: a image -> an image.  \nLine 286: an object category is able to produce detailed shapes of animal bodies which -> an object category which is able to produce detailed shapes of animal bodies.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ASzaLqRnCmD", "reviewer": "Reviewer_boNQ", "summary": "The paper proposes a new task formulation in the area of non-rigid reconstruction and parts discovery. The model is fit to a small collection of images (25–30) of unrelated instances of a category (such as animal species). It enables discovering 3D pose and shape of objects in all images, along with part segmentation. No supervision on the level of images is required, however the users have to provide the topology and a typical pose of the skeleton, assignment of image parts to image features’ clusters, and also there are specific constraints on the movement of certain joints (such as legs in case of quadrupeds). The model produces better PCK numbers than competing methods (Articulated CSM and 3D Safari) and visually convincing reconstructions on the examples shown.", "questions": "Major issues:\n* I suggest to allocate more text to position the method w.r.t. UMR [19] more clearly;\n* I suggest to evaluate reconstruction quality on datasets with 3D ground truth (see above);\n* can you clarify the comparison protocol with the baselines: are they evaluated on the hold-out evaluation set (vs. the proposed method that fits the model to all available images as far as I understood)? \n* please address the limitations below.\n\nOther issues:\n* In Chamfer loss (3), is the closest point chosen using the distance (4) rather than just geometric distance? If so, what is the motivation of that? I feel like the loss should enforce points to reproject in the correct locations; matching the features should not compensate for that.\n* in Experiments, baselines sometimes are trained using a different category; please indicate clearly in the results table where it is the case,\n", "limitations": "The paper lists some limitations, but crucially misses some others:\n* Can the method generalise to new images, i.e. can it infer the pose for a new instance without re-fitting the model? The competing methods only need to run the forward pass. That may be important for interactive applications.\n* The method uses the same shapes of the individual parts across instances (ll. 172–174), so the resulting meshes differ only in the parts’ rigid transformations. This is an approximations since in reality individuals differ in the bone lengths and deformation of their flesh.\n* Another consequence of that is, if I understand correctly, reposing the shape can result in gaps between the parts. In fact, lack of those gaps is guaranteed for training instances only thanks to rendering losses; this means that the invisible surface may not be represented properly,\n* the loss L_pose penalises the difference from the rest pose; it is usually difficult to find the weight of such loss so that the method is able to fit extreme poses,\n* the need for heuristics like L_ang (5) limits the applicability of the method to new categories where users would need to figure out similar constrints.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n* I see the method as an extension of UMR [19] to articulated shapes, which are naturally represented as a union of convex’ish primitives, where SCOPS features are upgraded to DINO. Extending to articulated shapes requires overcoming some technical challenges, such as adding regularisations and priors.\n\nClarity:\n* (+) In general, the paper is well written, and the method is mostly clear,\n* (−) some of the limitations are either not addressed or not mentioned in the introduction (such as the required category-level supervision), while l. 51 claims that the method is “completely self-supervised”,\n* (−) I am confused about the final optimisation algorithm; it runs block-coordinate optimisation (l. 256 and on), but then there is EM-like optimisation of features inside one of the steps; I think the paper would benefit from pseudo-code of the training loop,\n\nSignificance:\n* (+) getting rid of instance-level supervision is a great step; the method can scale to larger collections easily;\n* (+) the required prior information is indeed easier to obtain than e.g. category-specific mesh.\n\nExperiments:\n* (+) visual demonstrations look good;\n* (−) is the EM-like optimisation important? What would be the result if the original DINO features were used in (4)? (I think it corresponds to 1 iteration of “EM”);\n* (−) evaluating using 2D keypoint transfer may be used by other papers but it does not tell much about reconstruction quality; representing the animal’s shape with a flat surface would result in a relatively good PCK while such reconstruction is visually unacceptable. I suggest evaluating on datasets where 3D ground truth is available, such as synthetic datasets or human datasets (such as Human 3.6M) – without expectation of performing better than methods specialising on human reconstruction,\n* (+) Applications paragraph/results are nice.\n\n=============\n\nTypos:\n* line 80: In contrasted ← in contrast;\n* line 146: *k* ← *b*;\n* line 211–213: why is it called EM-like algorithm? I would call it block-coordinate descent; for EM, I would expect having an explicit likelihood function depending on latent random variables, and taking expectation over those at E-step,\n* line 285: “A-CSM assume high-quality skinned model”: IIRC, it only requires the mesh segmented to parts but no skinning weights, which is a much lighter assumption.\n* line 286: “animal bodies which”.\n\nUPD. The other reviewers pointed me towards the supplementary video that I had previously overlooked. It reassured me in the reasonable reconstruction quality, at least on the shown examples. I like the new problem setting and the method. My only remaining (but major) concern is the presentation and positioning of the method and the results: in the current manuscript, the relation to UMR (which is strong, at least on the ideas level) is essentially ignored; Experimental section wrongly claiming conclusive quantitative evaluation of 3D reconstruction; the limitations are not addressed, and the Introduction / abstract massively oversell the method (which it ironically does not even needs). I am changing my rating to borderline to not block the acceptance; I will thus leave it to ACs to decide whether we can accept the paper in this form.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9czt0pTJERN", "reviewer": "Reviewer_mH51", "summary": "The submission propose Lassie to obtain articulated 3D reconstructions of a skeleton-based animal class (shown are quadrupeds and bipeds) from an image collection of 20-30 images of different individual animals. Given a fairly generic animal skeleton (16 vertices and 15 edges), it learns a 3D part for each edge/bone. The 3D part is represented by regressing 3D offsets for a 3D unit sphere via a coordinate-based MLP. The shapes of the part are shared across instances, only a similarity transform is applied based on the skeleton of each instance. The method requires only little outside knowledge: It clusters DINO-ViT [5] image features to obtain a very rough 2D part annotation of the images, and it requires manual matching between these part annotations and the skeleton parts. The optimization determines instance-specific skeleton parameters and class-specific part parameters. The main loss is a novel semantic matching loss that has some similarity to the one used in BanMo since it transfers image features to the 3D reconstruction and then tries to match the reconstruction to the images using those features. \n\nThere is no prior work that tackles this problem setting. Comparisons to A-CSM and 3D Safari show better results for Lassie, in several distinct settings.", "questions": "Questions\n\n* The semantic loss is interesting. It would be nice to see a discussion on how it differs from BanMo's loss (see summary above).\n* I would like the authors to comment on why comparisons against CMR, CSM, U-CMR, VMR, etc. (see Weaknesses) is not necessary, e.g. because there is simply no code. \n* (Almost?) all methods that follow CMR (reconstructing animals/dynamic objects from image ensembles) evaluate on the CUB-200-2011 birds dataset. Why is there no evaluation on birds? Is there too much variablility? A-CSM [17], against which the submission compares, also evaluates on this dataset.\n* The checklist states that error bars are provided. But I do not see any numbers or any other indication that multiple runs/seeds were used anywhere. Am I overlooking something or is this answer in the checklist wrong?\n\n* While not crucial, the ablation on the number of images per animal in supplemental Sec. 2.2 uses the zebra class, which seems like a quiet easy class (see e.g. Fig. 9 in the supplement; all zebras have a pretty similar pose). I am curious how well a reduced number of images would work for giraffes or kangaroos or maybe horses, especially when keeping diverse poses and appearances in that subset of images.\n\nSuggestions (do not need to be addressed in the rebuttal but should be considered for a revised version of the paper):\n\n* I am not really satisfied with saying that no annotations are used (e.g. L.166). The DINO features give rise to 2D part segmentation masks. DINO is self-supervised, but still. \n* While I find the evaluation sufficient already, results on humans would be interesting. \n* The conclusion claims that no human annotations (L.342f) is used, but that's not correct (L.231ff). Please make sure that claims about annotations are precise everywhere.\n* Please provide the mathematical formulation of L_lap and L_norm (and all other terms that might not be defined) in the supplementary material.\n* In Fig. 10 and 11 in the supplemental document, please make the novel views of the same size as the input views.\n* The current uniformly colored visualization of the parts shows coarse-level correspondences across instances. It could be interesting to color each part with some kind of pattern (e.g. a rainbow or color gradient) to visualize fine-level correspondences.\n* L.167f says that the instances can have \"varying texture (appearance) due to pose, camera angle, and lighting\". What about the individual identity, i.e. different individuals have different fur? The method seems to not have an issue with it, I think it should be added to that list.\n* L.256ff state that alternative optimization of camera, pose, and shape is used. Does that mean that the camera is optimized at the beginning of training and then never again? Or is it camera, then pose, then shape, then camera, then pose, then shape, etc.? If it's the former, maybe calling it training stages instead of alternative optimization would be more descriptive.\n* What's the value of m in L.215?\n* Please render the images in a higher resolution. They are quite pixelated already.", "limitations": "The checklist states that potential negative societal impact is discussed. While I don't see any such potential negative impact, I don't believe that that is discussed anywhere in the paper, despite what the checklist says.\n\nLimitations are discussed, including failure cases in the supplement.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=====STRENGTHS=====\n\nThe problem setting is interesting and it is remarkable that the proposed method takes only 10 minutes on a consumer GPU to obtain quite decent articulated reconstructions from only 20-30 images of an animal class. This opens up the door for a lot of follow-up work and raises a bunch of interesting questions, although unfortunately code will not be released.\n\nWhile there are an MLP for the part shape and DINO image features, the majority of the method is not much concerned with neural networks, which is a nice change of pace.\n\nThe method is not exactly simple but also far from being convoluted. Its complexity seems appropriate to me and the design choices are argued for in the paper. \n\nThe evaluation is done well, including additional experiments in the supplement.\n\nThe paper is well written. The presentation is nice, including clear tables and figures.\n\nI appreciate the name of the method.\n\n=====WEAKNESSES=====\n\nThere are no significant weaknesses I can see. I do have a number of questions and suggestions though, see below.\n\nMinor writing issues:\n* L.12: \"We\" should be lower case\n* L.30: \"a\" -> \"the\"\n* L.146: \"k\" -> \"b\"\n* L.286: sentence ends in \"which\"\n\nRelated Work:\n\n* Missing related works that are from the same line of work as A-CSM [17]: Kanazawa et al. Learning category-specific mesh reconstruction from image collections (CMR; ECCV 2018) and its follow-up works, see e.g. Table 1 in Wu et al. DOVE: Learning Deformable 3D Objects by Watching Videos (arxiv preprint 2021):\n\n- Kulkarni et al. Canonical surface mapping via geometric cycle consistency (CSM; ICCV 2019), \n- Goel et al. Shape and viewpoints without keypoints (U-CMR; ECCV 2020), \n- Li et al. Self-supervised single-view 3D reconstruction via semantic consistency (UMR; ECCV 2020), \n- Li et al. Online adaptation for consistent mesh reconstruction in the wild (VMR; NeurIPS 2020),\n- Kokkinos et al. Learning monocular 3D reconstruction of articulated categories from motion (CVPR 2021),\n- Kokkinos et al. To The Point: Correspondence-driven monocular 3D category reconstruction (NeurIPS 2021)\n\n* Less crucial, several animal shape estimation methods that use SMAL or similar statistical models are not cited:\n\n- Biggs et al. Creatures great and SMAL: Recovering the shape and motion of animals from video (ACCV 2018)\n- Biggs et al. Who left the dogs out: 3D animal reconstruction with expectation maximization in the loop (ECCV 2020)\n- Badger et al. 3D bird reconstruction: a dataset, model, and shape recovery from a single view (ECCV 2020)\n- Wang et al. Birds of a feather: Capturing avian shape models from images (CVPR 2021)\n\n* The early important work of Cashman et al. What shape are dolphins? Building 3d morphable models from 2d images (TPAMI 2013) is also not cited.\n* Another work that reconstructs dynamic objects from in-the-wild image collections is Wu et al.'s Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild (CVPR 2020 best paper).\n* While NeRF and a couple static variants are mentioned, the NeRF field has also produced some works on dynamic reconstruction of general objects from video (not image collections; although see Banmo). These could be added for completion, see e.g. the recent survey Tewari et al. Advances in Neural Rendering (Eurographics 2022).\n\n\n---- post rebuttal\n\nMy rather minor concerns and questions have been addressed well. I increase my rating to accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept