LEPARD: Learning Explicit Part Discovery for 3D Articulated Shape Reconstruction
Authors: Di Liu
Anastasis Stathopoulos
Qilong Zhangli
Yunhe Gao
Dimitris N. Metaxas
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D computer vision
deep learning

Abstract: Reconstructing the 3D articulated shape of an animal from a single in-the-wild image is a challenging task. We propose LEPARD, a learning-based framework that discovers semantically meaningful 3D parts and reconstructs 3D shapes in a part-based manner. This is advantageous as 3D parts are robust to pose variations due to articulations and their shape is typically simpler than the overall shape of the object. In our framework, the parts are explicitly represented as parameterized primitive surfaces with global and local deformations in 3D that deform to match the image evidence. We propose a kinematics-inspired optimization to guide each transformation of the primitive deformation given 2D evidence. Similar to recent approaches, LEPARD is only trained using off-the-shelf deep features from DINO and does not require any form of 2D or 3D annotations. Experiments on 3D animal shape reconstruction, demonstrate significant improvement over existing alternatives in terms of both the overall reconstruction performance as well as the ability to discover semantically meaningful and consistent parts.

LEPARD: Learning Explicit Part Discovery for 3D Articulated Shape Reconstruction
Di Liu Qilong Zhangli
Yunhe Gao Rutgers University
Dimitris N. Metaxas
Abstract
Reconstructing the 3D articulated shape of an animal from single in-the-wild image is a challenging task. We propose LEPARD, a learning-based framework that discovers semantically meaningful 3D parts and reconstructs 3D shapes in a part-based manner. This is advantageous as 3D parts are robust to pose variations due to articulations and their shape is typically simpler than the overall shape of the object. In our framework, the parts are explicitly represented as parameterized primitive surfaces with global and local deformations in 3D that deform to match the image evidence. We propose a kinemattcs-inspire transformation of the primitive deformation given 2D evidence. Similar to recent approaches, LEPARD is only trained using off-the-shelf deep features from DINO and does require any form of 2D or 3D annotations. Experiments on 3D animal shape reconstruction, demonstrate significant improvement over existing alternatives in terms of both the overall reconstruction performance as well the ability to discover semantically meaningful and consistent parts.
optimization to guide each
Introduction
Predicting the 3D shape and part articulation of an object from a single image is a severely under- constrained and challenging problem. It can be applied to many downstream tasks, such as shape reconstruction [45, 61, 47, 50, 54, 391, segmentation [24, 51, 32, 72, 16, 33, 6, 71], editing [65, 21], re-targeting [12. 20, 69] and medical imaging applications [22, 34, 36, 17, 37, 35, 14, 23, 18, 44]. Successful approaches [26, 29] for predicting the 3D shape of humans rely on parametric human body model (e.g., SMPL [41]) built from thousands of mocap sequences and on strong supervision from 3D joint locations. Similar breakthroughs are not seen for other articulated object categories, like animals, 3D scanning of such categories is quite challenging. The lack of 3D annotations and appropriate parametric animal model has led to approaches that utilize pre-defined shape template and train with 2D supervision [27,
[27,31, 31, 30, 19, 28]. Assuming such a fixed shape template and supervision from 2D annotations is not optimal, but is necessary for training these systems. Recent works [67, 68, 60] have discarded both assumptions by learning shape prior (part-based [67, 68] or holistic shape [60]) and by using deep features from an off-the-shelf vision transformer [13], DINO- ViT [S]. for supervision. Although the approaches presented [67, 68, 60] seem promising. they exhibit certain limitations. In [60] siteeoor--spccffit prior mesh is obtained in pre-training step, which can be seen as a learned shape template. Then it is trained to predict the articulation and deformation based on this fixed template. LASSIE [67] and Hi-LASSIE [68] learn generic part priors that, similar to [60], can be articulated and deformed. However, learning the deformation field w.r. weak generic shape priors is hard task. Assuming that the prior shape far from the target shape of an object, the model has to compensate by predicting large deformations to match the image evidence.
In this paper, we propose a comprehensive framework for reconstructing the 3D shape and related articulations of object from single-view images that has several desired properties missing from
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

existing works. Similar to [67, 68], the 3D shape of an object is explicitly expressed as a set of part primitives. The primitives are parameterized surfaces (e.g., superquadrics) equipped with additional linear tapering and bending transformations to capture the target shape as faithfully as possible. Unlike [67, 68], the 3D primitives are not fixed across all instances and can deform to capture intra-category variations and accurately reconstruct 3D parts. To capture fine-grained shape details beyond the coverage of the primitive parametric deformations (termed global deformations), we employ diffeomorphic mapping to estimate local non-rigid deformations of a set of points sampled from the 3D surface of each part. Our approach uses global deformations to capture the salient part of the 3D shape and uses local deformations to further improve the 3D shape reconstruction quality. As such, the local deformations are typically small, which adds to the robustness of our method. Following prior work [66, 68, 60] we use deep features from vision transformer [13], DINO- ViT [5], as supervision to train our model. Inspired by the kinematics of 3D deformable models [57], we propose framework compute amage-based forces based the discrepancy of DINO features and the projected primitive parts. The image-based forces are then converted to generalized forces via kinematics modeling, which provides strong supervision for the deformation of the 3D part primitives.
We conduct extensive experiments on 3D articulated object shape reconstruction through part dis- covery. The Pascal-pant part [7] and LASSIE datasets [67] are used for training and evaluation following prior work [67, 68]. Both quantitative and qualitative evaluations demonstrate improved 3D recon- struction performance of our proposed approach compared to existing methods. Our approach even outperforms methods that rely on 3D skeletons or shape templates. In summary, our contributions as follows:
A new framework for reconstructing the 3D articulated shape of an object as set of deformable 3D primitive parts given only 2D evidence. For each primitive global deformations are used to reconstruct the corresponding 3D part, while local deformations increase the fidelity of the reconstruction.
A kinematics-inspired optimization process with perspective projection that allows converting the 3D primitive points to the generalized parameters corresponding to the transformations of each primitive.
Extensive quantitative and qualitative evaluations showcase the superiority of our approach over the existing sfate-of-the-aat methods.
2 Related Work
3D reconstruction of animals. Recently there have been several approaches that learn to reconstruct
the 3D shape of animals from image [27, 31, 30, 19, 73, 55, 60] or video inputs [28, 62-64]. Most previous methods make certain assumptions, such as pre-defined shape templates, statistical animal models, or the existence of annotated datasets. For example, some works [73, 4, 55] regress the parameters of a statistical shape model, SMAL [74]. However, this approach is only applicable to categories captured by the SMAL shape space. Some other methods rely on supervision from object silhouettes [27, 31, 30, 191. 2D keypoints [27] and the existence of template shape [31, 30, 19]. This limits the applicability of those works to animal categories that have such annotations. In contrast, our proposed approach does not require any 3D shape template or skeleton and is trained with self-supervised objective, which makes it easy to generalize to a wide range of animal species with no extra manual effort.
Optimization from multiple views and videos. Recently, Neural Radiance Fields (NeRF) [48, 3] have gained significant attention as a robust volumetric representation for multi-view reconstruction, particularly when accurate cameras are available. In related line of research, recent works [62-64] have focused on optimizing the 3D shapes of articulated objects using small number of monocular videos. These approaches employ meticulously designed optimization strategies that incorporate supervision from optical flow, object silhouettes, and DensePose [49] annotations. Another line of work [67, 68), leverages supervision from DINO- DINO-VIT [5] features and optimizes a part- bast- model on a small collection of images (~ 30) of particular animal category. LASSIE [67] and Hi- LASSIE [68] additionally include a test-time optimization process, in which per-instance articulation and part-refinement are employed. Our approach is closely related to LASSIE and Hi-LASSIE since we also learn to reconstruct 3D articulated shapes in a part-based manner. However, unlike those methods, our approach does not require any test-time processing.
2

---

Part-centered coordinate
d
Camera coordinate
Xproi
Xcam
Caam
Reconstructed shape
Image plane
Global coordinate
Figure 1: LEPARD geometry. We reconstruct an articulated animal shape by deforming a set of primitive parts with global deformation and local deformations d that are predicted in a part- centered coordinate system Φ. We also predict 3D translation and rotation transformations to place the reconstructed shape in global coordinate system 4. Global transformations are parameterized explicitly and offer an intuitive understanding of the shape (e.g., bending parameters make the back and tail of the tiger bend).
Part discovery. Deep feature factorization (DFF) [9] and follow-up works [1, 8, 25, 56] show that one can automatically obtain 2D corresponding part segments by clustering deep semantic features. In the 3D domain, the object parts can be discovered by using explicit representation primitives [58, 42, 43, 52, 53, 11] (e.g., cuboids, spheres, superquadrics), or learning part prior [66]. These methods mainly assume some form of supervision like 3D point clouds, keypoints or camera viewpoints. Similar to LASSIE and Hi-LASSIE, we discover parts based on deep features from DINO-VIT. However, we do not rely on pre-defined 3D skeleton like LASSIE or an intermediate skeleton representation as Hi-LASSIE.
3 Approach
3.1 Primitive Part Representation
Geometry. Given an image of an articulated object category, LEPARD aims to learn set of K primitive parts that compose its 3D shape. Each primitive is explicitly represented by group of parameters that describe its 3D shape and orientation. Following [57] each individual primitive is defined closed surface in a part-centered coordinate system (((4,, Given point (k) the surface of primitive k, its 3D location x computed follows:
(æ, y, z) in the global coordinate system & can be
X (k) +
(N)
(k)
+ (*)).
(1)
where g(M) (c)), represents the transformation the part-centered coordinate system (()) of primitive l to the global coordinate system Φ;, (k)) E R and (k) E 3x33 represent the translation and rotation of ((k) w.r.t. & and (k) denotes the relative position of the point on the primitive surface w.r.t. (²),,, which includes global deformation (kk) and local deformation d (k). The camera parameters (Cram, Ruam) used project onto the image. In Fig. 1, we illustrate the geometry of the proposed part-based shape representation that includes global and local deformations for each part. For the sake of simplicity, omit the superscript for the k-th primitive in the following.
Primitive deformations. We employ superquadrics to describe the global deformations of each part primitive. Each superquadric surface is explicitly defined by set of shape-related parameters:
cas u cos v a2 cos = ag sin
e = ag
where - 7/2 S u x/2, SUSS.
Here, is scaling parameter, (1, (22 ag denote the aspect ratio for V-, z- axes, respectively, and 81,82 are squareness parameters. To enable more flexible global deformations, we further include tapering and bending parameters. These additional global deformations are defined continuously

---

DINO Featunes
Ran
Reconstruction
Frondor
Image Sorces Farei
Inper Image
%
Medifeed Jacobinn 4pri
forin
Primitive Part Representation
Kinematies inspired Optimizaaiio
Figure 2: LEPARD training overview. Given an input image, we map it to a set of primitive parameters that describe K deformed parts via encoder-decoder network. The primitive parameters, 9.- 9o. 9s- 4a- are then used to reconstruct the 3D articulated shape x. During training, we project set of points from the primitives' surface onto the image using the predicted camera parameters Cram, Ream- We use DINO features as supervision to compute image-based forces Sproj that we further convert into generalized forces fpro,g supervise each transformation of the primitive part.
differentiable and commutative functions following [46, 39, 38]. To capture the finer shape details beyond the coverage of global deformations, we employ diffeomorphic point flow estimate the local non-rigid deformations d. Since the deformation with diffcomorphism is differentiable and invertible [2, 10], guarantees one-to-one mapping and preserves topology during the non-rigid deformations of the primitives. Please refer to [39, 38] for more details. We note that our approach is restricted these types of shapes and respective parameterizations. We can replace with any differentiable type of primitive. However, for the purposes of the articulated animal shapes this type of parameterization sufficient.
In summary, each part primitive is represented with a set of parameters q= [ge: qa q, 9al where q. and 4, contain the parameters the 3D translation and rotation respectively, that transform the part-centered coordinate system ó of the primitive to the global coordinate system 4, q, are the parameters of the global deformations, 4u are local deformations that are implemented deformation field and are added to the global shape, and [.] is the concatenation operator. Compared to the implicit function-based approaches such as NeRF [48] and NeRS [70], all these primitive parameters are defined explicitly for an intuitive understanding of the primitive deformation.
3.2 Primitive 3D Kinematics & Optimization
Primitive kinematics in 3D. We kinematics to define the relationship between any point the primitive surface and the corresponding primitive parameters q. This relationship expressed quantitatively by the model Jacobian matrix L, and this formulation allows us deform the primitive shape based on its parameters q. Specifically, the velocity of x is computed as follows:
Lq.
where denotes the first-order time derivative and L = [I. B, RJ. R [57] is the Model Jacobian matrix, where each of the four components transforms the 3D points into the translation, rotation, global and local deformation parameters of q. R is the rotation matrix that corresponds to the rotation between the part-centered coordinate system & of the primitive and the global coordinate system t. B = öRp /dg is related to the rotation matrix R and the relative position p of points on the primitive surface. Ös/Oq. is the Jacobian matrix. We refer the reader to the supplementary material for more details.
Optimization in 3D. In our modeling paradigm, minimize the energy of the primitive, defined using the principle of virtual work. The 3D forces on the primitive, SaD, result in displacements dx:
E1p 15 (Tb = fgdq.
where fg = LLL [57] are the generalized forces acting on the primitive parameters q. San are computed based on the discrepancy between points on the primitive and the target shape in 3D data space (e.g., wisn difference between the primitive surface and the target shape). The forces

---

the primitive are proportional to the distance from the target. Minimizing the generalized forces fu deforms the primitive to match the target shape and thus can be used training objective in our framework.
3.3 Primitive Part Discovery from Images
Our model is trained using a set of N in-the-wild images of articulated animals. We do not make use of any type of 2D/3D annotations or skeletons. We compute pseudo-labels using semantic clustering of self-supervised DINO [5] features similar to prior work [67, 68]. For each of the given images, we predict the parameters for the K primitive parts as defined previously. Our approach deforms set of primitives to fit the target shape under the influence of external forces fap- However, direct primitive parameter optimization using faD is not feasible since we not have access any form of 3D supervision. To this end, we project the predicted 3D primitive parts onto the image space and define losses based on the corresponding 2D forces to supervise the deformation each primitive. The training overview is given in Fig. 2.
Projection kinematies from 3D to 2D. We illustrate the relationship of kinematies between 3D and 2D via projective geometry. This allows us to project the primitives onto the image space and calculate the discrepancy between the projected primitives and 2D evidence. Then, we convert the image forces to their corresponding generalized forces that guide the deformation of the primitives. Specifically, using the estimated parameters for camera translation Caam and rotation Reams we can convert given 3D point to the camera coordinate system as follows:
Xcam
Ceam ceam
(5)
Under perspective projection, the point Neam (Jee Mes ze) projects onto image point Xproj (Fproj: Uproj) according to Jppaj = -he
%e Uproj = ʃ L where f is the focal length. By taking the time
derivative we have d'xproj Pdxxca: where
f/ze 0
0
- fxe/ 22 fue/ #cc
P
(6)
f/zo
Thus, from Eq. (5) we have:
dxproj Polxcam
Pil(Ceam + Ream x) PR RRaam
(7)
Given Eq. (3) we can rewrite Eq. (7) as:
d'xrroj
PReam dxx = PRea (Ldq) = (PR = Lpojdlq
(8)
where Eqno is the modified model Jacobian matrix that converts image points Xproj into primitive parameters q that determine translation, rotation, global and local deformations of the primitive.
Loss components. Since we only use 2D supervision during training, the primitive energy from each image we minimize is computed based on the image forces fproj:
8gm = -f Spicy -f Sm (Lprojdg)) -
Sprci. dq.
where the generalized forces Sproj.g - L guide the primitive deformation. For given image i from the training dataset, we minimize the discrepancy between our primitives and the target using the image-based forees follows:
K
K
C
K
(k.))) pmi-o
K
proj
(
(k..)) [
proj
(10)
where f(b,,) denotes the corresponding image forces of the k-th primitive part and is vector summation of all the forces at each sampling point on the projected primitive. The final loss used for training is computed by summing the generalized forces for all training samples as follows:
Lg Sproj.9e + fproj.co + fprci.a. + Sproj.qa"
(11)
5

---

where
N
K
N
K
(gng
T
(8) cam?
(plly) Tp(8.)
R cam
B ((b.)))
proj.ge
NK
NK
N
K
Tp(2.))
R
(1:t)) ((R,),
ppmj-ga
1
(ploy))
(P(2.0)
R R (1:.)
NK
NK
To obtain f(bi)) we render the predicted 3D primitive parts with a differentiable renderer [40] and compute the distance from the pseudo-mask part annotations:
p (g M(hig)))
(12)
where g(+)) and M (k) are the pscudo-mask and the k-th projected primitive for instance i, respec- tively. Apoj is constant modeling the strength of the image force fplo))
Force regularization. To enable more robust primitive fitting strategy, we incorporate regularization to the forces during training. We follow the physics-based deformable models (DMs) [46] and avoid the collisions between primitives by checking for primitive inter- penetrrenttation in each training iteration. two primitives penetrate each other, we allocate two equivalent and opposite collision forces fn and - f.. that are proportional to the distance between each pair of selected points on the two primitives. These two forces are added to the respective points on the two inter- penetreent primitives, respectively, to adjust the external forces frmj and thus push the primitives to separate from each other.
4 Experiments
Datasets and baselines. We emphasize that our aim is to reconstruct an articulated shape by discovering semantically meaningful parts in 3D. Thus, we mainly compare our approach with LASSIE [67] and Hi-LASSIE [68] who share the same goal. We conduct extensive experiments on few animal categories, including horses, giraffes, zebras, etc. For horses, we train our model using 11k images collected from Pascal [15], LASSIE [67] and DOVE [59] datasets. For the other categories, we only use the images from Pascal [15] and LASSIE [67] datasets fine-tur mun-tune our pre-trained model. To put our results into perspective, we also quantitatively compare with A-CSM [30] and 3D Safari [73], which learn to reconstruct the 3D articulated shape of animals holistically. Finally, we also provide qualitative comparisons with LASSIE using their released code. We only compare quantitatively with Hi-LASSIE, using the reported results in [68], since their code is not available at the time of submission.
Qualitative results. In Fig. we provide qualitative comparisons with LASSIE. We observe that LEPARD yields more geometrically accurate reconstructions with finer details compared to LASSIE. This is particularly evident in the cases of tiger and elephant, where the primitives used in LASSIE do not accurately correspond to the animal parts with complex shapes such as the tiger tail and the elephant nose. Additionally, LASSIE fails in the presence of additional objects in the background (see the predicted shape for the kangaroo in row 4). For the case of the kangaroo, our model accurately locates the head and correctly recovers the pose under the interference of the inaccurate clustering appearing on the top right of the DINO features. The superiority of our model in estimating accurate poses can also be observed in the case of the penguin (row 5 of Fig. 3).
Consistency visualization. In the first two rows of Fig. 4 we show that LEPARD predicts consistent parts across different samples of the same animal category, where LASSIE struggles (e.g., in the 5-th column, LASSIE predicts the elephant's 's nose as a leg). In addition, we visualize the semantic consistency of our model across different animal categories. In the last two rows of Fig. 4, we compare the reconstruction results of kangaroos and penguins. We observe that our approach employs the same primitive to consistently represent corresponding parts of different species, e.g., the heads of both penguins and kangaroos are represented with the same primitive part (highlighted in green). But LASSIE employs this same green part to represent the mouth penguins and the head of kangaroos, respectively.

---

Input
DINO
LASSIE
LASSIE
LEPARD
LEPARD
Figure 3: Qualitative results. We compare the recovered parts by LASSIE [67] and LEPARD (ours). observe that LEPARD discovers semantically meaningful and consistent parts, offering more faithful reconstructions than LASSIE. Unlike LASSIE, our approach is robust to the presence of additional objects the background (see the kangaroo in row 4). Failures LASSIE's prediction are indicated by arrows, whereas LEPARD avoids such failures.
Keypoint transfer. Due to the lack of ground-truth 3D annotations in our datasets, we follow common practice [30, 67, 68] and quantitatively evaluate our model using 2D keypoint transfer between each pair of images in the test set. In particular, given set of keypoints we map them onto the 3D primitive parts and then project them to the target image. then compute the percentage of correct keypoints (PCK) under tight threshold 0.05 max(h, w) (ie.. PCK @ @0.05), where and w are the image height and width respectively. For a successful 2D 2D-3D=+2 mapping. accurate 3D reconstructions for both the source and target images are necessary. We report results for the keypoint transfer evaluation in Table The results show that LEPARD achieves higher PCK compared to the baselines without performing test-time per-instance optimization.
source image,
Part transfer. Next, evaluate our approach on part transfer using the ground-truth part seg- mentation masks from the Pascal-Part dataset. Similar to 2D keypoint transfer, we transfer part annotations from source to target images through 2D-33D- +2D mapping utilizing the predicted 3D part primitives for the sources and target images. In this setting, we measure the performance with the percentage of correct pixels (PCP) metric, where pixel from the source image is considered to be transferred correctly if it is mapped to the same semantic part in the target image. The results

---

Input
LASSIE
LEPARD
Input
LASSIE
LEPARD
Figure 4: Consistency visualization. We compare the discovered parts by LEPARD and LASSIE [67]. The arrows indicate the correct correspondence of the parts among instances (e.g.. the left front legs of the elephants should be discovered by primitive parts in the same color) from the same (first two rows) and different (last two rows) animal categories. We observe that LEPARD discovers 3D parts with better semantic consistency (i.e., reconstructs the same parts using the same primitives).
Table I: Keypoint transfer evaluation. Results on the Pascal-Part and LASSIE datasets using PCK @0.05. LEPARD outperforms all methods for all animal categories by a significant margin.
Pascal Part dataset
LASSIE dataset
Horse
Cow
Sheep
Zebra
Tiger
Giraffe
Elephant
Kangaroo
Penguin
3D Safari [73]
57.1
50.3
50.5
62.1
50.3
32.5
29.9
20.7
28.9
A-CSM [30]
55.3
60.5
54.7
60.3
55.7
52.2
39.5
26.9
33.0
LASSIE [67]
58.0
62.4
55.5
63.3
62.4
60.5
40.3
31.5
40.6
Hi-LASSIE [68]
59.6
63.1
56.2
64.2
63.1
61.6
42.7
35.0
44.4
LEPARD
61.0
63.7
56.9
64.7
63.8
62.1
43.2
35.4
44.6
are reported on the right column of Table 2, demonstrating that LEPARD compares favorably to all baseline methods.
2D IoU. In addition, we evaluate LEPARD using overall and part IoU between the ground-truth and rendered masks. The results reported in Table 2. We follow the baselines [68, 67, 25, 5] and manually assign the discovered parts to the hest-matched parts in the Pascal-Part segmentation masks. From Table we observe that LEPARD outperforms all other methods in terms of overall IoU as well as Part IoU by large margin.
Effect of local deformations. We investigate the effect of local deformations in terms of reconstruc- tion accuracy and present qualitative results in Fig. 5. We observe that local deformations can capture fine-grained shape details and significantly improve the visual quality of the reconstructed shapes.
Model robustness and limitation. Fig. we evaluate on some held-out images from the Objaverse dataset for known categories such as elephants to show the robustness of our method. For the reconstruction of unknown categories, our method may require fine tuning on additional images, and will be included in our future work.

---

Table 2: Quantitative results the Pascal-Part [7] dataset. We report the overall IoU, part mask
IoU, as well as part transfer results measured by the percentage of correct pixels (PCP).
Overall IoU
Part IoU
Part Transfer (PCP)
Horse
Cow
Sheep
Horse
Cow
Sheep
Horse
Cow
Sheep
SCOPS [25]
62.9
67.7
63.2
23.0
19.1
26.8
DINO clustering [1]
81.3
85.1
83.9
26.3
21.8
30.8
3D Safari [73]
72.2
71.3
70.8
71.7
69.0
69.3
A-CSM [30]
72.5
73.4
71.9
73.8
71.1
72.5
LASSIE [67]
81.9
87.1
85.5
38.2
35.1
43.7
78.5
77.0
74.3
Hi-LASSIE [68]
83.4
88.
86.3
39.0
35.3
43.4
79.9
77.8
75.5
LEPARD
83.7
88.3
86.7
39.5
35.4
43.6
80.6
77.6
75.8
Input
LEPARD local
LEPARD local
LEPARD
LEPARD
Figure 5: Effect of local deformations. We test the effectiveness of the local deformations the Pascal-Part and the LASSIE dataset. We observe that local deformations have a significant impact on the visual quality of the reconstructed shapes.
Figure 6: Testing held-out images from known category. We show some qualitative results on the elephant category of Objaverse dataset.
5 Conclusion
We present LEPARD, a learning-based framework for discovering the 3D parts and shape of an articulated animal from an image without any 2D/3D annotations or skeletons. Our approach jointly optimizes a set of primitives which are explicitly defined by few shape-related parameters, and provides an intuitive understanding of the primitive deformation. To obtain high-fidelity reconstruc- tion that matches the 2D evidence provided by clustering DINO features, we propose kinematics mapping between 3D and 2D, and convert the 2D forces defined w.r.f. images to the generalized forces that supervise the motions and deformations of the primitive parts in 3D. Our approach does not require any test-time processing and discovers semantically meaningful and consistent 3D parts. We demonstrate the effectiveness of our approach through quantitative and qualitative evaluations where we achieve significant improvements over existing methods.
Acknowledgments
This research has been partially funded by research grants to D. Metaxas through NSF: IUCRC CARTA 1747778, 2235405, 2212301, 1951890, 2003874, and NIH-5ROIHL122661.

---

References
[1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features dense visual descriptors.
arXiv preprint arXiv:2112.05814, 2(3):4, 2021.
[2] Vincent Arsigny, Olivier Commowick, Xavier Pennec, and Nicholas Ayache. A log-euclidean framework for statistics diffeoomophisms. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 924-931. Springer. 2006. 4
[3] Jonathan Barron. Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Martin-Brualla, Brualla, and Pratul
Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021. 2
[4] Benjamin Biggs. Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipulla. Who left the dogs out? 3d animal reconstruction with expectation maximization in the loop. In ECCV, 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, In ICCV, 2021. 1, 2. 5.8
[6] Qi Chang. Zhennan Yan, Mu Zhou, Di Liu, Khalid Sawalha, Meng Ye. Qilong Zhangli, Mikael Kanski, Subhi Al' Aref, Leon Axel, et al. Deeprecon: Joint 2d cardiac segmentation and 3d volume reconstruction via speciffc generative method. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 567-577. Springer, 2022.
[7] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014. 2, 9 [8] Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. NeurIPS, 2021. 3
[9] Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery.
In ECCV, 2018. 3
[10] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu. Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. Medical image analysis. 17:226-236.
2019. 4
[11] Boyang Deng. Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.
Cvxnet: Learnable convex decomposition. In CVPR, 2020. 3
[12] Yu Deng, Jiaolong Yang. and Xin Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286-10296, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 1.2
[14] Matthias Eisemmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J Adler. Patrick Godau. Veronika Cheplygina, Michal Kozubek, Sharib Ali, et al. Biomedical image analysis competitions:
The state of current participation practice. arXiv preprini urXiv:2212.08566 2022. [15] Mark Everingham, SM Ali Eslami. Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer
vision, 111:98-136, 2015. 6
[16] Yunhe Gao, Zhuowei Li. Di Liu, Mu Zhou, Shaoting Zhang, and Dimitris Meta. Training like a
medical resident: Universal medical image segmentation via context prior learning. arXiv preprint
arXiv:2306.02441 12. 2023. 1
[17] Yunhe Gao, Mu Zhou, Di Liu, Zhennan Yan, Shaoting Zhang, and Dimitris N Metaxas. A data-scalable transformer for medical image segmentation: architecture, model efficiency, and benchmark, arXiv preprint
urXiv:2203.00/33 2022. 1
[18] Chuanbin Ge, Di Liu, Juan Liu, Bingshuai Liu, and Yi Xin. Automated recognition of arrhythmia using deep neural networks for 12-lead electrocardiograms with fractional time-frequency domain extension. Journal of Medical Imaging and Health Informatics. (0(11):26442267, 2020.
[19] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In ECCV.
2020. 1, 2
[20] Oshri Halimi, Or Litany, Emanuele Rodola, Alex M Bronstein, and Ron Kimmel. Unsupervised learning of dense shape correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4370-4379, 2019. 1
[21] Ligong Han, Song Wen. Qi Chen, Zhixing Zhang, Kunpeng Song. Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al. Improving negative-prompt inversion via proximal guidance. arXiv
preprint ar3:v:2306.0.m.. 05414, 14, 2023. 1
[22] Ali Hatamizadeh, Debleena Sengupta, and Demetri Terzopoulos. End-to-end trainable deep active contour
models for automated image segmentation: Delineating buildings aerial imagery. In ECCV, 2020. 1 [23] Xiaoxiao He, Chaowei Tan, Bo Liu, Liping Si, Weiwu Yao, Liang Zhao, Di Liu, Qilong Zhangli, Qi Chang, Kang Li, et al. Dealing with heterogeneous 3d mr knee images: A federated few-shot learning method
with dual knowledge distillation. arXiv preprint arXiv:2303. /4357, 2023. 1
[24] Ping Hu, Bing Shuai, Jun Liu, and Gang Wang. Deep level sets for salient object detection. In CVPR, 2017.
[25] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz. Scops:
Self-supervised co-part segmentation. In CVPR, 2019. 3, 8,
[26] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
10

---

[27] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specifie
mesh reconstruction from image collections. In ECCV, 2018. 1.2
[28]
Filippos Kokkinos and lasonas Kokkinos. Learning monocular 3d reconstruction of articulated categories
from motion. In CVPR, 2021. 1.2
[29]
Nikos Kolotouros. Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model fitting in the loop. In ICCV, 2019.
[30] Nilesh Kulkarmi, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-awant canonical
surface mapping. In CVPR, 2020, 1, 2, 6, 7, 8, 9
[31]
Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle
consistency. In ICCV, 2019. 1.2
[32] Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning. arXiv preprint arXiv23303.04447 2023.
[33] Di Liu, Yunhe Gao, Qilong Zhangli, Zhennan Yan, Mu Zhou, and Dimitris Mctaxas. Transfusion: Multi- view divergent fusion for medical image segmentation with transformers. arXiv preprint 2r11:22033000
2022.
[34] Di Liu, Chuanbin Ge, Yi Xin, Qin Li, and Ran Tao. Dispersion correction for optical coherence tomography by the stepped detection algorithm in the fractional fourier domain. Optics express, 28(5):5919 5935,
2020. 1
[35] Di Liu, Jiang Liu, Yihao Liu, Ran Tao, Jerry L Prince, and Aaron Carass. Label super resolution for 3d
magnetic resonance images using deformable u-nct. In Medical Imaging 2021: Image Processing, volume
1156, page 1159628, International Society for Optics and Photonics, 2021. 1
[36] Di Liu, Yi Xin, Qin Li, and Ran Tao. Dispersion correction for optical coherence tomography by parameter estimation in fractional fourier domain. In 2019 9 IEEE International Conference on Mechatronics and Automation (ICMA). pages 674-678. IEEE, 2019.
[37] Di Liu, Zhennan Yan. Qi Chang. Leon Axel, and Dimitris N Metaxas. Refined deep layer aggregation for multi-disease. multi-view & multi-center cardiac mr segmentation. In International Workshop on Statistical Atlases and Computational Models of the Heart, pages 315-322. Springer. 2021.
[38] Di Liu, Xiang Yu, Meng Yc, Qilong Zhangli, Zhuowei Li, Zhixing Zhang, and Dimitris N Mctaxas. Deformer: Integrating transformers with deformable models for 3d shape abstraction from a single image. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14236-14246, 2023.
[39] Di Liu, Long Zhao, Qilong Zhangli, Yunhe Gao, Ting Liu, and Dimitris N Metaxas. Deep deformable models: Learning 3d shape abstractions with part consistency. arXiv preprint arXiv2309.0/0335, 2023. 1,
[40] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In /CCV, 2019, 6
[41] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl:
skinned multi-person linear model. ACM TOG, 34(6):1-16, 2015. 1
[42] Tiange Luo. Kaichun Mo, Zhiao Huang. Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning to group:
A bottom-up framework for 3d part discovery in unseen categories. In ICLR, 2020. 3
[43] Priyanka Mandikal, Navancet KL, and R Venkatesh Babu. 3d-psrect: Part segmented 3d point cloud
reconstruction from single image. In ECCV, 2018. 3
[44] Carlos Martín-Ish Isla, Víctor M Campello, Cristian Izquierdo, Kaisar Kushibar, Carla Sendra- Balcells, Polyxeni Gkontra, Alireza Sojoudi. Mitchell J Fulton, Tewodros Weldebirhan Arega. Kumaradevan Punithakumar, et al. Deep learning segmentation of the right ventricle in cardiac mri: The m&ms challenge. IEFE Journal of Riomedical and Health Informatics, 2023.
[45] Lars Mescheder. Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupaney networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460-4470, 2019. 1
[46] Dimitris N Metaxas. Physics-bused deformable models: applications to computer vision, graphics and medical imaging, volume 389. Springer Science & Business Media, 2012. 4, 6
[47] Mateusz Michalkiewicz. Jhony K Pontes. Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.
Implicit surface representations layers in neural networks. In ICCV, 2019.
[48]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik. Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 2. 4
[49]
Natalia Neverova, David Novotny, Mare Szafraniec, Vasil Khalidov, Patrick Labatut, and Andrea Vedaldi. Continuous surface embeddings. In NeuriPS, 2020.
[50] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d recon- struction by learning particle dynamics. In Proceedings of the IEEE/CVF international conference on computer vision, pages $379-5389,2011 1
[51]
Chengjie Niu, Jun Li, and Kai Xu. Im2struct: Recovering 3d shape structure from single rgb image. In
CVPR, 2018 1
[52] Despoina Paschalidou, Luc Van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition of 3d objects from single rgb image. In CVPR, 2020. 3
[53] Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In CVPR, 2021. 3
11

---

[54]
Despoina Paschalidou, Ali Osman Ulusoy. and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In CVPR. 2019.
[55]
Nadine Rücgg, Silvia Zuffi, Konrad Schindler, and Michacl J Black. Barc: Learning to regress 3d dog shape from images by exploiting breed information. In CVPR, 2022. 2
[56] Aliaksandr Siarohin, Subhankar Roy, Stéphane Lathuilière. Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.
Motion-superviied part segmentation. In ICPR, 2021.
[57] Demetri Terzopoulos and Dimitri Mctaxas. Dynamic 3d models with local and global deformations: deformable superquadries. JEEE TPAMI, 13(7):703-77
1991. 2, 3, 4
[58] Shubham Tulsiani, Hao Su, Leonidas Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In CVPR, 2017.
[59] Shangzhe Wu, Tomas Jakah, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. International Journal of Computer Vision, pages 1-12, 2023. 6
[60] Shangzhe Wu, Ruining Li. Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning
articulated 3d animals in the wild. In CVPR, 2023, 1,2
[61] Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and UIrich Neumann. Dism: Deep implicit
surface network for high-quality single-view 3d reconstruction. NeurlPS, 2019. I
[62]
Gengshan Yang. Deqing Sun. Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from monocular video. In CVPR, 2021.
[63]
Gengshan Yang. Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. NeurlPS, 2021. Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo:
[64]
Building animatable 3d neural models from many casual videos. In CVPR, 2022.
[65]
Kaizhi Yang. Xiaoshuai Zhang. Zhiao Huang. Xuejin Chen, Zexiang Xu, and Hao Su. Movingparts: Motion-based 3d part discovery in dynamic radiance field. arXiv preprint arXiv:2303.0570 2023. 1 Chun-Han Yao, Wei-Chih Hung, Varun Jampani, and Ming-Hsuan Yang. Discovering 3d parts from image collections. In ICCV, 2021. 2,
[66]
[67]
Chun-Har Yao, Wei-Chil Hung. Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang. and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. In NeurIPS, 2022. 1, 5, 7.8.
[68]
Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming- Hing-Hsuann Yang, and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble. In CVPR,
2023. 1, 5, 7. 7,8,9
[69] Meng Ye, Mikael Kanski, Dong Yang. Qi Chang, Zhennan Yan, Qiaoying Huang. Leon Axel, and Dimitris Metaxas. Deeptag: An unsupervised deep learning method for motion tracking on cardiac tagging magnetic resonance images. In Proceddings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7261-7271, 2021.
[70] Jason Zhang. Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. Ners: neural reflectance surfaces for sparse-view 3d reconstruction the wild. NeurIPS, 2021. 4
[71] Qilong Zhangli, Jingru Yi, Di Liu, Xiaoxiao He, Zhaoyang Xia, Haiming Tang, He Wang, Mu Zhou, and Dimitris Metaxas. Region proposal rectification towards robust instance segmentation of biological images.
arXiv preprint arXiv:2203.00 022844666... 2022.
[72] Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3d-pmmn Generating shape primitives with recurrent neural networks. In ICCV, 2017.
[73] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- Wolf, and Michael Black. Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In ICCV, 2019. 2, 6, 8,
[74] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael Black. 3d menagerie: Modeling the 3d shape and pose of animals. In CVPR, 2017. 2
12


Reviews:
[{"review_id": "JCkaTw4WV4", "reviewer": "Reviewer_ifgQ", "summary": "The paper proposes a part-based method to reconstruct 3D shapes in a category-specific manner. Compare against its baseline LASSIE, the paper uses an elegant primitive part representation that could capture both global and local deformations to increase the fidelity of reconstruction. The method also does not need test-time optimization process like LASSIE. The proposed method outperforms LASSIE quantitatively and qualitatively. ", "questions": "1. How does the training objective guarantee semantic consistency. The paper adopts the same process of DINO features as LASSIE used for training. However, unlike LASSIE, the method does not need test-time optimization and the training objective does not explicitly model 2D-3D consistency. The author should explain how they achieve better results than LASSIE in more details. \n\n2. How to choose number of parts for each category? Different from LASSIE, the method does not take skeleton as input, so how to choose number of parts for unknown category. If the number of parts is high, will there be over-segment problem and lose the semantic meaning and consistency?", "limitations": "Please see comments above. It would also be quite interesting if the output parts could be further retargeted to novel poses. The part-based approach is hard for animation against parametric models. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed primitive part representation is novel. Compare against modeling deformation implicitly,  the parameterization of the representation is compact and intuitive for understanding. The new representation increase the fidelity of the reconstruction. \n\n2. No need for test-time processing.  The proposed method does not need per-instance optimization and only requires a forward-pass. \n\n3. No 3D or input requirements. The method takes only image as input, without the need of mesh template, category skeleton as additional input. For training, the method is self-supervised and does not need 2D/3D annotations and could be easily scale-up to in-the-wild data. \n\n4. Impressive qualitative results. The visual result shown in Fig. 4 is impressive. The results looks quite strong as the detailed structure of the animal could be fully recovered in a primitive-based approach. \n\n5. Outperform baseline method in all categories. ", "weaknesses": "1. The motivation of introducing image force in the training objective is not well explained. As LASSIE adopts simple silhouette loss by differentiable rendering, the paper introduces complex Jacobian matrices and generalized forces. What kind of benefit could we get from this and what is the difference and limitation of using silhouette loss against LASSIE. The author should include more experiments to demonstrate the superiority of such kinematic-inspired procedure. \n\n2. Several questions not fully understand. Please questions below. \n\n3. No discussion of the limitation of the method and when the method would fail. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we greatly appreciate your recognition of our work, and thank you for your valuable comments! We hope the following responses can address your concerns.\n\n>Q1: The motivation of introducing image force in the training objective is not well explained. As LASSIE adopts simple silhouette loss by differentiable rendering, the paper introduces complex Jacobian matrices and generalized forces. What kind of benefit could we get from this and what is the difference and limitation of using silhouette loss against LASSIE. The author should include more experiments to demonstrate the superiority of such kinematic-inspired procedures.\n\nA1: The image force is calculated using the pixel-level difference between the primitive parts and the 2D image, and measures how well the primitive surface needs to be deformed to match the ground truth shape in 2D. The image force can be viewed the same as the silhouette loss in LASSIE, which is obviously not enough for both approaches, especially in the scenario of 2D-3D reconstruction. As opposed to LASSIE which uses multiple separate regularization terms resulting in sub-optimal performance, we calculate image forces in 2D and convert them into generalized forces to enable a joint regularization of the shape-related parameters. This allows us to regularize each sub-transformation of each primitive part deformation, and provides more effective constraints during training without requiring knowledge of joint locations. The extensive experiments demonstrate the effectiveness and robustness of our optimization strategy. For comparison, in Table. 2 of the Author Rebuttal PDF, we provide an ablation study using only the image forces in the loss function and demonstrate consistent superior LEPARD performance against baselines.\n\n>Q2: How does the training objective guarantee semantic consistency. The paper adopts the same process of DINO features as LASSIE used for training. However, unlike LASSIE, the method does not need test-time optimization and the training objective does not explicitly model 2D-3D consistency. The author should explain how they achieve better results than LASSIE in more detail.\n\nA2: See also A1 for details on how our method works which is different from LASSIE. We use kinematic-based modeling which jointly constrains the parameters of translation, rotation, and deformations and results in a more robust solution. As opposed to LASSIE, these parameters are learnt and guide each primitive to deform and fit more accurately to a part of the shape. The kinematic constraints result in learning a more consistent parametric transformation among similar animal categories.\n\n>Q3: It would also be quite interesting if the output parts could be further retargeted to novel poses. The part-based approach is hard for animation against parametric models.\n\nA3: Our model is parametric (i.e., each part is fully parameterized by a few shape-related parameters) and thus is naturally suitable for reposing by changing the translation or rotation matrix. Note that since we use fully explicit representation, we can even change the bending or tapering degree of each part by adjusting the global deformation parameters. This is impossible to do with implicitly represented parts as in LASSIE and the other baseline methods since they use MLPs to directly estimate the point-wise deformation field. In addition to being more robust to missing data or gaps in the data compared to implicit methods, our explicit shape representation approach offers shape explainability (e.g., tapering, bending shape information).\n\n"}, {"review_id": "tAemPiy4Ie", "reviewer": "Reviewer_pQ1r", "summary": "The paper describes a method for fitting K superquadric geometric primitives (enhanced with tapering, bending, and diffeomorphic local deformations) to a set of images of an animal category (e.g. elephant). The main contribution is that the method requires no supervision, and uses 2D feature correspondence to impose constraints on 3D shapes.", "questions": "Elephant ears and trunk seem part of the same head primitive in Figure 5. Yet the shape is quite complex. Is that due to local deformation? If so, then figure 6 should be changed to use the elephant example instead of the more minor tiger example.\n\nHow do you ensure that rotation matrices R are orthonormal with positive determinants?\n\nHow would you incorporate the fact that animals are a kinematic chain (bones connected to each other), rather than a bag of shape primitives?\n\nDid you try running your method on humans? It would be very informative to see human results compared to human-specific methods.\n\nThe supplemental material shows interesting mappings between different animals. It would be impactful to show more of that analysis in the main paper. Are there any cross-category benchmarks to evaluate it on?\n ", "limitations": "The paper does not explicitly address its limitations in a traditional limitations section.\n\nIt is not explicitly stated whether the number of primitives, K, is computed automatically or manually specified. It would be a limitation if it has to be manually specified.\n\nThe method treats each primitive separately from others, which does not faithfully represent the kinematic chain that is an animal shape.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Superquadrics with limited deformations offer a good balance between geometric expressivity and parameter compactness.\n\nThe end-to-end image supervision of 3D geometry is fundamentally sound. Thus, relying only on 2D feature correspondences, and translating those to 3D constraints, is a robust way to aggregate information from an unstructured image collection.\n\nThe results are compelling and a clear improvement on previous work.", "weaknesses": "Animal bodies are kinematic chains, where one limb affects another. This approach does not take that in account.\n\nEven though the paper focuses on animals, it would be quite useful to see how it performs on humans and how it compares to human-specific baselines.\n\nThe paper could use a section discussing the limitations and future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we greatly appreciate your recognition of our work, and thank you for your valuable comments! We hope the following responses can address your concerns.\n\n>Q1: Animal bodies are kinematic chains, where one limb affects another. This approach does not take that into account. How would you incorporate the fact that animals are a kinematic chain (bones connected to each other), rather than a bag of shape primitives? \n\nA1: LEPARD is a general approach that is not limited to animal bodies and can also be applied to a wide range of objects such as inanimate (human-made, natural objects), and animate (animals, humans) objects. While in this paper, we focus on animals, however, our approach can be extended in future work to add kinematic constraints and other types of objects. In future work we plan to extend our approach to create a kinematically constrained primitive-based model of an animal or human, and fit to the target shape data (static or dynamic).  \n\n>Q2: Performance on humans.\n\nA2: In this paper we focus on 3D part discovery of articulated animal shapes from 2D images and mainly compare with those methods which work on the same task. In future work we plan to use our method on humans due to its general representation ability. \n\n>Q3: Elephant ears and trunk seem part of the same head primitive in Figure 5. Yet the shape is quite complex. Is that due to local deformation? If so, then Figure 6 should be changed to use the elephant example instead of the more minor tiger example.\n\nA3: The shape of ears and trunk of an elephant are reconstructed due to a combination of tapering, bending (i.e., global deformations) as well as local deformations. In Fig.5 of the main paper we ablate using the shape of tigers to better demonstrate the effect of local deformations.\n\n>Q4: How do you ensure that rotation matrices R are orthonormal with positive determinants?\n\nA4: We predict 3D rotations as normalized 4D unit quaternions, which are converted to valid rotation matrices (that are orthonormal with positive determinants) using a closed-form transformation matrix.\n\n>Q5: The supplemental material shows interesting mappings between different animals. It would be impactful to show more of that analysis in the main paper. Are there any cross-category benchmarks to evaluate it on?\n\nA5: To the best of our knowledge, there are no public cross-category benchmarks to show such 3D part level consistency across species. However, we will add sufficient qualitative comparisons for this to the main paper/supplementary.\n\n"}, {"review_id": "gyfz9PmRVV", "reviewer": "Reviewer_5yZr", "summary": "The paper presents LEPARD, a framework for reconstructing the 3D articulated shape of animals from a single in-the-wild image. It explicitly represents the parts as parameterized primitive surfaces (superquadrics) with global and local deformations in 3D. The authors employ a kinematics-inspired optimization to guide the deformations. Besides, LEPARD is trained solely using off-the-shelf deep features from DINO, without requiring any 2D or 3D annotations. Experiments on the Pascal-part and LASSIE datasets show the superiority of the proposed method.", "questions": "In L174, it is mentioned that \"the estimated parameters for camera translation and rotation\" are used. I am curious whether these camera parameters are trained by the image force only. Are there any additional supervisions or any missing implementation details (e.g., predicting multiple hypotheses as in A-CSM)? Because it can be quite ill-posed to optimize both camera parameters and part parameters.", "limitations": "The authors have not adequately addressed the limitations. It will be nice if the authors can discuss, e.g., 1) the effect of the quality of psuedo labels generated by DINO on the final performance; 2) the bottleneck of the current method for the further improvement.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The authors propose to use local non-rigid deformations to capture fine-grained shape details, which is different from previous works.\n- The authors propose a framework to compute image-based forces based on the discrepancy of DINO features and the projected primitive parts.", "weaknesses": "There are a limited number of categories in the datasets evaluated. Can the authors qualitatively evaluate their methods on held-out images from other sources for the same training categories and some unseen categories (maybe fine-tuning a small set of images)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your strong recognition of our work! We hope the following responses can address your concerns.\n\n>Q1: There are a limited number of categories in the datasets evaluated. Can the authors qualitatively evaluate their methods on held-out images from other sources for the same training categories and some unseen categories (maybe fine-tuning a small set of images)?\n\nA1: Thanks for the suggestion! We have tested on the Objaverse dataset for known categories such as elephants. The results are given in Fig. 2 of the Author Rebuttal PDF. Fine tuning on additional images for unknown categories might be out of the scope of this paper, and we will include it in future work.\n\n>Q2: In L174, it is mentioned that \"the estimated parameters for camera translation and rotation\" are used. I am curious whether these camera parameters are trained by the image force only. Are there any additional supervisions or any missing implementation details (e.g., predicting multiple hypotheses as in A-CSM)? Because it can be quite ill-posed to optimize both camera parameters and part parameters.\n\nA2: We do not use the multiple camera hypothesis approach from A-CSM [20] and U-CMR [14], but predict a single camera viewpoint similar to LASSIE [40] and Hi-LASSIE [41].\n\n>Q3: It will be nice if the authors can discuss the effect of the quality of pseudo labels generated by DINO on the final performance\n\nA3: Thanks for the suggestion! We added an experiment to test the effect of the DINO feature quality on the final performance. Specifically, we use the Pascal-Part dataset which provides GT masks for the evaluation of DINO feature quality. We sort the images based on the IoU between the DINO feature and the GT mask, and split these images into three groups according to their quality (IoU) rankings i.e., group 1, 2, 3 with the best, medium, and worst quality, respectively.  We then train a separate model on each group and evaluate their performance in terms of overall IoU and Part IoU. The average results over all three tested categories are reported in Table. 1 of the Author Rebuttal PDF. We observe that the performance of our approach only slightly degrades with lower DINO feature quality, which may also be due to the fewer training samples.\n\n\n\n>Q4: the bottleneck of the current method for further improvement.\n\nA4: The bottleneck is the performance of our method depends on the use of the correct number of parts for reconstruction. While this is not very restrictive, in future work we plan to extend our method so that it works without knowledge of the number of parts. Please see the general Author Rebuttal for details.\n\n"}, {"review_id": "ddoMq0KEFJ", "reviewer": "Reviewer_a5TN", "summary": "The paper introduces LEPARD, a framework for reconstructing the 3D shape of animals from single images. LEPARD reconstructs 3D shapes as parts, which are parameterized primitive surfaces with global and local deformations. LEPARD is trained using off-the-shelf deep features without the need for 2D or 3D annotations. Experimental results demonstrate more detailed shape reconstruction.", "questions": "Questions are listed in weaknesses.", "limitations": "There is no discussion about limitations in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "**Method:**\n- 3D reconstruction as parts is an interesting approach to 3D reconstruction and is generally less explored than holistic reconstruction. Exploration in this direction can be beneficial to the community.\n- The proposed method reconstructs details better by modeling coarse shapes as well as detailed deformations of each part. This is an intuitive approach and appears to be effective.\n\n**Experiments:**\n- The comparison with SOTA is thorough and complete to the best of my knowledge. The improvements over SOTA appear to be significant.\n- The ablation study clearly demonstrates the effectiveness of each component.", "weaknesses": "I have several confusion about the method.\n\n- The model parameters are not defined precisely. It’s unclear how q_c and q_theta are defined ( how are they just c and R? or is there any difference?)\n- Details regarding how the local deformation d is obtained from v_s are missing. The concept of stationary velocity field and Gaussian smoothing layer should be explained more in detail to help understanding. Also, the paper should include justification for using stationary velocity field and Gaussian smoothing layer.\n- I don’t understand Equation 5. The author introduces the model Jacobian matrix without defining what the model is. It’s also unclear why the velocity x is relevant in this static reconstruction setting. In addition, it’s unclear how Equation 5 is derived.\n- In L152-153, what is the practical meaning of “the energy of the primitive” and how is this energy and the force f_3d energy related to 3D shape reconstruction?\n- how is the loss in equation 12 derived?\n- In equation 14, shouldn’t G^i be part-based (as described in L162 and illustrated in Fig 3) and hence be denoted as G^(i,k)? Otherwise, why is the rendered part mask compared with the mask of the whole object? If G^i is part-based, how are the part labels from the DINO feature associated with the K primitives?\n\nI find it hard to understand the proposed method and hence cannot recommend acceptance given the current state. Please address my questions above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments! We will release the code for reproduction. We hope the following responses can address your concerns.\n\n>Q1: The model parameters are not defined precisely. It’s unclear how $q_c$ and $q_\\theta$ are defined.\n\nA1: $q_c = c \\in \\mathbb R ^3$ represents the 3D translation of each part, while $q_\\theta  \\in \\mathbb R ^4$ is a standard 4D quaternion formulation that represents the 3D rotation of each part. Please also see L134-135 of the main paper and the Notes in the Notation table (Table 3 of the supplementary material).\n\n>Q2: Details regarding how the local deformation d is obtained from $v$ are missing. The concept of SVF and Gaussian smoothing layer should be explained more in detail to help understanding. \n\nA2: The local deformation $\\textbf{d}$ is modeled as diffeomorphic point flows, which is a smooth and invertible spatial transformation (diffeomorphic mapping) and is widely used in shape modeling[1,2]. Diffeomorphic mappings allow for the computation of a time-dependent velocity field via an ordinary differential equation (ODE). However, such computations can be intricate and resource-intensive. To address this, a stationary velocity field (SVF) is usually employed [3]. This SVF maintains a constant velocity and simplifies the parameterization of diffeomorphisms. More specifically, a diffeomorphic mapping is achieved as the trajectory and integration of a smooth SVF $v$. The widely accepted practice[2] employs a scaling and squaring layer (S&S) for the integration, and utilizes convolution with suitable kernels, e.g., Gaussian kernels with positive scale,  to smooth out the velocity ﬁelds and achieve a more refined output - this process is termed the Gaussian smoothing layer. More comprehensive explanations of this can be found in Section A.3 of the supplementary.\n\n[1] Beg, M. Faisal, et al. \"Computing large deformation metric mappings via geodesic flows of diffeomorphisms.\" IJCV, 2005.\n\n[2] Amor, Boulbaba Ben, Sylvain Arguillère, and Ling Shao. \"ResNet-LDDMM: advancing the LDDMM framework using deep residual networks.\" TPAMI, 2022.\n\n[3] Arsigny, Vincent, et al. \"A log-euclidean framework for statistics on diffeomorphisms.\" MICCAI, 2006.\n\n>Q3: I don’t understand Equation 5. The author introduces the model Jacobian matrix without defining what the model is. It’s also unclear why the velocity x is relevant in this static reconstruction setting. In addition, it’s unclear how Equation 5 is derived.\n\nA3: Equation 5 shows how a change in a 3D point $x$ is translated to a change in the shape-related parameters of the primitive. Our approach is inspired by the kinematics of physics-based deformable models [33], which take the shape deformation as a dynamic system and thus the time derivative is used for $x$ and $q$. In our setting, at an arbitrary training iteration $t$, the 3D point $x$ from the primitive surface should match with the $x_{gt}$ point from the surface of the ground-truth shape. $\\dot{x}$ a.k.a $d{x}$ shows how this point should change to make this possible and $\\dot{q}$ a.k.a $dq$ is the corresponding change in the shape-related parameters of the primitive. Thus, in the static reconstruction setting the model is deformed during training to match the ground-truth shape. Please see Sec. A.4 of the supplementary material for detailed derivation of Equation 5.\n\n>Q4: In L152-153, what is the practical meaning of “the energy of the primitive” and how is this energy and the force $f_{3d}$ energy related to 3D shape reconstruction?\n\nA4: In the context of 3D shape reconstruction, the \"energy of the primitive\" refers to the amount of virtual work required to deform a primitive so that it aligns with a target shape. Here, \"virtual work\" is defined as the product of the virtual force and the distance over which this force acts, specifically the distance between the primitive surface and the target shape. During the training process, the goal is to minimize this energy, ensuring that the primitive shape deforms and closely matches the target shape. In other words, the lesser the energy, the closer our primitive is to the desired 3D shape.\n\n>Q5: how is the loss in equation 12 derived?\n\nA5: As we explained in Q4, we try to minimize the primitive energy to deform the primitive to match the target shape. The primitive energy refers to the amount of virtual work defined as the integration of “force” $f_\\text{proj}$  over the change of the points $dx_\\text{proj}$ (Eq. 11). Using Eq. 5, the change of points $dx_\\text{proj}$ is further expressed as the product of Jacobian $L_\\text{proj}$ and change of shape-related parameters $d q$. Thus we can convert the minimization of 2D point-wise image forces $f_\\text{proj}$ to the minimization of the parameter-based forces (generalized forces) $f_\\text{proj}$ using the Jacobian matrix, which leads to Eq. 12.\n\n>Q6: In equation 14, shouldn’t $G^i$ be part-based and hence be denoted as $G^{(i,k)}$? \n\nA6: Since the semantic clusters from DINO features do not essentially match the number of parts, we do not use the part-based ground truth but use the whole mask. During training, all primitive parts are initialized and deformed together to reach the boundary of the mask. To avoid the intersections between primitives, we follow PDMs and check for primitive inter-penetration in each training iteration. If two primitives penetrate each other, we assign two equivalent and opposite collision forces $f_n$ and $-f_n$ that are proportional to the distance between each pair of selected points on the two primitives. These two forces are added to the respective points on the two inter-penetrating primitives, respectively, to adjust the forces $f_\\text{proj}$ and thus push the primitives to separate from each other. In this way, the primitives are deformed to different parts of the shape under the kinematic-based optimization. We will add all the necessary details to the revised paper/supplementary. \n"}, {"review_id": "9ftiyznJTz", "reviewer": "Reviewer_Hc7X", "summary": "The paper studies the research problem of shelf(DINO)-supervised articulated 3D shape reconstruction. The key idea of the paper is to factorize shapes into different primitives, and to model the shape primitives using both global and local deformation parameters. The parameters of the model are optimized in a kinematics-inspired way, where the 3D kinematics are projected into 2D to utilize the DINO supervision. The experiments show the proposed model outperforms previous state-of-the-art methods, including both primitive and holistic reconstruction approaches.", "questions": "Please see the questions in the weaknesses. Overall, the high-level idea of global+local deformations modeling makes sense, and the qualitative results of the proposed method looks impressive. However, I do think the writing of the paper needs major improvement and many details need further clarification.", "limitations": "A detailed discussion about limitation and failure case is missing now and should be included.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The high-level idea of factorizing primitive parameterization into global and local deformations make sense.\n2. The performance of the proposed method has been well evaluated and clearly outperforms previous state-of-the-art methods.", "weaknesses": "1. As a non-expert in this field who does not have much knowledge about physics-based deformable model, I find this paper hard to follow and sometimes not self-contained. Not being able to understand some details about kinematics could be caused by my limited knowledge about physics, but, importantly, it's not clear to me at all why the kinematics-based optimization approach is adopted in this paper. Conceptually, I understand optimizing primitives without direct 3D primitive supervision requires some sort of regularization -- but what kind of prior knowledge or physical facts are encoded in this model, given that there are no real forces/materials? What are the assumptions/hypothesis that generate these forces? And what makes the local deformation to be small/local? It is really important to make the high-level intuition clear before dive into the details. I also wonder if one uses the global+local deformable model in this paper but with simple optimization methods (e.g. directly sample points from primitive surface and project to 2D, then minimize the mask L1/MSE/BCE loss), will such simple alternatives not converge/lead to inferior performance?\n\n2. In the quantitative ablation (Fig.6), the performance gap between the model w/o local and the full model does not seem too large (e.g. compared to the gap between LEPARD and Hi-LASSIE in Table 2). What is the typical std/error bar of this evaluation?\n\nOther comments/questions:\n1. It would have been beneficial to provide the dataset statistics.\n2. How is K decided? (L164)\n3. What is the limitation/failure cases of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: Motivation of kinematics-based optimization approach adopted in this paper. \n\nA1: Thanks for the comment! We realize that we may have made presumptions regarding the reader's familiarity with PDMs, and we appreciate the feedback. Let's break down the approach and the intuition behind our choices: \n\n**Virtual Forces**: The forces we reference in the paper aren't real forces we encounter in physical systems. Instead, these are \"virtual forces\". They're computed based on the virtual displacement of the surface points of the primitive[33]. They are used to measure how well the points on the primitive surface are deformed to match the target shape. \n\n**Kinematic-Based Optimization**: The choice to employ kinematic-based optimization stems from the conceptualization of the primitive deformation as a Lagrangian dynamic system[33]. This means the points on the primitive surface are continuously deforming to match the target shape during the training iterations, and also the shape-related parameters of the primitive change accordingly. During each training iteration, by minimizing the virtual forces applied to the primitives, we can obtain a reconstructed shape that is closest to the target.  However, we need more effective regularization due to the lack of 3D supervision. In addition to directly sampling points from the primitive surface and optimizing them (which might seem straightforward),  we seek to optimize groups of shape-related parameters that control the transformations of the primitive, i.e., translation, rotation, global and local deformations. This imposes tighter constraints, and is crucial especially when we do not have any 3D supervision but only 2D image-based evidence. Our method essentially provides more constrained regularization, making the model more robust and accurate in the absence of 3D supervision. We will carefully revise the paper by simplifying the text and adding intuition to make it easier to read.\n\n>Q2:  I also wonder if one uses the global+local deformable model in this paper but with simple optimization methods (e.g. directly sample points from the primitive surface and project to 2D, then minimize the mask L1/MSE/BCE loss), will such simple alternatives not converge/lead to inferior performance?\n\nA2: We understand your concern, and added an ablation study using our primitive parameterization with only image force loss (MSE). The results are given in Table. 2 of the Author Rebuttal PDF.\n\n>Q3: In the quantitative ablation (Fig.6), the performance gap between the model w/o local and the full model does not seem too large (e.g. compared to the gap between LEPARD and Hi-LASSIE in Table 2). What is the typical std/error bar of this evaluation?\n\nA3: The local deformations capture the fine-grained shape details and greatly improve the visual quality of the reconstructed shapes, while their effect is not entirely captured by the quantitative metrics that only improve slightly. We also added a std/err bar in Table. 2 of the Author Rebuttal PDF.\n\n>Q4: It would have been beneficial to provide the dataset statistics.\n\nA4: We train our model on Pascal-Part and LASSIE dataset following[35,40], and use the same evaluation protocol as LASSIE.\n\n>Q5: How is $K$ decided? Q6: Limitation\n\nA5, A6: Please see the general Author Rebuttal for details.\n"}]

Author rebuttal:
Dear AC and reviewers, we are grateful for the strong recognition and valuable comments of our work. In the following, we will first answer the common concern of all reviewers, followed by answers to each reviewer's comments.

The common concern is mainly the limitation of our approach.

We share the same assumption as LASSIE that all the four-limb animals share the same 3D part structure despite considerable shape variations across species, which allows LASSIE and LEPARD to apply the same number of parts to all four-limb animals. The only difference is that LASSIE uses more strict supervision, i.e., the number of joints K+1 (which determines the number of parts K) and their locations. As opposed to LASSIE, we use the less restrictive common knowledge K=16 for parts to all four-limb animals and we do not define or use the locations of the K (of the K+1 joints). We use kinematic-based modeling which allows us to discover the K parts. In addition, note that the number of parts is a common assumption for unsupervised 3D part reconstruction from a single image[1-5].  The experiments in the paper demonstrate the semantic consistency and reconstruction accuracy of LEPARD, which shows that our method works well in real-world scenarios/images. 

One potential limitation is that if we use other than K=16 for the number of parts, the performance may degrade because only up to 16 parts are visible in the images. For the same reason the performance of LASSIE may degrade if they change the number of joints that corresponds to K=16 parts. Our future work will include active part discovery by monitoring motion changes to overcome the need for initializing the number of parts.


[1] Tulsiani, Shubham, et al. "Learning shape abstractions by assembling volumetric primitives." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[2] Paschalidou, Despoina, Ali Osman Ulusoy, and Andreas Geiger. "Superquadrics revisited: Learning 3d shape parsing beyond cuboids." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Deng, Boyang, et al. "Cvxnet: Learnable convex decomposition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[4] Paschalidou, Despoina, et al. "Neural parts: Learning expressive 3d shape abstractions with invertible neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[5] Tertikas, Konstantinos, et al. "Generating Part-Aware Editable 3D Shapes Without 3D Supervision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.


Meta-review: This is a technically solid paper which introduces a novel part representation for unsupervised part-based 3D reconstruction. The qualitative and quantitative results show clear benefits over prior work using more restrictive representations while also removing the need for test-time optimizations. Although there were concerns regarding the limited applications to quadrupeds, this is the typical setting considered across prior works.

The reviewers all recommend (clear/weak/borderline) acceptance (including Reviewer XxbL based on their subjective comments), and the AC agrees. However, the authors are strongly encouraged to improve the presentation of the technical approach considering a broader audience in mind.
Decision: Accept (poster)