SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Di Liu Qilong Zhangli Yunhe Gao Dimitris N. Metaxas Rutgers University
Reconstructing the 3D articulated shape of an animal from a single in-the-wild image is a challenging task. We propose LEPARD, a learning-based framework that discovers semantically meaningful 3D parts and reconstructs 3D shapes in a part-based manner. This is advantageous as 3D parts are robust to pose variations due to articulations and their shape is typically simpler than the overall shape of the object. In our framework, the parts are explicitly represented as parameterized primitive surfaces with global and local deformations in 3D that deform to match the image evidence. We propose a kinematics-inspired optimization to guide each transformation of the primitive deformation given 2D evidence. Similar to recent approaches, LEPARD is only trained using off-the-shelf deep features from DINO and does not require any form of 2D or 3D annotations. Experiments on 3D animal shape reconstruction, demonstrate significant improvement over existing alternatives in terms of both the overall reconstruction performance as well as the ability to discover semantically meaningful and consistent parts.
Predicting the 3D shape and part articulation of an object from a single image is a severely under-constrained and challenging problem. It can be applied to many downstream tasks, such as shape reconstruction [45, 61, 47, 50, 54, 39], segmentation [24, 51, 32, 72, 16, 33, 6, 71], editing [65, 21], re-targeting [12, 20, 69] and medical imaging applications [22, 34, 36, 17, 37, 35, 14, 23, 18, 44]. Successful approaches [26, 29] for predicting the 3D shape of humans rely on a parametric human body model (e.g., SMPL [41]) built from thousands of mocap sequences and on strong supervision from 3D joint locations. Similar breakthroughs are not seen for other articulated object categories, like animals, as 3D scanning of such categories is quite challenging. The lack of 3D annotations and an appropriate parametric animal model has led to approaches that utilize a pre-defined shape template and train with 2D supervision [27, 31, 30, 19, 28]. Assuming such a fixed shape template and supervision from 2D annotations is not optimal, but is necessary for training these systems. Recent works [67, 68, 60] have discarded both assumptions by learning a shape prior (part-based [67, 68] or holistic shape [60]) and by using deep features from an off-the-shelf vision transformer [13], DINO-ViT [5], for supervision. Although the approaches presented in [67, 68, 60] seem promising, they exhibit certain limitations. In [60] a category-specific prior mesh is obtained in a pre-training step, which can be seen as a learned shape template. Then it is trained to predict the articulation and deformation based on this fixed template. LASSIE [67] and Hi-LASSIE [68] learn generic part priors that, similar to [60], can be articulated and deformed. However, learning the deformation field w.r.t weak generic shape priors is a hard task. Assuming that the prior shape is far from the target shape of an object, the model has to compensate by predicting large deformations to match the image evidence.
In this paper, we propose a comprehensive framework for reconstructing the 3D shape and related articulations of an object from single-view images that has several desired properties missing from
existing works. Similar to [67, 68], the 3D shape of an object is explicitly expressed as a set of part primitives. The primitives are parameterized surfaces (e.g., superquadrics) equipped with additional linear tapersing and bending transformations to capture the target shape as faithfully as possible. Unlike [67, 68], the 3D primitives are not fixed across all instances and can deform to capture intra-category variations and accurately reconstruct 3D parts. To capture fine-grained shape details beyond the coverage of the primitive parametric deformations (termed global deformations), we employ a diffeomorphic mapping to estimate local non-rigid deformations of a set of points sampled from the 3D surface of each part. Our approach uses global deformations to capture the salient part of the 3D shape and uses local deformations to further improve the 3D shape reconstruction quality. As such, the local deformations are typically small, which adds to the robustness of our method. Following prior work [66, 68, 60] we use deep features from a vision transformer [13], DINO-ViT [5], as supervision to train our model. Inspired by the kinematics of 3D deformable models [57], we propose a framework to compute image-based forces based on the discrepancy of DINO features and the projected primitive parts. The image-based forces are then converted to generalized forces via kinematics modeling, which provides strong supervision for the deformation of the 3D part primitives.
We conduct extensive experiments on 3D articulated object shape reconstruction through part discovery. The Pascal-part [7] and LASSIE datasets [67] are used for training and evaluation following prior work [67, 68]. Both quantitative and qualitative evaluations demonstrate improved 3D reconstruction performance of our proposed approach compared to existing methods. Our approach even outperforms methods that rely on 3D skeletons or shape templates. In summary, our contributions are as follows:
- A new framework for reconstructing the 3D articulated shape of an object as a set of deformable 3D primitive parts given only 2D evidence. For each primitive global deformations are used to reconstruct the corresponding 3D part, while local deformations increase the fidelity of the reconstruction.
- A kinematics-inspired optimization process with perspective projection that allows converting the 3D primitive points to the generalized parameters corresponding to the transformations of each primitive.
- Extensive quantitative and qualitative evaluations showcase the superiority of our approach over the existing state-of-the-art methods.
3D reconstruction of animals. Recently there have been several approaches that learn to reconstruct the 3D shape of animals from image [27, 31, 30, 19, 73, 4, 55, 60] or video inputs [28, 62-64]. Most previous methods make certain assumptions, such as pre-defined shape templates, statistical animal models, or the existence of annotated datasets. For example, some works [73, 4, 55] regress the parameters of a statistical shape model, SMAL [74]. However, this approach is only applicable to categories captured by the SMAL shape space. Some other methods rely on supervision from object silhouettes [27, 31, 30, 19], 2D keypoints [27] and the existence of a template shape [31, 30, 19]. This limits the applicability of those works to animal categories that have such annotations. In contrast, our proposed approach does not require any 3D shape template or skeleton and is trained with a self-supervised objective, which makes it easy to generalize to a wide range of animal species with no extra manual effort.
Optimization from multiple views and videos. Recently, Neural Radiance Fields (NeRF) [48, 3] have gained significant attention as a robust volumetric representation for multi-view reconstruction, particularly when accurate cameras are available. In a related line of research, recent works [62-64] have focused on optimizing the 3D shapes of articulated objects using a small number of monocular videos. These approaches employ meticulously designed optimization strategies that incorporate supervision from optical flow, object silhouettes, and DensePose [49] annotations. Another line of work [67, 68], leverages supervision from DINO-ViT [5] features and optimizes a part-based model on a small collection of images ( $\sim 30$ ) of a particular animal category. LASSIE [67] and Hi-LASSIE [68] additionally include a test-time optimization process, in which per-instance articulation and part-refinement are employed. Our approach is closely related to LASSIE and Hi-LASSIE since we also learn to reconstruct 3D articulated shapes in a part-based manner. However, unlike those methods, our approach does not require any test-time processing.
Figure 1: LEPARD geometry. We reconstruct an articulated animal shape by deforming a set of primitive parts with global deformation  $\mathbf{s}$  and local deformations  $\mathbf{d}$  that are predicted in a part-centered coordinate system  $\phi$ . We also predict 3D translation and rotation transformations to place the reconstructed shape in a global coordinate system  $\Phi$ . Global transformations are parameterized explicitly and offer an intuitive understanding of the shape (e.g., bending parameters make the back and tail of the tiger bend).
Part discovery. Deep feature factorization (DFF) [9] and follow-up works [1, 8, 25, 56] show that one can automatically obtain 2D corresponding part segments by clustering deep semantic features. In the 3D domain, the object parts can be discovered by using explicit representation primitives [58, 42, 43, 52, 53, 11] (e.g., cuboids, spheres, superquadrics), or learning part prior [66]. These methods mainly assume some form of supervision like 3D point clouds, keypoints or camera viewpoints. Similar to LASSIE and Hi-LASSIE, we discover parts based on deep features from DINO-ViT. However, we do not rely on a pre-defined 3D skeleton like LASSIE or an intermediate skeleton representation as Hi-LASSIE.
3.1 Primitive Part Representation
Geometry. Given an image of an articulated object category, LEPARD aims to learn a set of  $K$  primitive parts that compose its 3D shape. Each primitive is explicitly represented by a group of parameters that describe its 3D shape and orientation. Following [57] each individual primitive  $k$  is defined as a closed surface in a part-centered coordinate system  $\phi^{(k)}$ . Given a point  $\mathbf{p}^{(k)}$  on the surface of primitive  $k$ , its 3D location  $\mathbf{x} = (x,y,z)$  in the global coordinate system  $\Phi$  can be computed as follows:
$$
\mathbf {x} = \mathbf {c} ^ {(k)} + \mathbf {R} ^ {(k)} \mathbf {p} ^ {(k)} = \mathbf {c} ^ {(k)} + \mathbf {R} ^ {(k)} \left(\mathbf {s} ^ {(k)} + \mathbf {d} ^ {(k)}\right), \tag {1}
$$
where  $\delta^{(k)}\equiv (\mathbf{c}^{(k)},\mathbf{R}^{(k)})$  represents the transformation of the part-centered coordinate system  $\phi^{(k)}$  of primitive  $k$  to the global coordinate system  $\Phi$ ,  $\mathbf{c}^{(k)}\in \mathbb{R}^3$  and  $\mathbf{R}^{(k)}\in \mathbb{R}^{3\times 3}$  represent the translation and rotation of  $\phi^{(k)}$  w.r.t.  $\Phi$  and  $\mathbf{p}^{(k)}$  denotes the relative position of the point on the primitive surface w.r.t.  $\phi^{(k)}$ , which includes global deformation  $\mathbf{s}^{(k)}$  and local deformation  $\mathbf{d}^{(k)}$ . The camera parameters  $\pi \equiv (\mathbf{c}_{\mathrm{cam}},\mathbf{R}_{\mathrm{cam}})$  are used to project  $\mathbf{x}$  onto the image. In Fig. 1, we illustrate the geometry of the proposed part-based shape representation that includes global and local deformations for each part. For the sake of simplicity, we omit the superscript  $k$  for the  $k$ -th primitive in the following.
Primitive deformations. We employ superquadrics to describe the global deformations of each part primitive. Each superquadric surface  $\mathbf{e}$  is explicitly defined by a set of shape-related parameters:
$$
\mathbf {e} = a _ {0} \left[ \begin{array}{l} a _ {1} \cos^ {\varepsilon_ {1}} u \cos^ {\varepsilon_ {2}} v \\ a _ {2} \cos^ {\varepsilon_ {1}} u \sin^ {\varepsilon_ {2}} v \\ a _ {3} \sin^ {\varepsilon_ {1}} u \end{array} \right], \text {w h e r e} - \pi / 2 \leq u \leq \pi / 2, - \pi \leq v \leq \pi . \tag {2}
$$
Here,  $a_0$  is a scaling parameter,  $a_1, a_2, a_3$  denote the aspect ratio for  $x$ -,  $y$ -,  $z$ -axes, respectively, and  $\varepsilon_1, \varepsilon_2$  are squareness parameters. To enable more flexible global deformations, we further include tapering and bending parameters. These additional global deformations are defined as continuously
Figure 2: LEPARD training overview. Given an input image, we map it to a set of primitive parameters that describe  $K$  deformed parts via an encoder-decoder network. The primitive parameters,  $\mathbf{q}_c$ ,  $\mathbf{q}_{\theta}$ ,  $\mathbf{q}_s$ ,  $\mathbf{q}_d$ , are then used to reconstruct the 3D articulated shape  $\mathbf{x}$ . During training, we project a set of points from the primitives' surface onto the image using the predicted camera parameters  $\mathbf{c}_{\mathrm{cam}}$ ,  $\mathbf{R}_{\mathrm{cam}}$ . We use DINO features as supervision to compute image-based forces  $f_{\mathrm{proj}}$  that we further convert into generalized forces  $f_{\mathrm{proj},q}$  to supervise each transformation of the primitive part.
differentiable and commutative functions following [46, 39, 38]. To capture the finer shape details beyond the coverage of global deformations, we employ diffeomorphic point flow to estimate the local non-rigid deformations  $\mathbf{d}$ . Since the deformation with diffeomorphism is differentiable and invertible [2, 10], it guarantees one-to-one mapping and preserves topology during the non-rigid deformations of the primitives. Please refer to [39, 38] for more details. We note that our approach is not restricted to these types of shapes and respective parameterizations. We can replace it with any differentiable type of primitive. However, for the purposes of the articulated animal shapes this type of parameterization is sufficient.
In summary, each part primitive is represented with a set of parameters  $\mathbf{q} = [\mathbf{q}_c,\mathbf{q}_{\theta},\mathbf{q}_s,\mathbf{q}_d]$ , where  $\mathbf{q}_c$  and  $\mathbf{q}_{\theta}$  contain the parameters of the 3D translation and rotation respectively, that transform the part-centered coordinate system  $\phi$  of the primitive to the global coordinate system  $\Phi$ ,  $\mathbf{q}_s$  are the parameters of the global deformations,  $\mathbf{q}_d$  are local deformations that are implemented as a deformation field and are added to the global shape, and  $[\cdot ]$  is the concatenation operator. Compared to the implicit function-based approaches such as NeRF [48] and NeRS [70], all these primitive parameters are defined explicitly for an intuitive understanding of the primitive deformation.
3.2 Primitive 3D Kinematics & Optimization
**Primitive kinematics in 3D.** We use kinematics to define the relationship between any point  $\mathbf{x}$  on the primitive surface and the corresponding primitive parameters  $\mathbf{q}$ . This relationship is expressed quantitatively by the model Jacobian matrix  $\mathbf{L}$ , and this formulation allows us to deform the primitive shape based on its parameters  $\mathbf{q}$ . Specifically, the velocity of  $\mathbf{x}$  is computed as follows:
$$
\dot {\mathbf {x}} = \mathbf {L} \dot {\mathbf {q}}, \tag {3}
$$
where  $\cdot$  denotes the first-order time derivative and  $\mathbf{L} = [\mathbf{I},\mathbf{B},\mathbf{R}\mathbf{J},\mathbf{R}]$  [57] is the Model Jacobian matrix, where each of the four components transforms the 3D points  $\mathbf{x}$  into the translation, rotation, global and local deformation parameters of  $\mathbf{q}$ .  $\mathbf{R}$  is the rotation matrix that corresponds to the rotation between the part-centered coordinate system  $\phi$  of the primitive and the global coordinate system  $\Phi$ .  $\mathbf{B} = \partial \mathbf{R}\mathbf{p} / \partial \mathbf{q}_{\theta}$  is related to the rotation matrix  $\mathbf{R}$  and the relative position  $\mathbf{p}$  of points on the primitive surface.  $\mathbf{J} = \partial \mathbf{s} / \partial \mathbf{q}_{s}$  is the Jacobian matrix. We refer the reader to the supplementary material for more details.
Optimization in 3D. In our modeling paradigm, we minimize the energy of the primitive, defined using the principle of virtual work. The 3D forces on the primitive,  $f_{3\mathrm{D}}$ , result in displacements  $d\mathbf{x}$ :
$$
\mathcal {E} _ {f _ {\mathrm {3 D}}} = \int f _ {\mathrm {3 D}} ^ {\top} d \mathbf {x} = \int f _ {\mathrm {3 D}} ^ {\top} (\mathbf {L} d \mathbf {q}) = \int f _ {q} d \mathbf {q}. \tag {4}
$$
where  $f_{q} = f_{\mathrm{3D}}^{\top}\mathbf{L}$  [57] are the generalized forces acting on the primitive parameters  $\mathbf{q}$ .  $f_{\mathrm{3D}}$  are computed based on the discrepancy between points on the primitive and the target shape in 3D data space (e.g., point-wise difference between the primitive surface and the target shape). The forces on
the primitive are proportional to the distance from the target. Minimizing the generalized forces  $f_{q}$  deforms the primitive to match the target shape and thus can be used as a training objective in our framework.
3.3 Primitive Part Discovery from Images
Our model is trained using a set of  $N$  in-the-wild images of articulated animals. We do not make use of any type of 2D/3D annotations or skeletons. We compute pseudo-labels using semantic clustering of self-supervised DINO [5] features similar to prior work [67, 68]. For each of the given images, we predict the parameters for the  $K$  primitive parts as defined previously. Our approach deforms a set of primitives to fit the target shape under the influence of external forces  $f_{\mathrm{3D}}$ . However, direct primitive parameter optimization using  $f_{\mathrm{3D}}$  is not feasible since we do not have access to any form of 3D supervision. To this end, we project the predicted 3D primitive parts onto the image space and define losses based on the corresponding 2D forces to supervise the deformation of each primitive. The training overview is given in Fig. 2.
Projection kinematics from 3D to 2D. We illustrate the relationship of kinematics between 3D and 2D via projective geometry. This allows us to project the primitives onto the image space and calculate the discrepancy between the projected primitives and 2D evidence. Then, we convert the image forces to their corresponding generalized forces that guide the deformation of the primitives. Specifically, using the estimated parameters for camera translation  $\mathbf{c}_{\mathrm{cam}}$  and rotation  $\mathbf{R}_{\mathrm{cam}}$ , we can convert a given 3D point  $\mathbf{x}$  to the camera coordinate system as follows:
$$
\mathbf {x} _ {\text {c a m}} = \mathbf {c} _ {\text {c a m}} + \mathbf {R} _ {\text {c a m}} \mathbf {x}. \tag {5}
$$
Under perspective projection, the point  $\mathbf{x}_{\mathrm{cam}} = (x_c,y_c,z_c)$  projects onto an image point  $\mathbf{x}_{\mathrm{proj}} = (x_{\mathrm{proj}},y_{\mathrm{proj}})$  according to  $x_{\mathrm{proj}} = f\frac{x_c}{z_c},y_{\mathrm{proj}} = f\frac{y_c}{z_c}$ , where  $f$  is the focal length. By taking the time derivative we have  $d\mathbf{x}_{\mathrm{proj}} = \mathbf{P}d\mathbf{x}_{\mathrm{cam}}$ , where
$$
\mathbf {P} = \left[ \begin{array}{c c c} f / z _ {c} & 0 & - f x _ {c} / z _ {c} ^ {2} \\ 0 & f / z _ {c} & - f y _ {c} / z _ {c} ^ {2} \end{array} \right]. \tag {6}
$$
Thus, from Eq. (5) we have:
$$
d \mathbf {x} _ {\text {p r o j}} = \mathbf {P} d \mathbf {x} _ {\text {c a m}} = \mathbf {P} d \left(\mathbf {c} _ {\text {c a m}} + \mathbf {R} _ {\text {c a m}} \mathbf {x}\right) = \mathbf {P R} _ {\text {c a m}} d \mathbf {x}. \tag {7}
$$
Given Eq. (3) we can rewrite Eq. (7) as:
$$
d \mathbf {x} _ {\text {p r o j}} = \mathbf {P R} _ {\text {c a m}} d \mathbf {x} = \mathbf {P R} _ {\text {c a m}} (\mathbf {L} d \mathbf {q}) = \left(\mathbf {P R} _ {\text {c a m}} \mathbf {L}\right) d \mathbf {q} = \mathbf {L} _ {\text {p r o j}} d \mathbf {q}, \tag {8}
$$
where  $\mathbf{L}_{\mathrm{proj}}$  is the modified model Jacobian matrix that converts image points  $\mathbf{x}_{\mathrm{proj}}$  into primitive parameters  $\mathbf{q}$  that determine translation, rotation, global and local deformations of the primitive.
Loss components. Since we only use 2D supervision during training, the primitive energy from each image we minimize is computed based on the image forces  $f_{\mathrm{proj}}$ :
$$
\mathcal {E} _ {f _ {\mathrm {p r o j}}} = \int f _ {\mathrm {p r o j}} ^ {\top} d \mathbf {x} _ {\mathrm {p r o j}} = \int f _ {\mathrm {p r o j}} ^ {\top} \left(\mathbf {L} _ {\mathrm {p r o j}} d \mathbf {q}\right) = \int f _ {\mathrm {p r o j}, q} d \mathbf {q}, \tag {9}
$$
where the generalized forces  $f_{\mathrm{proj},q} = f_{\mathrm{proj}}^{\top}\mathbf{L}_{\mathrm{proj}}$  guide the primitive deformation. For a given image  $i$  from the training dataset, we minimize the discrepancy between our primitives and the target using the image-based forces as follows:
$$
\mathcal {L} _ {q} ^ {i} = \frac {1}{K} \sum_ {k = 1} ^ {K} f _ {\text {p r o j}, q} ^ {(k, i)} = \frac {1}{K} \sum_ {k = 1} ^ {K} \left(f _ {\text {p r o j}} ^ {(k, i)}\right) ^ {\top} \mathbf {L} _ {\text {p r o j}} ^ {(k, i)}, \tag {10}
$$
where  $f_{\mathrm{proj}}^{(k,i)}$  denotes the corresponding image forces of the  $k$ -th primitive part and is a vector summation of all the forces at each sampling point on the projected primitive. The final loss used for training is computed by summing the generalized forces for all  $N$  training samples as follows:
$$
\mathcal {L} _ {q} = f _ {\operatorname {p r o j}, q _ {c}} ^ {\top} + f _ {\operatorname {p r o j}, q _ {\theta}} ^ {\top} + f _ {\operatorname {p r o j}, q _ {s}} ^ {\top} + f _ {\operatorname {p r o j}, q _ {d}} ^ {\top}, \tag {11}
$$
$$
f _ {\operatorname {p r o j}, q _ {c}} ^ {\top} = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \left(f _ {\operatorname {p r o j}} ^ {(k, i)}\right) ^ {\top} \mathbf {P} ^ {(k, i)} \mathbf {R} _ {\mathrm {c a m}} ^ {(i)},
$$
$$
f _ {\operatorname {p r o j}, q _ {\theta}} ^ {\top} = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \left(f _ {\operatorname {p r o j}} ^ {(k, i)}\right) ^ {\top} \mathbf {P} ^ {(k, i)} \mathbf {R} _ {\mathrm {c a m}} ^ {(i)} \mathbf {B} ^ {(k, i)},
$$
$$
f _ {\operatorname {p r o j}, q _ {s}} ^ {\top} = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \left(f _ {\operatorname {p r o j}} ^ {(k, i)}\right) ^ {\top} \mathbf {P} ^ {(k, i)} \mathbf {R} _ {\operatorname {c a m}} ^ {(i)} \mathbf {R} ^ {(k, i)} \mathbf {J} ^ {(k, i)},
$$
$$
f _ {\mathrm {p r o j}, q _ {d}} ^ {\top} = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \left(f _ {\mathrm {p r o j}} ^ {(k, i)}\right) ^ {\top} \mathbf {P} ^ {(k, i)} \mathbf {R} _ {\mathrm {c a m}} ^ {(i)} \mathbf {R} ^ {(k, i)}.
$$
To obtain  $f_{\mathrm{proj}}^{(k,i)}$ , we render the predicted 3D primitive parts with a differentiable renderer [40] and compute the distance from the pseudo-mask part annotations:
$$
f _ {\operatorname {p r o j}} ^ {(k, i)} = \lambda_ {\operatorname {p r o j}} \left(\mathcal {G} ^ {(i)} - \mathcal {M} _ {\operatorname {p r o j}} ^ {(k, i)}\right), \tag {12}
$$
where  $\mathcal{G}^{(i)}$  and  $\mathcal{M}_{\mathrm{proj}}^{(k,i)}$  are the pseudo-mask and the  $k$ -th projected primitive for instance  $i$ , respectively.  $\lambda_{\mathrm{proj}}$  is a constant modeling the strength of the image force  $f_{\mathrm{proj}}^{(k,i)}$ .
Force regularization. To enable more robust primitive fitting strategy, we incorporate regularization to the forces during training. We follow the physics-based deformable models (DMs) [46] and avoid the collisions between primitives by checking for primitive inter-penetration in each training iteration. If two primitives penetrate each other, we allocate two equivalent and opposite collision forces  $f_{n}$  and  $-f_{n}$  that are proportional to the distance between each pair of selected points on the two primitives. These two forces are added to the respective points on the two inter-penetrating primitives, respectively, to adjust the external forces  $f_{\mathrm{proj}}$  and thus push the primitives to separate from each other.