SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets and baselines. We emphasize that our aim is to reconstruct an articulated shape by discovering semantically meaningful parts in 3D. Thus, we mainly compare our approach with LASSIE [67] and Hi-LASSIE [68] who share the same goal. We conduct extensive experiments on a few animal categories, including horses, giraffes, zebras, etc. For horses, we train our model using 11k images collected from Pascal [15], LASSIE [67] and DOVE [59] datasets. For the other categories, we only use the images from Pascal [15] and LASSIE [67] datasets to fine-tune our pre-trained model. To put our results into perspective, we also quantitatively compare with A-CSM [30] and 3D Safari [73], which learn to reconstruct the 3D articulated shape of animals holistically. Finally, we also provide qualitative comparisons with LASSIE using their released code. We only compare quantitatively with Hi-LASSIE, using the reported results in [68], since their code is not available at the time of submission.
Qualitative results. In Fig. 3 we provide qualitative comparisons with LASSIE. We observe that LEPARD yields more geometrically accurate reconstructions with finer details compared to LASSIE. This is particularly evident in the cases of tiger and elephant, where the primitives used in LASSIE do not accurately correspond to the animal parts with complex shapes such as the tiger tail and the elephant nose. Additionally, LASSIE fails in the presence of additional objects in the background (see the predicted shape for the kangaroo in row 4). For the case of the kangaroo, our model accurately locates the head and correctly recovers the pose under the interference of the inaccurate clustering appearing on the top right of the DINO features. The superiority of our model in estimating accurate poses can also be observed in the case of the penguin (row 5 of Fig. 3).
Consistency visualization. In the first two rows of Fig. 4 we show that LEPARD predicts consistent parts across different samples of the same animal category, where LASSIE struggles (e.g., in the 5-th column, LASSIE predicts the elephant's nose as a leg). In addition, we visualize the semantic consistency of our model across different animal categories. In the last two rows of Fig. 4, we compare the reconstruction results of kangaroos and penguins. We observe that our approach employs the same primitive to consistently represent corresponding parts of different species, e.g., the heads of both penguins and kangaroos are represented with the same primitive part (highlighted in green). But LASSIE employs this same green part to represent the mouth of penguins and the head of kangaroos, respectively.
Figure 3: Qualitative results. We compare the recovered parts by LASSIE [67] and LEPARD (ours). We observe that LEPARD discovers semantically meaningful and consistent parts, offering more faithful reconstructions than LASSIE. Unlike LASSIE, our approach is robust to the presence of additional objects in the background (see the kangaroo in row 4). Failures in LASSIE's prediction are indicated by arrows, whereas LEPARD avoids such failures.
Keypoint transfer. Due to the lack of ground-truth 3D annotations in our datasets, we follow a common practice [30, 67, 68] and quantitatively evaluate our model using 2D keypoint transfer between each pair of images in the test set. In particular, given a set of keypoints on a source image, we map them onto the 3D primitive parts and then project them to the target image. We then compute the percentage of correct keypoints (PCK) under a tight threshold  $0.05 \times \max(h, w)$  (i.e., PCK@0.05), where  $h$  and  $w$  are the image height and width respectively. For a successful 2D  $\rightarrow$  3D  $\rightarrow$  2D mapping, accurate 3D reconstructions for both the source and target images are necessary. We report results for the keypoint transfer evaluation in Table 1. The results show that LEPARD achieves higher PCK compared to the baselines without performing test-time per-instance optimization.
Part transfer. Next, we evaluate our approach on part transfer using the ground-truth part segmentation masks from the Pascal-Part dataset. Similar to 2D keypoint transfer, we transfer part annotations from source to target images through a  $2\mathrm{D}\rightarrow 3\mathrm{D}\rightarrow 2\mathrm{D}$  mapping utilizing the predicted 3D part primitives for the sources and target images. In this setting, we measure the performance with the percentage of correct pixels (PCP) metric, where a pixel from the source image is considered to be transferred correctly if it is mapped to the same semantic part in the target image. The results
Figure 4: Consistency visualization. We compare the discovered parts by LEPARD and LASSIE [67]. The arrows indicate the correct correspondence of the parts among instances (e.g., the left front legs of the elephants should be discovered by primitive parts in the same color) from the same (first two rows) and different (last two rows) animal categories. We observe that LEPARD discovers 3D parts with better semantic consistency (i.e., reconstructs the same parts using the same primitives).
Table 1: Keypoint transfer evaluation. Results on the Pascal-Part and LASSIE datasets using PCK@0.05. LEPARD outperforms all methods for all animal categories by a significant margin.
Pascal-Part dataset	LASSIE dataset
Horse	Cow	Sheep	Zebra	Tiger	Giraffe	Elephant	Kangaroo	Penguin
3D Safari [73]	57.1	50.3	50.5	62.1	50.3	32.5	29.9	20.7	28.9
A-CSM [30]	55.3	60.5	54.7	60.3	55.7	52.2	39.5	26.9	33.0
LASSIE [67]	58.0	62.4	55.5	63.3	62.4	60.5	40.3	31.5	40.6
Hi-LASSIE [68]	59.6	63.1	56.2	64.2	63.1	61.6	42.7	35.0	44.4
LEPARD	61.0	63.7	56.9	64.7	63.8	62.1	43.2	35.4	44.6
are reported on the right column of Table 2, demonstrating that LEPARD compares favorably to all baseline methods.
2D IoU. In addition, we evaluate LEPARD using overall and part IoU between the ground-truth and rendered masks. The results are reported in Table 2. We follow the baselines [68, 67, 25, 5] and manually assign the discovered parts to the best-matched parts in the Pascal-Part segmentation masks. From Table 2 we observe that LEPARD outperforms all other methods in terms of overall IoU as well as Part IoU by a large margin.
Effect of local deformations. We investigate the effect of local deformations in terms of reconstruction accuracy and present qualitative results in Fig. 5. We observe that local deformations can capture fine-grained shape details and significantly improve the visual quality of the reconstructed shapes.
Model robustness and limitation. In Fig. 6 we evaluate on some held-out images from the Objaverse dataset for known categories such as elephants to show the robustness of our method. For the reconstruction of unknown categories, our method may require fine tuning on additional images, and will be included in our future work.
Table 2: Quantitative results on the Pascal-Part [7] dataset. We report the overall IoU, part mask IoU, as well as part transfer results measured by the percentage of correct pixels (PCP).
Overall IoU	Part IoU	Part Transfer (PCP)
Horse	Cow	Sheep	Horse	Cow	Sheep	Horse	Cow	Sheep
SCOPS [25]	62.9	67.7	63.2	23.0	19.1	26.8	-	-	-
DINO clustering [1]	81.3	85.1	83.9	26.3	21.8	30.8	-	-	-
3D Safari [73]	72.2	71.3	70.8	-	-	-	71.7	69.0	69.3
A-CSM [30]	72.5	73.4	71.9	-	-	-	73.8	71.1	72.5
LASSIE [67]	81.9	87.1	85.5	38.2	35.1	43.7	78.5	77.0	74.3
Hi-LASSIE [68]	83.4	88.1	86.3	39.0	35.3	43.4	79.9	77.8	75.5
LEPARD	83.7	88.3	86.7	39.5	35.4	43.6	80.6	77.6	75.8
Figure 5: Effect of local deformations. We test the effectiveness of the local deformations on the Pascal-Part and the LASSIE dataset. We observe that local deformations have a significant impact on the visual quality of the reconstructed shapes.
LEPARD w/o local
LEPARD
Figure 6: Testing on held-out images from known category. We show some qualitative results on the elephant category of Objaverse dataset.
We present LEPARD, a learning-based framework for discovering the 3D parts and shape of an articulated animal from an image without any 2D/3D annotations or skeletons. Our approach jointly optimizes a set of primitives which are explicitly defined by a few shape-related parameters, and provides an intuitive understanding of the primitive deformation. To obtain high-fidelity reconstruction that matches the 2D evidence provided by clustering DINO features, we propose a kinematics mapping between 3D and 2D, and convert the 2D forces defined w.r.t. images to the generalized forces that supervise the motions and deformations of the primitive parts in 3D. Our approach does not require any test-time processing and discovers semantically meaningful and consistent 3D parts. We demonstrate the effectiveness of our approach through quantitative and qualitative evaluations where we achieve significant improvements over existing methods.
This research has been partially funded by research grants to D. Metaxas through NSF: IUCRC CARTA 1747778, 2235405, 2212301, 1951890, 2003874, and NIH-5R01HL127661.