SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{I}^j$  denotes the clean input image of instance  $j$  after input preprocessing and  $\hat{M}^j$  denotes the corresponding animal mask;  $\hat{I}^j$  is the rendered RGB image from the estimated 3D shape and texture; and  $\odot$  denotes element-wise product. The reconstruction loss is masked by the estimated foreground silhouette so that the texture optimization is only effected by the visible non-occluded animal pixels.
Distilling 3D reconstruction. In addition to the aforementioned losses, we propose to increase the shape and texture details by distilling 3D reconstruction. Here, we use DASS as a critic to evaluate how well a 3D reconstruction looks in its 2D renders, and calculate pixel gradients from the image target. Similar to prior diffusion-based methods [21, 18, 15], we render the 3D surfaces with random viewpoints, lighting, and background colors during training. Moreover, we design a pose exploration scheme to densify the articulation space in our sparse-image scenario. In particular, we randomly interpolate the estimated bone rotation  $(R^{j_1}, R^{j_2})$  of two instances  $(j_1, j_2)$ , and generate a new instance with novel pose  $R' = \alpha R^{j_1} + (1 - \alpha)R^{j_2}$  for rendering, where  $\alpha \in (0,1)$  is a random scalar. As such, we can better constrain the part deformation by diffusion prior and prevent irregular shape or disconnection between parts. As shown in Fig. 2, we then diffuse the latent codes of rendered images and obtain pixel gradients from the DASS module. The resulting gradients are back-propagated to update the part surface texture, deformation MLP, bone transformation, and camera viewpoints. In our experiments, we observe that the RGB gradients do not propagate well through the SoftRas [16] blending function, and we thus modify it with a layered blending approach proposed in [20].
**Optimization details.** The overall optimization objective can be expressed as the weighted sum of all the losses  $\mathcal{L} = \sum_{l\in \mathfrak{L}}\alpha_l\mathcal{L}_l$ , where  $\mathfrak{L} = \{\text{sil, sem, rot, lap, norm, text, dass}\}$  as described above. We optimize the shared and instance-specific shapes in two stages. That is, we first update the shared part MLPs along with camera viewpoints and pose parameters. Then, we fine-tune the instance-specific part MLPs and optimize texture images for each instance. All model parameters are updated using an Adam optimizer [10]. We render the images at  $512\times 512$  resolution and at  $128\times 128$  for the part texture images. More optimization details are described in the supplemental material.
3.5 Animation fine-tuning
One can easily animate the resulting 3D articulated animals by gradually rotating the skeleton bones and their corresponding parts surfaces. However, the rigid part transformations often result in disconnected shapes or texture around the joints. To improve the rendered animation in 2D, one can naively use DASS frame-by-frame on a sequence of articulated shapes. However this can produce artifacts like color flickering and shape inconsistency across the frames. As a remedy, we further propose a fine-tuning step, called Temporal-DASS (T-DASS), to generate high-quality and temporally consistent 2D animations based on the ARTIC3D outputs. Given a sequence of part transformations from simple interpolation across instances or motion re-targeting, we render the 3D surfaces as video frames  $\{J_k \in \mathbb{R}^{H \times W \times 3} (k \in \{1, \dots, K\})\}$  and encode them into latent codes  $\{z_k \in \mathbb{R}^{h \times w \times 3}\}$  through the SD encoder  $\mathcal{E}$ . Then, we design a reconstruction loss  $\mathcal{L}_{\text{recon}}$  and temporal consistency loss  $\mathcal{L}_{\text{temp}}$  to fine-tune the animation in the latent space. Similar to DASS, we obtain the reconstruction targets  $\{z_k'\}$  by accumulating latent SDS gradients  $\nabla z_k$  for multiple steps:  $z_k' = z_k - \nabla z_k$ . The reconstruction loss can then be written as:  $\mathcal{L}_{\text{recon}} = \sum_t \| (z_k - z_k') \|^2$ . To enforce temporal consistency, we exploit our 3D surface outputs and calculate accurate 2D correspondences across neighboring frames. Specifically, for each latent pixel in frame  $z_k$ , we find the closest visible 3D surfaces via mesh rasterization, then backtrack their 2D projection in frame  $z_{k-1}$ , forming a dense 2D flow field  $F_k \in \mathbb{R}^{h \times w \times 2}$ . Intuitively, the corresponding pixels should have similar latent codes. Hence, we use  $F_k$  to perform temporal warping on the latent codes  $z_{k-1}$ , denoted as:  $\text{warp}(z_{k-1}, F_k)$ , and define  $\mathcal{L}_{\text{temp}}$  as:
$$
\mathcal {L} _ {\text {t e m p}} = \sum_ {k = 2} ^ {K} \| \left(z _ {k} - \operatorname {w a r p} \left(z _ {k - 1}, F _ {k}\right) \right\| ^ {2}. \tag {3}
$$
We fine-tune the latent codes  $\{z_k\}$  with  $\mathcal{L}_{\text{recon}}$  and  $\mathcal{L}_{\text{temp}}$ , where  $\{F_k\}$  are pre-computed and  $\{z_k'\}$  are updated in each iteration. Finally, we can simply obtain the RGB video frames by passing the optimized latent codes through the SD decoder  $\{\mathcal{D}(z_k)\}$ . The proposed  $\mathcal{L}_{\text{recon}}$  encourages better shape and texture details in each frame, and  $\mathcal{L}_{\text{temp}}$  can effectively regularize latent updates temporally. Note that T-DASS optimizes the latent codes and takes temporal consistency into account, which is different from DASS which operates on each image individually.
Figure 4: E-LASSIE samples. We extend LASSIE [39] image sets with 15 occluded or truncated images per animal class and annotate the 2D keypoints for evaluation. These noisy images pose great challenges to sparse-image optimization since the per-instance 3D shapes can easily overfit to the visible parts and ignore the rest.
Datasets. Following [39, 38], we evaluate ARTIC3D on the Pascal-Part [5] and LASSIE [39] images. From Pascal-Part, we obtain images of horse, cow, and sheep, as well as their 2D keypoints automatically computed using the ground-truth 2D part masks. The LASSIE dataset includes web images of other animal species (zebra, tiger, giraffe, elephant, kangaroo, and penguin) and 2D keypoint annotations. Each image collection contains roughly 30 images of different instances with diverse appearances, which are manually filtered so that the animal bodies are fully visible in the images. To evaluate the model robustness in a more practical setting, we extend the LASSIE image sets with several noisy images where the animals are occluded or truncated. In particular, we collect 15 additional web images (CC-licensed) per class and annotate the 2D keypoints for evaluation. We call the extended image sets E-LASSIE and show some examples in Fig. 4. For the experiments on E-LASSIE, we optimize and evaluate on all the 45 images in each set.
Baselines. We mainly compare ARTIC3D with LASSIE [39] and Hi-LASSIE [38] as we deal with the same problem setting, namely sparse image optimization for articulated animal shapes. For reference, we also compare the results with several learning-based methods like A-CSM [11], MagicPony [15], and Farm3D [9]. Note that these approaches are not directly comparable to ARTIC3D since they train a feedforward network on large-scale image sets (not available in our scenario). Although related, some other recent works on 3D surface reconstruction either cannot handle articulations [12, 14, 7, 31, 40] or require different inputs [13, 34, 36]. As a stronger baseline, we implement Hi-LASSIE+, incorporating the standard SDS loss as in [27, 15, 18] (back-propagate latent gradients through encoder) during Hi-LASSIE [38] optimization for shape and texture.
Evaluation metrics. Considering the lack of ground-truth 3D annotations in our datasets, we follow a common practice [45, 11, 39, 38] to use keypoint transfer accuracy as a quantitative metric to evaluate 3D reconstruction. For each pair of images, we map the annotated 2D keypoints on source image onto the canonical 3D surfaces, re-project them to the target image via the estimated camera, pose, and shape, and compare the transferred keypoints with target annotations. To further evaluate the quality of textured outputs, we compute CLIP [22] features of the 3D output renders under densely sampled viewpoints, and calculate the feature similarity against text prompt as well as input images. While most prior arts on 3D shape generation [21, 23] only evaluate the image-text similarity, we also evaluate the image-image similarity since our outputs should be faithful to both the category-level textual description as well as instance-specific input images. We use a text prompt: "A photo of *" for each animal class "*'" in our experiments. A CLIP ViT-B/32 model is used to compute the average feature similarity over 36 uniformly sampled azimuth renders at a fixed elevation of 30 degrees. We show the main results here and more quantitative and qualitative comparisons in the supplemental material, including animation videos, user study, and more detailed ablation study.
Qualitative results. Fig. 1 shows some sample outputs of ARTIC3D. In Fig. 5, we compare the visual results of Hi-LASSIE, Hi-LASSIE+, and ARTIC3D on the E-LASSIE images. Both Hi-LASSIE and Hi-LASSIE+ produce irregular pose and shape for the invisible parts. Regarding surface texture, Hi-LASSIE reconstructs faithful texture from the input view but noisy in novel views, since it naively samples vertex colors from the input images. The output texture of Hi-LASSIE+ is generally less
Figure 5: Visual comparison of ARTIC3D and other baselines. For each input image, we show the 3D textured outputs from input (upper) and novel (lower) views. The results demonstrate that ARTIC3D is more robust to noisy images with occlusions or truncation, producing 3D shape and texture that are detailed and faithful to the input images.
noisy thanks to the SDS loss. By comparison, ARTIC3D accurately estimates the camera viewpoint, pose, shape, and texture even with the presence of occlusions or truncation. Note that there exists a tradeoff between realism and faithfulness to input images due to the highly ill-posed nature of our problem setting, especially for the unseen/occluded surface. For instance, whether a colored or black-and-white output is preferred given a grey-scale/poorly illuminated image, or whether we preserve the noisy texture (small occlusions like dirt, water splash, or shadow caused by rough surface / objects) in the original image. Since the dense surface visibility is hard to obtain, we optimize the shape and texture that are slightly biased towards realism (detailed and clean texture that resembles the animal class) in this paper. In our ablative analysis of the DASS module (Fig. 3), we show that we can control the realism-faithfulness tradeoff by tuning the diffusion timestep  $t$  and guidance weight  $w_{g}$ . Specifically, larger  $t$  and  $w_{g}$  allows DASS to hallucinate shape and texture that are not present in the original image. In addition, one can enforce a more faithful reconstruction by setting a higher weight of the texture reconstruction loss  $\alpha_{text}$ . More qualitative comparisons against other baselines can be found in the supplemental material.
Quantitative comparisons. We show comparisons of the keypoint transfer accuracy (PCK) in Tables 1. On both LASSIE and E-LASSIE image sets, Hi-LASSIE+ produces a marginal PCK gain from Hi-LASSIE [38] by naively applying the SDS loss. ARTIC3D, on the other hand, achieves consistently higher PCK than the baselines, especially on the noisy E-LASSIE images. The results demonstrate that our diffusion-guided strategies can effectively learn more detailed, accurate, and robust 3D shapes. The Pascal-Part results in Tab 2 further show that ARTIC3D performs favorably against the state-of-the-art optimization-based methods and are comparable to learning-based approaches. In Table 3, we show the CLIP similarity comparisons on the E-LASSIE images, which indicate that our textured outputs are more faithful to both the input images (instance-level) and text prompt (class-level) for most animal classes.
Animation and texture transfer. In Fig. 6, we compare the animations before and after our fine-tuning step via T-DASS. While the skeleton-based representation allows easy animation via rigid part transformation, the output part shapes and texture are often disconnected and irregular around the joints. The results show that T-DASS can effectively produce high-quality animations that are detailed in shape and texture and temporally consistent between frames. In addition to animation, our 3D part surfaces also enables convenient controllable syntheses like texture transfer and pose transfer between different instance or animal classes. Several examples of texture transfer are shown in Fig. 7. More visual results with video results of these applications are shown in the supplemental material.
Limitations. ARTIC3D relies on the 3D skeleton discovered by Hi-LASSIE [38] to initialize the parts. If the animal bodies are occluded or truncated in most images, the skeleton initialization tends to be inaccurate, and thus limiting ARTIC3D's ability to form realistic parts. Although our input preprocessing method can mitigate this issue to some extent, fluffy animals (e.g. sheep) with
Table 1: Keypoint transfer evaluations on the LASSIE [39] and E-LASSIE image sets. We report the average PCK@0.05 (↑) on all pairs of images. ARTIC3D performs favorably against the optimization-based prior arts on all animal classes. The larger performance gap in the E-LASSIE demonstrates that ARTIC3D is robust to noisy images.
Method	Image set	Elephant	Giraffe	Kangaroo	Penguin	Tiger	Zebra
LASSIE [39]	LASSIE	40.3	60.5	31.5	40.6	62.4	63.3
Hi-LASSIE [38]	LASSIE	42.7	61.6	35.0	44.4	63.1	64.2
Hi-LASSIE+ARTIC3D	LASSIE	43.3	61.5	35.5	44.6	63.4	64.0
LASSIE	44.1	61.9	36.7	45.3	64.0	64.8
Hi-LASSIE [38]	E-LASSIE	37.6	54.3	31.9	41.7	57.4	60.1
Hi-LASSIE+ARTIC3D	E-LASSIE	38.3	54.8	32.8	41.8	57.7	61.3
E-LASSIE	39.8	58.0	35.3	43.8	59.3	63.0
Table 2: Keypoint transfer results on Pascal-Part [6]. We report the mean PCK@0.1 (↑) on all pairs of images. * indicates learning-based models which are trained on a large-scale image set.
Method	Horse	Cow	Sheep
UMR* [14]	24.4	-	-
A-CSM* [11]	32.9	26.3	28.6
MagicPony* [33]	42.9	42.5	26.2
Farm3D* [9]	42.5	40.2	32.8
LASSIE [39]	42.2	37.5	27.5
Hi-LASSIE [38]	43.7	42.1	29.9
Hi-LASSIE+	43.3	42.3	30.5
ARTIC3D	44.4	43.0	31.9
Table 3: CLIP similarity  $(\uparrow)$  evaluations on the E-LASSIE images. For each animal class, we calculate cosine similarities  $s1 / s2$ , where  $s1$  is the image-image similarity (against masked input image) and  $s2$  is the image-text similarity (against text prompt).
Method	Elephant	Giraffe	Kangaroo	Penguin	Tiger	Zebra
Hi-LASSIE [38]	80.0 / 26.3	85.2 / 29.6	77.4 / 25.6	85.8 / 30.8	79.7 / 25.6	83.8 / 27.4
Hi-LASSIE+	79.0 / 27.7	84.7 / 30.2	78.3 / 29.1	82.9 / 32.3	75.3 / 25.3	81.9 / 27.6
ARTIC3D	82.6 / 28.4	85.3 / 30.7	81.6 / 29.9	85.5 / 33.1	80.0 / 27.8	84.1 / 29.4
Figure 6: Animation fine-tuning. Compared to the original animated outputs via rigid transformation (top), our animation fine-tuning (bottom) effectively improves the shape and texture details, especially around animal joints.
Figure 7: Texture transfer. Our part surface representation enables applications like pose or texture transfer. Given a source shape and target texture, we show the transferred texture between instances (left) and animal species (right).
ambiguous skeletal configuration can still pose challenges in skeleton discovery. In addition, the front-facing bias in diffusion models sometimes lead to unrealistic texture like multiple faces, which also affects our reconstruction quality. See the supplemental material for failure cases.
We propose ARTIC3D, a diffusion-guided framework to reconstruct 3D articulated shapes and texture from sparse and noisy web images. Specifically, we design a novel DASS module to efficiently calculate pixel gradients from score distillation for 3D surface optimization. We use the DASS module and its variants in the input preprocessing of noisy images; shape and texture optimization; as well as the animation fine-tuning stages. In addition to the existing Pascal-Part and LASSIE datasets, we also evaluate on the self-collected E-LASSIE images with occlusions and truncation. Results on these datasets demonstrate that ARTIC3D produces more robust, detailed, and realistic reconstructions against prior arts. As our future work, we hope to extend ARTIC3D to more general articulated objects.