SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chun-Han Yao $^{1*}$  Amit Raj $^{2}$  Wei-Chih Hung $^{3}$  Yuanzhen Li $^{2}$  Michael Rubinstein $^{2}$  Ming-Hsuan Yang $^{124}$  Varun Jampani $^{2}$
$^{1}$ UC Merced  $^{2}$ Google Research  $^{3}$ Waymo  $^{4}$ Yonsei University
Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging due to the ambiguities of camera viewpoint, pose, texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. Specifically, ARTIC3D is built upon a skeleton-based surface representation and is further guided by 2D diffusion priors from Stable Diffusion. First, we enhance the input images with occlusions/truncation via 2D diffusion to obtain cleaner mask estimates and semantic features. Second, we perform diffusion-guided 3D optimization to estimate shape and texture that are of high-fidelity and faithful to input images. We also propose a novel technique to calculate more stable image-level gradients via diffusion models compared to existing alternatives. Finally, we produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations. Extensive evaluations on multiple existing datasets as well as newly introduced noisy web image collections with occlusions and truncation demonstrate that ARTIC3D outputs are more robust to noisy images, higher quality in terms of shape and texture details, and more realistic when animated.
Articulated 3D animal shapes are widely used in applications such as AR/VR, gaming, and content creation. However, the articulated models are usually hard to obtain as manually creating them is labor intensive and 3D scanning real animals in the lab settings is highly infeasible. In this work, we aim to automatically estimate high-quality 3D articulated animal shapes directly from sparse and noisy web image collections. This is a highly ill-posed problem due to the variations across images with diverse backgrounds, lighting, camera viewpoints, animal poses, shapes, and textures, etc. In addition, we do not assume access to any 3D shape models or per-image annotations like keypoints and camera viewpoints in our in-the-wild setting.
While several recent methods [39, 33, 38] can produce animatable 3D shapes using a skeleton-based neural surface or pre-defined mesh template, the success is largely dependent on large-scale image datasets or manually-filtered clean images for training or optimization. Moreover, the output 3D shapes and textures are usually unrealistic when viewed from novel viewpoints or pose articulations. On the other hand, recent success of generative diffusion models [25, 28, 27] shows that one can generate high-quality images for a given text prompt. DreamFusion [21] and other recent works [15, 18, 23] further demonstrate the possibility to produce 3D objects/scenes simply using 2D diffusion as multi-view supervision. In this work, we leverage the powerful 2D diffusion prior
*Work done as a student researcher at Google.
Noisy web images
Articulated shapes (input and novel views)
Textured outputs (input and novel views)
Figure 1: Learning articulated 3D shapes from noisy web images. We propose ARTIC3D, a diffusion-guided optimization framework to estimate the 3D shape and texture of articulated animal bodies from sparse and noisy image in-the-wild. Results show that ARTIC3D outputs are detailed, animatable, and robust to occlusions or truncation.
Animated
Fine-tuned animation
to learn 3D articulated shapes, aiming to reconstruct and animate 3D animals from sparse noisy online images without any 2D or 3D annotations. Intuitively, one can improve the quality of 3D reconstructions by utilizing a diffusion prior similar to the score distillation sampling (SDS) loss proposed in DreamFusion [21]. In our experiments, nonetheless, we observe that naively applying the SDS loss on 3D surface optimization leads to unstable and inefficient training, producing undesirable artifacts like noisy surfaces or ambiguous texture.
In this work, we present ARTIC3D (ARTiculated Image Collections in 3D), a diffusion-guided optimization framework to learn articulated 3D shapes from sparse noisy image collections. We use the articulated part surface and skeleton from Hi-LASSIE [38], which allows explicit part manipulation and animation. We propose a novel Decoder-based Accumulative Score Sampling (DASS) module that can effectively leverage 2D diffusion model priors from Stable Diffusion [27] for 3D optimization. In contrast to existing works that back-propagate image gradients through the latent encoder, we propose a decoder-based multi-step strategy in DASS, which we find to provide more stable gradients for 3D optimization. To deal with noisy input images, we propose an input preprocessing scheme that use diffusion model to reason about occluded or truncated regions. In addition, we also propose techniques to create realistic animations from pose articulations.
We analyze ARTIC3D on the Pascal-Part [5] and LASSIE [39] datasets. To better demonstrate the robustness to noisy images, we extend LASSIE animal dataset [39] with noisy web animal images where animals are occluded and truncated. Both qualitative and quantitative results show that ARTIC3D produces 3D shapes and textures that are detailed, faithful to input images, and robust to partial observations. Moreover, our 3D articulated representation enables explicit pose transfer and realistic animation which are not feasible for prior diffusion-guided methods with neural volumetric representations. Fig. 1 shows sample 3D reconstructions and applications from ARTIC3D. The main contributions of this work are:
- We propose a diffusion-guided optimization framework called ARTIC3D, where we reconstruct 3D articulated shapes and textures from sparse noisy online images without using any pre-defined shape templates or per-image annotations like camera viewpoint or keypoints.
- We design several strategies to efficiently incorporate 2D diffusion priors in 3D surface optimization, including input preprocessing, decoding diffused latents as image targets, pose exploration, and animation fine-tuning.
- We introduce E-LASSIE, an extended LASSIE dataset [39], by collecting and annotating noisy web images with occlusions or truncation to evaluate model robustness. Both qualitative and quantitative results show that ARTIC3D outputs have higher-fidelity compared to prior arts in terms of 3D shape details, texture, and animation.
Animal shape and pose estimation. Earlier techniques on animal shape estimation used statistical body models [46, 45] that are learned either using animal figurines or a large number of annotated animal images. Some other works [35, 34, 36, 37], use video inputs to learn articulated shapes by exploiting dense correspondence information in video. However, these methods rely on optical flow correspondences between video frames, which are not available in our problem setting. Other techniques [12, 11] leverage a parametric mesh model and learn a linear blend skinning from images to obtain a posed mesh for different animal categories. MagicPony [33] learns a hybrid 3D representation of the animal instance from category specific image collections. Most related to our work are LASSIE [39] and Hi-LASSIE [38], which tackle the same problem setting of recovering 3D shapes from a sparse collection of animal images in the wild using either a manually annotated skeleton template or by discovering category-specific template from image collections. However, these approaches require carefully curated input data and fail to handle image collections with partial occlusions, truncation, or noise. By leveraging recent advances in diffusion models, we support reconstruction on a wider variety of input images.
3D reconstruction from sparse images. Several recent works [30, 42, 41, 32, 24, 2, 43, 3] have used implicit representations [19] to learn geometry and appearance from sparse image collections either by training in a category specific manner or assuming access to multi-view consistent sparse images during inference. However, most of these approaches demonstrate compelling results only on rigid objects. Zhang et al. [42] is another closely related work that finds a neural surface representation from sparse image collections but requires coarse camera initialization. By learning a part based mesh shape and texture, our framework naturally lends itself to modeling and animating articulated categories such as animals in the wild without any additional requirements on camera parameters.
Diffusion prior for 3D. Diffusion models [27, 28, 44] have recently gained popularity for generating high resolution images guided by various kinds of conditioning inputs. Diffusion models capture the distribution of real data which can be used as score function to guide 3D generation with score-distillation sampling (SDS) loss as first described in DreamFusion [21]. Several recent approaches [18, 15, 17, 29, 26, 23] leverage the SDS loss to generate 3D representations from either text or single or sparse image collections. Drawing inspiration from these lines of work, we propose a novel Decoder-based accumulative Score Sampling (DASS) that exploits the high quality images synthesized by the decoder and demonstrate improved performance over naive SDS loss.
3 Approach
Given 10-30 noisy web images of an animal species, ARTIC3D first preprocesses the images via 2D diffusion to obtain cleaner silhouette estimates, semantic features, and 3D skeleton initialization. We then jointly optimizes the camera viewpoint, pose articulation, part shapes and texture for each instance. Finally, we animate the 3D shapes with rigid bone transformations followed by diffusion-guided fine-tuning. Before introducing our diffusion-based strategies to improve the quality of 3D outputs, we briefly review the skeleton-based surface representation similar to [39, 38], as well as Stable Diffusion [27] that we use as diffusion prior.
While most 3D generation methods optimize a volumetric neural field to represent 3D rigid objects/scenes, we aim to produce 3D shapes that are articulated and animatable. To enable explicit part manipulation and realistic animation, we adopt a skeleton-based surface representation as in LASSIE [39] and Hi-LASSIE [38]. Nevertheless, unlike [39, 38] which naively sample surface texture from images, we optimize per-part texture images to obtain realistic instance textures from novel views. At a high level, [39, 38] focus on using geometry priors to learn detailed articulated shapes, whereas ARTIC3D further incorporates generative 2D diffusion priors to learn both shapes and texture in a more challenging scenario with noisy images.
3D Skeleton. Given a user-specified reference image in the collection, Hi-LASSIE [38] automatically discovers a 3D skeleton based on the geometric and semantic cues from DINO-ViT [4] feature clusters. The skeleton initializes a set of 3D joints and primitive part shapes, providing a good constraint of part transformation and connectivity. In our framework, we obtain cleaner feature
Figure 2: ARTIC3D overview. Given sparse web images of an animal species, ARTIC3D estimates the camera viewpoint, articulated pose, 3D part shapes, and surface texture for each instance. We propose a novel DASS module to efficiently compute image-level gradients from stable diffusion, which are applied in 1) input preprocessing, 2) shape and texture optimization, and 3) animation.
clusters by diffusing input images, then applying Hi-LASSIE as an off-the-shelf skeleton discovery method. For a fair comparison with existing works, we use the same reference image for skeleton discovery as in [38] in our experiments. Please refer to [38] for further details on skeleton discovery.
Neural part surfaces. Following [38], using the discovered 3D skeleton, we reconstruct a 3D part corresponding to each skeleton bone via a deformable neural surface [42]. The neural surfaces are parameterized by multi-layer perceptron networks (MLPs), mapping 3D surface points on a unit sphere to their xyz deformation. Given  $m$  uniformly sampled 3D points  $X \in \mathbb{R}^{3 \times m}$  on a spherical surface, we can deform the 3D shape of the  $i$ -th part through the part MLP as  $X \mapsto \mathcal{F}_i(X)$ . Then, the part surfaces are rigidly transformed by the scaling  $s_i \in \mathbb{R}$ , rotation  $R_i \in \mathbb{R}^{3 \times 3}$ , and translation  $t_i \in \mathbb{R}^3$  of each skeleton part  $i$ . The transformed part surface points  $V_i$  in the global coordinate can be written as:  $V_i = s_i R_i \mathcal{F}_i(X) + t_i$ . Please refer to [38] for further details.
Stable Diffusion architecture. Stable Diffusion (SD) [27] is a state-of-the-art text-to-image generative model that can synthesize high-quality images given a text prompt. SD mainly consists of 3 components: An image encoder  $\mathcal{E}$  that encodes a given image  $x$  into a latent code  $z$ ; a decoder network  $\mathcal{D}$  that converts the latent code back to image pixels; and a U-Net denoiser  $\epsilon_{\phi}$  that can iteratively denoise a noisy latent code. We use SD as a diffusion prior in our framework.
3.2 Decoder-based Accumulative Score Sampling (DASS)
To leverage the 2D diffusion prior for 3D shape learning, DreamFusion [21] proposes a score distillation sampling (SDS) loss to distill the images rendered from random views and propagate the image-level gradients to Neural Radiance Field (NeRF) parameters. To reduce the computational cost and improve training stability, recent works like Latent-NeRF [18] and Magic3D [15] perform distillation on the low-resolution latent codes in SD and back-propagate the gradients through the SD image encoder  $\mathcal{E}$ . Formally, let  $x$  be a rendered image from 3D model and  $z$  denote its latent codes from the SD image encoder  $\mathcal{E}$ . At each score distillation iteration, the latent codes  $z$  are noised to a random time step  $t$ , denoted as  $z_{t}$ , and denoised by the U-Net denoiser  $\epsilon_{\phi}$  of the diffusion model. The image-level SDS gradients can then be expressed as:
$$
\nabla_ {x} \mathcal {L} _ {\mathrm {S D S}} = w _ {t} \left(\epsilon_ {\phi} \left(z _ {t}; y, t\right) - \epsilon\right) \frac {\partial z}{\partial x}, \tag {1}
$$
where  $y$  denotes the guiding text embedding,  $\epsilon$  is the random noise added to the latent codes, and  $w_{t}$  is a constant multiplier which depends on diffusion timestep  $t$ . The denoiser  $\epsilon_{\phi}$  uses a guidance scale  $w_{g}$  to balance the text guidance and a classifier-free guidance [8] of an unconditional model.
Figure 3: Ablative visualizations of the DASS method. From the example input image (top left), we show the updated image after one optimization iteration using various ways to obtain image-level gradients or parameter settings: (a) shows that noised background in the input image encourages DASS to hallucinate the missing parts; (b) compares the standard SDS (back-propagate gradients through encoder) and our DASS (decoder-based) losses; (c) justifies our accumulating latent gradient approach as it leads to cleaner decoded output; (d) indicates that small timestep mostly modifies the texture, whereas large timestep changes the geometry more (sometimes removes or creates body parts); (e) demonstrates high-contrast colors and slightly disproportioned body with higher guidance weight (diffusion prior is biased towards larger heads and frontal views). Note that (b) uses the clean input in (a) for better visualization, whereas (c),(d),(e) are obtained from the noised input.
Although this common SDS loss is effective in generating NeRFs from text, we observe that naively applying it in our framework leads to unstable and inefficient training. As shown in Fig. 3 (b), the SDS gradients back-propagated through the encoder are often quite noisy, causing undesirable artifacts on 3D shapes and texture. Moreover, it requires the extra computation and memory usage for gradient back propagation, limiting the training batch size and thus decreasing stability.
To mitigate these issues, we propose a novel Decoder-based Accumulative Score Sampling (DASS) module, an alternative to calculate pixel gradients that are cleaner and more efficient. Fig. 2 illustrates the proposed DASS module. At a high level, given an input image  $x$ , we obtain a denoised image  $x'$  from the decoder as a reconstruction target, based on our observation that decoded outputs are generally less noisy. As shown in Fig. 2, we pass a rendered image through the encoder  $\mathcal{E}$  to obtain low-resolution latent codes, update the latents for  $n$  steps via score distillation, then decode the updated latents with the decoder  $\mathcal{D}$  as an image target. Formally, instead of explicitly calculating the partial derivative  $\partial z / \partial x$ , we use  $x - \mathcal{D}(z - \nabla z)$  as a proxy to  $\nabla x$ , where  $\nabla z$  is the accumulated latent gradients over  $n$  steps. This makes a linear assumption on  $\mathcal{D}$  around latents  $z$ , which we empirically find effective to approximate the pixel gradients. The target image  $x' = \mathcal{D}(z - \nabla z)$  can be directly used as an updated input (Section 3.3) or to compute a pixel-level DASS loss  $\mathcal{L}_{dass} = \| (x - \mathcal{D}(z - \nabla z))\|^2$  in 3D optimization (Section 3.4). Since the DASS module only involves one forward pass of the encoder and decoder, it costs roughly half the memory consumption during training compared to the original SDS loss.
The visualizations in Fig. 3 demonstrate that DASS produces cleaner images than the original SDS loss in one training step (b), and that the accumulated gradients can effectively reduce noise and fill in the missing parts (c). Moreover, we show that adding random noise to the background pixels can facilitate the shape completion by DASS (a). We also perform ablative analyses on other diffusion parameters like noise timestep (d) and guidance weight (e) in Fig. 3. In general, ARTIC3D favors moderate accumulation steps  $n \in (3,10)$  and lower timestep  $t \in (0.2,0.5)$  since higher variance can lead to 3D results that are not faithful to the input images. Also, we use a lower guidance weight  $w_{g} \in (10,30)$  so that our results do not suffer from over saturation effects common in prior works due to high guidance scale in SDS loss. We apply the DASS module in three different stages in our framework: input preprocessing, shape and texture optimization, and animation fine-tuning, which we describe in the following sections.
3.3 Input preprocessing for noisy images
Animal bodies in real-world images often have ambiguous appearance caused by noisy texture, dim lighting, occlusions, or truncation, as shown in Fig. 4. To better deal with noisy or partial observations, we propose a novel method to enhance the image quality and complete the missing parts. Given a sparse image collection  $\{I_j \in \mathbb{R}^{H \times W \times 3}\}$  ( $j \in \{1, \dots, N\}$  and  $N$  is typically between 10-30) of an animal species, we aim to obtain accurate silhouettes estimates  $\{\hat{M}_j \in \mathbb{R}^{H \times W}\}$  and clean semantic features  $\{K_j \in \mathbb{R}^{h \times w \times d}\}$  for each instance. As shown in Fig. 2, we roughly estimate the foreground masks via clustering salient features extracted by a trained DINO-ViT [4] network. Then, we apply DASS to diffuse the background-masked images, resulting in animal bodies with cleaner texture and complete shapes. Formally, we obtain an updated image  $I'$  by  $\mathcal{D}(z - \nabla z)$ , where  $z = \mathcal{E}(I)$ . Here, DASS serves as an image denoising and inpainting module, which can effectively generate a high-quality version of a noisy input via  $n$  latent updates and a single forward pass of  $\mathcal{D}$ . Following the noise-and-denoise nature of diffusion models, we show in Fig. 3 (a) that manually adding Gaussian noise to the background pixels in an input image encourages DASS to hallucinate the occluded parts while mostly preserving the visible regions. Finally, we re-apply DINO-ViT feature extraction and clustering [1] on the diffused images to obtain cleaner and more complete masks as well as semantic features. Fig. 2 (left) shows sample noisy input images and the corresponding output enhanced images and feature clusters. Note that Farm3D [9] uses SD [27] to generate animal images from text for 3D training, which, however, often contain irregular shapes (e.g., horses with 5 legs). On the contrary, our preprocessed images are more suitable for the sparse-image optimization framework since our goal is to reconstruct 3D shape and texture that are realistic and faithful to the input images.
3.4 Diffusion-guided optimization of shape and texture
Given the preprocessed images, silhouette estimates, and semantic features, we jointly optimize the camera viewpoint, pose articulation, 3D part shapes, and texture. Since we do not assume any 2D or 3D annotations, we follow Hi-LASSIE [38] and adopt an analysis-by-synthesis approach to reconstruct 3D shape and texture that are faithful to the input images. That is, we render the 3D part using a differentiable renderer [16] and compare them with the 2D images, pseudo ground-truth silhouettes, and DINO-ViT features. Fig. 2 (top) illustrates the shape and texture optimization.
LASSIE and Hi-LASSIE losses. Given the rendered silhouette  $\tilde{M}^j$  and pseudo ground-truth  $\hat{M}^j$  of instance  $j$ , the silhouette loss  $\mathcal{L}_{sil}$  can be written as:  $\mathcal{L}_{sil} = \sum_{j} \| \tilde{M}^j - \hat{M}^j \|^2$ . LASSIE [39] and Hi-LASSIE [38] further leverage the 2D correspondence of DINO features between images of the same animal class to define a semantic consistency loss  $\mathcal{L}_{sem}$ .  $\mathcal{L}_{sem}$  can be interpreted as the Chamfer distance between 3D surface points and 2D pixels, enforcing the aggregated 3D point features to project closer to the similar pixel features in all images. To regularize the pose articulations and part shapes, [39, 38] also apply a part rotation loss  $\mathcal{L}_{rot}$ , Laplacian mesh regularization  $\mathcal{L}_{lap}$ , and surface normal loss  $\mathcal{L}_{norm}$ . The part rotation loss  $\mathcal{L}_{rot} = \sum_{j} \| R^j - \bar{R} \|^2$  limits the angle offsets from resting pose, where  $R^j$  is the part rotations of instance  $j$  and  $\bar{R}$  denotes the part rotations of shared resting pose.  $\mathcal{L}_{lap}$  and  $\mathcal{L}_{norm}$  encourage smooth 3D surfaces by pulling each vertex towards the center of its neighbors and enforcing neighboring faces to have similar normals, respectively. We omit the details and refer the readers to [39, 38]. Considering that the reconstruction  $(\mathcal{L}_{sil}, \mathcal{L}_{sem})$  and regularization  $(\mathcal{L}_{rot}, \mathcal{L}_{lap}, \mathcal{L}_{norm})$  losses are generic and effective on articulated shapes, we use them in ARTIC3D along with novel texture reconstruction and DASS modules.
Texture reconstruction. Both [39, 38] directly sample texture from input RGB, resulting in unrealistic textures in occluded regions. To obtain more realistic textures, we also optimize a texture image  $T_{i}$  for each part. The vertex colors  $C \in \mathbb{R}^{3 \times m}$  are sampled via the pre-defined UV mapping  $S$  of surface points  $X$ . Formally, the surface color sampling of part  $i$  can be expressed as  $C_{i} = T_{i}(S(X))$ . The sampled surface texture are then symmetrized according to the symmetry plane defined in the 3D skeleton. Note that the texture images are optimized per instance since the animals in web images can have diverse texture. Similar to the  $\mathcal{L}_{lap}$ , we enforce the surface texture to be close to input image when rendered from the estimated input view. The texture reconstruction loss is defined as:
$$
\mathcal {L} _ {\text {t e x t}} = \sum_ {j} \| \hat {M} ^ {j} \odot (\hat {I} ^ {j} - \tilde {I} ^ {j}) \| ^ {2}, \tag {2}
$$